Neural Networks: Fundamentals and Applications

A practical introduction to neural networks with hands-on experience.

Delivery: Delivered from 13th June 2017 for 10 weeks. (2-hour Lecture and 1-hour hands-on tutorial per week).

Coordinator and Instructor: Dr. Rohitash Chandra (Research Fellow @CTDS UniSyd). Research interests in machine learning and neural networks.

Infor: https://www.researchgate.net/profile/Rohitash_Chandra

https://sydney.edu.au/science/people/rohitash.chandra.php

Tutor: Mr. Rafael Possas, Research Engineer, Sydney Informatics Hub (2017)

Guest Speakers: TBA

Duration: This will be a 10 weeks course

Programming Language: The course will use Python as the main language. C++ will be used as an additional programming language.

Assignments: Assignments can be done in any language of choice by the participant.

Location: Rm 538, Centre for Translational Data Science (CDTS), 5th Floor, School of Information TechnologiesBuildingg (J12), The University of Sydney, 1 Cleveland Street, NSW 2006.

Contact: rohitash.chandra (at) sydney.edu.au

Disclaimer: The course is not accredited towards any of the programmes at the University of Sydney. There will be no certificates issued upon completion of the course.

Feedforward Neural Networks
Backpropagation and Stochastic Gradient Descent (SGD)
Bayesian Neural Networks (with MCMC and SGD)
Recurrent Neural Networks
Long Short-Term Memory (LSTM)
Neuroevolution (Coevolution and Direct Neuro-evolution)
Deep Learning with Convolutional Neural Networks
Multi-task learning and Transfer Learning
Ensemble learning and Hybrid Methods
Modular Neural Networks

Applications: Pattern Classification, Time Series Prediction, and Computer Vision

Implement simple neural network architectures from scratch (without relying on machine learning libraries)
Develop rich applications using neural networks that involve real world problems
Become ready to work and contribute to challenging problems that arise in training and representation of knowledge in different neural network architectures.

Key Resource – Code

https://github.com/rohitash-chandra/NeuralNetworksCourse_CTDS

Prerequisites

If you are not familiar with Python, please take some time to get to know the basics. We will also use iPython notebook and

https://wiki.python.org/moin/SimplePrograms

https://www.programiz.com/python-programming/examples

As long as you have knowledge of some basic programming, you will be fine.

https://deeplearning4j.org/deeplearningforbeginners.html

https://www.quora.com/What-are-the-prerequisites-to-learn-neural-networks

WEEK 1: Intro

Introduction to neural network fundamentals to the audience with no background in machine learning or data science.

Lecture Notes:

Textbook: Machine Learning, Tom Mitchell, McGraw Hill, 1997.

Neural Networks Chapter 4: http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/mlbook/ch4.pdf

Chapter 2: Concept Learning: http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/mlbook/ch2.pdf

Other References:

Overview of Artificial Neural Networks https://en.wikipedia.org/wiki/Artificial_neural_network

Backpropagation Algorithm https://en.wikipedia.org/wiki/Backpropagation

Tutorial for Python: http://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/

Code: https://github.com/rohitash-chandra/NeuralNetworksCourse_CTDS/tree/master/week1

Neural Networks by Prof. Hinton

Videos

Assignment 1

Choose either Option I or Option II, or tackle both.

Group: 1 – 4 members.

Deadline: Begining of Week 3 of the course.

Option I: Fundamentals

Extend the basic FNN python code (fnn_v1.py, fnn_v2.py or fnn.v3.py) to include an additional hidden layer and compare the performance with original FNN with a single hidden layer. Carry out 30 experimental runs and report the mean and std or training and test datasets.
Extend your further so that it can be generalized for any number of hidden layers. Run experiences to see the effect from 1 – 5 hidden layers.
Experiment on different combinations of hidden layers and neurons for each hidden layer.
Select 5 other problems from UCI Machine Learning Repository and apply the method.
Apply the strategy used in (2 and 3) for chaotic time series problems that include Sunspot, Lazer, Mackey-Glass and Lorenz problems (Discussed in Week 2).

Option II: Application

Extend the application FNN code in Keras for additional number of hidden layers as given in Option I
Apply FNN in Keras for chaotic time series problems (Discussed in week 2)

Deliverable

Present your results in a Technical Report and discuss your major findings for the different problems ( 10 min presentation per group will be organized).
Submission: email pdf of the technical report to rohitash.chandra@sydney.edu.au
Deadline: 26th June 2017.
Best code will become part of the solution set for the course.

WEEK 2: NNs – Cont.

Notes

Neural Networks Chapter 4: http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/mlbook/ch4.pdf

Background on Differentiation and Chain Rule (Gradient Descent):

Basics Chain Rule: https://www.math.ucdavis.edu/~kouba/CalcOneDIRECTORY/chainruledirectory/ChainRule.html
Basics Chain Rule: http://www.sosmath.com/calculus/diff/der04/der04.html
Sigmoid differentiation: https://math.stackexchange.com/questions/78575/derivative-of-sigmoid-function-sigma-x-frac11e-x

Why Bias?

Videos

https://www.coursera.org/lecture/machine-learning/backpropagation-algorithm-1z9WW

https://www.coursera.org/lecture/machine-learning/backpropagation-intuition-du981

WEEK 3: Bayesian Neural networks

Code:

https://github.com/rohitash-chandra/NeuralNetworksCourse_CTDS/tree/master/week3

Notes:

MCMC basics

Bayesian Neural Networks

Lecture notes by Prof. Zoubin Ghahramani
Lecture notes by Prof. Sargur Srihari

Bayesian Neural Network Libraries:

Videos

Assignment 2

Option 1: Fundamentals

Extend FNN Python code and into a Recurrent Neural Network. You can use a reference to C++ Elman RNN code discussed in class. Implement BPTT for Python RNN.
Use time series prediction problems and Tomita Grammer (Chandra, 2011 Neurocomputing paper) problem to test your RNN
Bayesian Recurrent Neural Networks: Use MCMC for training RNNs

Option 2: Applications

Implement a simple (Elman) RNN from any existing libraries (Keras preferred) for time series prediction.
Use Keras implementation for RNN for any other problem of your choice (Speech, signature, or NLP)

References

WEEK 4 RNNs

Notes

Recurrent Neural Networks: Scholarpedia
RNNs (wiki)
RNN tutorial at WildML
Tutorial on Backpropagation Through Time – Towards LSTM
RNN on TensorFLow Tutorial

Resources

Code: https://github.com/rohitash-chandra/NeuralNetworksCourse_CTDS/tree/master/week4

Videos

RNN for Binary Addition by Hinton

https://www.youtube.com/watch?v=z61VFeALk3o

https://www.youtube.com/watch?v=Pp4oKq4kCYs

Keras RNNs https://www.youtube.com/watch?time_continue=2&v=4rG8IsKdC3U

Assignment 3

Option 1: Fundamentals

Extend FNN Python code and into a Recurrent Neural Network. You can use a reference to C++ Elman RNN code discussed in class. Implement BPTT for Python RNN.
Use time series prediction problems and Tomita Grammer (Chandra, 2011 Neurocomputing paper) problem to test your RNN
Bayesian Recurrent Neural Networks: Use MCMC for training RNNs

Option 2: Applications

Implement a simple (Elman) RNN from any existing libraries (Keras preferred) for time series prediction.
Use Keras implementation for RNN for any other problem of your choice (Speech, signature, or NLP)

References

WEEK 5: LSTMs

Notes

Original LSTM presentation, code, and examples: http://people.idsia.ch/~juergen/lstm/

Tutorials

Lecture notes

Code and Examples

Original LSTM code and tutorial: http://people.idsia.ch/~juergen/lstm/
Code from GRU tutorial: https://github.com/dennybritz/rnn-tutorial-gru-lstm/blob/master/gru_theano.py
Sentiment analysis example: http://deeplearning.net/tutorial/lstm.html
Video tutorial with code: https://pythonprogramming.net/recurrent-neural-network-rnn-lstm-machine-learning-tutorial/

Videos

Basics: https://www.youtube.com/watch?time_continue=1145&v=Ukgii7Yd_cU

LSTM by Hinton https://www.youtube.com/watch?v=lsV5rFbs-K0

LSTM for Time Series https://www.youtube.com/watch?v=2np77NOdnwk

RNN for Language Modelling: https://www.youtube.com/watch?v=Keqep_PKrY8

Week 6: CNNs

Notes

Lecture notes

Videos

Assignment 4

Use Keras for LSTM implementation for any selected pattern recognition, time series or classification problems that involve long term-dependencies. Language models could be also considered.
Use Keras for CNN implementation for any selected datasets that involve, face, object and gesture recognition.
Highlight an application when it’s necessary to have a combination of LSTMs and CNN’s and discuss implementation issues.

Week 7: Neuro-evolution

Notes

Tutorials

Videos

Week 8: Ensemble learning

Notes

Videos

https://www.youtube.com/watch?v=ix6IvwbVpw0

Week 9: MTL and Transfer Learning

Notes

Videos

http://videolectures.net/roks2013_pontil_learning/

Week 10: Modular NNs

Notes

Stony Brooks Uni on Modular Networks

http://www.evolvingai.org/modular

Videos

Extra : Deep Belief Nets

Research Project

Choose a topic and develop a proposal
Week 9: present research proposal
Week 10 – week 14: work on Project
Week 15: Demonstration of Results and Software
Week 16 – 18: Work on research paper

Option1: Neurocomputing methodologies

Langevin Dynamics Bayesian Recurrent Neural Networks for Multi-Step-Ahead Time Series Prediction (ref: https://arxiv.org/pdf/1704.02798.pdf . http://www.sciencedirect.com/science/article/pii/S0925231217303892)
Multi-Task Ensemble Stacking of Bayesian Neural Networks for Cyclone Track and Intensity Prediction
Multi-Task Stacking of CNNs for Unconstrained Mobile Face Recognition (ref: http://www.sciencedirect.com/science/article/pii/S1568494616306603)
Multi-task modular neural networks for “missing values” in classification problems
Approximate Bayesian computation framework for neuro-evolution

Option 2: Neurocomputing applications

Langevin dynamics via Bayesian Neural Networks for Cyclone Track Prediction (ref: http://ieeexplore.ieee.org/document/7727839/)
Bayesian multi-task learning for cyclone track and path prediction

Option 3: Others

Any idea or application of your choice

Additional Resources

Additional research papers: https://www.dropbox.com/sh/4ol9jtbox30r4tp/AAD1tiQdxniKVTQ4DaWSq1DPa?dl=0

furthermore: https://www.dropbox.com/sh/72cktrpsnug3736/AAC0mTvic8qyjWE12InUL56Fa?dl=0

Other related courses:

CS7792 Counterfactual Machine Learning , T. Joachims, Cornell University http://www.cs.cornell.edu/courses/cs7792/2016fa/

CS 294 Deep Reinforcement Learning, Spring 2017 http://rll.berkeley.edu/deeprlcourse/

CS287 Fall 2015 https://people.eecs.berkeley.edu/~pabbeel/cs287-fa15/

CS 323 – Automated Reasoning https://cs.stanford.edu/~ermon/cs323/index.html

CS 159: Advanced Topics in Machine Learning: Structured Prediction https://taehwanptl.github.io/

CSE 599 D1: Advanced Natural Language Processing http://courses.cs.washington.edu/courses/cse599d1/16sp/syllabus.html

SURVEYS: https://github.com/metrofun/machine-learning-surveys

Neural Networks: Fundamentals and Applications