OpenAI Scholars 2019 - The Syllabus
This week I started a 3 month program as a scholar at OpenAI - an amazing opportunity to study Deep Learning for 2 months & to work on a project for another month. I am focusing my efforts on getting to grips with Generative Models & the Visualization of deep networks. Til now, I have focused on applying DL models - gaining ‘just enough’ to get them working well for particular tasks, particularly in AR & VR.
I was lucky enough to learn (IRL) at the Fast.ai Practical Deep Learning for Coders session last year and it made me fall in love with DL in a whole new way. If you don’t know Fast.ai & the work of Jeremy Howard & Rachel Thomas then you are missing out on an “awesome renegade group of DL researchers” (in the words of MIT’s Lex Fridman … ;-)) In all seriousness, it was an amazing course with a focus on applying DL techniques from the get-go rather than building up slowly from a theoretical foundation. (An approach which is in keeping with that of David Perkins.) In essence, if we taught baseball the way we teach maths, then young kids would be learning spatial geometry & physics before being allowed onto the pitch with a ball.) In that same way, Fast.ai starts out applying & working with DL applications and then starts to fill in the details and foundational theory from there.
That is all to say that with this program & time at OpenAI, I now want to take the opportunity to step back & really ground myself in the fundamentals, as well as the application, of DL networks. For reference, I am including the outline of the syllabus I will be working on below - the full overview in the image & links to some of the readings & papers below that.
Stanford’s famous CS231n & CS236 will form a key part of my curriculum, as well as parts of CS228. In addition, I will be reading from core texts (Deep Learning Book for e.g.,) & relevant research papers - an initial reading list is included below.
Week 1: Set up for success…..
· Review Part 1 of Deep Learning Book (applied Math & Machine Learning Basics)
Msft Research Paper discussions – use as an intro to how to approach reading ML research papers:
Begin Research into Visualization
· Lessons from a year of distilling ML Research video
· Machine Learning for Visualization video
Week 2: Review & Cement the Key Foundations of DL
· Deep Learning Ch6. Deep FeedForward Networks
· Begin investigating visualization in DL with The Building Blocks of Interpretability
· Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps paper
· Investigate some of the ways batch size, learning rate, batch norm et al impact results
– Disciplined Approach to NN Hyperparameters Paper;
– Train longer, generalize better Paper;
– Don’t decay the Learning Rate, Increase the Batch Size Paper;
– Rethinking ImageNet PreTraining paper;
– How does Batch Normalization help Optimization? Paper
Week 3: Training Neural Nets w focus on CNN
• ImageNet Classification with Deep Convolutional Neural Networks Paper
• Improvement – Deep Residual Learning for Image Recognition Paper
• Multi-scale context Aggregation by Dilated Convolutions Paper (used in WaveNet – model referred to later in GAN section)
• Densely Connected Convolutional Networks Paper
Week 4: Sequence Models & Introduction to Generative Models
• Char-RNN blog post
• RNN Regularization paper
• DeepSpeech paper
• Detection – YOLO9000
Week 5: Deeper Dive – Generative Models
Read Visualization Papers & investigate Lucid Codebase
Use Readings & Resources below for Wks5-8 as most relevant:
Background & Generative Models overall:
· Chapter 10 of book Directed Graphical Models (Bayes nets) in Machine Learning: A Probablistic Perspective by Kevin Murphy
· Additional Tutorials if needed from various listed below:
- Tutorial on Deep Generative Models. Aditya Grover and Stefano Ermon. International Joint Conference on Artificial Intelligence, July 2018.
- Tutorial on Generative Adversarial Networks. Computer Vision and Pattern Recognition, June 2018.
- Tutorial on Deep Generative Models. Shakir Mohamed and Danilo Rezende. Uncertainty in Artificial Intelligence, July 2017.
- Tutorial on Generative Adversarial Networks. Ian Goodfellow. Neural Information Processing Systems, December 2016.
- Learning deep generative models. Ruslan Salakhutdinov. Annual Review of Statistics and Its Application, April 2015.
Normalizing Flow Models
· Tutorial by Eric Jang as needed
· Original normalising flow paper
· Auto_Encoding Variational Bayes (Original VAE paper)
· VAE tutorial paper
· Semi Supervised Learning Using Deep Generative Models paper
· Improve your estimate of the lower bound IWAE paper
· Variational Lossy Autoencoder paper
GANs & Variants
· Tutorial by Ian Goodfellow - Video
· Horse/Zebra in unpaired ImagetoImage Translation paper on CycleGAN
· Super image resolution with GAN paper