I got the chance to read this paper on Distributional Bellman published by DeepMind in July. Glossing over it the first time, my impression was that it would be an important paper, since the theory was sound and the experimental results were promising. However, it did not generate as much noise in the reinforcement learning community as I would have hoped. Nevertheless, as I thought the idea of Distributional Bellman was pretty neat, I decided to implement it (in Keras) and test it out myself. I hope this article can help interested readers better understanding the core concepts of Distributional Bellman.

After a brief stint with several interesting computer vision projects, include this and this, I’ve recently decided to take a break from computer vision and explore reinforcement learning, another exciting field. Similar to computer vision, the field of reinforcement learning has experienced several important breakthroughs made possible by the deep learning revolution. AlphaGo, a GO playing AI trained by reinforcement learning, became the first computer program to surpass human in the game of GO. If you think this is not impressive enough, in July this year, a team from OpenAI used reinforcement learning to train an AI that beats the world champion human player in DOTA 2. All of these were not possible if not for the recent advancement of reinforcement learning and deep learning.

I recently took part in the Nature Conservancy Fisheries Monitoring Competition organized by Kaggle. Of all the image related competitions I took part before, this is by far the toughest but most interesting competition in many regards. I managed to finish in the 3rd place in public leaderboard but dropped to 8th place in the private leaderboard. I will go over my solution and lessons learned in detail and hopefully they will be useful to someone who come across with similar problems.

Motivation

Those who have applied deep learning would know, being deep is both a curse and blessing. On the one hand, deep models are endowed with higher representation power; on the other hand, the increase in parameters means more data are necessary to train up the models so that they would generalize well to unseen data, to avoid a phenomena known as over-fitting. Unfortunately, in real world situation, it’s hard to collect enough high quality labeled data, usually in the order of magnitude of tens of thousands, to train our model with. Very often, we resort to a method called transfer learning, that is, to load up the model with pre-trained weights and continue running gradient descent on our own dataset. In practice, transfer learning, or so called fine-tuning, allows us to quickly train a decent model with very few training samples.

I recently took part in the Kaggle State Farm Distracted Driver Competition. This is an image recognition problem which deep learning is particular good at solving. Hence by taking part in this competition I got the perfect opportunity to acquaint myself with the deep learning modeling pipeline, which typically entails data preprocessing, data augmentation, train/fine-tune Convolutional Neural Networks, and ensemble multiple models to obtain the final output.