What parts of Deep Learning are modern?
Conclusion: outside of a very brief period in which pre-training with Unsupervised Learning was shown to be helpful, Deep Learning has largely been about hardware brute force, and learning how to use brute force to solve problems.
Terms I need to learn more about
-
Pattern Deformations
-
Hessian-free learning
-
Batch Normalisation (Thanks A Breitman)
- Competing Units
- Shallow NN-like models with few such stages have been around for many decades if not centuries, and models with several successive nonlinear layers of neurons date back at least to the 1960s and 1970s
- Training of deeper architectures (ANNs with more layers, including RNNs) only became feasible through the use of unsupervised learning techniques in the 2000s
- Training of deeper architectures without unsupervised learning became possible later
- More data
- More computation
- Better techniques
More data
I think this is actually conflated with More computation below. Without fast computation, you can't get through enough data.More computation
Page 23 of the reference - in 2010, a new MNIST record was set using backpropagation and pattern deformations, both of which are decades old.these results seemed to suggest that advances in exploiting modern computing hardware were more important than advances in algorithms