Abstract:<p><pre><code> Scaling up deep learning algorithms has been
shown to lead to increased performance in
benchmark tasks and to enable discovery of
complex high-level features. Recent eff orts
to train extremely large networks (with over
1 billion parameters) have relied on cloud-
like computing infrastructure and thousands
of CPU cores. In this paper, we present tech-
nical details and results from our own system
based on Commodity O ff-The-Shelf High Performance
Computing (COTS HPC) technology:
a cluster of GPU servers with Infi niband
interconnects and MPI. Our system is able to
train 1 billion parameter networks on just 3 machines
in a couple of days, and we show that it can scale
to networks with over 11 billion parameters using
just 16 machines. As this infrastructure is much
more easily marshaled by others, the approach
enables much wider-spread research with extremely
large neural networks.
</code></pre>
For $20,000, they were able to build a 1-billion-connection system comparable to the $1MM system they built the previous year. Also in this paper, Andrew Ng and others detail how for $100,000 they also created an 11-billion-connection deep learning system with 16 commodity servers, each loaded with 4 Nvidia GTX680 GPU cards.
Weird that a team led by Andrew Ng couldn't do better with the 11 billion parameters model than with the 1 billion one.<p>This is becoming accessible for everyone. It's both exciting and terrifying. It has the potential to save humanity from itself or condemn us to totalitarianism. I am sure machine learning and NLP and statistical models are what enables them to analyze the data they collect on us.<p>Big, fast noSQL tables, clustering technology (map-reduce) and machine learning are what allows these guys to do what they do. Our most prized toys became our enemies.