Nuit Blanche: Predicting the Future: The Steamrollers and Machine Learning.

Sunday, September 11, 2016

Predicting the Future: The Steamrollers and Machine Learning.

Four years ago, I wrote that when one wants to predict the future, it is always a good thing to rely on the steamrollers, i.e. the exponential trends on which we can surf to predict the future because they are highly efficient. Let see how this trend continues.

Since 2012, genome sequencing reached a second inflection point but I still expect a third inflection point due to the use of the nanopore sequencing technology as we had mentioned then. So we are good.

For Moore's law it is a little more complicated. While the number of transistors do increase (but probably not as fast a Moore's law anymore. In this most recent graph, the last five year show a slight inflection point) the performance is beginning to stall. Dennard scaling stopped around 2005

From HPC no more free lunch

and while the number of transistors increased, we are beginning to see a decrease in the use of that silicon. This last trend has a name: Dark Silicon. Bill Dally, whom we had invited at the now-not-gonna-happen NIPS workshop said the following three years ago:

“The number of transistors is still going up in a very healthy way,” Dally contends, “We’re still getting the Moore’s law increase in the number of devices, but without Dennard scaling to make them useful, all the things we care about – like clock frequency of our parts, single thread performance and even the number of cores we can put in our parts – is flattening out, leading to the end of scaling as we know it. We’re no longer getting this free lunch of technology giving us faster computers.”

This is a problem that needs to be solved because our 21st century economy hinges on continued computational progress. With the end of Dennard scaling, all computing power becomes limited, and performance going forward is determined by energy efficiency. As process technology can no longer be counted upon for exponential advances, the focus must shift to architecture and circuits, according to Dally.

Since Machine Learning has become the main driver in applications, it has become a target of interest for the silicon industry. In a way, one of the lessons of that story is for architecture of Machine Learning algorithms to be a better fit for silicon architectures. No wonder we recently saw the rise of Google's TPU, the numerous NVIDIA platforms like the DGX 1, the recent acquisition of silicon-machine learning companies like Nervana and Movidius, the effort of Baidu in the GPU clusters realm...

Let me finish this short blog post by pointing out that while silicon will become more efficient because Machine Learning algorithms are used as a constraint, Making sense of the world in the future will still require new technologies. From the recently released report produced by the Semiconductor Industry Association and the Semiconductor Research Corporation entitled "Rebooting the IT Revolution: A Call to Action", one can read that given the data explosion we are predicting, computing in 2040 may become problematic. From Appendix A Data Explosion Facts , A4. Total Effective Capacity to Compute Information in 2040:

For this benchmark energy per bit, computing will not be sustainable by 2040, when the energy required for computing will exceed the estimated world’s energy production. Thus, radical improvement in the energy efficiency of computing is needed.

And according to the authors of "Moore's law: the future of Si microelectronics"

The time frame to implement a radically new device is estimated to be ∼30 years.

We need to find less than radically new devices and better algorithms to make sense of this upcoming Miller's wave in 2025. It's time to get cranking.