Large Language Models and Transformers (Videos, Simons Institute for the Theory of Computing)

As some of you may know, LightOn has built a few Large Language Models, and we are now making them usable to Enterprise customers. In the meantime and on the theoretical side of things, the Simons Institute for the Theory of Computing has organized a workshop on the topic of Large Language Models and Transformers. The program is listed below, every link links to the video of the talk (that includes streaming this week).

Monday, Aug. 14, 2023

• 9:15 – 10:15 a.m. Sparks of Artificial General Intelligence, Yin Tat Lee (Microsoft Research)
• 11 a.m. – 12 p.m. Possible Impossibilities and Impossible Possibilities, Yejin Choi (University of Washington)
• 1:30 – 2:30 p.m. Towards Reliable Use of Large Language Models: Better Detection, Consistency, and Instruction-Tuning, Christopher D. Manning (Stanford University)
• 3 – 4 p.m. An observation on Generalization, Ilya Sutskever (OpenAI)
• 4 – 4:45 p.m. Panel Discussion (moderated by Alexei Efros)

Tuesday, Aug. 15, 2023

• 9 – 10 a.m. Understanding the Origins and Taxonomy of Neural Scaling Laws, Yasaman Bahri (Google DeepMind)
• 10 – 11 a.m. Scaling Data-Constrained Language Models, Sasha Rush (Cornell University & Hugging Face)
• 11:30 a.m. – 12:30 p.m. A Theory for Emergence of Complex Skills in Language Models, Sanjeev Arora (Princeton University)
• 2 – 3 p.m. Interpretability via Symbolic Distillation, Miles Cranmer (Flatiron Institute)
• 3:30 – 4:30 p.m. Build an Ecosystem, Not a Monolith, Colin Raffel (University of North Carolina & Hugging Face)
• 4:30 – 5:30 p.m. How to Use Self-Play for Language Models to Improve at Solving Programming Puzzles, Adam Tauman Kalai (Microsoft)

Wednesday, Aug. 16, 2023

• 9 – 10 a.m. Large Language Models Meet Copyright Law, Pamela Samuelson (UC Berkeley)
• 10 – 10:45 a.m. Panel Discussion (moderated by Shafi Goldwasser)
• 11:15 a.m. – 12:15 p.m. On Localization in Language Models, Yonatan Belinkov (Technion - Israel Institute of Technology)
• 2 – 3 p.m. Language Models as Statisticians, and as Adapted Organisms, Jacob Steinhardt (UC Berkeley)
• 3:30 – 4:30 p.m. Are Aligned Language Models “Adversarially Aligned”?, Nicholas Carlini (Google DeepMind)
• 4:30 – 5:30 p.m. Formalizing Explanations of Neural Network Behaviors, Paul Christiano (Alignment Research Center)

Thursday, Aug. 17, 2023

• 9 – 10 a.m. Meaning in the age of large language models, Steven Piantadosi (UC Berkeley)
• 10 – 11 a.m. Word Models to World Models, Josh Tenenbaum (MIT)
• 11:30 a.m. – 12:30 p.m. Beyond Language: Scaling up Robot Ontogeny, Jitendra Malik (UC Berkeley)
• 2 – 3 p.m. Are LLMs the Beginning or End of NLP?, Dan Klein (UC Berkeley)
• 3:30 – 4:30 p.m. Human-AI Interaction in the Age of Large Language Models, Diyi Yang (Stanford University)
• 4:30 – 5:30 p.m. Watermarking of Large Language Models, Scott Aaronson (UT Austin & OpenAI)

Friday, Aug. 18, 2023

• 9 – 10 a.m. In-Context Learning: A Case Study of Simple Function Classes, Gregory Valiant (Stanford University)
• 10 – 11 a.m. Pretraining Task Diversity and the Emergence of Non-Bayesian In-Context Learning for Regression, Surya Ganguli (Stanford University)
• 11:30 a.m. – 12:30 p.m. A data-centric view on reliable generalization: From ImageNet to LAION-5B, Ludwig Schmidt (University of Washington)
• 2 – 3:30 p.m. Short Talks
• 4 – 5 p.m. Short Talks

** Nuit Blanche is now on Twitter: @NuitBlog **

 
Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

2021, the year AI ate HPC … and more

Back in 2011, Marc Andreesen announced that Software was eating the world while everyone was trying to make sense of the realities of the cloud versus brick and mortar businesses. Eight years later, Tarry Singh articulated how AI was eating software; a year before GPT-3 and Codex would give solid ground to this prediction. Fast forward two years later, we just witnessed how AI ate HPC and we believe those are the first steps towards how AI is eating Learning, Creative and Office work.

Let me explain.

At LightOn, we have been working on getting AI to be transformative for everyone. For that to happen, we used the Jean Zay French national supercomputer for two different yet somehow related reasons this past year. First, our LightOn’s Optical Processing Unit hardware was integrated into this top105 supercomputer. Even though LightOn’s hardware is analog and uses a technology currently unknown to supercomputing, there are several good reasons the future of computing will use this technology. Relatedly, in a co-design fashion, we also used the Jean Zay facility to implement and run code for the building of Large Language/Foundation Models that we believe are key to Transformative AI. In March, we trained the largest French language model ever called Auriga and made it available to everyone through our PAGnol demo.

In July, we launched the Muse API, making our language models available for business use. Initially released in private beta, Muse has quickly gained its first customers, and a public commercial version with five languages is to be released in early 2022. Some of these early customers are using this new AI to redefine SEO or the experience for website creation.

“True happiness comes from the joy of deeds well done, the zest of creating things new” Antoine de Saint-Exupéry

Eventually, a major impact of these Large Language Models trained on HPC infrastructures will be the ability for everyone to personally learn faster and for office workers worldwide to get the job done in a fashion never seen before.

If you are a start-up company or an individual starting a business around this promise, don’t hesitate to join the Muse Partnership program, and let’s start a discussion around how Muse can help you.

These models will also have the same effect in creative work and in the discovery process.

Stay tuned, the true AI revolution is really coming!

 
Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

LightOn Photonic coprocessor integrated into European AI Supercomputer

** Nuit Blanche is now on Twitter: @NuitBlog **

This is history of computing in the making stuff!

Four years ago to the day, LightOn’s first Optical Processing Unit (OPU) had its first light in a Data Center showing that our technology was data center ready.

It is with immense pride and pleasure to announce that LightOn’s OPU has been installed in one of the world’s Top500 supercomputer as part of a pilot program with GENCI and IDRIS/CNRS.

The team at LightOn is immensely proud to write the future of computing in this world-first integration of a computing photonic device into an HPC infrastructure.

The press release can be found here.

Thank you GENCI and IDRIS/CNRS for making this happen!

 
Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

The Akronomicon: an Extreme-Scale Leaderboard

** Nuit Blanche is now on Twitter: @NuitBlog **

As larger models seem to be providing more context and more ability for zero-shot learning, Julien just created the Akronomicon: an Extreme-Scale Leaderboard featuring the world's largest Machine Learning Models. And yes, LightOn is on that board for the moment!

 Want to contribute? https://github.com/lightonai/akronomicon 

 
Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Virtual Workshop: Conceptual Understanding of Deep Learning (May 17th 9am-4pm PST)

** Nuit Blanche is now on Twitter: @NuitBlog **

Just got an email from Rina Panigrahy

Hi Igor,

I am an algorithms researcher at Google (http://theory.stanford.edu/~rinap) and I am organizing this workshop on "Conceptual Understanding of Deep Learning" (details below). It's trying to understand the Brain/Mind as an algorithm from a mathematical/theoretical perspective. I believe that a mathematical/algorithmic approach for understanding the Mind is crucial and very much missing. I'd appreciate any help I can get with advertising this on your blog/mailing-lists/twitter.

Best,

Rina

Here is the invite:

Please join us for a virtual Google workshop on “Conceptual Understanding of Deep Learning”

When: May 17th 9am-4pm PST.

Where: Live over Youtube,

Goal: How does the Brain/Mind (perhaps even an artificial one) work at an algorithmic level? While deep learning has produced tremendous technological strides in recent decades, there is an unsettling feeling of a lack of “conceptual” understanding of why it works and to what extent it will work in the current form. The goal of the workshop is to bring together theorists and practitioners to develop an understanding of the right algorithmic view of deep learning, characterizing the class of functions that can be learned, coming up with the right learning architecture that may (provably) learn multiple functions, concepts and remember them over time as humans do, theoretical understanding of language, logic, RL, meta learning and lifelong learning.

The speakers and panelists include Turing award winners Geoffrey Hinton, Leslie Valiant, and Godel Prize winner Christos Papadimitriou (full-details).

Panel Discussion: There will also be a panel discussion on the fundamental question of “Is there a mathematical model for the Mind?”. We will explore basic questions such as “Is there a provable algorithm that captures the essential capabilities of the mind?”, “How do we remember complex phenomena?”, “How is a knowledge graph created automatically?”, “How do we learn new concepts, function and action hierarchies over time?” and “Why do human decisions seem so interpretable?”

Twitter: #ConceptualDLWorkshop.

Please help advertise on mailing-lists/blog-posts and Retweet.

Hope to see you there!

Rina Panigrahy

 
Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Randomized Algorithms for Scientific Computing (RASC)

** Nuit Blanche is now on Twitter: @NuitBlog **

At LightOn, we build photonic hardware that performs random projections and it is nice to find a source of materials on the subject in one document. Here is a report comprehensively presenting how randomized algorithms are key to the future of computing:

Randomized Algorithms for Scientific Computing (RASC) by Aydin Buluc, Tamara G. Kolda, Stefan M. Wild, Mihai Anitescu, Anthony DeGennaro, John Jakeman, Chandrika Kamath, Ramakrishnan (Ramki)Kannan, Miles E. Lopes, Per-Gunnar Martinsson, Kary Myers, Jelani Nelson, Juan M. Restrepo, C. Seshadhri, Draguna Vrabie, Brendt Wohlberg, Stephen J. Wright, Chao Yang, Peter Zwart

Randomized algorithms have propelled advances in artificial intelligence and represent a foundational research area in advancing AI for Science. Future advancements in DOE Office of Science priority areas such as climate science, astrophysics, fusion, advanced materials, combustion, and quantum computing all require randomized algorithms for surmounting challenges of complexity, robustness, and scalability. This report summarizes the outcomes of that workshop, "Randomized Algorithms for Scientific Computing (RASC)," held virtually across four days in December 2020 and January 2021.

 
Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

The $1,000 GPT-3

** Nuit Blanche is now on Twitter: @NuitBlog **

Progress usually comes from a steady technology bootstrap…until it doesn’t.

Take for instance the race for the $1,000 genome that started in the early 2000s. Initially, sequencing the human genome meant a race between the well-funded public and private sectors but more importantly, the resources for the first breakthrough ended up costing upwards of $450M. Yet despite all the economic promise of genome sequencing, had Moore’s law been applied, sequencing one full genome would still cost $100,000 today. However, once the goal became clearer to everyone, a diversity of technologies and challengers emerged. This intense competition eventually yielded a growth faster than Moore’s Law. The main takeaway is that one cannot rely on the steady progress of one specific technology alone to commoditize tools.

Figure from NIH “Facts sheets about genomics: The cost of Sequencing a Human Genome”, Dec 7th, 2020.

What does this have to do with the current state of silicon computing and the new demand for Large Language Models (LLMs)? Everything if you ask us and here is how.

Less than a year into existence, Large Language Models like GPT-3 have already spawned a new generation of startups built on the ability of the model to respond to requests for which it was not trained. More importantly for us, hardware manufacturers are positing that one or several customers will be willing to put a billion dollars on the table to train an even larger model in the coming years.

Interestingly, much like the mass industrialization in the 1930s, the good folks at OpenAI are sketching new scaling laws for the industrialization of these larger models.

The sad truth is that extrapolating their findings to the training of a 10 Trillion parameters model involves a supercomputer running continuously for two decades. The minimum capital expenditure of this adventure is estimated in the realm of several hundreds of million dollars.

Much like what happened in sequencing, while silicon improvement and architecture may achieve speedups in the following years, it is fair to say that, even with Moore’s law, no foreseeable technology can reasonably train a fully scaled-up GPT-4 and grab the economic value associated with it.

Rebooting silicon with a different physics, light, and NvNs

For a real breakthrough to occur, much like what happened in the sequencing story, different technologies need to be jointly optimized. In our case, this means performing co-design with new hardware and physics but also going rogue on full programmability.

LightOn’s photonic hardware can produce massively parallel matrix-vector multiplications with an equivalent of 2 trillion parameters “for free”: this is about one-fifth of the number of parameters needed for GPT-4. Next comes revisiting the programmability. Current LightOn’s technology keeps these weights fixed by design. Co-design means finding the algorithms for which CPUs and GPUs can perform some of the most intelligent computations and how LightOn’s massive Non-von Neumann (NvN) hardware can do the heavy lifting. We already published how we are replacing backpropagation, the workhorse of Deep Learning, with an algorithm that unleashes the full potential of our hardware in distributed training. We are also working similarly on an inference step that will take full advantage of the massive number of parameters at our disposal. This involved effort relies in a heavy part thanks to our access to ½ million GPU hours on some of France’s and Europe’s largest supercomputers.

And this is just the beginning. There is a vast untapped potential for repurposing large swaths of optical technologies directed primarily for entertainment and telecommunication into computing.

The road towards a $1,000 GPT-3

Based on the GPT-3 training cost estimates, achieving a $1,000 GPT-3 requires four orders of magnitude improvements. Much like what occurred in 2007 with the genome sequencing revolution, Moore’s law may take care of the first two orders of magnitude in the coming decade but the next two rely on an outburst of new efficient technologies — hardware and algorithms. It just so happens that GPT-3 has close to 100 layers, so achieving two orders of magnitude savings may arise faster than you can imagine. Stay tuned!

Igor Carron is the CEO and co-founder at LightOn

 
Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Computing with Light: How LightOn intends to unlock Transformative AI

I gave a talk at #mathia2021 conference on March 9th, 2021 where I drew a parallel between the scaling laws that enabled industrialization in the 1920's and the new scaling laws in AI of the 2020's. AI is at its infancy and it needs to have guiding principles (as embedded in these empirical laws) and it also needs to develop new hardware. I showed how, in this context, LightOn can help unlock Transformative AI. Enjoy!

All these other presentations by Yann LeCun, Kathryn Hess, Michael Jordan, Emmanuel Candès and others can be found in this collection of videos on Vimeo. Let me note that Michael made a similar argument as mine where we think of current stage of AI at its infancy in terms of industrialization. 

 

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:

Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog

About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Unveiling LightOn Appliance

Today is a big day at LightOn as we unveil a hardware product, the Appliance, the world's first commercially available photonic co-processor for AI and HPC

If interested pre-ordering information is here: http://lighton.ai/lighton-appliance 

We have had a few of these optical processing units in our own LightOn Cloud for the past two years and just retired one after more than 800 days working full time.  

Here is the press release: 

The future is now! 

Leasing starts at 1900€/month or about US$2250/month 

 

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:

Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog

About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Video: LightOn unlocks Transformative AI

In the coming days, we'll be making another announcement but I wanted to first share a video we did recently. At LightOn, we don't build photonic computing hardware because it's fancy or cool (even though, it is cool) but because computing hardware is hitting the limits. I know what some say about Moore's law not being dead but the recent focus on Transformers and their attendant scaling laws makes it obvious that in order for more people to have access to these models, we need a new computing paradigm. Indeed not everyone can afford to spend a billion dollars in training these models. As Azeem was recently pointing out in one of his newsletters, this is how bad things will become:

The amazing thing is that we can start to compare the cost of training single AI models with the cost of building the physical fabs that make chips. TSMC’s state-of-the-art 3nm fab will run to around $20bn when it is completed in two years. A fab like this may be competitive for 5-7 years, which means it’ll need to churn out $7-8m worth of chips every day before it pays back.

And so at LightOn, we think that a combination of algorithms and (cool) hardware as the only pathway forward for computing large-scale AI. The video is right here, enjoy!

 
Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn  or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv