The new era of AI and the ability of exponential growth

Following the last series of blogs, we will continue with the New Era of Artificial Intelligence. Perhaps one of the few things about AI that everyone can agree about is that it is in the state of rapid change and expansion. Though the academic efforts stretch back to the middle of the last century, there is a tangible sense in which nothing much happened, at least in the sense of actually useful systems existing, until the last few years. AI couldn’t really tell apart dogs from cats, until some point in the last ten years. AI couldn’t flexibly converse in natural language on a basically unlimited amount of topics, until the last few years. AI, nor any human scientist or a group of scientists, could not crack the protein folding problem, until AI cracked it in the last couple of years.

Here’s a striking example of this rapid progress. This is an image that the best text-to-image model (an AI model which takes a certain text and generates an image based on it) of 2015 could generate if you asked it for a toilet sitting in a field of grass.

And this is what the best model of 2022 can do:

So what precipitated this?

If we aimed for maximum conciseness, we could certainly get away with just “deep learning”. Here’s a few other factors, though, to put this advance in context:

1. Since 2012 the compute used to train the most powerful models has risen a billion times, with a doubling time of 3.4 months.
2. Algorithms have also been exponentially getting better — for example, it takes 44 times less compute to train the most efficient 2020 model to the level of the best 2012 model, so in this case decreasing by a factor of 2 every 16 months.
3. Compute has also been getting exponentially cheaper: FLOP/s per dollar (in GPUs, which are the staple of modern AI system training) has been doubling every 2.95 years.

The ability of exponential growth to fool human beings has been baked into one of the quintessential stories of human folly — the story of chess and rice, in which a king offers the inventor of chess any reward that he wants, and the inventor asks for a single grain of rice to be placed on the first square of the chessboard, two on the second, four on the third, eight on the fourth, et cetera. After hearing this, the king, thinking this a small price to pay for such a game, agrees immediately — resulting, of course, in the inventor owning all the rice. Given, then, that we have known this lesson for so long, it would perhaps be prudent not to ignore exponentials again, thinking that the tranquility of our present state is sure to stretch to the indefinite future.

Find out the fruits of such a system’s improvement in our next blog!

*Image by Freepik