What is a Parameter?

Originally published on LinkedIn on September 28, 2022

A common word that you will hear when people are talking about certain types of artificial intelligence systems is “parameter.” This is a measure of the level of complexity of these systems, essentially a quantity of information that supports the functioning of the system. Recently Ray Kurzweil in an interview with Lex Fridman offered a simple example of a parameter to help illustrate this concept:

“elephants have trunks”

You can think of this as a single parameter. In itself it isn’t very helpful to identifying whether or not something is an elephant. We also use the word “trunk” when we talk about trees. And visually an elephant’s trunk might look similar to a hose. So to make sense of the world we need a lot of parameters – a lot of these bits of information that we then aggregate to describe the world. Human beings accumulate these bits of information as we grow up, experiencing the world around us and hearing the words that people use to describe what we are seeing and doing. An adult human might have hundreds of billions of “parameters.”

Some of the larger artificial intelligence systems now have an equivalent number of parameters. OpenAI’s system, GPT-3, (released in mid 2020) has 175 billion parameters. However these bits of information have of course not been accumulated through experiencing the world - they have been accumulated by collecting textual information from books, articles, conversations, Internet sources, etc. So what GPT-3 does with these parameter is in many ways less sophisticated than what a human would do. For example, interpreting the statement:

“the bowling ball fell on the vase and it broke”

and understanding that it was in fact the vase that broke and not the bowling ball requires knowledge of the world that comes easily through experience, but is difficult for an AI that is simply working with a large data set of accumulated text. On the flip side, GPT-3 has a much broader base of information of the world and can easily answer questions about things that humans would find obscure. For example, GPT-3 can tell you who the poet Joel Barlow is and that he is better known for his politics than his poetry and even write study notes on the key things to know about him. I asked GPT-3: “What are 5 key points I should know when studying the poet Joel Barlow?” and got the following output:

• Joel Barlow was an American poet and diplomat.
• He was born in 1754 in Reading, Connecticut.
• He graduated from Yale University in 1778.
• He served in the Continental Army during the American Revolution.
• He was a diplomat to France during the French Revolution.

Researchers are now developing even bigger models. Google has developed a system called PaLM that has over 500 billion parameters. There is a lot of research into the question of how the performance of these systems is impacted by the sheer size (or number of parameters) vs. how the parameters are being used and still quite a lot for us to learn in building systems for specific tasks much less general systems that can perform any task.

Where we are today – artificial intelligence systems are beginning to have, or exceed, a human level of complexity in the quantity of structured information - and we are beginning to understand how to use this complexity to allow these systems to perform specific tasks.

2026 update - frontier models now have over 1 trillion parameters and we expect 10 trillion parameter models in the next year. Each increase in the size of model has in the past increased the “capabilities” of these models