One of the most comprehensive pictures yet of how AI is developing and how we are getting to grips with it has been set out in Stanford University’s seventh annual AI Index report. The Index “tracks, collates, distills, and visualizes data related to artificial intelligence (AI)”.

The authors call the report “our most comprehensive to date” and say “it arrives at an important moment when AI’s influence on society has never been more pronounced”.

Introducing the findings, AI Index co-directors Ray Perrault and Jack Clark said: “A decade ago, the best AI systems in the world were unable to classify objects in images at a human level. AI struggled with language comprehension and could not solve math problems. Today, AI systems routinely exceed human performance on standard benchmarks.”

“Current AI technology still has significant problems. It cannot reliably deal with facts, perform complex reasoning, or explain its conclusions.”

Ray Perrault and Jack Clark, Stanford University

Progress has accelerated but, say Perrault and Clark, “current AI technology still has significant problems. It cannot reliably deal with facts, perform complex reasoning, or explain its conclusions”. They speak of two futures, one in which AI continues to be deployed, with major consequences for productivity and employment, and another where AI adoption is constrained by the limitations of the technology.

One trend identified is that the number of new large language models (LLMs) released across the world doubled in 2023 over the previous year. And while two-thirds were open-source, the highest performing systems were closed.

Top 10 takeouts from the AI report

AI beats humans on some tasks, but not on all.

AI outperforms humans on image classification, visual reasoning and English understanding benchmarks, but it is not as good at more complex tasks such as competition-level maths, visual commonsense reasoning, and planning.

Industry continues to dominate frontier AI research.

A total of 51 notable machine-learning models were produced by industry, while academia produced just 15. However, a record-high 21 models came from industry-academic collaboration.

Frontier models get way more expensive.

Training costs for state-of-the-art AI models are unprecedented. Examples given are the estimated $78m of compute used for OpenAI’s ChatGPT, and $191m for Google’s Gemini Ultra.

The United States leads China, the EU, and the UK as the leading source of top AI models.

Institutions in the US were responsible for originating 61 notable AI models during 2023, with 15 from the EU and 12 from China.

Robust and standardized evaluations for LLM responsibility are seriously lacking.

AI Index research revealed “a significant lack of standardization in responsible AI reporting”. Leading developers such as OpenAI, Google and Anthropic all primarliy test against different benchmarks.

Generative AI investment skyrockets.

Funding octupled from the 2022 figure to reach $25.2 billion, with major players including OpenAI, Anthropic, Hugging Face, and Inflection reporting substantial rounds of fundraising.

The data is in: AI makes workers more productive and leads to higher quality work.

Multiple studies revealed that “AI enables workers to complete tasks more quickly and to improve the quality of their output”, and that it bridges the skill-gap between low- and high-skilled workers. But those studies also found evidence that “using AI without proper oversight can lead to diminished performance”.

Scientific progress accelerates even further, thanks to AI.

AlphaDev, which makes algorithmic sorting more efficient, and GNoME, which facilitates the process of materials discovery were cited as two of the most significant science-related applications launched thanks to AI.

The number of AI regulations in the United States sharply increases.

In 2016, there was a single AI-related regulation in the US. By 2023, there were 25, a rise of 56.3% on the number in 2022.

People across the globe are more cognizant of AI’s potential impact – and more nervous.

Polling company Ipsos found 66% of people think AI will “dramatically affect” their lives within five years, a rise of 6% on previous polling, while 52% are nervous about AI products and services, up 13%. Pew research in the US also found more than half those surveyed, 52%, were more concerned than excited about AI.

Other notable information in the report

• AI patents worldwide rose 62.7%, contributing to a 31-fold increase in AI patents granted since 2010. China led the way with 61.1% of global patent origins, far ahead of the US which had 20.9%. The US share has decreased from 54.1% in 2010.
• Open-source AI research is booming. The number of AI-related products on GitHub rose to approximately 1.8 million in 2023.
• Multimodal models are on the rise, with models such as Google’s Geminin and OpenAI’s GPT4 able to handle images and text and, in some cases, audio.

• Benchmarking is improving, with harder measures emerging. Notable new standards the AI Index picks out are:

1. SWE-bench for coding;
2. HEIM for image generation;
3. MMMU for general reasoning;
4. MoCa for moral reasoning;
5. AgentBench for agent-based behavior; and
6. HaluEval for hallucinations. 

• In addition, better data, created by AI, is enhancing capabilities and laying the foundations for algorithmic advancements that will improve AI – a virtuous circle of development. And there is increasing use of human evaluation in benchmarking, with public sentiment an increasingly important consideration in assessing progress.
• Top AI-related concerns for business across the world are privacy, data security, and reliability. While organizations are taking steps to mitigate those risks, only a small proportion have actually been mitigated. Copyright violations are becoming an increasing concern. The number of incidents related to misuse of AI rose 32.3% year-on-year to 123.
• There is a serious lack of transparency among AI developers, hindering efforts to evaluate the safety and robustness of Ai systems. Additionally, debate about existing risks such as algorithmic discrimination and more existential potential risks is making it more difficult to identify which issues are scientifically founded, leading to challenges in formulating policy.
• Private investment in AI in the US reached $67.2 billion, up 22.1% year on year and almost 8.7 times more than China. Private investment in the EU, including the UK, declined by 14.1%.
• The AI Index provides some figures to illustrate the growth in regulatory work, adding to the key takeout mentioned above about the increase in AI regulation in the US.
• Significant legislation in the shape of the AI Act in the EU and President Biden’s Executive Order on AI was enacted. In the US, 181 pieces of AI-related legalisation were proposed at federal level in 2023, up from 88 the previous year. The number of U.S. regulatory agencies issuing AI regulations increased to 21 in 2023 from 17 in 2022.
• AI legislation was mentioned in legislative proceedings in 49 countries, including one country in every continent. Mentions of AI in legislative proceedings globally nearly doubled year on year to 2,175.