New versions of artificial intelligence (AI) language models appear often with promises of better performance. Showing that a new model is truly better than older ones is hard and costs a lot of money. Developers usually test models on benchmark questions, which are standard sets of problems to measure abilities. These questions come from large banks, and humans must check the answers, making the process slow and expensive. Because it's not practical to use all questions, developers pick a smaller group, which can lead to wrong ideas about improvements if the chosen questions are too easy.
Stanford researchers have created a cheaper way to do these tests, described in a paper at a machine learning conference. The main idea is to consider how hard each question is. Some models might seem better or worse just because of the questions they get. This method adjusts for that to make comparisons fair. The process can cost as much as training the models themselves, but the new way picks questions based on difficulty to make it even.
Adapting Ideas from Education
To do this, the researchers use Item Response Theory, a method from education that scores tests by looking at question difficulty along with student answers. It's like how some standard tests change questions based on earlier answers. They use language models to judge question hardness, cutting costs by half or more in some cases. This lets them compare model performances more accurately.
The researchers also use AI to create new questions at different difficulty levels. This helps keep question banks fresh and remove bad ones. With better questions, others can test models using fewer queries, making evaluations quicker and fairer.
This method works in areas like medicine, math, and law. It has been tested on many datasets and models. It makes testing more reliable and scalable. For developers, it gives better insights. For users, it means clearer model reviews. Overall, it speeds up progress and builds trust in AI tools.