Each Sunday, NPR host Will Shortz, The New York Instances’ crossword puzzle guru, will get to quiz 1000’s of listeners in a long-running section known as the Sunday Puzzle. Whereas written to be solvable with out too a lot foreknowledge, the brainteasers are normally difficult even for expert contestants.
That’s why some consultants assume they’re a promising method to take a look at the boundaries of AI’s problem-solving skills.
In a current research, a staff of researchers hailing from Wellesley School, Oberlin School, the College of Texas at Austin, Northeastern College, Charles College, and startup Cursor created an AI benchmark utilizing riddles from Sunday Puzzle episodes. The staff says their take a look at uncovered stunning insights, like that reasoning fashions — OpenAI’s o1, amongst others — typically “hand over” and supply solutions they know aren’t right.
“We wished to develop a benchmark with issues that people can perceive with solely common data,” Arjun Guha, a pc science school member at Northeastern and one of many co-authors on the research, instructed DailyTech.
The AI trade is in a little bit of a benchmarking quandary in the mean time. A lot of the exams generally used to guage AI fashions probe for expertise, like competency on PhD-level math and science questions, that aren’t related to the typical consumer. In the meantime, many benchmarks — even benchmarks launched comparatively lately — are shortly approaching the saturation level.
The benefits of a public radio quiz sport just like the Sunday Puzzle is that it doesn’t take a look at for esoteric data, and the challenges are phrased such that fashions can’t draw on “rote reminiscence” to resolve them, defined Guha.
“I feel what makes these issues onerous is that it’s actually troublesome to make significant progress on an issue till you resolve it — that’s when every thing clicks collectively unexpectedly,” Guha stated. “That requires a mixture of perception and a technique of elimination.”
No benchmark is ideal, after all. The Sunday Puzzle is U.S. centric and English solely. And since the quizzes are publicly out there, it’s attainable that fashions skilled on them can “cheat” in a way, though Guha says he hasn’t seen proof of this.
“New questions are launched each week, and we are able to anticipate the most recent inquiries to be really unseen,” he added. “We intend to maintain the benchmark contemporary and monitor how mannequin efficiency modifications over time.”
On the researchers’ benchmark, which consists of round 600 Sunday Puzzle riddles, reasoning fashions akin to o1 and DeepSeek’s R1 far outperform the remaining. Reasoning fashions completely fact-check themselves earlier than giving out outcomes, which helps them keep away from a number of the pitfalls that usually journey up AI fashions. The trade-off is that reasoning fashions take somewhat longer to reach at options — usually seconds to minutes longer.
No less than one mannequin, DeepSeek’s R1, offers options it is aware of to be improper for a number of the Sunday Puzzle questions. R1 will state verbatim “I hand over,” adopted by an incorrect reply chosen seemingly at random — habits this human can actually relate to.
The fashions make different weird decisions, like giving a improper reply solely to right away retract it, try to tease out a greater one, and fail once more. Additionally they get caught “pondering” perpetually and provides nonsensical explanations for solutions, or they arrive at an accurate reply straight away however then go on to think about various solutions for no apparent purpose.
“On onerous issues, R1 actually says that it’s getting ‘annoyed,’” Guha stated. “It was humorous to see how a mannequin emulates what a human may say. It stays to be seen how ‘frustration’ in reasoning can have an effect on the standard of mannequin outcomes.”
The present best-performing mannequin on the benchmark is o1 with a rating of 59%, adopted by the lately launched o3-mini set to excessive “reasoning effort” (47%). (R1 scored 35%.) As a subsequent step, the researchers plan to broaden their testing to further reasoning fashions, which they hope will assist to establish areas the place these fashions could be enhanced.
“You don’t want a PhD to be good at reasoning, so it needs to be attainable to design reasoning benchmarks that don’t require PhD-level data,” Guha stated. “A benchmark with broader entry permits a wider set of researchers to grasp and analyze the outcomes, which can in flip result in higher options sooner or later. Moreover, as state-of-the-art fashions are more and more deployed in settings that have an effect on everybody, we imagine everybody ought to be capable to intuit what these fashions are — and aren’t — able to.”
