AI might excel at certain tasks like coding or generating a podcast. But it struggles to pass a high-level history exam, a new paper has found. A team of researchers has created a new benchmark to test three top large language models (LLMs) — OpenAI’s GPT-4, Meta’s Llama, and Google’s Gemini — on historical questions. […]
AI isn’t very good at history, new paper finds Read More »
Former Google engineer and influential AI researcher François Chollet is co-founding a nonprofit to help develop benchmarks that’ll probe AI for “human-level” intelligence. The nonprofit, the ARC Prize Foundation, will be led by Greg Kamradt, an ex-Salesforce engineering director and founder of the AI product studio Leverage. Kamradt will serve as president and a member
AI researcher François Chollet is co-founding a nonprofit to build benchmarks for AGI Read More »
When a company releases a new AI video generator, it’s not long before someone uses it to make a video of actor Will Smith eating spaghetti. It’s become something of a meme as well as a benchmark: Seeing whether a new video generator can realistically render Smith slurping down a bowl of noodles. Smith himself
Will Smith eating spaghetti and other weird AI benchmarks that took off in 2024 Read More »
SaaS founders trying to figure out what it takes to raise their next round can refer to Point Nine’s famous yearly SaaS Funding Napkin. (The term refers to “back of the napkin” plans or calculations.) Now, European hardware deep tech teams have a similar resource from First Momentum, a pre-seed fund investing in technical B2B
Deep tech startups with very technical CEOs raise larger rounds, research finds Read More »
Anthropic is launching a program to fund the development of new types of benchmarks capable of evaluating the performance and impact of AI models, including generative models like its own Claude. Unveiled on Monday, Anthropic’s program will dole out grants to third-party organizations that can, as the company puts it in a blog post, “effectively
Anthropic looks to fund a new, more comprehensive generation of AI benchmarks Read More »
