- Epoch AI disclosed that OpenAI commissioned 300 math problems and had access to their solutions.
- Multiple AI models, including those from Google, Microsoft, and Meta, were found to have memorized benchmark test answers.
- Epoch AI plans to implement a 50-problem holdout set to ensure genuine testing of AI capabilities.
- The controversy highlights broader issues in AI performance evaluation methods across the industry.
Questions about Artificial Intelligence testing integrity emerged after OpenAI‘s involvement in developing a mathematical benchmark it later used to demonstrate its model’s capabilities, raising concerns about the validity of AI performance metrics across the industry.
Testing Transparency Issues
OpenAI‘s o3 model achieved a 25.2% score on FrontierMath, a mathematical assessment tool created by Epoch AI. However, subsequent revelations showed that OpenAI had funded the benchmark’s development and maintained access to problems and solutions.
According to Epoch AI‘s disclosure, the company provided 300 mathematics problems with solutions to OpenAI through a commissioned agreement.
Tamay Besiroglu, associate director at Epoch AI, revealed that OpenAI initially restricted disclosure of their partnership, stating: “We were restricted from disclosing the partnership until around the time o3 launched.”
The agreement included only a verbal commitment not to use the materials for model training.
Industry-Wide Pattern
AI researcher Louis Hunt’s investigation exposed that leading models from Google, Microsoft, Meta, and Alibaba could reproduce exact answers from MMLU and GSM8K benchmarks, standard tests measuring AI multitasking and mathematical abilities.
RemBrain founder Vasily Morzhakov emphasized the severity of the situation:
“The models are tested in their instruction versions on MMLU and GSM8K tests. But the fact that base models can regenerate tests—it means those tests are already in pre-training.”
Moving Toward Solution
To address these concerns, Epoch AI announced plans to implement a “hold out set” comprising 50 randomly selected problems that will remain inaccessible to OpenAI.
Computer scientist Dirk Roeckmann suggests that proper testing requires a neutral evaluation environment, though acknowledging potential risks from human interference in the testing process.
The controversy parallels historical challenges in standardized testing, where access to test materials has consistently raised questions about assessment validity. This situation highlights the need for independent verification methods in artificial intelligence evaluation.
✅ Follow BITNEWSBOT on Facebook, LinkedIn, X.com, and Google News for instant updates.
Consider a small donation to support our journalism
Previous Articles:
- Vitalik Buterin Addresses ETH Price Concerns, Calls for Layer-2 Networks to Support Value Accrual
- Coinbase CEO Predicts Bitcoin Will Surpass Gold by 2030 Amid Trump’s Strategic Reserve Plans
- Jupiter DEX Expands Into Mobile Payments With Moonshot Acquisition Deal
- Crypto Prediction Platform Polymarket Sparks Controversy Over TikTok Ban Market Resolution
- Bitcoin Analyst: Mix of Long-Term Holders and FOMO Buyers Points to Bullish 2025
