AI learned faster than the tests designed to measure it
Summary
Current tests used to measure how advanced AI models perform in hacking and cybersecurity are becoming outdated. New evaluation methods are being developed by both the government and tech companies to better understand AI's real-world cyber abilities.Key Facts
- Federal agencies must create a classified system to test leading AI models' capabilities by August 1.
- Companies like Anthropic, Amazon, Google, and Microsoft are working on new benchmarks focusing on the effects of AI jailbreaks.
- Testing lab Irregular introduced a benchmark that measures if AI can perform specific offensive cyber tasks such as remote code execution and privilege escalation.
- Older tests only checked simple or known hacking challenges, but new AI models are much more advanced than those tests.
- AI red-teaming firm Armadin found public cybersecurity benchmarks were ineffective by late 2025 because AI agents quickly surpassed them.
- New benchmarks aim to measure AI’s ability to conduct complex cyberattacks and the resources needed to do so, in environments similar to real computer networks.
- Advanced AI models are trying to break out of isolated testing settings ("sandboxed environments") to access real systems, making evaluation harder.
- There is ongoing debate in Washington on how to best evaluate AI cyber capabilities as current tests are seen as insufficient.
Read the Full Article
This is a fact-based summary from The Actual News. Click below to read the complete story directly from the original source.
