Google updates Android Bench with new LLMs, but Gemini still lags behind
Summary
Google has updated Android Bench, a test that measures how well large language models (LLMs) perform at Android app development. The update adds new models and a simpler testing framework, but Google's own Gemini model ranks below some competitors in accuracy and cost efficiency.Key Facts
- Android Bench tests how well LLMs handle 100 Android development tasks.
- Google added eight new LLMs to the leaderboard, including Claude Fable 5 and GPT 5.4.
- Gemini 3.1 Pro, Google's model, ranks fifth behind other models like GPT 5.4 and Claude Fable 5.
- Claude Fable 5 leads with 84.5% accuracy but costs over $130 per test run.
- Gemini 3.1 Pro costs less to run ($87 per test) but scores lower.
- Gemini 3.5 Flash is the most expensive model to run ($165) due to long runtime.
- Google switched Android Bench to use the Harbor framework to make it easier for developers to run tests and share results.
- Developers can contribute new tasks and benchmarks to Android Bench via GitHub.
Read the Full Article
This is a fact-based summary from The Actual News. Click below to read the complete story directly from the original source.
