Alphabet’s Kaggle adds local workflow for creating AI benchmark tasks

Alphabet’s Kaggle said it is rolling out “local development” for Kaggle Benchmarks, a change aimed at making it easier for engineers to create new evaluation tasks and push them into Kaggle’s benchmark ecosystem. The update, announced June 4, 2026, is designed to let developers build benchmark tasks from their own development setup rather than relying only on Kaggle’s web-based notebook editor. It also connects benchmark authoring to AI coding agents, so teams can describe evaluations in natural language and have agent tooling generate the underlying task structure.

The company framed the move around the evolving needs of AI model testing. As models move from chat-only behavior toward reasoning agents that can write code and use tools, Kaggle argued that “traditional benchmarks are no longer enough” on their own. Kaggle Benchmarks, which it says launched earlier, is intended to support community-driven, dynamic evaluations where tasks produce transparent leaderboards. In the June 4 announcement, Kaggle said the community has created more than 10,000 evaluation tasks since the benchmarks launched.

Under the local development update, Kaggle says developers can create, validate, push, run, and download benchmark tasks directly from local development environments, naming tools and workflows such as Antigravity, VS Code, Cursor, and “coding agents.” Previously, Kaggle said creating evaluation tasks required working within Kaggle’s web notebook editor. The practical shift, according to the announcement, is that teams can keep their preferred coding stack while still producing benchmark artifacts that can be executed and shared through Kaggle’s benchmarking system.

Kaggle also highlighted a second workflow, built around AI coding agents that author benchmark tasks. The company introduced a “write-kaggle-benchmarks” skill, which it described as a set of structured instructions that teaches a coding agent how to build benchmark tasks using the Kaggle Benchmarks software development kit and the Kaggle CLI (the command-line tool developers use to interact with Kaggle projects). Kaggle said agents can be instructed to install the skill, then accept a plain-language description of an evaluation and generate a working task on Kaggle. The post included an example of describing an evaluation involving a simple arithmetic question.

To clarify what it means by a “skill,” Kaggle’s agent-skills repository describes skills as “folders of instructions, scripts, and resources that agents can use to perform specialized tasks.” In other words, the “write-kaggle-benchmarks” skill is positioned as a reusable instruction pack for agents, rather than a one-off prompt. Kaggle said the new local workflow is supported by “new commands” built into the Kaggle CLI for Benchmarks, though it did not list those command names or provide additional implementation detail in the announcement.

Beyond the tooling update, Kaggle used the announcement to reiterate its broader philosophy for AI evaluation. It said it built Kaggle Benchmarks to “democratize trustworthy AI evaluations” and argued that if a capability can be measured, labs will compete to improve it. In the post, Kaggle emphasized that evaluations should reflect a diverse set of real-world challenges and called the local development launch a step toward enabling “anyone, anywhere” to build benchmarks that shape the direction of AI development.

What was not disclosed in the announcement leaves several operational questions open. Kaggle did not specify how local task creation interacts with authentication and permissions (for example, whether service accounts or user logins are required when pushing tasks), what compute and runtime constraints apply when running tasks locally versus on Kaggle’s infrastructure, or what versions of local IDEs and agent frameworks are supported. The company also did not provide adoption metrics, timeline for rolling out the “new commands” across all CLI environments, or any information about whether additional benchmark authoring features are planned beyond task creation and execution.

For teams watching the AI tooling landscape, the practical next question is whether this lowers the cost of benchmark experimentation enough to change who can publish new evaluation tasks and how quickly they can iterate. If local workflow plus agent-assisted authoring proves smooth, Kaggle Benchmarks could see faster expansion in the variety of tasks community members contribute, potentially increasing scrutiny over what models claim to do. The announcement’s “try it today” invitation suggests early testers may be the first to reveal the operational details that Kaggle did not lay out publicly.