What Do AI Agents Actually Change? An Empirical Taxonomy of Mutation Patterns in Performance-Improving Pull Requests
arXiv:2607.05666v1 Announce Type: cross Abstract: AI coding agents are black boxes: we cannot inspect how they generate code, but we can inspect what they change. This distinction matters for search-based software engineering (SBSE), where techniques such as genetic improvement (in the performance-optimisation application we study) depend on mutation operators that reflect how code is actually transformed. Fewer than 1% of the 33,596 agent PRs in AIDev-pop target performance, making each case a rare window into otherwise opaque agent behaviour. We classify 1,254 performance-relevant diff hunks
The proliferation of AI coding agents necessitates a deeper understanding of their real-world impact and how they optimize code, moving beyond theoretical capabilities to empirical observation.
This research provides critical empirical data on how AI agents transform code, particularly for performance, which is vital for search-based software engineering and the responsible development of autonomous AI systems.
Our understanding of AI agent behavior shifts from merely knowing they generate code to observing specific mutation patterns, allowing for more informed analysis and development of agentic systems.
- · AI agent developers
- · Software engineering researchers
- · Companies adopting AI for code optimization
- · Developers relying solely on black-box AI feedback
- · Traditional manual code optimization processes
Empirical data emerges on how AI agents modify code for performance improvements.
Improved tools and methodologies for analyzing and directing AI agent behavior in software development become possible.
The development of more transparent, controllable, and efficient AI coding agents accelerates, potentially leading to more sophisticated and trustworthy autonomous software development systems.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI