Testing · AI Systems
LLM Mutation Testing
Testing the AI grader by sneaking in intentional mistakes.
1. Who grades the grader?
When we use AI to grade other AI, how do we know the grader is paying attention?
Why is this a risk?
AI judges often like answers that sound confident and polite, even when a factual number or line of code is completely wrong.
2. Sneaking in the mutant
We intentionally change one tiny number or flip one fact to create a broken test copy.
What kind of mistakes do we inject?
We flip dates, swap names, invert true/false statements, and change numbers to see if the evaluation catches the error.
3. The catch score
If the grader catches the injected bugs, we know our evaluation test actually works.
What if a mutant survives?
If the grader passes a broken document, it means the test prompt is weak. We rewrite the evaluation rubric until all bugs are caught.
Longer version: Reference Entry on LLM Mutation Testing
Sources: Mitra, System Design for the LLM Era (2026); Zheng et al., Judging LLM-as-a-Judge (2023).