I was asked this in a backend interview last week and I don’t think I answered it well, so I’m curious how others would have handled it.
The setup they gave me: an ASP.NET Core service that calls an LLM to summarise support tickets and return a structured result. The team already has unit and integration tests for everything else. The interviewer asked how I would test this particular endpoint, given that the same input can produce different output on different runs.
What I said at the time:
Mock the model provider in unit tests, so the tests cover my own code — the request building, the parsing, the error handling — and not the model’s behaviour.
Validate the shape rather than the content: assert that the response deserialises, that required fields are present, that enums fall within allowed values.
Use a small fixed set of inputs with expected outputs for a separate, slower test suite that runs outside CI.
He pushed on the last point. His question was roughly: if the output varies, what exactly are you asserting against, and what counts as a pass? I said something about similarity thresholds and it was clearly not a satisfying answer.
Things I’ve thought about since:
Setting temperature to zero doesn’t fully solve it — output is more stable but not guaranteed identical, and provider-side changes can still shift it.
Asserting on semantic similarity needs a threshold, and I have no principled way of picking one.
Using another model to grade the output feels circular, and now you have two non-deterministic things instead of one.
Snapshot testing seems wrong here because the snapshot would need updating constantly.
So my questions:
In a real codebase, what do you actually assert on for this kind of endpoint?
Does any of this belong in CI, or is it a separate evaluation pipeline that runs on a schedule?
How do you catch a regression when the model provider updates a version underneath you — is that a test problem or a monitoring problem?
For anyone who has shipped something like this: what broke in production that your tests should have caught?
Interested in answers from people who have maintained one of these for a while rather than just built one, since I suspect the problems show up later.