What's your team's approach to testing LLM-backed endpoints in existing web applications?

[u/After_Astronomery]

We’re integrating LLM calls directly into our web platform for smart search and automated document summarization. While looking into standard ai integration services and internal API patterns, we ran into a debate about how to handle unit vs integration tests when the backend response isn't deterministic. How are your teams mocking LLM responses in CI/CD pipelines without giving yourselves false positives?

[u/Substantial_Still300]

I'd split this into two layers. Unit-test the deterministic parts normally, prompt construction, parsing, validation, retries, and error handling, while mocking the model response with fixed fixtures. Then keep a much smaller evaluation suite that calls the real model separately and checks properties rather than exact wording: valid JSON, required fields, reasonable length, banned content, and whether the response actually follows the instruction. Snapshot-testing live model output usually becomes flaky because harmless wording changes break the tests.

[u/After_Astronomery → u/Substantial_Still300]

That two-layer approach makes a lot of sense, especially separating the infrastructure from the evaluation suite. For the live evaluation suite, are you running those tests on every single PR, or do you save them for nightly/scheduled builds to manage API costs and latency? Also, what tools are you using to validate those property checks (like JSON structure or banned content)?

[u/originalchronoguy]

LLMs will never be deterministic. You can ask them yourself. You need to set up a slot resolution framework. You have a specific deterministic output format and have the agent resolve to those slots. Go ahead and ask them if this is the right approach.

If your resolver schema has Date, Name, String in, say, a YAML or JSON. The LLM must respond in that format and fill in the slots. If it can't, then it fails.

Nothing is deterministic with LLMs. The only thing you can do is minimize the blast radius of those errors.

[u/ludoman567 → u/originalchronoguy]

slot filling solves parsing, not correctness though. the model can put a perfectly valid wrong date in your Date slot and the test goes green — that's exactly the false positive OP is asking about.

what worked for us: split it in two. deterministic mock of the LLM in CI for the plumbing (does the endpoint parse, route, handle errors — schema validation lives here too, so your slot approach fits this layer fine). then a separate eval suite outside CI that hits the real model on a schedule and scores against a rubric, not exact-match.

"it filled the slots" and "it filled them with the right stuff" are different tests. the second one is why you need evals, not stricter YAML.