The LLM integration decision record: why the model selection and context management strategy you chose determines your quality ceiling and your inference cost surface

The document classifier that spent 28% of ARR on inference

A fintech startup builds an automated loan document classifier. The product ingests PDF uploads from loan applicants — tax returns, bank statements, pay stubs, utility bills — and classifies each document by type so that the loan processing team can route documents to the correct review queue without manually opening each file. Before the LLM integration, loan processors spent approximately 4 hours per application on document routing. The product promise is to reduce that to 20 minutes by automating the classification step. The founding team evaluates GPT-4 in a demo session: they upload ten sample documents to the ChatGPT interface, describe what each document is, and ask GPT-4 to classify the rest. GPT-4 correctly classifies nine of ten. GPT-3.5 correctly classifies seven of ten. The team chooses GPT-4.

The integration ships. Classification accuracy in production is 91% — better than GPT-3.5 on the demo sample but lower than the demo's 90% because the demo sample was manually selected representative documents and production includes edge cases: scanned PDFs with low OCR quality, bank statements from unusual institutions with non-standard formatting, tax returns from prior years with different field layouts. The team iterates the system prompt over six weeks, adding instructions to handle edge cases, and reaches 94% accuracy. They consider the integration complete. The system prompt is a 2,200-token instruction set that exists only in the production environment variable. No version history. No evaluation set. No documented accuracy baseline.

At launch, the startup has 40 loan applications per month, each with an average of 12 documents. Monthly inference cost: 40 applications × 12 documents × (average 4,500 input tokens per document + 200 output tokens) × GPT-4 pricing = approximately $290/month. Acceptable. Six months later, the startup runs a marketing campaign and reaches 200 applications per month. Monthly inference cost: 200 × 12 × 4,700 tokens × GPT-4 pricing = approximately $1,450/month. Twelve months in, the startup closes a partnership with a mortgage broker that sends 1,200 applications per month. Monthly inference cost: 1,400 × 12 × 4,700 tokens × GPT-4 pricing = approximately $10,100/month. Eighteen months in, at 1,800 applications per month: $14,000/month. The startup's ARR is $50,000. Inference costs are 28% of ARR and growing faster than revenue.

The team investigates switching to a cheaper model. They try GPT-3.5-turbo: accuracy drops from 94% to 78% — below the 85% threshold below which the loan processor team says the automation creates more work (having to verify incorrectly routed documents) than it saves. They try GPT-4o-mini: accuracy reaches 88% — also below threshold. They try fine-tuning GPT-4o-mini on their labeled dataset: accuracy reaches 93% — acceptable, and the inference cost for fine-tuned GPT-4o-mini is approximately $2,100/month at 1,800 applications. But fine-tuning requires a labeled dataset. The startup has been running for 18 months and has accumulated 21,600 classification results — but 6% of them were incorrect and were corrected by loan processors. The correction data was never captured. There is no labeled dataset. Building one requires manually labeling a sample of historical documents: a 3-week project for the team's one data engineer.

The decision that was never written down was not just the cost model — it was the production quality threshold (below which accuracy level does the automation create more work than it saves?), the model selection criteria at scale (the demo evaluated accuracy on 10 documents; the production constraint is cost at 21,600 documents per month), and the dataset management strategy (every correction made by a loan processor is a labeled training example — not capturing it costs 18 months of labeled data and 3 weeks of backfill work). All three were implicit in the founding demo session that produced "GPT-4 got 9 of 10 right."

The support classifier where the model update was silent

A SaaS company builds an LLM-powered customer support ticket classifier. Incoming support tickets are classified into one of 15 categories (billing, data export, API integration, account settings, etc.) and routed to the appropriate support team. The classifier is built on Claude claude-sonnet — at the time of integration, the current production version of the Sonnet model tier. The system prompt is 900 tokens: instructions to classify the ticket into one of the 15 categories, output requirements (JSON with a required category field and an optional confidence field), and 3 examples per category for few-shot prompting. The integration ships, the routing accuracy is 91% as measured by spot-check (50 tickets reviewed by a support manager who compares the AI classification to her own routing decision). The company considers the integration working.

Three months later, the provider updates the Sonnet model weights. The update is not announced — the API endpoint name does not change, the model version string in the response does not change, the provider's changelog has a brief entry noting "general quality improvements." The update changes how the model interprets few-shot examples in the system prompt. With the old weights, the few-shot examples were the primary signal for classification: the model found the example most similar to the input ticket and returned that category. With the new weights, the model reasons more explicitly about the category definitions before consulting the examples — a generally better behavior that produces better quality on diverse classification tasks, but slightly different on the specific ticket distribution of this SaaS company because the company's tickets have idiosyncratic phrasing that the old few-shot examples captured implicitly.

The accuracy drops from 91% to 84% on the company's ticket distribution. The API continues returning 200. The JSON format continues being valid. The category field continues being one of the 15 valid options. No error is raised. No alert fires. The company discovers the regression 23 days later when the head of support raises a question in Slack about why the billing team has been getting more non-billing tickets lately. The support manager spot-checks 50 tickets: 16 of 50 are misrouted. She escalates. The engineering team investigates. They discover the correlation between the provider update date and the onset of the regression. They roll back to the few-shot approach with more examples and more specific category definitions — and reach 93% accuracy after two weeks of prompt iteration. But the 23-day degradation has already caused support teams to process hundreds of incorrectly routed tickets, reducing the efficiency gain the classifier was supposed to produce and creating residual distrust of the AI routing among support staff.

The engineering leader reviewing this incident faces a classifier with no documented quality baseline, no automated quality measurement, no regression detection mechanism, and no prompt version history. The prompt was tuned for specific model weights that no longer exist behind the API endpoint, with no record of what the prompt said before each iteration or what quality trade-off each change made. The provider model update is not the root cause — model updates are a normal part of using managed LLM APIs. The root cause is that the quality measurement model was "the support manager spot-checks 50 tickets every few months" — a manual, low-frequency process that cannot detect a quality regression within 23 hours of the model update, let alone 23 days.

Three structural properties the LLM integration decision determines

1. The model selection and the quality-cost tradeoff surface

The model selection for a production LLM integration determines two surfaces simultaneously: the quality ceiling (the maximum accuracy the integration can achieve on the production task, given the model's capabilities and the prompt engineering possible within the context window) and the inference cost floor (the minimum cost per request given the model tier's pricing, the average input token count, and the required output length). These two surfaces are in tension — higher-capability models achieve higher quality ceilings but have higher inference cost floors — and the tension determines the viable model selection space for the integration.

The benchmark-to-production quality gap is the difference between what a model achieves on standardized benchmarks (MMLU, HumanEval, MATH) and what it achieves on the specific production task. This gap is non-trivial and non-predictable from benchmarks. A model that leads on MMLU may underperform a smaller model on a specific document classification task because the production task requires instruction following in a particular format that the smaller model was more specifically fine-tuned for. The only way to measure the production quality ceiling for a given model is to evaluate it on a labeled sample of production inputs — not demo inputs, not manually curated representative examples, but a stratified random sample from the actual production distribution including the edge cases and unusual inputs that constitute the long tail of real usage.

The cost model at production volume must be calculated at design time, not discovered post-launch. The cost calculation requires: (1) the average input token count per request, measured on a sample of production inputs using the provider's tokenizer, accounting for the system prompt tokens, the user input tokens, and any retrieved context tokens; (2) the average output token count per request, measured on a sample of actual completions rather than assumed from the output format specification (a JSON schema that fits in 150 tokens may generate 300 tokens of completion in practice if the model generates reasoning before the JSON); (3) the provider's input and output token prices; (4) the expected requests per month at current and projected traffic. The cost model should be calculated for the current model, the next cheaper model in the provider's lineup, and a fine-tuned version of the cheaper model — establishing the cost-quality tradeoff frontier before the integration is locked. The cost optimization decision record documents the general framework for infrastructure cost modeling; for LLM integrations the inference cost calculation must account for the non-linear relationship between model capability and token price (the 10x more expensive model is not 10x more accurate on a specific task) and the potential for prompt caching to reduce effective cost by 80–90% on stable system prompts.

The provider risk model documents the constraints that might force a model switch regardless of quality or cost preferences: rate limits at projected volume (does the provider's API rate limit in requests per minute constrain the integration at peak load?), pricing change history (has the provider raised prices or changed the pricing model in the past 18 months?), model deprecation policy (how much notice does the provider give before deprecating a model version?), and data processing terms (does the provider's privacy policy permit processing the data the integration will send — financial documents, customer PII, proprietary code?). Provider risk is a low-probability, high-disruption factor that is worth documenting explicitly because a forced model switch under time pressure (the provider announces a 30-day deprecation for the model the integration depends on) is much more disruptive than a planned migration. The build vs buy decision record documents the general framework for evaluating managed versus self-hosted infrastructure; for LLM integrations, self-hosted inference (running open-weight models on owned GPU infrastructure) eliminates provider risk and data transfer concerns but introduces operational complexity (GPU cluster management, model update management, inference optimization) that is rarely justified for teams that are not primarily AI infrastructure companies.

2. The context management strategy and the context window budget

The context management strategy is the decision about which information is sent to the model on each API call, how much of the context window each component occupies, and what happens when the conversation or the retrieved context exceeds the available space. It is the most consequential implementation decision in an LLM integration and the one most commonly deferred as an implementation detail that "we'll figure out when we need to."

Stateless context sends the system prompt and the current user input on each API call, with no conversation history. Stateless is the default implementation pattern — it requires no state management, produces deterministic per-request costs, and avoids context window overflow for single-turn interactions. It is correct for single-turn tasks (document classification, code generation with a complete specification, summarization) where each input is independent. It is incorrect for multi-turn tasks (support conversations that require understanding of prior messages, document editing sessions where the model must remember what the user changed in earlier turns, research assistants that need to recall what the user said they had already tried) where reasoning across prior turns is required for correct output. A stateless integration on a multi-turn product produces responses that ignore prior context, creating a user experience where the model "forgets" what was discussed a few messages earlier.

Stateful context includes the accumulated conversation history up to the context window limit. The context window budget allocation must be calculated for the maximum plausible conversation length, not the typical conversation length. For a customer support integration with an average conversation length of 8 turns at 300 tokens per turn: the typical context consumption is 2,400 tokens of conversation history, leaving ample room in a 128,000-token context window. But support conversations for complex billing disputes or migration issues can reach 40 turns, consuming 12,000 tokens of conversation history. If the system prompt is 900 tokens and the retrieved context is 3,000 tokens (3 retrieved articles at 1,000 tokens each), the total context at 40 turns is 900 + 3,000 + 12,000 = 15,900 tokens — still within the 128,000-token limit. But if the same integration uses a 16,000-token model and the long-tail conversation reaches 40 turns: 900 + 3,000 + 12,000 = 15,900 tokens, leaving only 100 tokens of headroom. The next turn's message pushes the context over the limit and the API returns a 400 error.

The context truncation policy specifies what to drop when the accumulated context exceeds the budget. The options have different quality implications: truncate oldest messages first (simplest, but drops the context that established the problem statement, which may be the most important); truncate middle messages (preserves the initial problem statement and the most recent messages, but creates a discontinuous narrative that can confuse the model); summarize dropped messages (compress older context with a summarization call, preserving semantic content at reduced token cost, but adding latency and cost for the summarization call); sliding window with pinned messages (keep the first N messages and the last M messages, dropping those in between; pin any messages with system annotations or key decisions). The truncation policy must be explicit in the ADR because it determines output quality for long-tail conversations — the ones where users are most frustrated and most likely to escalate.

Prompt caching reduces inference cost by caching the key-value computation for stable prefix sections of the prompt across multiple API calls. If the system prompt is 900 tokens and is identical across all requests, the provider can cache the computation for those 900 tokens and charge a lower rate for cached tokens on subsequent calls. For providers that support prompt caching (Anthropic supports prefix caching on Claude models; OpenAI supports it on GPT-4o models), the effective cost reduction on the system prompt portion is 80–90%. The caching strategy decision record documents the general framework for cache hierarchy and TTL decisions; for LLM integrations, prompt caching requires that the stable prefix (the system prompt and any static few-shot examples) is placed at the beginning of the prompt in a fixed order, and that the variable content (user input, retrieved context, conversation history) comes after. Interleaving static and variable content within the prompt defeats prefix caching.

3. The quality measurement model and the regression detection surface

LLM output quality is not binary. For structured output tasks (JSON classification, code generation with test suite, data extraction), quality is measurable against a ground truth: the output is either correct (the classification is the right category, the code passes the tests, the extracted field has the right value) or incorrect. For unstructured output tasks (summarization, customer support responses, creative content), quality requires human judgment or a secondary LLM to score — an approach called LLM-as-judge that has known biases (judge models prefer longer responses and responses that match their own style) but can approximate human evaluation at scale.

The quality measurement model must define three things before the integration ships: what correct means for the production task (for a classifier, a labeled test set of 500 inputs with ground truth categories; for a support response generator, a rubric with 5 dimensions scored 1–5 by a human reviewer or by an LLM judge); the measurement frequency (batch evaluation on a held-out test set daily, triggered evaluation on a random sample of production outputs weekly, or real-time evaluation via a parallel judge call on every production request); and the quality threshold below which the integration is considered degraded (below 85% F1 on classification, below 3.8/5 average on support response quality). The threshold must be calibrated against the production impact — what does a 5% drop in classifier accuracy cost the business in mis-routed tickets or human review time? That calculation sets the stakes for the threshold.

The regression detection surface is the gap between "the API returns 200" and "the output is correct for the production task." A provider model update that maintains JSON format compliance but shifts classification behavior by 7% F1 passes every technical health check (HTTP status, response format, latency SLO) while producing a business impact that accumulates invisibly for 23 days before a support manager notices a queue imbalance. The regression detection surface can only be closed by measuring quality, not by monitoring request success. The detection mechanism should be automated: a daily evaluation job runs the labeled test set through the production integration, scores the outputs, and alerts if the score drops below the threshold. The alert must fire before a 23-day regression accumulates into hundreds of mis-routed tickets. For a 500-sample test set, the daily evaluation job costs approximately 500 × (average tokens per request) × (provider pricing) — for most integrations, less than $5/day, which is the insurance premium for catching a quality regression within 24 hours.

The observability platform decision record documents the general framework for metrics, logs, and tracing; LLM integrations require observability dimensions that standard application monitoring does not cover: input token count distribution per request type (to detect when inputs are growing longer than expected, approaching context window limits), output token count distribution (to detect when the model is generating unusually long or short responses, which can indicate prompt interpretation changes), per-request quality scores from automated evaluation (to detect quality regressions within hours of a model update), and latency distribution by model tier (to set realistic p99 latency SLOs — LLM inference latency is orders of magnitude higher than database query latency and has a much longer tail). A feature flag wrapping the LLM integration (sending a percentage of requests to the new model or prompt while the rest go to the current one) enables the feature flag pattern for gradual model migrations and A/B testing of prompt changes — the rollout model and blast radius calculation from progressive delivery apply directly to LLM model rollouts.

Five ADR sections the LLM integration decision record needs

1. Model selection and capability mapping

Document the task requirements that drive model selection: the capability cluster needed (classification, reasoning over long documents, code generation, structured data extraction, multi-turn conversation, tool use), the required output format (free text, JSON with a schema, code in a specific language, a structured form), and the latency SLO (interactive integrations require p99 latency under 2 seconds; batch processing integrations can tolerate 30-second requests). Then document the evaluation results for the top three candidate models: the labeled test set size and source (a stratified random sample of production inputs, or a manually curated representative set with documented curation bias), the accuracy or quality score per model on the test set, and the estimated monthly inference cost per model at current and projected volume. The model selection decision should follow from these evaluations — not from demo performance or general benchmark ranking.

Document the fine-tuning decision: whether fine-tuning a smaller model was evaluated as an alternative to prompt-engineering the larger model, and the outcome (fine-tuning not evaluated because labeled dataset unavailable; fine-tuning evaluated and achieved X% accuracy on the test set at Y% of the inference cost of the larger model). If fine-tuning is deferred ("we'll fine-tune when we have enough labeled data"), document the labeled data collection strategy — how labeled examples are captured from production (every human correction of an AI output is a labeled training example), the minimum labeled dataset size before fine-tuning is evaluated, and the person responsible for initiating the fine-tuning evaluation. Without this strategy, the labeled dataset that would enable fine-tuning accumulates implicitly in human corrections that are never captured.

Document the provider fallback strategy: if the primary provider's API is unavailable (rate limit exceeded, service outage, or planned maintenance), what happens to requests? Options include failing open (returning a default value or skipping the AI step), queuing for retry (buffering requests for up to N minutes while waiting for the API to recover), or routing to a secondary provider (maintaining a fallback model from a different provider that can serve requests when the primary is unavailable). The error handling strategy decision record documents the general framework for graceful degradation and fallback behavior; for LLM integrations, the fallback strategy must account for the quality difference between the primary and fallback models — if the fallback model has significantly lower quality on the task, "fail open with a fallback model" may produce worse outcomes than "fail closed with a clear error message to the user."

2. Context management strategy and window budget

Document the context mode (stateless or stateful) and the rationale in terms of the task's turn-dependency (does correct output for the current turn require reasoning over prior turns?). For stateful integrations, document the context window budget: the token allocation for each component (system prompt, retrieved context, conversation history, output reservation) and the maximum plausible size for each component — not the average size, but the 95th percentile or the hard maximum. Show the arithmetic: context window size minus component maximums equals the headroom. If the headroom is negative (the components at maximum would overflow the context window), document the truncation policy that resolves the overflow.

The truncation policy must specify not just which messages are dropped but what the quality implications are for the task. A support conversation where the problem statement is in the first 3 messages and the resolution steps are in the last 10 requires a different truncation strategy than a research conversation where the most important context is in the middle (where the user provided the key constraint that narrowed the problem). Document the quality implication of truncation: if truncating the first 10 messages of a long conversation causes the model to lose the problem statement, what is the expected accuracy drop on the task? This analysis is not always possible before launch, but the truncation policy should specify that the quality impact will be measured after launch and the policy revised if the impact exceeds a threshold.

Document the retrieval-augmented generation design if the integration uses RAG: which knowledge bases are retrieved from, the retrieval model (vector similarity search, BM25, or hybrid), the number of chunks retrieved and their maximum size, the chunk placement in the prompt (before or after conversation history), and the quality impact of retrieval failures (what happens when the retrieval returns no relevant results or returns misleading results). RAG adds complexity in both the token budget (retrieved context consumes 20–40% of the context window for typical implementations) and the quality measurement (a response that is well-formatted and factually correct about the retrieved document but incorrect because the retrieval returned the wrong document requires quality measurement at the retrieval layer, not just the generation layer).

3. Prompt architecture and versioning policy

Document the prompt structure: the system prompt's components (role definition, task description, output format specification, few-shot examples, constraints), the ordering of components (stable content first for prompt caching effectiveness, variable content last), the versioning format (semantic version in a comment at the top of the prompt file, stored in the prompts/ directory in the application repository), and the changelog structure for each version (what changed, why, what quality evaluation was run, and the score delta). The prompt file is code — it should be reviewed before merge, tested against the evaluation set before deployment, and reverted through the same mechanism as a code rollback.

The evaluation gate for prompt changes specifies the minimum quality threshold that a new prompt version must achieve on the labeled test set before it can be deployed to production. The threshold should be documented as both an absolute score (new prompt must achieve at least 85% F1) and a relative score (new prompt must not regress more than 2% F1 from the current production prompt). The relative threshold protects against a new prompt that meets the absolute threshold but performs significantly worse than what users are currently experiencing. The evaluation gate should be automated as a CI step: when a prompt file change is merged to main, the CI pipeline runs the labeled evaluation set through the new prompt, calculates the quality score, and fails the deployment if the score is below the absolute or relative threshold.

Document the few-shot example management strategy: how examples are selected (a representative sample of correct input-output pairs from production, curated to cover edge cases and typical patterns), how they are updated (examples are updated when new edge cases are discovered, as part of the prompt versioning process with evaluation before deployment), and how many examples are appropriate (too few examples are insufficient for in-context learning on complex tasks; too many examples consume context window budget that could be used for retrieved context or conversation history). The API contract testing decision record documents the framework for testing API interfaces; the prompt evaluation set is the equivalent of API contract tests for the prompt interface — it defines the expected behavior of the model on a set of representative inputs and verifies that behavior is maintained across prompt changes and model updates.

4. Quality measurement and regression detection

Document the quality metric for the task: F1 or accuracy for classification tasks (with the specific definition of what counts as correct — exact category match, or acceptable if the model selects a category in the same cluster as the ground truth?); ROUGE or BERTScore for summarization; pass rate on a test suite for code generation; a rubric-based score for open-ended generation evaluated by a human or an LLM judge. Document the labeled test set: its size, the process by which labels were assigned (human annotators, subject matter expert review, consensus of two reviewers with a defined resolution process for disagreements), and the expected update cadence (when new input patterns emerge that are not represented in the test set, the test set should be extended).

Document the regression detection mechanism: the batch evaluation job that runs the test set against the production integration on a scheduled cadence, the alert threshold (percentage drop in quality metric that triggers an alert), the alert routing (who receives the alert and what the first response steps are), and the rollback procedure (for a quality regression caused by a prompt change, revert the prompt to the prior version; for a quality regression caused by a provider model update, add instructions to the system prompt to compensate for the new model behavior, or switch to a pinned prior model version if the provider supports it). The regression detection alert must fire faster than the business can accumulate damage from the regression — for a support ticket classifier, that means within 24 hours of the quality drop, not 23 days later when a support manager notices queue anomalies.

Document the production quality monitoring strategy distinct from the test set evaluation: a continuous sample of production requests with their outputs, evaluated for quality by a parallel LLM judge call or by human review of a random sample. The production monitoring catches degradation patterns that the static test set doesn't cover — new input types that have entered the production distribution since the test set was created, or edge cases that exist in production but were never included in the curated test set. The background job infrastructure decision record documents the execution model for scheduled evaluation jobs; LLM quality evaluation jobs have specific requirements: they must run against a static model version (using the same model the production integration uses, pinned to avoid the evaluation itself being affected by model updates), they must log the full input-output pair and the quality score for debugging, and they must be idempotent (rerunning the job on the same test set produces the same scores, not different scores due to model temperature settings).

5. Cost model and inference budget

Document the cost calculation at current and projected traffic: average input tokens per request (measured on a sample of production inputs, not estimated from the prompt specification), average output tokens per request (measured on a sample of production completions), provider pricing for input and output tokens at the selected model tier, requests per month at current traffic and at 12-month projected traffic. Show the monthly cost calculation for all three scenarios: current traffic, 12-month projection, and 3x the 12-month projection (the spike scenario). If the 3x spike cost is material (more than 20% of projected ARR), document the cost mitigation strategy: prompt caching to reduce effective input token cost, response caching to avoid repeated API calls for identical inputs, model downgrade for lower-stakes requests (batch processing, non-interactive classification), or request throttling at the application layer.

Document the cost monitoring strategy: alert thresholds for daily and monthly inference cost (absolute dollar amounts or percentage of budget), the dashboard or cost explorer view for tracking actual vs. projected costs, and the cost spike investigation process (which metrics to check first — is the spike due to increased request volume, increased average token count, or a model pricing change?). Cost spikes from increased average token count indicate a change in input distribution (users sending longer inputs) or a system prompt change that increased the prompt length. The observability platform decision record should include LLM-specific cost metrics alongside standard application metrics.

Document the token budget optimization strategies that are appropriate for the integration: prompt compression (removing redundant instructions from the system prompt without quality degradation — measured by running the compressed prompt through the evaluation set), few-shot example pruning (removing few-shot examples that are not contributing to quality — measured by ablation testing, removing one example at a time and measuring the quality delta), retrieved context sizing (measuring the marginal quality gain per additional retrieved chunk and stopping when the marginal gain falls below a threshold), and output length constraints (using the model's max_tokens parameter to cap response length for tasks with bounded output requirements). Each optimization should be documented as a deliberate decision with a measured quality impact, not applied heuristically — a 20% token reduction that costs 3% F1 is a reasonable trade; a 20% token reduction that costs 12% F1 is not, regardless of the cost savings.

Further reading

• Decisions never written down — the model selection rationale (what the founding demo measured and what production requires), the production quality threshold (below which accuracy the automation adds work rather than saving it), the context window budget (calculated at maximum conversation length, not typical length), and the labeled dataset collection strategy (capturing human corrections as training examples) are the four LLM integration decisions most commonly made implicitly in demo sessions and never documented
• Cost optimization decision record — LLM inference cost optimization requires the same framework as infrastructure cost optimization: a cost model calculated at projected volume (not current volume), optimization strategies ranked by cost savings vs. quality impact, and monitoring that alerts before the monthly budget is overrun rather than after; the specific techniques (prompt caching, response caching, model tier selection, fine-tuning smaller models) each require a documented quality-cost tradeoff evaluation
• Feature flag decision record — feature flags enable safe model migrations and prompt A/B testing by routing a percentage of production traffic to the new model or prompt while the majority continues on the current version; the blast radius calculation applies directly: a 5% rollout to a model with degraded quality exposes 5% of users to quality degradation before the quality regression detection fires; the rollback is a flag change (returning the 5% to the current model) rather than a full deployment
• Observability platform decision record — LLM integrations require observability dimensions that standard application monitoring does not cover: per-request token counts (to detect context window boundary approaches), output quality scores from automated evaluation (to detect quality regressions within 24 hours of a model update), cost per request by model tier and request type (to enable cost attribution and anomaly detection), and latency distribution including time-to-first-token (for streaming integrations where user-perceived latency is TTFT, not total request duration)
• Background job infrastructure decision record — asynchronous LLM inference (processing requests in a background queue rather than inline with the user request) is appropriate for non-interactive tasks (batch document classification, overnight report generation, asynchronous data enrichment) and decouples user-facing latency from inference latency; the background job infrastructure must support LLM-specific requirements: retries with exponential backoff on rate limit errors (provider APIs return 429 with a retry-after header), result storage for completed inferences, and idempotency to avoid duplicate API calls if a job is retried
• Caching strategy decision record — two types of caching apply to LLM integrations: prompt caching (provider-side caching of stable prompt prefixes, reducing inference cost by 80–90% on the cached portion) and response caching (application-side caching of complete responses for identical or semantically equivalent inputs, reducing API calls entirely for repeated queries); response caching requires a cache key that captures the meaningful variation in the input (not the raw text, which may vary in whitespace or punctuation without changing meaning) and a TTL that reflects when the model's response to the same input might change (a cached classification result for a document does not expire; a cached response to a current-events question expires rapidly)
• Error handling strategy decision record — LLM API errors have a different character than database or HTTP service errors: rate limit errors (429) are expected at scale and require exponential backoff with jitter rather than immediate retry; context window errors (400 with a specific error code) indicate a design constraint violation requiring prompt redesign rather than retry; model unavailability (503) during provider incidents requires a fallback strategy; the error handling strategy must account for the quality difference between primary and fallback models, and must specify when to fail open (return a default or degrade gracefully) versus fail closed (surface an error to the user)
• API contract testing decision record — the LLM system prompt defines the output contract (JSON schema, required fields, valid enumerated values, response length constraints); the labeled evaluation set is the contract test for the prompt — it verifies that the model produces compliant output for a representative sample of inputs; contract tests for LLM output must go beyond JSON schema validation (format compliance) to field value accuracy (the classification category is correct, not just a valid category from the schema); automated format validation in the application parsing layer and semantic quality evaluation in the evaluation job are two distinct layers of the contract testing strategy
• Build vs buy decision record — the self-hosted vs managed inference decision for LLMs is a specific instance of the build vs buy framework: managed inference (OpenAI, Anthropic, Google) provides low operational overhead, state-of-the-art models, and automatic infrastructure scaling at the cost of data transfer, provider lock-in, and pricing uncertainty; self-hosted inference (running open-weight models like Llama or Mistral on owned GPU infrastructure) provides data privacy, cost predictability at scale, and provider independence at the cost of GPU infrastructure operational complexity, model maintenance, and falling behind the managed provider's model quality improvements
• The new CTO onboarding problem — inheriting an LLM integration with no documented quality threshold, no labeled test set, no prompt version history, and no cost model means starting from scratch on every question: what is the current accuracy, what was the original rationale for choosing GPT-4, why does the system prompt contain specific instructions (are they compensating for an old model behavior that no longer exists?), what quality regression triggered the last prompt change? The LLM integration ADR is the document that converts the founding demo session's tribal knowledge into answerable questions for the next engineering leader