The model evaluation decision record: why the evaluation methodology you chose determines your production regression blind spot and your benchmark gaming surface

A 28-person developer productivity SaaS company built internal tooling for software engineering teams at mid-market SaaS companies — code navigation, automated PR description generation, and a context-aware code generation assistant that suggested implementations based on the team's codebase conventions. The code generation assistant was built on a fine-tuned language model, trained on a combination of open-source code repositories and a curated dataset of high-quality code generation examples assembled by the company's QA engineering team over six months. The QA team evaluated the fine-tuned model against a 1,200-example held-out test set drawn from the same curation process — examples were reserved from the curated dataset rather than seen during fine-tuning — and confirmed 87% accuracy on the held-out set using a combination of exact-match scoring for deterministic outputs and a rubric-scored manual review for generative completions. The model shipped to the company's twenty-three customers, integrated as an IDE extension used by the development teams at each customer's engineering organization.

Three weeks after the general availability release, the support queue contained fourteen tickets with a common pattern: the assistant was producing incomplete or incorrect suggestions specifically for code interacting with the customer's internal framework libraries. The company's largest customer — a 90-person engineering organization — had built a significant portion of their product on a proprietary API gateway framework and a reactive state management library, both maintained internally with no open-source equivalent. Their engineers reported that the assistant worked well for standard library calls and framework-agnostic logic but failed consistently on calls into the internal framework: it either hallucinated method signatures, ignored the framework's calling conventions, or produced implementations that were syntactically valid but semantically incompatible with how the framework's reactive model worked.

The investigation produced an uncomfortable finding. The 1,200-example held-out test set contained 24 examples involving internal or organization-specific framework usage — 2% of the set — because internal framework code had represented roughly the same fraction of the curated dataset that the QA team had assembled. The QA team had prioritized examples of high-quality, generalizable code patterns when curating: standard library usage, common algorithmic patterns, well-documented API integrations. Organization-specific internal framework code was not well-documented, was not generalizable across customers, and was not the focus of what the QA team was testing for. But the engineers at the company's customers used their internal frameworks constantly — at the largest customer, an analysis of the code generation prompts submitted in the first three weeks showed that 41% involved calls into or surrounding the company's internal framework libraries. The evaluation methodology had confirmed that the model performed well on the QA team's testing patterns. It had said nothing about the model's performance on the dominant category of actual production usage at the customers who relied most heavily on internal framework conventions.

The founding evaluation decision had not documented what population the held-out test set was intended to measure. The decision to evaluate against a held-out set from the curated dataset was made because it was the available data. Nobody had asked whether the curated dataset's distribution matched the distribution of production usage. Nobody had characterized what the major categories of production prompts would be across the customer base, measured how frequently each category would appear, and verified that the held-out set's composition reflected that distribution. The model had a 2% representation of the dominant failure category and an 87% accuracy score that was technically accurate and entirely misleading about the product's readiness for its intended use.

A 34-person content intelligence SaaS company built an AI-powered analysis platform for market research and competitive intelligence teams — a tool that processed large volumes of industry reports, analyst publications, earnings call transcripts, and news articles to extract strategic insights, competitive signals, and market trend summaries. The core pipeline used GPT-4o for document analysis, and the company had invested in building an automated evaluation harness to measure the quality of the model's outputs as they iterated on prompts, retrieval configurations, and post-processing logic. The automated judge was also GPT-4o — a different instance, with a separate system prompt designed for quality assessment rather than analysis — and had been calibrated against 800 human-rated examples drawn from the company's internal QA library, achieving 91% agreement with human raters on an overall quality rubric covering accuracy, completeness, and appropriate framing. The 91% calibration figure was treated as a validation that the automated judge was suitable as the primary evaluation signal for model and prompt changes going forward.

The automated judge ran on every prompt change, retrieval configuration update, and post-processing modification. Decisions about whether to ship changes were based primarily on whether the judge's quality scores improved or held steady, with human review reserved for cases where the judge flagged a significant regression. Over six months, the team iterated through fourteen prompt and retrieval configuration updates, each one evaluated by the judge and either shipped or reverted based on the judge's verdict. The judge reported consistent quality in the range of 4.0 to 4.4 out of 5, with no major regressions flagged across the fourteen iterations.

The signal that something was wrong came from two enterprise customer escalations in the same week, both involving document analyses that the customers described as missing the point. Specifically: both escalations involved analyses of multi-document sequences — earnings call transcripts evaluated across four consecutive quarters — where the model had described each quarter's content accurately but had failed to identify the directional pattern across the sequence. The customers needed to know that Q1 guidance was higher than Q2 actuals, which was lower than Q3 guidance, which had then been revised downward going into Q4 — a temporal sequence that required reasoning across the documents in order rather than analyzing each one independently. The model's outputs described each quarter without synthesizing the cross-quarter trend. The customers had flagged the same kind of failure in different documents.

The investigation pulled the evaluation records for the prior six months and filtered for analyses involving temporal sequence reasoning across multiple documents. The judge's quality scores for this category: an average of 4.1 out of 5, with 87% of evaluations rated as "high quality." The team pulled forty of the highest-rated examples in this category and asked a human reviewer — a former market research analyst who was not part of the evaluation team — to rate them against the original quality rubric. Her scores averaged 2.6 out of 5. Twenty-nine of the forty outputs she rated as insufficient were outputs the judge had rated as high quality. When she was asked to describe the failure pattern, she said the outputs were accurate about the individual periods but had not addressed what changed between periods, which was usually the most strategically relevant part of any multi-period analysis.

The root cause was architectural. GPT-4o and the GPT-4o judge shared a training lineage. Both had the same limitation on multi-document temporal sequence reasoning — a limitation that was well-known in the LLM community and that the team had noted in their own internal testing notes but had not formalized as a known failure category in the evaluation framework. The judge calibration had been performed on 800 examples from the internal QA library, which contained examples of strong single-document analyses and had not been constructed to include adversarial temporal sequence examples. The judge had been calibrated and found to agree with human raters at 91% — on the calibration set. On the calibration set, which did not include the failure category the subject model was systematically bad at. The judge could not identify failures the subject model consistently produced because the judge produced the same failures. Six months of model and prompt updates had been evaluated by a judge that was blind to the most important failure mode in the product's actual use case.

A 43-person SaaS company built an AI-powered document review platform for corporate legal teams — a tool that analyzed contracts, commercial agreements, and regulatory filings to flag non-standard clauses, identify material risks, and summarize key obligations across large document sets. The company had been operating since 2024 and had iterated through a series of model generations as foundation model quality improved. By 2026, the evaluation process was well-established: a fixed benchmark of 3,000 documents, each with human-assigned quality scores on a five-dimension rubric covering clause identification, risk characterization, obligation summary, cross-document consistency, and materiality ranking. The benchmark had been assembled in 2024 by a team of three contract lawyers who scored the original evaluation set. Subsequent model evaluations were performed by a team of two in-house legal content specialists who had been with the company since 2024 and had evaluated every model generation against the benchmark for nearly two years.

Over twelve model update sprints spanning sixteen months, the benchmark score rose from 72% to 84%. Each sprint evaluated two or three candidate model configurations and shipped the highest-scoring one. The team recorded each sprint's winner score and tracked the trend: the model was getting better. The legal content specialists provided qualitative feedback alongside the scores at each evaluation cycle, and their notes showed a consistent pattern — the model had been improving on clause identification accuracy and obligation summary coverage, while materiality ranking and cross-document consistency remained areas of ongoing development. The sprint notes described this as "continued progress on the harder dimensions."

In month fifteen, one of the two legal content specialists left the company to join a law firm. Her replacement was hired with experience in contract review but no prior exposure to the company's product or benchmark. Before beginning the sprint 12 evaluation, she spent a week calibrating against the benchmark using the original scoring rubric and a set of the original 2024 calibration examples that had been used to train the founding evaluation team. Her calibration agreement rate with the current team — measured on a fifty-example sample from the existing benchmark — was 61%. Well below the 85% threshold the company's evaluation process specified as the minimum inter-annotator agreement for a reliable benchmark.

She described two specific disagreements. First: outputs that described key clause relationships in detail but did not explicitly address how those clauses interacted under specific triggering conditions — what the original rubric called "interaction completeness." She was rating these as partial failures on the obligation summary dimension. The current team was rating them as passing. Second: outputs that correctly identified legal risks but listed them in the order they appeared in the document rather than ranked by potential financial or operational materiality — what the original rubric called "risk prioritization." She was rating these as failures on the materiality ranking dimension. The current team was rating them as neutral or passing. Both patterns had been in the model's characteristic output profile for at least eight sprints. The current team's sprint notes from sprint five described these patterns as "areas for improvement." By sprint eight the notes described them as "known limitations, model handles the core obligation surface well." By sprint ten there was no mention of them at all — they had become so familiar that they no longer registered as notable.

The new specialist re-evaluated the sprint 12 model against the full benchmark using her calibration from the original rubric. Her score: 69%. She then re-evaluated the sprint 1 model that was still available in the model registry. Her score for sprint 1: 74%. The model that had scored 72% on the original team's sprint 1 evaluation scored 74% under the rubric-anchored fresh evaluation. The model that had scored 84% on the original team's sprint 12 evaluation scored 69%. The benchmark had been measuring the primary team's recalibrated assessment of the model's output quality, not the model's quality against the original standard. Twelve sprints of selection for the highest-scoring candidate had selected for candidates the primary team's evolved calibration found acceptable — which, for two specific failure categories, was a lower standard than the original rubric required. The benchmark score trend was a record of how the team's tolerance for the model's failure modes had evolved, not a record of how the model's quality had improved.

Structural properties set by the model evaluation decision

Three structural properties are determined when a team establishes — or fails to establish — a model evaluation decision record: what the evaluation distribution selection reveals about the production blind spot created by the gap between test set composition and actual usage distribution, what the automated judge design determines about the shared failure mode surface between the judge and the evaluated model, and what the benchmark curation model determines about the label drift surface as the same annotators evaluate successive model generations against the same benchmark. None of these are labeled as decisions when the team assembles a held-out test set from the available data, chooses a same-family LLM as a convenient judge, or discovers that the benchmark annotators have been on the project since the benchmark was assembled. They emerge as operational failures when the model is deployed and customers encounter failures that the evaluation never measured, when the judge approves changes that human reviewers would reject, and when a benchmark score improvement cannot be reproduced by an evaluator who was not present for the drift.

Property 1: The evaluation distribution and the production blind spot. A held-out test set is a sample from a population. The evaluation measures the model's performance on that population. If the test set population does not match the production usage population, the evaluation is silent about the model's production performance — it answers a different question than the one the team needs answered. The production blind spot is not a random omission but a structured gap determined by the difference between the test set assembly process and the production usage distribution: QA-assembled test sets over-represent patterns QA engineers test; examples curated for quality over-represent well-behaved inputs; demonstration scenarios over-represent the product's happy path; internal usage examples over-represent the development team's workflows. The team that ships a model with 87% accuracy and a 2% representation of the dominant production failure category has not made an error in evaluation methodology — they have made an error in evaluation scoping, and the distinction matters because the fix is different. Methodology errors are fixed by improving the evaluation process. Scoping errors are fixed by characterizing the production usage distribution before assembling the test set, measuring the alignment between the test set composition and the production distribution, and treating distribution misalignment as a blocking issue rather than a known limitation. The production usage distribution is not static: it changes as the user base grows, as the product adds features, and as users find new use cases the product team did not anticipate. A test set that was well-aligned with production usage at launch may have a growing blind spot twelve months later if the production distribution has shifted and the test set composition has not been updated. The quarterly audit of test set distribution alignment is the structural fix — not a one-time assessment at benchmark assembly time but a recurring check that the evaluation is still measuring the right population. The LLM integration decision record connects at the architecture layer: the model selection, context window, prompt architecture, and retrieval configuration decisions that precede evaluation determine the input surface the model is expected to handle; the evaluation distribution must cover the full range of that input surface, including the edge cases and domain-specific inputs that are often underrepresented in general benchmark datasets but highly represented in the specific product's production usage.

Property 2: The judge model design and the shared failure mode surface. An automated LLM judge evaluates model outputs by generating an assessment of their quality. The judge's ability to identify a failure in the subject model's output depends on the judge's own ability to reason correctly about the assessment task — and if the judge shares the subject model's systematic weaknesses, it will assess outputs on the subject model's failure prompts using degraded judgment. The shared failure mode surface is the set of evaluation tasks on which the judge's quality assessment is unreliable because the judge has the same failure mode as the subject model. The circular evaluation architecture — same-family judge evaluating same-family subject — is the specific configuration that maximizes the shared failure mode surface: models from the same training lineage share systematic weaknesses by construction, and the calibration process that establishes the judge's agreement rate with human raters cannot detect shared failure modes that are absent from the calibration set. A 91% overall calibration agreement is compatible with a 40% agreement rate on the subject model's primary failure category, and a calibration set that does not include adversarial examples from that category will not surface the discrepancy. The structural fix is two-part: diversify the judge model's architecture or training lineage relative to the subject model, and construct the judge calibration set to explicitly include examples from the subject model's known failure categories. The known failure categories must be documented before calibration — which requires that the team has analyzed the model's failure modes and articulated them as testable categories, not just observed that "some outputs are not as good as others." The calibration agreement rate on the adversarial subset — the failure category examples — is the harder requirement than the overall calibration agreement rate, because it measures the judge's ability to do the specific thing the judge is most likely to fail at. A judge with 82% overall calibration agreement and 79% adversarial subset agreement is more reliable than a judge with 94% overall agreement and 48% adversarial subset agreement, and the standard aggregate calibration metric does not distinguish between these two profiles. The ML model serving decision record connects at the production monitoring layer: the serving decisions that determine which monitoring signals are available from the deployed model — user feedback collection, output logging, downstream metric correlation — provide the feedback channel through which production failure modes are discovered and added to the judge calibration set; a serving architecture that does not collect structured feedback on model output quality will not generate the adversarial examples the judge calibration requires.

Property 3: The benchmark curation model and the label drift surface. A benchmark is a measurement instrument. Like any measurement instrument, it must be calibrated periodically against an independent standard to ensure it is still measuring what it was designed to measure. Human evaluation benchmarks have a specific calibration failure mode that mechanical instruments do not: the annotators who operate the instrument develop familiarity with the model's output patterns over time, and that familiarity produces a gradual recalibration of their quality thresholds toward the model's characteristic output profile. The label drift surface is the set of quality dimensions on which the annotators' current threshold has diverged from the original rubric standard — the dimensions where the model's systematic errors have become so familiar that they no longer register as noteworthy. The drift is not detectable within the primary annotation team because every team member has experienced the same exposure and has recalibrated in the same direction. It is only detectable by comparison against an evaluator who has not been exposed to the model's failure profile — a fresh annotator calibrated against the original rubric, not against the current team's evolved standard. The structural fix is a periodic fresh annotator calibration check: at each major benchmark evaluation cycle, have a fresh annotator who has not previously evaluated this model score a sample of the current benchmark using the original rubric and measure their agreement with the primary team. A disagreement pattern that consistently clusters around specific quality dimensions is a drift signal on those dimensions. The fresh annotator protocol does not need to be large to be effective — a fifty-example sample scored by a single fresh annotator produces a reliable drift signal if the rubric is well-specified and the fresh annotator is calibrated against the original standard. The alternative structural fix — rotating a fraction of annotation work to fresh annotators on every evaluation cycle — is more robust but more operationally expensive; it converts the calibration check from a periodic audit into a continuous drift-detection mechanism. The data quality monitoring decision record connects at the data pipeline layer: the same principles that determine data quality monitoring for production data pipelines — defining quality dimensions, establishing measurement baselines, detecting anomalies and drift — apply to the evaluation data pipeline; the benchmark is a data asset with its own quality properties, and the annotation process is a pipeline that can produce systematic errors when its calibration drifts. The WhyChose extractor finds the model evaluation decisions in your AI session history — the evaluation harness design session where the held-out set was assembled from the available data without checking distribution alignment, the judge selection session where the same-family model was chosen for convenience, and the benchmark review session where the drift signal was present in the inter-annotator disagreement but was not recognized as a calibration failure.

The model evaluation ADR: five sections

Section 1: Evaluation distribution specification and production alignment documentation. Specify the population the evaluation is intended to measure and document the alignment between the test set composition and the production usage distribution. Before assembling the test set: characterize the production usage distribution by analyzing the major categories of inputs the model will handle — by domain, input type, complexity, and use case — and measure the expected frequency of each category in the production workload. Assemble the test set to reflect that distribution: categories that represent a large fraction of production usage must represent a corresponding fraction of the test set, not the fraction they happened to represent in the data collection source. Document the distribution alignment explicitly: for each major category in the production distribution, record its expected production frequency and its representation in the test set, and flag categories where the representation gap exceeds a defined threshold (for example, a category that represents more than 15% of production usage but less than 5% of the test set). A distribution gap at or above the flag threshold is a blocking issue: the test set should be augmented to close the gap before the evaluation is used for ship/no-ship decisions. Establish a quarterly test set audit: review the production usage distribution against the test set composition and update the test set when the distribution has shifted enough that one or more major categories have drifted outside the alignment threshold. Connect this section to the LLM integration decision record for the full input surface the model is expected to handle — the evaluation distribution must cover that surface, including the edge cases and domain-specific inputs that are characteristic of the specific product's usage patterns rather than the general benchmark datasets the model was pre-trained or fine-tuned on.

Section 2: Automated judge design policy and shared failure mode mitigation. Specify the judge model selection policy, the calibration set composition requirements, and the adversarial subset agreement threshold. Judge model selection: prefer a judge model from a different architecture family or training lineage than the subject model, particularly for tasks where the subject model has known systematic limitations. Document the rationale for the chosen judge model relative to this policy, including any trade-offs made for cost, latency, or API access reasons. Calibration set composition: the judge calibration set must include examples from each of the subject model's documented failure categories at a minimum 10% aggregate representation of the calibration set. Document the failure categories tested in the calibration set and the per-category sample sizes. Calibration thresholds: specify both an overall calibration agreement threshold and an adversarial subset calibration threshold. The adversarial subset threshold should be set at or above the overall threshold — 80% adversarial subset agreement when the overall threshold is 80%, not lower — because the adversarial subset is where the judge is most likely to fail and its performance there is the more important signal. Require re-calibration when: a new failure category is identified in production or through human review that was not represented in the original calibration set; the judge model is updated to a new version with changed capabilities; or the human review track (see Section 4) flags a category where the judge's high scores are consistently lower than human reviewer scores by more than 15 percentage points. Connect this section to the test strategy decision record for the framework for designing adversarial test inputs — the same principles that inform adversarial unit tests for application code apply to adversarial calibration examples for LLM judges; a failure mode that is known but not tested for is a risk that can be quantified and mitigated by explicit test design.

Section 3: Benchmark curation policy and label quality governance. Specify the benchmark assembly process, the rubric documentation requirements, and the drift detection protocol. Benchmark assembly: document the data sources, the sampling method, the rubric used for scoring, and the annotator calibration procedure used when the benchmark was assembled. The rubric must specify scoring criteria at sufficient detail that a new annotator calibrated only against the rubric documentation — not against the current team's calibration — can achieve 80% or higher agreement with the founding annotation team on a 50-example calibration test. Drift detection protocol: at each major model evaluation cycle (at least quarterly, or at each sprint if evaluation cycles are shorter), perform a fresh annotator check. The fresh annotator is an evaluator who has not previously scored this model or this benchmark, calibrated against the original rubric documentation rather than the current team's evaluation history. The fresh annotator scores a 50-example stratified sample from the benchmark. Calculate the fresh annotator's agreement rate with the primary team overall and per quality dimension. If the overall agreement rate falls below 80%, or if any individual quality dimension agreement rate falls below 70%, this is a drift signal on that dimension. Investigate the disagreements on flagged dimensions before using the primary team's scores for ship/no-ship decisions. Document the rubric anchoring process: when drift is confirmed on a dimension, re-anchor the primary team's calibration on that dimension using the original rubric and a set of ground-truth scoring examples before resuming benchmark evaluation. Connect this section to the data quality monitoring decision record for the monitoring principles — the benchmark is a data asset whose quality can drift silently, and the fresh annotator check is the quality monitoring alert that fires when the drift has exceeded the acceptable threshold.

Section 4: Regression test suite design and the production incident coverage requirement. Specify the regression test suite composition and the update requirement triggered by production incidents and customer escalations. Regression test suite: maintain a separate evaluation set of 20-50 examples specifically assembled from the model's documented failure categories and known edge cases. This set is not part of the main benchmark; it is the regression check run before any model update is shipped. The purpose of the regression set is not to measure overall quality but to verify that a proposed update has not regressed on the specific failure categories the team has already identified and characterized. Update requirement: when a production incident or customer escalation reveals a new failure category — an input type, reasoning task, or output pattern the model fails on that was not represented in the regression set — add at least five adversarial examples from that failure category to the regression set before the next model update evaluation. The regression set should grow monotonically across model generations, with each shipped generation's known failures represented in the test that follows. This creates a compounding record of the model's failure surface and ensures that each successive model generation is evaluated against the failures of all prior generations, not just the failures currently known at the time of the specific update. Connect this section to the feature flag decision record for the rollout model for model updates — model updates are often gated behind feature flags targeting a fraction of production traffic; the production traffic segment receiving the new model is the population from which new failure category examples will emerge, and the flag targeting model determines which users' inputs constitute the early signal; structuring the rollout to expose the new model to the full range of the production distribution — including the use cases most likely to surface new failure categories — rather than only to the lowest-risk user segment is a design choice with implications for the quality of the regression feedback the rollout produces.

Section 5: Evaluation refresh cadence and the benchmark staleness trigger. Specify the cadence at which the benchmark is reviewed for staleness and the criteria that trigger a full benchmark refresh. Review cadence: the benchmark is reviewed for staleness on a defined schedule — at minimum annually — and also on an event-triggered basis when specific signals appear. Event triggers for a staleness review: a fresh annotator check that confirms drift on two or more quality dimensions simultaneously; a production distribution audit that finds one or more major usage categories with representation below the alignment threshold; a model capability change — from a new foundation model version or a significant fine-tuning update — that introduces qualitatively different output characteristics the existing rubric's dimensions do not fully cover; or a product feature change that introduces new input types not covered by the current benchmark. Benchmark refresh process: a full benchmark refresh begins by re-characterizing the production usage distribution as of the refresh date, assembling a new set of test examples with composition aligned to the current distribution, and establishing new rubric-anchored ground-truth scores with fresh annotators. The existing benchmark is retained as a historical baseline — not replaced but supplemented — so that performance trends across generations can be computed on both the historical and refreshed benchmarks during the transition period. The transition period should run for at least two model evaluation cycles with both benchmarks scored, to confirm that the new benchmark does not introduce measurement discontinuities that would make the historical trend uninformative. Connect this section to the ML model serving decision record for the production monitoring infrastructure that generates the signals a staleness review depends on — the production distribution audit requires access to representative samples of production inputs, which requires that the serving infrastructure logs inputs in a form that is accessible for analysis, and the distribution characterization requires that the logging volume and sampling method produce a statistically representative sample of the full production usage distribution rather than a sample biased toward high-frequency or low-risk inputs.

FAQ

Further reading

• LLM integration decision record — the model selection, context window, prompt architecture, and retrieval configuration decisions that precede evaluation determine the full input surface the model is expected to handle; the evaluation distribution must cover that surface, including the edge cases and domain-specific inputs that are underrepresented in general benchmarks but heavily used in the specific product's production context.
• ML model serving decision record — the serving decisions that determine which monitoring signals are available from the deployed model — output logging, user feedback collection, downstream metric correlation — provide the feedback channel through which production failure modes are discovered and added to the regression test suite and judge calibration set; a serving architecture without structured output feedback will not generate the adversarial examples that an evaluation harness requires to stay current with the model's actual failure surface.
• Data quality monitoring decision record — the benchmark is a data asset whose quality properties can drift silently; the principles that determine data quality monitoring for production data pipelines — defining quality dimensions, establishing measurement baselines, detecting anomalies and systematic drift — apply to the evaluation pipeline; the fresh annotator calibration check is the quality monitoring alert that fires when the annotation calibration has drifted beyond the acceptable threshold.
• Test strategy decision record — model evaluation has structural analogues to software test strategy: the production-distribution test set corresponds to integration tests against representative production-like inputs; the failure mode catalogue corresponds to the adversarial unit test suite; the regression baseline corresponds to the test suite snapshot that defines what "passing" means before a change is merged; the same principles that determine test coverage requirements and regression prevention for application code determine evaluation coverage requirements and regression prevention for model updates.
• Feature flag decision record — model updates are often gated behind feature flags targeting a fraction of production traffic; the flag targeting model determines which users' inputs constitute the early signal for new failure category discovery; structuring the rollout to expose the new model to the full range of the production distribution — including the input categories most likely to surface new failures — rather than only to the lowest-risk user segment produces higher-quality regression feedback from the rollout period.
• Open-source extractor — find the model evaluation decisions buried in your AI chat history: the evaluation harness design session where the held-out set was assembled from the available data without checking distribution alignment, the judge selection session where the same-family model was chosen for convenience without documenting the adversarial calibration requirement, and the benchmark review session where the fresh annotator disagreement pattern was noted but not diagnosed as a calibration drift signal.