The content moderation decision record: why the moderation model you chose determines your false-positive suppression rate and your platform liability surface

A 25-person team built an online community platform for professional knowledge sharing — a space for practitioners in niche fields (supply chain logistics, healthcare administration, regulatory compliance) to discuss practice, share resources, and ask questions. The founding product decision was a post-moderation model: content published immediately, reviewed by community reports and a part-time moderator who checked the queue twice daily. The terms of service, written to match SaaS contract templates the founders had seen, included a clause that the platform would "actively moderate content to maintain a safe and professional environment." The founding session documented "post-moderation, human review, no blocklist" in a Notion page that was archived after the sprint.

For eleven months, the model worked. The communities were small, the users were professional, and the content was almost uniformly on-topic and civil. The part-time moderator reviewed the queue in under an hour each session and had found nothing to remove since the third month of operation. In the twelfth month, a coordinated campaign — apparently organized in a messaging group on another platform — targeted one of the healthcare administration communities with a wave of posts. The posts contained harassing content targeting specific named healthcare workers, including home address information for two community members and material that met the legal definition of targeted harassment under the jurisdiction where those members were located.

The first harassing post appeared at 9:17 PM on a Friday. The moderator's next scheduled review was Saturday morning at 10:00 AM. By 3:30 AM, 47 posts had been published. The harassed users had filed 23 reports, which sat in the unreviewed queue alongside the posts themselves. By the time the moderator logged in at 10:00 AM, the posts had been live for between six and thirteen hours. The harassed users had spent the night watching their personal addresses circulated on the platform, sending support tickets that received auto-replies, and posting increasingly urgent reports that remained unreviewed.

Two weeks later, the company received a legal notice from the attorney representing one of the harassed users. The notice alleged that the platform had failed to meet its contractual "actively moderated" obligation. The legal review that followed required the company to produce evidence of what "actively moderated" meant operationally. They had no documented definition. The terms of service said "actively moderated" but the founding session had never defined what violation window made a platform "actively moderated" versus inadequately moderated. The Notion page that documented the moderation decision was found in archived projects and offered no help — it said "post-moderation, human review" but nothing about response time commitments. The legal engagement lasted six months. The settlement included a required upgrade to continuous moderation coverage and a contractual commitment to maximum 4-hour violation window for harassment content. The cost — legal fees plus the staffing upgrade — exceeded the platform's revenue for the prior twelve months. None of this was in the founding session that documented "post-moderation, human review."

A 30-person B2B SaaS company built a project management platform with user-generated content: team members posted project updates, client feedback, and discussion threads inside the platform. Customers were enterprise and mid-market companies — regulated industries, professional services, financial services — and the platform's positioning emphasized reliability, professionalism, and appropriateness for client-facing communication. In month seven, the engineering team added an ML text classifier to detect off-topic, promotional, or inappropriate content in posts. The classifier was a fine-tuned BERT model trained on the team's manually labeled dataset of 12,000 posts. The founding session for the classifier documented "BERT classifier, threshold 0.85, auto-remove" in the project spec. The threshold was chosen because 0.85 precision sounded like a reasonable balance — the team did not calculate what 0.85 precision implied for the volume of false-positive enforcements at their actual post volume.

The platform processed approximately 2,800 posts per day across all tenants. At a 12% flag rate (the fraction of posts the classifier scored above 0.85), 336 posts per day triggered the classifier. At 0.85 precision, 15% of those — approximately 50 posts per day — were false positives: legitimate posts by professional users that the classifier incorrectly classified as inappropriate. The auto-remove action deleted those posts silently. Users whose posts were silently deleted received no notification that their post had been removed, because the notification system had been built for human-initiated moderation actions (the moderator clicked "remove and notify") and the classifier used the same action without the "notify" step.

Within three weeks of deploying the classifier, the customer success team flagged an increase in support tickets: "my post disappeared," "I can't see the update I just posted," "the system deleted my message." The support queue had grown from approximately 40 tickets per day to 95. Most tickets required a human support agent to inspect the moderation log, verify the false-positive classification, restore the post, and reply to the user with an explanation. Average handling time per moderation-related ticket was 12 minutes. The total cost was approximately 55 tickets per day × 12 minutes per ticket = 11 additional agent-hours per day, against a team of four support agents working 8-hour shifts. The moderation false positives were consuming 34% of the support team's capacity.

The engineering team lowered the threshold to 0.90 precision. At 0.90 precision, the false-positive rate dropped to 10%, reducing false-positive enforcements from 50 per day to approximately 34 per day. Support ticket volume dropped proportionally. But recall dropped from 73% to 58%: more genuinely inappropriate content slipped through. Three enterprise customers reported seeing content in shared project spaces that they found unprofessional, and one submitted a formal complaint referencing the platform's "professional environment" positioning in the sales materials. The team had moved the threshold without a documented calibration procedure, because the original decision had not specified what the acceptable false-positive rate was or what the acceptable slip-through rate for genuine violations was. There was no evaluation dataset to measure against. Each threshold adjustment was a guess validated by the support ticket volume two weeks later, not a principled choice against a documented target.

Six months after the classifier launch, the team hired a machine learning engineer who built the calibration process they should have had at the start: a labeled evaluation dataset of 5,000 posts, a precision-recall curve from threshold 0.70 to 0.99 at 0.01 increments, and a decision matrix that translated each threshold into a projected false-positive rate, a projected support ticket volume increase, and a projected slip-through violation rate. The operating threshold was chosen as 0.93 — the point where projected false-positive enforcement dropped to 15 per day (acceptable support queue impact) while recall remained above 65% (acceptable slip-through rate for the platform's risk tolerance). The calibration process also revealed that the classifier's training data had a systematic bias: it had been labeled predominantly by engineers who rated conversational, colloquial language as "inappropriate" more often than formal language, producing a classifier that suppressed informal communication between team members at a higher rate than formal project updates. The bias required a retraining run on a re-labeled dataset. All of this was discoverable in the founding session if the session had documented the acceptable false-positive rate, the support ticket volume projection at each threshold, and the calibration procedure — rather than just "threshold 0.85."

Structural properties set by the content moderation design decision

Three structural properties are determined when a platform decides how to moderate user-generated content. None appear explicitly in the founding session that picks a moderation model — they are the operational consequences of choices made under the pressure of launching a community or UGC feature before the moderation infrastructure exists to sustain it at scale.

Property 1: The moderation model and the violation window. The moderation model determines when content is reviewed relative to publication and what triggers review. The four primary models — pre-moderation, post-moderation, reactive moderation, and automated moderation — produce different violation windows (the time between when violating content is published and when it is removed) and different friction profiles for legitimate users.

Pre-moderation holds content in a queue before publication. The violation window is zero — no violating content reaches the community because all content is reviewed before it becomes visible. The cost is throughput: moderator review capacity must match or exceed the content submission rate, or the queue grows and legitimate users experience unacceptable delays before their content is visible. The pre-moderation model is correct when the regulatory or contractual context makes a zero violation window the overriding requirement: children's platforms (COPPA and KOSA in the US, DSA Article 28 in the EU require "effective" protection for minors), financial services platforms where content might constitute unlicensed financial advice, healthcare platforms where clinical misinformation could cause patient harm. The pre-moderation ADR must specify the maximum review latency SLA — the time between submission and a moderation decision — and the staffing model required to maintain that SLA under peak submission volume.

Post-moderation publishes content immediately and reviews it through proactive moderator sweeps, automated classifier flags, or user reports. The violation window is bounded by the detection mechanism and the moderator response time — hours in the professional-community failure above, minutes with a well-tuned classifier and a staffed moderation queue. Post-moderation is the correct model when user friction is the primary concern and the platform context accepts some violation window: developer forums, professional community platforms, general discussion communities. The post-moderation ADR must specify the violation window target by severity tier, the detection mechanisms that bound the window, and the moderator throughput required to meet the target. A 4-hour violation window for harassment content at 2,800 posts per day with an estimated 0.1% harassment rate requires 2-3 incoming harassment posts per day to be caught within 4 hours — achievable with daily moderation sweeps. A 4-hour violation window for harassment content on a platform with 280,000 posts per day requires either automated detection or a continuous moderation queue with multiple shifts, because 280 harassment posts per day distributed over 24 hours produce more than one harassment post per hour on average.

Reactive moderation relies on user reports as the primary detection mechanism. The violation window is bounded by the report rate (the time between content publication and first report) plus the moderator response time (the time between report and enforcement). Reactive moderation is the model with the lowest operational cost at low scale and the highest liability exposure at scale — platforms with high user volume generate enough genuinely violating content that relying on reports alone produces long violation windows for content that is not reported quickly. The reactive model is acceptable only when community self-policing is reliable (tight-knit communities where off-topic content is reported within minutes) and the content taxonomy excludes categories with mandatory reporting obligations (because CSAM is never acceptable to leave up pending report — it must be proactively detected). The notification system decision record documents the alerting infrastructure for moderators; the content moderation ADR must specify whether the report notification path has a different latency SLA than other notifications — report notifications to on-call moderators should arrive within seconds, not within the notification system's default batch window.

Automated moderation uses a classifier to make enforcement decisions at publication time or post-publication. The violation window for high-confidence classifier detections is near-zero; the violation window for content the classifier misses is unbounded (it defaults to the reactive or proactive sweep model for content below the threshold). Automated moderation is not a replacement for human moderation — it is a filter that handles high-confidence signals at volume so that human moderators can focus on ambiguous cases, appeals, and categories the classifier cannot reliably detect (context-dependent harassment, novel coordinated campaigns, jurisdiction-specific content). The classifier flag rate and the false-positive rate together determine how many posts per day enter the human review queue from the classifier alone — the human review queue sizing must account for classifier-generated volume in addition to user-reported volume.

Property 2: The classifier threshold and the appeal workflow capacity. The classifier threshold is not a single number — it is a function of the precision-recall curve evaluated against a labeled evaluation dataset, and the correct operating point on that curve is determined by the false-positive rate the platform is willing to impose on legitimate users. Choosing a threshold without specifying the acceptable false-positive rate is equivalent to setting a pricing model without specifying the acceptable churn rate: the number is meaningful only in context of the constraint it is meant to satisfy.

The false-positive rate (1 - precision at the chosen threshold) multiplied by the classifier flag rate and the daily post volume gives the number of false-positive enforcement actions per day. Each false-positive enforcement action generates a probability of an appeal — not all users appeal incorrectly moderated content, but a significant fraction do, particularly on platforms where the content was important to the user (a detailed project update, a professional reference, a carefully researched answer). The appeal workflow must have throughput equal to or greater than the inbound appeal rate, or appeals accumulate in the queue and resolution latency grows, producing secondary support tickets from users asking about the status of their appeal.

The appeal workflow design has three core decisions: the escalation path (who reviews an appeal, in what order, with what authority to reverse the classifier's decision); the resolution latency SLA (how long a user must wait before their appeal is resolved, during which time their content is suppressed); and the feedback loop to classifier retraining (whether resolved appeals are added to the classifier's training dataset so that systematic false-positive patterns are corrected in the next training run). The escalation path must be specified before the classifier is deployed, because the appeal volume will exceed the team's capacity to handle it ad hoc if the classifier is widely deployed before the appeal workflow exists. A single threshold change that reduces false positives by 20% reduces appeal volume by 20% and reduces the appeal workflow staffing requirement proportionally — the threshold and the appeal workflow are co-designed, not independent decisions.

The ML model serving decision record documents how the classifier is deployed and how prediction latency is managed; the content moderation ADR must specify the classifier update policy — how often the model is retrained, whether the threshold is re-evaluated at each retraining run, and who owns the threshold decision (product vs. engineering vs. legal). Classifier updates that change the precision-recall curve change the false-positive rate at the operating threshold, potentially increasing appeal volume without any explicit threshold change. The appeal workflow staffing must be notified of each classifier update with a projection of how the update changes expected appeal volume. The queue and messaging decision record documents the infrastructure for the moderation and appeal queues; the content moderation ADR must specify whether the moderation queue and the appeal queue share infrastructure or are separated — shared infrastructure risks appeal latency degrading when primary moderation volume spikes, because both queues compete for the same consumer throughput.

Property 3: The severity taxonomy and the regulatory compliance surface. The severity taxonomy is the classification of content categories by the harm they cause and the enforcement response they require. Spam requires removal. Harassment requires removal and may require user suspension. CSAM requires removal, content preservation, mandatory legal reporting to NCMEC within 24 hours, and user account suspension with law enforcement notification. The severity taxonomy determines the moderator training required for each category, the escalation path from automated detection to legal team notification, and the regulatory reporting timeline that binds the platform regardless of its internal policy choices.

The regulatory compliance surface for content moderation is jurisdiction-specific and category-specific. In the United States, CSAM reporting to NCMEC is mandatory (18 U.S.C. § 2258A) with a 24-hour reporting window from the moment the platform becomes aware of the content. "Aware" includes classifier detection — a classifier that detects CSAM with high confidence makes the platform legally aware of that content, triggering the 24-hour reporting clock. The platform must preserve the content (not delete it) until the report is filed and confirmed, because the report must include the content itself as evidence. The content preservation obligation during the reporting window conflicts with the default moderation action of immediate removal — the CSAM enforcement path must remove the content from visibility (so no user can see it) while preserving it in a restricted storage location accessible only to the legal team and the classifier detection system. The data retention decision record documents the general content retention policy; the content moderation ADR must specify a separate CSAM retention policy that overrides the general policy for the reporting window and specifies the destruction procedure after the report is confirmed.

In the European Union, the DSA imposes additional obligations beyond CSAM. Platforms with more than 2.5 million average monthly active users in the EU are designated as "very large online platforms" and are required to conduct annual risk assessments of systemic risks from their platform, including risks arising from content recommendation systems, implement risk mitigation measures, and submit to independent audits. The NetzDG in Germany (which applies to platforms with more than 2 million registered users in Germany regardless of overall size) requires removal of "clearly illegal" content — defined by reference to specific provisions of German law covering hate speech, Holocaust denial, and similar categories — within 24 hours of becoming aware of it. The NetzDG reporting requirement applies separately from the DSA. A platform that is large enough to trigger NetzDG obligations but did not account for NetzDG in its content moderation ADR faces removal timelines that require a continuous moderation queue in European time zones — a staffing requirement that is not derivable from the platform's general post-moderation model.

The audit log decision record documents the general audit log model; the content moderation ADR must specify a separate moderation action audit trail that records every moderation decision with the content at the time of moderation, the moderator or classifier that made the decision, the enforcement action taken, the timestamp, the appeal outcome if applicable, and the regulatory reporting outcome for CSAM-tier content. The moderation audit trail must be immutable and retained for the duration required by the longest applicable regulatory obligation — in the US, NCMEC recommends retaining CyberTipline-reported content records for at least three years; the NetzDG requires retaining records of removal decisions for ten weeks (for transparency reporting purposes) and two years (for regulatory oversight purposes). These retention requirements differ from the general data retention policy and from the GDPR's data minimization principle — the moderation audit trail is a legal hold that overrides the standard retention schedule.

What the founding session records and what it omits

The founding content moderation session typically records the moderation model ("post-moderation, human review"), the classifier model and threshold if one is used ("BERT, threshold 0.85"), and sometimes the content categories that are prohibited ("no spam, harassment, or illegal content"). What it does not record is the violation window target by severity tier, the false-positive rate target and the support ticket volume projection at the chosen classifier threshold, the appeal workflow design and throughput sizing, the regulatory compliance model for CSAM and any jurisdiction-specific obligations, the moderator escalation path for high-severity content, or the classifier update policy and its effect on the appeal workflow.

The omissions are consequential because they determine the operational cost of moderation at scale, the legal exposure when moderation fails to meet contractual or regulatory standards, and the user experience for the fraction of legitimate users whose content is incorrectly moderated. A post-moderation model with no violation window commitment exposes the platform to liability when the actual violation window exceeds what the context (contractual, regulatory, or reputational) requires. A classifier threshold without a false-positive rate target exposes the support team to a calculable volume of incorrect moderation tickets that was never staffed against. A regulatory compliance model that does not enumerate jurisdiction-specific obligations exposes the platform to reporting failures that carry criminal penalties.

The content moderation decision record does not need to be long. It needs to answer five questions: what is the moderation model for each severity tier, what is the violation window target for each tier, what is the acceptable false-positive rate and what support ticket volume does it imply at current post volume, what is the appeal workflow design and how is it sized, and which regulatory reporting obligations apply and what are their timelines. Five answers written down in the founding session avoid months of legal exposure, unplanned staffing costs, and support queue failures that each trace back to a decision that was made but not recorded.

The WhyChose decision extractor finds these founding moderation sessions in your ChatGPT and Claude export — the "how should we handle reports?" thread, the "what threshold should we use?" conversation, the "how do we handle GDPR and CCPA?" research session. It extracts the decision and the trade-off that was considered, without the surrounding context of clarification questions and scratch thinking that buries the actual choice.

The five ADR sections for a content moderation decision

Section 1: Moderation model selection and enforcement timing. Specify the moderation model for each content severity tier: which tiers use pre-moderation (content held before publication), which use post-moderation (content published then reviewed), and which use automated moderation (classifier makes the enforcement decision). Specify the violation window target for each tier — the maximum time between content publication and enforcement action — and the detection mechanism that bounds the window (proactive moderator sweep, classifier flag, user report). Specify the moderator shift coverage required to meet the violation window target at projected content volume: hours of coverage per day, number of moderators per shift, maximum queue depth before on-call escalation. Specify the escalation path when the moderator queue exceeds capacity — who is notified, what actions are taken to reduce queue depth, and what the maximum acceptable queue depth is before the platform's violation window commitments are at risk.

Section 2: Content severity taxonomy and category definitions. Enumerate the severity tiers and the content categories within each tier. Example tiers: Tier 1 (spam, off-topic) — remove, no user notification required, no account action; Tier 2 (harassment, hate speech) — remove, user notification with reason, warning on account, second violation triggers suspension; Tier 3 (graphic violence, illegal content) — remove, immediate user suspension, law enforcement referral assessment; Tier 4 (CSAM) — remove from visibility, preserve in restricted storage, mandatory NCMEC CyberTipline report within 24 hours, immediate account suspension, law enforcement notification. Specify the category definitions precisely enough to train moderators and classifier labelers consistently — "harassment" must be defined with examples and boundary cases, not left to moderator judgment, because inconsistent categorization produces inconsistent enforcement and creates grounds for user appeals on disparate treatment. The authorization model decision record documents the permission model; specify which moderator roles have authority to make Tier 3 and Tier 4 decisions — typically only senior moderators or legal team members, because Tier 3 and Tier 4 decisions have regulatory and legal consequences that require higher-authority review.

Section 3: Classifier specification and calibration procedure. Specify the classifier architecture, the training dataset composition (size, labeling methodology, label taxonomy alignment with Section 2's severity taxonomy), and the evaluation dataset (size, composition, how it differs from the training dataset to test generalization). Specify the operating threshold and the precision, recall, and false-positive rate at that threshold on the evaluation dataset. Calculate the daily false-positive enforcement volume at the current post volume and the current classifier flag rate: (post_volume × flag_rate × false_positive_rate) = false-positive enforcements per day. Specify the acceptable false-positive enforcement rate — the number of incorrect enforcement actions per day that the platform accepts as within operational tolerance — and confirm that the chosen threshold meets that rate. Specify the calibration procedure for threshold updates: when is the threshold re-evaluated (after each model update, after each major content volume change, quarterly), who owns the threshold decision, and what is the process for communicating threshold changes to the appeal workflow team. Specify the classifier update policy: how often is the model retrained, what triggers an off-schedule retraining (significant precision-recall degradation, new content category, regulatory requirement), and how are appeal outcomes fed back into the training dataset.

Section 4: Appeal workflow design and throughput sizing. Specify the appeal workflow for each moderation action type: what appeal mechanism is available to the user (in-product form, email, support ticket), what information the user must provide, who reviews the appeal (first-level review, escalated review), what authority the reviewer has (reverse the decision, uphold the decision, escalate to legal team), what the resolution latency SLA is (the time from appeal submission to resolution), and what notification is sent to the user at each stage. Calculate the inbound appeal volume: (false-positive enforcements per day) × (appeal rate) = appeals per day. The appeal rate is the fraction of incorrectly moderated users who submit an appeal — typically between 20% and 60% depending on how obvious the error is and how accessible the appeal mechanism is. Specify the staffing required to meet the resolution latency SLA at the projected appeal volume. Specify the feedback loop from resolved appeals to classifier improvement: how are appeal outcomes tagged (false positive confirmed, enforcement upheld), how are confirmed false positives added to the classifier evaluation dataset, and at what rate are confirmed false positives expected to trigger a classifier update. The logging strategy decision record documents the operational logging infrastructure; the moderation appeal workflow requires structured logs for each appeal decision — the content at moderation time, the classifier score, the moderator's appeal decision, the timestamp — to support classifier analysis and regulatory audits.

Section 5: Regulatory compliance model and mandatory reporting procedures. Enumerate the jurisdictions where the platform operates or has users and the content moderation regulatory obligations in each jurisdiction. For each jurisdiction, specify: the content categories subject to mandatory reporting obligations (CSAM in the US, UK, EU, and most jurisdictions; hate speech and Holocaust denial in Germany under NetzDG; terrorist content in the EU under TERREG), the reporting authority (NCMEC in the US, IWF in the UK, national authority in EU member states), the reporting timeline (24 hours for CSAM in the US), the content preservation obligation during the reporting window and the storage isolation requirement, the record retention requirement for moderation decisions (two years for NetzDG in Germany, three-year NCMEC recommendation for CSAM records), and the transparency reporting requirement (NetzDG requires biannual transparency reports, DSA requires annual transparency reports for very large platforms). Specify the detection mechanism for each mandatory-reporting category — CSAM must be detected proactively using PhotoDNA or equivalent perceptual hash matching against a known CSAM hash database (provided by NCMEC and IWF), because relying on user reports alone does not satisfy the "becoming aware" standard in most jurisdictions for content the platform could have detected automatically. Specify the escalation path when content in a mandatory-reporting category is detected: who on the legal team is notified, who files the report, how is the 24-hour window tracked, and who is responsible for confirming that the report was filed and received. The event-driven architecture decision record documents the event bus infrastructure; mandatory-reporting detection events must be published to a high-priority queue with guaranteed delivery and at-least-once processing — a missed CSAM detection event that results in a failure to file a required CyberTipline report is a federal offense, and the infrastructure must be designed to make missed events impossible, not merely unlikely. The API versioning decision record documents the versioning model for platform APIs; if the platform exposes a moderation API for enterprise customers to configure moderation settings, the API surface must be versioned with the same rigor as product APIs — a breaking change to a moderation API endpoint that causes a customer's integration to stop submitting reports creates the same liability exposure as the platform itself failing to report.

Further reading

• The notification system decision record — the alerting infrastructure for moderation queue notifications; moderator report alerts require sub-second delivery SLAs that differ from standard notification batching.
• The data retention decision record — how the CSAM preservation obligation and NetzDG retention requirements override the general data retention policy and interact with GDPR erasure requests.
• The authorization model decision record — the permission model for moderator roles; Tier 3 and Tier 4 moderation decisions require higher-authority permissions than standard content removal.
• The logging strategy decision record — the operational logging infrastructure that must be separated from the moderation audit trail to ensure the audit trail meets its immutability and retention requirements.
• The audit log decision record — the immutability contract and actor pseudonymization model for the moderation audit trail; moderation decisions are audit events that must satisfy both the NetzDG retention requirement and the GDPR right-to-erasure conflict.
• The ML model serving decision record — how the classifier is deployed and updated; classifier updates that change the precision-recall curve change the false-positive rate at the operating threshold without any explicit threshold change.
• The queue and messaging decision record — the infrastructure for the moderation queue and appeal queue; mandatory-reporting detection events require guaranteed-delivery queues with at-least-once processing semantics.
• The event-driven architecture decision record — the event bus model for moderation events; CSAM detection events must be published to a high-priority queue that cannot be silently dropped.
• The API versioning decision record — the versioning model for moderation APIs exposed to enterprise customers; breaking changes to moderation API endpoints create the same regulatory exposure as platform-level reporting failures.
• WhyChose decision extractor — finds the founding "how should we handle reports?" and "what threshold should we use?" sessions in your ChatGPT or Claude export and extracts the decision and the trade-offs that were considered.