The GraphQL subscription decision record: why the real-time transport model you chose determines your connection scale ceiling and your message delivery guarantee surface

Published 2026-07-14 · WhyChose

GraphQL subscription decisions are made when the team needs to push real-time updates to clients and wants a typed, schema-governed alternative to ad-hoc WebSocket messages or polling. The AI session that produces the subscription decision is thorough and technically grounded: it evaluates whether to use WebSocket with the graphql-ws protocol or the older subscriptions-transport-ws library; selects a pubsub backend (in-process EventEmitter for single-server deployments, Redis for horizontal scale); wires the subscription type into the GraphQL schema with the correct async iterator resolver pattern; and ships a working live-updating UI where events from the server arrive in the client in real time. The session deploys the first subscription endpoint, verifies that events are delivered, and confirms that the chosen pubsub backend integrates correctly with the existing infrastructure. The session produces a working subscription system whose correctness at founding scale — tens of concurrent connections, one server process, a manageable event rate — is not in doubt.

What the AI session does not produce is the subscription system's second half. The session answers "which transport protocol and pubsub backend?" and delivers a working real-time feature. It does not ask: what is the connection scale model — how many concurrent subscription connections can a single server process sustain before the OS file descriptor limit, the event loop's per-connection overhead, or the pubsub backend's fan-out cost becomes the binding constraint, and how does that ceiling interact with the team's horizontal scaling model if subscriptions require sticky sessions? What is the backpressure policy — when events are produced faster than a slow client can consume them, how many unconsumed events does the server hold in memory per connection before applying a slow-consumer action (drop oldest events, drop newest events, disconnect the client), and what is the maximum memory budget for event buffering across all concurrent connections? What is the message delivery guarantee — are subscriptions at-most-once (the default for most WebSocket implementations, where events that arrive while the client is disconnected or falling behind are discarded), at-least-once (requiring an acknowledgment from the client and server-side event persistence), or effectively-once (requiring a cursor-based replay model where the client can reconnect and receive missed events from a specific point in the event stream)? How do long-lived subscription connections interact with JWT token expiry — when a client's access token expires while a WebSocket connection is open, does the server close the connection, continue delivering events with an expired credential, or support in-flight token rotation? Each of these questions has an answer that is not derivable from "we use GraphQL subscriptions with graphql-ws over Redis pubsub" as the team frames the transport choice. Each answer determines whether subscriptions remain the correct real-time transport as concurrent user count grows and event rates increase, or become the ceiling on server memory, the source of OOM incidents, and the anchor to a connection affinity model whose cost was never estimated before it became the dominant infrastructure constraint. The answers exist in the AI sessions. They are the operational commitments behind the transport model choice. They are almost never written down.

Two ways GraphQL subscription decisions produce the wrong outcome at scale

The WebSocket connection scale ceiling

A developer tools startup adds GraphQL subscriptions in their second year to power a real-time activity feed in their primary dashboard. The decision is well-founded and technically sound: the team's GraphQL API is already established, the dashboard polls an activityFeed query every fifteen seconds, and subscriptions replace the polling pattern with a push model that reduces server load and improves the user experience by making updates appear immediately. The founding session selects graphql-ws over WebSocket (the current protocol recommendation, replacing the deprecated subscriptions-transport-ws), uses an in-process Node.js EventEmitter as the pubsub backend (appropriate for a single server process), and ships a working subscription within three days. At launch, the subscription powers the activity feed for the team's 60-person beta cohort. Concurrent subscription connections peak at 38 during a beta test session. The experience is excellent and the product team prioritizes the real-time feed as a key differentiator in the v1 launch.

Eighteen months after the beta, the startup has grown to 4,200 active users. The dashboard's activity feed subscription is the primary real-time surface, and usage patterns have shifted from the beta cohort's concentrated usage windows to a geographically distributed user base with overlapping work hours across time zones. On a Tuesday afternoon, a concurrent connection spike during a product launch announcement by a prominent customer peaks at 4,180 simultaneous subscription connections. At 4:23 PM, Datadog alerts fire on the Node.js process: the server is refusing new WebSocket connection upgrade requests with error code EMFILE — too many open files. Existing connections continue to function. New connections cannot be established. Users who close and reopen their dashboard during this period find their activity feed is not loading, with no explanation visible in the UI.

The root cause is the OS-level file descriptor limit. Each WebSocket connection requires a file descriptor — an OS resource that is bounded by the process's ulimit -n setting, commonly defaulted to 1,024 or 4,096 in many Linux distributions. The startup's server process had the default 4,096 file descriptor limit. At 4,180 simultaneous subscription connections — each requiring a file descriptor for the WebSocket socket — plus the file descriptors consumed by the server process itself (open log files, the database connection pool, the Redis connection, the Node.js runtime internals), the process hit the OS limit. The team raises the ulimit -n to 65,535 (a common adjustment for high-connection servers) within the hour, which restores new connection acceptance immediately.

The immediate fix reveals the structural problem: the team's horizontal scaling model assumes stateless HTTP services that can be load-balanced round-robin. The activity feed subscription uses an in-process EventEmitter as the pubsub backend — events are published to the EventEmitter inside one server process and delivered to subscription connections held by that same server process. If the team adds a second Node.js server process to distribute connection load, a user connected to server process A will not receive events published by an action processed by server process B, because process B's EventEmitter is not visible to process A. The only way to distribute subscription connections across processes while maintaining event delivery is to replace the in-process EventEmitter with a shared pubsub backend — Redis pubsub, NATS, or a message broker — that all server processes subscribe to and publish through. The engineering estimate to migrate from in-process EventEmitter to Redis pubsub is two weeks: designing the Redis channel naming schema, implementing the Redis subscriber in each server process, updating the subscription resolver to publish to Redis rather than the EventEmitter, and load testing the Redis fan-out behavior under the expected concurrent connection count. The migration to Redis pubsub also introduces a new operational dependency: the Redis instance becomes a high-availability requirement for the subscription system, because a Redis failure causes all active subscription connections to stop receiving events. The Redis instance requires a replication configuration, a failover mechanism, and monitoring that was not in the original infrastructure plan.

The founding session that chose graphql-ws and an in-process EventEmitter did not address: what is the maximum concurrent subscription connection count per server process at the expected file descriptor limit, and what is the per-connection memory overhead of a subscription connection in the chosen pubsub model? What is the pubsub backend's fan-out model — does it scale horizontally with the server process count, and if not, what is the migration path and its engineering cost? How do subscription connections interact with the load balancer — does the load balancer require sticky sessions (session affinity) to route a client's reconnection requests to the same server process that holds the client's subscription state, and if so, what happens to in-flight subscription connections when a server process restarts or is removed from the load balancer pool? Three questions that the founding session deferred as "we'll deal with scale when we have scale users" — which is a reasonable deferral — but did not document as open questions with a scale threshold at which they must be answered. At 4,000 concurrent connections, the team is answering the open questions under a production incident rather than during a planned migration.

The unbounded slow-consumer OOM incident

A fintech startup adds GraphQL subscriptions for live trade execution updates in their third year. The use case is high-value: traders on the platform watch their position portfolio update in real time as executions are confirmed, and the previous polling-based approach introduced a 3–5 second delay between execution confirmation and UI update that was unacceptable for active traders during volatile markets. The founding session designs a tradeExecuted subscription that pushes an event to all subscribers when a trade execution is confirmed for their account. The session uses graphql-ws over WebSocket, a Redis pubsub backend (the team already uses Redis for session caching and is familiar with it), and Apollo Server for the GraphQL endpoint. The subscription is tested at the team's peak-day load (roughly 180 concurrent traders active simultaneously), performs correctly, and ships to production. The real-time execution updates become a product differentiator that the sales team uses in enterprise demos.

Eight months after the subscription launches, the startup's user base has grown and a market volatility event occurs: a major index experiences a 3.2% intraday move driven by a macroeconomic announcement at 2:00 PM EST. In the thirty seconds following the announcement, the platform processes 2,847 trade executions across 312 active trader accounts. Each execution produces a tradeExecuted event published to Redis, which fans out to all subscription connections subscribed to the executing trader's account channel. Because the executions are concentrated among the most active traders — 40 traders with algorithmic strategies account for 2,100 of the 2,847 executions — the subscription connections for those 40 accounts receive between 30 and 80 events each in the thirty-second window. The Node.js server process's heap consumption grows from 380 MB at 1:59 PM to 2.1 GB at 2:03 PM. At 2:07 PM, the process receives a SIGKILL from the OOM killer. All 312 active subscription connections drop simultaneously. The process restarts in 12 seconds. Traders see their real-time position displays go blank during the most volatile 7-minute window of the trading day.

The post-mortem investigation identifies two contributing factors. The first is the absence of a per-connection event buffer limit. The graphql-ws library's internal AsyncIterator buffers events that have been produced by the subscription resolver but not yet sent to the client. When a client's WebSocket connection is slow — a mobile client on a congested network, a corporate laptop running a memory-heavy CI suite during a trading session — events accumulate in the AsyncIterator's internal buffer. Among the 312 active connections during the volatility event, six connections belonged to mobile clients using the startup's iOS app over cellular data. These six connections were consuming events at a rate of 3–7 events per second, while the tradeExecuted pubsub channel was producing 40–80 events per second during the peak window. The AsyncIterator buffer for each of the six slow connections accumulated 300–500 events within the 30-second burst window, each event containing the full execution payload (average 2.8 KB, including the account balance delta, position update, and execution metadata). Six connections × 450 average buffered events × 2.8 KB per event = 7.6 MB of buffered payloads for the slow connections alone — negligible in isolation. The second contributing factor is that the tradeExecuted subscription resolver was not implementing any backpressure control. The resolver published every execution event to Redis immediately, regardless of the consumer's consumption rate. Redis pubsub is fire-and-forget: there is no acknowledgment from subscribers, no backpressure from slow consumers to the publisher, and no mechanism for a slow subscriber to signal that it needs fewer events. The combination of Redis's fire-and-forget delivery and the AsyncIterator's unbounded buffering meant that the six slow mobile connections each held an AsyncIterator with a growing event queue that the event loop could not drain fast enough. The event queue grew until Node.js's heap was exhausted.

The founding session that chose graphql-ws and Redis pubsub did not address: what is the maximum per-connection event buffer size before the slow-consumer policy is applied, and what is that policy (drop oldest events, drop newest events, or disconnect the client)? What is the maximum total memory budget for in-flight event buffers across all concurrent connections at peak concurrent user count and peak event rate — and does the server process's heap allocation accommodate that budget plus the baseline application memory? What is the monitoring approach for detecting buffer accumulation before it reaches the OOM threshold — what metric measures per-connection buffer depth, and at what depth does an alert fire? Three operational commitments that were implicit in the subscription system design and that did not appear in any document until they were revealed by a production OOM incident during the highest-stakes usage window of the trading year.

Three structural properties that GraphQL subscription decisions determine

The transport model and the connection scale ceiling

The connection scale ceiling for a GraphQL subscription deployment is determined by three interacting constraints: the OS file descriptor limit per server process, the event loop overhead per active connection, and the pubsub backend's fan-out cost per event. Each constraint imposes a different type of ceiling and responds differently to horizontal scaling, and the transport model choice — WebSocket, Server-Sent Events over HTTP/2, or long-polling — determines the initial position of each constraint before any scaling decision is made.

WebSocket connections are full-duplex TCP connections, each consuming a file descriptor on the server for the lifetime of the connection. The practical ceiling for WebSocket connections on a single Node.js process is bounded by the configured ulimit -n value minus the file descriptors consumed by the process itself (database connections, log file handles, the Redis connection, and Node.js runtime internals typically consume 50–200 file descriptors). On a Linux server with ulimit -n set to 65,535, a single Node.js process can sustain approximately 60,000–64,000 concurrent WebSocket connections at the OS level — but the event loop overhead per connection imposes a lower practical ceiling. Each active WebSocket connection requires the event loop to process incoming ping frames, check for pending events from the pubsub backend, serialize and frame outgoing events, and handle connection state updates. At 10,000–20,000 concurrent connections on a single Node.js process, event loop lag typically increases to the point where subscription event delivery latency degrades noticeably (from single-digit millisecond delivery latency to 100–300ms), even without a slow-consumer accumulation problem. The practical connection scale ceiling for a Node.js process running graphql-ws is therefore a function of both the OS file descriptor limit and the event loop's per-connection overhead at the target delivery latency.

SSE (Server-Sent Events) over HTTP/2 changes the connection scale economics in a specific way: HTTP/2 multiplexing allows multiple SSE streams to share a single TCP connection. A browser client can open multiple SSE streams to the same origin over a single HTTP/2 connection, and the server sees multiple HTTP/2 streams rather than multiple TCP connections. This reduces the file descriptor consumption per logical subscription stream — multiple streams share a single file descriptor for the underlying TCP connection — and reduces the TCP connection overhead at the OS level. However, SSE is unidirectional (server-to-client only), which means the graphql-ws protocol's bidirectional features — client-to-server connection_init with parameters, in-flight token rotation via connection_init re-send, client-initiated ping — are not available. Teams that need these features cannot use SSE as a drop-in replacement for WebSocket. For subscriptions that are genuinely unidirectional (the server pushes events, the client only reads them), SSE over HTTP/2 is a viable alternative with better load balancer compatibility, built-in reconnection with Last-Event-ID cursor replay (the browser's EventSource API automatically sends the Last-Event-ID header on reconnection, enabling cursor-based replay without application-level implementation), and lower per-stream OS resource consumption at scale.

The pubsub backend's fan-out cost per event determines the per-event server overhead independent of the per-connection ceiling. An in-process EventEmitter delivers an event to N subscribed listeners by calling each listener function synchronously — the delivery cost scales linearly with the number of subscribed connections in the current process. At 1,000 connections and a high-rate event channel (1,000 events per second), the EventEmitter must call 1,000 listener functions per event, serializing and framing the payload for each connection — 1 million listener calls per second in the worst case, which saturates the event loop before the file descriptor limit is reached. A Redis pubsub backend shifts the fan-out responsibility to Redis: the server process subscribes to a Redis channel, receives each event from Redis, and then fans out to the WebSocket connections subscribed to that channel in the current process. The fan-out cost per event in the Redis model is bounded by the number of subscriptions in the current process (not across all processes), and Redis's delivery-to-subscriber latency adds a network round trip (typically 0.5–2ms for a co-located Redis instance) to the event delivery path. The per-event cost model must be documented in the subscription decision record: the pubsub backend's fan-out throughput at the expected peak event rate and concurrent connection count, the maximum event rate that the chosen backend can sustain without increasing per-event delivery latency above the product's latency target, and the horizontal scaling model for the backend itself (Redis Cluster for pubsub fan-out at scale; NATS JetStream for higher throughput requirements with persistence).

The horizontal scaling model for WebSocket subscriptions requires session affinity (sticky sessions) at the load balancer unless the pubsub backend broadcasts events to all server processes. Without session affinity, a client's WebSocket reconnection request may be routed to a different server process than the one holding the client's subscription state — the new process must re-establish the subscription from scratch, which causes a brief event delivery gap during the reconnection window. With Redis pubsub, events are published to a Redis channel and all server processes receive the event, so reconnection to a different server process does not miss events that were published during the reconnection window (assuming the reconnection is faster than the event rate requires — if the client was disconnected for 30 seconds and 200 events were published during that window, the client misses those 200 events unless a cursor-based replay model is in place). The sticky session requirement — or its Redis-pubsub-based relaxation — must be documented in the subscription decision record and communicated to the infrastructure team configuring the load balancer, because a load balancer reconfiguration that disables session affinity without knowing about the subscription system's state model will produce intermittent event delivery failures that are difficult to reproduce in testing and difficult to diagnose without understanding the connection affinity dependency.

The event delivery guarantee and the replay surface

The default message delivery model for GraphQL subscriptions with graphql-ws and Redis pubsub is at-most-once: each event published to the Redis channel is delivered to each connected subscriber at most once, with no guarantee of delivery if the subscriber is disconnected at the time of publication, no acknowledgment from the subscriber that the event was received and processed, and no mechanism for a subscriber to request replay of events that were missed during a disconnection. The at-most-once model is correct for subscriptions that deliver live ephemeral state — the current value of a sensor reading, the live cursor position of a remote collaborator, the current price of an asset — where the most recent value is always more useful than any historical value and replaying missed events would deliver stale data that the client has already superseded with a subsequent state fetch. The at-most-once model is incorrect for subscriptions that deliver append-only event streams where every event has business meaning — trade executions, audit log entries, order status transitions, payment confirmations — where a missed event is not "stale state" but a gap in the client's view of a sequence that the product requires to be complete.

The event delivery guarantee must be chosen at subscription schema design time, not added later, because the replay model requires infrastructure that is not present in the at-most-once baseline. A cursor-based replay model requires four components that are not part of the default graphql-ws setup. The first is event persistence: each event published to the subscription must be written to a persistent store (a database table, a Redis Stream, a message queue with retention) with a monotonically increasing cursor identifier (a sequence number, a timestamp with sufficient resolution to be unique per channel, or a UUID ordered by creation time). The event must be persisted before the pubsub publication — if the pubsub message is delivered before the event is persisted, a client that reconnects immediately after receiving the pubsub notification may find the event not yet visible in the persistent store's query. The second is cursor tracking: the client must receive the cursor identifier for each delivered event and persist it across reconnections, so that on reconnection the client can send the last-received cursor to the server. The cursor delivery mechanism depends on the transport: graphql-ws supports event extensions (a per-event metadata field in the graphql-ws protocol's next message type) that can carry the cursor without changing the subscription's data schema; SSE's Last-Event-ID header carries the cursor automatically if the server sets the id: field in the event stream. The third is a replay query: on reconnection with a cursor, the server must query the persistent event store for events with a cursor greater than the last-received cursor and deliver them to the client before switching to live delivery. The replay delivery must be ordered (events in cursor order) and must not have gaps — if the persistent store has a gap in its cursor sequence (due to a write failure or a transaction rollback), the replay must either include a gap event (a synthetic event indicating that a gap occurred) or fail the reconnection with an error that the client can surface as "some events may have been missed." The fourth is a replay window policy: how long events are retained in the persistent store for replay purposes. Retention must be long enough to cover the client's expected maximum disconnection duration (if clients can be disconnected for up to 24 hours, the retention window must be at least 24 hours) plus a safety margin. Events outside the retention window cannot be replayed, and a client that reconnects with a cursor outside the retention window must perform a full state reconciliation (re-fetching the current state via a query) rather than a replay.

The absence of an event delivery guarantee is not always wrong — many real-time features are correctly served by at-most-once delivery. The problem is when teams implement at-most-once delivery for use cases that require at-least-once delivery, because the gap between what the client sees and what the server processed is invisible until a business-critical event is missed. A trade execution subscription that delivers at-most-once is a UI convenience feature: it shows trades that happen to be delivered during an active session. A trade execution subscription that delivers at-least-once with cursor-based replay is an audit surface: it guarantees that the client's trade history is complete and can reconcile with the server's execution log. The decision record must document which guarantee each subscription type provides, which use cases are served by each guarantee level, and whether the product's business requirements for completeness and auditability require a cursor-based replay model for specific subscription types. Documenting the guarantee level per subscription type is the artifact that prevents a future engineer from adding a new subscription type for a business-critical event stream and inheriting the at-most-once default without realizing that the use case requires at-least-once delivery.

The backpressure model and the slow-consumer surface

Backpressure is the mechanism by which a consumer signals to a producer that it cannot consume events as fast as they are being produced. In a synchronous system (a function call returning a result), backpressure is implicit: the producer cannot produce the next event until the consumer processes the current one and the function returns. In an async push system — GraphQL subscriptions, Redis pubsub fan-out, WebSocket event delivery — the producer and consumer are decoupled, and the consumer's inability to keep up with the producer manifests as a growing buffer somewhere between the producer and the consumer. The location and size of the buffer determines the failure mode: an unbounded buffer in the server process heap produces OOM failures; an unbounded buffer in the WebSocket layer produces growing per-connection memory consumption and increasing event delivery latency for the slow connection; an unbounded buffer in the Redis pubsub system is bounded by Redis memory rather than server process heap, trading one OOM risk for another. No GraphQL subscription implementation has built-in backpressure in the direction from client to server — the WebSocket and HTTP/2 protocols have transport-level flow control (TCP receive window), but the graphql-ws protocol has no application-level mechanism for a slow client to signal that the server should produce events at a lower rate.

The backpressure policy must be implemented at the subscription resolver level, not delegated to the transport or the pubsub library. Three implementation patterns exist. The first is the buffer limit with drop policy: the subscription resolver wraps the AsyncIterator from the pubsub system in a buffering layer that tracks the number of unconsumed events per connection and applies a drop policy when the buffer exceeds a configured limit. The drop policy — drop oldest, drop newest, or disconnect — is applied synchronously when the event arrives, before the event is added to the buffer. A buffer limit of 500 events per connection with a drop-oldest policy means that a slow connection never accumulates more than 500 events in memory — the 501st event evicts the oldest buffered event. The memory budget per connection is bounded: 500 events × average payload size in bytes. The client experiences the drop as a gap in the event stream (events are missing), which requires the client to detect the gap and perform a state reconciliation query. The second is the debounce and aggregate policy: instead of buffering all events, the subscription resolver debounces or aggregates rapid events into a single delivery. For subscriptions that deliver live state (a stock price, a document cursor, a dashboard counter), a debounced delivery at 100ms intervals reduces the event rate by the ratio of the production rate to the debounce interval — a 1,000-events-per-second burst is reduced to 10 deliveries per second per connection — without requiring a drop policy. The debounce interval is a policy parameter that must be tuned per subscription type: a 100ms debounce for a stock price subscription is acceptable; a 100ms debounce for a trade execution confirmation is not, because each execution is individually meaningful and aggregating two executions into a single event loses information. The third is the circuit-breaker disconnect policy: when a connection's buffer depth exceeds a threshold, the server closes the subscription and sends a graphql-ws error message before closing the WebSocket connection. The client must reconnect and, if a cursor-based replay model is in place, request replay of missed events. The disconnect policy is the most aggressive option but guarantees that the server's memory consumption per connection is bounded by the buffer depth threshold at which the disconnect fires, rather than by the total events produced during the disconnection window. The backpressure policy must be documented per subscription type in the decision record — different subscription types may warrant different policies — and must include the monitoring requirement: a metric tracking per-connection buffer depth, exported to the observability system, with an alert at 50% of the configured buffer limit to provide advance warning before the slow-consumer policy fires.

The event loop's interaction with the backpressure model is the second structural constraint. Node.js processes events on a single thread: the event loop. When a subscription event arrives from the pubsub backend, the event loop serializes the event payload, frames it as a WebSocket message, and writes it to each subscribed connection's socket buffer. If 5,000 connections are subscribed and an event arrives, the event loop must perform 5,000 serialization and frame operations before it can process the next event or handle any incoming HTTP requests. At a 100-event-per-second production rate and 5,000 subscribed connections, the event loop must process 500,000 serialization operations per second — a load that saturates a single Node.js process before the file descriptor limit is reached. The subscription decision record must document the expected peak event rate per subscription channel, the expected peak connection count per subscription channel, and the product of these two numbers as the event loop serialization load. If the serialization load exceeds what a single Node.js process can sustain at the target delivery latency, the architecture must either shard subscriptions across multiple server processes (routing specific subscription channels to specific server processes, reducing the per-process connection count for each channel), use a dedicated subscription server process separate from the HTTP API process, or reduce the fan-out scope (sending each event only to the connections subscribed to the specific entity that changed, rather than broadcasting to all connections).

Three AI session types that embed GraphQL subscription decisions without documenting them

The real-time feature kickoff session is where the subscription transport is selected and the first subscription is deployed. The session is typically motivated by a product requirement: the team wants to eliminate polling from a high-frequency dashboard, add live collaborative cursors to a document editor, or push status updates to a long-running background job's progress indicator. The session evaluates the transport options — WebSocket with graphql-ws versus long-polling versus SSE — and selects WebSocket with graphql-ws as the current best practice for GraphQL subscriptions. The session implements the subscription type in the GraphQL schema, adds the pubsub backend (in-process EventEmitter or Redis), wires the subscription resolver's async iterator, and deploys a working live-updating UI. The session confirms that events are delivered with low latency and that the client reconnects correctly after a brief disconnection. What the session does not produce is the connection scale model (how many concurrent connections can the current pubsub backend sustain at the target delivery latency), the backpressure policy (what happens when a client falls behind the event production rate), or the event delivery guarantee (at-most-once or cursor-based replay, and under what conditions is a state reconciliation query required after a reconnection gap). Each of these is explicitly or implicitly deferred as "we'll address this when we have more users" — a reasonable deferral at the beta stage — without documenting what "more users" means as a specific threshold at which the deferred decisions must be made. The open-source extractor surfaces these founding real-time feature sessions from AI chat history, recovering the transport selection rationale, the pubsub backend choice, and the performance assumptions that were implicit in the founding team's shared context when the subscription system was first designed.

The horizontal scaling session is where the subscription system's stateful connection model first collides with the infrastructure team's load balancer configuration. The session is motivated by a capacity or reliability requirement: the team wants to add a second server process to distribute HTTP API load, or the on-call runbook specifies that service restarts should not require user-visible downtime. The infrastructure engineer adds a second Node.js process behind the load balancer with round-robin routing. The next morning, Datadog alerts fire: a subset of users is reporting that their real-time activity feed has stopped updating. The investigation quickly identifies the cause: users whose HTTP API requests are being routed to server process B are occasionally hitting WebSocket connections that were established against server process A, which has a different EventEmitter instance and does not receive the events published by API requests processed by process B. The fix is sticky sessions at the load balancer — configuring the load balancer to route all requests from a given client IP or session cookie to the same server process. The sticky session configuration is implemented and the alerts resolve. What the session does not produce is a documented sticky session requirement in the subscription architecture documentation, so that the next infrastructure engineer who reconfigures the load balancer understands why sticky sessions are required and what breaks if they are removed. Three months later, the infrastructure team migrates from the current load balancer configuration to a new Kubernetes ingress controller during a cloud infrastructure modernization. The Kubernetes ingress does not have sticky session configured by default. The same EventEmitter fan-out failure mode recurs for 72 hours before the root cause is identified and the Kubernetes service's sessionAffinity is set to ClientIP. The decision record that documented the sticky session requirement would have been consulted during the Kubernetes migration planning and prevented the 72-hour regression. The real-time architecture decision record is the parent document that the subscription decision record must reference: the horizontal scaling model for the real-time system — sticky sessions, shared pubsub, or connection-level routing — is a cross-cutting infrastructure decision that the subscription-level ADR cannot govern alone.

The mobile client optimization session is where subscriptions are extended to mobile clients for the first time, introducing connection lifecycle requirements that the desktop browser implementation did not expose. The session is motivated by a product requirement: the iOS or Android app should show the same real-time updates as the web dashboard. The session adds the graphql-ws client to the mobile app's networking layer, configures the subscription endpoint, and verifies that real-time events are delivered to the mobile client in a development environment. What the session does not address is the mobile client's connection lifecycle — how the subscription behaves when the mobile app moves to the background, when the device loses network connectivity, when the device switches from WiFi to cellular, or when the OS terminates the app's background network connections to conserve battery. Mobile operating systems aggressively manage background network connections: iOS may close background WebSocket connections within 30 seconds of the app entering the background, and Android's Doze mode may suppress network activity for minutes at a time for apps not in the foreground. A mobile client that has been in the background for two minutes will have a WebSocket connection that was closed by the OS without a clean graphql-ws ConnectionClose message — the server's connection is in a half-open state, still tracking the connection as active, until the server's ping-timeout mechanism fires (if configured) or the TCP keepalive timeout is reached (typically 2 hours on most Linux configurations). For those minutes or hours, the server is routing events to a WebSocket connection whose other end is closed, consuming memory for the event buffer without delivering events to the client. When the client returns to the foreground and the mobile WebSocket library reconnects, it reconnects with a new connection — the server now has two connections for the same logical user, one half-open and one active, until the half-open connection's TCP keepalive or ping-timeout cleans it up. The subscription decision record must address the mobile connection lifecycle explicitly: the server-side ping interval and connection timeout that detects half-open connections from mobile clients (graphql-ws ServerOptions.keepAlive set to 15–30 seconds for mobile-heavy workloads, versus 60–120 seconds for desktop-only workloads), the client-side reconnection policy (exponential backoff with jitter, maximum retry delay, retry ceiling to prevent battery drain during extended offline periods), and the state reconciliation requirement when the client reconnects after a background period (how the client determines what state to re-fetch to fill the gap in the event stream). The secrets management decision record is the companion document for the mobile connection lifecycle: the JWT token expiry during background periods is the most common authentication failure mode for mobile subscription clients, and the token refresh policy on reconnection must be documented alongside the reconnection semantics.

The five sections of a GraphQL subscription decision record

The first section documents the transport model selection and connection scale model. The transport selection rationale must be specific and evaluatable: "WebSocket with graphql-ws was selected because the subscription operations require bidirectional messaging (client sends connection_init parameters, server sends events, client sends ping responses), the team's load balancer supports WebSocket connections with sticky sessions, and the expected peak concurrent connection count of 5,000 connections per server process is within the file descriptor budget at ulimit -n 65,535" is more evaluatable than "we chose WebSocket because it's the industry standard for GraphQL subscriptions." The connection scale model must document: the configured file descriptor limit (ulimit -n) per server process; the reserved file descriptors for database connection pool, Redis connections, log file handles, and Node.js runtime (estimated 100–200 for a typical API server); the resulting maximum WebSocket connection count per server process; the per-connection memory overhead of a subscription connection in the chosen pubsub model (typically 15–50 KB in graphql-ws for the connection state, resolver context, and AsyncIterator — measured under the expected average event rate and payload size, not at idle); and the resulting maximum concurrent subscription connections per server process at the configured heap limit. The pubsub backend selection must document the fan-out model (in-process EventEmitter for single-process deployments, Redis pubsub for multi-process deployments), the fan-out throughput at the expected peak event rate and connection count (measured from a load test, not estimated from documentation), and the horizontal scaling model (Redis Cluster, NATS cluster, or a managed pubsub service) if the single-node pubsub throughput is within 50% of the expected peak load at 18-month projected growth. The load balancer configuration must document whether sticky sessions are required (required for in-process EventEmitter; may be optional for Redis pubsub deployments where events are broadcast to all processes), the sticky session mechanism (IP hash, cookie-based, or Kubernetes service sessionAffinity), and what breaks if sticky sessions are misconfigured — this must be in the decision record so that the infrastructure team understands the consequence of load balancer reconfiguration before making it.

The second section documents the connection lifecycle policy. The server-side ping interval and timeout must be documented separately for two client populations: browser clients and mobile clients. Browser WebSocket connections are managed by the browser's networking layer and are closed cleanly when the browser tab is closed or navigated away — the browser sends a WebSocket close frame, the server receives it, and the connection is cleaned up promptly. Mobile WebSocket connections are managed by the mobile OS's networking stack and may be closed without a clean close frame when the app moves to the background, the device sleeps, or the OS terminates background network activity. A server-side ping interval of 15–30 seconds (the graphql-ws ServerOptions.keepAlive value) with a 2× ping-timeout (30–60 seconds) detects half-open connections from mobile clients within 30–60 seconds of the client going offline, rather than waiting for TCP keepalive to fire (2 hours on default Linux configurations). The client-side reconnection policy must document: the initial reconnection delay after a connection drop (immediate reconnection for the first attempt, to minimize event delivery gap); the backoff schedule for subsequent reconnection failures (exponential backoff with jitter, starting at 1 second and doubling to a maximum of 60 seconds for browser clients, 120 seconds for mobile clients to avoid battery drain during extended offline periods); the maximum number of reconnection attempts before the client gives up and requires user intervention; and the state reconciliation query that the client performs after a successful reconnection to fill the gap in the event stream that accumulated during the disconnection window. The idle connection timeout must document: whether the server closes connections that have been inactive (no events delivered) for a defined period, and if so, the timeout value per subscription type (high-frequency subscriptions may be considered idle after 5 minutes without events; low-frequency subscriptions may be considered idle after 24 hours). The idle connection timeout prevents server resource accumulation from stale connections that the client has abandoned without a clean close — a user who closes the laptop lid and never reconnects leaves a WebSocket connection open on the server until either the ping-timeout or the idle timeout fires. The API gateway decision record must be consistent with the connection lifecycle policy: API gateways and reverse proxies have their own WebSocket connection timeout settings that interact with the application-level ping/pong mechanism, and a gateway that closes WebSocket connections after 60 seconds of inactivity will interfere with subscriptions whose event rate is lower than one event per 60 seconds even if the graphql-ws ping/pong is correctly configured.

The third section documents the event delivery guarantee and replay policy. The guarantee level must be documented per subscription type, not as a single system-level policy, because different subscription types within the same system may require different guarantees. The guarantee options are at-most-once (default for graphql-ws and Redis pubsub, no persistence or cursor required, appropriate for live ephemeral state), at-least-once (requires event persistence, cursor delivery, and server-side replay query, appropriate for append-only business event streams), and effectively-once (at-least-once delivery combined with client-side idempotency handling, appropriate for business event streams where duplicate delivery could produce incorrect client state). For each subscription type in the system, the decision record must specify: the guarantee level; if at-least-once or effectively-once, the persistence backend and retention window (events retained for 72 hours, or events retained until the client acknowledges receipt, or events retained for a configurable period per client tier); the cursor delivery mechanism (graphql-ws event extension field, SSE id: field, or an out-of-band cursor field in the subscription payload schema); and the reconnection behavior when the client's last-received cursor is outside the retention window (fail with a specific error code that the client handles by performing a full state reconciliation query, rather than silently replaying from the beginning of the retained window with a gap that the client does not know exists). The state reconciliation query must be documented as a first-class operation in the GraphQL schema — a query that returns the current state equivalent to what the client would have received if it had been connected and consumed all events — so that client developers know which query to call on reconnection and the schema evolves consistently with the subscription's event model. The GraphQL design decision record is the prerequisite document: the subscription system inherits the schema design conventions, the authorization model, and the pagination approach from the broader GraphQL API design, and the subscription's state reconciliation query must follow the same schema conventions as the equivalent read operation for the subscribed entity.

The fourth section documents the backpressure model and slow-consumer policy. The per-connection event buffer limit must be specified per subscription type, as a count and a byte budget: "the tradeExecuted subscription allows a maximum of 200 buffered events per connection (approximately 560 KB at the average execution payload size of 2.8 KB) before the slow-consumer policy is applied; the cursorMoved subscription allows a maximum of 50 buffered events per connection (approximately 25 KB at the average cursor payload size of 0.5 KB) before the slow-consumer policy is applied." The slow-consumer policy must specify the action applied at the buffer limit: drop-oldest (discard the oldest buffered event to make room for the new event), drop-newest (discard the new event if the buffer is full), or disconnect (close the subscription connection and require the client to reconnect). For subscriptions delivering live ephemeral state, drop-oldest is appropriate — the client receives the most recent events, and stale events in the buffer are less valuable than recent ones. For subscriptions delivering append-only business event streams, disconnect is appropriate — the client must reconnect and replay from cursor rather than receive a set of events with gaps. For both drop policies, the client must be able to detect that events were dropped and trigger a state reconciliation query — either through a subscription event that signals a gap, or through a client-side sequence number check that detects non-consecutive cursors. The implementation approach for the buffer limit must be documented: the AsyncIterator wrapper that tracks buffer depth and applies the drop or disconnect policy; the metric exported per connection (buffer depth as a gauge metric, drop count as a counter metric, disconnect count as a counter metric); and the alert threshold (fire an alert when P95 buffer depth across active connections exceeds 50% of the configured limit, giving the team advance warning before slow-consumer actions begin occurring at scale). The total memory budget must be documented as a calculation: maximum concurrent connections × per-connection buffer limit × average event payload size = maximum memory consumption attributable to subscription event buffering, which must fit within the server process's configured heap limit minus the baseline application memory. The GraphQL federation decision record is relevant when subscriptions span federated subgraphs: the backpressure model must account for subscription fan-out across a federation router and multiple subgraph subscription implementations, where the buffer limit must be enforced at the router level (closest to the client) rather than at the subgraph level, to prevent the router from accumulating events from subgraphs faster than slow clients consume them.

The fifth section documents the subscription authentication and authorization policy. The connection-level authentication mechanism must specify how the client presents credentials during connection establishment: via the WebSocket connection's initial HTTP upgrade request headers (requires the server to validate the token before upgrading from HTTP to WebSocket — the correct approach, since it prevents unauthorized clients from establishing connections), or via the graphql-ws ConnectionInit message payload (the client establishes the WebSocket connection first, then sends credentials in the connection_init payload — allows the connection to be established before authentication is verified, which is a weaker boundary but is required when the client cannot set custom headers on the WebSocket upgrade request, as is the case in browser environments using the native WebSocket API without a proxy). The token expiry policy must document: the mechanism for handling JWT expiry during a long-lived WebSocket connection (client-side token refresh before expiry with in-flight rotation via connection_init re-send; connection re-establishment at token expiry with cursor-based replay to fill the gap; or separate long-lived subscription tokens with a scoped refresh mechanism); the server-side validation timing (credentials validated at connection establishment only, or re-validated on each subscription operation within an established connection); and the server-side behavior when token expiry is detected (close the connection with a graphql-ws error message containing a specific error code that the client handles by initiating token refresh and reconnection, rather than closing the connection without a machine-readable error code that the client cannot distinguish from a transient network error). The per-subscription authorization policy must specify: whether each subscription operation is authorized at connection time (all operations on a connection are allowed if the connection-level token has the required scope) or at subscription time (each individual subscribe message is authorized against the token's claims and the subscription operation's required permissions, allowing a single connection to hold subscriptions to resources with different authorization requirements); and the authorization model for entity-scoped subscriptions (a tradeExecuted(accountId: ID!) subscription must verify that the authenticated user is authorized to subscribe to executions for the specified accountId, not just that they have a valid token). The new CTO onboarding problem in a subscription-heavy architecture is an authentication gap problem: an incoming technical leader will find subscription endpoints with connection-level authentication but no per-subscription authorization checks, long-lived connections running with tokens that expired months ago on clients that never implemented the token refresh policy, and no documentation of which subscription types require at-least-once delivery versus which are operating safely under at-most-once semantics.

The real-time feature kickoff session, the horizontal scaling session, and the mobile client optimization session each produce GraphQL subscription decisions whose long-term operational cost — a production OOM incident at peak trading volatility, discovered through a 7-minute real-time display blackout because no per-connection event buffer limit was defined when the subscription system was designed; or a 72-hour subscription delivery regression after a Kubernetes ingress migration, because the sticky session requirement was never written down and the infrastructure team did not know it existed; or a JWT expiry-driven subscription authentication failure on mobile clients during a product launch, because the token refresh policy for long-lived connections was deferred in the session that added mobile subscription support — exceeds what a connection scale model, a backpressure policy, and a token expiry document would have cost at the time the founding decisions were made. The decisions are in the AI sessions: the transport selection rationale, the pubsub backend choice, the assumption that in-process EventEmitter would be migrated to Redis before the second server process was added, the performance expectation that treated a 30-second event burst as out of scope for the beta load test. WhyChose's open-source extractor surfaces these founding real-time architecture sessions as structured decision records before the next slow-consumer OOM incident, the next sticky-session migration regression, or the next mobile token expiry authentication failure reveals the undocumented operational commitments as an unplanned engineering sprint or an enterprise customer escalation. The decisions never written down in a subscription-heavy system are the connection scale model, the backpressure policy, and the event delivery guarantee — the three operational commitments that determine whether the subscription system scales gracefully or breaks in ways that are invisible in a single-process development environment and visible only under production load, mobile network conditions, or market volatility events that the founding session's load test did not include.

Further reading