Why does Protobuf field number reuse cause silent data corruption instead of a deserialization error?

Protocol Buffers identifies each field in the wire format by its field number, not its name. When a message is encoded, each field is serialized as a tag (the field number and wire type packed into a single varint) followed by the field's value. When a consuming service deserializes a message, it reads each tag, identifies the field number, and maps that field number to the field defined in the proto schema the consumer was compiled against. If the producer and consumer have different proto schemas — specifically, if the producer defines field number 15 as a string and the consumer defines field number 15 as an int32 because the producer deprecated the original string field and added a new int32 field at the same field number — the consumer reads the wire-format bytes for a string-encoded value and attempts to parse them as an int32. The Protobuf decoder may succeed with a nonsense value, throw a parsing error, or produce a partially decoded message depending on the specific byte content of the string value and the target type's wire type. The 'silent' property occurs when the wire types happen to be compatible (for example, varint and fixed32 can overlap in some cases) and the decoder produces a value that passes range checks but is semantically incorrect. The field number reservation rule in the Protocol Buffers documentation — that removed field numbers must be reserved using the 'reserved' keyword and must never be reused — exists specifically to prevent this scenario. A proto file that marks a removed field as 'reserved 15' causes the proto compiler to reject any future field definition that uses field number 15, turning the silent corruption risk into a compile-time error. When the reservation rule is not in the serialization decision record, engineers who inherit the proto schema do not know why certain field numbers are reserved and cannot apply the rule to fields they deprecate themselves.

What is the difference between Avro schema registry compatibility modes (BACKWARD, FORWARD, FULL) and which should be the default?

Apache Avro schema evolution compatibility is checked against a compatibility mode configured per subject (per topic, per message type) in the Confluent Schema Registry or equivalent registry. BACKWARD compatibility means the new schema can read data written by the old schema — consumers on the new schema can process messages that were produced with the old schema. Adding an optional field with a default value is backward-compatible because old producers write messages without the new field, and the new consumer uses the default value when the field is absent. Removing a required field (a field without a default) is not backward-compatible because old messages may be missing the required field. FORWARD compatibility means the old schema can read data written by the new schema — consumers on the old schema can process messages produced with the new schema. Adding a required field breaks forward compatibility because old consumers do not know how to handle the new field. FULL compatibility requires both BACKWARD and FORWARD compatibility simultaneously — the new schema can read old messages and old consumers can read new messages. FULL_TRANSITIVE extends FULL compatibility to all registered versions of the schema, not just the previous version. For most production Kafka topics, FORWARD or FULL compatibility is the appropriate default: FORWARD ensures that when a new producer schema is deployed before all consumers are updated, old consumers can still process the new messages without failures; FULL additionally ensures that replayed old messages remain readable by new consumers. BACKWARD compatibility alone is insufficient for a production Kafka topic because it does not prevent a new producer from writing messages that old, still-deployed consumers cannot process. The compatibility mode must be set before the first schema is registered — changing the compatibility mode after schemas are registered requires validating all existing schemas against the new compatibility requirement, which may require schema migrations. The serialization decision record should document the chosen default compatibility mode, the rationale, and the override process for topics where a different mode is needed.

When should a team choose Protocol Buffers over JSON for service-to-service communication, and what does the choice constrain?

Protocol Buffers is the right choice when at least one of three conditions is true: (1) serialization and deserialization performance is a measured bottleneck — Protobuf's binary wire format is typically 3-10x smaller than equivalent JSON and 2-5x faster to serialize and deserialize at scale; (2) the team wants schema enforcement at compile time rather than at runtime — the .proto schema definition generates strongly-typed code in each target language, turning schema mismatches into build failures rather than production errors; (3) the service boundary crosses language ecosystems where each service uses a different implementation language — Protobuf's code generation produces idiomatic bindings in Go, Java, Python, C++, Rust, JavaScript, C#, and several others, sharing the .proto schema definition across implementations. The constraints that the Protobuf choice imposes are: the .proto schema files must be shared across all services that communicate using those types, requiring a shared schema repository or package distribution mechanism; field numbers must be managed with the reservation discipline (never reuse a removed field number, mark removed fields as reserved) for the life of the message type; and debugging a live Protobuf message without the schema requires the Protobuf reflection API or a decode-time schema reference — a binary payload logged from a production service without schema context is not human-readable without tooling. JSON is the right choice when debuggability and flexibility are more important than performance and compile-time schema enforcement — for external APIs consumed by third parties who control their own deserialization, for event payloads that need to be readable in log files and monitoring dashboards without special tooling, and for small-scale internal APIs where the schema evolution surface is small and manually managed. The serialization decision record should document the choice's rationale, the specific performance or ecosystem condition that motivated the choice over JSON, and the schema management process — because the constraints of the chosen format apply to every service that is added to the system after the initial choice, including services written by engineers who were not present when the format was selected.

What is an unknown field handling policy and why does it need to be documented for JSON APIs?

The unknown field handling policy for a JSON API is the specification of what a consumer does when it deserializes a JSON message and encounters a field that is not present in the consumer's expected schema — a field that was added by the producer after the consumer was deployed, or a field that exists in some versions of the producer's schema but not all. The policy has three common variants: (1) ignore unknown fields silently — the consumer deserializes only the fields it knows about and discards any unrecognized fields; this is forward-compatible because a consumer on an old schema can process messages from a producer on a new schema without errors; (2) reject unknown fields with an error — the consumer throws a deserialization exception if the JSON message contains any field not present in the consumer's schema; this is the default behavior of many strict JSON deserialization frameworks (Jackson's FAIL_ON_UNKNOWN_PROPERTIES in its default configuration before version 2.x, some OpenAPI-generated code) and is not forward-compatible; adding a new field to the producer's JSON schema causes all strict consumers to fail until they are updated; (3) pass through unknown fields — the consumer deserializes all fields including unrecognized ones, preserving them in a generic map or in the raw JSON, for re-serialization when the consumer acts as a pass-through intermediary. The policy must be consistent across all consumers that receive messages from a given producer — a mix of strict and lenient consumers means that adding a new field to the producer is safe for some consumers and breaking for others, with no compile-time or schema-registry signal to distinguish the two groups. The serialization decision record should document the required unknown field handling policy, the deserialization library configuration that enforces it for each language in use, and the process for verifying that new services implement the correct policy before they are deployed to receive production messages. Without documentation, the unknown field handling behavior defaults to whatever the chosen deserialization library's out-of-the-box configuration is — which differs between libraries, between language ecosystems, and between library versions, producing inconsistent behavior across the service fleet.

The data serialization decision record: why the format you chose determines your schema evolution safety and your cross-service compatibility ceiling

July 9, 2026 · WhyChose

Service-to-service data serialization starts with JSON because it is the obvious choice when configuring the first REST endpoint. Each service encodes its response by calling JSON.stringify or the equivalent and decodes incoming payloads by calling JSON.parse. The approach works, requires no schema definition, and produces payloads readable in any log aggregation tool or browser devtools panel. The format choice is not documented because it does not feel like a decision — it is the default. The first time the choice becomes a decision is when the team needs to add a field to a message type that is consumed by three services in two languages, and someone asks whether changing the message schema will require coordinated redeployment of all consumers or whether consumers on the old schema will continue working after the producer is deployed with the new field.

The answer depends on two things: whether the consuming services' JSON deserialization configuration ignores or rejects unknown fields, and whether any consuming service validates the incoming payload against a schema that explicitly rejects unrecognized keys. Neither is documented, because the serialization format was chosen in the original implementation session without a schema evolution policy. The team either runs a cross-service integration test to verify the behavior — which takes two days to set up and reveals that one of the three consumers uses a strict deserialization configuration that will reject the new field — or deploys the producer change and discovers the consumer behavior in production. Either path requires engineering time that would not have been needed if the serialization decision record had established a known unknown-field handling policy when the format was originally adopted.

The data serialization decision record is missing from most service architectures because the format choice is made implicitly, at the first integration point, by whoever sets up the first service-to-service call. The decision then propagates to every subsequent service by convention: new services are built to match what the existing services expect. The accumulated convention is not a documented policy — it is tribal knowledge that exists in the heads of the engineers who built the first services and in the implicit behavior of the deserialization libraries used in each language. What is not documented is the field lifecycle policy (how fields are added, deprecated, and removed across services), the compatibility requirement for the serialization format (whether consumers must be updated before producers or can lag behind), the language ecosystem constraints the format imposes (which languages have maintained serialization libraries for the chosen format), and the debugging tools available for diagnosing serialization failures in production. Those decisions belong in a decision record. Without one, the team that inherits the system three years after the first service-to-service call was written is operating under constraints that are visible only in the code and in the accumulated incidents produced when someone violated a constraint they did not know existed.

Two things that happen when the decision is not written down

The Protobuf field number reuse that corrupted a payment processing path

A 40-person platform company migrated their internal service communication from JSON to Protocol Buffers during a performance optimization project. The migration was motivated by a measurable bottleneck: their event processing pipeline was deserializing 120,000 JSON payloads per second at peak load, and profiling showed that JSON deserialization was consuming 23% of CPU time on the event processor service. A Protobuf migration benchmarked at 4x lower CPU for the same workload. The migration was implemented over six weeks, covering twelve message types and six services. The .proto files were stored in a shared internal repository, and each service pinned to a specific commit of the proto repository in its build system. The migration was documented in a Confluence page that covered the performance motivation, the migration timeline, and the proto compilation toolchain setup — but did not cover the field lifecycle policy: how fields should be added, deprecated, or removed from the proto schemas.

The payment processing path used a PaymentEvent proto message with fifteen fields. Field number 12 was a string field named legacy_reference_id that had been added during the early migration to preserve backward compatibility with a JSON-era field. By the time the migration was complete, legacy_reference_id was populated by the producer but not read by any consuming service — it existed to allow the JSON-era field to be present in the event without breaking existing consumers that had been updated to consume the proto format. Six months after the migration, an engineer cleaning up the proto schema removed legacy_reference_id from the PaymentEvent definition and added a new field idempotency_key at field number 12 — the same field number that legacy_reference_id had used — because the proto documentation showed field numbers 1-14 in use and field number 12 as absent (it had been removed), making field number 12 the natural next choice for a new field in the compact range.

The producer service was deployed with the updated PaymentEvent proto, which wrote the new idempotency_key string value at field number 12. Two consuming services were deployed simultaneously with the updated proto and read the new idempotency_key field correctly. A third consuming service — a payment reconciliation service that ran on a weekly batch schedule and had not yet been updated with the new proto — continued to run on the old proto definition, which mapped field number 12 to the string legacy_reference_id. From the reconciliation service's perspective, the incoming PaymentEvent messages had a field number 12 that the proto defined as a string, and the wire format contained a string value at field number 12. The Protobuf decoder for the reconciliation service read the idempotency_key value as the legacy_reference_id and stored it in the legacy_reference_id field of the decoded struct. The reconciliation service then exported its weekly payment summary with the legacy_reference_id field populated from the idempotency key values, not from the original reference IDs — which were no longer being written to field number 12. The corruption was silent: no deserialization error, no schema mismatch error, no monitoring alert. The reconciliation service successfully processed all payment events and generated a payment summary that contained incorrect reference IDs for all payments processed after the producer was updated.

The corruption was discovered six days later when the finance team attempted to reconcile the weekly payment summary against the payment processor's statement and found that the reference IDs in the internal summary did not match the reference IDs in the external processor statement. The investigation identified the field number reuse as the root cause. The reconciliation service was updated and the payment summary was regenerated from the raw event log, which required a backfill process that took eighteen hours. The proto schema was patched to add a reserved 12; statement alongside a comment explaining the field number reservation rule. The incident report asked why the field number reservation rule had not been applied to the original removal: the answer was that the engineer who removed legacy_reference_id did not know about the Protocol Buffers field number reservation requirement, because the Confluence page that documented the migration did not mention it, and the team had no serialization decision record that covered the field lifecycle policy. The proto documentation embedded in the protobuf.dev reference was available, but there was no organizational norm pointing engineers to that specific constraint before modifying a proto schema. A serialization decision record that included the field lifecycle policy — specifically, that removed field numbers must be marked reserved and that field number reuse causes silent deserialization corruption — would have given every engineer who touched a proto file the knowledge needed to avoid the incident without requiring them to read the full Protocol Buffers specification.

The Avro schema that passed the registry check but failed in production

A 25-person data engineering team built a Kafka-based event pipeline that used Apache Avro for message serialization with the Confluent Schema Registry for schema management. The pipeline had twelve producers written in Python and three consumers written in Java. The schema registry was configured with BACKWARD compatibility for all subjects, meaning each new schema version must be able to read messages written by the previous schema version. The team had three engineers who had deep familiarity with Avro schema evolution, and the schema change process was reasonably well-understood within the data engineering team: add an optional field with a default value, register the new schema with the registry, verify the registry accepts it under BACKWARD compatibility, deploy the new producer that writes the new field, then deploy the updated consumers that read the new field. The process had worked correctly for seven schema changes across four subjects in the previous eight months.

The eighth schema change added a new optional field to the UserActivityEvent Avro schema — a nested record field named session_context with a default value of null and a union type of ["null", {"type": "record", "name": "SessionContext", "fields": [...]}]. The schema was submitted to the registry, the BACKWARD compatibility check passed, and the producer was deployed. The three Java consumers were updated to read the session_context field and deployed. Two of the three consumers started successfully and began processing events from the topic. The third consumer — a consumer that performed aggregation over a sliding window of activity events — began throwing deserialization exceptions on approximately 15% of messages, with errors of the form org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -2147483648. The remaining 85% of messages were processed successfully.

The investigation revealed the root cause after four days: the Python Avro library version used by the producers (fastavro==1.4.7) had a bug in how it encoded default values for optional nested record fields when the record definition was a union type with null as the first type. Specifically, when the session_context field was null (the default), fastavro 1.4.7 encoded the union branch index correctly (index 0 for null) but did not write the subsequent null byte that the Avro binary encoding specification requires for the null type in a union. The Java consumer's Avro library (apache-avro 1.11.1) correctly implemented the specification and expected the null byte after the union branch index. When the consumer read a message from a Python producer running fastavro 1.4.7, it read the union branch index (0 for null), then attempted to read the required null byte and found instead the varint start of the next field in the message, interpreting it as a length field for what it expected to be a block count — a nonsense value that produced the negative length error. The 85% of messages that processed successfully were messages where the session_context was not null — they contained the full nested record, which both libraries encoded and decoded correctly.

The schema registry's BACKWARD compatibility check had validated the structure of the new schema definition — that it was a valid Avro schema, that the new field had a default value, and that the new schema's reader/writer compatibility was structurally correct. The compatibility check did not and could not have detected the library version bug in the producer's Avro serialization implementation. The fix required upgrading fastavro to version 1.7.0 on all Python producers (where the bug had been fixed) and replaying the 15% of failed messages from the consumer's dead letter queue. The replay required identifying affected messages by their offset range (corresponding to the deployment window of the buggy producer version) and verifying that each replayed message was processed correctly by the updated consumer. The entire remediation took eleven hours. The team's post-incident review identified that the decision to adopt Avro had been made without documenting the language ecosystem constraint — specifically, that the Python Avro library ecosystem had multiple competing libraries (avro, fastavro, confluent-kafka) with different version compatibility characteristics, and that the Java Avro library was the reference implementation against which the Avro spec was most closely validated. The serialization decision record should have specified the canonical library version for each language in use, the process for updating those versions, and the integration test requirement for verifying cross-language serialization compatibility before a schema change is deployed to production.

Three structural properties that are set at serialization format selection time

The schema evolution compatibility model and the field lifecycle policy

Every serialization format has a schema evolution model — the rules that govern what changes to a message type are backward-compatible (safe to deploy to the producer before all consumers are updated), forward-compatible (safe to deploy to consumers before the producer is updated), or breaking (requiring coordinated redeployment of all producers and consumers). The schema evolution model is determined at format selection time and cannot be changed without replacing the format across all services that use it. Understanding the schema evolution model before selecting a format is the most important input to the selection decision, because it determines the cost of every schema change for the life of the system.

Protocol Buffers has an additive-safe evolution model: adding a new optional field does not break existing consumers (unknown fields are ignored in the default configuration); removing a field does not break existing producers (the field is simply absent in messages from the old producer, and the new consumer uses a default value). The safety property comes with a requirement: removed field numbers must be reserved with the reserved keyword, and reserved field numbers must never be reused. Without the reservation rule, a removed field number is available for reuse in a future schema version, and reuse produces the silent corruption described above. The reservation rule is specific to Protocol Buffers and does not exist in JSON or Avro — it is one of the constraints that must be in the serialization decision record because it is not enforced automatically by the toolchain unless the reserved statement is present.

Apache Avro formalizes schema evolution through explicit compatibility modes that are validated by the schema registry before a new schema version is accepted. BACKWARD compatibility guarantees that new consumers can read old messages; FORWARD compatibility guarantees that old consumers can read new messages; FULL compatibility requires both simultaneously. The appropriate default for a Kafka topic whose consumers are deployed at different times than producers is FORWARD or FULL — BACKWARD alone is insufficient because a new producer with a BACKWARD-compatible schema can write messages that old consumers cannot process, and Kafka consumers that fail to process a message may block partition processing depending on their error handling configuration. The schema registry enforces the chosen compatibility mode structurally, but as the incident above illustrates, structural compatibility does not guarantee that the libraries implementing the format will serialize and deserialize correctly for every field type combination across every version. The serialization decision record should document the canonical library version for each language in use, the testing requirement for verifying cross-language serialization of each new schema before production deployment, and the process for upgrading library versions when bugs are discovered or security vulnerabilities patched.

JSON has no inherent schema evolution model — the schema evolution behavior depends entirely on the deserialization configuration of each consuming service. The three policies discussed earlier (ignore unknown fields, reject unknown fields, pass through unknown fields) must be specified and enforced explicitly across all consumers for the system to have a consistent schema evolution behavior. A fleet of JSON consumers with mixed deserialization policies — some ignoring unknown fields, some rejecting them — creates a situation where adding a new field to a producer is safe for some consumers and breaking for others, with no tooling to identify which consumers are affected before the change is deployed. The serialization decision record for a JSON-based system must document the required unknown-field handling policy and the framework configuration that enforces it for each language and library version in use. Without this documentation, the policy is whatever each engineer assumed when they configured their service's JSON deserialization library, and the assumption is not consistent across engineers or library versions. This connects to the API versioning decision record: the schema evolution policy for internal service-to-service messages and the versioning policy for external APIs have different requirements, but both need to be explicit — the same lack of documentation that causes internal deserialization incidents causes external API versioning incidents when the consumer is a third-party client rather than an internal service.

The language ecosystem surface and client library governance

Serialization format selection constrains the language choices available for future services. A format with poor support in a language that becomes important to the organization — because a team acquires deep expertise in that language, because a third-party library required for a critical feature is only available in that language, or because the team's hiring market produces candidates with expertise in that language — creates a choice between using the constrained language with a low-quality serialization library or migrating to a different format at higher cost. Documenting the language ecosystem constraint at format selection time does not prevent the organization from making incompatible choices later, but it makes the tradeoff visible at the point where the choice is being made rather than as an unpleasant discovery after a commitment has been made.

Protocol Buffers has first-party support for Go, Java, C++, Python, C#, JavaScript/TypeScript, Objective-C, PHP, Ruby, and Dart, with community support for Rust, Kotlin, and Swift. The first-party libraries are maintained by Google and track the Proto3 and Proto2 specifications closely. The code generation toolchain — protoc with per-language plugins — is part of the standard build process for services using Protobuf, which means each service's build system must be configured to compile the .proto files into language-specific classes or structs. Managing the build system dependency on protoc and the per-language plugin versions across multiple services in multiple languages is an operational overhead that does not exist with a schema-free format like JSON. The serialization decision record should document the build system integration approach — whether .proto files are compiled locally during each service's build, pre-compiled and distributed as language-specific packages, or compiled in CI and cached — because the approach affects how quickly a proto schema change propagates to all services and how the schema change is validated before it reaches production.

Apache Avro has strong Java support (the Apache Avro Java library is the reference implementation), reasonable Python support across multiple competing libraries (avro, fastavro, confluent-kafka) with documented version incompatibilities, and thinner support in other languages. Go's Avro support is provided by community libraries with varying spec compliance. Rust's Avro support is similarly community-maintained. If the organization uses Avro for a high-volume message type and then acquires expertise in Go for a new high-performance service that must consume that message type, the Go Avro library's spec compliance needs to be verified against the specific schema patterns in use — particularly the union types and nested record fields that have historically been the source of cross-language compatibility bugs. The serialization decision record should document the library chosen for each language in use, the spec compliance caveats known for each library, and the process for evaluating a new library when a new language is introduced. Without this documentation, the engineer who builds the first Go service to consume an Avro Kafka topic will choose a library based on GitHub stars and documentation quality, without knowing that the organization's Avro schemas use union types and nested records in patterns where the selected library may not match the Java reference implementation's behavior. This is the class of problem that produced the incident described above, and it is preventable with documented library governance rather than per-service library selection.

The language ecosystem constraint also applies to the schema registry tooling: the Confluent Schema Registry has first-party client support for Java, Python, .NET, Go, and JavaScript. Third-party clients exist for other languages. The schema registry client is used in both the producer (to register the schema and include the schema ID in each message) and the consumer (to fetch the schema by ID for deserialization). A language with a poorly maintained schema registry client creates a debugging surface when the schema ID cannot be resolved — the consumer falls back to schema resolution by fingerprint or fails entirely, producing an error that is not immediately attributable to the schema registry client rather than the Avro library. Document the schema registry client library for each language alongside the Avro library, and test the registry client's behavior when the registry is unavailable: if the consumer caches schema IDs locally, a registry outage does not affect in-flight deserialization; if the consumer fetches schema IDs on each message, a registry outage causes consumer failures that are not caused by the message content or the Avro library. The event-driven architecture decision record covers the Kafka infrastructure and partition model; the serialization decision record covers the message format governance; both are required to understand the full failure mode surface of a Kafka-based message pipeline, including the schema registry as a shared dependency in the consumer's critical path.

The debugging and observability surface for serialization failures

The choice of serialization format determines how easy it is to debug a production deserialization failure. For JSON, a failed deserialization typically produces a human-readable error that includes the field name, the expected type, and the received value: Cannot deserialize value of type String from Integer value "42" (token JsonToken.VALUE_NUMBER_INT). The error is self-descriptive because JSON is self-describing — the payload includes field names alongside values, and the error message can identify the problematic field by name. For Protocol Buffers, a deserialization failure typically identifies the field by number rather than name: Message field number 12 cannot be parsed as type int32. The field number is meaningful to an engineer who has access to the .proto file that defines the message, but requires looking up the proto definition to determine which field name corresponds to field number 12. In an incident at 2am, the additional lookup step is significant. For Avro, a deserialization failure produces an error that may reference the Avro binary specification rather than the application's schema: Malformed data. Length is negative: -2147483648 is a spec-level error that requires familiarity with the Avro binary encoding to trace back to the field that produced the malformed length.

Binary format observability requires tooling that is not needed for JSON. Logging a Protobuf message requires either decoding it to text format before logging (which requires the .proto schema to be available at log time) or logging the binary payload as a base64-encoded string (which cannot be read without the schema and a decoder). Production monitoring of Protobuf-encoded Kafka messages requires a consumer that knows the schema in order to decode messages for display in a monitoring tool. The schema registry partially addresses this for Avro — the schema ID embedded in each Avro message allows a consumer with schema registry access to decode the message for display — but creates a schema registry dependency in the monitoring path. Document the observability approach for binary formats in the serialization decision record: how are production messages decoded for debugging, which monitoring tools have been configured to decode the chosen format, and what the fallback is for decoding messages when the schema registry or the .proto repository is unavailable. Without this documentation, the first production deserialization incident requires an ad-hoc investigation to determine how to decode the binary payload before the root cause can even be identified.

The debugging surface also affects how new engineers learn the message format. A JSON payload in a development environment is readable without any tooling — an engineer can curl the service, pipe the output to jq, and read the field names and values directly. A Protobuf payload requires the engineer to know how to decode it using protoc --decode or a language-specific tool, and requires access to the .proto file. An Avro payload logged to a file requires the Avro schema to decode. For engineers joining the team, the barrier to understanding what a given service is sending and receiving is proportional to the format's self-descriptiveness. Document the developer tooling setup — the decoder tools, the schema browser, the Kafka consumer configuration for local development — in the same place as the serialization decision record, and treat it as part of the onboarding requirement rather than knowledge that engineers discover organically. The developer experience decision record covers the developer tooling philosophy; the serialization decision record's tooling section should reference it for the format-specific setup requirements. Both connect to the observability platform decision record: the monitoring infrastructure must be configured to decode binary message formats before it can provide meaningful metrics on message content and deserialization failure rates.

Five sections the data serialization decision record should address

1. Serialization format selection and wire format rationale

Document the serialization format chosen for each message boundary type — internal service-to-service REST or gRPC calls, internal event bus messages (Kafka, Pulsar, RabbitMQ), external API responses consumed by third parties — and the rationale for each selection. The format choice may legitimately differ by context: JSON for external API responses because third-party clients control their own deserialization and cannot be required to compile proto files; Protocol Buffers for internal gRPC calls between services the organization controls because the performance improvement is measurable and the schema enforcement is valuable; Avro for Kafka messages because the schema registry provides centralized schema governance and FORWARD compatibility enforcement for a high-volume event pipeline. Document the format choice per message boundary type, not as a single global decision, because different boundaries have different tradeoffs. Include the performance measurement that motivated the choice if applicable (the CPU and throughput numbers from the Protobuf migration benchmark, the message size comparison that motivated Avro over JSON), and the alternatives evaluated and rejected with one-sentence reasoning for each rejection. The rationale is particularly important for formats that impose toolchain requirements on future services: a team joining the organization months after the selection was made needs to know why their service must compile proto files and what the organizational standard for managing the .proto repository is, not just that they must do it.

Include the schema definition format for each serialization format: whether schemas are defined in .proto files (Protocol Buffers), Avro IDL or JSON-Schema (Avro), OpenAPI / JSON Schema (JSON APIs), or informally in code. The schema definition format determines where the schema is stored, how it is distributed to consuming services, and what tooling is available to validate schema changes before deployment. A .proto repository that is a separate git repository from each service requires a process for pinning each service to a proto repository commit and for propagating proto changes across services. An inline JSON Schema embedded in each service's configuration requires a process for ensuring that all services that participate in the same message exchange have synchronized schema definitions. The schema definition format and storage approach should be explicit because they are decisions that affect all future service additions, not just the initial services.

2. Schema registry strategy and compatibility enforcement

Document the schema registry configuration for formats that use one (Avro with Confluent Schema Registry, Protobuf with a schema registry, JSON Schema with a registry). Include the registry URL, the authentication mechanism, the subject naming strategy (whether subjects correspond to topic names, topic plus message type, or fully qualified schema names), the default compatibility mode and the rationale for the choice, and the process for overriding the default compatibility mode for a specific subject. The compatibility mode choice is the most consequential schema registry configuration — it determines what schema changes are allowed without coordination across all producers and consumers. Document the implication of the chosen mode for deployment order: if FORWARD compatibility is the default, a new consumer can be deployed before the new producer because the old producer's messages are compatible with the new consumer's schema; if BACKWARD compatibility is the default, the new producer must be deployed before the new consumer. The deployment order implication is not obvious to engineers who have not worked with schema registries before, and it should be explicit in the decision record because a violated deployment order in a BACKWARD-compatibility-configured registry can cause new consumers to receive messages they cannot deserialize.

Include the schema registration workflow in CI: how schemas are validated against the registry's compatibility rules before they are deployed. The typical workflow is: the PR that changes a .proto or Avro schema file triggers a CI step that submits the new schema to the registry in a test mode (validate without registering) and reports the compatibility check result. If the compatibility check fails, the PR is blocked until the schema change is made compatible or a compatibility override is requested and approved. Without CI integration, schema compatibility is checked manually by the engineer who registers the schema — a step that can be skipped under deployment pressure, leading to a schema change that reaches production and breaks consumers before the incompatibility is detected. The CI integration for schema compatibility checks connects to the CI/CD pipeline decision record: the schema compatibility check is a quality gate in the same category as tests and linting, and it should be enforced with the same pipeline integration.

3. Schema evolution compatibility policy and field lifecycle management

Document the field lifecycle policy for the chosen serialization format — how fields are added, deprecated, and removed. For Protocol Buffers, the field lifecycle policy must explicitly state the field number reservation requirement: when a field is deprecated and removed from active use, its field number must be added to the reserved list in the .proto definition, and reserved field numbers must never be reused in future field additions. The reservation statement should be accompanied by a comment explaining the original field name, its purpose, and the date it was reserved, so that engineers reading the .proto file can understand why certain numbers are absent from the active field list. Include the field naming convention for deprecated fields: whether deprecated fields are renamed with a DEPRECATED_ prefix before removal (to make the deprecation visible in generated code and to prevent future fields from accidentally using the same name), or simply marked with the deprecated = true option in the proto definition. The naming convention for deprecated fields is not enforced by the proto compiler — it must be part of the documented policy to be applied consistently.

For Avro, the field lifecycle policy must specify the process for adding a new optional field (add with a default value, register as BACKWARD compatible), deprecating an existing field (mark with a documentation annotation, plan the producer-consumer migration timeline), and removing a deprecated field (coordinate with all known consumers to verify they are not reading the field, register the removal under the configured compatibility mode, remove from all producer schema definitions). Include the migration timeline requirement for field removal: the minimum time between deprecation annotation and field removal, the notification process for teams that operate consumers reading the deprecated field, and the verification process for confirming that no consumer reads the field before it is removed from the schema. Without a minimum migration timeline and a notification process, field removal is an ad-hoc judgment call made by the engineer who wants to clean up the schema, and the timeline may be insufficient for teams running consumers on slow deployment schedules to update their consumers before the field disappears from the schema. This connects to the API versioning decision record — the deprecation lifecycle and migration timeline for internal message fields should follow a similar structure to the deprecation lifecycle for external API fields, even though the audience is internal teams rather than third-party developers.

For JSON APIs without a schema registry, the field lifecycle policy must specify the unknown field handling policy for all consumers and the mechanism for enforcing it. Document the specific framework configuration required to ignore unknown fields in each language and library version in use (for example, @JsonIgnoreProperties(ignoreUnknown = true) on Jackson-deserialized Java classes, FAIL_ON_UNKNOWN_PROPERTIES set to false in the Jackson ObjectMapper, strict_fields=False in a Pydantic model, or the equivalent for the organization's JSON libraries). Document the default-on configuration check: is the unknown-field handling policy enforced by a static analysis rule, a CI test that deserializes a test message with an extra field and verifies no error is thrown, or documented convention only? The enforcement mechanism determines whether the policy is reliably applied to new services or only to services built by engineers who know the convention.

4. Language ecosystem requirements and client library governance

Document the canonical library for each language in use and the minimum version required for each library. For Protocol Buffers, the canonical library for each language is the first-party Google-maintained library (google.golang.org/protobuf for Go, com.google.protobuf:protobuf-java for Java, protobuf on PyPI for Python), and the minimum version should be the version that supports the proto syntax in use (Proto2 vs Proto3) and the specific field options or features used in the organization's .proto files. For Avro, document the canonical library for each language, the known caveats for each library version (such as the fastavro 1.4.7 bug with nullable nested union fields), and the integration test requirement for cross-language serialization: before a new schema or a library version upgrade is deployed to production, a test that serializes a representative sample of each message type on the writer side and deserializes on the reader side in each language combination must pass. This test is the primary defense against the class of bugs that produced the incident above, where a library version bug in one language produced messages that another language's library correctly implemented the spec to reject.

Include the process for introducing a new language to the organization's service fleet. When the first service in a language that has not previously been used writes to a shared Kafka topic or calls a gRPC API, it introduces a new language-library combination to the existing cross-language compatibility surface. The process for introducing a new language should include: identifying the Avro or Protobuf library for the new language, reviewing its spec compliance documentation and known issues, running the cross-language serialization test suite with the new library added as both writer and reader, and verifying the library version against the minimum version required for the features used in the organization's schemas. This process should be documented before the first service in a new language is deployed to production, not discovered after a cross-language deserialization incident. The library governance also applies to library version upgrades: when a library is upgraded in one service, the cross-language test suite should be run to verify that the new library version is compatible with the library versions used by all other services in the fleet. A library version upgrade that breaks cross-language compatibility is harder to diagnose than a schema change because it does not change the schema registry and does not trigger the schema compatibility check.

5. Debugging tooling and observability instrumentation

Document the tooling available for decoding binary messages in each environment. For Protocol Buffers, the decoding tools are protoc --decode MessageType proto/message.proto for decoding a binary-encoded message given the .proto file, or language-specific tools like protoc-gen-go's text format marshaler for producing human-readable output. For Avro messages on Kafka, the Confluent Schema Registry provides a REST API for fetching schemas by ID (GET /schemas/ids/{id}), and the kafka-avro-console-consumer tool decodes Avro messages using the schema registry. For gRPC calls, grpcurl provides a curl-equivalent that decodes Protobuf responses using the gRPC server's reflection API (available when the server enables reflection) or a supplied .proto file. Document which of these tools are installed and configured in the development environment, which are available in the production debugging environment, and what the access requirements are (schema registry credentials, .proto repository access, gRPC server reflection enabled/disabled). An engineer who encounters a deserialization error in production for the first time should be able to read the serialization decision record's debugging section and know how to decode the failing message without googling the toolchain from scratch under incident pressure.

Include the observability instrumentation required for serialization failures. Each service that deserializes messages from a shared message type should emit a metric for deserialization success and failure rates, labeled by message type and schema version. A spike in deserialization failures on a specific message type is the signal for a schema evolution or library version incident — the spike should be visible in the monitoring dashboard before user-facing symptoms appear, not discovered after engineers investigate the downstream effects of messages that failed to process. The schema version label allows the monitoring to distinguish between failures caused by a new schema version incompatibility (the failure rate spikes after a producer deployment) and failures caused by a library bug (the failure rate spikes after a library upgrade). Document the metric names, the label set, and the alert threshold for each service. The observability platform decision record covers the monitoring infrastructure; the serialization decision record's observability section specifies the metrics and alerts that are specific to the serialization format and schema evolution surface. Both must be present for the organization to detect and respond to the class of incidents described in the opening narratives before they cause user-visible data corruption or extended processing failures.

The five sections above are unified by the observation that the serialization format choice is a fleet-level decision, not a service-level decision. When an engineer selects a JSON deserialization configuration for a new service, they are choosing whether that service participates in the organization's schema evolution policy or creates a new variant that other services must accommodate. When an engineer adds a new field to a proto message, they are changing the schema for every consumer of that message type, not just the service they are working on. When an engineer upgrades an Avro library in one service, they are introducing a new version into the cross-language compatibility surface that all consumers depend on. The serialization decision record is the mechanism by which fleet-level constraints — the field number reservation rule, the unknown field handling policy, the cross-language integration test requirement — are communicated to engineers who make service-level changes without understanding the fleet-level consequences. A two-day Protobuf field number corruption incident and a four-day Avro library version investigation are both cases where the engineer who made the change did not know about the fleet-level constraint their change violated. The decision record exists to make the constraint known at the moment the change is being made, not after the incident report asks why nobody knew.

What to do with the data serialization decision record

The data serialization decision record's primary audience is the engineer who adds the first field to a message type, the engineer who removes a deprecated field, and the engineer who builds the first service in a new language. Each of these engineers is making a change that affects the fleet-level serialization compatibility surface, and each needs to know the field lifecycle policy, the unknown-field handling requirement, or the library governance process before making the change — not after the incident that demonstrates the constraint was violated. The decision record is the document they read before they make the change, and the absence of a decision record means each engineer defaults to their best guess about the constraint, which will be correct for the engineers who have deep familiarity with the chosen format's specification and incorrect for the engineers who are encountering the format's edge cases for the first time.

If the serialization decision record was not written when the format was selected, the most useful reconstruction is not a comprehensive history of every schema change since the first service was built. The most useful starting point is to answer four questions today: what is the field lifecycle policy for the serialization format in use (specifically, what happens to field numbers or field names when a field is removed), what is the required unknown-field handling configuration for each language in use, what is the canonical library version for each language in use and what are the known compatibility caveats, and what is the toolchain for decoding binary messages in production when a deserialization failure needs to be investigated? Those four answers, documented in a single file in the shared schema repository alongside the .proto files or Avro schemas, are the serialization decision record that the next engineer who touches a schema will find most valuable.

The ADR template provides a starting structure. The WhyChose extractor can surface serialization-related decisions made in ChatGPT or Claude conversations during the format selection sprint — the performance benchmarking discussion where Protobuf was chosen over JSON, or the schema registry evaluation where Confluent was selected over a custom solution, may contain reasoning about the compatibility mode and the field lifecycle policy that was never written into a document and that is now recoverable from the chat export. The decisions never written down essay covers why serialization format choices are among the decisions whose absence is least visible day-to-day — everything serializes and deserializes correctly in the happy path — and most expensive to discover when a field number is reused, a library version diverges, or an unknown-field policy mismatch turns a producer schema update into a fleet-wide breaking change. The API schema design decision record and the queue and messaging decision record together with the serialization decision record form the complete wire format governance for a service-oriented architecture — three documents whose combined scope covers what data is sent, how it is encoded, and through which infrastructure it travels.