Use case 1: auto-registering Avro schemas for Dead Letter Topics
One of the core rules in our manifesto states that an Avro schema should be strictly defined and registered through Infrastructure as Code (IaC). This approach is generally superior as it provides a centralized, version-controlled source of truth for your data contracts, preventing schema drift and ensuring consistency.
But consider the case of a Dead Letter Topic (DLT). A DLT is designed to catch messages that fail processing for any number of reasons. By its very nature, we don't always know what message structures or Avro schemas will end up there. Forcing a strict, pre-defined schema on a DLT is counterintuitive and impractical. In this scenario, breaking the rule is the correct engineering decision. Allowing producers to auto-register their schemas upon publishing to a DLT provides the necessary flexibility to handle unforeseen failures without bringing the system to a halt.
Use case 2: handling breaking changes in schema compatibility
Another principle we adhere to is maintaining FULL_TRANSITIVE compatibility for our Avro schemas. This is a robust compatibility setting that allows producers and consumers to be updated in any order without breaking the data flow, providing maximum operational flexibility.
The challenge arises when you need to introduce an intentional, breaking change to a schema. According to strict protocol, the "correct" way to handle this is to create an entirely new topic for the new schema version, migrate all producers and consumers, and then decommission the old topic. This process is technically pure, but it's also incredibly cumbersome and can introduce significant operational overhead.
In these situations, a more pragmatic approach is often justified. By coordinating with the development teams, it can be far more efficient to temporarily relax the compatibility rules, delete the old schema, and register the new one with the breaking change. While this requires careful planning, it avoids the high cost of topic migration and allows the system to evolve more rapidly.
Conclusion: expertise is about judgment, not just rules
Having a strong manifesto is essential for building high-quality, scalable event-driven systems. But it's the expert judgment, the ability to analyze a specific context and decide whether to adhere to a rule or consciously break it, that truly drives a project's success. Rules provide the foundation, but pragmatism builds the optimal solution.
Need an expert partner who understands the deep technical nuances of event-driven architecture?
