On May 25, 2026, InfoQ published The Schema Proliferation Problem in Kafka and Flink Pipelines: How to Solve It, sparking broad discussion across the enterprise data infrastructure community. The article breaks down the proliferation of Avro, Protobuf, and JSON Schema definitions alongside surging Kafka topic counts, downstream consumer failures caused by compatibility violations, and semantically redundant schemas defined independently across business units, emphasizing the necessity of operating schema registries, naming conventions, compatibility policies, and lineage visualization in unison. On the same day, InfoQ also released Architecting Cloud-Native Kafka: From Tiered Storage Towards a Diskless Future, highlighting an era that demands both modernized Kafka architecture and governance at the schema layer.
For custom development firms supporting data platforms and event-driven architectures at mid-market enterprises, this presents an engagement opportunity for organizations facing the reality where "Kafka was deployed, but topics and schemas proliferated across departments beyond anyone's control" to implement "cross-organizational schema governance." Connecting with the data layer interoperability covered in BigQuery × Iceberg Open Lakehouse Engagements, the real-time ML infrastructure in Uber Eats Generative Recommender Implementation Engagements, and the data lineage visualization in Bintrail MySQL Time Travel Engagements, we outline schema governance tailored to the Kafka × Flink layer as an architectural methodology for custom development projects.
Why "schema governance is a turning point"
| Dimension | Existing "laissez-faire schema management" | Schema governance design |
|---|---|---|
| Schema duplication | 3–5 duplicate synonymous schemas | Single canonical schema |
| Compatibility checks | Manual / noticed only after outages | Automated via registry |
| Naming Conventions | By department/team | Company-wide unified |
| Evolution policy | Ad hoc | Explicitly define BACKWARD / FORWARD / FULL |
| Data lineage | Unknown | Visualized with OpenLineage / Marquez |
| Consumer impact | Discovered through incidents | Detected before deployment |
| Documentation | Outdated / None | Synchronized with registry |
| Reusability | Widespread copy-pasting | Converted into schema libraries |
In other words, schema governance is a design decision that transforms "secondary growth that begins the moment Kafka is launched" into "an operational foundation serving as organizational data contracts."
Three structural changes beneficial to custom development projects
Structure 1: From "departmental Kafka silos" to "company-wide data contracts"
In mid-sized companies, it is typical for marketing, sales, product, and SRE to each stand up their own Kafka topics independently, resulting in three different versions of the same "customer event." Schema governance establishes a mechanism to review, version-control, and guarantee compatibility across the company as "data contracts." This is the streaming layer version of the same philosophy as the lakehouse layer standardization covered in our BigQuery × Iceberg Interoperability Custom Development.
Structure 2: From "compatibility violations discovered via incidents" to "pre-deployment detection"
By enforcing compatibility policies (BACKWARD / FORWARD / FULL) through a schema registry and detecting violations in CI/CD, incidents where "a Friday production deployment wipes out all consumers by Monday" are physically prevented from happening. This implements the same culture of prevention at the Kafka layer as the pre-deployment verification covered in our custom pip dependency cooldowns development.
Structure 3: From "untraceable once pushed to Kafka" to "visible data lineage"
Visualizing data lineage using OpenLineage, Marquez, and DataHub creates an environment where you can answer within 30 seconds "which consumers are using this field." This applies the same philosophy behind data traceability covered in our custom Bintrail MySQL time travel development to event streams.
The five phases of "Kafka × Flink schema governance" provided in our custom development services
Phase 1: Current state assessment (2–3 weeks)
- Comprehensive inventory of all Kafka topics and schemas
- Detection of schema duplicates and compatibility violations
- Creation of a consumer dependency map
- Investigation of Flink job dependencies
- Review of incident history and RCAs
- Governance maturity assessment (5 levels)
Phase 2: Governance design (2–3 weeks)
- Naming conventions (domain.entity.event_type.v1)
- Compatibility policy (Mandatory: BACKWARD / Recommended: FULL)
- Schema evolution process (propose → review → merge)
- Data contract templates (owner / SLA / sensitivity level)
- Governance KPI design
Phase 3: Registry infrastructure setup (3–4 weeks)
- Selection among Confluent Schema Registry / Apicurio / Karapace
- Unification policy for Avro / Protobuf / JSON Schema
- CI/CD integration (compatibility checks)
- Schema catalog UI (Backstage / in-house)
- Data lineage (OpenLineage / DataHub / Marquez)
Phase 4: Legacy schema migration (4–6 weeks)
- Consolidation and refactoring of top 20 topics
- Canonicalization of duplicate schemas
- Phased consumer migration
- Sunsetting process for deprecated topics
- Migration progress dashboard
Phase 5: Monthly operational reviews (ongoing)
- Review of new schema proposals
- Inventory of compatibility violations and exception requests
- Updating data lineage
- Consumer SLA reviews
- Semi-annual review of conventions and policies
Standard technology stack set for custom development
| Layer | Recommended technology | Alternative |
|---|---|---|
| Message broker | Apache Kafka / Confluent Cloud | Redpanda / WarpStream |
| Stream processing | Apache Flink / Flink SQL | ksqlDB / Spark Structured Streaming |
| Schema registry | Confluent Schema Registry / Apicurio | Karapace |
| Serialization | Avro / Protobuf | JSON Schema |
| Data lineage | OpenLineage + Marquez / DataHub | Atlan |
| Catalog | Backstage / DataHub | Amundsen |
| CI/CD integration | GitHub Actions / GitLab CI | CircleCI |
| Observability | Grafana + Prometheus / Datadog | Honeycomb |
Which projects need this and which do not
| Projects requiring this | Projects not requiring this |
|---|---|
| 30+ Kafka topics / cross-departmental use | 5 or fewer topics / single-team dedicated use |
| Multi-language consumers (Java / Go / Python, etc.) | Single language / single service |
| ML features / audit requirements present | Ad-hoc analytics only |
| Data lake integration (BigQuery / Snowflake / S3) | Self-contained entirely within streams |
| Past experience with cascading failures across departments | Fully controlled company-wide by a single team |
Six clauses to include in client contracts
| Clause | Details | What the client should verify |
|---|---|---|
| Target scope | All Kafka clusters / specific environments only | Development / production boundaries |
| Data contract permissions | Segregation of duties for proposals, reviews, and approvals | RACI matrix |
| Compatibility SLA | Remediation deadlines after violation detection | Consumer blast radius |
| PII / Confidential data | Schema-level masking | Regulatory requirements |
| Handover Upon Project Completion | Conventions / Registry / Lineage | Internal operational continuity |
| Incident operations | Escalation and rollback procedures | 24/7 / business hours |
Client-side ROI projection (assuming 120 topics / 60 developers / 80 consumers)
| Item | Existing (laissez-faire schemas) | After governance implementation | Difference |
|---|---|---|---|
| Schema compatibility violation incidents | 18 incidents/year | 2 incidents/year | -16 incidents |
| Average time from violation to recovery | 6 hours | 30 min | -5.5 hours/incident |
| Duplicate schema consolidation effort | One-off 80 hours/year | Backlog clearance 8 hours/month | Approx. -16 hours |
| Consumer implementation rework | 12 cases / month | 2 incidents / month | -120 incidents/year |
| Audit response hours | 120 hours/year | 30 hours/year | -90 hours |
| Annual benefit | — | — | Equivalent to approx. 18 million JPY + improved audit compliance |
At a rate of 8,000 JPY per hour, this translates to annual labor savings exceeding 15 million JPY plus incident avoidance. When making investment decisions, the most practical approach is to estimate the payback period by comparing this cost reduction against the internal labor costs (or external contractor fees) required to build and operate the governance foundation.
Five common pitfalls
Pitfall 1: Making naming conventions overly strict
If you introduce six or seven hierarchy levels like <org>.<tenant>.<bounded_context>.<entity>.<event>.<v>, departments will resist and arbitrary naming practices will eventually return. Start with a maximum of four levels and review them semi-annually.
Pitfall 2: Enforcing FULL compatibility policy from day one
Enforcing FULL compatibility during the PoC stage prevents schemas from evolving and causes departmental frustration to boil over within the first few months. Operate initially with mandatory BACKWARD + recommended FULL.
Pitfall 3: Consolidating legacy schemas all at once
Attempting to consolidate all 200 topics within six months will overwhelm consumer teams and cause migrations to fail. Plan across 12-month increments using a cycle of deprecation notice → parallel run → decommission.
Pitfall 4: Leaving data lineage for later
If you set up only the registry and treat lineage as an afterthought, operations begin without anyone understanding "what breaks if this field is modified." Design the system so that lineage is included from the initial build phase.
Pitfall 5: Insufficient testing on the consumer side
Governing schemas only on the producer side while ignoring consumer-side test coverage will undermine the effectiveness of governance. Integrate consumer-driven contract tests directly into CI.
90-day action plan
| Week | Action |
|---|---|
| Week 1〜3 | Full topic/schema inventory + initial version of lineage |
| Week 4〜5 | Naming conventions + compatibility policy + data contract templates |
| Week 6〜9 | Registry + CI/CD integration + catalog UI construction |
| Week 10〜11 | Consolidation PoC for top 10 topics |
| Week 12 | Governance KPI dashboard launched |
| Week 13 | First monthly review + 12-month migration plan |
Conclusion — The real data platform challenge begins after Kafka is launched
The schema proliferation issue in Kafka × Flink is not something that ends once deployed; it is a challenge that inevitably emerges 12 to 24 months after launch, rather than "being done once set up." From the standpoint of supporting data platform governance for mid-sized enterprises through custom development, "schema governance"—which packages governance design, registry setup, lineage, and monthly operations into an integrated offering—serves as our new flagship service.
Situations such as having too many Kafka topics spawned across individual departments, frequent outages caused by compatibility violations, or having no visibility into data lineage require completely different solutions depending on topic count, consumer language stacks, and the presence of an existing registry. We provide tailored estimates for schema governance architecture design and migration roadmaps after reviewing your specific environment. Please feel free to reach out via our inquiry form.
Sources
- The Schema Proliferation Problem in Kafka and Flink Pipelines（InfoQ 2026-05-25）
- Architecting Cloud-Native Kafka: From Tiered Storage Towards a Diskless Future（InfoQ 2026-05-26）
- Custom Open Lakehouse Development with BigQuery × Iceberg (GH Media)
- Custom Implementation of Uber Eats-Style Generative Recommender (GH Media)
- Custom Bintrail MySQL Time Travel Development (GH Media)
