Beyond the bootstrap servers: lessons in building a mature Kafka platform
Apache Kafka has a beautifully simple core, which is a key reason for its widespread adoption. Setting up a cluster for a single project is straightforward. But scaling that to a multi-tenant, enterprise-wide platform that serves dozens or hundreds of teams is an entirely different beast. How do you provide a stable, resilient service without becoming a bottleneck?
This is the core challenge of platform engineering. To explore how to navigate this journey, we sat down with Jose Manuel Cristóbal, Director of Platform Engineering at Adidas, on our Talking Event-Driven podcast. He shared his experience of evolving Kafka from a single, on-prem project into a mature, cloud-based platform product.
The journey from project to platform
Like many organizations, the journey at Adidas began with a single, demanding project. This initial solution used Kafka as its core but was a bespoke setup with custom scripts and technologies like Apache Storm, all running on-prem. It was built for one purpose and one purpose only.
However, when other teams saw the power of the event-driven approach, they wanted in. This was the critical inflection point, the shift from a single-tenant project to a multi-tenant platform.
"It's a different beast," Jose explains. "It has nothing to do with running Kafka for one single project than compared to implementing any level of multi-tenancy."
This transition forced a change in thinking. The team had to move from simply building a solution to enabling others to build their own, all while managing the classic challenges of a shared environment, like noisy neighbors and the need for governance.
The platform team as an enabler, not a gatekeeper
One of the biggest risks for a platform team is becoming an ivory tower, an inaccessible group of experts who dictate all the rules. The Adidas team deliberately chose a different path, positioning themselves as a center of excellence that operates on a model of consultancy and shared responsibility.
"Apart from being simply the team that is doing the heavy lifting of a complex technology such as Kafka, we also act as the center of excellence for stream processing," Jose says. "It gives you the possibility of thinking out of the box and seeing the big picture because this is where you are really connected with the implementation of the use cases."
This approach has a dual benefit:
- It builds trust: By offering consultancy and even troubleshooting help, the platform team becomes a trusted partner rather than a gatekeeper.
- It creates a proactive feedback loop: Helping a team solve a problem early on prevents it from becoming a 2 a.m. production outage down the line. It makes life better for everyone and reduces the distance between the platform and the value it enables.
Building knowledge: teach your way, not just the tech
With a self-service model, empowering teams with the right knowledge is crucial. However, the platform team recognized that creating another "Kafka 101" course would be a waste of time. That content already exists in high quality all over the internet.
Instead, they focused on teaching what is unique to their environment: the Adidas way of doing things.
"What we do is explain how do we do things internally," Jose notes. "What is different in Adidas compared to other companies."
This includes custom training on:
- Internal automation: How to use their Git-based process for registering schemas, rather than interacting with the Schema Registry API directly.
- Opinionated choices: Explaining why certain compatibility levels are set by default or why schemas cannot be registered on the fly.
- Operational models: Onboarding new joiners to the specific conventions and guardrails of the platform.
This ensures that developers learn what they actually need to be productive and compliant within the organization, without the platform team having to become full-time teachers.
The platform is a product that is never "done"
A platform is not a project with a defined end date. It's a living product that must constantly evolve. For an established platform, the top priority isn't a flood of new features.
"People don't expect from you new and new and new features," Jose says. "They expect stability, they expect resiliency, they expect as many nines as you can give them."
Treating the platform as a product also means:
- Having a roadmap: Continuously improving the platform by adopting new technological advancements (like tiered storage in Kafka) to improve efficiency and save costs.
- Internal marketing & DevRel: A new feature is useless if no one adopts it. The platform team must proactively communicate benefits, share success stories, and use internal DevRel techniques to drive engagement.
- Cost transparency: Providing teams with clear data on the cost implications of their usage. This fosters a sense of ownership and encourages responsible resource consumption, often more effectively than a complex and rigid internal recharging model.
Conclusion: it's a strategy, not a silver bullet
A successful Kafka platform is the result of a deliberate, long-term strategy. It's about finding the right balance between flexibility and governance, empowering teams with the right knowledge, and treating your internal platform with the same care and attention as an external-facing product. By focusing on enabling others, you build a foundation that accelerates the entire organization.
If you're looking to move from ad-hoc clusters to a mature, enterprise-wide event streaming platform, we have the expertise to help you build the right strategy.
