Event-Based Architecture: a year in review

Welcome to "Event-Based Architecture: A Year In Review", as we look back at the influential trends that revolutionized the Event-Driven Architecture (EDA) landscape in 2023. With insights from Kris Van Vlaenderen, Wout Florin, and Bryan De Smaele, the managing partners at Cymo, we delve into a retrospective analysis to underline the key developments that shaped the dynamic world of EDAs.

This comprehensive overview gathers our experienced perspectives, technical breakdowns, and a glimpse into the future of EDA. And we, the team at Cymo, are proud to lead you through this retrospect and explore what lies beyond.

Growing Significance of Apache Flink

As the architectural complexities of EDA systems continue to grow into 2023, developers have turned to increasingly sophisticated tools to handle massive data streams. One such tool is Apache Flink, an open-source, unified stream-processing and batch-processing framework. Born out of the realization that Apache Kafka alone could not fully satisfy the needs of developing and deploying high-performance, reliable EDAs, Apache Flink emerged to bridge this gap.

Kafka indeed has its prowess as an event streaming platform, allowing for real-time data storage, processing, and transfer. However, for processing data that’s available on the platform you need to implement streaming logic. Kafka’s stream API is a java API, which limits it’s use for other environments. It’s also more low level than some users might wish for.

Flink enhances Kafka’s features, providing facilities for complex event processing, state management, and event-time processing, as well as ensuring fault tolerance. It facilitates comprehensive data computations, handling both stream and batch processing smoothly while maintaining high throughput and low-latency characteristics. By doing so, it enhances the overall designing and implementation of EDAs, allowing for the creation of more reliable, flexible, and complex applications.

Birth of the EDA Community

The EDA Community, a significant milestone for EDA enthusiasts, was brought to life by Cymo and the i8c team within the Cronos Groep. Bryan is proud of this creation, stating, "This vendor-agnostic EDA community enables members to exchange knowledge and insights, propelling the future of EDA together."

Tailored for both seasoned professionals and bright-eyed learners, this community provides an open forum for asking questions, exchanging thoughts, and unveiling new insights. Emerging from varied backgrounds, the community members share a common goal: shaping the future of EDA technology. Sign up and get connected today at www.eda.community.

The EDA Community is also thrilled to announce the upcoming EDA Community Meet-Up slated for Thursday, 22nd February 2024. This gathering is not merely a meet-up, but an opportunity to connect directly with peers and industry experts steeped in EDA.

Kafka Raft (Kraft) Scores High

Apache Kafka has been paired with Apache Zookeeper since its inception to handle critical functionality such as configuration management, distributed synchronization, and group services. However, the dependency on Zookeeper has often been a source of complexity and operational overhead. Enter Kraft (short for Kafka Raft), a significant development in Apache Kafka as it aims to replace Zookeeper and simplify Kafka's architecture.

Kris reflects on the change saying, "Although the shift to Kraft might appear mainly as an internal transformation in Kafka's architecture, it brings along tangible benefits for the users. The associated failover performance improvement and support for right-sized clusters are substantial advancements that shouldn't be overlooked. This modernized architecture puts Kafka back in line with newer players like RedPanda.”

The Rise of Event Catalogues and Event Management

As the complexity and importance of Event-Driven Architectures continue to grow, effective event management and the use of event catalogs have become increasingly crucial. An event catalog provides a centralized view of all the events in a system, detailing vital information about each one, such as its source, schema, and associated applications. Modern platforms like Solace PubSub+ Event Portal, IBM Event Catalog, and Confluent Stream Catalog are spearheading this development, all offering integrated event catalogs that streamline the handling and organization of event information in complex systems.

Technologies like AsyncAPI are also playing a key role in improving event management by offering comprehensive tooling for event-based APIs. AsyncAPI enables automatic generation of code, documentation, and testing tools based on the API specification, making the development process more efficient and resilient. Furthermore, it introduces a fresh approach to visualizing and understanding system architecture through event-driven views, which can significantly help developers and architects navigate increasingly intricate event landscapes.

Event Streaming Meets AI: The Rise of Real-Time Machine Learning

The convergence of event streaming and machine learning signifies another significant trend of 2023. The highly structured, continuous data flow in event streaming acts as perfect input for machine learning algorithms, significantly enhancing the ability to process and analyse data on the fly. The shift towards streaming machine learning in business environments is dismantling various bottlenecks associated with traditional batch processing models, such as the latency of insights and data lake maintenance overhead.

The adoption of streaming machine learning in 2023 has further been solidified as businesses are leaning towards real-time analytics and decision-making. The algorithmic ability to comprehend, learn, and make predictions or decisions based on real-time data streams ensures immediate, accurate responses. With event streaming facilitating the capture, processing, and transfer of business events, the integration of machine learning merges the power of prediction, detection, and real-time analysis.

Kannika: The Future of Event Driven Data Backup & Restore

Looking back to the EDA trends of 2023, data security and recovery have emerged as prime concerns. Addressing these critical areas, we at Cymo have developed Kannika - a premier backup and restore product. Designed to secure your event data, Kannika facilitates the offloading of your important business information to a file storage system of your choice.

With the flexibility of connection to various event-hubs and storage locations, uncomplicated backup procedures, quick data backup and restoration, and adaptable restore functionalities, it promises to simplify data protection. Plus, its deployable capabilities, both as a SaaS and within your secure cloud environment, make it an accessible choice for all. As we step further into 2023, solutions like Kannika are setting a new standard for data protection in the realm of EDAs, making them a trend worth watching.

Conclusion

Event-Driven Architecture witnessed an expansive trajectory of growth and innovation throughout 2023. From the increasing significance of Apache Flink and the Kafka Raft transition to the rise of event catalogs and seamless event management, the EDA landscape has evolved remarkably. The emergence of streaming machine learning, moreover, highlighted the synergy between real-time data streams and AI, crafting a new fabric for data analysis and decision making. Cymo’s own Kannika too left its mark, bolstering data protection standards for EDA.

In retrospect, there's no denying the transformative role each trend played in shaping the EDA domain in 2023. And as we groove into another promising year, we remain positive about the continued growth of EDAs, powered by an ever-evolving set of tools, practices, and innovative minds. Here’s to a future crafted by resilient event-driven architectures, paving the way for businesses to reach new pinnacles of success. This journey is not only a review, but also a toast to the exciting developments that await us.