So, you want to build a stream processing pipeline to do some serious data crunching on a massive data feed. Of course, it should be scalable and resilient to data loss. Easy, we’ll use Kafka and Kubernetes of course, but having to manage and integrate with tools like Kafka is complex and time-consuming. Not to mention the YAML-hell that a platform like Kubernetes introduces.
Meet Cloudflow! Cloudflow is Lightbend’s latest product aimed at reducing the time required to create, package and deploy streaming data pipelines on Kubernetes. It offers powerful abstractions allowing you to define the most complex streaming applications. It also seamlessly integrates with streaming platforms like Akka Streams, Flink and Spark.
In this talk, we’ll introduce the reasoning and concepts behind Cloudflow and we’ll demonstrate how it helped us build a stock market analysis tool. We’ll start by introducing the fundamentals of Cloudflow followed by an introduction of the problem domain. Then we go over each of the required components, from ingestion to final output and experience how Cloudflow enables us to rapidly develop our application. We’ll demonstrate how to test and validate our new application locally before pushing it to production. Finally, we’ll end up with a fully working solution running on Google Kubernetes Engine.