Event-Driven Cloud-Native ML Pipelines in Continuous Intelligence Systems

Abstract

Continuous intelligence (CI) systems represent the fusion of real‑time analytics, automated decision‑making, and feedback loops that continuously refine operational processes. At the core of these systems lie event‑driven, cloud‑native machine learning (ML) pipelines that respond instantaneously to incoming data, performing preprocessing, inference, and result dissemination without human intervention. In this work, we present an end‑to‑end design and empirical evaluation of such a pipeline. We begin by motivating the need for event‑driven architectures in CI, highlighting latency, throughput, and scalability requirements in domains ranging from fraud detection to IoT monitoring. We then describe our cloud‑native deployment, leveraging managed Kubernetes with Knative Eventing for serverless event routing, Apache Kafka on Confluent Cloud as the streaming backbone, and KServe for scalable model serving. Our implementation also integrates Kubeflow pipelines for periodic retraining and OpenTelemetry/Prometheus for observability. To assess performance, we simulate a continuous stream of synthetic sensor events at rates up to 5,000 events per second, measuring metrics such as end‑to‑end latency (ingestion to inference to sink), throughput, resource utilization, autoscaling behavior, and fault‑recovery time under injected failures. Experimental results demonstrate median latencies under 150 milliseconds, linear throughput scaling with cluster size, and recovery times below 10 seconds with fewer than 0.5% lost events. Observability data reveals how feature‑drift alerts and confidence‑based routing enhance model reliability in production.