Scaling Predictive Analytics to 1M+ Events per Second
Company
Engineering
Scaling Predictive Analytics to 1M+ Events per Second
A deep dive into our infrastructure choices for CrespoAI's real-time predictive engine and how we maintain <20ms latency.
Our first ingestion pipeline handled about 12,000 events per second and fell over predictably every Monday morning. Rebuilding it for two orders of magnitude more throughput meant giving up a few things we thought were requirements. The biggest one: exact ordering. Global ordering across a distributed stream is expensive and, for our workloads, almost never necessary.
Partition on the entity, not the timestamp Ordering guarantees per customer are cheap. Global ordering is not. Partitioning by account gave us the guarantee that actually mattered. Separate hot and cold paths Real-time detection reads a rolling in-memory window. Historical training reads columnar storage. Forcing one system to serve both was the original bottleneck. Backpressure over buffering Unbounded queues turn a slow consumer into an outage an hour later. Explicit backpressure fails loudly and immediately, which is easier to operate. Sample aggressively at read time Most queries do not need every event. Reservoir sampling at query time cut p99 latency by more than half with no measurable loss in accuracy.
We over-invested in exactly-once delivery early on. For analytics workloads, idempotent writes with at-least-once delivery would have delivered the same correctness for a fraction of the complexity.
"Scaling is mostly the discipline of noticing which guarantees you are paying for and never actually using."
No spam. Unsubscribe anytime.