Distributed stream processing
2025
SDE Spark
Synopsis Data Engine: a distributed system that maintains and queries probabilistic data structures over continuous Kafka streams, built on Apache Spark Structured Streaming.
- ▸ Unit, end-to-end and load-test harnesses covering every synopsis.
- ▸ Stateless hash routing across N slots; partial estimates are merged within the same micro-batch with no cross-batch buffering.
- ▸ Kafka in, Kafka out: JSON requests become typed add / estimate / delete events, and estimations are published back to Kafka.
Synopses
-
Count-Min
frequency estimates
-
Bloom filter
set membership
-
AMS
join / self-join size
-
HyperLogLog
distinct counts