Concept projects

Data engineering project ideas: Systems I’m designing next.

Four architecture-level designs I’m planning to build. They are concepts, not shipped products, laid out the way I’d approach each one in production.

Concept · Architecture

Streamline — Real-time streaming

A clickstream platform that ingests millions of events per hour, enriches them in flight and serves live metrics in under a few seconds.

Web / App events→Kafka→Spark Structured Streaming→ClickHouse→Grafana
Problems it solves
  • Exactly-once processing with idempotent sinks and checkpointing
  • Late and out-of-order events handled with watermarks
  • Back-pressure aware consumers so spikes never drop data
  • Schema registry to keep producers and consumers in sync
Planned stack
KafkaSparkClickHouseSchema RegistryDockerGrafana
Contact

Let’s build something solid.

[email protected]
LinkedIn GitHub Send an email