Streaming Analytics — Reference Architecture

The problem

Some decisions do not survive a nightly batch. A fraud check that runs tomorrow is a report, not a control. A machine that is about to fail does not care what the dashboard said at 06:00. A customer abandoning a basket is reachable for about four minutes.

The naive fix is to run the batch pipeline more often — every hour, then every fifteen minutes, then every five. That works for a while and then stops, because the cost grows linearly with frequency while the latency improves only down to the floor of a full recompute. At some point you have to stop recomputing the whole thing and start processing the change.

That switch brings one genuinely new problem with it. In batch, the input is complete: you process yesterday, and yesterday is over. In streaming, you never know whether the data is complete — only that nothing has arrived yet. Every hard part below follows from that one fact.

How it works

The layers are unchanged. What is new is the band underneath: a checkpoint and state store that every stream depends on and that nothing in the layer diagram shows you.

Step by step:

Events arrive on a bus, a CDC feed or as files. Each has a replayable position — an offset, a log sequence number, a filename.
readStream consumes from that position in micro-batches, bounded by a rate limit so one backlog cannot overwhelm the cluster.
Bronze appends the payload unparsed, with the source coordinates attached. This is both your audit trail and your replay source.
Silver parses, types, deduplicates within a watermark, and enriches by joining reference data.
Gold aggregates over event-time windows and writes a row per window and key.
Serving is whatever consumes it: a dashboard on the Gold table, an alert evaluated per micro-batch, or a push into a low-latency store for lookups.
The checkpoint underneath carries source offsets and aggregation state. It is what makes restarts correct, and it is the single most under-respected object in the system.

Streaming is a trigger, not an architecture

Bronze, Silver and Gold mean the same things here as in the batch medallion. The layering, the contracts and the governance carry over unchanged. What changes is that you can no longer wait for the input to be complete — so you must decide, explicitly, how long you are willing to wait instead.

Event time, processing time and the watermark

Two clocks exist in every streaming system: when something happened, and when you heard about it. They diverge constantly — a mobile app buffers while offline, a broker rebalances, a network partition heals and delivers an hour of backlog at once.

Aggregate by event time, always. Processing-time windows produce numbers that change depending on how your infrastructure behaved, which makes them unreproducible and, worse, quietly wrong in exactly the incidents you most want to analyse.

But event-time aggregation over an unbounded stream needs a bound, or the engine must keep every window open forever. That bound is the watermark: a declaration that you will accept data up to N behind the latest event you have seen, and discard anything later.

(events
   .withWatermark("event_time", "10 minutes")
   .groupBy(window("event_time", "5 minutes"), "device_id")
   .agg(avg("temperature").alias("avg_temp"),
        count("*").alias("readings")))

The watermark is a business decision

It is a direct trade between latency and completeness, and nobody but the business can price it. Ten minutes means a five-minute window is final fifteen minutes after it opens — and that readings older than ten minutes are dropped, silently, forever. Ask explicitly: how late is too late, and what happens to the data that misses the cut? If the answer is "we cannot lose any of it", route late events to a side table and reconcile in batch. Streaming and batch over the same Bronze is a normal design, not a failure.

The hard parts, in order of how often they bite

Problem	Why it happens	What to do
Duplicates	Sources are at-least-once; a producer retry sends the same event twice	`dropDuplicatesWithinWatermark` on a producer-assigned event ID
Late data	Devices buffer, networks heal, clocks drift	Set the watermark deliberately; measure how much it discards
Unbounded state	High-cardinality keys in a stateful operator never expire	Watermark every stateful operation; monitor state rows and store size
Small files	A short trigger writes a file per partition per batch	Longer trigger, fewer output partitions, scheduled compaction
Stream–stream joins	Both sides must be buffered until the join window closes	Bound both sides with watermarks and a time-range condition; expect the state to be large
Enrichment data changes	A static dimension read at stream start never updates	Read the dimension as a stream, or restart on a schedule and accept the staleness
Schema change at the source	A producer adds or renames a field	Keep Bronze unparsed; parse in Silver, where a change is a code change, not an outage
Checkpoint loss	Someone deletes the path, or copies a pipeline without changing it	Treat checkpoints as production state: unique per stream, backed up, never shared

Exactly-once, honestly

Structured Streaming gives you exactly-once results on a Delta sink, and it is worth being precise about how, because the guarantee does not extend past the sink.

Each micro-batch has an ID. The source offsets and the resulting Delta commit are recorded together, so on restart the engine knows which batch was committed and which must be reprocessed. Reprocessing a batch produces the same output rows, and the sink refuses to commit the same batch twice. Input may be read twice; output appears once.

This stops holding the moment the sink is something else. A webhook called from foreachBatch will fire again on a replayed batch. A row pushed to an external API will be pushed again. For anything outside the lakehouse, make the operation idempotent using the batch ID:

def publish(batch_df, batch_id):
    # Delta sinks are idempotent per (table, batch_id) via txnVersion/txnAppId.
    (batch_df.write
        .format("delta").mode("append")
        .option("txnVersion", batch_id)
        .option("txnAppId", "alerts_pipeline")
        .saveAsTable("gold.ops.alerts"))

    # External side effects are not. Key them yourself.
    for row in batch_df.where("severity = 'high'").collect():
        notifier.send(idempotency_key=f"{batch_id}:{row.device_id}", body=row.message)

(scored.writeStream
   .foreachBatch(publish)
   .option("checkpointLocation", f"{chk}/alerts")
   .trigger(processingTime="30 seconds")
   .start())

Choosing a trigger

The trigger is the main dial between latency and cost, and it is the one most often set from ambition rather than requirement.

Trigger	Typical latency	Compute	Use for
`availableNow`	Whatever the schedule is	Only while running	Incremental batch — most "streaming" pipelines belong here
`processingTime="5 minutes"`	Minutes	Always on, modest	Operational dashboards, near-real-time reporting
`processingTime="10 seconds"`	Tens of seconds	Always on	Monitoring, anomaly detection, alerting
Default (as fast as possible)	Seconds	Always on, higher	Genuinely time-critical reactions
Low-latency modes	Sub-second	Highest, and operationally demanding	Rare. Verify the requirement is worth it before committing

Start at the top of that table

A pipeline on availableNow scheduled every fifteen minutes uses the same code as a continuous one. Switching it on later is a configuration change, not a rewrite. Starting continuous and discovering nobody looks at the dashboard before nine in the morning is a year of compute you cannot get back.

Serving the results

Where the output goes is a separate decision from how it is computed, and it is where streaming projects most often lose the latency they just paid for.

Delta tables queried by SQL. The default, and fine for dashboards refreshing on a timer. Remember that a dashboard caching for ten minutes makes a ten-second trigger decorative.
Streaming tables and materialized views. Declared in SQL and refreshed incrementally — the lowest-effort route from a stream to something an analyst can query.
Alerting from foreachBatch. Evaluate the condition per batch and emit. Keep the emission idempotent and rate-limited; a flapping threshold will page someone two hundred times.
Back onto the bus. Writing the enriched stream to a topic lets applications consume it without touching the lakehouse. Often the cleanest boundary between analytics and operations.
A low-latency store for point lookups. Analytical tables are not built for single-row reads at high concurrency; push the serving layer to a store that is.

Operating it

A streaming pipeline is a long-running service, and it needs the monitoring of one rather than the monitoring of a nightly job. Four numbers tell you almost everything:

Input rows per second versus processed rows per second. If processing is consistently lower, the backlog grows without bound and the only question is when someone notices.
Batch duration against the trigger interval. Batches taking longer than the interval mean you are already behind.
State store size and row count. The leading indicator for the memory failure that eventually kills the stream — and it rises long before it breaks.
End-to-end lag: now minus the latest event time processed. The number to put in an SLA, because it is the one a user would recognise.

Beyond metrics, three operational habits matter:

Restarts are routine. Clusters are patched, code is deployed. A stream that cannot be restarted cleanly, repeatedly, is not production-ready.
Know your recovery path before you need it. Some changes — altering a stateful operator, changing aggregation keys — are not checkpoint-compatible, and the fix is a new checkpoint plus a backfill. Decide in advance how you backfill, and from where.
Keep the batch path. Bronze is append-only and replayable, so correcting a Silver or Gold bug across history is a batch job against the same tables. The organisations that run streaming well all have this; the ones that do not end up writing it during an incident.

When to use it — and when not to

Good fit

An action is taken automatically on the result, within minutes or seconds
The value of the insight decays fast — fraud, outages, abandonment, safety
Sensor, telemetry or clickstream data that already arrives as a stream
CDC replication where you want the lakehouse to track the source closely
Continuous arrival makes recompute wasteful even without a latency need
Model features that must match between training and online scoring

Poor fit

A human reads it in the morning — a scheduled batch is cheaper and simpler
The requirement is "real-time" but nobody can name the decision it drives
Results must be complete and reconciled, such as financial close
Logic changes weekly; every change is a redeploy of a live service
Sources are batch anyway — nightly extracts do not become fresher by streaming them
No team on call. Streaming pipelines fail at night, and they stay failed

The question to ask first

"What will be done differently if this number is five minutes old instead of five hours old — and who or what does it?" If the answer names an automated action or a person who is actually watching, build the stream. If it does not, you are buying continuous compute and a permanent on-call burden to make a dashboard feel modern.

Implementation checklist

Write down the decision the latency serves, and the latency it actually needs. Design to that number, not to the lowest one achievable.
Confirm the source is replayable and has sufficient retention to cover your worst outage.
Land Bronze unparsed and append-only, with topic, partition, offset and ingest time.
Give every stream its own checkpoint location, named after the stream, and never reuse one.
Set a watermark on every stateful operation, and agree its value with the business.
Deduplicate on a producer-assigned event ID, not on the full row.
Make every external side effect idempotent on the batch ID.
Set maxOffsetsPerTrigger or the equivalent before the first production backlog, not after.
Start on availableNow with a schedule. Move to a continuous trigger only when the requirement demands it.
Alert on end-to-end lag and state size, not only on the stream being alive.
Build and test the batch backfill path over Bronze while nothing is broken.
Rehearse a restart after a non-compatible code change, so the first time is not in production.

Build it with a skill

This checklist exists as a skill — streaming-pipeline makes the checkpoint, watermark and trigger decisions explicit, keys external side effects on the batch ID, and generates the batch backfill path alongside the stream.