Integration platform

The problem

Organisations rarely have one system. They have an ERP that owns orders, a CRM that owns customers, a PIM that owns products, and a growing set of tools that all need to know when any of those change. The default answer is point-to-point: the ERP calls the webshop, the webshop calls the data platform, someone writes a nightly job to patch the gaps.

That works until roughly the fourth system. After that, every new consumer means changing the producer, every schema change breaks something downstream, and nobody can answer "where did this value come from?" without reading code.

The pattern below breaks that coupling with one deliberate constraint: the bus announces that something changed, it does not describe what changed.

How it works

Five steps, and the fifth is the one that defines the pattern:

1. A record changes in a system of record — the one system allowed to own that entity.
2. A change publisher turns that into a thin event: what changed, where to find it, and the handful of attributes subscribers route on. This is a transactional outbox, a CDC stream, or a native webhook — whatever the source can offer reliably.
3. The bus filters and fans out. Each subscription declares the events it wants; nobody receives traffic they did not ask for.
4. Each subscriber receives the reference.
5. The subscriber reads the full record back from the source system's API and updates its own store.

What goes in the event

The envelope is CloudEvents 1.0 — a spec Event Grid, EventBridge and Pub/Sub all understand, which keeps the pattern portable.

{
  "specversion": "1.0",
  "type":        "customer.updated.v1",
  "source":      "/erp/prod/customers",
  "id":          "0f3a9f2c-8b21-4d0e-9e07-2b1f5a6c7d10",
  "subject":     "customer/NL-100482",
  "time":        "2026-08-11T09:14:22.117Z",
  "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
  "datacontenttype": "application/json",
  "data": {
    "entityType": "customer",
    "entityId":   "NL-100482",
    "operation":  "update",
    "sequence":   42,
    "self":       "https://api.contoso.com/erp/v1/customers/NL-100482",

    "country":    "NL",
    "segment":    "enterprise"
  }
}

Everything above the blank line is structural. Everything below it is a routing key — and those are the only business fields allowed on the bus.

Filtering and fan-out

Filtering belongs in the bus, not in subscriber code. A subscriber that receives everything and drops 98% of it is paying for compute, egress and on-call attention it does not need — and every new event type becomes a deployment for every consumer.

{
  "filter": {
    "includedEventTypes": ["customer.created.v1", "customer.updated.v1"],
    "advancedFilters": [
      { "operatorType": "StringIn", "key": "data.country", "values": ["NL", "BE"] },
      { "operatorType": "StringIn", "key": "data.segment", "values": ["enterprise"] }
    ]
  },
  "retryPolicy": { "maxDeliveryAttempts": 30, "eventTimeToLiveInMinutes": 1440 },
  "deadLetterDestination": { "endpointType": "StorageBlob", "...": "..." }
}

This is also why event types carry a version suffix (customer.updated.v1). Adding v2 alongside v1 lets subscribers migrate one at a time instead of all at once on a date somebody picked in a meeting.

Reading back from the source

The read-back is the part that gets under-engineered. A naive subscriber that issues one GET per event will turn a bulk update of 50,000 products into a denial-of-service attack on the PIM.

• Conditional reads. Store the ETag per entity and send If-None-Match. A 304 costs the source almost nothing.
• Coalesce. Buffer events per entity over a short window (5–30s) and read once. Ten updates to one order become one read — correct, because you only ever wanted the latest state.
• Batch endpoints. GET /customers?ids=a,b,c beats n round trips.
• Read replica or data product. Point subscribers at a read-optimised copy rather than the transactional database.
• Back off. Honour 429 and Retry-After; let the queue absorb the delay. The bus is your buffer — use it.

A subscriber handler, in full:

def handle(event):
    ref = event["data"]
    key = (ref["entityType"], ref["entityId"])

    if ref["operation"] == "delete":
        store.soft_delete(key, ref["sequence"])
        return

    # Read the current state from the system of record.
    record = source_api.get(ref["self"], if_none_match=store.etag(key))
    if record is None:                     # 304 Not Modified — nothing to do
        return

    # Last writer wins on the *source's* sequence, not on arrival order.
    if record["sequence"] <= store.sequence_of(key):
        return                             # stale or duplicate delivery

    store.upsert(key, record)

Deletes and the missing record

Deletes break the read-back, because the whole point of a delete is that there is nothing left to read. Handle it explicitly:

• The delete event is a tombstone — it must be actionable on its own, from entityId and sequence alone.
• A read that returns 404 is not an error. It means the entity is gone or the subscriber is not authorised to see it; either way, remove it locally.
• Prefer soft deletes in the source. GET returning {"status": "deleted"} keeps one code path instead of two.

Delivery guarantees you must design for

Failure handling and replay

Every subscription gets a dead-letter destination and an alert on its depth — a silently filling DLQ is the most common way this architecture fails in production. Because events are thin, replaying a DLQ is cheap and safe: re-delivering an old reference simply causes a fresh read of the current record.

That same property is also this pattern's sharpest edge. Replaying an event from last Tuesday gives you today's data, not Tuesday's. The bus is a change notification log, not an audit log. If you need point-in-time reconstruction, that is a job for the source system's history tables or a versioned store in the data platform — not for the bus.

Security and data protection

• The bus holds no personal data. Access requests, retention rules and breach scope stay with the systems of record — an enormous simplification under GDPR.
• Authorisation stays at the source. Each subscriber authenticates to the source API with its own identity and sees only its own scope. A subscription grants notification, not access.
• Managed identity, not keys. Publishers and handlers authenticate with workload identity; no connection strings in config.
• Private networking for the read path where the source is internal.
• Validate the event source. Handlers verify signature or issuer — an endpoint that trusts any POST is an open write path into your systems.

Observability

Three things make this operable, and all three are cheap if you build them in from the start:

• Trace context end to end. Propagate traceparent from the source transaction through the event, the handler and the read-back, so one trace spans all five steps.
• Lag per subscription. Measure handled_at − occurred_at. This is the number that tells you whether the platform is healthy; queue depth alone does not.
• Reconciliation. A periodic job comparing counts and checksums per entity type between source and subscriber. Events will be missed. Reconciliation is how you find out before your users do.

Thin events vs fat events

The alternative — putting the full entity in the event — is a legitimate pattern with a different set of costs. The choice is real, so make it deliberately.

A common middle ground is worth knowing: keep the bus thin, but let the publisher write a full snapshot to object storage and put the blob URL in self (the claim-check pattern). Subscribers keep one read-back code path, and the source no longer takes the read load.

When to use it — and when not to

Platform mapping

Nothing here is Azure-specific. The pattern maps onto every major bus, with one caveat per platform.

Implementation checklist

1. Name the system of record for every entity type. One owner each; write this down before writing code.
2. Agree the event envelope and the naming convention (<entity>.<verb>.v<n>). Version from day one.
3. Decide the routing keys per entity type — and defend the list. It only ever grows.
4. Implement publishing as a transactional outbox, not a best-effort call after commit.
5. Expose a read endpoint per entity type with ETag support and a batch variant.
6. Give every subscription its own dead-letter destination, retry policy and alert.
7. Make handlers idempotent and sequence-aware before the first one goes live, not after the first incident.
8. Propagate traceparent everywhere.
9. Ship the reconciliation job with the first subscriber, not with the fifth.
10. Publish the event catalogue where consumers can find it — a schema registry, or at minimum a versioned repository of the contracts.