Distributed Data Consistency Patterns#

In a monolith with a single database, you get ACID transactions. In microservices, each service owns its database. Cross-service consistency requires explicit patterns because distributed ACID transactions are impractical in most real systems.

CAP Theorem: Practical Implications#

The CAP theorem states that a distributed system can provide at most two of three guarantees: Consistency, Availability, and Partition tolerance. Since network partitions are inevitable, you choose between consistency and availability during partitions.

What this actually means for your system:

  • CP systems (consistency over availability): During a network partition, the system refuses to serve requests rather than return stale data. Example: a payment processing system that must never process duplicate charges. Use distributed locks or consensus protocols (etcd, ZooKeeper).

  • AP systems (availability over consistency): During a partition, the system serves requests but may return stale data. Example: a product catalog that shows slightly outdated inventory counts rather than going offline. Reconcile inconsistencies after the partition heals.

Most microservice systems are neither purely CP nor AP. Different operations within the same system make different tradeoffs. Checkout is CP (do not oversell). Product browsing is AP (showing a slightly stale price is better than a 500 error).

The Dual-Write Problem#

The most common consistency bug in microservices: writing to a database and publishing an event as two separate operations.

def place_order(order):
    db.save(order)                          # Step 1: write to database
    message_broker.publish("OrderPlaced", order)  # Step 2: publish event

If the process crashes between step 1 and step 2, the order exists in the database but no event was published. Downstream services never learn about the order. Reversing the order (publish first, then save) creates the opposite problem: the event goes out but the order is not persisted.

There is no way to make two writes to two different systems atomic without a coordination protocol. The outbox pattern and CDC solve this.

Outbox Pattern#

Write the event to the same database as the business data, in a single transaction. A separate process reads the outbox table and publishes to the message broker.

-- Single transaction: business write + event write
BEGIN;

INSERT INTO orders (id, customer_id, total, status)
VALUES ('order-123', 'cust-1', 99.50, 'placed');

INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload, created_at)
VALUES (
    'evt-456',
    'Order',
    'order-123',
    'OrderPlaced',
    '{"order_id": "order-123", "customer_id": "cust-1", "total": 99.50}',
    NOW()
);

COMMIT;

A relay process polls the outbox and publishes:

class OutboxRelay:
    def run(self):
        while True:
            events = db.query(
                "SELECT * FROM outbox WHERE published = false ORDER BY created_at LIMIT 100"
            )
            for event in events:
                kafka.produce(
                    topic=f"{event.aggregate_type.lower()}-events",
                    key=event.aggregate_id,
                    value=event.payload,
                    headers={"event_type": event.event_type}
                )
                db.execute(
                    "UPDATE outbox SET published = true WHERE id = %s",
                    event.id
                )
            time.sleep(0.5)

The relay may publish the same event twice if it crashes after publishing but before marking as published. Consumers must be idempotent. This is a fundamental tradeoff: at-least-once delivery with idempotent consumers is the standard pattern.

Change Data Capture with Debezium#

Instead of polling an outbox table, Debezium reads the database transaction log (WAL in PostgreSQL, binlog in MySQL) and streams changes to Kafka. This eliminates polling and catches every change with minimal latency.

Debezium with PostgreSQL Outbox#

{
  "name": "outbox-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "${secrets.DB_PASSWORD}",
    "database.dbname": "orders",
    "table.include.list": "public.outbox",
    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.table.field.event.id": "id",
    "transforms.outbox.table.field.event.key": "aggregate_id",
    "transforms.outbox.table.field.event.type": "event_type",
    "transforms.outbox.table.field.event.payload": "payload",
    "transforms.outbox.route.topic.replacement": "${routedByValue}-events"
  }
}

The EventRouter transform reads the outbox table changes and routes them to Kafka topics based on the aggregate type. The outbox rows can be deleted after Debezium captures them (Debezium reads the WAL, not the table, so deletion does not cause missed events).

Direct CDC (Without Outbox)#

You can also capture changes directly from business tables:

{
  "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
  "database.dbname": "orders",
  "table.include.list": "public.orders,public.order_items",
  "topic.prefix": "orders-db"
}

This publishes every INSERT, UPDATE, DELETE on the orders table to Kafka. Consumers see the full row state before and after each change. The downside: you are exposing your internal database schema as your event contract. Any table schema change potentially breaks consumers. The outbox pattern avoids this by decoupling the internal schema from the event schema.

Two-Phase Commit Limitations#

Two-phase commit (2PC) is the traditional solution for distributed transactions. A coordinator asks all participants to prepare, then tells them all to commit.

Why 2PC does not work well in microservices:

  • Blocking. Participants hold locks during the prepare phase. If the coordinator crashes, participants are stuck holding locks indefinitely.
  • Latency. Two network round trips (prepare + commit) across multiple services adds significant latency.
  • Availability. If any participant is unavailable, the entire transaction blocks. This couples availability across all participants.
  • Limited support. Most message brokers (Kafka, RabbitMQ) do not participate in 2PC. Most cloud databases do not support XA transactions.

2PC is acceptable within a single service that uses two data stores (e.g., a database and a cache) if both support XA and latency is acceptable. It is not viable across microservice boundaries.

Conflict Resolution Strategies#

When services process events concurrently or during network partitions, conflicts arise. Common resolution strategies:

Last-writer-wins (LWW): The most recent write (by timestamp or version number) wins. Simple but data loss is possible – concurrent updates overwrite each other silently.

UPDATE products SET price = 29.99, version = 5
WHERE id = 'prod-1' AND version = 4;
-- Returns 0 rows affected if version has changed → conflict detected

Merge resolution: For compatible changes, merge them. If user A updates the shipping address and user B updates the billing address, apply both. This requires domain-specific merge logic.

Conflict-free Replicated Data Types (CRDTs): Data structures designed for concurrent updates without conflicts. Counters, sets, and registers that mathematically guarantee convergence. Useful for specific use cases (distributed counters, shopping carts) but not general-purpose.

Application-level resolution: Present conflicts to users or business logic. “Two agents updated the same ticket – which change should win?” This is the most flexible but requires UI and workflow support.

Read-Your-Writes Consistency#

After a user performs a write, they expect to see their change on the next read. In an eventually consistent system, the read model may not have processed the write yet.

Solution 1: Read from Write Store After Write#

def create_order(request):
    order = Order.create(request)
    db.save(order)  # Write to primary store
    event_bus.publish(OrderCreated(order))

    # Return response from write store, not read store
    return OrderResponse.from_write_model(order)

def get_order(order_id, after_version=None):
    if after_version:
        # Wait for read model to catch up
        read_model.wait_for_version(order_id, after_version, timeout=5)
    return read_model.get(order_id)

Solution 2: Session-Sticky Reads#

Route reads to a replica that has seen all writes from the current session. In PostgreSQL with streaming replication:

# After a write, record the WAL position
write_position = db.execute("SELECT pg_current_wal_lsn()").scalar()
session["last_write_lsn"] = write_position

# On subsequent reads, ensure replica has caught up
def get_connection_for_read(session):
    if "last_write_lsn" in session:
        replica = get_replica_at_least(session["last_write_lsn"])
        return replica
    return any_replica()

Solution 3: Causal Consistency Tokens#

The write operation returns a token. The client passes this token on reads. The read service waits until its projection has processed up to that token:

POST /orders → 201 Created, X-Consistency-Token: evt-seq-42
GET /orders/123, X-Consistency-Token: evt-seq-42 → waits until projection >= 42

This is the cleanest approach for CQRS systems where the read model is eventually consistent. The client controls whether it needs strong consistency (passes the token) or accepts eventual consistency (omits the token).

Choosing Your Consistency Pattern#

Start with the simplest approach that meets your requirements. Outbox + Debezium covers most microservice data consistency needs. Add read-your-writes only where user experience demands it. Reserve CRDTs and complex merge resolution for genuinely concurrent-write-heavy workloads. Most systems need eventual consistency with good UX patterns, not distributed consensus.