The 200-Millisecond Yes: Designing a Card Authorization System That Survives Black Friday

Jul 9, 2026

Knowledge Café

engineering system-design payments low-latency scalability

Every time someone taps a card at a terminal, a question races around the planet: “Will you honor this?” The card network — Visa, Mastercard — routes that question to the card’s issuer, and the issuer has to answer. Fast.

Here’s the part that keeps issuer engineers honest: if you don’t answer in time, the network answers for you. It’s called stand-in processing (STIP) — the network approves or declines on your behalf using generic rules that know nothing about your customer’s real balance or your fraud posture. Every STIP’d transaction is either money you might lose or a good customer you might embarrass at a checkout counter.

So while the network’s official timeout is measured in seconds, nobody serious runs anywhere near it. Timeouts compound across merchant → acquirer → network → you, and every hop eats margin. A well-run issuer authorization stack targets p99 under 200ms measured at its own edge — and once you subtract network transit and protocol handling, your actual decision window is closer to 100ms of compute.

Let’s design a system that hits that number at 5,000 TPS on a normal Tuesday and doesn’t fall over when Black Friday multiplies it by four.

The Shape of the Problem

An authorization request arrives as an ISO 8583 message over a long-lived socket from the network. We need to decide: is this card real, active, and cryptographically legitimate? Does the customer have the funds? Does this smell like fraud? Then we respond with an approval code and — critically — place a hold on the funds.

sequenceDiagram
    participant POS as Merchant Terminal
    participant ACQ as Acquirer
    participant NET as Card Network
    participant EDGE as Issuer Edge Gateway
    participant AUTH as Authorization Engine

    POS->>ACQ: Auth request ($84.99)
    ACQ->>NET: ISO 8583 0100
    NET->>EDGE: Route to issuer (persistent socket)
    EDGE->>AUTH: Parsed, typed request
    AUTH->>AUTH: Validate, score, reserve funds
    AUTH-->>EDGE: Approve (response code 00)
    EDGE-->>NET: ISO 8583 0110
    NET-->>ACQ: Approved
    ACQ-->>POS: Receipt prints

Notice what’s not in this picture: no settlement, no statement posting, no money actually moving. An authorization is a promise, not a payment. That distinction is the single most important lever in this entire design — we’ll come back to it when we talk about the ledger.

The Budget: Where the Milliseconds Go

The first thing I do on any latency-critical system is write the budget down and defend it line by line. If it’s not written down, every team will spend it independently and you’ll discover at load test time that you’re 3x over.

Stage	Budget	Notes
Network transit (in + out)	~40ms	Not ours to spend, but ours to subtract
ISO 8583 parse / serialize	2ms	Zero-allocation parser, pre-compiled field maps
Card lookup + status checks	3ms	Cache hit or bust — no DB on this path
EMV cryptogram verification (HSM)	10ms	Hardware round trip, pooled sessions
Rules + velocity limits	5ms	In-memory counters
Fraud scoring	30ms	Hard timeout, fallback to rules
Funds check + hold	10ms	The interesting part — see below
Response assembly + headroom	100ms	p99 headroom for GC, retries, tail effects

Two principles fall out of this table. First: nothing on the hot path talks to anything slower than memory unless it absolutely must (the HSM and the durable hold write are the two exceptions, and each earns its place). Second: every line has a timeout and a fallback. A dependency without a degradation story isn’t a dependency — it’s a landmine.

Architecture at a Glance

graph TD
    NET["Card Network<br/>(Visa / Mastercard)"]

    subgraph "Edge Layer"
        GW["Protocol Gateway<br/>ISO 8583 over persistent TCP"]
    end

    subgraph "Hot Path — stateless, in-memory"
        AUTH["Authorization Engine"]
        RULES["Rules + Velocity Limits"]
        FRAUD["Fraud Scorer<br/>(30ms hard timeout)"]
        HSM["HSM Cluster<br/>EMV cryptogram checks"]
    end

    subgraph "Data Plane"
        L1["L1 In-Process Cache<br/>(BIN config, hotlists)"]
        REDIS["Redis Cluster<br/>card profiles + available balance"]
        LOG["Auth Event Log<br/>(Kafka, acks=all)"]
        LEDGER["Ledger DB<br/>(async consumers, batched)"]
    end

    NET <--> GW
    GW <--> AUTH
    AUTH --> RULES
    AUTH --> FRAUD
    AUTH --> HSM
    AUTH --> L1
    AUTH --> REDIS
    AUTH --> LOG
    LOG --> LEDGER

The authorization engine is stateless and horizontally scalable — every piece of state it needs is either in its own memory, in Redis, or arrives in the request. That’s what lets us handle Black Friday by adding pods instead of adding prayers.

Caching: The Only Way to Hit the Budget

You do not hit a 100ms compute budget by querying a relational database per transaction. You hit it with a deliberate, layered cache hierarchy where each layer has an explicit consistency contract.

L1 — in-process, microseconds. BIN range configuration, product rules, merchant category tables, and the lost/stolen card hotlist live in every engine node’s heap. These datasets are small (a hotlist is a few million entries — a bloom filter plus a compact set fits in a few hundred MB) and they change rarely. Refreshed by a background thread, never on the request path.

L2 — Redis, single-digit milliseconds. Card profiles: status, limits, expiry, PIN retry counters, velocity windows. Read-through with a TTL of minutes — but the TTL is the backstop, not the mechanism. The real consistency contract is event-driven invalidation: every write to the system-of-record database flows through CDC (Debezium → Kafka), and an invalidator service updates or evicts the Redis entry within tens of milliseconds. When a customer freezes their card in the app, that freeze must be honored on the very next swipe — “the TTL will expire eventually” is not an acceptable answer in payments.

Stampede protection. When a hot key expires or a node cold-starts, a thousand concurrent requests must not become a thousand identical loads. Request coalescing (singleflight) collapses them into one fetch; jittered TTLs keep expirations from synchronizing across the fleet.

And then there’s the thing you must never cache naively: the available balance. A stale card profile risks a slightly wrong decision. A stale balance risks money. Which brings us to the heart of this design.

The Balance Problem (a.k.a. Don’t Cache Money — Move It)

The naive design does this on every authorization: SELECT balance, check it, UPDATE balance, INSERT hold — a read-modify-write transaction against the ledger table. At 20,000 TPS peak, that design is dead on arrival, and not because of throughput in aggregate. It dies from contention on specific rows.

The fix has two parts. Part one is conceptual: remember that an authorization is a hold, not a posting. The accounting ledger doesn’t need to know about it synchronously — it needs to know about it durably. Part two is structural: make an in-memory store the authority for available-to-spend, and make the relational ledger an asynchronous consumer.

sequenceDiagram
    participant A as Auth Engine
    participant R as Redis (balance authority)
    participant K as Kafka (auth event log)
    participant L as Ledger DB (async)

    A->>R: EVAL reserve.lua — check AND decrement, one atomic round trip
    R-->>A: OK, hold_id (~1ms)
    A->>K: Append AuthorizationHeld event (acks=all)
    K-->>A: Committed (~3ms)
    A-->>A: Respond 00 — approved
    Note over K,L: Everything below is off the hot path
    K->>L: Consumer writes holds to ledger in 100ms batches

The details that make this safe rather than reckless:

Atomicity: the check-and-reserve is a single Lua script — one round trip, no race between “check balance” and “place hold.” If funds are insufficient, the script declines without mutating anything.
Durability: we do not send the approval to the network until the AuthorizationHeld event is acknowledged by the replicated log. Redis alone is not the durability story; the log is. If a Redis node dies, we rebuild its balances by replaying snapshots plus the log.
Reconciliation: the ledger consumer continuously reconciles Redis reservations against posted ledger state. Holds that never clear expire on the network’s schedule and are released back.

The relational database still owns the truth — every hold, clearing, and reversal lands there. It just doesn’t own the latency.

Hot-Spot Ledger Tables: Surviving Peak

Even with holds moved off the hot path, somebody eventually writes to the ledger — and during peak shopping hours, ledger writes are brutally skewed. A handful of accounts absorb a huge fraction of traffic: large corporate cards, BNPL settlement accounts, promotional wallets. Under update-in-place, every transaction against the same account queues on the same row lock, and your p99 doesn’t degrade gracefully — it falls off a cliff.

Three defenses, in the order I’d reach for them:

1. Append, don’t update. The ledger’s write path is INSERT-only: holds, clearings, reversals are immutable events. Inserts don’t take row locks on shared rows. Current balance is a derived value: a periodic snapshot plus the sum of deltas since. A compaction job rolls snapshots forward so the delta scan stays short.

2. Batch at the consumer. Because ledger writes arrive via Kafka rather than synchronously, the consumer controls the write shape: group commits, multi-row inserts, one fsync amortized across hundreds of transactions. The database sees a smooth, batched firehose instead of 20,000 individual sub-millisecond-deadline transactions.

3. Bucket the balances that must stay relational. If a product requirement forces a synchronously-updated SQL balance (some corporate products do), split each hot account’s balance across N sub-rows and reserve from one chosen by hash. Contention drops by roughly N. N is per-account and scales with observed hotness.

graph TD
    TXN["Incoming hold ($4,200)"]

    subgraph "One hot account — logical view"
        BAL["Available: $1,000,000"]
    end

    subgraph "Physical rows in ledger DB"
        B1["Bucket 1: $250k"]
        B2["Bucket 2: $250k"]
        B3["Bucket 3: $250k"]
        B4["Bucket 4: $250k"]
    end

    BAL -.-> B1
    BAL -.-> B2
    BAL -.-> B3
    BAL -.-> B4
    TXN -->|"hash(txn_id) mod 4"| B3
    B3 -->|"insufficient? try next bucket"| B4

One more subtle one: on insert-heavy journal tables, a strictly monotonic primary key concentrates all inserts on the last index page — a different flavor of the same hot-spot disease. Time-ordered-but-distributed keys (UUIDv7 or sharded sequences) keep the B-tree spread out.

Connection Pooling: The Unsexy Thing That Pages You at 2 A.M.

Nobody puts connection pooling on the architecture diagram, and it causes more peak-hour incidents than anything else on this page.

Inbound: you don’t accept a TCP connection per transaction. The network delivers traffic over a handful of long-lived, multiplexed sockets; responses are matched to requests by trace fields (STAN/RRN), not by connection. Your gateway’s job is to keep those pipes healthy — heartbeats, immediate reconnect, buffering discipline.

Outbound, size by arithmetic, not vibes. Little’s Law: concurrent connections needed = throughput × latency per operation. 20,000 TPS against Redis at 1ms is 20 in-flight operations — a pool of 50 per node is generous. Engineers who skip this math configure pools of 500 “to be safe” and then wonder why the database spends its life context-switching. Oversized pools don’t add safety; they add queueing inside the dependency, where you can’t see it.

Isolate pools by criticality. The hot path gets its own Redis pool, its own Kafka producer, its own HSM session pool. Batch jobs and reconciliation get separate ones. A single shared pool means one slow batch query can exhaust connections and starve authorizations — the classic pool-exhaustion cascade, and it always happens during peak, because that’s when everything is slow at once.

Fail fast. Pool checkout timeout on the hot path: 5–10ms. If there’s no connection available, we don’t wait — we fall through to the degradation path. Queueing for a connection is just latency you’ve chosen to hide from your own metrics.

When Things Go Wrong Anyway

At this volume, “when” is the only honest word. The system degrades in explicit, pre-decided steps:

Fraud scorer breaches its 30ms timeout → decide with in-memory rules alone (velocity + amount + geography). Slightly worse fraud recall for a few minutes beats declining everyone.
Redis partition → floor-limit mode: approve low-value, low-risk transactions below a configured threshold against last-known limits; decline or STIP the rest. This is a business decision made calmly in a design review, not improvised during an incident.
Total outage → the network’s STIP takes over with parameters we pre-agreed: per-transaction caps, category blocks, velocity ceilings. STIP is the airbag — you tune it before the crash.

And because networks retransmit and reversals arrive out of order, every operation is idempotent. The dedupe key (PAN hash + STAN + RRN + transmission timestamp) is checked in Redis before any state changes; a replayed request gets the original response replayed back, not a second hold.

What I’d Tell You to Remember

Write the latency budget down and make every line item defend itself. Unbudgeted milliseconds get spent by whoever integrates last.
An authorization is a hold, not a posting. Exploiting that gap — in-memory authority for available funds, durable log, async ledger — is what makes sub-200ms possible at all.
Cache with contracts, not TTLs. Event-driven invalidation for anything a customer can change; TTLs are backstops.
Hot rows are a design smell. Append instead of update, batch at the consumer, bucket what must stay relational.
Pool sizes come from Little’s Law, isolation comes from criticality, and the hot path never waits in line for a connection.
Degradation is a product feature. Decide the fallback ladder in a design review, encode it, and test it under load — because on Black Friday, you will use it.

The sub-200ms yes isn’t one clever trick. It’s fifty boring decisions, each defending a few milliseconds — written down, budgeted, and tested at four times the traffic you hope to see.