Architecture Patterns: Sync, Async, Events and Idempotency

Synchronous processing means the caller waits for a reply before continuing, while asynchronous processing means the caller sends a request and continues, receiving the result later. A synchronous call suits a balance check or a fraud score that must be known before the payment proceeds, because the hub cannot route a payment it has not yet approved. An asynchronous design suits settlement confirmations, ledger posting and notifications, where a delay of seconds does not change the outcome. A bank that makes every step synchronous will time out under load, while a bank that makes every step asynchronous will leave customers staring at an unknown status. The art is choosing per step. Event driven design treats every important change in a payment life as an event that other components can react to. When a payment moves from validated to screened, the hub publishes a PaymentScreened event with the payment identifier, the result and a timestamp. The orchestrator, the reporting service and the notification service each subscribe and act independently. This decouples components, because the validation engine does not need to know who consumes its result, and it creates a natural audit trail, because the event log is the history of the payment. A bank processing 2 million events a day can replay that log to rebuild state after an outage. Message queues and event streaming are the plumbing that makes asynchronous and event driven design work in practice. A message queue holds a task until a worker is ready, which smooths peaks, so a payroll file of 100,000 payments can be queued at 9 in the morning and processed steadily without overwhelming the fraud engine. Event streaming keeps an ordered, replayable log of events, which suits audit, reporting and downstream systems that need to catch up after downtime. The key difference is that a queue is usually consumed once and then removed, while a stream is retained and can be read again by new consumers. Retry and dead letter queues protect the hub when a downstream system is temporarily unavailable. A retry policy might attempt a failed core banking posting three times with delays of 5 seconds, 30 seconds and 5 minutes, using exponential backoff so the struggling system is not flooded. If the posting still fails, the message moves to a dead letter queue, which is a holding area for messages that need human attention or a fix before they can be retried. Without a dead letter queue, failed messages are either lost or retried forever. With one, operations can see exactly which 47 payments failed overnight and why. Idempotency means that processing the same payment instruction twice has the same effect as processing it once, and it is the main defence against duplicate payments. The hub assigns a unique idempotency key to each instruction, usually the client supplied instruction reference plus the debtor account, and stores the result of the first processing attempt. If the same key arrives again, whether from a client retry, a network timeout or a queue redelivery, the hub returns the stored result instead of sending the money again. Without idempotency, a single retry after a timeout could send a 500,000 dollar supplier payment twice, and the bank would have to recover the duplicate from the beneficiary. Ordering and exactly once delivery are the two guarantees that learners most often overpromise, so it is worth stating their limits plainly. Most queues guarantee at least once delivery, not exactly once, which means a message may arrive twice and the receiver must handle the duplicate. Ordering is usually guaranteed only within a single partition or queue, not across the whole system, so two payments from the same customer may be processed in a different order if they land on different partitions. The practical answer is to design for at least once delivery plus idempotency, and to order by business rules, such as value date and priority, rather than relying on the plumbing to preserve sequence.