Payments Training Hub
Message queues exist because PAYMENT OPERATIONS STARTED LIVE 01 Message queues - systems in a bank do not all run at the same ACTIVE 02 IBM MQ - traditional queue technology in large banks QUEUED 03 Retry logic - decides what happens when a payment fails QUEUED 04 Retry count - capped, often at five attempts, and each QUEUED 05 Correct the data if possible - They inspect the error QUEUED 06 Released in priority order - Fourth, when the system returns, the queued QUEUED Message queues exist because the systems in a bank do not all run at the same speed and are not all available at the same time. Step 1 of 6 Message queues exist because STARTED Message queues systems in a bank do not all run at the same ACTIVE IBM MQ traditional queue technology in large banks QUEUED Retry logic decides what happens when a payment fails QUEUED Retry count capped, often at five attempts, and each QUEUED Correct the data if possible They inspect the error QUEUED Released in priority order Fourth, when the system returns, the queued QUEUED Message queues exist because the systems in a bank do not all run at the same speed and are not all available at the same time.
IBM MQ PAYMENT OPERATIONS IN PROGRESS LIVE 01 Message queues - systems in a bank do not all run at the same DONE 02 IBM MQ - traditional queue technology in large banks ACTIVE 03 Retry logic - decides what happens when a payment fails QUEUED 04 Retry count - capped, often at five attempts, and each QUEUED 05 Correct the data if possible - They inspect the error QUEUED 06 Released in priority order - Fourth, when the system returns, the queued QUEUED IBM MQ is the traditional queue technology in large banks, and it has been used for payment processing for more than twenty years. Step 2 of 6 IBM MQ IN PROGRESS Message queues systems in a bank do not all run at the same DONE IBM MQ traditional queue technology in large banks ACTIVE Retry logic decides what happens when a payment fails QUEUED Retry count capped, often at five attempts, and each QUEUED Correct the data if possible They inspect the error QUEUED Released in priority order Fourth, when the system returns, the queued QUEUED IBM MQ is the traditional queue technology in large banks, and it has been used for payment processing for more than twenty years.
Retry logic PAYMENT OPERATIONS IN PROGRESS LIVE 01 Message queues - systems in a bank do not all run at the same DONE 02 IBM MQ - traditional queue technology in large banks DONE 03 Retry logic - decides what happens when a payment fails ACTIVE 04 Retry count - capped, often at five attempts, and each QUEUED 05 Correct the data if possible - They inspect the error QUEUED 06 Released in priority order - Fourth, when the system returns, the queued QUEUED Retry logic decides what happens when a payment fails to reach its destination on the first attempt. Step 3 of 6 Retry logic IN PROGRESS Message queues systems in a bank do not all run at the same DONE IBM MQ traditional queue technology in large banks DONE Retry logic decides what happens when a payment fails ACTIVE Retry count capped, often at five attempts, and each QUEUED Correct the data if possible They inspect the error QUEUED Released in priority order Fourth, when the system returns, the queued QUEUED Retry logic decides what happens when a payment fails to reach its destination on the first attempt.
Retry count PAYMENT OPERATIONS IN PROGRESS LIVE 01 Message queues - systems in a bank do not all run at the same DONE 02 IBM MQ - traditional queue technology in large banks DONE 03 Retry logic - decides what happens when a payment fails DONE 04 Retry count - capped, often at five attempts, and each ACTIVE 05 Correct the data if possible - They inspect the error QUEUED 06 Released in priority order - Fourth, when the system returns, the queued QUEUED The retry count is capped, often at five attempts, and each attempt is logged so operations can see exactly what happened. Step 4 of 6 Retry count IN PROGRESS Message queues systems in a bank do not all run at the same DONE IBM MQ traditional queue technology in large banks DONE Retry logic decides what happens when a payment fails DONE Retry count capped, often at five attempts, and each ACTIVE Correct the data if possible They inspect the error QUEUED Released in priority order Fourth, when the system returns, the queued QUEUED The retry count is capped, often at five attempts, and each attempt is logged so operations can see exactly what happened.
Correct the data if possible PAYMENT OPERATIONS IN PROGRESS LIVE 01 Message queues - systems in a bank do not all run at the same DONE 02 IBM MQ - traditional queue technology in large banks DONE 03 Retry logic - decides what happens when a payment fails DONE 04 Retry count - capped, often at five attempts, and each DONE 05 Correct the data if possible - They inspect the error ACTIVE 06 Released in priority order - Fourth, when the system returns, the queued QUEUED They inspect the error, correct the data if possible, and either resubmit the payment or return it to the customer. Step 5 of 6 Correct the data if possible IN PROGRESS Message queues systems in a bank do not all run at the same DONE IBM MQ traditional queue technology in large banks DONE Retry logic decides what happens when a payment fails DONE Retry count capped, often at five attempts, and each DONE Correct the data if possible They inspect the error ACTIVE Released in priority order Fourth, when the system returns, the queued QUEUED They inspect the error, correct the data if possible, and either resubmit the payment or return it to the customer.
Released in priority order PAYMENT OPERATIONS COMPLETE LIVE 01 Message queues - systems in a bank do not all run at the same DONE 02 IBM MQ - traditional queue technology in large banks DONE 03 Retry logic - decides what happens when a payment fails DONE 04 Retry count - capped, often at five attempts, and each DONE 05 Correct the data if possible - They inspect the error DONE 06 Released in priority order - Fourth, when the system returns, the queued ACTIVE Fourth, when the system returns, the queued payments are released in priority order, with urgent Step 6 of 6 Released in priority order COMPLETE Message queues systems in a bank do not all run at the same DONE IBM MQ traditional queue technology in large banks DONE Retry logic decides what happens when a payment fails DONE Retry count capped, often at five attempts, and each DONE Correct the data if possible They inspect the error DONE Released in priority order Fourth, when the system returns, the queued ACTIVE Fourth, when the system returns, the queued payments are released in priority order, with urgent

Key points of this lesson, in order.

0:00 / 3:33 Step 1 of 6

Queueing and Retry Logic

System Design Deep Dive, Lesson 3 of 7, Narration 3:33, D09 Payment architecture and technology

ReturnRetryQueue

What you will learn

Your objective. By the end of this lesson, you will be able to explain queueing and retry logic. Message queues exist because the systems in a bank do not all run at the same speed and are not all available at the same time.

Why it matters. Apache Kafka is the newer technology, and it is used where a bank needs to stream very large volumes of events, such as logging every status change of every payment for analytics.

Chapter objective: Learner applies system design to payments, covering throughput, latency, idempotency, exactly-once processing, and failure recovery, and designs a system for a stated load. Learner practises this skill on realistic banking cases, explaining the reasoning clearly and using correct terminology throughout the work.

Key learning points

  • Message queues exist because
  • IBM MQ
  • Retry logic
  • Retry count
  • Correct the data if possible
  • Released in priority order

Key terms

Return: A payment that is sent back to the sender after settlement because it cannot be credited to the beneficiary.

Retry: Repeating a failed request or message delivery after a delay, usually with a limit on attempts and a back off period.

Queue: A holding area in a payment system where payments wait for sufficient funds or for their turn to settle.

Banking architecture diagram showing Queueing and Retry Logic.

Practise this lesson

Exercise

Map the payment in this lesson through the hub components.

Use the hub components and their order in this lesson.

  1. What to do. List each component the payment passes through, what it does, and what happens if it fails.

Correct result. The payment enters through a channel, passes the gateway, validation, fraud and sanctions checks, routing, and the ledger, then leaves through the right clearing system. A failure at any component stops or holds the payment, and the hub must record where it stopped.

Why. The hub is where most payment requirements live. If you cannot map a payment through it, you cannot specify, test or fix it.

Check your understanding

Practice question

A system design review asks where the payment state should live so that an investigation can replay exactly what happened to a payment. What is the right answer?

Takeaway and next step

What to remember

A retry must never create a duplicate payment, so the hub checks the unique reference before sending again.

Send a payment through the hub and watch it pass each component named in this lesson.

Open the Payment Hub Architecture Simulator →

Builds on "Message Routing: From Channel to Rail".

Transcript and full text

Show transcript

Message queues exist because the systems in a bank do not all run at the same speed and are not all available at the same time. A channel may receive 10,000 payment instructions in a single minute during a payroll run, while the payment engine can only process 2,000 per minute. Without a queue, the extra 8,000 instructions would be rejected. With a queue, they wait safely and are processed in order. Queues also protect the bank when a downstream system is taken down for maintenance. The hub keeps accepting payments, stores them in the queue, and releases them when the system returns, so customers see no interruption. IBM MQ is the traditional queue technology in large banks, and it has been used for payment processing for more than twenty years. It guarantees that a message is delivered exactly once and in order, and it stores messages on disk so they survive a server restart. A payment placed on an IBM MQ queue will still be there after a power failure, which is essential when the message represents real money. Apache Kafka is the newer technology, and it is used where a bank needs to stream very large volumes of events, such as logging every status change of every payment for analytics. Kafka keeps a permanent record of events that many different systems can read, while IBM MQ is usually chosen where guaranteed single delivery of a payment instruction matters most. Retry logic decides what happens when a payment fails to reach its destination on the first attempt. A common pattern is exponential backoff, where the system waits a short time before the first retry, then waits twice as long before the second, and twice as long again before the third. A payment that fails at 10 in the morning might be retried at 1 minute past 10 in the morning, then at 3 minutes past 10 in the morning, then at 7 minutes past 10 in the morning, then at 15 minutes past 10 in the morning. This pattern gives a briefly overloaded system time to recover without being flooded by thousands of immediate retries. The retry count is capped, often at five attempts, and each attempt is logged so operations can see exactly what happened. A retry must never create a duplicate payment, so the hub checks the unique reference before sending again. When retries are exhausted, the payment moves to a dead letter queue, which is a special holding area for messages that could not be delivered. A payment lands in the dead letter queue for reasons such as a downstream system rejecting it repeatedly, a message format error that no retry can fix, or a missing piece of reference data. Operations analysts monitor the dead letter queue continuously, because every item in it represents a customer payment that has not moved. They inspect the error, correct the data if possible, and either resubmit the payment or return it to the customer. A bank that ignores its dead letter queue will quickly accumulate stuck payments and unhappy customers. When a downstream system is down for a long period, the bank follows a planned sequence. First, the hub stops releasing new payments to that system and holds them in the queue, while continuing to accept new instructions from customers. Second, operations confirms the size of the backlog and estimates the recovery time, and customer service is told what to say to callers. Third, if the outage passes a defined threshold, often two hours, the bank may switch to a contingency route, such as sending urgent payments through an alternative correspondent bank. Fourth, when the system returns, the queued payments are released in priority order, with urgent and high value payments first, and the backlog is cleared gradually so the recovering system is not overwhelmed.

Message queues exist because the systems in a bank do not all run at the same speed and are not all available at the same time. A channel may receive 10,000 payment instructions in a single minute during a payroll run, while the payment engine can only process 2,000 per minute. Without a queue, the extra 8,000 instructions would be rejected. With a queue, they wait safely and are processed in order. Queues also protect the bank when a downstream system is taken down for maintenance. The hub keeps accepting payments, stores them in the queue, and releases them when the system returns, so customers see no interruption.

IBM MQ is the traditional queue technology in large banks, and it has been used for payment processing for more than twenty years. It guarantees that a message is delivered exactly once and in order, and it stores messages on disk so they survive a server restart. A payment placed on an IBM MQ queue will still be there after a power failure, which is essential when the message represents real money. Apache Kafka is the newer technology, and it is used where a bank needs to stream very large volumes of events, such as logging every status change of every payment for analytics. Kafka keeps a permanent record of events that many different systems can read, while IBM MQ is usually chosen where guaranteed single delivery of a payment instruction matters most.

Retry logic decides what happens when a payment fails to reach its destination on the first attempt. A common pattern is exponential backoff, where the system waits a short time before the first retry, then waits twice as long before the second, and twice as long again before the third. A payment that fails at 10 in the morning might be retried at 1 minute past 10 in the morning, then at 3 minutes past 10 in the morning, then at 7 minutes past 10 in the morning, then at 15 minutes past 10 in the morning. This pattern gives a briefly overloaded system time to recover without being flooded by thousands of immediate retries. The retry count is capped, often at five attempts, and each attempt is logged so operations can see exactly what happened. A retry must never create a duplicate payment, so the hub checks the unique reference before sending again.

When retries are exhausted, the payment moves to a dead letter queue, which is a special holding area for messages that could not be delivered. A payment lands in the dead letter queue for reasons such as a downstream system rejecting it repeatedly, a message format error that no retry can fix, or a missing piece of reference data. Operations analysts monitor the dead letter queue continuously, because every item in it represents a customer payment that has not moved. They inspect the error, correct the data if possible, and either resubmit the payment or return it to the customer. A bank that ignores its dead letter queue will quickly accumulate stuck payments and unhappy customers.

When a downstream system is down for a long period, the bank follows a planned sequence. First, the hub stops releasing new payments to that system and holds them in the queue, while continuing to accept new instructions from customers. Second, operations confirms the size of the backlog and estimates the recovery time, and customer service is told what to say to callers. Third, if the outage passes a defined threshold, often two hours, the bank may switch to a contingency route, such as sending urgent payments through an alternative correspondent bank. Fourth, when the system returns, the queued payments are released in priority order, with urgent and high value payments first, and the backlog is cleared gradually so the recovering system is not overwhelmed.

Full lesson text (all scenes)

Completing a lesson updates your progress dashboard on the homepage.

Scene timestamps are estimated by allocating the narration text proportionally across the real audio length. The audio playback position is the source of truth: pause freezes the visual, seeking restores the correct scene, replay restarts the scene animation.