Payments Training Hub
Production support in payments 1 NOW Production 2 Incident containment 3 Five whys root cause 4 Blameless post-mortem 5 Production monitoring 6 Incident simulation acceptance criteria Production support in payments is a 24 hour discipline, because payments do not stop at 5pm. Step 1 of 6 Production support in payments 1 NOW Production 2 Incident containment 3 Five whys root cause 4 Blameless post-mortem 5 Production monitoring 6 Incident simulation acceptance criteria Production support in payments is a 24 hour discipline, because payments do not stop at 5pm. Step 1 of 6
Incident containment 1 Production 2 NOW Incident containment 3 Five whys root cause 4 Blameless post-mortem 5 Production monitoring 6 Incident simulation acceptance criteria The second is to contain the impact: stop new payments if necessary, queue the affected payments, and notify the customers. Step 2 of 6 Incident containment 1 Production 2 NOW Incident containment 3 Five whys root cause 4 Blameless post-mortem 5 Production monitoring 6 Incident simulation acceptance criteria The second is to contain the impact: stop new payments if necessary, queue the affected payments, and notify the customers. Step 2 of 6
Five whys root cause 1 Production 2 Incident containment 3 NOW Five whys root cause 4 Blameless post-mortem 5 Production monitoring 6 Incident simulation acceptance criteria Why? Step 3 of 6 Five whys root cause 1 Production 2 Incident containment 3 NOW Five whys root cause 4 Blameless post-mortem 5 Production monitoring 6 Incident simulation acceptance criteria Why? Step 3 of 6
Blameless post-mortem 1 Production 2 Incident containment 3 Five whys root cause 4 NOW Blameless post-mortem 5 Production monitoring 6 Incident simulation acceptance criteria A post-mortem that is written but never acted on is a ritual, not a learning exercise. Step 4 of 6 Blameless post-mortem 1 Production 2 Incident containment 3 Five whys root cause 4 NOW Blameless post-mortem 5 Production monitoring 6 Incident simulation acceptance criteria A post-mortem that is written but never acted on is a ritual, not a learning exercise. Step 4 of 6
Production monitoring 1 Production 2 Incident containment 3 Five whys root cause 4 Blameless post-mortem 5 NOW Production monitoring 6 Incident simulation acceptance criteria Monitoring is the early warning system. Step 5 of 6 Production monitoring 1 Production 2 Incident containment 3 Five whys root cause 4 Blameless post-mortem 5 NOW Production monitoring 6 Incident simulation acceptance criteria Monitoring is the early warning system. Step 5 of 6
Incident simulation acceptance criteria 1 Production 2 Incident containment 3 Five whys root cause 4 Blameless post-mortem 5 Production monitoring 6 NOW Incident simulation acceptance criteria The acceptance criteria must include a test where an incident is simulated end to end, from detection to recovery, with the full timeline recorded. Step 6 of 6 Incident simulation acceptance criteria 1 Production 2 Incident containment 3 Five whys root cause 4 Blameless post-mortem 5 Production monitoring 6 NOW Incident simulation acceptance criteria The acceptance criteria must include a test where an incident is simulated end to end, from detection to recovery, with the full timeline recorded. Step 6 of 6

Flow of the explanation in this lesson.

0:00 / 4:16 Step 1 of 6

Production support and root cause analysis: keeping payments running

BA practice depth, Lesson 8 of 8, Narration 4:16, D14 Business analysis and professional practice

ReconciliationIdempotencyPayment hub

What you will learn

Your objective. By the end of this lesson, you will be able to explain production support and root cause analysis: keeping payments running and apply it to a real banking scenario.

Why it matters. For a business analyst, production support is a set of requirements: the support model, the runbook, the incident procedure, the root cause process, the reconciliation investigation procedure, and the monitoring.

Chapter objective: Gherkin, traceability, SIT and UAT design, negative testing, defect management, root cause analysis

Key learning points

  • Production support in payments
  • Incident containment
  • Five whys root cause
  • Blameless post-mortem
  • Production monitoring
  • Incident simulation acceptance criteria

Key terms

Reconciliation: The process of comparing records from two sources, such as a bank statement and an internal ledger, to confirm they agree.

Idempotency: The property that repeating the same request has the same effect as sending it once. In payments, an idempotency key lets a client safely retry a request after a timeout without creating a duplicate...

Payment hub: A central system in a bank that receives payment instructions from all channels, validates them, applies screening and routing rules, and sends them to the correct clearing system or correspondent.

Payment flow diagram showing Production support and root cause analysis: keeping payments running.

Insights and practice scenarios

Worked insights and practice scenarios for this lesson. Each one is a reusable PaymentForge case, so the same case may appear in more than one lesson.

Practise this lesson

Exercise

Investigate the reconciliation break described in this lesson and find its root cause.

Use the three-way matching rule and the break categories in this lesson.

  1. What to do. Compare the instruction, the settlement record and the statement, name the break type, and state the corrective action.

Correct result. A break is found by matching amount, currency, date and reference across the three sources. The root cause is a timing difference, an amount difference, a missing entry, or a duplicate, and the corrective action follows the cause.

Why. Reconciliation is the last line of defence. A break that is not investigated is money that cannot be proved.

Check your understanding

Practice question

Which of the following best describes production support and root cause analysis: keeping payments running?

Takeaway and next step

What to remember

A post-mortem that is written but never acted on is a ritual, not a learning exercise.

Follow a payment through every stage, from instruction to beneficiary credit, and note where it can stop.

Open the Payment Lifecycle Simulator →

Builds on "Defect management: severity, triage and the defect lifecycle".

Transcript and full text

Show transcript

Production support in payments is a 24 hour discipline, because payments do not stop at 5pm. An instant payment can be sent at 3am on a Sunday, and if it fails, the customer expects it to be fixed at 3am on a Sunday. The support model must therefore include a rota, an on-call procedure, an escalation path, and a runbook for the common incidents. The runbook must state, for each incident type, the symptoms, the diagnosis steps, the workaround, and the escalation. A support team without a runbook will improvise, and improvisation in payments causes duplicate payments. Incident triage in production follows a fixed sequence. The first step is to assess the impact: how many payments are affected, what is the value, and is customer money at risk. The second is to contain the impact: stop new payments if necessary, queue the affected payments, and notify the customers. The third is to diagnose: check the logs, check the message queue, check the ledger, and identify the failing component. The fourth is to fix or work around: restart the component, route around it, or process manually. The fifth is to recover: reconcile, verify, and notify. Each step must be logged with a timestamp. The 5 whys is the simplest root cause technique, and it works well in payments. A payment was duplicated. Why? Because the retry sent the payment twice. Why? Because the idempotency key was not carried through the retry. Why? Because the retry logic was written before the idempotency requirement was added. Why? Because the requirement was added after the code was written. Why? Because the change process did not require a re-review. The root cause is the change process, not the code, and the fix is to require a re-review when a requirement changes. The analysis must go deep enough to find the process failure. A blameless post-mortem is the culture that makes root cause analysis work. The post-mortem asks what happened, what the impact was, what the timeline was, what the root cause was, and what will change. It does not ask who is to blame, because blame stops people from reporting incidents, and unreported incidents are the ones that cause the most harm. The post-mortem must be written, shared with the team, and tracked to completion, with each action owned and dated. A post-mortem that is written but never acted on is a ritual, not a learning exercise. Reconciliation investigations are the daily work of production support. When the nostro statement does not match the ledger, the analyst must find the break: a payment that was sent but not recorded, a charge that was deducted but not expected, a return that was received but not matched, or a value date that differs. The investigation follows the payment: check the message, check the ledger, check the statement, and find the point where they diverge. Each break type has a known cause and a known fix, and the analyst must know them. The investigation must be documented, because the same break will happen again. Monitoring is the early warning system. The payment hub must be monitored for throughput, for the queue depth, for the error rate, for the processing time, and for the ledger balance. An alert must fire when the queue depth exceeds a threshold, when the error rate rises, or when the ledger does not balance. The alert must go to the person who can act on it, not to a mailbox nobody reads. The monitoring must be tested, because a monitor that has never fired is a monitor that may not work when it is needed. For a business analyst, production support is a set of requirements: the support model, the runbook, the incident procedure, the root cause process, the reconciliation investigation procedure, and the monitoring. The acceptance criteria must include a test where an incident is simulated end to end, from detection to recovery, with the full timeline recorded. Last reviewed 7 November 2026. Further reading: the Site Reliability Engineering book by Google; the ITIL incident management guidance; the Bank of England operational resilience policy; the DORA incident reporting requirements, Regulation (EU) 2022/2554, Article 19.

Production support in payments is a 24 hour discipline, because payments do not stop at 5pm. An instant payment can be sent at 3am on a Sunday, and if it fails, the customer expects it to be fixed at 3am on a Sunday. The support model must therefore include a rota, an on-call procedure, an escalation path, and a runbook for the common incidents. The runbook must state, for each incident type, the symptoms, the diagnosis steps, the workaround, and the escalation. A support team without a runbook will improvise, and improvisation in payments causes duplicate payments.

Incident triage in production follows a fixed sequence. The first step is to assess the impact: how many payments are affected, what is the value, and is customer money at risk. The second is to contain the impact: stop new payments if necessary, queue the affected payments, and notify the customers. The third is to diagnose: check the logs, check the message queue, check the ledger, and identify the failing component. The fourth is to fix or work around: restart the component, route around it, or process manually. The fifth is to recover: reconcile, verify, and notify. Each step must be logged with a timestamp.

The 5 whys is the simplest root cause technique, and it works well in payments. A payment was duplicated. Why? Because the retry sent the payment twice. Why? Because the idempotency key was not carried through the retry. Why? Because the retry logic was written before the idempotency requirement was added. Why? Because the requirement was added after the code was written. Why? Because the change process did not require a re-review. The root cause is the change process, not the code, and the fix is to require a re-review when a requirement changes. The analysis must go deep enough to find the process failure.

A blameless post-mortem is the culture that makes root cause analysis work. The post-mortem asks what happened, what the impact was, what the timeline was, what the root cause was, and what will change. It does not ask who is to blame, because blame stops people from reporting incidents, and unreported incidents are the ones that cause the most harm. The post-mortem must be written, shared with the team, and tracked to completion, with each action owned and dated. A post-mortem that is written but never acted on is a ritual, not a learning exercise.

Reconciliation investigations are the daily work of production support. When the nostro statement does not match the ledger, the analyst must find the break: a payment that was sent but not recorded, a charge that was deducted but not expected, a return that was received but not matched, or a value date that differs. The investigation follows the payment: check the message, check the ledger, check the statement, and find the point where they diverge. Each break type has a known cause and a known fix, and the analyst must know them. The investigation must be documented, because the same break will happen again.

Monitoring is the early warning system. The payment hub must be monitored for throughput, for the queue depth, for the error rate, for the processing time, and for the ledger balance. An alert must fire when the queue depth exceeds a threshold, when the error rate rises, or when the ledger does not balance. The alert must go to the person who can act on it, not to a mailbox nobody reads. The monitoring must be tested, because a monitor that has never fired is a monitor that may not work when it is needed.

For a business analyst, production support is a set of requirements: the support model, the runbook, the incident procedure, the root cause process, the reconciliation investigation procedure, and the monitoring. The acceptance criteria must include a test where an incident is simulated end to end, from detection to recovery, with the full timeline recorded. Last reviewed 7 November 2026. Further reading: the Site Reliability Engineering book by Google; the ITIL incident management guidance; the Bank of England operational resilience policy; the DORA incident reporting requirements, Regulation (EU) 2022/2554, Article 19.

Full lesson text (all scenes)

Completing a lesson updates your progress dashboard on the homepage.

Scene timestamps are estimated by allocating the narration text proportionally across the real audio length. The audio playback position is the source of truth: pause freezes the visual, seeking restores the correct scene, replay restarts the scene animation.