An IT service recovery plan sets out timing requirements, the required data state, dependencies and the actions of the people involved. Testing establishes the conditions under which these requirements are met and the changes still needed.
This article covers RTO and RPO, a recovery example and a test-report template. The example and template support discussion of a particular service; they do not establish universal requirements.
RTO and RPO objectives
RTO—Recovery Time Objective is the recovery-time target used in planning. NIST’s definition concerns the time system components can remain in recovery before adverse effects on the organisation’s activities. For a specific service, state the timing convention and the condition that counts as recovery. [1]
RPO—Recovery Point Objective identifies the point in time to which data must be restored after an outage. Working requirements commonly express it as an acceptable interval of lost recent changes—for example, no more than 15 minutes. [2]
RTO and RPO are objectives. A test produces an observed recovery duration and an observed gap in recovered data. Writing a target in a document does not demonstrate that it can be met.
Service recovery criteria
A running server, an accessible login page and a completed database restore may be intermediate milestones. Define a minimum working user journey before testing.
For example, an employee signs in with the correct role, opens a record, creates a test transaction and receives confirmation that it reached a connected system. Recovery is complete when that agreed journey works with the required performance and access restrictions.
Not every function needs to return immediately. A limited operating mode may be acceptable, but its scope, permitted duration and handling of deferred work must be understood by the service owner.
Redundancy, backups and recovery planning
A redundant instance may help when a primary component fails. A data copy may make an earlier state available. A recovery plan connects data, configuration, infrastructure and human actions into a sequence that restores the service.
AWS’s disaster recovery guidance relates strategy selection to objectives, cost and complexity. It also notes that replication alone does not protect against data corruption or destruction without a way to return to an earlier point. This is an architectural principle, not a requirement to adopt a cloud or a particular product. [3]
In your evaluation, distinguish the scenarios covered by redundancy, the states retained by backups and the procedure that turns those states into a working process.
A worked recovery example
All values below are illustrative. They are not a standard, a universal recommendation or a result from an AI Code client project.
The selected service has an agreed RTO of 45 minutes and RPO of 15 minutes. For this example, the recovery clock starts when the service actually becomes unavailable.
| Event | Time |
|---|---|
| Latest consistent data point available for recovery | 09:55 |
| Service becomes unavailable | 10:00 |
| Incident detected | 10:04 |
| Recovery begins | 10:08 |
| Application starts | 10:18 |
| Integration with a dependent system is restored | 10:40 |
| Agreed user journey passes | 10:48 |
Observed service recovery takes 48 minutes from the outage: the RTO target is missed by 3 minutes. Starting the application at 10:18 does not establish an 18-minute service recovery because the agreed workflow was not yet available.
The data gap is 5 minutes, from 09:55 to 10:00. Provided the restored state is verified as consistent, the RPO target is met.
This is more informative than “recovery successful.” It identifies a specific gap: dependent integration and final validation must be included in the plan and may need improvement.
Service dependencies
For the chosen scenario, map the necessary components: identity and access, name resolution and connectivity, application configuration, data, integrations and observability.
For each component, ask where it will be restored from, who can perform the action, which components are needed first and whether instructions remain available when the primary environment fails. Check access to keys and secrets through the intended protected mechanism; do not copy them into an openly shared test report.
The recovered data point must represent a consistent state of the process. A backup file’s timestamp does not, by itself, establish that connected systems can resume without missing or duplicating operations.
What to record in the test report
The following is a proposed working template. Adapt it to the selected service and scenario.
| Field | Record |
|---|---|
| Service and owner | What is being restored and who accepts the result |
| Scenario | The failure or data-loss event being simulated |
| Test boundaries | Included dependencies and untested areas |
| Objectives | RTO, RPO and the conditions to which they apply |
| Timing convention | The precise starting event and time |
| Data point | The latest verified consistent state |
| Recovery progress | Important actions and timestamps |
| User validation | The end-to-end journey completed |
| Observed result | Recovery duration, data gap and limitations |
| Deviations | Unmet objectives and their causes |
| Next actions | Improvements, owners and a repeat test |
A passing test applies to the scenario and conditions actually exercised. Do not automatically extend it to different data volumes, locations or combinations of failures.
Regular recovery testing, including after significant changes, is discussed in the failure-management section of the AWS Well-Architected Framework. The organisation still needs to choose an appropriate cadence and a controlled way to test its environment. [4]
What changes when the service includes AI
For an AI or retrieval-augmented generation service, extend the recovery map beyond the application and database. Consider source documents, access rules, processing configuration, component versions and dependencies on external providers.
Treat a derived search index as reproducible from its sources only after verifying that it can be rebuilt and measuring the time required. If rebuilding takes longer than the agreed target, the architecture needs another recovery method or an explicitly accepted limited operating mode.
Verify that source permissions are correct after restoration. An assistant that answers questions again but has not recovered the right access controls is not a fully restored protected service.
For an important process, define behaviour without the model: whether conventional search remains available, work is queued or a person takes over. These are decisions for the particular system, not a universal recipe.
Test results
The report should identify the service restored, the source state, the scenario and the measured outcome.
Record objectives, test boundaries, measurements and observed limitations. Where results are not yet available, prepare an agreed test scenario and identify the conditions needed to run it.
Related reading: Architecture & Infrastructure · IT strategy and architecture · Building a modernisation plan.
Discuss infrastructure and service recovery.
Sources
- NIST CSRC — Recovery Time Objective. Terminology to be applied in the relevant system context.
- NIST CSRC — Recovery Point Objective. The required data recovery point.
- AWS Well-Architected, Reliability Pillar — Use defined recovery strategies to meet the recovery objectives. Strategy trade-offs and the limitations of replication.
- AWS Well-Architected Framework — Failure management, 27 June 2024 edition. Testing and recovery practices. This explicitly dated edition is not presented as the latest version.
The numerical example, reporting template and sequence of questions are original illustrations for this article, not extracts from a mandatory standard.