TC-21 Recoverability and rebuild
| Objective | TC-21 |
| Evidence level | Evidence, Integrate |
| Domain | Delivery fit |
| Owner | Raidiam |
| Phase | Formal Proof of Concept |
| Proven by | content-recovery, with the governance as code pipeline from scenes 01 and 07 |
On what is claimed here. Where this document describes the demonstration, it states what the Enhanced Vendor Demo of 24 August 2026 is built to prove, not what has already been built, recorded or verified. Status for every scene is tracked in objectives/tc-objectives.yaml.
1. What CBA asked for
Recoverability and rebuild approach for the PoC environment.
2. The answer
The entire federation is declared in git and applied by pipeline, so a rebuild is a re apply.
Federations, authorities, authorisation domains, capability roles and their metadata rows, organisations, authorisation servers, Agent Identities, authority claims and key references are all declared as files in a repository. A driver applies that declaration to the platform API idempotently: running it against an empty tenant creates the estate, and running it against a current tenant is a no operation. There is no hand wired configuration anywhere, which is a rule rather than an aspiration, because the moment one record is created by hand the rebuild stops being complete.
This gives four properties, and only the first is about disaster recovery.
| Property | What it means in practice |
|---|---|
| Rebuild | Recovery time is the time to run a pipeline against a fresh tenant, not the time to restore and validate a database |
| Reproducibility | The same declaration applied to a second tenant produces an identical topology, so a non production environment is a branch rather than a rebuild by hand |
| Drift detection | A re run produces a plan. A non empty plan against an unchanged repository means something changed outside the pipeline. That is a correctness signal and a security signal (content-observability.md). |
| Review | Every change to the estate arrives as a diff, is reviewable before it is applied, and carries the approval that merged it |
3. What git does not restore
This is the part of the answer that is usually missing, and it is where the real recovery risk sits.
| Not in git | Why | How it is recovered |
|---|---|---|
| Private key material | It must never be in a repository. Git holds public keys and references only. | Through the key management service that holds it. For the bring your own key path this is CBA's own key management service and CBA's existing key custody and recovery controls apply. Losing the trust anchor private key is the one unrecoverable event in this architecture, and it is recovered by the key management service or not at all. |
| Issued trust marks and their history | They are platform records produced by the act of granting, not declarations | Platform backup and restore. Re applying the declaration re grants the claims, which produces equivalent marks, but not the original issuance history. |
| The change history | It is the audit record of what happened, which by definition cannot be regenerated from a declaration of what should be | Platform backup and restore |
| Anything created by hand | Nothing should be. This row exists so the rule has somewhere to be written down. | It is not recovered. This is the cost of exceptions and the reason there are none. |
4. Entity identifiers are a recovery control
Entity identifiers and hostnames are fixed before the first record is written and are never changed. That is usually filed as a naming convention. It is a recovery control, because an identifier appears in every subordinate statement, in every consumer's client representation, in every recorded trust chain and in every audit record. A rebuild that produces different identifiers is not a recovery, it is a migration, and every consumer has to be touched. Holding identifiers stable is what makes the re apply transparent to consumers.
5. The rebuild procedure
- Provision a tenant.
- Restore or reference the key material from the key management service. Keys first, because entity configurations are signed with them and an identity without its key is not the same identity.
- Run the pipeline. It creates the estate from the declaration.
- Verify, and verification is not optional. For every entity: resolve it from the trust anchor and confirm the chain builds. For every trust mark that should be active: retrieve it and confirm its status. For every entity that should not resolve: confirm it does not. This is a script, run as part of the rebuild, not a manual checklist.
- Reconcile consumers. Where a consuming authorization server holds a cached client representation (the overlay model), confirm it re resolves against the rebuilt federation.
Step 5 is the one that gets forgotten. In the native target state it is unnecessary, because consumers hold nothing to reconcile. In the overlay stage it is a required step and belongs in the runbook.
6. What is demonstrated
The pipeline itself is demonstrated in scenes 01 and 07: a manifest is committed, a plan is produced, a merge applies it, a failed apply is retried without corrupting either system of record, and a correlation identifier threads the whole thing.
A full teardown and re apply is demonstrated if the schedule permits, and is stated here as conditional rather than promised. The property it would show, that the estate can be destroyed and rebuilt from the declaration, is exercised during build every time the environment is re created, so the confidence behind the claim comes from repeated use rather than from a single rehearsal.
7. Platform level backup and restore
The recoverability of the Raidiam Connect platform itself, including database backup frequency, retention, restore testing and the recovery point and recovery time objectives, is a service commitment rather than a design position. Those figures are supplied through ID Partners under the appropriate agreement, and are deliberately not stated here rather than estimated. Under the bring your own Postgres deployment option, backup and restore of the data store is CBA's own operation to CBA's own standards, and Raidiam supplies the schema and restore requirements (content-deployment-model.md).
8. What we need from CBA and ID Partners
- The recovery point and recovery time objectives CBA requires for the Formal Proof of Concept environment, so the deployment option can be chosen against them rather than after them.
- Whether CBA requires a documented and rehearsed recovery test as an exit criterion for the Proof of Concept.
- Where the trust anchor private key lives, and confirmation that CBA's key recovery controls cover it.
- Whether the git repository holding the declaration sits in CBA's own source control, which is the recommended position, since it makes the declaration a CBA asset rather than a vendor one.