TC-19 Observability and monitoring

On what is claimed here. Where this document describes the demonstration, it states what the Enhanced Vendor Demo of 24 August 2026 is built to prove, not what has already been built, recorded or verified. Status for every scene is tracked in objectives/tc-objectives.yaml.

1. What CBA asked for

Observability and monitoring sufficient to operate and investigate the Trust Controller.

The word that carries the weight is investigate. Health checks tell an operator that something is wrong. This objective is about whether they can find out what, and why, without a vendor on the call.

2. The four signal families

Availability alone is the trap. A Trust Controller can be entirely healthy while publishing the wrong thing, and it can be publishing correctly while a consumer refuses every agent for a reason that has nothing to do with it. Operating this system means watching all four.

3. Availability and latency

The endpoints that matter are the ones a consumer depends on at the moment it makes a decision:

• the entity configuration of each governed entity,
• subordinate statements at each authority,
• the subordinate listing,
• the resolve endpoint at the trust anchor,
• trust mark issuance and trust mark status.

Each is monitored for availability, status code distribution and latency distribution, not just a mean. The tail is what breaks consumers.

An operational lesson worth designing against, from a live deployment. On 3 August 2026, on a Raidiam operated directory, a resolution failed with a generic "failed to retrieve leaf entity configuration" error while the leaf itself was healthy and served the same document to an ordinary client in about 0.27 seconds. The cause was that the leaf's own discovery endpoint made outbound calls on the request path, which pushed a single response to 2.18 seconds, past the resolver's fetch timeout. Two design rules came out of it and both are stated here because they are the difference between a five minute diagnosis and a two hour one:

1. A discovery endpoint must never block on a third party. Publish from a background refreshed snapshot. Serving slightly stale metadata is strictly better than serving a timeout.
2. A resolution failure must distinguish timeout from unreachable from malformed, and state the timeout in the error. A single generic failure message for four different faults costs real time.

Both are carried into the demonstration components, and the second is raised as product feedback.

4. Publication health, which is the signal unique to a Trust Controller

An Agent Identity that is approved but does not resolve is an outage that no availability check will find, because every endpoint is returning 200. The check that finds it is a resolution probe per governed entity: resolve it from the trust anchor, confirm a chain is built, confirm the resulting metadata matches what the declared state in git says it should be, and confirm the trust marks that should be active are active and the ones that should not be are not.

Because the entire federation is declared in git and applied by pipeline (content-recovery.md), this comparison is available for free: the plan output of a re run is the drift report. A non empty plan against an unchanged repository means something changed outside the pipeline, which is both a correctness signal and a security signal.

Three specific alarms belong here:

• an entity that should resolve and does not,
• an entity that resolves but whose metadata differs from the declared state,
• a trust mark whose status differs from the declared state, in either direction. A mark that is still active after a withdrawal is the more serious of the two and should page.

5. Trust decisions

The signals that let an operator answer "why was this agent refused" sit at the consumer, and the demonstration surfaces them in the operations console rather than leaving them in logs:

Every denial in the demonstration renders its reason on screen, because the TC-03 and TC-08 acceptance criteria ask explicitly for denials that are observable and explainable. A denial that says only "access denied" fails the objective even when the enforcement is correct.

The five guardrail denials in scene 05 are each a different reason, and each shows a different one of these signals.

6. Change

Every state changing action is versioned by the platform with the actor, the time and the before and after state, available in the interface and through the API. Delivery to CBA's own tooling is by webhook and log stream, which is existing product capability rather than integration work.

The property that makes it usable in an investigation is correlation. Scene 07 follows a single correlation identifier from the committed manifest, through the platform change record, into the token issuance log and into the resource server decision log, filtered end to end on screen. That is what TC-18 asks for, and observability is where it is consumed.

7. Worked investigations

The real test of this section. Each of these is a question an operator will actually be asked.

"This agent stopped working an hour ago." Resolve the agent from the trust anchor. If it does not resolve, the change history says who changed what and when, and the correlation identifier leads back to the manifest and the merge that caused it. If it does resolve, compare the resolved metadata against the declared state, then look at the consumer: the refusal reason at the authorization server distinguishes "no chain" from "chain but no capability" from "envelope exceeded". Four checks, in order, and each eliminates a layer.

"We suspended this agent. Is it actually stopped?" Three questions, not one, because there are three propagation paths. Does it still resolve? Is its trust mark status still active at the authority? And has any consumer that holds a cached representation refreshed? Scene 06 measures all three with a stopwatch. The honest operational answer includes the access token lifetime at the consuming authorization server, because until an already issued token expires or is checked against the status list, withdrawal at the Trust Controller has not reached it. This is the number to have on hand before the question is asked.

"Authorization latency went up." Look at the resolve and entity configuration latency distributions first, then at whether the consumer is resolving on the request path or from cache, and whether that cache is shared across replicas or per process. A per process cache in a horizontally scaled consumer behaves like no cache at all after every deployment, and it is a recurring cause of exactly this symptom.

"Something changed that we did not approve." Re run the pipeline plan. A non empty plan against an unchanged repository is drift. The change history names the actor. If the actor is the pipeline credential, the correlation identifier leads to the merge. If it is a person, that is a separate conversation and the record supports it.

8. What is demonstrated versus what is described

9. What we need from CBA and ID Partners

1. Which monitoring platform and which SIEM receive the signals, and in which format.
2. Which of the alarms in section 4 CBA wants to page on, and which are tickets.
3. The access token lifetimes across the consuming authorization server estate, because the withdrawal worst case cannot be stated without them (content-overlay-compatibility.md).
4. Whether CBA requires the drift check to run on a schedule inside CBA's own tooling rather than the delivery pipeline.