# Concelier Authority Audit Runbook

_Last updated: 2025-10-22_

This runbook helps operators verify and monitor the StellaOps Concelier ⇆ Authority integration. It focuses on the `/jobs*` surface, which now requires StellaOps Authority tokens, and the corresponding audit/metric signals that expose authentication and bypass activity.

## 1. Prerequisites

- Authority integration is enabled in `concelier.yaml` (or via `CONCELIER_AUTHORITY__*` environment variables) with a valid `clientId`, secret, audience, and required scopes.
- OTLP metrics/log exporters are configured (`concelier.telemetry.*`) or container stdout is shipped to your SIEM.
- Operators have access to the Concelier job trigger endpoints via CLI or REST for smoke tests.
- The rollout table in `docs/10_CONCELIER_CLI_QUICKSTART.md` has been reviewed so stakeholders align on the staged → enforced toggle timeline.

### Configuration snippet

```yaml
concelier:
  authority:
    enabled: true
    allowAnonymousFallback: false          # keep true only during initial rollout
    issuer: "https://authority.internal"
    audiences:
      - "api://concelier"
    requiredScopes:
      - "concelier.jobs.trigger"
    bypassNetworks:
      - "127.0.0.1/32"
      - "::1/128"
    clientId: "concelier-jobs"
    clientSecretFile: "/run/secrets/concelier_authority_client"
    tokenClockSkewSeconds: 60
    resilience:
      enableRetries: true
      retryDelays:
        - "00:00:01"
        - "00:00:02"
        - "00:00:05"
      allowOfflineCacheFallback: true
      offlineCacheTolerance: "00:10:00"
```

> Store secrets outside source control. Concelier reads `clientSecretFile` on startup; rotate by updating the mounted file and restarting the service.

### Resilience tuning

- **Connected sites:** keep the default 1 s / 2 s / 5 s retry ladder so Concelier retries transient Authority hiccups but still surfaces outages quickly. Leave `allowOfflineCacheFallback=true` so cached discovery/JWKS data can bridge short Pathfinder restarts.
- **Air-gapped/Offline Kit installs:** extend `offlineCacheTolerance` (15–30 minutes) to keep the cached metadata valid between manual synchronisations. You can also disable retries (`enableRetries=false`) if infrastructure teams prefer to handle exponential backoff at the network layer; Concelier will fail fast but keep deterministic logs.
- Concelier resolves these knobs through `IOptionsMonitor<StellaOpsAuthClientOptions>`. Edits to `concelier.yaml` are applied on configuration reload; restart the container if you change environment variables or do not have file-watch reloads enabled.

## 2. Key Signals

### 2.1 Audit log channel

Concelier emits structured audit entries via the `Concelier.Authorization.Audit` logger for every `/jobs*` request once Authority enforcement is active.

```
Concelier authorization audit route=/jobs/definitions status=200 subject=ops@example.com clientId=concelier-cli scopes=concelier.jobs.trigger bypass=False remote=10.1.4.7
```

| Field        | Sample value            | Meaning                                                                                  |
|--------------|-------------------------|------------------------------------------------------------------------------------------|
| `route`      | `/jobs/definitions`     | Endpoint that processed the request.                                                     |
| `status`     | `200` / `401` / `409`   | Final HTTP status code returned to the caller.                                           |
| `subject`    | `ops@example.com`       | User or service principal subject (falls back to `(anonymous)` when unauthenticated).    |
| `clientId`   | `concelier-cli`           | OAuth client ID provided by Authority ( `(none)` if the token lacked the claim).         |
| `scopes`     | `concelier.jobs.trigger`  | Normalised scope list extracted from token claims; `(none)` if the token carried none.   |
| `bypass`     | `True` / `False`        | Indicates whether the request succeeded because its source IP matched a bypass CIDR.    |
| `remote`     | `10.1.4.7`              | Remote IP recorded from the connection / forwarded header test hooks.                    |

Use your logging backend (e.g., Loki) to index the logger name and filter for suspicious combinations:

- `status=401 AND bypass=True` – bypass network accepted an unauthenticated call (should be temporary during rollout).
- `status=202 AND scopes="(none)"` – a token without scopes triggered a job; tighten client configuration.
- Spike in `clientId="(none)"` – indicates upstream Authority is not issuing `client_id` claims or the CLI is outdated.

### 2.2 Metrics

Concelier publishes counters under the OTEL meter `StellaOps.Concelier.WebService.Jobs`. Tags: `job.kind`, `job.trigger`, `job.outcome`.

| Metric name                   | Description                                        | PromQL example |
|-------------------------------|----------------------------------------------------|----------------|
| `web.jobs.triggered`          | Accepted job trigger requests.                     | `sum by (job_kind) (rate(web_jobs_triggered_total[5m]))` |
| `web.jobs.trigger.conflict`   | Rejected triggers (already running, disabled…).    | `sum(rate(web_jobs_trigger_conflict_total[5m]))` |
| `web.jobs.trigger.failed`     | Server-side job failures.                          | `sum(rate(web_jobs_trigger_failed_total[5m]))` |

> Prometheus/OTEL collectors typically surface counters with `_total` suffix. Adjust queries to match your pipeline’s generated metric names.

Correlate audit logs with the following global meter exported via `Concelier.SourceDiagnostics`:

- `concelier.source.http.requests_total{concelier_source="jobs-run"}` – ensures REST/manual triggers route through Authority.
- If Grafana dashboards are deployed, extend the “Concelier Jobs” board with the above counters plus a table of recent audit log entries.

## 3. Alerting Guidance

1. **Unauthorized bypass attempt**  
   - Query: `sum(rate(log_messages_total{logger="Concelier.Authorization.Audit", status="401", bypass="True"}[5m])) > 0`  
   - Action: verify `bypassNetworks` list; confirm expected maintenance windows; rotate credentials if suspicious.

2. **Missing scopes**  
   - Query: `sum(rate(log_messages_total{logger="Concelier.Authorization.Audit", scopes="(none)", status="200"}[5m])) > 0`  
   - Action: audit Authority client registration; ensure `requiredScopes` includes `concelier.jobs.trigger`.

3. **Trigger failure surge**  
   - Query: `sum(rate(web_jobs_trigger_failed_total[10m])) > 0` with severity `warning` if sustained for 10 minutes.  
   - Action: inspect correlated audit entries and `Concelier.Telemetry` traces for job execution errors.

4. **Conflict spike**  
   - Query: `sum(rate(web_jobs_trigger_conflict_total[10m])) > 5` (tune threshold).  
   - Action: downstream scheduling may be firing repetitive triggers; ensure precedence is configured properly.

5. **Authority offline**  
   - Watch `Concelier.Authorization.Audit` logs for `status=503` or `status=500` along with `clientId="(none)"`. Investigate Authority availability before re-enabling anonymous fallback.

## 4. Rollout & Verification Procedure

1. **Pre-checks**
   - Align with the rollout phases documented in `docs/10_CONCELIER_CLI_QUICKSTART.md` (validation → rehearsal → enforced) and record the target dates in your change request.
   - Confirm `allowAnonymousFallback` is `false` in production; keep `true` only during staged validation.
   - Validate Authority issuer metadata is reachable from Concelier (`curl https://authority.internal/.well-known/openid-configuration` from the host).

2. **Smoke test with valid token**
   - Obtain a token via CLI: `stella auth login --scope concelier.jobs.trigger`.
   - Trigger a read-only endpoint: `curl -H "Authorization: Bearer $TOKEN" https://concelier.internal/jobs/definitions`.
   - Expect HTTP 200/202 and an audit log with `bypass=False`, `scopes=concelier.jobs.trigger`.

3. **Negative test without token**
   - Call the same endpoint without a token. Expect HTTP 401, `bypass=False`.
   - If the request succeeds, double-check `bypassNetworks` and ensure fallback is disabled.

4. **Bypass check (if applicable)**
   - From an allowed maintenance IP, call `/jobs/definitions` without a token. Confirm the audit log shows `bypass=True`. Review business justification and expiry date for such entries.

5. **Metrics validation**
   - Ensure `web.jobs.triggered` counter increments during accepted runs.
   - Exporters should show corresponding spans (`concelier.job.trigger`) if tracing is enabled.

## 5. Troubleshooting

| Symptom | Probable cause | Remediation |
|---------|----------------|-------------|
| Audit log shows `clientId=(none)` for all requests | Authority not issuing `client_id` claim or CLI outdated | Update StellaOps Authority configuration (`StellaOpsAuthorityOptions.Token.Claims.ClientId`), or upgrade the CLI token acquisition flow. |
| Requests succeed with `bypass=True` unexpectedly | Local network added to `bypassNetworks` or fallback still enabled | Remove/adjust the CIDR list, disable anonymous fallback, restart Concelier. |
| HTTP 401 with valid token | `requiredScopes` missing from client registration or token audience mismatch | Verify Authority client scopes (`concelier.jobs.trigger`) and ensure the token audience matches `audiences` config. |
| Metrics missing from Prometheus | Telemetry exporters disabled or filter missing OTEL meter | Set `concelier.telemetry.enableMetrics=true`, ensure collector includes `StellaOps.Concelier.WebService.Jobs` meter. |
| Sudden spike in `web.jobs.trigger.failed` | Downstream job failure or Authority timeout mid-request | Inspect Concelier job logs, re-run with tracing enabled, validate Authority latency. |

## 6. References

- `docs/21_INSTALL_GUIDE.md` – Authority configuration quick start.
- `docs/17_SECURITY_HARDENING_GUIDE.md` – Security guardrails and enforcement deadlines.
- `docs/ops/authority-monitoring.md` – Authority-side monitoring and alerting playbook.
- `StellaOps.Concelier.WebService/Filters/JobAuthorizationAuditFilter.cs` – source of audit log fields.