docs consoliation work
This commit is contained in:
StellaOps Bot
@@ -778,11 +778,13 @@ CREATE INDEX idx_large_table_col ON schema.large_table(column);
|
||||
"Persistence": {
|
||||
"Authority": "Postgres",
|
||||
"Scheduler": "Postgres",
|
||||
"Concelier": "Mongo"
|
||||
"Concelier": "Postgres"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> **Note:** MongoDB storage was deprecated in Sprint 4400. All modules now use PostgreSQL. MongoDB-related patterns in this document are retained for historical reference only.
|
||||
|
||||
### 13.2 Connection String Security
|
||||
|
||||
**RULE:** Connection strings MUST NOT be logged or included in exception messages.
|
||||
|
||||
@@ -1,33 +0,0 @@
|
||||
# Authority Rate Limit Tuning Outline (2025-10-11)
|
||||
|
||||
## Purpose
|
||||
- Drive the remaining work on SEC3.B (Security Guild) and PLG6.DOC (Docs Guild) by capturing the agreed baseline for Authority rate limits and related documentation deliverables.
|
||||
- Provide a single reference for lockout + rate limit interplay so Docs can lift accurate copy into `docs/security/rate-limits.md` and `docs/dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md`.
|
||||
|
||||
## Baseline Configuration
|
||||
- `/token`: fixed window, permitLimit 30, window 60s, queueLimit 0. Reduce to 10/60s for untrusted IP ranges; raise to 60/60s only with compensating controls (WAF + active monitoring).
|
||||
- `/authorize`: permitLimit 60, window 60s, queueLimit 10. Intended for interactive browser flows; lowering below 30 requires UX review.
|
||||
- `/internal/*`: disabled by default; recommended 5/60s with queueLimit 0 when bootstrap API exposed.
|
||||
- Configuration path: `authority.security.rateLimiting.<endpoint>` (e.g., `token.permitLimit`). YAML/ENV bindings follow the standard options hierarchy.
|
||||
- Retry metadata: middleware stamps `Retry-After` along with tags `authority.client_id`, `authority.remote_ip`, `authority.endpoint`. Docs should highlight these for operator dashboards.
|
||||
|
||||
## Parameter Matrix
|
||||
| Scenario | permitLimit | window | queueLimit | Notes |
|
||||
|----------|-------------|--------|------------|-------|
|
||||
| Default production | 30 | 60s | 0 | Works with anonymous quota (33 scans/day). |
|
||||
| High-trust clustered IPs | 60 | 60s | 5 | Requires `authorize_rate_limit_hits` alert ≤ 1% sustained. |
|
||||
| Air-gapped lab | 10 | 120s | 0 | Emphasise reduced concurrency + manual queue draining. |
|
||||
| Incident lockdown | 5 | 300s | 0 | Pair with lockout lowering to 3 attempts. |
|
||||
|
||||
## Lockout Interplay
|
||||
- Ensure Docs explain difference between rate limit (per IP/client) vs lockout (per subject). Provide table mapping retry-after headers to recommended support scripts.
|
||||
- Security Guild to define alert thresholds: trigger SOC ticket when 429 rate > 25% for 5 minutes or when limiter emits >100 events/hour per client.
|
||||
|
||||
## Observability
|
||||
- Surface metrics: `aspnetcore_rate_limiting_rejections_total{limiter="authority-token"}` and custom log tags from `AuthorityRateLimiterMetadataMiddleware`.
|
||||
- Recommend dashboard sections: request volume vs. rejections, top offending clientIds, per-endpoint heatmap.
|
||||
|
||||
## Action Items
|
||||
1. Security Guild (SEC3.B): incorporate matrix + alert rules into `docs/security/rate-limits.md`, add YAML examples for override blocks, and cross-link lockout policy doc.
|
||||
2. Docs Guild (PLG6.DOC): update developer guide section 9 with the middleware sequence and reference this outline for retry metadata + tuning guidance.
|
||||
3. Authority Core: validate appsettings sample includes the `security.rateLimiting` block with comments and link back to published doc once ready.
|
||||
@@ -43,7 +43,9 @@ If multiple connectors emit identical constraints, the merge layer should:
|
||||
2. Preserve a single normalized rule instance (thanks to `NormalizedVersionRuleEqualityComparer.Instance`).
|
||||
3. Attach `decisionReason="precedence"` if one source overrides another.
|
||||
|
||||
## 4. Example Mongo pipeline
|
||||
## 4. Example Query (Historical MongoDB Reference)
|
||||
|
||||
> **Note:** This section shows historical MongoDB syntax. The current system uses PostgreSQL.
|
||||
|
||||
Use the following aggregation to locate advisories that affect a specific SemVer:
|
||||
|
||||
@@ -102,7 +104,7 @@ For current database schema and query patterns, see `docs/db/SPECIFICATION.md`.
|
||||
|
||||
## 8. Storage projection reference
|
||||
|
||||
`NormalizedVersionDocumentFactory` copies each normalized rule into MongoDB using the shape below. Use this as a contract when reviewing connector fixtures or diagnosing merge/storage diffs:
|
||||
`NormalizedVersionDocumentFactory` copies each normalized rule into PostgreSQL using the shape below. Use this as a contract when reviewing connector fixtures or diagnosing merge/storage diffs:
|
||||
|
||||
```json
|
||||
{
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# StellaOps Attestor
|
||||
|
||||
# StellaOps Attestor
|
||||
|
||||
Attestor converts signed DSSE evidence from the Signer into transparency-log proofs and verifiable reports for every downstream surface (Policy Engine, Export Center, CLI, Console, Scheduler). It is the trust backbone that proves SBOM, scan, VEX, and policy artefacts were signed, witnessed, and preserved without tampering.
|
||||
|
||||
## Latest updates (2025-11-30)
|
||||
@@ -11,19 +11,19 @@ Attestor converts signed DSSE evidence from the Signer into transparency-log pro
|
||||
- **Evidence first:** organisations need portable, verifiable attestations that prove build provenance, SBOM availability, policy verdicts, and VEX statements.
|
||||
- **Policy enforcement:** verification policies ensure only approved issuers, key types, witnesses, and freshness windows are accepted.
|
||||
- **Sovereign/offline-ready:** Attestor archives envelopes, signatures, and proofs so air-gapped deployments can replay verification without contacting external services.
|
||||
|
||||
## Roles & surfaces
|
||||
- **Subjects:** immutable digests for container images, SBOMs, reports, and policy bundles.
|
||||
- **Issuers:** builders, scanners, policy engines, or operators signing DSSE envelopes using keyless (Fulcio), KMS/HSM, or FIDO2 keys.
|
||||
- **Consumers:** CLI/SDK, Console, Export Center, Scanner, Policy Engine, and Notify retrieving verification bundles or triggering policy checks.
|
||||
- **Scopes:** Authority issues `attestor.write`, `attestor.verify`, `attestor.read`, and administrative scopes for issuer/key management; every call is bound with mTLS + DPoP.
|
||||
|
||||
## Supported payloads
|
||||
- `StellaOps.BuildProvenance@1`, `StellaOps.SBOMAttestation@1`
|
||||
- `StellaOps.ScanResults@1`, `StellaOps.VEXAttestation@1`
|
||||
- `StellaOps.PolicyEvaluation@1`, `StellaOps.RiskProfileEvidence@1`
|
||||
All predicates capture subjects, issuer metadata, policy context, materials, optional witnesses, and versioned schemas. Unsupported predicates return `422 predicate_unsupported`.
|
||||
|
||||
|
||||
## Roles & surfaces
|
||||
- **Subjects:** immutable digests for container images, SBOMs, reports, and policy bundles.
|
||||
- **Issuers:** builders, scanners, policy engines, or operators signing DSSE envelopes using keyless (Fulcio), KMS/HSM, or FIDO2 keys.
|
||||
- **Consumers:** CLI/SDK, Console, Export Center, Scanner, Policy Engine, and Notify retrieving verification bundles or triggering policy checks.
|
||||
- **Scopes:** Authority issues `attestor.write`, `attestor.verify`, `attestor.read`, and administrative scopes for issuer/key management; every call is bound with mTLS + DPoP.
|
||||
|
||||
## Supported payloads
|
||||
- `StellaOps.BuildProvenance@1`, `StellaOps.SBOMAttestation@1`
|
||||
- `StellaOps.ScanResults@1`, `StellaOps.VEXAttestation@1`
|
||||
- `StellaOps.PolicyEvaluation@1`, `StellaOps.RiskProfileEvidence@1`
|
||||
All predicates capture subjects, issuer metadata, policy context, materials, optional witnesses, and versioned schemas. Unsupported predicates return `422 predicate_unsupported`.
|
||||
|
||||
## Trust & envelope model
|
||||
- DSSE envelopes are canonicalised, hashed, and stored alongside the Rekor UUID, index, and proof.
|
||||
- Signature modes span keyless (Fulcio), keyful (KMS/HSM), and hardware-backed (FIDO2). Multiple signatures are supported per envelope.
|
||||
@@ -40,27 +40,27 @@ All predicates capture subjects, issuer metadata, policy context, materials, opt
|
||||
- **Console:** Evidence browser, verification reports, chain-of-custody graph, issuer/key management, attestation workbench, and bulk verification flows.
|
||||
- **CLI / SDK:** `stella attest sign|verify|list|fetch|key` commands plus language SDKs to integrate build pipelines and offline verification scripts.
|
||||
- **Policy Studio:** Verification policies author required predicate types, issuers, witness requirements, and freshness windows; simulations show enforcement impact.
|
||||
|
||||
## Storage, offline & air-gap posture
|
||||
- MongoDB stores entry metadata, dedupe keys, and audit events; object storage optionally archives DSSE bundles.
|
||||
- Export Center packages attestation bundles (`stella export attestation-bundle`) for Offline Kit delivery.
|
||||
- Transparency logs can be mirrored; offline mode records gaps and provides compensating controls.
|
||||
|
||||
|
||||
## Storage, offline & air-gap posture
|
||||
- PostgreSQL stores entry metadata, dedupe keys, and audit events; object storage optionally archives DSSE bundles.
|
||||
- Export Center packages attestation bundles (`stella export attestation-bundle`) for Offline Kit delivery.
|
||||
- Transparency logs can be mirrored; offline mode records gaps and provides compensating controls.
|
||||
|
||||
## Observability & performance
|
||||
- Metrics: `attestor_submission_total`, `attestor_verify_seconds`, `attestor_cache_hit_ratio`, `attestor_rekor_latency_seconds`.
|
||||
- Logs capture tenant, issuer, subject digests, Rekor UUID, proof status, and policy verdict.
|
||||
- Performance target: ≥1 000 envelopes/minute per worker with cached verification, batched operations, and concurrency controls.
|
||||
- Observability assets: `operations/observability.md` and `operations/dashboards/attestor-observability.json` (offline import).
|
||||
|
||||
## Key integrations
|
||||
- Signer (DSSE source), Authority (scopes & tenancy), Export Center (attestation bundles), Policy Engine (verification policies), Scanner/Excititor (subject evidence), Notify (key rotation & verification alerts), Observability stack (dashboards/alerts).
|
||||
|
||||
## Backlog references
|
||||
- DOCS-ATTEST-73-001 … DOCS-ATTEST-75-002 (Attestor console, key management, air-gap bundles) in ../../TASKS.md.
|
||||
- EXPORT-ATTEST-75-002 (Export Center attestation packaging) in ../export-center/TASKS.md.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 19 – Attestor Console:** console experience, verification APIs, issuer/key governance, transparency integration, and offline bundles.
|
||||
- **Epic 10 – Export Center:** provenance alignment so exports carry signed manifests and attestation bundles.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
## Key integrations
|
||||
- Signer (DSSE source), Authority (scopes & tenancy), Export Center (attestation bundles), Policy Engine (verification policies), Scanner/Excititor (subject evidence), Notify (key rotation & verification alerts), Observability stack (dashboards/alerts).
|
||||
|
||||
## Backlog references
|
||||
- DOCS-ATTEST-73-001 … DOCS-ATTEST-75-002 (Attestor console, key management, air-gap bundles) in ../../TASKS.md.
|
||||
- EXPORT-ATTEST-75-002 (Export Center attestation packaging) in ../export-center/TASKS.md.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 19 – Attestor Console:** console experience, verification APIs, issuer/key governance, transparency integration, and offline bundles.
|
||||
- **Epic 10 – Export Center:** provenance alignment so exports carry signed manifests and attestation bundles.
|
||||
|
||||
> **Imposed rule:** Work of this type or tasks of this type on this component must also be applied everywhere else it should be applied.
|
||||
|
||||
@@ -482,7 +482,7 @@ The worker honours `bulkVerification.itemDelayMilliseconds` for throttling and r
|
||||
* **Proof acquisition**:
|
||||
|
||||
* In synchronous mode, poll the log for inclusion up to `proofTimeoutMs`.
|
||||
* In asynchronous mode, return `pending` and schedule a **proof fetcher** job (Mongo job doc + backoff).
|
||||
* In asynchronous mode, return `pending` and schedule a **proof fetcher** job (PostgreSQL job record + backoff).
|
||||
* **Mirrors/dual logs**:
|
||||
|
||||
* When `logPreference="both"`, submit to primary and mirror; store **both** UUIDs (primary canonical).
|
||||
|
||||
365
docs/modules/attestor/reg-evidence-architecture.md
Normal file
365
docs/modules/attestor/reg-evidence-architecture.md
Normal file
@@ -0,0 +1,365 @@
|
||||
# Resolved Execution Graph (REG) Evidence Architecture
|
||||
|
||||
> **Status:** Proposed
|
||||
> **Sprint Series:** 8100.0012.*
|
||||
> **Last Updated:** 2025-12-24
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the architectural enhancements to StellaOps' evidence and attestation subsystems based on the **Merkle-Hash REG** (Resolved Execution Graph) pattern. The core insight: when every node in an execution graph is identified by a **Merkle hash of its normalized content**, evidence can be attached to *content itself* rather than brittle positional indices.
|
||||
|
||||
## Motivation
|
||||
|
||||
### Problem Statement
|
||||
|
||||
StellaOps currently has robust foundations for content-addressed identifiers, Merkle trees, and attestations. However, three gaps limit the system's verifiability:
|
||||
|
||||
1. **Canonicalizer versioning:** Hash collisions possible if canonicalization logic changes
|
||||
2. **Fragmented evidence models:** Different modules use different evidence structures
|
||||
3. **Implicit graph roots:** Merkle roots are computed but not independently attested
|
||||
|
||||
### Target Benefits
|
||||
|
||||
| Benefit | Description |
|
||||
|---------|-------------|
|
||||
| **Reproducible proofs** | Verifiers re-hash node content and check against stored hashes—no fragile indices |
|
||||
| **Dedup & reuse** | Identical content across pipelines collapses to one ID; one piece of evidence justifies many occurrences |
|
||||
| **Audits + time travel** | Snapshot the graph's Merkle root; any replay matching the root guarantees identical nodes & evidence |
|
||||
| **Offline verification** | Attestations are self-contained; no network required to verify |
|
||||
| **Exception stability** | Exceptions bind to content hashes, not "row 42"; stable across rebuilds |
|
||||
|
||||
## Architecture
|
||||
|
||||
### Component Diagram
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ StellaOps REG Architecture │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
|
||||
│ │ StellaOps. │ │ StellaOps. │ │ StellaOps. │ │
|
||||
│ │ Canonical.Json │────▶│ Evidence.Core │◀────│ Attestor. │ │
|
||||
│ │ │ │ │ │ GraphRoot │ │
|
||||
│ │ • CanonVersion │ │ • IEvidence │ │ │ │
|
||||
│ │ • CanonJson │ │ • EvidenceRecord│ │ • IGraphRoot- │ │
|
||||
│ │ • Versioned │ │ • IEvidenceStore│ │ Attestor │ │
|
||||
│ │ Hashing │ │ • Adapters │ │ • Verification │ │
|
||||
│ └──────────────────┘ └──────────────────┘ └──────────────────┘ │
|
||||
│ │ │ │ │
|
||||
│ └────────────────────────┼────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Content-Addressed Store │ │
|
||||
│ │ │ │
|
||||
│ │ Evidence by subject_node_id │ Graph roots by root_hash │ │
|
||||
│ │ ──────────────────────────── │ ────────────────────────── │ │
|
||||
│ │ sha256:abc... → [evidence] │ sha256:xyz... → attestation │ │
|
||||
│ │ sha256:def... → [evidence] │ sha256:uvw... → attestation │ │
|
||||
│ └──────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
|
||||
│ │ Scanner │ │ Policy │ │ Excititor │ │
|
||||
│ │ ────────────── │ │ ────────────── │ │ ────────────── │ │
|
||||
│ │ • EvidenceBundle│────▶│ • Exception │────▶│ • VexObservation│ │
|
||||
│ │ • ProofSpine │ │ Application │ │ • VexLinkset │ │
|
||||
│ │ • RichGraph │ │ • PolicyVerdict │ │ │ │
|
||||
│ └──────────────────┘ └──────────────────┘ └──────────────────┘ │
|
||||
│ │ │ │ │
|
||||
│ └────────────────────────┼────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ Unified IEvidence│ │
|
||||
│ │ via Adapters │ │
|
||||
│ └──────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
1. Content Creation
|
||||
─────────────────
|
||||
Component/Node → Canonicalize(content, version) → SHA-256 → node_id
|
||||
|
||||
2. Evidence Attachment
|
||||
────────────────────
|
||||
Analysis Result → IEvidence { subject_node_id, type, payload, provenance }
|
||||
→ EvidenceId = hash(subject + type + payload + provenance)
|
||||
→ Store(evidence)
|
||||
|
||||
3. Graph Construction
|
||||
───────────────────
|
||||
Nodes + Edges → Sort(node_ids) + Sort(edge_ids) → Merkle Tree → root_hash
|
||||
|
||||
4. Graph Attestation
|
||||
──────────────────
|
||||
GraphRootAttestationRequest → GraphRootAttestor
|
||||
→ In-Toto Statement { subject: [root, artifact] }
|
||||
→ DSSE Sign
|
||||
→ Optional: Rekor Publish
|
||||
|
||||
5. Verification
|
||||
─────────────
|
||||
Download attestation → Verify signature
|
||||
→ Fetch nodes/edges by ID
|
||||
→ Recompute Merkle root
|
||||
→ Compare with attested root
|
||||
```
|
||||
|
||||
## Implementation Sprints
|
||||
|
||||
| Sprint | Title | Dependency | Key Deliverables |
|
||||
|--------|-------|------------|------------------|
|
||||
| 8100.0012.0001 | Canonicalizer Versioning | None | `CanonVersion`, `CanonicalizeVersioned()`, backward compatibility |
|
||||
| 8100.0012.0002 | Unified Evidence Model | 0001 | `IEvidence`, `EvidenceRecord`, `IEvidenceStore`, adapters |
|
||||
| 8100.0012.0003 | Graph Root Attestation | 0001, 0002 | `IGraphRootAttestor`, in-toto statements, Rekor integration |
|
||||
|
||||
### Sprint Sequence Diagram
|
||||
|
||||
```
|
||||
Week 1-2 Week 3-4 Week 5-6 Week 7-8
|
||||
──────── ──────── ──────── ────────
|
||||
│ │ │ │
|
||||
│ 0001: Canon │ │ │
|
||||
│ Versioning │ │ │
|
||||
│ ┌─────────┐ │ │ │
|
||||
│ │Wave 0-1 │────┼─▶ 0002: Unified│ │
|
||||
│ │Wave 2-3 │ │ Evidence │ │
|
||||
│ │Wave 4 │ │ ┌─────────┐ │ │
|
||||
│ └─────────┘ │ │Wave 0-1 │──┼─▶ 0003: Graph │
|
||||
│ │ │Wave 2-3 │ │ Root Attest │
|
||||
│ │ │Wave 4 │ │ ┌─────────┐ │
|
||||
│ │ └─────────┘ │ │Wave 0-1 │ │
|
||||
│ │ │ │Wave 2-4 │ │
|
||||
│ │ │ │Wave 5 │ │
|
||||
│ │ │ └─────────┘ │
|
||||
▼ ▼ ▼ ▼
|
||||
```
|
||||
|
||||
## Technical Specifications
|
||||
|
||||
### Canonicalization Version Marker
|
||||
|
||||
```json
|
||||
{
|
||||
"_canonVersion": "stella:canon:v1",
|
||||
"evidenceSource": "stellaops/scanner/reachability",
|
||||
"sbomEntryId": "sha256:abc123...:pkg:npm/lodash@4.17.21",
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
The `_canonVersion` field:
|
||||
- Uses underscore prefix to sort first lexicographically
|
||||
- Identifies the exact canonicalization algorithm
|
||||
- Enables graceful version migration
|
||||
- Is included in all content-addressed hash computations
|
||||
|
||||
### Unified Evidence Record
|
||||
|
||||
```typescript
|
||||
interface IEvidence {
|
||||
// Content-addressed binding
|
||||
subjectNodeId: string; // "sha256:{hex}" - what this evidence is about
|
||||
evidenceId: string; // "sha256:{hex}" - ID of this evidence record
|
||||
|
||||
// Type and payload
|
||||
evidenceType: EvidenceType; // reachability, scan, policy, vex, etc.
|
||||
payload: Uint8Array; // Canonical JSON bytes
|
||||
payloadSchemaVersion: string;
|
||||
|
||||
// Attestation
|
||||
signatures: EvidenceSignature[];
|
||||
|
||||
// Provenance
|
||||
provenance: {
|
||||
generatorId: string;
|
||||
generatorVersion: string;
|
||||
generatedAt: DateTimeOffset;
|
||||
inputsDigest?: string;
|
||||
correlationId?: string;
|
||||
};
|
||||
|
||||
// External storage
|
||||
externalPayloadCid?: string;
|
||||
}
|
||||
```
|
||||
|
||||
### Graph Root Attestation (In-Toto)
|
||||
|
||||
```json
|
||||
{
|
||||
"_type": "https://in-toto.io/Statement/v1",
|
||||
"subject": [
|
||||
{
|
||||
"name": "sha256:abc123...",
|
||||
"digest": { "sha256": "abc123..." }
|
||||
},
|
||||
{
|
||||
"name": "sha256:def456...",
|
||||
"digest": { "sha256": "def456..." }
|
||||
}
|
||||
],
|
||||
"predicateType": "https://stella-ops.org/attestation/graph-root/v1",
|
||||
"predicate": {
|
||||
"graphType": "ResolvedExecutionGraph",
|
||||
"rootHash": "sha256:abc123...",
|
||||
"nodeCount": 1247,
|
||||
"edgeCount": 3891,
|
||||
"nodeIds": ["sha256:...", ...],
|
||||
"edgeIds": ["sha256:...", ...],
|
||||
"inputs": {
|
||||
"policyDigest": "sha256:...",
|
||||
"feedsDigest": "sha256:...",
|
||||
"toolchainDigest": "sha256:...",
|
||||
"paramsDigest": "sha256:..."
|
||||
},
|
||||
"evidenceIds": ["sha256:...", ...],
|
||||
"canonVersion": "stella:canon:v1",
|
||||
"computedAt": "2025-12-24T10:30:00Z",
|
||||
"computedBy": "stellaops/attestor/graph-root",
|
||||
"computedByVersion": "1.0.0"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Verification Guarantees
|
||||
|
||||
### What Can Be Verified
|
||||
|
||||
| Claim | Verification Method |
|
||||
|-------|---------------------|
|
||||
| "This graph contains exactly these nodes" | Recompute Merkle root from node IDs |
|
||||
| "This evidence applies to node X" | Check evidence.subjectNodeId == node.id |
|
||||
| "This attestation was signed by key K" | Verify DSSE envelope signature |
|
||||
| "This graph was published to transparency log" | Check Rekor inclusion proof |
|
||||
| "These inputs produced this graph" | Check inputs.* digests match expectations |
|
||||
|
||||
### What Cannot Be Verified (Without Additional Trust)
|
||||
|
||||
| Claim | Reason | Mitigation |
|
||||
|-------|--------|------------|
|
||||
| "The analysis was performed correctly" | Semantic, not structural | Trust the generator; audit toolchain |
|
||||
| "No evidence was omitted" | Attester controls content | Require known evidence types; policy enforcement |
|
||||
| "The inputs are authentic" | External dependency | Chain attestations; verify feed signatures |
|
||||
|
||||
## Migration Path
|
||||
|
||||
### Phase 1: Parallel Operation (Sprints 0001-0003)
|
||||
|
||||
- New versioned hashing alongside legacy
|
||||
- New evidence model with adapters for existing types
|
||||
- Graph root attestations optional
|
||||
|
||||
### Phase 2: Gradual Adoption (Post-Sprints)
|
||||
|
||||
- Emit migration warnings for legacy hashes
|
||||
- Prefer IEvidence in new code
|
||||
- Enable graph root attestations by default
|
||||
|
||||
### Phase 3: Deprecation (Future)
|
||||
|
||||
- Remove legacy hash acceptance
|
||||
- Require IEvidence for all evidence storage
|
||||
- Require graph root attestations for verification
|
||||
|
||||
## Compatibility Considerations
|
||||
|
||||
### Existing Attestations
|
||||
|
||||
Attestations generated before canonicalizer versioning remain valid:
|
||||
- Verification detects legacy format (no `_canonVersion` field)
|
||||
- Falls back to legacy canonicalization
|
||||
- Logs warning for migration tracking
|
||||
|
||||
### Existing Evidence
|
||||
|
||||
Existing evidence types (`EvidenceBundle`, `EvidenceStatement`, etc.) continue working:
|
||||
- Adapters convert to `IEvidence` on demand
|
||||
- Original types remain in storage
|
||||
- Gradual migration via write-through
|
||||
|
||||
### API Stability
|
||||
|
||||
Public APIs remain backward compatible:
|
||||
- New methods added, not changed
|
||||
- Optional parameters for new features
|
||||
- Explicit opt-in for graph root attestations
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
| Operation | Impact | Mitigation |
|
||||
|-----------|--------|------------|
|
||||
| Version field injection | ~100 bytes per hash | Negligible |
|
||||
| Merkle tree computation | O(n log n) for n nodes | Existing algorithm; no change |
|
||||
| Graph root attestation | +1 DSSE sign per graph | Batching; async |
|
||||
| Evidence store queries | Index on subject_node_id | Composite index; pagination |
|
||||
| Rekor publishing | Network latency | Optional; async; batching |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Threat Model
|
||||
|
||||
| Threat | Mitigation |
|
||||
|--------|------------|
|
||||
| Hash collision attacks | SHA-256 with 256-bit security; version namespacing |
|
||||
| Signature forgery | DSSE with ECDSA/EdDSA; key rotation |
|
||||
| Evidence tampering | Content-addressed storage; Merkle verification |
|
||||
| Replay attacks | Timestamp in provenance; Rekor log index |
|
||||
| Canonicalization attacks | RFC 8785 compliance; explicit versioning |
|
||||
|
||||
### Key Management
|
||||
|
||||
Graph root attestations use the existing Signer module:
|
||||
- Keys managed via Authority plugin
|
||||
- Rotation policy applies
|
||||
- Certificate chains optional for external verification
|
||||
|
||||
## References
|
||||
|
||||
- [RFC 8785 - JSON Canonicalization Scheme (JCS)](https://datatracker.ietf.org/doc/html/rfc8785)
|
||||
- [in-toto Attestation Framework](https://github.com/in-toto/attestation)
|
||||
- [DSSE - Dead Simple Signing Envelope](https://github.com/secure-systems-lab/dsse)
|
||||
- [Sigstore Rekor](https://docs.sigstore.dev/rekor/overview/)
|
||||
- [Merkle Tree - Wikipedia](https://en.wikipedia.org/wiki/Merkle_tree)
|
||||
|
||||
## Appendix A: File Locations
|
||||
|
||||
| Component | Path |
|
||||
|-----------|------|
|
||||
| CanonVersion | `src/__Libraries/StellaOps.Canonical.Json/CanonVersion.cs` |
|
||||
| CanonJson (versioned) | `src/__Libraries/StellaOps.Canonical.Json/CanonJson.cs` |
|
||||
| IEvidence | `src/__Libraries/StellaOps.Evidence.Core/IEvidence.cs` |
|
||||
| EvidenceRecord | `src/__Libraries/StellaOps.Evidence.Core/EvidenceRecord.cs` |
|
||||
| IEvidenceStore | `src/__Libraries/StellaOps.Evidence.Core/IEvidenceStore.cs` |
|
||||
| Adapters | `src/__Libraries/StellaOps.Evidence.Core/Adapters/` |
|
||||
| IGraphRootAttestor | `src/Attestor/__Libraries/StellaOps.Attestor.GraphRoot/IGraphRootAttestor.cs` |
|
||||
| GraphRootAttestation | `src/Attestor/__Libraries/StellaOps.Attestor.GraphRoot/Models/` |
|
||||
|
||||
## Appendix B: Example Evidence Queries
|
||||
|
||||
```csharp
|
||||
// Get all reachability evidence for a package
|
||||
var evidence = await evidenceStore.GetBySubjectAsync(
|
||||
subjectNodeId: "sha256:abc123...pkg:npm/lodash@4.17.21",
|
||||
typeFilter: EvidenceType.Reachability);
|
||||
|
||||
// Verify a graph root attestation
|
||||
var result = await graphRootAttestor.VerifyAsync(
|
||||
envelope: downloadedEnvelope,
|
||||
nodes: fetchedNodes,
|
||||
edges: fetchedEdges);
|
||||
|
||||
if (!result.IsValid)
|
||||
throw new VerificationException(result.FailureReason);
|
||||
|
||||
// Check if evidence exists before creating duplicate
|
||||
if (!await evidenceStore.ExistsAsync(subjectNodeId, EvidenceType.Scan))
|
||||
{
|
||||
await evidenceStore.StoreAsync(newEvidence);
|
||||
}
|
||||
```
|
||||
@@ -1,5 +1,7 @@
|
||||
# Attestor TTL Validation Runbook
|
||||
|
||||
> **DEPRECATED:** This runbook tests MongoDB TTL indexes, which are no longer used. Attestor now uses PostgreSQL for persistence (Sprint 4400). See `docs/db/SPECIFICATION.md` for current database schema.
|
||||
|
||||
> **Purpose:** confirm MongoDB TTL indexes and Redis expirations for the attestation dedupe store behave as expected on a production-like stack.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -2,13 +2,13 @@
|
||||
|
||||
Authority is the platform OIDC/OAuth2 control plane that mints short-lived, sender-constrained operational tokens (OpToks) for every StellaOps service and tool.
|
||||
|
||||
## Latest updates (2025-12-04)
|
||||
- Added gap remediation package for AU1–AU10 and RR1–RR10 (31-Nov-2025 FINDINGS) under `docs/modules/authority/gaps/`; includes deliverable map + evidence layout.
|
||||
- Sprint tracker `docs/implplan/SPRINT_0314_0001_0001_docs_modules_authority.md` and module `TASKS.md` mirror status.
|
||||
- Monitoring/observability references consolidated; Grafana JSON remains offline import (`operations/grafana-dashboard.json`).
|
||||
- Prior content retained: OpTok/DPoP/mTLS responsibilities, backup/restore, key rotation.
|
||||
|
||||
## Responsibilities
|
||||
## Latest updates (2025-12-04)
|
||||
- Added gap remediation package for AU1–AU10 and RR1–RR10 (31-Nov-2025 FINDINGS) under `docs/modules/authority/gaps/`; includes deliverable map + evidence layout.
|
||||
- Sprint tracker `docs/implplan/SPRINT_0314_0001_0001_docs_modules_authority.md` and module `TASKS.md` mirror status.
|
||||
- Monitoring/observability references consolidated; Grafana JSON remains offline import (`operations/grafana-dashboard.json`).
|
||||
- Prior content retained: OpTok/DPoP/mTLS responsibilities, backup/restore, key rotation.
|
||||
|
||||
## Responsibilities
|
||||
- Expose device-code, auth-code, and client-credential flows with DPoP or mTLS binding.
|
||||
- Manage signing keys, JWKS rotation, and PoE integration for plan enforcement.
|
||||
- Emit structured audit events and enforce tenant-aware scope policies.
|
||||
@@ -24,20 +24,20 @@ Authority is the platform OIDC/OAuth2 control plane that mints short-lived, send
|
||||
- CLI/UI for login flows and token management.
|
||||
- Scheduler/Scanner for machine-to-machine scope enforcement.
|
||||
|
||||
## Operational notes
|
||||
- MongoDB for tenant, client, and token state.
|
||||
- Key material in KMS/HSM with rotation runbooks (`operations/key-rotation.md`).
|
||||
- Monitoring runbook (`operations/monitoring.md`) and offline-import Grafana JSON (`operations/grafana-dashboard.json`).
|
||||
## Operational notes
|
||||
- PostgreSQL (schema `authority`) for tenant, client, and token state.
|
||||
- Key material in KMS/HSM with rotation runbooks (`operations/key-rotation.md`).
|
||||
- Monitoring runbook (`operations/monitoring.md`) and offline-import Grafana JSON (`operations/grafana-dashboard.json`).
|
||||
|
||||
## Related resources
|
||||
- ./operations/backup-restore.md
|
||||
- ./operations/key-rotation.md
|
||||
- ./operations/monitoring.md
|
||||
- ./operations/grafana-dashboard.json
|
||||
- ./crypto-provider-contract.md
|
||||
- ./gaps/2025-12-04-auth-gaps-au1-au10.md
|
||||
- ./gaps/2025-12-04-rekor-receipt-gaps-rr1-rr10.md
|
||||
- Sprint/status mirrors: `docs/implplan/SPRINT_0314_0001_0001_docs_modules_authority.md`, `docs/modules/authority/TASKS.md`
|
||||
## Related resources
|
||||
- ./operations/backup-restore.md
|
||||
- ./operations/key-rotation.md
|
||||
- ./operations/monitoring.md
|
||||
- ./operations/grafana-dashboard.json
|
||||
- ./crypto-provider-contract.md
|
||||
- ./gaps/2025-12-04-auth-gaps-au1-au10.md
|
||||
- ./gaps/2025-12-04-rekor-receipt-gaps-rr1-rr10.md
|
||||
- Sprint/status mirrors: `docs/implplan/SPRINT_0314_0001_0001_docs_modules_authority.md`, `docs/modules/authority/TASKS.md`
|
||||
|
||||
## Backlog references
|
||||
- DOCS-SEC-62-001 (scope hardening doc) in ../../TASKS.md.
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
# StellaOps Concelier
|
||||
|
||||
Concelier ingests signed advisories from dozens of sources and converts them into immutable observations plus linksets under the Aggregation-Only Contract (AOC).
|
||||
|
||||
## Responsibilities
|
||||
# StellaOps Concelier
|
||||
|
||||
Concelier ingests signed advisories from dozens of sources and converts them into immutable observations plus linksets under the Aggregation-Only Contract (AOC).
|
||||
|
||||
## Responsibilities
|
||||
- Fetch and normalise vulnerability advisories via restart-time connectors.
|
||||
- Persist observations and correlation linksets without precedence decisions.
|
||||
- Emit deterministic exports (JSON, Trivy DB) for downstream policy evaluation.
|
||||
- Coordinate offline/air-gap updates via Offline Kit bundles.
|
||||
- Serve paragraph-anchored advisory chunks for Advisory AI consumers without breaking the Aggregation-Only Contract.
|
||||
|
||||
|
||||
## Key components
|
||||
- `StellaOps.Concelier.WebService` orchestration host.
|
||||
- Connector libraries under `StellaOps.Concelier.Connector.*`.
|
||||
@@ -16,12 +16,12 @@ Concelier ingests signed advisories from dozens of sources and converts them int
|
||||
|
||||
## Recent updates
|
||||
- **2025-11-07:** Paragraph-anchored `/advisories/{advisoryKey}/chunks` endpoint shipped for Advisory AI paragraph retrieval. Details and rollout notes live in [`../../updates/2025-11-07-concelier-advisory-chunks.md`](../../updates/2025-11-07-concelier-advisory-chunks.md).
|
||||
|
||||
## Integrations & dependencies
|
||||
- MongoDB for canonical observations and schedules.
|
||||
- Policy Engine / Export Center / CLI for evidence consumption.
|
||||
- Notify and UI for advisory deltas.
|
||||
|
||||
|
||||
## Integrations & dependencies
|
||||
- PostgreSQL (schema `vuln`) for canonical observations and schedules.
|
||||
- Policy Engine / Export Center / CLI for evidence consumption.
|
||||
- Notify and UI for advisory deltas.
|
||||
|
||||
## Operational notes
|
||||
- Connector runbooks in ./operations/connectors/.
|
||||
- Mirror operations for Offline Kit parity.
|
||||
@@ -38,6 +38,6 @@ Concelier ingests signed advisories from dozens of sources and converts them int
|
||||
- DOCS-LNM-22-001, DOCS-LNM-22-007 in ../../TASKS.md.
|
||||
- Connector-specific TODOs in `src/Concelier/**/TASKS.md`.
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 1 – AOC enforcement:** uphold raw observation invariants, provenance requirements, linkset-only enrichment, and AOC verifier guardrails across every connector.
|
||||
- **Epic 10 – Export Center:** expose deterministic advisory exports and metadata required by JSON/Trivy/mirror bundles.
|
||||
## Epic alignment
|
||||
- **Epic 1 – AOC enforcement:** uphold raw observation invariants, provenance requirements, linkset-only enrichment, and AOC verifier guardrails across every connector.
|
||||
- **Epic 10 – Export Center:** expose deterministic advisory exports and metadata required by JSON/Trivy/mirror bundles.
|
||||
|
||||
@@ -45,7 +45,7 @@ Expect all logs at `Information`. Ensure OTEL exporters include the scope `Stell
|
||||
- `eventId=1002` with `reason="equal_rank"` - indicates precedence table gaps; page merge owners.
|
||||
- `eventId=1002` with `reason="mismatch"` - severity disagreement; open connector bug if sustained.
|
||||
3. **Job health**
|
||||
- `stella db merge` exit code `1` signifies unresolved conflicts. Pipe to automation that captures logs and notifies #concelier-ops.
|
||||
- `stella db merge` exit code `1` signifies unresolved conflicts. Pipe to automation that captures logs and notifies #concelier-ops.
|
||||
|
||||
### Threshold updates (2025-10-12)
|
||||
|
||||
@@ -58,7 +58,7 @@ Expect all logs at `Information`. Ensure OTEL exporters include the scope `Stell
|
||||
## 4. Triage Workflow
|
||||
|
||||
1. **Confirm job context**
|
||||
- `stella db merge` (CLI) or `POST /jobs/merge:reconcile` (API) to rehydrate the merge job. Use `--verbose` to stream structured logs during triage.
|
||||
- `stella db merge` (CLI) or `POST /jobs/merge:reconcile` (API) to rehydrate the merge job. Use `--verbose` to stream structured logs during triage.
|
||||
2. **Inspect metrics**
|
||||
- Correlate spikes in `concelier.merge.conflicts` with `primary_source`/`suppressed_source` tags from `concelier.merge.overrides`.
|
||||
3. **Pull structured logs**
|
||||
@@ -94,7 +94,7 @@ Expect all logs at `Information`. Ensure OTEL exporters include the scope `Stell
|
||||
## 6. Resolution Playbook
|
||||
|
||||
1. **Connector data fix**
|
||||
- Re-run the offending connector stages (`stella db fetch --source ghsa --stage map` etc.).
|
||||
- Re-run the offending connector stages (`stella db fetch --source ghsa --stage map` etc.).
|
||||
- Once fixed, rerun merge and verify `decisionReason` reflects `freshness` or `precedence` as expected.
|
||||
2. **Temporary precedence override**
|
||||
- Edit `etc/concelier.yaml`:
|
||||
@@ -131,7 +131,7 @@ Expect all logs at `Information`. Ensure OTEL exporters include the scope `Stell
|
||||
- Canonical conflict rules: `src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md`.
|
||||
- Merge engine internals: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/AdvisoryPrecedenceMerger.cs`.
|
||||
- Metrics definitions: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/AdvisoryMergeService.cs` (identity conflicts) and `AdvisoryPrecedenceMerger`.
|
||||
- Storage audit trail: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/MergeEventWriter.cs`, `src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/MergeEvents`.
|
||||
- Storage audit trail: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/MergeEventWriter.cs`, `src/Concelier/__Libraries/StellaOps.Concelier.Storage/MergeEvents`.
|
||||
|
||||
Keep this runbook synchronized with future sprint notes and update alert thresholds as baseline volumes change.
|
||||
|
||||
|
||||
@@ -25,13 +25,13 @@ concelier:
|
||||
|
||||
## 2. Staging Smoke Test
|
||||
|
||||
1. Deploy the configuration and restart the Concelier workers to ensure the Apple connector options are bound.
|
||||
2. Trigger a full connector cycle:
|
||||
- CLI: run `stella db fetch --source vndr-apple --stage fetch`, then `--stage parse`, then `--stage map`.
|
||||
- REST: `POST /jobs/run { "kind": "source:vndr-apple:fetch", "chain": ["source:vndr-apple:parse", "source:vndr-apple:map"] }`
|
||||
3. Validate metrics exported under meter `StellaOps.Concelier.Connector.Vndr.Apple`:
|
||||
- `apple.fetch.items` (documents fetched)
|
||||
- `apple.fetch.failures`
|
||||
1. Deploy the configuration and restart the Concelier workers to ensure the Apple connector options are bound.
|
||||
2. Trigger a full connector cycle:
|
||||
- CLI: run `stella db fetch --source vndr-apple --stage fetch`, then `--stage parse`, then `--stage map`.
|
||||
- REST: `POST /jobs/run { "kind": "source:vndr-apple:fetch", "chain": ["source:vndr-apple:parse", "source:vndr-apple:map"] }`
|
||||
3. Validate metrics exported under meter `StellaOps.Concelier.Connector.Vndr.Apple`:
|
||||
- `apple.fetch.items` (documents fetched)
|
||||
- `apple.fetch.failures`
|
||||
- `apple.fetch.unchanged`
|
||||
- `apple.parse.failures`
|
||||
- `apple.map.affected.count` (histogram of affected package counts)
|
||||
@@ -42,10 +42,10 @@ concelier:
|
||||
- `Apple software index fetch … processed=X newDocuments=Y`
|
||||
- `Apple advisory parse complete … aliases=… affected=…`
|
||||
- `Mapped Apple advisory … pendingMappings=0`
|
||||
6. Confirm MongoDB state:
|
||||
- `raw_documents` store contains the HT article HTML with metadata (`apple.articleId`, `apple.postingDate`).
|
||||
- `dtos` store has `schemaVersion="apple.security.update.v1"`.
|
||||
- `advisories` collection includes keys `HTxxxxxx` with normalized SemVer rules.
|
||||
6. Confirm PostgreSQL state (schema `vuln`):
|
||||
- `raw_documents` table contains the HT article HTML with metadata (`apple.articleId`, `apple.postingDate`).
|
||||
- `dtos` table has `schemaVersion="apple.security.update.v1"`.
|
||||
- `advisories` table includes keys `HTxxxxxx` with normalized SemVer rules.
|
||||
- `source_states` entry for `apple` shows a recent `cursor.lastPosted`.
|
||||
|
||||
## 3. Production Monitoring
|
||||
|
||||
@@ -96,7 +96,7 @@ curl -s -b cookies.txt \
|
||||
|
||||
Iterate `page` until the response `content` array is empty. Pages 0–9 currently cover 2014→present. Persist JSON responses (plus SHA256) for Offline Kit parity.
|
||||
|
||||
> **Shortcut** – run `python src/Tools/certbund_offline_snapshot.py --output seed-data/cert-bund`
|
||||
> **Shortcut** – run `python src/Tools/certbund_offline_snapshot.py --output seed-data/cert-bund`
|
||||
> to bootstrap the session, capture the paginated search responses, and regenerate
|
||||
> the manifest/checksum files automatically. Supply `--cookie-file` and `--xsrf-token`
|
||||
> if the portal requires a browser-derived session (see options via `--help`).
|
||||
@@ -104,7 +104,7 @@ Iterate `page` until the response `content` array is empty. Pages 0–9 currentl
|
||||
### 3.3 Export bundles
|
||||
|
||||
```bash
|
||||
python src/Tools/certbund_offline_snapshot.py \
|
||||
python src/Tools/certbund_offline_snapshot.py \
|
||||
--output seed-data/cert-bund \
|
||||
--start-year 2014 \
|
||||
--end-year "$(date -u +%Y)"
|
||||
@@ -124,7 +124,7 @@ operating offline.
|
||||
### 3.4 Connector-driven catch-up
|
||||
|
||||
1. Temporarily raise `maxAdvisoriesPerFetch` (e.g. 150) and reduce `requestDelay`.
|
||||
2. Run `stella db fetch --source cert-bund --stage fetch`, then `--stage parse`, then `--stage map` until the fetch log reports `enqueued=0`.
|
||||
2. Run `stella db fetch --source cert-bund --stage fetch`, then `--stage parse`, then `--stage map` until the fetch log reports `enqueued=0`.
|
||||
3. Restore defaults and capture the cursor snapshot for audit.
|
||||
|
||||
---
|
||||
@@ -142,5 +142,5 @@ operating offline.
|
||||
1. Observe `certbund.feed.fetch.success` and `certbund.detail.fetch.success` increments after runs; `certbund.feed.coverage.days` should hover near the observed RSS window.
|
||||
2. Ensure summary logs report `truncated=false` in steady state—`true` indicates the fetch cap was hit.
|
||||
3. During backfills, watch `certbund.feed.enqueued.count` trend to zero.
|
||||
4. Spot-check stored advisories in Mongo to confirm `language="de"` and reference URLs match the portal detail endpoint.
|
||||
4. Spot-check stored advisories in PostgreSQL (schema `vuln`) to confirm `language="de"` and reference URLs match the portal detail endpoint.
|
||||
5. For Offline Kit exports, validate SHA256 hashes before distribution.
|
||||
|
||||
@@ -34,7 +34,7 @@ concelier:
|
||||
|
||||
1. Deploy the updated configuration and restart the Concelier service so the connector picks up the credentials.
|
||||
2. Trigger one end-to-end cycle:
|
||||
- Concelier CLI: run `stella db fetch --source cve --stage fetch`, then `--stage parse`, then `--stage map`.
|
||||
- Concelier CLI: run `stella db fetch --source cve --stage fetch`, then `--stage parse`, then `--stage map`.
|
||||
- REST fallback: `POST /jobs/run { "kind": "source:cve:fetch", "chain": ["source:cve:parse", "source:cve:map"] }`
|
||||
3. Observe the following metrics (exported via OTEL meter `StellaOps.Concelier.Connector.Cve`):
|
||||
- `cve.fetch.attempts`, `cve.fetch.success`, `cve.fetch.documents`, `cve.fetch.failures`, `cve.fetch.unchanged`
|
||||
@@ -42,7 +42,7 @@ concelier:
|
||||
- `cve.map.success`
|
||||
4. Verify Prometheus shows matching `concelier.source.http.requests_total{concelier_source="cve"}` deltas (list vs detail phases) while `concelier.source.http.failures_total{concelier_source="cve"}` stays flat.
|
||||
5. Confirm the info-level summary log `CVEs fetch window … pages=X detailDocuments=Y detailFailures=Z` appears once per fetch run and shows `detailFailures=0`.
|
||||
6. Verify the MongoDB advisory store contains fresh CVE advisories (`advisoryKey` prefix `cve/`) and that the source cursor (`source_states` collection) advanced.
|
||||
6. Verify the PostgreSQL advisory store (schema `vuln`) contains fresh CVE advisories (`advisoryKey` prefix `cve/`) and that the source cursor (`source_states` table) advanced.
|
||||
|
||||
### 1.3 Production Monitoring
|
||||
|
||||
@@ -107,7 +107,7 @@ Treat repeated schema failures or growing anomaly counts as an upstream regressi
|
||||
|
||||
1. Deploy the configuration and restart Concelier.
|
||||
2. Trigger a pipeline run:
|
||||
- CLI: run `stella db fetch --source kev --stage fetch`, then `--stage parse`, then `--stage map`.
|
||||
- CLI: run `stella db fetch --source kev --stage fetch`, then `--stage parse`, then `--stage map`.
|
||||
- REST: `POST /jobs/run { "kind": "source:kev:fetch", "chain": ["source:kev:parse", "source:kev:map"] }`
|
||||
3. Verify the metrics exposed by meter `StellaOps.Concelier.Connector.Kev`:
|
||||
- `kev.fetch.attempts`, `kev.fetch.success`, `kev.fetch.unchanged`, `kev.fetch.failures`
|
||||
@@ -115,7 +115,7 @@ Treat repeated schema failures or growing anomaly counts as an upstream regressi
|
||||
- `kev.map.advisories` (tag `catalogVersion`)
|
||||
4. Confirm `concelier.source.http.requests_total{concelier_source="kev"}` increments once per fetch and that the paired `concelier.source.http.failures_total` stays flat (zero increase).
|
||||
5. Inspect the info logs `Fetched KEV catalog document … pendingDocuments=…` and `Parsed KEV catalog document … entries=…`—they should appear exactly once per run and `Mapped X/Y… skipped=0` should match the `kev.map.advisories` delta.
|
||||
6. Confirm MongoDB documents exist for the catalog JSON (`raw_documents` & `dtos`) and that advisories with prefix `kev/` are written.
|
||||
6. Confirm PostgreSQL records exist for the catalog JSON (`raw_documents` and `dtos` tables in schema `vuln`) and that advisories with prefix `kev/` are written.
|
||||
|
||||
### 2.4 Production Monitoring
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@ concelier:
|
||||
|
||||
1. Restart the Concelier workers so the KISA options bind.
|
||||
2. Run a full connector cycle:
|
||||
- CLI: run `stella db fetch --source kisa --stage fetch`, then `--stage parse`, then `--stage map`.
|
||||
- CLI: run `stella db fetch --source kisa --stage fetch`, then `--stage parse`, then `--stage map`.
|
||||
- REST: `POST /jobs/run { "kind": "source:kisa:fetch", "chain": ["source:kisa:parse", "source:kisa:map"] }`
|
||||
3. Confirm telemetry (Meter `StellaOps.Concelier.Connector.Kisa`):
|
||||
- `kisa.feed.success`, `kisa.feed.items`
|
||||
@@ -38,10 +38,10 @@ concelier:
|
||||
- `KISA fetched detail for {Idx} … category={Category}`
|
||||
- `KISA mapped advisory {AdvisoryId} (severity={Severity})`
|
||||
- Absence of warnings such as `document missing GridFS payload`.
|
||||
5. Validate MongoDB state:
|
||||
- `raw_documents.metadata` has `kisa.idx`, `kisa.category`, `kisa.title`.
|
||||
- DTO store contains `schemaVersion="kisa.detail.v1"`.
|
||||
- Advisories include aliases (`IDX`, CVE) and `language="ko"`.
|
||||
5. Validate PostgreSQL state (schema `vuln`):
|
||||
- `raw_documents` table metadata has `kisa.idx`, `kisa.category`, `kisa.title`.
|
||||
- `dtos` table contains `schemaVersion="kisa.detail.v1"`.
|
||||
- `advisories` table includes aliases (`IDX`, CVE) and `language="ko"`.
|
||||
- `source_states` entry for `kisa` shows recent `cursor.lastFetchAt`.
|
||||
|
||||
## 3. Production Monitoring
|
||||
|
||||
@@ -41,7 +41,7 @@ concelier:
|
||||
|
||||
### 4.1 State seeding helper
|
||||
|
||||
Use `src/Tools/SourceStateSeeder` to queue historical advisories (detail JSON + optional CVRF artefacts) for replay without manual Mongo edits. Example seed file:
|
||||
Use `src/Tools/SourceStateSeeder` to queue historical advisories (detail JSON + optional CVRF artefacts) for replay without manual Mongo edits. Example seed file:
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -71,9 +71,9 @@ Use `src/Tools/SourceStateSeeder` to queue historical advisories (detail JSON +
|
||||
Run the helper:
|
||||
|
||||
```bash
|
||||
dotnet run --project src/Tools/SourceStateSeeder -- \
|
||||
--connection-string "mongodb://localhost:27017" \
|
||||
--database concelier \
|
||||
dotnet run --project src/Tools/SourceStateSeeder -- \
|
||||
--connection-string "Host=localhost;Database=stellaops;Username=stella;Password=..." \
|
||||
--schema vuln \
|
||||
--input seeds/msrc-backfill.json
|
||||
```
|
||||
|
||||
|
||||
@@ -1,19 +1,19 @@
|
||||
# Observation Event Transport (advisory.observation.updated@1)
|
||||
|
||||
Purpose: document how to emit `advisory.observation.updated@1` events via Mongo outbox with optional NATS JetStream transport.
|
||||
Purpose: document how to emit `advisory.observation.updated@1` events via PostgreSQL outbox with optional NATS JetStream transport.
|
||||
|
||||
## Configuration (appsettings.yaml / config)
|
||||
```yaml
|
||||
advisoryObservationEvents:
|
||||
enabled: false # set true to publish beyond Mongo outbox
|
||||
transport: "mongo" # "mongo" (no-op publisher) or "nats"
|
||||
enabled: false # set true to publish beyond PostgreSQL outbox
|
||||
transport: "postgres" # "postgres" (no-op publisher) or "nats"
|
||||
natsUrl: "nats://127.0.0.1:4222"
|
||||
subject: "concelier.advisory.observation.updated.v1"
|
||||
deadLetterSubject: "concelier.advisory.observation.updated.dead.v1"
|
||||
stream: "CONCELIER_OBS"
|
||||
```
|
||||
|
||||
Defaults: disabled, transport `mongo`; subject/stream as above.
|
||||
Defaults: disabled, transport `postgres`; subject/stream as above.
|
||||
|
||||
## Flow
|
||||
1) Observation sink writes event to `advisory_observation_events` (idempotent on `observationHash`).
|
||||
@@ -24,7 +24,7 @@ Defaults: disabled, transport `mongo`; subject/stream as above.
|
||||
- Ensure NATS JetStream is reachable before enabling `transport: nats` to avoid retry noise.
|
||||
- Stream is auto-created if missing with current subject; size capped at 512 KiB per message.
|
||||
- Dead-letter subject reserved; not yet wired—keep for future schema validation failures.
|
||||
- Backlog monitoring: count documents in `advisory_observation_events` with `publishedAt: null`.
|
||||
- Backlog monitoring: count rows in `advisory_observation_events` table with `published_at IS NULL`.
|
||||
|
||||
## Testing
|
||||
- Without NATS: leave `enabled=false`; app continues writing outbox only.
|
||||
|
||||
@@ -1,489 +1,489 @@
|
||||
# component_architecture_devops.md — **Stella Ops Release & Operations** (2025Q4)
|
||||
|
||||
> Draws from the AOC guardrails, Orchestrator, Export Center, and Observability module plans to describe how Stella Ops is built, signed, distributed, and operated.
|
||||
|
||||
> **Scope.** Implementation‑ready blueprint for **how Stella Ops is built, versioned, signed, distributed, upgraded, licensed (PoE)**, and operated in customer environments (online and air‑gapped). Covers reproducible builds, supply‑chain attestations, registries, offline kits, migration/rollback, artifact lifecycle (RustFS default + Mongo, S3 fallback), monitoring SLOs, and customer activation.
|
||||
|
||||
---
|
||||
|
||||
## 0) Product vision (operations lens)
|
||||
|
||||
Stella Ops must be **trustable at a glance** and **boringly operable**:
|
||||
|
||||
* Every release ships with **first‑party SBOMs, provenance, and signatures**; services verify **each other’s** integrity at runtime.
|
||||
* Customers can deploy by **digest** and stay aligned with **LTS/stable/edge** channels.
|
||||
* Paid customers receive **attestation authority** (Signer accepts their PoE) while the core platform remains **free to run**.
|
||||
* Air‑gapped customers receive **offline kits** with verifiable digests and deterministic import.
|
||||
* Artifacts expire predictably; operators know what’s kept, for how long, and why.
|
||||
|
||||
---
|
||||
|
||||
## 1) Release trains & versioning
|
||||
|
||||
### 1.1 Channels
|
||||
|
||||
* **LTS** (12‑month support window): quarterly cadence (Q1/Q2/Q3/Q4).
|
||||
* **Stable** (default): monthly rollup (bug fixes + compatible features).
|
||||
* **Edge**: weekly; for early adopters, no guarantees.
|
||||
|
||||
### 1.2 Version strings
|
||||
|
||||
Semantic core + calendar tag:
|
||||
|
||||
```
|
||||
<MAJOR>.<MINOR>.<PATCH> (<YYYY>.<MM>) e.g., 2.4.1 (2027.06)
|
||||
```
|
||||
|
||||
* **MAJOR**: breaking API/DB changes (rare).
|
||||
* **MINOR**: new features, compatible schema migrations (expand/contract pattern).
|
||||
* **PATCH**: bug fixes, perf and security updates.
|
||||
* **Calendar tag** exposes **release year** used by Signer for **PoE window checks**.
|
||||
|
||||
### 1.3 Component alignment
|
||||
|
||||
A release is a **bundle** of image digests + charts + manifests. All services in a bundle are **wire‑compatible**. Mixed minor versions are allowed within a bounded skew:
|
||||
|
||||
* **Web UI ↔ backend**: `±1 minor`.
|
||||
* **Scanner ↔ Policy/Excititor/Concelier**: `±1 minor`.
|
||||
* **Authority/Signer/Attestor triangle**: **must** be same minor (crypto and DPoP/mTLS binding rules).
|
||||
|
||||
At startup, services **self‑advertise** their semver & channel; the UI surfaces **mismatch warnings**.
|
||||
|
||||
---
|
||||
|
||||
## 2) Supply‑chain pipeline (how a release is built)
|
||||
|
||||
### 2.1 Deterministic builds
|
||||
|
||||
* **Builders**: isolated **BuildKit** workers with pinned base images (digest only).
|
||||
* **Pinning**: lock files or `go.mod`, `package-lock.json`, `global.json`, `Directory.Packages.props` are **frozen** at tag.
|
||||
* **Reproducibility**: timestamps normalized; source date epoch; deterministic zips/tars.
|
||||
* **Multi‑arch**: linux/amd64 + linux/arm64 (Windows images track M2 roadmap).
|
||||
|
||||
### 2.2 First‑party SBOMs & provenance
|
||||
|
||||
* Each image gets **CycloneDX (JSON+Protobuf) SBOM** and **SLSA‑style provenance** attached as **OCI referrers**.
|
||||
* Scanner’s **Buildx generator** is used to produce SBOMs *during* build; a separate post‑build scan verifies parity (red flag if drift).
|
||||
* **Release manifest** (see §6.1) lists all digests and SBOM/attestation refs.
|
||||
|
||||
### 2.3 Signing & transparency
|
||||
|
||||
* Images are **cosign‑signed** (keyless) with a Stella Ops release identity; inclusion in a **transparency log** (Rekor) is required.
|
||||
* SBOM and provenance attestations are **DSSE** and also transparency‑logged.
|
||||
* Release keys (Fulcio roots or public keys) are embedded in **Signer** policy (for **scanner‑release validation** at customer side).
|
||||
|
||||
### 2.4 Gates & tests
|
||||
|
||||
* **Static**: linters, codegen checks, protobuf API freeze (backward‑compat tests).
|
||||
# component_architecture_devops.md — **Stella Ops Release & Operations** (2025Q4)
|
||||
|
||||
> Draws from the AOC guardrails, Orchestrator, Export Center, and Observability module plans to describe how Stella Ops is built, signed, distributed, and operated.
|
||||
|
||||
> **Scope.** Implementation‑ready blueprint for **how Stella Ops is built, versioned, signed, distributed, upgraded, licensed (PoE)**, and operated in customer environments (online and air‑gapped). Covers reproducible builds, supply‑chain attestations, registries, offline kits, migration/rollback, artifact lifecycle (RustFS default + PostgreSQL, S3 fallback), monitoring SLOs, and customer activation.
|
||||
|
||||
---
|
||||
|
||||
## 0) Product vision (operations lens)
|
||||
|
||||
Stella Ops must be **trustable at a glance** and **boringly operable**:
|
||||
|
||||
* Every release ships with **first‑party SBOMs, provenance, and signatures**; services verify **each other’s** integrity at runtime.
|
||||
* Customers can deploy by **digest** and stay aligned with **LTS/stable/edge** channels.
|
||||
* Paid customers receive **attestation authority** (Signer accepts their PoE) while the core platform remains **free to run**.
|
||||
* Air‑gapped customers receive **offline kits** with verifiable digests and deterministic import.
|
||||
* Artifacts expire predictably; operators know what’s kept, for how long, and why.
|
||||
|
||||
---
|
||||
|
||||
## 1) Release trains & versioning
|
||||
|
||||
### 1.1 Channels
|
||||
|
||||
* **LTS** (12‑month support window): quarterly cadence (Q1/Q2/Q3/Q4).
|
||||
* **Stable** (default): monthly rollup (bug fixes + compatible features).
|
||||
* **Edge**: weekly; for early adopters, no guarantees.
|
||||
|
||||
### 1.2 Version strings
|
||||
|
||||
Semantic core + calendar tag:
|
||||
|
||||
```
|
||||
<MAJOR>.<MINOR>.<PATCH> (<YYYY>.<MM>) e.g., 2.4.1 (2027.06)
|
||||
```
|
||||
|
||||
* **MAJOR**: breaking API/DB changes (rare).
|
||||
* **MINOR**: new features, compatible schema migrations (expand/contract pattern).
|
||||
* **PATCH**: bug fixes, perf and security updates.
|
||||
* **Calendar tag** exposes **release year** used by Signer for **PoE window checks**.
|
||||
|
||||
### 1.3 Component alignment
|
||||
|
||||
A release is a **bundle** of image digests + charts + manifests. All services in a bundle are **wire‑compatible**. Mixed minor versions are allowed within a bounded skew:
|
||||
|
||||
* **Web UI ↔ backend**: `±1 minor`.
|
||||
* **Scanner ↔ Policy/Excititor/Concelier**: `±1 minor`.
|
||||
* **Authority/Signer/Attestor triangle**: **must** be same minor (crypto and DPoP/mTLS binding rules).
|
||||
|
||||
At startup, services **self‑advertise** their semver & channel; the UI surfaces **mismatch warnings**.
|
||||
|
||||
---
|
||||
|
||||
## 2) Supply‑chain pipeline (how a release is built)
|
||||
|
||||
### 2.1 Deterministic builds
|
||||
|
||||
* **Builders**: isolated **BuildKit** workers with pinned base images (digest only).
|
||||
* **Pinning**: lock files or `go.mod`, `package-lock.json`, `global.json`, `Directory.Packages.props` are **frozen** at tag.
|
||||
* **Reproducibility**: timestamps normalized; source date epoch; deterministic zips/tars.
|
||||
* **Multi‑arch**: linux/amd64 + linux/arm64 (Windows images track M2 roadmap).
|
||||
|
||||
### 2.2 First‑party SBOMs & provenance
|
||||
|
||||
* Each image gets **CycloneDX (JSON+Protobuf) SBOM** and **SLSA‑style provenance** attached as **OCI referrers**.
|
||||
* Scanner’s **Buildx generator** is used to produce SBOMs *during* build; a separate post‑build scan verifies parity (red flag if drift).
|
||||
* **Release manifest** (see §6.1) lists all digests and SBOM/attestation refs.
|
||||
|
||||
### 2.3 Signing & transparency
|
||||
|
||||
* Images are **cosign‑signed** (keyless) with a Stella Ops release identity; inclusion in a **transparency log** (Rekor) is required.
|
||||
* SBOM and provenance attestations are **DSSE** and also transparency‑logged.
|
||||
* Release keys (Fulcio roots or public keys) are embedded in **Signer** policy (for **scanner‑release validation** at customer side).
|
||||
|
||||
### 2.4 Gates & tests
|
||||
|
||||
* **Static**: linters, codegen checks, protobuf API freeze (backward‑compat tests).
|
||||
* **Unit/integration**: per-component, plus **end-to-end** flows (scan→vex→policy→sign→attest).
|
||||
* **Perf SLOs**: hot paths (SBOM compose, diff, export) measured against budgets.
|
||||
* **Security**: dependency audit vs Concelier export; container hardening tests; minimal caps.
|
||||
* **Deployment assets**: `Build Test Deploy` workflow’s `profile-validation` job installs Helm and runs `helm lint` + `helm template` against `deploy/helm/stellaops` for every `values*.yaml`, catching ConfigMap/templating drift before merges.
|
||||
* **Analyzer smoke**: restart-time language plug-ins (currently Python) verified via `dotnet run --project src/Tools/LanguageAnalyzerSmoke` to ensure manifest integrity plus cold vs warm determinism (< 30 s / < 5 s budgets); the harness logs deviations from repository goldens for follow-up.
|
||||
* **Canary cohort**: internal staging + selected customers; one week on **edge** before **stable** tag.
|
||||
|
||||
### 2.5 Debug-store artefacts
|
||||
|
||||
* Every release exports stripped debug information for ELF binaries discovered in service images. Debug files follow the GNU build-id layout (`debug/.build-id/<aa>/<rest>.debug`) and are generated via `objcopy --only-keep-debug`.
|
||||
* `debug/debug-manifest.json` captures build-id → component/image/source mappings with SHA-256 checksums so operators can mirror the directory into debuginfod or offline symbol stores. The manifest (and its `.sha256` companion) ships with every release bundle and Offline Kit.
|
||||
|
||||
---
|
||||
|
||||
## 3) Distribution & activation
|
||||
|
||||
### 3.1 Registries
|
||||
|
||||
* **Primary**: `registry.stella-ops.org` (OCI v2, supports Referrers API).
|
||||
* **Mirrors**: GHCR (read‑only), regional mirrors for latency.
|
||||
* Operational runbook: see `docs/modules/concelier/operations/mirror.md` for deployment profiles, CDN guidance, and sync automation.
|
||||
* **Pull by digest only** in Kubernetes/Compose manifests.
|
||||
|
||||
**Gating policy**:
|
||||
|
||||
* **Core images** (Authority, Scanner, Concelier, Excititor, Attestor, UI): public **read**.
|
||||
* **Enterprise add‑ons** (if any) and **pre‑release**: private repos via the **Registry Token Service** (`src/Registry/StellaOps.Registry.TokenService`) which exchanges Authority-issued OpToks for short-lived Docker registry bearer tokens.
|
||||
|
||||
> Monetization lever is **signing** (PoE gate), not image pulls, so the core remains simple to consume.
|
||||
|
||||
### 3.2 OAuth2 token service (for private repos)
|
||||
|
||||
* Docker Registry’s token flow backed by **Authority**:
|
||||
|
||||
1. Client hits registry (`401` with `WWW-Authenticate: Bearer realm=…`).
|
||||
2. Client gets an **access token** from the token service (validated by Authority) with `scope=repository:…:pull`.
|
||||
3. Registry allows pull for the requested repo.
|
||||
* Tokens are **short‑lived** (60–300 s) and **DPoP‑bound**.
|
||||
|
||||
The token service enforces plan gating via `registry-token.yaml` (see `docs/modules/registry/operations/token-service.md`) and exposes Prometheus metrics (`registry_token_issued_total`, `registry_token_rejected_total`). Revoked licence identifiers halt issuance even when scope requirements are met.
|
||||
|
||||
### 3.3 Offline kits (air‑gapped)
|
||||
|
||||
* Tarball per release channel:
|
||||
|
||||
```
|
||||
stellaops-kit-<ver>-<channel>.tar.zst
|
||||
/images/ OCI layout with all first-party images (multi-arch)
|
||||
/sboms/ CycloneDX JSON+PB for each image
|
||||
/attest/ DSSE bundles + Rekor proofs
|
||||
/charts/ Helm charts + values templates
|
||||
/compose/ docker-compose.yml + .env template
|
||||
/plugins/ Concelier/Excititor connectors (restart-time)
|
||||
/policy/ example policies
|
||||
/manifest/ release.yaml (see §6.1)
|
||||
```
|
||||
* Import via CLI `offline kit import`; checks digests and signatures before load.
|
||||
|
||||
---
|
||||
|
||||
## 4) Licensing (PoE) & monetization
|
||||
|
||||
**Principle**: **Only paid Stella Ops issues valid signed attestations.** Running the stack is free; signing requires PoE.
|
||||
|
||||
### 4.1 PoE issuance
|
||||
|
||||
* Customers purchase a plan and obtain a **PoE artifact** from `www.stella-ops.org`:
|
||||
|
||||
* **PoE‑JWT** (DPoP/mTLS‑bound) **or** **PoE mTLS client certificate**.
|
||||
* Contains: `license_id`, `plan`, `valid_release_year`, `max_version`, `exp`, optional `tenant/customer` IDs.
|
||||
|
||||
### 4.2 Online enforcement
|
||||
|
||||
* **Signer** calls **Licensing /license/introspect** on every signing request (see signer doc).
|
||||
* If **revoked/expired/out‑of‑window** → deny with machine‑readable reason.
|
||||
* All **valid** bundles are DSSE‑signed and **Attestor** logs them; Rekor UUID returned.
|
||||
* UI badges: “**Verified by Stella Ops**” with link to the public log.
|
||||
|
||||
### 4.3 Air‑gapped / offline
|
||||
|
||||
* Customers obtain a **time‑boxed PoE lease** (signed JSON, 7–30 days).
|
||||
* Signer accepts the lease and emits **provisional** attestations (clearly labeled).
|
||||
* When connectivity returns, a background job **endorses** the provisional entries with the cloud service, updating their status to **verified**.
|
||||
* Operators can export a **verification bundle** for auditors even before endorsement (contains DSSE + local Rekor proof + lease snapshot).
|
||||
|
||||
### 4.4 Stolen/abused PoE
|
||||
|
||||
* Customers report theft; **Licensing** flags `license_id` as **revoked**.
|
||||
* Subsequent Signer requests **deny**; previous attestations remain but can be marked **contested** (UI shows badge, optional re‑sign path upon new PoE).
|
||||
|
||||
---
|
||||
|
||||
## 5) Deployment path (customer side)
|
||||
|
||||
### 5.1 First install
|
||||
|
||||
* **Helm** (Kubernetes) or **Compose** (VMs). Example (K8s):
|
||||
|
||||
```bash
|
||||
helm repo add stellaops https://charts.stella-ops.org
|
||||
helm install stella stellaops/platform \
|
||||
--version 2.4.0 \
|
||||
--set global.channel=stable \
|
||||
--set authority.issuer=https://authority.stella.local \
|
||||
--set scanner.minio.endpoint=http://minio.stella.local:9000 \
|
||||
--set scanner.mongo.uri=mongodb://mongo/scanner \
|
||||
--set concelier.mongo.uri=mongodb://mongo/concelier \
|
||||
--set excititor.mongo.uri=mongodb://mongo/excititor
|
||||
```
|
||||
|
||||
* Post‑install job registers **Authority clients** (Scanner, Signer, Attestor, UI) and prints **bootstrap** URLs and client credentials (sealed secrets).
|
||||
* UI banner shows **release bundle** and verification state (cosign OK? Rekor OK?).
|
||||
|
||||
### 5.2 Updates
|
||||
|
||||
* **Blue/green**: pull new bundle by **digest**; deploy side‑by‑side; cut traffic.
|
||||
|
||||
* **Rolling**: upgrade stateful components in safe order:
|
||||
|
||||
1. Authority (stateless, dual‑key rotation ready)
|
||||
2. Signer/Attestor (same minor)
|
||||
3. Scanner WebService & Workers
|
||||
4. Concelier, then Excititor (schema migrations are expand/contract)
|
||||
5. UI last
|
||||
|
||||
* **DB migrations** are **expand/contract**:
|
||||
|
||||
* Phase A (release N): **add** new fields/indexes, write old+new.
|
||||
* Phase B (N+1): **read** new fields; **drop** old.
|
||||
* Rollback is a matter of redeploying previous images and keeping both schemas valid.
|
||||
|
||||
### 5.3 Rollback
|
||||
|
||||
* Images referenced by **digest**; keep previous release manifest `K` versions back.
|
||||
* `helm rollback` or compose `docker compose -f release-K.yml up -d`.
|
||||
* Mongo migrations are additive; **no destructive changes** within a single minor.
|
||||
|
||||
---
|
||||
|
||||
## 6) Release payloads & manifests
|
||||
|
||||
### 6.1 Release manifest (`release.yaml`)
|
||||
|
||||
```yaml
|
||||
release:
|
||||
version: "2.4.1"
|
||||
channel: "stable"
|
||||
date: "2027-06-20T12:00:00Z"
|
||||
calendar: "2027.06"
|
||||
components:
|
||||
- name: scanner-webservice
|
||||
image: registry.stella-ops.org/stellaops/scanner-web@sha256:aa..bb
|
||||
sbom: oci://.../referrers/cdx-json@sha256:11..22
|
||||
provenance: oci://.../attest/provenance@sha256:33..44
|
||||
signature: { rekorUUID: "…" }
|
||||
- name: signer
|
||||
image: registry.stella-ops.org/stellaops/signer@sha256:cc..dd
|
||||
signature: { rekorUUID: "…" }
|
||||
charts:
|
||||
- name: platform
|
||||
version: "2.4.1"
|
||||
digest: "sha256:ee..ff"
|
||||
compose:
|
||||
file: "docker-compose.yml"
|
||||
digest: "sha256:77..88"
|
||||
checksums:
|
||||
sha256: "… digest of this release.yaml …"
|
||||
```
|
||||
|
||||
The manifest is **cosign‑signed**; UI/CLI can verify a bundle without talking to registries.
|
||||
|
||||
> Deployment guardrails – The repository keeps channel-aligned Compose bundles
|
||||
> in `deploy/compose/` and Helm overlays in `deploy/helm/stellaops/`. Both sets
|
||||
> pull their digests from `deploy/releases/` and are validated by
|
||||
> `deploy/tools/validate-profiles.sh` to guarantee lint/dry-run cleanliness.
|
||||
|
||||
### 6.2 Image labels (release metadata)
|
||||
|
||||
Each image sets OCI labels:
|
||||
|
||||
```
|
||||
org.opencontainers.image.version = "2.4.1"
|
||||
org.opencontainers.image.revision = "<git sha>"
|
||||
org.opencontainers.image.created = "2027-06-20T12:00:00Z"
|
||||
org.stellaops.release.calendar = "2027.06"
|
||||
org.stellaops.release.channel = "stable"
|
||||
org.stellaops.build.slsaProvenance = "oci://…"
|
||||
```
|
||||
|
||||
Signer validates **scanner** image’s cosign identity + calendar tag for **release window** checks.
|
||||
|
||||
---
|
||||
|
||||
## 7) Artifact lifecycle & storage (RustFS/Mongo)
|
||||
|
||||
### 7.1 Buckets & prefixes (RustFS)
|
||||
|
||||
```
|
||||
rustfs://stellaops/
|
||||
scanner/
|
||||
layers/<sha256>/sbom.cdx.json.zst
|
||||
images/<imgDigest>/inventory.cdx.pb
|
||||
images/<imgDigest>/usage.cdx.pb
|
||||
diffs/<old>_<new>/diff.json.zst
|
||||
attest/<artifactSha256>.dsse.json
|
||||
concelier/
|
||||
json/<exportId>/...
|
||||
trivy/<exportId>/...
|
||||
excititor/
|
||||
exports/<exportId>/...
|
||||
attestor/
|
||||
dsse/<bundleSha256>.json
|
||||
proof/<rekorUuid>.json
|
||||
```
|
||||
|
||||
### 7.2 ILM classes
|
||||
|
||||
* **`short`**: working artifacts (diffs, queues) — TTL 7–14 days.
|
||||
* **`default`**: SBOMs & indexes — TTL 90–180 days (configurable).
|
||||
* **`compliance`**: signed reports & attested exports — retention enforced via RustFS hold or S3 Object Lock (governance/compliance) 1–7 years.
|
||||
|
||||
### 7.3 Artifact Lifecycle Controller (ALC)
|
||||
|
||||
* A background worker (part of Scanner.WebService) enforces **TTL** and **reference counting**:
|
||||
|
||||
* Artifacts referenced by **reports** or **tickets** are pinned.
|
||||
* ILM actions logged; UI shows per‑class usage & upcoming purges.
|
||||
|
||||
> **Migration note.** Follow `docs/modules/scanner/operations/rustfs-migration.md` when transitioning existing
|
||||
> MinIO buckets to RustFS. The provided migrator is idempotent and safe to rerun per prefix.
|
||||
|
||||
### 7.4 Mongo retention
|
||||
|
||||
* **Scanner**: `runtime.events` use TTL (e.g., 30–90 days); **catalog** permanent.
|
||||
* **Concelier/Excititor**: raw docs keep **last N windows**; canonical stores permanent.
|
||||
* **Attestor**: `entries` permanent; `dedupe` TTL 24–48h.
|
||||
|
||||
### 7.5 Mongo server baseline
|
||||
|
||||
* **Minimum supported server:** MongoDB **4.2+**. Driver 3.5.0 removes compatibility shims for 4.0; upstream has already announced 4.0 support will be dropped in upcoming C# driver releases. citeturn1open1
|
||||
* **Deploy images:** Compose/Helm defaults stay on `mongo:7.x`. For air-gapped installs, refresh Offline Kit bundles so the packaged `mongod` matches ≥4.2.
|
||||
* **Upgrade guard:** During rollout, verify replica sets reach FCV `4.2` or above before swapping binaries; automation should hard-stop if FCV is <4.2.
|
||||
|
||||
---
|
||||
|
||||
## 8) Observability & SLOs (operations)
|
||||
|
||||
* **Uptime SLO**: 99.9% for Signer/Authority/Attestor; 99.5% for Scanner WebService; Excititor/Concelier 99.0%.
|
||||
* **Error budgets**: tracked per month; dashboards show burn rates.
|
||||
* **Golden signals**:
|
||||
|
||||
* **Latency**: token issuance, sign→attest round‑trip, scan enqueue→emit, export build.
|
||||
* **Saturation**: queue depth, Mongo write IOPS, RustFS throughput / queue depth (or S3 metrics when in fallback mode).
|
||||
* **Traffic**: scans/min, attestations/min, webhook admits/min.
|
||||
* **Errors**: 5xx rates, cosign verification failures, Rekor timeouts.
|
||||
|
||||
Prometheus + OTLP; Grafana dashboards ship in the charts.
|
||||
|
||||
---
|
||||
|
||||
## 9) Security & compliance operations
|
||||
|
||||
* **Key rotation**:
|
||||
|
||||
* Authority JWKS: 60‑day cadence, dual‑key overlap.
|
||||
* Release signing identities: rotate per minor or quarterly.
|
||||
* Sigstore roots mirrored and pinned; alarms on drift.
|
||||
|
||||
* **FIPS mode** (Gov build):
|
||||
|
||||
* Enforce `ES256` + KMS/HSM; disable Ed25519; MLS ciphers only.
|
||||
* Local **Rekor v2** and **Fulcio** alternatives; **air‑gapped** CA.
|
||||
|
||||
* **Vulnerability response**:
|
||||
|
||||
* Concelier red-flag advisories trigger accelerated **stable** patch rollout; UI/CLI “security patch available” notice.
|
||||
* 2025-10: Pinned `MongoDB.Driver` **3.5.0** and `SharpCompress` **0.41.0** across services (DEVOPS-SEC-10-301) to eliminate NU1902/NU1903 warnings surfaced during scanner cache/worker test runs; repacked the local `Mongo2Go` feed so test fixtures inherit the patched dependencies; future bumps follow the same central override pattern.
|
||||
|
||||
* **Backups/DR**:
|
||||
|
||||
* Mongo nightly snapshots; MinIO versioning + replication (if configured).
|
||||
* Restore runbooks tested quarterly with synthetic data.
|
||||
|
||||
---
|
||||
|
||||
## 10) Customer update flow (how versions are fetched & activated)
|
||||
|
||||
### 10.1 Online clusters
|
||||
|
||||
* **UI** surfaces update banner with **release manifest** diff and risk notes.
|
||||
* Operator approves → **Controller** pulls new images by digest; health‑checks; moves traffic; deprecates old revision.
|
||||
* Post‑switch, **schema Phase B** migrations (if any) run automatically.
|
||||
|
||||
### 10.2 Air‑gapped clusters
|
||||
|
||||
* Operator downloads **offline kit** from a mirror → `stellaops offline kit import`.
|
||||
* Controller validates bundle checksums and **cosign signatures**; applies charts/compose by digest.
|
||||
* After install, **verify** page shows green checks: image sigs, SBOMs attached, provenance logged.
|
||||
|
||||
### 10.3 CLI self‑update (optional)
|
||||
|
||||
* `stellaops self-update` pulls a **signed release manifest** and verifies the **CLI binary** with cosign before swapping (admin can disable).
|
||||
|
||||
---
|
||||
|
||||
## 11) Compatibility & deprecation policy
|
||||
|
||||
* **APIs** are stable within a **major**; breaking changes imply **MAJOR++** and deprecation period of one minor.
|
||||
* **Storage**: expand/contract; “drop old fields” only after one minor grace.
|
||||
* **Config**: feature flags (default off) for risky features (e.g., eBPF).
|
||||
|
||||
---
|
||||
|
||||
## 12) Runbooks (selected)
|
||||
|
||||
### 12.1 Lost PoE
|
||||
|
||||
1. Suspend **automatic attestation** jobs.
|
||||
2. Use CLI `stellaops signer status` to confirm `entitlement_denied`.
|
||||
3. Obtain new PoE from portal; verify on Signer `/poe/verify`.
|
||||
4. Re‑enable; optionally **re‑sign** last N reports (UI button → batch).
|
||||
|
||||
### 12.2 Rekor outage (self‑hosted)
|
||||
|
||||
* Attestor returns `202 (pending)` with queued proof fetch.
|
||||
* Keep DSSE bundles locally; re‑submit on schedule; UI badge shows **Pending**.
|
||||
* If outage > SLA, you can switch to a **mirror** log in config; Attestor writes to both when restored.
|
||||
|
||||
### 12.3 Emergency downgrade
|
||||
|
||||
* Identify prior release manifest (UI → Admin → Releases).
|
||||
* `helm rollback stella <revision>` (or compose apply previous file).
|
||||
* Services tolerate skew per §1.3; ensure **Signer/Authority/Attestor** are rolled together.
|
||||
|
||||
---
|
||||
|
||||
## 13) Example: cluster bootstrap (Compose)
|
||||
|
||||
```yaml
|
||||
version: "3.9"
|
||||
services:
|
||||
authority:
|
||||
image: registry.stella-ops.org/stellaops/authority@sha256:...
|
||||
env_file: ./env/authority.env
|
||||
ports: ["8440:8440"]
|
||||
signer:
|
||||
image: registry.stella-ops.org/stellaops/signer@sha256:...
|
||||
depends_on: [authority]
|
||||
environment:
|
||||
- SIGNER__POE__LICENSING__INTROSPECTURL=https://www.stella-ops.org/api/v1/license/introspect
|
||||
attestor:
|
||||
image: registry.stella-ops.org/stellaops/attestor@sha256:...
|
||||
depends_on: [signer]
|
||||
scanner-web:
|
||||
image: registry.stella-ops.org/stellaops/scanner-web@sha256:...
|
||||
environment:
|
||||
- SCANNER__S3__ENDPOINT=http://minio:9000
|
||||
scanner-worker:
|
||||
image: registry.stella-ops.org/stellaops/scanner-worker@sha256:...
|
||||
deploy: { replicas: 4 }
|
||||
concelier:
|
||||
image: registry.stella-ops.org/stellaops/concelier@sha256:...
|
||||
excititor:
|
||||
image: registry.stella-ops.org/stellaops/excititor@sha256:...
|
||||
web-ui:
|
||||
image: registry.stella-ops.org/stellaops/web-ui@sha256:...
|
||||
mongo:
|
||||
image: mongo:7
|
||||
minio:
|
||||
image: minio/minio:RELEASE.2025-07-10T00-00-00Z
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14) Governance & keys (who owns the trust root)
|
||||
|
||||
* **Release key policy**: only the Release Engineering group can push signed releases; 4‑eyes approval; TUF‑style manifest possible in future.
|
||||
* **Signer acceptance policy**: embedded release identities are updated **only** via minor upgrade; emergency CRL supported.
|
||||
* **Customer keys**: none needed for core use; enterprise add‑ons may require per‑customer registries and keys.
|
||||
|
||||
---
|
||||
|
||||
## 15) Roadmap (Ops)
|
||||
|
||||
* **Windows containers GA** (Scanner + Zastava).
|
||||
* **Key Transparency** for Signer certs.
|
||||
* **Delta‑kit** (offline) for incremental updates.
|
||||
* **Operator CRDs** (K8s) to manage policy and ILM declaratively.
|
||||
* **SBOM **protobuf** as default transport at rest (smaller, faster).
|
||||
|
||||
---
|
||||
|
||||
### Appendix A — Minimal SLO monitors
|
||||
|
||||
* `authority.tokens_issued_total` slope ≈ normal.
|
||||
* `signer.requests_total{result="success"}/minute` > 0 (when scans occur).
|
||||
* `attestor.submit_latency_seconds{quantile=0.95}` < 0.3.
|
||||
* `scanner.scan_latency_seconds{quantile=0.95}` < target per image size.
|
||||
* `concelier.export.duration_seconds` stable; `excititor.consensus.conflicts_total` not exploding after policy changes.
|
||||
* RustFS request error rate near zero (or `s3_requests_errors_total` when operating against S3); Mongo `opcounters` hit expected baseline.
|
||||
|
||||
### Appendix B — Upgrade safety checklist
|
||||
|
||||
* Verify **release manifest** signature.
|
||||
* Ensure **Signer/Authority/Attestor** are same minor.
|
||||
* Verify **DB backups** < 24h old.
|
||||
* Confirm **ILM** won’t purge compliance artifacts during upgrade window.
|
||||
* Roll **one component** at a time; watch SLOs; abort on regression.
|
||||
|
||||
---
|
||||
|
||||
**End — component_architecture_devops.md**
|
||||
* **Analyzer smoke**: restart-time language plug-ins (currently Python) verified via `dotnet run --project src/Tools/LanguageAnalyzerSmoke` to ensure manifest integrity plus cold vs warm determinism (< 30 s / < 5 s budgets); the harness logs deviations from repository goldens for follow-up.
|
||||
* **Canary cohort**: internal staging + selected customers; one week on **edge** before **stable** tag.
|
||||
|
||||
### 2.5 Debug-store artefacts
|
||||
|
||||
* Every release exports stripped debug information for ELF binaries discovered in service images. Debug files follow the GNU build-id layout (`debug/.build-id/<aa>/<rest>.debug`) and are generated via `objcopy --only-keep-debug`.
|
||||
* `debug/debug-manifest.json` captures build-id → component/image/source mappings with SHA-256 checksums so operators can mirror the directory into debuginfod or offline symbol stores. The manifest (and its `.sha256` companion) ships with every release bundle and Offline Kit.
|
||||
|
||||
---
|
||||
|
||||
## 3) Distribution & activation
|
||||
|
||||
### 3.1 Registries
|
||||
|
||||
* **Primary**: `registry.stella-ops.org` (OCI v2, supports Referrers API).
|
||||
* **Mirrors**: GHCR (read‑only), regional mirrors for latency.
|
||||
* Operational runbook: see `docs/modules/concelier/operations/mirror.md` for deployment profiles, CDN guidance, and sync automation.
|
||||
* **Pull by digest only** in Kubernetes/Compose manifests.
|
||||
|
||||
**Gating policy**:
|
||||
|
||||
* **Core images** (Authority, Scanner, Concelier, Excititor, Attestor, UI): public **read**.
|
||||
* **Enterprise add‑ons** (if any) and **pre‑release**: private repos via the **Registry Token Service** (`src/Registry/StellaOps.Registry.TokenService`) which exchanges Authority-issued OpToks for short-lived Docker registry bearer tokens.
|
||||
|
||||
> Monetization lever is **signing** (PoE gate), not image pulls, so the core remains simple to consume.
|
||||
|
||||
### 3.2 OAuth2 token service (for private repos)
|
||||
|
||||
* Docker Registry’s token flow backed by **Authority**:
|
||||
|
||||
1. Client hits registry (`401` with `WWW-Authenticate: Bearer realm=…`).
|
||||
2. Client gets an **access token** from the token service (validated by Authority) with `scope=repository:…:pull`.
|
||||
3. Registry allows pull for the requested repo.
|
||||
* Tokens are **short‑lived** (60–300 s) and **DPoP‑bound**.
|
||||
|
||||
The token service enforces plan gating via `registry-token.yaml` (see `docs/modules/registry/operations/token-service.md`) and exposes Prometheus metrics (`registry_token_issued_total`, `registry_token_rejected_total`). Revoked licence identifiers halt issuance even when scope requirements are met.
|
||||
|
||||
### 3.3 Offline kits (air‑gapped)
|
||||
|
||||
* Tarball per release channel:
|
||||
|
||||
```
|
||||
stellaops-kit-<ver>-<channel>.tar.zst
|
||||
/images/ OCI layout with all first-party images (multi-arch)
|
||||
/sboms/ CycloneDX JSON+PB for each image
|
||||
/attest/ DSSE bundles + Rekor proofs
|
||||
/charts/ Helm charts + values templates
|
||||
/compose/ docker-compose.yml + .env template
|
||||
/plugins/ Concelier/Excititor connectors (restart-time)
|
||||
/policy/ example policies
|
||||
/manifest/ release.yaml (see §6.1)
|
||||
```
|
||||
* Import via CLI `offline kit import`; checks digests and signatures before load.
|
||||
|
||||
---
|
||||
|
||||
## 4) Licensing (PoE) & monetization
|
||||
|
||||
**Principle**: **Only paid Stella Ops issues valid signed attestations.** Running the stack is free; signing requires PoE.
|
||||
|
||||
### 4.1 PoE issuance
|
||||
|
||||
* Customers purchase a plan and obtain a **PoE artifact** from `www.stella-ops.org`:
|
||||
|
||||
* **PoE‑JWT** (DPoP/mTLS‑bound) **or** **PoE mTLS client certificate**.
|
||||
* Contains: `license_id`, `plan`, `valid_release_year`, `max_version`, `exp`, optional `tenant/customer` IDs.
|
||||
|
||||
### 4.2 Online enforcement
|
||||
|
||||
* **Signer** calls **Licensing /license/introspect** on every signing request (see signer doc).
|
||||
* If **revoked/expired/out‑of‑window** → deny with machine‑readable reason.
|
||||
* All **valid** bundles are DSSE‑signed and **Attestor** logs them; Rekor UUID returned.
|
||||
* UI badges: “**Verified by Stella Ops**” with link to the public log.
|
||||
|
||||
### 4.3 Air‑gapped / offline
|
||||
|
||||
* Customers obtain a **time‑boxed PoE lease** (signed JSON, 7–30 days).
|
||||
* Signer accepts the lease and emits **provisional** attestations (clearly labeled).
|
||||
* When connectivity returns, a background job **endorses** the provisional entries with the cloud service, updating their status to **verified**.
|
||||
* Operators can export a **verification bundle** for auditors even before endorsement (contains DSSE + local Rekor proof + lease snapshot).
|
||||
|
||||
### 4.4 Stolen/abused PoE
|
||||
|
||||
* Customers report theft; **Licensing** flags `license_id` as **revoked**.
|
||||
* Subsequent Signer requests **deny**; previous attestations remain but can be marked **contested** (UI shows badge, optional re‑sign path upon new PoE).
|
||||
|
||||
---
|
||||
|
||||
## 5) Deployment path (customer side)
|
||||
|
||||
### 5.1 First install
|
||||
|
||||
* **Helm** (Kubernetes) or **Compose** (VMs). Example (K8s):
|
||||
|
||||
```bash
|
||||
helm repo add stellaops https://charts.stella-ops.org
|
||||
helm install stella stellaops/platform \
|
||||
--version 2.4.0 \
|
||||
--set global.channel=stable \
|
||||
--set authority.issuer=https://authority.stella.local \
|
||||
--set scanner.minio.endpoint=http://minio.stella.local:9000 \
|
||||
--set scanner.mongo.uri=postgresb://mongo/scanner \
|
||||
--set concelier.mongo.uri=postgresb://mongo/concelier \
|
||||
--set excititor.mongo.uri=postgresb://mongo/excititor
|
||||
```
|
||||
|
||||
* Post‑install job registers **Authority clients** (Scanner, Signer, Attestor, UI) and prints **bootstrap** URLs and client credentials (sealed secrets).
|
||||
* UI banner shows **release bundle** and verification state (cosign OK? Rekor OK?).
|
||||
|
||||
### 5.2 Updates
|
||||
|
||||
* **Blue/green**: pull new bundle by **digest**; deploy side‑by‑side; cut traffic.
|
||||
|
||||
* **Rolling**: upgrade stateful components in safe order:
|
||||
|
||||
1. Authority (stateless, dual‑key rotation ready)
|
||||
2. Signer/Attestor (same minor)
|
||||
3. Scanner WebService & Workers
|
||||
4. Concelier, then Excititor (schema migrations are expand/contract)
|
||||
5. UI last
|
||||
|
||||
* **DB migrations** are **expand/contract**:
|
||||
|
||||
* Phase A (release N): **add** new fields/indexes, write old+new.
|
||||
* Phase B (N+1): **read** new fields; **drop** old.
|
||||
* Rollback is a matter of redeploying previous images and keeping both schemas valid.
|
||||
|
||||
### 5.3 Rollback
|
||||
|
||||
* Images referenced by **digest**; keep previous release manifest `K` versions back.
|
||||
* `helm rollback` or compose `docker compose -f release-K.yml up -d`.
|
||||
* PostgreSQL migrations are additive; **no destructive changes** within a single minor.
|
||||
|
||||
---
|
||||
|
||||
## 6) Release payloads & manifests
|
||||
|
||||
### 6.1 Release manifest (`release.yaml`)
|
||||
|
||||
```yaml
|
||||
release:
|
||||
version: "2.4.1"
|
||||
channel: "stable"
|
||||
date: "2027-06-20T12:00:00Z"
|
||||
calendar: "2027.06"
|
||||
components:
|
||||
- name: scanner-webservice
|
||||
image: registry.stella-ops.org/stellaops/scanner-web@sha256:aa..bb
|
||||
sbom: oci://.../referrers/cdx-json@sha256:11..22
|
||||
provenance: oci://.../attest/provenance@sha256:33..44
|
||||
signature: { rekorUUID: "…" }
|
||||
- name: signer
|
||||
image: registry.stella-ops.org/stellaops/signer@sha256:cc..dd
|
||||
signature: { rekorUUID: "…" }
|
||||
charts:
|
||||
- name: platform
|
||||
version: "2.4.1"
|
||||
digest: "sha256:ee..ff"
|
||||
compose:
|
||||
file: "docker-compose.yml"
|
||||
digest: "sha256:77..88"
|
||||
checksums:
|
||||
sha256: "… digest of this release.yaml …"
|
||||
```
|
||||
|
||||
The manifest is **cosign‑signed**; UI/CLI can verify a bundle without talking to registries.
|
||||
|
||||
> Deployment guardrails – The repository keeps channel-aligned Compose bundles
|
||||
> in `deploy/compose/` and Helm overlays in `deploy/helm/stellaops/`. Both sets
|
||||
> pull their digests from `deploy/releases/` and are validated by
|
||||
> `deploy/tools/validate-profiles.sh` to guarantee lint/dry-run cleanliness.
|
||||
|
||||
### 6.2 Image labels (release metadata)
|
||||
|
||||
Each image sets OCI labels:
|
||||
|
||||
```
|
||||
org.opencontainers.image.version = "2.4.1"
|
||||
org.opencontainers.image.revision = "<git sha>"
|
||||
org.opencontainers.image.created = "2027-06-20T12:00:00Z"
|
||||
org.stellaops.release.calendar = "2027.06"
|
||||
org.stellaops.release.channel = "stable"
|
||||
org.stellaops.build.slsaProvenance = "oci://…"
|
||||
```
|
||||
|
||||
Signer validates **scanner** image’s cosign identity + calendar tag for **release window** checks.
|
||||
|
||||
---
|
||||
|
||||
## 7) Artifact lifecycle & storage (RustFS/PostgreSQL)
|
||||
|
||||
### 7.1 Buckets & prefixes (RustFS)
|
||||
|
||||
```
|
||||
rustfs://stellaops/
|
||||
scanner/
|
||||
layers/<sha256>/sbom.cdx.json.zst
|
||||
images/<imgDigest>/inventory.cdx.pb
|
||||
images/<imgDigest>/usage.cdx.pb
|
||||
diffs/<old>_<new>/diff.json.zst
|
||||
attest/<artifactSha256>.dsse.json
|
||||
concelier/
|
||||
json/<exportId>/...
|
||||
trivy/<exportId>/...
|
||||
excititor/
|
||||
exports/<exportId>/...
|
||||
attestor/
|
||||
dsse/<bundleSha256>.json
|
||||
proof/<rekorUuid>.json
|
||||
```
|
||||
|
||||
### 7.2 ILM classes
|
||||
|
||||
* **`short`**: working artifacts (diffs, queues) — TTL 7–14 days.
|
||||
* **`default`**: SBOMs & indexes — TTL 90–180 days (configurable).
|
||||
* **`compliance`**: signed reports & attested exports — retention enforced via RustFS hold or S3 Object Lock (governance/compliance) 1–7 years.
|
||||
|
||||
### 7.3 Artifact Lifecycle Controller (ALC)
|
||||
|
||||
* A background worker (part of Scanner.WebService) enforces **TTL** and **reference counting**:
|
||||
|
||||
* Artifacts referenced by **reports** or **tickets** are pinned.
|
||||
* ILM actions logged; UI shows per‑class usage & upcoming purges.
|
||||
|
||||
> **Migration note.** Follow `docs/modules/scanner/operations/rustfs-migration.md` when transitioning existing
|
||||
> MinIO buckets to RustFS. The provided migrator is idempotent and safe to rerun per prefix.
|
||||
|
||||
### 7.4 PostgreSQL retention
|
||||
|
||||
* **Scanner**: `runtime.events` use TTL (e.g., 30–90 days); **catalog** permanent.
|
||||
* **Concelier/Excititor**: raw docs keep **last N windows**; canonical stores permanent.
|
||||
* **Attestor**: `entries` permanent; `dedupe` TTL 24–48h.
|
||||
|
||||
### 7.5 PostgreSQL server baseline
|
||||
|
||||
* **Minimum supported server:** MongoDB **4.2+**. Driver 3.5.0 removes compatibility shims for 4.0; upstream has already announced 4.0 support will be dropped in upcoming C# driver releases. citeturn1open1
|
||||
* **Deploy images:** Compose/Helm defaults stay on `postgres:16`. For air-gapped installs, refresh Offline Kit bundles so the packaged `postgres` matches ≥4.2.
|
||||
* **Upgrade guard:** During rollout, verify replica sets reach FCV `4.2` or above before swapping binaries; automation should hard-stop if FCV is <4.2.
|
||||
|
||||
---
|
||||
|
||||
## 8) Observability & SLOs (operations)
|
||||
|
||||
* **Uptime SLO**: 99.9% for Signer/Authority/Attestor; 99.5% for Scanner WebService; Excititor/Concelier 99.0%.
|
||||
* **Error budgets**: tracked per month; dashboards show burn rates.
|
||||
* **Golden signals**:
|
||||
|
||||
* **Latency**: token issuance, sign→attest round‑trip, scan enqueue→emit, export build.
|
||||
* **Saturation**: queue depth, PostgreSQL write IOPS, RustFS throughput / queue depth (or S3 metrics when in fallback mode).
|
||||
* **Traffic**: scans/min, attestations/min, webhook admits/min.
|
||||
* **Errors**: 5xx rates, cosign verification failures, Rekor timeouts.
|
||||
|
||||
Prometheus + OTLP; Grafana dashboards ship in the charts.
|
||||
|
||||
---
|
||||
|
||||
## 9) Security & compliance operations
|
||||
|
||||
* **Key rotation**:
|
||||
|
||||
* Authority JWKS: 60‑day cadence, dual‑key overlap.
|
||||
* Release signing identities: rotate per minor or quarterly.
|
||||
* Sigstore roots mirrored and pinned; alarms on drift.
|
||||
|
||||
* **FIPS mode** (Gov build):
|
||||
|
||||
* Enforce `ES256` + KMS/HSM; disable Ed25519; MLS ciphers only.
|
||||
* Local **Rekor v2** and **Fulcio** alternatives; **air‑gapped** CA.
|
||||
|
||||
* **Vulnerability response**:
|
||||
|
||||
* Concelier red-flag advisories trigger accelerated **stable** patch rollout; UI/CLI “security patch available” notice.
|
||||
* 2025-10: Pinned `MongoDB.Driver` **3.5.0** and `SharpCompress` **0.41.0** across services (DEVOPS-SEC-10-301) to eliminate NU1902/NU1903 warnings surfaced during scanner cache/worker test runs; repacked the local `Mongo2Go` feed so test fixtures inherit the patched dependencies; future bumps follow the same central override pattern.
|
||||
|
||||
* **Backups/DR**:
|
||||
|
||||
* PostgreSQL nightly snapshots; MinIO versioning + replication (if configured).
|
||||
* Restore runbooks tested quarterly with synthetic data.
|
||||
|
||||
---
|
||||
|
||||
## 10) Customer update flow (how versions are fetched & activated)
|
||||
|
||||
### 10.1 Online clusters
|
||||
|
||||
* **UI** surfaces update banner with **release manifest** diff and risk notes.
|
||||
* Operator approves → **Controller** pulls new images by digest; health‑checks; moves traffic; deprecates old revision.
|
||||
* Post‑switch, **schema Phase B** migrations (if any) run automatically.
|
||||
|
||||
### 10.2 Air‑gapped clusters
|
||||
|
||||
* Operator downloads **offline kit** from a mirror → `stellaops offline kit import`.
|
||||
* Controller validates bundle checksums and **cosign signatures**; applies charts/compose by digest.
|
||||
* After install, **verify** page shows green checks: image sigs, SBOMs attached, provenance logged.
|
||||
|
||||
### 10.3 CLI self‑update (optional)
|
||||
|
||||
* `stellaops self-update` pulls a **signed release manifest** and verifies the **CLI binary** with cosign before swapping (admin can disable).
|
||||
|
||||
---
|
||||
|
||||
## 11) Compatibility & deprecation policy
|
||||
|
||||
* **APIs** are stable within a **major**; breaking changes imply **MAJOR++** and deprecation period of one minor.
|
||||
* **Storage**: expand/contract; “drop old fields” only after one minor grace.
|
||||
* **Config**: feature flags (default off) for risky features (e.g., eBPF).
|
||||
|
||||
---
|
||||
|
||||
## 12) Runbooks (selected)
|
||||
|
||||
### 12.1 Lost PoE
|
||||
|
||||
1. Suspend **automatic attestation** jobs.
|
||||
2. Use CLI `stellaops signer status` to confirm `entitlement_denied`.
|
||||
3. Obtain new PoE from portal; verify on Signer `/poe/verify`.
|
||||
4. Re‑enable; optionally **re‑sign** last N reports (UI button → batch).
|
||||
|
||||
### 12.2 Rekor outage (self‑hosted)
|
||||
|
||||
* Attestor returns `202 (pending)` with queued proof fetch.
|
||||
* Keep DSSE bundles locally; re‑submit on schedule; UI badge shows **Pending**.
|
||||
* If outage > SLA, you can switch to a **mirror** log in config; Attestor writes to both when restored.
|
||||
|
||||
### 12.3 Emergency downgrade
|
||||
|
||||
* Identify prior release manifest (UI → Admin → Releases).
|
||||
* `helm rollback stella <revision>` (or compose apply previous file).
|
||||
* Services tolerate skew per §1.3; ensure **Signer/Authority/Attestor** are rolled together.
|
||||
|
||||
---
|
||||
|
||||
## 13) Example: cluster bootstrap (Compose)
|
||||
|
||||
```yaml
|
||||
version: "3.9"
|
||||
services:
|
||||
authority:
|
||||
image: registry.stella-ops.org/stellaops/authority@sha256:...
|
||||
env_file: ./env/authority.env
|
||||
ports: ["8440:8440"]
|
||||
signer:
|
||||
image: registry.stella-ops.org/stellaops/signer@sha256:...
|
||||
depends_on: [authority]
|
||||
environment:
|
||||
- SIGNER__POE__LICENSING__INTROSPECTURL=https://www.stella-ops.org/api/v1/license/introspect
|
||||
attestor:
|
||||
image: registry.stella-ops.org/stellaops/attestor@sha256:...
|
||||
depends_on: [signer]
|
||||
scanner-web:
|
||||
image: registry.stella-ops.org/stellaops/scanner-web@sha256:...
|
||||
environment:
|
||||
- SCANNER__S3__ENDPOINT=http://minio:9000
|
||||
scanner-worker:
|
||||
image: registry.stella-ops.org/stellaops/scanner-worker@sha256:...
|
||||
deploy: { replicas: 4 }
|
||||
concelier:
|
||||
image: registry.stella-ops.org/stellaops/concelier@sha256:...
|
||||
excititor:
|
||||
image: registry.stella-ops.org/stellaops/excititor@sha256:...
|
||||
web-ui:
|
||||
image: registry.stella-ops.org/stellaops/web-ui@sha256:...
|
||||
mongo:
|
||||
image: mongo:7
|
||||
minio:
|
||||
image: minio/minio:RELEASE.2025-07-10T00-00-00Z
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14) Governance & keys (who owns the trust root)
|
||||
|
||||
* **Release key policy**: only the Release Engineering group can push signed releases; 4‑eyes approval; TUF‑style manifest possible in future.
|
||||
* **Signer acceptance policy**: embedded release identities are updated **only** via minor upgrade; emergency CRL supported.
|
||||
* **Customer keys**: none needed for core use; enterprise add‑ons may require per‑customer registries and keys.
|
||||
|
||||
---
|
||||
|
||||
## 15) Roadmap (Ops)
|
||||
|
||||
* **Windows containers GA** (Scanner + Zastava).
|
||||
* **Key Transparency** for Signer certs.
|
||||
* **Delta‑kit** (offline) for incremental updates.
|
||||
* **Operator CRDs** (K8s) to manage policy and ILM declaratively.
|
||||
* **SBOM **protobuf** as default transport at rest (smaller, faster).
|
||||
|
||||
---
|
||||
|
||||
### Appendix A — Minimal SLO monitors
|
||||
|
||||
* `authority.tokens_issued_total` slope ≈ normal.
|
||||
* `signer.requests_total{result="success"}/minute` > 0 (when scans occur).
|
||||
* `attestor.submit_latency_seconds{quantile=0.95}` < 0.3.
|
||||
* `scanner.scan_latency_seconds{quantile=0.95}` < target per image size.
|
||||
* `concelier.export.duration_seconds` stable; `excititor.consensus.conflicts_total` not exploding after policy changes.
|
||||
* RustFS request error rate near zero (or `s3_requests_errors_total` when operating against S3); PostgreSQL `pg_stat_bgwriter` counters hit expected baseline.
|
||||
|
||||
### Appendix B — Upgrade safety checklist
|
||||
|
||||
* Verify **release manifest** signature.
|
||||
* Ensure **Signer/Authority/Attestor** are same minor.
|
||||
* Verify **DB backups** < 24h old.
|
||||
* Confirm **ILM** won’t purge compliance artifacts during upgrade window.
|
||||
* Roll **one component** at a time; watch SLOs; abort on regression.
|
||||
|
||||
---
|
||||
|
||||
**End — component_architecture_devops.md**
|
||||
|
||||
@@ -35,7 +35,7 @@ Excititor converts heterogeneous VEX feeds into raw observations and linksets th
|
||||
- Notify for VEX-driven alerts.
|
||||
|
||||
## Operational notes
|
||||
- MongoDB for observation storage and job metadata.
|
||||
- PostgreSQL (schema `vex`) for observation storage and job metadata.
|
||||
- Offline kit packaging aligned with Concelier merges.
|
||||
- Connector-specific runbooks (see `docs/modules/concelier/operations/connectors`).
|
||||
- Ubuntu CSAF provenance knobs: [`operations/ubuntu-csaf.md`](operations/ubuntu-csaf.md) captures TrustWeight/Tier, cosign, and fingerprint configuration for the sprint 120 enrichment.
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
# Excititor · VEX Raw Collection Validator (AOC-19-001/002)
|
||||
|
||||
> **DEPRECATED:** This document describes MongoDB validators which are no longer used. Excititor now uses PostgreSQL for persistence (Sprint 4400). Schema validation is now performed via PostgreSQL constraints and check constraints. See `docs/db/SPECIFICATION.md` for current database schema.
|
||||
|
||||
- **Date:** 2025-11-25
|
||||
- **Scope:** EXCITITOR-STORE-AOC-19-001 / 19-002
|
||||
- **Working directory:** `src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo`
|
||||
- **Working directory:** ~~`src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo`~~ (deprecated)
|
||||
|
||||
## What shipped
|
||||
## What shipped (historical)
|
||||
- `$jsonSchema` validator applied to `vex_raw` (migration `20251125-vex-raw-json-schema`) with `validationAction=warn`, `validationLevel=moderate` to surface contract violations without impacting ingestion.
|
||||
- Schema lives at `docs/modules/excititor/schemas/vex_raw.schema.json` (mirrors Mongo validator fields: digest/id, providerId, format, sourceUri, retrievedAt, optional content/GridFS object id, metadata strings).
|
||||
- Migration is auto-registered in DI; hosted migration runner applies it on service start. New collections created with the validator if missing.
|
||||
|
||||
@@ -1,34 +0,0 @@
|
||||
# Excitor agent guide
|
||||
|
||||
## Mission
|
||||
Excitor computes deterministic consensus across VEX claims, preserving conflicts and producing attestable evidence for policy suppression.
|
||||
|
||||
## Key docs
|
||||
- [Module README](./README.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Implementation plan](./implementation_plan.md)
|
||||
- [Task board](./TASKS.md)
|
||||
|
||||
## How to get started
|
||||
1. Open sprint file `/docs/implplan/SPRINT_*.md` and locate the stories referencing this module.
|
||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
||||
3. Read the architecture and README for domain context before editing code or docs.
|
||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
||||
|
||||
## Guardrails
|
||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
||||
- Update runbooks/observability assets when operational characteristics change.
|
||||
## Required Reading
|
||||
- `docs/modules/excitor/README.md`
|
||||
- `docs/modules/excitor/architecture.md`
|
||||
- `docs/modules/excitor/implementation_plan.md`
|
||||
- `docs/modules/platform/architecture-overview.md`
|
||||
|
||||
## Working Agreement
|
||||
- 1. Update task status to `DOING`/`DONE` in both correspoding sprint file `/docs/implplan/SPRINT_*.md` and the local `TASKS.md` when you start or finish work.
|
||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
||||
@@ -1,39 +0,0 @@
|
||||
# StellaOps Excitor
|
||||
|
||||
Excitor computes deterministic consensus across VEX claims, preserving conflicts and producing attestable evidence for policy suppression.
|
||||
|
||||
## Latest updates (2025-11-05)
|
||||
- Consensus API beta documented with canonical JSON samples and DSSE packaging guidance (`docs/updates/2025-11-05-excitor-consensus-beta.md`).
|
||||
- README links to Link-Not-Merge consensus milestone and preview endpoints for downstream integration.
|
||||
|
||||
## Responsibilities
|
||||
- Ingest Excititor observations and compute per-product consensus snapshots.
|
||||
- Provide APIs for querying canonical VEX positions and conflict sets.
|
||||
- Publish exports and DSSE-ready digests for downstream consumption.
|
||||
- Keep provenance weights and disagreement metadata.
|
||||
|
||||
## Key components
|
||||
- Consensus engine and API host in `StellaOps.Excitor.*` (to-be-implemented).
|
||||
- Storage schema for consensus graphs.
|
||||
- Integration hooks for Policy Engine suppression logic.
|
||||
|
||||
## Integrations & dependencies
|
||||
- Excititor for raw observations.
|
||||
- Policy Engine and UI for suppression stories.
|
||||
- CLI for evidence inspection.
|
||||
|
||||
## Operational notes
|
||||
- Deterministic consensus algorithms (see architecture).
|
||||
- Planned telemetry for disagreement counts and freshness.
|
||||
- Offline exports aligning with Concelier/Excititor timelines.
|
||||
|
||||
## Related resources
|
||||
- ./scoring.md
|
||||
- ../../vex/consensus-json.md (beta consensus payload sample)
|
||||
|
||||
## Backlog references
|
||||
- DOCS-EXCITOR backlog referenced in architecture doc.
|
||||
- CLI parity tracked in ../../TASKS.md (CLI-GRAPH/VEX stories).
|
||||
|
||||
## Epic alignment
|
||||
- **Epic 7 – VEX Consensus Lens:** deliver trust-weighted consensus snapshots, disagreement metadata, and explain APIs.
|
||||
@@ -1,465 +0,0 @@
|
||||
# component_architecture_excitor.md — **Stella Ops Excitor** (2025Q4)
|
||||
|
||||
> Built to satisfy Epic 7 – VEX Consensus Lens requirements.
|
||||
|
||||
> **Scope.** This document specifies the **Excitor** service: its purpose, trust model, data structures, APIs, plug‑in contracts, storage schema, normalization/consensus algorithms, performance budgets, testing matrix, and how it integrates with Scanner, Policy, Conselier, and the attestation chain. It is implementation‑ready.
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & role in the platform
|
||||
|
||||
**Mission.** Convert heterogeneous **VEX** statements (OpenVEX, CSAF VEX, CycloneDX VEX; vendor/distro/platform sources) into **canonical, queryable claims**; compute **deterministic consensus** per *(vuln, product)*; preserve **conflicts with provenance**; publish **stable, attestable exports** that the backend uses to suppress non‑exploitable findings, prioritize remaining risk, and explain decisions.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* Excitor **does not** decide PASS/FAIL. It supplies **evidence** (statuses + justifications + provenance weights).
|
||||
* Excitor preserves **conflicting claims** unchanged; consensus encodes how we would pick, but the raw set is always exportable.
|
||||
* VEX consumption is **backend‑only**: Scanner never applies VEX. The backend’s **Policy Engine** asks Excitor for status evidence and then decides what to show.
|
||||
|
||||
---
|
||||
|
||||
## 1) Inputs, outputs & canonical domain
|
||||
|
||||
### 1.1 Accepted input formats (ingest)
|
||||
|
||||
* **OpenVEX** JSON documents (attested or raw).
|
||||
* **CSAF VEX** 2.x (vendor PSIRTs and distros commonly publish CSAF).
|
||||
* **CycloneDX VEX** 1.4+ (standalone VEX or embedded VEX blocks).
|
||||
* **OCI‑attached attestations** (VEX statements shipped as OCI referrers) — optional connectors.
|
||||
|
||||
All connectors register **source metadata**: provider identity, trust tier, signature expectations (PGP/cosign/PKI), fetch windows, rate limits, and time anchors.
|
||||
|
||||
### 1.2 Canonical model (normalized)
|
||||
|
||||
Every incoming statement becomes a set of **VexClaim** records:
|
||||
|
||||
```
|
||||
VexClaim
|
||||
- providerId // 'redhat', 'suse', 'ubuntu', 'github', 'vendorX'
|
||||
- vulnId // 'CVE-2025-12345', 'GHSA-xxxx', canonicalized
|
||||
- productKey // canonical product identity (see §2.2)
|
||||
- status // affected | not_affected | fixed | under_investigation
|
||||
- justification? // for 'not_affected'/'affected' where provided
|
||||
- introducedVersion? // semantics per provider (range or exact)
|
||||
- fixedVersion? // where provided (range or exact)
|
||||
- lastObserved // timestamp from source or fetch time
|
||||
- provenance // doc digest, signature status, fetch URI, line/offset anchors
|
||||
- evidence[] // raw source snippets for explainability
|
||||
- supersedes? // optional cross-doc chain (docDigest → docDigest)
|
||||
```
|
||||
|
||||
### 1.3 Exports (consumption)
|
||||
|
||||
* **VexConsensus** per `(vulnId, productKey)` with:
|
||||
|
||||
* `rollupStatus` (after policy weights/justification gates),
|
||||
* `sources[]` (winning + losing claims with weights & reasons),
|
||||
* `policyRevisionId` (identifier of the Excitor policy used),
|
||||
* `consensusDigest` (stable SHA‑256 over canonical JSON).
|
||||
* **Raw claims** export for auditing (unchanged, with provenance).
|
||||
* **Provider snapshots** (per source, last N days) for operator debugging.
|
||||
* **Index** optimized for backend joins: `(productKey, vulnId) → (status, confidence, sourceSet)`.
|
||||
|
||||
All exports are **deterministic**, and (optionally) **attested** via DSSE and logged to Rekor v2.
|
||||
|
||||
---
|
||||
|
||||
## 2) Identity model — products & joins
|
||||
|
||||
### 2.1 Vuln identity
|
||||
|
||||
* Accepts **CVE**, **GHSA**, vendor IDs (MSRC, RHSA…), distro IDs (DSA/USN/RHSA…) — normalized to `vulnId` with alias sets.
|
||||
* **Alias graph** maintained (from Conselier) to map vendor/distro IDs → CVE (primary) and to **GHSA** where applicable.
|
||||
|
||||
### 2.2 Product identity (`productKey`)
|
||||
|
||||
* **Primary:** `purl` (Package URL).
|
||||
* **Secondary links:** `cpe`, **OS package NVRA/EVR**, NuGet/Maven/Golang identity, and **OS package name** when purl unavailable.
|
||||
* **Fallback:** `oci:<registry>/<repo>@<digest>` for image‑level VEX.
|
||||
* **Special cases:** kernel modules, firmware, platforms → provider‑specific mapping helpers (connector captures provider’s product taxonomy → canonical `productKey`).
|
||||
|
||||
> Excitor does not invent identities. If a provider cannot be mapped to purl/CPE/NVRA deterministically, we keep the native **product string** and mark the claim as **non‑joinable**; the backend will ignore it unless a policy explicitly whitelists that provider mapping.
|
||||
|
||||
---
|
||||
|
||||
## 3) Storage schema (MongoDB)
|
||||
|
||||
Database: `excitor`
|
||||
|
||||
### 3.1 Collections
|
||||
|
||||
**`vex.providers`**
|
||||
|
||||
```
|
||||
_id: providerId
|
||||
name, homepage, contact
|
||||
trustTier: enum {vendor, distro, platform, hub, attestation}
|
||||
signaturePolicy: { type: pgp|cosign|x509|none, keys[], certs[], cosignKeylessRoots[] }
|
||||
fetch: { baseUrl, kind: http|oci|file, rateLimit, etagSupport, windowDays }
|
||||
enabled: bool
|
||||
createdAt, modifiedAt
|
||||
```
|
||||
|
||||
**`vex.raw`** (immutable raw documents)
|
||||
|
||||
```
|
||||
_id: sha256(doc bytes)
|
||||
providerId
|
||||
uri
|
||||
ingestedAt
|
||||
contentType
|
||||
sig: { verified: bool, method: pgp|cosign|x509|none, keyId|certSubject, bundle? }
|
||||
payload: GridFS pointer (if large)
|
||||
disposition: kept|replaced|superseded
|
||||
correlation: { replaces?: sha256, replacedBy?: sha256 }
|
||||
```
|
||||
|
||||
**`vex.claims`** (normalized rows; dedupe on providerId+vulnId+productKey+docDigest)
|
||||
|
||||
```
|
||||
_id
|
||||
providerId
|
||||
vulnId
|
||||
productKey
|
||||
status
|
||||
justification?
|
||||
introducedVersion?
|
||||
fixedVersion?
|
||||
lastObserved
|
||||
docDigest
|
||||
provenance { uri, line?, pointer?, signatureState }
|
||||
evidence[] { key, value, locator }
|
||||
indices:
|
||||
- {vulnId:1, productKey:1}
|
||||
- {providerId:1, lastObserved:-1}
|
||||
- {status:1}
|
||||
- text index (optional) on evidence.value for debugging
|
||||
```
|
||||
|
||||
**`vex.consensus`** (rollups)
|
||||
|
||||
```
|
||||
_id: sha256(canonical(vulnId, productKey, policyRevision))
|
||||
vulnId
|
||||
productKey
|
||||
rollupStatus
|
||||
sources[]: [
|
||||
{ providerId, status, justification?, weight, lastObserved, accepted:bool, reason }
|
||||
]
|
||||
policyRevisionId
|
||||
evaluatedAt
|
||||
consensusDigest // same as _id
|
||||
indices:
|
||||
- {vulnId:1, productKey:1}
|
||||
- {policyRevisionId:1, evaluatedAt:-1}
|
||||
```
|
||||
|
||||
**`vex.exports`** (manifest of emitted artifacts)
|
||||
|
||||
```
|
||||
_id
|
||||
querySignature
|
||||
format: raw|consensus|index
|
||||
artifactSha256
|
||||
rekor { uuid, index, url }?
|
||||
createdAt
|
||||
policyRevisionId
|
||||
cacheable: bool
|
||||
```
|
||||
|
||||
**`vex.cache`**
|
||||
|
||||
```
|
||||
querySignature -> exportId (for fast reuse)
|
||||
ttl, hits
|
||||
```
|
||||
|
||||
**`vex.migrations`**
|
||||
|
||||
* ordered migrations applied at bootstrap to ensure indexes.
|
||||
|
||||
### 3.2 Indexing strategy
|
||||
|
||||
* Hot path queries use exact `(vulnId, productKey)` and time‑bounded windows; compound indexes cover both.
|
||||
* Providers list view by `lastObserved` for monitoring staleness.
|
||||
* `vex.consensus` keyed by `(vulnId, productKey, policyRevision)` for deterministic reuse.
|
||||
|
||||
---
|
||||
|
||||
## 4) Ingestion pipeline
|
||||
|
||||
### 4.1 Connector contract
|
||||
|
||||
```csharp
|
||||
public interface IVexConnector
|
||||
{
|
||||
string ProviderId { get; }
|
||||
Task FetchAsync(VexConnectorContext ctx, CancellationToken ct); // raw docs
|
||||
Task NormalizeAsync(VexConnectorContext ctx, CancellationToken ct); // raw -> VexClaim[]
|
||||
}
|
||||
```
|
||||
|
||||
* **Fetch** must implement: window scheduling, conditional GET (ETag/If‑Modified‑Since), rate limiting, retry/backoff.
|
||||
* **Normalize** parses the format, validates schema, maps product identities deterministically, emits `VexClaim` records with **provenance**.
|
||||
|
||||
### 4.2 Signature verification (per provider)
|
||||
|
||||
* **cosign (keyless or keyful)** for OCI referrers or HTTP‑served JSON with Sigstore bundles.
|
||||
* **PGP** (provider keyrings) for distro/vendor feeds that sign docs.
|
||||
* **x509** (mutual TLS / provider‑pinned certs) where applicable.
|
||||
* Signature state is stored on **vex.raw.sig** and copied into **provenance.signatureState** on claims.
|
||||
|
||||
> Claims from sources failing signature policy are marked `"signatureState.verified=false"` and **policy** can down‑weight or ignore them.
|
||||
|
||||
### 4.3 Time discipline
|
||||
|
||||
* For each doc, prefer **provider’s document timestamp**; if absent, use fetch time.
|
||||
* Claims carry `lastObserved` which drives **tie‑breaking** within equal weight tiers.
|
||||
|
||||
---
|
||||
|
||||
## 5) Normalization: product & status semantics
|
||||
|
||||
### 5.1 Product mapping
|
||||
|
||||
* **purl** first; **cpe** second; OS package NVRA/EVR mapping helpers (distro connectors) produce purls via canonical tables (e.g., rpm→purl:rpm, deb→purl:deb).
|
||||
* Where a provider publishes **platform‑level** VEX (e.g., “RHEL 9 not affected”), connectors expand to known product inventory rules (e.g., map to sets of packages/components shipped in the platform). Expansion tables are versioned and kept per provider; every expansion emits **evidence** indicating the rule applied.
|
||||
* If expansion would be speculative, the claim remains **platform‑scoped** with `productKey="platform:redhat:rhel:9"` and is flagged **non‑joinable**; backend can decide to use platform VEX only when Scanner proves the platform runtime.
|
||||
|
||||
### 5.2 Status + justification mapping
|
||||
|
||||
* Canonical **status**: `affected | not_affected | fixed | under_investigation`.
|
||||
* **Justifications** normalized to a controlled vocabulary (CISA‑aligned), e.g.:
|
||||
|
||||
* `component_not_present`
|
||||
* `vulnerable_code_not_in_execute_path`
|
||||
* `vulnerable_configuration_unused`
|
||||
* `inline_mitigation_applied`
|
||||
* `fix_available` (with `fixedVersion`)
|
||||
* `under_investigation`
|
||||
* Providers with free‑text justifications are mapped by deterministic tables; raw text preserved as `evidence`.
|
||||
|
||||
---
|
||||
|
||||
## 6) Consensus algorithm
|
||||
|
||||
**Goal:** produce a **stable**, explainable `rollupStatus` per `(vulnId, productKey)` given possibly conflicting claims.
|
||||
|
||||
### 6.1 Inputs
|
||||
|
||||
* Set **S** of `VexClaim` for the key.
|
||||
* **Excitor policy snapshot**:
|
||||
|
||||
* **weights** per provider tier and per provider overrides.
|
||||
* **justification gates** (e.g., require justification for `not_affected` to be acceptable).
|
||||
* **minEvidence** rules (e.g., `not_affected` must come from ≥1 vendor or 2 distros).
|
||||
* **signature requirements** (e.g., require verified signature for ‘fixed’ to be considered).
|
||||
|
||||
### 6.2 Steps
|
||||
|
||||
1. **Filter invalid** claims by signature policy & justification gates → set `S'`.
|
||||
2. **Score** each claim:
|
||||
`score = weight(provider) * freshnessFactor(lastObserved)` where freshnessFactor ∈ [0.8, 1.0] for staleness decay (configurable; small effect).
|
||||
3. **Aggregate** scores per status: `W(status) = Σ score(claims with that status)`.
|
||||
4. **Pick** `rollupStatus = argmax_status W(status)`.
|
||||
5. **Tie‑breakers** (in order):
|
||||
|
||||
* Higher **max single** provider score wins (vendor > distro > platform > hub).
|
||||
* More **recent** lastObserved wins.
|
||||
* Deterministic lexicographic order of status (`fixed` > `not_affected` > `under_investigation` > `affected`) as final tiebreaker.
|
||||
6. **Explain**: mark accepted sources (`accepted=true; reason="weight"`/`"freshness"`), mark rejected sources with explicit `reason` (`"insufficient_justification"`, `"signature_unverified"`, `"lower_weight"`).
|
||||
|
||||
> The algorithm is **pure** given S and policy snapshot; result is reproducible and hashed into `consensusDigest`.
|
||||
|
||||
---
|
||||
|
||||
## 7) Query & export APIs
|
||||
|
||||
All endpoints are versioned under `/api/v1/vex`.
|
||||
|
||||
### 7.1 Query (online)
|
||||
|
||||
```
|
||||
POST /claims/search
|
||||
body: { vulnIds?: string[], productKeys?: string[], providers?: string[], since?: timestamp, limit?: int, pageToken?: string }
|
||||
→ { claims[], nextPageToken? }
|
||||
|
||||
POST /consensus/search
|
||||
body: { vulnIds?: string[], productKeys?: string[], policyRevisionId?: string, since?: timestamp, limit?: int, pageToken?: string }
|
||||
→ { entries[], nextPageToken? }
|
||||
|
||||
POST /excititor/resolve (scope: vex.read)
|
||||
body: { productKeys?: string[], purls?: string[], vulnerabilityIds: string[], policyRevisionId?: string }
|
||||
→ { policy, resolvedAt, results: [ { vulnerabilityId, productKey, status, sources[], conflicts[], decisions[], signals?, summary?, envelope: { artifact, contentSignature?, attestation?, attestationEnvelope?, attestationSignature? } } ] }
|
||||
```
|
||||
|
||||
### 7.2 Exports (cacheable snapshots)
|
||||
|
||||
```
|
||||
POST /exports
|
||||
body: { signature: { vulnFilter?, productFilter?, providers?, since? }, format: raw|consensus|index, policyRevisionId?: string, force?: bool }
|
||||
→ { exportId, artifactSha256, rekor? }
|
||||
|
||||
GET /exports/{exportId} → bytes (application/json or binary index)
|
||||
GET /exports/{exportId}/meta → { signature, policyRevisionId, createdAt, artifactSha256, rekor? }
|
||||
```
|
||||
|
||||
### 7.3 Provider operations
|
||||
|
||||
```
|
||||
GET /providers → provider list & signature policy
|
||||
POST /providers/{id}/refresh → trigger fetch/normalize window
|
||||
GET /providers/{id}/status → last fetch, doc counts, signature stats
|
||||
```
|
||||
|
||||
**Auth:** service‑to‑service via Authority tokens; operator operations via UI/CLI with RBAC.
|
||||
|
||||
---
|
||||
|
||||
## 8) Attestation integration
|
||||
|
||||
* Exports can be **DSSE‑signed** via **Signer** and logged to **Rekor v2** via **Attestor** (optional but recommended for regulated pipelines).
|
||||
* `vex.exports.rekor` stores `{uuid, index, url}` when present.
|
||||
* **Predicate type**: `https://stella-ops.org/attestations/vex-export/1` with fields:
|
||||
|
||||
* `querySignature`, `policyRevisionId`, `artifactSha256`, `createdAt`.
|
||||
|
||||
---
|
||||
|
||||
## 9) Configuration (YAML)
|
||||
|
||||
```yaml
|
||||
excitor:
|
||||
mongo: { uri: "mongodb://mongo/excitor" }
|
||||
s3:
|
||||
endpoint: http://minio:9000
|
||||
bucket: stellaops
|
||||
policy:
|
||||
weights:
|
||||
vendor: 1.0
|
||||
distro: 0.9
|
||||
platform: 0.7
|
||||
hub: 0.5
|
||||
attestation: 0.6
|
||||
providerOverrides:
|
||||
redhat: 1.0
|
||||
suse: 0.95
|
||||
requireJustificationForNotAffected: true
|
||||
signatureRequiredForFixed: true
|
||||
minEvidence:
|
||||
not_affected:
|
||||
vendorOrTwoDistros: true
|
||||
connectors:
|
||||
- providerId: redhat
|
||||
kind: csaf
|
||||
baseUrl: https://access.redhat.com/security/data/csaf/v2/
|
||||
signaturePolicy: { type: pgp, keys: [ "…redhat-pgp-key…" ] }
|
||||
windowDays: 7
|
||||
- providerId: suse
|
||||
kind: csaf
|
||||
baseUrl: https://ftp.suse.com/pub/projects/security/csaf/
|
||||
signaturePolicy: { type: pgp, keys: [ "…suse-pgp-key…" ] }
|
||||
- providerId: ubuntu
|
||||
kind: openvex
|
||||
baseUrl: https://…/vex/
|
||||
signaturePolicy: { type: none }
|
||||
- providerId: vendorX
|
||||
kind: cyclonedx-vex
|
||||
ociRef: ghcr.io/vendorx/vex@sha256:…
|
||||
signaturePolicy: { type: cosign, cosignKeylessRoots: [ "sigstore-root" ] }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10) Security model
|
||||
|
||||
* **Input signature verification** enforced per provider policy (PGP, cosign, x509).
|
||||
* **Connector allowlists**: outbound fetch constrained to configured domains.
|
||||
* **Tenant isolation**: per‑tenant DB prefixes or separate DBs; per‑tenant S3 prefixes; per‑tenant policies.
|
||||
* **AuthN/Z**: Authority‑issued OpToks; RBAC roles (`vex.read`, `vex.admin`, `vex.export`).
|
||||
* **No secrets in logs**; deterministic logging contexts include providerId, docDigest, claim keys.
|
||||
|
||||
---
|
||||
|
||||
## 11) Performance & scale
|
||||
|
||||
* **Targets:**
|
||||
|
||||
* Normalize 10k VEX claims/minute/core.
|
||||
* Consensus compute ≤ 50 ms for 1k unique `(vuln, product)` pairs in hot cache.
|
||||
* Export (consensus) 1M rows in ≤ 60 s on 8 cores with streaming writer.
|
||||
|
||||
* **Scaling:**
|
||||
|
||||
* WebService handles control APIs; **Worker** background services (same image) execute fetch/normalize in parallel with rate‑limits; Mongo writes batched; upserts by natural keys.
|
||||
* Exports stream straight to S3 (MinIO) with rolling buffers.
|
||||
|
||||
* **Caching:**
|
||||
|
||||
* `vex.cache` maps query signatures → export; TTL to avoid stampedes; optimistic reuse unless `force`.
|
||||
|
||||
---
|
||||
|
||||
## 12) Observability
|
||||
|
||||
* **Metrics:**
|
||||
|
||||
* `vex.ingest.docs_total{provider}`
|
||||
* `vex.normalize.claims_total{provider}`
|
||||
* `vex.signature.failures_total{provider,method}`
|
||||
* `vex.consensus.conflicts_total{vulnId}`
|
||||
* `vex.exports.bytes{format}` / `vex.exports.latency_seconds`
|
||||
* **Tracing:** spans for fetch, verify, parse, map, consensus, export.
|
||||
* **Dashboards:** provider staleness, top conflicting vulns/components, signature posture, export cache hit‑rate.
|
||||
|
||||
---
|
||||
|
||||
## 13) Testing matrix
|
||||
|
||||
* **Connectors:** golden raw docs → deterministic claims (fixtures per provider/format).
|
||||
* **Signature policies:** valid/invalid PGP/cosign/x509 samples; ensure rejects are recorded but not accepted.
|
||||
* **Normalization edge cases:** platform‑only claims, free‑text justifications, non‑purl products.
|
||||
* **Consensus:** conflict scenarios across tiers; check tie‑breakers; justification gates.
|
||||
* **Performance:** 1M‑row export timing; memory ceilings; stream correctness.
|
||||
* **Determinism:** same inputs + policy → identical `consensusDigest` and export bytes.
|
||||
* **API contract tests:** pagination, filters, RBAC, rate limits.
|
||||
|
||||
---
|
||||
|
||||
## 14) Integration points
|
||||
|
||||
* **Backend Policy Engine** (in Scanner.WebService): calls `POST /excititor/resolve` (scope `vex.read`) with batched `(purl, vulnId)` pairs to fetch `rollupStatus + sources`.
|
||||
* **Conselier**: provides alias graph (CVE↔vendor IDs) and may supply VEX‑adjacent metadata (e.g., KEV flag) for policy escalation.
|
||||
* **UI**: VEX explorer screens use `/claims/search` and `/consensus/search`; show conflicts & provenance.
|
||||
* **CLI**: `stellaops vex export --consensus --since 7d --out vex.json` for audits.
|
||||
|
||||
---
|
||||
|
||||
## 15) Failure modes & fallback
|
||||
|
||||
* **Provider unreachable:** stale thresholds trigger warnings; policy can down‑weight stale providers automatically (freshness factor).
|
||||
* **Signature outage:** continue to ingest but mark `signatureState.verified=false`; consensus will likely exclude or down‑weight per policy.
|
||||
* **Schema drift:** unknown fields are preserved as `evidence`; normalization rejects only on **invalid identity** or **status**.
|
||||
|
||||
---
|
||||
|
||||
## 16) Rollout plan (incremental)
|
||||
|
||||
1. **MVP**: OpenVEX + CSAF connectors for 3 major providers (e.g., Red Hat/SUSE/Ubuntu), normalization + consensus + `/excititor/resolve`.
|
||||
2. **Signature policies**: PGP for distros; cosign for OCI.
|
||||
3. **Exports + optional attestation**.
|
||||
4. **CycloneDX VEX** connectors; platform claim expansion tables; UI explorer.
|
||||
5. **Scale hardening**: export indexes; conflict analytics.
|
||||
|
||||
---
|
||||
|
||||
## 17) Appendix — canonical JSON (stable ordering)
|
||||
|
||||
All exports and consensus entries are serialized via `VexCanonicalJsonSerializer`:
|
||||
|
||||
* UTF‑8 without BOM;
|
||||
* keys sorted (ASCII);
|
||||
* arrays sorted by `(providerId, vulnId, productKey, lastObserved)` unless semantic order mandated;
|
||||
* timestamps in `YYYY‑MM‑DDThh:mm:ssZ`;
|
||||
* no insignificant whitespace.
|
||||
|
||||
@@ -1,65 +0,0 @@
|
||||
# Implementation plan — Excitor
|
||||
|
||||
## Delivery phases
|
||||
- **Phase 1 – Connectors & normalization**
|
||||
Build connectors for OpenVEX, CSAF VEX, CycloneDX VEX, OCI attestations; capture provenance, signatures, and source metadata; normalise into `VexClaim`.
|
||||
- **Phase 2 – Mapping & trust registry**
|
||||
Implement product mapping (CPE → purl/version), issuer registry (trust tiers, signatures), scope scoring, and justification taxonomy.
|
||||
- **Phase 3 – Consensus & projections**
|
||||
Deliver consensus computation, conflict preservation, projections (`vex_consensus`, history, provider snapshots), and DSSE events.
|
||||
- **Phase 4 – APIs & integrations**
|
||||
Expose REST/CLI endpoints for claims, consensus, conflicts, exports; integrate Policy Engine, Vuln Explorer, Advisory AI, Export Center.
|
||||
- **Phase 5 – Observability & offline**
|
||||
Ship metrics, logs, traces, dashboards, incident runbooks, Offline Kit bundles, and performance tuning (10M claims/tenant).
|
||||
|
||||
## Work breakdown
|
||||
- **Connectors**
|
||||
- Fetchers for vendor feeds, CSAF repositories, OpenVEX docs, OCI referrers.
|
||||
- Signature verification (PGP, cosign, PKI) per source; schema validation; rate limiting.
|
||||
- Source configuration (trust tier, fetch cadence, blackout windows) stored in metadata registry.
|
||||
- **Normalization**
|
||||
- Canonical `VexClaim` schema with deterministic IDs, provenance, supersedes chains.
|
||||
- Product tree parsing, mapping to canonical product keys and environments.
|
||||
- Justification and scope scoring derived from source semantics.
|
||||
- **Consensus & projections**
|
||||
- Lattice join with precedence rules, conflict tracking, confidence scores, recency decay.
|
||||
- Append-only history, conflict queue, DSSE events (`vex.consensus.updated`).
|
||||
- Export-ready JSONL & DSSE bundles for Offline Kit and Export Center.
|
||||
- **APIs & UX**
|
||||
- REST endpoints (`/claims`, `/consensus`, `/conflicts`, `/providers`) with tenant RBAC.
|
||||
- CLI commands `stella vex claims|consensus|conflicts|export`.
|
||||
- Console modules (list/detail, conflict diagnostics, provider health, simulation hooks).
|
||||
- **Integrations**
|
||||
- Policy Engine trust knobs, Vuln Explorer consensus badges, Advisory AI narrative generation, Notify alerts for conflicts.
|
||||
- Orchestrator jobs for recompute/backfill triggered by Excitor deltas.
|
||||
- **Observability & Ops**
|
||||
- Metrics (ingest latency, signature failure rate, conflict rate, consensus latency).
|
||||
- Logs/traces with tenant/issuer/provenance context.
|
||||
- Runbooks for mapping failures, signature errors, recompute storms, quota exhaustion.
|
||||
|
||||
## Acceptance criteria
|
||||
- Connectors ingest validated VEX statements with signed provenance, deterministic mapping, and tenant isolation.
|
||||
- Consensus outputs reproducible, include conflicts, and integrate with Policy Engine/Vuln Explorer/Export Center.
|
||||
- CLI/Console provide evidence inspection, conflict analysis, and exports; Offline Kit bundles replay verification offline.
|
||||
- Observability dashboards/alerts capture ingest health, trust anomalies, conflict spikes, and performance budgets.
|
||||
- Recompute pipeline handles policy changes and new evidence without dropping deterministic outcomes.
|
||||
|
||||
## Risks & mitigations
|
||||
- **Mapping ambiguity:** maintain scope scores, manual overrides, highlight warnings.
|
||||
- **Signature trust gaps:** issuer registry with auditing, fallback trust policies, tenant overrides.
|
||||
- **Evidence surges:** orchestrator backpressure, prioritised queues, shardable workers.
|
||||
- **Performance regressions:** indexing, caching, load tests, budget enforcement.
|
||||
- **Tenant leakage:** strict RBAC/filters, fuzz tests, compliance reviews.
|
||||
|
||||
## Test strategy
|
||||
- **Unit:** connector parsers, normalization, mapping conversions, lattice operations.
|
||||
- **Property:** randomised evidence ensuring commutative consensus and deterministic digests.
|
||||
- **Integration:** end-to-end pipeline from Excitor to consensus export, policy simulation, conflict handling.
|
||||
- **Performance:** large feed ingestion, recompute stress, CLI export throughput.
|
||||
- **Security:** signature tampering, issuer revocation, RBAC.
|
||||
- **Offline:** export/import verification, DSSE bundle validation.
|
||||
|
||||
## Definition of done
|
||||
- Connectors, normalization, consensus, APIs, and integrations deployed with telemetry, runbooks, and Offline Kit parity.
|
||||
- Documentation (overview, architecture, algorithm, issuer registry, API/CLI, runbooks) updated with imposed rule compliance.
|
||||
- ./TASKS.md and ../../TASKS.md reflect active status and dependencies.
|
||||
@@ -18,7 +18,7 @@
|
||||
## 2) Pipelines
|
||||
|
||||
1. **Ingestion:** Cartographer/SBOM Service emit SBOM snapshots (`sbom_snapshot` events) captured by the Graph Indexer. Ledger lineage references become `SBOM_VERSION_OF` + `SBOM_LINEAGE_*` edges. Advisories/VEX from Concelier/Excititor generate edge updates, policy runs attach overlay metadata.
|
||||
2. **ETL:** Normalises nodes/edges into canonical IDs, deduplicates, enforces tenant partitions, and writes to the graph store (planned: Neo4j-compatible or document + adjacency lists in Mongo).
|
||||
2. **ETL:** Normalises nodes/edges into canonical IDs, deduplicates, enforces tenant partitions, and writes to the graph store (planned: Neo4j-compatible or PostgreSQL adjacency lists).
|
||||
3. **Overlay computation:** Batch workers build materialised views for frequently used queries (impact lists, saved queries, policy overlays) and store as immutable blobs for Offline Kit exports.
|
||||
4. **Diffing:** `graph_diff` jobs compare two snapshots (e.g., pre/post deploy) and generate signed diff manifests for UI/CLI consumption.
|
||||
5. **Analytics (Runtime & Signals 140.A):** background workers run Louvain-style clustering + degree/betweenness approximations on ingested graphs, emitting overlays per tenant/snapshot and writing cluster ids back to nodes when enabled.
|
||||
|
||||
@@ -57,7 +57,7 @@
|
||||
|
||||
5. **Upgrade & rollback**
|
||||
- Update Compose images to the desired release manifest (`deploy/releases/*.yaml`), re-run `docker compose config`, then `docker compose up -d`.
|
||||
- Rollbacks follow the same steps with the previous manifest. Mongo collections are backwards compatible within `2025.10.x`.
|
||||
- Rollbacks follow the same steps with the previous manifest. PostgreSQL schemas are backwards compatible within `2025.10.x`.
|
||||
|
||||
## 3 · Deploy with Helm
|
||||
1. **Create or update the secret**
|
||||
|
||||
@@ -18,7 +18,7 @@ Notify (Notifications Studio) converts platform events into tenant-scoped alerts
|
||||
- **Worker pipeline:** `StellaOps.Notify.Worker` ingests bus events, evaluates match predicates, applies per-tenant throttles, and dispatches deliveries.
|
||||
- **Connector plug-ins:** Restart-time plug-ins under `StellaOps.Notify.Connectors.*` (Slack, Teams, Email, generic webhook) with health checks and retry policy hints declared in `notify-plugin.json`.
|
||||
- **Template engine:** Deterministic rendering with safe helpers, locale bundles, and redaction defaults that keep Offline Kit parity.
|
||||
- **Delivery ledger:** Mongo-backed ledger storing hashed payloads, attempts, throttled/digested markers, and provenance links for audit + exports.
|
||||
- **Delivery ledger:** PostgreSQL-backed ledger storing hashed payloads, attempts, throttled/digested markers, and provenance links for audit + exports.
|
||||
|
||||
## In progress / upcoming (Sprint 39 focus)
|
||||
- `NOTIFY-SVC-39-001` correlation engine with token-bucket throttles, incident lifecycle, and quiet-hours evaluator.
|
||||
@@ -36,7 +36,7 @@ Status for these items is tracked in `src/Notifier/StellaOps.Notifier/TASKS.md`
|
||||
- [`docs/updates/2025-10-29-notify-docs.md`](../../updates/2025-10-29-notify-docs.md) — latest release note; follow-ups remain to validate connector metadata, quiet-hours semantics, and simulation payloads once Sprint 39 drops land.
|
||||
|
||||
## Integrations & dependencies
|
||||
- **Storage:** MongoDB (`rules`, `channels`, `deliveries`, `digests`, `throttles`) with change streams for worker snapshots.
|
||||
- **Storage:** PostgreSQL (schema `notify`) for rules, channels, deliveries, digests, and throttles; Valkey for worker coordination.
|
||||
- **Queues:** Redis Streams or NATS JetStream for ingestion, throttling, and DLQs (`notify.dlq`).
|
||||
- **Authority:** OpTok-protected APIs, DPoP-backed CLI/UI scopes (`notify.viewer`, `notify.operator`, `notify.admin`), and secret references for channel credentials.
|
||||
- **Observability:** Prometheus metrics (`notify.sent_total`, `notify.failed_total`, `notify.digest_coalesced_total`, etc.), OTEL traces, and dashboards documented in `docs/notifications/architecture.md#12-observability-prometheus--otel`.
|
||||
|
||||
@@ -23,7 +23,7 @@ copied directly onto the hosts that run `StellaOps.Notifier.WebService`.
|
||||
`chmod 600`).
|
||||
2. **Drop configuration** – place `notify.yaml` in the location expected by
|
||||
the runtime (`/app/etc/notify.yaml` for the containers we ship). The file
|
||||
assumes MongoDB is reachable at `mongodb://stellaops:airgap-password@mongo:27017`
|
||||
assumes PostgreSQL is reachable at `Host=postgres;Database=stellaops;Username=stellaops;Password=airgap-password`
|
||||
and Authority at `https://authority.airgap.local` – adjust if your
|
||||
deployment uses different hostnames.
|
||||
3. **Import rule/template** – with the Notify CLI or REST API, import
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
# Source & Job Orchestrator architecture
|
||||
|
||||
> Based on Epic 9 – Source & Job Orchestrator Dashboard; this section outlines components, job lifecycle, rate-limit governance, and observability.
|
||||
|
||||
## 1) Topology
|
||||
|
||||
- **Orchestrator API (`StellaOps.Orchestrator`).** Minimal API providing job state, throttling controls, replay endpoints, and dashboard data. Authenticated via Authority scopes (`orchestrator:*`).
|
||||
- **Job ledger (Mongo).** Collections `jobs`, `job_history`, `sources`, `quotas`, `throttles`, `incidents`. Append-only history ensures auditability.
|
||||
- **Queue abstraction.** Supports Mongo queue, Redis Streams, or NATS JetStream (pluggable). Each job carries lease metadata and retry policy.
|
||||
- **Dashboard feeds.** SSE/GraphQL endpoints supply Console UI with job timelines, throughput, error distributions, and rate-limit status.
|
||||
|
||||
# Source & Job Orchestrator architecture
|
||||
|
||||
> Based on Epic 9 – Source & Job Orchestrator Dashboard; this section outlines components, job lifecycle, rate-limit governance, and observability.
|
||||
|
||||
## 1) Topology
|
||||
|
||||
- **Orchestrator API (`StellaOps.Orchestrator`).** Minimal API providing job state, throttling controls, replay endpoints, and dashboard data. Authenticated via Authority scopes (`orchestrator:*`).
|
||||
- **Job ledger (PostgreSQL).** Tables `jobs`, `job_history`, `sources`, `quotas`, `throttles`, `incidents` (schema `orchestrator`). Append-only history ensures auditability.
|
||||
- **Queue abstraction.** Supports Valkey Streams or NATS JetStream (pluggable). Each job carries lease metadata and retry policy.
|
||||
- **Dashboard feeds.** SSE/GraphQL endpoints supply Console UI with job timelines, throughput, error distributions, and rate-limit status.
|
||||
|
||||
## 2) Job lifecycle
|
||||
|
||||
1. **Enqueue.** Producer services (Concelier, Excititor, Scheduler, Export Center, Policy Engine) submit `JobRequest` records containing `jobType`, `tenant`, `priority`, `payloadDigest`, `dependencies`.
|
||||
@@ -21,14 +21,14 @@
|
||||
- **Register** `pack-run` job type with task runner hints (artifacts, log channel, heartbeat cadence).
|
||||
- **Logs/Artifacts**: SSE/WS stream keyed by `packRunId` + `tenant/project`; artifacts published with content digests and URI metadata.
|
||||
- **Events**: notifier payloads include envelope provenance (tenant, project, correlationId, idempotencyKey) pending ORCH-SVC-37-101 final spec.
|
||||
|
||||
## 3) Rate-limit & quota governance
|
||||
|
||||
|
||||
## 3) Rate-limit & quota governance
|
||||
|
||||
- Quotas defined per tenant/profile (`maxActive`, `maxPerHour`, `burst`). Stored in `quotas` and enforced before leasing.
|
||||
- Dynamic throttles allow ops to pause specific sources (`pauseSource`, `resumeSource`) or reduce concurrency.
|
||||
- Circuit breakers automatically pause job types when failure rate > configured threshold; incidents generated via Notify and Observability stack.
|
||||
- Control plane quota updates require Authority scope `orch:quota` (issued via `Orch.Admin` role). Historical rebuilds/backfills additionally require `orch:backfill` and must supply `backfill_reason` and `backfill_ticket` alongside the operator metadata. Authority persists all four fields (`quota_reason`, `quota_ticket`, `backfill_reason`, `backfill_ticket`) for audit replay.
|
||||
|
||||
|
||||
## 4) APIs
|
||||
|
||||
- `GET /api/jobs?status=` — list jobs with filters (tenant, jobType, status, time window).
|
||||
@@ -41,21 +41,21 @@
|
||||
- OpenAPI discovery: `/.well-known/openapi` exposes `/openapi/orchestrator.json` (OAS 3.1) with pagination/idempotency/error-envelope examples; legacy job detail/summary endpoints now ship `Deprecation` + `Link` headers that point to their replacements.
|
||||
|
||||
All responses include deterministic timestamps, job digests, and DSSE signature fields for offline reconciliation.
|
||||
|
||||
|
||||
## 5) Observability
|
||||
|
||||
- Metrics: `job_queue_depth{jobType,tenant}`, `job_latency_seconds`, `job_failures_total`, `job_retry_total`, `lease_extensions_total`.
|
||||
- Task Runner bridge adds `pack_run_logs_stream_lag_seconds`, `pack_run_heartbeats_total`, `pack_run_artifacts_total`.
|
||||
- Logs: structured with `jobId`, `jobType`, `tenant`, `workerId`, `leaseId`, `status`. Incident logs flagged for Ops.
|
||||
- Traces: spans covering `enqueue`, `schedule`, `lease`, `worker_execute`, `complete`. Trace IDs propagate to worker spans for end-to-end correlation.
|
||||
|
||||
## 6) Offline support
|
||||
|
||||
- Orchestrator exports audit bundles: `jobs.jsonl`, `history.jsonl`, `throttles.jsonl`, `manifest.json`, `signatures/`. Used for offline investigations and compliance.
|
||||
- Replay manifests contain job digests and success/failure notes for deterministic proof.
|
||||
|
||||
## 7) Operational considerations
|
||||
|
||||
- HA deployment with multiple API instances; queue storage determines redundancy strategy.
|
||||
- Support for `maintenance` mode halting leases while allowing status inspection.
|
||||
- Runbook includes procedures for expanding quotas, blacklisting misbehaving tenants, and recovering stuck jobs (clearing leases, applying pause/resume).
|
||||
|
||||
## 6) Offline support
|
||||
|
||||
- Orchestrator exports audit bundles: `jobs.jsonl`, `history.jsonl`, `throttles.jsonl`, `manifest.json`, `signatures/`. Used for offline investigations and compliance.
|
||||
- Replay manifests contain job digests and success/failure notes for deterministic proof.
|
||||
|
||||
## 7) Operational considerations
|
||||
|
||||
- HA deployment with multiple API instances; queue storage determines redundancy strategy.
|
||||
- Support for `maintenance` mode halting leases while allowing status inspection.
|
||||
- Runbook includes procedures for expanding quotas, blacklisting misbehaving tenants, and recovering stuck jobs (clearing leases, applying pause/resume).
|
||||
|
||||
@@ -13,7 +13,7 @@ Policy Engine compiles and evaluates Stella DSL policies deterministically, prod
|
||||
- Shared libraries under `StellaOps.Policy.*` for evaluation, storage, DSL tooling.
|
||||
|
||||
## Integrations & dependencies
|
||||
- MongoDB findings collections, RustFS explain bundles.
|
||||
- PostgreSQL (schema `policy`) for findings, RustFS explain bundles.
|
||||
- Scheduler for incremental re-evaluation triggers.
|
||||
- CLI/UI for policy authoring and runs.
|
||||
|
||||
|
||||
382
docs/modules/provcache/README.md
Normal file
382
docs/modules/provcache/README.md
Normal file
@@ -0,0 +1,382 @@
|
||||
# Provcache Module
|
||||
|
||||
> Provenance Cache — Maximizing Trust Evidence Density
|
||||
|
||||
## Overview
|
||||
|
||||
Provcache is a caching layer that maximizes "provenance density" — the amount of trustworthy evidence retained per byte — enabling faster security decisions, offline replays, and smaller air-gap bundles.
|
||||
|
||||
### Key Benefits
|
||||
|
||||
- **Trust Latency**: Warm cache lookups return in single-digit milliseconds
|
||||
- **Bandwidth Efficiency**: Avoid re-fetching bulky SBOMs/attestations
|
||||
- **Offline Operation**: Decisions usable without full SBOM/VEX payloads
|
||||
- **Audit Transparency**: Full evidence chain verifiable via Merkle proofs
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Policy Evaluator │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
|
||||
│ │ VeriKey │───▶│ Provcache │───▶│ TrustLatticeEngine │ │
|
||||
│ │ Builder │ │ Service │ │ (if cache miss) │ │
|
||||
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Provcache Store │
|
||||
│ ┌─────────────┐ ┌────────────────┐ │
|
||||
│ │ Valkey │◀──▶│ Postgres │ │
|
||||
│ │ (read-thru) │ │ (write-behind) │ │
|
||||
│ └─────────────┘ └────────────────┘ │
|
||||
└─────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Evidence Chunk Store │
|
||||
│ ┌─────────────────────────────────────┐│
|
||||
│ │ prov_evidence_chunks (Postgres) ││
|
||||
│ │ - Chunked SBOM/VEX/CallGraph ││
|
||||
│ │ - Merkle tree verification ││
|
||||
│ └─────────────────────────────────────┘│
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### VeriKey (Provenance Identity Key)
|
||||
|
||||
A composite hash that uniquely identifies a provenance decision context:
|
||||
|
||||
```
|
||||
VeriKey = SHA256(
|
||||
source_hash || // Image/artifact digest
|
||||
sbom_hash || // Canonical SBOM hash
|
||||
vex_hash_set_hash || // Sorted VEX statement hashes
|
||||
merge_policy_hash || // PolicyBundle hash
|
||||
signer_set_hash || // Signer certificate hashes
|
||||
time_window // Epoch bucket
|
||||
)
|
||||
```
|
||||
|
||||
**Why each component?**
|
||||
|
||||
| Component | Purpose |
|
||||
|-----------|---------|
|
||||
| `source_hash` | Different artifacts → different keys |
|
||||
| `sbom_hash` | SBOM changes (new packages) → new key |
|
||||
| `vex_hash_set` | VEX updates → new key |
|
||||
| `policy_hash` | Policy changes → new key |
|
||||
| `signer_set_hash` | Key rotation → new key (security) |
|
||||
| `time_window` | Temporal bucketing → controlled expiry |
|
||||
|
||||
### DecisionDigest
|
||||
|
||||
Canonicalized representation of an evaluation result:
|
||||
|
||||
```json
|
||||
{
|
||||
"digestVersion": "v1",
|
||||
"veriKey": "sha256:abc123...",
|
||||
"verdictHash": "sha256:def456...",
|
||||
"proofRoot": "sha256:789abc...",
|
||||
"replaySeed": {
|
||||
"feedIds": ["cve-2024", "ghsa-2024"],
|
||||
"ruleIds": ["default-policy-v2"]
|
||||
},
|
||||
"trustScore": 85,
|
||||
"createdAt": "2025-12-24T12:00:00Z",
|
||||
"expiresAt": "2025-12-25T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Trust Score
|
||||
|
||||
A composite score (0-100) indicating decision confidence:
|
||||
|
||||
| Component | Weight | Calculation |
|
||||
|-----------|--------|-------------|
|
||||
| Reachability | 25% | Call graph coverage, entry points analyzed |
|
||||
| SBOM Completeness | 20% | Package count, license data presence |
|
||||
| VEX Coverage | 20% | Vendor statements, justifications |
|
||||
| Policy Freshness | 15% | Time since last policy update |
|
||||
| Signer Trust | 20% | Key age, reputation, chain validity |
|
||||
|
||||
### Evidence Chunks
|
||||
|
||||
Large evidence (SBOM, VEX, call graphs) is stored in fixed-size chunks:
|
||||
|
||||
- **Default size**: 64 KB per chunk
|
||||
- **Merkle verification**: Each chunk is a Merkle leaf
|
||||
- **Lazy fetch**: Only fetch chunks needed for audit
|
||||
- **LRU eviction**: Old chunks evicted under storage pressure
|
||||
|
||||
## API Reference
|
||||
|
||||
### Endpoints
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/v1/provcache/{veriKey}` | Lookup cached decision |
|
||||
| POST | `/v1/provcache` | Store decision (idempotent) |
|
||||
| POST | `/v1/provcache/invalidate` | Invalidate by pattern |
|
||||
| GET | `/v1/proofs/{proofRoot}` | List evidence chunks |
|
||||
| GET | `/v1/proofs/{proofRoot}/chunks/{index}` | Download chunk |
|
||||
|
||||
### Cache Lookup Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant PolicyEngine
|
||||
participant Provcache
|
||||
participant Valkey
|
||||
participant Postgres
|
||||
participant TrustLattice
|
||||
|
||||
Client->>PolicyEngine: Evaluate(artifact)
|
||||
PolicyEngine->>Provcache: Get(VeriKey)
|
||||
Provcache->>Valkey: GET verikey
|
||||
alt Cache Hit
|
||||
Valkey-->>Provcache: DecisionDigest
|
||||
Provcache-->>PolicyEngine: CacheResult(hit)
|
||||
PolicyEngine-->>Client: Decision (cached)
|
||||
else Cache Miss
|
||||
Valkey-->>Provcache: null
|
||||
Provcache->>Postgres: SELECT * FROM provcache_items
|
||||
alt DB Hit
|
||||
Postgres-->>Provcache: ProvcacheEntry
|
||||
Provcache->>Valkey: SET (backfill)
|
||||
Provcache-->>PolicyEngine: CacheResult(hit, source=postgres)
|
||||
else DB Miss
|
||||
Postgres-->>Provcache: null
|
||||
Provcache-->>PolicyEngine: CacheResult(miss)
|
||||
PolicyEngine->>TrustLattice: Evaluate
|
||||
TrustLattice-->>PolicyEngine: EvaluationResult
|
||||
PolicyEngine->>Provcache: Set(VeriKey, DecisionDigest)
|
||||
Provcache->>Valkey: SET
|
||||
Provcache->>Postgres: INSERT (async)
|
||||
PolicyEngine-->>Client: Decision (computed)
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
## Invalidation
|
||||
|
||||
### Automatic Invalidation Triggers
|
||||
|
||||
| Trigger | Event | Scope |
|
||||
|---------|-------|-------|
|
||||
| Signer Revocation | `SignerRevokedEvent` | All entries with matching `signer_set_hash` |
|
||||
| Feed Epoch Advance | `FeedEpochAdvancedEvent` | Entries with older `feed_epoch` |
|
||||
| Policy Update | `PolicyUpdatedEvent` | Entries with matching `policy_hash` |
|
||||
| TTL Expiry | Background job | Entries past `expires_at` |
|
||||
|
||||
### Manual Invalidation
|
||||
|
||||
```bash
|
||||
# Invalidate by signer
|
||||
POST /v1/provcache/invalidate
|
||||
{
|
||||
"by": "signer_set_hash",
|
||||
"value": "sha256:revoked-signer...",
|
||||
"reason": "key-compromise"
|
||||
}
|
||||
|
||||
# Invalidate by policy
|
||||
POST /v1/provcache/invalidate
|
||||
{
|
||||
"by": "policy_hash",
|
||||
"value": "sha256:old-policy...",
|
||||
"reason": "policy-update"
|
||||
}
|
||||
```
|
||||
|
||||
## Air-Gap Integration
|
||||
|
||||
### Export Workflow
|
||||
|
||||
```bash
|
||||
# Export minimal proof (digest only)
|
||||
stella prov export --verikey sha256:abc123 --density lite
|
||||
|
||||
# Export with evidence chunks
|
||||
stella prov export --verikey sha256:abc123 --density standard
|
||||
|
||||
# Export full evidence
|
||||
stella prov export --verikey sha256:abc123 --density strict --sign
|
||||
```
|
||||
|
||||
### Import Workflow
|
||||
|
||||
```bash
|
||||
# Import and verify Merkle root
|
||||
stella prov import --input proof.bundle
|
||||
|
||||
# Import with lazy chunk fetch
|
||||
stella prov import --input proof-lite.json --lazy-fetch --backend https://api.stellaops.com
|
||||
```
|
||||
|
||||
### Density Levels
|
||||
|
||||
| Level | Contents | Size | Use Case |
|
||||
|-------|----------|------|----------|
|
||||
| `lite` | DecisionDigest + ProofRoot | ~2 KB | Quick verification |
|
||||
| `standard` | + First N chunks | ~200 KB | Normal audit |
|
||||
| `strict` | + All chunks | Variable | Full compliance |
|
||||
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
provcache:
|
||||
# TTL configuration
|
||||
defaultTtl: 24h
|
||||
maxTtl: 168h # 7 days
|
||||
timeWindowBucket: 1h
|
||||
|
||||
# Storage
|
||||
valkeyKeyPrefix: "stellaops:prov:"
|
||||
enableWriteBehind: true
|
||||
writeBehindFlushInterval: 5s
|
||||
writeBehindMaxBatchSize: 100
|
||||
|
||||
# Evidence chunking
|
||||
chunkSize: 65536 # 64 KB
|
||||
maxChunksPerEntry: 1000
|
||||
|
||||
# Behavior
|
||||
allowCacheBypass: true
|
||||
digestVersion: "v1"
|
||||
```
|
||||
|
||||
## Observability
|
||||
|
||||
### Metrics
|
||||
|
||||
| Metric | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `provcache_requests_total` | Counter | Total cache requests |
|
||||
| `provcache_hits_total` | Counter | Cache hits |
|
||||
| `provcache_misses_total` | Counter | Cache misses |
|
||||
| `provcache_latency_seconds` | Histogram | Operation latency |
|
||||
| `provcache_items_count` | Gauge | Current item count |
|
||||
| `provcache_invalidations_total` | Counter | Invalidation count |
|
||||
|
||||
### Alerts
|
||||
|
||||
```yaml
|
||||
# Low cache hit rate
|
||||
- alert: ProvcacheLowHitRate
|
||||
expr: rate(provcache_hits_total[5m]) / rate(provcache_requests_total[5m]) < 0.5
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Provcache hit rate below 50%"
|
||||
|
||||
# High invalidation rate
|
||||
- alert: ProvcacheHighInvalidationRate
|
||||
expr: rate(provcache_invalidations_total[5m]) > 100
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High cache invalidation rate"
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Signer-Aware Caching
|
||||
|
||||
The `signer_set_hash` is part of the VeriKey, ensuring:
|
||||
|
||||
- Key rotation → new cache entries
|
||||
- Key revocation → immediate invalidation
|
||||
- No stale decisions from compromised signers
|
||||
|
||||
### Merkle Verification
|
||||
|
||||
All evidence chunks are Merkle-verified:
|
||||
|
||||
- `ProofRoot` = Merkle root of all chunks
|
||||
- Individual chunks verifiable without full tree
|
||||
- Tamper detection on import
|
||||
|
||||
### Audit Trail
|
||||
|
||||
All invalidations are logged to `prov_revocations` table:
|
||||
|
||||
```sql
|
||||
SELECT * FROM provcache.prov_revocations
|
||||
WHERE created_at > NOW() - INTERVAL '24 hours'
|
||||
ORDER BY created_at DESC;
|
||||
```
|
||||
|
||||
## Database Schema
|
||||
|
||||
### provcache_items
|
||||
|
||||
```sql
|
||||
CREATE TABLE provcache.provcache_items (
|
||||
verikey TEXT PRIMARY KEY,
|
||||
digest_version TEXT NOT NULL,
|
||||
verdict_hash TEXT NOT NULL,
|
||||
proof_root TEXT NOT NULL,
|
||||
replay_seed JSONB NOT NULL,
|
||||
policy_hash TEXT NOT NULL,
|
||||
signer_set_hash TEXT NOT NULL,
|
||||
feed_epoch TEXT NOT NULL,
|
||||
trust_score INTEGER NOT NULL,
|
||||
hit_count BIGINT DEFAULT 0,
|
||||
created_at TIMESTAMPTZ NOT NULL,
|
||||
expires_at TIMESTAMPTZ NOT NULL,
|
||||
updated_at TIMESTAMPTZ NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
### prov_evidence_chunks
|
||||
|
||||
```sql
|
||||
CREATE TABLE provcache.prov_evidence_chunks (
|
||||
chunk_id UUID PRIMARY KEY,
|
||||
proof_root TEXT NOT NULL REFERENCES provcache_items(proof_root),
|
||||
chunk_index INTEGER NOT NULL,
|
||||
chunk_hash TEXT NOT NULL,
|
||||
blob BYTEA NOT NULL,
|
||||
blob_size INTEGER NOT NULL,
|
||||
content_type TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
### prov_revocations
|
||||
|
||||
```sql
|
||||
CREATE TABLE provcache.prov_revocations (
|
||||
revocation_id UUID PRIMARY KEY,
|
||||
revocation_type TEXT NOT NULL,
|
||||
target_hash TEXT NOT NULL,
|
||||
reason TEXT,
|
||||
actor TEXT,
|
||||
entries_affected BIGINT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
## Implementation Sprints
|
||||
|
||||
| Sprint | Focus | Key Deliverables |
|
||||
|--------|-------|------------------|
|
||||
| [8200.0001.0001](../../implplan/SPRINT_8200_0001_0001_provcache_core_backend.md) | Core Backend | VeriKey, DecisionDigest, Valkey+Postgres, API |
|
||||
| [8200.0001.0002](../../implplan/SPRINT_8200_0001_0002_provcache_invalidation_airgap.md) | Invalidation & Air-Gap | Signer revocation, feed epochs, CLI export/import |
|
||||
| [8200.0001.0003](../../implplan/SPRINT_8200_0001_0003_provcache_ux_observability.md) | UX & Observability | UI badges, proof tree, Grafana, OCI attestation |
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Policy Engine Architecture](../policy/README.md)
|
||||
- [TrustLattice Engine](../policy/design/policy-deterministic-evaluator.md)
|
||||
- [Offline Kit Documentation](../../24_OFFLINE_KIT.md)
|
||||
- [Air-Gap Controller](../airgap/README.md)
|
||||
- [Authority Key Rotation](../authority/README.md)
|
||||
@@ -1,8 +1,9 @@
|
||||
# Router · Messaging Transport over Valkey (Draft v0.1)
|
||||
# Router · Messaging Transport over Valkey
|
||||
|
||||
## Status
|
||||
- Draft; intended for implementation via a dedicated sprint.
|
||||
- Last updated: 2025-12-23 (UTC).
|
||||
- **Implemented** in Sprint 8100.0011.0003.
|
||||
- Core components: Gateway DI wiring, GatewayHostedService integration, GatewayTransportClient dispatch.
|
||||
- Last updated: 2025-12-24 (UTC).
|
||||
|
||||
## Purpose
|
||||
Enable Gateway ↔ microservice Router traffic over an offline-friendly, Redis-compatible transport (Valkey) by using the existing **Messaging** transport layer:
|
||||
@@ -11,15 +12,18 @@ Enable Gateway ↔ microservice Router traffic over an offline-friendly, Redis-c
|
||||
|
||||
This supports environments where direct TCP/TLS microservice connections are undesirable, and where an internal message bus is the preferred control plane.
|
||||
|
||||
## What Exists Today (Repository Reality)
|
||||
- Messaging transport server/client:
|
||||
- `src/__Libraries/StellaOps.Router.Transport.Messaging/MessagingTransportServer.cs`
|
||||
- `src/__Libraries/StellaOps.Router.Transport.Messaging/MessagingTransportClient.cs`
|
||||
- Valkey-backed message queue factory:
|
||||
- `src/__Libraries/StellaOps.Messaging.Transport.Valkey/ValkeyMessageQueueFactory.cs`
|
||||
- Gateway WebService currently starts only TCP/TLS transport servers:
|
||||
- `src/Gateway/StellaOps.Gateway.WebService/Program.cs`
|
||||
- `src/Gateway/StellaOps.Gateway.WebService/Services/GatewayHostedService.cs`
|
||||
## Implementation Summary
|
||||
|
||||
### Libraries
|
||||
- `StellaOps.Router.Transport.Messaging` — Router transport layer over messaging
|
||||
- `StellaOps.Messaging.Transport.Valkey` — Valkey/Redis backend for messaging
|
||||
- `StellaOps.Messaging` — Core messaging abstractions and DI
|
||||
|
||||
### Gateway Integration
|
||||
- **Program.cs** — Conditional registration of `ValkeyTransportPlugin` and `AddMessagingTransportServer()`
|
||||
- **GatewayOptions.cs** — `GatewayMessagingTransportOptions` for Valkey connection and queue configuration
|
||||
- **GatewayHostedService.cs** — Start/stop `MessagingTransportServer`, subscribe to events
|
||||
- **GatewayTransportClient.cs** — Dispatch to `TransportType.Messaging` connections
|
||||
|
||||
## High-Level Flow
|
||||
1) Microservice connects via messaging transport:
|
||||
@@ -39,14 +43,35 @@ The Messaging transport uses a small set of queues (names are configurable):
|
||||
- **Per-service request queues**: gateway publishes REQUEST frames targeted to a service
|
||||
- **Dead letter queues** (optional): for messages that exceed retries/leases
|
||||
|
||||
## Configuration (Draft)
|
||||
### Gateway
|
||||
- Register Valkey messaging services (`StellaOps.Messaging.Transport.Valkey`)
|
||||
- Add messaging transport server (`AddMessagingTransportServer`)
|
||||
- Add Gateway config section for messaging transport options:
|
||||
- Valkey connection info (host/port/auth)
|
||||
- queue naming prefix
|
||||
- consumer group / lease duration / dead-letter suffix
|
||||
## Configuration
|
||||
|
||||
### Gateway YAML Configuration
|
||||
```yaml
|
||||
Gateway:
|
||||
Transports:
|
||||
Messaging:
|
||||
Enabled: true
|
||||
ConnectionString: "valkey:6379"
|
||||
Database: 0
|
||||
RequestQueueTemplate: "router:requests:{service}"
|
||||
ResponseQueueName: "router:responses"
|
||||
ConsumerGroup: "router-gateway"
|
||||
RequestTimeout: "30s"
|
||||
LeaseDuration: "5m"
|
||||
BatchSize: 10
|
||||
HeartbeatInterval: "10s"
|
||||
```
|
||||
|
||||
### Gateway DI Registration
|
||||
```csharp
|
||||
// In Program.cs (already implemented)
|
||||
if (bootstrapOptions.Transports.Messaging.Enabled)
|
||||
{
|
||||
builder.Services.AddMessagingTransport<ValkeyTransportPlugin>(
|
||||
builder.Configuration, "Gateway:Transports:Messaging");
|
||||
builder.Services.AddMessagingTransportServer();
|
||||
}
|
||||
```
|
||||
|
||||
### Microservice
|
||||
- Register Valkey messaging services (`StellaOps.Messaging.Transport.Valkey`)
|
||||
@@ -63,13 +88,20 @@ The Messaging transport uses a small set of queues (names are configurable):
|
||||
- The Gateway must not trust client-supplied identity headers; it must overwrite reserved headers before dispatch.
|
||||
- See `docs/modules/gateway/identity-header-policy.md`.
|
||||
|
||||
## Gaps / Implementation Work
|
||||
1) Wire Messaging transport into Gateway:
|
||||
- start/stop `MessagingTransportServer`
|
||||
- subscribe to HELLO/HEARTBEAT/RESPONSE events and reuse existing HELLO validation + routing state updates
|
||||
2) Extend Gateway transport client to support `TransportType.Messaging` for dispatch.
|
||||
3) Add config mapping and deployment examples (compose/helm) for Valkey transport.
|
||||
4) Add integration tests covering:
|
||||
## Implementation Status
|
||||
|
||||
### Completed (Sprint 8100.0011.0003)
|
||||
1. ✅ Wire Messaging transport into Gateway:
|
||||
- start/stop `MessagingTransportServer` in `GatewayHostedService`
|
||||
- subscribe to `OnHelloReceived`, `OnHeartbeatReceived`, `OnResponseReceived`, `OnConnectionClosed` events
|
||||
- reuse routing state updates and claims store updates
|
||||
2. ✅ Extend Gateway transport client to support `TransportType.Messaging` for dispatch.
|
||||
3. ✅ Add config options (`GatewayMessagingTransportOptions`) and DI mappings.
|
||||
|
||||
### Remaining Work
|
||||
1. Add deployment examples (compose/helm) for Valkey transport.
|
||||
2. Add integration tests using ValkeyFixture:
|
||||
- microservice HELLO registration via messaging
|
||||
- request dispatch + response return
|
||||
3. Validate streaming support (or document as out-of-scope).
|
||||
|
||||
|
||||
@@ -93,7 +93,7 @@
|
||||
|----------|-------|----------|
|
||||
| `ruby_packages.json` | `RubyPackageInventory { scanId, imageDigest, generatedAt, packages[] }` where each package mirrors `{id, name, version, source, provenance, groups[], platform, runtime.*}` | SBOM Composer, Policy Engine |
|
||||
|
||||
`ruby_packages.json` records are persisted in Mongo’s `ruby.packages` collection via the `RubyPackageInventoryStore`. Scanner.WebService exposes the same payload through `GET /api/scans/{scanId}/ruby-packages` so Policy, CLI, and Offline Kit consumers can reuse the canonical inventory without re-running the analyzer. Each document is keyed by `scanId` and includes the resolved `imageDigest` plus the UTC timestamp recorded by the Worker.
|
||||
`ruby_packages.json` records are persisted in PostgreSQL's `scanner.ruby_packages` table via the `RubyPackageInventoryStore`. Scanner.WebService exposes the same payload through `GET /api/scans/{scanId}/ruby-packages` so Policy, CLI, and Offline Kit consumers can reuse the canonical inventory without re-running the analyzer. Each record is keyed by `scanId` and includes the resolved `imageDigest` plus the UTC timestamp recorded by the Worker.
|
||||
| `ruby_runtime_edges.json` | Edges `{from, to, reason, confidence}` | EntryTrace overlay, Policy explain traces |
|
||||
| `ruby_capabilities.json` | Capability `{kind, location, evidenceHash, params}` | Policy Engine (capability predicates) |
|
||||
| `ruby_observation.json` | Summary document (packages, runtime edges, capability flags) | Surface manifest, Policy explain traces |
|
||||
|
||||
@@ -37,7 +37,7 @@
|
||||
- `scanner.attestation.attestorEndpoint` → Attestor base URL.
|
||||
- `attestor.rekor.api` & `attestor.rekor.pubkey` set for the target log.
|
||||
3. **Storage**
|
||||
- Mongo collections `attestations` & `rekorProofs` sized for retention (7–30 days recommended).
|
||||
- PostgreSQL tables `attestations` & `rekor_proofs` sized for retention (7–30 days recommended).
|
||||
- Object store tier with at-rest encryption for DSSE payloads.
|
||||
4. **Observability**
|
||||
- Metrics: `attestor_rekor_success_total`, `attestor_rekor_retry_total`, `rekor_inclusion_latency`.
|
||||
|
||||
@@ -74,7 +74,7 @@ Restart the worker so the analyzer reloads the updated bundle. Bundles are immut
|
||||
- Scanner.Analysis.SecretFindingsTtl
|
||||
```
|
||||
|
||||
The migration adds `secretFindings` documents to `ScanAnalysisStore` with the standard TTL (default 90 days). Adjust Mongo TTL via the deployment overlay if longer retention is required.
|
||||
The migration adds `secretFindings` records to `ScanAnalysisStore` with the standard TTL (default 90 days). Adjust PostgreSQL retention policy via the deployment overlay if longer retention is required.
|
||||
|
||||
3. **Activate policy ingestion** (WebService):
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
| Phase V · Sprint 0134 (PHP fixtures/runtime/package) | Green | PHP analyzer fixtures, runtime evidence, and packaging shipped; docs updated. | Keep fixture hashes stable; rerun benchmarks when dependencies change. |
|
||||
| Phase VI · Sprint 0135 (Python container + Ruby VFS/edges) | Green | Python container/zipapp adapters shipped; Ruby VFS/dependency edges/observations/runtime capture packaged; EntryTrace 18-502/503 delivered. | Maintain determinism; re-run EntryTrace suite in CI. |
|
||||
| Phase VII · Sprint 0136 (EntryTrace surface/CLI) | Green | EntryTrace phase VII tasks 18-504/505/506 completed; CLI/WebService surfaces show best-terminal metadata and confidence. | Keep NDJSON schema stable; rerun worker payload tests in CI. |
|
||||
| Sprint 0138 (Ruby parity & future analyzers) | Amber/Red | Ruby parity shipped; Mongo package inventory live. PHP pipeline SCANNER-ENG-0010 blocked on composer/autoload design + restore stability (design at `docs/modules/scanner/design/php-autoload-design.md`); Deno scope drafted (`docs/modules/scanner/design/deno-analyzer-scope.md`); Dart/Swift scope drafted (`docs/modules/scanner/design/dart-swift-analyzer-scope.md`); Kubernetes/VM roadmap pending. | Implement PHP autoload parser/fixtures per design; add Deno fixtures and validation evidence; align with Zastava/Runtime and update readiness once fixtures land. |
|
||||
| Sprint 0138 (Ruby parity & future analyzers) | Amber/Red | Ruby parity shipped; PostgreSQL package inventory live. PHP pipeline SCANNER-ENG-0010 blocked on composer/autoload design + restore stability (design at `docs/modules/scanner/design/php-autoload-design.md`); Deno scope drafted (`docs/modules/scanner/design/deno-analyzer-scope.md`); Dart/Swift scope drafted (`docs/modules/scanner/design/dart-swift-analyzer-scope.md`); Kubernetes/VM roadmap pending. | Implement PHP autoload parser/fixtures per design; add Deno fixtures and validation evidence; align with Zastava/Runtime and update readiness once fixtures land. |
|
||||
|
||||
## Overall
|
||||
- Green areas: native analyzers, PHP fixtures/runtime packaging, Ruby analyzer, Python container adapters, EntryTrace phases VI–VII.
|
||||
|
||||
@@ -14,7 +14,7 @@ Scheduler detects advisory/VEX deltas, computes impact windows, and orchestrates
|
||||
- Shared libraries under `StellaOps.Scheduler.*`.
|
||||
|
||||
## Integrations & dependencies
|
||||
- MongoDB for impact models.
|
||||
- PostgreSQL (schema `scheduler`) for impact models.
|
||||
- Redis/NATS for queueing.
|
||||
- Policy Engine, Scanner, Notify.
|
||||
|
||||
|
||||
215
docs/modules/signals/architecture.md
Normal file
215
docs/modules/signals/architecture.md
Normal file
@@ -0,0 +1,215 @@
|
||||
# Signals Module Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
The **Signals** module provides a unified evidence-weighted scoring system for vulnerability findings. It aggregates evidence from multiple sources (reachability analysis, runtime observations, backport detection, exploit intelligence, source trust, and mitigations) into a single 0-100 score that enables rapid triage.
|
||||
|
||||
## Module Purpose
|
||||
|
||||
- **Unify Scoring:** Combine disparate evidence signals into one actionable score
|
||||
- **Enable Triage:** Support sorting, filtering, and prioritization by evidence strength
|
||||
- **Maintain Determinism:** Same inputs + policy = same score, always
|
||||
- **Provide Transparency:** Decomposable scores with explanations
|
||||
|
||||
## Location
|
||||
|
||||
```
|
||||
src/Signals/
|
||||
├── StellaOps.Signals/
|
||||
│ └── EvidenceWeightedScore/
|
||||
│ ├── EvidenceWeightedScoreCalculator.cs
|
||||
│ ├── EvidenceWeightedScoreInput.cs
|
||||
│ ├── EvidenceWeightedScoreResult.cs
|
||||
│ ├── Normalizers/
|
||||
│ │ ├── BackportEvidenceNormalizer.cs
|
||||
│ │ ├── ExploitLikelihoodNormalizer.cs
|
||||
│ │ ├── MitigationNormalizer.cs
|
||||
│ │ ├── ReachabilityNormalizer.cs
|
||||
│ │ ├── RuntimeSignalNormalizer.cs
|
||||
│ │ └── SourceTrustNormalizer.cs
|
||||
│ ├── Guardrails/
|
||||
│ │ └── ScoreGuardrails.cs
|
||||
│ └── Policy/
|
||||
│ ├── EvidenceWeightPolicy.cs
|
||||
│ └── EvidenceWeightPolicyProvider.cs
|
||||
└── __Tests/
|
||||
└── StellaOps.Signals.Tests/
|
||||
└── EvidenceWeightedScore/
|
||||
```
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### Evidence Dimensions
|
||||
|
||||
| Dimension | Symbol | Description | Source Module |
|
||||
|-----------|--------|-------------|---------------|
|
||||
| Reachability | RCH | Code path reachability to vulnerable sink | Policy |
|
||||
| Runtime | RTS | Live observation strength (eBPF/dyld/ETW) | Policy |
|
||||
| Backport | BKP | Patch evidence from distro/changelog/binary | Concelier |
|
||||
| Exploit | XPL | Exploit probability (EPSS + KEV) | Scanner, Concelier |
|
||||
| Source Trust | SRC | VEX source trustworthiness | Excititor |
|
||||
| Mitigations | MIT | Active protective controls | Policy |
|
||||
|
||||
### Scoring Formula
|
||||
|
||||
```
|
||||
Score = clamp01(W_rch*RCH + W_rts*RTS + W_bkp*BKP + W_xpl*XPL + W_src*SRC - W_mit*MIT) * 100
|
||||
```
|
||||
|
||||
Note: MIT is **subtractive** — mitigations reduce risk.
|
||||
|
||||
### Guardrails
|
||||
|
||||
Hard caps and floors based on evidence conditions:
|
||||
|
||||
1. **Not-Affected Cap:** If vendor says not-affected (BKP=1) and no runtime contradiction (RTS<0.6), cap at 15
|
||||
2. **Runtime Floor:** If strong live signal (RTS>=0.8), floor at 60
|
||||
3. **Speculative Cap:** If no reachability and no runtime (RCH=0, RTS=0), cap at 45
|
||||
|
||||
### Score Buckets
|
||||
|
||||
| Bucket | Range | Meaning |
|
||||
|--------|-------|---------|
|
||||
| ActNow | 90-100 | Strong evidence of exploitable risk; immediate action |
|
||||
| ScheduleNext | 70-89 | Likely real; schedule for next sprint |
|
||||
| Investigate | 40-69 | Moderate evidence; investigate when touching component |
|
||||
| Watchlist | 0-39 | Low/insufficient evidence; monitor |
|
||||
|
||||
## Architecture
|
||||
|
||||
### Component Diagram
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Signals Module │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────────────┐ ┌──────────────────────────────────┐ │
|
||||
│ │ NormalizerAggr. │───▶│ EvidenceWeightedScoreCalculator │ │
|
||||
│ └────────┬────────┘ └──────────────┬───────────────────┘ │
|
||||
│ │ │ │
|
||||
│ ▼ ▼ │
|
||||
│ ┌─────────────────┐ ┌──────────────────────────────────┐ │
|
||||
│ │ Normalizers │ │ ScoreGuardrails │ │
|
||||
│ │ ┌─────┐ ┌─────┐ │ └──────────────────────────────────┘ │
|
||||
│ │ │ BKP │ │ XPL │ │ │
|
||||
│ │ └─────┘ └─────┘ │ ┌──────────────────────────────────┐ │
|
||||
│ │ ┌─────┐ ┌─────┐ │ │ EvidenceWeightPolicyProvider │ │
|
||||
│ │ │ MIT │ │ RCH │ │ └──────────────────────────────────┘ │
|
||||
│ │ └─────┘ └─────┘ │ │
|
||||
│ │ ┌─────┐ ┌─────┐ │ │
|
||||
│ │ │ RTS │ │ SRC │ │ │
|
||||
│ │ └─────┘ └─────┘ │ │
|
||||
│ └─────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────┐ ┌───────────────────────────────────────┐
|
||||
│ External Modules │ │ Consumers │
|
||||
│ │ │ │
|
||||
│ • Policy (RCH, RTS) │ │ • Policy Engine (verdict enrichment) │
|
||||
│ • Concelier (BKP) │ │ • Findings API (score endpoints) │
|
||||
│ • Scanner (XPL) │ │ • Web UI (score components) │
|
||||
│ • Excititor (SRC) │ │ │
|
||||
└─────────────────────┘ └───────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Data Flow
|
||||
|
||||
1. **Finding arrives** for scoring
|
||||
2. **NormalizerAggregator** collects evidence from source modules
|
||||
3. **Individual normalizers** convert raw evidence to 0-1 values
|
||||
4. **EvidenceWeightedScoreCalculator** applies formula
|
||||
5. **ScoreGuardrails** apply caps/floors
|
||||
6. **Result** returned with score, bucket, breakdown, explanations
|
||||
|
||||
### Policy Configuration
|
||||
|
||||
Weight policies are loaded from YAML and can vary by:
|
||||
- **Tenant:** Different organizations may prioritize differently
|
||||
- **Environment:** Production may weight runtime higher than dev
|
||||
- **Policy version:** Changes are versioned for reproducibility
|
||||
|
||||
```yaml
|
||||
version: "ews.v1"
|
||||
profile: production
|
||||
|
||||
weights:
|
||||
rch: 0.30
|
||||
rts: 0.25
|
||||
bkp: 0.15
|
||||
xpl: 0.15
|
||||
src: 0.10
|
||||
mit: 0.10
|
||||
|
||||
guardrails:
|
||||
not_affected_cap:
|
||||
enabled: true
|
||||
max_score: 15
|
||||
runtime_floor:
|
||||
enabled: true
|
||||
min_score: 60
|
||||
speculative_cap:
|
||||
enabled: true
|
||||
max_score: 45
|
||||
```
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Inbound (Evidence Sources)
|
||||
|
||||
| Source | Data | Interface |
|
||||
|--------|------|-----------|
|
||||
| Policy/ConfidenceCalculator | ReachabilityEvidence, RuntimeEvidence | Direct type reference |
|
||||
| Concelier/BackportProofService | ProofBlob | Direct type reference |
|
||||
| Scanner/EpssPriorityCalculator | EPSS score, percentile | Direct type reference |
|
||||
| Concelier/VendorRiskSignalExtractor | KEV status | Direct type reference |
|
||||
| Excititor/TrustVector | Trust vector components | Direct type reference |
|
||||
| Policy/GateMultipliers | Active mitigations | Direct type reference |
|
||||
|
||||
### Outbound (Consumers)
|
||||
|
||||
| Consumer | Usage |
|
||||
|----------|-------|
|
||||
| Policy Engine | Verdict enrichment, score-based rules |
|
||||
| Findings API | Score endpoints (calculate, get, history) |
|
||||
| Web UI | Score display components |
|
||||
| Webhooks | Score change notifications |
|
||||
| Attestor | Scoring proofs in attestations |
|
||||
|
||||
## Determinism
|
||||
|
||||
The Signals module maintains strict determinism:
|
||||
|
||||
1. **Same inputs + same policy = same score:** No randomness, no time-dependent factors in formula
|
||||
2. **Policy versioning:** Every policy has a digest; scores include policy digest
|
||||
3. **Reproducible proofs:** Scoring proofs can be verified by recalculation
|
||||
4. **Stable ordering:** Input order doesn't affect result
|
||||
|
||||
## Performance
|
||||
|
||||
| Metric | Target |
|
||||
|--------|--------|
|
||||
| Single score calculation | <10ms |
|
||||
| Batch (100 findings) | <1s |
|
||||
| Policy load | <100ms |
|
||||
| Cache hit ratio | >80% |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
| Level | Focus | Location |
|
||||
|-------|-------|----------|
|
||||
| L0 Unit | Calculator, normalizers, guardrails | `*.Tests` |
|
||||
| L0 Property | Monotonicity, bounds, determinism | `*.Tests/Properties` |
|
||||
| S1 Integration | Cross-module evidence flow | `*.Tests/Integration` |
|
||||
| S1 Snapshot | Result JSON structure | `*.Tests/Snapshots` |
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- Product Advisory: `docs/product-advisories/24-Dec-2025 - Evidence-Weighted Score Model.md`
|
||||
- Sprint Plans: `docs/implplan/SPRINT_8200_0012_*.md`
|
||||
- Policy Confidence (deprecated): `docs/modules/policy/confidence-scoring.md`
|
||||
- Backport Detection: `docs/modules/concelier/backport-detection.md`
|
||||
- EPSS Enrichment: `docs/modules/scanner/epss-enrichment.md`
|
||||
- Trust Vector: `docs/modules/excititor/trust-vector.md`
|
||||
@@ -400,7 +400,7 @@ signer:
|
||||
|
||||
* Run ≥ 2 replicas; front with L7 LB; **sticky** not required.
|
||||
* Redis for replay/quota caches (HA).
|
||||
* Audit sink (Mongo/Postgres) in primary region; asynchronous write with local fallback buffer.
|
||||
* Audit sink (PostgreSQL) in primary region; asynchronous write with local fallback buffer.
|
||||
* Fulcio/KMS clients configured with retries/backoff; circuit breakers.
|
||||
|
||||
---
|
||||
|
||||
@@ -1,99 +0,0 @@
|
||||
# Task Runner Collections — Initial Migration
|
||||
|
||||
Last updated: 2025-11-06
|
||||
|
||||
This migration seeds the MongoDB collections that back the Task Runner service. It is implemented as `20251106-task-runner-baseline.mongosh` under the platform migration runner and must be applied **before** enabling the TaskRunner service in any environment.
|
||||
|
||||
## Collections
|
||||
|
||||
### `pack_runs`
|
||||
|
||||
| Field | Type | Notes |
|
||||
|------------------|-----------------|-----------------------------------------------------------|
|
||||
| `_id` | `string` | Run identifier (same as `runId`). |
|
||||
| `planHash` | `string` | Deterministic hash produced by the planner. |
|
||||
| `plan` | `object` | Full `TaskPackPlan` payload used to execute the run. |
|
||||
| `failurePolicy` | `object` | Retry/backoff directives resolved at plan time. |
|
||||
| `requestedAt` | `date` | Timestamp when the client requested the run. |
|
||||
| `createdAt` | `date` | Timestamp when the run was persisted. |
|
||||
| `updatedAt` | `date` | Timestamp of the last mutation. |
|
||||
| `steps` | `array<object>` | Flattened step records (`stepId`, `status`, attempts…). |
|
||||
| `tenantId` | `string` | Optional multi-tenant scope (reserved for future phases). |
|
||||
|
||||
**Indexes**
|
||||
|
||||
1. `{ _id: 1 }` — implicit primary key / uniqueness guarantee.
|
||||
2. `{ updatedAt: -1 }` — serves `GET /runs` listings and staleness checks.
|
||||
3. `{ tenantId: 1, updatedAt: -1 }` — activated once tenancy is enforced; remains sparse until then.
|
||||
|
||||
### `pack_run_logs`
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---------------|-----------------|--------------------------------------------------------|
|
||||
| `_id` | `ObjectId` | Generated per log entry. |
|
||||
| `runId` | `string` | Foreign key to `pack_runs._id`. |
|
||||
| `sequence` | `long` | Monotonic counter assigned by the writer. |
|
||||
| `timestamp` | `date` | UTC timestamp of the log event. |
|
||||
| `level` | `string` | `trace`, `debug`, `info`, `warn`, `error`. |
|
||||
| `eventType` | `string` | Machine-friendly event identifier (e.g. `step.started`). |
|
||||
| `message` | `string` | Human-readable summary. |
|
||||
| `stepId` | `string` | Optional step identifier. |
|
||||
| `metadata` | `object` | Deterministic key/value payload (string-only values). |
|
||||
|
||||
**Indexes**
|
||||
|
||||
1. `{ runId: 1, sequence: 1 }` (unique) — guarantees ordered retrieval and enforces idempotence.
|
||||
2. `{ runId: 1, timestamp: 1 }` — accelerates replay and time-window queries.
|
||||
3. `{ timestamp: 1 }` — optional TTL (disabled by default) for retention policies.
|
||||
|
||||
### `pack_artifacts`
|
||||
|
||||
| Field | Type | Notes |
|
||||
|--------------|------------|-------------------------------------------------------------|
|
||||
| `_id` | `ObjectId` | Generated per artifact record. |
|
||||
| `runId` | `string` | Foreign key to `pack_runs._id`. |
|
||||
| `name` | `string` | Output name from the Task Pack manifest. |
|
||||
| `type` | `string` | `file`, `object`, or other future evidence categories. |
|
||||
| `sourcePath` | `string` | Local path captured during execution (nullable). |
|
||||
| `storedPath` | `string` | Object store path or bundle-relative URI (nullable). |
|
||||
| `status` | `string` | `pending`, `copied`, `materialized`, `skipped`. |
|
||||
| `notes` | `string` | Free-form notes (deterministic messages only). |
|
||||
| `capturedAt` | `date` | UTC timestamp recorded by the worker. |
|
||||
|
||||
**Indexes**
|
||||
|
||||
1. `{ runId: 1, name: 1 }` (unique) — ensures a run emits at most one record per output.
|
||||
2. `{ runId: 1 }` — supports artifact listing alongside run inspection.
|
||||
|
||||
## Execution Order
|
||||
|
||||
1. Create collections with `validator` envelopes mirroring the field expectations above (if MongoDB schema validation is enabled in the environment).
|
||||
2. Apply the indexes in the order listed — unique indexes first to surface data issues early.
|
||||
3. Backfill existing filesystem-backed runs by importing the serialized state/log/artifact manifests into the new collections. A dedicated importer script (`tools/taskrunner/import-filesystem-state.ps1`) accompanies the migration.
|
||||
4. Switch the Task Runner service configuration to point at the Mongo-backed stores (`TaskRunner:Storage:Mode = "Mongo"`), then redeploy workers and web service.
|
||||
|
||||
## Rollback
|
||||
|
||||
To revert, switch the Task Runner configuration back to the filesystem provider and stop the Mongo migration runner. Collections can remain in place; they are append-only and harmless when unused.
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
Enable the Mongo-backed stores by updating the worker and web service configuration (Compose/Helm values or `appsettings*.json`):
|
||||
|
||||
```json
|
||||
"TaskRunner": {
|
||||
"Storage": {
|
||||
"Mode": "mongo",
|
||||
"Mongo": {
|
||||
"ConnectionString": "mongodb://127.0.0.1:27017/taskrunner",
|
||||
"Database": "taskrunner",
|
||||
"RunsCollection": "pack_runs",
|
||||
"LogsCollection": "pack_run_logs",
|
||||
"ArtifactsCollection": "pack_artifacts",
|
||||
"ApprovalsCollection": "pack_run_approvals"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The worker uses the mirrored structure under the `Worker` section. Omit the `Database` property to fall back to the name embedded in the connection string.
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
> **Ownership:** Console Guild • Docs Guild
|
||||
> **Delivery scope:** `StellaOps.Web` Angular workspace, Console Web Gateway routes (`/console/*`), Downloads manifest surfacing, SSE fan-out for Scheduler & telemetry.
|
||||
> **Related docs:** [Console operator guide](../../15_UI_GUIDE.md), [Admin workflows](../../console/admin-tenants.md), [Air-gap workflows](../../console/airgap.md), [Console security posture](../../security/console-security.md), [Console observability](../../console/observability.md), [UI telemetry](../../observability/ui-telemetry.md), [Deployment guide](../../deploy/console.md)
|
||||
> **Related docs:** [Console operator guide](../../15_UI_GUIDE.md), [Admin workflows](../../console/admin-tenants.md), [Air-gap workflows](../../console/airgap.md), [Console security posture](../../security/console-security.md), [Console observability](../../console/observability.md), [UI telemetry](../../observability/ui-telemetry.md), [Deployment guide](../../deploy/console.md)
|
||||
|
||||
This dossier describes the end-to-end architecture of the StellaOps Console as delivered in Sprint 23. It covers the Angular workspace layout, API/gateway integration points, live-update channels, performance budgets, offline workflows, and observability hooks needed to keep the console deterministic and air-gap friendly.
|
||||
|
||||
@@ -18,7 +18,7 @@ This dossier describes the end-to-end architecture of the StellaOps Console as d
|
||||
- Stream live status (ingestion deltas, scheduler progress, telemetry) via SSE with graceful degradation to polling when offline or throttled.
|
||||
- Maintain CLI parity by embedding `stella` commands alongside interactive actions.
|
||||
|
||||
Non-goals: authoring ingestion logic, mutating Policy overlays, exposing internal Mongo collections, or performing cryptographic signing in-browser.
|
||||
Non-goals: authoring ingestion logic, mutating Policy overlays, exposing internal database tables, or performing cryptographic signing in-browser.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ VEX Lens produces a deterministic, provenance-rich consensus view of VEX stateme
|
||||
- Provide simulation/export APIs and Offline Kit bundles so tenants can rehearse policy changes and mirror consensus data in air-gapped environments.
|
||||
|
||||
## Architecture snapshot (Sprint 30 groundwork)
|
||||
- **StellaOps.VexLens service & workers** — orchestrate normalisation, trust weighting, lattice join, and persistence into `vex_consensus`, `vex_consensus_history`, and `vex_conflict_queue` collections.
|
||||
- **StellaOps.VexLens service & workers** — orchestrate normalisation, trust weighting, lattice join, and persistence into `vex_consensus`, `vex_consensus_history`, and `vex_conflict_queue` tables (PostgreSQL).
|
||||
- **Issuer Directory integration** — maintains publisher metadata, keys, and trust tiers that feed weighting engines and revocation workflows.
|
||||
- **Consensus APIs** — `/v1/vex/consensus`, `/v1/vex/conflicts`, `/v1/vex/trust/weights`, and export streams with DSSE manifests for Offline Kit + Export Center.
|
||||
- **Explainability traces** — capture derived-from chains, conflicting issuers, and trust deltas to power UI drilldowns and CLI audits.
|
||||
@@ -39,7 +39,7 @@ VEX Lens produces a deterministic, provenance-rich consensus view of VEX stateme
|
||||
- **Notify / Task Runner** receive conflict and override events for operator actions once notification bridges ship.
|
||||
|
||||
## Data & observability
|
||||
- Collections: `vex_consensus`, `vex_consensus_history`, `vex_conflict_queue`, plus issuer registry tables managed with tenant isolation and deterministic indexes.
|
||||
- PostgreSQL tables: `vex_consensus`, `vex_consensus_history`, `vex_conflict_queue`, plus issuer registry tables managed with tenant isolation and deterministic indexes.
|
||||
- Metrics: `vex_consensus_conflicts_total`, `vex_consensus_latency_seconds`, `vex_consensus_recompute_seconds{reason}`, signature failure counters.
|
||||
- Traces/logs: `consensus.group`, `consensus.join`, `consensus.persist` spans with correlation IDs and issuer details; structured logs capture trust adjustments and reconciliation outcomes.
|
||||
- Offline bundles include `consensus.jsonl`, `conflicts.jsonl`, manifest + DSSE signatures, enabling mirror deployments and replay validation.
|
||||
@@ -47,7 +47,9 @@ VEX Lens produces a deterministic, provenance-rich consensus view of VEX stateme
|
||||
## Key docs & references
|
||||
- [`architecture.md`](architecture.md) — implementation-ready blueprint covering inputs, algorithm, APIs, storage, observability, and exports.
|
||||
- [`implementation_plan.md`](implementation_plan.md) — phased delivery roadmap and acceptance criteria.
|
||||
- [`scoring.md`](scoring.md) — future risk scoring model and formula reference.
|
||||
- [`../../vex/aggregation.md`](../../vex/aggregation.md) — Aggregation-Only Contract boundaries for VEX ingestion and downstream consumers.
|
||||
- **Operations:** [`operations/deployment.md`](operations/deployment.md), [`operations/offline-kit.md`](operations/offline-kit.md) — deployment guides and offline bundle preparation.
|
||||
- Sprint tracking in `docs/implplan/SPRINT_0332_0001_0001_docs_modules_vex_lens.md`; module engineering tasks in `src/VexLens/StellaOps.VexLens/TASKS.md`; doc TASKS mirror in `docs/modules/vex-lens/TASKS.md`.
|
||||
|
||||
## Epic alignment
|
||||
|
||||
@@ -192,10 +192,10 @@ vexlens:
|
||||
|
||||
storage:
|
||||
# Local storage only
|
||||
mongodb:
|
||||
connectionString: mongodb://localhost:27017
|
||||
postgresql:
|
||||
connectionString: Host=localhost;Database=stellaops;Username=stellaops;Password=password
|
||||
# No external cache
|
||||
redis:
|
||||
valkey:
|
||||
enabled: false
|
||||
|
||||
time:
|
||||
@@ -1,319 +0,0 @@
|
||||
# component_architecture_vexlens.md — **Stella Ops VexLens** (2025Q4)
|
||||
|
||||
> Supports deliverables from Epic 30 – VEX Consensus Engine and Epic 31 – Advisory AI Integration.
|
||||
|
||||
> **Scope.** Implementation-ready architecture for **VexLens**: the consensus engine for computing authoritative VEX (Vulnerability Exploitability eXchange) status from multiple overlapping statements. It supports trust-weighted voting, lattice-based conflict resolution, and provides policy integration for vulnerability decisioning.
|
||||
|
||||
---
|
||||
|
||||
## 0) Mission & Boundaries
|
||||
|
||||
**Mission.** Compute deterministic VEX consensus status from multiple sources with full audit trail, enabling automated vulnerability triage based on exploitability data.
|
||||
|
||||
**Boundaries.**
|
||||
|
||||
* **VexLens does not fetch VEX documents** — it receives normalized statements from Excititor or direct API input.
|
||||
* **VexLens does not store raw VEX documents** — it stores computed projections and consensus results.
|
||||
* **VexLens does not make policy decisions** — it provides VEX status to Policy Engine for final determination.
|
||||
|
||||
---
|
||||
|
||||
## 1) Responsibilities (contract)
|
||||
|
||||
1. **Normalize** VEX documents from OpenVEX, CSAF VEX, CycloneDX VEX, and SPDX VEX formats.
|
||||
2. **Map products** using PURL and CPE identifiers with configurable matching strictness.
|
||||
3. **Verify signatures** on VEX documents (DSSE, JWS, PGP, PKCS#7).
|
||||
4. **Compute trust weights** based on issuer authority, signature status, freshness, and other factors.
|
||||
5. **Compute consensus** using configurable modes:
|
||||
- **HighestWeight**: Single highest-weighted statement wins
|
||||
- **WeightedVote**: Weighted voting among all statements
|
||||
- **Lattice**: Most conservative status wins (affected > under_investigation > not_affected > fixed)
|
||||
- **AuthoritativeFirst**: Authoritative sources override others
|
||||
- **MostRecent**: Most recent statement wins
|
||||
6. **Store projections** for historical tracking and audit.
|
||||
7. **Emit events** on consensus computation, status changes, and conflict detection.
|
||||
8. **Integrate** with Policy Engine for vulnerability suppression and severity adjustment.
|
||||
|
||||
---
|
||||
|
||||
## 2) External Dependencies
|
||||
|
||||
* **Excititor**: Provides normalized VEX statements from connectors.
|
||||
* **Policy Engine**: Consumes VEX consensus for vulnerability decisioning.
|
||||
* **Vuln Explorer**: Enriches vulnerability data with VEX status.
|
||||
* **Orchestrator**: Schedules consensus compute jobs for batch processing.
|
||||
* **Authority**: Validates issuer trust and key fingerprints.
|
||||
* **Config stores**: PostgreSQL (projections, issuer directory), Redis (caches).
|
||||
|
||||
---
|
||||
|
||||
## 3) API Surface
|
||||
|
||||
Base path: `/api/v1/vexlens`. Full OpenAPI spec at `docs/api/vexlens-openapi.yaml`.
|
||||
|
||||
### 3.1 Consensus Operations
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/consensus` | POST | Compute consensus for a vulnerability-product pair |
|
||||
| `/consensus/batch` | POST | Compute consensus for multiple pairs in batch |
|
||||
|
||||
### 3.2 Projection Queries
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/projections` | GET | Query consensus projections with filtering |
|
||||
| `/projections/{projectionId}` | GET | Get a projection by ID |
|
||||
| `/projections/latest` | GET | Get latest projection for a vuln-product pair |
|
||||
| `/projections/history` | GET | Get projection history |
|
||||
|
||||
### 3.3 Issuer Directory
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/issuers` | GET | List registered issuers |
|
||||
| `/issuers` | POST | Register a new issuer |
|
||||
| `/issuers/{issuerId}` | GET | Get issuer details |
|
||||
| `/issuers/{issuerId}` | DELETE | Revoke an issuer |
|
||||
| `/issuers/{issuerId}/keys` | POST | Add a key to an issuer |
|
||||
| `/issuers/{issuerId}/keys/{fingerprint}` | DELETE | Revoke a key |
|
||||
|
||||
### 3.4 Statistics
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/statistics` | GET | Get consensus statistics |
|
||||
|
||||
---
|
||||
|
||||
## 4) Data Flow
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
|
||||
│ Excititor │────▶│ Normalizer │────▶│ Trust Weighting │
|
||||
│ (VEX Docs) │ │ (OpenVEX, │ │ (9 factors) │
|
||||
└─────────────┘ │ CSAF, CDX) │ └────────┬────────┘
|
||||
└──────────────┘ │
|
||||
▼
|
||||
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
|
||||
│ Policy │◀────│ Projection │◀────│ Consensus │
|
||||
│ Engine │ │ Store │ │ Engine │
|
||||
└─────────────┘ └──────────────┘ └─────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Events │
|
||||
│ (Computed, │
|
||||
│ StatusChange,│
|
||||
│ Conflict) │
|
||||
└──────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5) VEX Status Lattice
|
||||
|
||||
VexLens uses a status lattice for conservative conflict resolution:
|
||||
|
||||
```
|
||||
affected (most restrictive)
|
||||
│
|
||||
▼
|
||||
under_investigation
|
||||
│
|
||||
▼
|
||||
not_affected
|
||||
│
|
||||
▼
|
||||
fixed (least restrictive)
|
||||
```
|
||||
|
||||
In lattice mode, the most restrictive status always wins. This ensures that when sources disagree, the system errs on the side of caution.
|
||||
|
||||
---
|
||||
|
||||
## 6) Trust Weight Factors
|
||||
|
||||
| Factor | Weight | Description |
|
||||
|--------|--------|-------------|
|
||||
| IssuerBase | 25% | Base trust from issuer directory |
|
||||
| SignatureStatus | 15% | Valid/invalid/unsigned signature |
|
||||
| Freshness | 15% | Document age with exponential decay |
|
||||
| IssuerCategory | 10% | Vendor > Distributor > Aggregator |
|
||||
| IssuerTier | 10% | Authoritative > Trusted > Untrusted |
|
||||
| StatusQuality | 10% | Has justification, specific status |
|
||||
| TransparencyLog | 5% | Sigstore Rekor entry |
|
||||
| SourceMatch | 5% | Source URI pattern match |
|
||||
| ProductAuthority | 5% | Issuer is authoritative for product |
|
||||
|
||||
---
|
||||
|
||||
## 7) Configuration
|
||||
|
||||
```yaml
|
||||
vexlens:
|
||||
consensus:
|
||||
defaultMode: WeightedVote # HighestWeight, WeightedVote, Lattice, AuthoritativeFirst, MostRecent
|
||||
minimumConfidence: 0.1
|
||||
conflictThreshold: 0.3
|
||||
requireJustificationForNotAffected: false
|
||||
trust:
|
||||
freshnessHalfLifeDays: 90
|
||||
minimumFreshness: 0.3
|
||||
allowUnsigned: true
|
||||
unsignedPenalty: 0.3
|
||||
allowUnknownIssuers: true
|
||||
unknownIssuerPenalty: 0.5
|
||||
storage:
|
||||
projectionRetentionDays: 365
|
||||
eventRetentionDays: 90
|
||||
issuerDirectory:
|
||||
source: postgresql # postgresql, file, api
|
||||
refreshIntervalMinutes: 60
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8) Storage Schema
|
||||
|
||||
### 8.1 Consensus Projection
|
||||
|
||||
```json
|
||||
{
|
||||
"projectionId": "proj-abc123",
|
||||
"vulnerabilityId": "CVE-2024-1234",
|
||||
"productKey": "pkg:npm/lodash@4.17.21",
|
||||
"tenantId": "tenant-001",
|
||||
"status": "not_affected",
|
||||
"justification": "vulnerable_code_not_present",
|
||||
"confidenceScore": 0.95,
|
||||
"outcome": "Unanimous",
|
||||
"statementCount": 3,
|
||||
"conflictCount": 0,
|
||||
"rationaleSummary": "Unanimous consensus from 3 authoritative sources",
|
||||
"computedAt": "2025-12-06T12:00:00Z",
|
||||
"storedAt": "2025-12-06T12:00:01Z",
|
||||
"previousProjectionId": null,
|
||||
"statusChanged": true
|
||||
}
|
||||
```
|
||||
|
||||
### 8.2 Issuer Record
|
||||
|
||||
```json
|
||||
{
|
||||
"issuerId": "npm-security",
|
||||
"name": "npm Security Team",
|
||||
"category": "Vendor",
|
||||
"trustTier": "Authoritative",
|
||||
"status": "Active",
|
||||
"keyFingerprints": [
|
||||
{
|
||||
"fingerprint": "ABCD1234EFGH5678",
|
||||
"keyType": "Pgp",
|
||||
"algorithm": "EdDSA",
|
||||
"status": "Active",
|
||||
"registeredAt": "2025-01-01T00:00:00Z",
|
||||
"expiresAt": null
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"description": "Official npm security advisories",
|
||||
"uri": "https://www.npmjs.com/advisories",
|
||||
"email": "security@npmjs.com"
|
||||
},
|
||||
"registeredAt": "2025-01-01T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9) Events
|
||||
|
||||
### 9.1 ConsensusComputedEvent
|
||||
|
||||
Emitted after every consensus computation.
|
||||
|
||||
```json
|
||||
{
|
||||
"eventId": "evt-abc123",
|
||||
"projectionId": "proj-abc123",
|
||||
"vulnerabilityId": "CVE-2024-1234",
|
||||
"productKey": "pkg:npm/lodash@4.17.21",
|
||||
"status": "not_affected",
|
||||
"confidenceScore": 0.95,
|
||||
"outcome": "Unanimous",
|
||||
"statementCount": 3,
|
||||
"computedAt": "2025-12-06T12:00:00Z",
|
||||
"emittedAt": "2025-12-06T12:00:01Z"
|
||||
}
|
||||
```
|
||||
|
||||
### 9.2 ConsensusStatusChangedEvent
|
||||
|
||||
Emitted when consensus status changes from previous projection.
|
||||
|
||||
### 9.3 ConsensusConflictDetectedEvent
|
||||
|
||||
Emitted when conflicts are detected during consensus computation.
|
||||
|
||||
---
|
||||
|
||||
## 10) Observability
|
||||
|
||||
### 10.1 Metrics (OpenTelemetry)
|
||||
|
||||
| Metric | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `vexlens.consensus.computed_total` | Counter | Total consensus computations |
|
||||
| `vexlens.consensus.conflicts_total` | Counter | Total conflicts detected |
|
||||
| `vexlens.consensus.confidence` | Histogram | Confidence score distribution |
|
||||
| `vexlens.consensus.duration_seconds` | Histogram | Computation duration |
|
||||
| `vexlens.consensus.status_changes_total` | Counter | Status changes detected |
|
||||
| `vexlens.normalization.documents_total` | Counter | Documents normalized |
|
||||
| `vexlens.trust.weight_value` | Histogram | Trust weight distribution |
|
||||
| `vexlens.issuer.registered_total` | Counter | Issuers registered |
|
||||
|
||||
### 10.2 Traces
|
||||
|
||||
Activity source: `StellaOps.VexLens`
|
||||
|
||||
| Activity | Description |
|
||||
|----------|-------------|
|
||||
| `vexlens.normalize` | VEX document normalization |
|
||||
| `vexlens.compute_trust_weight` | Trust weight computation |
|
||||
| `vexlens.compute_consensus` | Consensus computation |
|
||||
| `vexlens.store_projection` | Projection storage |
|
||||
| `vexlens.query_projections` | Projection query |
|
||||
|
||||
### 10.3 Logging
|
||||
|
||||
Structured logging with event IDs in `VexLensLogEvents`:
|
||||
- 1xxx: Normalization events
|
||||
- 2xxx: Product mapping events
|
||||
- 3xxx: Signature verification events
|
||||
- 4xxx: Trust weight events
|
||||
- 5xxx: Consensus events
|
||||
- 6xxx: Projection events
|
||||
- 7xxx: Issuer directory events
|
||||
|
||||
---
|
||||
|
||||
## 11) Security Considerations
|
||||
|
||||
1. **Issuer Trust**: All issuers must be registered with verified key fingerprints.
|
||||
2. **Signature Verification**: Documents should be cryptographically signed for production use.
|
||||
3. **Tenant Isolation**: Projections are scoped to tenants; no cross-tenant data access.
|
||||
4. **Audit Trail**: All consensus computations are logged with full rationale.
|
||||
5. **Determinism**: All computations are deterministic for reproducibility.
|
||||
|
||||
---
|
||||
|
||||
## 12) Test Matrix
|
||||
|
||||
| Test Category | Coverage | Notes |
|
||||
|---------------|----------|-------|
|
||||
| Unit tests | Normalizer, Parser, Trust, Consensus | 89+ tests |
|
||||
| Determinism harness | Normalization, Trust, Consensus | Verify reproducibility |
|
||||
| Integration tests | API service, Storage, Events | End-to-end flows |
|
||||
| Property-based tests | Lattice semantics, Weight computation | Invariant verification |
|
||||
@@ -60,7 +60,7 @@ CLI mirrors these endpoints (`stella findings list|view|update|export`). Console
|
||||
## 6) Identity & access integration
|
||||
|
||||
- **Scopes** – `vuln:view`, `vuln:investigate`, `vuln:operate`, `vuln:audit` map to read-only, triage, workflow, and audit experiences respectively. The deprecated `vuln:read` scope is still honoured for legacy tokens but is no longer advertised.
|
||||
- **Attribute filters (ABAC)** – Authority enforces per-service-account filters via the client-credential parameters `vuln_env`, `vuln_owner`, and `vuln_business_tier`. Service accounts define the allowed values in `authority.yaml` (`attributes` block). Tokens include the resolved filters as claims (`stellaops:vuln_env`, `stellaops:vuln_owner`, `stellaops:vuln_business_tier`), and tokens persisted to Mongo retain the same values for audit and revocation.
|
||||
- **Attribute filters (ABAC)** – Authority enforces per-service-account filters via the client-credential parameters `vuln_env`, `vuln_owner`, and `vuln_business_tier`. Service accounts define the allowed values in `authority.yaml` (`attributes` block). Tokens include the resolved filters as claims (`stellaops:vuln_env`, `stellaops:vuln_owner`, `stellaops:vuln_business_tier`), and tokens persisted to PostgreSQL retain the same values for audit and revocation.
|
||||
- **Audit trail** – Every token issuance emits `authority.vuln_attr.*` audit properties that mirror the resolved filter set, along with `delegation.service_account` and ordered `delegation.actor[n]` entries so Vuln Explorer can correlate access decisions.
|
||||
- **Permalinks** – Signed permalinks inherit the caller’s ABAC filters; consuming services must enforce the embedded claims in addition to scope checks when resolving permalinks.
|
||||
- **Attachment tokens** – Authority mints short-lived tokens (`POST /vuln/attachments/tokens/issue`) to gate evidence downloads. Verification (`POST /vuln/attachments/tokens/verify`) enforces tenant, scope, and ABAC metadata, and emits `vuln.attachment.token.*` audit records.
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
- Trace exemplar anchors: `traceparent` headers are copied into logs; exporters stay disabled by default for air-gap. Enable by setting `Telemetry:ExportEnabled=true` and pointing to on-prem Tempo/Jaeger.
|
||||
|
||||
## Health/diagnostics
|
||||
- `/health/liveness` and `/health/readiness` (HTTP 200 expected; readiness checks Mongo + cache reachability).
|
||||
- `/health/liveness` and `/health/readiness` (HTTP 200 expected; readiness checks PostgreSQL + cache reachability).
|
||||
- `/status` returns build version, git commit, and enabled features; safe for anonymous fetch in sealed environments.
|
||||
- Ledger replay check: `GET /v1/findings?projectionMode=verify` emits `X-Vuln-Projection-Head` for quick consistency probes.
|
||||
|
||||
|
||||
@@ -17,11 +17,11 @@ Digests coalesce multiple matching events into a single notification when rules
|
||||
|
||||
## 2. Storage model
|
||||
|
||||
Digest state lives in Mongo (`digests` collection) and mirrors the schema described in [modules/notify/architecture.md](../modules/notify/architecture.md#7-data-model-mongo):
|
||||
Digest state lives in PostgreSQL (`notify.digests` table) and mirrors the schema described in [modules/notify/architecture.md](../modules/notify/architecture.md#7-data-model):
|
||||
|
||||
```json
|
||||
{
|
||||
"_id": "tenant-dev:act-email-compliance:1h",
|
||||
"id": "tenant-dev:act-email-compliance:1h",
|
||||
"tenantId": "tenant-dev",
|
||||
"actionKey": "act-email-compliance",
|
||||
"window": "1h",
|
||||
@@ -63,7 +63,7 @@ Digest state lives in Mongo (`digests` collection) and mirrors the schema descri
|
||||
| Endpoint | Description | Notes |
|
||||
|----------|-------------|-------|
|
||||
| `POST /digests` | Issues administrative commands (e.g., force flush, reopen) for a specific action/window. | Request body specifies the command target; requires `notify.admin`. |
|
||||
| `GET /digests/{actionKey}` | Returns the currently open window (if any) for the referenced action. | Supports operators/CLI inspecting pending digests; requires `notify.viewer`. |
|
||||
| `GET /digests/{actionKey}` | Returns the currently open window (if any) for the referenced action. | Supports operators/CLI inspecting pending digests; requires `notify.viewer`. |
|
||||
| `DELETE /digests/{actionKey}` | Drops the open window without notifying (emergency stop). | Emits an audit record; use sparingly. |
|
||||
|
||||
All routes honour the tenant header and reuse the standard Notify rate limits.
|
||||
@@ -76,7 +76,7 @@ All routes honour the tenant header and reuse the standard Notify rate limits.
|
||||
- **Throttles.** Digest generation respects action throttles; setting an aggressive throttle together with a digest window may result in deliberate skips (logged as `Throttled` in the delivery ledger).
|
||||
- **Quiet hours.** Future sprint work (`NOTIFY-SVC-39-004`) integrates quiet-hour calendars. When enabled, flush timers pause during quiet windows and resume afterwards.
|
||||
- **Back-pressure.** When the window reaches the configured item cap before the timer, the worker flushes early and starts a new window immediately.
|
||||
- **Crash resilience.** Workers rebuild in-flight windows from Mongo on startup; partially flushed windows remain closed after success or reopened if the flush fails.
|
||||
- **Crash resilience.** Workers rebuild in-flight windows from PostgreSQL on startup; partially flushed windows remain closed after success or reopened if the flush fails.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -21,10 +21,10 @@ Notifications Studio turns raw platform events into concise, tenant-scoped alert
|
||||
|------------|--------------|----------|
|
||||
| Rules engine | Declarative matchers for event kinds, severities, namespaces, VEX context, KEV flags, and more. | [`notifications/rules.md`](rules.md) |
|
||||
| Channel catalog | Slack, Teams, Email, Webhook connectors loaded via restart-time plug-ins; metadata stored without secrets. | [`notifications/architecture.md`](architecture.md) |
|
||||
| Templates | Locale-aware, deterministic rendering via safe helpers; channel defaults plus tenant-specific overrides, including the attestation lifecycle suite (`tmpl-attest-*`). | [`notifications/templates.md`](templates.md#7-attestation--signing-lifecycle-templates-notify-attest-74-001) |
|
||||
| Digests | Coalesce bursts into periodic summaries with deterministic IDs and audit trails. | [`notifications/digests.md`](digests.md) |
|
||||
| Delivery ledger | Tracks rendered payload hashes, attempts, throttles, and outcomes for every action. | [`modules/notify/architecture.md`](../modules/notify/architecture.md#7-data-model-mongo) |
|
||||
| Ack tokens | DSSE-signed acknowledgement tokens with webhook allowlists and escalation guardrails enforced by Authority. | [`modules/notify/architecture.md`](../modules/notify/architecture.md#81-ack-tokens--escalation-workflows) |
|
||||
| Templates | Locale-aware, deterministic rendering via safe helpers; channel defaults plus tenant-specific overrides, including the attestation lifecycle suite (`tmpl-attest-*`). | [`notifications/templates.md`](templates.md#7-attestation--signing-lifecycle-templates-notify-attest-74-001) |
|
||||
| Digests | Coalesce bursts into periodic summaries with deterministic IDs and audit trails. | [`notifications/digests.md`](digests.md) |
|
||||
| Delivery ledger | Tracks rendered payload hashes, attempts, throttles, and outcomes for every action. | [`modules/notify/architecture.md`](../modules/notify/architecture.md#7-data-model) |
|
||||
| Ack tokens | DSSE-signed acknowledgement tokens with webhook allowlists and escalation guardrails enforced by Authority. | [`modules/notify/architecture.md`](../modules/notify/architecture.md#81-ack-tokens--escalation-workflows) |
|
||||
|
||||
---
|
||||
|
||||
@@ -35,7 +35,7 @@ Notifications Studio turns raw platform events into concise, tenant-scoped alert
|
||||
3. **Connectors deliver.** Channel plug-ins send the rendered payload to Slack/Teams/Email/Webhook targets and report back attempts and outcomes.
|
||||
4. **Consumers investigate.** Operators pivot from message links into Console dashboards, SBOM views, or policy overlays with correlation IDs preserved.
|
||||
|
||||
The Notify WebService fronts worker state with REST APIs used by the UI and CLI. Tenants authenticate via StellaOps Authority scopes `notify.viewer`, `notify.operator`, and (for escalated actions) `notify.admin`. All operations require the tenant header (`X-StellaOps-Tenant`) to preserve sovereignty boundaries.
|
||||
The Notify WebService fronts worker state with REST APIs used by the UI and CLI. Tenants authenticate via StellaOps Authority scopes `notify.viewer`, `notify.operator`, and (for escalated actions) `notify.admin`. All operations require the tenant header (`X-StellaOps-Tenant`) to preserve sovereignty boundaries.
|
||||
|
||||
---
|
||||
|
||||
@@ -43,12 +43,12 @@ The Notify WebService fronts worker state with REST APIs used by the UI and CLI.
|
||||
|
||||
| Area | Guidance |
|
||||
|------|----------|
|
||||
| **Tenancy** | Each rule, channel, template, and delivery belongs to exactly one tenant. Cross-tenant sharing is intentionally unsupported. |
|
||||
| **Determinism** | Configuration persistence normalises strings and sorts collections. Template rendering produces identical `bodyHash` values when inputs match; attestation events always reference the canonical `tmpl-attest-*` keys documented in the template guide. |
|
||||
| **Scaling** | Workers scale horizontally; per-tenant rule snapshots are cached and refreshed from Mongo change streams. Redis (or equivalent) guards throttles and locks. |
|
||||
| **Offline** | Offline Kits include plug-ins, default templates, and seed rules. Operators can edit YAML/JSON manifests before air-gapped deployment. |
|
||||
| **Security** | Channel secrets use indirection (`secretRef`), Authority-protected OAuth clients secure API access, and delivery payloads are redacted before storage where required. |
|
||||
| **Module boundaries** | 2025-11-02 decision: keep `src/Notify/` as the shared notification toolkit and `src/Notifier/` as the Notifications Studio runtime host until a packaging RFC covers the implications of merging. |
|
||||
| **Tenancy** | Each rule, channel, template, and delivery belongs to exactly one tenant. Cross-tenant sharing is intentionally unsupported. |
|
||||
| **Determinism** | Configuration persistence normalises strings and sorts collections. Template rendering produces identical `bodyHash` values when inputs match; attestation events always reference the canonical `tmpl-attest-*` keys documented in the template guide. |
|
||||
| **Scaling** | Workers scale horizontally; per-tenant rule snapshots are cached and refreshed from PostgreSQL change notifications. Valkey (or Redis-compatible) guards throttles and locks. |
|
||||
| **Offline** | Offline Kits include plug-ins, default templates, and seed rules. Operators can edit YAML/JSON manifests before air-gapped deployment. |
|
||||
| **Security** | Channel secrets use indirection (`secretRef`), Authority-protected OAuth clients secure API access, and delivery payloads are redacted before storage where required. |
|
||||
| **Module boundaries** | 2025-11-02 decision: keep `src/Notify/` as the shared notification toolkit and `src/Notifier/` as the Notifications Studio runtime host until a packaging RFC covers the implications of merging. |
|
||||
|
||||
---
|
||||
|
||||
@@ -56,8 +56,8 @@ The Notify WebService fronts worker state with REST APIs used by the UI and CLI.
|
||||
|
||||
| Step | Goal | Reference |
|
||||
|------|------|-----------|
|
||||
| 1 | Deploy Notify WebService + Worker with Mongo and Redis | [`modules/notify/architecture.md`](../modules/notify/architecture.md#1-runtime-shape--projects) |
|
||||
| 2 | Register OAuth clients/scopes in Authority | [`etc/authority.yaml.sample`](../../etc/authority.yaml.sample) |
|
||||
| 1 | Deploy Notify WebService + Worker with PostgreSQL and Valkey | [`modules/notify/architecture.md`](../modules/notify/architecture.md#1-runtime-shape--projects) |
|
||||
| 2 | Register OAuth clients/scopes in Authority | [`etc/authority.yaml.sample`](../../etc/authority.yaml.sample) |
|
||||
| 3 | Install channel plug-ins and capture secret references | [`plugins/notify`](../../plugins) |
|
||||
| 4 | Create a tenant rule and test preview | [`POST /channels/{id}/test`](../modules/notify/architecture.md#8-external-apis-webservice) |
|
||||
| 5 | Inspect deliveries and digests | `/api/v1/notify/deliveries`, `/api/v1/notify/digests` |
|
||||
|
||||
@@ -0,0 +1,413 @@
|
||||
# Product Advisory: Deterministic Resolver Architecture
|
||||
|
||||
**Date:** 2025-12-24
|
||||
**Author:** Architecture Guild
|
||||
**Status:** Approved for Implementation
|
||||
**Sprint Epoch:** 9100
|
||||
**Priority:** P0 (Critical Path for Audit Compliance)
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
This advisory defines the architecture for a **unified deterministic resolver** that evaluates vulnerability graphs using only declared evidence edges and produces cryptographically verifiable, reproducible verdicts. The resolver guarantees: **same inputs → same traversal order → same per-node inputs → same verdicts → same output digest**.
|
||||
|
||||
This is a foundational capability for:
|
||||
- Auditor-friendly verification ("same inputs → same verdicts")
|
||||
- Offline replay by vendors/distributors
|
||||
- Single-digest comparison across runs
|
||||
- CI/CD gate assertions
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Current implementation has strong building blocks but lacks unified orchestration:
|
||||
|
||||
| Component | Location | Gap |
|
||||
|-----------|----------|-----|
|
||||
| Topological Sort | `DeterministicGraphOrderer` | Traversal only; no verdict output |
|
||||
| Lattice Evaluation | `TrustLatticeEngine`, `PolicyEvaluator` | No traversal sequence in output |
|
||||
| Content Addressing | `ContentAddressedIdGenerator` | Node IDs only; no edge IDs |
|
||||
| Purity Analysis | `ProhibitedPatternAnalyzer` | Static only; no runtime enforcement |
|
||||
| Digest Computation | Various | No unified "run digest" |
|
||||
|
||||
**Result:** Cannot produce a single `ResolutionResult` that captures:
|
||||
1. The exact traversal sequence used
|
||||
2. Per-node verdicts with individual digests
|
||||
3. A composite `FinalDigest` for verification
|
||||
|
||||
---
|
||||
|
||||
## Solution Architecture
|
||||
|
||||
### Core Data Model
|
||||
|
||||
```csharp
|
||||
// New: src/__Libraries/StellaOps.Resolver/
|
||||
|
||||
/// <summary>
|
||||
/// Content-addressed node identifier.
|
||||
/// Format: sha256(kind || ":" || normalized-key)
|
||||
/// </summary>
|
||||
public sealed record NodeId(string Value) : IComparable<NodeId>
|
||||
{
|
||||
public int CompareTo(NodeId? other) =>
|
||||
string.Compare(Value, other?.Value, StringComparison.Ordinal);
|
||||
|
||||
public static NodeId From(string kind, string normalizedKey) =>
|
||||
new(HexSha256($"{kind}:{normalizedKey}"));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Content-addressed edge identifier.
|
||||
/// Format: sha256(srcID || "->" || edgeKind || "->" || dstID)
|
||||
/// </summary>
|
||||
public sealed record EdgeId(string Value) : IComparable<EdgeId>
|
||||
{
|
||||
public static EdgeId From(NodeId src, string kind, NodeId dst) =>
|
||||
new(HexSha256($"{src.Value}->{kind}->{dst.Value}"));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Graph edge with content-addressed identity and optional cycle-cut marker.
|
||||
/// </summary>
|
||||
public sealed record Edge(
|
||||
EdgeId Id,
|
||||
NodeId Src,
|
||||
string Kind,
|
||||
NodeId Dst,
|
||||
JsonElement Attrs,
|
||||
bool IsCycleCut = false);
|
||||
|
||||
/// <summary>
|
||||
/// Graph node with content-addressed identity.
|
||||
/// </summary>
|
||||
public sealed record Node(
|
||||
NodeId Id,
|
||||
string Kind,
|
||||
JsonElement Attrs);
|
||||
|
||||
/// <summary>
|
||||
/// Immutable policy with content-addressed identity.
|
||||
/// </summary>
|
||||
public sealed record Policy(
|
||||
string Version,
|
||||
JsonElement Rules,
|
||||
string ConstantsDigest);
|
||||
|
||||
/// <summary>
|
||||
/// Individual verdict with its own content-addressed digest.
|
||||
/// </summary>
|
||||
public sealed record Verdict(
|
||||
NodeId Node,
|
||||
string Status,
|
||||
JsonElement Evidence,
|
||||
string VerdictDigest);
|
||||
|
||||
/// <summary>
|
||||
/// Complete resolution result with all digests for verification.
|
||||
/// </summary>
|
||||
public sealed record ResolutionResult(
|
||||
ImmutableArray<NodeId> TraversalSequence,
|
||||
ImmutableArray<Verdict> Verdicts,
|
||||
string GraphDigest,
|
||||
string PolicyDigest,
|
||||
string FinalDigest);
|
||||
```
|
||||
|
||||
### Resolver Algorithm
|
||||
|
||||
```text
|
||||
normalize(graph); // IDs, fields, ordering
|
||||
validate(graph); // No implicit data, cycles declared
|
||||
assert no-ambient-inputs(); // Runtime purity check
|
||||
|
||||
order = topoSortWithLexTieBreak(graph); // Uses only evidence edges
|
||||
verdicts = []
|
||||
|
||||
for node in order:
|
||||
inbound = gatherInboundEvidence(node) // Pure, no IO
|
||||
verdict = Evaluate(node, inbound, policy) // Pure, no IO
|
||||
verdicts.append(canonicalize(verdict) with VerdictDigest)
|
||||
|
||||
result = {
|
||||
traversal: order,
|
||||
verdicts: verdicts,
|
||||
graphDigest: digest(canonical(graph)),
|
||||
policyDigest: digest(canonical(policy))
|
||||
}
|
||||
result.finalDigest = digest(canonical(result))
|
||||
return result
|
||||
```
|
||||
|
||||
### Key Invariants
|
||||
|
||||
1. **Canonical IDs**: `NodeId = sha256(kind:normalized-key)`, `EdgeId = sha256(src->kind->dst)`
|
||||
2. **Canonical Serialization**: Alphabetical keys, sorted arrays, fixed numerics, ISO-8601 Z timestamps
|
||||
3. **Fixed Traversal**: Kahn's algorithm with lexicographic tie-breaker (`SortedSet<NodeId>`)
|
||||
4. **Explicit Cycles**: Cycles require `IsCycleCut = true` edge; unmarked cycles → invalid graph
|
||||
5. **Evidence-Only Evaluation**: Node verdict = f(node, inbound_edges, policy) — no ambient IO
|
||||
6. **Stable Outputs**: `(TraversalSequence, Verdicts[], GraphDigest, PolicyDigest, FinalDigest)`
|
||||
|
||||
---
|
||||
|
||||
## Gap Analysis vs Current Implementation
|
||||
|
||||
### Already Implemented (No Work Needed)
|
||||
|
||||
| Requirement | Implementation | Location |
|
||||
|-------------|---------------|----------|
|
||||
| Canonical JSON | RFC 8785 compliant | `CanonicalJsonSerializer`, `Rfc8785JsonCanonicalizer` |
|
||||
| Alphabetical key sorting | `StableDictionaryConverter` | `StellaOps.Canonicalization` |
|
||||
| ISO-8601 UTC timestamps | `Iso8601DateTimeConverter` | `StellaOps.Canonicalization` |
|
||||
| Topological sort with lex tie-break | `SortedSet<string>` | `DeterministicGraphOrderer:134` |
|
||||
| K4 Four-Valued Lattice | Complete | `K4Lattice.cs` |
|
||||
| VEX Lattice Merging | With traces | `OpenVexStatementMerger` |
|
||||
| Merkle Tree Proofs | SHA256-based | `DeterministicMerkleTreeBuilder` |
|
||||
| Static Purity Analysis | 16+ patterns | `ProhibitedPatternAnalyzer` |
|
||||
| Injected Timestamp | Context-based | `PolicyEvaluationContext.Now` |
|
||||
| Graph Content Hash | SHA256 | `DeterministicGraphOrderer.ComputeCanonicalHash()` |
|
||||
|
||||
### Gaps to Implement
|
||||
|
||||
| Gap | Priority | Sprint | Description |
|
||||
|-----|----------|--------|-------------|
|
||||
| Unified Resolver | P0 | 9100.0001.0001 | Single entry point producing `ResolutionResult` |
|
||||
| Cycle-Cut Edges | P1 | 9100.0001.0002 | `IsCycleCut` edge property; validation |
|
||||
| EdgeId | P2 | 9100.0001.0003 | Content-addressed edge identifiers |
|
||||
| FinalDigest | P1 | 9100.0002.0001 | Composite run-level digest |
|
||||
| Per-Node VerdictDigest | P2 | 9100.0002.0002 | Individual verdict digests |
|
||||
| Runtime Purity | P1 | 9100.0003.0001 | Runtime enforcement beyond static analysis |
|
||||
| Graph Validation + NFC | P3 | 9100.0003.0002 | Pre-traversal validation; NFC normalization |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Core Resolver Package (Sprint 9100.0001.*)
|
||||
|
||||
**Goal:** Create `StellaOps.Resolver` library with unified resolver pattern.
|
||||
|
||||
1. **Sprint 9100.0001.0001** — Core Resolver
|
||||
- `DeterministicResolver` class
|
||||
- `ResolutionResult` record
|
||||
- Integration with existing `DeterministicGraphOrderer`
|
||||
- Integration with existing `TrustLatticeEngine`
|
||||
|
||||
2. **Sprint 9100.0001.0002** — Cycle-Cut Edges
|
||||
- `IsCycleCut` property on edges
|
||||
- Cycle validation (unmarked cycles → error)
|
||||
- Graph validation before traversal
|
||||
|
||||
3. **Sprint 9100.0001.0003** — Content-Addressed EdgeId
|
||||
- `EdgeId` record
|
||||
- Edge content addressing
|
||||
- Merkle tree inclusion of edges
|
||||
|
||||
### Phase 2: Digest Infrastructure (Sprint 9100.0002.*)
|
||||
|
||||
**Goal:** Implement comprehensive digest chain for verification.
|
||||
|
||||
1. **Sprint 9100.0002.0001** — FinalDigest
|
||||
- Composite digest computation
|
||||
- `sha256(canonical({graphDigest, policyDigest, verdicts[]}))`
|
||||
- Integration with attestation system
|
||||
|
||||
2. **Sprint 9100.0002.0002** — Per-Node VerdictDigest
|
||||
- Verdict-level content addressing
|
||||
- Drill-down debugging ("which node changed?")
|
||||
- Delta detection between runs
|
||||
|
||||
### Phase 3: Purity & Validation (Sprint 9100.0003.*)
|
||||
|
||||
**Goal:** Harden determinism guarantees with runtime enforcement.
|
||||
|
||||
1. **Sprint 9100.0003.0001** — Runtime Purity Enforcement
|
||||
- Runtime guards beyond static analysis
|
||||
- Dependency injection shims for ambient services
|
||||
- Test harness for purity verification
|
||||
|
||||
2. **Sprint 9100.0003.0002** — Graph Validation & NFC
|
||||
- Pre-traversal validation ("no implicit data")
|
||||
- Unicode NFC normalization for string fields
|
||||
- Evidence completeness assertions
|
||||
|
||||
---
|
||||
|
||||
## Testing Requirements
|
||||
|
||||
### Mandatory Test Types
|
||||
|
||||
1. **Replay Test**: Same input twice → identical `FinalDigest`
|
||||
2. **Permutation Test**: Shuffle input nodes/edges → identical outputs
|
||||
3. **Cycle Test**: Introduce cycle → fail unless `IsCycleCut` edge present
|
||||
4. **Ambience Test**: Forbid calls to time/env/network during evaluation
|
||||
5. **Serialization Test**: Canonicalization changes → predictable digest changes
|
||||
|
||||
### Property-Based Tests
|
||||
|
||||
```csharp
|
||||
// Idempotency
|
||||
[Property]
|
||||
public void Resolver_SameInput_SameOutput(Graph graph)
|
||||
{
|
||||
var result1 = resolver.Run(graph);
|
||||
var result2 = resolver.Run(graph);
|
||||
Assert.Equal(result1.FinalDigest, result2.FinalDigest);
|
||||
}
|
||||
|
||||
// Order Independence
|
||||
[Property]
|
||||
public void Resolver_ShuffledInputs_SameOutput(Graph graph, int seed)
|
||||
{
|
||||
var shuffled = ShuffleNodesAndEdges(graph, seed);
|
||||
var result1 = resolver.Run(graph);
|
||||
var result2 = resolver.Run(shuffled);
|
||||
Assert.Equal(result1.FinalDigest, result2.FinalDigest);
|
||||
}
|
||||
|
||||
// Cycle Detection
|
||||
[Property]
|
||||
public void Resolver_UnmarkedCycle_Throws(Graph graphWithCycle)
|
||||
{
|
||||
Assume.That(!graphWithCycle.Edges.Any(e => e.IsCycleCut));
|
||||
Assert.Throws<InvalidGraphException>(() => resolver.Run(graphWithCycle));
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Existing Components to Integrate
|
||||
|
||||
| Component | Integration |
|
||||
|-----------|-------------|
|
||||
| `DeterministicGraphOrderer` | Use for traversal order computation |
|
||||
| `TrustLatticeEngine` | Use for K4 lattice evaluation |
|
||||
| `CanonicalJsonSerializer` | Use for all serialization |
|
||||
| `ContentAddressedIdGenerator` | Extend for EdgeId and VerdictId |
|
||||
| `ProhibitedPatternAnalyzer` | Combine with runtime guards |
|
||||
| `PolicyEvaluationContext` | Use for injected inputs |
|
||||
|
||||
### New Modules
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `StellaOps.Resolver` | Core resolver library |
|
||||
| `StellaOps.Resolver.Testing` | Test fixtures and harnesses |
|
||||
| `StellaOps.Resolver.Attestation` | Integration with proof chain |
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. **Unified API**: Single `resolver.Run(graph)` produces complete `ResolutionResult`
|
||||
2. **Deterministic**: 100% pass rate on replay/permutation/cycle tests
|
||||
3. **Auditable**: `FinalDigest` enables single-value verification
|
||||
4. **Performant**: No more than 10% overhead vs current fragmented approach
|
||||
5. **Documented**: Full API documentation and integration guide
|
||||
|
||||
---
|
||||
|
||||
## Appendix: C# Reference Implementation
|
||||
|
||||
```csharp
|
||||
public sealed class DeterministicResolver
|
||||
{
|
||||
private readonly Policy _policy;
|
||||
private readonly string _policyDigest;
|
||||
private readonly DeterministicGraphOrderer _orderer;
|
||||
private readonly TrustLatticeEngine _lattice;
|
||||
private readonly CanonicalJsonSerializer _serializer;
|
||||
|
||||
public DeterministicResolver(
|
||||
Policy policy,
|
||||
DeterministicGraphOrderer orderer,
|
||||
TrustLatticeEngine lattice,
|
||||
CanonicalJsonSerializer serializer)
|
||||
{
|
||||
_policy = Canon(policy);
|
||||
_policyDigest = HexSha256(serializer.SerializeToBytes(_policy));
|
||||
_orderer = orderer;
|
||||
_lattice = lattice;
|
||||
_serializer = serializer;
|
||||
}
|
||||
|
||||
public ResolutionResult Run(EvidenceGraph graph)
|
||||
{
|
||||
// 1. Canonicalize and validate
|
||||
var canonical = _orderer.Canonicalize(graph);
|
||||
ValidateCycles(canonical);
|
||||
EnsureNoAmbientInputs();
|
||||
|
||||
// 2. Traverse in deterministic order
|
||||
var traversal = canonical.Nodes
|
||||
.Select(n => NodeId.Parse(n.Id))
|
||||
.ToImmutableArray();
|
||||
|
||||
// 3. Evaluate each node purely
|
||||
var verdicts = new List<Verdict>(traversal.Length);
|
||||
foreach (var nodeId in traversal)
|
||||
{
|
||||
var node = GetNode(canonical, nodeId);
|
||||
var inbound = GatherInboundEvidence(canonical, nodeId);
|
||||
var verdict = EvaluatePure(node, inbound, _policy);
|
||||
var verdictBytes = _serializer.SerializeToBytes(verdict);
|
||||
verdicts.Add(verdict with { VerdictDigest = HexSha256(verdictBytes) });
|
||||
}
|
||||
|
||||
// 4. Compute digests
|
||||
var verdictsArray = verdicts.ToImmutableArray();
|
||||
var resultWithoutFinal = new ResolutionResult(
|
||||
traversal,
|
||||
verdictsArray,
|
||||
canonical.ContentHash,
|
||||
_policyDigest,
|
||||
"");
|
||||
|
||||
var finalBytes = _serializer.SerializeToBytes(resultWithoutFinal);
|
||||
return resultWithoutFinal with { FinalDigest = HexSha256(finalBytes) };
|
||||
}
|
||||
|
||||
private void ValidateCycles(CanonicalGraph graph)
|
||||
{
|
||||
// Detect cycles; require IsCycleCut for each
|
||||
var cycles = DetectCycles(graph);
|
||||
var unmarked = cycles.Where(c => !c.CutEdge.IsCycleCut).ToList();
|
||||
if (unmarked.Any())
|
||||
{
|
||||
throw new InvalidGraphException(
|
||||
$"Graph contains {unmarked.Count} unmarked cycle(s). " +
|
||||
"Add IsCycleCut=true to break cycles.");
|
||||
}
|
||||
}
|
||||
|
||||
private Verdict EvaluatePure(Node node, InboundEvidence evidence, Policy policy)
|
||||
{
|
||||
// Pure evaluation: no IO, no ambient state
|
||||
return _lattice.Evaluate(node, evidence, policy);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Documents
|
||||
|
||||
- `docs/modules/attestor/proof-chain-specification.md`
|
||||
- `docs/modules/policy/architecture.md`
|
||||
- `docs/testing/testing-strategy-models.md`
|
||||
- `src/Policy/__Libraries/StellaOps.Policy/TrustLattice/K4Lattice.cs`
|
||||
- `src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Ordering/DeterministicGraphOrderer.cs`
|
||||
|
||||
---
|
||||
|
||||
## Approval
|
||||
|
||||
| Role | Name | Date | Decision |
|
||||
|------|------|------|----------|
|
||||
| Architecture Lead | — | 2025-12-24 | Approved |
|
||||
| Policy Guild Lead | — | 2025-12-24 | Approved |
|
||||
| Scanner Guild Lead | — | 2025-12-24 | Approved |
|
||||
@@ -0,0 +1,271 @@
|
||||
# Product Advisory: Evidence-Weighted Score Model
|
||||
|
||||
**Date:** 24-Dec-2025
|
||||
**Status:** Approved for Implementation
|
||||
**Sprint Batch:** 8200.0012
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
This advisory introduces a **unified evidence-weighted scoring model** that combines six evidence dimensions into a single 0-100 score per vulnerability finding. The score enables rapid triage by surfacing the most "real" risks with transparent, decomposable evidence.
|
||||
|
||||
### Key Benefits
|
||||
|
||||
- **Prioritize quickly:** Sort by score to fix the most "real" risks first
|
||||
- **Reduce noise:** Thin, speculative findings sink to the bottom
|
||||
- **Stay explainable:** Score decomposes into evidence "slices" on hover
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The current platform has **four independent scoring systems** that don't interoperate:
|
||||
|
||||
1. `ConfidenceCalculator` (Policy module) — 5 factors, outputs 0-1
|
||||
2. `ScorePolicy` (Policy module) — 4 factors in basis points
|
||||
3. `EvidenceDensityScorer` (Scanner module) — 8 factors, outputs 0-1
|
||||
4. `TrustVector` (Excititor module) — 3 factors, outputs 0-1
|
||||
|
||||
Users cannot sort findings by a single "risk strength" metric. They must mentally combine multiple signals, leading to:
|
||||
|
||||
- Inconsistent prioritization across teams
|
||||
- Missed high-risk findings buried in noise
|
||||
- Difficulty explaining triage decisions to stakeholders
|
||||
|
||||
---
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
### Scoring Model
|
||||
|
||||
```
|
||||
Score = clamp01(
|
||||
W_rch*RCH + W_rts*RTS + W_bkp*BKP + W_xpl*XPL + W_src*SRC - W_mit*MIT
|
||||
) * 100
|
||||
```
|
||||
|
||||
### Evidence Dimensions
|
||||
|
||||
| Symbol | Name | Description | Weight |
|
||||
|--------|------|-------------|--------|
|
||||
| **RCH** | Reachability | Static/dynamic reachability confidence | 0.30 |
|
||||
| **RTS** | Runtime | Runtime signal strength (eBPF, hits, recency) | 0.25 |
|
||||
| **BKP** | Backport | Backport/patch evidence strength | 0.15 |
|
||||
| **XPL** | Exploit | Exploit likelihood (EPSS + KEV) | 0.15 |
|
||||
| **SRC** | Source Trust | Source trust (vendor > distro > community) | 0.10 |
|
||||
| **MIT** | Mitigations | Active mitigations (subtractive) | 0.10 |
|
||||
|
||||
### Guardrails
|
||||
|
||||
| Condition | Action | Rationale |
|
||||
|-----------|--------|-----------|
|
||||
| BKP=1 + not_affected + RTS<0.6 | Cap at 15 | Vendor confirms not affected |
|
||||
| RTS >= 0.8 | Floor at 60 | Strong live signal |
|
||||
| RCH=0 + RTS=0 | Cap at 45 | Speculative finding |
|
||||
|
||||
### Score Buckets
|
||||
|
||||
| Bucket | Range | Action |
|
||||
|--------|-------|--------|
|
||||
| **ActNow** | 90-100 | Immediate action required |
|
||||
| **ScheduleNext** | 70-89 | Schedule for next sprint |
|
||||
| **Investigate** | 40-69 | Investigate when touching component |
|
||||
| **Watchlist** | 0-39 | Low priority, monitor |
|
||||
|
||||
---
|
||||
|
||||
## Gap Analysis
|
||||
|
||||
### Existing Infrastructure (Reusable)
|
||||
|
||||
| Component | Location | Reuse |
|
||||
|-----------|----------|-------|
|
||||
| Reachability states | `Policy/ConfidenceCalculator.cs` | Direct mapping to RCH |
|
||||
| Runtime evidence | `Policy/RuntimeEvidence` | Direct mapping to RTS |
|
||||
| Backport detection | `Concelier/BackportProofService.cs` | Needs normalization |
|
||||
| EPSS bands | `Scanner/EpssPriorityCalculator.cs` | Needs normalization |
|
||||
| KEV status | `Concelier/VendorRiskSignalExtractor.cs` | Direct flag |
|
||||
| Trust vector | `Excititor/TrustVector.cs` | Direct mapping to SRC |
|
||||
| Gate multipliers | `Policy/GateMultipliersBps` | Needs inversion to MIT |
|
||||
|
||||
### New Components Required
|
||||
|
||||
1. **Unified Score Calculator** — Core formula implementation
|
||||
2. **Normalizers** — Convert raw evidence to 0-1 inputs
|
||||
3. **Guardrails Engine** — Apply caps/floors
|
||||
4. **API Endpoints** — Expose scores via REST
|
||||
5. **UI Components** — Score pill, breakdown popover, badges
|
||||
|
||||
---
|
||||
|
||||
## Implementation Sprints
|
||||
|
||||
| Sprint | Focus | Key Deliverables |
|
||||
|--------|-------|------------------|
|
||||
| **8200.0012.0001** | Core Library | Calculator, input models, weight policy, guardrails |
|
||||
| **8200.0012.0002** | Normalizers | BKP, XPL, MIT, RCH, RTS, SRC normalizers |
|
||||
| **8200.0012.0003** | Policy Integration | Score-based rules, verdict enrichment, attestation |
|
||||
| **8200.0012.0004** | API Endpoints | Single/batch score, history, webhooks |
|
||||
| **8200.0012.0005** | Frontend UI | Score pill, breakdown, badges, history chart |
|
||||
|
||||
**Total Tasks:** 268 across 5 sprints
|
||||
**Estimated Duration:** 6 months
|
||||
|
||||
---
|
||||
|
||||
## API Shape
|
||||
|
||||
### Score Response
|
||||
|
||||
```json
|
||||
{
|
||||
"findingId": "CVE-2024-1234@pkg:deb/debian/curl@7.64.0-4",
|
||||
"score": 78,
|
||||
"bucket": "ScheduleNext",
|
||||
"inputs": {
|
||||
"rch": 0.85, "rts": 0.40, "bkp": 0.00,
|
||||
"xpl": 0.70, "src": 0.80, "mit": 0.10
|
||||
},
|
||||
"weights": {
|
||||
"rch": 0.30, "rts": 0.25, "bkp": 0.15,
|
||||
"xpl": 0.15, "src": 0.10, "mit": 0.10
|
||||
},
|
||||
"flags": ["live-signal", "proven-path"],
|
||||
"explanations": [
|
||||
"Static path to vulnerable sink (confidence: 85%)",
|
||||
"EPSS: 0.8% probability (High band)",
|
||||
"Distro VEX signed (trust: 80%)"
|
||||
],
|
||||
"caps": {
|
||||
"speculativeCap": false,
|
||||
"notAffectedCap": false,
|
||||
"runtimeFloor": false
|
||||
},
|
||||
"policyDigest": "sha256:abc123...",
|
||||
"calculatedAt": "2026-01-15T14:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## UI Design
|
||||
|
||||
### Score Pill
|
||||
|
||||
```
|
||||
┌───────┐
|
||||
│ 78 │ ← White text on amber background (ScheduleNext bucket)
|
||||
└───────┘
|
||||
```
|
||||
|
||||
### Breakdown Popover
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Evidence Score: 78/100 │
|
||||
│ Bucket: Schedule Next Sprint │
|
||||
├─────────────────────────────────────────┤
|
||||
│ Reachability ████████▒▒ 0.85 │
|
||||
│ Runtime ████▒▒▒▒▒▒ 0.40 │
|
||||
│ Backport ▒▒▒▒▒▒▒▒▒▒ 0.00 │
|
||||
│ Exploit ███████▒▒▒ 0.70 │
|
||||
│ Source Trust ████████▒▒ 0.80 │
|
||||
│ Mitigations -█▒▒▒▒▒▒▒▒▒ 0.10 │
|
||||
├─────────────────────────────────────────┤
|
||||
│ 🟢 Live signal detected │
|
||||
│ ✓ Proven reachability path │
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Governance & Tuning
|
||||
|
||||
1. **Policy-Driven Weights:** Different environments (prod/staging/dev) can have different weight profiles
|
||||
2. **Versioned Policies:** Policy changes are versioned and signed; same inputs + policy = same score
|
||||
3. **Audit Trail:** All score calculations logged with inputs, policy digest, and result
|
||||
4. **Threshold Tuning:** Bucket thresholds configurable per tenant
|
||||
|
||||
---
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
### Phase 1: Dual Emit (3 months)
|
||||
- Both Confidence and EWS emitted in verdicts
|
||||
- Feature flag controls which is primary
|
||||
- Telemetry compares rankings
|
||||
|
||||
### Phase 2: EWS Primary (3 months)
|
||||
- EWS is default; Confidence available for compatibility
|
||||
- Deprecation warnings for Confidence consumers
|
||||
|
||||
### Phase 3: Confidence Removal (v3.0)
|
||||
- Confidence scoring removed
|
||||
- EWS is sole scoring system
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|------------|
|
||||
| Weight tuning requires iteration | Suboptimal prioritization | Conservative defaults; telemetry |
|
||||
| Migration confusion | User errors | Clear docs; gradual rollout |
|
||||
| Performance regression | Slower evaluation | Caching; benchmarks (<50ms) |
|
||||
| Attestation size increase | Storage cost | Compact proof format |
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. **Adoption:** 80% of findings viewed with EWS score within 6 months
|
||||
2. **Triage Time:** 30% reduction in mean-time-to-triage
|
||||
3. **False Positive Rate:** <5% of "ActNow" findings downgraded after review
|
||||
4. **Determinism:** 100% of scores reproducible given same inputs + policy
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Normalization Details
|
||||
|
||||
### BKP (Backport) Normalization
|
||||
|
||||
| Evidence Tier | BKP Range |
|
||||
|---------------|-----------|
|
||||
| No evidence | 0.00 |
|
||||
| Distro advisory text | 0.40-0.55 |
|
||||
| Changelog mention | 0.45-0.60 |
|
||||
| Patch header match | 0.70-0.85 |
|
||||
| HunkSig match | 0.80-0.92 |
|
||||
| Binary fingerprint | 0.90-1.00 |
|
||||
|
||||
### XPL (Exploit) Normalization
|
||||
|
||||
| Signal | XPL Value |
|
||||
|--------|-----------|
|
||||
| KEV present | floor 0.40 |
|
||||
| EPSS top 1% | 0.90-1.00 |
|
||||
| EPSS top 5% | 0.70-0.89 |
|
||||
| EPSS top 25% | 0.40-0.69 |
|
||||
| EPSS below 25% | 0.20-0.39 |
|
||||
| No EPSS data | 0.30 |
|
||||
|
||||
### MIT (Mitigation) Normalization
|
||||
|
||||
| Mitigation | MIT Value |
|
||||
|------------|-----------|
|
||||
| Feature flag disabled | 0.20-0.40 |
|
||||
| Auth required | 0.10-0.20 |
|
||||
| Admin only | 0.15-0.25 |
|
||||
| Seccomp profile | 0.10-0.25 |
|
||||
| Network isolation | 0.05-0.15 |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- Sprint Files: `docs/implplan/SPRINT_8200_0012_*.md`
|
||||
- Module Docs: `docs/modules/signals/architecture.md` (to be created)
|
||||
- Existing Confidence: `docs/modules/policy/confidence-scoring.md`
|
||||
- EPSS Enrichment: `docs/modules/scanner/epss-enrichment.md`
|
||||
- Backport Detection: `docs/modules/concelier/backport-detection.md`
|
||||
@@ -37,7 +37,7 @@ This guide translates the deterministic reachability blueprint into concrete wor
|
||||
| Stream | Owner Guild(s) | Key deliverables |
|
||||
|--------|----------------|------------------|
|
||||
| **Native symbols & callgraphs** | Scanner Worker · Symbols Guild | Ship `Scanner.Symbols.Native` + `Scanner.CallGraph.Native`, integrate Symbol Manifest v1, demangle Itanium/MSVC names, emit `FuncNode`/`CallEdge` CAS bundles (task `SCANNER-NATIVE-401-015`). |
|
||||
| **Reachability store** | Signals · BE-Base Platform | Provision shared Mongo collections (`func_nodes`, `call_edges`, `cve_func_hits`), indexes, and repositories plus REST hooks for reuse (task `SIG-STORE-401-016`). |
|
||||
| **Reachability store** | Signals · BE-Base Platform | Provision shared PostgreSQL tables (`func_nodes`, `call_edges`, `cve_func_hits`), indexes, and repositories plus REST hooks for reuse (task `SIG-STORE-401-016`). |
|
||||
| **Language lifters** | Scanner Worker | CLI/hosted lifters for DotNet, Go, Node/Deno, JVM, Rust, Swift, Binary, Shell with CAS uploads and richgraph output |
|
||||
| **Signals ingestion & scoring** | Signals | `/callgraphs`, `/runtime-facts` (JSON + NDJSON/gzip), `/graphs/{id}`, `/reachability/recompute` GA; CAS-backed storage, runtime dedupe, BFS+predicates scoring |
|
||||
| **Runtime capture** | Zastava + Runtime Guild | EntryTrace/eBPF samplers, NDJSON batches (symbol IDs + timestamps + counts) |
|
||||
|
||||
@@ -21,8 +21,8 @@
|
||||
- Empty runtime stream is rejected.
|
||||
|
||||
## Storage and cache
|
||||
- Stored alongside reachability facts in Mongo collection `reachability_facts`.
|
||||
- Runtime hits cached in Redis via `reachability_cache:*` entries; invalidated on ingest.
|
||||
- Stored alongside reachability facts in PostgreSQL table `reachability_facts`.
|
||||
- Runtime hits cached in Valkey via `reachability_cache:*` entries; invalidated on ingest.
|
||||
|
||||
## Interaction with scoring
|
||||
- Ingest triggers recompute: runtime hits added to prior facts’ hits, targets set to symbols observed, entryPoints taken from callgraph.
|
||||
|
||||
286
docs/reproducibility.md
Normal file
286
docs/reproducibility.md
Normal file
@@ -0,0 +1,286 @@
|
||||
# StellaOps Reproducibility Specification
|
||||
|
||||
This document defines the reproducibility guarantees, verdict identity computation, and replay procedures for StellaOps artifacts.
|
||||
|
||||
## Overview
|
||||
|
||||
StellaOps provides **deterministic, reproducible outputs** for all security artifacts:
|
||||
- SBOM generation (CycloneDX 1.6, SPDX 3.0.1)
|
||||
- VEX statements (OpenVEX)
|
||||
- Policy verdicts
|
||||
- Delta computations
|
||||
- DSSE attestations and Sigstore bundles
|
||||
|
||||
**Core Guarantee:** Given identical inputs (image digest, advisory feeds, policies, tool versions), StellaOps produces byte-for-byte identical outputs with matching content-addressed identifiers.
|
||||
|
||||
## Verdict Identity Formula
|
||||
|
||||
### Content-Addressed Verdict ID
|
||||
|
||||
All verdicts use content-addressed identifiers computed as:
|
||||
|
||||
```
|
||||
VerdictId = SHA256(Canonicalize(VerdictPayload))
|
||||
```
|
||||
|
||||
Where `VerdictPayload` includes:
|
||||
- **Delta ID**: Content hash of the security delta
|
||||
- **Blocking Drivers**: Sorted list of risk-increasing factors
|
||||
- **Warning Drivers**: Sorted list of advisory factors
|
||||
- **Applied Exceptions**: Sorted list of exception IDs covering findings
|
||||
- **Gate Level**: Recommended gate (G0-G4)
|
||||
- **Input Stamps**: Hashes of all inputs (see below)
|
||||
|
||||
### Input Stamps
|
||||
|
||||
Every artifact includes `InputStamps` capturing the provenance of all inputs:
|
||||
|
||||
```json
|
||||
{
|
||||
"feedSnapshotHash": "sha256:abc123...",
|
||||
"policyManifestHash": "sha256:def456...",
|
||||
"sourceCodeHash": "sha256:789ghi...",
|
||||
"baseImageDigest": "sha256:jkl012...",
|
||||
"vexDocumentHashes": ["sha256:mno345..."],
|
||||
"toolchainVersion": "1.0.0",
|
||||
"custom": {}
|
||||
}
|
||||
```
|
||||
|
||||
### Determinism Manifest
|
||||
|
||||
The `DeterminismManifest` (schema v1.0) tracks artifact reproducibility:
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": "1.0",
|
||||
"artifact": {
|
||||
"type": "verdict",
|
||||
"name": "scan-verdict",
|
||||
"version": "2025-12-24T12:00:00Z",
|
||||
"format": "StellaOps.DeltaVerdict@1"
|
||||
},
|
||||
"canonicalHash": {
|
||||
"algorithm": "SHA-256",
|
||||
"value": "abc123def456...",
|
||||
"encoding": "hex"
|
||||
},
|
||||
"inputs": {
|
||||
"feedSnapshotHash": "sha256:...",
|
||||
"policyManifestHash": "sha256:...",
|
||||
"baseImageDigest": "sha256:..."
|
||||
},
|
||||
"toolchain": {
|
||||
"platform": ".NET 10.0",
|
||||
"components": [
|
||||
{"name": "StellaOps.Scanner", "version": "1.0.0"},
|
||||
{"name": "StellaOps.Policy", "version": "1.0.0"}
|
||||
]
|
||||
},
|
||||
"reproducibility": {
|
||||
"clockFixed": true,
|
||||
"orderingGuarantee": "stable-sort",
|
||||
"normalizationRules": ["UTF-8", "LF", "canonical-json"]
|
||||
},
|
||||
"generatedAt": "2025-12-24T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Canonical JSON Serialization
|
||||
|
||||
All JSON outputs follow RFC 8785 (JSON Canonicalization Scheme):
|
||||
1. Keys sorted lexicographically
|
||||
2. No whitespace between tokens
|
||||
3. Unicode escaping for non-ASCII
|
||||
4. Numbers without leading zeros
|
||||
5. UTF-8 encoding
|
||||
|
||||
## DSSE Attestation Format
|
||||
|
||||
### Envelope Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"payloadType": "application/vnd.in-toto+json",
|
||||
"payload": "<base64url-encoded statement>",
|
||||
"signatures": [
|
||||
{
|
||||
"keyid": "sha256:...",
|
||||
"sig": "<base64url-encoded signature>"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### In-toto Statement
|
||||
|
||||
```json
|
||||
{
|
||||
"_type": "https://in-toto.io/Statement/v1",
|
||||
"subject": [
|
||||
{
|
||||
"name": "registry.example.com/image:tag",
|
||||
"digest": {"sha256": "abc123..."}
|
||||
}
|
||||
],
|
||||
"predicateType": "https://stellaops.io/attestation/verdict/v1",
|
||||
"predicate": {
|
||||
"verdictId": "sha256:...",
|
||||
"status": "pass",
|
||||
"gate": "G2",
|
||||
"inputs": {...},
|
||||
"evidence": [...]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Sigstore Bundle Format
|
||||
|
||||
StellaOps produces Sigstore bundles (v0.3) for offline verification:
|
||||
|
||||
```json
|
||||
{
|
||||
"$mediaType": "application/vnd.dev.sigstore.bundle.v0.3+json",
|
||||
"verificationMaterial": {
|
||||
"certificate": {...},
|
||||
"tlogEntries": [{
|
||||
"logIndex": "12345",
|
||||
"logId": {...},
|
||||
"inclusionProof": {...}
|
||||
}]
|
||||
},
|
||||
"dsseEnvelope": {...}
|
||||
}
|
||||
```
|
||||
|
||||
## Replay Procedure
|
||||
|
||||
### Prerequisites
|
||||
1. Offline bundle containing:
|
||||
- Advisory feed snapshot
|
||||
- Policy pack
|
||||
- VEX documents
|
||||
- Tool binaries (pinned versions)
|
||||
|
||||
### Steps
|
||||
|
||||
```bash
|
||||
# 1. Extract offline bundle
|
||||
stella offline extract --bundle offline-kit.tar.gz --output ./replay
|
||||
|
||||
# 2. Set deterministic environment
|
||||
export STELLAOPS_DETERMINISTIC_SEED=42
|
||||
export STELLAOPS_CLOCK_FIXED=2025-12-24T12:00:00Z
|
||||
|
||||
# 3. Run scan with pinned inputs
|
||||
stella scan \
|
||||
--image registry.example.com/image@sha256:abc123 \
|
||||
--feeds ./replay/feeds \
|
||||
--policies ./replay/policies \
|
||||
--output ./replay/output
|
||||
|
||||
# 4. Verify hash matches original
|
||||
stella verify \
|
||||
--manifest ./replay/output/manifest.json \
|
||||
--expected-hash sha256:def456...
|
||||
|
||||
# 5. Verify DSSE attestation
|
||||
stella attest verify \
|
||||
--bundle ./replay/output/bundle.sigstore \
|
||||
--policy verification-policy.yaml
|
||||
```
|
||||
|
||||
### Verification Policy
|
||||
|
||||
```yaml
|
||||
apiVersion: stellaops.io/v1
|
||||
kind: VerificationPolicy
|
||||
metadata:
|
||||
name: audit-verification
|
||||
spec:
|
||||
requiredPredicateTypes:
|
||||
- https://stellaops.io/attestation/verdict/v1
|
||||
trustedIssuers:
|
||||
- https://accounts.stellaops.io
|
||||
maxAge: 90d
|
||||
requireRekorEntry: true
|
||||
unknownBudget:
|
||||
maxTotal: 5
|
||||
action: fail
|
||||
```
|
||||
|
||||
## Unknown Budget Attestation
|
||||
|
||||
Policy thresholds are attested in verdict bundles:
|
||||
|
||||
```json
|
||||
{
|
||||
"predicateType": "https://stellaops.io/attestation/budget-check/v1",
|
||||
"predicate": {
|
||||
"environment": "production",
|
||||
"budgetConfig": {
|
||||
"maxUnknownCount": 5,
|
||||
"maxCumulativeUncertainty": 2.0,
|
||||
"reasonLimits": {
|
||||
"Reachability": 0,
|
||||
"Identity": 2,
|
||||
"Provenance": 2
|
||||
}
|
||||
},
|
||||
"actualCounts": {
|
||||
"total": 3,
|
||||
"byReason": {"Identity": 2, "Provenance": 1}
|
||||
},
|
||||
"result": "pass",
|
||||
"configHash": "sha256:..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Schema Versions
|
||||
|
||||
| Format | Version | Schema Location |
|
||||
|--------|---------|-----------------|
|
||||
| CycloneDX | 1.6 | `docs/schemas/cyclonedx-bom-1.6.schema.json` |
|
||||
| SPDX | 3.0.1 | `docs/schemas/spdx-3.0.1.schema.json` |
|
||||
| OpenVEX | 0.2.0 | `docs/schemas/openvex-0.2.0.schema.json` |
|
||||
| Sigstore Bundle | 0.3 | `docs/schemas/sigstore-bundle-0.3.schema.json` |
|
||||
| DeterminismManifest | 1.0 | `docs/schemas/determinism-manifest-1.0.schema.json` |
|
||||
|
||||
## CI Integration
|
||||
|
||||
### Schema Validation
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/schema-validation.yml
|
||||
- name: Validate CycloneDX
|
||||
run: |
|
||||
sbom-utility validate \
|
||||
--input-file ${{ matrix.fixture }} \
|
||||
--schema docs/schemas/cyclonedx-bom-1.6.schema.json
|
||||
```
|
||||
|
||||
### Determinism Gate
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/determinism-gate.yml
|
||||
- name: Verify Verdict Hash
|
||||
run: |
|
||||
HASH1=$(stella scan --image test:latest --output /tmp/run1 | jq -r '.verdictId')
|
||||
HASH2=$(stella scan --image test:latest --output /tmp/run2 | jq -r '.verdictId')
|
||||
[ "$HASH1" = "$HASH2" ] || exit 1
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Testing Strategy](testing/testing-strategy-models.md)
|
||||
- [Determinism Verification](testing/determinism-verification.md)
|
||||
- [DSSE Attestation Guide](modules/attestor/README.md)
|
||||
- [Offline Operation](24_OFFLINE_KIT.md)
|
||||
- [Proof Bundle Spec](modules/triage/proof-bundle-spec.md)
|
||||
|
||||
## Changelog
|
||||
|
||||
| Version | Date | Changes |
|
||||
|---------|------|---------|
|
||||
| 1.0 | 2025-12-24 | Initial specification based on product advisory gap analysis |
|
||||
@@ -44,7 +44,7 @@ Event names follow dotted notation:
|
||||
|
||||
## Persistence
|
||||
|
||||
The Authority host converts audit records into `AuthorityLoginAttemptDocument` rows for MongoDB persistence. Documents must:
|
||||
The Authority host converts audit records into `AuthorityLoginAttemptDocument` rows for PostgreSQL persistence (schema `authority`). Documents must:
|
||||
|
||||
- Preserve `CorrelationId`, `SubjectId`, `ClientId`, `Plugin`, `Outcome`, `Reason`, and `OccurredAt`.
|
||||
- Store remote address in `remoteAddress` only after classification as PII.
|
||||
|
||||
@@ -64,7 +64,7 @@ See [Unknowns Ranking Algorithm](./unknowns-ranking.md) for the complete formula
|
||||
|
||||
## 6. Storage & CAS
|
||||
|
||||
- Primary store: append-only KV/graph in Mongo (collections `unknowns`, `unknown_metrics`).
|
||||
- Primary store: append-only KV/graph in PostgreSQL (tables `unknowns`, `unknown_metrics`).
|
||||
- Evidence blobs: CAS under `cas://unknowns/{sha256}` for large payloads (runtime traces, partial SBOMs).
|
||||
- Include analyzer fingerprint + schema version in each record for replay.
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ The Packs Registry stores, verifies, and serves Task Pack bundles across environ
|
||||
|
||||
- **Service name:** `StellaOps.PacksRegistry`
|
||||
- **Interfaces:** REST/GraphQL API, OCI-compatible registry endpoints, event streams for mirroring.
|
||||
- **Data stores:** MongoDB (`packs`, `pack_versions`, `pack_provenance`), object storage (bundle blobs, signatures), timeline events.
|
||||
- **Data stores:** PostgreSQL (`packs`, `pack_versions`, `pack_provenance` tables), object storage (bundle blobs, signatures), timeline events.
|
||||
- **Dependencies:** Authority scopes (`packs.*`), Export Center (manifests), DevOps signing service, Notifications (optional).
|
||||
|
||||
---
|
||||
@@ -145,7 +145,7 @@ Extensions must be deterministic and derived from signed bundle data.
|
||||
|
||||
## 9 · Operations
|
||||
|
||||
- **Backups:** Daily snapshots of Mongo collections + object storage, retained for 30 days.
|
||||
- **Backups:** Daily snapshots of PostgreSQL tables + object storage, retained for 30 days.
|
||||
- **Retention:** Old versions retained indefinitely; mark as `deprecated` instead of deleting.
|
||||
- **Maintenance:**
|
||||
- Run `registry vacuum` weekly to prune orphaned blobs.
|
||||
|
||||
Reference in New Issue
Block a user