docs consoliation work
This commit is contained in:
@@ -778,11 +778,13 @@ CREATE INDEX idx_large_table_col ON schema.large_table(column);
|
|||||||
"Persistence": {
|
"Persistence": {
|
||||||
"Authority": "Postgres",
|
"Authority": "Postgres",
|
||||||
"Scheduler": "Postgres",
|
"Scheduler": "Postgres",
|
||||||
"Concelier": "Mongo"
|
"Concelier": "Postgres"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **Note:** MongoDB storage was deprecated in Sprint 4400. All modules now use PostgreSQL. MongoDB-related patterns in this document are retained for historical reference only.
|
||||||
|
|
||||||
### 13.2 Connection String Security
|
### 13.2 Connection String Security
|
||||||
|
|
||||||
**RULE:** Connection strings MUST NOT be logged or included in exception messages.
|
**RULE:** Connection strings MUST NOT be logged or included in exception messages.
|
||||||
|
|||||||
@@ -1,33 +0,0 @@
|
|||||||
# Authority Rate Limit Tuning Outline (2025-10-11)
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
- Drive the remaining work on SEC3.B (Security Guild) and PLG6.DOC (Docs Guild) by capturing the agreed baseline for Authority rate limits and related documentation deliverables.
|
|
||||||
- Provide a single reference for lockout + rate limit interplay so Docs can lift accurate copy into `docs/security/rate-limits.md` and `docs/dev/31_AUTHORITY_PLUGIN_DEVELOPER_GUIDE.md`.
|
|
||||||
|
|
||||||
## Baseline Configuration
|
|
||||||
- `/token`: fixed window, permitLimit 30, window 60s, queueLimit 0. Reduce to 10/60s for untrusted IP ranges; raise to 60/60s only with compensating controls (WAF + active monitoring).
|
|
||||||
- `/authorize`: permitLimit 60, window 60s, queueLimit 10. Intended for interactive browser flows; lowering below 30 requires UX review.
|
|
||||||
- `/internal/*`: disabled by default; recommended 5/60s with queueLimit 0 when bootstrap API exposed.
|
|
||||||
- Configuration path: `authority.security.rateLimiting.<endpoint>` (e.g., `token.permitLimit`). YAML/ENV bindings follow the standard options hierarchy.
|
|
||||||
- Retry metadata: middleware stamps `Retry-After` along with tags `authority.client_id`, `authority.remote_ip`, `authority.endpoint`. Docs should highlight these for operator dashboards.
|
|
||||||
|
|
||||||
## Parameter Matrix
|
|
||||||
| Scenario | permitLimit | window | queueLimit | Notes |
|
|
||||||
|----------|-------------|--------|------------|-------|
|
|
||||||
| Default production | 30 | 60s | 0 | Works with anonymous quota (33 scans/day). |
|
|
||||||
| High-trust clustered IPs | 60 | 60s | 5 | Requires `authorize_rate_limit_hits` alert ≤ 1% sustained. |
|
|
||||||
| Air-gapped lab | 10 | 120s | 0 | Emphasise reduced concurrency + manual queue draining. |
|
|
||||||
| Incident lockdown | 5 | 300s | 0 | Pair with lockout lowering to 3 attempts. |
|
|
||||||
|
|
||||||
## Lockout Interplay
|
|
||||||
- Ensure Docs explain difference between rate limit (per IP/client) vs lockout (per subject). Provide table mapping retry-after headers to recommended support scripts.
|
|
||||||
- Security Guild to define alert thresholds: trigger SOC ticket when 429 rate > 25% for 5 minutes or when limiter emits >100 events/hour per client.
|
|
||||||
|
|
||||||
## Observability
|
|
||||||
- Surface metrics: `aspnetcore_rate_limiting_rejections_total{limiter="authority-token"}` and custom log tags from `AuthorityRateLimiterMetadataMiddleware`.
|
|
||||||
- Recommend dashboard sections: request volume vs. rejections, top offending clientIds, per-endpoint heatmap.
|
|
||||||
|
|
||||||
## Action Items
|
|
||||||
1. Security Guild (SEC3.B): incorporate matrix + alert rules into `docs/security/rate-limits.md`, add YAML examples for override blocks, and cross-link lockout policy doc.
|
|
||||||
2. Docs Guild (PLG6.DOC): update developer guide section 9 with the middleware sequence and reference this outline for retry metadata + tuning guidance.
|
|
||||||
3. Authority Core: validate appsettings sample includes the `security.rateLimiting` block with comments and link back to published doc once ready.
|
|
||||||
@@ -43,7 +43,9 @@ If multiple connectors emit identical constraints, the merge layer should:
|
|||||||
2. Preserve a single normalized rule instance (thanks to `NormalizedVersionRuleEqualityComparer.Instance`).
|
2. Preserve a single normalized rule instance (thanks to `NormalizedVersionRuleEqualityComparer.Instance`).
|
||||||
3. Attach `decisionReason="precedence"` if one source overrides another.
|
3. Attach `decisionReason="precedence"` if one source overrides another.
|
||||||
|
|
||||||
## 4. Example Mongo pipeline
|
## 4. Example Query (Historical MongoDB Reference)
|
||||||
|
|
||||||
|
> **Note:** This section shows historical MongoDB syntax. The current system uses PostgreSQL.
|
||||||
|
|
||||||
Use the following aggregation to locate advisories that affect a specific SemVer:
|
Use the following aggregation to locate advisories that affect a specific SemVer:
|
||||||
|
|
||||||
@@ -102,7 +104,7 @@ For current database schema and query patterns, see `docs/db/SPECIFICATION.md`.
|
|||||||
|
|
||||||
## 8. Storage projection reference
|
## 8. Storage projection reference
|
||||||
|
|
||||||
`NormalizedVersionDocumentFactory` copies each normalized rule into MongoDB using the shape below. Use this as a contract when reviewing connector fixtures or diagnosing merge/storage diffs:
|
`NormalizedVersionDocumentFactory` copies each normalized rule into PostgreSQL using the shape below. Use this as a contract when reviewing connector fixtures or diagnosing merge/storage diffs:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -42,7 +42,7 @@ All predicates capture subjects, issuer metadata, policy context, materials, opt
|
|||||||
- **Policy Studio:** Verification policies author required predicate types, issuers, witness requirements, and freshness windows; simulations show enforcement impact.
|
- **Policy Studio:** Verification policies author required predicate types, issuers, witness requirements, and freshness windows; simulations show enforcement impact.
|
||||||
|
|
||||||
## Storage, offline & air-gap posture
|
## Storage, offline & air-gap posture
|
||||||
- MongoDB stores entry metadata, dedupe keys, and audit events; object storage optionally archives DSSE bundles.
|
- PostgreSQL stores entry metadata, dedupe keys, and audit events; object storage optionally archives DSSE bundles.
|
||||||
- Export Center packages attestation bundles (`stella export attestation-bundle`) for Offline Kit delivery.
|
- Export Center packages attestation bundles (`stella export attestation-bundle`) for Offline Kit delivery.
|
||||||
- Transparency logs can be mirrored; offline mode records gaps and provides compensating controls.
|
- Transparency logs can be mirrored; offline mode records gaps and provides compensating controls.
|
||||||
|
|
||||||
|
|||||||
@@ -482,7 +482,7 @@ The worker honours `bulkVerification.itemDelayMilliseconds` for throttling and r
|
|||||||
* **Proof acquisition**:
|
* **Proof acquisition**:
|
||||||
|
|
||||||
* In synchronous mode, poll the log for inclusion up to `proofTimeoutMs`.
|
* In synchronous mode, poll the log for inclusion up to `proofTimeoutMs`.
|
||||||
* In asynchronous mode, return `pending` and schedule a **proof fetcher** job (Mongo job doc + backoff).
|
* In asynchronous mode, return `pending` and schedule a **proof fetcher** job (PostgreSQL job record + backoff).
|
||||||
* **Mirrors/dual logs**:
|
* **Mirrors/dual logs**:
|
||||||
|
|
||||||
* When `logPreference="both"`, submit to primary and mirror; store **both** UUIDs (primary canonical).
|
* When `logPreference="both"`, submit to primary and mirror; store **both** UUIDs (primary canonical).
|
||||||
|
|||||||
365
docs/modules/attestor/reg-evidence-architecture.md
Normal file
365
docs/modules/attestor/reg-evidence-architecture.md
Normal file
@@ -0,0 +1,365 @@
|
|||||||
|
# Resolved Execution Graph (REG) Evidence Architecture
|
||||||
|
|
||||||
|
> **Status:** Proposed
|
||||||
|
> **Sprint Series:** 8100.0012.*
|
||||||
|
> **Last Updated:** 2025-12-24
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This document describes the architectural enhancements to StellaOps' evidence and attestation subsystems based on the **Merkle-Hash REG** (Resolved Execution Graph) pattern. The core insight: when every node in an execution graph is identified by a **Merkle hash of its normalized content**, evidence can be attached to *content itself* rather than brittle positional indices.
|
||||||
|
|
||||||
|
## Motivation
|
||||||
|
|
||||||
|
### Problem Statement
|
||||||
|
|
||||||
|
StellaOps currently has robust foundations for content-addressed identifiers, Merkle trees, and attestations. However, three gaps limit the system's verifiability:
|
||||||
|
|
||||||
|
1. **Canonicalizer versioning:** Hash collisions possible if canonicalization logic changes
|
||||||
|
2. **Fragmented evidence models:** Different modules use different evidence structures
|
||||||
|
3. **Implicit graph roots:** Merkle roots are computed but not independently attested
|
||||||
|
|
||||||
|
### Target Benefits
|
||||||
|
|
||||||
|
| Benefit | Description |
|
||||||
|
|---------|-------------|
|
||||||
|
| **Reproducible proofs** | Verifiers re-hash node content and check against stored hashes—no fragile indices |
|
||||||
|
| **Dedup & reuse** | Identical content across pipelines collapses to one ID; one piece of evidence justifies many occurrences |
|
||||||
|
| **Audits + time travel** | Snapshot the graph's Merkle root; any replay matching the root guarantees identical nodes & evidence |
|
||||||
|
| **Offline verification** | Attestations are self-contained; no network required to verify |
|
||||||
|
| **Exception stability** | Exceptions bind to content hashes, not "row 42"; stable across rebuilds |
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Component Diagram
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ StellaOps REG Architecture │
|
||||||
|
├─────────────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ │
|
||||||
|
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
|
||||||
|
│ │ StellaOps. │ │ StellaOps. │ │ StellaOps. │ │
|
||||||
|
│ │ Canonical.Json │────▶│ Evidence.Core │◀────│ Attestor. │ │
|
||||||
|
│ │ │ │ │ │ GraphRoot │ │
|
||||||
|
│ │ • CanonVersion │ │ • IEvidence │ │ │ │
|
||||||
|
│ │ • CanonJson │ │ • EvidenceRecord│ │ • IGraphRoot- │ │
|
||||||
|
│ │ • Versioned │ │ • IEvidenceStore│ │ Attestor │ │
|
||||||
|
│ │ Hashing │ │ • Adapters │ │ • Verification │ │
|
||||||
|
│ └──────────────────┘ └──────────────────┘ └──────────────────┘ │
|
||||||
|
│ │ │ │ │
|
||||||
|
│ └────────────────────────┼────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌──────────────────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Content-Addressed Store │ │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ Evidence by subject_node_id │ Graph roots by root_hash │ │
|
||||||
|
│ │ ──────────────────────────── │ ────────────────────────── │ │
|
||||||
|
│ │ sha256:abc... → [evidence] │ sha256:xyz... → attestation │ │
|
||||||
|
│ │ sha256:def... → [evidence] │ sha256:uvw... → attestation │ │
|
||||||
|
│ └──────────────────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
|
||||||
|
│ │ Scanner │ │ Policy │ │ Excititor │ │
|
||||||
|
│ │ ────────────── │ │ ────────────── │ │ ────────────── │ │
|
||||||
|
│ │ • EvidenceBundle│────▶│ • Exception │────▶│ • VexObservation│ │
|
||||||
|
│ │ • ProofSpine │ │ Application │ │ • VexLinkset │ │
|
||||||
|
│ │ • RichGraph │ │ • PolicyVerdict │ │ │ │
|
||||||
|
│ └──────────────────┘ └──────────────────┘ └──────────────────┘ │
|
||||||
|
│ │ │ │ │
|
||||||
|
│ └────────────────────────┼────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌──────────────────┐ │
|
||||||
|
│ │ Unified IEvidence│ │
|
||||||
|
│ │ via Adapters │ │
|
||||||
|
│ └──────────────────┘ │
|
||||||
|
│ │
|
||||||
|
└─────────────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
1. Content Creation
|
||||||
|
─────────────────
|
||||||
|
Component/Node → Canonicalize(content, version) → SHA-256 → node_id
|
||||||
|
|
||||||
|
2. Evidence Attachment
|
||||||
|
────────────────────
|
||||||
|
Analysis Result → IEvidence { subject_node_id, type, payload, provenance }
|
||||||
|
→ EvidenceId = hash(subject + type + payload + provenance)
|
||||||
|
→ Store(evidence)
|
||||||
|
|
||||||
|
3. Graph Construction
|
||||||
|
───────────────────
|
||||||
|
Nodes + Edges → Sort(node_ids) + Sort(edge_ids) → Merkle Tree → root_hash
|
||||||
|
|
||||||
|
4. Graph Attestation
|
||||||
|
──────────────────
|
||||||
|
GraphRootAttestationRequest → GraphRootAttestor
|
||||||
|
→ In-Toto Statement { subject: [root, artifact] }
|
||||||
|
→ DSSE Sign
|
||||||
|
→ Optional: Rekor Publish
|
||||||
|
|
||||||
|
5. Verification
|
||||||
|
─────────────
|
||||||
|
Download attestation → Verify signature
|
||||||
|
→ Fetch nodes/edges by ID
|
||||||
|
→ Recompute Merkle root
|
||||||
|
→ Compare with attested root
|
||||||
|
```
|
||||||
|
|
||||||
|
## Implementation Sprints
|
||||||
|
|
||||||
|
| Sprint | Title | Dependency | Key Deliverables |
|
||||||
|
|--------|-------|------------|------------------|
|
||||||
|
| 8100.0012.0001 | Canonicalizer Versioning | None | `CanonVersion`, `CanonicalizeVersioned()`, backward compatibility |
|
||||||
|
| 8100.0012.0002 | Unified Evidence Model | 0001 | `IEvidence`, `EvidenceRecord`, `IEvidenceStore`, adapters |
|
||||||
|
| 8100.0012.0003 | Graph Root Attestation | 0001, 0002 | `IGraphRootAttestor`, in-toto statements, Rekor integration |
|
||||||
|
|
||||||
|
### Sprint Sequence Diagram
|
||||||
|
|
||||||
|
```
|
||||||
|
Week 1-2 Week 3-4 Week 5-6 Week 7-8
|
||||||
|
──────── ──────── ──────── ────────
|
||||||
|
│ │ │ │
|
||||||
|
│ 0001: Canon │ │ │
|
||||||
|
│ Versioning │ │ │
|
||||||
|
│ ┌─────────┐ │ │ │
|
||||||
|
│ │Wave 0-1 │────┼─▶ 0002: Unified│ │
|
||||||
|
│ │Wave 2-3 │ │ Evidence │ │
|
||||||
|
│ │Wave 4 │ │ ┌─────────┐ │ │
|
||||||
|
│ └─────────┘ │ │Wave 0-1 │──┼─▶ 0003: Graph │
|
||||||
|
│ │ │Wave 2-3 │ │ Root Attest │
|
||||||
|
│ │ │Wave 4 │ │ ┌─────────┐ │
|
||||||
|
│ │ └─────────┘ │ │Wave 0-1 │ │
|
||||||
|
│ │ │ │Wave 2-4 │ │
|
||||||
|
│ │ │ │Wave 5 │ │
|
||||||
|
│ │ │ └─────────┘ │
|
||||||
|
▼ ▼ ▼ ▼
|
||||||
|
```
|
||||||
|
|
||||||
|
## Technical Specifications
|
||||||
|
|
||||||
|
### Canonicalization Version Marker
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"_canonVersion": "stella:canon:v1",
|
||||||
|
"evidenceSource": "stellaops/scanner/reachability",
|
||||||
|
"sbomEntryId": "sha256:abc123...:pkg:npm/lodash@4.17.21",
|
||||||
|
...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The `_canonVersion` field:
|
||||||
|
- Uses underscore prefix to sort first lexicographically
|
||||||
|
- Identifies the exact canonicalization algorithm
|
||||||
|
- Enables graceful version migration
|
||||||
|
- Is included in all content-addressed hash computations
|
||||||
|
|
||||||
|
### Unified Evidence Record
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface IEvidence {
|
||||||
|
// Content-addressed binding
|
||||||
|
subjectNodeId: string; // "sha256:{hex}" - what this evidence is about
|
||||||
|
evidenceId: string; // "sha256:{hex}" - ID of this evidence record
|
||||||
|
|
||||||
|
// Type and payload
|
||||||
|
evidenceType: EvidenceType; // reachability, scan, policy, vex, etc.
|
||||||
|
payload: Uint8Array; // Canonical JSON bytes
|
||||||
|
payloadSchemaVersion: string;
|
||||||
|
|
||||||
|
// Attestation
|
||||||
|
signatures: EvidenceSignature[];
|
||||||
|
|
||||||
|
// Provenance
|
||||||
|
provenance: {
|
||||||
|
generatorId: string;
|
||||||
|
generatorVersion: string;
|
||||||
|
generatedAt: DateTimeOffset;
|
||||||
|
inputsDigest?: string;
|
||||||
|
correlationId?: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
// External storage
|
||||||
|
externalPayloadCid?: string;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Graph Root Attestation (In-Toto)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"_type": "https://in-toto.io/Statement/v1",
|
||||||
|
"subject": [
|
||||||
|
{
|
||||||
|
"name": "sha256:abc123...",
|
||||||
|
"digest": { "sha256": "abc123..." }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "sha256:def456...",
|
||||||
|
"digest": { "sha256": "def456..." }
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"predicateType": "https://stella-ops.org/attestation/graph-root/v1",
|
||||||
|
"predicate": {
|
||||||
|
"graphType": "ResolvedExecutionGraph",
|
||||||
|
"rootHash": "sha256:abc123...",
|
||||||
|
"nodeCount": 1247,
|
||||||
|
"edgeCount": 3891,
|
||||||
|
"nodeIds": ["sha256:...", ...],
|
||||||
|
"edgeIds": ["sha256:...", ...],
|
||||||
|
"inputs": {
|
||||||
|
"policyDigest": "sha256:...",
|
||||||
|
"feedsDigest": "sha256:...",
|
||||||
|
"toolchainDigest": "sha256:...",
|
||||||
|
"paramsDigest": "sha256:..."
|
||||||
|
},
|
||||||
|
"evidenceIds": ["sha256:...", ...],
|
||||||
|
"canonVersion": "stella:canon:v1",
|
||||||
|
"computedAt": "2025-12-24T10:30:00Z",
|
||||||
|
"computedBy": "stellaops/attestor/graph-root",
|
||||||
|
"computedByVersion": "1.0.0"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification Guarantees
|
||||||
|
|
||||||
|
### What Can Be Verified
|
||||||
|
|
||||||
|
| Claim | Verification Method |
|
||||||
|
|-------|---------------------|
|
||||||
|
| "This graph contains exactly these nodes" | Recompute Merkle root from node IDs |
|
||||||
|
| "This evidence applies to node X" | Check evidence.subjectNodeId == node.id |
|
||||||
|
| "This attestation was signed by key K" | Verify DSSE envelope signature |
|
||||||
|
| "This graph was published to transparency log" | Check Rekor inclusion proof |
|
||||||
|
| "These inputs produced this graph" | Check inputs.* digests match expectations |
|
||||||
|
|
||||||
|
### What Cannot Be Verified (Without Additional Trust)
|
||||||
|
|
||||||
|
| Claim | Reason | Mitigation |
|
||||||
|
|-------|--------|------------|
|
||||||
|
| "The analysis was performed correctly" | Semantic, not structural | Trust the generator; audit toolchain |
|
||||||
|
| "No evidence was omitted" | Attester controls content | Require known evidence types; policy enforcement |
|
||||||
|
| "The inputs are authentic" | External dependency | Chain attestations; verify feed signatures |
|
||||||
|
|
||||||
|
## Migration Path
|
||||||
|
|
||||||
|
### Phase 1: Parallel Operation (Sprints 0001-0003)
|
||||||
|
|
||||||
|
- New versioned hashing alongside legacy
|
||||||
|
- New evidence model with adapters for existing types
|
||||||
|
- Graph root attestations optional
|
||||||
|
|
||||||
|
### Phase 2: Gradual Adoption (Post-Sprints)
|
||||||
|
|
||||||
|
- Emit migration warnings for legacy hashes
|
||||||
|
- Prefer IEvidence in new code
|
||||||
|
- Enable graph root attestations by default
|
||||||
|
|
||||||
|
### Phase 3: Deprecation (Future)
|
||||||
|
|
||||||
|
- Remove legacy hash acceptance
|
||||||
|
- Require IEvidence for all evidence storage
|
||||||
|
- Require graph root attestations for verification
|
||||||
|
|
||||||
|
## Compatibility Considerations
|
||||||
|
|
||||||
|
### Existing Attestations
|
||||||
|
|
||||||
|
Attestations generated before canonicalizer versioning remain valid:
|
||||||
|
- Verification detects legacy format (no `_canonVersion` field)
|
||||||
|
- Falls back to legacy canonicalization
|
||||||
|
- Logs warning for migration tracking
|
||||||
|
|
||||||
|
### Existing Evidence
|
||||||
|
|
||||||
|
Existing evidence types (`EvidenceBundle`, `EvidenceStatement`, etc.) continue working:
|
||||||
|
- Adapters convert to `IEvidence` on demand
|
||||||
|
- Original types remain in storage
|
||||||
|
- Gradual migration via write-through
|
||||||
|
|
||||||
|
### API Stability
|
||||||
|
|
||||||
|
Public APIs remain backward compatible:
|
||||||
|
- New methods added, not changed
|
||||||
|
- Optional parameters for new features
|
||||||
|
- Explicit opt-in for graph root attestations
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
| Operation | Impact | Mitigation |
|
||||||
|
|-----------|--------|------------|
|
||||||
|
| Version field injection | ~100 bytes per hash | Negligible |
|
||||||
|
| Merkle tree computation | O(n log n) for n nodes | Existing algorithm; no change |
|
||||||
|
| Graph root attestation | +1 DSSE sign per graph | Batching; async |
|
||||||
|
| Evidence store queries | Index on subject_node_id | Composite index; pagination |
|
||||||
|
| Rekor publishing | Network latency | Optional; async; batching |
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
### Threat Model
|
||||||
|
|
||||||
|
| Threat | Mitigation |
|
||||||
|
|--------|------------|
|
||||||
|
| Hash collision attacks | SHA-256 with 256-bit security; version namespacing |
|
||||||
|
| Signature forgery | DSSE with ECDSA/EdDSA; key rotation |
|
||||||
|
| Evidence tampering | Content-addressed storage; Merkle verification |
|
||||||
|
| Replay attacks | Timestamp in provenance; Rekor log index |
|
||||||
|
| Canonicalization attacks | RFC 8785 compliance; explicit versioning |
|
||||||
|
|
||||||
|
### Key Management
|
||||||
|
|
||||||
|
Graph root attestations use the existing Signer module:
|
||||||
|
- Keys managed via Authority plugin
|
||||||
|
- Rotation policy applies
|
||||||
|
- Certificate chains optional for external verification
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [RFC 8785 - JSON Canonicalization Scheme (JCS)](https://datatracker.ietf.org/doc/html/rfc8785)
|
||||||
|
- [in-toto Attestation Framework](https://github.com/in-toto/attestation)
|
||||||
|
- [DSSE - Dead Simple Signing Envelope](https://github.com/secure-systems-lab/dsse)
|
||||||
|
- [Sigstore Rekor](https://docs.sigstore.dev/rekor/overview/)
|
||||||
|
- [Merkle Tree - Wikipedia](https://en.wikipedia.org/wiki/Merkle_tree)
|
||||||
|
|
||||||
|
## Appendix A: File Locations
|
||||||
|
|
||||||
|
| Component | Path |
|
||||||
|
|-----------|------|
|
||||||
|
| CanonVersion | `src/__Libraries/StellaOps.Canonical.Json/CanonVersion.cs` |
|
||||||
|
| CanonJson (versioned) | `src/__Libraries/StellaOps.Canonical.Json/CanonJson.cs` |
|
||||||
|
| IEvidence | `src/__Libraries/StellaOps.Evidence.Core/IEvidence.cs` |
|
||||||
|
| EvidenceRecord | `src/__Libraries/StellaOps.Evidence.Core/EvidenceRecord.cs` |
|
||||||
|
| IEvidenceStore | `src/__Libraries/StellaOps.Evidence.Core/IEvidenceStore.cs` |
|
||||||
|
| Adapters | `src/__Libraries/StellaOps.Evidence.Core/Adapters/` |
|
||||||
|
| IGraphRootAttestor | `src/Attestor/__Libraries/StellaOps.Attestor.GraphRoot/IGraphRootAttestor.cs` |
|
||||||
|
| GraphRootAttestation | `src/Attestor/__Libraries/StellaOps.Attestor.GraphRoot/Models/` |
|
||||||
|
|
||||||
|
## Appendix B: Example Evidence Queries
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
// Get all reachability evidence for a package
|
||||||
|
var evidence = await evidenceStore.GetBySubjectAsync(
|
||||||
|
subjectNodeId: "sha256:abc123...pkg:npm/lodash@4.17.21",
|
||||||
|
typeFilter: EvidenceType.Reachability);
|
||||||
|
|
||||||
|
// Verify a graph root attestation
|
||||||
|
var result = await graphRootAttestor.VerifyAsync(
|
||||||
|
envelope: downloadedEnvelope,
|
||||||
|
nodes: fetchedNodes,
|
||||||
|
edges: fetchedEdges);
|
||||||
|
|
||||||
|
if (!result.IsValid)
|
||||||
|
throw new VerificationException(result.FailureReason);
|
||||||
|
|
||||||
|
// Check if evidence exists before creating duplicate
|
||||||
|
if (!await evidenceStore.ExistsAsync(subjectNodeId, EvidenceType.Scan))
|
||||||
|
{
|
||||||
|
await evidenceStore.StoreAsync(newEvidence);
|
||||||
|
}
|
||||||
|
```
|
||||||
@@ -1,5 +1,7 @@
|
|||||||
# Attestor TTL Validation Runbook
|
# Attestor TTL Validation Runbook
|
||||||
|
|
||||||
|
> **DEPRECATED:** This runbook tests MongoDB TTL indexes, which are no longer used. Attestor now uses PostgreSQL for persistence (Sprint 4400). See `docs/db/SPECIFICATION.md` for current database schema.
|
||||||
|
|
||||||
> **Purpose:** confirm MongoDB TTL indexes and Redis expirations for the attestation dedupe store behave as expected on a production-like stack.
|
> **Purpose:** confirm MongoDB TTL indexes and Redis expirations for the attestation dedupe store behave as expected on a production-like stack.
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|||||||
@@ -25,7 +25,7 @@ Authority is the platform OIDC/OAuth2 control plane that mints short-lived, send
|
|||||||
- Scheduler/Scanner for machine-to-machine scope enforcement.
|
- Scheduler/Scanner for machine-to-machine scope enforcement.
|
||||||
|
|
||||||
## Operational notes
|
## Operational notes
|
||||||
- MongoDB for tenant, client, and token state.
|
- PostgreSQL (schema `authority`) for tenant, client, and token state.
|
||||||
- Key material in KMS/HSM with rotation runbooks (`operations/key-rotation.md`).
|
- Key material in KMS/HSM with rotation runbooks (`operations/key-rotation.md`).
|
||||||
- Monitoring runbook (`operations/monitoring.md`) and offline-import Grafana JSON (`operations/grafana-dashboard.json`).
|
- Monitoring runbook (`operations/monitoring.md`) and offline-import Grafana JSON (`operations/grafana-dashboard.json`).
|
||||||
|
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ Concelier ingests signed advisories from dozens of sources and converts them int
|
|||||||
- **2025-11-07:** Paragraph-anchored `/advisories/{advisoryKey}/chunks` endpoint shipped for Advisory AI paragraph retrieval. Details and rollout notes live in [`../../updates/2025-11-07-concelier-advisory-chunks.md`](../../updates/2025-11-07-concelier-advisory-chunks.md).
|
- **2025-11-07:** Paragraph-anchored `/advisories/{advisoryKey}/chunks` endpoint shipped for Advisory AI paragraph retrieval. Details and rollout notes live in [`../../updates/2025-11-07-concelier-advisory-chunks.md`](../../updates/2025-11-07-concelier-advisory-chunks.md).
|
||||||
|
|
||||||
## Integrations & dependencies
|
## Integrations & dependencies
|
||||||
- MongoDB for canonical observations and schedules.
|
- PostgreSQL (schema `vuln`) for canonical observations and schedules.
|
||||||
- Policy Engine / Export Center / CLI for evidence consumption.
|
- Policy Engine / Export Center / CLI for evidence consumption.
|
||||||
- Notify and UI for advisory deltas.
|
- Notify and UI for advisory deltas.
|
||||||
|
|
||||||
|
|||||||
@@ -131,7 +131,7 @@ Expect all logs at `Information`. Ensure OTEL exporters include the scope `Stell
|
|||||||
- Canonical conflict rules: `src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md`.
|
- Canonical conflict rules: `src/DEDUP_CONFLICTS_RESOLUTION_ALGO.md`.
|
||||||
- Merge engine internals: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/AdvisoryPrecedenceMerger.cs`.
|
- Merge engine internals: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/AdvisoryPrecedenceMerger.cs`.
|
||||||
- Metrics definitions: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/AdvisoryMergeService.cs` (identity conflicts) and `AdvisoryPrecedenceMerger`.
|
- Metrics definitions: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/AdvisoryMergeService.cs` (identity conflicts) and `AdvisoryPrecedenceMerger`.
|
||||||
- Storage audit trail: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/MergeEventWriter.cs`, `src/Concelier/__Libraries/StellaOps.Concelier.Storage.Mongo/MergeEvents`.
|
- Storage audit trail: `src/Concelier/__Libraries/StellaOps.Concelier.Merge/Services/MergeEventWriter.cs`, `src/Concelier/__Libraries/StellaOps.Concelier.Storage/MergeEvents`.
|
||||||
|
|
||||||
Keep this runbook synchronized with future sprint notes and update alert thresholds as baseline volumes change.
|
Keep this runbook synchronized with future sprint notes and update alert thresholds as baseline volumes change.
|
||||||
|
|
||||||
|
|||||||
@@ -42,10 +42,10 @@ concelier:
|
|||||||
- `Apple software index fetch … processed=X newDocuments=Y`
|
- `Apple software index fetch … processed=X newDocuments=Y`
|
||||||
- `Apple advisory parse complete … aliases=… affected=…`
|
- `Apple advisory parse complete … aliases=… affected=…`
|
||||||
- `Mapped Apple advisory … pendingMappings=0`
|
- `Mapped Apple advisory … pendingMappings=0`
|
||||||
6. Confirm MongoDB state:
|
6. Confirm PostgreSQL state (schema `vuln`):
|
||||||
- `raw_documents` store contains the HT article HTML with metadata (`apple.articleId`, `apple.postingDate`).
|
- `raw_documents` table contains the HT article HTML with metadata (`apple.articleId`, `apple.postingDate`).
|
||||||
- `dtos` store has `schemaVersion="apple.security.update.v1"`.
|
- `dtos` table has `schemaVersion="apple.security.update.v1"`.
|
||||||
- `advisories` collection includes keys `HTxxxxxx` with normalized SemVer rules.
|
- `advisories` table includes keys `HTxxxxxx` with normalized SemVer rules.
|
||||||
- `source_states` entry for `apple` shows a recent `cursor.lastPosted`.
|
- `source_states` entry for `apple` shows a recent `cursor.lastPosted`.
|
||||||
|
|
||||||
## 3. Production Monitoring
|
## 3. Production Monitoring
|
||||||
|
|||||||
@@ -142,5 +142,5 @@ operating offline.
|
|||||||
1. Observe `certbund.feed.fetch.success` and `certbund.detail.fetch.success` increments after runs; `certbund.feed.coverage.days` should hover near the observed RSS window.
|
1. Observe `certbund.feed.fetch.success` and `certbund.detail.fetch.success` increments after runs; `certbund.feed.coverage.days` should hover near the observed RSS window.
|
||||||
2. Ensure summary logs report `truncated=false` in steady state—`true` indicates the fetch cap was hit.
|
2. Ensure summary logs report `truncated=false` in steady state—`true` indicates the fetch cap was hit.
|
||||||
3. During backfills, watch `certbund.feed.enqueued.count` trend to zero.
|
3. During backfills, watch `certbund.feed.enqueued.count` trend to zero.
|
||||||
4. Spot-check stored advisories in Mongo to confirm `language="de"` and reference URLs match the portal detail endpoint.
|
4. Spot-check stored advisories in PostgreSQL (schema `vuln`) to confirm `language="de"` and reference URLs match the portal detail endpoint.
|
||||||
5. For Offline Kit exports, validate SHA256 hashes before distribution.
|
5. For Offline Kit exports, validate SHA256 hashes before distribution.
|
||||||
|
|||||||
@@ -42,7 +42,7 @@ concelier:
|
|||||||
- `cve.map.success`
|
- `cve.map.success`
|
||||||
4. Verify Prometheus shows matching `concelier.source.http.requests_total{concelier_source="cve"}` deltas (list vs detail phases) while `concelier.source.http.failures_total{concelier_source="cve"}` stays flat.
|
4. Verify Prometheus shows matching `concelier.source.http.requests_total{concelier_source="cve"}` deltas (list vs detail phases) while `concelier.source.http.failures_total{concelier_source="cve"}` stays flat.
|
||||||
5. Confirm the info-level summary log `CVEs fetch window … pages=X detailDocuments=Y detailFailures=Z` appears once per fetch run and shows `detailFailures=0`.
|
5. Confirm the info-level summary log `CVEs fetch window … pages=X detailDocuments=Y detailFailures=Z` appears once per fetch run and shows `detailFailures=0`.
|
||||||
6. Verify the MongoDB advisory store contains fresh CVE advisories (`advisoryKey` prefix `cve/`) and that the source cursor (`source_states` collection) advanced.
|
6. Verify the PostgreSQL advisory store (schema `vuln`) contains fresh CVE advisories (`advisoryKey` prefix `cve/`) and that the source cursor (`source_states` table) advanced.
|
||||||
|
|
||||||
### 1.3 Production Monitoring
|
### 1.3 Production Monitoring
|
||||||
|
|
||||||
@@ -115,7 +115,7 @@ Treat repeated schema failures or growing anomaly counts as an upstream regressi
|
|||||||
- `kev.map.advisories` (tag `catalogVersion`)
|
- `kev.map.advisories` (tag `catalogVersion`)
|
||||||
4. Confirm `concelier.source.http.requests_total{concelier_source="kev"}` increments once per fetch and that the paired `concelier.source.http.failures_total` stays flat (zero increase).
|
4. Confirm `concelier.source.http.requests_total{concelier_source="kev"}` increments once per fetch and that the paired `concelier.source.http.failures_total` stays flat (zero increase).
|
||||||
5. Inspect the info logs `Fetched KEV catalog document … pendingDocuments=…` and `Parsed KEV catalog document … entries=…`—they should appear exactly once per run and `Mapped X/Y… skipped=0` should match the `kev.map.advisories` delta.
|
5. Inspect the info logs `Fetched KEV catalog document … pendingDocuments=…` and `Parsed KEV catalog document … entries=…`—they should appear exactly once per run and `Mapped X/Y… skipped=0` should match the `kev.map.advisories` delta.
|
||||||
6. Confirm MongoDB documents exist for the catalog JSON (`raw_documents` & `dtos`) and that advisories with prefix `kev/` are written.
|
6. Confirm PostgreSQL records exist for the catalog JSON (`raw_documents` and `dtos` tables in schema `vuln`) and that advisories with prefix `kev/` are written.
|
||||||
|
|
||||||
### 2.4 Production Monitoring
|
### 2.4 Production Monitoring
|
||||||
|
|
||||||
|
|||||||
@@ -38,10 +38,10 @@ concelier:
|
|||||||
- `KISA fetched detail for {Idx} … category={Category}`
|
- `KISA fetched detail for {Idx} … category={Category}`
|
||||||
- `KISA mapped advisory {AdvisoryId} (severity={Severity})`
|
- `KISA mapped advisory {AdvisoryId} (severity={Severity})`
|
||||||
- Absence of warnings such as `document missing GridFS payload`.
|
- Absence of warnings such as `document missing GridFS payload`.
|
||||||
5. Validate MongoDB state:
|
5. Validate PostgreSQL state (schema `vuln`):
|
||||||
- `raw_documents.metadata` has `kisa.idx`, `kisa.category`, `kisa.title`.
|
- `raw_documents` table metadata has `kisa.idx`, `kisa.category`, `kisa.title`.
|
||||||
- DTO store contains `schemaVersion="kisa.detail.v1"`.
|
- `dtos` table contains `schemaVersion="kisa.detail.v1"`.
|
||||||
- Advisories include aliases (`IDX`, CVE) and `language="ko"`.
|
- `advisories` table includes aliases (`IDX`, CVE) and `language="ko"`.
|
||||||
- `source_states` entry for `kisa` shows recent `cursor.lastFetchAt`.
|
- `source_states` entry for `kisa` shows recent `cursor.lastFetchAt`.
|
||||||
|
|
||||||
## 3. Production Monitoring
|
## 3. Production Monitoring
|
||||||
|
|||||||
@@ -72,8 +72,8 @@ Run the helper:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
dotnet run --project src/Tools/SourceStateSeeder -- \
|
dotnet run --project src/Tools/SourceStateSeeder -- \
|
||||||
--connection-string "mongodb://localhost:27017" \
|
--connection-string "Host=localhost;Database=stellaops;Username=stella;Password=..." \
|
||||||
--database concelier \
|
--schema vuln \
|
||||||
--input seeds/msrc-backfill.json
|
--input seeds/msrc-backfill.json
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -1,19 +1,19 @@
|
|||||||
# Observation Event Transport (advisory.observation.updated@1)
|
# Observation Event Transport (advisory.observation.updated@1)
|
||||||
|
|
||||||
Purpose: document how to emit `advisory.observation.updated@1` events via Mongo outbox with optional NATS JetStream transport.
|
Purpose: document how to emit `advisory.observation.updated@1` events via PostgreSQL outbox with optional NATS JetStream transport.
|
||||||
|
|
||||||
## Configuration (appsettings.yaml / config)
|
## Configuration (appsettings.yaml / config)
|
||||||
```yaml
|
```yaml
|
||||||
advisoryObservationEvents:
|
advisoryObservationEvents:
|
||||||
enabled: false # set true to publish beyond Mongo outbox
|
enabled: false # set true to publish beyond PostgreSQL outbox
|
||||||
transport: "mongo" # "mongo" (no-op publisher) or "nats"
|
transport: "postgres" # "postgres" (no-op publisher) or "nats"
|
||||||
natsUrl: "nats://127.0.0.1:4222"
|
natsUrl: "nats://127.0.0.1:4222"
|
||||||
subject: "concelier.advisory.observation.updated.v1"
|
subject: "concelier.advisory.observation.updated.v1"
|
||||||
deadLetterSubject: "concelier.advisory.observation.updated.dead.v1"
|
deadLetterSubject: "concelier.advisory.observation.updated.dead.v1"
|
||||||
stream: "CONCELIER_OBS"
|
stream: "CONCELIER_OBS"
|
||||||
```
|
```
|
||||||
|
|
||||||
Defaults: disabled, transport `mongo`; subject/stream as above.
|
Defaults: disabled, transport `postgres`; subject/stream as above.
|
||||||
|
|
||||||
## Flow
|
## Flow
|
||||||
1) Observation sink writes event to `advisory_observation_events` (idempotent on `observationHash`).
|
1) Observation sink writes event to `advisory_observation_events` (idempotent on `observationHash`).
|
||||||
@@ -24,7 +24,7 @@ Defaults: disabled, transport `mongo`; subject/stream as above.
|
|||||||
- Ensure NATS JetStream is reachable before enabling `transport: nats` to avoid retry noise.
|
- Ensure NATS JetStream is reachable before enabling `transport: nats` to avoid retry noise.
|
||||||
- Stream is auto-created if missing with current subject; size capped at 512 KiB per message.
|
- Stream is auto-created if missing with current subject; size capped at 512 KiB per message.
|
||||||
- Dead-letter subject reserved; not yet wired—keep for future schema validation failures.
|
- Dead-letter subject reserved; not yet wired—keep for future schema validation failures.
|
||||||
- Backlog monitoring: count documents in `advisory_observation_events` with `publishedAt: null`.
|
- Backlog monitoring: count rows in `advisory_observation_events` table with `published_at IS NULL`.
|
||||||
|
|
||||||
## Testing
|
## Testing
|
||||||
- Without NATS: leave `enabled=false`; app continues writing outbox only.
|
- Without NATS: leave `enabled=false`; app continues writing outbox only.
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
> Draws from the AOC guardrails, Orchestrator, Export Center, and Observability module plans to describe how Stella Ops is built, signed, distributed, and operated.
|
> Draws from the AOC guardrails, Orchestrator, Export Center, and Observability module plans to describe how Stella Ops is built, signed, distributed, and operated.
|
||||||
|
|
||||||
> **Scope.** Implementation‑ready blueprint for **how Stella Ops is built, versioned, signed, distributed, upgraded, licensed (PoE)**, and operated in customer environments (online and air‑gapped). Covers reproducible builds, supply‑chain attestations, registries, offline kits, migration/rollback, artifact lifecycle (RustFS default + Mongo, S3 fallback), monitoring SLOs, and customer activation.
|
> **Scope.** Implementation‑ready blueprint for **how Stella Ops is built, versioned, signed, distributed, upgraded, licensed (PoE)**, and operated in customer environments (online and air‑gapped). Covers reproducible builds, supply‑chain attestations, registries, offline kits, migration/rollback, artifact lifecycle (RustFS default + PostgreSQL, S3 fallback), monitoring SLOs, and customer activation.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -180,9 +180,9 @@ helm install stella stellaops/platform \
|
|||||||
--set global.channel=stable \
|
--set global.channel=stable \
|
||||||
--set authority.issuer=https://authority.stella.local \
|
--set authority.issuer=https://authority.stella.local \
|
||||||
--set scanner.minio.endpoint=http://minio.stella.local:9000 \
|
--set scanner.minio.endpoint=http://minio.stella.local:9000 \
|
||||||
--set scanner.mongo.uri=mongodb://mongo/scanner \
|
--set scanner.mongo.uri=postgresb://mongo/scanner \
|
||||||
--set concelier.mongo.uri=mongodb://mongo/concelier \
|
--set concelier.mongo.uri=postgresb://mongo/concelier \
|
||||||
--set excititor.mongo.uri=mongodb://mongo/excititor
|
--set excititor.mongo.uri=postgresb://mongo/excititor
|
||||||
```
|
```
|
||||||
|
|
||||||
* Post‑install job registers **Authority clients** (Scanner, Signer, Attestor, UI) and prints **bootstrap** URLs and client credentials (sealed secrets).
|
* Post‑install job registers **Authority clients** (Scanner, Signer, Attestor, UI) and prints **bootstrap** URLs and client credentials (sealed secrets).
|
||||||
@@ -210,7 +210,7 @@ helm install stella stellaops/platform \
|
|||||||
|
|
||||||
* Images referenced by **digest**; keep previous release manifest `K` versions back.
|
* Images referenced by **digest**; keep previous release manifest `K` versions back.
|
||||||
* `helm rollback` or compose `docker compose -f release-K.yml up -d`.
|
* `helm rollback` or compose `docker compose -f release-K.yml up -d`.
|
||||||
* Mongo migrations are additive; **no destructive changes** within a single minor.
|
* PostgreSQL migrations are additive; **no destructive changes** within a single minor.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -268,7 +268,7 @@ Signer validates **scanner** image’s cosign identity + calendar tag for **rele
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 7) Artifact lifecycle & storage (RustFS/Mongo)
|
## 7) Artifact lifecycle & storage (RustFS/PostgreSQL)
|
||||||
|
|
||||||
### 7.1 Buckets & prefixes (RustFS)
|
### 7.1 Buckets & prefixes (RustFS)
|
||||||
|
|
||||||
@@ -306,16 +306,16 @@ rustfs://stellaops/
|
|||||||
> **Migration note.** Follow `docs/modules/scanner/operations/rustfs-migration.md` when transitioning existing
|
> **Migration note.** Follow `docs/modules/scanner/operations/rustfs-migration.md` when transitioning existing
|
||||||
> MinIO buckets to RustFS. The provided migrator is idempotent and safe to rerun per prefix.
|
> MinIO buckets to RustFS. The provided migrator is idempotent and safe to rerun per prefix.
|
||||||
|
|
||||||
### 7.4 Mongo retention
|
### 7.4 PostgreSQL retention
|
||||||
|
|
||||||
* **Scanner**: `runtime.events` use TTL (e.g., 30–90 days); **catalog** permanent.
|
* **Scanner**: `runtime.events` use TTL (e.g., 30–90 days); **catalog** permanent.
|
||||||
* **Concelier/Excititor**: raw docs keep **last N windows**; canonical stores permanent.
|
* **Concelier/Excititor**: raw docs keep **last N windows**; canonical stores permanent.
|
||||||
* **Attestor**: `entries` permanent; `dedupe` TTL 24–48h.
|
* **Attestor**: `entries` permanent; `dedupe` TTL 24–48h.
|
||||||
|
|
||||||
### 7.5 Mongo server baseline
|
### 7.5 PostgreSQL server baseline
|
||||||
|
|
||||||
* **Minimum supported server:** MongoDB **4.2+**. Driver 3.5.0 removes compatibility shims for 4.0; upstream has already announced 4.0 support will be dropped in upcoming C# driver releases. citeturn1open1
|
* **Minimum supported server:** MongoDB **4.2+**. Driver 3.5.0 removes compatibility shims for 4.0; upstream has already announced 4.0 support will be dropped in upcoming C# driver releases. citeturn1open1
|
||||||
* **Deploy images:** Compose/Helm defaults stay on `mongo:7.x`. For air-gapped installs, refresh Offline Kit bundles so the packaged `mongod` matches ≥4.2.
|
* **Deploy images:** Compose/Helm defaults stay on `postgres:16`. For air-gapped installs, refresh Offline Kit bundles so the packaged `postgres` matches ≥4.2.
|
||||||
* **Upgrade guard:** During rollout, verify replica sets reach FCV `4.2` or above before swapping binaries; automation should hard-stop if FCV is <4.2.
|
* **Upgrade guard:** During rollout, verify replica sets reach FCV `4.2` or above before swapping binaries; automation should hard-stop if FCV is <4.2.
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -327,7 +327,7 @@ rustfs://stellaops/
|
|||||||
* **Golden signals**:
|
* **Golden signals**:
|
||||||
|
|
||||||
* **Latency**: token issuance, sign→attest round‑trip, scan enqueue→emit, export build.
|
* **Latency**: token issuance, sign→attest round‑trip, scan enqueue→emit, export build.
|
||||||
* **Saturation**: queue depth, Mongo write IOPS, RustFS throughput / queue depth (or S3 metrics when in fallback mode).
|
* **Saturation**: queue depth, PostgreSQL write IOPS, RustFS throughput / queue depth (or S3 metrics when in fallback mode).
|
||||||
* **Traffic**: scans/min, attestations/min, webhook admits/min.
|
* **Traffic**: scans/min, attestations/min, webhook admits/min.
|
||||||
* **Errors**: 5xx rates, cosign verification failures, Rekor timeouts.
|
* **Errors**: 5xx rates, cosign verification failures, Rekor timeouts.
|
||||||
|
|
||||||
@@ -355,7 +355,7 @@ Prometheus + OTLP; Grafana dashboards ship in the charts.
|
|||||||
|
|
||||||
* **Backups/DR**:
|
* **Backups/DR**:
|
||||||
|
|
||||||
* Mongo nightly snapshots; MinIO versioning + replication (if configured).
|
* PostgreSQL nightly snapshots; MinIO versioning + replication (if configured).
|
||||||
* Restore runbooks tested quarterly with synthetic data.
|
* Restore runbooks tested quarterly with synthetic data.
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -474,7 +474,7 @@ services:
|
|||||||
* `attestor.submit_latency_seconds{quantile=0.95}` < 0.3.
|
* `attestor.submit_latency_seconds{quantile=0.95}` < 0.3.
|
||||||
* `scanner.scan_latency_seconds{quantile=0.95}` < target per image size.
|
* `scanner.scan_latency_seconds{quantile=0.95}` < target per image size.
|
||||||
* `concelier.export.duration_seconds` stable; `excititor.consensus.conflicts_total` not exploding after policy changes.
|
* `concelier.export.duration_seconds` stable; `excititor.consensus.conflicts_total` not exploding after policy changes.
|
||||||
* RustFS request error rate near zero (or `s3_requests_errors_total` when operating against S3); Mongo `opcounters` hit expected baseline.
|
* RustFS request error rate near zero (or `s3_requests_errors_total` when operating against S3); PostgreSQL `pg_stat_bgwriter` counters hit expected baseline.
|
||||||
|
|
||||||
### Appendix B — Upgrade safety checklist
|
### Appendix B — Upgrade safety checklist
|
||||||
|
|
||||||
|
|||||||
@@ -35,7 +35,7 @@ Excititor converts heterogeneous VEX feeds into raw observations and linksets th
|
|||||||
- Notify for VEX-driven alerts.
|
- Notify for VEX-driven alerts.
|
||||||
|
|
||||||
## Operational notes
|
## Operational notes
|
||||||
- MongoDB for observation storage and job metadata.
|
- PostgreSQL (schema `vex`) for observation storage and job metadata.
|
||||||
- Offline kit packaging aligned with Concelier merges.
|
- Offline kit packaging aligned with Concelier merges.
|
||||||
- Connector-specific runbooks (see `docs/modules/concelier/operations/connectors`).
|
- Connector-specific runbooks (see `docs/modules/concelier/operations/connectors`).
|
||||||
- Ubuntu CSAF provenance knobs: [`operations/ubuntu-csaf.md`](operations/ubuntu-csaf.md) captures TrustWeight/Tier, cosign, and fingerprint configuration for the sprint 120 enrichment.
|
- Ubuntu CSAF provenance knobs: [`operations/ubuntu-csaf.md`](operations/ubuntu-csaf.md) captures TrustWeight/Tier, cosign, and fingerprint configuration for the sprint 120 enrichment.
|
||||||
|
|||||||
@@ -1,10 +1,12 @@
|
|||||||
# Excititor · VEX Raw Collection Validator (AOC-19-001/002)
|
# Excititor · VEX Raw Collection Validator (AOC-19-001/002)
|
||||||
|
|
||||||
|
> **DEPRECATED:** This document describes MongoDB validators which are no longer used. Excititor now uses PostgreSQL for persistence (Sprint 4400). Schema validation is now performed via PostgreSQL constraints and check constraints. See `docs/db/SPECIFICATION.md` for current database schema.
|
||||||
|
|
||||||
- **Date:** 2025-11-25
|
- **Date:** 2025-11-25
|
||||||
- **Scope:** EXCITITOR-STORE-AOC-19-001 / 19-002
|
- **Scope:** EXCITITOR-STORE-AOC-19-001 / 19-002
|
||||||
- **Working directory:** `src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo`
|
- **Working directory:** ~~`src/Excititor/__Libraries/StellaOps.Excititor.Storage.Mongo`~~ (deprecated)
|
||||||
|
|
||||||
## What shipped
|
## What shipped (historical)
|
||||||
- `$jsonSchema` validator applied to `vex_raw` (migration `20251125-vex-raw-json-schema`) with `validationAction=warn`, `validationLevel=moderate` to surface contract violations without impacting ingestion.
|
- `$jsonSchema` validator applied to `vex_raw` (migration `20251125-vex-raw-json-schema`) with `validationAction=warn`, `validationLevel=moderate` to surface contract violations without impacting ingestion.
|
||||||
- Schema lives at `docs/modules/excititor/schemas/vex_raw.schema.json` (mirrors Mongo validator fields: digest/id, providerId, format, sourceUri, retrievedAt, optional content/GridFS object id, metadata strings).
|
- Schema lives at `docs/modules/excititor/schemas/vex_raw.schema.json` (mirrors Mongo validator fields: digest/id, providerId, format, sourceUri, retrievedAt, optional content/GridFS object id, metadata strings).
|
||||||
- Migration is auto-registered in DI; hosted migration runner applies it on service start. New collections created with the validator if missing.
|
- Migration is auto-registered in DI; hosted migration runner applies it on service start. New collections created with the validator if missing.
|
||||||
|
|||||||
@@ -1,34 +0,0 @@
|
|||||||
# Excitor agent guide
|
|
||||||
|
|
||||||
## Mission
|
|
||||||
Excitor computes deterministic consensus across VEX claims, preserving conflicts and producing attestable evidence for policy suppression.
|
|
||||||
|
|
||||||
## Key docs
|
|
||||||
- [Module README](./README.md)
|
|
||||||
- [Architecture](./architecture.md)
|
|
||||||
- [Implementation plan](./implementation_plan.md)
|
|
||||||
- [Task board](./TASKS.md)
|
|
||||||
|
|
||||||
## How to get started
|
|
||||||
1. Open sprint file `/docs/implplan/SPRINT_*.md` and locate the stories referencing this module.
|
|
||||||
2. Review ./TASKS.md for local follow-ups and confirm status transitions (TODO → DOING → DONE/BLOCKED).
|
|
||||||
3. Read the architecture and README for domain context before editing code or docs.
|
|
||||||
4. Coordinate cross-module changes in the main /AGENTS.md description and through the sprint plan.
|
|
||||||
|
|
||||||
## Guardrails
|
|
||||||
- Honour the Aggregation-Only Contract where applicable (see ../../ingestion/aggregation-only-contract.md).
|
|
||||||
- Preserve determinism: sort outputs, normalise timestamps (UTC ISO-8601), and avoid machine-specific artefacts.
|
|
||||||
- Keep Offline Kit parity in mind—document air-gapped workflows for any new feature.
|
|
||||||
- Update runbooks/observability assets when operational characteristics change.
|
|
||||||
## Required Reading
|
|
||||||
- `docs/modules/excitor/README.md`
|
|
||||||
- `docs/modules/excitor/architecture.md`
|
|
||||||
- `docs/modules/excitor/implementation_plan.md`
|
|
||||||
- `docs/modules/platform/architecture-overview.md`
|
|
||||||
|
|
||||||
## Working Agreement
|
|
||||||
- 1. Update task status to `DOING`/`DONE` in both correspoding sprint file `/docs/implplan/SPRINT_*.md` and the local `TASKS.md` when you start or finish work.
|
|
||||||
- 2. Review this charter and the Required Reading documents before coding; confirm prerequisites are met.
|
|
||||||
- 3. Keep changes deterministic (stable ordering, timestamps, hashes) and align with offline/air-gap expectations.
|
|
||||||
- 4. Coordinate doc updates, tests, and cross-guild communication whenever contracts or workflows change.
|
|
||||||
- 5. Revert to `TODO` if you pause the task without shipping changes; leave notes in commit/PR descriptions for context.
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# StellaOps Excitor
|
|
||||||
|
|
||||||
Excitor computes deterministic consensus across VEX claims, preserving conflicts and producing attestable evidence for policy suppression.
|
|
||||||
|
|
||||||
## Latest updates (2025-11-05)
|
|
||||||
- Consensus API beta documented with canonical JSON samples and DSSE packaging guidance (`docs/updates/2025-11-05-excitor-consensus-beta.md`).
|
|
||||||
- README links to Link-Not-Merge consensus milestone and preview endpoints for downstream integration.
|
|
||||||
|
|
||||||
## Responsibilities
|
|
||||||
- Ingest Excititor observations and compute per-product consensus snapshots.
|
|
||||||
- Provide APIs for querying canonical VEX positions and conflict sets.
|
|
||||||
- Publish exports and DSSE-ready digests for downstream consumption.
|
|
||||||
- Keep provenance weights and disagreement metadata.
|
|
||||||
|
|
||||||
## Key components
|
|
||||||
- Consensus engine and API host in `StellaOps.Excitor.*` (to-be-implemented).
|
|
||||||
- Storage schema for consensus graphs.
|
|
||||||
- Integration hooks for Policy Engine suppression logic.
|
|
||||||
|
|
||||||
## Integrations & dependencies
|
|
||||||
- Excititor for raw observations.
|
|
||||||
- Policy Engine and UI for suppression stories.
|
|
||||||
- CLI for evidence inspection.
|
|
||||||
|
|
||||||
## Operational notes
|
|
||||||
- Deterministic consensus algorithms (see architecture).
|
|
||||||
- Planned telemetry for disagreement counts and freshness.
|
|
||||||
- Offline exports aligning with Concelier/Excititor timelines.
|
|
||||||
|
|
||||||
## Related resources
|
|
||||||
- ./scoring.md
|
|
||||||
- ../../vex/consensus-json.md (beta consensus payload sample)
|
|
||||||
|
|
||||||
## Backlog references
|
|
||||||
- DOCS-EXCITOR backlog referenced in architecture doc.
|
|
||||||
- CLI parity tracked in ../../TASKS.md (CLI-GRAPH/VEX stories).
|
|
||||||
|
|
||||||
## Epic alignment
|
|
||||||
- **Epic 7 – VEX Consensus Lens:** deliver trust-weighted consensus snapshots, disagreement metadata, and explain APIs.
|
|
||||||
@@ -1,465 +0,0 @@
|
|||||||
# component_architecture_excitor.md — **Stella Ops Excitor** (2025Q4)
|
|
||||||
|
|
||||||
> Built to satisfy Epic 7 – VEX Consensus Lens requirements.
|
|
||||||
|
|
||||||
> **Scope.** This document specifies the **Excitor** service: its purpose, trust model, data structures, APIs, plug‑in contracts, storage schema, normalization/consensus algorithms, performance budgets, testing matrix, and how it integrates with Scanner, Policy, Conselier, and the attestation chain. It is implementation‑ready.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 0) Mission & role in the platform
|
|
||||||
|
|
||||||
**Mission.** Convert heterogeneous **VEX** statements (OpenVEX, CSAF VEX, CycloneDX VEX; vendor/distro/platform sources) into **canonical, queryable claims**; compute **deterministic consensus** per *(vuln, product)*; preserve **conflicts with provenance**; publish **stable, attestable exports** that the backend uses to suppress non‑exploitable findings, prioritize remaining risk, and explain decisions.
|
|
||||||
|
|
||||||
**Boundaries.**
|
|
||||||
|
|
||||||
* Excitor **does not** decide PASS/FAIL. It supplies **evidence** (statuses + justifications + provenance weights).
|
|
||||||
* Excitor preserves **conflicting claims** unchanged; consensus encodes how we would pick, but the raw set is always exportable.
|
|
||||||
* VEX consumption is **backend‑only**: Scanner never applies VEX. The backend’s **Policy Engine** asks Excitor for status evidence and then decides what to show.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1) Inputs, outputs & canonical domain
|
|
||||||
|
|
||||||
### 1.1 Accepted input formats (ingest)
|
|
||||||
|
|
||||||
* **OpenVEX** JSON documents (attested or raw).
|
|
||||||
* **CSAF VEX** 2.x (vendor PSIRTs and distros commonly publish CSAF).
|
|
||||||
* **CycloneDX VEX** 1.4+ (standalone VEX or embedded VEX blocks).
|
|
||||||
* **OCI‑attached attestations** (VEX statements shipped as OCI referrers) — optional connectors.
|
|
||||||
|
|
||||||
All connectors register **source metadata**: provider identity, trust tier, signature expectations (PGP/cosign/PKI), fetch windows, rate limits, and time anchors.
|
|
||||||
|
|
||||||
### 1.2 Canonical model (normalized)
|
|
||||||
|
|
||||||
Every incoming statement becomes a set of **VexClaim** records:
|
|
||||||
|
|
||||||
```
|
|
||||||
VexClaim
|
|
||||||
- providerId // 'redhat', 'suse', 'ubuntu', 'github', 'vendorX'
|
|
||||||
- vulnId // 'CVE-2025-12345', 'GHSA-xxxx', canonicalized
|
|
||||||
- productKey // canonical product identity (see §2.2)
|
|
||||||
- status // affected | not_affected | fixed | under_investigation
|
|
||||||
- justification? // for 'not_affected'/'affected' where provided
|
|
||||||
- introducedVersion? // semantics per provider (range or exact)
|
|
||||||
- fixedVersion? // where provided (range or exact)
|
|
||||||
- lastObserved // timestamp from source or fetch time
|
|
||||||
- provenance // doc digest, signature status, fetch URI, line/offset anchors
|
|
||||||
- evidence[] // raw source snippets for explainability
|
|
||||||
- supersedes? // optional cross-doc chain (docDigest → docDigest)
|
|
||||||
```
|
|
||||||
|
|
||||||
### 1.3 Exports (consumption)
|
|
||||||
|
|
||||||
* **VexConsensus** per `(vulnId, productKey)` with:
|
|
||||||
|
|
||||||
* `rollupStatus` (after policy weights/justification gates),
|
|
||||||
* `sources[]` (winning + losing claims with weights & reasons),
|
|
||||||
* `policyRevisionId` (identifier of the Excitor policy used),
|
|
||||||
* `consensusDigest` (stable SHA‑256 over canonical JSON).
|
|
||||||
* **Raw claims** export for auditing (unchanged, with provenance).
|
|
||||||
* **Provider snapshots** (per source, last N days) for operator debugging.
|
|
||||||
* **Index** optimized for backend joins: `(productKey, vulnId) → (status, confidence, sourceSet)`.
|
|
||||||
|
|
||||||
All exports are **deterministic**, and (optionally) **attested** via DSSE and logged to Rekor v2.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2) Identity model — products & joins
|
|
||||||
|
|
||||||
### 2.1 Vuln identity
|
|
||||||
|
|
||||||
* Accepts **CVE**, **GHSA**, vendor IDs (MSRC, RHSA…), distro IDs (DSA/USN/RHSA…) — normalized to `vulnId` with alias sets.
|
|
||||||
* **Alias graph** maintained (from Conselier) to map vendor/distro IDs → CVE (primary) and to **GHSA** where applicable.
|
|
||||||
|
|
||||||
### 2.2 Product identity (`productKey`)
|
|
||||||
|
|
||||||
* **Primary:** `purl` (Package URL).
|
|
||||||
* **Secondary links:** `cpe`, **OS package NVRA/EVR**, NuGet/Maven/Golang identity, and **OS package name** when purl unavailable.
|
|
||||||
* **Fallback:** `oci:<registry>/<repo>@<digest>` for image‑level VEX.
|
|
||||||
* **Special cases:** kernel modules, firmware, platforms → provider‑specific mapping helpers (connector captures provider’s product taxonomy → canonical `productKey`).
|
|
||||||
|
|
||||||
> Excitor does not invent identities. If a provider cannot be mapped to purl/CPE/NVRA deterministically, we keep the native **product string** and mark the claim as **non‑joinable**; the backend will ignore it unless a policy explicitly whitelists that provider mapping.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3) Storage schema (MongoDB)
|
|
||||||
|
|
||||||
Database: `excitor`
|
|
||||||
|
|
||||||
### 3.1 Collections
|
|
||||||
|
|
||||||
**`vex.providers`**
|
|
||||||
|
|
||||||
```
|
|
||||||
_id: providerId
|
|
||||||
name, homepage, contact
|
|
||||||
trustTier: enum {vendor, distro, platform, hub, attestation}
|
|
||||||
signaturePolicy: { type: pgp|cosign|x509|none, keys[], certs[], cosignKeylessRoots[] }
|
|
||||||
fetch: { baseUrl, kind: http|oci|file, rateLimit, etagSupport, windowDays }
|
|
||||||
enabled: bool
|
|
||||||
createdAt, modifiedAt
|
|
||||||
```
|
|
||||||
|
|
||||||
**`vex.raw`** (immutable raw documents)
|
|
||||||
|
|
||||||
```
|
|
||||||
_id: sha256(doc bytes)
|
|
||||||
providerId
|
|
||||||
uri
|
|
||||||
ingestedAt
|
|
||||||
contentType
|
|
||||||
sig: { verified: bool, method: pgp|cosign|x509|none, keyId|certSubject, bundle? }
|
|
||||||
payload: GridFS pointer (if large)
|
|
||||||
disposition: kept|replaced|superseded
|
|
||||||
correlation: { replaces?: sha256, replacedBy?: sha256 }
|
|
||||||
```
|
|
||||||
|
|
||||||
**`vex.claims`** (normalized rows; dedupe on providerId+vulnId+productKey+docDigest)
|
|
||||||
|
|
||||||
```
|
|
||||||
_id
|
|
||||||
providerId
|
|
||||||
vulnId
|
|
||||||
productKey
|
|
||||||
status
|
|
||||||
justification?
|
|
||||||
introducedVersion?
|
|
||||||
fixedVersion?
|
|
||||||
lastObserved
|
|
||||||
docDigest
|
|
||||||
provenance { uri, line?, pointer?, signatureState }
|
|
||||||
evidence[] { key, value, locator }
|
|
||||||
indices:
|
|
||||||
- {vulnId:1, productKey:1}
|
|
||||||
- {providerId:1, lastObserved:-1}
|
|
||||||
- {status:1}
|
|
||||||
- text index (optional) on evidence.value for debugging
|
|
||||||
```
|
|
||||||
|
|
||||||
**`vex.consensus`** (rollups)
|
|
||||||
|
|
||||||
```
|
|
||||||
_id: sha256(canonical(vulnId, productKey, policyRevision))
|
|
||||||
vulnId
|
|
||||||
productKey
|
|
||||||
rollupStatus
|
|
||||||
sources[]: [
|
|
||||||
{ providerId, status, justification?, weight, lastObserved, accepted:bool, reason }
|
|
||||||
]
|
|
||||||
policyRevisionId
|
|
||||||
evaluatedAt
|
|
||||||
consensusDigest // same as _id
|
|
||||||
indices:
|
|
||||||
- {vulnId:1, productKey:1}
|
|
||||||
- {policyRevisionId:1, evaluatedAt:-1}
|
|
||||||
```
|
|
||||||
|
|
||||||
**`vex.exports`** (manifest of emitted artifacts)
|
|
||||||
|
|
||||||
```
|
|
||||||
_id
|
|
||||||
querySignature
|
|
||||||
format: raw|consensus|index
|
|
||||||
artifactSha256
|
|
||||||
rekor { uuid, index, url }?
|
|
||||||
createdAt
|
|
||||||
policyRevisionId
|
|
||||||
cacheable: bool
|
|
||||||
```
|
|
||||||
|
|
||||||
**`vex.cache`**
|
|
||||||
|
|
||||||
```
|
|
||||||
querySignature -> exportId (for fast reuse)
|
|
||||||
ttl, hits
|
|
||||||
```
|
|
||||||
|
|
||||||
**`vex.migrations`**
|
|
||||||
|
|
||||||
* ordered migrations applied at bootstrap to ensure indexes.
|
|
||||||
|
|
||||||
### 3.2 Indexing strategy
|
|
||||||
|
|
||||||
* Hot path queries use exact `(vulnId, productKey)` and time‑bounded windows; compound indexes cover both.
|
|
||||||
* Providers list view by `lastObserved` for monitoring staleness.
|
|
||||||
* `vex.consensus` keyed by `(vulnId, productKey, policyRevision)` for deterministic reuse.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4) Ingestion pipeline
|
|
||||||
|
|
||||||
### 4.1 Connector contract
|
|
||||||
|
|
||||||
```csharp
|
|
||||||
public interface IVexConnector
|
|
||||||
{
|
|
||||||
string ProviderId { get; }
|
|
||||||
Task FetchAsync(VexConnectorContext ctx, CancellationToken ct); // raw docs
|
|
||||||
Task NormalizeAsync(VexConnectorContext ctx, CancellationToken ct); // raw -> VexClaim[]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
* **Fetch** must implement: window scheduling, conditional GET (ETag/If‑Modified‑Since), rate limiting, retry/backoff.
|
|
||||||
* **Normalize** parses the format, validates schema, maps product identities deterministically, emits `VexClaim` records with **provenance**.
|
|
||||||
|
|
||||||
### 4.2 Signature verification (per provider)
|
|
||||||
|
|
||||||
* **cosign (keyless or keyful)** for OCI referrers or HTTP‑served JSON with Sigstore bundles.
|
|
||||||
* **PGP** (provider keyrings) for distro/vendor feeds that sign docs.
|
|
||||||
* **x509** (mutual TLS / provider‑pinned certs) where applicable.
|
|
||||||
* Signature state is stored on **vex.raw.sig** and copied into **provenance.signatureState** on claims.
|
|
||||||
|
|
||||||
> Claims from sources failing signature policy are marked `"signatureState.verified=false"` and **policy** can down‑weight or ignore them.
|
|
||||||
|
|
||||||
### 4.3 Time discipline
|
|
||||||
|
|
||||||
* For each doc, prefer **provider’s document timestamp**; if absent, use fetch time.
|
|
||||||
* Claims carry `lastObserved` which drives **tie‑breaking** within equal weight tiers.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5) Normalization: product & status semantics
|
|
||||||
|
|
||||||
### 5.1 Product mapping
|
|
||||||
|
|
||||||
* **purl** first; **cpe** second; OS package NVRA/EVR mapping helpers (distro connectors) produce purls via canonical tables (e.g., rpm→purl:rpm, deb→purl:deb).
|
|
||||||
* Where a provider publishes **platform‑level** VEX (e.g., “RHEL 9 not affected”), connectors expand to known product inventory rules (e.g., map to sets of packages/components shipped in the platform). Expansion tables are versioned and kept per provider; every expansion emits **evidence** indicating the rule applied.
|
|
||||||
* If expansion would be speculative, the claim remains **platform‑scoped** with `productKey="platform:redhat:rhel:9"` and is flagged **non‑joinable**; backend can decide to use platform VEX only when Scanner proves the platform runtime.
|
|
||||||
|
|
||||||
### 5.2 Status + justification mapping
|
|
||||||
|
|
||||||
* Canonical **status**: `affected | not_affected | fixed | under_investigation`.
|
|
||||||
* **Justifications** normalized to a controlled vocabulary (CISA‑aligned), e.g.:
|
|
||||||
|
|
||||||
* `component_not_present`
|
|
||||||
* `vulnerable_code_not_in_execute_path`
|
|
||||||
* `vulnerable_configuration_unused`
|
|
||||||
* `inline_mitigation_applied`
|
|
||||||
* `fix_available` (with `fixedVersion`)
|
|
||||||
* `under_investigation`
|
|
||||||
* Providers with free‑text justifications are mapped by deterministic tables; raw text preserved as `evidence`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6) Consensus algorithm
|
|
||||||
|
|
||||||
**Goal:** produce a **stable**, explainable `rollupStatus` per `(vulnId, productKey)` given possibly conflicting claims.
|
|
||||||
|
|
||||||
### 6.1 Inputs
|
|
||||||
|
|
||||||
* Set **S** of `VexClaim` for the key.
|
|
||||||
* **Excitor policy snapshot**:
|
|
||||||
|
|
||||||
* **weights** per provider tier and per provider overrides.
|
|
||||||
* **justification gates** (e.g., require justification for `not_affected` to be acceptable).
|
|
||||||
* **minEvidence** rules (e.g., `not_affected` must come from ≥1 vendor or 2 distros).
|
|
||||||
* **signature requirements** (e.g., require verified signature for ‘fixed’ to be considered).
|
|
||||||
|
|
||||||
### 6.2 Steps
|
|
||||||
|
|
||||||
1. **Filter invalid** claims by signature policy & justification gates → set `S'`.
|
|
||||||
2. **Score** each claim:
|
|
||||||
`score = weight(provider) * freshnessFactor(lastObserved)` where freshnessFactor ∈ [0.8, 1.0] for staleness decay (configurable; small effect).
|
|
||||||
3. **Aggregate** scores per status: `W(status) = Σ score(claims with that status)`.
|
|
||||||
4. **Pick** `rollupStatus = argmax_status W(status)`.
|
|
||||||
5. **Tie‑breakers** (in order):
|
|
||||||
|
|
||||||
* Higher **max single** provider score wins (vendor > distro > platform > hub).
|
|
||||||
* More **recent** lastObserved wins.
|
|
||||||
* Deterministic lexicographic order of status (`fixed` > `not_affected` > `under_investigation` > `affected`) as final tiebreaker.
|
|
||||||
6. **Explain**: mark accepted sources (`accepted=true; reason="weight"`/`"freshness"`), mark rejected sources with explicit `reason` (`"insufficient_justification"`, `"signature_unverified"`, `"lower_weight"`).
|
|
||||||
|
|
||||||
> The algorithm is **pure** given S and policy snapshot; result is reproducible and hashed into `consensusDigest`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 7) Query & export APIs
|
|
||||||
|
|
||||||
All endpoints are versioned under `/api/v1/vex`.
|
|
||||||
|
|
||||||
### 7.1 Query (online)
|
|
||||||
|
|
||||||
```
|
|
||||||
POST /claims/search
|
|
||||||
body: { vulnIds?: string[], productKeys?: string[], providers?: string[], since?: timestamp, limit?: int, pageToken?: string }
|
|
||||||
→ { claims[], nextPageToken? }
|
|
||||||
|
|
||||||
POST /consensus/search
|
|
||||||
body: { vulnIds?: string[], productKeys?: string[], policyRevisionId?: string, since?: timestamp, limit?: int, pageToken?: string }
|
|
||||||
→ { entries[], nextPageToken? }
|
|
||||||
|
|
||||||
POST /excititor/resolve (scope: vex.read)
|
|
||||||
body: { productKeys?: string[], purls?: string[], vulnerabilityIds: string[], policyRevisionId?: string }
|
|
||||||
→ { policy, resolvedAt, results: [ { vulnerabilityId, productKey, status, sources[], conflicts[], decisions[], signals?, summary?, envelope: { artifact, contentSignature?, attestation?, attestationEnvelope?, attestationSignature? } } ] }
|
|
||||||
```
|
|
||||||
|
|
||||||
### 7.2 Exports (cacheable snapshots)
|
|
||||||
|
|
||||||
```
|
|
||||||
POST /exports
|
|
||||||
body: { signature: { vulnFilter?, productFilter?, providers?, since? }, format: raw|consensus|index, policyRevisionId?: string, force?: bool }
|
|
||||||
→ { exportId, artifactSha256, rekor? }
|
|
||||||
|
|
||||||
GET /exports/{exportId} → bytes (application/json or binary index)
|
|
||||||
GET /exports/{exportId}/meta → { signature, policyRevisionId, createdAt, artifactSha256, rekor? }
|
|
||||||
```
|
|
||||||
|
|
||||||
### 7.3 Provider operations
|
|
||||||
|
|
||||||
```
|
|
||||||
GET /providers → provider list & signature policy
|
|
||||||
POST /providers/{id}/refresh → trigger fetch/normalize window
|
|
||||||
GET /providers/{id}/status → last fetch, doc counts, signature stats
|
|
||||||
```
|
|
||||||
|
|
||||||
**Auth:** service‑to‑service via Authority tokens; operator operations via UI/CLI with RBAC.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 8) Attestation integration
|
|
||||||
|
|
||||||
* Exports can be **DSSE‑signed** via **Signer** and logged to **Rekor v2** via **Attestor** (optional but recommended for regulated pipelines).
|
|
||||||
* `vex.exports.rekor` stores `{uuid, index, url}` when present.
|
|
||||||
* **Predicate type**: `https://stella-ops.org/attestations/vex-export/1` with fields:
|
|
||||||
|
|
||||||
* `querySignature`, `policyRevisionId`, `artifactSha256`, `createdAt`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 9) Configuration (YAML)
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
excitor:
|
|
||||||
mongo: { uri: "mongodb://mongo/excitor" }
|
|
||||||
s3:
|
|
||||||
endpoint: http://minio:9000
|
|
||||||
bucket: stellaops
|
|
||||||
policy:
|
|
||||||
weights:
|
|
||||||
vendor: 1.0
|
|
||||||
distro: 0.9
|
|
||||||
platform: 0.7
|
|
||||||
hub: 0.5
|
|
||||||
attestation: 0.6
|
|
||||||
providerOverrides:
|
|
||||||
redhat: 1.0
|
|
||||||
suse: 0.95
|
|
||||||
requireJustificationForNotAffected: true
|
|
||||||
signatureRequiredForFixed: true
|
|
||||||
minEvidence:
|
|
||||||
not_affected:
|
|
||||||
vendorOrTwoDistros: true
|
|
||||||
connectors:
|
|
||||||
- providerId: redhat
|
|
||||||
kind: csaf
|
|
||||||
baseUrl: https://access.redhat.com/security/data/csaf/v2/
|
|
||||||
signaturePolicy: { type: pgp, keys: [ "…redhat-pgp-key…" ] }
|
|
||||||
windowDays: 7
|
|
||||||
- providerId: suse
|
|
||||||
kind: csaf
|
|
||||||
baseUrl: https://ftp.suse.com/pub/projects/security/csaf/
|
|
||||||
signaturePolicy: { type: pgp, keys: [ "…suse-pgp-key…" ] }
|
|
||||||
- providerId: ubuntu
|
|
||||||
kind: openvex
|
|
||||||
baseUrl: https://…/vex/
|
|
||||||
signaturePolicy: { type: none }
|
|
||||||
- providerId: vendorX
|
|
||||||
kind: cyclonedx-vex
|
|
||||||
ociRef: ghcr.io/vendorx/vex@sha256:…
|
|
||||||
signaturePolicy: { type: cosign, cosignKeylessRoots: [ "sigstore-root" ] }
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 10) Security model
|
|
||||||
|
|
||||||
* **Input signature verification** enforced per provider policy (PGP, cosign, x509).
|
|
||||||
* **Connector allowlists**: outbound fetch constrained to configured domains.
|
|
||||||
* **Tenant isolation**: per‑tenant DB prefixes or separate DBs; per‑tenant S3 prefixes; per‑tenant policies.
|
|
||||||
* **AuthN/Z**: Authority‑issued OpToks; RBAC roles (`vex.read`, `vex.admin`, `vex.export`).
|
|
||||||
* **No secrets in logs**; deterministic logging contexts include providerId, docDigest, claim keys.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 11) Performance & scale
|
|
||||||
|
|
||||||
* **Targets:**
|
|
||||||
|
|
||||||
* Normalize 10k VEX claims/minute/core.
|
|
||||||
* Consensus compute ≤ 50 ms for 1k unique `(vuln, product)` pairs in hot cache.
|
|
||||||
* Export (consensus) 1M rows in ≤ 60 s on 8 cores with streaming writer.
|
|
||||||
|
|
||||||
* **Scaling:**
|
|
||||||
|
|
||||||
* WebService handles control APIs; **Worker** background services (same image) execute fetch/normalize in parallel with rate‑limits; Mongo writes batched; upserts by natural keys.
|
|
||||||
* Exports stream straight to S3 (MinIO) with rolling buffers.
|
|
||||||
|
|
||||||
* **Caching:**
|
|
||||||
|
|
||||||
* `vex.cache` maps query signatures → export; TTL to avoid stampedes; optimistic reuse unless `force`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 12) Observability
|
|
||||||
|
|
||||||
* **Metrics:**
|
|
||||||
|
|
||||||
* `vex.ingest.docs_total{provider}`
|
|
||||||
* `vex.normalize.claims_total{provider}`
|
|
||||||
* `vex.signature.failures_total{provider,method}`
|
|
||||||
* `vex.consensus.conflicts_total{vulnId}`
|
|
||||||
* `vex.exports.bytes{format}` / `vex.exports.latency_seconds`
|
|
||||||
* **Tracing:** spans for fetch, verify, parse, map, consensus, export.
|
|
||||||
* **Dashboards:** provider staleness, top conflicting vulns/components, signature posture, export cache hit‑rate.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 13) Testing matrix
|
|
||||||
|
|
||||||
* **Connectors:** golden raw docs → deterministic claims (fixtures per provider/format).
|
|
||||||
* **Signature policies:** valid/invalid PGP/cosign/x509 samples; ensure rejects are recorded but not accepted.
|
|
||||||
* **Normalization edge cases:** platform‑only claims, free‑text justifications, non‑purl products.
|
|
||||||
* **Consensus:** conflict scenarios across tiers; check tie‑breakers; justification gates.
|
|
||||||
* **Performance:** 1M‑row export timing; memory ceilings; stream correctness.
|
|
||||||
* **Determinism:** same inputs + policy → identical `consensusDigest` and export bytes.
|
|
||||||
* **API contract tests:** pagination, filters, RBAC, rate limits.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 14) Integration points
|
|
||||||
|
|
||||||
* **Backend Policy Engine** (in Scanner.WebService): calls `POST /excititor/resolve` (scope `vex.read`) with batched `(purl, vulnId)` pairs to fetch `rollupStatus + sources`.
|
|
||||||
* **Conselier**: provides alias graph (CVE↔vendor IDs) and may supply VEX‑adjacent metadata (e.g., KEV flag) for policy escalation.
|
|
||||||
* **UI**: VEX explorer screens use `/claims/search` and `/consensus/search`; show conflicts & provenance.
|
|
||||||
* **CLI**: `stellaops vex export --consensus --since 7d --out vex.json` for audits.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 15) Failure modes & fallback
|
|
||||||
|
|
||||||
* **Provider unreachable:** stale thresholds trigger warnings; policy can down‑weight stale providers automatically (freshness factor).
|
|
||||||
* **Signature outage:** continue to ingest but mark `signatureState.verified=false`; consensus will likely exclude or down‑weight per policy.
|
|
||||||
* **Schema drift:** unknown fields are preserved as `evidence`; normalization rejects only on **invalid identity** or **status**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 16) Rollout plan (incremental)
|
|
||||||
|
|
||||||
1. **MVP**: OpenVEX + CSAF connectors for 3 major providers (e.g., Red Hat/SUSE/Ubuntu), normalization + consensus + `/excititor/resolve`.
|
|
||||||
2. **Signature policies**: PGP for distros; cosign for OCI.
|
|
||||||
3. **Exports + optional attestation**.
|
|
||||||
4. **CycloneDX VEX** connectors; platform claim expansion tables; UI explorer.
|
|
||||||
5. **Scale hardening**: export indexes; conflict analytics.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 17) Appendix — canonical JSON (stable ordering)
|
|
||||||
|
|
||||||
All exports and consensus entries are serialized via `VexCanonicalJsonSerializer`:
|
|
||||||
|
|
||||||
* UTF‑8 without BOM;
|
|
||||||
* keys sorted (ASCII);
|
|
||||||
* arrays sorted by `(providerId, vulnId, productKey, lastObserved)` unless semantic order mandated;
|
|
||||||
* timestamps in `YYYY‑MM‑DDThh:mm:ssZ`;
|
|
||||||
* no insignificant whitespace.
|
|
||||||
|
|
||||||
@@ -1,65 +0,0 @@
|
|||||||
# Implementation plan — Excitor
|
|
||||||
|
|
||||||
## Delivery phases
|
|
||||||
- **Phase 1 – Connectors & normalization**
|
|
||||||
Build connectors for OpenVEX, CSAF VEX, CycloneDX VEX, OCI attestations; capture provenance, signatures, and source metadata; normalise into `VexClaim`.
|
|
||||||
- **Phase 2 – Mapping & trust registry**
|
|
||||||
Implement product mapping (CPE → purl/version), issuer registry (trust tiers, signatures), scope scoring, and justification taxonomy.
|
|
||||||
- **Phase 3 – Consensus & projections**
|
|
||||||
Deliver consensus computation, conflict preservation, projections (`vex_consensus`, history, provider snapshots), and DSSE events.
|
|
||||||
- **Phase 4 – APIs & integrations**
|
|
||||||
Expose REST/CLI endpoints for claims, consensus, conflicts, exports; integrate Policy Engine, Vuln Explorer, Advisory AI, Export Center.
|
|
||||||
- **Phase 5 – Observability & offline**
|
|
||||||
Ship metrics, logs, traces, dashboards, incident runbooks, Offline Kit bundles, and performance tuning (10M claims/tenant).
|
|
||||||
|
|
||||||
## Work breakdown
|
|
||||||
- **Connectors**
|
|
||||||
- Fetchers for vendor feeds, CSAF repositories, OpenVEX docs, OCI referrers.
|
|
||||||
- Signature verification (PGP, cosign, PKI) per source; schema validation; rate limiting.
|
|
||||||
- Source configuration (trust tier, fetch cadence, blackout windows) stored in metadata registry.
|
|
||||||
- **Normalization**
|
|
||||||
- Canonical `VexClaim` schema with deterministic IDs, provenance, supersedes chains.
|
|
||||||
- Product tree parsing, mapping to canonical product keys and environments.
|
|
||||||
- Justification and scope scoring derived from source semantics.
|
|
||||||
- **Consensus & projections**
|
|
||||||
- Lattice join with precedence rules, conflict tracking, confidence scores, recency decay.
|
|
||||||
- Append-only history, conflict queue, DSSE events (`vex.consensus.updated`).
|
|
||||||
- Export-ready JSONL & DSSE bundles for Offline Kit and Export Center.
|
|
||||||
- **APIs & UX**
|
|
||||||
- REST endpoints (`/claims`, `/consensus`, `/conflicts`, `/providers`) with tenant RBAC.
|
|
||||||
- CLI commands `stella vex claims|consensus|conflicts|export`.
|
|
||||||
- Console modules (list/detail, conflict diagnostics, provider health, simulation hooks).
|
|
||||||
- **Integrations**
|
|
||||||
- Policy Engine trust knobs, Vuln Explorer consensus badges, Advisory AI narrative generation, Notify alerts for conflicts.
|
|
||||||
- Orchestrator jobs for recompute/backfill triggered by Excitor deltas.
|
|
||||||
- **Observability & Ops**
|
|
||||||
- Metrics (ingest latency, signature failure rate, conflict rate, consensus latency).
|
|
||||||
- Logs/traces with tenant/issuer/provenance context.
|
|
||||||
- Runbooks for mapping failures, signature errors, recompute storms, quota exhaustion.
|
|
||||||
|
|
||||||
## Acceptance criteria
|
|
||||||
- Connectors ingest validated VEX statements with signed provenance, deterministic mapping, and tenant isolation.
|
|
||||||
- Consensus outputs reproducible, include conflicts, and integrate with Policy Engine/Vuln Explorer/Export Center.
|
|
||||||
- CLI/Console provide evidence inspection, conflict analysis, and exports; Offline Kit bundles replay verification offline.
|
|
||||||
- Observability dashboards/alerts capture ingest health, trust anomalies, conflict spikes, and performance budgets.
|
|
||||||
- Recompute pipeline handles policy changes and new evidence without dropping deterministic outcomes.
|
|
||||||
|
|
||||||
## Risks & mitigations
|
|
||||||
- **Mapping ambiguity:** maintain scope scores, manual overrides, highlight warnings.
|
|
||||||
- **Signature trust gaps:** issuer registry with auditing, fallback trust policies, tenant overrides.
|
|
||||||
- **Evidence surges:** orchestrator backpressure, prioritised queues, shardable workers.
|
|
||||||
- **Performance regressions:** indexing, caching, load tests, budget enforcement.
|
|
||||||
- **Tenant leakage:** strict RBAC/filters, fuzz tests, compliance reviews.
|
|
||||||
|
|
||||||
## Test strategy
|
|
||||||
- **Unit:** connector parsers, normalization, mapping conversions, lattice operations.
|
|
||||||
- **Property:** randomised evidence ensuring commutative consensus and deterministic digests.
|
|
||||||
- **Integration:** end-to-end pipeline from Excitor to consensus export, policy simulation, conflict handling.
|
|
||||||
- **Performance:** large feed ingestion, recompute stress, CLI export throughput.
|
|
||||||
- **Security:** signature tampering, issuer revocation, RBAC.
|
|
||||||
- **Offline:** export/import verification, DSSE bundle validation.
|
|
||||||
|
|
||||||
## Definition of done
|
|
||||||
- Connectors, normalization, consensus, APIs, and integrations deployed with telemetry, runbooks, and Offline Kit parity.
|
|
||||||
- Documentation (overview, architecture, algorithm, issuer registry, API/CLI, runbooks) updated with imposed rule compliance.
|
|
||||||
- ./TASKS.md and ../../TASKS.md reflect active status and dependencies.
|
|
||||||
@@ -18,7 +18,7 @@
|
|||||||
## 2) Pipelines
|
## 2) Pipelines
|
||||||
|
|
||||||
1. **Ingestion:** Cartographer/SBOM Service emit SBOM snapshots (`sbom_snapshot` events) captured by the Graph Indexer. Ledger lineage references become `SBOM_VERSION_OF` + `SBOM_LINEAGE_*` edges. Advisories/VEX from Concelier/Excititor generate edge updates, policy runs attach overlay metadata.
|
1. **Ingestion:** Cartographer/SBOM Service emit SBOM snapshots (`sbom_snapshot` events) captured by the Graph Indexer. Ledger lineage references become `SBOM_VERSION_OF` + `SBOM_LINEAGE_*` edges. Advisories/VEX from Concelier/Excititor generate edge updates, policy runs attach overlay metadata.
|
||||||
2. **ETL:** Normalises nodes/edges into canonical IDs, deduplicates, enforces tenant partitions, and writes to the graph store (planned: Neo4j-compatible or document + adjacency lists in Mongo).
|
2. **ETL:** Normalises nodes/edges into canonical IDs, deduplicates, enforces tenant partitions, and writes to the graph store (planned: Neo4j-compatible or PostgreSQL adjacency lists).
|
||||||
3. **Overlay computation:** Batch workers build materialised views for frequently used queries (impact lists, saved queries, policy overlays) and store as immutable blobs for Offline Kit exports.
|
3. **Overlay computation:** Batch workers build materialised views for frequently used queries (impact lists, saved queries, policy overlays) and store as immutable blobs for Offline Kit exports.
|
||||||
4. **Diffing:** `graph_diff` jobs compare two snapshots (e.g., pre/post deploy) and generate signed diff manifests for UI/CLI consumption.
|
4. **Diffing:** `graph_diff` jobs compare two snapshots (e.g., pre/post deploy) and generate signed diff manifests for UI/CLI consumption.
|
||||||
5. **Analytics (Runtime & Signals 140.A):** background workers run Louvain-style clustering + degree/betweenness approximations on ingested graphs, emitting overlays per tenant/snapshot and writing cluster ids back to nodes when enabled.
|
5. **Analytics (Runtime & Signals 140.A):** background workers run Louvain-style clustering + degree/betweenness approximations on ingested graphs, emitting overlays per tenant/snapshot and writing cluster ids back to nodes when enabled.
|
||||||
|
|||||||
@@ -57,7 +57,7 @@
|
|||||||
|
|
||||||
5. **Upgrade & rollback**
|
5. **Upgrade & rollback**
|
||||||
- Update Compose images to the desired release manifest (`deploy/releases/*.yaml`), re-run `docker compose config`, then `docker compose up -d`.
|
- Update Compose images to the desired release manifest (`deploy/releases/*.yaml`), re-run `docker compose config`, then `docker compose up -d`.
|
||||||
- Rollbacks follow the same steps with the previous manifest. Mongo collections are backwards compatible within `2025.10.x`.
|
- Rollbacks follow the same steps with the previous manifest. PostgreSQL schemas are backwards compatible within `2025.10.x`.
|
||||||
|
|
||||||
## 3 · Deploy with Helm
|
## 3 · Deploy with Helm
|
||||||
1. **Create or update the secret**
|
1. **Create or update the secret**
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ Notify (Notifications Studio) converts platform events into tenant-scoped alerts
|
|||||||
- **Worker pipeline:** `StellaOps.Notify.Worker` ingests bus events, evaluates match predicates, applies per-tenant throttles, and dispatches deliveries.
|
- **Worker pipeline:** `StellaOps.Notify.Worker` ingests bus events, evaluates match predicates, applies per-tenant throttles, and dispatches deliveries.
|
||||||
- **Connector plug-ins:** Restart-time plug-ins under `StellaOps.Notify.Connectors.*` (Slack, Teams, Email, generic webhook) with health checks and retry policy hints declared in `notify-plugin.json`.
|
- **Connector plug-ins:** Restart-time plug-ins under `StellaOps.Notify.Connectors.*` (Slack, Teams, Email, generic webhook) with health checks and retry policy hints declared in `notify-plugin.json`.
|
||||||
- **Template engine:** Deterministic rendering with safe helpers, locale bundles, and redaction defaults that keep Offline Kit parity.
|
- **Template engine:** Deterministic rendering with safe helpers, locale bundles, and redaction defaults that keep Offline Kit parity.
|
||||||
- **Delivery ledger:** Mongo-backed ledger storing hashed payloads, attempts, throttled/digested markers, and provenance links for audit + exports.
|
- **Delivery ledger:** PostgreSQL-backed ledger storing hashed payloads, attempts, throttled/digested markers, and provenance links for audit + exports.
|
||||||
|
|
||||||
## In progress / upcoming (Sprint 39 focus)
|
## In progress / upcoming (Sprint 39 focus)
|
||||||
- `NOTIFY-SVC-39-001` correlation engine with token-bucket throttles, incident lifecycle, and quiet-hours evaluator.
|
- `NOTIFY-SVC-39-001` correlation engine with token-bucket throttles, incident lifecycle, and quiet-hours evaluator.
|
||||||
@@ -36,7 +36,7 @@ Status for these items is tracked in `src/Notifier/StellaOps.Notifier/TASKS.md`
|
|||||||
- [`docs/updates/2025-10-29-notify-docs.md`](../../updates/2025-10-29-notify-docs.md) — latest release note; follow-ups remain to validate connector metadata, quiet-hours semantics, and simulation payloads once Sprint 39 drops land.
|
- [`docs/updates/2025-10-29-notify-docs.md`](../../updates/2025-10-29-notify-docs.md) — latest release note; follow-ups remain to validate connector metadata, quiet-hours semantics, and simulation payloads once Sprint 39 drops land.
|
||||||
|
|
||||||
## Integrations & dependencies
|
## Integrations & dependencies
|
||||||
- **Storage:** MongoDB (`rules`, `channels`, `deliveries`, `digests`, `throttles`) with change streams for worker snapshots.
|
- **Storage:** PostgreSQL (schema `notify`) for rules, channels, deliveries, digests, and throttles; Valkey for worker coordination.
|
||||||
- **Queues:** Redis Streams or NATS JetStream for ingestion, throttling, and DLQs (`notify.dlq`).
|
- **Queues:** Redis Streams or NATS JetStream for ingestion, throttling, and DLQs (`notify.dlq`).
|
||||||
- **Authority:** OpTok-protected APIs, DPoP-backed CLI/UI scopes (`notify.viewer`, `notify.operator`, `notify.admin`), and secret references for channel credentials.
|
- **Authority:** OpTok-protected APIs, DPoP-backed CLI/UI scopes (`notify.viewer`, `notify.operator`, `notify.admin`), and secret references for channel credentials.
|
||||||
- **Observability:** Prometheus metrics (`notify.sent_total`, `notify.failed_total`, `notify.digest_coalesced_total`, etc.), OTEL traces, and dashboards documented in `docs/notifications/architecture.md#12-observability-prometheus--otel`.
|
- **Observability:** Prometheus metrics (`notify.sent_total`, `notify.failed_total`, `notify.digest_coalesced_total`, etc.), OTEL traces, and dashboards documented in `docs/notifications/architecture.md#12-observability-prometheus--otel`.
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ copied directly onto the hosts that run `StellaOps.Notifier.WebService`.
|
|||||||
`chmod 600`).
|
`chmod 600`).
|
||||||
2. **Drop configuration** – place `notify.yaml` in the location expected by
|
2. **Drop configuration** – place `notify.yaml` in the location expected by
|
||||||
the runtime (`/app/etc/notify.yaml` for the containers we ship). The file
|
the runtime (`/app/etc/notify.yaml` for the containers we ship). The file
|
||||||
assumes MongoDB is reachable at `mongodb://stellaops:airgap-password@mongo:27017`
|
assumes PostgreSQL is reachable at `Host=postgres;Database=stellaops;Username=stellaops;Password=airgap-password`
|
||||||
and Authority at `https://authority.airgap.local` – adjust if your
|
and Authority at `https://authority.airgap.local` – adjust if your
|
||||||
deployment uses different hostnames.
|
deployment uses different hostnames.
|
||||||
3. **Import rule/template** – with the Notify CLI or REST API, import
|
3. **Import rule/template** – with the Notify CLI or REST API, import
|
||||||
|
|||||||
@@ -5,8 +5,8 @@
|
|||||||
## 1) Topology
|
## 1) Topology
|
||||||
|
|
||||||
- **Orchestrator API (`StellaOps.Orchestrator`).** Minimal API providing job state, throttling controls, replay endpoints, and dashboard data. Authenticated via Authority scopes (`orchestrator:*`).
|
- **Orchestrator API (`StellaOps.Orchestrator`).** Minimal API providing job state, throttling controls, replay endpoints, and dashboard data. Authenticated via Authority scopes (`orchestrator:*`).
|
||||||
- **Job ledger (Mongo).** Collections `jobs`, `job_history`, `sources`, `quotas`, `throttles`, `incidents`. Append-only history ensures auditability.
|
- **Job ledger (PostgreSQL).** Tables `jobs`, `job_history`, `sources`, `quotas`, `throttles`, `incidents` (schema `orchestrator`). Append-only history ensures auditability.
|
||||||
- **Queue abstraction.** Supports Mongo queue, Redis Streams, or NATS JetStream (pluggable). Each job carries lease metadata and retry policy.
|
- **Queue abstraction.** Supports Valkey Streams or NATS JetStream (pluggable). Each job carries lease metadata and retry policy.
|
||||||
- **Dashboard feeds.** SSE/GraphQL endpoints supply Console UI with job timelines, throughput, error distributions, and rate-limit status.
|
- **Dashboard feeds.** SSE/GraphQL endpoints supply Console UI with job timelines, throughput, error distributions, and rate-limit status.
|
||||||
|
|
||||||
## 2) Job lifecycle
|
## 2) Job lifecycle
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ Policy Engine compiles and evaluates Stella DSL policies deterministically, prod
|
|||||||
- Shared libraries under `StellaOps.Policy.*` for evaluation, storage, DSL tooling.
|
- Shared libraries under `StellaOps.Policy.*` for evaluation, storage, DSL tooling.
|
||||||
|
|
||||||
## Integrations & dependencies
|
## Integrations & dependencies
|
||||||
- MongoDB findings collections, RustFS explain bundles.
|
- PostgreSQL (schema `policy`) for findings, RustFS explain bundles.
|
||||||
- Scheduler for incremental re-evaluation triggers.
|
- Scheduler for incremental re-evaluation triggers.
|
||||||
- CLI/UI for policy authoring and runs.
|
- CLI/UI for policy authoring and runs.
|
||||||
|
|
||||||
|
|||||||
382
docs/modules/provcache/README.md
Normal file
382
docs/modules/provcache/README.md
Normal file
@@ -0,0 +1,382 @@
|
|||||||
|
# Provcache Module
|
||||||
|
|
||||||
|
> Provenance Cache — Maximizing Trust Evidence Density
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Provcache is a caching layer that maximizes "provenance density" — the amount of trustworthy evidence retained per byte — enabling faster security decisions, offline replays, and smaller air-gap bundles.
|
||||||
|
|
||||||
|
### Key Benefits
|
||||||
|
|
||||||
|
- **Trust Latency**: Warm cache lookups return in single-digit milliseconds
|
||||||
|
- **Bandwidth Efficiency**: Avoid re-fetching bulky SBOMs/attestations
|
||||||
|
- **Offline Operation**: Decisions usable without full SBOM/VEX payloads
|
||||||
|
- **Audit Transparency**: Full evidence chain verifiable via Merkle proofs
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Policy Evaluator │
|
||||||
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
|
||||||
|
│ │ VeriKey │───▶│ Provcache │───▶│ TrustLatticeEngine │ │
|
||||||
|
│ │ Builder │ │ Service │ │ (if cache miss) │ │
|
||||||
|
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────┐
|
||||||
|
│ Provcache Store │
|
||||||
|
│ ┌─────────────┐ ┌────────────────┐ │
|
||||||
|
│ │ Valkey │◀──▶│ Postgres │ │
|
||||||
|
│ │ (read-thru) │ │ (write-behind) │ │
|
||||||
|
│ └─────────────┘ └────────────────┘ │
|
||||||
|
└─────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────┐
|
||||||
|
│ Evidence Chunk Store │
|
||||||
|
│ ┌─────────────────────────────────────┐│
|
||||||
|
│ │ prov_evidence_chunks (Postgres) ││
|
||||||
|
│ │ - Chunked SBOM/VEX/CallGraph ││
|
||||||
|
│ │ - Merkle tree verification ││
|
||||||
|
│ └─────────────────────────────────────┘│
|
||||||
|
└─────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Core Concepts
|
||||||
|
|
||||||
|
### VeriKey (Provenance Identity Key)
|
||||||
|
|
||||||
|
A composite hash that uniquely identifies a provenance decision context:
|
||||||
|
|
||||||
|
```
|
||||||
|
VeriKey = SHA256(
|
||||||
|
source_hash || // Image/artifact digest
|
||||||
|
sbom_hash || // Canonical SBOM hash
|
||||||
|
vex_hash_set_hash || // Sorted VEX statement hashes
|
||||||
|
merge_policy_hash || // PolicyBundle hash
|
||||||
|
signer_set_hash || // Signer certificate hashes
|
||||||
|
time_window // Epoch bucket
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why each component?**
|
||||||
|
|
||||||
|
| Component | Purpose |
|
||||||
|
|-----------|---------|
|
||||||
|
| `source_hash` | Different artifacts → different keys |
|
||||||
|
| `sbom_hash` | SBOM changes (new packages) → new key |
|
||||||
|
| `vex_hash_set` | VEX updates → new key |
|
||||||
|
| `policy_hash` | Policy changes → new key |
|
||||||
|
| `signer_set_hash` | Key rotation → new key (security) |
|
||||||
|
| `time_window` | Temporal bucketing → controlled expiry |
|
||||||
|
|
||||||
|
### DecisionDigest
|
||||||
|
|
||||||
|
Canonicalized representation of an evaluation result:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"digestVersion": "v1",
|
||||||
|
"veriKey": "sha256:abc123...",
|
||||||
|
"verdictHash": "sha256:def456...",
|
||||||
|
"proofRoot": "sha256:789abc...",
|
||||||
|
"replaySeed": {
|
||||||
|
"feedIds": ["cve-2024", "ghsa-2024"],
|
||||||
|
"ruleIds": ["default-policy-v2"]
|
||||||
|
},
|
||||||
|
"trustScore": 85,
|
||||||
|
"createdAt": "2025-12-24T12:00:00Z",
|
||||||
|
"expiresAt": "2025-12-25T12:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Trust Score
|
||||||
|
|
||||||
|
A composite score (0-100) indicating decision confidence:
|
||||||
|
|
||||||
|
| Component | Weight | Calculation |
|
||||||
|
|-----------|--------|-------------|
|
||||||
|
| Reachability | 25% | Call graph coverage, entry points analyzed |
|
||||||
|
| SBOM Completeness | 20% | Package count, license data presence |
|
||||||
|
| VEX Coverage | 20% | Vendor statements, justifications |
|
||||||
|
| Policy Freshness | 15% | Time since last policy update |
|
||||||
|
| Signer Trust | 20% | Key age, reputation, chain validity |
|
||||||
|
|
||||||
|
### Evidence Chunks
|
||||||
|
|
||||||
|
Large evidence (SBOM, VEX, call graphs) is stored in fixed-size chunks:
|
||||||
|
|
||||||
|
- **Default size**: 64 KB per chunk
|
||||||
|
- **Merkle verification**: Each chunk is a Merkle leaf
|
||||||
|
- **Lazy fetch**: Only fetch chunks needed for audit
|
||||||
|
- **LRU eviction**: Old chunks evicted under storage pressure
|
||||||
|
|
||||||
|
## API Reference
|
||||||
|
|
||||||
|
### Endpoints
|
||||||
|
|
||||||
|
| Method | Path | Description |
|
||||||
|
|--------|------|-------------|
|
||||||
|
| GET | `/v1/provcache/{veriKey}` | Lookup cached decision |
|
||||||
|
| POST | `/v1/provcache` | Store decision (idempotent) |
|
||||||
|
| POST | `/v1/provcache/invalidate` | Invalidate by pattern |
|
||||||
|
| GET | `/v1/proofs/{proofRoot}` | List evidence chunks |
|
||||||
|
| GET | `/v1/proofs/{proofRoot}/chunks/{index}` | Download chunk |
|
||||||
|
|
||||||
|
### Cache Lookup Flow
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
sequenceDiagram
|
||||||
|
participant Client
|
||||||
|
participant PolicyEngine
|
||||||
|
participant Provcache
|
||||||
|
participant Valkey
|
||||||
|
participant Postgres
|
||||||
|
participant TrustLattice
|
||||||
|
|
||||||
|
Client->>PolicyEngine: Evaluate(artifact)
|
||||||
|
PolicyEngine->>Provcache: Get(VeriKey)
|
||||||
|
Provcache->>Valkey: GET verikey
|
||||||
|
alt Cache Hit
|
||||||
|
Valkey-->>Provcache: DecisionDigest
|
||||||
|
Provcache-->>PolicyEngine: CacheResult(hit)
|
||||||
|
PolicyEngine-->>Client: Decision (cached)
|
||||||
|
else Cache Miss
|
||||||
|
Valkey-->>Provcache: null
|
||||||
|
Provcache->>Postgres: SELECT * FROM provcache_items
|
||||||
|
alt DB Hit
|
||||||
|
Postgres-->>Provcache: ProvcacheEntry
|
||||||
|
Provcache->>Valkey: SET (backfill)
|
||||||
|
Provcache-->>PolicyEngine: CacheResult(hit, source=postgres)
|
||||||
|
else DB Miss
|
||||||
|
Postgres-->>Provcache: null
|
||||||
|
Provcache-->>PolicyEngine: CacheResult(miss)
|
||||||
|
PolicyEngine->>TrustLattice: Evaluate
|
||||||
|
TrustLattice-->>PolicyEngine: EvaluationResult
|
||||||
|
PolicyEngine->>Provcache: Set(VeriKey, DecisionDigest)
|
||||||
|
Provcache->>Valkey: SET
|
||||||
|
Provcache->>Postgres: INSERT (async)
|
||||||
|
PolicyEngine-->>Client: Decision (computed)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
```
|
||||||
|
|
||||||
|
## Invalidation
|
||||||
|
|
||||||
|
### Automatic Invalidation Triggers
|
||||||
|
|
||||||
|
| Trigger | Event | Scope |
|
||||||
|
|---------|-------|-------|
|
||||||
|
| Signer Revocation | `SignerRevokedEvent` | All entries with matching `signer_set_hash` |
|
||||||
|
| Feed Epoch Advance | `FeedEpochAdvancedEvent` | Entries with older `feed_epoch` |
|
||||||
|
| Policy Update | `PolicyUpdatedEvent` | Entries with matching `policy_hash` |
|
||||||
|
| TTL Expiry | Background job | Entries past `expires_at` |
|
||||||
|
|
||||||
|
### Manual Invalidation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Invalidate by signer
|
||||||
|
POST /v1/provcache/invalidate
|
||||||
|
{
|
||||||
|
"by": "signer_set_hash",
|
||||||
|
"value": "sha256:revoked-signer...",
|
||||||
|
"reason": "key-compromise"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Invalidate by policy
|
||||||
|
POST /v1/provcache/invalidate
|
||||||
|
{
|
||||||
|
"by": "policy_hash",
|
||||||
|
"value": "sha256:old-policy...",
|
||||||
|
"reason": "policy-update"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Air-Gap Integration
|
||||||
|
|
||||||
|
### Export Workflow
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Export minimal proof (digest only)
|
||||||
|
stella prov export --verikey sha256:abc123 --density lite
|
||||||
|
|
||||||
|
# Export with evidence chunks
|
||||||
|
stella prov export --verikey sha256:abc123 --density standard
|
||||||
|
|
||||||
|
# Export full evidence
|
||||||
|
stella prov export --verikey sha256:abc123 --density strict --sign
|
||||||
|
```
|
||||||
|
|
||||||
|
### Import Workflow
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Import and verify Merkle root
|
||||||
|
stella prov import --input proof.bundle
|
||||||
|
|
||||||
|
# Import with lazy chunk fetch
|
||||||
|
stella prov import --input proof-lite.json --lazy-fetch --backend https://api.stellaops.com
|
||||||
|
```
|
||||||
|
|
||||||
|
### Density Levels
|
||||||
|
|
||||||
|
| Level | Contents | Size | Use Case |
|
||||||
|
|-------|----------|------|----------|
|
||||||
|
| `lite` | DecisionDigest + ProofRoot | ~2 KB | Quick verification |
|
||||||
|
| `standard` | + First N chunks | ~200 KB | Normal audit |
|
||||||
|
| `strict` | + All chunks | Variable | Full compliance |
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
provcache:
|
||||||
|
# TTL configuration
|
||||||
|
defaultTtl: 24h
|
||||||
|
maxTtl: 168h # 7 days
|
||||||
|
timeWindowBucket: 1h
|
||||||
|
|
||||||
|
# Storage
|
||||||
|
valkeyKeyPrefix: "stellaops:prov:"
|
||||||
|
enableWriteBehind: true
|
||||||
|
writeBehindFlushInterval: 5s
|
||||||
|
writeBehindMaxBatchSize: 100
|
||||||
|
|
||||||
|
# Evidence chunking
|
||||||
|
chunkSize: 65536 # 64 KB
|
||||||
|
maxChunksPerEntry: 1000
|
||||||
|
|
||||||
|
# Behavior
|
||||||
|
allowCacheBypass: true
|
||||||
|
digestVersion: "v1"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Observability
|
||||||
|
|
||||||
|
### Metrics
|
||||||
|
|
||||||
|
| Metric | Type | Description |
|
||||||
|
|--------|------|-------------|
|
||||||
|
| `provcache_requests_total` | Counter | Total cache requests |
|
||||||
|
| `provcache_hits_total` | Counter | Cache hits |
|
||||||
|
| `provcache_misses_total` | Counter | Cache misses |
|
||||||
|
| `provcache_latency_seconds` | Histogram | Operation latency |
|
||||||
|
| `provcache_items_count` | Gauge | Current item count |
|
||||||
|
| `provcache_invalidations_total` | Counter | Invalidation count |
|
||||||
|
|
||||||
|
### Alerts
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# Low cache hit rate
|
||||||
|
- alert: ProvcacheLowHitRate
|
||||||
|
expr: rate(provcache_hits_total[5m]) / rate(provcache_requests_total[5m]) < 0.5
|
||||||
|
for: 10m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
annotations:
|
||||||
|
summary: "Provcache hit rate below 50%"
|
||||||
|
|
||||||
|
# High invalidation rate
|
||||||
|
- alert: ProvcacheHighInvalidationRate
|
||||||
|
expr: rate(provcache_invalidations_total[5m]) > 100
|
||||||
|
for: 5m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
annotations:
|
||||||
|
summary: "High cache invalidation rate"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
### Signer-Aware Caching
|
||||||
|
|
||||||
|
The `signer_set_hash` is part of the VeriKey, ensuring:
|
||||||
|
|
||||||
|
- Key rotation → new cache entries
|
||||||
|
- Key revocation → immediate invalidation
|
||||||
|
- No stale decisions from compromised signers
|
||||||
|
|
||||||
|
### Merkle Verification
|
||||||
|
|
||||||
|
All evidence chunks are Merkle-verified:
|
||||||
|
|
||||||
|
- `ProofRoot` = Merkle root of all chunks
|
||||||
|
- Individual chunks verifiable without full tree
|
||||||
|
- Tamper detection on import
|
||||||
|
|
||||||
|
### Audit Trail
|
||||||
|
|
||||||
|
All invalidations are logged to `prov_revocations` table:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT * FROM provcache.prov_revocations
|
||||||
|
WHERE created_at > NOW() - INTERVAL '24 hours'
|
||||||
|
ORDER BY created_at DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Database Schema
|
||||||
|
|
||||||
|
### provcache_items
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE provcache.provcache_items (
|
||||||
|
verikey TEXT PRIMARY KEY,
|
||||||
|
digest_version TEXT NOT NULL,
|
||||||
|
verdict_hash TEXT NOT NULL,
|
||||||
|
proof_root TEXT NOT NULL,
|
||||||
|
replay_seed JSONB NOT NULL,
|
||||||
|
policy_hash TEXT NOT NULL,
|
||||||
|
signer_set_hash TEXT NOT NULL,
|
||||||
|
feed_epoch TEXT NOT NULL,
|
||||||
|
trust_score INTEGER NOT NULL,
|
||||||
|
hit_count BIGINT DEFAULT 0,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL,
|
||||||
|
expires_at TIMESTAMPTZ NOT NULL,
|
||||||
|
updated_at TIMESTAMPTZ NOT NULL
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
### prov_evidence_chunks
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE provcache.prov_evidence_chunks (
|
||||||
|
chunk_id UUID PRIMARY KEY,
|
||||||
|
proof_root TEXT NOT NULL REFERENCES provcache_items(proof_root),
|
||||||
|
chunk_index INTEGER NOT NULL,
|
||||||
|
chunk_hash TEXT NOT NULL,
|
||||||
|
blob BYTEA NOT NULL,
|
||||||
|
blob_size INTEGER NOT NULL,
|
||||||
|
content_type TEXT NOT NULL,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
### prov_revocations
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE provcache.prov_revocations (
|
||||||
|
revocation_id UUID PRIMARY KEY,
|
||||||
|
revocation_type TEXT NOT NULL,
|
||||||
|
target_hash TEXT NOT NULL,
|
||||||
|
reason TEXT,
|
||||||
|
actor TEXT,
|
||||||
|
entries_affected BIGINT NOT NULL,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
## Implementation Sprints
|
||||||
|
|
||||||
|
| Sprint | Focus | Key Deliverables |
|
||||||
|
|--------|-------|------------------|
|
||||||
|
| [8200.0001.0001](../../implplan/SPRINT_8200_0001_0001_provcache_core_backend.md) | Core Backend | VeriKey, DecisionDigest, Valkey+Postgres, API |
|
||||||
|
| [8200.0001.0002](../../implplan/SPRINT_8200_0001_0002_provcache_invalidation_airgap.md) | Invalidation & Air-Gap | Signer revocation, feed epochs, CLI export/import |
|
||||||
|
| [8200.0001.0003](../../implplan/SPRINT_8200_0001_0003_provcache_ux_observability.md) | UX & Observability | UI badges, proof tree, Grafana, OCI attestation |
|
||||||
|
|
||||||
|
## Related Documentation
|
||||||
|
|
||||||
|
- [Policy Engine Architecture](../policy/README.md)
|
||||||
|
- [TrustLattice Engine](../policy/design/policy-deterministic-evaluator.md)
|
||||||
|
- [Offline Kit Documentation](../../24_OFFLINE_KIT.md)
|
||||||
|
- [Air-Gap Controller](../airgap/README.md)
|
||||||
|
- [Authority Key Rotation](../authority/README.md)
|
||||||
@@ -1,8 +1,9 @@
|
|||||||
# Router · Messaging Transport over Valkey (Draft v0.1)
|
# Router · Messaging Transport over Valkey
|
||||||
|
|
||||||
## Status
|
## Status
|
||||||
- Draft; intended for implementation via a dedicated sprint.
|
- **Implemented** in Sprint 8100.0011.0003.
|
||||||
- Last updated: 2025-12-23 (UTC).
|
- Core components: Gateway DI wiring, GatewayHostedService integration, GatewayTransportClient dispatch.
|
||||||
|
- Last updated: 2025-12-24 (UTC).
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Enable Gateway ↔ microservice Router traffic over an offline-friendly, Redis-compatible transport (Valkey) by using the existing **Messaging** transport layer:
|
Enable Gateway ↔ microservice Router traffic over an offline-friendly, Redis-compatible transport (Valkey) by using the existing **Messaging** transport layer:
|
||||||
@@ -11,15 +12,18 @@ Enable Gateway ↔ microservice Router traffic over an offline-friendly, Redis-c
|
|||||||
|
|
||||||
This supports environments where direct TCP/TLS microservice connections are undesirable, and where an internal message bus is the preferred control plane.
|
This supports environments where direct TCP/TLS microservice connections are undesirable, and where an internal message bus is the preferred control plane.
|
||||||
|
|
||||||
## What Exists Today (Repository Reality)
|
## Implementation Summary
|
||||||
- Messaging transport server/client:
|
|
||||||
- `src/__Libraries/StellaOps.Router.Transport.Messaging/MessagingTransportServer.cs`
|
### Libraries
|
||||||
- `src/__Libraries/StellaOps.Router.Transport.Messaging/MessagingTransportClient.cs`
|
- `StellaOps.Router.Transport.Messaging` — Router transport layer over messaging
|
||||||
- Valkey-backed message queue factory:
|
- `StellaOps.Messaging.Transport.Valkey` — Valkey/Redis backend for messaging
|
||||||
- `src/__Libraries/StellaOps.Messaging.Transport.Valkey/ValkeyMessageQueueFactory.cs`
|
- `StellaOps.Messaging` — Core messaging abstractions and DI
|
||||||
- Gateway WebService currently starts only TCP/TLS transport servers:
|
|
||||||
- `src/Gateway/StellaOps.Gateway.WebService/Program.cs`
|
### Gateway Integration
|
||||||
- `src/Gateway/StellaOps.Gateway.WebService/Services/GatewayHostedService.cs`
|
- **Program.cs** — Conditional registration of `ValkeyTransportPlugin` and `AddMessagingTransportServer()`
|
||||||
|
- **GatewayOptions.cs** — `GatewayMessagingTransportOptions` for Valkey connection and queue configuration
|
||||||
|
- **GatewayHostedService.cs** — Start/stop `MessagingTransportServer`, subscribe to events
|
||||||
|
- **GatewayTransportClient.cs** — Dispatch to `TransportType.Messaging` connections
|
||||||
|
|
||||||
## High-Level Flow
|
## High-Level Flow
|
||||||
1) Microservice connects via messaging transport:
|
1) Microservice connects via messaging transport:
|
||||||
@@ -39,14 +43,35 @@ The Messaging transport uses a small set of queues (names are configurable):
|
|||||||
- **Per-service request queues**: gateway publishes REQUEST frames targeted to a service
|
- **Per-service request queues**: gateway publishes REQUEST frames targeted to a service
|
||||||
- **Dead letter queues** (optional): for messages that exceed retries/leases
|
- **Dead letter queues** (optional): for messages that exceed retries/leases
|
||||||
|
|
||||||
## Configuration (Draft)
|
## Configuration
|
||||||
### Gateway
|
|
||||||
- Register Valkey messaging services (`StellaOps.Messaging.Transport.Valkey`)
|
### Gateway YAML Configuration
|
||||||
- Add messaging transport server (`AddMessagingTransportServer`)
|
```yaml
|
||||||
- Add Gateway config section for messaging transport options:
|
Gateway:
|
||||||
- Valkey connection info (host/port/auth)
|
Transports:
|
||||||
- queue naming prefix
|
Messaging:
|
||||||
- consumer group / lease duration / dead-letter suffix
|
Enabled: true
|
||||||
|
ConnectionString: "valkey:6379"
|
||||||
|
Database: 0
|
||||||
|
RequestQueueTemplate: "router:requests:{service}"
|
||||||
|
ResponseQueueName: "router:responses"
|
||||||
|
ConsumerGroup: "router-gateway"
|
||||||
|
RequestTimeout: "30s"
|
||||||
|
LeaseDuration: "5m"
|
||||||
|
BatchSize: 10
|
||||||
|
HeartbeatInterval: "10s"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Gateway DI Registration
|
||||||
|
```csharp
|
||||||
|
// In Program.cs (already implemented)
|
||||||
|
if (bootstrapOptions.Transports.Messaging.Enabled)
|
||||||
|
{
|
||||||
|
builder.Services.AddMessagingTransport<ValkeyTransportPlugin>(
|
||||||
|
builder.Configuration, "Gateway:Transports:Messaging");
|
||||||
|
builder.Services.AddMessagingTransportServer();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
### Microservice
|
### Microservice
|
||||||
- Register Valkey messaging services (`StellaOps.Messaging.Transport.Valkey`)
|
- Register Valkey messaging services (`StellaOps.Messaging.Transport.Valkey`)
|
||||||
@@ -63,13 +88,20 @@ The Messaging transport uses a small set of queues (names are configurable):
|
|||||||
- The Gateway must not trust client-supplied identity headers; it must overwrite reserved headers before dispatch.
|
- The Gateway must not trust client-supplied identity headers; it must overwrite reserved headers before dispatch.
|
||||||
- See `docs/modules/gateway/identity-header-policy.md`.
|
- See `docs/modules/gateway/identity-header-policy.md`.
|
||||||
|
|
||||||
## Gaps / Implementation Work
|
## Implementation Status
|
||||||
1) Wire Messaging transport into Gateway:
|
|
||||||
- start/stop `MessagingTransportServer`
|
### Completed (Sprint 8100.0011.0003)
|
||||||
- subscribe to HELLO/HEARTBEAT/RESPONSE events and reuse existing HELLO validation + routing state updates
|
1. ✅ Wire Messaging transport into Gateway:
|
||||||
2) Extend Gateway transport client to support `TransportType.Messaging` for dispatch.
|
- start/stop `MessagingTransportServer` in `GatewayHostedService`
|
||||||
3) Add config mapping and deployment examples (compose/helm) for Valkey transport.
|
- subscribe to `OnHelloReceived`, `OnHeartbeatReceived`, `OnResponseReceived`, `OnConnectionClosed` events
|
||||||
4) Add integration tests covering:
|
- reuse routing state updates and claims store updates
|
||||||
|
2. ✅ Extend Gateway transport client to support `TransportType.Messaging` for dispatch.
|
||||||
|
3. ✅ Add config options (`GatewayMessagingTransportOptions`) and DI mappings.
|
||||||
|
|
||||||
|
### Remaining Work
|
||||||
|
1. Add deployment examples (compose/helm) for Valkey transport.
|
||||||
|
2. Add integration tests using ValkeyFixture:
|
||||||
- microservice HELLO registration via messaging
|
- microservice HELLO registration via messaging
|
||||||
- request dispatch + response return
|
- request dispatch + response return
|
||||||
|
3. Validate streaming support (or document as out-of-scope).
|
||||||
|
|
||||||
|
|||||||
@@ -93,7 +93,7 @@
|
|||||||
|----------|-------|----------|
|
|----------|-------|----------|
|
||||||
| `ruby_packages.json` | `RubyPackageInventory { scanId, imageDigest, generatedAt, packages[] }` where each package mirrors `{id, name, version, source, provenance, groups[], platform, runtime.*}` | SBOM Composer, Policy Engine |
|
| `ruby_packages.json` | `RubyPackageInventory { scanId, imageDigest, generatedAt, packages[] }` where each package mirrors `{id, name, version, source, provenance, groups[], platform, runtime.*}` | SBOM Composer, Policy Engine |
|
||||||
|
|
||||||
`ruby_packages.json` records are persisted in Mongo’s `ruby.packages` collection via the `RubyPackageInventoryStore`. Scanner.WebService exposes the same payload through `GET /api/scans/{scanId}/ruby-packages` so Policy, CLI, and Offline Kit consumers can reuse the canonical inventory without re-running the analyzer. Each document is keyed by `scanId` and includes the resolved `imageDigest` plus the UTC timestamp recorded by the Worker.
|
`ruby_packages.json` records are persisted in PostgreSQL's `scanner.ruby_packages` table via the `RubyPackageInventoryStore`. Scanner.WebService exposes the same payload through `GET /api/scans/{scanId}/ruby-packages` so Policy, CLI, and Offline Kit consumers can reuse the canonical inventory without re-running the analyzer. Each record is keyed by `scanId` and includes the resolved `imageDigest` plus the UTC timestamp recorded by the Worker.
|
||||||
| `ruby_runtime_edges.json` | Edges `{from, to, reason, confidence}` | EntryTrace overlay, Policy explain traces |
|
| `ruby_runtime_edges.json` | Edges `{from, to, reason, confidence}` | EntryTrace overlay, Policy explain traces |
|
||||||
| `ruby_capabilities.json` | Capability `{kind, location, evidenceHash, params}` | Policy Engine (capability predicates) |
|
| `ruby_capabilities.json` | Capability `{kind, location, evidenceHash, params}` | Policy Engine (capability predicates) |
|
||||||
| `ruby_observation.json` | Summary document (packages, runtime edges, capability flags) | Surface manifest, Policy explain traces |
|
| `ruby_observation.json` | Summary document (packages, runtime edges, capability flags) | Surface manifest, Policy explain traces |
|
||||||
|
|||||||
@@ -37,7 +37,7 @@
|
|||||||
- `scanner.attestation.attestorEndpoint` → Attestor base URL.
|
- `scanner.attestation.attestorEndpoint` → Attestor base URL.
|
||||||
- `attestor.rekor.api` & `attestor.rekor.pubkey` set for the target log.
|
- `attestor.rekor.api` & `attestor.rekor.pubkey` set for the target log.
|
||||||
3. **Storage**
|
3. **Storage**
|
||||||
- Mongo collections `attestations` & `rekorProofs` sized for retention (7–30 days recommended).
|
- PostgreSQL tables `attestations` & `rekor_proofs` sized for retention (7–30 days recommended).
|
||||||
- Object store tier with at-rest encryption for DSSE payloads.
|
- Object store tier with at-rest encryption for DSSE payloads.
|
||||||
4. **Observability**
|
4. **Observability**
|
||||||
- Metrics: `attestor_rekor_success_total`, `attestor_rekor_retry_total`, `rekor_inclusion_latency`.
|
- Metrics: `attestor_rekor_success_total`, `attestor_rekor_retry_total`, `rekor_inclusion_latency`.
|
||||||
|
|||||||
@@ -74,7 +74,7 @@ Restart the worker so the analyzer reloads the updated bundle. Bundles are immut
|
|||||||
- Scanner.Analysis.SecretFindingsTtl
|
- Scanner.Analysis.SecretFindingsTtl
|
||||||
```
|
```
|
||||||
|
|
||||||
The migration adds `secretFindings` documents to `ScanAnalysisStore` with the standard TTL (default 90 days). Adjust Mongo TTL via the deployment overlay if longer retention is required.
|
The migration adds `secretFindings` records to `ScanAnalysisStore` with the standard TTL (default 90 days). Adjust PostgreSQL retention policy via the deployment overlay if longer retention is required.
|
||||||
|
|
||||||
3. **Activate policy ingestion** (WebService):
|
3. **Activate policy ingestion** (WebService):
|
||||||
|
|
||||||
|
|||||||
@@ -13,7 +13,7 @@
|
|||||||
| Phase V · Sprint 0134 (PHP fixtures/runtime/package) | Green | PHP analyzer fixtures, runtime evidence, and packaging shipped; docs updated. | Keep fixture hashes stable; rerun benchmarks when dependencies change. |
|
| Phase V · Sprint 0134 (PHP fixtures/runtime/package) | Green | PHP analyzer fixtures, runtime evidence, and packaging shipped; docs updated. | Keep fixture hashes stable; rerun benchmarks when dependencies change. |
|
||||||
| Phase VI · Sprint 0135 (Python container + Ruby VFS/edges) | Green | Python container/zipapp adapters shipped; Ruby VFS/dependency edges/observations/runtime capture packaged; EntryTrace 18-502/503 delivered. | Maintain determinism; re-run EntryTrace suite in CI. |
|
| Phase VI · Sprint 0135 (Python container + Ruby VFS/edges) | Green | Python container/zipapp adapters shipped; Ruby VFS/dependency edges/observations/runtime capture packaged; EntryTrace 18-502/503 delivered. | Maintain determinism; re-run EntryTrace suite in CI. |
|
||||||
| Phase VII · Sprint 0136 (EntryTrace surface/CLI) | Green | EntryTrace phase VII tasks 18-504/505/506 completed; CLI/WebService surfaces show best-terminal metadata and confidence. | Keep NDJSON schema stable; rerun worker payload tests in CI. |
|
| Phase VII · Sprint 0136 (EntryTrace surface/CLI) | Green | EntryTrace phase VII tasks 18-504/505/506 completed; CLI/WebService surfaces show best-terminal metadata and confidence. | Keep NDJSON schema stable; rerun worker payload tests in CI. |
|
||||||
| Sprint 0138 (Ruby parity & future analyzers) | Amber/Red | Ruby parity shipped; Mongo package inventory live. PHP pipeline SCANNER-ENG-0010 blocked on composer/autoload design + restore stability (design at `docs/modules/scanner/design/php-autoload-design.md`); Deno scope drafted (`docs/modules/scanner/design/deno-analyzer-scope.md`); Dart/Swift scope drafted (`docs/modules/scanner/design/dart-swift-analyzer-scope.md`); Kubernetes/VM roadmap pending. | Implement PHP autoload parser/fixtures per design; add Deno fixtures and validation evidence; align with Zastava/Runtime and update readiness once fixtures land. |
|
| Sprint 0138 (Ruby parity & future analyzers) | Amber/Red | Ruby parity shipped; PostgreSQL package inventory live. PHP pipeline SCANNER-ENG-0010 blocked on composer/autoload design + restore stability (design at `docs/modules/scanner/design/php-autoload-design.md`); Deno scope drafted (`docs/modules/scanner/design/deno-analyzer-scope.md`); Dart/Swift scope drafted (`docs/modules/scanner/design/dart-swift-analyzer-scope.md`); Kubernetes/VM roadmap pending. | Implement PHP autoload parser/fixtures per design; add Deno fixtures and validation evidence; align with Zastava/Runtime and update readiness once fixtures land. |
|
||||||
|
|
||||||
## Overall
|
## Overall
|
||||||
- Green areas: native analyzers, PHP fixtures/runtime packaging, Ruby analyzer, Python container adapters, EntryTrace phases VI–VII.
|
- Green areas: native analyzers, PHP fixtures/runtime packaging, Ruby analyzer, Python container adapters, EntryTrace phases VI–VII.
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ Scheduler detects advisory/VEX deltas, computes impact windows, and orchestrates
|
|||||||
- Shared libraries under `StellaOps.Scheduler.*`.
|
- Shared libraries under `StellaOps.Scheduler.*`.
|
||||||
|
|
||||||
## Integrations & dependencies
|
## Integrations & dependencies
|
||||||
- MongoDB for impact models.
|
- PostgreSQL (schema `scheduler`) for impact models.
|
||||||
- Redis/NATS for queueing.
|
- Redis/NATS for queueing.
|
||||||
- Policy Engine, Scanner, Notify.
|
- Policy Engine, Scanner, Notify.
|
||||||
|
|
||||||
|
|||||||
215
docs/modules/signals/architecture.md
Normal file
215
docs/modules/signals/architecture.md
Normal file
@@ -0,0 +1,215 @@
|
|||||||
|
# Signals Module Architecture
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The **Signals** module provides a unified evidence-weighted scoring system for vulnerability findings. It aggregates evidence from multiple sources (reachability analysis, runtime observations, backport detection, exploit intelligence, source trust, and mitigations) into a single 0-100 score that enables rapid triage.
|
||||||
|
|
||||||
|
## Module Purpose
|
||||||
|
|
||||||
|
- **Unify Scoring:** Combine disparate evidence signals into one actionable score
|
||||||
|
- **Enable Triage:** Support sorting, filtering, and prioritization by evidence strength
|
||||||
|
- **Maintain Determinism:** Same inputs + policy = same score, always
|
||||||
|
- **Provide Transparency:** Decomposable scores with explanations
|
||||||
|
|
||||||
|
## Location
|
||||||
|
|
||||||
|
```
|
||||||
|
src/Signals/
|
||||||
|
├── StellaOps.Signals/
|
||||||
|
│ └── EvidenceWeightedScore/
|
||||||
|
│ ├── EvidenceWeightedScoreCalculator.cs
|
||||||
|
│ ├── EvidenceWeightedScoreInput.cs
|
||||||
|
│ ├── EvidenceWeightedScoreResult.cs
|
||||||
|
│ ├── Normalizers/
|
||||||
|
│ │ ├── BackportEvidenceNormalizer.cs
|
||||||
|
│ │ ├── ExploitLikelihoodNormalizer.cs
|
||||||
|
│ │ ├── MitigationNormalizer.cs
|
||||||
|
│ │ ├── ReachabilityNormalizer.cs
|
||||||
|
│ │ ├── RuntimeSignalNormalizer.cs
|
||||||
|
│ │ └── SourceTrustNormalizer.cs
|
||||||
|
│ ├── Guardrails/
|
||||||
|
│ │ └── ScoreGuardrails.cs
|
||||||
|
│ └── Policy/
|
||||||
|
│ ├── EvidenceWeightPolicy.cs
|
||||||
|
│ └── EvidenceWeightPolicyProvider.cs
|
||||||
|
└── __Tests/
|
||||||
|
└── StellaOps.Signals.Tests/
|
||||||
|
└── EvidenceWeightedScore/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Core Concepts
|
||||||
|
|
||||||
|
### Evidence Dimensions
|
||||||
|
|
||||||
|
| Dimension | Symbol | Description | Source Module |
|
||||||
|
|-----------|--------|-------------|---------------|
|
||||||
|
| Reachability | RCH | Code path reachability to vulnerable sink | Policy |
|
||||||
|
| Runtime | RTS | Live observation strength (eBPF/dyld/ETW) | Policy |
|
||||||
|
| Backport | BKP | Patch evidence from distro/changelog/binary | Concelier |
|
||||||
|
| Exploit | XPL | Exploit probability (EPSS + KEV) | Scanner, Concelier |
|
||||||
|
| Source Trust | SRC | VEX source trustworthiness | Excititor |
|
||||||
|
| Mitigations | MIT | Active protective controls | Policy |
|
||||||
|
|
||||||
|
### Scoring Formula
|
||||||
|
|
||||||
|
```
|
||||||
|
Score = clamp01(W_rch*RCH + W_rts*RTS + W_bkp*BKP + W_xpl*XPL + W_src*SRC - W_mit*MIT) * 100
|
||||||
|
```
|
||||||
|
|
||||||
|
Note: MIT is **subtractive** — mitigations reduce risk.
|
||||||
|
|
||||||
|
### Guardrails
|
||||||
|
|
||||||
|
Hard caps and floors based on evidence conditions:
|
||||||
|
|
||||||
|
1. **Not-Affected Cap:** If vendor says not-affected (BKP=1) and no runtime contradiction (RTS<0.6), cap at 15
|
||||||
|
2. **Runtime Floor:** If strong live signal (RTS>=0.8), floor at 60
|
||||||
|
3. **Speculative Cap:** If no reachability and no runtime (RCH=0, RTS=0), cap at 45
|
||||||
|
|
||||||
|
### Score Buckets
|
||||||
|
|
||||||
|
| Bucket | Range | Meaning |
|
||||||
|
|--------|-------|---------|
|
||||||
|
| ActNow | 90-100 | Strong evidence of exploitable risk; immediate action |
|
||||||
|
| ScheduleNext | 70-89 | Likely real; schedule for next sprint |
|
||||||
|
| Investigate | 40-69 | Moderate evidence; investigate when touching component |
|
||||||
|
| Watchlist | 0-39 | Low/insufficient evidence; monitor |
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Component Diagram
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Signals Module │
|
||||||
|
├─────────────────────────────────────────────────────────────────┤
|
||||||
|
│ │
|
||||||
|
│ ┌─────────────────┐ ┌──────────────────────────────────┐ │
|
||||||
|
│ │ NormalizerAggr. │───▶│ EvidenceWeightedScoreCalculator │ │
|
||||||
|
│ └────────┬────────┘ └──────────────┬───────────────────┘ │
|
||||||
|
│ │ │ │
|
||||||
|
│ ▼ ▼ │
|
||||||
|
│ ┌─────────────────┐ ┌──────────────────────────────────┐ │
|
||||||
|
│ │ Normalizers │ │ ScoreGuardrails │ │
|
||||||
|
│ │ ┌─────┐ ┌─────┐ │ └──────────────────────────────────┘ │
|
||||||
|
│ │ │ BKP │ │ XPL │ │ │
|
||||||
|
│ │ └─────┘ └─────┘ │ ┌──────────────────────────────────┐ │
|
||||||
|
│ │ ┌─────┐ ┌─────┐ │ │ EvidenceWeightPolicyProvider │ │
|
||||||
|
│ │ │ MIT │ │ RCH │ │ └──────────────────────────────────┘ │
|
||||||
|
│ │ └─────┘ └─────┘ │ │
|
||||||
|
│ │ ┌─────┐ ┌─────┐ │ │
|
||||||
|
│ │ │ RTS │ │ SRC │ │ │
|
||||||
|
│ │ └─────┘ └─────┘ │ │
|
||||||
|
│ └─────────────────┘ │
|
||||||
|
│ │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
┌─────────────────────┐ ┌───────────────────────────────────────┐
|
||||||
|
│ External Modules │ │ Consumers │
|
||||||
|
│ │ │ │
|
||||||
|
│ • Policy (RCH, RTS) │ │ • Policy Engine (verdict enrichment) │
|
||||||
|
│ • Concelier (BKP) │ │ • Findings API (score endpoints) │
|
||||||
|
│ • Scanner (XPL) │ │ • Web UI (score components) │
|
||||||
|
│ • Excititor (SRC) │ │ │
|
||||||
|
└─────────────────────┘ └───────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Flow
|
||||||
|
|
||||||
|
1. **Finding arrives** for scoring
|
||||||
|
2. **NormalizerAggregator** collects evidence from source modules
|
||||||
|
3. **Individual normalizers** convert raw evidence to 0-1 values
|
||||||
|
4. **EvidenceWeightedScoreCalculator** applies formula
|
||||||
|
5. **ScoreGuardrails** apply caps/floors
|
||||||
|
6. **Result** returned with score, bucket, breakdown, explanations
|
||||||
|
|
||||||
|
### Policy Configuration
|
||||||
|
|
||||||
|
Weight policies are loaded from YAML and can vary by:
|
||||||
|
- **Tenant:** Different organizations may prioritize differently
|
||||||
|
- **Environment:** Production may weight runtime higher than dev
|
||||||
|
- **Policy version:** Changes are versioned for reproducibility
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
version: "ews.v1"
|
||||||
|
profile: production
|
||||||
|
|
||||||
|
weights:
|
||||||
|
rch: 0.30
|
||||||
|
rts: 0.25
|
||||||
|
bkp: 0.15
|
||||||
|
xpl: 0.15
|
||||||
|
src: 0.10
|
||||||
|
mit: 0.10
|
||||||
|
|
||||||
|
guardrails:
|
||||||
|
not_affected_cap:
|
||||||
|
enabled: true
|
||||||
|
max_score: 15
|
||||||
|
runtime_floor:
|
||||||
|
enabled: true
|
||||||
|
min_score: 60
|
||||||
|
speculative_cap:
|
||||||
|
enabled: true
|
||||||
|
max_score: 45
|
||||||
|
```
|
||||||
|
|
||||||
|
## Integration Points
|
||||||
|
|
||||||
|
### Inbound (Evidence Sources)
|
||||||
|
|
||||||
|
| Source | Data | Interface |
|
||||||
|
|--------|------|-----------|
|
||||||
|
| Policy/ConfidenceCalculator | ReachabilityEvidence, RuntimeEvidence | Direct type reference |
|
||||||
|
| Concelier/BackportProofService | ProofBlob | Direct type reference |
|
||||||
|
| Scanner/EpssPriorityCalculator | EPSS score, percentile | Direct type reference |
|
||||||
|
| Concelier/VendorRiskSignalExtractor | KEV status | Direct type reference |
|
||||||
|
| Excititor/TrustVector | Trust vector components | Direct type reference |
|
||||||
|
| Policy/GateMultipliers | Active mitigations | Direct type reference |
|
||||||
|
|
||||||
|
### Outbound (Consumers)
|
||||||
|
|
||||||
|
| Consumer | Usage |
|
||||||
|
|----------|-------|
|
||||||
|
| Policy Engine | Verdict enrichment, score-based rules |
|
||||||
|
| Findings API | Score endpoints (calculate, get, history) |
|
||||||
|
| Web UI | Score display components |
|
||||||
|
| Webhooks | Score change notifications |
|
||||||
|
| Attestor | Scoring proofs in attestations |
|
||||||
|
|
||||||
|
## Determinism
|
||||||
|
|
||||||
|
The Signals module maintains strict determinism:
|
||||||
|
|
||||||
|
1. **Same inputs + same policy = same score:** No randomness, no time-dependent factors in formula
|
||||||
|
2. **Policy versioning:** Every policy has a digest; scores include policy digest
|
||||||
|
3. **Reproducible proofs:** Scoring proofs can be verified by recalculation
|
||||||
|
4. **Stable ordering:** Input order doesn't affect result
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
| Metric | Target |
|
||||||
|
|--------|--------|
|
||||||
|
| Single score calculation | <10ms |
|
||||||
|
| Batch (100 findings) | <1s |
|
||||||
|
| Policy load | <100ms |
|
||||||
|
| Cache hit ratio | >80% |
|
||||||
|
|
||||||
|
## Testing Strategy
|
||||||
|
|
||||||
|
| Level | Focus | Location |
|
||||||
|
|-------|-------|----------|
|
||||||
|
| L0 Unit | Calculator, normalizers, guardrails | `*.Tests` |
|
||||||
|
| L0 Property | Monotonicity, bounds, determinism | `*.Tests/Properties` |
|
||||||
|
| S1 Integration | Cross-module evidence flow | `*.Tests/Integration` |
|
||||||
|
| S1 Snapshot | Result JSON structure | `*.Tests/Snapshots` |
|
||||||
|
|
||||||
|
## Related Documentation
|
||||||
|
|
||||||
|
- Product Advisory: `docs/product-advisories/24-Dec-2025 - Evidence-Weighted Score Model.md`
|
||||||
|
- Sprint Plans: `docs/implplan/SPRINT_8200_0012_*.md`
|
||||||
|
- Policy Confidence (deprecated): `docs/modules/policy/confidence-scoring.md`
|
||||||
|
- Backport Detection: `docs/modules/concelier/backport-detection.md`
|
||||||
|
- EPSS Enrichment: `docs/modules/scanner/epss-enrichment.md`
|
||||||
|
- Trust Vector: `docs/modules/excititor/trust-vector.md`
|
||||||
@@ -400,7 +400,7 @@ signer:
|
|||||||
|
|
||||||
* Run ≥ 2 replicas; front with L7 LB; **sticky** not required.
|
* Run ≥ 2 replicas; front with L7 LB; **sticky** not required.
|
||||||
* Redis for replay/quota caches (HA).
|
* Redis for replay/quota caches (HA).
|
||||||
* Audit sink (Mongo/Postgres) in primary region; asynchronous write with local fallback buffer.
|
* Audit sink (PostgreSQL) in primary region; asynchronous write with local fallback buffer.
|
||||||
* Fulcio/KMS clients configured with retries/backoff; circuit breakers.
|
* Fulcio/KMS clients configured with retries/backoff; circuit breakers.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
@@ -1,99 +0,0 @@
|
|||||||
# Task Runner Collections — Initial Migration
|
|
||||||
|
|
||||||
Last updated: 2025-11-06
|
|
||||||
|
|
||||||
This migration seeds the MongoDB collections that back the Task Runner service. It is implemented as `20251106-task-runner-baseline.mongosh` under the platform migration runner and must be applied **before** enabling the TaskRunner service in any environment.
|
|
||||||
|
|
||||||
## Collections
|
|
||||||
|
|
||||||
### `pack_runs`
|
|
||||||
|
|
||||||
| Field | Type | Notes |
|
|
||||||
|------------------|-----------------|-----------------------------------------------------------|
|
|
||||||
| `_id` | `string` | Run identifier (same as `runId`). |
|
|
||||||
| `planHash` | `string` | Deterministic hash produced by the planner. |
|
|
||||||
| `plan` | `object` | Full `TaskPackPlan` payload used to execute the run. |
|
|
||||||
| `failurePolicy` | `object` | Retry/backoff directives resolved at plan time. |
|
|
||||||
| `requestedAt` | `date` | Timestamp when the client requested the run. |
|
|
||||||
| `createdAt` | `date` | Timestamp when the run was persisted. |
|
|
||||||
| `updatedAt` | `date` | Timestamp of the last mutation. |
|
|
||||||
| `steps` | `array<object>` | Flattened step records (`stepId`, `status`, attempts…). |
|
|
||||||
| `tenantId` | `string` | Optional multi-tenant scope (reserved for future phases). |
|
|
||||||
|
|
||||||
**Indexes**
|
|
||||||
|
|
||||||
1. `{ _id: 1 }` — implicit primary key / uniqueness guarantee.
|
|
||||||
2. `{ updatedAt: -1 }` — serves `GET /runs` listings and staleness checks.
|
|
||||||
3. `{ tenantId: 1, updatedAt: -1 }` — activated once tenancy is enforced; remains sparse until then.
|
|
||||||
|
|
||||||
### `pack_run_logs`
|
|
||||||
|
|
||||||
| Field | Type | Notes |
|
|
||||||
|---------------|-----------------|--------------------------------------------------------|
|
|
||||||
| `_id` | `ObjectId` | Generated per log entry. |
|
|
||||||
| `runId` | `string` | Foreign key to `pack_runs._id`. |
|
|
||||||
| `sequence` | `long` | Monotonic counter assigned by the writer. |
|
|
||||||
| `timestamp` | `date` | UTC timestamp of the log event. |
|
|
||||||
| `level` | `string` | `trace`, `debug`, `info`, `warn`, `error`. |
|
|
||||||
| `eventType` | `string` | Machine-friendly event identifier (e.g. `step.started`). |
|
|
||||||
| `message` | `string` | Human-readable summary. |
|
|
||||||
| `stepId` | `string` | Optional step identifier. |
|
|
||||||
| `metadata` | `object` | Deterministic key/value payload (string-only values). |
|
|
||||||
|
|
||||||
**Indexes**
|
|
||||||
|
|
||||||
1. `{ runId: 1, sequence: 1 }` (unique) — guarantees ordered retrieval and enforces idempotence.
|
|
||||||
2. `{ runId: 1, timestamp: 1 }` — accelerates replay and time-window queries.
|
|
||||||
3. `{ timestamp: 1 }` — optional TTL (disabled by default) for retention policies.
|
|
||||||
|
|
||||||
### `pack_artifacts`
|
|
||||||
|
|
||||||
| Field | Type | Notes |
|
|
||||||
|--------------|------------|-------------------------------------------------------------|
|
|
||||||
| `_id` | `ObjectId` | Generated per artifact record. |
|
|
||||||
| `runId` | `string` | Foreign key to `pack_runs._id`. |
|
|
||||||
| `name` | `string` | Output name from the Task Pack manifest. |
|
|
||||||
| `type` | `string` | `file`, `object`, or other future evidence categories. |
|
|
||||||
| `sourcePath` | `string` | Local path captured during execution (nullable). |
|
|
||||||
| `storedPath` | `string` | Object store path or bundle-relative URI (nullable). |
|
|
||||||
| `status` | `string` | `pending`, `copied`, `materialized`, `skipped`. |
|
|
||||||
| `notes` | `string` | Free-form notes (deterministic messages only). |
|
|
||||||
| `capturedAt` | `date` | UTC timestamp recorded by the worker. |
|
|
||||||
|
|
||||||
**Indexes**
|
|
||||||
|
|
||||||
1. `{ runId: 1, name: 1 }` (unique) — ensures a run emits at most one record per output.
|
|
||||||
2. `{ runId: 1 }` — supports artifact listing alongside run inspection.
|
|
||||||
|
|
||||||
## Execution Order
|
|
||||||
|
|
||||||
1. Create collections with `validator` envelopes mirroring the field expectations above (if MongoDB schema validation is enabled in the environment).
|
|
||||||
2. Apply the indexes in the order listed — unique indexes first to surface data issues early.
|
|
||||||
3. Backfill existing filesystem-backed runs by importing the serialized state/log/artifact manifests into the new collections. A dedicated importer script (`tools/taskrunner/import-filesystem-state.ps1`) accompanies the migration.
|
|
||||||
4. Switch the Task Runner service configuration to point at the Mongo-backed stores (`TaskRunner:Storage:Mode = "Mongo"`), then redeploy workers and web service.
|
|
||||||
|
|
||||||
## Rollback
|
|
||||||
|
|
||||||
To revert, switch the Task Runner configuration back to the filesystem provider and stop the Mongo migration runner. Collections can remain in place; they are append-only and harmless when unused.
|
|
||||||
|
|
||||||
## Configuration Reference
|
|
||||||
|
|
||||||
Enable the Mongo-backed stores by updating the worker and web service configuration (Compose/Helm values or `appsettings*.json`):
|
|
||||||
|
|
||||||
```json
|
|
||||||
"TaskRunner": {
|
|
||||||
"Storage": {
|
|
||||||
"Mode": "mongo",
|
|
||||||
"Mongo": {
|
|
||||||
"ConnectionString": "mongodb://127.0.0.1:27017/taskrunner",
|
|
||||||
"Database": "taskrunner",
|
|
||||||
"RunsCollection": "pack_runs",
|
|
||||||
"LogsCollection": "pack_run_logs",
|
|
||||||
"ArtifactsCollection": "pack_artifacts",
|
|
||||||
"ApprovalsCollection": "pack_run_approvals"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
The worker uses the mirrored structure under the `Worker` section. Omit the `Database` property to fall back to the name embedded in the connection string.
|
|
||||||
@@ -18,7 +18,7 @@ This dossier describes the end-to-end architecture of the StellaOps Console as d
|
|||||||
- Stream live status (ingestion deltas, scheduler progress, telemetry) via SSE with graceful degradation to polling when offline or throttled.
|
- Stream live status (ingestion deltas, scheduler progress, telemetry) via SSE with graceful degradation to polling when offline or throttled.
|
||||||
- Maintain CLI parity by embedding `stella` commands alongside interactive actions.
|
- Maintain CLI parity by embedding `stella` commands alongside interactive actions.
|
||||||
|
|
||||||
Non-goals: authoring ingestion logic, mutating Policy overlays, exposing internal Mongo collections, or performing cryptographic signing in-browser.
|
Non-goals: authoring ingestion logic, mutating Policy overlays, exposing internal database tables, or performing cryptographic signing in-browser.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ VEX Lens produces a deterministic, provenance-rich consensus view of VEX stateme
|
|||||||
- Provide simulation/export APIs and Offline Kit bundles so tenants can rehearse policy changes and mirror consensus data in air-gapped environments.
|
- Provide simulation/export APIs and Offline Kit bundles so tenants can rehearse policy changes and mirror consensus data in air-gapped environments.
|
||||||
|
|
||||||
## Architecture snapshot (Sprint 30 groundwork)
|
## Architecture snapshot (Sprint 30 groundwork)
|
||||||
- **StellaOps.VexLens service & workers** — orchestrate normalisation, trust weighting, lattice join, and persistence into `vex_consensus`, `vex_consensus_history`, and `vex_conflict_queue` collections.
|
- **StellaOps.VexLens service & workers** — orchestrate normalisation, trust weighting, lattice join, and persistence into `vex_consensus`, `vex_consensus_history`, and `vex_conflict_queue` tables (PostgreSQL).
|
||||||
- **Issuer Directory integration** — maintains publisher metadata, keys, and trust tiers that feed weighting engines and revocation workflows.
|
- **Issuer Directory integration** — maintains publisher metadata, keys, and trust tiers that feed weighting engines and revocation workflows.
|
||||||
- **Consensus APIs** — `/v1/vex/consensus`, `/v1/vex/conflicts`, `/v1/vex/trust/weights`, and export streams with DSSE manifests for Offline Kit + Export Center.
|
- **Consensus APIs** — `/v1/vex/consensus`, `/v1/vex/conflicts`, `/v1/vex/trust/weights`, and export streams with DSSE manifests for Offline Kit + Export Center.
|
||||||
- **Explainability traces** — capture derived-from chains, conflicting issuers, and trust deltas to power UI drilldowns and CLI audits.
|
- **Explainability traces** — capture derived-from chains, conflicting issuers, and trust deltas to power UI drilldowns and CLI audits.
|
||||||
@@ -39,7 +39,7 @@ VEX Lens produces a deterministic, provenance-rich consensus view of VEX stateme
|
|||||||
- **Notify / Task Runner** receive conflict and override events for operator actions once notification bridges ship.
|
- **Notify / Task Runner** receive conflict and override events for operator actions once notification bridges ship.
|
||||||
|
|
||||||
## Data & observability
|
## Data & observability
|
||||||
- Collections: `vex_consensus`, `vex_consensus_history`, `vex_conflict_queue`, plus issuer registry tables managed with tenant isolation and deterministic indexes.
|
- PostgreSQL tables: `vex_consensus`, `vex_consensus_history`, `vex_conflict_queue`, plus issuer registry tables managed with tenant isolation and deterministic indexes.
|
||||||
- Metrics: `vex_consensus_conflicts_total`, `vex_consensus_latency_seconds`, `vex_consensus_recompute_seconds{reason}`, signature failure counters.
|
- Metrics: `vex_consensus_conflicts_total`, `vex_consensus_latency_seconds`, `vex_consensus_recompute_seconds{reason}`, signature failure counters.
|
||||||
- Traces/logs: `consensus.group`, `consensus.join`, `consensus.persist` spans with correlation IDs and issuer details; structured logs capture trust adjustments and reconciliation outcomes.
|
- Traces/logs: `consensus.group`, `consensus.join`, `consensus.persist` spans with correlation IDs and issuer details; structured logs capture trust adjustments and reconciliation outcomes.
|
||||||
- Offline bundles include `consensus.jsonl`, `conflicts.jsonl`, manifest + DSSE signatures, enabling mirror deployments and replay validation.
|
- Offline bundles include `consensus.jsonl`, `conflicts.jsonl`, manifest + DSSE signatures, enabling mirror deployments and replay validation.
|
||||||
@@ -47,7 +47,9 @@ VEX Lens produces a deterministic, provenance-rich consensus view of VEX stateme
|
|||||||
## Key docs & references
|
## Key docs & references
|
||||||
- [`architecture.md`](architecture.md) — implementation-ready blueprint covering inputs, algorithm, APIs, storage, observability, and exports.
|
- [`architecture.md`](architecture.md) — implementation-ready blueprint covering inputs, algorithm, APIs, storage, observability, and exports.
|
||||||
- [`implementation_plan.md`](implementation_plan.md) — phased delivery roadmap and acceptance criteria.
|
- [`implementation_plan.md`](implementation_plan.md) — phased delivery roadmap and acceptance criteria.
|
||||||
|
- [`scoring.md`](scoring.md) — future risk scoring model and formula reference.
|
||||||
- [`../../vex/aggregation.md`](../../vex/aggregation.md) — Aggregation-Only Contract boundaries for VEX ingestion and downstream consumers.
|
- [`../../vex/aggregation.md`](../../vex/aggregation.md) — Aggregation-Only Contract boundaries for VEX ingestion and downstream consumers.
|
||||||
|
- **Operations:** [`operations/deployment.md`](operations/deployment.md), [`operations/offline-kit.md`](operations/offline-kit.md) — deployment guides and offline bundle preparation.
|
||||||
- Sprint tracking in `docs/implplan/SPRINT_0332_0001_0001_docs_modules_vex_lens.md`; module engineering tasks in `src/VexLens/StellaOps.VexLens/TASKS.md`; doc TASKS mirror in `docs/modules/vex-lens/TASKS.md`.
|
- Sprint tracking in `docs/implplan/SPRINT_0332_0001_0001_docs_modules_vex_lens.md`; module engineering tasks in `src/VexLens/StellaOps.VexLens/TASKS.md`; doc TASKS mirror in `docs/modules/vex-lens/TASKS.md`.
|
||||||
|
|
||||||
## Epic alignment
|
## Epic alignment
|
||||||
|
|||||||
@@ -192,10 +192,10 @@ vexlens:
|
|||||||
|
|
||||||
storage:
|
storage:
|
||||||
# Local storage only
|
# Local storage only
|
||||||
mongodb:
|
postgresql:
|
||||||
connectionString: mongodb://localhost:27017
|
connectionString: Host=localhost;Database=stellaops;Username=stellaops;Password=password
|
||||||
# No external cache
|
# No external cache
|
||||||
redis:
|
valkey:
|
||||||
enabled: false
|
enabled: false
|
||||||
|
|
||||||
time:
|
time:
|
||||||
@@ -1,319 +0,0 @@
|
|||||||
# component_architecture_vexlens.md — **Stella Ops VexLens** (2025Q4)
|
|
||||||
|
|
||||||
> Supports deliverables from Epic 30 – VEX Consensus Engine and Epic 31 – Advisory AI Integration.
|
|
||||||
|
|
||||||
> **Scope.** Implementation-ready architecture for **VexLens**: the consensus engine for computing authoritative VEX (Vulnerability Exploitability eXchange) status from multiple overlapping statements. It supports trust-weighted voting, lattice-based conflict resolution, and provides policy integration for vulnerability decisioning.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 0) Mission & Boundaries
|
|
||||||
|
|
||||||
**Mission.** Compute deterministic VEX consensus status from multiple sources with full audit trail, enabling automated vulnerability triage based on exploitability data.
|
|
||||||
|
|
||||||
**Boundaries.**
|
|
||||||
|
|
||||||
* **VexLens does not fetch VEX documents** — it receives normalized statements from Excititor or direct API input.
|
|
||||||
* **VexLens does not store raw VEX documents** — it stores computed projections and consensus results.
|
|
||||||
* **VexLens does not make policy decisions** — it provides VEX status to Policy Engine for final determination.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1) Responsibilities (contract)
|
|
||||||
|
|
||||||
1. **Normalize** VEX documents from OpenVEX, CSAF VEX, CycloneDX VEX, and SPDX VEX formats.
|
|
||||||
2. **Map products** using PURL and CPE identifiers with configurable matching strictness.
|
|
||||||
3. **Verify signatures** on VEX documents (DSSE, JWS, PGP, PKCS#7).
|
|
||||||
4. **Compute trust weights** based on issuer authority, signature status, freshness, and other factors.
|
|
||||||
5. **Compute consensus** using configurable modes:
|
|
||||||
- **HighestWeight**: Single highest-weighted statement wins
|
|
||||||
- **WeightedVote**: Weighted voting among all statements
|
|
||||||
- **Lattice**: Most conservative status wins (affected > under_investigation > not_affected > fixed)
|
|
||||||
- **AuthoritativeFirst**: Authoritative sources override others
|
|
||||||
- **MostRecent**: Most recent statement wins
|
|
||||||
6. **Store projections** for historical tracking and audit.
|
|
||||||
7. **Emit events** on consensus computation, status changes, and conflict detection.
|
|
||||||
8. **Integrate** with Policy Engine for vulnerability suppression and severity adjustment.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2) External Dependencies
|
|
||||||
|
|
||||||
* **Excititor**: Provides normalized VEX statements from connectors.
|
|
||||||
* **Policy Engine**: Consumes VEX consensus for vulnerability decisioning.
|
|
||||||
* **Vuln Explorer**: Enriches vulnerability data with VEX status.
|
|
||||||
* **Orchestrator**: Schedules consensus compute jobs for batch processing.
|
|
||||||
* **Authority**: Validates issuer trust and key fingerprints.
|
|
||||||
* **Config stores**: PostgreSQL (projections, issuer directory), Redis (caches).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3) API Surface
|
|
||||||
|
|
||||||
Base path: `/api/v1/vexlens`. Full OpenAPI spec at `docs/api/vexlens-openapi.yaml`.
|
|
||||||
|
|
||||||
### 3.1 Consensus Operations
|
|
||||||
|
|
||||||
| Endpoint | Method | Description |
|
|
||||||
|----------|--------|-------------|
|
|
||||||
| `/consensus` | POST | Compute consensus for a vulnerability-product pair |
|
|
||||||
| `/consensus/batch` | POST | Compute consensus for multiple pairs in batch |
|
|
||||||
|
|
||||||
### 3.2 Projection Queries
|
|
||||||
|
|
||||||
| Endpoint | Method | Description |
|
|
||||||
|----------|--------|-------------|
|
|
||||||
| `/projections` | GET | Query consensus projections with filtering |
|
|
||||||
| `/projections/{projectionId}` | GET | Get a projection by ID |
|
|
||||||
| `/projections/latest` | GET | Get latest projection for a vuln-product pair |
|
|
||||||
| `/projections/history` | GET | Get projection history |
|
|
||||||
|
|
||||||
### 3.3 Issuer Directory
|
|
||||||
|
|
||||||
| Endpoint | Method | Description |
|
|
||||||
|----------|--------|-------------|
|
|
||||||
| `/issuers` | GET | List registered issuers |
|
|
||||||
| `/issuers` | POST | Register a new issuer |
|
|
||||||
| `/issuers/{issuerId}` | GET | Get issuer details |
|
|
||||||
| `/issuers/{issuerId}` | DELETE | Revoke an issuer |
|
|
||||||
| `/issuers/{issuerId}/keys` | POST | Add a key to an issuer |
|
|
||||||
| `/issuers/{issuerId}/keys/{fingerprint}` | DELETE | Revoke a key |
|
|
||||||
|
|
||||||
### 3.4 Statistics
|
|
||||||
|
|
||||||
| Endpoint | Method | Description |
|
|
||||||
|----------|--------|-------------|
|
|
||||||
| `/statistics` | GET | Get consensus statistics |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4) Data Flow
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
|
|
||||||
│ Excititor │────▶│ Normalizer │────▶│ Trust Weighting │
|
|
||||||
│ (VEX Docs) │ │ (OpenVEX, │ │ (9 factors) │
|
|
||||||
└─────────────┘ │ CSAF, CDX) │ └────────┬────────┘
|
|
||||||
└──────────────┘ │
|
|
||||||
▼
|
|
||||||
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
|
|
||||||
│ Policy │◀────│ Projection │◀────│ Consensus │
|
|
||||||
│ Engine │ │ Store │ │ Engine │
|
|
||||||
└─────────────┘ └──────────────┘ └─────────────────┘
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
┌──────────────┐
|
|
||||||
│ Events │
|
|
||||||
│ (Computed, │
|
|
||||||
│ StatusChange,│
|
|
||||||
│ Conflict) │
|
|
||||||
└──────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5) VEX Status Lattice
|
|
||||||
|
|
||||||
VexLens uses a status lattice for conservative conflict resolution:
|
|
||||||
|
|
||||||
```
|
|
||||||
affected (most restrictive)
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
under_investigation
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
not_affected
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
fixed (least restrictive)
|
|
||||||
```
|
|
||||||
|
|
||||||
In lattice mode, the most restrictive status always wins. This ensures that when sources disagree, the system errs on the side of caution.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6) Trust Weight Factors
|
|
||||||
|
|
||||||
| Factor | Weight | Description |
|
|
||||||
|--------|--------|-------------|
|
|
||||||
| IssuerBase | 25% | Base trust from issuer directory |
|
|
||||||
| SignatureStatus | 15% | Valid/invalid/unsigned signature |
|
|
||||||
| Freshness | 15% | Document age with exponential decay |
|
|
||||||
| IssuerCategory | 10% | Vendor > Distributor > Aggregator |
|
|
||||||
| IssuerTier | 10% | Authoritative > Trusted > Untrusted |
|
|
||||||
| StatusQuality | 10% | Has justification, specific status |
|
|
||||||
| TransparencyLog | 5% | Sigstore Rekor entry |
|
|
||||||
| SourceMatch | 5% | Source URI pattern match |
|
|
||||||
| ProductAuthority | 5% | Issuer is authoritative for product |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 7) Configuration
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
vexlens:
|
|
||||||
consensus:
|
|
||||||
defaultMode: WeightedVote # HighestWeight, WeightedVote, Lattice, AuthoritativeFirst, MostRecent
|
|
||||||
minimumConfidence: 0.1
|
|
||||||
conflictThreshold: 0.3
|
|
||||||
requireJustificationForNotAffected: false
|
|
||||||
trust:
|
|
||||||
freshnessHalfLifeDays: 90
|
|
||||||
minimumFreshness: 0.3
|
|
||||||
allowUnsigned: true
|
|
||||||
unsignedPenalty: 0.3
|
|
||||||
allowUnknownIssuers: true
|
|
||||||
unknownIssuerPenalty: 0.5
|
|
||||||
storage:
|
|
||||||
projectionRetentionDays: 365
|
|
||||||
eventRetentionDays: 90
|
|
||||||
issuerDirectory:
|
|
||||||
source: postgresql # postgresql, file, api
|
|
||||||
refreshIntervalMinutes: 60
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 8) Storage Schema
|
|
||||||
|
|
||||||
### 8.1 Consensus Projection
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"projectionId": "proj-abc123",
|
|
||||||
"vulnerabilityId": "CVE-2024-1234",
|
|
||||||
"productKey": "pkg:npm/lodash@4.17.21",
|
|
||||||
"tenantId": "tenant-001",
|
|
||||||
"status": "not_affected",
|
|
||||||
"justification": "vulnerable_code_not_present",
|
|
||||||
"confidenceScore": 0.95,
|
|
||||||
"outcome": "Unanimous",
|
|
||||||
"statementCount": 3,
|
|
||||||
"conflictCount": 0,
|
|
||||||
"rationaleSummary": "Unanimous consensus from 3 authoritative sources",
|
|
||||||
"computedAt": "2025-12-06T12:00:00Z",
|
|
||||||
"storedAt": "2025-12-06T12:00:01Z",
|
|
||||||
"previousProjectionId": null,
|
|
||||||
"statusChanged": true
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 8.2 Issuer Record
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"issuerId": "npm-security",
|
|
||||||
"name": "npm Security Team",
|
|
||||||
"category": "Vendor",
|
|
||||||
"trustTier": "Authoritative",
|
|
||||||
"status": "Active",
|
|
||||||
"keyFingerprints": [
|
|
||||||
{
|
|
||||||
"fingerprint": "ABCD1234EFGH5678",
|
|
||||||
"keyType": "Pgp",
|
|
||||||
"algorithm": "EdDSA",
|
|
||||||
"status": "Active",
|
|
||||||
"registeredAt": "2025-01-01T00:00:00Z",
|
|
||||||
"expiresAt": null
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"description": "Official npm security advisories",
|
|
||||||
"uri": "https://www.npmjs.com/advisories",
|
|
||||||
"email": "security@npmjs.com"
|
|
||||||
},
|
|
||||||
"registeredAt": "2025-01-01T00:00:00Z"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 9) Events
|
|
||||||
|
|
||||||
### 9.1 ConsensusComputedEvent
|
|
||||||
|
|
||||||
Emitted after every consensus computation.
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"eventId": "evt-abc123",
|
|
||||||
"projectionId": "proj-abc123",
|
|
||||||
"vulnerabilityId": "CVE-2024-1234",
|
|
||||||
"productKey": "pkg:npm/lodash@4.17.21",
|
|
||||||
"status": "not_affected",
|
|
||||||
"confidenceScore": 0.95,
|
|
||||||
"outcome": "Unanimous",
|
|
||||||
"statementCount": 3,
|
|
||||||
"computedAt": "2025-12-06T12:00:00Z",
|
|
||||||
"emittedAt": "2025-12-06T12:00:01Z"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 9.2 ConsensusStatusChangedEvent
|
|
||||||
|
|
||||||
Emitted when consensus status changes from previous projection.
|
|
||||||
|
|
||||||
### 9.3 ConsensusConflictDetectedEvent
|
|
||||||
|
|
||||||
Emitted when conflicts are detected during consensus computation.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 10) Observability
|
|
||||||
|
|
||||||
### 10.1 Metrics (OpenTelemetry)
|
|
||||||
|
|
||||||
| Metric | Type | Description |
|
|
||||||
|--------|------|-------------|
|
|
||||||
| `vexlens.consensus.computed_total` | Counter | Total consensus computations |
|
|
||||||
| `vexlens.consensus.conflicts_total` | Counter | Total conflicts detected |
|
|
||||||
| `vexlens.consensus.confidence` | Histogram | Confidence score distribution |
|
|
||||||
| `vexlens.consensus.duration_seconds` | Histogram | Computation duration |
|
|
||||||
| `vexlens.consensus.status_changes_total` | Counter | Status changes detected |
|
|
||||||
| `vexlens.normalization.documents_total` | Counter | Documents normalized |
|
|
||||||
| `vexlens.trust.weight_value` | Histogram | Trust weight distribution |
|
|
||||||
| `vexlens.issuer.registered_total` | Counter | Issuers registered |
|
|
||||||
|
|
||||||
### 10.2 Traces
|
|
||||||
|
|
||||||
Activity source: `StellaOps.VexLens`
|
|
||||||
|
|
||||||
| Activity | Description |
|
|
||||||
|----------|-------------|
|
|
||||||
| `vexlens.normalize` | VEX document normalization |
|
|
||||||
| `vexlens.compute_trust_weight` | Trust weight computation |
|
|
||||||
| `vexlens.compute_consensus` | Consensus computation |
|
|
||||||
| `vexlens.store_projection` | Projection storage |
|
|
||||||
| `vexlens.query_projections` | Projection query |
|
|
||||||
|
|
||||||
### 10.3 Logging
|
|
||||||
|
|
||||||
Structured logging with event IDs in `VexLensLogEvents`:
|
|
||||||
- 1xxx: Normalization events
|
|
||||||
- 2xxx: Product mapping events
|
|
||||||
- 3xxx: Signature verification events
|
|
||||||
- 4xxx: Trust weight events
|
|
||||||
- 5xxx: Consensus events
|
|
||||||
- 6xxx: Projection events
|
|
||||||
- 7xxx: Issuer directory events
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 11) Security Considerations
|
|
||||||
|
|
||||||
1. **Issuer Trust**: All issuers must be registered with verified key fingerprints.
|
|
||||||
2. **Signature Verification**: Documents should be cryptographically signed for production use.
|
|
||||||
3. **Tenant Isolation**: Projections are scoped to tenants; no cross-tenant data access.
|
|
||||||
4. **Audit Trail**: All consensus computations are logged with full rationale.
|
|
||||||
5. **Determinism**: All computations are deterministic for reproducibility.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 12) Test Matrix
|
|
||||||
|
|
||||||
| Test Category | Coverage | Notes |
|
|
||||||
|---------------|----------|-------|
|
|
||||||
| Unit tests | Normalizer, Parser, Trust, Consensus | 89+ tests |
|
|
||||||
| Determinism harness | Normalization, Trust, Consensus | Verify reproducibility |
|
|
||||||
| Integration tests | API service, Storage, Events | End-to-end flows |
|
|
||||||
| Property-based tests | Lattice semantics, Weight computation | Invariant verification |
|
|
||||||
@@ -60,7 +60,7 @@ CLI mirrors these endpoints (`stella findings list|view|update|export`). Console
|
|||||||
## 6) Identity & access integration
|
## 6) Identity & access integration
|
||||||
|
|
||||||
- **Scopes** – `vuln:view`, `vuln:investigate`, `vuln:operate`, `vuln:audit` map to read-only, triage, workflow, and audit experiences respectively. The deprecated `vuln:read` scope is still honoured for legacy tokens but is no longer advertised.
|
- **Scopes** – `vuln:view`, `vuln:investigate`, `vuln:operate`, `vuln:audit` map to read-only, triage, workflow, and audit experiences respectively. The deprecated `vuln:read` scope is still honoured for legacy tokens but is no longer advertised.
|
||||||
- **Attribute filters (ABAC)** – Authority enforces per-service-account filters via the client-credential parameters `vuln_env`, `vuln_owner`, and `vuln_business_tier`. Service accounts define the allowed values in `authority.yaml` (`attributes` block). Tokens include the resolved filters as claims (`stellaops:vuln_env`, `stellaops:vuln_owner`, `stellaops:vuln_business_tier`), and tokens persisted to Mongo retain the same values for audit and revocation.
|
- **Attribute filters (ABAC)** – Authority enforces per-service-account filters via the client-credential parameters `vuln_env`, `vuln_owner`, and `vuln_business_tier`. Service accounts define the allowed values in `authority.yaml` (`attributes` block). Tokens include the resolved filters as claims (`stellaops:vuln_env`, `stellaops:vuln_owner`, `stellaops:vuln_business_tier`), and tokens persisted to PostgreSQL retain the same values for audit and revocation.
|
||||||
- **Audit trail** – Every token issuance emits `authority.vuln_attr.*` audit properties that mirror the resolved filter set, along with `delegation.service_account` and ordered `delegation.actor[n]` entries so Vuln Explorer can correlate access decisions.
|
- **Audit trail** – Every token issuance emits `authority.vuln_attr.*` audit properties that mirror the resolved filter set, along with `delegation.service_account` and ordered `delegation.actor[n]` entries so Vuln Explorer can correlate access decisions.
|
||||||
- **Permalinks** – Signed permalinks inherit the caller’s ABAC filters; consuming services must enforce the embedded claims in addition to scope checks when resolving permalinks.
|
- **Permalinks** – Signed permalinks inherit the caller’s ABAC filters; consuming services must enforce the embedded claims in addition to scope checks when resolving permalinks.
|
||||||
- **Attachment tokens** – Authority mints short-lived tokens (`POST /vuln/attachments/tokens/issue`) to gate evidence downloads. Verification (`POST /vuln/attachments/tokens/verify`) enforces tenant, scope, and ABAC metadata, and emits `vuln.attachment.token.*` audit records.
|
- **Attachment tokens** – Authority mints short-lived tokens (`POST /vuln/attachments/tokens/issue`) to gate evidence downloads. Verification (`POST /vuln/attachments/tokens/verify`) enforces tenant, scope, and ABAC metadata, and emits `vuln.attachment.token.*` audit records.
|
||||||
|
|||||||
@@ -20,7 +20,7 @@
|
|||||||
- Trace exemplar anchors: `traceparent` headers are copied into logs; exporters stay disabled by default for air-gap. Enable by setting `Telemetry:ExportEnabled=true` and pointing to on-prem Tempo/Jaeger.
|
- Trace exemplar anchors: `traceparent` headers are copied into logs; exporters stay disabled by default for air-gap. Enable by setting `Telemetry:ExportEnabled=true` and pointing to on-prem Tempo/Jaeger.
|
||||||
|
|
||||||
## Health/diagnostics
|
## Health/diagnostics
|
||||||
- `/health/liveness` and `/health/readiness` (HTTP 200 expected; readiness checks Mongo + cache reachability).
|
- `/health/liveness` and `/health/readiness` (HTTP 200 expected; readiness checks PostgreSQL + cache reachability).
|
||||||
- `/status` returns build version, git commit, and enabled features; safe for anonymous fetch in sealed environments.
|
- `/status` returns build version, git commit, and enabled features; safe for anonymous fetch in sealed environments.
|
||||||
- Ledger replay check: `GET /v1/findings?projectionMode=verify` emits `X-Vuln-Projection-Head` for quick consistency probes.
|
- Ledger replay check: `GET /v1/findings?projectionMode=verify` emits `X-Vuln-Projection-Head` for quick consistency probes.
|
||||||
|
|
||||||
|
|||||||
@@ -17,11 +17,11 @@ Digests coalesce multiple matching events into a single notification when rules
|
|||||||
|
|
||||||
## 2. Storage model
|
## 2. Storage model
|
||||||
|
|
||||||
Digest state lives in Mongo (`digests` collection) and mirrors the schema described in [modules/notify/architecture.md](../modules/notify/architecture.md#7-data-model-mongo):
|
Digest state lives in PostgreSQL (`notify.digests` table) and mirrors the schema described in [modules/notify/architecture.md](../modules/notify/architecture.md#7-data-model):
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"_id": "tenant-dev:act-email-compliance:1h",
|
"id": "tenant-dev:act-email-compliance:1h",
|
||||||
"tenantId": "tenant-dev",
|
"tenantId": "tenant-dev",
|
||||||
"actionKey": "act-email-compliance",
|
"actionKey": "act-email-compliance",
|
||||||
"window": "1h",
|
"window": "1h",
|
||||||
@@ -76,7 +76,7 @@ All routes honour the tenant header and reuse the standard Notify rate limits.
|
|||||||
- **Throttles.** Digest generation respects action throttles; setting an aggressive throttle together with a digest window may result in deliberate skips (logged as `Throttled` in the delivery ledger).
|
- **Throttles.** Digest generation respects action throttles; setting an aggressive throttle together with a digest window may result in deliberate skips (logged as `Throttled` in the delivery ledger).
|
||||||
- **Quiet hours.** Future sprint work (`NOTIFY-SVC-39-004`) integrates quiet-hour calendars. When enabled, flush timers pause during quiet windows and resume afterwards.
|
- **Quiet hours.** Future sprint work (`NOTIFY-SVC-39-004`) integrates quiet-hour calendars. When enabled, flush timers pause during quiet windows and resume afterwards.
|
||||||
- **Back-pressure.** When the window reaches the configured item cap before the timer, the worker flushes early and starts a new window immediately.
|
- **Back-pressure.** When the window reaches the configured item cap before the timer, the worker flushes early and starts a new window immediately.
|
||||||
- **Crash resilience.** Workers rebuild in-flight windows from Mongo on startup; partially flushed windows remain closed after success or reopened if the flush fails.
|
- **Crash resilience.** Workers rebuild in-flight windows from PostgreSQL on startup; partially flushed windows remain closed after success or reopened if the flush fails.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ Notifications Studio turns raw platform events into concise, tenant-scoped alert
|
|||||||
| Channel catalog | Slack, Teams, Email, Webhook connectors loaded via restart-time plug-ins; metadata stored without secrets. | [`notifications/architecture.md`](architecture.md) |
|
| Channel catalog | Slack, Teams, Email, Webhook connectors loaded via restart-time plug-ins; metadata stored without secrets. | [`notifications/architecture.md`](architecture.md) |
|
||||||
| Templates | Locale-aware, deterministic rendering via safe helpers; channel defaults plus tenant-specific overrides, including the attestation lifecycle suite (`tmpl-attest-*`). | [`notifications/templates.md`](templates.md#7-attestation--signing-lifecycle-templates-notify-attest-74-001) |
|
| Templates | Locale-aware, deterministic rendering via safe helpers; channel defaults plus tenant-specific overrides, including the attestation lifecycle suite (`tmpl-attest-*`). | [`notifications/templates.md`](templates.md#7-attestation--signing-lifecycle-templates-notify-attest-74-001) |
|
||||||
| Digests | Coalesce bursts into periodic summaries with deterministic IDs and audit trails. | [`notifications/digests.md`](digests.md) |
|
| Digests | Coalesce bursts into periodic summaries with deterministic IDs and audit trails. | [`notifications/digests.md`](digests.md) |
|
||||||
| Delivery ledger | Tracks rendered payload hashes, attempts, throttles, and outcomes for every action. | [`modules/notify/architecture.md`](../modules/notify/architecture.md#7-data-model-mongo) |
|
| Delivery ledger | Tracks rendered payload hashes, attempts, throttles, and outcomes for every action. | [`modules/notify/architecture.md`](../modules/notify/architecture.md#7-data-model) |
|
||||||
| Ack tokens | DSSE-signed acknowledgement tokens with webhook allowlists and escalation guardrails enforced by Authority. | [`modules/notify/architecture.md`](../modules/notify/architecture.md#81-ack-tokens--escalation-workflows) |
|
| Ack tokens | DSSE-signed acknowledgement tokens with webhook allowlists and escalation guardrails enforced by Authority. | [`modules/notify/architecture.md`](../modules/notify/architecture.md#81-ack-tokens--escalation-workflows) |
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -45,7 +45,7 @@ The Notify WebService fronts worker state with REST APIs used by the UI and CLI.
|
|||||||
|------|----------|
|
|------|----------|
|
||||||
| **Tenancy** | Each rule, channel, template, and delivery belongs to exactly one tenant. Cross-tenant sharing is intentionally unsupported. |
|
| **Tenancy** | Each rule, channel, template, and delivery belongs to exactly one tenant. Cross-tenant sharing is intentionally unsupported. |
|
||||||
| **Determinism** | Configuration persistence normalises strings and sorts collections. Template rendering produces identical `bodyHash` values when inputs match; attestation events always reference the canonical `tmpl-attest-*` keys documented in the template guide. |
|
| **Determinism** | Configuration persistence normalises strings and sorts collections. Template rendering produces identical `bodyHash` values when inputs match; attestation events always reference the canonical `tmpl-attest-*` keys documented in the template guide. |
|
||||||
| **Scaling** | Workers scale horizontally; per-tenant rule snapshots are cached and refreshed from Mongo change streams. Redis (or equivalent) guards throttles and locks. |
|
| **Scaling** | Workers scale horizontally; per-tenant rule snapshots are cached and refreshed from PostgreSQL change notifications. Valkey (or Redis-compatible) guards throttles and locks. |
|
||||||
| **Offline** | Offline Kits include plug-ins, default templates, and seed rules. Operators can edit YAML/JSON manifests before air-gapped deployment. |
|
| **Offline** | Offline Kits include plug-ins, default templates, and seed rules. Operators can edit YAML/JSON manifests before air-gapped deployment. |
|
||||||
| **Security** | Channel secrets use indirection (`secretRef`), Authority-protected OAuth clients secure API access, and delivery payloads are redacted before storage where required. |
|
| **Security** | Channel secrets use indirection (`secretRef`), Authority-protected OAuth clients secure API access, and delivery payloads are redacted before storage where required. |
|
||||||
| **Module boundaries** | 2025-11-02 decision: keep `src/Notify/` as the shared notification toolkit and `src/Notifier/` as the Notifications Studio runtime host until a packaging RFC covers the implications of merging. |
|
| **Module boundaries** | 2025-11-02 decision: keep `src/Notify/` as the shared notification toolkit and `src/Notifier/` as the Notifications Studio runtime host until a packaging RFC covers the implications of merging. |
|
||||||
@@ -56,7 +56,7 @@ The Notify WebService fronts worker state with REST APIs used by the UI and CLI.
|
|||||||
|
|
||||||
| Step | Goal | Reference |
|
| Step | Goal | Reference |
|
||||||
|------|------|-----------|
|
|------|------|-----------|
|
||||||
| 1 | Deploy Notify WebService + Worker with Mongo and Redis | [`modules/notify/architecture.md`](../modules/notify/architecture.md#1-runtime-shape--projects) |
|
| 1 | Deploy Notify WebService + Worker with PostgreSQL and Valkey | [`modules/notify/architecture.md`](../modules/notify/architecture.md#1-runtime-shape--projects) |
|
||||||
| 2 | Register OAuth clients/scopes in Authority | [`etc/authority.yaml.sample`](../../etc/authority.yaml.sample) |
|
| 2 | Register OAuth clients/scopes in Authority | [`etc/authority.yaml.sample`](../../etc/authority.yaml.sample) |
|
||||||
| 3 | Install channel plug-ins and capture secret references | [`plugins/notify`](../../plugins) |
|
| 3 | Install channel plug-ins and capture secret references | [`plugins/notify`](../../plugins) |
|
||||||
| 4 | Create a tenant rule and test preview | [`POST /channels/{id}/test`](../modules/notify/architecture.md#8-external-apis-webservice) |
|
| 4 | Create a tenant rule and test preview | [`POST /channels/{id}/test`](../modules/notify/architecture.md#8-external-apis-webservice) |
|
||||||
|
|||||||
@@ -0,0 +1,413 @@
|
|||||||
|
# Product Advisory: Deterministic Resolver Architecture
|
||||||
|
|
||||||
|
**Date:** 2025-12-24
|
||||||
|
**Author:** Architecture Guild
|
||||||
|
**Status:** Approved for Implementation
|
||||||
|
**Sprint Epoch:** 9100
|
||||||
|
**Priority:** P0 (Critical Path for Audit Compliance)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
This advisory defines the architecture for a **unified deterministic resolver** that evaluates vulnerability graphs using only declared evidence edges and produces cryptographically verifiable, reproducible verdicts. The resolver guarantees: **same inputs → same traversal order → same per-node inputs → same verdicts → same output digest**.
|
||||||
|
|
||||||
|
This is a foundational capability for:
|
||||||
|
- Auditor-friendly verification ("same inputs → same verdicts")
|
||||||
|
- Offline replay by vendors/distributors
|
||||||
|
- Single-digest comparison across runs
|
||||||
|
- CI/CD gate assertions
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Problem Statement
|
||||||
|
|
||||||
|
Current implementation has strong building blocks but lacks unified orchestration:
|
||||||
|
|
||||||
|
| Component | Location | Gap |
|
||||||
|
|-----------|----------|-----|
|
||||||
|
| Topological Sort | `DeterministicGraphOrderer` | Traversal only; no verdict output |
|
||||||
|
| Lattice Evaluation | `TrustLatticeEngine`, `PolicyEvaluator` | No traversal sequence in output |
|
||||||
|
| Content Addressing | `ContentAddressedIdGenerator` | Node IDs only; no edge IDs |
|
||||||
|
| Purity Analysis | `ProhibitedPatternAnalyzer` | Static only; no runtime enforcement |
|
||||||
|
| Digest Computation | Various | No unified "run digest" |
|
||||||
|
|
||||||
|
**Result:** Cannot produce a single `ResolutionResult` that captures:
|
||||||
|
1. The exact traversal sequence used
|
||||||
|
2. Per-node verdicts with individual digests
|
||||||
|
3. A composite `FinalDigest` for verification
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Solution Architecture
|
||||||
|
|
||||||
|
### Core Data Model
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
// New: src/__Libraries/StellaOps.Resolver/
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Content-addressed node identifier.
|
||||||
|
/// Format: sha256(kind || ":" || normalized-key)
|
||||||
|
/// </summary>
|
||||||
|
public sealed record NodeId(string Value) : IComparable<NodeId>
|
||||||
|
{
|
||||||
|
public int CompareTo(NodeId? other) =>
|
||||||
|
string.Compare(Value, other?.Value, StringComparison.Ordinal);
|
||||||
|
|
||||||
|
public static NodeId From(string kind, string normalizedKey) =>
|
||||||
|
new(HexSha256($"{kind}:{normalizedKey}"));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Content-addressed edge identifier.
|
||||||
|
/// Format: sha256(srcID || "->" || edgeKind || "->" || dstID)
|
||||||
|
/// </summary>
|
||||||
|
public sealed record EdgeId(string Value) : IComparable<EdgeId>
|
||||||
|
{
|
||||||
|
public static EdgeId From(NodeId src, string kind, NodeId dst) =>
|
||||||
|
new(HexSha256($"{src.Value}->{kind}->{dst.Value}"));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Graph edge with content-addressed identity and optional cycle-cut marker.
|
||||||
|
/// </summary>
|
||||||
|
public sealed record Edge(
|
||||||
|
EdgeId Id,
|
||||||
|
NodeId Src,
|
||||||
|
string Kind,
|
||||||
|
NodeId Dst,
|
||||||
|
JsonElement Attrs,
|
||||||
|
bool IsCycleCut = false);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Graph node with content-addressed identity.
|
||||||
|
/// </summary>
|
||||||
|
public sealed record Node(
|
||||||
|
NodeId Id,
|
||||||
|
string Kind,
|
||||||
|
JsonElement Attrs);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Immutable policy with content-addressed identity.
|
||||||
|
/// </summary>
|
||||||
|
public sealed record Policy(
|
||||||
|
string Version,
|
||||||
|
JsonElement Rules,
|
||||||
|
string ConstantsDigest);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Individual verdict with its own content-addressed digest.
|
||||||
|
/// </summary>
|
||||||
|
public sealed record Verdict(
|
||||||
|
NodeId Node,
|
||||||
|
string Status,
|
||||||
|
JsonElement Evidence,
|
||||||
|
string VerdictDigest);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Complete resolution result with all digests for verification.
|
||||||
|
/// </summary>
|
||||||
|
public sealed record ResolutionResult(
|
||||||
|
ImmutableArray<NodeId> TraversalSequence,
|
||||||
|
ImmutableArray<Verdict> Verdicts,
|
||||||
|
string GraphDigest,
|
||||||
|
string PolicyDigest,
|
||||||
|
string FinalDigest);
|
||||||
|
```
|
||||||
|
|
||||||
|
### Resolver Algorithm
|
||||||
|
|
||||||
|
```text
|
||||||
|
normalize(graph); // IDs, fields, ordering
|
||||||
|
validate(graph); // No implicit data, cycles declared
|
||||||
|
assert no-ambient-inputs(); // Runtime purity check
|
||||||
|
|
||||||
|
order = topoSortWithLexTieBreak(graph); // Uses only evidence edges
|
||||||
|
verdicts = []
|
||||||
|
|
||||||
|
for node in order:
|
||||||
|
inbound = gatherInboundEvidence(node) // Pure, no IO
|
||||||
|
verdict = Evaluate(node, inbound, policy) // Pure, no IO
|
||||||
|
verdicts.append(canonicalize(verdict) with VerdictDigest)
|
||||||
|
|
||||||
|
result = {
|
||||||
|
traversal: order,
|
||||||
|
verdicts: verdicts,
|
||||||
|
graphDigest: digest(canonical(graph)),
|
||||||
|
policyDigest: digest(canonical(policy))
|
||||||
|
}
|
||||||
|
result.finalDigest = digest(canonical(result))
|
||||||
|
return result
|
||||||
|
```
|
||||||
|
|
||||||
|
### Key Invariants
|
||||||
|
|
||||||
|
1. **Canonical IDs**: `NodeId = sha256(kind:normalized-key)`, `EdgeId = sha256(src->kind->dst)`
|
||||||
|
2. **Canonical Serialization**: Alphabetical keys, sorted arrays, fixed numerics, ISO-8601 Z timestamps
|
||||||
|
3. **Fixed Traversal**: Kahn's algorithm with lexicographic tie-breaker (`SortedSet<NodeId>`)
|
||||||
|
4. **Explicit Cycles**: Cycles require `IsCycleCut = true` edge; unmarked cycles → invalid graph
|
||||||
|
5. **Evidence-Only Evaluation**: Node verdict = f(node, inbound_edges, policy) — no ambient IO
|
||||||
|
6. **Stable Outputs**: `(TraversalSequence, Verdicts[], GraphDigest, PolicyDigest, FinalDigest)`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Gap Analysis vs Current Implementation
|
||||||
|
|
||||||
|
### Already Implemented (No Work Needed)
|
||||||
|
|
||||||
|
| Requirement | Implementation | Location |
|
||||||
|
|-------------|---------------|----------|
|
||||||
|
| Canonical JSON | RFC 8785 compliant | `CanonicalJsonSerializer`, `Rfc8785JsonCanonicalizer` |
|
||||||
|
| Alphabetical key sorting | `StableDictionaryConverter` | `StellaOps.Canonicalization` |
|
||||||
|
| ISO-8601 UTC timestamps | `Iso8601DateTimeConverter` | `StellaOps.Canonicalization` |
|
||||||
|
| Topological sort with lex tie-break | `SortedSet<string>` | `DeterministicGraphOrderer:134` |
|
||||||
|
| K4 Four-Valued Lattice | Complete | `K4Lattice.cs` |
|
||||||
|
| VEX Lattice Merging | With traces | `OpenVexStatementMerger` |
|
||||||
|
| Merkle Tree Proofs | SHA256-based | `DeterministicMerkleTreeBuilder` |
|
||||||
|
| Static Purity Analysis | 16+ patterns | `ProhibitedPatternAnalyzer` |
|
||||||
|
| Injected Timestamp | Context-based | `PolicyEvaluationContext.Now` |
|
||||||
|
| Graph Content Hash | SHA256 | `DeterministicGraphOrderer.ComputeCanonicalHash()` |
|
||||||
|
|
||||||
|
### Gaps to Implement
|
||||||
|
|
||||||
|
| Gap | Priority | Sprint | Description |
|
||||||
|
|-----|----------|--------|-------------|
|
||||||
|
| Unified Resolver | P0 | 9100.0001.0001 | Single entry point producing `ResolutionResult` |
|
||||||
|
| Cycle-Cut Edges | P1 | 9100.0001.0002 | `IsCycleCut` edge property; validation |
|
||||||
|
| EdgeId | P2 | 9100.0001.0003 | Content-addressed edge identifiers |
|
||||||
|
| FinalDigest | P1 | 9100.0002.0001 | Composite run-level digest |
|
||||||
|
| Per-Node VerdictDigest | P2 | 9100.0002.0002 | Individual verdict digests |
|
||||||
|
| Runtime Purity | P1 | 9100.0003.0001 | Runtime enforcement beyond static analysis |
|
||||||
|
| Graph Validation + NFC | P3 | 9100.0003.0002 | Pre-traversal validation; NFC normalization |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implementation Phases
|
||||||
|
|
||||||
|
### Phase 1: Core Resolver Package (Sprint 9100.0001.*)
|
||||||
|
|
||||||
|
**Goal:** Create `StellaOps.Resolver` library with unified resolver pattern.
|
||||||
|
|
||||||
|
1. **Sprint 9100.0001.0001** — Core Resolver
|
||||||
|
- `DeterministicResolver` class
|
||||||
|
- `ResolutionResult` record
|
||||||
|
- Integration with existing `DeterministicGraphOrderer`
|
||||||
|
- Integration with existing `TrustLatticeEngine`
|
||||||
|
|
||||||
|
2. **Sprint 9100.0001.0002** — Cycle-Cut Edges
|
||||||
|
- `IsCycleCut` property on edges
|
||||||
|
- Cycle validation (unmarked cycles → error)
|
||||||
|
- Graph validation before traversal
|
||||||
|
|
||||||
|
3. **Sprint 9100.0001.0003** — Content-Addressed EdgeId
|
||||||
|
- `EdgeId` record
|
||||||
|
- Edge content addressing
|
||||||
|
- Merkle tree inclusion of edges
|
||||||
|
|
||||||
|
### Phase 2: Digest Infrastructure (Sprint 9100.0002.*)
|
||||||
|
|
||||||
|
**Goal:** Implement comprehensive digest chain for verification.
|
||||||
|
|
||||||
|
1. **Sprint 9100.0002.0001** — FinalDigest
|
||||||
|
- Composite digest computation
|
||||||
|
- `sha256(canonical({graphDigest, policyDigest, verdicts[]}))`
|
||||||
|
- Integration with attestation system
|
||||||
|
|
||||||
|
2. **Sprint 9100.0002.0002** — Per-Node VerdictDigest
|
||||||
|
- Verdict-level content addressing
|
||||||
|
- Drill-down debugging ("which node changed?")
|
||||||
|
- Delta detection between runs
|
||||||
|
|
||||||
|
### Phase 3: Purity & Validation (Sprint 9100.0003.*)
|
||||||
|
|
||||||
|
**Goal:** Harden determinism guarantees with runtime enforcement.
|
||||||
|
|
||||||
|
1. **Sprint 9100.0003.0001** — Runtime Purity Enforcement
|
||||||
|
- Runtime guards beyond static analysis
|
||||||
|
- Dependency injection shims for ambient services
|
||||||
|
- Test harness for purity verification
|
||||||
|
|
||||||
|
2. **Sprint 9100.0003.0002** — Graph Validation & NFC
|
||||||
|
- Pre-traversal validation ("no implicit data")
|
||||||
|
- Unicode NFC normalization for string fields
|
||||||
|
- Evidence completeness assertions
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Testing Requirements
|
||||||
|
|
||||||
|
### Mandatory Test Types
|
||||||
|
|
||||||
|
1. **Replay Test**: Same input twice → identical `FinalDigest`
|
||||||
|
2. **Permutation Test**: Shuffle input nodes/edges → identical outputs
|
||||||
|
3. **Cycle Test**: Introduce cycle → fail unless `IsCycleCut` edge present
|
||||||
|
4. **Ambience Test**: Forbid calls to time/env/network during evaluation
|
||||||
|
5. **Serialization Test**: Canonicalization changes → predictable digest changes
|
||||||
|
|
||||||
|
### Property-Based Tests
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
// Idempotency
|
||||||
|
[Property]
|
||||||
|
public void Resolver_SameInput_SameOutput(Graph graph)
|
||||||
|
{
|
||||||
|
var result1 = resolver.Run(graph);
|
||||||
|
var result2 = resolver.Run(graph);
|
||||||
|
Assert.Equal(result1.FinalDigest, result2.FinalDigest);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Order Independence
|
||||||
|
[Property]
|
||||||
|
public void Resolver_ShuffledInputs_SameOutput(Graph graph, int seed)
|
||||||
|
{
|
||||||
|
var shuffled = ShuffleNodesAndEdges(graph, seed);
|
||||||
|
var result1 = resolver.Run(graph);
|
||||||
|
var result2 = resolver.Run(shuffled);
|
||||||
|
Assert.Equal(result1.FinalDigest, result2.FinalDigest);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Cycle Detection
|
||||||
|
[Property]
|
||||||
|
public void Resolver_UnmarkedCycle_Throws(Graph graphWithCycle)
|
||||||
|
{
|
||||||
|
Assume.That(!graphWithCycle.Edges.Any(e => e.IsCycleCut));
|
||||||
|
Assert.Throws<InvalidGraphException>(() => resolver.Run(graphWithCycle));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Integration Points
|
||||||
|
|
||||||
|
### Existing Components to Integrate
|
||||||
|
|
||||||
|
| Component | Integration |
|
||||||
|
|-----------|-------------|
|
||||||
|
| `DeterministicGraphOrderer` | Use for traversal order computation |
|
||||||
|
| `TrustLatticeEngine` | Use for K4 lattice evaluation |
|
||||||
|
| `CanonicalJsonSerializer` | Use for all serialization |
|
||||||
|
| `ContentAddressedIdGenerator` | Extend for EdgeId and VerdictId |
|
||||||
|
| `ProhibitedPatternAnalyzer` | Combine with runtime guards |
|
||||||
|
| `PolicyEvaluationContext` | Use for injected inputs |
|
||||||
|
|
||||||
|
### New Modules
|
||||||
|
|
||||||
|
| Module | Purpose |
|
||||||
|
|--------|---------|
|
||||||
|
| `StellaOps.Resolver` | Core resolver library |
|
||||||
|
| `StellaOps.Resolver.Testing` | Test fixtures and harnesses |
|
||||||
|
| `StellaOps.Resolver.Attestation` | Integration with proof chain |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
1. **Unified API**: Single `resolver.Run(graph)` produces complete `ResolutionResult`
|
||||||
|
2. **Deterministic**: 100% pass rate on replay/permutation/cycle tests
|
||||||
|
3. **Auditable**: `FinalDigest` enables single-value verification
|
||||||
|
4. **Performant**: No more than 10% overhead vs current fragmented approach
|
||||||
|
5. **Documented**: Full API documentation and integration guide
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix: C# Reference Implementation
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
public sealed class DeterministicResolver
|
||||||
|
{
|
||||||
|
private readonly Policy _policy;
|
||||||
|
private readonly string _policyDigest;
|
||||||
|
private readonly DeterministicGraphOrderer _orderer;
|
||||||
|
private readonly TrustLatticeEngine _lattice;
|
||||||
|
private readonly CanonicalJsonSerializer _serializer;
|
||||||
|
|
||||||
|
public DeterministicResolver(
|
||||||
|
Policy policy,
|
||||||
|
DeterministicGraphOrderer orderer,
|
||||||
|
TrustLatticeEngine lattice,
|
||||||
|
CanonicalJsonSerializer serializer)
|
||||||
|
{
|
||||||
|
_policy = Canon(policy);
|
||||||
|
_policyDigest = HexSha256(serializer.SerializeToBytes(_policy));
|
||||||
|
_orderer = orderer;
|
||||||
|
_lattice = lattice;
|
||||||
|
_serializer = serializer;
|
||||||
|
}
|
||||||
|
|
||||||
|
public ResolutionResult Run(EvidenceGraph graph)
|
||||||
|
{
|
||||||
|
// 1. Canonicalize and validate
|
||||||
|
var canonical = _orderer.Canonicalize(graph);
|
||||||
|
ValidateCycles(canonical);
|
||||||
|
EnsureNoAmbientInputs();
|
||||||
|
|
||||||
|
// 2. Traverse in deterministic order
|
||||||
|
var traversal = canonical.Nodes
|
||||||
|
.Select(n => NodeId.Parse(n.Id))
|
||||||
|
.ToImmutableArray();
|
||||||
|
|
||||||
|
// 3. Evaluate each node purely
|
||||||
|
var verdicts = new List<Verdict>(traversal.Length);
|
||||||
|
foreach (var nodeId in traversal)
|
||||||
|
{
|
||||||
|
var node = GetNode(canonical, nodeId);
|
||||||
|
var inbound = GatherInboundEvidence(canonical, nodeId);
|
||||||
|
var verdict = EvaluatePure(node, inbound, _policy);
|
||||||
|
var verdictBytes = _serializer.SerializeToBytes(verdict);
|
||||||
|
verdicts.Add(verdict with { VerdictDigest = HexSha256(verdictBytes) });
|
||||||
|
}
|
||||||
|
|
||||||
|
// 4. Compute digests
|
||||||
|
var verdictsArray = verdicts.ToImmutableArray();
|
||||||
|
var resultWithoutFinal = new ResolutionResult(
|
||||||
|
traversal,
|
||||||
|
verdictsArray,
|
||||||
|
canonical.ContentHash,
|
||||||
|
_policyDigest,
|
||||||
|
"");
|
||||||
|
|
||||||
|
var finalBytes = _serializer.SerializeToBytes(resultWithoutFinal);
|
||||||
|
return resultWithoutFinal with { FinalDigest = HexSha256(finalBytes) };
|
||||||
|
}
|
||||||
|
|
||||||
|
private void ValidateCycles(CanonicalGraph graph)
|
||||||
|
{
|
||||||
|
// Detect cycles; require IsCycleCut for each
|
||||||
|
var cycles = DetectCycles(graph);
|
||||||
|
var unmarked = cycles.Where(c => !c.CutEdge.IsCycleCut).ToList();
|
||||||
|
if (unmarked.Any())
|
||||||
|
{
|
||||||
|
throw new InvalidGraphException(
|
||||||
|
$"Graph contains {unmarked.Count} unmarked cycle(s). " +
|
||||||
|
"Add IsCycleCut=true to break cycles.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private Verdict EvaluatePure(Node node, InboundEvidence evidence, Policy policy)
|
||||||
|
{
|
||||||
|
// Pure evaluation: no IO, no ambient state
|
||||||
|
return _lattice.Evaluate(node, evidence, policy);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Related Documents
|
||||||
|
|
||||||
|
- `docs/modules/attestor/proof-chain-specification.md`
|
||||||
|
- `docs/modules/policy/architecture.md`
|
||||||
|
- `docs/testing/testing-strategy-models.md`
|
||||||
|
- `src/Policy/__Libraries/StellaOps.Policy/TrustLattice/K4Lattice.cs`
|
||||||
|
- `src/Scanner/__Libraries/StellaOps.Scanner.Reachability/Ordering/DeterministicGraphOrderer.cs`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Approval
|
||||||
|
|
||||||
|
| Role | Name | Date | Decision |
|
||||||
|
|------|------|------|----------|
|
||||||
|
| Architecture Lead | — | 2025-12-24 | Approved |
|
||||||
|
| Policy Guild Lead | — | 2025-12-24 | Approved |
|
||||||
|
| Scanner Guild Lead | — | 2025-12-24 | Approved |
|
||||||
@@ -0,0 +1,271 @@
|
|||||||
|
# Product Advisory: Evidence-Weighted Score Model
|
||||||
|
|
||||||
|
**Date:** 24-Dec-2025
|
||||||
|
**Status:** Approved for Implementation
|
||||||
|
**Sprint Batch:** 8200.0012
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
This advisory introduces a **unified evidence-weighted scoring model** that combines six evidence dimensions into a single 0-100 score per vulnerability finding. The score enables rapid triage by surfacing the most "real" risks with transparent, decomposable evidence.
|
||||||
|
|
||||||
|
### Key Benefits
|
||||||
|
|
||||||
|
- **Prioritize quickly:** Sort by score to fix the most "real" risks first
|
||||||
|
- **Reduce noise:** Thin, speculative findings sink to the bottom
|
||||||
|
- **Stay explainable:** Score decomposes into evidence "slices" on hover
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Problem Statement
|
||||||
|
|
||||||
|
The current platform has **four independent scoring systems** that don't interoperate:
|
||||||
|
|
||||||
|
1. `ConfidenceCalculator` (Policy module) — 5 factors, outputs 0-1
|
||||||
|
2. `ScorePolicy` (Policy module) — 4 factors in basis points
|
||||||
|
3. `EvidenceDensityScorer` (Scanner module) — 8 factors, outputs 0-1
|
||||||
|
4. `TrustVector` (Excititor module) — 3 factors, outputs 0-1
|
||||||
|
|
||||||
|
Users cannot sort findings by a single "risk strength" metric. They must mentally combine multiple signals, leading to:
|
||||||
|
|
||||||
|
- Inconsistent prioritization across teams
|
||||||
|
- Missed high-risk findings buried in noise
|
||||||
|
- Difficulty explaining triage decisions to stakeholders
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Proposed Solution
|
||||||
|
|
||||||
|
### Scoring Model
|
||||||
|
|
||||||
|
```
|
||||||
|
Score = clamp01(
|
||||||
|
W_rch*RCH + W_rts*RTS + W_bkp*BKP + W_xpl*XPL + W_src*SRC - W_mit*MIT
|
||||||
|
) * 100
|
||||||
|
```
|
||||||
|
|
||||||
|
### Evidence Dimensions
|
||||||
|
|
||||||
|
| Symbol | Name | Description | Weight |
|
||||||
|
|--------|------|-------------|--------|
|
||||||
|
| **RCH** | Reachability | Static/dynamic reachability confidence | 0.30 |
|
||||||
|
| **RTS** | Runtime | Runtime signal strength (eBPF, hits, recency) | 0.25 |
|
||||||
|
| **BKP** | Backport | Backport/patch evidence strength | 0.15 |
|
||||||
|
| **XPL** | Exploit | Exploit likelihood (EPSS + KEV) | 0.15 |
|
||||||
|
| **SRC** | Source Trust | Source trust (vendor > distro > community) | 0.10 |
|
||||||
|
| **MIT** | Mitigations | Active mitigations (subtractive) | 0.10 |
|
||||||
|
|
||||||
|
### Guardrails
|
||||||
|
|
||||||
|
| Condition | Action | Rationale |
|
||||||
|
|-----------|--------|-----------|
|
||||||
|
| BKP=1 + not_affected + RTS<0.6 | Cap at 15 | Vendor confirms not affected |
|
||||||
|
| RTS >= 0.8 | Floor at 60 | Strong live signal |
|
||||||
|
| RCH=0 + RTS=0 | Cap at 45 | Speculative finding |
|
||||||
|
|
||||||
|
### Score Buckets
|
||||||
|
|
||||||
|
| Bucket | Range | Action |
|
||||||
|
|--------|-------|--------|
|
||||||
|
| **ActNow** | 90-100 | Immediate action required |
|
||||||
|
| **ScheduleNext** | 70-89 | Schedule for next sprint |
|
||||||
|
| **Investigate** | 40-69 | Investigate when touching component |
|
||||||
|
| **Watchlist** | 0-39 | Low priority, monitor |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Gap Analysis
|
||||||
|
|
||||||
|
### Existing Infrastructure (Reusable)
|
||||||
|
|
||||||
|
| Component | Location | Reuse |
|
||||||
|
|-----------|----------|-------|
|
||||||
|
| Reachability states | `Policy/ConfidenceCalculator.cs` | Direct mapping to RCH |
|
||||||
|
| Runtime evidence | `Policy/RuntimeEvidence` | Direct mapping to RTS |
|
||||||
|
| Backport detection | `Concelier/BackportProofService.cs` | Needs normalization |
|
||||||
|
| EPSS bands | `Scanner/EpssPriorityCalculator.cs` | Needs normalization |
|
||||||
|
| KEV status | `Concelier/VendorRiskSignalExtractor.cs` | Direct flag |
|
||||||
|
| Trust vector | `Excititor/TrustVector.cs` | Direct mapping to SRC |
|
||||||
|
| Gate multipliers | `Policy/GateMultipliersBps` | Needs inversion to MIT |
|
||||||
|
|
||||||
|
### New Components Required
|
||||||
|
|
||||||
|
1. **Unified Score Calculator** — Core formula implementation
|
||||||
|
2. **Normalizers** — Convert raw evidence to 0-1 inputs
|
||||||
|
3. **Guardrails Engine** — Apply caps/floors
|
||||||
|
4. **API Endpoints** — Expose scores via REST
|
||||||
|
5. **UI Components** — Score pill, breakdown popover, badges
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implementation Sprints
|
||||||
|
|
||||||
|
| Sprint | Focus | Key Deliverables |
|
||||||
|
|--------|-------|------------------|
|
||||||
|
| **8200.0012.0001** | Core Library | Calculator, input models, weight policy, guardrails |
|
||||||
|
| **8200.0012.0002** | Normalizers | BKP, XPL, MIT, RCH, RTS, SRC normalizers |
|
||||||
|
| **8200.0012.0003** | Policy Integration | Score-based rules, verdict enrichment, attestation |
|
||||||
|
| **8200.0012.0004** | API Endpoints | Single/batch score, history, webhooks |
|
||||||
|
| **8200.0012.0005** | Frontend UI | Score pill, breakdown, badges, history chart |
|
||||||
|
|
||||||
|
**Total Tasks:** 268 across 5 sprints
|
||||||
|
**Estimated Duration:** 6 months
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## API Shape
|
||||||
|
|
||||||
|
### Score Response
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"findingId": "CVE-2024-1234@pkg:deb/debian/curl@7.64.0-4",
|
||||||
|
"score": 78,
|
||||||
|
"bucket": "ScheduleNext",
|
||||||
|
"inputs": {
|
||||||
|
"rch": 0.85, "rts": 0.40, "bkp": 0.00,
|
||||||
|
"xpl": 0.70, "src": 0.80, "mit": 0.10
|
||||||
|
},
|
||||||
|
"weights": {
|
||||||
|
"rch": 0.30, "rts": 0.25, "bkp": 0.15,
|
||||||
|
"xpl": 0.15, "src": 0.10, "mit": 0.10
|
||||||
|
},
|
||||||
|
"flags": ["live-signal", "proven-path"],
|
||||||
|
"explanations": [
|
||||||
|
"Static path to vulnerable sink (confidence: 85%)",
|
||||||
|
"EPSS: 0.8% probability (High band)",
|
||||||
|
"Distro VEX signed (trust: 80%)"
|
||||||
|
],
|
||||||
|
"caps": {
|
||||||
|
"speculativeCap": false,
|
||||||
|
"notAffectedCap": false,
|
||||||
|
"runtimeFloor": false
|
||||||
|
},
|
||||||
|
"policyDigest": "sha256:abc123...",
|
||||||
|
"calculatedAt": "2026-01-15T14:30:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## UI Design
|
||||||
|
|
||||||
|
### Score Pill
|
||||||
|
|
||||||
|
```
|
||||||
|
┌───────┐
|
||||||
|
│ 78 │ ← White text on amber background (ScheduleNext bucket)
|
||||||
|
└───────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Breakdown Popover
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────┐
|
||||||
|
│ Evidence Score: 78/100 │
|
||||||
|
│ Bucket: Schedule Next Sprint │
|
||||||
|
├─────────────────────────────────────────┤
|
||||||
|
│ Reachability ████████▒▒ 0.85 │
|
||||||
|
│ Runtime ████▒▒▒▒▒▒ 0.40 │
|
||||||
|
│ Backport ▒▒▒▒▒▒▒▒▒▒ 0.00 │
|
||||||
|
│ Exploit ███████▒▒▒ 0.70 │
|
||||||
|
│ Source Trust ████████▒▒ 0.80 │
|
||||||
|
│ Mitigations -█▒▒▒▒▒▒▒▒▒ 0.10 │
|
||||||
|
├─────────────────────────────────────────┤
|
||||||
|
│ 🟢 Live signal detected │
|
||||||
|
│ ✓ Proven reachability path │
|
||||||
|
└─────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Governance & Tuning
|
||||||
|
|
||||||
|
1. **Policy-Driven Weights:** Different environments (prod/staging/dev) can have different weight profiles
|
||||||
|
2. **Versioned Policies:** Policy changes are versioned and signed; same inputs + policy = same score
|
||||||
|
3. **Audit Trail:** All score calculations logged with inputs, policy digest, and result
|
||||||
|
4. **Threshold Tuning:** Bucket thresholds configurable per tenant
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Migration Strategy
|
||||||
|
|
||||||
|
### Phase 1: Dual Emit (3 months)
|
||||||
|
- Both Confidence and EWS emitted in verdicts
|
||||||
|
- Feature flag controls which is primary
|
||||||
|
- Telemetry compares rankings
|
||||||
|
|
||||||
|
### Phase 2: EWS Primary (3 months)
|
||||||
|
- EWS is default; Confidence available for compatibility
|
||||||
|
- Deprecation warnings for Confidence consumers
|
||||||
|
|
||||||
|
### Phase 3: Confidence Removal (v3.0)
|
||||||
|
- Confidence scoring removed
|
||||||
|
- EWS is sole scoring system
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Risks & Mitigations
|
||||||
|
|
||||||
|
| Risk | Impact | Mitigation |
|
||||||
|
|------|--------|------------|
|
||||||
|
| Weight tuning requires iteration | Suboptimal prioritization | Conservative defaults; telemetry |
|
||||||
|
| Migration confusion | User errors | Clear docs; gradual rollout |
|
||||||
|
| Performance regression | Slower evaluation | Caching; benchmarks (<50ms) |
|
||||||
|
| Attestation size increase | Storage cost | Compact proof format |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
1. **Adoption:** 80% of findings viewed with EWS score within 6 months
|
||||||
|
2. **Triage Time:** 30% reduction in mean-time-to-triage
|
||||||
|
3. **False Positive Rate:** <5% of "ActNow" findings downgraded after review
|
||||||
|
4. **Determinism:** 100% of scores reproducible given same inputs + policy
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix: Normalization Details
|
||||||
|
|
||||||
|
### BKP (Backport) Normalization
|
||||||
|
|
||||||
|
| Evidence Tier | BKP Range |
|
||||||
|
|---------------|-----------|
|
||||||
|
| No evidence | 0.00 |
|
||||||
|
| Distro advisory text | 0.40-0.55 |
|
||||||
|
| Changelog mention | 0.45-0.60 |
|
||||||
|
| Patch header match | 0.70-0.85 |
|
||||||
|
| HunkSig match | 0.80-0.92 |
|
||||||
|
| Binary fingerprint | 0.90-1.00 |
|
||||||
|
|
||||||
|
### XPL (Exploit) Normalization
|
||||||
|
|
||||||
|
| Signal | XPL Value |
|
||||||
|
|--------|-----------|
|
||||||
|
| KEV present | floor 0.40 |
|
||||||
|
| EPSS top 1% | 0.90-1.00 |
|
||||||
|
| EPSS top 5% | 0.70-0.89 |
|
||||||
|
| EPSS top 25% | 0.40-0.69 |
|
||||||
|
| EPSS below 25% | 0.20-0.39 |
|
||||||
|
| No EPSS data | 0.30 |
|
||||||
|
|
||||||
|
### MIT (Mitigation) Normalization
|
||||||
|
|
||||||
|
| Mitigation | MIT Value |
|
||||||
|
|------------|-----------|
|
||||||
|
| Feature flag disabled | 0.20-0.40 |
|
||||||
|
| Auth required | 0.10-0.20 |
|
||||||
|
| Admin only | 0.15-0.25 |
|
||||||
|
| Seccomp profile | 0.10-0.25 |
|
||||||
|
| Network isolation | 0.05-0.15 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- Sprint Files: `docs/implplan/SPRINT_8200_0012_*.md`
|
||||||
|
- Module Docs: `docs/modules/signals/architecture.md` (to be created)
|
||||||
|
- Existing Confidence: `docs/modules/policy/confidence-scoring.md`
|
||||||
|
- EPSS Enrichment: `docs/modules/scanner/epss-enrichment.md`
|
||||||
|
- Backport Detection: `docs/modules/concelier/backport-detection.md`
|
||||||
@@ -37,7 +37,7 @@ This guide translates the deterministic reachability blueprint into concrete wor
|
|||||||
| Stream | Owner Guild(s) | Key deliverables |
|
| Stream | Owner Guild(s) | Key deliverables |
|
||||||
|--------|----------------|------------------|
|
|--------|----------------|------------------|
|
||||||
| **Native symbols & callgraphs** | Scanner Worker · Symbols Guild | Ship `Scanner.Symbols.Native` + `Scanner.CallGraph.Native`, integrate Symbol Manifest v1, demangle Itanium/MSVC names, emit `FuncNode`/`CallEdge` CAS bundles (task `SCANNER-NATIVE-401-015`). |
|
| **Native symbols & callgraphs** | Scanner Worker · Symbols Guild | Ship `Scanner.Symbols.Native` + `Scanner.CallGraph.Native`, integrate Symbol Manifest v1, demangle Itanium/MSVC names, emit `FuncNode`/`CallEdge` CAS bundles (task `SCANNER-NATIVE-401-015`). |
|
||||||
| **Reachability store** | Signals · BE-Base Platform | Provision shared Mongo collections (`func_nodes`, `call_edges`, `cve_func_hits`), indexes, and repositories plus REST hooks for reuse (task `SIG-STORE-401-016`). |
|
| **Reachability store** | Signals · BE-Base Platform | Provision shared PostgreSQL tables (`func_nodes`, `call_edges`, `cve_func_hits`), indexes, and repositories plus REST hooks for reuse (task `SIG-STORE-401-016`). |
|
||||||
| **Language lifters** | Scanner Worker | CLI/hosted lifters for DotNet, Go, Node/Deno, JVM, Rust, Swift, Binary, Shell with CAS uploads and richgraph output |
|
| **Language lifters** | Scanner Worker | CLI/hosted lifters for DotNet, Go, Node/Deno, JVM, Rust, Swift, Binary, Shell with CAS uploads and richgraph output |
|
||||||
| **Signals ingestion & scoring** | Signals | `/callgraphs`, `/runtime-facts` (JSON + NDJSON/gzip), `/graphs/{id}`, `/reachability/recompute` GA; CAS-backed storage, runtime dedupe, BFS+predicates scoring |
|
| **Signals ingestion & scoring** | Signals | `/callgraphs`, `/runtime-facts` (JSON + NDJSON/gzip), `/graphs/{id}`, `/reachability/recompute` GA; CAS-backed storage, runtime dedupe, BFS+predicates scoring |
|
||||||
| **Runtime capture** | Zastava + Runtime Guild | EntryTrace/eBPF samplers, NDJSON batches (symbol IDs + timestamps + counts) |
|
| **Runtime capture** | Zastava + Runtime Guild | EntryTrace/eBPF samplers, NDJSON batches (symbol IDs + timestamps + counts) |
|
||||||
|
|||||||
@@ -21,8 +21,8 @@
|
|||||||
- Empty runtime stream is rejected.
|
- Empty runtime stream is rejected.
|
||||||
|
|
||||||
## Storage and cache
|
## Storage and cache
|
||||||
- Stored alongside reachability facts in Mongo collection `reachability_facts`.
|
- Stored alongside reachability facts in PostgreSQL table `reachability_facts`.
|
||||||
- Runtime hits cached in Redis via `reachability_cache:*` entries; invalidated on ingest.
|
- Runtime hits cached in Valkey via `reachability_cache:*` entries; invalidated on ingest.
|
||||||
|
|
||||||
## Interaction with scoring
|
## Interaction with scoring
|
||||||
- Ingest triggers recompute: runtime hits added to prior facts’ hits, targets set to symbols observed, entryPoints taken from callgraph.
|
- Ingest triggers recompute: runtime hits added to prior facts’ hits, targets set to symbols observed, entryPoints taken from callgraph.
|
||||||
|
|||||||
286
docs/reproducibility.md
Normal file
286
docs/reproducibility.md
Normal file
@@ -0,0 +1,286 @@
|
|||||||
|
# StellaOps Reproducibility Specification
|
||||||
|
|
||||||
|
This document defines the reproducibility guarantees, verdict identity computation, and replay procedures for StellaOps artifacts.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
StellaOps provides **deterministic, reproducible outputs** for all security artifacts:
|
||||||
|
- SBOM generation (CycloneDX 1.6, SPDX 3.0.1)
|
||||||
|
- VEX statements (OpenVEX)
|
||||||
|
- Policy verdicts
|
||||||
|
- Delta computations
|
||||||
|
- DSSE attestations and Sigstore bundles
|
||||||
|
|
||||||
|
**Core Guarantee:** Given identical inputs (image digest, advisory feeds, policies, tool versions), StellaOps produces byte-for-byte identical outputs with matching content-addressed identifiers.
|
||||||
|
|
||||||
|
## Verdict Identity Formula
|
||||||
|
|
||||||
|
### Content-Addressed Verdict ID
|
||||||
|
|
||||||
|
All verdicts use content-addressed identifiers computed as:
|
||||||
|
|
||||||
|
```
|
||||||
|
VerdictId = SHA256(Canonicalize(VerdictPayload))
|
||||||
|
```
|
||||||
|
|
||||||
|
Where `VerdictPayload` includes:
|
||||||
|
- **Delta ID**: Content hash of the security delta
|
||||||
|
- **Blocking Drivers**: Sorted list of risk-increasing factors
|
||||||
|
- **Warning Drivers**: Sorted list of advisory factors
|
||||||
|
- **Applied Exceptions**: Sorted list of exception IDs covering findings
|
||||||
|
- **Gate Level**: Recommended gate (G0-G4)
|
||||||
|
- **Input Stamps**: Hashes of all inputs (see below)
|
||||||
|
|
||||||
|
### Input Stamps
|
||||||
|
|
||||||
|
Every artifact includes `InputStamps` capturing the provenance of all inputs:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"feedSnapshotHash": "sha256:abc123...",
|
||||||
|
"policyManifestHash": "sha256:def456...",
|
||||||
|
"sourceCodeHash": "sha256:789ghi...",
|
||||||
|
"baseImageDigest": "sha256:jkl012...",
|
||||||
|
"vexDocumentHashes": ["sha256:mno345..."],
|
||||||
|
"toolchainVersion": "1.0.0",
|
||||||
|
"custom": {}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Determinism Manifest
|
||||||
|
|
||||||
|
The `DeterminismManifest` (schema v1.0) tracks artifact reproducibility:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"schemaVersion": "1.0",
|
||||||
|
"artifact": {
|
||||||
|
"type": "verdict",
|
||||||
|
"name": "scan-verdict",
|
||||||
|
"version": "2025-12-24T12:00:00Z",
|
||||||
|
"format": "StellaOps.DeltaVerdict@1"
|
||||||
|
},
|
||||||
|
"canonicalHash": {
|
||||||
|
"algorithm": "SHA-256",
|
||||||
|
"value": "abc123def456...",
|
||||||
|
"encoding": "hex"
|
||||||
|
},
|
||||||
|
"inputs": {
|
||||||
|
"feedSnapshotHash": "sha256:...",
|
||||||
|
"policyManifestHash": "sha256:...",
|
||||||
|
"baseImageDigest": "sha256:..."
|
||||||
|
},
|
||||||
|
"toolchain": {
|
||||||
|
"platform": ".NET 10.0",
|
||||||
|
"components": [
|
||||||
|
{"name": "StellaOps.Scanner", "version": "1.0.0"},
|
||||||
|
{"name": "StellaOps.Policy", "version": "1.0.0"}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"reproducibility": {
|
||||||
|
"clockFixed": true,
|
||||||
|
"orderingGuarantee": "stable-sort",
|
||||||
|
"normalizationRules": ["UTF-8", "LF", "canonical-json"]
|
||||||
|
},
|
||||||
|
"generatedAt": "2025-12-24T12:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Canonical JSON Serialization
|
||||||
|
|
||||||
|
All JSON outputs follow RFC 8785 (JSON Canonicalization Scheme):
|
||||||
|
1. Keys sorted lexicographically
|
||||||
|
2. No whitespace between tokens
|
||||||
|
3. Unicode escaping for non-ASCII
|
||||||
|
4. Numbers without leading zeros
|
||||||
|
5. UTF-8 encoding
|
||||||
|
|
||||||
|
## DSSE Attestation Format
|
||||||
|
|
||||||
|
### Envelope Structure
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"payloadType": "application/vnd.in-toto+json",
|
||||||
|
"payload": "<base64url-encoded statement>",
|
||||||
|
"signatures": [
|
||||||
|
{
|
||||||
|
"keyid": "sha256:...",
|
||||||
|
"sig": "<base64url-encoded signature>"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### In-toto Statement
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"_type": "https://in-toto.io/Statement/v1",
|
||||||
|
"subject": [
|
||||||
|
{
|
||||||
|
"name": "registry.example.com/image:tag",
|
||||||
|
"digest": {"sha256": "abc123..."}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"predicateType": "https://stellaops.io/attestation/verdict/v1",
|
||||||
|
"predicate": {
|
||||||
|
"verdictId": "sha256:...",
|
||||||
|
"status": "pass",
|
||||||
|
"gate": "G2",
|
||||||
|
"inputs": {...},
|
||||||
|
"evidence": [...]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Sigstore Bundle Format
|
||||||
|
|
||||||
|
StellaOps produces Sigstore bundles (v0.3) for offline verification:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"$mediaType": "application/vnd.dev.sigstore.bundle.v0.3+json",
|
||||||
|
"verificationMaterial": {
|
||||||
|
"certificate": {...},
|
||||||
|
"tlogEntries": [{
|
||||||
|
"logIndex": "12345",
|
||||||
|
"logId": {...},
|
||||||
|
"inclusionProof": {...}
|
||||||
|
}]
|
||||||
|
},
|
||||||
|
"dsseEnvelope": {...}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Replay Procedure
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
1. Offline bundle containing:
|
||||||
|
- Advisory feed snapshot
|
||||||
|
- Policy pack
|
||||||
|
- VEX documents
|
||||||
|
- Tool binaries (pinned versions)
|
||||||
|
|
||||||
|
### Steps
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Extract offline bundle
|
||||||
|
stella offline extract --bundle offline-kit.tar.gz --output ./replay
|
||||||
|
|
||||||
|
# 2. Set deterministic environment
|
||||||
|
export STELLAOPS_DETERMINISTIC_SEED=42
|
||||||
|
export STELLAOPS_CLOCK_FIXED=2025-12-24T12:00:00Z
|
||||||
|
|
||||||
|
# 3. Run scan with pinned inputs
|
||||||
|
stella scan \
|
||||||
|
--image registry.example.com/image@sha256:abc123 \
|
||||||
|
--feeds ./replay/feeds \
|
||||||
|
--policies ./replay/policies \
|
||||||
|
--output ./replay/output
|
||||||
|
|
||||||
|
# 4. Verify hash matches original
|
||||||
|
stella verify \
|
||||||
|
--manifest ./replay/output/manifest.json \
|
||||||
|
--expected-hash sha256:def456...
|
||||||
|
|
||||||
|
# 5. Verify DSSE attestation
|
||||||
|
stella attest verify \
|
||||||
|
--bundle ./replay/output/bundle.sigstore \
|
||||||
|
--policy verification-policy.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verification Policy
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
apiVersion: stellaops.io/v1
|
||||||
|
kind: VerificationPolicy
|
||||||
|
metadata:
|
||||||
|
name: audit-verification
|
||||||
|
spec:
|
||||||
|
requiredPredicateTypes:
|
||||||
|
- https://stellaops.io/attestation/verdict/v1
|
||||||
|
trustedIssuers:
|
||||||
|
- https://accounts.stellaops.io
|
||||||
|
maxAge: 90d
|
||||||
|
requireRekorEntry: true
|
||||||
|
unknownBudget:
|
||||||
|
maxTotal: 5
|
||||||
|
action: fail
|
||||||
|
```
|
||||||
|
|
||||||
|
## Unknown Budget Attestation
|
||||||
|
|
||||||
|
Policy thresholds are attested in verdict bundles:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"predicateType": "https://stellaops.io/attestation/budget-check/v1",
|
||||||
|
"predicate": {
|
||||||
|
"environment": "production",
|
||||||
|
"budgetConfig": {
|
||||||
|
"maxUnknownCount": 5,
|
||||||
|
"maxCumulativeUncertainty": 2.0,
|
||||||
|
"reasonLimits": {
|
||||||
|
"Reachability": 0,
|
||||||
|
"Identity": 2,
|
||||||
|
"Provenance": 2
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"actualCounts": {
|
||||||
|
"total": 3,
|
||||||
|
"byReason": {"Identity": 2, "Provenance": 1}
|
||||||
|
},
|
||||||
|
"result": "pass",
|
||||||
|
"configHash": "sha256:..."
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Schema Versions
|
||||||
|
|
||||||
|
| Format | Version | Schema Location |
|
||||||
|
|--------|---------|-----------------|
|
||||||
|
| CycloneDX | 1.6 | `docs/schemas/cyclonedx-bom-1.6.schema.json` |
|
||||||
|
| SPDX | 3.0.1 | `docs/schemas/spdx-3.0.1.schema.json` |
|
||||||
|
| OpenVEX | 0.2.0 | `docs/schemas/openvex-0.2.0.schema.json` |
|
||||||
|
| Sigstore Bundle | 0.3 | `docs/schemas/sigstore-bundle-0.3.schema.json` |
|
||||||
|
| DeterminismManifest | 1.0 | `docs/schemas/determinism-manifest-1.0.schema.json` |
|
||||||
|
|
||||||
|
## CI Integration
|
||||||
|
|
||||||
|
### Schema Validation
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# .gitea/workflows/schema-validation.yml
|
||||||
|
- name: Validate CycloneDX
|
||||||
|
run: |
|
||||||
|
sbom-utility validate \
|
||||||
|
--input-file ${{ matrix.fixture }} \
|
||||||
|
--schema docs/schemas/cyclonedx-bom-1.6.schema.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### Determinism Gate
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# .gitea/workflows/determinism-gate.yml
|
||||||
|
- name: Verify Verdict Hash
|
||||||
|
run: |
|
||||||
|
HASH1=$(stella scan --image test:latest --output /tmp/run1 | jq -r '.verdictId')
|
||||||
|
HASH2=$(stella scan --image test:latest --output /tmp/run2 | jq -r '.verdictId')
|
||||||
|
[ "$HASH1" = "$HASH2" ] || exit 1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Related Documentation
|
||||||
|
|
||||||
|
- [Testing Strategy](testing/testing-strategy-models.md)
|
||||||
|
- [Determinism Verification](testing/determinism-verification.md)
|
||||||
|
- [DSSE Attestation Guide](modules/attestor/README.md)
|
||||||
|
- [Offline Operation](24_OFFLINE_KIT.md)
|
||||||
|
- [Proof Bundle Spec](modules/triage/proof-bundle-spec.md)
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
| Version | Date | Changes |
|
||||||
|
|---------|------|---------|
|
||||||
|
| 1.0 | 2025-12-24 | Initial specification based on product advisory gap analysis |
|
||||||
@@ -44,7 +44,7 @@ Event names follow dotted notation:
|
|||||||
|
|
||||||
## Persistence
|
## Persistence
|
||||||
|
|
||||||
The Authority host converts audit records into `AuthorityLoginAttemptDocument` rows for MongoDB persistence. Documents must:
|
The Authority host converts audit records into `AuthorityLoginAttemptDocument` rows for PostgreSQL persistence (schema `authority`). Documents must:
|
||||||
|
|
||||||
- Preserve `CorrelationId`, `SubjectId`, `ClientId`, `Plugin`, `Outcome`, `Reason`, and `OccurredAt`.
|
- Preserve `CorrelationId`, `SubjectId`, `ClientId`, `Plugin`, `Outcome`, `Reason`, and `OccurredAt`.
|
||||||
- Store remote address in `remoteAddress` only after classification as PII.
|
- Store remote address in `remoteAddress` only after classification as PII.
|
||||||
|
|||||||
@@ -64,7 +64,7 @@ See [Unknowns Ranking Algorithm](./unknowns-ranking.md) for the complete formula
|
|||||||
|
|
||||||
## 6. Storage & CAS
|
## 6. Storage & CAS
|
||||||
|
|
||||||
- Primary store: append-only KV/graph in Mongo (collections `unknowns`, `unknown_metrics`).
|
- Primary store: append-only KV/graph in PostgreSQL (tables `unknowns`, `unknown_metrics`).
|
||||||
- Evidence blobs: CAS under `cas://unknowns/{sha256}` for large payloads (runtime traces, partial SBOMs).
|
- Evidence blobs: CAS under `cas://unknowns/{sha256}` for large payloads (runtime traces, partial SBOMs).
|
||||||
- Include analyzer fingerprint + schema version in each record for replay.
|
- Include analyzer fingerprint + schema version in each record for replay.
|
||||||
|
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ The Packs Registry stores, verifies, and serves Task Pack bundles across environ
|
|||||||
|
|
||||||
- **Service name:** `StellaOps.PacksRegistry`
|
- **Service name:** `StellaOps.PacksRegistry`
|
||||||
- **Interfaces:** REST/GraphQL API, OCI-compatible registry endpoints, event streams for mirroring.
|
- **Interfaces:** REST/GraphQL API, OCI-compatible registry endpoints, event streams for mirroring.
|
||||||
- **Data stores:** MongoDB (`packs`, `pack_versions`, `pack_provenance`), object storage (bundle blobs, signatures), timeline events.
|
- **Data stores:** PostgreSQL (`packs`, `pack_versions`, `pack_provenance` tables), object storage (bundle blobs, signatures), timeline events.
|
||||||
- **Dependencies:** Authority scopes (`packs.*`), Export Center (manifests), DevOps signing service, Notifications (optional).
|
- **Dependencies:** Authority scopes (`packs.*`), Export Center (manifests), DevOps signing service, Notifications (optional).
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -145,7 +145,7 @@ Extensions must be deterministic and derived from signed bundle data.
|
|||||||
|
|
||||||
## 9 · Operations
|
## 9 · Operations
|
||||||
|
|
||||||
- **Backups:** Daily snapshots of Mongo collections + object storage, retained for 30 days.
|
- **Backups:** Daily snapshots of PostgreSQL tables + object storage, retained for 30 days.
|
||||||
- **Retention:** Old versions retained indefinitely; mark as `deprecated` instead of deleting.
|
- **Retention:** Old versions retained indefinitely; mark as `deprecated` instead of deleting.
|
||||||
- **Maintenance:**
|
- **Maintenance:**
|
||||||
- Run `registry vacuum` weekly to prune orphaned blobs.
|
- Run `registry vacuum` weekly to prune orphaned blobs.
|
||||||
|
|||||||
Reference in New Issue
Block a user