This commit is contained in:
master
@@ -86,6 +86,7 @@ AUTH-TEN-49-001 | DOING (2025-11-02) | Implement service accounts & delegation t
|
||||
> 2025-11-02: Service account store + configuration wired, delegation quotas enforced, token persistence extended with `serviceAccountId`/`tokenKind`/`actorChain`, docs & samples refreshed, and new tests cover delegated issuance/persistence.
|
||||
> 2025-11-02: Updated bootstrap test fixtures to use AuthorityDelegation seed types and verified `/internal/service-accounts` endpoints respond as expected via targeted Authority tests.
|
||||
> 2025-11-02: Documented bootstrap admin API usage (`/internal/service-accounts/**`) and clarified that repeated seeding preserves Mongo `_id`/`createdAt` values to avoid immutable field errors.
|
||||
> 2025-11-03: Patched Authority test harness to seed enabled service-account records deterministically and restored `StellaOps.Authority.Tests` to green (covers `/internal/service-accounts` listing + revocation paths).
|
||||
AUTH-VULN-29-001 | DONE (2025-11-03) | Define Vuln Explorer scopes/roles (`vuln:view`, `vuln:investigate`, `vuln:operate`, `vuln:audit`) with ABAC attributes (env, owner, business_tier) and update discovery metadata/offline kit defaults. Dependencies: AUTH-POLICY-27-001. | Authority Core & Security Guild (src/Authority/StellaOps.Authority/TASKS.md)
|
||||
AUTH-VULN-29-002 | DONE (2025-11-03) | Enforce CSRF/anti-forgery tokens for workflow actions, sign attachment tokens, and record audit logs with ledger event hashes. Dependencies: AUTH-VULN-29-001, LEDGER-29-002. | Authority Core & Security Guild (src/Authority/StellaOps.Authority/TASKS.md)
|
||||
AUTH-VULN-29-003 | DOING (2025-11-03) | Update security docs/config samples for Vuln Explorer roles, ABAC policies, attachment signing, and ledger verification guidance. Dependencies: AUTH-VULN-29-001..002. | Authority Core & Docs Guild (src/Authority/StellaOps.Authority/TASKS.md)
|
||||
|
||||
423
docs/replay/DETERMINISTIC_REPLAY.md
Normal file
423
docs/replay/DETERMINISTIC_REPLAY.md
Normal file
@@ -0,0 +1,423 @@
|
||||
# Stella Ops — Deterministic Replay Specification
|
||||
|
||||
Version: 1.0
|
||||
Status: Draft / Internal Technical Reference
|
||||
Audience: Core developers, module maintainers, audit engineers.
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Deterministic Replay allows any completed Stella Ops scan to be **reproduced byte-for-byte** with full cryptographic validation.
|
||||
It guarantees that SBOMs, Findings, and VEX evaluations can be re-executed later to:
|
||||
|
||||
- prove historical compliance decisions,
|
||||
- attribute changes precisely to feeds, rules, or tools,
|
||||
- support dual-signing (FIPS + regional crypto),
|
||||
- and anchor cryptographic evidence in offline or public ledgers.
|
||||
|
||||
Replay requires that all inputs and environmental conditions are **captured, hashed, and sealed** at scan time.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture Overview
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Scanner.WebService] --> B[Replay Manifest]
|
||||
A --> C[InputBundle]
|
||||
A --> D[OutputBundle]
|
||||
B --> E[DSSE Envelope]
|
||||
C --> F[Feedser Snapshot Export]
|
||||
C --> G[Policy/Lattice Bundle]
|
||||
D --> H[DSSE Outputs (SBOM, Findings, VEX)]
|
||||
E --> I[MongoDB: replay_runs]
|
||||
C --> J[Blob Store: Input/Output Bundles]
|
||||
````
|
||||
|
||||
### Core Artifacts
|
||||
|
||||
| Artifact | Description | Format |
|
||||
| ------------------- | ------------------------------------------------------ | -------------------------- |
|
||||
| **Replay Manifest** | Immutable JSON describing all scan inputs and outputs. | JSON (canonicalized) |
|
||||
| **InputBundle** | Feeds, rules, policies, tool binaries (hashed). | `.tar.zst` |
|
||||
| **OutputBundle** | SBOM, Findings, VEX, logs. | `.tar.zst` |
|
||||
| **DSSE Envelope** | Signed metadata for each artifact. | JSON / JWS |
|
||||
| **Merkle Map** | Layer and feed chunk trees. | JSON (embedded or sidecar) |
|
||||
|
||||
---
|
||||
|
||||
## 3. Replay Manifest Schema (v1)
|
||||
|
||||
### 3.1 Top-level Layout
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"schemaVersion": "1.0",
|
||||
"scan": {
|
||||
"id": "uuid",
|
||||
"time": "2025-10-29T13:05:33Z",
|
||||
"mode": "record",
|
||||
"scannerVersion": "10.1.3",
|
||||
"cryptoProfile": "FIPS-140-3+GOST-R-34.10-2012"
|
||||
},
|
||||
"subject": {
|
||||
"ociDigest": "sha256:abcd...",
|
||||
"layers": [
|
||||
{ "layerDigest": "...", "merkleRoot": "...", "leafCount": 144 }
|
||||
]
|
||||
},
|
||||
"inputs": {
|
||||
"feeds": [
|
||||
{
|
||||
"name": "nvd",
|
||||
"snapshotHash": "sha256:...",
|
||||
"snapshotTime": "2025-10-29T12:00:00Z",
|
||||
"merkleRoot": "..."
|
||||
}
|
||||
],
|
||||
"rulesBundleHash": "sha256:...",
|
||||
"tools": [
|
||||
{ "name": "sbomer", "version": "10.1.3", "sha256": "..." },
|
||||
{ "name": "scanner", "version": "10.1.3", "sha256": "..." },
|
||||
{ "name": "vexer", "version": "10.1.3", "sha256": "..." }
|
||||
],
|
||||
"env": {
|
||||
"os": "linux",
|
||||
"arch": "x64",
|
||||
"locale": "en_US.UTF-8",
|
||||
"tz": "UTC",
|
||||
"seed": "H(scan.id||merkleRootAllLayers)",
|
||||
"flags": ["offline"]
|
||||
}
|
||||
},
|
||||
"policy": {
|
||||
"latticeHash": "sha256:...",
|
||||
"mutes": [
|
||||
{ "id": "MUTE-1234", "reason": "vendor ack", "approvedBy": "authority@example.com", "approvedAt": "2025-10-29T12:55Z" }
|
||||
],
|
||||
"trustProfile": "sha256:..."
|
||||
},
|
||||
"outputs": {
|
||||
"sbomHash": "sha256:...",
|
||||
"findingsHash": "sha256:...",
|
||||
"vexHash": "sha256:...",
|
||||
"logHash": "sha256:..."
|
||||
},
|
||||
"provenance": {
|
||||
"signer": "scanner.authority",
|
||||
"dsseEnvelopeHash": "sha256:...",
|
||||
"rekorEntry": "optional"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Deterministic Execution Rules
|
||||
|
||||
### 4.1 Environment Normalization
|
||||
|
||||
* **Clock:** frozen to `scan.time` unless a rule explicitly requires “now”.
|
||||
* **Random seed:** derived as `H(scan.id || MerkleRootAllLayers)`.
|
||||
* **Locale/TZ:** enforced per manifest; deviations cause validation error.
|
||||
* **Filesystem normalization:**
|
||||
|
||||
* Normalize perms to 0644/0755.
|
||||
* Path separators = `/`.
|
||||
* Newlines = LF.
|
||||
* JSON key order = lexical.
|
||||
|
||||
### 4.2 Concurrency & I/O
|
||||
|
||||
* File traversal: stable lexicographic order.
|
||||
* Parallel jobs: ordered reduction by subject path.
|
||||
* Temporary directories: ephemeral but deterministic hash seeds.
|
||||
|
||||
### 4.3 Feeds & Policies
|
||||
|
||||
* All network I/O disabled; feeds must be read from snapshot bundles.
|
||||
* Policies and suppressions must resolve by hash, not name.
|
||||
|
||||
---
|
||||
|
||||
## 5. DSSE and Signing
|
||||
|
||||
### 5.1 Envelope Structure
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"payloadType": "application/vnd.stella.replay.manifest+json",
|
||||
"payload": "<base64-encoded canonical JSON>",
|
||||
"signatures": [
|
||||
{ "keyid": "authority-root-fips", "sig": "..." },
|
||||
{ "keyid": "authority-root-gost", "sig": "..." }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 5.2 Verification Steps
|
||||
|
||||
1. Decode payload → verify canonical form.
|
||||
2. Verify each signature chain against RootPack (offline trust anchors).
|
||||
3. Recompute hash and compare to `dsseEnvelopeHash` in manifest.
|
||||
4. Optionally verify Rekor inclusion proof.
|
||||
|
||||
---
|
||||
|
||||
## 6. CLI Interface
|
||||
|
||||
### 6.1 Recording a Scan
|
||||
|
||||
```bash
|
||||
stella scan image:tag --record ./out/
|
||||
```
|
||||
|
||||
Produces:
|
||||
|
||||
```
|
||||
out/
|
||||
├─ manifest.json
|
||||
├─ manifest.dsse.json
|
||||
├─ inputbundle.tar.zst
|
||||
├─ outputbundle.tar.zst
|
||||
└─ signatures/
|
||||
```
|
||||
|
||||
### 6.2 Verifying
|
||||
|
||||
```bash
|
||||
stella verify manifest.json
|
||||
```
|
||||
|
||||
* Checks all hashes and DSSE envelopes.
|
||||
* Prints summary:
|
||||
|
||||
```
|
||||
✅ Verified: SBOM, Findings, VEX, Tools, Feeds, Policy
|
||||
```
|
||||
|
||||
### 6.3 Replaying
|
||||
|
||||
```bash
|
||||
stella replay manifest.json --strict
|
||||
stella replay manifest.json --what-if --vary=feeds
|
||||
```
|
||||
|
||||
* `--strict`: all inputs locked; identical result expected.
|
||||
* `--what-if`: varies only specified dimension(s).
|
||||
|
||||
### 6.4 Diffing
|
||||
|
||||
```bash
|
||||
stella diff manifestA.json manifestB.json
|
||||
```
|
||||
|
||||
Shows field-level differences (feed snapshot, tool, or policy hash).
|
||||
|
||||
---
|
||||
|
||||
## 7. MongoDB Schema
|
||||
|
||||
### 7.1 `replay_runs`
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"_id": "uuid",
|
||||
"manifestHash": "sha256:...",
|
||||
"status": "verified|failed|replayed",
|
||||
"createdAt": "...",
|
||||
"updatedAt": "...",
|
||||
"signatures": [{ "profile": "FIPS", "verified": true }],
|
||||
"outputs": {
|
||||
"sbom": "sha256:...",
|
||||
"findings": "sha256:..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 7.2 `bundles`
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"_id": "sha256:...",
|
||||
"type": "input|output|rootpack",
|
||||
"size": 4123123,
|
||||
"location": "/var/lib/stella/bundles/<sha>.tar.zst"
|
||||
}
|
||||
```
|
||||
|
||||
### 7.3 `subjects`
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"ociDigest": "sha256:abcd...",
|
||||
"layers": [
|
||||
{ "layerDigest": "...", "merkleRoot": "...", "leafCount": 120 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Layer Merkle Implementation
|
||||
|
||||
### 8.1 Algorithm
|
||||
|
||||
```csharp
|
||||
static string ComputeMerkleRoot(string layerTarPath)
|
||||
{
|
||||
const int ChunkSize = 4 * 1024 * 1024;
|
||||
var hashes = new List<byte[]>();
|
||||
using var fs = File.OpenRead(layerTarPath);
|
||||
var buffer = new byte[ChunkSize];
|
||||
int read;
|
||||
using var sha = SHA256.Create();
|
||||
while ((read = fs.Read(buffer, 0, buffer.Length)) > 0)
|
||||
hashes.Add(sha.ComputeHash(buffer, 0, read));
|
||||
while (hashes.Count > 1)
|
||||
hashes = hashes
|
||||
.Select((h, i) => (h, i))
|
||||
.GroupBy(x => x.i / 2)
|
||||
.Select(g => sha.ComputeHash(g.SelectMany(x => x.h).ToArray()))
|
||||
.ToList();
|
||||
return Convert.ToHexString(hashes.Single());
|
||||
}
|
||||
```
|
||||
|
||||
### 8.2 Stored Values
|
||||
|
||||
```json
|
||||
{
|
||||
"layerDigest": "sha256:...",
|
||||
"merkleRoot": "b81f...",
|
||||
"leafCount": 240,
|
||||
"leavesHash": "sha256:..."
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Replay Engine Implementation Notes (.NET 10)
|
||||
|
||||
### 9.1 Manifest Parsing
|
||||
|
||||
Use `System.Text.Json` with deterministic ordering:
|
||||
|
||||
```csharp
|
||||
var options = new JsonSerializerOptions {
|
||||
WriteIndented = false,
|
||||
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
||||
TypeInfoResolverChain = { new OrderedResolver() }
|
||||
};
|
||||
```
|
||||
|
||||
### 9.2 Stable Output
|
||||
|
||||
Normalize SBOM/Findings/VEX JSON:
|
||||
|
||||
```csharp
|
||||
string Canonicalize(string json) =>
|
||||
JsonSerializer.Serialize(
|
||||
JsonSerializer.Deserialize<JsonDocument>(json),
|
||||
options);
|
||||
```
|
||||
|
||||
### 9.3 Verification Flow
|
||||
|
||||
```csharp
|
||||
var manifest = Manifest.Load("manifest.json");
|
||||
VerifySignatures(manifest);
|
||||
VerifyHashes(manifest);
|
||||
if (mode == Strict) RunPipeline(manifest);
|
||||
else RunPipelineWithVariation(manifest, vary);
|
||||
```
|
||||
|
||||
### 9.4 Failure Modes
|
||||
|
||||
| Condition | Action |
|
||||
| -------------------------------- | ----------------------------- |
|
||||
| Missing snapshot or bundle | Error: `InputBundleMissing` |
|
||||
| Feed hash mismatch | Error: `FeedSnapshotDrift` |
|
||||
| Tool binary hash mismatch | Reject replay |
|
||||
| Output hash drift in strict mode | Mark as failed, emit diff log |
|
||||
| Invalid signature | Reject manifest |
|
||||
|
||||
---
|
||||
|
||||
## 10. Crypto Profiles and RootPack
|
||||
|
||||
### 10.1 Example Profiles
|
||||
|
||||
| Profile | Algorithms | Notes |
|
||||
| -------------- | ------------------------------------- | ----------------------- |
|
||||
| **FIPS-140-3** | ECDSA-P256 / SHA-256 / AES-GCM | Default for US/EU |
|
||||
| **GOST** | GOST R 34.10-2012 / GOST R 34.11-2012 | Russia |
|
||||
| **SM** | SM2 / SM3 / SM4 | China |
|
||||
| **eIDAS** | RSA-PSS / SHA-256 | EU qualified signatures |
|
||||
|
||||
### 10.2 Dual-Signing Example
|
||||
|
||||
```bash
|
||||
stella sign manifest.json --profiles=FIPS,GOST
|
||||
```
|
||||
|
||||
Produces:
|
||||
|
||||
```
|
||||
signatures/
|
||||
├─ manifest.dsse.fips.json
|
||||
└─ manifest.dsse.gost.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. Test Strategy
|
||||
|
||||
| Test | Description | Expected Result |
|
||||
| ---------------------- | ------------------------------------ | --------------------------- |
|
||||
| **Golden Replay** | Repeat identical scan → same outputs | ✅ identical hashes |
|
||||
| **Feed Drift Test** | Replay with updated feeds | Only `inputs.feeds` changes |
|
||||
| **Tool Upgrade Test** | Replay with new scanner version | Reject or diff by `tools` |
|
||||
| **Policy Change Test** | Different lattice/mutes | Diff by `policy` section |
|
||||
| **Cross-Arch Test** | x64 vs arm64 | Identical outputs |
|
||||
| **Corrupted Bundle** | Tamper bundle | Verification fails |
|
||||
|
||||
---
|
||||
|
||||
## 12. Example Verification Output
|
||||
|
||||
```
|
||||
$ stella verify manifest.json
|
||||
|
||||
[✓] Manifest integrity: OK
|
||||
[✓] DSSE signatures (FIPS,GOST): OK
|
||||
[✓] Feeds snapshot hash: OK
|
||||
[✓] Policy + mutes hash: OK
|
||||
[✓] Toolchain hash: OK
|
||||
[✓] SBOM/VEX outputs: OK
|
||||
|
||||
Result: VERIFIED
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 13. Future Extensions
|
||||
|
||||
* Support **SPDX 3.0.1** alongside CycloneDX 1.6.
|
||||
* Add **per-file Merkle proofs** for local scans.
|
||||
* Ledger anchoring (Rekor, distributed Proof-Market).
|
||||
* Post-quantum signatures (Dilithium/Falcon).
|
||||
* Replay orchestration API (`/api/replay/:id`).
|
||||
|
||||
---
|
||||
|
||||
## 14. Summary
|
||||
|
||||
Deterministic Replay freezes every element of a scan:
|
||||
|
||||
> *image → feeds → policy → toolchain → environment → outputs → signatures.*
|
||||
|
||||
By enforcing canonical input/output states and verifiable cryptographic bindings, Stella Ops achieves **regulatory-grade replayability**, **regional crypto compliance**, and **immutable provenance** across all scans.
|
||||
|
||||
---
|
||||
113
docs/replay/DEVS_GUIDE_REPLAY.md
Normal file
113
docs/replay/DEVS_GUIDE_REPLAY.md
Normal file
@@ -0,0 +1,113 @@
|
||||
# Stella Ops — Developer Guide: Deterministic Replay
|
||||
|
||||
## Purpose
|
||||
Deterministic Replay ensures any past scan can be re-executed byte-for-byte, producing identical SBOM, Findings, and VEX results, cryptographically verifiable for audits or compliance.
|
||||
|
||||
Replay is the foundation for:
|
||||
- **Audit proofs** (exact past state reproduction)
|
||||
- **Diff analysis** (feeds, policies, tool versions)
|
||||
- **Cross-region verification** (same outputs on different hosts)
|
||||
- **Long-term cryptographic trust** (re-sign with new crypto profiles)
|
||||
|
||||
---
|
||||
|
||||
## Core Concepts
|
||||
|
||||
| Term | Description |
|
||||
|------|--------------|
|
||||
| **Replay Manifest** | Immutable JSON describing all inputs, tools, env, and outputs of a scan. |
|
||||
| **InputBundle** | Snapshot of feeds, rules, policies, and toolchain binaries used. |
|
||||
| **OutputBundle** | SBOM, Findings, VEX, and logs from a completed scan. |
|
||||
| **Layer Merkle** | Per-layer hash tree for precise deduplication and drift detection. |
|
||||
| **DSSE Envelope** | Digital signature wrapper for each attestation (SBOM, Findings, Manifest, etc.). |
|
||||
|
||||
---
|
||||
|
||||
## What to Freeze
|
||||
|
||||
| Category | Example Contents | Required in Manifest |
|
||||
|-----------|------------------|----------------------|
|
||||
| **Subject** | OCI image digest, per-layer Merkle roots | ✅ |
|
||||
| **Outputs** | SBOM, Findings, VEX, logs (content hashes) | ✅ |
|
||||
| **Toolchain** | Sbomer, Scanner, Vexer binaries + versions + SHA256 | ✅ |
|
||||
| **Feeds/VEX sources** | Full or pruned snapshot with Merkle proofs | ✅ |
|
||||
| **Policy Bundle** | Lattice rules, mutes, trust profiles, thresholds | ✅ |
|
||||
| **Environment** | OS, arch, locale, TZ, deterministic seed, runtime flags | ✅ |
|
||||
| **Crypto Profile** | Algorithm suites (FIPS, GOST, SM, eIDAS) | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## Replay Modes
|
||||
|
||||
| Mode | Purpose | Input Variation | Expected Output |
|
||||
|------|----------|-----------------|-----------------|
|
||||
| **Strict Replay** | Audit proof | None | Bit-for-bit identical |
|
||||
| **What-If Replay** | Change impact analysis | One dimension (feeds/tools/policy) | Deterministic diff |
|
||||
|
||||
Example:
|
||||
```
|
||||
|
||||
stella replay manifest.json --strict
|
||||
stella replay manifest.json --what-if --vary=feeds
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Developer Responsibilities
|
||||
|
||||
| Module | Role |
|
||||
|---------|------|
|
||||
| **Scanner.WebService** | Capture full input set and produce Replay Manifest + DSSE sigs. |
|
||||
| **Sbomer** | Generate deterministic SBOM; normalize ordering and JSON formatting. |
|
||||
| **Vexer/Excititor** | Apply lattice and mutes from policy bundle; record gating logic. |
|
||||
| **Feedser/Concelier** | Freeze and export feed snapshots or Merkle proofs. |
|
||||
| **Authority** | Manage signer keys and crypto profiles; issue DSSE envelopes. |
|
||||
| **CLI** | Provide `scan --record`, `replay`, `verify`, `diff` commands. |
|
||||
|
||||
---
|
||||
|
||||
## Workflow
|
||||
|
||||
1. `stella scan image:tag --record out/`
|
||||
- Generates Replay Manifest, InputBundle, OutputBundle, DSSE sigs.
|
||||
2. `stella verify manifest.json`
|
||||
- Validates hashes, signatures, and completeness.
|
||||
3. `stella replay manifest.json --strict`
|
||||
- Re-executes in sealed mode; expect byte-identical results.
|
||||
4. `stella replay manifest.json --what-if --vary=feeds`
|
||||
- Runs with new feeds; diff is attributed to feeds only.
|
||||
5. `stella diff manifestA manifestB`
|
||||
- Attribute differences by hash comparison.
|
||||
|
||||
---
|
||||
|
||||
## Storage
|
||||
|
||||
- **Mongo collections**
|
||||
- `replay_runs`: manifest + DSSE envelopes + status
|
||||
- `bundles`: content-addressed (input/output/rootpack)
|
||||
- `subjects`: OCI digests, Merkle roots per layer
|
||||
- **File store**
|
||||
- Bundles stored as `<sha256>.tar.zst`
|
||||
|
||||
---
|
||||
|
||||
## Developer Checklist
|
||||
|
||||
- [ ] All inputs (feeds, policies, tools, env) hashed and recorded.
|
||||
- [ ] JSON normalization: key order, number format, newline mode.
|
||||
- [ ] Random seed = `H(scan.id || MerkleRootAllLayers)`.
|
||||
- [ ] Clock fixed to `scan.time` unless policy requires “now”.
|
||||
- [ ] DSSE multi-sig supported (FIPS + regional).
|
||||
- [ ] Manifest signed + optionally anchored to Rekor ledger.
|
||||
- [ ] Replay comparison mode tested across x64/arm64.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
See also:
|
||||
- `DETERMINISTIC_REPLAY.md` — detailed manifest schema & CLI examples.
|
||||
- `../docs/CRYPTO_SOVEREIGN_READY.md` — RootPack and dual-signature handling.
|
||||
|
||||
---
|
||||
Reference in New Issue
Block a user