feat(graph-api): Add schema review notes for upcoming Graph API changes
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled

feat(sbomservice): Add placeholder for SHA256SUMS in LNM v1 fixtures

docs(devportal): Create README for SDK archives in public directory

build(devportal): Implement offline bundle build script

test(devportal): Add link checker script for validating links in documentation

test(devportal): Create performance check script for dist folder size

test(devportal): Implement accessibility check script using Playwright and Axe

docs(devportal): Add SDK quickstart guide with examples for Node.js, Python, and cURL

feat(excititor): Implement MongoDB storage for airgap import records

test(findings): Add unit tests for export filters hash determinism

feat(findings): Define attestation contracts for ledger web service

feat(graph): Add MongoDB options and service collection extensions for graph indexing

test(graph): Implement integration tests for MongoDB provider and service collection extensions

feat(zastava): Define configuration options for Zastava surface secrets

build(tests): Create script to run Concelier linkset tests with TRX output
This commit is contained in:
StellaOps Bot
2025-11-22 19:22:30 +02:00
parent ca09400069
commit 48702191be
76 changed files with 3878 additions and 1081 deletions

View File

@@ -99,12 +99,38 @@
## 4. Backups & restores
### 4.1 Backup quickstart (PostgreSQL)
| Task | Command / guidance |
| --- | --- |
| Online backup | `pg_dump -Fc --dbname="$LEDGER_DB" --file ledger-$(date -u +%Y%m%d).dump` (run hourly for WAL + daily full dumps). |
| Point-in-time recovery | Enable WAL archiving; document target `recovery_target_time`. |
| Projection rebuild | After restore, run `dotnet run --project tools/LedgerReplayHarness -- --connection "$LEDGER_DB" --tenant all` to regenerate projections and verify hashes. |
| Evidence bundles | Store Merkle root anchors + replay DSSE bundles alongside DB backups for audit parity. |
| Full dump | `pg_dump -Fc --dbname="$LEDGER_DB" --file ledger-$(date -u +%Y%m%d).dump` (run daily). |
| WAL archiving | `archive_mode=on`, `archive_command='test ! -f /wal/%f && cp %p /wal/%f'`; retain ≥7 days or per policy. |
| Integrity check | `pg_restore -l ledger-YYYYMMDD.dump | head` (validate readable) + verify `ledger_merkle_roots` count matches production before pruning. |
### 4.2 Restore + replay
1. Restore database (full + WAL).
```bash
pg_restore -C -d postgres ledger-YYYYMMDD.dump
```
2. Run projection replay/determinism harness to repopulate projections and validate hashes:
```bash
dotnet run --project src/Findings/tools/LedgerReplayHarness \
-- --connection "$LEDGER_DB" \
--tenant all \
--maxParallel 8 \
--report out/harness/restore-report.json
```
3. Compare Merkle roots: query `select root_hash from ledger_merkle_roots order by anchored_at desc limit 5;` and ensure harness report `merkleRoot` matches latest root.
4. Recreate indexes/materialized views if disabled during restore (see `migrations/` for schema reference).
### 4.3 Evidence & audit artefacts
- Store the following with each backup set:
- `ledger_merkle_roots` CSV export.
- Replay harness signed report (`harness-report.json` + `.sig`).
- Latest Grafana dashboard JSON and alert rules used during the period.
- Keep anchor references (`anchor_reference`) when external anchoring is enabled; include external proof bundle if present.
## 5. Offline / air-gapped workflow
@@ -115,6 +141,8 @@
- Package ledger service binaries + migrations using `ops/offline-kit/build_offline_kit.py --include ledger`.
- Document sealed-mode restrictions: disable outbound attachments unless egress policy allows Evidence Locker endpoints; set `LEDGER__ATTACHMENTS__ALLOWEGRESS=false`.
**Path placeholder (waiting on DevOps):** Helm/Compose/offline-kit overlay directories are pending centralisation under `ops/deployment`/`ops/offline-kit`. Until paths are assigned, keep environment-specific overlays local to `docs/modules/findings-ledger/deployment.md` examples and avoid committing manifests outside this module.
## 6. Post-deploy checklist
- [ ] Health + metrics endpoints respond.

View File

@@ -110,6 +110,240 @@ paths:
application/json:
schema:
$ref: '#/components/schemas/FindingProjectionPage'
/v1/ledger/export/findings:
get:
summary: Export findings in deterministic order
operationId: exportFindings
tags: [export]
parameters:
- $ref: '#/components/parameters/TenantId'
- $ref: '#/components/parameters/Shape'
- $ref: '#/components/parameters/SinceSequence'
- $ref: '#/components/parameters/UntilSequence'
- $ref: '#/components/parameters/SinceObservedAt'
- $ref: '#/components/parameters/UntilObservedAt'
- name: finding_status
in: query
schema: { type: string }
- name: severity
in: query
schema: { type: number }
- name: risk_profile_version
in: query
schema: { type: string }
- $ref: '#/components/parameters/PageSize'
- $ref: '#/components/parameters/PageToken'
responses:
'200':
description: Paged findings export
headers:
X-Stella-Next-Page-Token:
schema: { type: string }
X-Stella-Result-Count:
schema: { type: integer }
content:
application/json:
schema:
$ref: '#/components/schemas/FindingExportPage'
application/x-ndjson:
schema:
type: string
description: NDJSON stream of FindingExportItem
/v1/ledger/export/vex:
get:
summary: Export VEX statements
operationId: exportVex
tags: [export]
parameters:
- $ref: '#/components/parameters/TenantId'
- $ref: '#/components/parameters/Shape'
- $ref: '#/components/parameters/SinceSequence'
- $ref: '#/components/parameters/UntilSequence'
- $ref: '#/components/parameters/SinceObservedAt'
- $ref: '#/components/parameters/UntilObservedAt'
- name: product_id
in: query
schema: { type: string }
- name: advisory_id
in: query
schema: { type: string }
- name: status
in: query
schema: { type: string }
- name: statement_type
in: query
schema: { type: string }
- $ref: '#/components/parameters/PageSize'
- $ref: '#/components/parameters/PageToken'
responses:
'200':
description: Paged VEX export
headers:
X-Stella-Next-Page-Token:
schema: { type: string }
X-Stella-Result-Count:
schema: { type: integer }
content:
application/json:
schema:
$ref: '#/components/schemas/VexExportPage'
application/x-ndjson:
schema:
type: string
description: NDJSON stream of VexExportItem
/v1/ledger/export/advisories:
get:
summary: Export advisories
operationId: exportAdvisories
tags: [export]
parameters:
- $ref: '#/components/parameters/TenantId'
- $ref: '#/components/parameters/Shape'
- $ref: '#/components/parameters/SinceSequence'
- $ref: '#/components/parameters/UntilSequence'
- $ref: '#/components/parameters/SinceObservedAt'
- $ref: '#/components/parameters/UntilObservedAt'
- name: severity
in: query
schema: { type: string }
- name: source
in: query
schema: { type: string }
- name: cwe_id
in: query
schema: { type: string }
- name: kev
in: query
schema: { type: boolean }
- name: cvss_version
in: query
schema: { type: string }
- name: cvss_score_min
in: query
schema: { type: number }
- name: cvss_score_max
in: query
schema: { type: number }
- $ref: '#/components/parameters/PageSize'
- $ref: '#/components/parameters/PageToken'
responses:
'200':
description: Paged advisory export
headers:
X-Stella-Next-Page-Token:
schema: { type: string }
X-Stella-Result-Count:
schema: { type: integer }
content:
application/json:
schema:
$ref: '#/components/schemas/AdvisoryExportPage'
application/x-ndjson:
schema:
type: string
description: NDJSON stream of AdvisoryExportItem
/v1/ledger/export/sboms:
get:
summary: Export SBOMs
operationId: exportSboms
tags: [export]
parameters:
- $ref: '#/components/parameters/TenantId'
- $ref: '#/components/parameters/Shape'
- $ref: '#/components/parameters/SinceSequence'
- $ref: '#/components/parameters/UntilSequence'
- $ref: '#/components/parameters/SinceObservedAt'
- $ref: '#/components/parameters/UntilObservedAt'
- name: subject_digest
in: query
schema: { type: string }
- name: sbom_format
in: query
schema: { type: string }
- name: component_purl
in: query
schema: { type: string }
- name: contains_native
in: query
schema: { type: boolean }
- name: slsa_build_type
in: query
schema: { type: string }
- $ref: '#/components/parameters/PageSize'
- $ref: '#/components/parameters/PageToken'
responses:
'200':
description: Paged SBOM export
headers:
X-Stella-Next-Page-Token:
schema: { type: string }
X-Stella-Result-Count:
schema: { type: integer }
content:
application/json:
schema:
$ref: '#/components/schemas/SbomExportPage'
application/x-ndjson:
schema:
type: string
description: NDJSON stream of SbomExportItem
/v1/ledger/attestations:
get:
summary: List attestation verifications
operationId: listLedgerAttestations
tags: [attestation]
parameters:
- $ref: '#/components/parameters/TenantId'
- name: artifactId
in: query
schema: { type: string }
- name: findingId
in: query
schema: { type: string }
- name: attestationId
in: query
schema: { type: string }
- name: status
in: query
schema:
type: string
enum: [verified, failed, unknown]
- name: sinceRecordedAt
in: query
schema: { type: string, format: date-time }
- name: untilRecordedAt
in: query
schema: { type: string, format: date-time }
- $ref: '#/components/parameters/AttestationLimit'
- $ref: '#/components/parameters/PageToken'
responses:
'200':
description: Paged attestation verifications
headers:
X-Stella-Next-Page-Token:
schema: { type: string }
X-Stella-Result-Count:
schema: { type: integer }
content:
application/json:
schema:
$ref: '#/components/schemas/AttestationExportPage'
application/x-ndjson:
schema:
type: string
description: NDJSON stream of AttestationExportItem
/.well-known/openapi:
get:
summary: Serve Findings Ledger OpenAPI document
operationId: getOpenApi
tags: [metadata]
responses:
'200':
description: OpenAPI YAML document
content:
application/yaml:
schema:
type: string
components:
securitySchemes:
bearerAuth:
@@ -125,6 +359,56 @@ components:
required: true
schema:
type: string
Shape:
name: shape
in: query
required: true
schema:
type: string
enum: [canonical, compact]
SinceSequence:
name: since_sequence
in: query
schema:
type: integer
minimum: 0
UntilSequence:
name: until_sequence
in: query
schema:
type: integer
minimum: 0
SinceObservedAt:
name: since_observed_at
in: query
schema:
type: string
format: date-time
UntilObservedAt:
name: until_observed_at
in: query
schema:
type: string
format: date-time
PageSize:
name: page_size
in: query
schema:
type: integer
default: 500
maximum: 5000
PageToken:
name: page_token
in: query
schema:
type: string
AttestationLimit:
name: limit
in: query
schema:
type: integer
default: 200
maximum: 1000
schemas:
LedgerEvent:
type: object
@@ -241,6 +525,16 @@ components:
severity:
type: number
format: double
riskScore:
type: number
format: double
riskSeverity:
type: string
riskProfileVersion:
type: string
riskExplanationId:
type: string
format: uuid
labels:
type: object
additionalProperties: true
@@ -262,3 +556,116 @@ components:
$ref: '#/components/schemas/FindingProjection'
nextCursor:
type: string
ExportProvenance:
type: object
properties:
policyVersion: { type: string }
cycleHash: { type: string }
ledgerEventHash: { type: string }
FindingExportItem:
type: object
properties:
eventSequence: { type: integer }
observedAt: { type: string, format: date-time }
findingId: { type: string }
policyVersion: { type: string }
status: { type: string }
severity: { type: number, format: double }
cycleHash: { type: string }
evidenceBundleRef: { type: string }
provenance:
$ref: '#/components/schemas/ExportProvenance'
labels:
type: object
additionalProperties: true
VexExportItem:
type: object
properties:
eventSequence: { type: integer }
observedAt: { type: string, format: date-time }
vexStatementId: { type: string }
productId: { type: string }
status: { type: string }
statementType: { type: string }
knownExploited: { type: boolean }
cycleHash: { type: string }
provenance:
$ref: '#/components/schemas/ExportProvenance'
AdvisoryExportItem:
type: object
properties:
eventSequence: { type: integer }
published: { type: string, format: date-time }
advisoryId: { type: string }
source: { type: string }
title: { type: string }
severity: { type: string }
cvssScore: { type: number, format: double }
cvssVector: { type: string }
kev: { type: boolean }
cycleHash: { type: string }
provenance:
$ref: '#/components/schemas/ExportProvenance'
SbomExportItem:
type: object
properties:
eventSequence: { type: integer }
createdAt: { type: string, format: date-time }
sbomId: { type: string }
subjectDigest: { type: string }
sbomFormat: { type: string }
componentsCount: { type: integer }
hasVulnerabilities: { type: boolean }
cycleHash: { type: string }
provenance:
$ref: '#/components/schemas/ExportProvenance'
FindingExportPage:
type: object
properties:
items:
type: array
items: { $ref: '#/components/schemas/FindingExportItem' }
nextPageToken: { type: string }
VexExportPage:
type: object
properties:
items:
type: array
items: { $ref: '#/components/schemas/VexExportItem' }
nextPageToken: { type: string }
AdvisoryExportPage:
type: object
properties:
items:
type: array
items: { $ref: '#/components/schemas/AdvisoryExportItem' }
nextPageToken: { type: string }
SbomExportPage:
type: object
properties:
items:
type: array
items: { $ref: '#/components/schemas/SbomExportItem' }
nextPageToken: { type: string }
AttestationExportItem:
type: object
properties:
attestationId: { type: string }
artifactId: { type: string }
findingId: { type: string }
verificationStatus: { type: string }
verificationTime: { type: string, format: date-time }
dsseDigest: { type: string }
rekorEntryId: { type: string }
evidenceBundleRef: { type: string }
ledgerEventId: { type: string, format: uuid }
recordedAt: { type: string, format: date-time }
merkleLeafHash: { type: string }
rootHash: { type: string }
AttestationExportPage:
type: object
properties:
items:
type: array
items: { $ref: '#/components/schemas/AttestationExportItem' }
nextPageToken: { type: string }

View File

@@ -241,6 +241,30 @@ Checkpoint store for the projection background worker. Ensures idempotent replay
Seed row inserted on migration ensures catch-up from epoch (`1970-01-01T00:00:00Z` with empty GUID).
### 4.5 `ledger_attestations`
Deterministic view of DSSE verification results used by `/v1/ledger/attestations`. Rows are written by the provenance/verification pipeline and keyed per tenant.
| Column | Type | Description |
|--------|------|-------------|
| `tenant_id` | `text` | Partition key. |
| `attestation_id` | `uuid` | Primary key within tenant. |
| `artifact_id` | `text` | OCI digest or SBOM identifier verified. |
| `finding_id` | `text` | Optional finding linkage. |
| `verification_status` | `text` | `verified`, `failed`, or `unknown`. |
| `verification_time` | `timestamptz` | When verification completed. |
| `dsse_digest` | `text` | Lower-case SHA-256 of DSSE envelope. |
| `rekor_entry_id` | `text` | Optional transparency log UUID. |
| `evidence_bundle_ref` | `text` | Optional evidence bundle reference. |
| `ledger_event_id` | `uuid` | Ledger event that linked the attestation. |
| `recorded_at` | `timestamptz` | Ingestion timestamp used for paging. |
| `merkle_leaf_hash` | `text` | Leaf hash for anchoring proofs. |
| `root_hash` | `text` | Anchor root hash. |
| `cycle_hash` | `text` | Projection cycle hash for determinism. |
| `projection_version` | `text` | Projection version identifier. |
Ordering and pagination: `ORDER BY recorded_at ASC, attestation_id ASC` with cursor token `{recordedAt, attestationId, filtersHash}`. Indexes: PK `(tenant_id, attestation_id)`, paging index `(tenant_id, recorded_at, attestation_id)`, lookups on `(tenant_id, artifact_id, recorded_at DESC)` and `(tenant_id, verification_status, recorded_at DESC)`.
## 5. Hashing & verification
1. Canonical serialize the envelope (§2.3).

View File

@@ -5,7 +5,7 @@ Graph Indexer + Graph API build the tenant-scoped knowledge graph that powers bl
## Scope & responsibilities
- Ingest SBOM snapshots, advisory/VEX events, policy overlays, and runtime signals to maintain a first-party graph representation with deterministic node/edge identities.
- Serve APIs and saved-query tooling for impact analysis, dependency traversal, diffing, and policy/VEX overlays with explainable provenance.
- Supply Graph Explorer UI/CLI experiences, plus Offline Kit exports (`nodes.jsonl`, `edges.jsonl`, `overlays/`) with DSSE manifests for air-gapped replay.
- Supply Graph Explorer UI/CLI experiences, plus Offline Kit exports (`nodes.jsonl`, `edges.jsonl`, `overlays/`) with DSSE manifests for air-gapped replay. Analytics overlays are emitted as NDJSON (`overlays/clusters.ndjson`, `overlays/centrality.ndjson`) with deterministic ordering; Mongo-backed providers support production wiring.
- Maintain the [Graph Index Canonical Schema](schema.md) and coordinate query/overlay lifecycle with Scheduler, Policy Engine, Vulnerability Explorer, and Export Center.
## Architecture snapshot (Sprint 30 groundwork)

View File

@@ -4,6 +4,7 @@
- Helm/Compose should expose two timers for analytics: `GRAPH_ANALYTICS_CLUSTER_INTERVAL` and `GRAPH_ANALYTICS_CENTRALITY_INTERVAL` (ISO-8601 duration, default 5m). Map to `GraphAnalyticsOptions`.
- Change-stream/backfill worker toggles via `GRAPH_CHANGE_POLL_INTERVAL`, `GRAPH_BACKFILL_INTERVAL`, `GRAPH_CHANGE_MAX_RETRIES`, `GRAPH_CHANGE_RETRY_BACKOFF`.
- Mongo bindings (optional): `GRAPH_CHANGE_COLLECTION`, `GRAPH_CHANGE_SEQUENCE_FIELD`, `GRAPH_CHANGE_NODE_FIELD`, `GRAPH_CHANGE_EDGE_FIELD`, `GRAPH_CHANGE_IDEMPOTENCY_COLLECTION`, `GRAPH_ANALYTICS_SNAPSHOT_COLLECTION`, `GRAPH_ANALYTICS_PROGRESS_COLLECTION`.
- Mongo connection: `STELLAOPS_GRAPH_MONGO_CONNECTION` and `STELLAOPS_GRAPH_MONGO_DB` feed `AddGraphMongoDatabase` for clients/services.
- New Mongo collections:
- `graph_cluster_overlays` — cluster assignments (`tenant`, `snapshot_id`, `node_id`, `cluster_id`, `generated_at`).
- `graph_centrality_overlays` — degree + betweenness approximations per node.

View File

@@ -0,0 +1,28 @@
# Graph API schema review notes (planned)
Date: 2025-11-24 (target)
Scope: Review OpenAPI/JSON schema for search/query/paths/diff/export, tiles, budgets, and overlays alignment (GRAPH-API-28-001).
## Attendees
- Graph API Guild: TBD
- Policy Engine Guild: TBD
- QA Guild (observer): TBD
## Pre-reads
- `docs/api/graph-gateway-spec-draft.yaml`
- `docs/modules/graph/prep/2025-11-22-graph-api-schema-outline.md`
- Policy overlay contract references: `POLICY-ENGINE-30-001..003`
## Agenda
- Validate tile envelope shape and budget semantics.
- Confirm overlay payload fields and versioning handshake with Policy Engine.
- Decide DSL vs structured filter scope for `/graph/query` v1.
- Agree on export manifest shape and size caps for PNG/SVG.
## Decisions
- TODO (capture during review)
## Open items / follow-ups
- TODO
## Outcomes snapshot
- TODO (link to sprint Execution Log once review completes)

View File

@@ -0,0 +1 @@
# Pending fixture drop — replace with real SHA256 hashes when LNM v1 fixtures are published.

View File

@@ -25,7 +25,22 @@ Document a repeatable AirGap parity review for `/sbom/paths`, `/sbom/versions`,
- Minutes + decisions appended to this file (Execution Notes section) with timestamps and owners.
- Metrics table with p50/p95/p99 latency, error rate, and cache hit ratio.
- Actions list with owners and due dates; blockers mirrored to sprint 0140/0142 Decisions & Risks.
- Fixture hash list appended (from `SHA256SUMS`) with date and signer.
## Data capture templates
### Metrics
| Metric | p50 | p95 | p99 | Error rate | Notes |
| --- | --- | --- | --- | --- | --- |
| `/sbom/paths` latency (ms) | | | | | |
| `/sbom/versions` latency (ms) | | | | | |
| Event ingest → emit (ms) | | | | | |
| Cache hit ratio | | | | | |
### Decisions & follow-ups
| Decision / Action | Owner | Due | Status | Notes |
| --- | --- | --- | --- | --- |
| | | | | |
## Execution Notes
- 2025-11-22: Template published; awaiting fixtures and review scheduling.