docs consolidation
This commit is contained in:
StellaOps Bot
@@ -14,7 +14,8 @@ Proof chains in StellaOps consist of cryptographically-linked attestations:
|
||||
1. **Evidence statements** - Raw vulnerability findings
|
||||
2. **Reasoning statements** - Policy evaluation traces
|
||||
3. **VEX verdict statements** - Final vulnerability status determinations
|
||||
4. **Proof spine** - Merkle tree aggregating all components
|
||||
4. **Graph root statements** - Merkle root commitments to graph analysis results
|
||||
5. **Proof spine** - Merkle tree aggregating all components
|
||||
|
||||
In online mode, proof chains include Rekor inclusion proofs for transparency. In air-gap mode, verification proceeds without Rekor but maintains cryptographic integrity.
|
||||
|
||||
@@ -244,6 +245,174 @@ stellaops proof verify-batch \
|
||||
|
||||
---
|
||||
|
||||
## Graph Root Attestation Verification (Offline)
|
||||
|
||||
Graph root attestations provide tamper-evident commitment to graph analysis results. In air-gap mode, these attestations can be verified without network access.
|
||||
|
||||
### Verify Graph Root Attestation
|
||||
|
||||
```bash
|
||||
# Verify a single graph root attestation
|
||||
stellaops graph-root verify --offline \
|
||||
--envelope graph-root.dsse \
|
||||
--anchor-file trust-anchors.json
|
||||
|
||||
# Expected output:
|
||||
# Graph Root Verification
|
||||
# ═══════════════════════
|
||||
# ✓ DSSE signature verified
|
||||
# ✓ Predicate type: graph-root.stella/v1
|
||||
# ✓ Graph type: ReachabilityGraph
|
||||
# ✓ Canon version: stella:canon:v1
|
||||
# ⊘ Rekor verification skipped (offline mode)
|
||||
#
|
||||
# Overall: VERIFIED (offline)
|
||||
```
|
||||
|
||||
### Verify with Node/Edge Reconstruction
|
||||
|
||||
When you have the original graph data, you can recompute and verify the Merkle root:
|
||||
|
||||
```bash
|
||||
# Verify with reconstruction
|
||||
stellaops graph-root verify --offline \
|
||||
--envelope graph-root.dsse \
|
||||
--nodes nodes.json \
|
||||
--edges edges.json \
|
||||
--anchor-file trust-anchors.json
|
||||
|
||||
# Expected output:
|
||||
# Graph Root Verification (with reconstruction)
|
||||
# ═════════════════════════════════════════════
|
||||
# ✓ DSSE signature verified
|
||||
# ✓ Nodes canonicalized: 1234 entries
|
||||
# ✓ Edges canonicalized: 5678 entries
|
||||
# ✓ Merkle root recomputed: sha256:abc123...
|
||||
# ✓ Merkle root matches claimed: sha256:abc123...
|
||||
#
|
||||
# Overall: VERIFIED (reconstructed)
|
||||
```
|
||||
|
||||
### Graph Data File Formats
|
||||
|
||||
**nodes.json** - Array of node identifiers:
|
||||
```json
|
||||
{
|
||||
"canonVersion": "stella:canon:v1",
|
||||
"nodes": [
|
||||
"pkg:npm/lodash@4.17.21",
|
||||
"pkg:npm/express@4.18.2",
|
||||
"pkg:npm/body-parser@1.20.0"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**edges.json** - Array of edge identifiers:
|
||||
```json
|
||||
{
|
||||
"canonVersion": "stella:canon:v1",
|
||||
"edges": [
|
||||
"pkg:npm/express@4.18.2->pkg:npm/body-parser@1.20.0",
|
||||
"pkg:npm/express@4.18.2->pkg:npm/lodash@4.17.21"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Verification Steps (Detailed)
|
||||
|
||||
The offline graph root verification algorithm:
|
||||
|
||||
1. **Parse DSSE envelope** - Extract payload and signatures
|
||||
2. **Decode in-toto statement** - Parse subject and predicate
|
||||
3. **Verify signature** - Check DSSE signature against trust anchor allowed keys
|
||||
4. **Validate predicate type** - Confirm `graph-root.stella/v1`
|
||||
5. **Extract Merkle root** - Get claimed root from predicate
|
||||
6. **If reconstruction requested**:
|
||||
- Load nodes.json and edges.json
|
||||
- Verify canon version matches predicate
|
||||
- Sort nodes lexicographically
|
||||
- Sort edges lexicographically
|
||||
- Concatenate sorted lists
|
||||
- Build SHA-256 Merkle tree
|
||||
- Compare computed root to claimed root
|
||||
7. **Emit verification result**
|
||||
|
||||
### Programmatic Verification (.NET)
|
||||
|
||||
```csharp
|
||||
using StellaOps.Attestor.GraphRoot;
|
||||
|
||||
// Load trust anchors
|
||||
var anchors = await TrustAnchors.LoadFromFileAsync("trust-anchors.json");
|
||||
|
||||
// Create verifier
|
||||
var verifier = new GraphRootAttestor(signer, canonicalJsonSerializer);
|
||||
|
||||
// Load envelope
|
||||
var envelope = await DsseEnvelope.LoadAsync("graph-root.dsse");
|
||||
|
||||
// Verify without reconstruction
|
||||
var result = await verifier.VerifyAsync(
|
||||
envelope,
|
||||
trustAnchors: anchors,
|
||||
verifyRekor: false);
|
||||
|
||||
// Verify with reconstruction
|
||||
var nodeIds = new[] { "pkg:npm/lodash@4.17.21", "pkg:npm/express@4.18.2" };
|
||||
var edgeIds = new[] { "pkg:npm/express@4.18.2->pkg:npm/lodash@4.17.21" };
|
||||
|
||||
var fullResult = await verifier.VerifyAsync(
|
||||
envelope,
|
||||
nodeIds: nodeIds,
|
||||
edgeIds: edgeIds,
|
||||
trustAnchors: anchors,
|
||||
verifyRekor: false);
|
||||
|
||||
Console.WriteLine($"Verified: {fullResult.IsValid}");
|
||||
Console.WriteLine($"Merkle root: {fullResult.MerkleRoot}");
|
||||
```
|
||||
|
||||
### Integration with Proof Spine
|
||||
|
||||
Graph roots can be included in proof spines for comprehensive verification:
|
||||
|
||||
```bash
|
||||
# Export proof bundle with graph roots
|
||||
stellaops proof export \
|
||||
--entry sha256:abc123:pkg:npm/lodash@4.17.21 \
|
||||
--include-graph-roots \
|
||||
--output proof-bundle.zip
|
||||
|
||||
# Bundle now includes:
|
||||
# proof-bundle.zip
|
||||
# ├── proof-spine.json
|
||||
# ├── evidence/
|
||||
# ├── reasoning.json
|
||||
# ├── vex-verdict.json
|
||||
# ├── graph-roots/ # Graph root attestations
|
||||
# │ ├── reachability.dsse
|
||||
# │ └── dependency.dsse
|
||||
# ├── envelopes/
|
||||
# └── VERIFY.md
|
||||
|
||||
# Verify with graph roots
|
||||
stellaops proof verify --offline \
|
||||
--bundle-file proof-bundle.zip \
|
||||
--verify-graph-roots \
|
||||
--anchor-file trust-anchors.json
|
||||
```
|
||||
|
||||
### Determinism Requirements
|
||||
|
||||
For offline verification to succeed:
|
||||
|
||||
1. **Same canonicalization** - Use `stella:canon:v1` consistently
|
||||
2. **Same ordering** - Lexicographic sort for nodes and edges
|
||||
3. **Same encoding** - UTF-8 for all string operations
|
||||
4. **Same hash algorithm** - SHA-256 for Merkle tree
|
||||
|
||||
---
|
||||
|
||||
## Key Rotation in Air-Gap Mode
|
||||
|
||||
When keys are rotated, trust anchor updates must be distributed:
|
||||
|
||||
@@ -1,156 +0,0 @@
|
||||
# VEX Raw Migration Rollback Guide
|
||||
|
||||
> **DEPRECATED:** This document describes MongoDB migration rollback procedures which are no longer used. Excititor now uses PostgreSQL for persistence (Sprint 4400). See `docs/db/SPECIFICATION.md` for current schema and migration procedures.
|
||||
|
||||
This document describes how to rollback migrations applied to the `vex_raw` table.
|
||||
|
||||
## Migration: 20251127-vex-raw-idempotency-indexes
|
||||
|
||||
### Description
|
||||
Adds unique idempotency indexes to enforce content-addressed storage:
|
||||
- `idx_provider_sourceUri_digest_unique`: Prevents duplicate documents from same provider/source
|
||||
- `idx_digest_providerId`: Optimizes evidence queries by digest
|
||||
- `idx_retrievedAt`: Supports time-based queries and future TTL operations
|
||||
|
||||
### Rollback Steps
|
||||
|
||||
#### Option 1: MongoDB Shell
|
||||
|
||||
```javascript
|
||||
// Connect to your MongoDB instance
|
||||
mongosh "mongodb://localhost:27017/excititor"
|
||||
|
||||
// Drop the idempotency indexes
|
||||
db.vex_raw.dropIndex("idx_provider_sourceUri_digest_unique")
|
||||
db.vex_raw.dropIndex("idx_digest_providerId")
|
||||
db.vex_raw.dropIndex("idx_retrievedAt")
|
||||
|
||||
// Verify indexes are dropped
|
||||
db.vex_raw.getIndexes()
|
||||
```
|
||||
|
||||
#### Option 2: Programmatic Rollback (C#)
|
||||
|
||||
```csharp
|
||||
using StellaOps.Excititor.Storage.Mongo.Migrations;
|
||||
|
||||
// Get the database instance
|
||||
var database = client.GetDatabase("excititor");
|
||||
|
||||
// Execute rollback
|
||||
await database.RollbackIdempotencyIndexesAsync(cancellationToken);
|
||||
|
||||
// Verify rollback
|
||||
var verified = await database.VerifyIdempotencyIndexesExistAsync(cancellationToken);
|
||||
Console.WriteLine($"Indexes exist after rollback: {verified}"); // Should be false
|
||||
```
|
||||
|
||||
#### Option 3: MongoDB Compass
|
||||
|
||||
1. Connect to your MongoDB instance
|
||||
2. Navigate to the `excititor` database
|
||||
3. Select the `vex_raw` collection
|
||||
4. Go to the "Indexes" tab
|
||||
5. Click "Drop Index" for each of:
|
||||
- `idx_provider_sourceUri_digest_unique`
|
||||
- `idx_digest_providerId`
|
||||
- `idx_retrievedAt`
|
||||
|
||||
### Impact of Rollback
|
||||
|
||||
**Before rollback (indexes present):**
|
||||
- Documents are prevented from being duplicated
|
||||
- Evidence queries are optimized
|
||||
- Unique constraint enforced
|
||||
|
||||
**After rollback (indexes dropped):**
|
||||
- Duplicate documents may be inserted
|
||||
- Evidence queries may be slower
|
||||
- No unique constraint enforcement
|
||||
|
||||
### Re-applying the Migration
|
||||
|
||||
To re-apply the migration after rollback:
|
||||
|
||||
```javascript
|
||||
// MongoDB shell
|
||||
db.vex_raw.createIndex(
|
||||
{ "providerId": 1, "sourceUri": 1, "digest": 1 },
|
||||
{ unique: true, name: "idx_provider_sourceUri_digest_unique", background: true }
|
||||
)
|
||||
|
||||
db.vex_raw.createIndex(
|
||||
{ "digest": 1, "providerId": 1 },
|
||||
{ name: "idx_digest_providerId", background: true }
|
||||
)
|
||||
|
||||
db.vex_raw.createIndex(
|
||||
{ "retrievedAt": 1 },
|
||||
{ name: "idx_retrievedAt", background: true }
|
||||
)
|
||||
```
|
||||
|
||||
Or run the migration runner:
|
||||
|
||||
```bash
|
||||
stellaops excititor migrate --run 20251127-vex-raw-idempotency-indexes
|
||||
```
|
||||
|
||||
## Migration: 20251125-vex-raw-json-schema
|
||||
|
||||
### Description
|
||||
Adds a JSON Schema validator to the `vex_raw` collection with `validationAction: warn`.
|
||||
|
||||
### Rollback Steps
|
||||
|
||||
```javascript
|
||||
// MongoDB shell - remove the validator
|
||||
db.runCommand({
|
||||
collMod: "vex_raw",
|
||||
validator: {},
|
||||
validationAction: "off",
|
||||
validationLevel: "off"
|
||||
})
|
||||
|
||||
// Verify validator is removed
|
||||
db.getCollectionInfos({ name: "vex_raw" })[0].options
|
||||
```
|
||||
|
||||
### Impact of Rollback
|
||||
|
||||
- Documents will no longer be validated against the schema
|
||||
- Invalid documents may be inserted
|
||||
- Existing documents are not affected
|
||||
|
||||
## General Rollback Guidelines
|
||||
|
||||
1. **Always backup first**: Create a backup before any rollback operation
|
||||
2. **Test in staging**: Verify rollback procedure in a non-production environment
|
||||
3. **Monitor performance**: Watch for query performance changes after rollback
|
||||
4. **Document changes**: Log all rollback operations for audit purposes
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Index Drop Fails
|
||||
|
||||
If you see "IndexNotFound" errors, the index may have already been dropped or was never created:
|
||||
|
||||
```javascript
|
||||
// Check existing indexes
|
||||
db.vex_raw.getIndexes()
|
||||
```
|
||||
|
||||
### Validator Removal Fails
|
||||
|
||||
If the validator command fails, verify you have the correct permissions:
|
||||
|
||||
```javascript
|
||||
// Check current user roles
|
||||
db.runCommand({ usersInfo: 1 })
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [VEX Raw Schema Validation](vex-raw-schema-validation.md)
|
||||
- [MongoDB Index Management](https://www.mongodb.com/docs/manual/indexes/)
|
||||
- [Excititor Architecture](../modules/excititor/architecture.md)
|
||||
@@ -1,199 +0,0 @@
|
||||
# VEX Raw Schema Validation - Offline Kit
|
||||
|
||||
> **DEPRECATED:** This document describes MongoDB validation procedures which are no longer used. Excititor now uses PostgreSQL for persistence (Sprint 4400). Schema validation is performed via PostgreSQL constraints. See `docs/db/SPECIFICATION.md` for current schema.
|
||||
|
||||
This document describes how operators can validate the integrity of VEX raw evidence stored in the database, ensuring that Excititor stores only immutable, content-addressed documents.
|
||||
|
||||
## Overview
|
||||
|
||||
The `vex_raw` collection stores raw VEX documents with content-addressed storage (documents are keyed by their cryptographic hash). This ensures immutability - documents cannot be modified after insertion without changing their key.
|
||||
|
||||
## Schema Definition
|
||||
|
||||
The MongoDB JSON Schema enforces the following structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"$jsonSchema": {
|
||||
"bsonType": "object",
|
||||
"title": "VEX Raw Document Schema",
|
||||
"description": "Schema for immutable VEX evidence storage",
|
||||
"required": ["_id", "providerId", "format", "sourceUri", "retrievedAt", "digest"],
|
||||
"properties": {
|
||||
"_id": {
|
||||
"bsonType": "string",
|
||||
"description": "Content digest serving as immutable key"
|
||||
},
|
||||
"providerId": {
|
||||
"bsonType": "string",
|
||||
"minLength": 1,
|
||||
"description": "VEX provider identifier"
|
||||
},
|
||||
"format": {
|
||||
"bsonType": "string",
|
||||
"enum": ["csaf", "cyclonedx", "openvex"],
|
||||
"description": "VEX document format"
|
||||
},
|
||||
"sourceUri": {
|
||||
"bsonType": "string",
|
||||
"minLength": 1,
|
||||
"description": "Original source URI"
|
||||
},
|
||||
"retrievedAt": {
|
||||
"bsonType": "date",
|
||||
"description": "Timestamp when document was fetched"
|
||||
},
|
||||
"digest": {
|
||||
"bsonType": "string",
|
||||
"minLength": 32,
|
||||
"description": "Content hash (SHA-256 hex)"
|
||||
},
|
||||
"content": {
|
||||
"bsonType": ["binData", "string"],
|
||||
"description": "Raw document content"
|
||||
},
|
||||
"gridFsObjectId": {
|
||||
"bsonType": ["objectId", "null", "string"],
|
||||
"description": "GridFS reference for large documents"
|
||||
},
|
||||
"metadata": {
|
||||
"bsonType": "object",
|
||||
"description": "Provider-specific metadata"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Offline Validation Steps
|
||||
|
||||
### 1. Export the Schema
|
||||
|
||||
The schema can be exported from the application using the validator tooling:
|
||||
|
||||
```bash
|
||||
# Using the Excititor CLI
|
||||
stellaops excititor schema export --collection vex_raw --output vex-raw-schema.json
|
||||
|
||||
# Or via MongoDB shell
|
||||
mongosh --eval "db.getCollectionInfos({name: 'vex_raw'})[0].options.validator" > vex-raw-schema.json
|
||||
```
|
||||
|
||||
### 2. Validate Documents in MongoDB Shell
|
||||
|
||||
```javascript
|
||||
// Connect to your MongoDB instance
|
||||
mongosh "mongodb://localhost:27017/excititor"
|
||||
|
||||
// Get all documents that violate the schema
|
||||
db.runCommand({
|
||||
validate: "vex_raw",
|
||||
full: true
|
||||
})
|
||||
|
||||
// Or check individual documents
|
||||
db.vex_raw.find().forEach(function(doc) {
|
||||
var result = db.runCommand({
|
||||
validate: "vex_raw",
|
||||
documentId: doc._id
|
||||
});
|
||||
if (!result.valid) {
|
||||
print("Invalid: " + doc._id);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
### 3. Programmatic Validation (C#)
|
||||
|
||||
```csharp
|
||||
using StellaOps.Excititor.Storage.Mongo.Validation;
|
||||
|
||||
// Validate a single document
|
||||
var result = VexRawSchemaValidator.Validate(document);
|
||||
if (!result.IsValid)
|
||||
{
|
||||
foreach (var violation in result.Violations)
|
||||
{
|
||||
Console.WriteLine($"{violation.Field}: {violation.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
// Batch validation
|
||||
var batchResult = VexRawSchemaValidator.ValidateBatch(documents);
|
||||
Console.WriteLine($"Valid: {batchResult.ValidCount}, Invalid: {batchResult.InvalidCount}");
|
||||
```
|
||||
|
||||
### 4. Export Schema for External Tools
|
||||
|
||||
```csharp
|
||||
// Get schema as JSON for external validation tools
|
||||
var schemaJson = VexRawSchemaValidator.GetJsonSchemaAsJson();
|
||||
File.WriteAllText("vex-raw-schema.json", schemaJson);
|
||||
```
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
Use this checklist to verify schema compliance:
|
||||
|
||||
- [ ] All documents have required fields (_id, providerId, format, sourceUri, retrievedAt, digest)
|
||||
- [ ] The `_id` matches the `digest` value (content-addressed)
|
||||
- [ ] Format is one of: csaf, cyclonedx, openvex
|
||||
- [ ] Digest is at least 32 characters (SHA-256 hex)
|
||||
- [ ] No documents have been modified after insertion (verify via digest recomputation)
|
||||
|
||||
## Immutability Verification
|
||||
|
||||
To verify documents haven't been tampered with:
|
||||
|
||||
```javascript
|
||||
// MongoDB shell - verify content matches digest
|
||||
db.vex_raw.find().forEach(function(doc) {
|
||||
var content = doc.content;
|
||||
if (content) {
|
||||
// Compute SHA-256 of content
|
||||
var computedDigest = hex_md5(content); // Use appropriate hash function
|
||||
if (computedDigest !== doc.digest) {
|
||||
print("TAMPERED: " + doc._id);
|
||||
}
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
## Auditing
|
||||
|
||||
For compliance auditing, export a validation report:
|
||||
|
||||
```bash
|
||||
# Generate validation report
|
||||
stellaops excititor validate --collection vex_raw --report validation-report.json
|
||||
|
||||
# The report includes:
|
||||
# - Total document count
|
||||
# - Valid/invalid counts
|
||||
# - List of violations by document
|
||||
# - Schema version used for validation
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Violations
|
||||
|
||||
1. **Missing required field**: Ensure all required fields are present
|
||||
2. **Invalid format**: Format must be exactly "csaf", "cyclonedx", or "openvex"
|
||||
3. **Digest too short**: Digest must be at least 32 hex characters
|
||||
4. **Wrong type**: Check field types match schema requirements
|
||||
|
||||
### Recovery
|
||||
|
||||
If invalid documents are found:
|
||||
|
||||
1. Do NOT modify documents in place (violates immutability)
|
||||
2. Export the invalid documents for analysis
|
||||
3. Re-ingest from original sources with correct data
|
||||
4. Document the incident in audit logs
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Excititor Architecture](../modules/excititor/architecture.md)
|
||||
- [VEX Storage Design](../modules/excititor/storage.md)
|
||||
- [Offline Operation Guide](../24_OFFLINE_KIT.md)
|
||||
Reference in New Issue
Block a user