stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity

up
Some checks failed
AOC Guard CI / aoc-guard (push) Has been cancelled
Details
AOC Guard CI / aoc-verify (push) Has been cancelled
Details
Concelier Attestation Tests / attestation-tests (push) Has been cancelled
Details
Docs CI / lint-and-preview (push) Has been cancelled
Details
Policy Lint & Smoke / policy-lint (push) Has been cancelled
Details

Browse Source
This commit is contained in:
StellaOps Bot
2025-11-28 09:40:40 +02:00
parent 1c6730a1d2
commit 05da719048
206 changed files with 34741 additions and 1751 deletions
Download Patch File Download Diff File Expand all files Collapse all files

414
docs/modules/scanner/guides/surface-fs-workflow.md Normal file
View File

@@ -0,0 +1,414 @@
# Surface.FS Workflow Guide
> **Version:** 1.0 (2025-11-28)
>
> **Audience:** Scanner Worker/WebService integrators, Zastava operators, Offline Kit builders
## Overview
Surface.FS provides a content-addressable storage layer for Scanner-derived artefacts. This guide covers the end-to-end workflow from artefact generation to consumption, including offline bundle handling.
## Workflow Stages
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Scanner Worker │───▶│ Surface.FS │───▶│ Consumers │
│ - Scan image │ │ - Store manifest│ │ - WebService │
│ - Generate │ │ - Store payload │ │ - Zastava │
│ artefacts │ │ - Local cache │ │ - CLI │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Generate: │ │ Store: │ │ Consume: │
│ - Layer frags │ │ - RustFS/S3 │ │ - Report API │
│ - EntryTrace │ │ - Local disk │ │ - Drift detect │
│ - SBOM frags │ │ - Offline kit │ │ - Rescan plan │
└─────────────────┘ └─────────────────┘ └─────────────────┘
```
## Stage 1: Artefact Generation (Scanner Worker)
### 1.1 Configure Surface.FS
```csharp
// In Scanner Worker startup
builder.Services.AddSurfaceFileCache();
builder.Services.AddSurfaceManifestStore();
```
Environment variables (see [Surface.Env guide](../design/surface-env.md)):
```bash
SCANNER_SURFACE_FS_ENDPOINT=http://rustfs:8080
SCANNER_SURFACE_FS_BUCKET=surface-cache
SCANNER_SURFACE_CACHE_ROOT=/var/lib/stellaops/surface
SCANNER_SURFACE_TENANT=default
```
### 1.2 Generate and Publish Artefacts
```csharp
public async Task<ScanResult> ExecuteScanAsync(ScanJob job, CancellationToken ct)
{
// 1. Run analyzers to generate artefacts
var layerFragments = await AnalyzeLayersAsync(job.Image, ct);
var entryTrace = await AnalyzeEntryPointsAsync(job.Image, ct);
var sbomFragments = await GenerateSbomAsync(job.Image, ct);
// 2. Create manifest document
var manifest = new SurfaceManifestDocument
{
Schema = "stellaops.surface.manifest@1",
Tenant = _environment.Settings.Tenant,
ImageDigest = job.Image.Digest,
ScanId = job.Id,
GeneratedAt = DateTimeOffset.UtcNow,
Source = new SurfaceManifestSource
{
Component = "scanner.worker",
Version = _version,
WorkerInstance = Environment.MachineName,
Attempt = job.Attempt
},
Artifacts = new List<SurfaceManifestArtifact>()
};
// 3. Add artefacts to manifest
foreach (var fragment in layerFragments)
{
var payloadUri = await _manifestWriter.StorePayloadAsync(
fragment.Content,
"layer.fragments",
ct);
manifest.Artifacts.Add(new SurfaceManifestArtifact
{
Kind = "layer.fragments",
Uri = payloadUri,
Digest = fragment.Digest,
MediaType = "application/vnd.stellaops.layer-fragments+json",
Format = "json",
SizeBytes = fragment.Content.Length
});
}
// 4. Publish manifest
var result = await _manifestWriter.PublishAsync(manifest, ct);
_logger.LogInformation(
"Published manifest {Digest} with {Count} artefacts",
result.Digest,
manifest.Artifacts.Count);
return new ScanResult
{
ManifestUri = result.Uri,
ManifestDigest = result.Digest
};
}
```
### 1.3 Cache EntryTrace Results
```csharp
public async Task<EntryTraceGraph?> GetOrComputeEntryTraceAsync(
ImageReference image,
EntryTraceOptions options,
CancellationToken ct)
{
// Create deterministic cache key
var cacheKey = new SurfaceCacheKey(
@namespace: "entrytrace.graph",
tenant: _environment.Settings.Tenant,
digest: ComputeOptionsHash(options, image.Digest));
// Try cache first
var cached = await _cache.TryGetAsync<EntryTraceGraph>(cacheKey, ct);
if (cached is not null)
{
_logger.LogDebug("EntryTrace cache hit for {Key}", cacheKey);
return cached;
}
// Compute and cache
var graph = await ComputeEntryTraceAsync(image, options, ct);
await _cache.SetAsync(cacheKey, graph, ct);
return graph;
}
```
## Stage 2: Storage (Surface.FS)
### 2.1 Manifest Storage Layout
```
<bucket>/
├── manifests/
│ └── <tenant>/
│ └── <digest[0..1]>/
│ └── <digest[2..3]>/
│ └── <digest>.json
└── payloads/
└── <tenant>/
└── <kind>/
└── sha256/
└── <digest[0..1]>/
└── <digest[2..3]>/
└── <digest>.json.zst
```
### 2.2 Local Cache Layout
```
<cache_root>/
├── manifests/ # Manifest JSON files
│ └── <tenant>/...
├── cache/ # Hot artefacts
│ └── <namespace>/
│ └── <tenant>/
│ └── <digest>
└── temp/ # In-progress writes
```
### 2.3 Manifest URI Format
```
cas://<bucket>/<prefix>/<tenant>/<digest[0..1]>/<digest[2..3]>/<digest>.json
```
Example:
```
cas://surface-cache/manifests/acme/ab/cd/abcdef0123456789...json
```
## Stage 3: Consumption
### 3.1 WebService API
```http
GET /api/v1/scans/{id}
```
Response includes Surface manifest pointer:
```json
{
"id": "scan-1234",
"status": "completed",
"surface": {
"manifestUri": "cas://surface-cache/manifests/acme/ab/cd/...",
"manifestDigest": "sha256:abcdef...",
"artifacts": [
{
"kind": "layer.fragments",
"uri": "cas://surface-cache/payloads/acme/layer.fragments/...",
"digest": "sha256:123456...",
"mediaType": "application/vnd.stellaops.layer-fragments+json"
}
]
}
}
```
### 3.2 Zastava Drift Detection
```csharp
public async Task<DriftResult> DetectDriftAsync(
string imageDigest,
CancellationToken ct)
{
// 1. Fetch baseline manifest
var manifestUri = await _surfacePointerService.GetManifestUriAsync(imageDigest, ct);
var manifest = await _manifestReader.TryGetByUriAsync(manifestUri, ct);
if (manifest is null)
{
return DriftResult.NoBaseline();
}
// 2. Get EntryTrace artefact
var entryTraceArtifact = manifest.Artifacts
.FirstOrDefault(a => a.Kind == "entrytrace.graph");
if (entryTraceArtifact is null)
{
return DriftResult.NoEntryTrace();
}
// 3. Compare with runtime
var baseline = await _payloadStore.GetAsync<EntryTraceGraph>(
entryTraceArtifact.Uri, ct);
var runtime = await _runtimeCollector.CollectAsync(ct);
return CompareGraphs(baseline, runtime);
}
```
### 3.3 Scheduler Rescan Planning
```csharp
public async Task<RescanPlan> CreateRescanPlanAsync(
string imageDigest,
CancellationToken ct)
{
// 1. Read manifest to understand what was scanned
var manifest = await _manifestReader.TryGetByDigestAsync(imageDigest, ct);
if (manifest is null || IsExpired(manifest))
{
return RescanPlan.FullRescan();
}
// 2. Check for layer changes
var layerArtifact = manifest.Artifacts
.FirstOrDefault(a => a.Kind == "layer.fragments");
if (layerArtifact is not null)
{
var layers = await _payloadStore.GetAsync<LayerFragments>(
layerArtifact.Uri, ct);
var changedLayers = await DetectChangedLayersAsync(layers, ct);
if (changedLayers.Any())
{
return RescanPlan.IncrementalRescan(changedLayers);
}
}
return RescanPlan.NoRescanNeeded();
}
```
## Offline Kit Workflow
### Export (Online Environment)
```bash
# 1. Build offline kit with Surface manifests
python ops/offline-kit/build_offline_kit.py \
--version 2025.10.0 \
--include-surface-manifests \
--output-dir out/offline-kit
# 2. Kit structure includes:
# offline/
# surface/
# manifests/
# <tenant>/<digest[0..1]>/<digest[2..3]>/<digest>.json
# payloads/
# <tenant>/<kind>/sha256/<digest[0..1]>/<digest[2..3]>/<digest>.json.zst
# manifest-index.json
```
### Import (Air-Gapped Environment)
```csharp
public async Task ImportOfflineKitAsync(
string kitPath,
CancellationToken ct)
{
var surfacePath = Path.Combine(kitPath, "surface");
var indexPath = Path.Combine(surfacePath, "manifest-index.json");
var index = await LoadIndexAsync(indexPath, ct);
foreach (var entry in index.Manifests)
{
// 1. Load and verify manifest
var manifestPath = Path.Combine(surfacePath, entry.RelativePath);
var manifest = await LoadManifestAsync(manifestPath, ct);
// 2. Verify digest
var computedDigest = ComputeDigest(manifest);
if (computedDigest != entry.Digest)
{
throw new InvalidOperationException(
$"Manifest digest mismatch: expected {entry.Digest}, got {computedDigest}");
}
// 3. Import via Surface.FS API
await _manifestWriter.PublishAsync(manifest, ct);
_logger.LogInformation(
"Imported manifest {Digest} for image {Image}",
entry.Digest,
manifest.ImageDigest);
}
// 4. Import payloads
foreach (var payload in index.Payloads)
{
var payloadPath = Path.Combine(surfacePath, payload.RelativePath);
await _payloadStore.ImportAsync(payloadPath, payload.Uri, ct);
}
}
```
### Offline Operation
Once imported, Surface.FS consumers operate normally:
```csharp
// Same code works online and offline
var manifest = await _manifestReader.TryGetByUriAsync(manifestUri, ct);
var payload = await _payloadStore.GetAsync(artifact.Uri, ct);
```
## Configuration Reference
### SurfaceManifestStoreOptions
| Option | Default | Description |
|--------|---------|-------------|
| `Bucket` | `surface-cache` | Object store bucket |
| `ManifestPrefix` | `manifests` | Prefix for manifest objects |
| `PayloadPrefix` | `payloads` | Prefix for payload objects |
| `LocalManifestRoot` | `<cache>/manifests` | Local manifest directory |
### SurfaceCacheOptions
| Option | Default | Description |
|--------|---------|-------------|
| `Root` | `<temp>/stellaops/surface` | Cache root directory |
| `QuotaMegabytes` | `4096` | Cache size limit |
| `EvictionThreshold` | `0.9` | Trigger eviction at 90% quota |
## Metrics
| Metric | Labels | Description |
|--------|--------|-------------|
| `surface_manifest_published_total` | `tenant`, `kind` | Manifests published |
| `surface_manifest_cache_hit_total` | `namespace`, `tenant` | Cache hits |
| `surface_manifest_publish_duration_ms` | `tenant` | Publish latency |
| `surface_payload_persisted_total` | `kind` | Payloads stored |
## Troubleshooting
### Manifest Not Found
1. Check tenant matches between writer and reader
2. Verify Surface.FS endpoint is reachable
3. Check bucket permissions
4. Review `surface_manifest_published_total` metric
### Cache Miss Despite Expected Hit
1. Verify cache key components match (namespace, tenant, digest)
2. Check cache quota - eviction may have occurred
3. Review `surface_manifest_cache_hit_total` metric
### Offline Import Failures
1. Verify manifest digest matches index
2. Check file permissions on import path
3. Ensure Surface.FS endpoint is writable
4. Review import logs for specific errors
## References
- [Surface.FS Design](../design/surface-fs.md)
- [Surface.Env Design](../design/surface-env.md)
- [Surface.Validation Guide](./surface-validation-extensibility.md)
- [Offline Kit Documentation](../../../../24_OFFLINE_KIT.md)