git.stella-ops.org/docs/modules/scanner/guides/surface-fs-workflow.md

Surface.FS Workflow Guide

Version: 1.0 (2025-11-28)

Audience: Scanner Worker/WebService integrators, Zastava operators, Offline Kit builders

Overview

Surface.FS provides a content-addressable storage layer for Scanner-derived artefacts. This guide covers the end-to-end workflow from artefact generation to consumption, including offline bundle handling.

Workflow Stages

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Scanner Worker │───▶│  Surface.FS     │───▶│  Consumers      │
│  - Scan image   │    │  - Store manifest│   │  - WebService   │
│  - Generate     │    │  - Store payload │   │  - Zastava      │
│    artefacts    │    │  - Local cache   │   │  - CLI          │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                      │                      │
         ▼                      ▼                      ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Generate:      │    │  Store:         │    │  Consume:       │
│  - Layer frags  │    │  - RustFS/S3    │    │  - Report API   │
│  - EntryTrace   │    │  - Local disk   │    │  - Drift detect │
│  - SBOM frags   │    │  - Offline kit  │    │  - Rescan plan  │
└─────────────────┘    └─────────────────┘    └─────────────────┘

Stage 1: Artefact Generation (Scanner Worker)

1.1 Configure Surface.FS

// In Scanner Worker startup
builder.Services.AddSurfaceFileCache();
builder.Services.AddSurfaceManifestStore();

Environment variables (see Surface.Env guide):

SCANNER_SURFACE_FS_ENDPOINT=http://rustfs:8080
SCANNER_SURFACE_FS_BUCKET=surface-cache
SCANNER_SURFACE_CACHE_ROOT=/var/lib/stellaops/surface
SCANNER_SURFACE_TENANT=default

1.2 Generate and Publish Artefacts

public async Task<ScanResult> ExecuteScanAsync(ScanJob job, CancellationToken ct)
{
    // 1. Run analyzers to generate artefacts
    var layerFragments = await AnalyzeLayersAsync(job.Image, ct);
    var entryTrace = await AnalyzeEntryPointsAsync(job.Image, ct);
    var sbomFragments = await GenerateSbomAsync(job.Image, ct);

    // 2. Create manifest document
    var manifest = new SurfaceManifestDocument
    {
        Schema = "stellaops.surface.manifest@1",
        Tenant = _environment.Settings.Tenant,
        ImageDigest = job.Image.Digest,
        ScanId = job.Id,
        GeneratedAt = DateTimeOffset.UtcNow,
        Source = new SurfaceManifestSource
        {
            Component = "scanner.worker",
            Version = _version,
            WorkerInstance = Environment.MachineName,
            Attempt = job.Attempt
        },
        Artifacts = new List<SurfaceManifestArtifact>()
    };

    // 3. Add artefacts to manifest
    foreach (var fragment in layerFragments)
    {
        var payloadUri = await _manifestWriter.StorePayloadAsync(
            fragment.Content,
            "layer.fragments",
            ct);

        manifest.Artifacts.Add(new SurfaceManifestArtifact
        {
            Kind = "layer.fragments",
            Uri = payloadUri,
            Digest = fragment.Digest,
            MediaType = "application/vnd.stellaops.layer-fragments+json",
            Format = "json",
            SizeBytes = fragment.Content.Length
        });
    }

    // 4. Publish manifest
    var result = await _manifestWriter.PublishAsync(manifest, ct);

    _logger.LogInformation(
        "Published manifest {Digest} with {Count} artefacts",
        result.Digest,
        manifest.Artifacts.Count);

    return new ScanResult
    {
        ManifestUri = result.Uri,
        ManifestDigest = result.Digest
    };
}

1.3 Cache EntryTrace Results

public async Task<EntryTraceGraph?> GetOrComputeEntryTraceAsync(
    ImageReference image,
    EntryTraceOptions options,
    CancellationToken ct)
{
    // Create deterministic cache key
    var cacheKey = new SurfaceCacheKey(
        @namespace: "entrytrace.graph",
        tenant: _environment.Settings.Tenant,
        digest: ComputeOptionsHash(options, image.Digest));

    // Try cache first
    var cached = await _cache.TryGetAsync<EntryTraceGraph>(cacheKey, ct);
    if (cached is not null)
    {
        _logger.LogDebug("EntryTrace cache hit for {Key}", cacheKey);
        return cached;
    }

    // Compute and cache
    var graph = await ComputeEntryTraceAsync(image, options, ct);
    await _cache.SetAsync(cacheKey, graph, ct);

    return graph;
}

Stage 2: Storage (Surface.FS)

2.1 Manifest Storage Layout

<bucket>/
├── manifests/
│   └── <tenant>/
│       └── <digest[0..1]>/
│           └── <digest[2..3]>/
│               └── <digest>.json
└── payloads/
    └── <tenant>/
        └── <kind>/
            └── sha256/
                └── <digest[0..1]>/
                    └── <digest[2..3]>/
                        └── <digest>.json.zst

2.2 Local Cache Layout

<cache_root>/
├── manifests/          # Manifest JSON files
│   └── <tenant>/...
├── cache/              # Hot artefacts
│   └── <namespace>/
│       └── <tenant>/
│           └── <digest>
└── temp/               # In-progress writes

2.3 Manifest URI Format

cas://<bucket>/<prefix>/<tenant>/<digest[0..1]>/<digest[2..3]>/<digest>.json

Example:

cas://surface-cache/manifests/acme/ab/cd/abcdef0123456789...json

Stage 3: Consumption

3.1 WebService API

GET /api/v1/scans/{id}

Response includes Surface manifest pointer:

{
  "id": "scan-1234",
  "status": "completed",
  "surface": {
    "manifestUri": "cas://surface-cache/manifests/acme/ab/cd/...",
    "manifestDigest": "sha256:abcdef...",
    "artifacts": [
      {
        "kind": "layer.fragments",
        "uri": "cas://surface-cache/payloads/acme/layer.fragments/...",
        "digest": "sha256:123456...",
        "mediaType": "application/vnd.stellaops.layer-fragments+json"
      }
    ]
  }
}

3.2 Zastava Drift Detection

public async Task<DriftResult> DetectDriftAsync(
    string imageDigest,
    CancellationToken ct)
{
    // 1. Fetch baseline manifest
    var manifestUri = await _surfacePointerService.GetManifestUriAsync(imageDigest, ct);
    var manifest = await _manifestReader.TryGetByUriAsync(manifestUri, ct);

    if (manifest is null)
    {
        return DriftResult.NoBaseline();
    }

    // 2. Get EntryTrace artefact
    var entryTraceArtifact = manifest.Artifacts
        .FirstOrDefault(a => a.Kind == "entrytrace.graph");

    if (entryTraceArtifact is null)
    {
        return DriftResult.NoEntryTrace();
    }

    // 3. Compare with runtime
    var baseline = await _payloadStore.GetAsync<EntryTraceGraph>(
        entryTraceArtifact.Uri, ct);

    var runtime = await _runtimeCollector.CollectAsync(ct);

    return CompareGraphs(baseline, runtime);
}

3.3 Scheduler Rescan Planning

public async Task<RescanPlan> CreateRescanPlanAsync(
    string imageDigest,
    CancellationToken ct)
{
    // 1. Read manifest to understand what was scanned
    var manifest = await _manifestReader.TryGetByDigestAsync(imageDigest, ct);

    if (manifest is null || IsExpired(manifest))
    {
        return RescanPlan.FullRescan();
    }

    // 2. Check for layer changes
    var layerArtifact = manifest.Artifacts
        .FirstOrDefault(a => a.Kind == "layer.fragments");

    if (layerArtifact is not null)
    {
        var layers = await _payloadStore.GetAsync<LayerFragments>(
            layerArtifact.Uri, ct);

        var changedLayers = await DetectChangedLayersAsync(layers, ct);

        if (changedLayers.Any())
        {
            return RescanPlan.IncrementalRescan(changedLayers);
        }
    }

    return RescanPlan.NoRescanNeeded();
}

Offline Kit Workflow

Export (Online Environment)

# 1. Build offline kit with Surface manifests
python ops/offline-kit/build_offline_kit.py \
  --version 2025.10.0 \
  --include-surface-manifests \
  --output-dir out/offline-kit

# 2. Kit structure includes:
# offline/
#   surface/
#     manifests/
#       <tenant>/<digest[0..1]>/<digest[2..3]>/<digest>.json
#     payloads/
#       <tenant>/<kind>/sha256/<digest[0..1]>/<digest[2..3]>/<digest>.json.zst
#     manifest-index.json

Import (Air-Gapped Environment)

public async Task ImportOfflineKitAsync(
    string kitPath,
    CancellationToken ct)
{
    var surfacePath = Path.Combine(kitPath, "surface");
    var indexPath = Path.Combine(surfacePath, "manifest-index.json");

    var index = await LoadIndexAsync(indexPath, ct);

    foreach (var entry in index.Manifests)
    {
        // 1. Load and verify manifest
        var manifestPath = Path.Combine(surfacePath, entry.RelativePath);
        var manifest = await LoadManifestAsync(manifestPath, ct);

        // 2. Verify digest
        var computedDigest = ComputeDigest(manifest);
        if (computedDigest != entry.Digest)
        {
            throw new InvalidOperationException(
                $"Manifest digest mismatch: expected {entry.Digest}, got {computedDigest}");
        }

        // 3. Import via Surface.FS API
        await _manifestWriter.PublishAsync(manifest, ct);

        _logger.LogInformation(
            "Imported manifest {Digest} for image {Image}",
            entry.Digest,
            manifest.ImageDigest);
    }

    // 4. Import payloads
    foreach (var payload in index.Payloads)
    {
        var payloadPath = Path.Combine(surfacePath, payload.RelativePath);
        await _payloadStore.ImportAsync(payloadPath, payload.Uri, ct);
    }
}

Offline Operation

Once imported, Surface.FS consumers operate normally:

// Same code works online and offline
var manifest = await _manifestReader.TryGetByUriAsync(manifestUri, ct);
var payload = await _payloadStore.GetAsync(artifact.Uri, ct);

Configuration Reference

SurfaceManifestStoreOptions

Option	Default	Description
Bucket	surface-cache	Object store bucket
ManifestPrefix	manifests	Prefix for manifest objects
PayloadPrefix	payloads	Prefix for payload objects
LocalManifestRoot	<cache>/manifests	Local manifest directory

SurfaceCacheOptions

Option	Default	Description
Root	<temp>/stellaops/surface	Cache root directory
QuotaMegabytes	4096	Cache size limit
EvictionThreshold	0.9	Trigger eviction at 90% quota

Metrics

Metric	Labels	Description
surface_manifest_published_total	tenant, kind	Manifests published
surface_manifest_cache_hit_total	namespace, tenant	Cache hits
surface_manifest_publish_duration_ms	tenant	Publish latency
surface_payload_persisted_total	kind	Payloads stored

Troubleshooting

Manifest Not Found

1. Check tenant matches between writer and reader
2. Verify Surface.FS endpoint is reachable
3. Check bucket permissions
4. Review surface_manifest_published_total metric

Cache Miss Despite Expected Hit

1. Verify cache key components match (namespace, tenant, digest)
2. Check cache quota - eviction may have occurred
3. Review surface_manifest_cache_hit_total metric

Offline Import Failures

1. Verify manifest digest matches index
2. Check file permissions on import path
3. Ensure Surface.FS endpoint is writable
4. Review import logs for specific errors

References

• Surface.FS Design
• Surface.Env Design
• Surface.Validation Guide
• Offline Kit Documentation