git.stella-ops.org/docs/modules/scanner/design/surface-fs.md

# Surface.FS Design (Epic: SURFACE-SHARING)

> **Status:** Draft v1.0 — aligns with tasks `SURFACE-FS-01..06`, `SCANNER-SURFACE-01..05`, `ZASTAVA-SURFACE-01..02`, `SCHED-SURFACE-01`, `OPS-SECRETS-01..02`.
>
> **Audience:** Scanner Worker/WebService, Zastava, Scheduler, DevOps.

## 1. Purpose

Surface.FS provides a unified content-addressable cache for Scanner-derived artefacts (layer manifests, entry traces, SBOM fragments, runtime deltas). It enables:

- Sharing scan results between Worker, WebService, Zastava Observer/Webhook, Scheduler planners, Export Center, and future CLI operations.
- Deterministic reproduction of scan evidence (manifests and payloads) in both connected and air-gapped environments.
- Efficient data movement by storing manifests once and referencing them via stable pointers.

## 2. Core Concepts

### 2.1 Artefact Key

Each artefact is addressed by a tuple `(tenant, surfaceKind, contentDigest)` where `contentDigest` is a SHA256 of the canonical payload. `surfaceKind` identifies artefact type (see Manifest schema below).

### 2.2 Manifest

Manifests describe the artefact metadata and storage pointers. They are stored in the `surface-manifests` bucket and fetched by consumers before retrieving bulk data.

```json
{
  "schema": "stellaops.surface.manifest@1",
  "tenant": "acme",
  "kind": "layer-entry-trace",
  "digest": "sha256:ab12...",
  "createdAt": "2025-10-29T12:00:00Z",
  "expiresAt": "2025-11-05T12:00:00Z",
  "source": {
    "scannerBuild": "stellaops/scanner@sha256:deadbeef",
    "imageDigest": "sha256:cafe...",
    "scanId": "scan-1234"
  },
  "storage": {
    "bucket": "surface-cache",
    "objectKey": "tenants/acme/layer-entry-trace/sha256/ab/12/.../payload.json.zst",
    "sizeBytes": 524288,
    "contentType": "application/json+zstd"
  },
  "integrity": {
    "hash": "sha256:ab12...",
    "signature": null
  }
}
```

### 2.3 Payload Storage

Large payloads (SBOM fragments, entry traces, runtime events) live in the same object store as manifests (RustFS/S3). Manifests record relative paths so offline bundles can copy both manifest and payload without modification.

## 3. APIs

Surface.FS exposes a gRPC/HTTP API consumed by .NET clients:

| Method | Description |
|--------|-------------|
| `PutManifest(PutManifestRequest)` | Stores manifest + optional payload. Idempotent via `digest`. |
| `GetManifest(GetManifestRequest)` | Returns manifest metadata; 404 if missing. |
| `GetPayload(GetPayloadRequest)` | Streams payload bytes (optionally decompressing). |
| `ListManifests(ListManifestRequest)` | Enumerates manifests for tenant/kind with pagination. |
| `DeleteManifest(DeleteManifestRequest)` | (Optional) Removes manifest/payload based on retention policies. |

.NET client wraps these calls and handles retries using Polly policies.

## 4. Library Responsibilities

Surface.FS library for .NET hosts provides:

- `ISurfaceManifestWriter` / `ISurfaceManifestReader` interfaces.
- Content-addressed path builder (`SurfacePathBuilder`).
- Tenant namespace isolation and bucket configuration (via Surface.Env).
- Local cache management (using `SCANNER_SURFACE_CACHE_ROOT` and quota).
- Metrics: `surface_manifest_put_seconds`, `surface_manifest_cache_hit_total`, etc.

## 5. Retention & Eviction

- Manifests include optional `expiresAt`; Worker defaults to 30 days for SBOM fragments, 7 days for entry traces.
- Background job `SurfaceCacheMaintenanceService` evicts local cache entries exceeding quota, oldest-first.
- Object storage retention policies are managed by DevOps; library exposes metrics but does not auto-delete unless instructed.

## 6. Offline Kit Handling

Offline kits include:

```
offline/surface/
  manifests/
    tenants/<tenant>/<kind>/<digest>.json
  payloads/
    tenants/<tenant>/<kind>/<digest>.json.zst
  manifest-index.json
```

Import script calls `PutManifest` for each manifest, verifying digests. This enables Zastava and Scheduler running offline to consume cached data without re-scanning.

## 7. Security & Tenancy

- Tenant ID is mandatory; Surface.Validation enforces match with Authority token.
- Manifests/payloads stored in tenant-specific prefixes to prevent leakage.
- Optional manifest signing (future) will use `Surface.Secrets` to load signing keys.
- TLS enforced between hosts and Surface.FS endpoint; certificate pins configured via Surface.Env.

## 8. Observability

- Logs include manifest SHA, tenant, and kind; payload paths truncated for brevity.
- Metrics exported via Prometheus with labels `{tenant, kind, result}`.
- Tracing spans: `surface.fs.put`, `surface.fs.get`, `surface.fs.cache`.

## 9. Testing Strategy

- Unit tests for path builder, manifest serializer, and local cache eviction.
- Integration tests using embedded RustFS or MinIO container to validate API interactions.
- Offline kit tests verifying export/import cycle round-trips manifests and payloads.

## 10. Future Enhancements

- Manifest signing (DSSE) to support tamper detection in hostile environments.
- Differential manifests to optimise large SBOM updates.
- Cross-region replication for multi-site deployments.

## 11. References

- Surface.Env Design (`docs/modules/scanner/design/surface-env.md`)
- Surface.Secrets Design (`docs/modules/scanner/design/surface-secrets.md`)
- Surface.Validation Design (`docs/modules/scanner/design/surface-validation.md`)
- Zastava Deployment Runbook (`docs/modules/devops/runbooks/zastava-deployment.md`)