feat: Add guild charters and task boards for various components
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Docs CI / lint-and-preview (push) Has been cancelled
- Introduced guild charters for Scanner Deno, PHP, Ruby, Native, WebService, Java, Surface.Env, Surface.FS, Surface.Secrets, Surface.Validation, UI, Zastava Observer, Zastava Webhook, Zastava Core, and Plugin Platform. - Each charter outlines the mission, scope, required reading, and working agreements for the respective guilds. - Created task boards for Surface.Env, Surface.FS, Surface.Secrets, Surface.Validation, and Zastava components to track progress and dependencies. - Ensured all documents emphasize determinism, offline readiness, security, and integration with shared Surface libraries.
This commit is contained in:
130
docs/modules/scanner/design/surface-fs.md
Normal file
130
docs/modules/scanner/design/surface-fs.md
Normal file
@@ -0,0 +1,130 @@
|
||||
# Surface.FS Design (Epic: SURFACE-SHARING)
|
||||
|
||||
> **Status:** Draft v1.0 — aligns with tasks `SURFACE-FS-01..06`, `SCANNER-SURFACE-01..05`, `ZASTAVA-SURFACE-01..02`, `SCHED-SURFACE-01`, `OPS-SECRETS-01..02`.
|
||||
>
|
||||
> **Audience:** Scanner Worker/WebService, Zastava, Scheduler, DevOps.
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Surface.FS provides a unified content-addressable cache for Scanner-derived artefacts (layer manifests, entry traces, SBOM fragments, runtime deltas). It enables:
|
||||
|
||||
- Sharing scan results between Worker, WebService, Zastava Observer/Webhook, Scheduler planners, Export Center, and future CLI operations.
|
||||
- Deterministic reproduction of scan evidence (manifests and payloads) in both connected and air-gapped environments.
|
||||
- Efficient data movement by storing manifests once and referencing them via stable pointers.
|
||||
|
||||
## 2. Core Concepts
|
||||
|
||||
### 2.1 Artefact Key
|
||||
|
||||
Each artefact is addressed by a tuple `(tenant, surfaceKind, contentDigest)` where `contentDigest` is a SHA256 of the canonical payload. `surfaceKind` identifies artefact type (see Manifest schema below).
|
||||
|
||||
### 2.2 Manifest
|
||||
|
||||
Manifests describe the artefact metadata and storage pointers. They are stored in the `surface-manifests` bucket and fetched by consumers before retrieving bulk data.
|
||||
|
||||
```json
|
||||
{
|
||||
"schema": "stellaops.surface.manifest@1",
|
||||
"tenant": "acme",
|
||||
"kind": "layer-entry-trace",
|
||||
"digest": "sha256:ab12...",
|
||||
"createdAt": "2025-10-29T12:00:00Z",
|
||||
"expiresAt": "2025-11-05T12:00:00Z",
|
||||
"source": {
|
||||
"scannerBuild": "stellaops/scanner@sha256:deadbeef",
|
||||
"imageDigest": "sha256:cafe...",
|
||||
"scanId": "scan-1234"
|
||||
},
|
||||
"storage": {
|
||||
"bucket": "surface-cache",
|
||||
"objectKey": "tenants/acme/layer-entry-trace/sha256/ab/12/.../payload.json.zst",
|
||||
"sizeBytes": 524288,
|
||||
"contentType": "application/json+zstd"
|
||||
},
|
||||
"integrity": {
|
||||
"hash": "sha256:ab12...",
|
||||
"signature": null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2.3 Payload Storage
|
||||
|
||||
Large payloads (SBOM fragments, entry traces, runtime events) live in the same object store as manifests (RustFS/S3). Manifests record relative paths so offline bundles can copy both manifest and payload without modification.
|
||||
|
||||
## 3. APIs
|
||||
|
||||
Surface.FS exposes a gRPC/HTTP API consumed by .NET clients:
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `PutManifest(PutManifestRequest)` | Stores manifest + optional payload. Idempotent via `digest`. |
|
||||
| `GetManifest(GetManifestRequest)` | Returns manifest metadata; 404 if missing. |
|
||||
| `GetPayload(GetPayloadRequest)` | Streams payload bytes (optionally decompressing). |
|
||||
| `ListManifests(ListManifestRequest)` | Enumerates manifests for tenant/kind with pagination. |
|
||||
| `DeleteManifest(DeleteManifestRequest)` | (Optional) Removes manifest/payload based on retention policies. |
|
||||
|
||||
.NET client wraps these calls and handles retries using Polly policies.
|
||||
|
||||
## 4. Library Responsibilities
|
||||
|
||||
Surface.FS library for .NET hosts provides:
|
||||
|
||||
- `ISurfaceManifestWriter` / `ISurfaceManifestReader` interfaces.
|
||||
- Content-addressed path builder (`SurfacePathBuilder`).
|
||||
- Tenant namespace isolation and bucket configuration (via Surface.Env).
|
||||
- Local cache management (using `SCANNER_SURFACE_CACHE_ROOT` and quota).
|
||||
- Metrics: `surface_manifest_put_seconds`, `surface_manifest_cache_hit_total`, etc.
|
||||
|
||||
## 5. Retention & Eviction
|
||||
|
||||
- Manifests include optional `expiresAt`; Worker defaults to 30 days for SBOM fragments, 7 days for entry traces.
|
||||
- Background job `SurfaceCacheMaintenanceService` evicts local cache entries exceeding quota, oldest-first.
|
||||
- Object storage retention policies are managed by DevOps; library exposes metrics but does not auto-delete unless instructed.
|
||||
|
||||
## 6. Offline Kit Handling
|
||||
|
||||
Offline kits include:
|
||||
|
||||
```
|
||||
offline/surface/
|
||||
manifests/
|
||||
tenants/<tenant>/<kind>/<digest>.json
|
||||
payloads/
|
||||
tenants/<tenant>/<kind>/<digest>.json.zst
|
||||
manifest-index.json
|
||||
```
|
||||
|
||||
Import script calls `PutManifest` for each manifest, verifying digests. This enables Zastava and Scheduler running offline to consume cached data without re-scanning.
|
||||
|
||||
## 7. Security & Tenancy
|
||||
|
||||
- Tenant ID is mandatory; Surface.Validation enforces match with Authority token.
|
||||
- Manifests/payloads stored in tenant-specific prefixes to prevent leakage.
|
||||
- Optional manifest signing (future) will use `Surface.Secrets` to load signing keys.
|
||||
- TLS enforced between hosts and Surface.FS endpoint; certificate pins configured via Surface.Env.
|
||||
|
||||
## 8. Observability
|
||||
|
||||
- Logs include manifest SHA, tenant, and kind; payload paths truncated for brevity.
|
||||
- Metrics exported via Prometheus with labels `{tenant, kind, result}`.
|
||||
- Tracing spans: `surface.fs.put`, `surface.fs.get`, `surface.fs.cache`.
|
||||
|
||||
## 9. Testing Strategy
|
||||
|
||||
- Unit tests for path builder, manifest serializer, and local cache eviction.
|
||||
- Integration tests using embedded RustFS or MinIO container to validate API interactions.
|
||||
- Offline kit tests verifying export/import cycle round-trips manifests and payloads.
|
||||
|
||||
## 10. Future Enhancements
|
||||
|
||||
- Manifest signing (DSSE) to support tamper detection in hostile environments.
|
||||
- Differential manifests to optimise large SBOM updates.
|
||||
- Cross-region replication for multi-site deployments.
|
||||
|
||||
## 11. References
|
||||
|
||||
- Surface.Env Design (`docs/modules/scanner/design/surface-env.md`)
|
||||
- Surface.Secrets Design (`docs/modules/scanner/design/surface-secrets.md`)
|
||||
- Surface.Validation Design (`docs/modules/scanner/design/surface-validation.md`)
|
||||
- Zastava Deployment Runbook (`docs/modules/devops/runbooks/zastava-deployment.md`)
|
||||
Reference in New Issue
Block a user