109 lines
6.0 KiB
Markdown
109 lines
6.0 KiB
Markdown
# Scanner Cache Configuration Guide
|
|
|
|
The scanner cache stores layer-level SBOM fragments and file content that can be reused across scans. This document explains how to configure and operate the cache subsystem introduced in Sprint 10 (Group SP10-G5).
|
|
|
|
## 1. Overview
|
|
|
|
- **Layer cache** persists SBOM fragments per layer digest under `<root>/layers/<digest>/` with deterministic metadata (`meta.json`).
|
|
- **File CAS** (content-addressable store) keeps deduplicated blobs (e.g., analyzer fixtures, imported SBOM layers) under `<root>/cas/<prefix>/<hash>/`.
|
|
- **Maintenance** runs via `ScannerCacheMaintenanceService`, evicting expired entries and compacting the cache to stay within size limits.
|
|
- **Metrics** emit on the `StellaOps.Scanner.Cache` meter with counters for hits, misses, evictions, and byte histograms.
|
|
- **Offline workflows** use the CAS import/export helpers to package cache warmups inside the Offline Kit.
|
|
|
|
## 2. Configuration keys (`scanner:cache`)
|
|
|
|
| Key | Default | Description |
|
|
| --- | --- | --- |
|
|
| `enabled` | `true` | Globally disable cache if `false`. |
|
|
| `rootPath` | `cache/scanner` | Base directory for cache data. Use an SSD-backed path for best warm-scan latency. |
|
|
| `layersDirectoryName` | `layers` | Subdirectory for layer cache entries. |
|
|
| `fileCasDirectoryName` | `cas` | Subdirectory for file CAS entries. |
|
|
| `layerTtl` | `45.00:00:00` | Time-to-live for layer cache entries (`TimeSpan`). `0` disables TTL eviction. |
|
|
| `fileTtl` | `30.00:00:00` | Time-to-live for CAS entries. `0` disables TTL eviction. |
|
|
| `maxBytes` | `5368709120` (5 GiB) | Hard cap for combined cache footprint. Compaction trims data back to `warmBytesThreshold`. |
|
|
| `warmBytesThreshold` | `maxBytes / 5` | Target size after compaction. |
|
|
| `coldBytesThreshold` | `maxBytes * 0.8` | Upper bound that triggers compaction. |
|
|
| `enableAutoEviction` | `true` | If `false`, callers must invoke `ILayerCacheStore.CompactAsync` / `IFileContentAddressableStore.CompactAsync` manually. |
|
|
| `maintenanceInterval` | `00:15:00` | Interval for the maintenance hosted service. |
|
|
| `enableFileCas` | `true` | Disable to prevent CAS usage (APIs throw on `PutAsync`). |
|
|
| `importDirectory` / `exportDirectory` | `null` | Optional defaults for offline import/export tooling. |
|
|
|
|
> **Tip:** configure `scanner:cache:rootPath` to a dedicated volume and mount it into worker containers when running in Kubernetes or Nomad.
|
|
|
|
## 3. Metrics
|
|
|
|
Instrumentation lives in `ScannerCacheMetrics` on meter `StellaOps.Scanner.Cache`.
|
|
|
|
| Instrument | Unit | Description |
|
|
| --- | --- | --- |
|
|
| `scanner.layer_cache_hits_total` | count | Layer cache hit counter. Tag: `layer`. |
|
|
| `scanner.layer_cache_misses_total` | count | Layer cache miss counter. Tag: `layer`. |
|
|
| `scanner.layer_cache_evictions_total` | count | Layer entries evicted due to TTL or compaction. Tag: `layer`. |
|
|
| `scanner.layer_cache_bytes` | bytes | Histogram of per-entry payload size when stored. |
|
|
| `scanner.file_cas_hits_total` | count | File CAS hit counter. Tag: `sha256`. |
|
|
| `scanner.file_cas_misses_total` | count | File CAS miss counter. Tag: `sha256`. |
|
|
| `scanner.file_cas_evictions_total` | count | CAS eviction counter. Tag: `sha256`. |
|
|
| `scanner.file_cas_bytes` | bytes | Histogram of CAS payload sizes on insert. |
|
|
|
|
## 4. Import / Export workflow
|
|
|
|
1. **Export warm cache**
|
|
```bash
|
|
dotnet tool run stellaops-cache export --destination ./offline-kit/cache
|
|
```
|
|
Internally this calls `IFileContentAddressableStore.ExportAsync` which copies each CAS entry (metadata + `content.bin`).
|
|
|
|
2. **Import on air-gapped hosts**
|
|
```bash
|
|
dotnet tool run stellaops-cache import --source ./offline-kit/cache
|
|
```
|
|
The import API merges newer metadata and skips older snapshots automatically.
|
|
|
|
3. **Layer cache seeding**
|
|
Layer cache entries are deterministic and can be packaged the same way (copy `<root>/layers`). For now we keep seeding optional because layers are larger; follow-up tooling can compress directories as needed.
|
|
|
|
## 5. Hosted maintenance loop
|
|
|
|
`ScannerCacheMaintenanceService` runs as a background service within Scanner Worker or WebService hosts when `AddScannerCache` is registered. Behaviour:
|
|
|
|
- At startup it performs an immediate eviction/compaction run.
|
|
- Every `maintenanceInterval` it triggers:
|
|
- `ILayerCacheStore.EvictExpiredAsync`
|
|
- `ILayerCacheStore.CompactAsync`
|
|
- `IFileContentAddressableStore.EvictExpiredAsync`
|
|
- `IFileContentAddressableStore.CompactAsync`
|
|
- Failures are logged at `Error` with preserved stack traces; the next tick continues normally.
|
|
|
|
Set `enableAutoEviction=false` when hosting the cache inside ephemeral build pipelines that want to drive eviction explicitly.
|
|
|
|
## 6. API surface summary
|
|
|
|
```csharp
|
|
public interface ILayerCacheStore
|
|
{
|
|
ValueTask<LayerCacheEntry?> TryGetAsync(string layerDigest, CancellationToken ct = default);
|
|
Task<LayerCacheEntry> PutAsync(LayerCachePutRequest request, CancellationToken ct = default);
|
|
Task RemoveAsync(string layerDigest, CancellationToken ct = default);
|
|
Task<int> EvictExpiredAsync(CancellationToken ct = default);
|
|
Task<int> CompactAsync(CancellationToken ct = default);
|
|
Task<Stream?> OpenArtifactAsync(string layerDigest, string artifactName, CancellationToken ct = default);
|
|
}
|
|
|
|
public interface IFileContentAddressableStore
|
|
{
|
|
ValueTask<FileCasEntry?> TryGetAsync(string sha256, CancellationToken ct = default);
|
|
Task<FileCasEntry> PutAsync(FileCasPutRequest request, CancellationToken ct = default);
|
|
Task<bool> RemoveAsync(string sha256, CancellationToken ct = default);
|
|
Task<int> EvictExpiredAsync(CancellationToken ct = default);
|
|
Task<int> CompactAsync(CancellationToken ct = default);
|
|
Task<int> ExportAsync(string destinationDirectory, CancellationToken ct = default);
|
|
Task<int> ImportAsync(string sourceDirectory, CancellationToken ct = default);
|
|
}
|
|
```
|
|
|
|
Register both stores via `services.AddScannerCache(configuration);` in WebService or Worker hosts.
|
|
|
|
---
|
|
|
|
_Last updated: 2025-10-19_
|