Resolve Concelier/Excititor merge conflicts
This commit is contained in:
root
108
docs/dev/SCANNER_CACHE_CONFIGURATION.md
Normal file
108
docs/dev/SCANNER_CACHE_CONFIGURATION.md
Normal file
@@ -0,0 +1,108 @@
|
||||
# Scanner Cache Configuration Guide
|
||||
|
||||
The scanner cache stores layer-level SBOM fragments and file content that can be reused across scans. This document explains how to configure and operate the cache subsystem introduced in Sprint 10 (Group SP10-G5).
|
||||
|
||||
## 1. Overview
|
||||
|
||||
- **Layer cache** persists SBOM fragments per layer digest under `<root>/layers/<digest>/` with deterministic metadata (`meta.json`).
|
||||
- **File CAS** (content-addressable store) keeps deduplicated blobs (e.g., analyzer fixtures, imported SBOM layers) under `<root>/cas/<prefix>/<hash>/`.
|
||||
- **Maintenance** runs via `ScannerCacheMaintenanceService`, evicting expired entries and compacting the cache to stay within size limits.
|
||||
- **Metrics** emit on the `StellaOps.Scanner.Cache` meter with counters for hits, misses, evictions, and byte histograms.
|
||||
- **Offline workflows** use the CAS import/export helpers to package cache warmups inside the Offline Kit.
|
||||
|
||||
## 2. Configuration keys (`scanner:cache`)
|
||||
|
||||
| Key | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `enabled` | `true` | Globally disable cache if `false`. |
|
||||
| `rootPath` | `cache/scanner` | Base directory for cache data. Use an SSD-backed path for best warm-scan latency. |
|
||||
| `layersDirectoryName` | `layers` | Subdirectory for layer cache entries. |
|
||||
| `fileCasDirectoryName` | `cas` | Subdirectory for file CAS entries. |
|
||||
| `layerTtl` | `45.00:00:00` | Time-to-live for layer cache entries (`TimeSpan`). `0` disables TTL eviction. |
|
||||
| `fileTtl` | `30.00:00:00` | Time-to-live for CAS entries. `0` disables TTL eviction. |
|
||||
| `maxBytes` | `5368709120` (5 GiB) | Hard cap for combined cache footprint. Compaction trims data back to `warmBytesThreshold`. |
|
||||
| `warmBytesThreshold` | `maxBytes / 5` | Target size after compaction. |
|
||||
| `coldBytesThreshold` | `maxBytes * 0.8` | Upper bound that triggers compaction. |
|
||||
| `enableAutoEviction` | `true` | If `false`, callers must invoke `ILayerCacheStore.CompactAsync` / `IFileContentAddressableStore.CompactAsync` manually. |
|
||||
| `maintenanceInterval` | `00:15:00` | Interval for the maintenance hosted service. |
|
||||
| `enableFileCas` | `true` | Disable to prevent CAS usage (APIs throw on `PutAsync`). |
|
||||
| `importDirectory` / `exportDirectory` | `null` | Optional defaults for offline import/export tooling. |
|
||||
|
||||
> **Tip:** configure `scanner:cache:rootPath` to a dedicated volume and mount it into worker containers when running in Kubernetes or Nomad.
|
||||
|
||||
## 3. Metrics
|
||||
|
||||
Instrumentation lives in `ScannerCacheMetrics` on meter `StellaOps.Scanner.Cache`.
|
||||
|
||||
| Instrument | Unit | Description |
|
||||
| --- | --- | --- |
|
||||
| `scanner.layer_cache_hits_total` | count | Layer cache hit counter. Tag: `layer`. |
|
||||
| `scanner.layer_cache_misses_total` | count | Layer cache miss counter. Tag: `layer`. |
|
||||
| `scanner.layer_cache_evictions_total` | count | Layer entries evicted due to TTL or compaction. Tag: `layer`. |
|
||||
| `scanner.layer_cache_bytes` | bytes | Histogram of per-entry payload size when stored. |
|
||||
| `scanner.file_cas_hits_total` | count | File CAS hit counter. Tag: `sha256`. |
|
||||
| `scanner.file_cas_misses_total` | count | File CAS miss counter. Tag: `sha256`. |
|
||||
| `scanner.file_cas_evictions_total` | count | CAS eviction counter. Tag: `sha256`. |
|
||||
| `scanner.file_cas_bytes` | bytes | Histogram of CAS payload sizes on insert. |
|
||||
|
||||
## 4. Import / Export workflow
|
||||
|
||||
1. **Export warm cache**
|
||||
```bash
|
||||
dotnet tool run stellaops-cache export --destination ./offline-kit/cache
|
||||
```
|
||||
Internally this calls `IFileContentAddressableStore.ExportAsync` which copies each CAS entry (metadata + `content.bin`).
|
||||
|
||||
2. **Import on air-gapped hosts**
|
||||
```bash
|
||||
dotnet tool run stellaops-cache import --source ./offline-kit/cache
|
||||
```
|
||||
The import API merges newer metadata and skips older snapshots automatically.
|
||||
|
||||
3. **Layer cache seeding**
|
||||
Layer cache entries are deterministic and can be packaged the same way (copy `<root>/layers`). For now we keep seeding optional because layers are larger; follow-up tooling can compress directories as needed.
|
||||
|
||||
## 5. Hosted maintenance loop
|
||||
|
||||
`ScannerCacheMaintenanceService` runs as a background service within Scanner Worker or WebService hosts when `AddScannerCache` is registered. Behaviour:
|
||||
|
||||
- At startup it performs an immediate eviction/compaction run.
|
||||
- Every `maintenanceInterval` it triggers:
|
||||
- `ILayerCacheStore.EvictExpiredAsync`
|
||||
- `ILayerCacheStore.CompactAsync`
|
||||
- `IFileContentAddressableStore.EvictExpiredAsync`
|
||||
- `IFileContentAddressableStore.CompactAsync`
|
||||
- Failures are logged at `Error` with preserved stack traces; the next tick continues normally.
|
||||
|
||||
Set `enableAutoEviction=false` when hosting the cache inside ephemeral build pipelines that want to drive eviction explicitly.
|
||||
|
||||
## 6. API surface summary
|
||||
|
||||
```csharp
|
||||
public interface ILayerCacheStore
|
||||
{
|
||||
ValueTask<LayerCacheEntry?> TryGetAsync(string layerDigest, CancellationToken ct = default);
|
||||
Task<LayerCacheEntry> PutAsync(LayerCachePutRequest request, CancellationToken ct = default);
|
||||
Task RemoveAsync(string layerDigest, CancellationToken ct = default);
|
||||
Task<int> EvictExpiredAsync(CancellationToken ct = default);
|
||||
Task<int> CompactAsync(CancellationToken ct = default);
|
||||
Task<Stream?> OpenArtifactAsync(string layerDigest, string artifactName, CancellationToken ct = default);
|
||||
}
|
||||
|
||||
public interface IFileContentAddressableStore
|
||||
{
|
||||
ValueTask<FileCasEntry?> TryGetAsync(string sha256, CancellationToken ct = default);
|
||||
Task<FileCasEntry> PutAsync(FileCasPutRequest request, CancellationToken ct = default);
|
||||
Task<bool> RemoveAsync(string sha256, CancellationToken ct = default);
|
||||
Task<int> EvictExpiredAsync(CancellationToken ct = default);
|
||||
Task<int> CompactAsync(CancellationToken ct = default);
|
||||
Task<int> ExportAsync(string destinationDirectory, CancellationToken ct = default);
|
||||
Task<int> ImportAsync(string sourceDirectory, CancellationToken ct = default);
|
||||
}
|
||||
```
|
||||
|
||||
Register both stores via `services.AddScannerCache(configuration);` in WebService or Worker hosts.
|
||||
|
||||
---
|
||||
|
||||
_Last updated: 2025-10-19_
|
||||
Reference in New Issue
Block a user