feat(rate-limiting): Implement core rate limiting functionality with configuration, decision-making, metrics, middleware, and service registration

- Add RateLimitConfig for configuration management with YAML binding support.
- Introduce RateLimitDecision to encapsulate the result of rate limit checks.
- Implement RateLimitMetrics for OpenTelemetry metrics tracking.
- Create RateLimitMiddleware for enforcing rate limits on incoming requests.
- Develop RateLimitService to orchestrate instance and environment rate limit checks.
- Add RateLimitServiceCollectionExtensions for dependency injection registration.
This commit is contained in:
master
2025-12-17 18:02:37 +02:00
parent 394b57f6bf
commit 8bbfe4d2d2
211 changed files with 47179 additions and 1590 deletions

View File

@@ -0,0 +1,682 @@
# Scanner WebService API — Score Proofs & Reachability Extensions
**Version**: 2.0
**Base URL**: `/api/v1/scanner`
**Authentication**: Bearer token (OpTok with DPoP/mTLS)
**Sprint**: SPRINT_3500_0002_0003, SPRINT_3500_0003_0003
---
## Overview
This document specifies API extensions to `Scanner.WebService` for:
1. Scan manifests and deterministic replay
2. Proof bundles (score proofs + reachability evidence)
3. Call-graph ingestion and reachability analysis
4. Unknowns management
**Design Principles**:
- All endpoints return canonical JSON (deterministic serialization)
- Idempotency via `Content-Digest` headers (SHA-256)
- DSSE signatures returned for all proof artifacts
- Offline-first (bundles downloadable for air-gap verification)
---
## Endpoints
### 1. Create Scan with Manifest
**POST** `/api/v1/scanner/scans`
**Description**: Creates a new scan with deterministic manifest.
**Request Body**:
```json
{
"artifactDigest": "sha256:abc123...",
"artifactPurl": "pkg:oci/myapp@sha256:abc123...",
"scannerVersion": "1.0.0",
"workerVersion": "1.0.0",
"concelierSnapshotHash": "sha256:feed123...",
"excititorSnapshotHash": "sha256:vex456...",
"latticePolicyHash": "sha256:policy789...",
"deterministic": true,
"seed": "AQIDBA==", // base64-encoded 32 bytes
"knobs": {
"maxDepth": "10",
"indirectCallResolution": "conservative"
}
}
```
**Response** (201 Created):
```json
{
"scanId": "550e8400-e29b-41d4-a716-446655440000",
"manifestHash": "sha256:manifest123...",
"createdAt": "2025-12-17T12:00:00Z",
"_links": {
"self": "/api/v1/scanner/scans/550e8400-e29b-41d4-a716-446655440000",
"manifest": "/api/v1/scanner/scans/550e8400-e29b-41d4-a716-446655440000/manifest"
}
}
```
**Headers**:
- `Content-Digest`: `sha256=<base64-hash>` (idempotency key)
- `Location`: `/api/v1/scanner/scans/{scanId}`
**Errors**:
- `400 Bad Request` — Invalid manifest (missing required fields)
- `409 Conflict` — Scan with same `manifestHash` already exists
- `422 Unprocessable Entity` — Snapshot hashes not found in Concelier/Excititor
**Idempotency**: Requests with same `Content-Digest` return existing scan (no duplicate creation).
---
### 2. Retrieve Scan Manifest
**GET** `/api/v1/scanner/scans/{scanId}/manifest`
**Description**: Retrieves the canonical JSON manifest with DSSE signature.
**Response** (200 OK):
```json
{
"manifest": {
"scanId": "550e8400-e29b-41d4-a716-446655440000",
"createdAtUtc": "2025-12-17T12:00:00Z",
"artifactDigest": "sha256:abc123...",
"artifactPurl": "pkg:oci/myapp@sha256:abc123...",
"scannerVersion": "1.0.0",
"workerVersion": "1.0.0",
"concelierSnapshotHash": "sha256:feed123...",
"excititorSnapshotHash": "sha256:vex456...",
"latticePolicyHash": "sha256:policy789...",
"deterministic": true,
"seed": "AQIDBA==",
"knobs": {
"maxDepth": "10"
}
},
"manifestHash": "sha256:manifest123...",
"dsseEnvelope": {
"payloadType": "application/vnd.stellaops.scan-manifest.v1+json",
"payload": "eyJzY2FuSWQiOiIuLi4ifQ==", // base64 canonical JSON
"signatures": [
{
"keyid": "ecdsa-p256-key-001",
"sig": "MEUCIQDx..."
}
]
}
}
```
**Headers**:
- `Content-Type`: `application/json`
- `ETag`: `"<manifestHash>"`
**Errors**:
- `404 Not Found` — Scan ID not found
**Caching**: `ETag` supports conditional `If-None-Match` requests (304 Not Modified).
---
### 3. Replay Score Computation
**POST** `/api/v1/scanner/scans/{scanId}/score/replay`
**Description**: Recomputes score proofs from manifest without rescanning binaries. Used when feeds/policies change.
**Request Body**:
```json
{
"overrides": {
"concelierSnapshotHash": "sha256:newfeed...", // Optional: use different feed
"excititorSnapshotHash": "sha256:newvex...", // Optional: use different VEX
"latticePolicyHash": "sha256:newpolicy..." // Optional: use different policy
}
}
```
**Response** (200 OK):
```json
{
"scanId": "550e8400-e29b-41d4-a716-446655440000",
"replayedAt": "2025-12-17T13:00:00Z",
"scoreProof": {
"rootHash": "sha256:proof123...",
"nodes": [
{
"id": "input-1",
"kind": "Input",
"ruleId": "inputs.v1",
"delta": 0.0,
"total": 0.0,
"nodeHash": "sha256:node1..."
},
{
"id": "delta-cvss",
"kind": "Delta",
"ruleId": "score.cvss_base.weighted",
"parentIds": ["input-1"],
"evidenceRefs": ["cvss:9.1"],
"delta": 0.50,
"total": 0.50,
"nodeHash": "sha256:node2..."
}
]
},
"proofBundleUri": "/api/v1/scanner/scans/550e8400-e29b-41d4-a716-446655440000/proofs/sha256:proof123...",
"_links": {
"bundle": "/api/v1/scanner/scans/550e8400-e29b-41d4-a716-446655440000/proofs/sha256:proof123..."
}
}
```
**Errors**:
- `404 Not Found` — Scan ID not found
- `422 Unprocessable Entity` — Override snapshot not found
**Use Case**: Nightly rescore job when Concelier publishes new advisory snapshot.
---
### 4. Upload Call-Graph
**POST** `/api/v1/scanner/scans/{scanId}/callgraphs`
**Description**: Uploads call-graph extracted by language-specific workers (.NET, Java, etc.).
**Request Body** (`application/json`):
```json
{
"schema": "stella.callgraph.v1",
"language": "dotnet",
"artifacts": [
{
"artifactKey": "MyApp.WebApi.dll",
"kind": "assembly",
"sha256": "sha256:artifact123..."
}
],
"nodes": [
{
"nodeId": "sha256:node1...",
"artifactKey": "MyApp.WebApi.dll",
"symbolKey": "MyApp.Controllers.OrdersController::Get(System.Guid)",
"visibility": "public",
"isEntrypointCandidate": true
}
],
"edges": [
{
"from": "sha256:node1...",
"to": "sha256:node2...",
"kind": "static",
"reason": "direct_call",
"weight": 1.0
}
],
"entrypoints": [
{
"nodeId": "sha256:node1...",
"kind": "http",
"route": "/api/orders/{id}",
"framework": "aspnetcore"
}
]
}
```
**Headers**:
- `Content-Digest`: `sha256=<hash>` (idempotency)
**Response** (202 Accepted):
```json
{
"scanId": "550e8400-e29b-41d4-a716-446655440000",
"callGraphDigest": "sha256:cg123...",
"nodesCount": 1234,
"edgesCount": 5678,
"entrypointsCount": 12,
"status": "accepted",
"_links": {
"reachability": "/api/v1/scanner/scans/550e8400-e29b-41d4-a716-446655440000/reachability/compute"
}
}
```
**Errors**:
- `400 Bad Request` — Invalid call-graph schema
- `404 Not Found` — Scan ID not found
- `413 Payload Too Large` — Call-graph >100MB
**Idempotency**: Same `Content-Digest` → returns existing call-graph.
---
### 5. Compute Reachability
**POST** `/api/v1/scanner/scans/{scanId}/reachability/compute`
**Description**: Triggers reachability analysis for uploaded call-graph + SBOM + vulnerabilities.
**Request Body**: Empty (uses existing scan data)
**Response** (202 Accepted):
```json
{
"scanId": "550e8400-e29b-41d4-a716-446655440000",
"jobId": "reachability-job-001",
"status": "queued",
"estimatedDuration": "30s",
"_links": {
"status": "/api/v1/scanner/jobs/reachability-job-001",
"results": "/api/v1/scanner/scans/550e8400-e29b-41d4-a716-446655440000/reachability/findings"
}
}
```
**Polling**: Use `GET /api/v1/scanner/jobs/{jobId}` to check status.
**Errors**:
- `404 Not Found` — Scan ID not found
- `422 Unprocessable Entity` — Call-graph not uploaded yet
---
### 6. Get Reachability Findings
**GET** `/api/v1/scanner/scans/{scanId}/reachability/findings`
**Description**: Retrieves reachability verdicts for all vulnerabilities.
**Query Parameters**:
- `status` (optional): Filter by `REACHABLE`, `UNREACHABLE`, `POSSIBLY_REACHABLE`, `UNKNOWN`
- `cveId` (optional): Filter by CVE ID
**Response** (200 OK):
```json
{
"scanId": "550e8400-e29b-41d4-a716-446655440000",
"computedAt": "2025-12-17T12:30:00Z",
"findings": [
{
"cveId": "CVE-2024-1234",
"purl": "pkg:npm/lodash@4.17.20",
"status": "REACHABLE_STATIC",
"confidence": 0.70,
"path": [
{
"nodeId": "sha256:entrypoint...",
"symbolKey": "MyApp.Controllers.OrdersController::Get(System.Guid)"
},
{
"nodeId": "sha256:intermediate...",
"symbolKey": "MyApp.Services.OrderService::Process(Order)"
},
{
"nodeId": "sha256:vuln...",
"symbolKey": "Lodash.merge(Object, Object)"
}
],
"evidence": {
"pathLength": 3,
"staticEdgesOnly": true,
"runtimeConfirmed": false
},
"_links": {
"explain": "/api/v1/scanner/scans/{scanId}/reachability/explain?cve=CVE-2024-1234&purl=pkg:npm/lodash@4.17.20"
}
}
],
"summary": {
"total": 45,
"reachable": 3,
"unreachable": 38,
"possiblyReachable": 4,
"unknown": 0
}
}
```
**Errors**:
- `404 Not Found` — Scan ID not found or reachability not computed
---
### 7. Explain Reachability
**GET** `/api/v1/scanner/scans/{scanId}/reachability/explain`
**Description**: Provides detailed explanation for a reachability verdict.
**Query Parameters**:
- `cve` (required): CVE ID
- `purl` (required): Package URL
**Response** (200 OK):
```json
{
"cveId": "CVE-2024-1234",
"purl": "pkg:npm/lodash@4.17.20",
"status": "REACHABLE_STATIC",
"confidence": 0.70,
"explanation": {
"shortestPath": [
{
"depth": 0,
"nodeId": "sha256:entry...",
"symbolKey": "MyApp.Controllers.OrdersController::Get(System.Guid)",
"entrypointKind": "http",
"route": "/api/orders/{id}"
},
{
"depth": 1,
"nodeId": "sha256:inter...",
"symbolKey": "MyApp.Services.OrderService::Process(Order)",
"edgeKind": "static",
"edgeReason": "direct_call"
},
{
"depth": 2,
"nodeId": "sha256:vuln...",
"symbolKey": "Lodash.merge(Object, Object)",
"edgeKind": "static",
"edgeReason": "direct_call",
"vulnerableFunction": true
}
],
"whyReachable": [
"Static call path exists from HTTP entrypoint /api/orders/{id}",
"All edges are statically proven (no heuristics)",
"Vulnerable function Lodash.merge() is directly invoked"
],
"confidenceFactors": {
"staticPathExists": 0.50,
"noHeuristicEdges": 0.20,
"runtimeConfirmed": 0.00
}
},
"alternativePaths": 2, // Number of other paths found
"_links": {
"callGraph": "/api/v1/scanner/scans/{scanId}/callgraphs/sha256:cg123.../graph.json"
}
}
```
**Errors**:
- `404 Not Found` — Scan, CVE, or PURL not found
---
### 8. Fetch Proof Bundle
**GET** `/api/v1/scanner/scans/{scanId}/proofs/{rootHash}`
**Description**: Downloads proof bundle zip archive for offline verification.
**Path Parameters**:
- `rootHash`: Proof root hash (e.g., `sha256:proof123...`)
**Response** (200 OK):
**Headers**:
- `Content-Type`: `application/zip`
- `Content-Disposition`: `attachment; filename="proof-{scanId}-{rootHash}.zip"`
- `X-Proof-Root-Hash`: `{rootHash}`
- `X-Manifest-Hash`: `{manifestHash}`
**Body**: Binary zip archive containing:
- `manifest.json` — Canonical scan manifest
- `manifest.dsse.json` — DSSE signature of manifest
- `score_proof.json` — Proof ledger (array of ProofNodes)
- `proof_root.dsse.json` — DSSE signature of proof root
- `meta.json` — Metadata (created timestamp, etc.)
**Errors**:
- `404 Not Found` — Scan or proof root hash not found
**Use Case**: Air-gap verification (`stella proof verify --bundle proof.zip`).
---
### 9. List Unknowns
**GET** `/api/v1/scanner/unknowns`
**Description**: Lists unknowns (missing evidence) ranked by priority.
**Query Parameters**:
- `band` (optional): Filter by `HOT`, `WARM`, `COLD`
- `limit` (optional): Max results (default: 100, max: 1000)
- `offset` (optional): Pagination offset
**Response** (200 OK):
```json
{
"unknowns": [
{
"unknownId": "unk-001",
"pkgId": "pkg:npm/lodash",
"pkgVersion": "4.17.20",
"digestAnchor": "sha256:...",
"reasons": ["missing_vex", "ambiguous_version"],
"score": 0.72,
"band": "HOT",
"popularity": 0.85,
"potentialExploit": 0.60,
"uncertainty": 0.75,
"evidence": {
"deployments": 42,
"epss": 0.58,
"kev": false
},
"createdAt": "2025-12-15T10:00:00Z",
"_links": {
"escalate": "/api/v1/scanner/unknowns/unk-001/escalate"
}
}
],
"pagination": {
"total": 156,
"limit": 100,
"offset": 0,
"next": "/api/v1/scanner/unknowns?band=HOT&limit=100&offset=100"
}
}
```
**Errors**:
- `400 Bad Request` — Invalid band value
---
### 10. Escalate Unknown to Rescan
**POST** `/api/v1/scanner/unknowns/{unknownId}/escalate`
**Description**: Escalates an unknown to trigger immediate rescan/re-analysis.
**Request Body**: Empty
**Response** (202 Accepted):
```json
{
"unknownId": "unk-001",
"escalatedAt": "2025-12-17T12:00:00Z",
"rescanJobId": "rescan-job-001",
"status": "queued",
"_links": {
"job": "/api/v1/scanner/jobs/rescan-job-001"
}
}
```
**Errors**:
- `404 Not Found` — Unknown ID not found
- `409 Conflict` — Unknown already escalated (rescan in progress)
---
## Data Models
### ScanManifest
See `src/__Libraries/StellaOps.Scanner.Core/Models/ScanManifest.cs` for full definition.
### ProofNode
```typescript
interface ProofNode {
id: string;
kind: "Input" | "Transform" | "Delta" | "Score";
ruleId: string;
parentIds: string[];
evidenceRefs: string[];
delta: number;
total: number;
actor: string;
tsUtc: string; // ISO 8601
seed: string; // base64
nodeHash: string; // sha256:...
}
```
### DsseEnvelope
```typescript
interface DsseEnvelope {
payloadType: string;
payload: string; // base64 canonical JSON
signatures: DsseSignature[];
}
interface DsseSignature {
keyid: string;
sig: string; // base64
}
```
### ReachabilityStatus
```typescript
enum ReachabilityStatus {
UNREACHABLE = "UNREACHABLE",
POSSIBLY_REACHABLE = "POSSIBLY_REACHABLE",
REACHABLE_STATIC = "REACHABLE_STATIC",
REACHABLE_PROVEN = "REACHABLE_PROVEN",
UNKNOWN = "UNKNOWN"
}
```
---
## Error Responses
All errors follow RFC 7807 (Problem Details):
```json
{
"type": "https://stella-ops.org/errors/scan-not-found",
"title": "Scan Not Found",
"status": 404,
"detail": "Scan ID '550e8400-e29b-41d4-a716-446655440000' does not exist.",
"instance": "/api/v1/scanner/scans/550e8400-e29b-41d4-a716-446655440000",
"traceId": "trace-001"
}
```
### Error Types
| Type | Status | Description |
|------|--------|-------------|
| `scan-not-found` | 404 | Scan ID not found |
| `invalid-manifest` | 400 | Manifest validation failed |
| `duplicate-scan` | 409 | Scan with same manifest hash exists |
| `snapshot-not-found` | 422 | Concelier/Excititor snapshot not found |
| `callgraph-not-uploaded` | 422 | Call-graph required before reachability |
| `payload-too-large` | 413 | Request body exceeds size limit |
| `proof-not-found` | 404 | Proof root hash not found |
| `unknown-not-found` | 404 | Unknown ID not found |
| `escalation-conflict` | 409 | Unknown already escalated |
---
## Rate Limiting
**Limits**:
- `POST /scans`: 100 requests/hour per tenant
- `POST /scans/{id}/score/replay`: 1000 requests/hour per tenant
- `POST /callgraphs`: 100 requests/hour per tenant
- `POST /reachability/compute`: 100 requests/hour per tenant
- `GET` endpoints: 10,000 requests/hour per tenant
**Headers**:
- `X-RateLimit-Limit`: Maximum requests per window
- `X-RateLimit-Remaining`: Remaining requests
- `X-RateLimit-Reset`: Unix timestamp when limit resets
**Error** (429 Too Many Requests):
```json
{
"type": "https://stella-ops.org/errors/rate-limit-exceeded",
"title": "Rate Limit Exceeded",
"status": 429,
"detail": "Exceeded 100 requests/hour for POST /scans. Retry after 1234567890.",
"retryAfter": 1234567890
}
```
---
## Webhooks (Future)
**Planned for Sprint 3500.0004.0003**:
```
POST /api/v1/scanner/webhooks
Register webhook for events: scan.completed, reachability.computed, unknown.escalated
```
---
## OpenAPI Specification
**File**: `src/Api/StellaOps.Api.OpenApi/scanner/openapi.yaml`
Update with new endpoints (Sprint 3500.0002.0003).
---
## References
- `SPRINT_3500_0002_0001_score_proofs_foundations.md` — Implementation sprint
- `SPRINT_3500_0002_0003_proof_replay_api.md` — API implementation sprint
- `SPRINT_3500_0003_0003_graph_attestations_rekor.md` — Reachability API sprint
- `docs/07_HIGH_LEVEL_ARCHITECTURE.md` — API contracts section
- `docs/db/schemas/scanner_schema_specification.md` — Database schema
---
**Last Updated**: 2025-12-17
**API Version**: 2.0
**Next Review**: Sprint 3500.0004.0001 (CLI integration)