Restructure solution layout by module

This commit is contained in:
master
2025-10-28 15:10:40 +02:00
parent 95daa159c4
commit d870da18ce
4103 changed files with 192899 additions and 187024 deletions

View File

@@ -0,0 +1,68 @@
## StellaOps.Concelier.Connector.Acsc
Australian Cyber Security Centre (ACSC) connector that ingests RSS/Atom advisories, sanitises embedded HTML, and maps entries into canonical `Advisory` records for Concelier.
### Configuration
Settings live under `concelier:sources:acsc` (see `AcscOptions`):
| Setting | Description | Default |
| --- | --- | --- |
| `baseEndpoint` | Base URI for direct ACSC requests (trailing slash required). | `https://www.cyber.gov.au/` |
| `relayEndpoint` | Optional relay host to fall back to when Akamai refuses HTTP/2. | empty |
| `preferRelayByDefault` | Default endpoint preference when no cursor state exists. | `false` |
| `enableRelayFallback` | Allows automatic relay fallback when direct fetch fails. | `true` |
| `forceRelay` | Forces all fetches through the relay (skips direct attempts). | `false` |
| `feeds` | Array of feed descriptors (`slug`, `relativePath`, `enabled`). | alerts/advisories enabled |
| `requestTimeout` | Per-request timeout override. | 45 seconds |
| `failureBackoff` | Backoff window when fetch fails. | 5 minutes |
| `initialBackfill` | Sliding window used to seed published cursors. | 120 days |
| `userAgent` | Outbound `User-Agent` header. | `StellaOps/Concelier (+https://stella-ops.org)` |
| `requestVersion`/`versionPolicy` | HTTP version negotiation knobs. | HTTP/2 with downgrade |
The dependency injection routine registers the connector plus scheduled jobs:
| Job | Cron | Purpose |
| --- | --- | --- |
| `source:acsc:fetch` | `7,37 * * * *` | Fetch RSS/Atom feeds (direct + relay fallback). |
| `source:acsc:parse` | `12,42 * * * *` | Persist sanitised DTOs (`acsc.feed.v1`). |
| `source:acsc:map` | `17,47 * * * *` | Map DTO entries into canonical advisories. |
| `source:acsc:probe` | `25,55 * * * *` | Verify direct endpoint health and adjust cursor preference. |
### Metrics
Emitted via `AcscDiagnostics` (`Meter` = `StellaOps.Concelier.Connector.Acsc`):
| Instrument | Unit | Description |
| --- | --- | --- |
| `acsc.fetch.attempts` | operations | Feed fetch attempts (tags: `feed`, `mode`). |
| `acsc.fetch.success` | operations | Successful fetches. |
| `acsc.fetch.failures` | operations | Failed fetches before retry backoff. |
| `acsc.fetch.unchanged` | operations | 304 Not Modified responses. |
| `acsc.fetch.fallbacks` | operations | Relay fallbacks triggered (`reason` tag). |
| `acsc.cursor.published_updates` | feeds | Published cursor updates per feed slug. |
| `acsc.parse.attempts` | documents | Parse attempts per feed. |
| `acsc.parse.success` | documents | Successful RSS → DTO conversions. |
| `acsc.parse.failures` | documents | Parse failures (tags: `feed`, `reason`). |
| `acsc.map.success` | advisories | Advisories emitted from a mapping pass. |
### Logging
Key log messages include:
- Fetch successes/failures, HTTP status codes, and relay fallbacks.
- Parse failures with reasons (download, schema, sanitisation).
- Mapping summaries showing advisory counts per document.
- Probe results toggling relay usage.
Logs include feed slug metadata for troubleshooting parallel ingestion.
### Tests & fixtures
`StellaOps.Concelier.Connector.Acsc.Tests` exercises the fetch→parse→map pipeline using canned RSS content. Deterministic snapshots live in `Acsc/Fixtures`. To refresh them after intentional behavioural changes:
```bash
UPDATE_ACSC_FIXTURES=1 dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Acsc.Tests/StellaOps.Concelier.Connector.Acsc.Tests.csproj
```
Remember to review the generated `.actual.json` files when assertions fail without fixture updates.
### Operational notes
- Keep the relay endpoint allowlisted for air-gapped deployments; the probe job will automatically switch back to direct fetching when Akamai stabilises.
- Mapping currently emits vendor `affectedPackages` from “Systems/Products affected” fields; expand range primitives once structured version data appears in ACSC feeds.
- The connector is offline-friendly—no outbound calls beyond the configured feeds.