50abd2137f
Add 12 new sprint files (Integrations, Graph, JobEngine, FE, Router, AdvisoryAI), archive completed scheduler UI sprint, update module architecture docs (router, graph, jobengine, web, integrations), and add Gitea entrypoint script for local dev. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
365 lines
16 KiB
Markdown
365 lines
16 KiB
Markdown
# Integrations Architecture
|
|
|
|
## Overview
|
|
|
|
The Integrations module provides a unified catalog for external service connections including SCM providers (GitHub, GitLab, Bitbucket), container registries (Harbor, ECR, GCR, ACR), CI systems, and runtime hosts. It implements a plugin-based architecture for extensibility while maintaining consistent security and observability patterns.
|
|
|
|
## Architecture Diagram
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ Integrations Module │
|
|
├─────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌──────────────────────┐ ┌──────────────────────┐ │
|
|
│ │ WebService Host │ │ Plugin Loader │ │
|
|
│ │ (ASP.NET Core) │────│ (DI Registration) │ │
|
|
│ └──────────┬───────────┘ └──────────┬───────────┘ │
|
|
│ │ │ │
|
|
│ ┌──────────▼───────────────────────────▼───────────┐ │
|
|
│ │ Integration Catalog │ │
|
|
│ │ - Registration CRUD │ │
|
|
│ │ - Health Polling │ │
|
|
│ │ - Test Connection │ │
|
|
│ └──────────┬───────────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ┌──────────▼───────────────────────────────────────┐ │
|
|
│ │ Plugin Contracts │ │
|
|
│ │ - IIntegrationConnectorPlugin │ │
|
|
│ │ - IScmAnnotationClient │ │
|
|
│ │ - IRegistryConnector │ │
|
|
│ └──────────────────────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ┌──────────▼───────────────────────────────────────┐ │
|
|
│ │ Provider Plugins │ │
|
|
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │
|
|
│ │ │ GitHub │ │ GitLab │ │ Harbor │ │ ECR │ │ │
|
|
│ │ │ App │ │ │ │ │ │ │ │ │
|
|
│ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │
|
|
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │
|
|
│ │ │ GCR │ │ ACR │ │InMemory │ │ │
|
|
│ │ │ │ │ │ │ (test) │ │ │
|
|
│ │ └─────────┘ └─────────┘ └─────────┘ │ │
|
|
│ └──────────────────────────────────────────────────┘ │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Core Components
|
|
|
|
### Integration Catalog
|
|
|
|
The central registry for all external service connections:
|
|
|
|
- **Registration**: Store connection configuration with encrypted credentials
|
|
- **Health Monitoring**: Periodic health checks with status tracking
|
|
- **Test Connection**: On-demand connectivity verification
|
|
- **Lifecycle Events**: Emit events for Scheduler/Signals integration
|
|
|
|
### Plugin System
|
|
|
|
Extensible plugin architecture for provider support:
|
|
|
|
```csharp
|
|
public interface IIntegrationConnectorPlugin : IAvailabilityPlugin
|
|
{
|
|
IntegrationType Type { get; }
|
|
IntegrationProvider Provider { get; }
|
|
Task<TestConnectionResult> TestConnectionAsync(IntegrationConfig config, CancellationToken ct);
|
|
Task<HealthCheckResult> CheckHealthAsync(IntegrationConfig config, CancellationToken ct);
|
|
}
|
|
```
|
|
|
|
### SCM Annotation Client
|
|
|
|
Interface for PR/MR comments and status checks:
|
|
|
|
```csharp
|
|
public interface IScmAnnotationClient
|
|
{
|
|
Task<ScmOperationResult<ScmCommentResponse>> PostCommentAsync(
|
|
ScmCommentRequest request, CancellationToken ct);
|
|
|
|
Task<ScmOperationResult<ScmStatusResponse>> PostStatusAsync(
|
|
ScmStatusRequest request, CancellationToken ct);
|
|
|
|
Task<ScmOperationResult<ScmCheckRunResponse>> CreateCheckRunAsync(
|
|
ScmCheckRunRequest request, CancellationToken ct);
|
|
}
|
|
```
|
|
|
|
## SCM Annotation Architecture
|
|
|
|
### Comment and Status Flow
|
|
|
|
```
|
|
┌────────────┐ ┌─────────────┐ ┌────────────────┐ ┌──────────┐
|
|
│ Scanner │────▶│ Integrations│────▶│ SCM Annotation │────▶│ GitHub/ │
|
|
│ Service │ │ Service │ │ Client │ │ GitLab │
|
|
└────────────┘ └─────────────┘ └────────────────┘ └──────────┘
|
|
│ │
|
|
│ ┌─────────────────┐ │
|
|
└────────▶│ Annotation │◀───────────┘
|
|
│ Payload Builder │
|
|
└─────────────────┘
|
|
```
|
|
|
|
### Supported Operations
|
|
|
|
| Operation | GitHub | GitLab |
|
|
|-----------|--------|--------|
|
|
| PR/MR Comment | Issue comment / Review comment | MR Note / Discussion |
|
|
| Commit Status | Commit status API | Commit status API |
|
|
| Check Run | Checks API with annotations | Pipeline status (emulated) |
|
|
| Inline Annotation | Check run annotation | MR discussion on line |
|
|
|
|
### Payload Models
|
|
|
|
#### Comment Request
|
|
|
|
```csharp
|
|
public record ScmCommentRequest
|
|
{
|
|
public required string Owner { get; init; }
|
|
public required string Repository { get; init; }
|
|
public required int PullRequestNumber { get; init; }
|
|
public required string Body { get; init; }
|
|
public string? CommentId { get; init; } // For updates
|
|
public bool UpdateExisting { get; init; } = true;
|
|
}
|
|
```
|
|
|
|
#### Status Request
|
|
|
|
```csharp
|
|
public record ScmStatusRequest
|
|
{
|
|
public required string Owner { get; init; }
|
|
public required string Repository { get; init; }
|
|
public required string CommitSha { get; init; }
|
|
public required ScmStatusState State { get; init; }
|
|
public required string Context { get; init; }
|
|
public string? Description { get; init; }
|
|
public string? TargetUrl { get; init; }
|
|
}
|
|
|
|
public enum ScmStatusState
|
|
{
|
|
Pending,
|
|
Success,
|
|
Failure,
|
|
Error
|
|
}
|
|
```
|
|
|
|
#### Check Run Request
|
|
|
|
```csharp
|
|
public record ScmCheckRunRequest
|
|
{
|
|
public required string Owner { get; init; }
|
|
public required string Repository { get; init; }
|
|
public required string HeadSha { get; init; }
|
|
public required string Name { get; init; }
|
|
public string? Status { get; init; } // queued, in_progress, completed
|
|
public string? Conclusion { get; init; } // success, failure, neutral, etc.
|
|
public string? Summary { get; init; }
|
|
public string? Text { get; init; }
|
|
public IReadOnlyList<ScmCheckRunAnnotation>? Annotations { get; init; }
|
|
}
|
|
|
|
public record ScmCheckRunAnnotation
|
|
{
|
|
public required string Path { get; init; }
|
|
public required int StartLine { get; init; }
|
|
public required int EndLine { get; init; }
|
|
public required string AnnotationLevel { get; init; } // notice, warning, failure
|
|
public required string Message { get; init; }
|
|
public string? Title { get; init; }
|
|
}
|
|
```
|
|
|
|
## Provider Implementations
|
|
|
|
### GitHub App Plugin
|
|
|
|
- Uses GitHub App authentication (installation tokens)
|
|
- Supports: PR comments, commit status, check runs with annotations
|
|
- Handles rate limiting with exponential backoff
|
|
- Maps StellaOps severity to GitHub annotation levels
|
|
|
|
### GitLab Plugin
|
|
|
|
- Uses Personal Access Token or CI Job Token
|
|
- Supports: MR notes, discussions, commit status
|
|
- Emulates check runs via pipeline status + MR discussions
|
|
- Handles project path encoding for API calls
|
|
|
|
## Security
|
|
|
|
### Credential Management
|
|
|
|
- All credentials stored as AuthRef URIs
|
|
- Resolved at runtime through Authority
|
|
- No plaintext secrets in configuration
|
|
- Audit trail for credential access
|
|
|
|
### Token Scopes
|
|
|
|
| Provider | Required Scopes |
|
|
|----------|----------------|
|
|
| GitHub App | `checks:write`, `pull_requests:write`, `statuses:write` |
|
|
| GitLab | `api`, `read_repository`, `write_repository` |
|
|
|
|
## Error Handling
|
|
|
|
### Offline-Safe Operations
|
|
|
|
All SCM operations return `ScmOperationResult<T>`:
|
|
|
|
```csharp
|
|
public record ScmOperationResult<T>
|
|
{
|
|
public bool Success { get; init; }
|
|
public T? Result { get; init; }
|
|
public string? ErrorMessage { get; init; }
|
|
public bool IsTransient { get; init; } // Retry-able
|
|
public bool SkippedOffline { get; init; }
|
|
}
|
|
```
|
|
|
|
### Retry Policy
|
|
|
|
- Transient errors (rate limit, network): Retry with exponential backoff
|
|
- Permanent errors (auth, not found): Fail immediately
|
|
- Offline mode: Skip with warning, log payload for manual posting
|
|
|
|
## Observability
|
|
|
|
### Metrics
|
|
|
|
| Metric | Type | Labels |
|
|
|--------|------|--------|
|
|
| `integrations_health_check_total` | Counter | `provider`, `status` |
|
|
| `integrations_test_connection_duration_seconds` | Histogram | `provider` |
|
|
| `scm_annotation_total` | Counter | `provider`, `operation`, `status` |
|
|
| `scm_annotation_duration_seconds` | Histogram | `provider`, `operation` |
|
|
|
|
### Structured Logging
|
|
|
|
All operations log with:
|
|
- `integrationId`: Registration ID
|
|
- `provider`: GitHub, GitLab, etc.
|
|
- `operation`: comment, status, check_run
|
|
- `prNumber` / `commitSha`: Target reference
|
|
|
|
## Current Catalog Contract
|
|
|
|
The live Integration Catalog contract is served by the Integrations WebService and is the source of truth for provider discovery and resource discovery.
|
|
|
|
### Provider Metadata
|
|
|
|
- `GET /api/v1/integrations/providers` returns `ProviderInfo[]` with `name`, `type`, `provider`, `isTestOnly`, `supportsDiscovery`, and `supportedResourceTypes`.
|
|
- Test-only providers are hidden by default. `GET /api/v1/integrations/providers?includeTestOnly=true` exposes providers such as `InMemory` for explicit test/dev workflows.
|
|
- Built-in provider coverage now includes Harbor, Docker Registry, GitLab Container Registry, GitHub App, Gitea, GitLab Server, GitLab CI, Jenkins, Nexus, Vault, Consul, eBPF Agent, the `S3Compatible` object-storage provider, feed mirror providers (`StellaOpsMirror`, `NvdMirror`, `OsvMirror`), and the hidden test-only `InMemory` plugin.
|
|
|
|
### Discovery
|
|
|
|
- `POST /api/v1/integrations/{id}/discover` accepts:
|
|
|
|
```json
|
|
{
|
|
"resourceType": "repositories",
|
|
"filter": {
|
|
"namePattern": "team/*"
|
|
}
|
|
}
|
|
```
|
|
|
|
- Successful responses return `DiscoverIntegrationResponse` with the normalized `resourceType`, the ordered `supportedResourceTypes`, and the discovered `resources`.
|
|
- Unsupported discovery requests return `400 Bad Request` with the supported resource types for that provider.
|
|
- Missing or cross-tenant integration IDs return `404 Not Found`.
|
|
|
|
### Discovery-Capable Providers
|
|
|
|
- OCI registries: `repositories`, `tags`
|
|
- SCM: `projects`, `repositories`
|
|
- CI/CD: `jobs`, `pipelines`
|
|
- Feed mirror, object-storage, and secrets/runtime providers currently expose health/test flows only
|
|
|
|
### Credential Resolution
|
|
|
|
- Integration secrets are supplied as AuthRef URIs such as `authref://vault/gitlab#access-token`
|
|
- The runtime resolver is Vault-backed; there is no product-path stub resolver in the shipped service
|
|
- Registry connectors accept bearer tokens and `username:password` or `username:token` secrets for Basic auth-compatible registries
|
|
|
|
## Related Documentation
|
|
|
|
- [CI/CD Gate Flow](../../flows/10-cicd-gate-flow.md)
|
|
- [Authority Architecture](../authority/architecture.md)
|
|
- [Scanner Architecture](../scanner/architecture.md)
|
|
|
|
## AI Code Guard Standalone Run (Sprint 20260208_040)
|
|
|
|
This sprint adds deterministic standalone execution for AI Code Guard checks in the Integrations WebService.
|
|
|
|
### API Surface
|
|
|
|
- Endpoint: `POST /api/v1/integrations/ai-code-guard/run`
|
|
- Mapped in: `src/Integrations/StellaOps.Integrations.WebService/IntegrationEndpoints.cs`
|
|
- Service contract: `IAiCodeGuardRunService` in `src/Integrations/StellaOps.Integrations.WebService/AiCodeGuard/AiCodeGuardRunService.cs`
|
|
|
|
The endpoint executes the equivalent of `stella guard run` behavior through an offline-safe API surface inside the Integrations module.
|
|
|
|
### YAML-Driven Configuration
|
|
|
|
Configuration is parsed by `AiCodeGuardPipelineConfigLoader`:
|
|
|
|
- `secrets` / `enableSecretsScan`
|
|
- `attribution` / `enableAttributionCheck`
|
|
- `license` / `enableLicenseHygiene`
|
|
- `maxFindings`
|
|
- `allowedSpdxLicenses` / `licenseAllowList`
|
|
- `customSecretPatterns` / `secretPatterns`
|
|
|
|
The loader is deterministic and rejects unsupported keys or invalid values with explicit `FormatException` errors.
|
|
|
|
### Scanning Behavior
|
|
|
|
`AiCodeGuardRunService` adds deterministic checks for:
|
|
|
|
- Secrets (built-in + optional custom regex patterns)
|
|
- Attribution markers
|
|
- SPDX license presence / allow-list validation
|
|
|
|
Output ordering is stable:
|
|
|
|
1. Severity descending
|
|
2. Path ordinal
|
|
3. Line number
|
|
4. Rule ID
|
|
5. Finding ID
|
|
|
|
### Contracts
|
|
|
|
New contracts in `src/Integrations/__Libraries/StellaOps.Integrations.Contracts/AiCodeGuardRunContracts.cs`:
|
|
|
|
- `AiCodeGuardRunRequest`
|
|
- `AiCodeGuardSourceFile`
|
|
- `AiCodeGuardRunConfiguration`
|
|
- `AiCodeGuardRunResponse`
|
|
|
|
### Test Evidence
|
|
|
|
Validated in `src/Integrations/__Tests/StellaOps.Integrations.Tests/AiCodeGuardRunServiceTests.cs`:
|
|
|
|
- Deterministic repeated output
|
|
- YAML configuration application and max-finding truncation
|
|
- Invalid YAML validation failure
|
|
|
|
Execution command:
|
|
|
|
- `dotnet test src/Integrations/__Tests/StellaOps.Integrations.Tests/StellaOps.Integrations.Tests.csproj -p:BuildProjectReferences=false --no-restore`
|
|
|
|
Result on 2026-02-08: passed (`37/37`).
|