old sprints work, new sprints for exposing functionality via cli, improve code_of_conduct and other agents instructions
This commit is contained in:
master
260
docs/architecture/integrations.md
Normal file
260
docs/architecture/integrations.md
Normal file
@@ -0,0 +1,260 @@
|
||||
# Integrations Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
The Integrations module provides a unified catalog for external service connections including SCM providers (GitHub, GitLab, Bitbucket), container registries (Harbor, ECR, GCR, ACR), CI systems, and runtime hosts. It implements a plugin-based architecture for extensibility while maintaining consistent security and observability patterns.
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ Integrations Module │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────────────────┐ ┌──────────────────────┐ │
|
||||
│ │ WebService Host │ │ Plugin Loader │ │
|
||||
│ │ (ASP.NET Core) │────│ (DI Registration) │ │
|
||||
│ └──────────┬───────────┘ └──────────┬───────────┘ │
|
||||
│ │ │ │
|
||||
│ ┌──────────▼───────────────────────────▼───────────┐ │
|
||||
│ │ Integration Catalog │ │
|
||||
│ │ - Registration CRUD │ │
|
||||
│ │ - Health Polling │ │
|
||||
│ │ - Test Connection │ │
|
||||
│ └──────────┬───────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────▼───────────────────────────────────────┐ │
|
||||
│ │ Plugin Contracts │ │
|
||||
│ │ - IIntegrationConnectorPlugin │ │
|
||||
│ │ - IScmAnnotationClient │ │
|
||||
│ │ - IRegistryConnector │ │
|
||||
│ └──────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────▼───────────────────────────────────────┐ │
|
||||
│ │ Provider Plugins │ │
|
||||
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │
|
||||
│ │ │ GitHub │ │ GitLab │ │ Harbor │ │ ECR │ │ │
|
||||
│ │ │ App │ │ │ │ │ │ │ │ │
|
||||
│ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │
|
||||
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │
|
||||
│ │ │ GCR │ │ ACR │ │InMemory │ │ │
|
||||
│ │ │ │ │ │ │ (test) │ │ │
|
||||
│ │ └─────────┘ └─────────┘ └─────────┘ │ │
|
||||
│ └──────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Core Components
|
||||
|
||||
### Integration Catalog
|
||||
|
||||
The central registry for all external service connections:
|
||||
|
||||
- **Registration**: Store connection configuration with encrypted credentials
|
||||
- **Health Monitoring**: Periodic health checks with status tracking
|
||||
- **Test Connection**: On-demand connectivity verification
|
||||
- **Lifecycle Events**: Emit events for Scheduler/Signals integration
|
||||
|
||||
### Plugin System
|
||||
|
||||
Extensible plugin architecture for provider support:
|
||||
|
||||
```csharp
|
||||
public interface IIntegrationConnectorPlugin : IAvailabilityPlugin
|
||||
{
|
||||
IntegrationType Type { get; }
|
||||
IntegrationProvider Provider { get; }
|
||||
Task<TestConnectionResult> TestConnectionAsync(IntegrationConfig config, CancellationToken ct);
|
||||
Task<HealthCheckResult> CheckHealthAsync(IntegrationConfig config, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
### SCM Annotation Client
|
||||
|
||||
Interface for PR/MR comments and status checks:
|
||||
|
||||
```csharp
|
||||
public interface IScmAnnotationClient
|
||||
{
|
||||
Task<ScmOperationResult<ScmCommentResponse>> PostCommentAsync(
|
||||
ScmCommentRequest request, CancellationToken ct);
|
||||
|
||||
Task<ScmOperationResult<ScmStatusResponse>> PostStatusAsync(
|
||||
ScmStatusRequest request, CancellationToken ct);
|
||||
|
||||
Task<ScmOperationResult<ScmCheckRunResponse>> CreateCheckRunAsync(
|
||||
ScmCheckRunRequest request, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
## SCM Annotation Architecture
|
||||
|
||||
### Comment and Status Flow
|
||||
|
||||
```
|
||||
┌────────────┐ ┌─────────────┐ ┌────────────────┐ ┌──────────┐
|
||||
│ Scanner │────▶│ Integrations│────▶│ SCM Annotation │────▶│ GitHub/ │
|
||||
│ Service │ │ Service │ │ Client │ │ GitLab │
|
||||
└────────────┘ └─────────────┘ └────────────────┘ └──────────┘
|
||||
│ │
|
||||
│ ┌─────────────────┐ │
|
||||
└────────▶│ Annotation │◀───────────┘
|
||||
│ Payload Builder │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
### Supported Operations
|
||||
|
||||
| Operation | GitHub | GitLab |
|
||||
|-----------|--------|--------|
|
||||
| PR/MR Comment | Issue comment / Review comment | MR Note / Discussion |
|
||||
| Commit Status | Commit status API | Commit status API |
|
||||
| Check Run | Checks API with annotations | Pipeline status (emulated) |
|
||||
| Inline Annotation | Check run annotation | MR discussion on line |
|
||||
|
||||
### Payload Models
|
||||
|
||||
#### Comment Request
|
||||
|
||||
```csharp
|
||||
public record ScmCommentRequest
|
||||
{
|
||||
public required string Owner { get; init; }
|
||||
public required string Repository { get; init; }
|
||||
public required int PullRequestNumber { get; init; }
|
||||
public required string Body { get; init; }
|
||||
public string? CommentId { get; init; } // For updates
|
||||
public bool UpdateExisting { get; init; } = true;
|
||||
}
|
||||
```
|
||||
|
||||
#### Status Request
|
||||
|
||||
```csharp
|
||||
public record ScmStatusRequest
|
||||
{
|
||||
public required string Owner { get; init; }
|
||||
public required string Repository { get; init; }
|
||||
public required string CommitSha { get; init; }
|
||||
public required ScmStatusState State { get; init; }
|
||||
public required string Context { get; init; }
|
||||
public string? Description { get; init; }
|
||||
public string? TargetUrl { get; init; }
|
||||
}
|
||||
|
||||
public enum ScmStatusState
|
||||
{
|
||||
Pending,
|
||||
Success,
|
||||
Failure,
|
||||
Error
|
||||
}
|
||||
```
|
||||
|
||||
#### Check Run Request
|
||||
|
||||
```csharp
|
||||
public record ScmCheckRunRequest
|
||||
{
|
||||
public required string Owner { get; init; }
|
||||
public required string Repository { get; init; }
|
||||
public required string HeadSha { get; init; }
|
||||
public required string Name { get; init; }
|
||||
public string? Status { get; init; } // queued, in_progress, completed
|
||||
public string? Conclusion { get; init; } // success, failure, neutral, etc.
|
||||
public string? Summary { get; init; }
|
||||
public string? Text { get; init; }
|
||||
public IReadOnlyList<ScmCheckRunAnnotation>? Annotations { get; init; }
|
||||
}
|
||||
|
||||
public record ScmCheckRunAnnotation
|
||||
{
|
||||
public required string Path { get; init; }
|
||||
public required int StartLine { get; init; }
|
||||
public required int EndLine { get; init; }
|
||||
public required string AnnotationLevel { get; init; } // notice, warning, failure
|
||||
public required string Message { get; init; }
|
||||
public string? Title { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
## Provider Implementations
|
||||
|
||||
### GitHub App Plugin
|
||||
|
||||
- Uses GitHub App authentication (installation tokens)
|
||||
- Supports: PR comments, commit status, check runs with annotations
|
||||
- Handles rate limiting with exponential backoff
|
||||
- Maps StellaOps severity to GitHub annotation levels
|
||||
|
||||
### GitLab Plugin
|
||||
|
||||
- Uses Personal Access Token or CI Job Token
|
||||
- Supports: MR notes, discussions, commit status
|
||||
- Emulates check runs via pipeline status + MR discussions
|
||||
- Handles project path encoding for API calls
|
||||
|
||||
## Security
|
||||
|
||||
### Credential Management
|
||||
|
||||
- All credentials stored as AuthRef URIs
|
||||
- Resolved at runtime through Authority
|
||||
- No plaintext secrets in configuration
|
||||
- Audit trail for credential access
|
||||
|
||||
### Token Scopes
|
||||
|
||||
| Provider | Required Scopes |
|
||||
|----------|----------------|
|
||||
| GitHub App | `checks:write`, `pull_requests:write`, `statuses:write` |
|
||||
| GitLab | `api`, `read_repository`, `write_repository` |
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Offline-Safe Operations
|
||||
|
||||
All SCM operations return `ScmOperationResult<T>`:
|
||||
|
||||
```csharp
|
||||
public record ScmOperationResult<T>
|
||||
{
|
||||
public bool Success { get; init; }
|
||||
public T? Result { get; init; }
|
||||
public string? ErrorMessage { get; init; }
|
||||
public bool IsTransient { get; init; } // Retry-able
|
||||
public bool SkippedOffline { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
### Retry Policy
|
||||
|
||||
- Transient errors (rate limit, network): Retry with exponential backoff
|
||||
- Permanent errors (auth, not found): Fail immediately
|
||||
- Offline mode: Skip with warning, log payload for manual posting
|
||||
|
||||
## Observability
|
||||
|
||||
### Metrics
|
||||
|
||||
| Metric | Type | Labels |
|
||||
|--------|------|--------|
|
||||
| `integrations_health_check_total` | Counter | `provider`, `status` |
|
||||
| `integrations_test_connection_duration_seconds` | Histogram | `provider` |
|
||||
| `scm_annotation_total` | Counter | `provider`, `operation`, `status` |
|
||||
| `scm_annotation_duration_seconds` | Histogram | `provider`, `operation` |
|
||||
|
||||
### Structured Logging
|
||||
|
||||
All operations log with:
|
||||
- `integrationId`: Registration ID
|
||||
- `provider`: GitHub, GitLab, etc.
|
||||
- `operation`: comment, status, check_run
|
||||
- `prNumber` / `commitSha`: Target reference
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [CI/CD Gate Flow](../../flows/10-cicd-gate-flow.md)
|
||||
- [Authority Architecture](../authority/architecture.md)
|
||||
- [Scanner Architecture](../scanner/architecture.md)
|
||||
Reference in New Issue
Block a user