wip: doctor/cli/docs/api to vector db consolidation; api hardening for descriptions, tenant, and scopes; migrations and conversions of all DALs to EF v10
This commit is contained in:
master
122
docs/modules/integrations/architecture.md
Normal file
122
docs/modules/integrations/architecture.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# Integrations Architecture
|
||||
|
||||
> Technical architecture for the central integration catalog and connector hub.
|
||||
|
||||
## Overview
|
||||
|
||||
The Integrations module provides a unified service for managing connections to external tools (SCM providers, container registries, CI systems). It uses a plugin-based architecture where each external provider is implemented as a connector plugin, enabling new integrations to be added without modifying core logic. All integration state is persisted in PostgreSQL with tenant-scoped isolation.
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Plugin extensibility** - New providers are added as plugins, not core code changes
|
||||
2. **Credential indirection** - Secrets are referenced via AuthRef URIs pointing to external vaults; the integration catalog never stores raw credentials
|
||||
3. **Tenant isolation** - Every integration record is scoped to a tenant; queries are filtered at the data layer
|
||||
4. **Health-first** - Every integration has a health check contract; unhealthy integrations surface alerts
|
||||
|
||||
## Components
|
||||
|
||||
```
|
||||
Integrations/
|
||||
├── StellaOps.Integrations.WebService/ # HTTP API (Minimal API)
|
||||
├── StellaOps.Integrations.Core/ # Business logic and plugin orchestration
|
||||
├── StellaOps.Integrations.Contracts/ # Shared DTOs and interfaces
|
||||
├── StellaOps.Integrations.Persistence/ # PostgreSQL via IntegrationDbContext
|
||||
├── Plugins/
|
||||
│ ├── StellaOps.Integrations.GitHubApp/ # GitHub App connector
|
||||
│ ├── StellaOps.Integrations.GitLab/ # GitLab connector
|
||||
│ ├── StellaOps.Integrations.Harbor/ # Harbor registry connector
|
||||
│ └── StellaOps.Integrations.InMemory/ # In-memory connector for testing
|
||||
└── __Tests/
|
||||
└── StellaOps.Integrations.Tests/ # Unit and integration tests
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
[External Tool] <── health check ── [Integrations WebService]
|
||||
│
|
||||
[Client Module] ── GET /integrations ──> │
|
||||
│
|
||||
[Admin/CLI] ── POST /integrations ──> │ ── [IntegrationDbContext] ── [PostgreSQL]
|
||||
│
|
||||
[Plugin Loader] ── discover plugins ──> │ ── [Plugin Framework]
|
||||
```
|
||||
|
||||
1. **Registration:** An administrator or automated onboarding flow creates an integration record via `POST /integrations`, providing the provider type, configuration JSON, and an AuthRef URI for credentials.
|
||||
2. **Discovery:** The `PluginLoader` scans configured paths for available connector plugins and registers their capabilities in the plugin metadata table.
|
||||
3. **Health checking:** A background timer invokes each integration's health check endpoint (provider-specific) and persists the result. Failures update the integration status and emit alerts.
|
||||
4. **Consumption:** Downstream modules (Scanner, Orchestrator, Signals) query `GET /integrations/{type}` to retrieve connection details for a specific provider type, filtered by tenant.
|
||||
|
||||
## Database Schema
|
||||
|
||||
Database: `stellaops_integrations` (PostgreSQL)
|
||||
|
||||
| Table | Purpose |
|
||||
|-------|---------|
|
||||
| `integrations` | Primary catalog of registered integrations (id, tenant_id, type, name, config_json, auth_ref_uri, status, created_at, updated_at) |
|
||||
| `plugin_metadata` | Discovered plugin descriptors (plugin_id, type, version, capabilities, path) |
|
||||
| `health_checks` | Health check history (integration_id, checked_at, status, latency_ms, error_message) |
|
||||
| `audit_logs` | Audit trail for integration CRUD operations (actor, action, integration_id, timestamp, details) |
|
||||
|
||||
## Endpoints
|
||||
|
||||
### Integration CRUD (`/api/v1/integrations`)
|
||||
- `GET /` - List integrations for current tenant (optional `?type=` filter)
|
||||
- `GET /{id}` - Get integration by ID
|
||||
- `POST /` - Register a new integration (body: type, name, config, authRefUri)
|
||||
- `PUT /{id}` - Update integration configuration
|
||||
- `DELETE /{id}` - Remove integration (soft delete with audit)
|
||||
|
||||
### Health (`/api/v1/integrations`)
|
||||
- `POST /{id}/health-check` - Trigger an on-demand health check for a specific integration
|
||||
- `GET /{id}/health` - Retrieve latest health status and history
|
||||
|
||||
### Type-filtered queries
|
||||
- `GET /by-type/{type}` - List integrations of a specific provider type (e.g., `github`, `harbor`)
|
||||
|
||||
### Plugin discovery
|
||||
- `POST /plugins/discover` - Trigger plugin discovery scan and register new connectors
|
||||
|
||||
## Plugin Architecture
|
||||
|
||||
Each connector plugin implements a standard interface:
|
||||
|
||||
```csharp
|
||||
public interface IIntegrationPlugin
|
||||
{
|
||||
string ProviderType { get; }
|
||||
Task<IntegrationHealthResult> CheckHealthAsync(IntegrationConfig config, CancellationToken ct);
|
||||
Task<bool> ValidateConfigAsync(JsonDocument config, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Built-in plugins:**
|
||||
- **GitHubApp** - GitHub App installation authentication, repository listing, webhook setup
|
||||
- **GitLab** - Personal/project access token authentication, project discovery
|
||||
- **Harbor** - Robot account authentication, project and repository enumeration
|
||||
- **InMemory** - Deterministic test double for integration tests and offline development
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- **AuthRef URI credential model:** Credentials are stored in an external vault (e.g., HashiCorp Vault, Azure Key Vault). The integration catalog stores only the URI reference (`authref://vault/path/to/secret`), never the raw secret.
|
||||
- **Tenant scoping:** All database queries include a mandatory tenant filter enforced at the `DbContext` level via query filters.
|
||||
- **Audit logging:** Every create, update, and delete operation is recorded in the audit log with actor identity, timestamp, and change details.
|
||||
- **Plugin sandboxing:** Connector plugins run within the Plugin Framework's trust boundary; untrusted plugins are process-isolated.
|
||||
|
||||
## Observability
|
||||
|
||||
- **Metrics:** `integration_health_status{type,tenant}`, `integration_health_latency_ms`, `integration_crud_total{action}`
|
||||
- **Logs:** Structured logs with `integrationId`, `tenantId`, `providerType`, `action`
|
||||
- **Traces:** Spans for health checks, plugin discovery, and CRUD operations
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
- Health checks run on a configurable interval (default: 5 minutes) per integration
|
||||
- Plugin discovery is triggered on startup and on-demand; results are cached
|
||||
- Integration queries use indexed tenant_id + type columns for fast filtering
|
||||
|
||||
## References
|
||||
|
||||
- [Module README](./README.md)
|
||||
- [Plugin Framework Architecture](../plugin/architecture.md)
|
||||
- [Scanner Architecture](../scanner/architecture.md)
|
||||
Reference in New Issue
Block a user