wip: doctor/cli/docs/api to vector db consolidation; api hardening for descriptions, tenant, and scopes; migrations and conversions of all DALs to EF v10
This commit is contained in:
master
53
docs/modules/integrations/README.md
Normal file
53
docs/modules/integrations/README.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# Integrations
|
||||
|
||||
> Central catalog and connector hub for managing external tool integrations across the Stella Ops platform.
|
||||
|
||||
## Purpose
|
||||
|
||||
Integrations provides a unified API for registering, configuring, and health-checking third-party service connectors such as GitHub, GitLab, and Harbor. It serves as the single source of truth for all external tool configurations, enabling other modules to discover and consume integration details without embedding provider-specific logic.
|
||||
|
||||
## Quick Links
|
||||
|
||||
- [Architecture](./architecture.md) - Technical design and implementation details
|
||||
|
||||
## Status
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| **Maturity** | Production |
|
||||
| **Source** | `src/Integrations/` |
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Plugin-based connector architecture:** Extensible provider system with built-in connectors for GitHub App, GitLab, Harbor, and in-memory testing
|
||||
- **Health checks:** Per-integration health probing with status tracking and alerting
|
||||
- **Credential management:** AuthRef URI scheme for vault-referenced credentials; no plaintext secrets stored in the integration catalog
|
||||
- **Tenant isolation:** All integrations are scoped to a tenant; cross-tenant access is prohibited
|
||||
- **AiCodeGuard pipeline support:** Integrations participate in AI-assisted code review pipelines
|
||||
- **Plugin discovery:** Automatic detection and registration of available connector plugins
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Upstream (this module depends on)
|
||||
- **Authority** - Authentication and authorization for API access
|
||||
- **Plugin Framework** - Plugin lifecycle and discovery infrastructure
|
||||
|
||||
### Downstream (modules that depend on this)
|
||||
- **Scanner** - Retrieves repository and registry connection details for scanning operations
|
||||
- **Orchestrator** - Reads integration configs for CI/CD pipeline orchestration
|
||||
- **Signals** - Uses integration metadata for SCM webhook routing and validation
|
||||
|
||||
## Configuration
|
||||
|
||||
Key settings:
|
||||
- PostgreSQL connection (database: `stellaops_integrations`)
|
||||
- Authority audiences and scopes
|
||||
- Plugin search paths for connector discovery
|
||||
- Health check intervals and timeout thresholds
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Plugin Framework](../plugin/) - Underlying plugin infrastructure
|
||||
- [Scanner](../scanner/) - Primary consumer of integration configs
|
||||
- [Orchestrator](../orchestrator/) - Pipeline orchestration using integrations
|
||||
- [Signals](../signals/) - SCM webhook processing
|
||||
122
docs/modules/integrations/architecture.md
Normal file
122
docs/modules/integrations/architecture.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# Integrations Architecture
|
||||
|
||||
> Technical architecture for the central integration catalog and connector hub.
|
||||
|
||||
## Overview
|
||||
|
||||
The Integrations module provides a unified service for managing connections to external tools (SCM providers, container registries, CI systems). It uses a plugin-based architecture where each external provider is implemented as a connector plugin, enabling new integrations to be added without modifying core logic. All integration state is persisted in PostgreSQL with tenant-scoped isolation.
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Plugin extensibility** - New providers are added as plugins, not core code changes
|
||||
2. **Credential indirection** - Secrets are referenced via AuthRef URIs pointing to external vaults; the integration catalog never stores raw credentials
|
||||
3. **Tenant isolation** - Every integration record is scoped to a tenant; queries are filtered at the data layer
|
||||
4. **Health-first** - Every integration has a health check contract; unhealthy integrations surface alerts
|
||||
|
||||
## Components
|
||||
|
||||
```
|
||||
Integrations/
|
||||
├── StellaOps.Integrations.WebService/ # HTTP API (Minimal API)
|
||||
├── StellaOps.Integrations.Core/ # Business logic and plugin orchestration
|
||||
├── StellaOps.Integrations.Contracts/ # Shared DTOs and interfaces
|
||||
├── StellaOps.Integrations.Persistence/ # PostgreSQL via IntegrationDbContext
|
||||
├── Plugins/
|
||||
│ ├── StellaOps.Integrations.GitHubApp/ # GitHub App connector
|
||||
│ ├── StellaOps.Integrations.GitLab/ # GitLab connector
|
||||
│ ├── StellaOps.Integrations.Harbor/ # Harbor registry connector
|
||||
│ └── StellaOps.Integrations.InMemory/ # In-memory connector for testing
|
||||
└── __Tests/
|
||||
└── StellaOps.Integrations.Tests/ # Unit and integration tests
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
[External Tool] <── health check ── [Integrations WebService]
|
||||
│
|
||||
[Client Module] ── GET /integrations ──> │
|
||||
│
|
||||
[Admin/CLI] ── POST /integrations ──> │ ── [IntegrationDbContext] ── [PostgreSQL]
|
||||
│
|
||||
[Plugin Loader] ── discover plugins ──> │ ── [Plugin Framework]
|
||||
```
|
||||
|
||||
1. **Registration:** An administrator or automated onboarding flow creates an integration record via `POST /integrations`, providing the provider type, configuration JSON, and an AuthRef URI for credentials.
|
||||
2. **Discovery:** The `PluginLoader` scans configured paths for available connector plugins and registers their capabilities in the plugin metadata table.
|
||||
3. **Health checking:** A background timer invokes each integration's health check endpoint (provider-specific) and persists the result. Failures update the integration status and emit alerts.
|
||||
4. **Consumption:** Downstream modules (Scanner, Orchestrator, Signals) query `GET /integrations/{type}` to retrieve connection details for a specific provider type, filtered by tenant.
|
||||
|
||||
## Database Schema
|
||||
|
||||
Database: `stellaops_integrations` (PostgreSQL)
|
||||
|
||||
| Table | Purpose |
|
||||
|-------|---------|
|
||||
| `integrations` | Primary catalog of registered integrations (id, tenant_id, type, name, config_json, auth_ref_uri, status, created_at, updated_at) |
|
||||
| `plugin_metadata` | Discovered plugin descriptors (plugin_id, type, version, capabilities, path) |
|
||||
| `health_checks` | Health check history (integration_id, checked_at, status, latency_ms, error_message) |
|
||||
| `audit_logs` | Audit trail for integration CRUD operations (actor, action, integration_id, timestamp, details) |
|
||||
|
||||
## Endpoints
|
||||
|
||||
### Integration CRUD (`/api/v1/integrations`)
|
||||
- `GET /` - List integrations for current tenant (optional `?type=` filter)
|
||||
- `GET /{id}` - Get integration by ID
|
||||
- `POST /` - Register a new integration (body: type, name, config, authRefUri)
|
||||
- `PUT /{id}` - Update integration configuration
|
||||
- `DELETE /{id}` - Remove integration (soft delete with audit)
|
||||
|
||||
### Health (`/api/v1/integrations`)
|
||||
- `POST /{id}/health-check` - Trigger an on-demand health check for a specific integration
|
||||
- `GET /{id}/health` - Retrieve latest health status and history
|
||||
|
||||
### Type-filtered queries
|
||||
- `GET /by-type/{type}` - List integrations of a specific provider type (e.g., `github`, `harbor`)
|
||||
|
||||
### Plugin discovery
|
||||
- `POST /plugins/discover` - Trigger plugin discovery scan and register new connectors
|
||||
|
||||
## Plugin Architecture
|
||||
|
||||
Each connector plugin implements a standard interface:
|
||||
|
||||
```csharp
|
||||
public interface IIntegrationPlugin
|
||||
{
|
||||
string ProviderType { get; }
|
||||
Task<IntegrationHealthResult> CheckHealthAsync(IntegrationConfig config, CancellationToken ct);
|
||||
Task<bool> ValidateConfigAsync(JsonDocument config, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
|
||||
**Built-in plugins:**
|
||||
- **GitHubApp** - GitHub App installation authentication, repository listing, webhook setup
|
||||
- **GitLab** - Personal/project access token authentication, project discovery
|
||||
- **Harbor** - Robot account authentication, project and repository enumeration
|
||||
- **InMemory** - Deterministic test double for integration tests and offline development
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- **AuthRef URI credential model:** Credentials are stored in an external vault (e.g., HashiCorp Vault, Azure Key Vault). The integration catalog stores only the URI reference (`authref://vault/path/to/secret`), never the raw secret.
|
||||
- **Tenant scoping:** All database queries include a mandatory tenant filter enforced at the `DbContext` level via query filters.
|
||||
- **Audit logging:** Every create, update, and delete operation is recorded in the audit log with actor identity, timestamp, and change details.
|
||||
- **Plugin sandboxing:** Connector plugins run within the Plugin Framework's trust boundary; untrusted plugins are process-isolated.
|
||||
|
||||
## Observability
|
||||
|
||||
- **Metrics:** `integration_health_status{type,tenant}`, `integration_health_latency_ms`, `integration_crud_total{action}`
|
||||
- **Logs:** Structured logs with `integrationId`, `tenantId`, `providerType`, `action`
|
||||
- **Traces:** Spans for health checks, plugin discovery, and CRUD operations
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
- Health checks run on a configurable interval (default: 5 minutes) per integration
|
||||
- Plugin discovery is triggered on startup and on-demand; results are cached
|
||||
- Integration queries use indexed tenant_id + type columns for fast filtering
|
||||
|
||||
## References
|
||||
|
||||
- [Module README](./README.md)
|
||||
- [Plugin Framework Architecture](../plugin/architecture.md)
|
||||
- [Scanner Architecture](../scanner/architecture.md)
|
||||
Reference in New Issue
Block a user