diff --git a/devops/compose/README.md b/devops/compose/README.md index 593460606..1fdc654fb 100644 --- a/devops/compose/README.md +++ b/devops/compose/README.md @@ -281,6 +281,41 @@ docker compose \ docker compose -f docker-compose.integrations.yml ps gitea ``` +Register the default local-ready integration catalog once the stack is up: + +```powershell +powershell -ExecutionPolicy Bypass -File scripts/register-local-integrations.ps1 ` + -Tenant demo-prod +``` + +The helper creates and verifies the 13 turnkey local providers on a fresh +machine. GitLab server/CI and the GitLab registry remain opt-in because they +require Vault-backed PAT material. The scripted local path is: + +```powershell +powershell -ExecutionPolicy Bypass -File scripts/bootstrap-local-gitlab-secrets.ps1 ` + -VerifyRegistry + +powershell -ExecutionPolicy Bypass -File scripts/register-local-integrations.ps1 ` + -Tenant demo-prod ` + -IncludeGitLab + +powershell -ExecutionPolicy Bypass -File scripts/register-local-integrations.ps1 ` + -Tenant demo-prod ` + -IncludeGitLab ` + -IncludeGitLabRegistry +``` + +Or run the GitLab-backed registration in one step: + +```powershell +powershell -ExecutionPolicy Bypass -File scripts/register-local-integrations.ps1 ` + -Tenant demo-prod ` + -IncludeGitLab ` + -IncludeGitLabRegistry ` + -BootstrapGitLabSecrets +``` + **Hosts file entries** (add to `C:\Windows\System32\drivers\etc\hosts`): ``` 127.1.2.1 gitea.stella-ops.local @@ -322,7 +357,7 @@ vault kv put secret/nexus admin-password="your-password" Gitea is now bootstrapped by the compose service itself: a fresh `stellaops-gitea-data` volume creates the default local admin user and the repository root before the container reports healthy. Personal access tokens remain a manual step because Gitea only reveals the token value when it is created. -When you enable the optional GitLab registry surface (`GITLAB_ENABLE_REGISTRY=true`), register it through the `GitLabContainerRegistry` provider with `authref://vault/gitlab#registry-basic`. The local Docker registry connector now follows the registry's Bearer challenge and exchanges that `username:personal-access-token` secret against `jwt/auth` before retrying catalog and tag probes. +For GitLab, `scripts/bootstrap-local-gitlab-secrets.ps1` is the preferred local bootstrap path. It reuses a valid `secret/gitlab` secret when possible and otherwise rotates the local `stella-local-integration` PAT, then writes `authref://vault/gitlab#access-token` plus `authref://vault/gitlab#registry-basic` into the dev Vault. When you enable the optional GitLab registry surface (`GITLAB_ENABLE_REGISTRY=true`), register it through the `GitLabContainerRegistry` provider with `authref://vault/gitlab#registry-basic`. The local Docker registry connector now follows the registry's Bearer challenge and exchanges that `username:personal-access-token` secret against `jwt/auth` before retrying catalog and tag probes. `docker-compose.testing.yml` is a separate infrastructure-test lane. It starts `postgres-test`, `valkey-test`, mocks, and an isolated Gitea profile on different ports; it does not start Consul or GitLab. Use `docker-compose.integrations.yml` only when you need real third-party providers for connector validation. diff --git a/devops/compose/router-gateway-local.json b/devops/compose/router-gateway-local.json index 29120623b..f66774b13 100644 --- a/devops/compose/router-gateway-local.json +++ b/devops/compose/router-gateway-local.json @@ -150,6 +150,7 @@ { "Type": "Microservice", "Path": "^/api/v1/timeline(.*)", "IsRegex": true, "TranslatesTo": "http://timeline.stella-ops.local/api/v1/timeline$1" }, { "Type": "Microservice", "Path": "^/api/v1/audit(.*)", "IsRegex": true, "TranslatesTo": "http://timeline.stella-ops.local/api/v1/audit$1" }, { "Type": "Microservice", "Path": "^/api/v1/export(.*)", "IsRegex": true, "TranslatesTo": "https://exportcenter.stella-ops.local/api/v1/export$1" }, + { "Type": "Microservice", "Path": "^/api/v1/concelier(.*)", "IsRegex": true, "TranslatesTo": "http://concelier.stella-ops.local/api/v1/concelier$1" }, { "Type": "Microservice", "Path": "^/api/v1/advisory-sources(.*)", "IsRegex": true, "TranslatesTo": "http://concelier.stella-ops.local/api/v1/advisory-sources$1" }, { "Type": "Microservice", "Path": "^/api/v1/notifier/delivery(.*)", "IsRegex": true, "TranslatesTo": "http://notify.stella-ops.local/api/v2/notify/deliveries$1" }, { "Type": "Microservice", "Path": "^/api/v1/notifier/(.*)", "IsRegex": true, "TranslatesTo": "http://notify.stella-ops.local/api/v2/notify/$1" }, diff --git a/docs/INSTALL_GUIDE.md b/docs/INSTALL_GUIDE.md index 61bb7fec9..236c0e22f 100755 --- a/docs/INSTALL_GUIDE.md +++ b/docs/INSTALL_GUIDE.md @@ -171,6 +171,50 @@ docker compose -f docker-compose.stella-ops.yml ps curl -k https://stella-ops.local # should return the Angular UI ``` +For a fresh local developer install, populate the live integration catalog with: + +```powershell +powershell -ExecutionPolicy Bypass -File scripts/register-local-integrations.ps1 ` + -Tenant demo-prod +``` + +This converges the default local-ready lane to 13 healthy providers: +Harbor fixture, Docker Registry, Nexus, GitHub App fixture, Gitea, Jenkins, +Vault, Consul, eBPF runtime-host fixture, MinIO, and the three feed mirror +providers (`StellaOpsMirror`, `NvdMirror`, `OsvMirror`). + +GitLab server/CI and the GitLab registry remain opt-in because they require +Vault-backed credentials. The scripted local path is: + +```powershell +powershell -ExecutionPolicy Bypass -File scripts/bootstrap-local-gitlab-secrets.ps1 ` + -VerifyRegistry + +powershell -ExecutionPolicy Bypass -File scripts/register-local-integrations.ps1 ` + -Tenant demo-prod ` + -IncludeGitLab + +powershell -ExecutionPolicy Bypass -File scripts/register-local-integrations.ps1 ` + -Tenant demo-prod ` + -IncludeGitLab ` + -IncludeGitLabRegistry +``` + +Or run the GitLab-backed registration in one step: + +```powershell +powershell -ExecutionPolicy Bypass -File scripts/register-local-integrations.ps1 ` + -Tenant demo-prod ` + -IncludeGitLab ` + -IncludeGitLabRegistry ` + -BootstrapGitLabSecrets +``` + +`scripts/bootstrap-local-gitlab-secrets.ps1` reuses a valid `secret/gitlab` +secret when possible and otherwise rotates the local `stella-local-integration` +PAT, then writes both `authref://vault/gitlab#access-token` and +`authref://vault/gitlab#registry-basic` into the dev Vault. + ## Air-gapped deployments For offline/air-gapped environments, use the sealed CI compose file and offline telemetry overlay: diff --git a/docs/implplan/SPRINT_20260413_002_Integrations_gitlab_secret_bootstrap_automation.md b/docs/implplan/SPRINT_20260413_002_Integrations_gitlab_secret_bootstrap_automation.md new file mode 100644 index 000000000..b137636e5 --- /dev/null +++ b/docs/implplan/SPRINT_20260413_002_Integrations_gitlab_secret_bootstrap_automation.md @@ -0,0 +1,67 @@ +# Sprint 20260413-002 - GitLab Secret Bootstrap Automation + +## Topic & Scope +- Remove the last manual local GitLab secret-seeding step from the local integrations lane by scripting PAT rotation and Vault writeback. +- Keep GitLab local integrations opt-in, but make the credential bootstrap reusable and idempotent from repo-owned scripts. +- Sync the setup docs with the automated GitLab path and verify the one-command registration flow on this machine. +- Working directory: `scripts/`. +- Expected evidence: script runs, Vault secret metadata, GitLab PAT inventory, integration registration output, and updated setup docs. + +## Dependencies & Concurrency +- Depends on the archived local scratch-setup sprint `docs-archived/implplan/SPRINT_20260413_001_Platform_scratch_setup_local_integrations.md`. +- Safe parallelism is low because the bootstrap rotates live local GitLab PAT material and updates a shared Vault path. +- Cross-directory edits are allowed for `scripts/**`, `docs/**`, and `devops/**` only. + +## Documentation Prerequisites +- `docs/INSTALL_GUIDE.md` +- `docs/integrations/LOCAL_SERVICES.md` +- `devops/compose/README.md` +- `docs-archived/implplan/SPRINT_20260413_001_Platform_scratch_setup_local_integrations.md` + +## Delivery Tracker + +### GITLAB-001 - Automate local GitLab PAT bootstrap into Vault +Status: DONE +Dependency: none +Owners: Developer / Ops Integrator +Task description: +- Add a repo-owned script that can bootstrap or rotate the local GitLab personal access token used by the GitLab SCM, CI, and registry integrations. +- The script must reuse the existing `secret/gitlab` material when it still verifies, rotate stale tokens deterministically, and write the resulting auth material back into the dev Vault without requiring manual UI work. + +Completion criteria: +- [x] A local script can create or rotate the GitLab PAT and write `access-token` plus `registry-basic` into `secret/gitlab`. +- [x] The script reuses a valid local secret instead of rotating on every run. +- [x] The script verifies the GitLab API token and optional registry token exchange before reporting success. + +### GITLAB-002 - Wire and verify the automated GitLab registration path +Status: DONE +Dependency: GITLAB-001 +Owners: Developer / Documentation Author +Task description: +- Wire the local integration helper so GitLab-backed registration can invoke the bootstrap automatically when requested. +- Update the setup docs to make the automated GitLab path the documented local default, keeping manual Vault writes only as an override path. + +Completion criteria: +- [x] `scripts/register-local-integrations.ps1` can invoke the GitLab bootstrap automatically. +- [x] The install and local-service docs describe the automated GitLab path and the one-command registration flow. +- [x] The GitLab-backed integration registration path is rerun successfully on this machine with the automated bootstrap enabled. + +## Execution Log +| Date (UTC) | Update | Owner | +| --- | --- | --- | +| 2026-04-13 | Sprint created to automate the remaining local GitLab PAT and Vault bootstrap step after the archived local scratch-setup sprint closed green. | Developer | +| 2026-04-13 | Confirmed the local GitLab CE admin OAuth password-grant, PAT list, PAT create, and PAT revoke APIs all work against the running local heavy GitLab profile. | Developer | +| 2026-04-13 | Added `scripts/bootstrap-local-gitlab-secrets.ps1` to reuse or rotate the local `stella-local-integration` PAT, verify API plus registry token exchange, and write `secret/gitlab` metadata into the dev Vault. | Developer | +| 2026-04-13 | Updated `scripts/register-local-integrations.ps1` with `-BootstrapGitLabSecrets`, then re-ran `powershell -ExecutionPolicy Bypass -File scripts/register-local-integrations.ps1 -Tenant demo-prod -IncludeGitLab -IncludeGitLabRegistry -BootstrapGitLabSecrets`; result: all 16 local integrations converged `Healthy`. | Developer | +| 2026-04-13 | Updated `docs/INSTALL_GUIDE.md`, `docs/integrations/LOCAL_SERVICES.md`, and `devops/compose/README.md` to make the scripted GitLab bootstrap the documented local path and keep manual Vault writes as the override path. | Developer | + +## Decisions & Risks +- Decision: the local GitLab bootstrap continues to use the deterministic local root account (`root` / `Stella2026!`) because that credential is already the documented dev default in `docker-compose.integrations.yml`. +- Decision: the bootstrap PAT name is fixed to `stella-local-integration`; reruns revoke earlier active PATs with the same name before creating a replacement token so the local token inventory does not grow without bound. +- Decision: the script stores `bootstrap-user`, `token-name`, `expires-at`, and `rotated-at` alongside `access-token` and `registry-basic` in `secret/gitlab` so local operators can inspect the current bootstrap state without exposing extra secret sources. +- Risk: there is no existing PowerShell unit-test harness in this repo for local ops scripts, so verification for this change is live-script execution against the local GitLab and Vault containers plus the integration registration pass. +- Risk: the GitLab local integrations remain opt-in because the heavy GitLab profile and optional registry surface still carry materially higher resource cost than the default local provider lane. + +## Next Checkpoints +- If the local GitLab root password is changed in compose, update `scripts/bootstrap-local-gitlab-secrets.ps1` defaults and the associated docs in the same change. +- If the local GitLab topology moves away from the root account, replace the bootstrap user model with a dedicated seeded local service user and update the Vault metadata schema accordingly. diff --git a/docs/implplan/SPRINT_20260413_003_Web_ui_driven_local_setup_rerun.md b/docs/implplan/SPRINT_20260413_003_Web_ui_driven_local_setup_rerun.md new file mode 100644 index 000000000..a30841215 --- /dev/null +++ b/docs/implplan/SPRINT_20260413_003_Web_ui_driven_local_setup_rerun.md @@ -0,0 +1,78 @@ +# Sprint 20260413-003 -- UI Driven Local Setup Rerun + +## Topic & Scope +- Re-run the Stella Ops local setup from zero Stella runtime state and use the browser UI as the operator surface instead of the CLI registration path. +- Verify which parts of the setup wizard and integrations hub are truly live and which parts still depend on script/API-only bootstrap behind the scenes. +- Leave behind truthful evidence: what the UI can complete today, what was completed via UI, and any remaining product gaps that still prevent a full UI-only bootstrap. +- Working directory: `.`. +- Expected evidence: sprint log, live browser verification, container health, and final integration health results. + +## Dependencies & Concurrency +- Required docs: `docs/modules/platform/architecture-overview.md`, `docs/INSTALL_GUIDE.md`, `docs/integrations/LOCAL_SERVICES.md`, `src/Web/StellaOps.Web/AGENTS.md`. +- Builds on the local-stack convergence work in [SPRINT_20260409_002_Platform_local_stack_regression_retest.md](/C:/dev/New%20folder/git.stella-ops.org/docs/implplan/SPRINT_20260409_002_Platform_local_stack_regression_retest.md) and the local-integration automation in [SPRINT_20260413_002_Integrations_gitlab_secret_bootstrap_automation.md](/C:/dev/New%20folder/git.stella-ops.org/docs/implplan/SPRINT_20260413_002_Integrations_gitlab_secret_bootstrap_automation.md). +- Safe parallelism: none during wipe/setup because the environment reset is global to the machine. Browser verification can run only after the platform is reachable. + +## Documentation Prerequisites +- `docs/modules/platform/architecture-overview.md` +- `docs/INSTALL_GUIDE.md` +- `docs/integrations/LOCAL_SERVICES.md` +- `src/Web/StellaOps.Web/AGENTS.md` + +## Delivery Tracker + +### UISETUP-001 - Reset the local Stella runtime to zero state +Status: DOING +Dependency: none +Owners: Developer / QA +Task description: +- Remove Stella-owned runtime containers, volumes, networks, and local images so the rerun starts from zero Stella platform state rather than from the already converged environment. +- Re-run the documented setup entrypoint needed to bring the platform back to a reachable browser state. + +Completion criteria: +- [ ] Stella-owned containers, volumes, and networks are removed before the rerun. +- [ ] The documented local setup entrypoint is executed successfully after the wipe. +- [ ] `https://stella-ops.local` becomes reachable again for browser-driven setup. + +### UISETUP-002 - Drive the operator setup through the browser UI +Status: TODO +Dependency: UISETUP-001 +Owners: Developer / QA +Task description: +- Use the live browser UI to sign in, complete the setup wizard flow, and exercise the integrations onboarding surfaces instead of provisioning through the CLI registration helper. +- Record which configuration steps are truly persisted/provisioned by the UI versus which ones are session-only or still backend-limited. + +Completion criteria: +- [ ] The setup wizard is exercised through the live browser against the rebuilt stack. +- [ ] The integrations UI is used for the available local integration onboarding flows. +- [ ] Any step that is not truly UI-provisioned is captured explicitly with supporting evidence. + +### UISETUP-003 - Close documentation and evidence gaps for the UI path +Status: TODO +Dependency: UISETUP-002 +Owners: Developer / Documentation +Task description: +- Update operator-facing docs if the actual UI-driven setup path differs from the current documented local bootstrap. +- Record the final verified state, remaining non-UI gaps, and any follow-up implementation needs in the sprint log and linked docs. + +Completion criteria: +- [ ] Docs reflect the verified UI-driven setup reality. +- [ ] Final health and integration results are logged in the execution log. +- [ ] Remaining non-UI blockers are called out explicitly rather than glossed over. + +## Execution Log +| Date (UTC) | Update | Owner | +| --- | --- | --- | +| 2026-04-13 | Sprint created for a zero-state local rerun that must use the live browser UI for setup and integration onboarding wherever the product currently supports it. | Developer | +| 2026-04-13 | Removed all `stellaops-*` containers, `compose_*` / `stellaops_*` volumes, the `stellaops` / `stellaops_frontdoor` networks, and all `stellaops/*:dev` images to return the machine to zero Stella runtime state before the rerun. | Developer | +| 2026-04-13 | Started the documented machine-level bootstrap with `scripts/setup.ps1 -QaIntegrationFixtures`; this restores the platform and fixture-backed frontdoor but not the full real-provider integrations compose lane. | Developer | +| 2026-04-13 | Code inspection ahead of the live browser run found two likely UI-path gaps to validate: the setup wizard backend persists much of its state only in-session, and the integrations onboarding wizard currently requires a non-empty `AuthRef URI` even though the backend API itself accepts null auth refs for local no-auth connectors. | Developer | + +## Decisions & Risks +- Decision: this rerun uses the real browser UI as the operator surface and treats CLI/bootstrap helpers only as fallback evidence if the product lacks a true UI path. +- Risk: the setup wizard backend may still persist parts of the flow only in-memory/session scope, which would make a strict UI-only bootstrap impossible without follow-on implementation work. +- Risk: some local integrations may still require credentials to exist before the UI can complete onboarding, especially GitLab and Vault-backed secrets. + +## Next Checkpoints +- Complete the zero-state wipe and restore the platform to a reachable browser state. +- Drive the setup wizard and integrations flows through the UI. +- Record the exact boundary between UI-complete setup and script/API-only gaps. diff --git a/docs/integrations/LOCAL_SERVICES.md b/docs/integrations/LOCAL_SERVICES.md index f0c0190e0..8a26d664b 100644 --- a/docs/integrations/LOCAL_SERVICES.md +++ b/docs/integrations/LOCAL_SERVICES.md @@ -86,6 +86,59 @@ docker compose -f docker-compose.integrations.yml ps docker compose -f docker-compose.integrations.yml ps gitea ``` +### 4. Register the local integration catalog + +After the core stack plus the local provider lanes are running, register the +catalog entries that Stella Ops can exercise immediately from a fresh local +install: + +```powershell +powershell -ExecutionPolicy Bypass -File scripts/register-local-integrations.ps1 ` + -Tenant demo-prod +``` + +This converges the default local-ready lane to 13 healthy entries: +- Harbor fixture +- Docker Registry +- Nexus +- GitHub App fixture +- Gitea +- Jenkins +- Vault +- Consul +- eBPF runtime-host fixture +- MinIO (`S3Compatible`) +- StellaOps mirror +- NVD mirror +- OSV mirror + +Optional GitLab providers require Vault-backed credentials. The recommended +local flow is: + +```powershell +# Reuse or rotate the local GitLab bootstrap PAT and write it to Vault. +powershell -ExecutionPolicy Bypass -File scripts/bootstrap-local-gitlab-secrets.ps1 ` + -VerifyRegistry + +# Register SCM + CI using the bootstrapped authref://vault/gitlab#access-token +powershell -ExecutionPolicy Bypass -File scripts/register-local-integrations.ps1 ` + -Tenant demo-prod ` + -IncludeGitLab + +# Also requires GitLab registry enabled; uses authref://vault/gitlab#registry-basic +powershell -ExecutionPolicy Bypass -File scripts/register-local-integrations.ps1 ` + -Tenant demo-prod ` + -IncludeGitLab ` + -IncludeGitLabRegistry + +# Or do the GitLab-backed registration in one step +powershell -ExecutionPolicy Bypass -File scripts/register-local-integrations.ps1 ` + -Tenant demo-prod ` + -IncludeGitLab ` + -IncludeGitLabRegistry ` + -BootstrapGitLabSecrets +``` + `docker-compose.testing.yml` is the separate infrastructure-test lane. It starts `postgres-test`, `valkey-test`, mocks, and an isolated Gitea profile on different ports; it does not start Consul or GitLab. --- @@ -206,7 +259,7 @@ vault kv put secret/jenkins api-token="your-jenkins-token" # Store Nexus admin password vault kv put secret/nexus admin-password="your-nexus-password" -# Store GitLab PATs for API and registry access +# Store GitLab PATs for API and registry access (manual override path) vault kv put secret/gitlab access-token="glpat-your-token" registry-basic="root:glpat-your-token" ``` @@ -326,12 +379,14 @@ GITLAB_ENABLE_REGISTRY=true GITLAB_ENABLE_PACKAGES=true \ **Stella Ops integration config (SCM / CI):** - Endpoint: `http://gitlab.stella-ops.local:8929` - AuthRef: `authref://vault/gitlab#access-token` +- Bootstrap helper: `powershell -ExecutionPolicy Bypass -File scripts/bootstrap-local-gitlab-secrets.ps1` **Stella Ops integration config (Registry):** - Endpoint: `http://gitlab.stella-ops.local:5050` - AuthRef: `authref://vault/gitlab#registry-basic` - Secret format: `username:personal-access-token` (local default: `root:`) - The Docker registry connector follows GitLab's `WWW-Authenticate: Bearer` challenge and exchanges this basic secret against `/jwt/auth` before retrying catalog and tag probes. +- `scripts/bootstrap-local-gitlab-secrets.ps1 -VerifyRegistry` reuses a valid local Vault secret when possible and otherwise rotates the local `stella-local-integration` PAT before writing both authrefs. --- diff --git a/scripts/bootstrap-local-gitlab-secrets.ps1 b/scripts/bootstrap-local-gitlab-secrets.ps1 new file mode 100644 index 000000000..6e6385a37 --- /dev/null +++ b/scripts/bootstrap-local-gitlab-secrets.ps1 @@ -0,0 +1,351 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Bootstraps local GitLab credentials into the dev Vault for local integrations. +.DESCRIPTION + Reuses the current `secret/gitlab` credentials when they still pass the local + GitLab API probe (and optional registry token exchange probe). Otherwise, the + script signs in to the local GitLab CE instance as the configured root user, + revokes any earlier bootstrap personal access tokens with the same name, + creates a replacement token, verifies it, and stores: + + - `access-token` + - `registry-basic` + + under the configured Vault KV v2 path. +.PARAMETER GitLabUrl + Base URL of the local GitLab CE web service. +.PARAMETER GitLabRegistryUrl + Base URL of the local GitLab container registry endpoint. +.PARAMETER GitLabUsername + Local GitLab admin username used for the bootstrap flow. +.PARAMETER GitLabPassword + Local GitLab admin password used for the bootstrap flow. +.PARAMETER VaultUrl + Base URL of the local Vault dev server. +.PARAMETER VaultToken + Vault token used to read and write the local secret path. +.PARAMETER VaultSecretPath + KV v2 path in `/` form. Defaults to `secret/gitlab`. +.PARAMETER TokenName + GitLab personal access token name used for the local integration bootstrap. +.PARAMETER TokenLifetimeDays + Lifetime of the generated GitLab personal access token. +.PARAMETER Rotate + Forces PAT rotation even when the current Vault secret still verifies cleanly. +.PARAMETER VerifyRegistry + Also verify that the stored or generated `registry-basic` secret can exchange + against GitLab's `/jwt/auth` registry token endpoint. +#> +[CmdletBinding()] +param( + [string]$GitLabUrl = 'http://gitlab.stella-ops.local:8929', + [string]$GitLabRegistryUrl = 'http://gitlab.stella-ops.local:5050', + [string]$GitLabUsername = 'root', + [string]$GitLabPassword = 'Stella2026!', + [string]$VaultUrl = 'http://vault.stella-ops.local:8200', + [string]$VaultToken = 'stellaops-dev-root-token-2026', + [string]$VaultSecretPath = 'secret/gitlab', + [string]$TokenName = 'stella-local-integration', + [ValidateRange(1, 365)] + [int]$TokenLifetimeDays = 30, + [ValidateRange(10, 900)] + [int]$ReadinessTimeoutSeconds = 600, + [switch]$Rotate, + [switch]$VerifyRegistry +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +$GitLabUrl = $GitLabUrl.TrimEnd('/') +$GitLabRegistryUrl = $GitLabRegistryUrl.TrimEnd('/') +$VaultUrl = $VaultUrl.TrimEnd('/') + +function Get-HttpStatusCode { + param([Parameter(Mandatory)]$ErrorRecord) + + $response = $ErrorRecord.Exception.Response + if ($null -eq $response) { + return $null + } + + if ($response.PSObject.Properties.Name -contains 'StatusCode') { + $statusCode = $response.StatusCode + if ($statusCode -is [int]) { + return $statusCode + } + + try { + return [int]$statusCode + } catch { + return $null + } + } + + return $null +} + +function Wait-HttpReady { + param( + [Parameter(Mandatory)][string]$Name, + [Parameter(Mandatory)][string]$Uri, + [hashtable]$Headers, + [int]$TimeoutSeconds = 300 + ) + + $deadline = (Get-Date).ToUniversalTime().AddSeconds($TimeoutSeconds) + do { + try { + Invoke-RestMethod -Method GET -Uri $Uri -Headers $Headers -TimeoutSec 30 -ErrorAction Stop | Out-Null + return + } catch { + if ((Get-Date).ToUniversalTime() -ge $deadline) { + throw "Timed out waiting for $Name at $Uri. Last error: $($_.Exception.Message)" + } + + Start-Sleep -Seconds 5 + } + } while ($true) +} + +function Split-VaultSecretPath { + param([Parameter(Mandatory)][string]$Path) + + $normalized = $Path.Trim('/') + $segments = $normalized -split '/', 2 + if ($segments.Count -ne 2 -or [string]::IsNullOrWhiteSpace($segments[0]) -or [string]::IsNullOrWhiteSpace($segments[1])) { + throw "VaultSecretPath must be in '/' form. Received '$Path'." + } + + return [pscustomobject]@{ + Mount = $segments[0] + Path = $segments[1] + } +} + +function Invoke-VaultJson { + param( + [Parameter(Mandatory)][ValidateSet('GET', 'POST')][string]$Method, + [Parameter(Mandatory)][string]$Path, + [object]$Body + ) + + $parameters = @{ + Method = $Method + Uri = "$VaultUrl/v1/$Path" + Headers = @{ 'X-Vault-Token' = $VaultToken } + TimeoutSec = 30 + ErrorAction = 'Stop' + } + + if ($null -ne $Body) { + $parameters['ContentType'] = 'application/json' + $parameters['Body'] = $Body | ConvertTo-Json -Depth 10 + } + + return Invoke-RestMethod @parameters +} + +function Get-VaultGitLabSecret { + param([Parameter(Mandatory)]$VaultPathParts) + + try { + $response = Invoke-VaultJson -Method GET -Path "$($VaultPathParts.Mount)/data/$($VaultPathParts.Path)" + return $response.data.data + } catch { + $statusCode = Get-HttpStatusCode -ErrorRecord $_ + if ($statusCode -eq 404) { + return $null + } + + throw + } +} + +function Set-VaultGitLabSecret { + param( + [Parameter(Mandatory)]$VaultPathParts, + [Parameter(Mandatory)][hashtable]$Data + ) + + Invoke-VaultJson -Method POST -Path "$($VaultPathParts.Mount)/data/$($VaultPathParts.Path)" -Body @{ data = $Data } | Out-Null +} + +function Get-GitLabOAuthToken { + $response = Invoke-RestMethod -Method POST -Uri "$GitLabUrl/oauth/token" -ContentType 'application/x-www-form-urlencoded' -Body @{ + grant_type = 'password' + username = $GitLabUsername + password = $GitLabPassword + } -TimeoutSec 60 -ErrorAction Stop + + if ([string]::IsNullOrWhiteSpace($response.access_token)) { + throw "GitLab OAuth password grant did not return an access token." + } + + return "$($response.access_token)" +} + +function Get-GitLabAdminHeaders { + $deadline = (Get-Date).ToUniversalTime().AddSeconds($ReadinessTimeoutSeconds) + do { + try { + $oauthToken = Get-GitLabOAuthToken + return @{ + Authorization = "Bearer $oauthToken" + } + } catch { + if ((Get-Date).ToUniversalTime() -ge $deadline) { + throw "Timed out waiting for GitLab admin login at $GitLabUrl. Last error: $($_.Exception.Message)" + } + + Start-Sleep -Seconds 5 + } + } while ($true) +} + +function Get-GitLabCurrentUser { + param([Parameter(Mandatory)][hashtable]$Headers) + + return Invoke-RestMethod -Method GET -Uri "$GitLabUrl/api/v4/user" -Headers $Headers -TimeoutSec 30 -ErrorAction Stop +} + +function Get-GitLabPersonalAccessTokens { + param( + [Parameter(Mandatory)][hashtable]$Headers, + [Parameter(Mandatory)][int]$UserId + ) + + $response = Invoke-RestMethod -Method GET -Uri "$GitLabUrl/api/v4/personal_access_tokens?user_id=$UserId" -Headers $Headers -TimeoutSec 30 -ErrorAction Stop + return @($response) +} + +function Revoke-GitLabPersonalAccessToken { + param( + [Parameter(Mandatory)][hashtable]$Headers, + [Parameter(Mandatory)][int]$TokenId + ) + + Invoke-RestMethod -Method DELETE -Uri "$GitLabUrl/api/v4/personal_access_tokens/$TokenId" -Headers $Headers -TimeoutSec 30 -ErrorAction Stop | Out-Null +} + +function New-GitLabPersonalAccessToken { + param( + [Parameter(Mandatory)][hashtable]$Headers, + [Parameter(Mandatory)][int]$UserId, + [Parameter(Mandatory)][string]$Name, + [Parameter(Mandatory)][string[]]$Scopes, + [Parameter(Mandatory)][string]$ExpiresAt + ) + + $body = @{ + name = $Name + description = 'Stella Ops local integration bootstrap' + expires_at = $ExpiresAt + scopes = $Scopes + } | ConvertTo-Json -Depth 5 + + return Invoke-RestMethod -Method POST -Uri "$GitLabUrl/api/v4/users/$UserId/personal_access_tokens" -Headers $Headers -ContentType 'application/json' -Body $body -TimeoutSec 30 -ErrorAction Stop +} + +function Test-GitLabApiToken { + param([Parameter(Mandatory)][string]$Token) + + if ([string]::IsNullOrWhiteSpace($Token)) { + return $false + } + + try { + $response = Invoke-RestMethod -Method GET -Uri "$GitLabUrl/api/v4/version" -Headers @{ 'PRIVATE-TOKEN' = $Token } -TimeoutSec 30 -ErrorAction Stop + return -not [string]::IsNullOrWhiteSpace($response.version) + } catch { + return $false + } +} + +function Test-GitLabRegistryCredential { + param([Parameter(Mandatory)][string]$RegistryBasic) + + if ([string]::IsNullOrWhiteSpace($RegistryBasic) -or -not $RegistryBasic.Contains(':')) { + return $false + } + + try { + Invoke-WebRequest -Method GET -Uri "$GitLabRegistryUrl/v2/" -TimeoutSec 30 -ErrorAction Stop | Out-Null + } catch { + $statusCode = Get-HttpStatusCode -ErrorRecord $_ + if ($statusCode -ne 401) { + return $false + } + } + + try { + $encoded = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes($RegistryBasic)) + $response = Invoke-RestMethod -Method GET -Uri "$GitLabUrl/jwt/auth?service=container_registry&scope=registry:catalog:*" -Headers @{ Authorization = "Basic $encoded" } -TimeoutSec 30 -ErrorAction Stop + return -not [string]::IsNullOrWhiteSpace($response.token) + } catch { + return $false + } +} + +$vaultPathParts = Split-VaultSecretPath -Path $VaultSecretPath + +Write-Host "Waiting for Vault and GitLab to become ready..." -ForegroundColor Cyan +Wait-HttpReady -Name 'Vault' -Uri "$VaultUrl/v1/sys/health" -TimeoutSeconds $ReadinessTimeoutSeconds + +$existingSecret = Get-VaultGitLabSecret -VaultPathParts $vaultPathParts +if (-not $Rotate -and $null -ne $existingSecret) { + $apiValid = Test-GitLabApiToken -Token "$($existingSecret.'access-token')" + $registryValid = $true + if ($VerifyRegistry) { + $registryValid = Test-GitLabRegistryCredential -RegistryBasic "$($existingSecret.'registry-basic')" + } + + if ($apiValid -and $registryValid) { + Write-Host "Existing GitLab bootstrap secret at '$VaultSecretPath' is still valid; reusing it." -ForegroundColor Green + Write-Host "AuthRefs: authref://vault/gitlab#access-token, authref://vault/gitlab#registry-basic" + return + } + + Write-Host "Existing GitLab bootstrap secret at '$VaultSecretPath' is stale or incomplete; rotating it." -ForegroundColor Yellow +} + +Write-Host "Signing in to GitLab and reconciling the local bootstrap PAT..." -ForegroundColor Cyan +$adminHeaders = Get-GitLabAdminHeaders +$currentUser = Get-GitLabCurrentUser -Headers $adminHeaders +$expiresAt = (Get-Date).ToUniversalTime().AddDays($TokenLifetimeDays).ToString('yyyy-MM-dd') + +$tokensToRevoke = Get-GitLabPersonalAccessTokens -Headers $adminHeaders -UserId ([int]$currentUser.id) | + Where-Object { $_.name -eq $TokenName -and $_.active -and -not $_.revoked } + +foreach ($token in $tokensToRevoke) { + Revoke-GitLabPersonalAccessToken -Headers $adminHeaders -TokenId ([int]$token.id) +} + +$newToken = New-GitLabPersonalAccessToken -Headers $adminHeaders -UserId ([int]$currentUser.id) -Name $TokenName -Scopes @('api', 'read_registry') -ExpiresAt $expiresAt +$registryBasic = "${GitLabUsername}:$($newToken.token)" + +Write-Host "Verifying the new GitLab PAT against the API and registry surfaces..." -ForegroundColor Cyan +if (-not (Test-GitLabApiToken -Token "$($newToken.token)")) { + throw "The newly created GitLab personal access token failed the API probe." +} + +if ($VerifyRegistry -and -not (Test-GitLabRegistryCredential -RegistryBasic $registryBasic)) { + throw "The newly created GitLab personal access token failed the registry token exchange probe. Ensure the GitLab registry surface is enabled." +} + +$secretData = @{ + 'access-token' = "$($newToken.token)" + 'registry-basic' = $registryBasic + 'bootstrap-user' = $GitLabUsername + 'token-name' = $TokenName + 'expires-at' = "$($newToken.expires_at)" + 'rotated-at' = (Get-Date).ToUniversalTime().ToString('o') +} + +Set-VaultGitLabSecret -VaultPathParts $vaultPathParts -Data $secretData + +Write-Host "Bootstrapped GitLab PAT material into Vault path '$VaultSecretPath'." -ForegroundColor Green +Write-Host "AuthRefs: authref://vault/gitlab#access-token, authref://vault/gitlab#registry-basic" +Write-Host "Token name: $TokenName" +Write-Host "Expires at: $($newToken.expires_at)" diff --git a/scripts/register-local-integrations.ps1 b/scripts/register-local-integrations.ps1 new file mode 100644 index 000000000..9698c2635 --- /dev/null +++ b/scripts/register-local-integrations.ps1 @@ -0,0 +1,301 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Registers the locally reachable integration catalog entries for a tenant. +.DESCRIPTION + Uses the live Integrations API exposed by the local Docker stack to create + any missing local-capable providers, then runs test and health checks for + each entry so the catalog converges to a ready local lane. +.PARAMETER Tenant + Tenant identifier used for the catalog operations. Defaults to demo-prod. +.PARAMETER BaseUrl + Base URL for the local Integrations API. Defaults to the host-mapped + integrations-web endpoint. +.PARAMETER IncludeGitLab + Also register the GitLab Server and GitLab CI providers. This requires + authref://vault/gitlab#access-token to be populated in Vault. +.PARAMETER IncludeGitLabRegistry + Also register the GitLab Container Registry provider. This requires the + heavy GitLab profile with registry enabled plus authref://vault/gitlab#registry-basic. +.PARAMETER BootstrapGitLabSecrets + When used with `-IncludeGitLab` or `-IncludeGitLabRegistry`, bootstrap or + rotate the local GitLab PAT material into Vault automatically before the + GitLab-backed integrations are registered. +#> +[CmdletBinding()] +param( + [string]$Tenant = 'demo-prod', + [string]$BaseUrl = 'http://127.1.0.42', + [switch]$IncludeGitLab, + [switch]$IncludeGitLabRegistry, + [switch]$BootstrapGitLabSecrets +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' +$BaseUrl = $BaseUrl.TrimEnd('/') + +$Headers = @{ + 'X-StellaOps-Tenant' = $Tenant + 'X-StellaOps-Actor' = 'local-scratch-setup' +} + +if ($BootstrapGitLabSecrets -and ($IncludeGitLab -or $IncludeGitLabRegistry)) { + & (Join-Path $PSScriptRoot 'bootstrap-local-gitlab-secrets.ps1') -VerifyRegistry:$IncludeGitLabRegistry +} + +function Invoke-IntegrationApi { + param( + [Parameter(Mandatory)] + [ValidateSet('GET', 'POST')] + [string]$Method, + + [Parameter(Mandatory)] + [string]$Path, + + [object]$Body + ) + + $invokeParameters = @{ + Method = $Method + Uri = "$BaseUrl$Path" + Headers = $Headers + TimeoutSec = 30 + ErrorAction = 'Stop' + } + + if ($null -ne $Body) { + $invokeParameters['ContentType'] = 'application/json' + $invokeParameters['Body'] = $Body | ConvertTo-Json -Depth 10 + } + + return Invoke-RestMethod @invokeParameters +} + +function Get-HealthName { + param([int]$Status) + + switch ($Status) { + 1 { return 'Healthy' } + 2 { return 'Degraded' } + 3 { return 'Unhealthy' } + default { return 'Unknown' } + } +} + +function New-IntegrationDefinition { + param( + [Parameter(Mandatory)][string]$Name, + [Parameter(Mandatory)][string]$Description, + [Parameter(Mandatory)][int]$Type, + [Parameter(Mandatory)][int]$Provider, + [Parameter(Mandatory)][string]$Endpoint, + [string]$AuthRefUri, + [string]$OrganizationId, + [hashtable]$ExtendedConfig, + [string[]]$Tags + ) + + return [ordered]@{ + name = $Name + description = $Description + type = $Type + provider = $Provider + endpoint = $Endpoint + authRefUri = $AuthRefUri + organizationId = $OrganizationId + extendedConfig = $ExtendedConfig + tags = $Tags + } +} + +$definitions = @( + (New-IntegrationDefinition ` + -Name 'Local Harbor Fixture' ` + -Description 'Local Harbor mock fixture for registry onboarding and health checks.' ` + -Type 1 ` + -Provider 100 ` + -Endpoint 'http://harbor-fixture.stella-ops.local' ` + -OrganizationId 'local-fixtures' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'registry')), + (New-IntegrationDefinition ` + -Name 'Local Docker Registry' ` + -Description 'Local open OCI registry for catalog and tag probe validation.' ` + -Type 1 ` + -Provider 104 ` + -Endpoint 'http://registry.stella-ops.local:5000' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'registry')), + (New-IntegrationDefinition ` + -Name 'Local Nexus Registry' ` + -Description 'Local Nexus Repository Manager for registry integration checks.' ` + -Type 1 ` + -Provider 107 ` + -Endpoint 'http://nexus.stella-ops.local:8081' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'registry')), + (New-IntegrationDefinition ` + -Name 'Local GitHub App Fixture' ` + -Description 'Deterministic GitHub App fixture for SCM integration checks.' ` + -Type 2 ` + -Provider 200 ` + -Endpoint 'http://github-app-fixture.stella-ops.local' ` + -OrganizationId 'local-fixtures' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'scm')), + (New-IntegrationDefinition ` + -Name 'Local Gitea Server' ` + -Description 'Local Gitea service for SCM connectivity and repository discovery.' ` + -Type 2 ` + -Provider 203 ` + -Endpoint 'http://gitea.stella-ops.local:3000' ` + -OrganizationId 'local' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'scm')), + (New-IntegrationDefinition ` + -Name 'Local Jenkins' ` + -Description 'Local Jenkins service for CI/CD integration checks.' ` + -Type 3 ` + -Provider 302 ` + -Endpoint 'http://jenkins.stella-ops.local:8080' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'cicd')), + (New-IntegrationDefinition ` + -Name 'Local eBPF Runtime Host' ` + -Description 'Local runtime-host fixture exposing the eBPF agent contract.' ` + -Type 5 ` + -Provider 500 ` + -Endpoint 'http://runtime-host-fixture.stella-ops.local' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'runtime-host')), + (New-IntegrationDefinition ` + -Name 'Local StellaOps Mirror' ` + -Description 'Local Concelier mirror health surface for the StellaOps mirror provider.' ` + -Type 6 ` + -Provider 600 ` + -Endpoint 'http://concelier.stella-ops.local' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'feed-mirror')), + (New-IntegrationDefinition ` + -Name 'Local NVD Mirror' ` + -Description 'Local Concelier mirror health surface for the NVD mirror provider.' ` + -Type 6 ` + -Provider 601 ` + -Endpoint 'http://concelier.stella-ops.local' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'feed-mirror')), + (New-IntegrationDefinition ` + -Name 'Local OSV Mirror' ` + -Description 'Local Concelier mirror health surface for the OSV mirror provider.' ` + -Type 6 ` + -Provider 602 ` + -Endpoint 'http://concelier.stella-ops.local' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'feed-mirror')), + (New-IntegrationDefinition ` + -Name 'Local Vault' ` + -Description 'Local HashiCorp Vault dev server for secrets integration checks.' ` + -Type 9 ` + -Provider 550 ` + -Endpoint 'http://vault.stella-ops.local:8200' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'secrets')), + (New-IntegrationDefinition ` + -Name 'Local Consul' ` + -Description 'Local Consul server for settings and service-discovery checks.' ` + -Type 9 ` + -Provider 551 ` + -Endpoint 'http://consul.stella-ops.local:8500' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'secrets')), + (New-IntegrationDefinition ` + -Name 'Local MinIO' ` + -Description 'Local MinIO server for S3-compatible storage integration checks.' ` + -Type 10 ` + -Provider 450 ` + -Endpoint 'http://minio.stella-ops.local:9000' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'storage')) +) + +if ($IncludeGitLab) { + $definitions += @( + (New-IntegrationDefinition ` + -Name 'Local GitLab Server' ` + -Description 'Local GitLab server for SCM connectivity and discovery probes.' ` + -Type 2 ` + -Provider 201 ` + -Endpoint 'http://gitlab.stella-ops.local:8929' ` + -AuthRefUri 'authref://vault/gitlab#access-token' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'scm')), + (New-IntegrationDefinition ` + -Name 'Local GitLab CI' ` + -Description 'Local GitLab CI surface for CI/CD connectivity checks.' ` + -Type 3 ` + -Provider 301 ` + -Endpoint 'http://gitlab.stella-ops.local:8929' ` + -AuthRefUri 'authref://vault/gitlab#access-token' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'cicd')) + ) +} + +if ($IncludeGitLabRegistry) { + $definitions += New-IntegrationDefinition ` + -Name 'Local GitLab Container Registry' ` + -Description 'Local GitLab container registry surface. Requires authref://vault/gitlab#registry-basic.' ` + -Type 1 ` + -Provider 109 ` + -Endpoint 'http://gitlab.stella-ops.local:5050' ` + -AuthRefUri 'authref://vault/gitlab#registry-basic' ` + -ExtendedConfig @{ scheduleType = 'manual' } ` + -Tags @('local', 'scratch-setup', 'registry') +} + +$existingResponse = Invoke-IntegrationApi -Method GET -Path '/api/v1/integrations?pageSize=200' +$existingItems = @($existingResponse.items) +$results = New-Object System.Collections.Generic.List[object] + +foreach ($definition in $definitions) { + $match = $existingItems | Where-Object { + $_.provider -eq $definition.provider -and $_.endpoint -eq $definition.endpoint + } | Select-Object -First 1 + + if ($null -eq $match) { + $created = Invoke-IntegrationApi -Method POST -Path '/api/v1/integrations/' -Body $definition + $id = $created.id + $action = 'created' + } else { + $id = $match.id + $action = 'existing' + } + + $test = Invoke-IntegrationApi -Method POST -Path "/api/v1/integrations/$id/test" + $health = Invoke-IntegrationApi -Method GET -Path "/api/v1/integrations/$id/health" + + $results.Add([pscustomobject]@{ + Name = $definition.name + Provider = $definition.provider + Action = $action + TestSuccess = [bool]$test.success + Health = Get-HealthName -Status ([int]$health.status) + Endpoint = $definition.endpoint + Id = "$id" + }) +} + +$results | + Sort-Object Name | + Format-Table Name, Action, TestSuccess, Health, Endpoint -AutoSize | + Out-String | + Write-Host + +$failures = @($results | Where-Object { -not $_.TestSuccess -or $_.Health -ne 'Healthy' }) +if ($failures.Count -gt 0) { + Write-Error "Local integration registration completed with $($failures.Count) failing or non-healthy entry/entries for tenant '$Tenant'." + exit 1 +} + +Write-Host "Registered and verified $($results.Count) local integration entries for tenant '$Tenant' via $BaseUrl." -ForegroundColor Green