Update docs, sprint plans, and compose configuration

Add 12 new sprint files (Integrations, Graph, JobEngine, FE, Router, AdvisoryAI), archive completed scheduler UI sprint, update module architecture docs (router, graph, jobengine, web, integrations), and add Gitea entrypoint script for local dev. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 08:53:50 +03:00
parent 8e823792a3
commit 50abd2137f
36 changed files with 1723 additions and 69 deletions
--- a/devops/compose/README.md
+++ b/devops/compose/README.md
@@ -9,8 +9,9 @@ Consolidated Docker Compose configuration for the StellaOps platform. All profil
 | Run the full platform | `docker compose -f docker-compose.stella-ops.yml up -d` |
 | Add observability | `docker compose -f docker-compose.stella-ops.yml -f docker-compose.telemetry.yml up -d` |
 | Start QA integration fixtures | `docker compose -f docker-compose.integration-fixtures.yml up -d` |
-| Start 3rd-party integration services | `docker compose -f docker-compose.integrations.yml up -d` |
-| Start GitLab CE (heavy, ~4 GB RAM) | `docker compose -f docker-compose.integrations.yml --profile heavy up -d gitlab` |
+| Start the default low-idle 3rd-party integration lane | `docker compose -f docker-compose.integrations.yml up -d` |
+| Start Consul KV only when needed | `docker compose -f docker-compose.integrations.yml --profile consul up -d consul` |
+| Start GitLab CE (heavy, low-idle defaults) | `docker compose -f docker-compose.integrations.yml --profile heavy up -d gitlab` |
 | Run integration E2E test suite | See [Integration Test Suite](#integration-test-suite) |
 | Run CI/testing infrastructure | `docker compose -f docker-compose.testing.yml --profile ci up -d` |
 | Deploy with China compliance | See [China Compliance](#china-compliance-sm2sm3sm4) |
@@ -30,7 +31,7 @@ Consolidated Docker Compose configuration for the StellaOps platform. All profil
 | `docker-compose.telemetry.yml` | **Observability**: OpenTelemetry collector, Prometheus, Tempo, Loki |
 | `docker-compose.testing.yml` | **CI/Testing**: Test databases, mock services, Gitea for integration tests |
 | `docker-compose.dev.yml` | **Minimal dev infrastructure**: PostgreSQL, Valkey, RustFS only |
-| `docker-compose.integrations.yml` | **Integration services**: Gitea, Jenkins, Nexus, Vault, Docker Registry, MinIO, GitLab |
+| `docker-compose.integrations.yml` | **Integration services**: Gitea, Jenkins, Nexus, Vault, Docker Registry, MinIO, plus opt-in Consul and GitLab |

 ### Specialized Infrastructure

@@ -158,6 +159,8 @@ docker compose -f docker-compose.stella-ops.yml up -d
 pwsh ./scripts/router-mode-redeploy.ps1 -Mode microservice
 ```

+The local compose defaults intentionally keep router control traffic calm: `ROUTER_MESSAGING_HEARTBEAT_INTERVAL` defaults to `30s` so the stack does not churn small heartbeat traffic every 10 seconds across the full service fleet. Messaging endpoint/schema/OpenAPI replay is no longer periodic; it now happens on service startup, gateway-state recovery, or explicit administration resync. `ROUTER_REGISTRATION_REFRESH_INTERVAL_SECONDS` remains exposed only as a compatibility knob for older assumptions and non-messaging experiments.
+
 Validation endpoints:

 ```bash
@@ -241,20 +244,31 @@ These fixtures are deterministic QA aids only; they are not production dependenc
 Real 3rd-party services for local integration testing. Unlike the QA fixtures above (which are nginx mocks), these are fully functional instances that exercise actual connector plugin code paths.

 ```bash
-# Start all lightweight integration services (after the main stack is up)
+# Start the default low-idle integration lane (after the main stack is up)
 docker compose -f docker-compose.integrations.yml up -d

 # Start specific services only
 docker compose -f docker-compose.integrations.yml up -d gitea vault jenkins

+# Start Consul only when you need the Consul connector
+docker compose -f docker-compose.integrations.yml --profile consul up -d consul
+
 # Start GitLab CE (heavy — requires ~4 GB RAM, ~3 min startup)
+# Default GitLab tuning keeps SCM/API coverage and disables registry extras.
 docker compose -f docker-compose.integrations.yml --profile heavy up -d gitlab

+# Re-enable GitLab registry/package surfaces for dedicated registry tests
+GITLAB_ENABLE_REGISTRY=true GITLAB_ENABLE_PACKAGES=true \
+  docker compose -f docker-compose.integrations.yml --profile heavy up -d gitlab
+
 # Combine with mock fixtures for full coverage
 docker compose \
  -f docker-compose.integrations.yml \
  -f docker-compose.integration-fixtures.yml \
  up -d
+
+# Confirm the deterministic Gitea bootstrap completed
+docker compose -f docker-compose.integrations.yml ps gitea
 ```

 **Hosts file entries** (add to `C:\Windows\System32\drivers\etc\hosts`):
@@ -266,19 +280,21 @@ docker compose \
 127.1.2.5  registry.stella-ops.local
 127.1.2.6  minio.stella-ops.local
 127.1.2.7  gitlab.stella-ops.local
+127.1.2.8  consul.stella-ops.local
 ```

 **Service reference:**

 | Service | Type | Address | Credentials | Integration Provider |
 |---------|------|---------|-------------|---------------------|
-| Gitea | SCM | `http://gitea.stella-ops.local:3000` | Create on first login | `Gitea` |
+| Gitea | SCM | `http://gitea.stella-ops.local:3000` | `stellaops` / `Stella2026!` on fresh volumes | `Gitea` |
 | Jenkins | CI/CD | `http://jenkins.stella-ops.local:8080` | Setup wizard disabled | `Jenkins` |
 | Nexus | Registry | `http://nexus.stella-ops.local:8081` | admin / see `admin.password` | `Nexus` |
 | Vault | Secrets | `http://vault.stella-ops.local:8200` | Token: `stellaops-dev-root-token-2026` | — |
+| Consul | Settings/KV | `http://consul.stella-ops.local:8500` | none (single-node local server, opt-in profile) | `Consul` |
 | Docker Registry | Registry | `http://registry.stella-ops.local:5000` | None (open dev) | `DockerHub` |
 | MinIO | S3 Storage | `http://minio.stella-ops.local:9001` | `stellaops` / `Stella2026!` | — |
-| GitLab CE | SCM+CI+Registry | `http://gitlab.stella-ops.local:8929` | root / `Stella2026!` | `GitLabServer` |
+| GitLab CE | SCM+CI(+Registry opt-in) | `http://gitlab.stella-ops.local:8929` | root / `Stella2026!` | `GitLabServer` |

 **Credential resolution:** Integration connectors resolve secrets via `authref://vault/{path}#{key}` URIs. The Integrations service resolves these from Vault automatically in dev mode. Store credentials with:

@@ -294,6 +310,10 @@ vault kv put secret/jenkins api-token="user:token"
 vault kv put secret/nexus admin-password="your-password"
 ```

+Gitea is now bootstrapped by the compose service itself: a fresh `stellaops-gitea-data` volume creates the default local admin user and the repository root before the container reports healthy. Personal access tokens remain a manual step because Gitea only reveals the token value when it is created.
+
+`docker-compose.testing.yml` is a separate infrastructure-test lane. It starts `postgres-test`, `valkey-test`, mocks, and an isolated Gitea profile on different ports; it does not start Consul or GitLab. Use `docker-compose.integrations.yml` only when you need real third-party providers for connector validation.
+
 **Backend connector plugins** (8 total, loaded in Integrations service):

 | Plugin | Type | Provider | Health Endpoint |
@@ -327,6 +347,7 @@ vault kv put secret/nexus admin-password="your-password"
 | 127.1.2.5 | docker-registry | 5000 |
 | 127.1.2.6 | minio | 9000, 9001 |
 | 127.1.2.7 | gitlab (heavy) | 8929, 2224, 5050 |
+| 127.1.2.8 | consul (optional) | 8500 |

 For detailed setup instructions per service, see [`docs/integrations/LOCAL_SERVICES.md`](../../docs/integrations/LOCAL_SERVICES.md).

@@ -354,6 +375,9 @@ docker compose -f docker-compose.integration-fixtures.yml up -d
 # 3. Start real 3rd-party services
 docker compose -f docker-compose.integrations.yml up -d

+# 3a. (Optional) Start Consul only when validating the Consul connector
+docker compose -f docker-compose.integrations.yml --profile consul up -d consul
+
 # 4. (Optional) Start GitLab for full SCM coverage
 docker compose -f docker-compose.integrations.yml --profile heavy up -d gitlab
 ```
--- a/devops/compose/docker-compose.integrations.yml
+++ b/devops/compose/docker-compose.integrations.yml
@@ -8,12 +8,16 @@
 #   The main stellaops network must exist (started via docker-compose.stella-ops.yml).
 #
 # Usage:
-#   # Start all integration services
+#   # Start the default low-idle integration lane
 #   docker compose -f devops/compose/docker-compose.integrations.yml up -d
 #
 #   # Start specific services only
 #   docker compose -f devops/compose/docker-compose.integrations.yml up -d gitea jenkins vault
 #
+#   # Start optional higher-idle providers only when needed
+#   docker compose -f devops/compose/docker-compose.integrations.yml --profile consul up -d consul
+#   docker compose -f devops/compose/docker-compose.integrations.yml --profile heavy up -d gitlab
+#
 #   # Start integration services + mock fixtures together
 #   docker compose \
 #     -f devops/compose/docker-compose.integrations.yml \
@@ -42,8 +46,8 @@ networks:
 volumes:
  gitea-data:
    name: stellaops-gitea-data
-  gitea-db:
-    name: stellaops-gitea-db
+  gitea-config:
+    name: stellaops-gitea-config
  jenkins-data:
    name: stellaops-jenkins-data
  nexus-data:
@@ -60,6 +64,8 @@ volumes:
    name: stellaops-gitlab-data
  gitlab-logs:
    name: stellaops-gitlab-logs
+  consul-data:
+    name: stellaops-consul-data

 services:
  # ===========================================================================
@@ -67,39 +73,49 @@ services:
  # ===========================================================================
  # Integration type: SCM (Gitea provider)
  # URL: http://gitea.stella-ops.local:3000
-  # Admin: stellaops / Stella2026!
+  # Admin: stellaops / Stella2026! (fresh volumes auto-bootstrap on container start)
  # API: http://gitea.stella-ops.local:3000/api/v1
  # ===========================================================================
  gitea:
    image: gitea/gitea:1.22-rootless
    container_name: stellaops-gitea
    restart: unless-stopped
+    entrypoint: ["/bin/sh", "/stellaops-gitea-entrypoint.sh"]
    ports:
      - "127.1.2.1:3000:3000"
      - "127.1.2.1:2222:2222"
    environment:
      - GITEA__database__DB_TYPE=sqlite3
+      - GITEA__database__PATH=/var/lib/gitea/data/gitea.db
      - GITEA__server__ROOT_URL=http://gitea.stella-ops.local:3000
      - GITEA__server__DOMAIN=gitea.stella-ops.local
      - GITEA__server__HTTP_PORT=3000
      - GITEA__server__SSH_PORT=2222
      - GITEA__server__SSH_DOMAIN=gitea.stella-ops.local
-      - GITEA__service__DISABLE_REGISTRATION=false
+      - GITEA__service__DISABLE_REGISTRATION=true
      - GITEA__service__REQUIRE_SIGNIN_VIEW=false
      - GITEA__actions__ENABLED=true
      - GITEA__api__ENABLE_SWAGGER=true
      - GITEA__security__INSTALL_LOCK=true
      - GITEA__security__SECRET_KEY=stellaops-dev-secret-key-2026
      - GITEA__security__INTERNAL_TOKEN=stellaops-internal-token-2026-dev
+      - GITEA_LOCAL_ADMIN_USERNAME=stellaops
+      - GITEA_LOCAL_ADMIN_PASSWORD=Stella2026!
+      - GITEA_LOCAL_ADMIN_EMAIL=stellaops@gitea.stella-ops.local
    volumes:
      - gitea-data:/var/lib/gitea
-      - gitea-db:/var/lib/gitea/db
+      - gitea-config:/etc/gitea
+      - ./scripts/gitea-entrypoint.sh:/stellaops-gitea-entrypoint.sh:ro
    networks:
      stellaops:
        aliases:
          - gitea.stella-ops.local
    healthcheck:
-      test: ["CMD-SHELL", "wget -qO- http://localhost:3000/api/v1/version || exit 1"]
+      test:
+        [
+          "CMD-SHELL",
+          "wget -qO- http://localhost:3000/api/v1/version >/dev/null 2>&1 && test -f /var/lib/gitea/data/.local-admin-ready"
+        ]
      interval: 30s
      timeout: 10s
      retries: 5
@@ -114,7 +130,7 @@ services:
  # ===========================================================================
  # Integration type: CI/CD (Jenkins provider)
  # URL: http://jenkins.stella-ops.local:8080
-  # Admin: admin / Stella2026!
+  # Auth: anonymous access by default; create an admin user manually if you need authenticated API flows
  # API: http://jenkins.stella-ops.local:8080/api/json
  # ===========================================================================
  jenkins:
@@ -297,8 +313,12 @@ services:
  # ===========================================================================
  # Integration type: Secrets Manager (Consul provider)
  # URL: http://consul.stella-ops.local:8500
-  # No auth (dev mode)
+  # No auth (single-node local mode)
  # API: http://consul.stella-ops.local:8500/v1/status/leader
+  #
+  # Profile: consul - opt in only when validating the Consul connector.
+  # Runs as a single-node local server with the UI enabled. This preserves
+  # the HTTP KV surface while avoiding the higher idle CPU cost of `agent -dev`.
  # ===========================================================================
  consul:
    image: hashicorp/consul:1.19
@@ -306,21 +326,25 @@ services:
    restart: unless-stopped
    ports:
      - "127.1.2.8:8500:8500"
-    command: agent -dev -client=0.0.0.0
+    command: agent -server -bootstrap-expect=1 -ui -client=0.0.0.0 -data-dir=/consul/data -log-level=warn
+    volumes:
+      - consul-data:/consul/data
    networks:
      stellaops:
        aliases:
          - consul.stella-ops.local
    healthcheck:
      test: ["CMD-SHELL", "consul members || exit 1"]
-      interval: 15s
+      interval: 60s
      timeout: 5s
      retries: 5
      start_period: 10s
    labels:
      com.stellaops.integration: "secrets"
      com.stellaops.provider: "consul"
-      com.stellaops.profile: "integrations"
+      com.stellaops.profile: "integrations-optional"
+    profiles:
+      - consul

  # ===========================================================================
  # GITLAB CE — Full Git SCM + CI/CD + Container Registry (optional, heavy)
@@ -332,7 +356,10 @@ services:
  # Requires: ~4 GB RAM, ~2 min startup
  #
  # Profile: heavy — only start when explicitly requested:
-  #   docker compose -f docker-compose.integrations.yml up -d gitlab
+  #   docker compose -f docker-compose.integrations.yml --profile heavy up -d gitlab
+  #
+  # Local defaults bias for lower idle CPU. SCM/API coverage remains available,
+  # while registry/KAS extras stay disabled unless you opt in via env vars.
  # ===========================================================================
  gitlab:
    image: gitlab/gitlab-ce:17.8.1-ce.0
@@ -348,12 +375,20 @@ services:
        gitlab_rails['initial_root_password'] = 'Stella2026!'
        gitlab_rails['gitlab_shell_ssh_port'] = 2224
        registry_external_url 'http://gitlab.stella-ops.local:5050'
-        registry['enable'] = true
+        registry['enable'] = ${GITLAB_ENABLE_REGISTRY:-false}
+        registry_nginx['enable'] = ${GITLAB_ENABLE_REGISTRY:-false}
+        gitlab_kas['enable'] = false
        prometheus_monitoring['enable'] = false
-        sidekiq['max_concurrency'] = 5
-        puma['workers'] = 2
-        puma['min_threads'] = 1
-        puma['max_threads'] = 2
+        gitlab_rails['usage_ping_enabled'] = false
+        gitlab_rails['runners_registration_enabled'] = false
+        gitlab_rails['packages_enabled'] = ${GITLAB_ENABLE_PACKAGES:-false}
+        sidekiq['concurrency'] = ${GITLAB_SIDEKIQ_CONCURRENCY:-2}
+        sidekiq['metrics_enabled'] = false
+        sidekiq['health_checks_enabled'] = false
+        puma['worker_processes'] = ${GITLAB_PUMA_WORKERS:-1}
+        puma['min_threads'] = ${GITLAB_PUMA_MIN_THREADS:-1}
+        puma['max_threads'] = ${GITLAB_PUMA_MAX_THREADS:-2}
+        nginx['worker_processes'] = 1
        postgresql['shared_buffers'] = '128MB'
        gitlab_rails['env'] = { 'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000' }
    volumes:
--- a/devops/compose/docker-compose.stella-ops.yml
+++ b/devops/compose/docker-compose.stella-ops.yml
@@ -63,7 +63,8 @@ x-router-microservice-defaults: &router-microservice-defaults
  Router__Messaging__RequestTimeout: "30s"
  Router__Messaging__LeaseDuration: "5m"
  Router__Messaging__BatchSize: "10"
-  Router__Messaging__HeartbeatInterval: "10s"
+  Router__Messaging__HeartbeatInterval: "${ROUTER_MESSAGING_HEARTBEAT_INTERVAL:-30s}"
+  Router__RegistrationRefreshIntervalSeconds: "${ROUTER_REGISTRATION_REFRESH_INTERVAL_SECONDS:-30}"
  Router__Messaging__valkey__ConnectionString: "cache.stella-ops.local:6379"
  Router__Messaging__valkey__Database: "0"
  Router__Messaging__valkey__QueueWaitTimeoutSeconds: "${VALKEY_QUEUE_WAIT_TIMEOUT:-0}"
@@ -352,7 +353,7 @@ services:
      Gateway__Transports__Messaging__RequestTimeout: "30s"
      Gateway__Transports__Messaging__LeaseDuration: "5m"
      Gateway__Transports__Messaging__BatchSize: "10"
-      Gateway__Transports__Messaging__HeartbeatInterval: "10s"
+      Gateway__Transports__Messaging__HeartbeatInterval: "${ROUTER_MESSAGING_HEARTBEAT_INTERVAL:-30s}"
      # Identity envelope signing (gateway -> microservice auth)
      Gateway__Auth__IdentityEnvelopeSigningKey: "${STELLAOPS_IDENTITY_ENVELOPE_SIGNING_KEY}"
      # Audience validation disabled until authority includes aud in access tokens
@@ -2337,6 +2338,8 @@ services:
      ADVISORYAI__AdvisoryAI__Inference__Remote__BaseAddress: "${ADVISORY_AI_REMOTE_BASEADDRESS:-}"
      ADVISORYAI__AdvisoryAI__Inference__Remote__ApiKey: "${ADVISORY_AI_REMOTE_APIKEY:-}"
      ADVISORYAI__KnowledgeSearch__ConnectionString: *postgres-connection
+      ADVISORYAI__KnowledgeSearch__DatabaseApplicationName: "${ADVISORY_AI_KNOWLEDGESEARCH_DB_APPLICATION_NAME:-stellaops-advisory-ai-web/knowledge-search}"
+      ADVISORYAI__KnowledgeSearch__DatabaseConnectionIdleLifetimeSeconds: "${ADVISORY_AI_KNOWLEDGESEARCH_DB_IDLE_LIFETIME_SECONDS:-900}"
      ADVISORYAI__KnowledgeSearch__FindingsAdapterEnabled: "true"
      ADVISORYAI__KnowledgeSearch__FindingsAdapterBaseUrl: "http://scanner.stella-ops.local"
      ADVISORYAI__KnowledgeSearch__VexAdapterEnabled: "true"
--- a/devops/compose/scripts/gitea-entrypoint.sh
+++ b/devops/compose/scripts/gitea-entrypoint.sh
@@ -0,0 +1,66 @@
+#!/bin/sh
+
+set -eu
+
+GITEA_CONFIG="${GITEA_APP_INI:-/etc/gitea/app.ini}"
+GITEA_WORK_PATH="${GITEA_WORK_DIR:-/var/lib/gitea}"
+GITEA_HTTP_PORT="${GITEA__server__HTTP_PORT:-3000}"
+GITEA_ADMIN_USERNAME="${GITEA_LOCAL_ADMIN_USERNAME:-stellaops}"
+GITEA_ADMIN_PASSWORD="${GITEA_LOCAL_ADMIN_PASSWORD:-Stella2026!}"
+GITEA_ADMIN_EMAIL="${GITEA_LOCAL_ADMIN_EMAIL:-stellaops@gitea.stella-ops.local}"
+GITEA_BOOTSTRAP_SENTINEL="${GITEA_BOOTSTRAP_SENTINEL:-${GITEA_WORK_PATH}/data/.local-admin-ready}"
+
+mkdir -p "${GITEA_WORK_PATH}/git/repositories"
+
+/usr/local/bin/docker-entrypoint.sh "$@" &
+gitea_pid=$!
+
+cleanup() {
+    if kill -0 "${gitea_pid}" 2>/dev/null; then
+        kill "${gitea_pid}" 2>/dev/null || true
+        wait "${gitea_pid}" 2>/dev/null || true
+    fi
+}
+
+trap cleanup INT TERM
+
+ready=0
+attempt=0
+while [ "${attempt}" -lt 60 ]; do
+    if wget -qO- "http://127.0.0.1:${GITEA_HTTP_PORT}/api/v1/version" >/dev/null 2>&1; then
+        ready=1
+        break
+    fi
+
+    if ! kill -0 "${gitea_pid}" 2>/dev/null; then
+        wait "${gitea_pid}"
+        exit 1
+    fi
+
+    attempt=$((attempt + 1))
+    sleep 1
+done
+
+if [ "${ready}" -ne 1 ]; then
+    echo "Gitea did not become ready for local admin bootstrap" >&2
+    wait "${gitea_pid}"
+    exit 1
+fi
+
+existing_admins="$(gitea admin user list --config "${GITEA_CONFIG}" --work-path "${GITEA_WORK_PATH}" --admin | tail -n +2 | awk 'BEGIN { first = 1 } NF { if (!first) printf ","; printf "%s", $2; first = 0 }')"
+if [ -z "${existing_admins}" ]; then
+    gitea admin user create \
+        --config "${GITEA_CONFIG}" \
+        --work-path "${GITEA_WORK_PATH}" \
+        --username "${GITEA_ADMIN_USERNAME}" \
+        --password "${GITEA_ADMIN_PASSWORD}" \
+        --email "${GITEA_ADMIN_EMAIL}" \
+        --admin \
+        --must-change-password=false
+else
+    echo "Gitea admin bootstrap skipped; existing admin users: ${existing_admins}"
+fi
+
+touch "${GITEA_BOOTSTRAP_SENTINEL}"
+
+wait "${gitea_pid}"