stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 1 Releases Wiki Activity
Files
f98cea3bcfb4730b238ab8605ce6d48c2998aa44
git.stella-ops.org/docs/modules/orchestrator/architecture.md
master f98cea3bcf Add Authority Advisory AI and API Lifecycle Configuration 
- Introduced AuthorityAdvisoryAiOptions and related classes for managing advisory AI configurations, including remote inference options and tenant-specific settings.
- Added AuthorityApiLifecycleOptions to control API lifecycle settings, including legacy OAuth endpoint configurations.
- Implemented validation and normalization methods for both advisory AI and API lifecycle options to ensure proper configuration.
- Created AuthorityNotificationsOptions and its related classes for managing notification settings, including ack tokens, webhooks, and escalation options.
- Developed IssuerDirectoryClient and related models for interacting with the issuer directory service, including caching mechanisms and HTTP client configurations.
- Added support for dependency injection through ServiceCollectionExtensions for the Issuer Directory Client.
- Updated project file to include necessary package references for the new Issuer Directory Client library.
2025-11-02 13:50:25 +02:00

4.1 KiB
Raw Blame History

Source & Job Orchestrator architecture

Based on Epic 9 – Source & Job Orchestrator Dashboard; this section outlines components, job lifecycle, rate-limit governance, and observability.

1) Topology

  • Orchestrator API (StellaOps.Orchestrator). Minimal API providing job state, throttling controls, replay endpoints, and dashboard data. Authenticated via Authority scopes (orchestrator:*).
  • Job ledger (Mongo). Collections jobs, job_history, sources, quotas, throttles, incidents. Append-only history ensures auditability.
  • Queue abstraction. Supports Mongo queue, Redis Streams, or NATS JetStream (pluggable). Each job carries lease metadata and retry policy.
  • Dashboard feeds. SSE/GraphQL endpoints supply Console UI with job timelines, throughput, error distributions, and rate-limit status.

2) Job lifecycle

  1. Enqueue. Producer services (Concelier, Excititor, Scheduler, Export Center, Policy Engine) submit JobRequest records containing jobType, tenant, priority, payloadDigest, dependencies.
  2. Scheduling. Orchestrator applies quotas and rate limits per {tenant, jobType}. Jobs exceeding limits are staged in pending queue with next eligible timestamp.
  3. Leasing. Workers poll LeaseJob endpoint; Orchestrator returns job with leaseId, leaseUntil, and instrumentation tokens. Lease renewal required for long-running tasks.
  4. Completion. Worker reports status (succeeded, failed, canceled, timed_out). On success the job is archived; on failure Orchestrator applies retry policy (exponential backoff, max attempts). Incidents escalate to Ops if thresholds exceeded.
  5. Replay. Operators trigger POST /jobs/{id}/replay which clones job payload, sets replayOf pointer, and requeues with high priority while preserving determinism metadata.

3) Rate-limit & quota governance

  • Quotas defined per tenant/profile (maxActive, maxPerHour, burst). Stored in quotas and enforced before leasing.
  • Dynamic throttles allow ops to pause specific sources (pauseSource, resumeSource) or reduce concurrency.
  • Circuit breakers automatically pause job types when failure rate > configured threshold; incidents generated via Notify and Observability stack.
  • Control plane quota updates require Authority scope orch:quota (issued via Orch.Admin role). Token requests include quota_reason (mandatory) and optional quota_ticket; Authority persists both values for audit replay.

4) APIs

  • GET /api/jobs?status= — list jobs with filters (tenant, jobType, status, time window).
  • GET /api/jobs/{id} — job detail (payload digest, attempts, worker, lease history, metrics).
  • POST /api/jobs/{id}/cancel — cancel running/pending job with audit reason.
  • POST /api/jobs/{id}/replay — schedule replay.
  • POST /api/limits/throttle — apply throttle (requires elevated scope).
  • GET /api/dashboard/metrics — aggregated metrics for Console dashboards.

All responses include deterministic timestamps, job digests, and DSSE signature fields for offline reconciliation.

5) Observability

  • Metrics: job_queue_depth{jobType,tenant}, job_latency_seconds, job_failures_total, job_retry_total, lease_extensions_total.
  • Logs: structured with jobId, jobType, tenant, workerId, leaseId, status. Incident logs flagged for Ops.
  • Traces: spans covering enqueue, schedule, lease, worker_execute, complete. Trace IDs propagate to worker spans for end-to-end correlation.

6) Offline support

  • Orchestrator exports audit bundles: jobs.jsonl, history.jsonl, throttles.jsonl, manifest.json, signatures/. Used for offline investigations and compliance.
  • Replay manifests contain job digests and success/failure notes for deterministic proof.

7) Operational considerations

  • HA deployment with multiple API instances; queue storage determines redundancy strategy.
  • Support for maintenance mode halting leases while allowing status inspection.
  • Runbook includes procedures for expanding quotas, blacklisting misbehaving tenants, and recovering stuck jobs (clearing leases, applying pause/resume).