stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
88a85cdd92394204a7a8877e62f9f32df07848d9
git.stella-ops.org/docs/modules/advisory-ai/runs.md
master 17d0631b8e sprints work
2026-01-10 20:38:13 +02:00

18 KiB
Raw Blame History

AI Runs Framework

Sprint: SPRINT_20260109_011_003_BE_ai_runs_framework Status: Active Last Updated: 2026-01-10

The AI Runs Framework provides an auditable container for AI-assisted investigations, capturing the complete lifecycle from initial query through tool calls, artifact generation, and approvals.

Overview

"Chat is not auditable, repeatable, actionable with guardrails, or collaborative."

The Run concept transforms ephemeral chat into:

  • Auditable: Every interaction logged with timestamps and content digests
  • Repeatable: Deterministic replay possible for verification
  • Actionable: Artifacts produced (evidence packs, VEX statements, decisions)
  • Collaborative: Handoffs, approvals, shared context across team members

Run Lifecycle

Runs progress through defined states with explicit transitions:

Created -> Active -> PendingApproval -> Completed
                  \-> Cancelled
                  \-> Failed

States

State Description Allowed Transitions
Created Run initialized, no activity yet Active, Cancelled
Active Conversation in progress PendingApproval, Completed, Cancelled, Failed
PendingApproval Action requires user approval Active, Completed, Cancelled
Completed Run finished, attestation generated (terminal)
Cancelled Run cancelled by user (terminal)
Failed Run failed due to error (terminal)

Lifecycle Diagram

+----------+    +---------+    +------------+    +----------+
| Created  | -> | Active  | -> | Pending    | -> | Complete |
|          |    |         |    | Approval   |    |          |
+----------+    +---------+    +------------+    +----------+
     |              |               |                 |
     v              v               v                 v
+--------------------------------------------------+
|                 Run Timeline                      |
|  +-------+ +------+ +---------+ +--------+ +---+ |
|  |Created| | User | |Assistant| | Action | |...| |
|  | Event | | Turn | |  Turn   | |Proposed| |   | |
|  +-------+ +------+ +---------+ +--------+ +---+ |
+--------------------------------------------------+
                        |
                        v
+--------------------------------------------------+
|              Artifacts Produced                   |
|  +---------+ +--------+ +------+ +-----+         |
|  | Evidence| |Decision| |Action| | VEX |         |
|  |   Pack  | | Record | |Result| |Stmt |         |
|  +---------+ +--------+ +------+ +-----+         |
+--------------------------------------------------+
                        |
                        v
+--------------------------------------------------+
|           Run Attestation (DSSE)                  |
|  - Content digest of all turns                   |
|  - Evidence references                           |
|  - Artifact digests                              |
|  - Signed by platform key                        |
+--------------------------------------------------+

API Reference

Create Run

Creates a new Run from an existing conversation.

POST /api/v1/advisory-ai/runs
Content-Type: application/json
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>

{
  "conversationId": "conv-abc123",
  "context": {
    "findingId": "f-456",
    "cveId": "CVE-2024-1234",
    "component": "pkg:npm/lodash@4.17.21",
    "scanId": "scan-789",
    "sbomId": "sbom-xyz"
  }
}

Response (201 Created):

{
  "runId": "run-abc123",
  "tenantId": "tenant-xyz",
  "userId": "user@example.com",
  "conversationId": "conv-abc123",
  "status": "Created",
  "createdAt": "2026-01-10T14:30:00Z",
  "context": {
    "findingId": "f-456",
    "cveId": "CVE-2024-1234",
    "component": "pkg:npm/lodash@4.17.21",
    "scanId": "scan-789",
    "sbomId": "sbom-xyz"
  },
  "timeline": [],
  "artifacts": []
}

Get Run

Retrieves a Run with its complete timeline and artifacts.

GET /api/v1/advisory-ai/runs/{runId}
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>

Response (200 OK):

{
  "runId": "run-abc123",
  "tenantId": "tenant-xyz",
  "userId": "user@example.com",
  "conversationId": "conv-abc123",
  "status": "Active",
  "createdAt": "2026-01-10T14:30:00Z",
  "completedAt": null,
  "context": { ... },
  "timeline": [
    {
      "eventId": "evt-001",
      "eventType": "RunCreated",
      "timestamp": "2026-01-10T14:30:00Z",
      "actor": "system",
      "summary": "Run created from conversation conv-abc123"
    },
    {
      "eventId": "evt-002",
      "eventType": "UserTurn",
      "timestamp": "2026-01-10T14:30:05Z",
      "actor": "user:user@example.com",
      "summary": "Is CVE-2024-1234 exploitable in our environment?",
      "relatedTurnId": "turn-001"
    },
    {
      "eventId": "evt-003",
      "eventType": "AssistantTurn",
      "timestamp": "2026-01-10T14:30:08Z",
      "actor": "assistant",
      "summary": "Based on the reachability analysis...",
      "details": {
        "contentDigest": "sha256:abc123...",
        "groundingScore": 0.92
      },
      "relatedTurnId": "turn-002"
    }
  ],
  "artifacts": [],
  "attestationDigest": null
}

Get Timeline

Returns paginated timeline events for a Run.

GET /api/v1/advisory-ai/runs/{runId}/timeline?limit=50&cursor=evt-050
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>

Response (200 OK):

{
  "runId": "run-abc123",
  "events": [ ... ],
  "cursor": "evt-100",
  "hasMore": true
}

Get Artifacts

Lists all artifacts attached to a Run.

GET /api/v1/advisory-ai/runs/{runId}/artifacts
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>

Response (200 OK):

{
  "runId": "run-abc123",
  "artifacts": [
    {
      "artifactId": "art-001",
      "type": "EvidencePack",
      "name": "CVE-2024-1234 Evidence Pack",
      "contentDigest": "sha256:def456...",
      "uri": "evidence://packs/art-001",
      "createdAt": "2026-01-10T14:35:00Z",
      "metadata": {
        "cveId": "CVE-2024-1234",
        "component": "pkg:npm/lodash@4.17.21"
      }
    }
  ]
}

Complete Run

Marks a Run as complete and generates the attestation.

POST /api/v1/advisory-ai/runs/{runId}/complete
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>

Response (200 OK):

{
  "runId": "run-abc123",
  "status": "Completed",
  "completedAt": "2026-01-10T14:40:00Z",
  "attestationDigest": "sha256:xyz789...",
  "attestation": {
    "runId": "run-abc123",
    "tenantId": "tenant-xyz",
    "userId": "user@example.com",
    "modelInfo": {
      "modelId": "gpt-4-turbo",
      "modelVersion": "2024-04-09"
    },
    "turns": [ ... ],
    "overallGroundingScore": 0.89
  }
}

Cancel Run

Cancels an active Run.

POST /api/v1/advisory-ai/runs/{runId}/cancel
Content-Type: application/json
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>

{
  "reason": "Investigation no longer needed - issue resolved"
}

Response (200 OK):

{
  "runId": "run-abc123",
  "status": "Cancelled",
  "cancelledAt": "2026-01-10T14:35:00Z",
  "reason": "Investigation no longer needed - issue resolved"
}

Replay Run

Replays a Run for verification and determinism checking.

POST /api/v1/advisory-ai/runs/{runId}/replay
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>

Response (200 OK):

{
  "runId": "run-abc123",
  "deterministic": true,
  "originalDigest": "sha256:abc123...",
  "replayDigest": "sha256:abc123...",
  "differences": []
}

Response (Non-deterministic):

{
  "runId": "run-abc123",
  "deterministic": false,
  "originalDigest": "sha256:abc123...",
  "replayDigest": "sha256:def456...",
  "differences": [
    "Turn 2: original=sha256:111..., replay=sha256:222..."
  ]
}

List Runs

Lists Runs with filtering and pagination.

GET /api/v1/advisory-ai/runs?userId=user@example.com&status=Active&limit=20
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>

Query Parameters:

Parameter Type Description
userId string Filter by user
findingId string Filter by finding
status string Filter by status (Created, Active, PendingApproval, Completed, Cancelled, Failed)
since datetime Runs created after this time
until datetime Runs created before this time
limit integer Page size (default: 50, max: 100)
cursor string Pagination cursor

Response (200 OK):

{
  "runs": [ ... ],
  "cursor": "run-xyz",
  "hasMore": true
}

Timeline Event Types

The timeline captures all significant events during a Run.

Event Type Actor Description
RunCreated system Run initialized
UserTurn user:{userId} User message added
AssistantTurn assistant AI response generated
ToolCall assistant AI invoked a tool (search, lookup, etc.)
ActionProposed assistant AI proposed an action
ApprovalRequested system Action requires user approval
ApprovalGranted user:{userId} User approved action
ApprovalDenied user:{userId} User denied action
ActionExecuted system Action was executed
ActionFailed system Action execution failed
ArtifactCreated system Artifact attached to Run
RunCompleted system Run completed with attestation
RunCancelled user:{userId} Run cancelled
RunFailed system Run failed due to error

Event Details

Each event type may include additional details:

AssistantTurn:

{
  "contentDigest": "sha256:...",
  "groundingScore": 0.92,
  "tokenCount": 847
}

ActionProposed:

{
  "actionType": "approve",
  "label": "Accept Risk",
  "parameters": {
    "cveId": "CVE-2024-1234",
    "rationale": "Not exploitable in our environment"
  }
}

ArtifactCreated:

{
  "artifactId": "art-001",
  "artifactType": "EvidencePack",
  "contentDigest": "sha256:..."
}

Artifact Types

Runs can produce various artifact types:

Type Description Use Case
EvidencePack Bundle of evidence supporting a decision Risk acceptance, VEX justification
DecisionRecord Formal record of a security decision Audit trail, compliance
VexStatement VEX statement draft or final Vulnerability disclosure
ActionResult Result of an executed action Remediation tracking
Explanation AI-generated explanation User understanding
Report Generated report document Stakeholder communication

Artifact Structure

{
  "artifactId": "art-001",
  "type": "EvidencePack",
  "name": "CVE-2024-1234 Evidence Pack",
  "contentDigest": "sha256:def456...",
  "uri": "evidence://packs/art-001",
  "createdAt": "2026-01-10T14:35:00Z",
  "metadata": {
    "cveId": "CVE-2024-1234",
    "component": "pkg:npm/lodash@4.17.21",
    "scanId": "scan-789",
    "format": "evidence-pack/v1"
  }
}

Replay Verification

Runs can be replayed to verify determinism. This is crucial for:

  • Compliance audits: Proving AI outputs are reproducible
  • Debugging: Understanding how AI reached conclusions
  • Trust: Demonstrating consistent behavior

Replay Requirements

For successful replay:

  1. Temperature = 0: No randomness in token selection
  2. Fixed seed: Same seed across replays
  3. Model match: Same model weights (verified by digest)
  4. Prompt match: Identical prompts (verified by hash)
  5. Context match: Same input context

Replay Results

Result Meaning
deterministic: true Replay produced identical output
deterministic: false Output differs (see differences array)

Common Divergence Causes

Cause Detection Resolution
Different model Model version mismatch Use pinned model version
Non-zero temperature Parameter check Set temperature to 0
Different seed Seed mismatch Use consistent seed
Prompt template change Template version mismatch Pin template version
Context ordering Context hash mismatch Sort context deterministically

Run Attestation

Completed Runs produce DSSE-signed attestations containing:

{
  "_type": "https://stellaops.org/attestation/ai-run/v1",
  "runId": "run-abc123",
  "tenantId": "tenant-xyz",
  "userId": "user@example.com",
  "conversationId": "conv-abc123",
  "startedAt": "2026-01-10T14:30:00Z",
  "completedAt": "2026-01-10T14:40:00Z",
  "model": {
    "modelId": "gpt-4-turbo",
    "modelVersion": "2024-04-09",
    "provider": "azure-openai"
  },
  "promptTemplate": {
    "templateId": "security-investigate",
    "version": "1.2.0",
    "digest": "sha256:..."
  },
  "context": {
    "findingId": "f-456",
    "cveId": "CVE-2024-1234",
    "policyId": "policy-001",
    "evidenceUris": [
      "sbom://scan-789/lodash",
      "reach://api-gateway:lodash.get"
    ]
  },
  "turns": [
    {
      "turnId": "turn-001",
      "role": "User",
      "contentDigest": "sha256:...",
      "timestamp": "2026-01-10T14:30:05Z"
    },
    {
      "turnId": "turn-002",
      "role": "Assistant",
      "contentDigest": "sha256:...",
      "timestamp": "2026-01-10T14:30:08Z",
      "claims": [
        {
          "text": "CVE-2024-1234 is reachable through...",
          "groundingScore": 0.92,
          "groundedBy": ["reach://api-gateway:lodash.get"],
          "verified": true
        }
      ]
    }
  ],
  "overallGroundingScore": 0.89,
  "artifacts": [
    {
      "artifactId": "art-001",
      "type": "EvidencePack",
      "contentDigest": "sha256:..."
    }
  ]
}

Attestation Verification

POST /api/v1/advisory-ai/runs/{runId}/attestation/verify
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>

Response:

{
  "valid": true,
  "runId": "run-abc123",
  "attestationDigest": "sha256:...",
  "signatureValid": true,
  "contentValid": true,
  "verifiedAt": "2026-01-10T15:00:00Z"
}

UI Guide

Run Timeline View

The Run Timeline component provides a visual representation of all events:

+------------------------------------------------------------------+
| Run: run-abc123                                     [Active]      |
| Started: Jan 10, 2026 14:30                                       |
+------------------------------------------------------------------+
|                                                                   |
|  o  Run Created                                        14:30:00   |
|  |  Run initialized from conversation conv-abc123                 |
|  |                                                                |
|  o  User Turn                                          14:30:05   |
|  |  user@example.com                                              |
|  |  "Is CVE-2024-1234 exploitable in our environment?"            |
|  |                                                                |
|  o  Assistant Turn                                     14:30:08   |
|  |  assistant                                                     |
|  |  "Based on the reachability analysis [reach:api-gateway:...]"  |
|  |  Grounding: 92%                                                |
|  |                                                                |
|  o  Action Proposed                                    14:30:09   |
|  |  [Accept Risk] [Create VEX] [Escalate]                         |
|  |                                                                |
|  o  Artifact Created                                   14:35:00   |
|     Evidence Pack: CVE-2024-1234                                  |
|                                                                   |
+------------------------------------------------------------------+
| Artifacts (1)                                                     |
| +----------------------------+                                    |
| | Evidence Pack              |                                    |
| | CVE-2024-1234 Evidence Pack|                                    |
| | sha256:def456...           |                                    |
| +----------------------------+                                    |
+------------------------------------------------------------------+

Key UI Components

Component Purpose
RunTimelineComponent Displays event timeline
RunStatusBadge Shows current Run status with color coding
EventIcon Icon for each event type
ArtifactCard Displays artifact with download link
AttestationBadge Shows attestation status and verification
RunListComponent Paginated list of Runs

Status Colors

Status Color Meaning
Created Gray Initialized, no activity
Active Blue In progress
PendingApproval Yellow Waiting for user action
Completed Green Successfully finished
Cancelled Orange User cancelled
Failed Red Error occurred

Configuration

AdvisoryAI:
  Runs:
    Enabled: true
    AutoCreate: true          # Auto-create Run from first conversation turn
    RetentionDays: 90         # How long to keep completed Runs
    AttestOnComplete: true    # Generate attestation on completion
    ReplayEnabled: true       # Allow replay verification

  Timeline:
    MaxEventsPerRun: 1000     # Maximum timeline events per Run
    ContentDigestAlgorithm: sha256

  Artifacts:
    MaxPerRun: 50             # Maximum artifacts per Run
    MaxSizeBytes: 10485760    # 10 MB max artifact size

Error Handling

Status Code Error Description
400 InvalidRequest Malformed request body
401 Unauthorized Missing or invalid token
403 Forbidden Insufficient permissions for tenant/run
404 RunNotFound Run does not exist
409 InvalidStateTransition Cannot transition Run to requested state
429 RateLimited Too many requests
500 InternalError Server error

See Also

Last updated: 10-Jan-2026