stella-ops.org/git.stella-ops.org
1
0
Fork 0
Code Issues Pull Requests Actions 2 Releases Wiki Activity
Files
88a85cdd92394204a7a8877e62f9f32df07848d9
git.stella-ops.org/docs/modules/advisory-ai/runs.md
master 17d0631b8e sprints work
2026-01-10 20:38:13 +02:00

679 lines
18 KiB
Markdown
Raw Blame History

# AI Runs Framework
> **Sprint:** SPRINT_20260109_011_003_BE_ai_runs_framework
> **Status:** Active
> **Last Updated:** 2026-01-10
The AI Runs Framework provides an auditable container for AI-assisted investigations, capturing the complete lifecycle from initial query through tool calls, artifact generation, and approvals.
## Overview
> "Chat is not auditable, repeatable, actionable with guardrails, or collaborative."
The Run concept transforms ephemeral chat into:
- **Auditable:** Every interaction logged with timestamps and content digests
- **Repeatable:** Deterministic replay possible for verification
- **Actionable:** Artifacts produced (evidence packs, VEX statements, decisions)
- **Collaborative:** Handoffs, approvals, shared context across team members
---
## Run Lifecycle
Runs progress through defined states with explicit transitions:
```
Created -> Active -> PendingApproval -> Completed
\-> Cancelled
\-> Failed
```
### States
| State | Description | Allowed Transitions |
|-------|-------------|---------------------|
| `Created` | Run initialized, no activity yet | Active, Cancelled |
| `Active` | Conversation in progress | PendingApproval, Completed, Cancelled, Failed |
| `PendingApproval` | Action requires user approval | Active, Completed, Cancelled |
| `Completed` | Run finished, attestation generated | (terminal) |
| `Cancelled` | Run cancelled by user | (terminal) |
| `Failed` | Run failed due to error | (terminal) |
### Lifecycle Diagram
```
+----------+ +---------+ +------------+ +----------+
| Created | -> | Active | -> | Pending | -> | Complete |
| | | | | Approval | | |
+----------+ +---------+ +------------+ +----------+
| | | |
v v v v
+--------------------------------------------------+
| Run Timeline |
| +-------+ +------+ +---------+ +--------+ +---+ |
| |Created| | User | |Assistant| | Action | |...| |
| | Event | | Turn | | Turn | |Proposed| | | |
| +-------+ +------+ +---------+ +--------+ +---+ |
+--------------------------------------------------+
|
v
+--------------------------------------------------+
| Artifacts Produced |
| +---------+ +--------+ +------+ +-----+ |
| | Evidence| |Decision| |Action| | VEX | |
| | Pack | | Record | |Result| |Stmt | |
| +---------+ +--------+ +------+ +-----+ |
+--------------------------------------------------+
|
v
+--------------------------------------------------+
| Run Attestation (DSSE) |
| - Content digest of all turns |
| - Evidence references |
| - Artifact digests |
| - Signed by platform key |
+--------------------------------------------------+
```
---
## API Reference
### Create Run
Creates a new Run from an existing conversation.
```http
POST /api/v1/advisory-ai/runs
Content-Type: application/json
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>
{
"conversationId": "conv-abc123",
"context": {
"findingId": "f-456",
"cveId": "CVE-2024-1234",
"component": "pkg:npm/lodash@4.17.21",
"scanId": "scan-789",
"sbomId": "sbom-xyz"
}
}
```
**Response (201 Created):**
```json
{
"runId": "run-abc123",
"tenantId": "tenant-xyz",
"userId": "user@example.com",
"conversationId": "conv-abc123",
"status": "Created",
"createdAt": "2026-01-10T14:30:00Z",
"context": {
"findingId": "f-456",
"cveId": "CVE-2024-1234",
"component": "pkg:npm/lodash@4.17.21",
"scanId": "scan-789",
"sbomId": "sbom-xyz"
},
"timeline": [],
"artifacts": []
}
```
### Get Run
Retrieves a Run with its complete timeline and artifacts.
```http
GET /api/v1/advisory-ai/runs/{runId}
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>
```
**Response (200 OK):**
```json
{
"runId": "run-abc123",
"tenantId": "tenant-xyz",
"userId": "user@example.com",
"conversationId": "conv-abc123",
"status": "Active",
"createdAt": "2026-01-10T14:30:00Z",
"completedAt": null,
"context": { ... },
"timeline": [
{
"eventId": "evt-001",
"eventType": "RunCreated",
"timestamp": "2026-01-10T14:30:00Z",
"actor": "system",
"summary": "Run created from conversation conv-abc123"
},
{
"eventId": "evt-002",
"eventType": "UserTurn",
"timestamp": "2026-01-10T14:30:05Z",
"actor": "user:user@example.com",
"summary": "Is CVE-2024-1234 exploitable in our environment?",
"relatedTurnId": "turn-001"
},
{
"eventId": "evt-003",
"eventType": "AssistantTurn",
"timestamp": "2026-01-10T14:30:08Z",
"actor": "assistant",
"summary": "Based on the reachability analysis...",
"details": {
"contentDigest": "sha256:abc123...",
"groundingScore": 0.92
},
"relatedTurnId": "turn-002"
}
],
"artifacts": [],
"attestationDigest": null
}
```
### Get Timeline
Returns paginated timeline events for a Run.
```http
GET /api/v1/advisory-ai/runs/{runId}/timeline?limit=50&cursor=evt-050
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>
```
**Response (200 OK):**
```json
{
"runId": "run-abc123",
"events": [ ... ],
"cursor": "evt-100",
"hasMore": true
}
```
### Get Artifacts
Lists all artifacts attached to a Run.
```http
GET /api/v1/advisory-ai/runs/{runId}/artifacts
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>
```
**Response (200 OK):**
```json
{
"runId": "run-abc123",
"artifacts": [
{
"artifactId": "art-001",
"type": "EvidencePack",
"name": "CVE-2024-1234 Evidence Pack",
"contentDigest": "sha256:def456...",
"uri": "evidence://packs/art-001",
"createdAt": "2026-01-10T14:35:00Z",
"metadata": {
"cveId": "CVE-2024-1234",
"component": "pkg:npm/lodash@4.17.21"
}
}
]
}
```
### Complete Run
Marks a Run as complete and generates the attestation.
```http
POST /api/v1/advisory-ai/runs/{runId}/complete
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>
```
**Response (200 OK):**
```json
{
"runId": "run-abc123",
"status": "Completed",
"completedAt": "2026-01-10T14:40:00Z",
"attestationDigest": "sha256:xyz789...",
"attestation": {
"runId": "run-abc123",
"tenantId": "tenant-xyz",
"userId": "user@example.com",
"modelInfo": {
"modelId": "gpt-4-turbo",
"modelVersion": "2024-04-09"
},
"turns": [ ... ],
"overallGroundingScore": 0.89
}
}
```
### Cancel Run
Cancels an active Run.
```http
POST /api/v1/advisory-ai/runs/{runId}/cancel
Content-Type: application/json
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>
{
"reason": "Investigation no longer needed - issue resolved"
}
```
**Response (200 OK):**
```json
{
"runId": "run-abc123",
"status": "Cancelled",
"cancelledAt": "2026-01-10T14:35:00Z",
"reason": "Investigation no longer needed - issue resolved"
}
```
### Replay Run
Replays a Run for verification and determinism checking.
```http
POST /api/v1/advisory-ai/runs/{runId}/replay
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>
```
**Response (200 OK):**
```json
{
"runId": "run-abc123",
"deterministic": true,
"originalDigest": "sha256:abc123...",
"replayDigest": "sha256:abc123...",
"differences": []
}
```
**Response (Non-deterministic):**
```json
{
"runId": "run-abc123",
"deterministic": false,
"originalDigest": "sha256:abc123...",
"replayDigest": "sha256:def456...",
"differences": [
"Turn 2: original=sha256:111..., replay=sha256:222..."
]
}
```
### List Runs
Lists Runs with filtering and pagination.
```http
GET /api/v1/advisory-ai/runs?userId=user@example.com&status=Active&limit=20
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>
```
**Query Parameters:**
| Parameter | Type | Description |
|-----------|------|-------------|
| `userId` | string | Filter by user |
| `findingId` | string | Filter by finding |
| `status` | string | Filter by status (Created, Active, PendingApproval, Completed, Cancelled, Failed) |
| `since` | datetime | Runs created after this time |
| `until` | datetime | Runs created before this time |
| `limit` | integer | Page size (default: 50, max: 100) |
| `cursor` | string | Pagination cursor |
**Response (200 OK):**
```json
{
"runs": [ ... ],
"cursor": "run-xyz",
"hasMore": true
}
```
---
## Timeline Event Types
The timeline captures all significant events during a Run.
| Event Type | Actor | Description |
|------------|-------|-------------|
| `RunCreated` | system | Run initialized |
| `UserTurn` | user:{userId} | User message added |
| `AssistantTurn` | assistant | AI response generated |
| `ToolCall` | assistant | AI invoked a tool (search, lookup, etc.) |
| `ActionProposed` | assistant | AI proposed an action |
| `ApprovalRequested` | system | Action requires user approval |
| `ApprovalGranted` | user:{userId} | User approved action |
| `ApprovalDenied` | user:{userId} | User denied action |
| `ActionExecuted` | system | Action was executed |
| `ActionFailed` | system | Action execution failed |
| `ArtifactCreated` | system | Artifact attached to Run |
| `RunCompleted` | system | Run completed with attestation |
| `RunCancelled` | user:{userId} | Run cancelled |
| `RunFailed` | system | Run failed due to error |
### Event Details
Each event type may include additional details:
**AssistantTurn:**
```json
{
"contentDigest": "sha256:...",
"groundingScore": 0.92,
"tokenCount": 847
}
```
**ActionProposed:**
```json
{
"actionType": "approve",
"label": "Accept Risk",
"parameters": {
"cveId": "CVE-2024-1234",
"rationale": "Not exploitable in our environment"
}
}
```
**ArtifactCreated:**
```json
{
"artifactId": "art-001",
"artifactType": "EvidencePack",
"contentDigest": "sha256:..."
}
```
---
## Artifact Types
Runs can produce various artifact types:
| Type | Description | Use Case |
|------|-------------|----------|
| `EvidencePack` | Bundle of evidence supporting a decision | Risk acceptance, VEX justification |
| `DecisionRecord` | Formal record of a security decision | Audit trail, compliance |
| `VexStatement` | VEX statement draft or final | Vulnerability disclosure |
| `ActionResult` | Result of an executed action | Remediation tracking |
| `Explanation` | AI-generated explanation | User understanding |
| `Report` | Generated report document | Stakeholder communication |
### Artifact Structure
```json
{
"artifactId": "art-001",
"type": "EvidencePack",
"name": "CVE-2024-1234 Evidence Pack",
"contentDigest": "sha256:def456...",
"uri": "evidence://packs/art-001",
"createdAt": "2026-01-10T14:35:00Z",
"metadata": {
"cveId": "CVE-2024-1234",
"component": "pkg:npm/lodash@4.17.21",
"scanId": "scan-789",
"format": "evidence-pack/v1"
}
}
```
---
## Replay Verification
Runs can be replayed to verify determinism. This is crucial for:
- **Compliance audits:** Proving AI outputs are reproducible
- **Debugging:** Understanding how AI reached conclusions
- **Trust:** Demonstrating consistent behavior
### Replay Requirements
For successful replay:
1. **Temperature = 0:** No randomness in token selection
2. **Fixed seed:** Same seed across replays
3. **Model match:** Same model weights (verified by digest)
4. **Prompt match:** Identical prompts (verified by hash)
5. **Context match:** Same input context
### Replay Results
| Result | Meaning |
|--------|---------|
| `deterministic: true` | Replay produced identical output |
| `deterministic: false` | Output differs (see differences array) |
### Common Divergence Causes
| Cause | Detection | Resolution |
|-------|-----------|------------|
| Different model | Model version mismatch | Use pinned model version |
| Non-zero temperature | Parameter check | Set temperature to 0 |
| Different seed | Seed mismatch | Use consistent seed |
| Prompt template change | Template version mismatch | Pin template version |
| Context ordering | Context hash mismatch | Sort context deterministically |
---
## Run Attestation
Completed Runs produce DSSE-signed attestations containing:
```json
{
"_type": "https://stellaops.org/attestation/ai-run/v1",
"runId": "run-abc123",
"tenantId": "tenant-xyz",
"userId": "user@example.com",
"conversationId": "conv-abc123",
"startedAt": "2026-01-10T14:30:00Z",
"completedAt": "2026-01-10T14:40:00Z",
"model": {
"modelId": "gpt-4-turbo",
"modelVersion": "2024-04-09",
"provider": "azure-openai"
},
"promptTemplate": {
"templateId": "security-investigate",
"version": "1.2.0",
"digest": "sha256:..."
},
"context": {
"findingId": "f-456",
"cveId": "CVE-2024-1234",
"policyId": "policy-001",
"evidenceUris": [
"sbom://scan-789/lodash",
"reach://api-gateway:lodash.get"
]
},
"turns": [
{
"turnId": "turn-001",
"role": "User",
"contentDigest": "sha256:...",
"timestamp": "2026-01-10T14:30:05Z"
},
{
"turnId": "turn-002",
"role": "Assistant",
"contentDigest": "sha256:...",
"timestamp": "2026-01-10T14:30:08Z",
"claims": [
{
"text": "CVE-2024-1234 is reachable through...",
"groundingScore": 0.92,
"groundedBy": ["reach://api-gateway:lodash.get"],
"verified": true
}
]
}
],
"overallGroundingScore": 0.89,
"artifacts": [
{
"artifactId": "art-001",
"type": "EvidencePack",
"contentDigest": "sha256:..."
}
]
}
```
### Attestation Verification
```http
POST /api/v1/advisory-ai/runs/{runId}/attestation/verify
Authorization: Bearer <token>
X-StellaOps-Tenant: <tenant-id>
```
**Response:**
```json
{
"valid": true,
"runId": "run-abc123",
"attestationDigest": "sha256:...",
"signatureValid": true,
"contentValid": true,
"verifiedAt": "2026-01-10T15:00:00Z"
}
```
---
## UI Guide
### Run Timeline View
The Run Timeline component provides a visual representation of all events:
```
+------------------------------------------------------------------+
| Run: run-abc123 [Active] |
| Started: Jan 10, 2026 14:30 |
+------------------------------------------------------------------+
| |
| o Run Created 14:30:00 |
| | Run initialized from conversation conv-abc123 |
| | |
| o User Turn 14:30:05 |
| | user@example.com |
| | "Is CVE-2024-1234 exploitable in our environment?" |
| | |
| o Assistant Turn 14:30:08 |
| | assistant |
| | "Based on the reachability analysis [reach:api-gateway:...]" |
| | Grounding: 92% |
| | |
| o Action Proposed 14:30:09 |
| | [Accept Risk] [Create VEX] [Escalate] |
| | |
| o Artifact Created 14:35:00 |
| Evidence Pack: CVE-2024-1234 |
| |
+------------------------------------------------------------------+
| Artifacts (1) |
| +----------------------------+ |
| | Evidence Pack | |
| | CVE-2024-1234 Evidence Pack| |
| | sha256:def456... | |
| +----------------------------+ |
+------------------------------------------------------------------+
```
### Key UI Components
| Component | Purpose |
|-----------|---------|
| `RunTimelineComponent` | Displays event timeline |
| `RunStatusBadge` | Shows current Run status with color coding |
| `EventIcon` | Icon for each event type |
| `ArtifactCard` | Displays artifact with download link |
| `AttestationBadge` | Shows attestation status and verification |
| `RunListComponent` | Paginated list of Runs |
### Status Colors
| Status | Color | Meaning |
|--------|-------|---------|
| Created | Gray | Initialized, no activity |
| Active | Blue | In progress |
| PendingApproval | Yellow | Waiting for user action |
| Completed | Green | Successfully finished |
| Cancelled | Orange | User cancelled |
| Failed | Red | Error occurred |
---
## Configuration
```yaml
AdvisoryAI:
Runs:
Enabled: true
AutoCreate: true # Auto-create Run from first conversation turn
RetentionDays: 90 # How long to keep completed Runs
AttestOnComplete: true # Generate attestation on completion
ReplayEnabled: true # Allow replay verification
Timeline:
MaxEventsPerRun: 1000 # Maximum timeline events per Run
ContentDigestAlgorithm: sha256
Artifacts:
MaxPerRun: 50 # Maximum artifacts per Run
MaxSizeBytes: 10485760 # 10 MB max artifact size
```
---
## Error Handling
| Status Code | Error | Description |
|-------------|-------|-------------|
| 400 | InvalidRequest | Malformed request body |
| 401 | Unauthorized | Missing or invalid token |
| 403 | Forbidden | Insufficient permissions for tenant/run |
| 404 | RunNotFound | Run does not exist |
| 409 | InvalidStateTransition | Cannot transition Run to requested state |
| 429 | RateLimited | Too many requests |
| 500 | InternalError | Server error |
---
## See Also
- [AdvisoryAI Architecture](architecture.md)
- [Chat Interface](chat-interface.md)
- [AI Attestations](guides/ai-attestations.md)
- [Evidence Locker](/docs/modules/evidence-locker/architecture.md)
- [Attestor Module](/docs/modules/attestor/architecture.md)
---
_Last updated: 10-Jan-2026_
Reference in New Issue View Git Blame Copy Permalink