test fixes and new product advisories work
This commit is contained in:
master
128
docs/reachability/README.md
Normal file
128
docs/reachability/README.md
Normal file
@@ -0,0 +1,128 @@
|
||||
# eBPF Reachability Evidence System
|
||||
|
||||
This documentation covers the eBPF-based runtime reachability evidence collection system in StellaOps.
|
||||
|
||||
## Overview
|
||||
|
||||
The eBPF reachability system provides kernel-level syscall tracing to prove which code paths, files, and network connections were (or weren't) executed in production. This evidence complements static analysis by providing runtime proof of actual behavior.
|
||||
|
||||
## Key Capabilities
|
||||
|
||||
- **Syscall Tracing**: Capture file access (`openat`), process execution (`exec`), and network connections (`inet_sock_set_state`)
|
||||
- **User-Space Probes**: Monitor libc network functions and OpenSSL TLS operations
|
||||
- **Container Awareness**: Automatic correlation of events to container IDs and image digests
|
||||
- **Signed Evidence Chains**: DSSE-signed chunks with Rekor transparency log integration
|
||||
- **Deterministic Output**: Canonical NDJSON format for reproducible evidence
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Linux kernel 5.x+ with BTF support (4.14+ with external BTF)
|
||||
- Container runtime (containerd, Docker, or CRI-O)
|
||||
- StellaOps CLI installed
|
||||
|
||||
### Enable Runtime Evidence Collection
|
||||
|
||||
```bash
|
||||
# Start the runtime signal collector
|
||||
stella signals start --target /var/lib/stellaops/evidence
|
||||
|
||||
# Verify collection is active
|
||||
stella signals status
|
||||
|
||||
# View recent signals
|
||||
stella signals inspect sha256:abc123...
|
||||
|
||||
# Verify evidence chain integrity
|
||||
stella signals verify-chain /var/lib/stellaops/evidence
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
```yaml
|
||||
# stellaops.yaml
|
||||
signals:
|
||||
enabled: true
|
||||
output_directory: /var/lib/stellaops/evidence
|
||||
rotation:
|
||||
max_size_mb: 100
|
||||
max_age_hours: 1
|
||||
signing:
|
||||
enabled: true
|
||||
key_id: fulcio # or KMS key reference
|
||||
submit_to_rekor: true
|
||||
filters:
|
||||
target_containers: [] # Empty = all containers
|
||||
path_allowlist:
|
||||
- /etc/**
|
||||
- /var/lib/**
|
||||
path_denylist:
|
||||
- /proc/**
|
||||
- /sys/**
|
||||
```
|
||||
|
||||
## Documentation Index
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [ebpf-architecture.md](ebpf-architecture.md) | System design and data flow |
|
||||
| [evidence-schema.md](evidence-schema.md) | NDJSON schema reference |
|
||||
| [probe-reference.md](probe-reference.md) | Tracepoint and uprobe details |
|
||||
| [deployment-guide.md](deployment-guide.md) | Kernel requirements and installation |
|
||||
| [operator-runbook.md](operator-runbook.md) | Operations and troubleshooting |
|
||||
| [security-model.md](security-model.md) | Threat model and mitigations |
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ User Space │
|
||||
│ ┌─────────────┐ ┌──────────────┐ ┌─────────────────────────┐ │
|
||||
│ │ Zastava │ │ Scanner │ │ RuntimeSignalCollector │ │
|
||||
│ │ Container │ │ Reachability │ │ │ │
|
||||
│ │ Tracker │ │ Merger │ │ ┌─────────────────┐ │ │
|
||||
│ └──────┬──────┘ └──────┬───────┘ │ │ EventParser │ │ │
|
||||
│ │ │ │ └────────┬────────┘ │ │
|
||||
│ │ │ │ │ │ │
|
||||
│ └────────┬───────┘ │ ┌────────▼────────┐ │ │
|
||||
│ │ │ │ CgroupResolver │ │ │
|
||||
│ ┌────────▼────────┐ │ └────────┬────────┘ │ │
|
||||
│ │ RuntimeEvent │ │ │ │ │
|
||||
│ │ Enricher │◄────────┤ ┌────────▼────────┐ │ │
|
||||
│ └────────┬────────┘ │ │SymbolResolver │ │ │
|
||||
│ │ │ └────────┬────────┘ │ │
|
||||
│ ┌────────▼────────┐ │ │ │ │
|
||||
│ │ NDJSON Writer │◄────────┼───────────┘ │ │
|
||||
│ └────────┬────────┘ │ │ │
|
||||
│ │ └─────────────────────────┘ │
|
||||
│ ┌────────▼────────┐ │
|
||||
│ │ ChunkFinalizer │──────► Signer ──────► Rekor │
|
||||
│ └─────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
──────────┼──────────
|
||||
│
|
||||
┌─────────────────────────────┼───────────────────────────────────┐
|
||||
│ Kernel │Space │
|
||||
│ │ │
|
||||
│ ┌──────────────────────────▼───────────────────────────────┐ │
|
||||
│ │ Ring Buffer │ │
|
||||
│ └──────────────────────────▲───────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────────┐ ┌────────┴───────┐ ┌──────────────────┐ │
|
||||
│ │ Tracepoints │ │ Uprobes │ │ BPF Maps │ │
|
||||
│ │ │ │ │ │ │ │
|
||||
│ │ sys_openat │ │ libc:connect │ │ cgroup_filter │ │
|
||||
│ │ sched_exec │ │ libc:accept │ │ symbol_cache │ │
|
||||
│ │ inet_sock │ │ SSL_read/write │ │ pid_namespace │ │
|
||||
│ └──────────────┘ └────────────────┘ └──────────────────┘ │
|
||||
│ │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Signals Module Architecture](../modules/signals/architecture.md)
|
||||
- [Evidence Schema Conventions](../11_DATA_SCHEMAS.md)
|
||||
- [Zastava Container Tracking](../modules/zastava/architecture.md)
|
||||
397
docs/reachability/deployment-guide.md
Normal file
397
docs/reachability/deployment-guide.md
Normal file
@@ -0,0 +1,397 @@
|
||||
# Deployment Guide
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Kernel Requirements
|
||||
|
||||
**Minimum:** Linux 4.14 with eBPF support
|
||||
**Recommended:** Linux 5.8+ with BTF and ring buffer support
|
||||
|
||||
#### Verify Kernel Configuration
|
||||
|
||||
```bash
|
||||
# Check eBPF support
|
||||
zcat /proc/config.gz 2>/dev/null | grep -E "CONFIG_BPF|CONFIG_DEBUG_INFO_BTF" || \
|
||||
cat /boot/config-$(uname -r) | grep -E "CONFIG_BPF|CONFIG_DEBUG_INFO_BTF"
|
||||
|
||||
# Required settings:
|
||||
# CONFIG_BPF=y
|
||||
# CONFIG_BPF_SYSCALL=y
|
||||
# CONFIG_BPF_JIT=y (recommended)
|
||||
# CONFIG_DEBUG_INFO_BTF=y (for CO-RE)
|
||||
```
|
||||
|
||||
#### Verify BTF Availability
|
||||
|
||||
```bash
|
||||
# Check for BTF in kernel
|
||||
ls -la /sys/kernel/btf/vmlinux
|
||||
|
||||
# If missing, check BTFHub or kernel debug packages
|
||||
```
|
||||
|
||||
### Container Runtime
|
||||
|
||||
Supported runtimes:
|
||||
- containerd 1.4+
|
||||
- Docker 20.10+
|
||||
- CRI-O 1.20+
|
||||
|
||||
Verify cgroup v2 is available (recommended):
|
||||
```bash
|
||||
mount | grep cgroup2
|
||||
# Expected: cgroup2 on /sys/fs/cgroup type cgroup2
|
||||
```
|
||||
|
||||
### Permissions
|
||||
|
||||
The collector requires elevated privileges:
|
||||
|
||||
**Option 1: Root**
|
||||
```bash
|
||||
sudo stella signals start
|
||||
```
|
||||
|
||||
**Option 2: Capabilities (preferred)**
|
||||
```bash
|
||||
# Grant required capabilities
|
||||
sudo setcap cap_bpf,cap_perfmon,cap_sys_ptrace+ep /usr/bin/stella
|
||||
|
||||
# Or run with specific capabilities
|
||||
sudo capsh --caps="cap_bpf,cap_perfmon,cap_sys_ptrace+eip" -- -c "stella signals start"
|
||||
```
|
||||
|
||||
Required capabilities:
|
||||
- `CAP_BPF`: Load and manage eBPF programs
|
||||
- `CAP_PERFMON`: Access performance monitoring (ring buffer)
|
||||
- `CAP_SYS_PTRACE`: Attach uprobes to processes
|
||||
|
||||
## Installation
|
||||
|
||||
### Standard Installation
|
||||
|
||||
```bash
|
||||
# Install StellaOps CLI
|
||||
curl -fsSL https://stella.ops/install.sh | bash
|
||||
|
||||
# Verify installation
|
||||
stella version
|
||||
stella signals --help
|
||||
```
|
||||
|
||||
### Air-Gap Installation
|
||||
|
||||
For disconnected environments, use the offline bundle:
|
||||
|
||||
```bash
|
||||
# Download bundle (on connected machine)
|
||||
stella bundle create --include-probes ebpf-reachability \
|
||||
--output stellaops-offline.tar.gz
|
||||
|
||||
# Transfer to air-gapped system
|
||||
scp stellaops-offline.tar.gz airgap-host:
|
||||
|
||||
# Install on air-gapped system
|
||||
tar -xzf stellaops-offline.tar.gz
|
||||
cd stellaops-offline
|
||||
./install.sh
|
||||
```
|
||||
|
||||
The bundle includes:
|
||||
- Pre-compiled eBPF probes for common kernel versions
|
||||
- BTF files for kernels without built-in BTF
|
||||
- All runtime dependencies
|
||||
|
||||
### Pre-Compiled Probes
|
||||
|
||||
If CO-RE probes fail to load, use kernel-specific probes:
|
||||
|
||||
```bash
|
||||
# List available pre-compiled probes
|
||||
stella signals probes list
|
||||
|
||||
# Install probes for specific kernel
|
||||
stella signals probes install --kernel $(uname -r)
|
||||
|
||||
# Verify probe compatibility
|
||||
stella signals probes verify
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Basic Configuration
|
||||
|
||||
Create `/etc/stellaops/signals.yaml`:
|
||||
|
||||
```yaml
|
||||
signals:
|
||||
enabled: true
|
||||
|
||||
# Output directory for evidence files
|
||||
output_directory: /var/lib/stellaops/evidence
|
||||
|
||||
# Ring buffer size (default 256KB)
|
||||
ring_buffer_size: 262144
|
||||
|
||||
# Maximum events per second (0 = unlimited)
|
||||
max_events_per_second: 0
|
||||
|
||||
# Rotation settings
|
||||
rotation:
|
||||
max_size_mb: 100
|
||||
max_age_hours: 1
|
||||
|
||||
# Signing configuration
|
||||
signing:
|
||||
enabled: true
|
||||
key_id: fulcio # or KMS key ARN
|
||||
submit_to_rekor: true
|
||||
```
|
||||
|
||||
### Probe Selection
|
||||
|
||||
Enable specific probes:
|
||||
|
||||
```yaml
|
||||
signals:
|
||||
probes:
|
||||
# Tracepoints
|
||||
sys_enter_openat: true
|
||||
sched_process_exec: true
|
||||
inet_sock_set_state: true
|
||||
|
||||
# Uprobes
|
||||
libc_connect: true
|
||||
libc_accept: true
|
||||
openssl_read: false # Disable if not needed
|
||||
openssl_write: false
|
||||
```
|
||||
|
||||
### Filtering
|
||||
|
||||
Configure what to capture:
|
||||
|
||||
```yaml
|
||||
signals:
|
||||
filters:
|
||||
# Target specific containers (empty = all)
|
||||
target_containers: []
|
||||
|
||||
# Target specific namespaces
|
||||
target_namespaces: []
|
||||
|
||||
# File path filtering
|
||||
paths:
|
||||
allowlist:
|
||||
- /etc/**
|
||||
- /var/lib/**
|
||||
- /home/**
|
||||
denylist:
|
||||
- /proc/**
|
||||
- /sys/**
|
||||
- /dev/**
|
||||
|
||||
# Network filtering
|
||||
networks:
|
||||
# Capture connections to these CIDRs
|
||||
allowlist:
|
||||
- 10.0.0.0/8
|
||||
- 172.16.0.0/12
|
||||
# Exclude these destinations
|
||||
denylist:
|
||||
- 127.0.0.0/8
|
||||
```
|
||||
|
||||
### Resource Limits
|
||||
|
||||
Prevent runaway resource usage:
|
||||
|
||||
```yaml
|
||||
signals:
|
||||
resources:
|
||||
# Maximum memory for caches
|
||||
max_cache_memory_mb: 256
|
||||
|
||||
# Symbol cache entries
|
||||
symbol_cache_max_entries: 100000
|
||||
|
||||
# Container cache TTL
|
||||
container_cache_ttl_seconds: 300
|
||||
|
||||
# Event rate limiting
|
||||
max_events_per_second: 50000
|
||||
```
|
||||
|
||||
## Starting the Collector
|
||||
|
||||
### Systemd Service
|
||||
|
||||
```bash
|
||||
# Enable and start
|
||||
sudo systemctl enable stellaops-signals
|
||||
sudo systemctl start stellaops-signals
|
||||
|
||||
# Check status
|
||||
sudo systemctl status stellaops-signals
|
||||
|
||||
# View logs
|
||||
sudo journalctl -u stellaops-signals -f
|
||||
```
|
||||
|
||||
### Manual Start
|
||||
|
||||
```bash
|
||||
# Start with default configuration
|
||||
stella signals start
|
||||
|
||||
# Start with custom config
|
||||
stella signals start --config /path/to/signals.yaml
|
||||
|
||||
# Start with verbose logging
|
||||
stella signals start --verbose
|
||||
|
||||
# Start in foreground (for debugging)
|
||||
stella signals start --foreground
|
||||
```
|
||||
|
||||
### Docker Deployment
|
||||
|
||||
```dockerfile
|
||||
FROM stellaops/signals-collector:latest
|
||||
|
||||
# Mount host systems
|
||||
VOLUME /sys/kernel/debug
|
||||
VOLUME /sys/fs/cgroup
|
||||
VOLUME /proc
|
||||
|
||||
# Evidence output
|
||||
VOLUME /var/lib/stellaops/evidence
|
||||
|
||||
# Run with required capabilities
|
||||
# docker run --privileged or with specific caps
|
||||
```
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name stellaops-signals \
|
||||
--privileged \
|
||||
-v /sys/kernel/debug:/sys/kernel/debug:ro \
|
||||
-v /sys/fs/cgroup:/sys/fs/cgroup:ro \
|
||||
-v /proc:/host/proc:ro \
|
||||
-v /var/lib/stellaops/evidence:/evidence \
|
||||
stellaops/signals-collector:latest
|
||||
```
|
||||
|
||||
### Kubernetes DaemonSet
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: stellaops-signals
|
||||
namespace: stellaops
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: stellaops-signals
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: stellaops-signals
|
||||
spec:
|
||||
hostPID: true
|
||||
hostNetwork: true
|
||||
containers:
|
||||
- name: collector
|
||||
image: stellaops/signals-collector:latest
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- name: sys-kernel-debug
|
||||
mountPath: /sys/kernel/debug
|
||||
readOnly: true
|
||||
- name: sys-fs-cgroup
|
||||
mountPath: /sys/fs/cgroup
|
||||
readOnly: true
|
||||
- name: proc
|
||||
mountPath: /host/proc
|
||||
readOnly: true
|
||||
- name: evidence
|
||||
mountPath: /var/lib/stellaops/evidence
|
||||
volumes:
|
||||
- name: sys-kernel-debug
|
||||
hostPath:
|
||||
path: /sys/kernel/debug
|
||||
- name: sys-fs-cgroup
|
||||
hostPath:
|
||||
path: /sys/fs/cgroup
|
||||
- name: proc
|
||||
hostPath:
|
||||
path: /proc
|
||||
- name: evidence
|
||||
hostPath:
|
||||
path: /var/lib/stellaops/evidence
|
||||
type: DirectoryOrCreate
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
### Verify Probes Attached
|
||||
|
||||
```bash
|
||||
# List attached probes
|
||||
stella signals status
|
||||
|
||||
# Expected output:
|
||||
# Probes:
|
||||
# tracepoint/syscalls/sys_enter_openat: attached
|
||||
# tracepoint/sched/sched_process_exec: attached
|
||||
# tracepoint/sock/inet_sock_set_state: attached
|
||||
# uprobe/libc.so.6:connect: attached
|
||||
# uprobe/libc.so.6:accept: attached
|
||||
```
|
||||
|
||||
### Verify Events Flowing
|
||||
|
||||
```bash
|
||||
# Watch live events
|
||||
stella signals watch
|
||||
|
||||
# Check event counts
|
||||
stella signals stats
|
||||
|
||||
# Expected output:
|
||||
# Events collected: 15234
|
||||
# Events/second: 847
|
||||
# Ring buffer usage: 12%
|
||||
```
|
||||
|
||||
### Verify Evidence Files
|
||||
|
||||
```bash
|
||||
# List evidence chunks
|
||||
ls -la /var/lib/stellaops/evidence/
|
||||
|
||||
# Verify chain integrity
|
||||
stella signals verify-chain /var/lib/stellaops/evidence/
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
See [operator-runbook.md](operator-runbook.md) for detailed troubleshooting procedures.
|
||||
|
||||
### Quick Checks
|
||||
|
||||
```bash
|
||||
# Check kernel support
|
||||
stella signals check-kernel
|
||||
|
||||
# Verify permissions
|
||||
stella signals check-permissions
|
||||
|
||||
# Test probe loading
|
||||
stella signals test-probes
|
||||
|
||||
# Validate configuration
|
||||
stella signals validate-config --config /etc/stellaops/signals.yaml
|
||||
```
|
||||
232
docs/reachability/ebpf-architecture.md
Normal file
232
docs/reachability/ebpf-architecture.md
Normal file
@@ -0,0 +1,232 @@
|
||||
# eBPF Reachability Architecture
|
||||
|
||||
## System Overview
|
||||
|
||||
The eBPF reachability system captures kernel-level events to provide cryptographic proof of runtime behavior. It uses Linux eBPF (extended Berkeley Packet Filter) with CO-RE (Compile Once, Run Everywhere) for portable deployment across kernel versions.
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Minimal Kernel Footprint**: eBPF programs perform only essential filtering and data capture
|
||||
2. **User-Space Enrichment**: Complex lookups (symbols, containers, SBOMs) happen in user space
|
||||
3. **Deterministic Output**: Same inputs produce byte-identical NDJSON output
|
||||
4. **Chain of Custody**: Every evidence chunk is cryptographically signed and linked
|
||||
|
||||
## Component Architecture
|
||||
|
||||
### Kernel-Space Components
|
||||
|
||||
#### Ring Buffer (`BPF_MAP_TYPE_RINGBUF`)
|
||||
- Single shared buffer for all event types (default 256KB)
|
||||
- Lock-free, multi-producer design
|
||||
- Automatic backpressure via `bpf_ringbuf_reserve()` failures
|
||||
|
||||
#### Tracepoint Probes
|
||||
| Probe | Event Type | Purpose |
|
||||
|-------|------------|---------|
|
||||
| `tracepoint/syscalls/sys_enter_openat` | File access | Track which files are opened |
|
||||
| `tracepoint/sched/sched_process_exec` | Process execution | Track binary invocations |
|
||||
| `tracepoint/sock/inet_sock_set_state` | TCP state | Track network connections |
|
||||
|
||||
#### Uprobe Probes
|
||||
| Probe | Library | Purpose |
|
||||
|-------|---------|---------|
|
||||
| `uprobe/libc.so:connect` | glibc/musl | Outbound network connections |
|
||||
| `uprobe/libc.so:accept` | glibc/musl | Inbound connections |
|
||||
| `uprobe/libssl.so:SSL_read` | OpenSSL | TLS traffic monitoring |
|
||||
| `uprobe/libssl.so:SSL_write` | OpenSSL | TLS traffic monitoring |
|
||||
|
||||
#### BPF Maps for Filtering
|
||||
```c
|
||||
// Cgroup filter for container targeting
|
||||
struct {
|
||||
__uint(type, BPF_MAP_TYPE_HASH);
|
||||
__uint(max_entries, 1024);
|
||||
__type(key, u64); // cgroup_id
|
||||
__type(value, u8); // 1 = include
|
||||
} cgroup_filter SEC(".maps");
|
||||
|
||||
// Namespace filter for multi-tenant isolation
|
||||
struct {
|
||||
__uint(type, BPF_MAP_TYPE_HASH);
|
||||
__uint(max_entries, 256);
|
||||
__type(key, u64); // namespace inode
|
||||
__type(value, u8); // 1 = include
|
||||
} namespace_filter SEC(".maps");
|
||||
```
|
||||
|
||||
### User-Space Components
|
||||
|
||||
#### CoreProbeLoader
|
||||
Manages eBPF program lifecycle:
|
||||
- Loads compiled `.bpf.o` files via libbpf
|
||||
- Attaches probes to tracepoints/uprobes
|
||||
- Configures BPF maps for filtering
|
||||
- Handles graceful detachment and cleanup
|
||||
|
||||
#### EventParser
|
||||
Parses binary events from ring buffer:
|
||||
- Fixed-size header with event type discriminator
|
||||
- Type-specific payload parsing
|
||||
- Timestamp normalization (boot time to wall clock)
|
||||
|
||||
#### CgroupContainerResolver
|
||||
Maps kernel cgroup IDs to container identities:
|
||||
- Parses `/proc/{pid}/cgroup` for container runtime paths
|
||||
- Supports containerd, Docker, CRI-O path formats
|
||||
- Caches mappings with configurable TTL
|
||||
|
||||
#### EnhancedSymbolResolver
|
||||
Resolves addresses to human-readable symbols:
|
||||
- Parses `/proc/{pid}/maps` for ASLR offsets
|
||||
- Reads ELF symbol tables (`.symtab`, `.dynsym`)
|
||||
- Optional DWARF debug info for line numbers
|
||||
- LRU cache with bounded memory usage
|
||||
|
||||
#### RuntimeEventEnricher
|
||||
Decorates events with container and SBOM metadata:
|
||||
- Container ID and image digest correlation
|
||||
- SBOM component (PURL) lookup
|
||||
- Graceful degradation on missing metadata
|
||||
|
||||
#### RuntimeEvidenceNdjsonWriter
|
||||
Produces deterministic NDJSON output:
|
||||
- Canonical JSON serialization (sorted keys, no whitespace variance)
|
||||
- Rolling BLAKE3 hash for content verification
|
||||
- Size and time-based rotation with callbacks
|
||||
|
||||
#### EvidenceChunkFinalizer
|
||||
Signs and links evidence chunks:
|
||||
- Creates in-toto statements with chunk metadata
|
||||
- Requests DSSE signatures via Signer service
|
||||
- Submits to Rekor transparency log
|
||||
- Maintains chain state (previous_chunk_id linkage)
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
1. Kernel Event
|
||||
│
|
||||
├─► Tracepoint/Uprobe fires
|
||||
│ └─► BPF program captures event data
|
||||
│ └─► Filter by cgroup/namespace (optional)
|
||||
│ └─► Submit to ring buffer
|
||||
│
|
||||
2. Ring Buffer Drain
|
||||
│
|
||||
├─► EventParser reads binary data
|
||||
│ └─► Deserialize to typed event struct
|
||||
│ └─► Validate event integrity
|
||||
│
|
||||
3. Resolution & Enrichment
|
||||
│
|
||||
├─► CgroupResolver: cgroup_id → container_id
|
||||
├─► SymbolResolver: address → symbol name
|
||||
├─► StateProvider: container_id → image_ref
|
||||
├─► DigestResolver: image_ref → image_digest
|
||||
└─► SbomProvider: image_digest → purls[]
|
||||
│
|
||||
4. Serialization
|
||||
│
|
||||
├─► RuntimeEvidenceNdjsonWriter
|
||||
│ ├─► Canonical JSON serialization
|
||||
│ ├─► Append to current chunk file
|
||||
│ └─► Update rolling hash
|
||||
│
|
||||
5. Rotation & Signing
|
||||
│
|
||||
├─► Size/time threshold reached
|
||||
│ └─► Close current chunk
|
||||
│ └─► ChunkFinalizer
|
||||
│ ├─► Create in-toto statement
|
||||
│ ├─► Sign with DSSE
|
||||
│ ├─► Submit to Rekor
|
||||
│ └─► Link to previous chunk
|
||||
│
|
||||
6. Verification
|
||||
│
|
||||
└─► stella signals verify-chain
|
||||
├─► Parse DSSE envelopes
|
||||
├─► Verify signatures
|
||||
├─► Check chain linkage
|
||||
└─► Validate time monotonicity
|
||||
```
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Kernel-Space
|
||||
- Ring buffer prevents event loss under load (backpressure)
|
||||
- In-kernel filtering reduces user-space processing
|
||||
- BTF enables zero-copy field access
|
||||
|
||||
### User-Space
|
||||
| Operation | Target Latency |
|
||||
|-----------|---------------|
|
||||
| Cached symbol lookup | < 1ms p99 |
|
||||
| Uncached symbol lookup | < 10ms p99 |
|
||||
| Container enrichment | < 10ms p99 |
|
||||
| NDJSON write | < 1ms p99 |
|
||||
|
||||
### Throughput
|
||||
- Target: 100,000 events/second sustained
|
||||
- Rate limiting available for resource-constrained environments
|
||||
|
||||
## Memory Budget
|
||||
|
||||
| Component | Default | Configurable |
|
||||
|-----------|---------|--------------|
|
||||
| Ring buffer | 256 KB | Yes |
|
||||
| Symbol cache | 100,000 entries | Yes |
|
||||
| Container cache | 5 min TTL | Yes |
|
||||
| Write buffer | 64 KB | Yes |
|
||||
|
||||
## Failure Modes
|
||||
|
||||
### Ring Buffer Overflow
|
||||
- **Symptom**: Events dropped, warning logged
|
||||
- **Mitigation**: Increase buffer size or enable rate limiting
|
||||
|
||||
### Symbol Resolution Failure
|
||||
- **Symptom**: Address shown as `addr:0x{hex}`
|
||||
- **Mitigation**: Ensure debug symbols available or accept address-only evidence
|
||||
|
||||
### Container Resolution Failure
|
||||
- **Symptom**: `container_id = "unknown:{cgroup_id}"`
|
||||
- **Mitigation**: Verify Zastava integration, check cgroup path format support
|
||||
|
||||
### Signing Failure
|
||||
- **Symptom**: Chunk saved without signature, warning logged
|
||||
- **Mitigation**: Check Signer service availability, verify Fulcio/KMS connectivity
|
||||
|
||||
## CO-RE (Compile Once, Run Everywhere)
|
||||
|
||||
The system uses BTF (BPF Type Format) for kernel-version-independent field access:
|
||||
|
||||
```c
|
||||
// Access kernel struct fields without hardcoded offsets
|
||||
struct task_struct *task = (void *)bpf_get_current_task();
|
||||
pid_t pid = BPF_CORE_READ(task, pid);
|
||||
pid_t tgid = BPF_CORE_READ(task, tgid);
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
- Kernel 5.2+ with built-in BTF (recommended)
|
||||
- Kernel 4.14+ with external BTF from btfhub
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Zastava (Container State)
|
||||
- `IContainerIdentityResolver` interface
|
||||
- Container lifecycle events (start/stop)
|
||||
- Image reference to digest mapping
|
||||
|
||||
### Scanner (Reachability Merger)
|
||||
- `EbpfSignalMerger` combines runtime with static analysis
|
||||
- Symbol hash correlation via `RuntimeNodeHash`
|
||||
|
||||
### Signer (Evidence Signing)
|
||||
- `IAttestationSigningService` for DSSE signatures
|
||||
- `IRekorClient` for transparency log submission
|
||||
|
||||
### SBOM Service (Component Correlation)
|
||||
- `ISbomComponentProvider` for PURL lookup
|
||||
- Image digest to component mapping
|
||||
281
docs/reachability/evidence-schema.md
Normal file
281
docs/reachability/evidence-schema.md
Normal file
@@ -0,0 +1,281 @@
|
||||
# Runtime Evidence Schema Reference
|
||||
|
||||
## Overview
|
||||
|
||||
Runtime evidence is serialized as NDJSON (Newline-Delimited JSON), with one event per line. The schema ensures deterministic output for reproducible evidence chains.
|
||||
|
||||
## Schema Location
|
||||
|
||||
- JSON Schema: `docs/schemas/runtime-evidence-v1.json`
|
||||
- C# Models: `src/Signals/__Libraries/StellaOps.Signals.Ebpf/Schema/`
|
||||
|
||||
## Common Fields
|
||||
|
||||
Every evidence record includes these base fields:
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `ts_ns` | integer | Yes | Nanoseconds since system boot |
|
||||
| `src` | string | Yes | Event source identifier |
|
||||
| `pid` | integer | Yes | Process ID |
|
||||
| `tid` | integer | No | Thread ID (if available) |
|
||||
| `cgroup_id` | integer | Yes | Kernel cgroup ID |
|
||||
| `container_id` | string | No | Container ID (enriched) |
|
||||
| `image_digest` | string | No | Image digest (enriched) |
|
||||
| `comm` | string | No | Process command name (max 16 chars) |
|
||||
| `event` | object | Yes | Type-specific event data |
|
||||
|
||||
## Event Types
|
||||
|
||||
### File Access (`file_access`)
|
||||
|
||||
Captured from `sys_enter_openat` tracepoint.
|
||||
|
||||
```json
|
||||
{
|
||||
"ts_ns": 1234567890123456789,
|
||||
"src": "tracepoint:syscalls:sys_enter_openat",
|
||||
"pid": 1234,
|
||||
"cgroup_id": 5678,
|
||||
"container_id": "abc123def456",
|
||||
"image_digest": "sha256:...",
|
||||
"comm": "nginx",
|
||||
"event": {
|
||||
"type": "file_access",
|
||||
"path": "/etc/nginx/nginx.conf",
|
||||
"flags": 0,
|
||||
"mode": 0,
|
||||
"access": "read"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `path` | string | File path (max 256 chars) |
|
||||
| `flags` | integer | Open flags (`O_RDONLY`, `O_WRONLY`, etc.) |
|
||||
| `mode` | integer | File mode (for creation) |
|
||||
| `access` | string | Derived access type: `read`, `write`, `read_write` |
|
||||
|
||||
### Process Execution (`process_exec`)
|
||||
|
||||
Captured from `sched_process_exec` tracepoint.
|
||||
|
||||
```json
|
||||
{
|
||||
"ts_ns": 1234567890123456789,
|
||||
"src": "tracepoint:sched:sched_process_exec",
|
||||
"pid": 1234,
|
||||
"cgroup_id": 5678,
|
||||
"container_id": "abc123def456",
|
||||
"image_digest": "sha256:...",
|
||||
"comm": "python3",
|
||||
"event": {
|
||||
"type": "process_exec",
|
||||
"filename": "/usr/bin/python3",
|
||||
"ppid": 1000,
|
||||
"argv": ["python3", "script.py", "--config", "/etc/app.conf"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `filename` | string | Executed binary path |
|
||||
| `ppid` | integer | Parent process ID |
|
||||
| `argv` | string[] | Command arguments (limited to first 4) |
|
||||
|
||||
### TCP State Change (`tcp_state`)
|
||||
|
||||
Captured from `inet_sock_set_state` tracepoint.
|
||||
|
||||
```json
|
||||
{
|
||||
"ts_ns": 1234567890123456789,
|
||||
"src": "tracepoint:sock:inet_sock_set_state",
|
||||
"pid": 1234,
|
||||
"cgroup_id": 5678,
|
||||
"container_id": "abc123def456",
|
||||
"image_digest": "sha256:...",
|
||||
"comm": "curl",
|
||||
"event": {
|
||||
"type": "tcp_state",
|
||||
"family": "ipv4",
|
||||
"old_state": "SYN_SENT",
|
||||
"new_state": "ESTABLISHED",
|
||||
"src_addr": "10.0.0.5",
|
||||
"src_port": 45678,
|
||||
"dst_addr": "93.184.216.34",
|
||||
"dst_port": 443
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `family` | string | Address family: `ipv4` or `ipv6` |
|
||||
| `old_state` | string | Previous TCP state |
|
||||
| `new_state` | string | New TCP state |
|
||||
| `src_addr` | string | Source IP address |
|
||||
| `src_port` | integer | Source port |
|
||||
| `dst_addr` | string | Destination IP address |
|
||||
| `dst_port` | integer | Destination port |
|
||||
|
||||
TCP States: `CLOSED`, `LISTEN`, `SYN_SENT`, `SYN_RECV`, `ESTABLISHED`, `FIN_WAIT1`, `FIN_WAIT2`, `CLOSE_WAIT`, `CLOSING`, `LAST_ACK`, `TIME_WAIT`
|
||||
|
||||
### Network Operation (`network_op`)
|
||||
|
||||
Captured from libc `connect`/`accept` uprobes.
|
||||
|
||||
```json
|
||||
{
|
||||
"ts_ns": 1234567890123456789,
|
||||
"src": "uprobe:libc.so.6:connect",
|
||||
"pid": 1234,
|
||||
"cgroup_id": 5678,
|
||||
"container_id": "abc123def456",
|
||||
"image_digest": "sha256:...",
|
||||
"comm": "app",
|
||||
"event": {
|
||||
"type": "network_op",
|
||||
"operation": "connect",
|
||||
"family": "ipv4",
|
||||
"addr": "10.0.1.100",
|
||||
"port": 5432,
|
||||
"result": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `operation` | string | `connect` or `accept` |
|
||||
| `family` | string | Address family |
|
||||
| `addr` | string | Remote address |
|
||||
| `port` | integer | Remote port |
|
||||
| `result` | integer | Return value (0 = success) |
|
||||
|
||||
### SSL Operation (`ssl_op`)
|
||||
|
||||
Captured from OpenSSL `SSL_read`/`SSL_write` uprobes.
|
||||
|
||||
```json
|
||||
{
|
||||
"ts_ns": 1234567890123456789,
|
||||
"src": "uprobe:libssl.so.3:SSL_write",
|
||||
"pid": 1234,
|
||||
"cgroup_id": 5678,
|
||||
"container_id": "abc123def456",
|
||||
"image_digest": "sha256:...",
|
||||
"comm": "nginx",
|
||||
"event": {
|
||||
"type": "ssl_op",
|
||||
"operation": "write",
|
||||
"requested_bytes": 1024,
|
||||
"actual_bytes": 1024,
|
||||
"ssl_ptr": 140234567890
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `operation` | string | `read` or `write` |
|
||||
| `requested_bytes` | integer | Bytes requested |
|
||||
| `actual_bytes` | integer | Bytes actually transferred |
|
||||
| `ssl_ptr` | integer | SSL context pointer (for correlation) |
|
||||
|
||||
### Symbol Call (`symbol_call`)
|
||||
|
||||
Captured from function uprobes.
|
||||
|
||||
```json
|
||||
{
|
||||
"ts_ns": 1234567890123456789,
|
||||
"src": "uprobe:app:vulnerable_parse_json",
|
||||
"pid": 1234,
|
||||
"cgroup_id": 5678,
|
||||
"container_id": "abc123def456",
|
||||
"image_digest": "sha256:...",
|
||||
"comm": "app",
|
||||
"event": {
|
||||
"type": "symbol_call",
|
||||
"symbol": "vulnerable_parse_json",
|
||||
"library": "/usr/lib/libapp.so",
|
||||
"offset": 4096,
|
||||
"address": 140234571986
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `symbol` | string | Function symbol name |
|
||||
| `library` | string | Library/binary path |
|
||||
| `offset` | integer | Offset within library |
|
||||
| `address` | integer | Runtime address |
|
||||
|
||||
## Determinism Requirements
|
||||
|
||||
For byte-identical output across runs:
|
||||
|
||||
1. **Field Ordering**: All JSON keys sorted alphabetically
|
||||
2. **Number Format**: Integers as-is, no floating point variance
|
||||
3. **String Encoding**: UTF-8 with NFC normalization
|
||||
4. **Null Handling**: Null fields omitted (not `"field": null`)
|
||||
5. **Whitespace**: No trailing whitespace, single newline per record
|
||||
|
||||
## Chunk Metadata
|
||||
|
||||
Each evidence chunk includes metadata in its DSSE attestation:
|
||||
|
||||
```json
|
||||
{
|
||||
"predicateType": "stella.ops/runtime-evidence@v1",
|
||||
"predicate": {
|
||||
"chunk_id": "sha256:abc123...",
|
||||
"chunk_sequence": 42,
|
||||
"previous_chunk_id": "sha256:def456...",
|
||||
"event_count": 150000,
|
||||
"time_range": {
|
||||
"start": "2026-01-27T10:00:00Z",
|
||||
"end": "2026-01-27T11:00:00Z"
|
||||
},
|
||||
"collector_version": "1.0.0",
|
||||
"kernel_version": "5.15.0-generic",
|
||||
"compression": null,
|
||||
"host_id": "node-01.cluster.local",
|
||||
"container_ids": ["abc123", "def456"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Validation
|
||||
|
||||
Evidence can be validated against the JSON Schema:
|
||||
|
||||
```bash
|
||||
# Validate single file
|
||||
stella evidence validate evidence-chunk-001.ndjson
|
||||
|
||||
# Validate and show statistics
|
||||
stella evidence validate --stats evidence-chunk-001.ndjson
|
||||
```
|
||||
|
||||
## Migration from v0 Schemas
|
||||
|
||||
If using earlier per-language schemas, migrate to v1 unified schema:
|
||||
|
||||
1. Update field names to snake_case
|
||||
2. Wrap type-specific fields in `event` object
|
||||
3. Add `src` field with probe identifier
|
||||
4. Ensure `ts_ns` uses nanoseconds since boot
|
||||
|
||||
Example migration:
|
||||
```json
|
||||
// v0 (old)
|
||||
{"timestamp": 1234567890, "type": "file", "path": "/etc/config"}
|
||||
|
||||
// v1 (new)
|
||||
{"ts_ns": 1234567890000000000, "src": "tracepoint:syscalls:sys_enter_openat", "pid": 1234, "cgroup_id": 5678, "event": {"type": "file_access", "path": "/etc/config", "flags": 0, "mode": 0, "access": "read"}}
|
||||
```
|
||||
467
docs/reachability/operator-runbook.md
Normal file
467
docs/reachability/operator-runbook.md
Normal file
@@ -0,0 +1,467 @@
|
||||
# Operator Runbook
|
||||
|
||||
## Overview
|
||||
|
||||
This runbook provides operational procedures for managing the eBPF reachability evidence collection system.
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Key Metrics
|
||||
|
||||
Monitor these metrics for system health:
|
||||
|
||||
| Metric | Description | Alert Threshold |
|
||||
|--------|-------------|-----------------|
|
||||
| `stellaops_signals_events_total` | Total events collected | N/A (info) |
|
||||
| `stellaops_signals_events_rate` | Events per second | > 100,000 (high load) |
|
||||
| `stellaops_signals_ringbuf_usage` | Ring buffer utilization % | > 80% (overflow risk) |
|
||||
| `stellaops_signals_drops_total` | Events dropped | > 0 (investigate) |
|
||||
| `stellaops_signals_enrich_latency_p99` | Enrichment latency | > 50ms (degraded) |
|
||||
| `stellaops_signals_chunks_signed` | Signed chunks count | N/A (info) |
|
||||
| `stellaops_signals_rekor_failures` | Rekor submission failures | > 0 (investigate) |
|
||||
|
||||
### Health Checks
|
||||
|
||||
```bash
|
||||
# Quick health check
|
||||
stella signals health
|
||||
|
||||
# Detailed status
|
||||
stella signals status --verbose
|
||||
|
||||
# Prometheus metrics
|
||||
curl localhost:9090/metrics | grep stellaops_signals
|
||||
```
|
||||
|
||||
### Log Analysis
|
||||
|
||||
```bash
|
||||
# View recent logs
|
||||
journalctl -u stellaops-signals --since "1 hour ago"
|
||||
|
||||
# Filter by severity
|
||||
journalctl -u stellaops-signals -p err
|
||||
|
||||
# Follow live
|
||||
journalctl -u stellaops-signals -f
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
### Issue: Probe Failed to Attach
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Error: Failed to attach tracepoint/syscalls/sys_enter_openat: permission denied
|
||||
```
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Check capabilities
|
||||
getcap /usr/bin/stella
|
||||
|
||||
# Check kernel config
|
||||
cat /boot/config-$(uname -r) | grep CONFIG_BPF
|
||||
|
||||
# Check seccomp/AppArmor
|
||||
dmesg | grep -i "bpf\|seccomp\|apparmor"
|
||||
```
|
||||
|
||||
**Resolution:**
|
||||
1. Ensure proper capabilities:
|
||||
```bash
|
||||
sudo setcap cap_bpf,cap_perfmon,cap_sys_ptrace+ep /usr/bin/stella
|
||||
```
|
||||
2. Or run as root:
|
||||
```bash
|
||||
sudo stella signals start
|
||||
```
|
||||
3. Check AppArmor/SELinux isn't blocking
|
||||
|
||||
---
|
||||
|
||||
### Issue: Ring Buffer Overflow
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Warning: Ring buffer full, 1523 events dropped
|
||||
```
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Check buffer usage
|
||||
stella signals stats | grep ringbuf
|
||||
|
||||
# Check event rate
|
||||
stella signals stats | grep rate
|
||||
```
|
||||
|
||||
**Resolution:**
|
||||
1. Increase buffer size:
|
||||
```yaml
|
||||
signals:
|
||||
ring_buffer_size: 1048576 # 1MB
|
||||
```
|
||||
2. Enable rate limiting:
|
||||
```yaml
|
||||
signals:
|
||||
max_events_per_second: 50000
|
||||
```
|
||||
3. Add more aggressive filtering:
|
||||
```yaml
|
||||
signals:
|
||||
filters:
|
||||
paths:
|
||||
denylist:
|
||||
- /proc/**
|
||||
- /sys/**
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: High Memory Usage
|
||||
|
||||
**Symptoms:**
|
||||
- OOM kills
|
||||
- High RSS in process stats
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Check memory breakdown
|
||||
stella signals stats --memory
|
||||
|
||||
# Check cache sizes
|
||||
stella signals cache-stats
|
||||
```
|
||||
|
||||
**Resolution:**
|
||||
1. Reduce cache sizes:
|
||||
```yaml
|
||||
signals:
|
||||
resources:
|
||||
symbol_cache_max_entries: 50000
|
||||
max_cache_memory_mb: 128
|
||||
```
|
||||
2. Reduce container cache TTL:
|
||||
```yaml
|
||||
signals:
|
||||
resources:
|
||||
container_cache_ttl_seconds: 60
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Symbol Resolution Failures
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Symbol: addr:0x7f4a3b2c1000 (unresolved)
|
||||
```
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Check if binary has symbols
|
||||
nm /path/to/binary | head
|
||||
|
||||
# Check if debuginfo available
|
||||
file /path/to/binary | grep "not stripped"
|
||||
```
|
||||
|
||||
**Resolution:**
|
||||
1. Install debug symbols:
|
||||
```bash
|
||||
# Debian/Ubuntu
|
||||
apt install libc6-dbg
|
||||
|
||||
# RHEL/CentOS
|
||||
debuginfo-install glibc
|
||||
```
|
||||
2. Accept address-only evidence (still valuable for correlation)
|
||||
|
||||
---
|
||||
|
||||
### Issue: Container Resolution Failures
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
container_id: unknown:1234567890
|
||||
```
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Check cgroup path format
|
||||
cat /proc/<pid>/cgroup
|
||||
|
||||
# Verify container runtime
|
||||
docker ps
|
||||
crictl ps
|
||||
```
|
||||
|
||||
**Resolution:**
|
||||
1. Verify Zastava integration is running
|
||||
2. Check container runtime is supported (containerd/Docker/CRI-O)
|
||||
3. Restart collector to refresh container mappings
|
||||
|
||||
---
|
||||
|
||||
### Issue: Evidence Chain Verification Failure
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
$ stella signals verify-chain /var/lib/stellaops/evidence/
|
||||
Chain Status: ✗ INVALID
|
||||
Error: Chain broken at chunk 42
|
||||
```
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Get detailed report
|
||||
stella signals verify-chain /var/lib/stellaops/evidence/ --verbose --format json
|
||||
```
|
||||
|
||||
**Resolution:**
|
||||
1. Check for missing chunk files
|
||||
2. Check for disk corruption
|
||||
3. If intentional restart, document gap in audit trail
|
||||
4. Re-initialize chain if necessary:
|
||||
```bash
|
||||
stella signals reset-chain --confirm
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Rekor Submission Failures
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Warning: Failed to submit to Rekor: connection refused
|
||||
```
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Check Rekor connectivity
|
||||
curl https://rekor.sigstore.dev/api/v1/log
|
||||
|
||||
# Check signing service
|
||||
stella signer status
|
||||
```
|
||||
|
||||
**Resolution:**
|
||||
1. Check network connectivity to Rekor
|
||||
2. Verify Fulcio/OIDC tokens are valid
|
||||
3. Switch to offline mode temporarily:
|
||||
```yaml
|
||||
signals:
|
||||
signing:
|
||||
submit_to_rekor: false
|
||||
```
|
||||
4. Retry failed submissions later:
|
||||
```bash
|
||||
stella signals resubmit-pending
|
||||
```
|
||||
|
||||
## Operational Procedures
|
||||
|
||||
### Procedure: Rotate Evidence Directory
|
||||
|
||||
When evidence directory is full or needs archival:
|
||||
|
||||
```bash
|
||||
# 1. Stop collector gracefully
|
||||
stella signals stop
|
||||
|
||||
# 2. Archive current evidence
|
||||
tar -czvf evidence-$(date +%Y%m%d).tar.gz /var/lib/stellaops/evidence/
|
||||
|
||||
# 3. Verify archive integrity
|
||||
stella signals verify-chain evidence-$(date +%Y%m%d).tar.gz
|
||||
|
||||
# 4. Move to long-term storage
|
||||
aws s3 cp evidence-$(date +%Y%m%d).tar.gz s3://evidence-archive/
|
||||
|
||||
# 5. Clear old evidence (keep chain state)
|
||||
stella signals cleanup --keep-chain-state --older-than 7d
|
||||
|
||||
# 6. Restart collector
|
||||
stella signals start
|
||||
```
|
||||
|
||||
### Procedure: Update Collector
|
||||
|
||||
```bash
|
||||
# 1. Check current version
|
||||
stella version
|
||||
|
||||
# 2. Download new version
|
||||
curl -fsSL https://stella.ops/install.sh | bash -s -- --version 1.2.0
|
||||
|
||||
# 3. Verify probe compatibility
|
||||
stella signals test-probes
|
||||
|
||||
# 4. Restart service
|
||||
sudo systemctl restart stellaops-signals
|
||||
|
||||
# 5. Verify operation
|
||||
stella signals status
|
||||
```
|
||||
|
||||
### Procedure: Recover from Crash
|
||||
|
||||
```bash
|
||||
# 1. Check service status
|
||||
systemctl status stellaops-signals
|
||||
|
||||
# 2. Check for core dumps
|
||||
coredumpctl list | grep stella
|
||||
|
||||
# 3. Review logs for cause
|
||||
journalctl -u stellaops-signals --since "30 min ago"
|
||||
|
||||
# 4. Verify chain state
|
||||
stella signals verify-chain /var/lib/stellaops/evidence/
|
||||
|
||||
# 5. Restart service
|
||||
sudo systemctl start stellaops-signals
|
||||
|
||||
# 6. Monitor for recurrence
|
||||
watch -n 5 'stella signals stats'
|
||||
```
|
||||
|
||||
### Procedure: Air-Gap Evidence Export
|
||||
|
||||
```bash
|
||||
# 1. Create signed export bundle
|
||||
stella signals export \
|
||||
--from 2026-01-01 \
|
||||
--to 2026-01-31 \
|
||||
--include-proofs \
|
||||
--output january-evidence.tar.gz
|
||||
|
||||
# 2. Generate verification manifest
|
||||
stella signals manifest january-evidence.tar.gz > manifest.json
|
||||
|
||||
# 3. Transfer to verification system
|
||||
scp january-evidence.tar.gz manifest.json airgap-verifier:
|
||||
|
||||
# 4. On verifier, import and verify
|
||||
stella signals import january-evidence.tar.gz
|
||||
stella signals verify-chain --offline /imported/evidence/
|
||||
```
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
### Full Configuration Example
|
||||
|
||||
```yaml
|
||||
signals:
|
||||
enabled: true
|
||||
output_directory: /var/lib/stellaops/evidence
|
||||
|
||||
# Ring buffer (kernel space)
|
||||
ring_buffer_size: 262144 # 256KB
|
||||
|
||||
# Rate limiting
|
||||
max_events_per_second: 0 # unlimited
|
||||
|
||||
# Rotation
|
||||
rotation:
|
||||
max_size_mb: 100
|
||||
max_age_hours: 1
|
||||
|
||||
# Signing
|
||||
signing:
|
||||
enabled: true
|
||||
key_id: fulcio
|
||||
submit_to_rekor: true
|
||||
|
||||
# Probes
|
||||
probes:
|
||||
sys_enter_openat: true
|
||||
sched_process_exec: true
|
||||
inet_sock_set_state: true
|
||||
libc_connect: true
|
||||
libc_accept: true
|
||||
openssl_read: true
|
||||
openssl_write: true
|
||||
|
||||
# Filters
|
||||
filters:
|
||||
target_containers: []
|
||||
target_namespaces: []
|
||||
paths:
|
||||
allowlist:
|
||||
- /etc/**
|
||||
- /var/lib/**
|
||||
denylist:
|
||||
- /proc/**
|
||||
- /sys/**
|
||||
- /dev/**
|
||||
networks:
|
||||
allowlist: []
|
||||
denylist:
|
||||
- 127.0.0.0/8
|
||||
|
||||
# Resources
|
||||
resources:
|
||||
max_cache_memory_mb: 256
|
||||
symbol_cache_max_entries: 100000
|
||||
container_cache_ttl_seconds: 300
|
||||
|
||||
# Observability
|
||||
metrics:
|
||||
enabled: true
|
||||
port: 9090
|
||||
logging:
|
||||
level: info
|
||||
format: json
|
||||
```
|
||||
|
||||
## Emergency Procedures
|
||||
|
||||
### Emergency: Disable Collection
|
||||
|
||||
If collector is causing system issues:
|
||||
|
||||
```bash
|
||||
# Immediate stop
|
||||
sudo systemctl stop stellaops-signals
|
||||
|
||||
# Disable on boot
|
||||
sudo systemctl disable stellaops-signals
|
||||
|
||||
# Remove all probes manually
|
||||
sudo bpftool prog list | grep stella | awk '{print $1}' | xargs -I{} sudo bpftool prog detach {}
|
||||
```
|
||||
|
||||
### Emergency: Clear Corrupted State
|
||||
|
||||
If state is corrupted and normal recovery fails:
|
||||
|
||||
```bash
|
||||
# Stop service
|
||||
sudo systemctl stop stellaops-signals
|
||||
|
||||
# Backup current state
|
||||
cp -r /var/lib/stellaops/evidence /var/lib/stellaops/evidence.backup
|
||||
|
||||
# Clear state
|
||||
rm -rf /var/lib/stellaops/evidence/*
|
||||
|
||||
# Re-initialize
|
||||
stella signals init
|
||||
|
||||
# Start fresh
|
||||
sudo systemctl start stellaops-signals
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
For issues not covered in this runbook:
|
||||
|
||||
1. Check [GitHub Issues](https://github.com/stellaops/stellaops/issues)
|
||||
2. Search [Documentation](https://docs.stella.ops/)
|
||||
3. Contact support with:
|
||||
- Output of `stella signals status --verbose`
|
||||
- Relevant log excerpts
|
||||
- Kernel version (`uname -a`)
|
||||
- Configuration file (sanitized)
|
||||
275
docs/reachability/probe-reference.md
Normal file
275
docs/reachability/probe-reference.md
Normal file
@@ -0,0 +1,275 @@
|
||||
# Probe Reference
|
||||
|
||||
## Overview
|
||||
|
||||
This document details each eBPF probe used for runtime evidence collection, including kernel requirements, captured data, and known limitations.
|
||||
|
||||
## Tracepoint Probes
|
||||
|
||||
### sys_enter_openat
|
||||
|
||||
**Location:** `tracepoint/syscalls/sys_enter_openat`
|
||||
|
||||
**Purpose:** Capture file access operations to prove which files were read or written.
|
||||
|
||||
**Kernel Requirement:** 2.6.16+ (openat syscall), 4.14+ for eBPF attachment
|
||||
|
||||
**Source File:** `src/Signals/__Libraries/StellaOps.Signals.Ebpf/Probes/Bpf/syscall_openat.bpf.c`
|
||||
|
||||
**Captured Fields:**
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `timestamp_ns` | u64 | Nanoseconds since boot |
|
||||
| `pid` | u32 | Process ID |
|
||||
| `tid` | u32 | Thread ID |
|
||||
| `cgroup_id` | u64 | Kernel cgroup ID |
|
||||
| `dfd` | int | Directory file descriptor |
|
||||
| `flags` | int | Open flags (O_RDONLY, O_WRONLY, etc.) |
|
||||
| `mode` | u16 | File mode for creation |
|
||||
| `filename` | char[256] | File path |
|
||||
| `comm` | char[16] | Process command name |
|
||||
|
||||
**Filtering:**
|
||||
- Cgroup-based: Only capture events from specified containers
|
||||
- Path-based: Allowlist/denylist patterns applied in user space
|
||||
|
||||
**Fallback:** For kernels without `openat` (pre-2.6.16), attaches to `sys_enter_open` instead.
|
||||
|
||||
**Performance Impact:** ~1-2% CPU at 10,000 opens/second
|
||||
|
||||
---
|
||||
|
||||
### sched_process_exec
|
||||
|
||||
**Location:** `tracepoint/sched/sched_process_exec`
|
||||
|
||||
**Purpose:** Capture process execution to prove which binaries were invoked.
|
||||
|
||||
**Kernel Requirement:** 3.4+ for tracepoint, 4.14+ for eBPF attachment
|
||||
|
||||
**Source File:** `src/Signals/__Libraries/StellaOps.Signals.Ebpf/Probes/Bpf/syscall_exec.bpf.c`
|
||||
|
||||
**Captured Fields:**
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `timestamp_ns` | u64 | Nanoseconds since boot |
|
||||
| `pid` | u32 | Process ID (after exec) |
|
||||
| `ppid` | u32 | Parent process ID |
|
||||
| `cgroup_id` | u64 | Kernel cgroup ID |
|
||||
| `filename` | char[256] | Executed binary path |
|
||||
| `comm` | char[16] | Process command name |
|
||||
| `argv0` | char[128] | First argument |
|
||||
|
||||
**Argv Capture:**
|
||||
- Limited to first 4 arguments for safety
|
||||
- Each argument truncated to 128 bytes
|
||||
- Uses `bpf_probe_read_user_str()` with bounds checking
|
||||
|
||||
**Interpreter Detection:**
|
||||
- Recognizes shebangs for Python, Node, Ruby, Shell scripts
|
||||
- Maps `/usr/bin/python script.py` to script path
|
||||
|
||||
**Performance Impact:** Minimal (exec rate typically low)
|
||||
|
||||
---
|
||||
|
||||
### inet_sock_set_state
|
||||
|
||||
**Location:** `tracepoint/sock/inet_sock_set_state`
|
||||
|
||||
**Purpose:** Capture TCP connection lifecycle to prove network communication patterns.
|
||||
|
||||
**Kernel Requirement:** 4.16+ (tracepoint added), BTF recommended for CO-RE
|
||||
|
||||
**Source File:** `src/Signals/__Libraries/StellaOps.Signals.Ebpf/Probes/Bpf/syscall_network.bpf.c`
|
||||
|
||||
**Captured Fields:**
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `timestamp_ns` | u64 | Nanoseconds since boot |
|
||||
| `pid` | u32 | Process ID |
|
||||
| `cgroup_id` | u64 | Kernel cgroup ID |
|
||||
| `oldstate` | u8 | Previous TCP state |
|
||||
| `newstate` | u8 | New TCP state |
|
||||
| `sport` | u16 | Source port |
|
||||
| `dport` | u16 | Destination port |
|
||||
| `family` | u8 | AF_INET (2) or AF_INET6 (10) |
|
||||
| `saddr_v4` / `saddr_v6` | u32 / u8[16] | Source address |
|
||||
| `daddr_v4` / `daddr_v6` | u32 / u8[16] | Destination address |
|
||||
| `comm` | char[16] | Process command name |
|
||||
|
||||
**State Transition Filtering:**
|
||||
- Default: Only `* -> ESTABLISHED` and `* -> CLOSE`
|
||||
- Configurable: All transitions for debugging
|
||||
|
||||
**Address Formatting:**
|
||||
- IPv4: Dotted decimal (e.g., `192.168.1.1`)
|
||||
- IPv6: RFC 5952 compressed (e.g., `2001:db8::1`)
|
||||
|
||||
**Performance Impact:** ~1% CPU at high connection rate
|
||||
|
||||
---
|
||||
|
||||
## Uprobe Probes
|
||||
|
||||
### libc connect/accept
|
||||
|
||||
**Location:**
|
||||
- `uprobe/libc.so.6:connect`
|
||||
- `uretprobe/libc.so.6:connect`
|
||||
- `uprobe/libc.so.6:accept`
|
||||
- `uprobe/libc.so.6:accept4`
|
||||
|
||||
**Purpose:** Capture network operations at libc level as alternative to kernel tracepoints.
|
||||
|
||||
**Library Support:**
|
||||
- glibc: `libc.so.6`
|
||||
- musl: `libc.musl-*.so.1`
|
||||
|
||||
**Source File:** `src/Signals/__Libraries/StellaOps.Signals.Ebpf/Probes/Bpf/uprobe_libc.bpf.c`
|
||||
|
||||
**Captured Fields (connect):**
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `timestamp_ns` | u64 | Nanoseconds since boot |
|
||||
| `pid` | u32 | Process ID |
|
||||
| `cgroup_id` | u64 | Kernel cgroup ID |
|
||||
| `fd` | int | Socket file descriptor |
|
||||
| `family` | u16 | Address family |
|
||||
| `addr` | varies | Remote address |
|
||||
| `port` | u16 | Remote port |
|
||||
| `comm` | char[16] | Process command name |
|
||||
| `result` | int | Return value (from uretprobe) |
|
||||
|
||||
**Library Path Resolution:**
|
||||
1. Parse `/etc/ld.so.cache` for library locations
|
||||
2. Fall back to common paths (`/lib/x86_64-linux-gnu/`, etc.)
|
||||
3. Handle container-specific paths via `/proc/{pid}/root`
|
||||
|
||||
**Byte Counting (optional):**
|
||||
- `uprobe/libc.so.6:read` and `uprobe/libc.so.6:write`
|
||||
- Tracks bytes per file descriptor
|
||||
- Aggregated to prevent event flood
|
||||
|
||||
---
|
||||
|
||||
### OpenSSL SSL_read/SSL_write
|
||||
|
||||
**Location:**
|
||||
- `uprobe/libssl.so.3:SSL_read`
|
||||
- `uretprobe/libssl.so.3:SSL_read`
|
||||
- `uprobe/libssl.so.3:SSL_write`
|
||||
- `uretprobe/libssl.so.3:SSL_write`
|
||||
|
||||
**Purpose:** Capture TLS traffic volumes without decryption.
|
||||
|
||||
**Library Support:**
|
||||
- OpenSSL 1.1.x: `libssl.so.1.1`
|
||||
- OpenSSL 3.x: `libssl.so.3`
|
||||
- LibreSSL: `libssl.so.*` (best-effort)
|
||||
- BoringSSL: Limited support
|
||||
|
||||
**Source File:** `src/Signals/__Libraries/StellaOps.Signals.Ebpf/Probes/Bpf/uprobe_openssl.bpf.c`
|
||||
|
||||
**Captured Fields:**
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `timestamp_ns` | u64 | Nanoseconds since boot |
|
||||
| `pid` | u32 | Process ID |
|
||||
| `cgroup_id` | u64 | Kernel cgroup ID |
|
||||
| `operation` | u8 | READ (0) or WRITE (1) |
|
||||
| `requested_bytes` | u32 | Bytes requested |
|
||||
| `actual_bytes` | u32 | Bytes transferred (from uretprobe) |
|
||||
| `ssl_ptr` | u64 | SSL context pointer |
|
||||
| `comm` | char[16] | Process command name |
|
||||
|
||||
**Session Correlation:**
|
||||
- `ssl_ptr` can correlate with `SSL_get_fd` for socket mapping
|
||||
- Optional: `SSL_get_peer_certificate` for peer info
|
||||
|
||||
**Byte Aggregation:**
|
||||
- High-throughput connections aggregate to periodic summaries
|
||||
- Prevents event flood on bulk data transfer
|
||||
|
||||
---
|
||||
|
||||
### Function Tracer (Generic)
|
||||
|
||||
**Location:** `uprobe/{binary}:{symbol}`
|
||||
|
||||
**Purpose:** Attach to arbitrary function symbols for custom evidence.
|
||||
|
||||
**Source File:** `src/Signals/__Libraries/StellaOps.Signals.Ebpf/Probes/Bpf/function_tracer.bpf.c`
|
||||
|
||||
**Captured Fields:**
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `timestamp_ns` | u64 | Nanoseconds since boot |
|
||||
| `pid` | u32 | Process ID |
|
||||
| `cgroup_id` | u64 | Kernel cgroup ID |
|
||||
| `address` | u64 | Runtime address |
|
||||
| `symbol_id` | u32 | Symbol identifier (from BPF map) |
|
||||
| `comm` | char[16] | Process command name |
|
||||
|
||||
**Symbol Resolution:**
|
||||
- User-space resolves address to symbol via ELF tables
|
||||
- ASLR offset calculated from `/proc/{pid}/maps`
|
||||
- Cached for performance
|
||||
|
||||
---
|
||||
|
||||
## Kernel Version Compatibility
|
||||
|
||||
| Feature | Minimum Kernel | Recommended |
|
||||
|---------|---------------|-------------|
|
||||
| Basic eBPF | 4.14 | 5.x+ |
|
||||
| BTF (CO-RE) | 5.2 | 5.8+ |
|
||||
| Ring buffer | 5.8 | 5.8+ |
|
||||
| `sys_enter_openat` | 4.14 | 5.x+ |
|
||||
| `sched_process_exec` | 4.14 | 5.x+ |
|
||||
| `inet_sock_set_state` | 4.16 | 5.x+ |
|
||||
| Uprobes | 4.14 | 5.x+ |
|
||||
|
||||
## Known Limitations
|
||||
|
||||
### Tracepoints
|
||||
- **sys_enter_openat**: Path may be relative; resolution requires dfd lookup
|
||||
- **sched_process_exec**: Argv reading limited by verifier complexity
|
||||
- **inet_sock_set_state**: UDP not covered; use kprobe for UDP if needed
|
||||
|
||||
### Uprobes
|
||||
- **Library resolution**: May fail for statically linked binaries
|
||||
- **musl libc**: Some symbol names differ from glibc
|
||||
- **OpenSSL**: Version detection required for correct symbol names
|
||||
- **Stripped binaries**: Uprobes require symbol tables
|
||||
|
||||
### General
|
||||
- **eBPF verifier**: Complex programs may be rejected
|
||||
- **Container namespaces**: Paths may differ from host view
|
||||
- **High event rate**: Ring buffer overflow possible under extreme load
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Probe Failed to Attach
|
||||
```
|
||||
Error: Failed to attach tracepoint/syscalls/sys_enter_openat
|
||||
```
|
||||
- Check kernel version supports the tracepoint
|
||||
- Verify eBPF is enabled (`CONFIG_BPF=y`, `CONFIG_BPF_SYSCALL=y`)
|
||||
- Check permissions (CAP_BPF or root required)
|
||||
|
||||
### Missing BTF
|
||||
```
|
||||
Error: BTF not found for kernel version
|
||||
```
|
||||
- Install kernel BTF package (`linux-image-*-dbg` on Debian/Ubuntu)
|
||||
- Use BTFHub for external BTF files
|
||||
- Fall back to pre-compiled probes for specific kernel
|
||||
|
||||
### Ring Buffer Overflow
|
||||
```
|
||||
Warning: Ring buffer full, events dropped
|
||||
```
|
||||
- Increase buffer size: `--ring-buffer-size 1M`
|
||||
- Enable more aggressive filtering
|
||||
- Enable rate limiting: `--max-events-per-second 10000`
|
||||
311
docs/reachability/security-model.md
Normal file
311
docs/reachability/security-model.md
Normal file
@@ -0,0 +1,311 @@
|
||||
# Security Model
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the security model for the eBPF reachability evidence system, including threat model, trust boundaries, and mitigations.
|
||||
|
||||
## Trust Boundaries
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Untrusted Zone │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Monitored Workloads │ │
|
||||
│ │ (containers, processes generating events) │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
══════════╪══════════ Trust Boundary 1
|
||||
│
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Kernel Space (Trusted) │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ eBPF Verifier (enforces safety) │ │
|
||||
│ │ ├─ Memory bounds checking │ │
|
||||
│ │ ├─ No unbounded loops │ │
|
||||
│ │ └─ Restricted kernel API access │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ eBPF Programs (verified safe) │ │
|
||||
│ │ └─ Ring buffer output only │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
══════════╪══════════ Trust Boundary 2
|
||||
│
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Collector (Trusted Component) │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ RuntimeSignalCollector │ │
|
||||
│ │ ├─ Privileged (CAP_BPF, CAP_PERFMON, CAP_SYS_PTRACE) │ │
|
||||
│ │ ├─ Reads ring buffer │ │
|
||||
│ │ └─ Writes signed evidence │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
══════════╪══════════ Trust Boundary 3
|
||||
│
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Evidence Storage │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Signed NDJSON Chunks │ │
|
||||
│ │ ├─ DSSE signatures (Fulcio/KMS) │ │
|
||||
│ │ ├─ Rekor inclusion proofs │ │
|
||||
│ │ └─ Chain linkage (previous_chunk_id) │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Threat Model
|
||||
|
||||
### Threat 1: Malicious Workload Evasion
|
||||
|
||||
**Description:** Attacker attempts to hide malicious activity from evidence collection.
|
||||
|
||||
**Attack Vectors:**
|
||||
- Disable/bypass eBPF probes
|
||||
- Use syscalls not monitored
|
||||
- Operate from unmonitored namespaces
|
||||
|
||||
**Mitigations:**
|
||||
- Collector runs with elevated privileges, not accessible to workloads
|
||||
- Comprehensive probe coverage (syscalls + uprobes)
|
||||
- Namespace filtering ensures coverage of target workloads
|
||||
- Kernel-level capture cannot be bypassed from user space
|
||||
|
||||
**Residual Risk:** Novel syscalls or kernel exploits may evade monitoring.
|
||||
|
||||
---
|
||||
|
||||
### Threat 2: Evidence Tampering
|
||||
|
||||
**Description:** Attacker attempts to modify evidence after collection.
|
||||
|
||||
**Attack Vectors:**
|
||||
- Modify NDJSON files on disk
|
||||
- Delete evidence chunks
|
||||
- Break chain linkage
|
||||
|
||||
**Mitigations:**
|
||||
- DSSE signatures on each chunk (Fulcio ephemeral keys or KMS)
|
||||
- Rekor transparency log provides tamper-evident timestamps
|
||||
- Chain linkage (previous_chunk_id) detects deletions/insertions
|
||||
- Verification CLI detects any modifications
|
||||
|
||||
**Residual Risk:** Attacker with Signer key access could forge valid signatures (mitigated by Fulcio/OIDC).
|
||||
|
||||
---
|
||||
|
||||
### Threat 3: Collector Compromise
|
||||
|
||||
**Description:** Attacker gains control of the collector process.
|
||||
|
||||
**Attack Vectors:**
|
||||
- Exploit vulnerability in collector code
|
||||
- Compromise host and access collector credentials
|
||||
- Supply chain attack on collector binary
|
||||
|
||||
**Mitigations:**
|
||||
- Minimal attack surface (single-purpose daemon)
|
||||
- Capability-based privileges (not full root)
|
||||
- Signed releases with provenance attestations
|
||||
- Collector cannot modify already-signed chunks
|
||||
|
||||
**Residual Risk:** Zero-day in collector could allow evidence manipulation before signing.
|
||||
|
||||
---
|
||||
|
||||
### Threat 4: Denial of Service
|
||||
|
||||
**Description:** Attacker overwhelms evidence collection system.
|
||||
|
||||
**Attack Vectors:**
|
||||
- Generate excessive events to overflow ring buffer
|
||||
- Exhaust disk space with evidence
|
||||
- CPU exhaustion through complex enrichment
|
||||
|
||||
**Mitigations:**
|
||||
- Ring buffer backpressure (events dropped, not crash)
|
||||
- Rate limiting configurable
|
||||
- Disk space monitoring with rotation
|
||||
- Bounded caches prevent memory exhaustion
|
||||
|
||||
**Residual Risk:** Sustained attack could cause evidence gaps (documented in chain).
|
||||
|
||||
---
|
||||
|
||||
### Threat 5: Privacy/Data Exfiltration
|
||||
|
||||
**Description:** Evidence contains sensitive information exposed to unauthorized parties.
|
||||
|
||||
**Attack Vectors:**
|
||||
- File paths reveal sensitive locations
|
||||
- Command arguments contain secrets
|
||||
- Network destinations reveal infrastructure
|
||||
|
||||
**Mitigations:**
|
||||
- Path filtering (denylist sensitive paths)
|
||||
- Argument truncation and filtering
|
||||
- Network CIDR filtering
|
||||
- Evidence access controlled by filesystem permissions
|
||||
- Encryption at rest (optional)
|
||||
|
||||
**Residual Risk:** Metadata leakage possible even with filtering.
|
||||
|
||||
---
|
||||
|
||||
### Threat 6: Replay/Injection Attacks
|
||||
|
||||
**Description:** Attacker injects fabricated evidence or replays old evidence.
|
||||
|
||||
**Attack Vectors:**
|
||||
- Inject false events into evidence stream
|
||||
- Replay signed chunks from different time period
|
||||
- Forge DSSE envelopes
|
||||
|
||||
**Mitigations:**
|
||||
- Ring buffer is kernel-only write
|
||||
- Timestamps from kernel (monotonic, not settable by user space)
|
||||
- Chain linkage prevents replay (previous_chunk_id)
|
||||
- Rekor timestamps provide external time anchor
|
||||
- DSSE signatures with certificate transparency
|
||||
|
||||
**Residual Risk:** Attacker with collector access could inject events before signing.
|
||||
|
||||
## Security Controls
|
||||
|
||||
### Kernel-Level Controls
|
||||
|
||||
| Control | Description |
|
||||
|---------|-------------|
|
||||
| eBPF Verifier | Validates program safety before loading |
|
||||
| BTF | Type-safe kernel access without hardcoded offsets |
|
||||
| Capability Checks | BPF_PROG_LOAD requires CAP_BPF |
|
||||
| LSM Hooks | AppArmor/SELinux can restrict BPF operations |
|
||||
|
||||
### Collector Controls
|
||||
|
||||
| Control | Description |
|
||||
|---------|-------------|
|
||||
| Minimal Privileges | Only CAP_BPF, CAP_PERFMON, CAP_SYS_PTRACE |
|
||||
| Sandboxing | Systemd hardening (NoNewPrivileges, ProtectSystem) |
|
||||
| Input Validation | Bounds checking on all kernel data |
|
||||
| Secure Defaults | Signing enabled, Rekor submission enabled |
|
||||
|
||||
### Evidence Controls
|
||||
|
||||
| Control | Description |
|
||||
|---------|-------------|
|
||||
| DSSE Signing | Cryptographic integrity for each chunk |
|
||||
| Chain Linking | Tamper-evident sequence |
|
||||
| Rekor Inclusion | Public timestamp and immutability |
|
||||
| Offline Verification | No trust in online services required |
|
||||
|
||||
## Hardening Recommendations
|
||||
|
||||
### Collector Hardening
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/stellaops-signals.service.d/hardening.conf
|
||||
[Service]
|
||||
# Prevent privilege escalation
|
||||
NoNewPrivileges=yes
|
||||
|
||||
# Protect system directories
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
ProtectKernelTunables=yes
|
||||
ProtectKernelModules=yes
|
||||
|
||||
# Allow only necessary capabilities
|
||||
CapabilityBoundingSet=CAP_BPF CAP_PERFMON CAP_SYS_PTRACE
|
||||
|
||||
# Restrict syscalls
|
||||
SystemCallFilter=@system-service
|
||||
SystemCallFilter=~@privileged
|
||||
|
||||
# Network isolation (if not needed)
|
||||
PrivateNetwork=yes
|
||||
|
||||
# Read-only evidence directory (write via tmpfs)
|
||||
ReadWritePaths=/var/lib/stellaops/evidence
|
||||
```
|
||||
|
||||
### Access Control
|
||||
|
||||
```bash
|
||||
# Evidence directory permissions
|
||||
chmod 750 /var/lib/stellaops/evidence
|
||||
chown stellaops:stellaops-readers /var/lib/stellaops/evidence
|
||||
|
||||
# Configuration permissions
|
||||
chmod 640 /etc/stellaops/signals.yaml
|
||||
chown root:stellaops /etc/stellaops/signals.yaml
|
||||
```
|
||||
|
||||
### Encryption at Rest
|
||||
|
||||
```yaml
|
||||
# Enable encrypted evidence storage
|
||||
signals:
|
||||
encryption:
|
||||
enabled: true
|
||||
key_id: arn:aws:kms:us-east-1:123456789:key/abc-123
|
||||
```
|
||||
|
||||
## Compliance Mapping
|
||||
|
||||
### SOC 2
|
||||
|
||||
| Control | Implementation |
|
||||
|---------|----------------|
|
||||
| CC6.1 Logical Access | Capability-based privileges |
|
||||
| CC6.6 System Boundaries | Trust boundaries documented |
|
||||
| CC7.2 System Monitoring | Comprehensive event capture |
|
||||
| CC8.1 Change Management | Signed collector releases |
|
||||
|
||||
### NIST 800-53
|
||||
|
||||
| Control | Implementation |
|
||||
|---------|----------------|
|
||||
| AU-3 Content of Audit Records | Rich event schema |
|
||||
| AU-9 Protection of Audit Information | DSSE signing, Rekor |
|
||||
| AU-10 Non-repudiation | Chain linkage, transparency log |
|
||||
| SI-4 System Monitoring | eBPF-based collection |
|
||||
|
||||
### PCI-DSS
|
||||
|
||||
| Requirement | Implementation |
|
||||
|-------------|----------------|
|
||||
| 10.2 Audit Trails | Syscall/uprobe logging |
|
||||
| 10.5 Secure Audit Trails | Cryptographic signing |
|
||||
| 10.7 Audit History | Configurable retention |
|
||||
|
||||
## Incident Response
|
||||
|
||||
### Evidence Integrity Alert
|
||||
|
||||
If chain verification fails:
|
||||
|
||||
1. **Isolate** affected evidence chunks
|
||||
2. **Preserve** surrounding chunks and Rekor proofs
|
||||
3. **Analyze** verification report for failure cause
|
||||
4. **Report** gap in audit trail to compliance
|
||||
5. **Investigate** root cause (crash, attack, bug)
|
||||
|
||||
### Collector Compromise
|
||||
|
||||
If collector compromise suspected:
|
||||
|
||||
1. **Stop** collector immediately
|
||||
2. **Preserve** last signed chunk for forensics
|
||||
3. **Rotate** signing keys if KMS-based
|
||||
4. **Audit** Rekor for unexpected submissions
|
||||
5. **Reinstall** collector from verified source
|
||||
6. **Resume** collection with new chain
|
||||
|
||||
## Security Contacts
|
||||
|
||||
Report security issues to: security@stella.ops
|
||||
|
||||
PGP Key: [keys.stella.ops/security.asc](https://keys.stella.ops/security.asc)
|
||||
Reference in New Issue
Block a user