save progress
This commit is contained in:
StellaOps Bot
736
.gitea/docs/scripts.md
Normal file
736
.gitea/docs/scripts.md
Normal file
@@ -0,0 +1,736 @@
|
||||
# CI/CD Scripts Inventory
|
||||
|
||||
Complete documentation of all scripts in `.gitea/scripts/`.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
.gitea/scripts/
|
||||
├── build/ # Build orchestration
|
||||
├── evidence/ # Evidence bundle management
|
||||
├── metrics/ # Performance metrics
|
||||
├── release/ # Release automation
|
||||
├── sign/ # Artifact signing
|
||||
├── test/ # Test execution
|
||||
├── util/ # Utilities
|
||||
└── validate/ # Validation scripts
|
||||
```
|
||||
|
||||
## Exit Code Conventions
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| 0 | Success |
|
||||
| 1 | General error |
|
||||
| 2 | Missing configuration/key |
|
||||
| 3 | Missing required file |
|
||||
| 69 | Tool not found (EX_UNAVAILABLE) |
|
||||
|
||||
---
|
||||
|
||||
## Build Scripts (`scripts/build/`)
|
||||
|
||||
### build-cli.sh
|
||||
|
||||
Multi-platform CLI build with SBOM generation and signing.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
RIDS=linux-x64,win-x64,osx-arm64 ./build-cli.sh
|
||||
```
|
||||
|
||||
**Environment Variables:**
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `RIDS` | `linux-x64,win-x64,osx-arm64` | Comma-separated runtime identifiers |
|
||||
| `CONFIG` | `Release` | Build configuration |
|
||||
| `SBOM_TOOL` | `syft` | SBOM generator (`syft` or `none`) |
|
||||
| `SIGN` | `false` | Enable artifact signing |
|
||||
| `COSIGN_KEY` | - | Path to Cosign key file |
|
||||
|
||||
**Output:**
|
||||
```
|
||||
out/cli/
|
||||
├── linux-x64/
|
||||
│ ├── publish/
|
||||
│ ├── stella-cli-linux-x64.tar.gz
|
||||
│ ├── stella-cli-linux-x64.tar.gz.sha256
|
||||
│ └── stella-cli-linux-x64.tar.gz.sbom.json
|
||||
├── win-x64/
|
||||
│ ├── publish/
|
||||
│ ├── stella-cli-win-x64.zip
|
||||
│ └── ...
|
||||
└── manifest.json
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Builds self-contained single-file executables
|
||||
- Includes CLI plugins (Aoc, Symbols)
|
||||
- Generates SHA-256 checksums
|
||||
- Optional SBOM generation via Syft
|
||||
- Optional Cosign signing
|
||||
|
||||
---
|
||||
|
||||
### build-multiarch.sh
|
||||
|
||||
Multi-architecture Docker image builds using buildx.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
IMAGE=scanner PLATFORMS=linux/amd64,linux/arm64 ./build-multiarch.sh
|
||||
```
|
||||
|
||||
**Environment Variables:**
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `IMAGE` | - | Image name (required) |
|
||||
| `PLATFORMS` | `linux/amd64,linux/arm64` | Target platforms |
|
||||
| `REGISTRY` | `git.stella-ops.org` | Container registry |
|
||||
| `TAG` | `latest` | Image tag |
|
||||
| `PUSH` | `false` | Push to registry |
|
||||
|
||||
---
|
||||
|
||||
### build-airgap-bundle.sh
|
||||
|
||||
Build offline/air-gapped deployment bundle.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
VERSION=2026.04 ./build-airgap-bundle.sh
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
out/airgap/
|
||||
├── images.tar # All container images
|
||||
├── helm-charts.tar.gz # Helm charts
|
||||
├── compose.tar.gz # Docker Compose files
|
||||
├── checksums.txt
|
||||
└── manifest.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Scripts (`scripts/test/`)
|
||||
|
||||
### determinism-run.sh
|
||||
|
||||
Run determinism verification tests.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./determinism-run.sh
|
||||
```
|
||||
|
||||
**Purpose:**
|
||||
- Executes tests filtered by `Determinism` category
|
||||
- Collects TRX test results
|
||||
- Generates summary and artifacts archive
|
||||
|
||||
**Output:**
|
||||
```
|
||||
out/scanner-determinism/
|
||||
├── determinism.trx
|
||||
├── summary.txt
|
||||
└── determinism-artifacts.tgz
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### run-fixtures-check.sh
|
||||
|
||||
Validate test fixtures against expected schemas.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./run-fixtures-check.sh [--update]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
- `--update`: Update golden fixtures if mismatched
|
||||
|
||||
---
|
||||
|
||||
## Validation Scripts (`scripts/validate/`)
|
||||
|
||||
### validate-sbom.sh
|
||||
|
||||
Validate CycloneDX SBOM files.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./validate-sbom.sh <sbom-file>
|
||||
./validate-sbom.sh --all
|
||||
./validate-sbom.sh --schema custom.json sample.json
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--all` | Validate all fixtures in `src/__Tests/__Benchmarks/golden-corpus/` |
|
||||
| `--schema <path>` | Custom schema file |
|
||||
|
||||
**Dependencies:**
|
||||
- `sbom-utility` (auto-installed if missing)
|
||||
|
||||
**Exit Codes:**
|
||||
- `0`: All validations passed
|
||||
- `1`: Validation failed
|
||||
|
||||
---
|
||||
|
||||
### validate-spdx.sh
|
||||
|
||||
Validate SPDX SBOM files.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./validate-spdx.sh <spdx-file>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### validate-vex.sh
|
||||
|
||||
Validate VEX documents (OpenVEX, CSAF).
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./validate-vex.sh <vex-file>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### validate-helm.sh
|
||||
|
||||
Validate Helm charts.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./validate-helm.sh [chart-path]
|
||||
```
|
||||
|
||||
**Default Path:** `devops/helm/stellaops`
|
||||
|
||||
**Checks:**
|
||||
- `helm lint`
|
||||
- Template rendering
|
||||
- Schema validation
|
||||
|
||||
---
|
||||
|
||||
### validate-compose.sh
|
||||
|
||||
Validate Docker Compose files.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./validate-compose.sh [profile]
|
||||
```
|
||||
|
||||
**Profiles:**
|
||||
- `dev` - Development
|
||||
- `stage` - Staging
|
||||
- `prod` - Production
|
||||
- `airgap` - Air-gapped
|
||||
|
||||
---
|
||||
|
||||
### validate-licenses.sh
|
||||
|
||||
Check dependency licenses for compliance.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./validate-licenses.sh
|
||||
```
|
||||
|
||||
**Checks:**
|
||||
- NuGet packages via `dotnet-delice`
|
||||
- npm packages via `license-checker`
|
||||
- Reports blocked licenses (GPL-2.0-only, SSPL, etc.)
|
||||
|
||||
---
|
||||
|
||||
### validate-migrations.sh
|
||||
|
||||
Validate database migrations.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./validate-migrations.sh
|
||||
```
|
||||
|
||||
**Checks:**
|
||||
- Migration naming conventions
|
||||
- Forward/rollback pairs
|
||||
- Idempotency
|
||||
|
||||
---
|
||||
|
||||
### validate-workflows.sh
|
||||
|
||||
Validate Gitea Actions workflow YAML files.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./validate-workflows.sh
|
||||
```
|
||||
|
||||
**Checks:**
|
||||
- YAML syntax
|
||||
- Required fields
|
||||
- Action version pinning
|
||||
|
||||
---
|
||||
|
||||
### verify-binaries.sh
|
||||
|
||||
Verify binary integrity.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./verify-binaries.sh <binary-path> [checksum-file]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Signing Scripts (`scripts/sign/`)
|
||||
|
||||
### sign-signals.sh
|
||||
|
||||
Sign Signals artifacts with Cosign.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./sign-signals.sh
|
||||
```
|
||||
|
||||
**Environment Variables:**
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `COSIGN_KEY_FILE` | Path to signing key |
|
||||
| `COSIGN_PRIVATE_KEY_B64` | Base64-encoded private key |
|
||||
| `COSIGN_PASSWORD` | Key password |
|
||||
| `COSIGN_ALLOW_DEV_KEY` | Allow development key (`1`) |
|
||||
| `OUT_DIR` | Output directory |
|
||||
|
||||
**Key Resolution Order:**
|
||||
1. `COSIGN_KEY_FILE` environment variable
|
||||
2. `COSIGN_PRIVATE_KEY_B64` environment variable (decoded)
|
||||
3. `tools/cosign/cosign.key`
|
||||
4. `tools/cosign/cosign.dev.key` (if `COSIGN_ALLOW_DEV_KEY=1`)
|
||||
|
||||
**Signed Artifacts:**
|
||||
- `confidence_decay_config.yaml`
|
||||
- `unknowns_scoring_manifest.json`
|
||||
- `heuristics.catalog.json`
|
||||
|
||||
**Output:**
|
||||
```
|
||||
evidence-locker/signals/{date}/
|
||||
├── confidence_decay_config.sigstore.json
|
||||
├── unknowns_scoring_manifest.sigstore.json
|
||||
├── heuristics_catalog.sigstore.json
|
||||
└── SHA256SUMS
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### sign-policy.sh
|
||||
|
||||
Sign policy artifacts.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./sign-policy.sh <policy-file>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### sign-authority-gaps.sh
|
||||
|
||||
Sign authority gap attestations.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./sign-authority-gaps.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Release Scripts (`scripts/release/`)
|
||||
|
||||
### build_release.py
|
||||
|
||||
Main release pipeline orchestration.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python build_release.py --channel stable --version 2026.04
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
|
||||
| Argument | Description |
|
||||
|----------|-------------|
|
||||
| `--channel` | Release channel (`stable`, `beta`, `nightly`) |
|
||||
| `--version` | Version string |
|
||||
| `--config` | Component config file |
|
||||
| `--dry-run` | Don't push artifacts |
|
||||
|
||||
**Dependencies:**
|
||||
- docker (with buildx)
|
||||
- cosign
|
||||
- helm
|
||||
- npm/node
|
||||
- dotnet SDK
|
||||
|
||||
---
|
||||
|
||||
### verify_release.py
|
||||
|
||||
Post-release verification.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python verify_release.py --version 2026.04
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### bump-service-version.py
|
||||
|
||||
Manage service versions in `Directory.Versions.props`.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
# Bump version
|
||||
python bump-service-version.py --service scanner --bump minor
|
||||
|
||||
# Set explicit version
|
||||
python bump-service-version.py --service scanner --version 2.0.0
|
||||
|
||||
# List versions
|
||||
python bump-service-version.py --list
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
|
||||
| Argument | Description |
|
||||
|----------|-------------|
|
||||
| `--service` | Service name (e.g., `scanner`, `authority`) |
|
||||
| `--bump` | Bump type (`major`, `minor`, `patch`) |
|
||||
| `--version` | Explicit version to set |
|
||||
| `--list` | List all service versions |
|
||||
| `--dry-run` | Don't write changes |
|
||||
|
||||
---
|
||||
|
||||
### read-service-version.sh
|
||||
|
||||
Read current service version.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./read-service-version.sh scanner
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
1.2.3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### generate-docker-tag.sh
|
||||
|
||||
Generate Docker tag with datetime suffix.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./generate-docker-tag.sh 1.2.3
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
1.2.3+20250128143022
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### generate_changelog.py
|
||||
|
||||
AI-assisted changelog generation.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python generate_changelog.py --version 2026.04 --codename Nova
|
||||
```
|
||||
|
||||
**Environment Variables:**
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `AI_API_KEY` | AI service API key |
|
||||
| `AI_API_URL` | AI service endpoint (optional) |
|
||||
|
||||
**Features:**
|
||||
- Parses git commits since last release
|
||||
- Categorizes by type (Breaking, Security, Features, Fixes)
|
||||
- Groups by module
|
||||
- AI-assisted summary generation
|
||||
- Fallback to rule-based generation
|
||||
|
||||
---
|
||||
|
||||
### generate_suite_docs.py
|
||||
|
||||
Generate suite release documentation.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python generate_suite_docs.py --version 2026.04 --codename Nova
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
docs/releases/2026.04/
|
||||
├── README.md
|
||||
├── CHANGELOG.md
|
||||
├── services.md
|
||||
├── upgrade-guide.md
|
||||
├── checksums.txt
|
||||
└── manifest.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### generate_compose.py
|
||||
|
||||
Generate pinned Docker Compose files.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python generate_compose.py --version 2026.04
|
||||
```
|
||||
|
||||
**Output:**
|
||||
- `docker-compose.yml` - Standard deployment
|
||||
- `docker-compose.airgap.yml` - Air-gapped deployment
|
||||
|
||||
---
|
||||
|
||||
### collect_versions.py
|
||||
|
||||
Collect service versions from `Directory.Versions.props`.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python collect_versions.py --format json
|
||||
python collect_versions.py --format yaml
|
||||
python collect_versions.py --format markdown
|
||||
python collect_versions.py --format env
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### check_cli_parity.py
|
||||
|
||||
Verify CLI version parity across platforms.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python check_cli_parity.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Evidence Scripts (`scripts/evidence/`)
|
||||
|
||||
### upload-all-evidence.sh
|
||||
|
||||
Upload all evidence bundles to Evidence Locker.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./upload-all-evidence.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### signals-upload-evidence.sh
|
||||
|
||||
Upload Signals evidence.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./signals-upload-evidence.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### zastava-upload-evidence.sh
|
||||
|
||||
Upload Zastava evidence.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./zastava-upload-evidence.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Metrics Scripts (`scripts/metrics/`)
|
||||
|
||||
### compute-reachability-metrics.sh
|
||||
|
||||
Compute reachability analysis metrics.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./compute-reachability-metrics.sh
|
||||
```
|
||||
|
||||
**Output Metrics:**
|
||||
- Total functions analyzed
|
||||
- Reachable functions
|
||||
- Coverage percentage
|
||||
- Analysis duration
|
||||
|
||||
---
|
||||
|
||||
### compute-ttfs-metrics.sh
|
||||
|
||||
Compute Time-to-First-Scan metrics.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./compute-ttfs-metrics.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### enforce-performance-slos.sh
|
||||
|
||||
Enforce performance SLOs.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./enforce-performance-slos.sh
|
||||
```
|
||||
|
||||
**Checked SLOs:**
|
||||
- Build time < 30 minutes
|
||||
- Test coverage > 80%
|
||||
- TTFS < 60 seconds
|
||||
|
||||
---
|
||||
|
||||
## Utility Scripts (`scripts/util/`)
|
||||
|
||||
### cleanup-runner-space.sh
|
||||
|
||||
Clean up runner disk space.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./cleanup-runner-space.sh
|
||||
```
|
||||
|
||||
**Actions:**
|
||||
- Remove Docker build cache
|
||||
- Clean NuGet cache
|
||||
- Remove old test results
|
||||
- Prune unused images
|
||||
|
||||
---
|
||||
|
||||
### dotnet-filter.sh
|
||||
|
||||
Filter .NET projects for selective builds.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./dotnet-filter.sh --changed
|
||||
./dotnet-filter.sh --module Scanner
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### enable-openssl11-shim.sh
|
||||
|
||||
Enable OpenSSL 1.1 compatibility shim.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./enable-openssl11-shim.sh
|
||||
```
|
||||
|
||||
**Purpose:**
|
||||
Required for certain cryptographic operations on newer Linux distributions that have removed OpenSSL 1.1.
|
||||
|
||||
---
|
||||
|
||||
## Script Development Guidelines
|
||||
|
||||
### Required Elements
|
||||
|
||||
1. **Shebang:**
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
```
|
||||
|
||||
2. **Strict Mode:**
|
||||
```bash
|
||||
set -euo pipefail
|
||||
```
|
||||
|
||||
3. **Sprint Reference:**
|
||||
```bash
|
||||
# DEVOPS-XXX-YY-ZZZ: Description
|
||||
# Sprint: SPRINT_XXXX_XXXX_XXXX - Topic
|
||||
```
|
||||
|
||||
4. **Usage Documentation:**
|
||||
```bash
|
||||
# Usage:
|
||||
# ./script.sh <required-arg> [optional-arg]
|
||||
```
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Use environment variables with defaults:**
|
||||
```bash
|
||||
CONFIG="${CONFIG:-Release}"
|
||||
```
|
||||
|
||||
2. **Validate required tools:**
|
||||
```bash
|
||||
if ! command -v dotnet >/dev/null 2>&1; then
|
||||
echo "dotnet CLI not found" >&2
|
||||
exit 69
|
||||
fi
|
||||
```
|
||||
|
||||
3. **Use absolute paths:**
|
||||
```bash
|
||||
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
```
|
||||
|
||||
4. **Handle cleanup:**
|
||||
```bash
|
||||
trap 'rm -f "$TMP_FILE"' EXIT
|
||||
```
|
||||
|
||||
5. **Use logging functions:**
|
||||
```bash
|
||||
log_info() { echo "[INFO] $*"; }
|
||||
log_error() { echo "[ERROR] $*" >&2; }
|
||||
```
|
||||
Reference in New Issue
Block a user