319 lines
7.0 KiB
Markdown
319 lines
7.0 KiB
Markdown
# Reproducible Build Environment Requirements
|
|
|
|
**Sprint:** SPRINT_1227_0002_0001_LB_reproducible_builders
|
|
**Task:** T12 — Document build environment requirements
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
This document describes the environment requirements for running reproducible distro package builds. The build system supports Alpine, Debian, and RHEL package ecosystems.
|
|
|
|
---
|
|
|
|
## Hardware Requirements
|
|
|
|
### Minimum Requirements
|
|
|
|
| Resource | Minimum | Recommended |
|
|
|----------|---------|-------------|
|
|
| CPU | 4 cores | 8+ cores |
|
|
| RAM | 8 GB | 16+ GB |
|
|
| Disk | 50 GB SSD | 200+ GB NVMe |
|
|
| Network | 10 Mbps | 100+ Mbps |
|
|
|
|
### Storage Breakdown
|
|
|
|
| Directory | Purpose | Estimated Size |
|
|
|-----------|---------|----------------|
|
|
| `/var/lib/docker` | Docker images and containers | 30 GB |
|
|
| `/var/cache/stellaops/builds` | Build cache | 50 GB |
|
|
| `/var/cache/stellaops/sources` | Source package cache | 20 GB |
|
|
| `/var/cache/stellaops/artifacts` | Output artifacts | 50 GB |
|
|
|
|
---
|
|
|
|
## Software Requirements
|
|
|
|
### Host System
|
|
|
|
| Component | Version | Purpose |
|
|
|-----------|---------|---------|
|
|
| Docker | 24.0+ | Container runtime |
|
|
| Docker Compose | 2.20+ | Multi-container orchestration |
|
|
| .NET SDK | 10.0 | Worker service runtime |
|
|
| objdump | binutils 2.40+ | Binary analysis |
|
|
| readelf | binutils 2.40+ | ELF parsing |
|
|
|
|
### Container Images
|
|
|
|
The build system uses the following base images:
|
|
|
|
| Builder | Base Image | Tag |
|
|
|---------|------------|-----|
|
|
| Alpine | `alpine` | `3.19`, `3.18` |
|
|
| Debian | `debian` | `bookworm`, `bullseye` |
|
|
| RHEL | `almalinux` | `9`, `8` |
|
|
|
|
---
|
|
|
|
## Environment Variables
|
|
|
|
### Required Variables
|
|
|
|
```bash
|
|
# Build configuration
|
|
export STELLAOPS_BUILD_CACHE=/var/cache/stellaops/builds
|
|
export STELLAOPS_SOURCE_CACHE=/var/cache/stellaops/sources
|
|
export STELLAOPS_ARTIFACT_DIR=/var/cache/stellaops/artifacts
|
|
|
|
# Reproducibility settings
|
|
export TZ=UTC
|
|
export LC_ALL=C.UTF-8
|
|
export SOURCE_DATE_EPOCH=$(date +%s)
|
|
|
|
# Docker settings
|
|
export DOCKER_BUILDKIT=1
|
|
export COMPOSE_DOCKER_CLI_BUILD=1
|
|
```
|
|
|
|
### Optional Variables
|
|
|
|
```bash
|
|
# Parallel build settings
|
|
export STELLAOPS_MAX_CONCURRENT_BUILDS=2
|
|
export STELLAOPS_BUILD_TIMEOUT=1800 # 30 minutes
|
|
|
|
# Proxy settings (if behind corporate firewall)
|
|
export HTTP_PROXY=http://proxy:8080
|
|
export HTTPS_PROXY=http://proxy:8080
|
|
export NO_PROXY=localhost,127.0.0.1
|
|
```
|
|
|
|
---
|
|
|
|
## Builder-Specific Requirements
|
|
|
|
### Alpine Builder
|
|
|
|
```dockerfile
|
|
# Required packages in builder image
|
|
apk add --no-cache \
|
|
alpine-sdk \
|
|
abuild \
|
|
sudo \
|
|
binutils \
|
|
elfutils \
|
|
build-base
|
|
```
|
|
|
|
**Normalization requirements:**
|
|
- `SOURCE_DATE_EPOCH` must be set
|
|
- Use `abuild -r` with reproducible flags
|
|
- Archive ordering: `--sort=name`
|
|
|
|
### Debian Builder
|
|
|
|
```dockerfile
|
|
# Required packages in builder image
|
|
apt-get install -y \
|
|
build-essential \
|
|
devscripts \
|
|
dpkg-dev \
|
|
fakeroot \
|
|
binutils \
|
|
elfutils \
|
|
debhelper
|
|
```
|
|
|
|
**Normalization requirements:**
|
|
- Use `dpkg-buildpackage -b` with reproducible flags
|
|
- Set `DEB_BUILD_OPTIONS=reproducible`
|
|
- Apply `dh_strip_nondeterminism` post-build
|
|
|
|
### RHEL Builder
|
|
|
|
```dockerfile
|
|
# Required packages in builder image (AlmaLinux 9)
|
|
dnf install -y \
|
|
mock \
|
|
rpm-build \
|
|
rpmdevtools \
|
|
binutils \
|
|
elfutils
|
|
```
|
|
|
|
**Normalization requirements:**
|
|
- Use mock with `--enable-network=false`
|
|
- Configure mock for deterministic builds
|
|
- Set `%_buildhost stellaops.build`
|
|
|
|
---
|
|
|
|
## Compiler Flags for Reproducibility
|
|
|
|
### C/C++ Flags
|
|
|
|
```bash
|
|
CFLAGS="-fno-record-gcc-switches -fdebug-prefix-map=$(pwd)=/build -grecord-gcc-switches=off"
|
|
CXXFLAGS="${CFLAGS}"
|
|
LDFLAGS="-Wl,--build-id=sha1"
|
|
```
|
|
|
|
### Additional Flags
|
|
|
|
```bash
|
|
# Disable date/time macros
|
|
-Wdate-time -Werror=date-time
|
|
|
|
# Normalize paths
|
|
-fmacro-prefix-map=$(pwd)=/build
|
|
-ffile-prefix-map=$(pwd)=/build
|
|
```
|
|
|
|
---
|
|
|
|
## Archive Determinism
|
|
|
|
### ar (Static Libraries)
|
|
|
|
```bash
|
|
# Use deterministic mode
|
|
ar --enable-deterministic-archives crs libfoo.a *.o
|
|
|
|
# Or set environment variable
|
|
export AR_FLAGS=--enable-deterministic-archives
|
|
```
|
|
|
|
### tar (Package Archives)
|
|
|
|
```bash
|
|
# Deterministic tar creation
|
|
tar --sort=name \
|
|
--mtime="@${SOURCE_DATE_EPOCH}" \
|
|
--owner=0 \
|
|
--group=0 \
|
|
--numeric-owner \
|
|
-cf archive.tar directory/
|
|
```
|
|
|
|
### zip/gzip
|
|
|
|
```bash
|
|
# Use gzip -n to avoid timestamp
|
|
gzip -n file
|
|
|
|
# Use mtime for consistent timestamps
|
|
touch -d "@${SOURCE_DATE_EPOCH}" file
|
|
```
|
|
|
|
---
|
|
|
|
## Network Requirements
|
|
|
|
### Outbound Access Required
|
|
|
|
| Destination | Port | Purpose |
|
|
|-------------|------|---------|
|
|
| `dl-cdn.alpinelinux.org` | 443 | Alpine packages |
|
|
| `deb.debian.org` | 443 | Debian packages |
|
|
| `vault.centos.org` | 443 | CentOS/RHEL sources |
|
|
| `mirror.almalinux.org` | 443 | AlmaLinux packages |
|
|
| `git.*.org` | 443 | Upstream source repos |
|
|
|
|
### Air-Gapped Operation
|
|
|
|
For air-gapped environments:
|
|
|
|
1. Pre-download source packages
|
|
2. Configure local mirrors
|
|
3. Set `STELLAOPS_OFFLINE_MODE=true`
|
|
4. Use cached build artifacts
|
|
|
|
---
|
|
|
|
## Security Considerations
|
|
|
|
### Container Isolation
|
|
|
|
- Builders run in unprivileged containers
|
|
- No host network access
|
|
- Read-only source mounts
|
|
- Ephemeral containers (destroyed after build)
|
|
|
|
### Signing Keys
|
|
|
|
- Build outputs are unsigned by default
|
|
- DSSE signing requires configured key material
|
|
- Keys stored in `/etc/stellaops/keys/` or HSM
|
|
|
|
### Build Verification
|
|
|
|
```bash
|
|
# Verify reproducibility
|
|
sha256sum build1/output/* > checksums1.txt
|
|
sha256sum build2/output/* > checksums2.txt
|
|
diff checksums1.txt checksums2.txt
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
| Issue | Cause | Resolution |
|
|
|-------|-------|------------|
|
|
| Build timestamp differs | `SOURCE_DATE_EPOCH` not set | Export variable before build |
|
|
| Path in debug info | Missing `-fdebug-prefix-map` | Add to CFLAGS |
|
|
| ar archive differs | Deterministic mode disabled | Use `--enable-deterministic-archives` |
|
|
| tar ordering differs | Random file order | Use `--sort=name` |
|
|
|
|
### Debugging Reproducibility
|
|
|
|
```bash
|
|
# Compare two builds byte-by-byte
|
|
diffoscope build1/output/libfoo.so build2/output/libfoo.so
|
|
|
|
# Check for timestamp differences
|
|
objdump -t binary | grep -i time
|
|
|
|
# Verify no random UUIDs
|
|
strings binary | grep -E '[0-9a-f]{8}-[0-9a-f]{4}'
|
|
```
|
|
|
|
---
|
|
|
|
## Monitoring and Metrics
|
|
|
|
### Key Metrics
|
|
|
|
| Metric | Description | Target |
|
|
|--------|-------------|--------|
|
|
| `build_reproducibility_rate` | % of reproducible builds | > 95% |
|
|
| `build_duration_seconds` | Time to complete build | < 1800 |
|
|
| `fingerprint_extraction_rate` | Functions per second | > 1000 |
|
|
| `build_cache_hit_rate` | Cache effectiveness | > 80% |
|
|
|
|
### Health Checks
|
|
|
|
```bash
|
|
# Verify builder containers are ready
|
|
docker ps --filter "name=repro-builder"
|
|
|
|
# Check cache disk usage
|
|
df -h /var/cache/stellaops/
|
|
|
|
# Verify build queue
|
|
curl -s http://localhost:9090/metrics | grep stellaops_build
|
|
```
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- [Reproducible Builds](https://reproducible-builds.org/)
|
|
- [Debian Reproducible Builds](https://wiki.debian.org/ReproducibleBuilds)
|
|
- [Alpine Reproducibility](https://wiki.alpinelinux.org/wiki/Reproducible_Builds)
|
|
- [RPM Reproducibility](https://rpm-software-management.github.io/rpm/manual/reproducibility.html)
|