save progress
This commit is contained in:
StellaOps Bot
318
devops/docker/repro-builders/BUILD_ENVIRONMENT.md
Normal file
318
devops/docker/repro-builders/BUILD_ENVIRONMENT.md
Normal file
@@ -0,0 +1,318 @@
|
||||
# Reproducible Build Environment Requirements
|
||||
|
||||
**Sprint:** SPRINT_1227_0002_0001_LB_reproducible_builders
|
||||
**Task:** T12 — Document build environment requirements
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the environment requirements for running reproducible distro package builds. The build system supports Alpine, Debian, and RHEL package ecosystems.
|
||||
|
||||
---
|
||||
|
||||
## Hardware Requirements
|
||||
|
||||
### Minimum Requirements
|
||||
|
||||
| Resource | Minimum | Recommended |
|
||||
|----------|---------|-------------|
|
||||
| CPU | 4 cores | 8+ cores |
|
||||
| RAM | 8 GB | 16+ GB |
|
||||
| Disk | 50 GB SSD | 200+ GB NVMe |
|
||||
| Network | 10 Mbps | 100+ Mbps |
|
||||
|
||||
### Storage Breakdown
|
||||
|
||||
| Directory | Purpose | Estimated Size |
|
||||
|-----------|---------|----------------|
|
||||
| `/var/lib/docker` | Docker images and containers | 30 GB |
|
||||
| `/var/cache/stellaops/builds` | Build cache | 50 GB |
|
||||
| `/var/cache/stellaops/sources` | Source package cache | 20 GB |
|
||||
| `/var/cache/stellaops/artifacts` | Output artifacts | 50 GB |
|
||||
|
||||
---
|
||||
|
||||
## Software Requirements
|
||||
|
||||
### Host System
|
||||
|
||||
| Component | Version | Purpose |
|
||||
|-----------|---------|---------|
|
||||
| Docker | 24.0+ | Container runtime |
|
||||
| Docker Compose | 2.20+ | Multi-container orchestration |
|
||||
| .NET SDK | 10.0 | Worker service runtime |
|
||||
| objdump | binutils 2.40+ | Binary analysis |
|
||||
| readelf | binutils 2.40+ | ELF parsing |
|
||||
|
||||
### Container Images
|
||||
|
||||
The build system uses the following base images:
|
||||
|
||||
| Builder | Base Image | Tag |
|
||||
|---------|------------|-----|
|
||||
| Alpine | `alpine` | `3.19`, `3.18` |
|
||||
| Debian | `debian` | `bookworm`, `bullseye` |
|
||||
| RHEL | `almalinux` | `9`, `8` |
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
### Required Variables
|
||||
|
||||
```bash
|
||||
# Build configuration
|
||||
export STELLAOPS_BUILD_CACHE=/var/cache/stellaops/builds
|
||||
export STELLAOPS_SOURCE_CACHE=/var/cache/stellaops/sources
|
||||
export STELLAOPS_ARTIFACT_DIR=/var/cache/stellaops/artifacts
|
||||
|
||||
# Reproducibility settings
|
||||
export TZ=UTC
|
||||
export LC_ALL=C.UTF-8
|
||||
export SOURCE_DATE_EPOCH=$(date +%s)
|
||||
|
||||
# Docker settings
|
||||
export DOCKER_BUILDKIT=1
|
||||
export COMPOSE_DOCKER_CLI_BUILD=1
|
||||
```
|
||||
|
||||
### Optional Variables
|
||||
|
||||
```bash
|
||||
# Parallel build settings
|
||||
export STELLAOPS_MAX_CONCURRENT_BUILDS=2
|
||||
export STELLAOPS_BUILD_TIMEOUT=1800 # 30 minutes
|
||||
|
||||
# Proxy settings (if behind corporate firewall)
|
||||
export HTTP_PROXY=http://proxy:8080
|
||||
export HTTPS_PROXY=http://proxy:8080
|
||||
export NO_PROXY=localhost,127.0.0.1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Builder-Specific Requirements
|
||||
|
||||
### Alpine Builder
|
||||
|
||||
```dockerfile
|
||||
# Required packages in builder image
|
||||
apk add --no-cache \
|
||||
alpine-sdk \
|
||||
abuild \
|
||||
sudo \
|
||||
binutils \
|
||||
elfutils \
|
||||
build-base
|
||||
```
|
||||
|
||||
**Normalization requirements:**
|
||||
- `SOURCE_DATE_EPOCH` must be set
|
||||
- Use `abuild -r` with reproducible flags
|
||||
- Archive ordering: `--sort=name`
|
||||
|
||||
### Debian Builder
|
||||
|
||||
```dockerfile
|
||||
# Required packages in builder image
|
||||
apt-get install -y \
|
||||
build-essential \
|
||||
devscripts \
|
||||
dpkg-dev \
|
||||
fakeroot \
|
||||
binutils \
|
||||
elfutils \
|
||||
debhelper
|
||||
```
|
||||
|
||||
**Normalization requirements:**
|
||||
- Use `dpkg-buildpackage -b` with reproducible flags
|
||||
- Set `DEB_BUILD_OPTIONS=reproducible`
|
||||
- Apply `dh_strip_nondeterminism` post-build
|
||||
|
||||
### RHEL Builder
|
||||
|
||||
```dockerfile
|
||||
# Required packages in builder image (AlmaLinux 9)
|
||||
dnf install -y \
|
||||
mock \
|
||||
rpm-build \
|
||||
rpmdevtools \
|
||||
binutils \
|
||||
elfutils
|
||||
```
|
||||
|
||||
**Normalization requirements:**
|
||||
- Use mock with `--enable-network=false`
|
||||
- Configure mock for deterministic builds
|
||||
- Set `%_buildhost stellaops.build`
|
||||
|
||||
---
|
||||
|
||||
## Compiler Flags for Reproducibility
|
||||
|
||||
### C/C++ Flags
|
||||
|
||||
```bash
|
||||
CFLAGS="-fno-record-gcc-switches -fdebug-prefix-map=$(pwd)=/build -grecord-gcc-switches=off"
|
||||
CXXFLAGS="${CFLAGS}"
|
||||
LDFLAGS="-Wl,--build-id=sha1"
|
||||
```
|
||||
|
||||
### Additional Flags
|
||||
|
||||
```bash
|
||||
# Disable date/time macros
|
||||
-Wdate-time -Werror=date-time
|
||||
|
||||
# Normalize paths
|
||||
-fmacro-prefix-map=$(pwd)=/build
|
||||
-ffile-prefix-map=$(pwd)=/build
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Archive Determinism
|
||||
|
||||
### ar (Static Libraries)
|
||||
|
||||
```bash
|
||||
# Use deterministic mode
|
||||
ar --enable-deterministic-archives crs libfoo.a *.o
|
||||
|
||||
# Or set environment variable
|
||||
export AR_FLAGS=--enable-deterministic-archives
|
||||
```
|
||||
|
||||
### tar (Package Archives)
|
||||
|
||||
```bash
|
||||
# Deterministic tar creation
|
||||
tar --sort=name \
|
||||
--mtime="@${SOURCE_DATE_EPOCH}" \
|
||||
--owner=0 \
|
||||
--group=0 \
|
||||
--numeric-owner \
|
||||
-cf archive.tar directory/
|
||||
```
|
||||
|
||||
### zip/gzip
|
||||
|
||||
```bash
|
||||
# Use gzip -n to avoid timestamp
|
||||
gzip -n file
|
||||
|
||||
# Use mtime for consistent timestamps
|
||||
touch -d "@${SOURCE_DATE_EPOCH}" file
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Network Requirements
|
||||
|
||||
### Outbound Access Required
|
||||
|
||||
| Destination | Port | Purpose |
|
||||
|-------------|------|---------|
|
||||
| `dl-cdn.alpinelinux.org` | 443 | Alpine packages |
|
||||
| `deb.debian.org` | 443 | Debian packages |
|
||||
| `vault.centos.org` | 443 | CentOS/RHEL sources |
|
||||
| `mirror.almalinux.org` | 443 | AlmaLinux packages |
|
||||
| `git.*.org` | 443 | Upstream source repos |
|
||||
|
||||
### Air-Gapped Operation
|
||||
|
||||
For air-gapped environments:
|
||||
|
||||
1. Pre-download source packages
|
||||
2. Configure local mirrors
|
||||
3. Set `STELLAOPS_OFFLINE_MODE=true`
|
||||
4. Use cached build artifacts
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Container Isolation
|
||||
|
||||
- Builders run in unprivileged containers
|
||||
- No host network access
|
||||
- Read-only source mounts
|
||||
- Ephemeral containers (destroyed after build)
|
||||
|
||||
### Signing Keys
|
||||
|
||||
- Build outputs are unsigned by default
|
||||
- DSSE signing requires configured key material
|
||||
- Keys stored in `/etc/stellaops/keys/` or HSM
|
||||
|
||||
### Build Verification
|
||||
|
||||
```bash
|
||||
# Verify reproducibility
|
||||
sha256sum build1/output/* > checksums1.txt
|
||||
sha256sum build2/output/* > checksums2.txt
|
||||
diff checksums1.txt checksums2.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
| Issue | Cause | Resolution |
|
||||
|-------|-------|------------|
|
||||
| Build timestamp differs | `SOURCE_DATE_EPOCH` not set | Export variable before build |
|
||||
| Path in debug info | Missing `-fdebug-prefix-map` | Add to CFLAGS |
|
||||
| ar archive differs | Deterministic mode disabled | Use `--enable-deterministic-archives` |
|
||||
| tar ordering differs | Random file order | Use `--sort=name` |
|
||||
|
||||
### Debugging Reproducibility
|
||||
|
||||
```bash
|
||||
# Compare two builds byte-by-byte
|
||||
diffoscope build1/output/libfoo.so build2/output/libfoo.so
|
||||
|
||||
# Check for timestamp differences
|
||||
objdump -t binary | grep -i time
|
||||
|
||||
# Verify no random UUIDs
|
||||
strings binary | grep -E '[0-9a-f]{8}-[0-9a-f]{4}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring and Metrics
|
||||
|
||||
### Key Metrics
|
||||
|
||||
| Metric | Description | Target |
|
||||
|--------|-------------|--------|
|
||||
| `build_reproducibility_rate` | % of reproducible builds | > 95% |
|
||||
| `build_duration_seconds` | Time to complete build | < 1800 |
|
||||
| `fingerprint_extraction_rate` | Functions per second | > 1000 |
|
||||
| `build_cache_hit_rate` | Cache effectiveness | > 80% |
|
||||
|
||||
### Health Checks
|
||||
|
||||
```bash
|
||||
# Verify builder containers are ready
|
||||
docker ps --filter "name=repro-builder"
|
||||
|
||||
# Check cache disk usage
|
||||
df -h /var/cache/stellaops/
|
||||
|
||||
# Verify build queue
|
||||
curl -s http://localhost:9090/metrics | grep stellaops_build
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [Reproducible Builds](https://reproducible-builds.org/)
|
||||
- [Debian Reproducible Builds](https://wiki.debian.org/ReproducibleBuilds)
|
||||
- [Alpine Reproducibility](https://wiki.alpinelinux.org/wiki/Reproducible_Builds)
|
||||
- [RPM Reproducibility](https://rpm-software-management.github.io/rpm/manual/reproducibility.html)
|
||||
Reference in New Issue
Block a user