save progress

2025-12-28 01:40:35 +02:00
parent 3bfbbae115
commit cec4265a40
694 changed files with 88052 additions and 24718 deletions
--- a/devops/docker/repro-builders/BUILD_ENVIRONMENT.md
+++ b/devops/docker/repro-builders/BUILD_ENVIRONMENT.md
@@ -0,0 +1,318 @@
+# Reproducible Build Environment Requirements
+
+**Sprint:** SPRINT_1227_0002_0001_LB_reproducible_builders
+**Task:** T12 — Document build environment requirements
+
+---
+
+## Overview
+
+This document describes the environment requirements for running reproducible distro package builds. The build system supports Alpine, Debian, and RHEL package ecosystems.
+
+---
+
+## Hardware Requirements
+
+### Minimum Requirements
+
+| Resource | Minimum | Recommended |
+|----------|---------|-------------|
+| CPU | 4 cores | 8+ cores |
+| RAM | 8 GB | 16+ GB |
+| Disk | 50 GB SSD | 200+ GB NVMe |
+| Network | 10 Mbps | 100+ Mbps |
+
+### Storage Breakdown
+
+| Directory | Purpose | Estimated Size |
+|-----------|---------|----------------|
+| `/var/lib/docker` | Docker images and containers | 30 GB |
+| `/var/cache/stellaops/builds` | Build cache | 50 GB |
+| `/var/cache/stellaops/sources` | Source package cache | 20 GB |
+| `/var/cache/stellaops/artifacts` | Output artifacts | 50 GB |
+
+---
+
+## Software Requirements
+
+### Host System
+
+| Component | Version | Purpose |
+|-----------|---------|---------|
+| Docker | 24.0+ | Container runtime |
+| Docker Compose | 2.20+ | Multi-container orchestration |
+| .NET SDK | 10.0 | Worker service runtime |
+| objdump | binutils 2.40+ | Binary analysis |
+| readelf | binutils 2.40+ | ELF parsing |
+
+### Container Images
+
+The build system uses the following base images:
+
+| Builder | Base Image | Tag |
+|---------|------------|-----|
+| Alpine | `alpine` | `3.19`, `3.18` |
+| Debian | `debian` | `bookworm`, `bullseye` |
+| RHEL | `almalinux` | `9`, `8` |
+
+---
+
+## Environment Variables
+
+### Required Variables
+
+```bash
+# Build configuration
+export STELLAOPS_BUILD_CACHE=/var/cache/stellaops/builds
+export STELLAOPS_SOURCE_CACHE=/var/cache/stellaops/sources
+export STELLAOPS_ARTIFACT_DIR=/var/cache/stellaops/artifacts
+
+# Reproducibility settings
+export TZ=UTC
+export LC_ALL=C.UTF-8
+export SOURCE_DATE_EPOCH=$(date +%s)
+
+# Docker settings
+export DOCKER_BUILDKIT=1
+export COMPOSE_DOCKER_CLI_BUILD=1
+```
+
+### Optional Variables
+
+```bash
+# Parallel build settings
+export STELLAOPS_MAX_CONCURRENT_BUILDS=2
+export STELLAOPS_BUILD_TIMEOUT=1800  # 30 minutes
+
+# Proxy settings (if behind corporate firewall)
+export HTTP_PROXY=http://proxy:8080
+export HTTPS_PROXY=http://proxy:8080
+export NO_PROXY=localhost,127.0.0.1
+```
+
+---
+
+## Builder-Specific Requirements
+
+### Alpine Builder
+
+```dockerfile
+# Required packages in builder image
+apk add --no-cache \
+    alpine-sdk \
+    abuild \
+    sudo \
+    binutils \
+    elfutils \
+    build-base
+```
+
+**Normalization requirements:**
+- `SOURCE_DATE_EPOCH` must be set
+- Use `abuild -r` with reproducible flags
+- Archive ordering: `--sort=name`
+
+### Debian Builder
+
+```dockerfile
+# Required packages in builder image
+apt-get install -y \
+    build-essential \
+    devscripts \
+    dpkg-dev \
+    fakeroot \
+    binutils \
+    elfutils \
+    debhelper
+```
+
+**Normalization requirements:**
+- Use `dpkg-buildpackage -b` with reproducible flags
+- Set `DEB_BUILD_OPTIONS=reproducible`
+- Apply `dh_strip_nondeterminism` post-build
+
+### RHEL Builder
+
+```dockerfile
+# Required packages in builder image (AlmaLinux 9)
+dnf install -y \
+    mock \
+    rpm-build \
+    rpmdevtools \
+    binutils \
+    elfutils
+```
+
+**Normalization requirements:**
+- Use mock with `--enable-network=false`
+- Configure mock for deterministic builds
+- Set `%_buildhost stellaops.build`
+
+---
+
+## Compiler Flags for Reproducibility
+
+### C/C++ Flags
+
+```bash
+CFLAGS="-fno-record-gcc-switches -fdebug-prefix-map=$(pwd)=/build -grecord-gcc-switches=off"
+CXXFLAGS="${CFLAGS}"
+LDFLAGS="-Wl,--build-id=sha1"
+```
+
+### Additional Flags
+
+```bash
+# Disable date/time macros
+-Wdate-time -Werror=date-time
+
+# Normalize paths
+-fmacro-prefix-map=$(pwd)=/build
+-ffile-prefix-map=$(pwd)=/build
+```
+
+---
+
+## Archive Determinism
+
+### ar (Static Libraries)
+
+```bash
+# Use deterministic mode
+ar --enable-deterministic-archives crs libfoo.a *.o
+
+# Or set environment variable
+export AR_FLAGS=--enable-deterministic-archives
+```
+
+### tar (Package Archives)
+
+```bash
+# Deterministic tar creation
+tar --sort=name \
+    --mtime="@${SOURCE_DATE_EPOCH}" \
+    --owner=0 \
+    --group=0 \
+    --numeric-owner \
+    -cf archive.tar directory/
+```
+
+### zip/gzip
+
+```bash
+# Use gzip -n to avoid timestamp
+gzip -n file
+
+# Use mtime for consistent timestamps
+touch -d "@${SOURCE_DATE_EPOCH}" file
+```
+
+---
+
+## Network Requirements
+
+### Outbound Access Required
+
+| Destination | Port | Purpose |
+|-------------|------|---------|
+| `dl-cdn.alpinelinux.org` | 443 | Alpine packages |
+| `deb.debian.org` | 443 | Debian packages |
+| `vault.centos.org` | 443 | CentOS/RHEL sources |
+| `mirror.almalinux.org` | 443 | AlmaLinux packages |
+| `git.*.org` | 443 | Upstream source repos |
+
+### Air-Gapped Operation
+
+For air-gapped environments:
+
+1. Pre-download source packages
+2. Configure local mirrors
+3. Set `STELLAOPS_OFFLINE_MODE=true`
+4. Use cached build artifacts
+
+---
+
+## Security Considerations
+
+### Container Isolation
+
+- Builders run in unprivileged containers
+- No host network access
+- Read-only source mounts
+- Ephemeral containers (destroyed after build)
+
+### Signing Keys
+
+- Build outputs are unsigned by default
+- DSSE signing requires configured key material
+- Keys stored in `/etc/stellaops/keys/` or HSM
+
+### Build Verification
+
+```bash
+# Verify reproducibility
+sha256sum build1/output/* > checksums1.txt
+sha256sum build2/output/* > checksums2.txt
+diff checksums1.txt checksums2.txt
+```
+
+---
+
+## Troubleshooting
+
+### Common Issues
+
+| Issue | Cause | Resolution |
+|-------|-------|------------|
+| Build timestamp differs | `SOURCE_DATE_EPOCH` not set | Export variable before build |
+| Path in debug info | Missing `-fdebug-prefix-map` | Add to CFLAGS |
+| ar archive differs | Deterministic mode disabled | Use `--enable-deterministic-archives` |
+| tar ordering differs | Random file order | Use `--sort=name` |
+
+### Debugging Reproducibility
+
+```bash
+# Compare two builds byte-by-byte
+diffoscope build1/output/libfoo.so build2/output/libfoo.so
+
+# Check for timestamp differences
+objdump -t binary | grep -i time
+
+# Verify no random UUIDs
+strings binary | grep -E '[0-9a-f]{8}-[0-9a-f]{4}'
+```
+
+---
+
+## Monitoring and Metrics
+
+### Key Metrics
+
+| Metric | Description | Target |
+|--------|-------------|--------|
+| `build_reproducibility_rate` | % of reproducible builds | > 95% |
+| `build_duration_seconds` | Time to complete build | < 1800 |
+| `fingerprint_extraction_rate` | Functions per second | > 1000 |
+| `build_cache_hit_rate` | Cache effectiveness | > 80% |
+
+### Health Checks
+
+```bash
+# Verify builder containers are ready
+docker ps --filter "name=repro-builder"
+
+# Check cache disk usage
+df -h /var/cache/stellaops/
+
+# Verify build queue
+curl -s http://localhost:9090/metrics | grep stellaops_build
+```
+
+---
+
+## References
+
+- [Reproducible Builds](https://reproducible-builds.org/)
+- [Debian Reproducible Builds](https://wiki.debian.org/ReproducibleBuilds)
+- [Alpine Reproducibility](https://wiki.alpinelinux.org/wiki/Reproducible_Builds)
+- [RPM Reproducibility](https://rpm-software-management.github.io/rpm/manual/reproducibility.html)
--- a/devops/docker/repro-builders/alpine/Dockerfile
+++ b/devops/docker/repro-builders/alpine/Dockerfile
@@ -0,0 +1,62 @@
+# Alpine Reproducible Builder
+# Creates deterministic builds of Alpine packages for fingerprint diffing
+#
+# Usage:
+#   docker build -t repro-builder-alpine:3.20 --build-arg RELEASE=3.20 .
+#   docker run -v ./output:/output repro-builder-alpine:3.20 build openssl 3.0.7-r0
+
+ARG RELEASE=3.20
+FROM alpine:${RELEASE}
+
+ARG RELEASE
+ENV ALPINE_RELEASE=${RELEASE}
+
+# Install build tools and dependencies
+RUN apk add --no-cache \
+    alpine-sdk \
+    abuild \
+    sudo \
+    git \
+    curl \
+    binutils \
+    elfutils \
+    coreutils \
+    tar \
+    gzip \
+    xz \
+    patch \
+    diffutils \
+    file \
+    && rm -rf /var/cache/apk/*
+
+# Create build user (abuild requires non-root)
+RUN adduser -D -G abuild builder \
+    && echo "builder ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers \
+    && mkdir -p /var/cache/distfiles \
+    && chown -R builder:abuild /var/cache/distfiles
+
+# Setup abuild
+USER builder
+WORKDIR /home/builder
+
+# Generate abuild keys
+RUN abuild-keygen -a -i -n
+
+# Copy normalization and build scripts
+COPY --chown=builder:abuild scripts/normalize.sh /usr/local/bin/normalize.sh
+COPY --chown=builder:abuild scripts/build.sh /usr/local/bin/build.sh
+COPY --chown=builder:abuild scripts/extract-functions.sh /usr/local/bin/extract-functions.sh
+
+RUN chmod +x /usr/local/bin/*.sh
+
+# Environment for reproducibility
+ENV TZ=UTC
+ENV LC_ALL=C.UTF-8
+ENV LANG=C.UTF-8
+
+# Build output directory
+VOLUME /output
+WORKDIR /build
+
+ENTRYPOINT ["/usr/local/bin/build.sh"]
+CMD ["--help"]
--- a/devops/docker/repro-builders/alpine/scripts/build.sh
+++ b/devops/docker/repro-builders/alpine/scripts/build.sh
@@ -0,0 +1,226 @@
+#!/bin/sh
+# Alpine Reproducible Build Script
+# Builds packages with deterministic settings for fingerprint generation
+#
+# Usage: build.sh [build|diff] <package> <version> [patch_url...]
+#
+# Examples:
+#   build.sh build openssl 3.0.7-r0
+#   build.sh diff openssl 3.0.7-r0 3.0.8-r0
+#   build.sh build openssl 3.0.7-r0 https://patch.url/CVE-2023-1234.patch
+
+set -eu
+
+COMMAND="${1:-help}"
+PACKAGE="${2:-}"
+VERSION="${3:-}"
+OUTPUT_DIR="${OUTPUT_DIR:-/output}"
+
+log() {
+    echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*" >&2
+}
+
+show_help() {
+    cat <<EOF
+Alpine Reproducible Builder
+
+Usage:
+    build.sh build <package> <version> [patch_urls...]
+        Build a package with reproducible settings
+    
+    build.sh diff <package> <vuln_version> <patched_version>
+        Build two versions and compute fingerprint diff
+    
+    build.sh --help
+        Show this help message
+
+Environment:
+    SOURCE_DATE_EPOCH    Override timestamp (extracted from APKBUILD if not set)
+    OUTPUT_DIR           Output directory (default: /output)
+    CFLAGS              Additional compiler flags
+    LDFLAGS             Additional linker flags
+
+Examples:
+    build.sh build openssl 3.0.7-r0
+    build.sh build curl 8.1.0-r0 https://patch/CVE-2023-1234.patch
+    build.sh diff openssl 3.0.7-r0 3.0.8-r0
+EOF
+}
+
+setup_reproducible_env() {
+    local pkg="$1"
+    local ver="$2"
+    
+    # Extract SOURCE_DATE_EPOCH from APKBUILD if not set
+    if [ -z "${SOURCE_DATE_EPOCH:-}" ]; then
+        if [ -f "aports/main/$pkg/APKBUILD" ]; then
+            # Use pkgrel date or fallback to current
+            SOURCE_DATE_EPOCH=$(stat -c %Y "aports/main/$pkg/APKBUILD" 2>/dev/null || date +%s)
+        else
+            SOURCE_DATE_EPOCH=$(date +%s)
+        fi
+        export SOURCE_DATE_EPOCH
+    fi
+    
+    log "SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH"
+    
+    # Reproducible compiler flags
+    export CFLAGS="${CFLAGS:-} -fno-record-gcc-switches -fdebug-prefix-map=$(pwd)=/build"
+    export CXXFLAGS="${CXXFLAGS:-} ${CFLAGS}"
+    export LDFLAGS="${LDFLAGS:-}"
+    
+    # Locale for deterministic sorting
+    export LC_ALL=C.UTF-8
+    export TZ=UTC
+}
+
+fetch_source() {
+    local pkg="$1"
+    local ver="$2"
+    
+    log "Fetching source for $pkg-$ver"
+    
+    # Clone aports if needed
+    if [ ! -d "aports" ]; then
+        git clone --depth 1 https://gitlab.alpinelinux.org/alpine/aports.git
+    fi
+    
+    # Find package
+    local pkg_dir=""
+    for repo in main community testing; do
+        if [ -d "aports/$repo/$pkg" ]; then
+            pkg_dir="aports/$repo/$pkg"
+            break
+        fi
+    done
+    
+    if [ -z "$pkg_dir" ]; then
+        log "ERROR: Package $pkg not found in aports"
+        return 1
+    fi
+    
+    # Checkout specific version if needed
+    cd "$pkg_dir"
+    abuild fetch
+    abuild unpack
+}
+
+apply_patches() {
+    local src_dir="$1"
+    shift
+    
+    for patch_url in "$@"; do
+        log "Applying patch: $patch_url"
+        curl -sSL "$patch_url" | patch -d "$src_dir" -p1
+    done
+}
+
+build_package() {
+    local pkg="$1"
+    local ver="$2"
+    shift 2
+    local patches="$@"
+    
+    log "Building $pkg-$ver"
+    
+    setup_reproducible_env "$pkg" "$ver"
+    
+    cd /build
+    fetch_source "$pkg" "$ver"
+    
+    if [ -n "$patches" ]; then
+        apply_patches "src/$pkg-*" $patches
+    fi
+    
+    # Build with reproducible settings
+    abuild -r
+    
+    # Copy output
+    local out_dir="$OUTPUT_DIR/$pkg-$ver"
+    mkdir -p "$out_dir"
+    cp -r ~/packages/*/*.apk "$out_dir/" 2>/dev/null || true
+    
+    # Extract binaries and fingerprints
+    for apk in "$out_dir"/*.apk; do
+        [ -f "$apk" ] || continue
+        local apk_name=$(basename "$apk" .apk)
+        mkdir -p "$out_dir/extracted/$apk_name"
+        tar -xzf "$apk" -C "$out_dir/extracted/$apk_name"
+        
+        # Extract function fingerprints
+        /usr/local/bin/extract-functions.sh "$out_dir/extracted/$apk_name" > "$out_dir/$apk_name.functions.json"
+    done
+    
+    log "Build complete: $out_dir"
+}
+
+diff_versions() {
+    local pkg="$1"
+    local vuln_ver="$2"
+    local patched_ver="$3"
+    
+    log "Building and diffing $pkg: $vuln_ver vs $patched_ver"
+    
+    # Build vulnerable version
+    build_package "$pkg" "$vuln_ver"
+    
+    # Build patched version
+    build_package "$pkg" "$patched_ver"
+    
+    # Compute diff
+    local diff_out="$OUTPUT_DIR/$pkg-diff-$vuln_ver-vs-$patched_ver.json"
+    
+    # Simple diff of function fingerprints
+    jq -s '
+        .[0] as $vuln |
+        .[1] as $patched |
+        {
+            package: "'"$pkg"'",
+            vulnerable_version: "'"$vuln_ver"'",
+            patched_version: "'"$patched_ver"'",
+            vulnerable_functions: ($vuln | length),
+            patched_functions: ($patched | length),
+            added: [($patched[] | select(.name as $n | ($vuln | map(.name) | index($n)) == null))],
+            removed: [($vuln[] | select(.name as $n | ($patched | map(.name) | index($n)) == null))],
+            modified: [
+                $vuln[] | .name as $n | .hash as $h |
+                ($patched[] | select(.name == $n and .hash != $h)) |
+                {name: $n, vuln_hash: $h, patched_hash: .hash}
+            ]
+        }
+    ' \
+        "$OUTPUT_DIR/$pkg-$vuln_ver"/*.functions.json \
+        "$OUTPUT_DIR/$pkg-$patched_ver"/*.functions.json \
+        > "$diff_out"
+    
+    log "Diff complete: $diff_out"
+}
+
+case "$COMMAND" in
+    build)
+        if [ -z "$PACKAGE" ] || [ -z "$VERSION" ]; then
+            log "ERROR: Package and version required"
+            show_help
+            exit 1
+        fi
+        shift 2  # Remove command, package, version
+        build_package "$PACKAGE" "$VERSION" "$@"
+        ;;
+    diff)
+        PATCHED_VERSION="${4:-}"
+        if [ -z "$PACKAGE" ] || [ -z "$VERSION" ] || [ -z "$PATCHED_VERSION" ]; then
+            log "ERROR: Package, vulnerable version, and patched version required"
+            show_help
+            exit 1
+        fi
+        diff_versions "$PACKAGE" "$VERSION" "$PATCHED_VERSION"
+        ;;
+    --help|help)
+        show_help
+        ;;
+    *)
+        log "ERROR: Unknown command: $COMMAND"
+        show_help
+        exit 1
+        ;;
+esac
--- a/devops/docker/repro-builders/alpine/scripts/extract-functions.sh
+++ b/devops/docker/repro-builders/alpine/scripts/extract-functions.sh
@@ -0,0 +1,71 @@
+#!/bin/sh
+# Extract function fingerprints from ELF binaries
+# Outputs JSON array with function name, offset, size, and hashes
+#
+# Usage: extract-functions.sh <directory>
+#
+# Dependencies: objdump, readelf, sha256sum, jq
+
+set -eu
+
+DIR="${1:-.}"
+
+extract_functions_from_binary() {
+    local binary="$1"
+    
+    # Skip non-ELF files
+    file "$binary" | grep -q "ELF" || return 0
+    
+    # Get function symbols
+    objdump -t "$binary" 2>/dev/null | \
+    awk '/\.text.*[0-9a-f]+.*F/ {
+        # Fields: addr flags section size name
+        gsub(/\*.*\*/, "", $1)  # Clean address
+        if ($5 != "" && $4 != "00000000" && $4 != "0000000000000000") {
+            printf "%s %s %s\n", $1, $4, $NF
+        }
+    }' | while read -r offset size name; do
+        # Skip compiler-generated symbols
+        case "$name" in
+            __*|_GLOBAL_*|.plt*|.text*|frame_dummy|register_tm_clones|deregister_tm_clones)
+                continue
+                ;;
+        esac
+        
+        # Convert hex size to decimal
+        dec_size=$((16#$size))
+        
+        # Skip tiny functions (likely padding)
+        [ "$dec_size" -lt 16 ] && continue
+        
+        # Extract function bytes and compute hash
+        # Using objdump to get disassembly and hash the opcodes
+        local hash=$(objdump -d --start-address="0x$offset" --stop-address="0x$((16#$offset + dec_size))" "$binary" 2>/dev/null | \
+            grep "^[[:space:]]*[0-9a-f]*:" | \
+            awk '{for(i=2;i<=NF;i++){if($i~/^[0-9a-f]{2}$/){printf "%s", $i}}}' | \
+            sha256sum | cut -d' ' -f1)
+        
+        # Output JSON object
+        printf '{"name":"%s","offset":"0x%s","size":%d,"hash":"%s"}\n' \
+            "$name" "$offset" "$dec_size" "${hash:-unknown}"
+    done
+}
+
+# Find all ELF binaries in directory
+echo "["
+first=true
+find "$DIR" -type f -executable 2>/dev/null | while read -r binary; do
+    # Check if ELF
+    file "$binary" 2>/dev/null | grep -q "ELF" || continue
+    
+    extract_functions_from_binary "$binary" | while read -r json; do
+        [ -z "$json" ] && continue
+        if [ "$first" = "true" ]; then
+            first=false
+        else
+            echo ","
+        fi
+        echo "$json"
+    done
+done
+echo "]"
--- a/devops/docker/repro-builders/alpine/scripts/normalize.sh
+++ b/devops/docker/repro-builders/alpine/scripts/normalize.sh
@@ -0,0 +1,65 @@
+#!/bin/sh
+# Normalization scripts for reproducible builds
+# Strips non-deterministic content from build artifacts
+#
+# Usage: normalize.sh <directory>
+
+set -eu
+
+DIR="${1:-.}"
+
+log() {
+    echo "[normalize] $*" >&2
+}
+
+# Strip timestamps from __DATE__ and __TIME__ macros
+strip_date_time() {
+    log "Stripping date/time macros..."
+    # Already handled by SOURCE_DATE_EPOCH in modern GCC
+}
+
+# Normalize build paths
+normalize_paths() {
+    log "Normalizing build paths..."
+    # Handled by -fdebug-prefix-map
+}
+
+# Normalize ar archives for deterministic ordering
+normalize_archives() {
+    log "Normalizing ar archives..."
+    find "$DIR" -name "*.a" -type f | while read -r archive; do
+        if ar --version 2>&1 | grep -q "GNU ar"; then
+            # GNU ar with deterministic mode
+            ar -rcsD "$archive.tmp" "$archive" && mv "$archive.tmp" "$archive" 2>/dev/null || true
+        fi
+    done
+}
+
+# Strip debug sections that contain non-deterministic info
+strip_debug_timestamps() {
+    log "Stripping debug timestamps..."
+    find "$DIR" -type f \( -name "*.o" -o -name "*.so" -o -name "*.so.*" -o -executable \) | while read -r obj; do
+        # Check if ELF
+        file "$obj" 2>/dev/null | grep -q "ELF" || continue
+        
+        # Strip build-id if not needed (we regenerate it)
+        # objcopy --remove-section=.note.gnu.build-id "$obj" 2>/dev/null || true
+        
+        # Remove timestamps from DWARF debug info
+        # This is typically handled by SOURCE_DATE_EPOCH
+    done
+}
+
+# Normalize tar archives
+normalize_tars() {
+    log "Normalizing tar archives..."
+    # When creating tars, use:
+    # tar --sort=name --mtime="@${SOURCE_DATE_EPOCH}" --owner=0 --group=0 --numeric-owner
+}
+
+# Run all normalizations
+normalize_paths
+normalize_archives
+strip_debug_timestamps
+
+log "Normalization complete"
--- a/devops/docker/repro-builders/debian/Dockerfile
+++ b/devops/docker/repro-builders/debian/Dockerfile
@@ -0,0 +1,59 @@
+# Debian Reproducible Builder
+# Creates deterministic builds of Debian packages for fingerprint diffing
+#
+# Usage:
+#   docker build -t repro-builder-debian:bookworm --build-arg RELEASE=bookworm .
+#   docker run -v ./output:/output repro-builder-debian:bookworm build openssl 3.0.7-1
+
+ARG RELEASE=bookworm
+FROM debian:${RELEASE}
+
+ARG RELEASE
+ENV DEBIAN_RELEASE=${RELEASE}
+ENV DEBIAN_FRONTEND=noninteractive
+
+# Install build tools
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    devscripts \
+    dpkg-dev \
+    equivs \
+    fakeroot \
+    git \
+    curl \
+    ca-certificates \
+    binutils \
+    elfutils \
+    coreutils \
+    patch \
+    diffutils \
+    file \
+    jq \
+    && rm -rf /var/lib/apt/lists/*
+
+# Create build user
+RUN useradd -m -s /bin/bash builder \
+    && echo "builder ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
+
+USER builder
+WORKDIR /home/builder
+
+# Copy scripts
+COPY --chown=builder:builder scripts/build.sh /usr/local/bin/build.sh
+COPY --chown=builder:builder scripts/extract-functions.sh /usr/local/bin/extract-functions.sh
+COPY --chown=builder:builder scripts/normalize.sh /usr/local/bin/normalize.sh
+
+USER root
+RUN chmod +x /usr/local/bin/*.sh
+USER builder
+
+# Environment for reproducibility
+ENV TZ=UTC
+ENV LC_ALL=C.UTF-8
+ENV LANG=C.UTF-8
+
+VOLUME /output
+WORKDIR /build
+
+ENTRYPOINT ["/usr/local/bin/build.sh"]
+CMD ["--help"]
--- a/devops/docker/repro-builders/debian/scripts/build.sh
+++ b/devops/docker/repro-builders/debian/scripts/build.sh
@@ -0,0 +1,233 @@
+#!/bin/bash
+# Debian Reproducible Build Script
+# Builds packages with deterministic settings for fingerprint generation
+#
+# Usage: build.sh [build|diff] <package> <version> [patch_url...]
+
+set -euo pipefail
+
+COMMAND="${1:-help}"
+PACKAGE="${2:-}"
+VERSION="${3:-}"
+OUTPUT_DIR="${OUTPUT_DIR:-/output}"
+
+log() {
+    echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*" >&2
+}
+
+show_help() {
+    cat <<EOF
+Debian Reproducible Builder
+
+Usage:
+    build.sh build <package> <version> [patch_urls...]
+        Build a package with reproducible settings
+    
+    build.sh diff <package> <vuln_version> <patched_version>
+        Build two versions and compute fingerprint diff
+    
+    build.sh --help
+        Show this help message
+
+Environment:
+    SOURCE_DATE_EPOCH    Override timestamp (extracted from changelog if not set)
+    OUTPUT_DIR           Output directory (default: /output)
+    DEB_BUILD_OPTIONS    Additional build options
+
+Examples:
+    build.sh build openssl 3.0.7-1
+    build.sh diff curl 8.1.0-1 8.1.0-2
+EOF
+}
+
+setup_reproducible_env() {
+    local pkg="$1"
+    
+    # Reproducible build flags
+    export DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS:-} reproducible=+all"
+    export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(date +%s)}"
+    
+    # Compiler flags for reproducibility
+    export CFLAGS="${CFLAGS:-} -fno-record-gcc-switches -fdebug-prefix-map=$(pwd)=/build"
+    export CXXFLAGS="${CXXFLAGS:-} ${CFLAGS}"
+    
+    export LC_ALL=C.UTF-8
+    export TZ=UTC
+    
+    log "SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH"
+}
+
+fetch_source() {
+    local pkg="$1"
+    local ver="$2"
+    
+    log "Fetching source for $pkg=$ver"
+    
+    mkdir -p /build/src
+    cd /build/src
+    
+    # Enable source repositories
+    sudo sed -i 's/^# deb-src/deb-src/' /etc/apt/sources.list.d/*.sources 2>/dev/null || \
+    sudo sed -i 's/^# deb-src/deb-src/' /etc/apt/sources.list 2>/dev/null || true
+    sudo apt-get update
+    
+    # Fetch source
+    if [ -n "$ver" ]; then
+        apt-get source "${pkg}=${ver}" || apt-get source "$pkg"
+    else
+        apt-get source "$pkg"
+    fi
+    
+    # Find extracted directory
+    local src_dir=$(ls -d "${pkg}"*/ 2>/dev/null | head -1)
+    if [ -z "$src_dir" ]; then
+        log "ERROR: Could not find source directory for $pkg"
+        return 1
+    fi
+    
+    # Extract SOURCE_DATE_EPOCH from changelog
+    if [ -z "${SOURCE_DATE_EPOCH:-}" ]; then
+        if [ -f "$src_dir/debian/changelog" ]; then
+            SOURCE_DATE_EPOCH=$(dpkg-parsechangelog -l "$src_dir/debian/changelog" -S Timestamp 2>/dev/null || date +%s)
+            export SOURCE_DATE_EPOCH
+        fi
+    fi
+    
+    echo "$src_dir"
+}
+
+install_build_deps() {
+    local src_dir="$1"
+    
+    log "Installing build dependencies"
+    cd "$src_dir"
+    sudo apt-get build-dep -y . || true
+}
+
+apply_patches() {
+    local src_dir="$1"
+    shift
+    
+    cd "$src_dir"
+    for patch_url in "$@"; do
+        log "Applying patch: $patch_url"
+        curl -sSL "$patch_url" | patch -p1
+    done
+}
+
+build_package() {
+    local pkg="$1"
+    local ver="$2"
+    shift 2
+    local patches="${@:-}"
+    
+    log "Building $pkg version $ver"
+    
+    setup_reproducible_env "$pkg"
+    
+    cd /build
+    local src_dir=$(fetch_source "$pkg" "$ver")
+    
+    install_build_deps "$src_dir"
+    
+    if [ -n "$patches" ]; then
+        apply_patches "$src_dir" $patches
+    fi
+    
+    cd "$src_dir"
+    
+    # Build with reproducible settings
+    dpkg-buildpackage -b -us -uc
+    
+    # Copy output
+    local out_dir="$OUTPUT_DIR/$pkg-$ver"
+    mkdir -p "$out_dir"
+    cp -r /build/src/*.deb "$out_dir/" 2>/dev/null || true
+    
+    # Extract and fingerprint
+    for deb in "$out_dir"/*.deb; do
+        [ -f "$deb" ] || continue
+        local deb_name=$(basename "$deb" .deb)
+        mkdir -p "$out_dir/extracted/$deb_name"
+        dpkg-deb -x "$deb" "$out_dir/extracted/$deb_name"
+        
+        # Extract function fingerprints
+        /usr/local/bin/extract-functions.sh "$out_dir/extracted/$deb_name" > "$out_dir/$deb_name.functions.json"
+    done
+    
+    log "Build complete: $out_dir"
+}
+
+diff_versions() {
+    local pkg="$1"
+    local vuln_ver="$2"
+    local patched_ver="$3"
+    
+    log "Building and diffing $pkg: $vuln_ver vs $patched_ver"
+    
+    # Build vulnerable version
+    build_package "$pkg" "$vuln_ver"
+    
+    # Clean build environment
+    rm -rf /build/src/*
+    
+    # Build patched version  
+    build_package "$pkg" "$patched_ver"
+    
+    # Compute diff
+    local diff_out="$OUTPUT_DIR/$pkg-diff-$vuln_ver-vs-$patched_ver.json"
+    
+    jq -s '
+        .[0] as $vuln |
+        .[1] as $patched |
+        {
+            package: "'"$pkg"'",
+            vulnerable_version: "'"$vuln_ver"'",
+            patched_version: "'"$patched_ver"'",
+            vulnerable_functions: ($vuln | length),
+            patched_functions: ($patched | length),
+            added: [($patched[] | select(.name as $n | ($vuln | map(.name) | index($n)) == null))],
+            removed: [($vuln[] | select(.name as $n | ($patched | map(.name) | index($n)) == null))],
+            modified: [
+                $vuln[] | .name as $n | .hash as $h |
+                ($patched[] | select(.name == $n and .hash != $h)) |
+                {name: $n, vuln_hash: $h, patched_hash: .hash}
+            ]
+        }
+    ' \
+        "$OUTPUT_DIR/$pkg-$vuln_ver"/*.functions.json \
+        "$OUTPUT_DIR/$pkg-$patched_ver"/*.functions.json \
+        > "$diff_out" 2>/dev/null || log "Warning: Could not compute diff"
+    
+    log "Diff complete: $diff_out"
+}
+
+case "$COMMAND" in
+    build)
+        if [ -z "$PACKAGE" ]; then
+            log "ERROR: Package required"
+            show_help
+            exit 1
+        fi
+        shift 2  # Remove command, package
+        [ -n "${VERSION:-}" ] && shift  # Remove version if present
+        build_package "$PACKAGE" "${VERSION:-}" "$@"
+        ;;
+    diff)
+        PATCHED_VERSION="${4:-}"
+        if [ -z "$PACKAGE" ] || [ -z "$VERSION" ] || [ -z "$PATCHED_VERSION" ]; then
+            log "ERROR: Package, vulnerable version, and patched version required"
+            show_help
+            exit 1
+        fi
+        diff_versions "$PACKAGE" "$VERSION" "$PATCHED_VERSION"
+        ;;
+    --help|help)
+        show_help
+        ;;
+    *)
+        log "ERROR: Unknown command: $COMMAND"
+        show_help
+        exit 1
+        ;;
+esac
--- a/devops/docker/repro-builders/debian/scripts/extract-functions.sh
+++ b/devops/docker/repro-builders/debian/scripts/extract-functions.sh
@@ -0,0 +1,67 @@
+#!/bin/bash
+# Extract function fingerprints from ELF binaries
+# Outputs JSON array with function name, offset, size, and hashes
+
+set -euo pipefail
+
+DIR="${1:-.}"
+
+extract_functions_from_binary() {
+    local binary="$1"
+    
+    # Skip non-ELF files
+    file "$binary" 2>/dev/null | grep -q "ELF" || return 0
+    
+    # Get function symbols with objdump
+    objdump -t "$binary" 2>/dev/null | \
+    awk '/\.text.*[0-9a-f]+.*F/ {
+        gsub(/\*.*\*/, "", $1)
+        if ($5 != "" && length($4) > 0) {
+            size = strtonum("0x" $4)
+            if (size >= 16) {
+                print $1, $4, $NF
+            }
+        }
+    }' | while read -r offset size name; do
+        # Skip compiler-generated symbols
+        case "$name" in
+            __*|_GLOBAL_*|.plt*|.text*|frame_dummy|register_tm_clones|deregister_tm_clones|_start|_init|_fini)
+                continue
+                ;;
+        esac
+        
+        # Convert hex size
+        dec_size=$((16#$size))
+        
+        # Compute hash of function bytes
+        local hash=$(objdump -d --start-address="0x$offset" --stop-address="$((16#$offset + dec_size))" "$binary" 2>/dev/null | \
+            grep -E "^[[:space:]]*[0-9a-f]+:" | \
+            awk '{for(i=2;i<=NF;i++){if($i~/^[0-9a-f]{2}$/){printf "%s", $i}}}' | \
+            sha256sum | cut -d' ' -f1)
+        
+        [ -n "$hash" ] || hash="unknown"
+        
+        printf '{"name":"%s","offset":"0x%s","size":%d,"hash":"%s"}\n' \
+            "$name" "$offset" "$dec_size" "$hash"
+    done
+}
+
+# Output JSON array
+echo "["
+first=true
+
+find "$DIR" -type f \( -executable -o -name "*.so" -o -name "*.so.*" \) 2>/dev/null | while read -r binary; do
+    file "$binary" 2>/dev/null | grep -q "ELF" || continue
+    
+    extract_functions_from_binary "$binary" | while read -r json; do
+        [ -z "$json" ] && continue
+        if [ "$first" = "true" ]; then
+            first=false
+            echo "$json"
+        else
+            echo ",$json"
+        fi
+    done
+done
+
+echo "]"
--- a/devops/docker/repro-builders/debian/scripts/normalize.sh
+++ b/devops/docker/repro-builders/debian/scripts/normalize.sh
@@ -0,0 +1,29 @@
+#!/bin/bash
+# Normalization scripts for Debian reproducible builds
+
+set -euo pipefail
+
+DIR="${1:-.}"
+
+log() {
+    echo "[normalize] $*" >&2
+}
+
+normalize_archives() {
+    log "Normalizing ar archives..."
+    find "$DIR" -name "*.a" -type f | while read -r archive; do
+        if ar --version 2>&1 | grep -q "GNU ar"; then
+            ar -rcsD "$archive.tmp" "$archive" 2>/dev/null && mv "$archive.tmp" "$archive" || true
+        fi
+    done
+}
+
+strip_debug_timestamps() {
+    log "Stripping debug timestamps..."
+    # Handled by SOURCE_DATE_EPOCH and DEB_BUILD_OPTIONS
+}
+
+normalize_archives
+strip_debug_timestamps
+
+log "Normalization complete"
--- a/devops/docker/repro-builders/rhel/Dockerfile
+++ b/devops/docker/repro-builders/rhel/Dockerfile
@@ -0,0 +1,85 @@
+# RHEL-compatible Reproducible Build Container
+# Sprint: SPRINT_1227_0002_0001 (Reproducible Builders)
+# Task: T3 - RHEL builder with mock-based package building
+#
+# Uses AlmaLinux 9 as RHEL-compatible base for open source builds.
+# Production RHEL builds require valid subscription.
+
+ARG BASE_IMAGE=almalinux:9
+FROM ${BASE_IMAGE} AS builder
+
+LABEL org.opencontainers.image.title="StellaOps RHEL Reproducible Builder"
+LABEL org.opencontainers.image.description="RHEL-compatible reproducible build environment for security patching"
+LABEL org.opencontainers.image.vendor="StellaOps"
+LABEL org.opencontainers.image.source="https://github.com/stellaops/stellaops"
+
+# Install build dependencies
+RUN dnf -y update && \
+    dnf -y install \
+        # Core build tools
+        rpm-build \
+        rpmdevtools \
+        rpmlint \
+        mock \
+        # Compiler toolchain
+        gcc \
+        gcc-c++ \
+        make \
+        cmake \
+        autoconf \
+        automake \
+        libtool \
+        # Package management
+        dnf-plugins-core \
+        yum-utils \
+        createrepo_c \
+        # Binary analysis
+        binutils \
+        elfutils \
+        gdb \
+        # Reproducibility
+        diffoscope \
+        # Source control
+        git \
+        patch \
+        # Utilities
+        wget \
+        curl \
+        jq \
+        python3 \
+        python3-pip && \
+    dnf clean all
+
+# Create mock user (mock requires non-root)
+RUN useradd -m mockbuild && \
+    usermod -a -G mock mockbuild
+
+# Set up rpmbuild directories
+RUN mkdir -p /build/{BUILD,RPMS,SOURCES,SPECS,SRPMS} && \
+    chown -R mockbuild:mockbuild /build
+
+# Copy build scripts
+COPY scripts/build.sh /usr/local/bin/build.sh
+COPY scripts/extract-functions.sh /usr/local/bin/extract-functions.sh
+COPY scripts/normalize.sh /usr/local/bin/normalize.sh
+COPY scripts/mock-build.sh /usr/local/bin/mock-build.sh
+
+RUN chmod +x /usr/local/bin/*.sh
+
+# Set reproducibility environment
+ENV TZ=UTC
+ENV LC_ALL=C.UTF-8
+ENV LANG=C.UTF-8
+
+# Deterministic compiler flags
+ENV CFLAGS="-fno-record-gcc-switches -fdebug-prefix-map=/build=/buildroot -O2 -g"
+ENV CXXFLAGS="${CFLAGS}"
+
+# Mock configuration for reproducible builds
+COPY mock/stellaops-repro.cfg /etc/mock/stellaops-repro.cfg
+
+WORKDIR /build
+USER mockbuild
+
+ENTRYPOINT ["/usr/local/bin/build.sh"]
+CMD ["--help"]
--- a/devops/docker/repro-builders/rhel/mock/stellaops-repro.cfg
+++ b/devops/docker/repro-builders/rhel/mock/stellaops-repro.cfg
@@ -0,0 +1,71 @@
+# StellaOps Reproducible Build Mock Configuration
+# Sprint: SPRINT_1227_0002_0001 (Reproducible Builders)
+#
+# Mock configuration optimized for reproducible RHEL/AlmaLinux builds
+
+config_opts['root'] = 'stellaops-repro'
+config_opts['target_arch'] = 'x86_64'
+config_opts['legal_host_arches'] = ('x86_64',)
+config_opts['chroot_setup_cmd'] = 'install @buildsys-build'
+config_opts['dist'] = 'el9'
+config_opts['releasever'] = '9'
+
+# Reproducibility settings
+config_opts['use_host_resolv'] = False
+config_opts['rpmbuild_networking'] = False
+config_opts['cleanup_on_success'] = True
+config_opts['cleanup_on_failure'] = True
+
+# Deterministic build settings
+config_opts['macros']['SOURCE_DATE_EPOCH'] = '%{getenv:SOURCE_DATE_EPOCH}'
+config_opts['macros']['_buildhost'] = 'stellaops.build'
+config_opts['macros']['debug_package'] = '%{nil}'
+config_opts['macros']['_default_patch_fuzz'] = '0'
+
+# Compiler flags for reproducibility
+config_opts['macros']['optflags'] = '-O2 -g -fno-record-gcc-switches -fdebug-prefix-map=%{_builddir}=/buildroot'
+
+# Environment normalization
+config_opts['environment']['TZ'] = 'UTC'
+config_opts['environment']['LC_ALL'] = 'C.UTF-8'
+config_opts['environment']['LANG'] = 'C.UTF-8'
+
+# Use AlmaLinux as RHEL-compatible base
+config_opts['dnf.conf'] = """
+[main]
+keepcache=1
+debuglevel=2
+reposdir=/dev/null
+logfile=/var/log/yum.log
+retries=20
+obsoletes=1
+gpgcheck=0
+assumeyes=1
+syslog_ident=mock
+syslog_device=
+metadata_expire=0
+mdpolicy=group:primary
+best=1
+install_weak_deps=0
+protected_packages=
+module_platform_id=platform:el9
+user_agent={{ user_agent }}
+
+[baseos]
+name=AlmaLinux $releasever - BaseOS
+mirrorlist=https://mirrors.almalinux.org/mirrorlist/$releasever/baseos
+enabled=1
+gpgcheck=0
+
+[appstream]
+name=AlmaLinux $releasever - AppStream
+mirrorlist=https://mirrors.almalinux.org/mirrorlist/$releasever/appstream
+enabled=1
+gpgcheck=0
+
+[crb]
+name=AlmaLinux $releasever - CRB
+mirrorlist=https://mirrors.almalinux.org/mirrorlist/$releasever/crb
+enabled=1
+gpgcheck=0
+"""
--- a/devops/docker/repro-builders/rhel/scripts/build.sh
+++ b/devops/docker/repro-builders/rhel/scripts/build.sh
@@ -0,0 +1,213 @@
+#!/bin/bash
+# RHEL Reproducible Build Script
+# Sprint: SPRINT_1227_0002_0001 (Reproducible Builders)
+#
+# Usage: build.sh --srpm <url_or_path> [--patch <patch_file>] [--output <dir>]
+
+set -euo pipefail
+
+# Default values
+OUTPUT_DIR="/build/output"
+WORK_DIR="/build/work"
+SRPM=""
+PATCH_FILE=""
+SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-}"
+
+usage() {
+    cat <<EOF
+RHEL Reproducible Build Script
+
+Usage: $0 [OPTIONS]
+
+Options:
+    --srpm <path>       Path or URL to SRPM file (required)
+    --patch <path>      Path to security patch file (optional)
+    --output <dir>      Output directory (default: /build/output)
+    --epoch <timestamp> SOURCE_DATE_EPOCH value (default: from changelog)
+    --help              Show this help message
+
+Examples:
+    $0 --srpm openssl-3.0.7-1.el9.src.rpm --patch CVE-2023-0286.patch
+    $0 --srpm https://mirror/srpms/curl-8.0.1-1.el9.src.rpm
+
+EOF
+    exit 0
+}
+
+log() {
+    echo "[$(date -u '+%Y-%m-%dT%H:%M:%SZ')] $*"
+}
+
+error() {
+    log "ERROR: $*" >&2
+    exit 1
+}
+
+# Parse arguments
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        --srpm)
+            SRPM="$2"
+            shift 2
+            ;;
+        --patch)
+            PATCH_FILE="$2"
+            shift 2
+            ;;
+        --output)
+            OUTPUT_DIR="$2"
+            shift 2
+            ;;
+        --epoch)
+            SOURCE_DATE_EPOCH="$2"
+            shift 2
+            ;;
+        --help)
+            usage
+            ;;
+        *)
+            error "Unknown option: $1"
+            ;;
+    esac
+done
+
+[[ -z "${SRPM}" ]] && error "SRPM path required. Use --srpm <path>"
+
+# Create directories
+mkdir -p "${OUTPUT_DIR}" "${WORK_DIR}"
+cd "${WORK_DIR}"
+
+log "Starting RHEL reproducible build"
+log "SRPM: ${SRPM}"
+
+# Download or copy SRPM
+if [[ "${SRPM}" =~ ^https?:// ]]; then
+    log "Downloading SRPM..."
+    curl -fsSL -o source.src.rpm "${SRPM}"
+    SRPM="source.src.rpm"
+elif [[ ! -f "${SRPM}" ]]; then
+    error "SRPM file not found: ${SRPM}"
+fi
+
+# Install SRPM
+log "Installing SRPM..."
+rpm2cpio "${SRPM}" | cpio -idmv
+
+# Extract SOURCE_DATE_EPOCH from changelog if not provided
+if [[ -z "${SOURCE_DATE_EPOCH}" ]]; then
+    SPEC_FILE=$(find . -name "*.spec" | head -1)
+    if [[ -n "${SPEC_FILE}" ]]; then
+        # Extract date from first changelog entry
+        CHANGELOG_DATE=$(grep -m1 '^\*' "${SPEC_FILE}" | sed 's/^\* //' | cut -d' ' -f1-3)
+        if [[ -n "${CHANGELOG_DATE}" ]]; then
+            SOURCE_DATE_EPOCH=$(date -d "${CHANGELOG_DATE}" +%s 2>/dev/null || echo "")
+        fi
+    fi
+    
+    if [[ -z "${SOURCE_DATE_EPOCH}" ]]; then
+        SOURCE_DATE_EPOCH=$(date +%s)
+        log "Warning: Using current time for SOURCE_DATE_EPOCH"
+    fi
+fi
+
+export SOURCE_DATE_EPOCH
+log "SOURCE_DATE_EPOCH: ${SOURCE_DATE_EPOCH}"
+
+# Apply security patch if provided
+if [[ -n "${PATCH_FILE}" ]]; then
+    if [[ ! -f "${PATCH_FILE}" ]]; then
+        error "Patch file not found: ${PATCH_FILE}"
+    fi
+    
+    log "Applying security patch: ${PATCH_FILE}"
+    
+    # Copy patch to SOURCES
+    PATCH_NAME=$(basename "${PATCH_FILE}")
+    cp "${PATCH_FILE}" SOURCES/
+    
+    # Add patch to spec file
+    SPEC_FILE=$(find . -name "*.spec" | head -1)
+    if [[ -n "${SPEC_FILE}" ]]; then
+        # Find last Patch line or Source line
+        LAST_PATCH=$(grep -n '^Patch[0-9]*:' "${SPEC_FILE}" | tail -1 | cut -d: -f1)
+        if [[ -z "${LAST_PATCH}" ]]; then
+            LAST_PATCH=$(grep -n '^Source[0-9]*:' "${SPEC_FILE}" | tail -1 | cut -d: -f1)
+        fi
+        
+        # Calculate next patch number
+        PATCH_NUM=$(grep -c '^Patch[0-9]*:' "${SPEC_FILE}" || echo 0)
+        PATCH_NUM=$((PATCH_NUM + 100))  # Use 100+ for security patches
+        
+        # Insert patch declaration
+        sed -i "${LAST_PATCH}a Patch${PATCH_NUM}: ${PATCH_NAME}" "${SPEC_FILE}"
+        
+        # Add %patch to %prep if not using autosetup
+        if ! grep -q '%autosetup' "${SPEC_FILE}"; then
+            PREP_LINE=$(grep -n '^%prep' "${SPEC_FILE}" | head -1 | cut -d: -f1)
+            if [[ -n "${PREP_LINE}" ]]; then
+                # Find last %patch line in %prep
+                LAST_PATCH_LINE=$(sed -n "${PREP_LINE},\$p" "${SPEC_FILE}" | grep -n '^%patch' | tail -1 | cut -d: -f1)
+                if [[ -n "${LAST_PATCH_LINE}" ]]; then
+                    INSERT_LINE=$((PREP_LINE + LAST_PATCH_LINE))
+                else
+                    INSERT_LINE=$((PREP_LINE + 1))
+                fi
+                sed -i "${INSERT_LINE}a %patch${PATCH_NUM} -p1" "${SPEC_FILE}"
+            fi
+        fi
+    fi
+fi
+
+# Set up rpmbuild tree
+log "Setting up rpmbuild tree..."
+rpmdev-setuptree || true
+
+# Copy sources and spec
+cp -r SOURCES/* ~/rpmbuild/SOURCES/ 2>/dev/null || true
+cp *.spec ~/rpmbuild/SPECS/ 2>/dev/null || true
+
+# Build using mock for isolation and reproducibility
+log "Building with mock (stellaops-repro config)..."
+SPEC_FILE=$(find ~/rpmbuild/SPECS -name "*.spec" | head -1)
+
+if [[ -n "${SPEC_FILE}" ]]; then
+    # Build SRPM first
+    rpmbuild -bs "${SPEC_FILE}"
+    
+    BUILT_SRPM=$(find ~/rpmbuild/SRPMS -name "*.src.rpm" | head -1)
+    
+    if [[ -n "${BUILT_SRPM}" ]]; then
+        # Build with mock
+        mock -r stellaops-repro --rebuild "${BUILT_SRPM}" --resultdir="${OUTPUT_DIR}/rpms"
+    else
+        error "SRPM build failed"
+    fi
+else
+    error "No spec file found"
+fi
+
+# Extract function fingerprints from built RPMs
+log "Extracting function fingerprints..."
+for rpm in "${OUTPUT_DIR}/rpms"/*.rpm; do
+    if [[ -f "${rpm}" ]] && [[ ! "${rpm}" =~ \.src\.rpm$ ]]; then
+        /usr/local/bin/extract-functions.sh "${rpm}" "${OUTPUT_DIR}/fingerprints"
+    fi
+done
+
+# Generate build manifest
+log "Generating build manifest..."
+cat > "${OUTPUT_DIR}/manifest.json" <<EOF
+{
+  "builder": "rhel",
+  "base_image": "${BASE_IMAGE:-almalinux:9}",
+  "source_date_epoch": ${SOURCE_DATE_EPOCH},
+  "build_timestamp": "$(date -u '+%Y-%m-%dT%H:%M:%SZ')",
+  "srpm": "${SRPM}",
+  "patch_applied": $(if [[ -n "${PATCH_FILE}" ]]; then echo "\"${PATCH_FILE}\""; else echo "null"; fi),
+  "rpm_outputs": $(find "${OUTPUT_DIR}/rpms" -name "*.rpm" ! -name "*.src.rpm" -printf '"%f",' 2>/dev/null | sed 's/,$//' | sed 's/^/[/' | sed 's/$/]/'),
+  "fingerprint_files": $(find "${OUTPUT_DIR}/fingerprints" -name "*.json" -printf '"%f",' 2>/dev/null | sed 's/,$//' | sed 's/^/[/' | sed 's/$/]/')
+}
+EOF
+
+log "Build complete. Output in: ${OUTPUT_DIR}"
+log "Manifest: ${OUTPUT_DIR}/manifest.json"
--- a/devops/docker/repro-builders/rhel/scripts/extract-functions.sh
+++ b/devops/docker/repro-builders/rhel/scripts/extract-functions.sh
@@ -0,0 +1,73 @@
+#!/bin/bash
+# RHEL Function Extraction Script
+# Sprint: SPRINT_1227_0002_0001 (Reproducible Builders)
+#
+# Extracts function-level fingerprints from RPM packages
+
+set -euo pipefail
+
+RPM_PATH="${1:-}"
+OUTPUT_DIR="${2:-/build/fingerprints}"
+
+[[ -z "${RPM_PATH}" ]] && { echo "Usage: $0 <rpm_path> [output_dir]"; exit 1; }
+[[ ! -f "${RPM_PATH}" ]] && { echo "RPM not found: ${RPM_PATH}"; exit 1; }
+
+mkdir -p "${OUTPUT_DIR}"
+
+RPM_NAME=$(rpm -qp --qf '%{NAME}' "${RPM_PATH}" 2>/dev/null)
+RPM_VERSION=$(rpm -qp --qf '%{VERSION}-%{RELEASE}' "${RPM_PATH}" 2>/dev/null)
+
+WORK_DIR=$(mktemp -d)
+trap "rm -rf ${WORK_DIR}" EXIT
+
+cd "${WORK_DIR}"
+
+# Extract RPM contents
+rpm2cpio "${RPM_PATH}" | cpio -idmv 2>/dev/null
+
+# Find ELF binaries
+find . -type f -exec file {} \; | grep -E 'ELF.*(executable|shared object)' | cut -d: -f1 | while read -r binary; do
+    BINARY_NAME=$(basename "${binary}")
+    BINARY_PATH="${binary#./}"
+    
+    # Get build-id if present
+    BUILD_ID=$(readelf -n "${binary}" 2>/dev/null | grep 'Build ID:' | awk '{print $3}' || echo "")
+    
+    # Extract function symbols
+    OUTPUT_FILE="${OUTPUT_DIR}/${RPM_NAME}_${BINARY_NAME}.json"
+    
+    {
+        echo "{"
+        echo "  \"package\": \"${RPM_NAME}\","
+        echo "  \"version\": \"${RPM_VERSION}\","
+        echo "  \"binary\": \"${BINARY_PATH}\","
+        echo "  \"build_id\": \"${BUILD_ID}\","
+        echo "  \"extracted_at\": \"$(date -u '+%Y-%m-%dT%H:%M:%SZ')\","
+        echo "  \"functions\": ["
+        
+        # Extract function addresses and sizes using nm and objdump
+        FIRST=true
+        nm -S --defined-only "${binary}" 2>/dev/null | grep -E '^[0-9a-f]+ [0-9a-f]+ [Tt]' | while read -r addr size type name; do
+            if [[ "${FIRST}" == "true" ]]; then
+                FIRST=false
+            else
+                echo ","
+            fi
+            
+            # Calculate function hash from disassembly
+            FUNC_HASH=$(objdump -d --start-address=0x${addr} --stop-address=$((0x${addr} + 0x${size})) "${binary}" 2>/dev/null | \
+                        grep -E '^\s+[0-9a-f]+:' | awk '{$1=""; print}' | sha256sum | cut -d' ' -f1)
+            
+            printf '    {"name": "%s", "address": "0x%s", "size": %d, "hash": "%s"}' \
+                "${name}" "${addr}" "$((0x${size}))" "${FUNC_HASH}"
+        done || true
+        
+        echo ""
+        echo "  ]"
+        echo "}"
+    } > "${OUTPUT_FILE}"
+    
+    echo "Extracted: ${OUTPUT_FILE}"
+done
+
+echo "Function extraction complete for: ${RPM_NAME}"
--- a/devops/docker/repro-builders/rhel/scripts/mock-build.sh
+++ b/devops/docker/repro-builders/rhel/scripts/mock-build.sh
@@ -0,0 +1,34 @@
+#!/bin/bash
+# RHEL Mock Build Script
+# Sprint: SPRINT_1227_0002_0001 (Reproducible Builders)
+#
+# Builds SRPMs using mock for isolation and reproducibility
+
+set -euo pipefail
+
+SRPM="${1:-}"
+RESULT_DIR="${2:-/build/output}"
+CONFIG="${3:-stellaops-repro}"
+
+[[ -z "${SRPM}" ]] && { echo "Usage: $0 <srpm> [result_dir] [mock_config]"; exit 1; }
+[[ ! -f "${SRPM}" ]] && { echo "SRPM not found: ${SRPM}"; exit 1; }
+
+mkdir -p "${RESULT_DIR}"
+
+echo "Building SRPM with mock: ${SRPM}"
+echo "Config: ${CONFIG}"
+echo "Output: ${RESULT_DIR}"
+
+# Initialize mock if needed
+mock -r "${CONFIG}" --init
+
+# Build with reproducibility settings
+mock -r "${CONFIG}" \
+    --rebuild "${SRPM}" \
+    --resultdir="${RESULT_DIR}" \
+    --define "SOURCE_DATE_EPOCH ${SOURCE_DATE_EPOCH:-$(date +%s)}" \
+    --define "_buildhost stellaops.build" \
+    --define "debug_package %{nil}"
+
+echo "Build complete. Results in: ${RESULT_DIR}"
+ls -la "${RESULT_DIR}"
--- a/devops/docker/repro-builders/rhel/scripts/normalize.sh
+++ b/devops/docker/repro-builders/rhel/scripts/normalize.sh
@@ -0,0 +1,83 @@
+#!/bin/bash
+# RHEL Build Normalization Script
+# Sprint: SPRINT_1227_0002_0001 (Reproducible Builders)
+#
+# Normalizes RPM build environment for reproducibility
+
+set -euo pipefail
+
+# Normalize environment
+export TZ=UTC
+export LC_ALL=C.UTF-8
+export LANG=C.UTF-8
+
+# Deterministic compiler flags
+export CFLAGS="${CFLAGS:--fno-record-gcc-switches -fdebug-prefix-map=$(pwd)=/buildroot -O2 -g}"
+export CXXFLAGS="${CXXFLAGS:-${CFLAGS}}"
+
+# Disable debug info that varies
+export DEB_BUILD_OPTIONS="nostrip noopt"
+
+# RPM-specific reproducibility
+export RPM_BUILD_NCPUS=1
+
+# Normalize timestamps in archives
+normalize_ar() {
+    local archive="$1"
+    if command -v llvm-ar &>/dev/null; then
+        llvm-ar --format=gnu --enable-deterministic-archives rcs "${archive}.new" "${archive}"
+        mv "${archive}.new" "${archive}"
+    fi
+}
+
+# Normalize timestamps in tar archives
+normalize_tar() {
+    local archive="$1"
+    local mtime="${SOURCE_DATE_EPOCH:-0}"
+    
+    # Repack with deterministic settings
+    local tmp_dir=$(mktemp -d)
+    tar -xf "${archive}" -C "${tmp_dir}"
+    tar --sort=name \
+        --mtime="@${mtime}" \
+        --owner=0 --group=0 \
+        --numeric-owner \
+        -cf "${archive}.new" -C "${tmp_dir}" .
+    mv "${archive}.new" "${archive}"
+    rm -rf "${tmp_dir}"
+}
+
+# Normalize __pycache__ timestamps
+normalize_python() {
+    find . -name '__pycache__' -type d -exec rm -rf {} + 2>/dev/null || true
+    find . -name '*.pyc' -delete 2>/dev/null || true
+}
+
+# Strip build paths from binaries
+strip_build_paths() {
+    local binary="$1"
+    if command -v objcopy &>/dev/null; then
+        # Remove .note.gnu.build-id if it contains build path
+        objcopy --remove-section=.note.gnu.build-id "${binary}" 2>/dev/null || true
+    fi
+}
+
+# Main normalization
+normalize_build() {
+    echo "Normalizing build environment..."
+    
+    # Normalize Python bytecode
+    normalize_python
+    
+    # Find and normalize archives
+    find . -name '*.a' -type f | while read -r ar; do
+        normalize_ar "${ar}"
+    done
+    
+    echo "Normalization complete"
+}
+
+# If sourced, export functions; if executed, run normalization
+if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
+    normalize_build
+fi