From 73769ca19ae39c1424a59e042fa03e375bbc5622 Mon Sep 17 00:00:00 2001 From: Hunter Halloran Date: Fri, 30 Jan 2026 12:02:27 -0500 Subject: [PATCH] feat: Add flake and ragenix package generation and dev environment --- NIX_FLAKE_MIGRATION.md | 223 +++++++++++++++++++++++++++++++++++++++++ default.nix | 155 +--------------------------- usda-vision | 2 +- 3 files changed, 229 insertions(+), 151 deletions(-) create mode 100644 NIX_FLAKE_MIGRATION.md diff --git a/NIX_FLAKE_MIGRATION.md b/NIX_FLAKE_MIGRATION.md new file mode 100644 index 0000000..2a30e68 --- /dev/null +++ b/NIX_FLAKE_MIGRATION.md @@ -0,0 +1,223 @@ +# Nix Flake Migration Summary + +## What Changed + +The USDA Vision repository has been migrated from inline Nix derivations to a self-contained flake with integrated secrets management. + +### Before +``` +usda-dash-config/ +├── default.nix # Contains all build logic inline +└── usda-vision/ # Git submodule + ├── .env # Plaintext secrets (git-ignored) + └── .env.azure # Plaintext secrets (git-ignored) +``` + +### After +``` +usda-dash-config/ +├── default.nix # Now references usda-vision flake +└── usda-vision/ # Git submodule with its own flake + ├── flake.nix # Flake definition + ├── package.nix # Application build + ├── camera-sdk.nix # Camera SDK build + ├── secrets.nix # ragenix config + └── secrets/ + ├── secrets.nix # Public keys + ├── env.age # Encrypted secrets (safe to commit) + └── env.azure.age # Encrypted secrets (safe to commit) +``` + +## Key Improvements + +### 1. Self-Contained Flake +- **Standalone Development**: The `usda-vision` directory now has its own complete build and development environment +- **Independent Versioning**: Can pin dependencies independently from parent project +- **Reusable**: Can be used in other projects via flake inputs + +### 2. Development Shell +```bash +cd usda-vision +nix develop +``` + +Provides: +- Docker & Docker Compose +- Node.js 20 (npm, pnpm) +- Python 3.11 (pip, virtualenv) +- Supabase CLI +- Camera SDK (pre-configured) +- ragenix for secrets +- All utilities (jq, yq, rsync, etc.) + +### 3. Encrypted Secrets with ragenix +- **Git-Safe**: Encrypted `.age` files can be committed safely +- **Key-Based Access**: Only users with the right age/SSH keys can decrypt +- **No Manual Distribution**: Secrets are in the repo (encrypted) +- **Audit Trail**: Changes to secrets tracked in git history + +### 4. NixOS Module +The flake exports a NixOS module for easy deployment: + +```nix +{ + imports = [ inputs.usda-vision.nixosModules.default ]; + + services.usda-vision = { + enable = true; + secretsFile = config.age.secrets.usda-vision-env.path; + }; +} +``` + +## Files Created + +### In `usda-vision/`: +- [`flake.nix`](usda-vision/flake.nix) - Main flake definition +- [`package.nix`](usda-vision/package.nix) - Application package +- [`camera-sdk.nix`](usda-vision/camera-sdk.nix) - Camera SDK package +- [`secrets.nix`](usda-vision/secrets.nix) - ragenix configuration +- [`FLAKE_SETUP.md`](usda-vision/FLAKE_SETUP.md) - Complete documentation +- [`setup-dev.sh`](usda-vision/setup-dev.sh) - Quick setup script + +### In `usda-vision/secrets/`: +- [`secrets.nix`](usda-vision/secrets/secrets.nix) - Public key list +- [`README.md`](usda-vision/secrets/README.md) - Secrets documentation +- [`.gitignore`](usda-vision/secrets/.gitignore) - Protect unencrypted files + +## Files Modified + +### [`default.nix`](default.nix) +- Removed inline build derivations (200+ lines) +- Now imports packages from usda-vision flake +- Cleaner, more maintainable + +### [`usda-vision/.gitignore`](usda-vision/.gitignore) +- Added Nix-specific ignores (`result`, `.direnv/`) + +## Migration Guide + +### For Developers + +1. **Setup age key** (one-time): + ```bash + cd usda-vision + ./setup-dev.sh # Interactive setup + ``` + +2. **Enter dev environment**: + ```bash + nix develop + ``` + +3. **Encrypt secrets** (if you have them): + ```bash + ragenix -e secrets/env.age + # Paste your .env contents, save, exit + ``` + +### For System Administrators + +1. **Generate age key** on the deployment server: + ```bash + mkdir -p /root/.config/age + age-keygen -o /root/.config/age/keys.txt + ``` + +2. **Add public key** to `usda-vision/secrets/secrets.nix` + +3. **Re-encrypt secrets** (from your dev machine): + ```bash + cd usda-vision + nix develop + ragenix -r # Re-key all secrets + git commit secrets/ -m "Add server key" + git push + ``` + +4. **Configure NixOS module** (see FLAKE_SETUP.md) + +## Testing + +### Build the package: +```bash +cd usda-vision +nix build +``` + +### Test development shell: +```bash +nix develop +# Should see welcome message with available commands +``` + +### Test from parent directory: +```bash +cd usda-dash-config +nix build .#usda-vision # If default.nix is updated +``` + +## Rollback Plan + +If needed, revert to the previous setup: + +```bash +git revert +``` + +The old inline derivations are preserved in git history. + +## Security Considerations + +### Secrets in Git +- ✅ Encrypted files (`.age`) are safe to commit +- ❌ Never commit `.env` or unencrypted secrets +- ✅ `.gitignore` is configured to prevent accidents +- ✅ Age encryption is secure (modern crypto) + +### Key Management +- Private keys stay on developer machines +- Public keys are in the repo +- Server keys generated on-server (never transmitted) +- Multiple keys can decrypt the same secrets (team access) + +### Migration of Existing Secrets +- Old `.env` files should be deleted after encryption +- Verify `.gitignore` is working: `git status` +- Consider rotating sensitive credentials after migration + +## Benefits Summary + +| Aspect | Before | After | +|--------|--------|-------| +| **Build** | Inline in parent | Self-contained flake | +| **Dev Environment** | Manual setup | `nix develop` | +| **Secrets** | Plaintext, git-ignored | Encrypted, in git | +| **Dependencies** | Parent's nixpkgs | Pinned in flake.lock | +| **Reusability** | Coupled to parent | Standalone | +| **Documentation** | Scattered | Centralized | + +## Next Steps + +1. **Encrypt existing secrets** + - Run `setup-dev.sh` in usda-vision + - Encrypt `.env` files with ragenix + - Delete plaintext versions + +2. **Test locally** + - `nix develop` in usda-vision + - Verify all tools available + - Test docker-compose workflow + +3. **Update CI/CD** (if applicable) + - Add age keys to CI secrets + - Update build commands to use flake + +4. **Team onboarding** + - Share FLAKE_SETUP.md + - Collect public keys + - Add to secrets.nix + +## Questions? + +See [FLAKE_SETUP.md](usda-vision/FLAKE_SETUP.md) for detailed documentation. diff --git a/default.nix b/default.nix index 4657f0b..a53a951 100644 --- a/default.nix +++ b/default.nix @@ -30,157 +30,12 @@ }: let - # Camera SDK derivation - extracts and installs the SDK - camera-sdk = pkgs.stdenv.mkDerivation { - pname = "mindvision-camera-sdk"; - version = "2.1.0.49"; - - # Use the camera_sdk directory as source - src = ./usda-vision/camera-management-api/camera_sdk; - - nativeBuildInputs = [ pkgs.makeWrapper ]; - buildInputs = [ pkgs.libusb1 ]; - - unpackPhase = '' - cp -r $src/* . - tar xzf "linuxSDK_V2.1.0.49(250108).tar.gz" - cd "linuxSDK_V2.1.0.49(250108)" - ''; - - installPhase = '' - mkdir -p $out/lib $out/include - - # Copy x64 library files (SDK has arch-specific subdirs) - if [ -d lib/x64 ]; then - cp -r lib/x64/* $out/lib/ || true - fi - - # Copy header files - if [ -d include ]; then - cp -r include/* $out/include/ || true - fi - - # Make libraries executable - chmod +x $out/lib/*.so* 2>/dev/null || true - ''; - - meta = { - description = "MindVision Camera SDK"; - platforms = pkgs.lib.platforms.linux; - }; - }; + # Import the usda-vision flake + usda-vision-flake = (builtins.getFlake (toString ./usda-vision)).packages.${pkgs.system}; - # Create a derivation that packages the usda-vision directory - usda-vision-app = pkgs.stdenv.mkDerivation { - pname = "usda-vision"; - version = "1.0.0"; - - # Use the directory from this repository with explicit source filtering - # The ./usda-vision path is relative to this module file - src = lib.cleanSourceWith { - src = ./usda-vision; - filter = path: type: - let - baseName = baseNameOf path; - in - # Exclude git, but include everything else - baseName != ".git" && - baseName != ".cursor" && - baseName != "__pycache__" && - baseName != "node_modules" && - baseName != ".venv"; - }; - - nativeBuildInputs = [ pkgs.makeWrapper pkgs.rsync ]; - - # Don't run these phases, we'll do everything in installPhase - dontBuild = true; - dontConfigure = true; - - installPhase = '' - mkdir -p $out/opt/usda-vision - - # Debug: show what's in source - echo "Source directory contents:" - ls -la $src/ || true - - # Process docker-compose.yml - replace paths, hostnames, and configure SDK from Nix - if [ -f $src/docker-compose.yml ]; then - # Basic path and hostname replacements with sed - ${pkgs.gnused}/bin/sed \ - -e 's|env_file:.*management-dashboard-web-app/\.env|env_file: /var/lib/usda-vision/.env|g' \ - -e 's|\./management-dashboard-web-app/\.env|/var/lib/usda-vision/.env|g' \ - -e 's|\./management-dashboard-web-app|/var/lib/usda-vision/management-dashboard-web-app|g' \ - -e 's|\./media-api|/var/lib/usda-vision/media-api|g' \ - -e 's|\./video-remote|/var/lib/usda-vision/video-remote|g' \ - -e 's|\./scheduling-remote|/var/lib/usda-vision/scheduling-remote|g' \ - -e 's|\./vision-system-remote|/var/lib/usda-vision/vision-system-remote|g' \ - -e 's|\./camera-management-api|/var/lib/usda-vision/camera-management-api|g' \ - -e 's|exp-dash|192.168.1.156|g' \ - -e 's|localhost|192.168.1.156|g' \ - -e '/^ - \/etc\/timezone:\/etc\/timezone:ro$/a\ - ${camera-sdk}/lib:/opt/camera-sdk/lib:ro' \ - -e 's|LD_LIBRARY_PATH=/usr/local/lib:/lib:/usr/lib|LD_LIBRARY_PATH=/opt/camera-sdk/lib:/usr/local/lib:/lib:/usr/lib|' \ - $src/docker-compose.yml > $TMPDIR/docker-compose-step1.yml - - # Remove SDK installation blocks using awk for better multi-line handling - ${pkgs.gawk}/bin/awk ' - /# Only install system packages if not already installed/ { skip=1 } - skip && /^ fi$/ { skip=0; next } - /# Install camera SDK if not already installed/ { skip_sdk=1 } - skip_sdk && /^ fi;$/ { skip_sdk=0; next } - !skip && !skip_sdk { print } - ' $TMPDIR/docker-compose-step1.yml > $TMPDIR/docker-compose.yml - - rm -f $TMPDIR/docker-compose-step1.yml - fi - - # Copy all application files using rsync with chmod, excluding files we'll provide separately - ${pkgs.rsync}/bin/rsync -av --chmod=Du+w --exclude='.git' --exclude='docker-compose.yml' --exclude='.env' --exclude='management-dashboard-web-app/.env' $src/ $out/opt/usda-vision/ - - # Copy the processed docker-compose.yml - if [ -f $TMPDIR/docker-compose.yml ]; then - cp $TMPDIR/docker-compose.yml $out/opt/usda-vision/docker-compose.yml - fi - - # Verify files were copied - echo "Destination directory contents:" - ls -la $out/opt/usda-vision/ || true - - # Create convenience scripts - mkdir -p $out/bin - - cat > $out/bin/usda-vision-start <<'EOF' - #!/usr/bin/env bash - cd $out/opt/usda-vision - ${pkgs.docker-compose}/bin/docker-compose up -d --build - EOF - - cat > $out/bin/usda-vision-stop <<'EOF' - #!/usr/bin/env bash - cd $out/opt/usda-vision - ${pkgs.docker-compose}/bin/docker-compose down - EOF - - cat > $out/bin/usda-vision-logs <<'EOF' - #!/usr%bin/env bash - cd $out/opt/usda-vision - ${pkgs.docker-compose}/bin/docker-compose logs -f "$@" - EOF - - cat > $out/bin/usda-vision-restart <<'EOF' - #!/usr/bin/env bash - cd $out/opt/usda-vision - ${pkgs.docker-compose}/bin/docker-compose restart "$@" - EOF - - chmod +x $out/bin/usda-vision-* - ''; - - meta = { - description = "USDA Vision camera management system"; - maintainers = [ "UGA Innovation Factory" ]; - }; - }; + # Get packages from the flake + camera-sdk = usda-vision-flake.camera-sdk; + usda-vision-app = usda-vision-flake.usda-vision; in { diff --git a/usda-vision b/usda-vision index 8308415..b384f0f 160000 --- a/usda-vision +++ b/usda-vision @@ -1 +1 @@ -Subproject commit 83084158b54ab8149981b5b9572cd8367a896fa6 +Subproject commit b384f0f676e1f5f8d3a5144ea07ff8a87dd20391