feat: Add flake and ragenix package generation and dev environment
This commit is contained in:
223
NIX_FLAKE_MIGRATION.md
Normal file
223
NIX_FLAKE_MIGRATION.md
Normal file
@@ -0,0 +1,223 @@
|
||||
# Nix Flake Migration Summary
|
||||
|
||||
## What Changed
|
||||
|
||||
The USDA Vision repository has been migrated from inline Nix derivations to a self-contained flake with integrated secrets management.
|
||||
|
||||
### Before
|
||||
```
|
||||
usda-dash-config/
|
||||
├── default.nix # Contains all build logic inline
|
||||
└── usda-vision/ # Git submodule
|
||||
├── .env # Plaintext secrets (git-ignored)
|
||||
└── .env.azure # Plaintext secrets (git-ignored)
|
||||
```
|
||||
|
||||
### After
|
||||
```
|
||||
usda-dash-config/
|
||||
├── default.nix # Now references usda-vision flake
|
||||
└── usda-vision/ # Git submodule with its own flake
|
||||
├── flake.nix # Flake definition
|
||||
├── package.nix # Application build
|
||||
├── camera-sdk.nix # Camera SDK build
|
||||
├── secrets.nix # ragenix config
|
||||
└── secrets/
|
||||
├── secrets.nix # Public keys
|
||||
├── env.age # Encrypted secrets (safe to commit)
|
||||
└── env.azure.age # Encrypted secrets (safe to commit)
|
||||
```
|
||||
|
||||
## Key Improvements
|
||||
|
||||
### 1. Self-Contained Flake
|
||||
- **Standalone Development**: The `usda-vision` directory now has its own complete build and development environment
|
||||
- **Independent Versioning**: Can pin dependencies independently from parent project
|
||||
- **Reusable**: Can be used in other projects via flake inputs
|
||||
|
||||
### 2. Development Shell
|
||||
```bash
|
||||
cd usda-vision
|
||||
nix develop
|
||||
```
|
||||
|
||||
Provides:
|
||||
- Docker & Docker Compose
|
||||
- Node.js 20 (npm, pnpm)
|
||||
- Python 3.11 (pip, virtualenv)
|
||||
- Supabase CLI
|
||||
- Camera SDK (pre-configured)
|
||||
- ragenix for secrets
|
||||
- All utilities (jq, yq, rsync, etc.)
|
||||
|
||||
### 3. Encrypted Secrets with ragenix
|
||||
- **Git-Safe**: Encrypted `.age` files can be committed safely
|
||||
- **Key-Based Access**: Only users with the right age/SSH keys can decrypt
|
||||
- **No Manual Distribution**: Secrets are in the repo (encrypted)
|
||||
- **Audit Trail**: Changes to secrets tracked in git history
|
||||
|
||||
### 4. NixOS Module
|
||||
The flake exports a NixOS module for easy deployment:
|
||||
|
||||
```nix
|
||||
{
|
||||
imports = [ inputs.usda-vision.nixosModules.default ];
|
||||
|
||||
services.usda-vision = {
|
||||
enable = true;
|
||||
secretsFile = config.age.secrets.usda-vision-env.path;
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
## Files Created
|
||||
|
||||
### In `usda-vision/`:
|
||||
- [`flake.nix`](usda-vision/flake.nix) - Main flake definition
|
||||
- [`package.nix`](usda-vision/package.nix) - Application package
|
||||
- [`camera-sdk.nix`](usda-vision/camera-sdk.nix) - Camera SDK package
|
||||
- [`secrets.nix`](usda-vision/secrets.nix) - ragenix configuration
|
||||
- [`FLAKE_SETUP.md`](usda-vision/FLAKE_SETUP.md) - Complete documentation
|
||||
- [`setup-dev.sh`](usda-vision/setup-dev.sh) - Quick setup script
|
||||
|
||||
### In `usda-vision/secrets/`:
|
||||
- [`secrets.nix`](usda-vision/secrets/secrets.nix) - Public key list
|
||||
- [`README.md`](usda-vision/secrets/README.md) - Secrets documentation
|
||||
- [`.gitignore`](usda-vision/secrets/.gitignore) - Protect unencrypted files
|
||||
|
||||
## Files Modified
|
||||
|
||||
### [`default.nix`](default.nix)
|
||||
- Removed inline build derivations (200+ lines)
|
||||
- Now imports packages from usda-vision flake
|
||||
- Cleaner, more maintainable
|
||||
|
||||
### [`usda-vision/.gitignore`](usda-vision/.gitignore)
|
||||
- Added Nix-specific ignores (`result`, `.direnv/`)
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### For Developers
|
||||
|
||||
1. **Setup age key** (one-time):
|
||||
```bash
|
||||
cd usda-vision
|
||||
./setup-dev.sh # Interactive setup
|
||||
```
|
||||
|
||||
2. **Enter dev environment**:
|
||||
```bash
|
||||
nix develop
|
||||
```
|
||||
|
||||
3. **Encrypt secrets** (if you have them):
|
||||
```bash
|
||||
ragenix -e secrets/env.age
|
||||
# Paste your .env contents, save, exit
|
||||
```
|
||||
|
||||
### For System Administrators
|
||||
|
||||
1. **Generate age key** on the deployment server:
|
||||
```bash
|
||||
mkdir -p /root/.config/age
|
||||
age-keygen -o /root/.config/age/keys.txt
|
||||
```
|
||||
|
||||
2. **Add public key** to `usda-vision/secrets/secrets.nix`
|
||||
|
||||
3. **Re-encrypt secrets** (from your dev machine):
|
||||
```bash
|
||||
cd usda-vision
|
||||
nix develop
|
||||
ragenix -r # Re-key all secrets
|
||||
git commit secrets/ -m "Add server key"
|
||||
git push
|
||||
```
|
||||
|
||||
4. **Configure NixOS module** (see FLAKE_SETUP.md)
|
||||
|
||||
## Testing
|
||||
|
||||
### Build the package:
|
||||
```bash
|
||||
cd usda-vision
|
||||
nix build
|
||||
```
|
||||
|
||||
### Test development shell:
|
||||
```bash
|
||||
nix develop
|
||||
# Should see welcome message with available commands
|
||||
```
|
||||
|
||||
### Test from parent directory:
|
||||
```bash
|
||||
cd usda-dash-config
|
||||
nix build .#usda-vision # If default.nix is updated
|
||||
```
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If needed, revert to the previous setup:
|
||||
|
||||
```bash
|
||||
git revert <commit-hash>
|
||||
```
|
||||
|
||||
The old inline derivations are preserved in git history.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Secrets in Git
|
||||
- ✅ Encrypted files (`.age`) are safe to commit
|
||||
- ❌ Never commit `.env` or unencrypted secrets
|
||||
- ✅ `.gitignore` is configured to prevent accidents
|
||||
- ✅ Age encryption is secure (modern crypto)
|
||||
|
||||
### Key Management
|
||||
- Private keys stay on developer machines
|
||||
- Public keys are in the repo
|
||||
- Server keys generated on-server (never transmitted)
|
||||
- Multiple keys can decrypt the same secrets (team access)
|
||||
|
||||
### Migration of Existing Secrets
|
||||
- Old `.env` files should be deleted after encryption
|
||||
- Verify `.gitignore` is working: `git status`
|
||||
- Consider rotating sensitive credentials after migration
|
||||
|
||||
## Benefits Summary
|
||||
|
||||
| Aspect | Before | After |
|
||||
|--------|--------|-------|
|
||||
| **Build** | Inline in parent | Self-contained flake |
|
||||
| **Dev Environment** | Manual setup | `nix develop` |
|
||||
| **Secrets** | Plaintext, git-ignored | Encrypted, in git |
|
||||
| **Dependencies** | Parent's nixpkgs | Pinned in flake.lock |
|
||||
| **Reusability** | Coupled to parent | Standalone |
|
||||
| **Documentation** | Scattered | Centralized |
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Encrypt existing secrets**
|
||||
- Run `setup-dev.sh` in usda-vision
|
||||
- Encrypt `.env` files with ragenix
|
||||
- Delete plaintext versions
|
||||
|
||||
2. **Test locally**
|
||||
- `nix develop` in usda-vision
|
||||
- Verify all tools available
|
||||
- Test docker-compose workflow
|
||||
|
||||
3. **Update CI/CD** (if applicable)
|
||||
- Add age keys to CI secrets
|
||||
- Update build commands to use flake
|
||||
|
||||
4. **Team onboarding**
|
||||
- Share FLAKE_SETUP.md
|
||||
- Collect public keys
|
||||
- Add to secrets.nix
|
||||
|
||||
## Questions?
|
||||
|
||||
See [FLAKE_SETUP.md](usda-vision/FLAKE_SETUP.md) for detailed documentation.
|
||||
155
default.nix
155
default.nix
@@ -30,157 +30,12 @@
|
||||
}:
|
||||
|
||||
let
|
||||
# Camera SDK derivation - extracts and installs the SDK
|
||||
camera-sdk = pkgs.stdenv.mkDerivation {
|
||||
pname = "mindvision-camera-sdk";
|
||||
version = "2.1.0.49";
|
||||
|
||||
# Use the camera_sdk directory as source
|
||||
src = ./usda-vision/camera-management-api/camera_sdk;
|
||||
|
||||
nativeBuildInputs = [ pkgs.makeWrapper ];
|
||||
buildInputs = [ pkgs.libusb1 ];
|
||||
|
||||
unpackPhase = ''
|
||||
cp -r $src/* .
|
||||
tar xzf "linuxSDK_V2.1.0.49(250108).tar.gz"
|
||||
cd "linuxSDK_V2.1.0.49(250108)"
|
||||
'';
|
||||
|
||||
installPhase = ''
|
||||
mkdir -p $out/lib $out/include
|
||||
|
||||
# Copy x64 library files (SDK has arch-specific subdirs)
|
||||
if [ -d lib/x64 ]; then
|
||||
cp -r lib/x64/* $out/lib/ || true
|
||||
fi
|
||||
|
||||
# Copy header files
|
||||
if [ -d include ]; then
|
||||
cp -r include/* $out/include/ || true
|
||||
fi
|
||||
|
||||
# Make libraries executable
|
||||
chmod +x $out/lib/*.so* 2>/dev/null || true
|
||||
'';
|
||||
|
||||
meta = {
|
||||
description = "MindVision Camera SDK";
|
||||
platforms = pkgs.lib.platforms.linux;
|
||||
};
|
||||
};
|
||||
# Import the usda-vision flake
|
||||
usda-vision-flake = (builtins.getFlake (toString ./usda-vision)).packages.${pkgs.system};
|
||||
|
||||
# Create a derivation that packages the usda-vision directory
|
||||
usda-vision-app = pkgs.stdenv.mkDerivation {
|
||||
pname = "usda-vision";
|
||||
version = "1.0.0";
|
||||
|
||||
# Use the directory from this repository with explicit source filtering
|
||||
# The ./usda-vision path is relative to this module file
|
||||
src = lib.cleanSourceWith {
|
||||
src = ./usda-vision;
|
||||
filter = path: type:
|
||||
let
|
||||
baseName = baseNameOf path;
|
||||
in
|
||||
# Exclude git, but include everything else
|
||||
baseName != ".git" &&
|
||||
baseName != ".cursor" &&
|
||||
baseName != "__pycache__" &&
|
||||
baseName != "node_modules" &&
|
||||
baseName != ".venv";
|
||||
};
|
||||
|
||||
nativeBuildInputs = [ pkgs.makeWrapper pkgs.rsync ];
|
||||
|
||||
# Don't run these phases, we'll do everything in installPhase
|
||||
dontBuild = true;
|
||||
dontConfigure = true;
|
||||
|
||||
installPhase = ''
|
||||
mkdir -p $out/opt/usda-vision
|
||||
|
||||
# Debug: show what's in source
|
||||
echo "Source directory contents:"
|
||||
ls -la $src/ || true
|
||||
|
||||
# Process docker-compose.yml - replace paths, hostnames, and configure SDK from Nix
|
||||
if [ -f $src/docker-compose.yml ]; then
|
||||
# Basic path and hostname replacements with sed
|
||||
${pkgs.gnused}/bin/sed \
|
||||
-e 's|env_file:.*management-dashboard-web-app/\.env|env_file: /var/lib/usda-vision/.env|g' \
|
||||
-e 's|\./management-dashboard-web-app/\.env|/var/lib/usda-vision/.env|g' \
|
||||
-e 's|\./management-dashboard-web-app|/var/lib/usda-vision/management-dashboard-web-app|g' \
|
||||
-e 's|\./media-api|/var/lib/usda-vision/media-api|g' \
|
||||
-e 's|\./video-remote|/var/lib/usda-vision/video-remote|g' \
|
||||
-e 's|\./scheduling-remote|/var/lib/usda-vision/scheduling-remote|g' \
|
||||
-e 's|\./vision-system-remote|/var/lib/usda-vision/vision-system-remote|g' \
|
||||
-e 's|\./camera-management-api|/var/lib/usda-vision/camera-management-api|g' \
|
||||
-e 's|exp-dash|192.168.1.156|g' \
|
||||
-e 's|localhost|192.168.1.156|g' \
|
||||
-e '/^ - \/etc\/timezone:\/etc\/timezone:ro$/a\ - ${camera-sdk}/lib:/opt/camera-sdk/lib:ro' \
|
||||
-e 's|LD_LIBRARY_PATH=/usr/local/lib:/lib:/usr/lib|LD_LIBRARY_PATH=/opt/camera-sdk/lib:/usr/local/lib:/lib:/usr/lib|' \
|
||||
$src/docker-compose.yml > $TMPDIR/docker-compose-step1.yml
|
||||
|
||||
# Remove SDK installation blocks using awk for better multi-line handling
|
||||
${pkgs.gawk}/bin/awk '
|
||||
/# Only install system packages if not already installed/ { skip=1 }
|
||||
skip && /^ fi$/ { skip=0; next }
|
||||
/# Install camera SDK if not already installed/ { skip_sdk=1 }
|
||||
skip_sdk && /^ fi;$/ { skip_sdk=0; next }
|
||||
!skip && !skip_sdk { print }
|
||||
' $TMPDIR/docker-compose-step1.yml > $TMPDIR/docker-compose.yml
|
||||
|
||||
rm -f $TMPDIR/docker-compose-step1.yml
|
||||
fi
|
||||
|
||||
# Copy all application files using rsync with chmod, excluding files we'll provide separately
|
||||
${pkgs.rsync}/bin/rsync -av --chmod=Du+w --exclude='.git' --exclude='docker-compose.yml' --exclude='.env' --exclude='management-dashboard-web-app/.env' $src/ $out/opt/usda-vision/
|
||||
|
||||
# Copy the processed docker-compose.yml
|
||||
if [ -f $TMPDIR/docker-compose.yml ]; then
|
||||
cp $TMPDIR/docker-compose.yml $out/opt/usda-vision/docker-compose.yml
|
||||
fi
|
||||
|
||||
# Verify files were copied
|
||||
echo "Destination directory contents:"
|
||||
ls -la $out/opt/usda-vision/ || true
|
||||
|
||||
# Create convenience scripts
|
||||
mkdir -p $out/bin
|
||||
|
||||
cat > $out/bin/usda-vision-start <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
cd $out/opt/usda-vision
|
||||
${pkgs.docker-compose}/bin/docker-compose up -d --build
|
||||
EOF
|
||||
|
||||
cat > $out/bin/usda-vision-stop <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
cd $out/opt/usda-vision
|
||||
${pkgs.docker-compose}/bin/docker-compose down
|
||||
EOF
|
||||
|
||||
cat > $out/bin/usda-vision-logs <<'EOF'
|
||||
#!/usr%bin/env bash
|
||||
cd $out/opt/usda-vision
|
||||
${pkgs.docker-compose}/bin/docker-compose logs -f "$@"
|
||||
EOF
|
||||
|
||||
cat > $out/bin/usda-vision-restart <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
cd $out/opt/usda-vision
|
||||
${pkgs.docker-compose}/bin/docker-compose restart "$@"
|
||||
EOF
|
||||
|
||||
chmod +x $out/bin/usda-vision-*
|
||||
'';
|
||||
|
||||
meta = {
|
||||
description = "USDA Vision camera management system";
|
||||
maintainers = [ "UGA Innovation Factory" ];
|
||||
};
|
||||
};
|
||||
# Get packages from the flake
|
||||
camera-sdk = usda-vision-flake.camera-sdk;
|
||||
usda-vision-app = usda-vision-flake.usda-vision;
|
||||
in
|
||||
|
||||
{
|
||||
|
||||
Submodule usda-vision updated: 83084158b5...b384f0f676
Reference in New Issue
Block a user