feat: Add flake and ragenix package generation and dev environment
This commit is contained in:
223
NIX_FLAKE_MIGRATION.md
Normal file
223
NIX_FLAKE_MIGRATION.md
Normal file
@@ -0,0 +1,223 @@
|
|||||||
|
# Nix Flake Migration Summary
|
||||||
|
|
||||||
|
## What Changed
|
||||||
|
|
||||||
|
The USDA Vision repository has been migrated from inline Nix derivations to a self-contained flake with integrated secrets management.
|
||||||
|
|
||||||
|
### Before
|
||||||
|
```
|
||||||
|
usda-dash-config/
|
||||||
|
├── default.nix # Contains all build logic inline
|
||||||
|
└── usda-vision/ # Git submodule
|
||||||
|
├── .env # Plaintext secrets (git-ignored)
|
||||||
|
└── .env.azure # Plaintext secrets (git-ignored)
|
||||||
|
```
|
||||||
|
|
||||||
|
### After
|
||||||
|
```
|
||||||
|
usda-dash-config/
|
||||||
|
├── default.nix # Now references usda-vision flake
|
||||||
|
└── usda-vision/ # Git submodule with its own flake
|
||||||
|
├── flake.nix # Flake definition
|
||||||
|
├── package.nix # Application build
|
||||||
|
├── camera-sdk.nix # Camera SDK build
|
||||||
|
├── secrets.nix # ragenix config
|
||||||
|
└── secrets/
|
||||||
|
├── secrets.nix # Public keys
|
||||||
|
├── env.age # Encrypted secrets (safe to commit)
|
||||||
|
└── env.azure.age # Encrypted secrets (safe to commit)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key Improvements
|
||||||
|
|
||||||
|
### 1. Self-Contained Flake
|
||||||
|
- **Standalone Development**: The `usda-vision` directory now has its own complete build and development environment
|
||||||
|
- **Independent Versioning**: Can pin dependencies independently from parent project
|
||||||
|
- **Reusable**: Can be used in other projects via flake inputs
|
||||||
|
|
||||||
|
### 2. Development Shell
|
||||||
|
```bash
|
||||||
|
cd usda-vision
|
||||||
|
nix develop
|
||||||
|
```
|
||||||
|
|
||||||
|
Provides:
|
||||||
|
- Docker & Docker Compose
|
||||||
|
- Node.js 20 (npm, pnpm)
|
||||||
|
- Python 3.11 (pip, virtualenv)
|
||||||
|
- Supabase CLI
|
||||||
|
- Camera SDK (pre-configured)
|
||||||
|
- ragenix for secrets
|
||||||
|
- All utilities (jq, yq, rsync, etc.)
|
||||||
|
|
||||||
|
### 3. Encrypted Secrets with ragenix
|
||||||
|
- **Git-Safe**: Encrypted `.age` files can be committed safely
|
||||||
|
- **Key-Based Access**: Only users with the right age/SSH keys can decrypt
|
||||||
|
- **No Manual Distribution**: Secrets are in the repo (encrypted)
|
||||||
|
- **Audit Trail**: Changes to secrets tracked in git history
|
||||||
|
|
||||||
|
### 4. NixOS Module
|
||||||
|
The flake exports a NixOS module for easy deployment:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
{
|
||||||
|
imports = [ inputs.usda-vision.nixosModules.default ];
|
||||||
|
|
||||||
|
services.usda-vision = {
|
||||||
|
enable = true;
|
||||||
|
secretsFile = config.age.secrets.usda-vision-env.path;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Files Created
|
||||||
|
|
||||||
|
### In `usda-vision/`:
|
||||||
|
- [`flake.nix`](usda-vision/flake.nix) - Main flake definition
|
||||||
|
- [`package.nix`](usda-vision/package.nix) - Application package
|
||||||
|
- [`camera-sdk.nix`](usda-vision/camera-sdk.nix) - Camera SDK package
|
||||||
|
- [`secrets.nix`](usda-vision/secrets.nix) - ragenix configuration
|
||||||
|
- [`FLAKE_SETUP.md`](usda-vision/FLAKE_SETUP.md) - Complete documentation
|
||||||
|
- [`setup-dev.sh`](usda-vision/setup-dev.sh) - Quick setup script
|
||||||
|
|
||||||
|
### In `usda-vision/secrets/`:
|
||||||
|
- [`secrets.nix`](usda-vision/secrets/secrets.nix) - Public key list
|
||||||
|
- [`README.md`](usda-vision/secrets/README.md) - Secrets documentation
|
||||||
|
- [`.gitignore`](usda-vision/secrets/.gitignore) - Protect unencrypted files
|
||||||
|
|
||||||
|
## Files Modified
|
||||||
|
|
||||||
|
### [`default.nix`](default.nix)
|
||||||
|
- Removed inline build derivations (200+ lines)
|
||||||
|
- Now imports packages from usda-vision flake
|
||||||
|
- Cleaner, more maintainable
|
||||||
|
|
||||||
|
### [`usda-vision/.gitignore`](usda-vision/.gitignore)
|
||||||
|
- Added Nix-specific ignores (`result`, `.direnv/`)
|
||||||
|
|
||||||
|
## Migration Guide
|
||||||
|
|
||||||
|
### For Developers
|
||||||
|
|
||||||
|
1. **Setup age key** (one-time):
|
||||||
|
```bash
|
||||||
|
cd usda-vision
|
||||||
|
./setup-dev.sh # Interactive setup
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Enter dev environment**:
|
||||||
|
```bash
|
||||||
|
nix develop
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Encrypt secrets** (if you have them):
|
||||||
|
```bash
|
||||||
|
ragenix -e secrets/env.age
|
||||||
|
# Paste your .env contents, save, exit
|
||||||
|
```
|
||||||
|
|
||||||
|
### For System Administrators
|
||||||
|
|
||||||
|
1. **Generate age key** on the deployment server:
|
||||||
|
```bash
|
||||||
|
mkdir -p /root/.config/age
|
||||||
|
age-keygen -o /root/.config/age/keys.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Add public key** to `usda-vision/secrets/secrets.nix`
|
||||||
|
|
||||||
|
3. **Re-encrypt secrets** (from your dev machine):
|
||||||
|
```bash
|
||||||
|
cd usda-vision
|
||||||
|
nix develop
|
||||||
|
ragenix -r # Re-key all secrets
|
||||||
|
git commit secrets/ -m "Add server key"
|
||||||
|
git push
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Configure NixOS module** (see FLAKE_SETUP.md)
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
### Build the package:
|
||||||
|
```bash
|
||||||
|
cd usda-vision
|
||||||
|
nix build
|
||||||
|
```
|
||||||
|
|
||||||
|
### Test development shell:
|
||||||
|
```bash
|
||||||
|
nix develop
|
||||||
|
# Should see welcome message with available commands
|
||||||
|
```
|
||||||
|
|
||||||
|
### Test from parent directory:
|
||||||
|
```bash
|
||||||
|
cd usda-dash-config
|
||||||
|
nix build .#usda-vision # If default.nix is updated
|
||||||
|
```
|
||||||
|
|
||||||
|
## Rollback Plan
|
||||||
|
|
||||||
|
If needed, revert to the previous setup:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git revert <commit-hash>
|
||||||
|
```
|
||||||
|
|
||||||
|
The old inline derivations are preserved in git history.
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
### Secrets in Git
|
||||||
|
- ✅ Encrypted files (`.age`) are safe to commit
|
||||||
|
- ❌ Never commit `.env` or unencrypted secrets
|
||||||
|
- ✅ `.gitignore` is configured to prevent accidents
|
||||||
|
- ✅ Age encryption is secure (modern crypto)
|
||||||
|
|
||||||
|
### Key Management
|
||||||
|
- Private keys stay on developer machines
|
||||||
|
- Public keys are in the repo
|
||||||
|
- Server keys generated on-server (never transmitted)
|
||||||
|
- Multiple keys can decrypt the same secrets (team access)
|
||||||
|
|
||||||
|
### Migration of Existing Secrets
|
||||||
|
- Old `.env` files should be deleted after encryption
|
||||||
|
- Verify `.gitignore` is working: `git status`
|
||||||
|
- Consider rotating sensitive credentials after migration
|
||||||
|
|
||||||
|
## Benefits Summary
|
||||||
|
|
||||||
|
| Aspect | Before | After |
|
||||||
|
|--------|--------|-------|
|
||||||
|
| **Build** | Inline in parent | Self-contained flake |
|
||||||
|
| **Dev Environment** | Manual setup | `nix develop` |
|
||||||
|
| **Secrets** | Plaintext, git-ignored | Encrypted, in git |
|
||||||
|
| **Dependencies** | Parent's nixpkgs | Pinned in flake.lock |
|
||||||
|
| **Reusability** | Coupled to parent | Standalone |
|
||||||
|
| **Documentation** | Scattered | Centralized |
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
1. **Encrypt existing secrets**
|
||||||
|
- Run `setup-dev.sh` in usda-vision
|
||||||
|
- Encrypt `.env` files with ragenix
|
||||||
|
- Delete plaintext versions
|
||||||
|
|
||||||
|
2. **Test locally**
|
||||||
|
- `nix develop` in usda-vision
|
||||||
|
- Verify all tools available
|
||||||
|
- Test docker-compose workflow
|
||||||
|
|
||||||
|
3. **Update CI/CD** (if applicable)
|
||||||
|
- Add age keys to CI secrets
|
||||||
|
- Update build commands to use flake
|
||||||
|
|
||||||
|
4. **Team onboarding**
|
||||||
|
- Share FLAKE_SETUP.md
|
||||||
|
- Collect public keys
|
||||||
|
- Add to secrets.nix
|
||||||
|
|
||||||
|
## Questions?
|
||||||
|
|
||||||
|
See [FLAKE_SETUP.md](usda-vision/FLAKE_SETUP.md) for detailed documentation.
|
||||||
155
default.nix
155
default.nix
@@ -30,157 +30,12 @@
|
|||||||
}:
|
}:
|
||||||
|
|
||||||
let
|
let
|
||||||
# Camera SDK derivation - extracts and installs the SDK
|
# Import the usda-vision flake
|
||||||
camera-sdk = pkgs.stdenv.mkDerivation {
|
usda-vision-flake = (builtins.getFlake (toString ./usda-vision)).packages.${pkgs.system};
|
||||||
pname = "mindvision-camera-sdk";
|
|
||||||
version = "2.1.0.49";
|
|
||||||
|
|
||||||
# Use the camera_sdk directory as source
|
|
||||||
src = ./usda-vision/camera-management-api/camera_sdk;
|
|
||||||
|
|
||||||
nativeBuildInputs = [ pkgs.makeWrapper ];
|
|
||||||
buildInputs = [ pkgs.libusb1 ];
|
|
||||||
|
|
||||||
unpackPhase = ''
|
|
||||||
cp -r $src/* .
|
|
||||||
tar xzf "linuxSDK_V2.1.0.49(250108).tar.gz"
|
|
||||||
cd "linuxSDK_V2.1.0.49(250108)"
|
|
||||||
'';
|
|
||||||
|
|
||||||
installPhase = ''
|
|
||||||
mkdir -p $out/lib $out/include
|
|
||||||
|
|
||||||
# Copy x64 library files (SDK has arch-specific subdirs)
|
|
||||||
if [ -d lib/x64 ]; then
|
|
||||||
cp -r lib/x64/* $out/lib/ || true
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Copy header files
|
|
||||||
if [ -d include ]; then
|
|
||||||
cp -r include/* $out/include/ || true
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Make libraries executable
|
|
||||||
chmod +x $out/lib/*.so* 2>/dev/null || true
|
|
||||||
'';
|
|
||||||
|
|
||||||
meta = {
|
|
||||||
description = "MindVision Camera SDK";
|
|
||||||
platforms = pkgs.lib.platforms.linux;
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# Create a derivation that packages the usda-vision directory
|
# Get packages from the flake
|
||||||
usda-vision-app = pkgs.stdenv.mkDerivation {
|
camera-sdk = usda-vision-flake.camera-sdk;
|
||||||
pname = "usda-vision";
|
usda-vision-app = usda-vision-flake.usda-vision;
|
||||||
version = "1.0.0";
|
|
||||||
|
|
||||||
# Use the directory from this repository with explicit source filtering
|
|
||||||
# The ./usda-vision path is relative to this module file
|
|
||||||
src = lib.cleanSourceWith {
|
|
||||||
src = ./usda-vision;
|
|
||||||
filter = path: type:
|
|
||||||
let
|
|
||||||
baseName = baseNameOf path;
|
|
||||||
in
|
|
||||||
# Exclude git, but include everything else
|
|
||||||
baseName != ".git" &&
|
|
||||||
baseName != ".cursor" &&
|
|
||||||
baseName != "__pycache__" &&
|
|
||||||
baseName != "node_modules" &&
|
|
||||||
baseName != ".venv";
|
|
||||||
};
|
|
||||||
|
|
||||||
nativeBuildInputs = [ pkgs.makeWrapper pkgs.rsync ];
|
|
||||||
|
|
||||||
# Don't run these phases, we'll do everything in installPhase
|
|
||||||
dontBuild = true;
|
|
||||||
dontConfigure = true;
|
|
||||||
|
|
||||||
installPhase = ''
|
|
||||||
mkdir -p $out/opt/usda-vision
|
|
||||||
|
|
||||||
# Debug: show what's in source
|
|
||||||
echo "Source directory contents:"
|
|
||||||
ls -la $src/ || true
|
|
||||||
|
|
||||||
# Process docker-compose.yml - replace paths, hostnames, and configure SDK from Nix
|
|
||||||
if [ -f $src/docker-compose.yml ]; then
|
|
||||||
# Basic path and hostname replacements with sed
|
|
||||||
${pkgs.gnused}/bin/sed \
|
|
||||||
-e 's|env_file:.*management-dashboard-web-app/\.env|env_file: /var/lib/usda-vision/.env|g' \
|
|
||||||
-e 's|\./management-dashboard-web-app/\.env|/var/lib/usda-vision/.env|g' \
|
|
||||||
-e 's|\./management-dashboard-web-app|/var/lib/usda-vision/management-dashboard-web-app|g' \
|
|
||||||
-e 's|\./media-api|/var/lib/usda-vision/media-api|g' \
|
|
||||||
-e 's|\./video-remote|/var/lib/usda-vision/video-remote|g' \
|
|
||||||
-e 's|\./scheduling-remote|/var/lib/usda-vision/scheduling-remote|g' \
|
|
||||||
-e 's|\./vision-system-remote|/var/lib/usda-vision/vision-system-remote|g' \
|
|
||||||
-e 's|\./camera-management-api|/var/lib/usda-vision/camera-management-api|g' \
|
|
||||||
-e 's|exp-dash|192.168.1.156|g' \
|
|
||||||
-e 's|localhost|192.168.1.156|g' \
|
|
||||||
-e '/^ - \/etc\/timezone:\/etc\/timezone:ro$/a\ - ${camera-sdk}/lib:/opt/camera-sdk/lib:ro' \
|
|
||||||
-e 's|LD_LIBRARY_PATH=/usr/local/lib:/lib:/usr/lib|LD_LIBRARY_PATH=/opt/camera-sdk/lib:/usr/local/lib:/lib:/usr/lib|' \
|
|
||||||
$src/docker-compose.yml > $TMPDIR/docker-compose-step1.yml
|
|
||||||
|
|
||||||
# Remove SDK installation blocks using awk for better multi-line handling
|
|
||||||
${pkgs.gawk}/bin/awk '
|
|
||||||
/# Only install system packages if not already installed/ { skip=1 }
|
|
||||||
skip && /^ fi$/ { skip=0; next }
|
|
||||||
/# Install camera SDK if not already installed/ { skip_sdk=1 }
|
|
||||||
skip_sdk && /^ fi;$/ { skip_sdk=0; next }
|
|
||||||
!skip && !skip_sdk { print }
|
|
||||||
' $TMPDIR/docker-compose-step1.yml > $TMPDIR/docker-compose.yml
|
|
||||||
|
|
||||||
rm -f $TMPDIR/docker-compose-step1.yml
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Copy all application files using rsync with chmod, excluding files we'll provide separately
|
|
||||||
${pkgs.rsync}/bin/rsync -av --chmod=Du+w --exclude='.git' --exclude='docker-compose.yml' --exclude='.env' --exclude='management-dashboard-web-app/.env' $src/ $out/opt/usda-vision/
|
|
||||||
|
|
||||||
# Copy the processed docker-compose.yml
|
|
||||||
if [ -f $TMPDIR/docker-compose.yml ]; then
|
|
||||||
cp $TMPDIR/docker-compose.yml $out/opt/usda-vision/docker-compose.yml
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Verify files were copied
|
|
||||||
echo "Destination directory contents:"
|
|
||||||
ls -la $out/opt/usda-vision/ || true
|
|
||||||
|
|
||||||
# Create convenience scripts
|
|
||||||
mkdir -p $out/bin
|
|
||||||
|
|
||||||
cat > $out/bin/usda-vision-start <<'EOF'
|
|
||||||
#!/usr/bin/env bash
|
|
||||||
cd $out/opt/usda-vision
|
|
||||||
${pkgs.docker-compose}/bin/docker-compose up -d --build
|
|
||||||
EOF
|
|
||||||
|
|
||||||
cat > $out/bin/usda-vision-stop <<'EOF'
|
|
||||||
#!/usr/bin/env bash
|
|
||||||
cd $out/opt/usda-vision
|
|
||||||
${pkgs.docker-compose}/bin/docker-compose down
|
|
||||||
EOF
|
|
||||||
|
|
||||||
cat > $out/bin/usda-vision-logs <<'EOF'
|
|
||||||
#!/usr%bin/env bash
|
|
||||||
cd $out/opt/usda-vision
|
|
||||||
${pkgs.docker-compose}/bin/docker-compose logs -f "$@"
|
|
||||||
EOF
|
|
||||||
|
|
||||||
cat > $out/bin/usda-vision-restart <<'EOF'
|
|
||||||
#!/usr/bin/env bash
|
|
||||||
cd $out/opt/usda-vision
|
|
||||||
${pkgs.docker-compose}/bin/docker-compose restart "$@"
|
|
||||||
EOF
|
|
||||||
|
|
||||||
chmod +x $out/bin/usda-vision-*
|
|
||||||
'';
|
|
||||||
|
|
||||||
meta = {
|
|
||||||
description = "USDA Vision camera management system";
|
|
||||||
maintainers = [ "UGA Innovation Factory" ];
|
|
||||||
};
|
|
||||||
};
|
|
||||||
in
|
in
|
||||||
|
|
||||||
{
|
{
|
||||||
|
|||||||
Submodule usda-vision updated: 83084158b5...b384f0f676
Reference in New Issue
Block a user