athenix/secrets/README.md

Secrets Management with Agenix

This directory contains age-encrypted secrets for Athenix hosts. Secrets are automatically loaded based on directory structure.

Directory Structure

secrets/
├── global/              # Secrets installed on ALL systems
│   ├── default.nix      # Optional: Custom config for global secrets
│   └── example.age      # Decrypted to /run/agenix/example on all hosts
├── nix-builder/         # Secrets only for nix-builder host
│   ├── default.nix      # Optional: Custom config for nix-builder secrets
│   └── ssh_host_ed25519_key.age
└── usda-dash/           # Secrets only for usda-dash host
    └── ssh_host_ed25519_key.age

How It Works

1. Global secrets (./secrets/global/*.age) are installed on every system
2. Host-specific secrets (./secrets/{hostname}/*.age) are only installed on matching hosts
3. Only .age encrypted files are loaded; .pub public keys are ignored
4. Secrets are decrypted at boot to /run/agenix/{secret-name} with mode 0400 and owner root:root
5. Custom configurations can be defined in default.nix files within each directory

Creating Secrets

1. Generate Age Keys

For a new host, generate an age identity:

# On the target system
mkdir -p /etc/age
age-keygen -o /etc/age/identity.key
chmod 600 /etc/age/identity.key

Or use SSH host keys (automatically done by Athenix):

# Get the age public key from SSH host key
nix shell nixpkgs#ssh-to-age -c sh -c 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age'

2. Store Public Keys

Save the public key to secrets/{hostname}/ for reference:

# Example for nix-builder
echo "age1..." > secrets/nix-builder/identity.pub

Or from SSH host key:

cat /etc/ssh/ssh_host_ed25519_key.pub > secrets/nix-builder/ssh_host_ed25519_key.pub

Then convert SSH keys to age format:

cd secrets/
./update-age-keys.sh

This creates .age.pub files that secrets.nix uses for ragenix recipient configuration.

3. Encrypt Secrets

Encrypt a secret for specific hosts:

# For a single host
age -r age1publickey... -o secrets/nix-builder/my-secret.age <<< "secret value"

# For multiple hosts (recipient list)
age -R recipients.txt -o secrets/global/shared-secret.age < plaintext-file

# Using SSH public keys
age -R secrets/nix-builder/ssh_host_ed25519_key.pub \
    -o secrets/nix-builder/ssh_host_key.age < /etc/ssh/ssh_host_ed25519_key

4. Using ragenix CLI (Recommended)

The ragenix CLI tool simplifies secret management. The secrets/secrets.nix file automatically discovers hosts and their keys from the directory structure:

# Install ragenix
nix shell github:yaxitech/ragenix

# Edit a secret (creates if doesn't exist)
# Recipients are automatically determined based on the path:
# - secrets/global/*.age -> encrypted for ALL hosts + admins
# - secrets/{hostname}/*.age -> encrypted for that host + global keys + admins
ragenix -e secrets/global/example.age

# Re-key all secrets after adding/removing hosts
ragenix -r

The secrets.nix file automatically:

• Discovers hosts from directory names in secrets/
• Reads age public keys from .age.pub files in each directory
• Generates recipient lists based on secret location (global vs host-specific)
• Includes admin keys from secrets/admins/*.age.pub for editing

To add admin keys for editing secrets:

# Generate personal age key
age-keygen -o ~/.config/age/personal.key

# Extract public key and add to secrets
grep "public key:" ~/.config/age/personal.key | cut -d: -f2 | tr -d ' ' > secrets/admins/your-name.age.pub

Using Secrets in Configuration

Secrets are automatically loaded. Reference them in your NixOS configuration:

# Example: Using a secret for a service
services.myservice = {
  enable = true;
  passwordFile = config.age.secrets.my-password.path;  # /run/agenix/my-password
};

# Example: Setting up SSH host key from secret
services.openssh = {
  hostKeys = [{
    path = config.age.secrets.ssh_host_ed25519_key.path;
    type = "ed25519";
  }];
};

Custom Secret Configuration

For secrets needing custom permissions, use athenix.sw.secrets.extraSecrets:

# In inventory.nix or host config
athenix.sw.secrets.extraSecrets = {
  "nginx-cert" = {
    file = ./secrets/custom/cert.age;
    mode = "0440";
    owner = "nginx";
    group = "nginx";
  };
};

Using default.nix in Secret Directories

Alternatively, create a default.nix file in the secret directory to configure all secrets in that directory:

# secrets/global/default.nix
{
  "example" = {
    mode = "0440";        # Custom file mode (default: "0400")
    owner = "nginx";      # Custom owner (default: "root")
    group = "nginx";      # Custom group (default: "root")
    path = "/run/secrets/example";  # Custom path (default: /run/agenix/{name})
  };
  
  "api-key" = {
    mode = "0400";
    owner = "myservice";
    group = "myservice";
  };
}

The default.nix file should return an attribute set where:

• Keys are secret names (without the .age extension)
• Values are configuration objects with optional fields:
  • mode - File permissions (string, e.g., "0440")
  • owner - File owner (string, e.g., "nginx")
  • group - File group (string, e.g., "nginx")
  • path - Custom installation path (string, e.g., "/custom/path")

Secrets not listed in default.nix will use default settings.

Security Best Practices

1. Never commit unencrypted secrets - Only .age and .pub files belong in this directory
2. Use host-specific secrets when possible - Limit exposure by using hostname directories
3. Rotate secrets regularly - Re-encrypt with new keys periodically
4. Backup age identity keys - Store /etc/age/identity.key securely offline
5. Use SSH keys - Leverage existing SSH host keys for age encryption when possible
6. Pin to commits - When using external secrets modules, always use rev = "commit-hash"

Converting SSH Keys to Age Format

# Convert SSH public key to age public key
nix shell nixpkgs#ssh-to-age -c ssh-to-age < secrets/nix-builder/ssh_host_ed25519_key.pub

# Convert SSH private key to age identity (for editing secrets)
nix shell nixpkgs#ssh-to-age -c ssh-to-age -private-key -i ~/.ssh/id_ed25519

Disabling Automatic Secrets

To disable automatic secret loading:

# In inventory.nix or host config
athenix.sw.secrets.enable = false;

Troubleshooting

Secret not found

• Ensure the .age file exists in secrets/global/ or secrets/{hostname}/
• Check hostname matches directory name: echo $HOSTNAME on the target system
• Run nix flake check to verify secrets are discovered

Permission denied

• Verify secret permissions in /run/agenix/
• Check if custom permissions are needed (use extraSecrets)
• Ensure the service user/group has access to the secret file

Age decrypt failed

• Verify the host's age identity exists: ls -l /etc/age/identity.key
• Check that the secret was encrypted with the host's public key
• Confirm SSH host key hasn't changed (would change derived age key)

References

• ragenix GitHub
• agenix upstream
• age encryption tool