athenix/secrets/README.md

# Secrets Management with Agenix

This directory contains age-encrypted secrets for Athenix hosts. Secrets are automatically loaded based on directory structure.

## Directory Structure

```
secrets/
├── global/              # Secrets installed on ALL systems
│   ├── default.nix      # Optional: Custom config for global secrets
│   └── example.age      # Decrypted to /run/agenix/example on all hosts
├── nix-builder/         # Secrets only for nix-builder host
│   ├── default.nix      # Optional: Custom config for nix-builder secrets
│   └── ssh_host_ed25519_key.age
└── usda-dash/           # Secrets only for usda-dash host
    └── ssh_host_ed25519_key.age
```

## How It Works

1. **Global secrets** (`./secrets/global/*.age`) are installed on every system
2. **Host-specific secrets** (`./secrets/{hostname}/*.age`) are only installed on matching hosts
3. Only `.age` encrypted files are loaded; `.pub` public keys are ignored
4. Secrets are decrypted at boot to `/run/agenix/{secret-name}` with mode `0400` and owner `root:root`
5. **Custom configurations** can be defined in `default.nix` files within each directory

## Creating Secrets

### 1. Generate Age Keys

For a new host, generate an age identity:
```bash
# On the target system
mkdir -p /etc/age
age-keygen -o /etc/age/identity.key
chmod 600 /etc/age/identity.key
```

Or use SSH host keys (automatically done by Athenix):
```bash
# Get the age public key from SSH host key
nix shell nixpkgs#ssh-to-age -c sh -c 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age'
```

### 2. Store Public Keys

Save the public key to `secrets/{hostname}/` for reference:
```bash
# Example for nix-builder
echo "age1..." > secrets/nix-builder/identity.pub
```

Or from SSH host key:
```bash
cat /etc/ssh/ssh_host_ed25519_key.pub > secrets/nix-builder/ssh_host_ed25519_key.pub
```

**Then convert SSH keys to age format:**
```bash
cd secrets/
./update-age-keys.sh
```

This creates `.age.pub` files that `secrets.nix` uses for ragenix recipient configuration.

### 3. Encrypt Secrets

Encrypt a secret for specific hosts:
```bash
# For a single host
age -r age1publickey... -o secrets/nix-builder/my-secret.age <<< "secret value"

# For multiple hosts (recipient list)
age -R recipients.txt -o secrets/global/shared-secret.age < plaintext-file

# Using SSH public keys
age -R secrets/nix-builder/ssh_host_ed25519_key.pub \
    -o secrets/nix-builder/ssh_host_key.age < /etc/ssh/ssh_host_ed25519_key
```

### 4. Creating and Editing Secrets

**For new secrets**, use the helper script (automatically determines recipients):

```bash
cd secrets/

# Create a host-specific secret
./create-secret.sh usda-dash/database-url.age <<< "postgresql://..."

# Create a global secret
echo "shared-api-key" | ./create-secret.sh global/api-key.age

# From a file
./create-secret.sh nix-builder/ssh-key.age < ~/.ssh/id_ed25519
```

The script automatically includes the correct recipients:
- **Host-specific**: that host's keys + global keys + admin keys
- **Global**: all host keys + admin keys

**To edit existing secrets**, use `ragenix`:

```bash
# Install ragenix
nix shell github:yaxitech/ragenix

# Edit an existing secret (you must have a decryption key)
ragenix -e secrets/global/existing-secret.age

# Re-key all secrets after adding new hosts
ragenix -r
```

**Why create with `age` first?** Ragenix requires the `.age` file to exist before editing. The `secrets/secrets.nix` configuration auto-discovers recipients from the directory structure, but ragenix doesn't support wildcard patterns for creating new files.

**Recipient management** is automatic:
- **Global secrets** (`secrets/global/*.age`): encrypted for ALL hosts + admins
- **Host secrets** (`secrets/{hostname}/*.age`): encrypted for that host + global keys + admins
- **Admin keys** from `secrets/admins/*.age.pub` allow editing from your workstation

After creating new .age files with `age`, use `ragenix -r` to re-key all secrets with the updated recipient configuration.

To add admin keys for editing secrets:
```bash
# Generate personal age key
age-keygen -o ~/.config/age/personal.key

# Extract public key and add to secrets
grep "public key:" ~/.config/age/personal.key | cut -d: -f2 | tr -d ' ' > secrets/admins/your-name.age.pub
```

## Using Secrets in Configuration

Secrets are automatically loaded. Reference them in your NixOS configuration:

```nix
# Example: Using a secret for a service
services.myservice = {
  enable = true;
  passwordFile = config.age.secrets.my-password.path;  # /run/agenix/my-password
};

# Example: Setting up SSH host key from secret
services.openssh = {
  hostKeys = [{
    path = config.age.secrets.ssh_host_ed25519_key.path;
    type = "ed25519";
  }];
};
```

## Custom Secret Configuration

For secrets needing custom permissions, use `athenix.sw.secrets.extraSecrets`:

```nix
# In inventory.nix or host config
athenix.sw.secrets.extraSecrets = {
  "nginx-cert" = {
    file = ./secrets/custom/cert.age;
    mode = "0440";
    owner = "nginx";
    group = "nginx";
  };
};
```

### Using default.nix in Secret Directories

Alternatively, create a `default.nix` file in the secret directory to configure all secrets in that directory:

```nix
# secrets/global/default.nix
{
  "example" = {
    mode = "0440";        # Custom file mode (default: "0400")
    owner = "nginx";      # Custom owner (default: "root")
    group = "nginx";      # Custom group (default: "root")
    path = "/run/secrets/example";  # Custom path (default: /run/agenix/{name})
  };
  
  "api-key" = {
    mode = "0400";
    owner = "myservice";
    group = "myservice";
  };
}
```

The `default.nix` file should return an attribute set where:
- **Keys** are secret names (without the `.age` extension)
- **Values** are configuration objects with optional fields:
  - `mode` - File permissions (string, e.g., `"0440"`)
  - `owner` - File owner (string, e.g., `"nginx"`)
  - `group` - File group (string, e.g., `"nginx"`)
  - `path` - Custom installation path (string, e.g., `"/custom/path"`)
  
Secrets not listed in `default.nix` will use default settings.

## Security Best Practices

1. **Never commit unencrypted secrets** - Only `.age` and `.pub` files belong in this directory
2. **Use host-specific secrets** when possible - Limit exposure by using hostname directories
3. **Rotate secrets regularly** - Re-encrypt with new keys periodically
4. **Backup age identity keys** - Store `/etc/age/identity.key` securely offline
5. **Use SSH keys** - Leverage existing SSH host keys for age encryption when possible
6. **Pin to commits** - When using external secrets modules, always use `rev = "commit-hash"`

## Converting SSH Keys to Age Format

```bash
# Convert SSH public key to age public key
nix shell nixpkgs#ssh-to-age -c ssh-to-age < secrets/nix-builder/ssh_host_ed25519_key.pub

# Convert SSH private key to age identity (for editing secrets)
nix shell nixpkgs#ssh-to-age -c ssh-to-age -private-key -i ~/.ssh/id_ed25519
```

## Disabling Automatic Secrets

To disable automatic secret loading:

```nix
# In inventory.nix or host config
athenix.sw.secrets.enable = false;
```

## Troubleshooting

### Secret not found
- Ensure the `.age` file exists in `secrets/global/` or `secrets/{hostname}/`
- Check `hostname` matches directory name: `echo $HOSTNAME` on the target system
- Run `nix flake check` to verify secrets are discovered

### Permission denied
- Verify secret permissions in `/run/agenix/`
- Check if custom permissions are needed (use `extraSecrets`)
- Ensure the service user/group has access to the secret file

### Age decrypt failed
- Verify the host's age identity exists: `ls -l /etc/age/identity.key`
- Check that the secret was encrypted with the host's public key
- Confirm SSH host key hasn't changed (would change derived age key)

## References

- [ragenix GitHub](https://github.com/yaxitech/ragenix)
- [agenix upstream](https://github.com/ryantm/agenix)
- [age encryption tool](https://age-encryption.org/)