Refactor Supabase services in docker-compose.yml for better organization and testing

- Commented out all Supabase services to facilitate testing with Supabase CLI. - Updated README to include Supabase directory in project structure. - Adjusted documentation for migration paths in Supabase Docker Compose guide. - Enhanced docker-compose-reset.sh to explicitly remove Supabase volumes and wait for migrations to complete.
2025-12-18 18:27:04 -05:00
parent 93c68768d8
commit 8cb45cbe03
28 changed files with 7264 additions and 253 deletions
--- a/docs/DOCKER_COMPOSE_ORGANIZATION.md
+++ b/docs/DOCKER_COMPOSE_ORGANIZATION.md
@@ -0,0 +1,104 @@
+# Docker Compose Service Organization
+
+## Service Namespaces
+
+Services in the docker-compose.yml are organized using labels to create logical namespaces. This allows for better organization and easier management of related services.
+
+## Supabase Namespace
+
+All Supabase services are grouped under the `com.usda-vision.service=supabase` label namespace:
+
+### Services in Supabase Namespace
+
+- **supabase-db** - PostgreSQL database (group: `database`)
+- **supabase-rest** - PostgREST API (group: `api`)
+- **supabase-auth** - GoTrue authentication (group: `auth`)
+- **supabase-realtime** - Realtime subscriptions (group: `realtime`)
+- **supabase-storage** - Storage API (group: `storage`)
+- **supabase-studio** - Supabase Studio UI (group: `studio`)
+- **supabase-meta** - Database metadata service (group: `meta`)
+- **supabase-migrate** - Migration runner (group: `migration`)
+- **supabase-inbucket** - Email testing server (group: `email`)
+
+### Managing Supabase Services
+
+#### List all Supabase services:
+```bash
+docker compose ps --filter "label=com.usda-vision.service=supabase"
+```
+
+#### View logs for all Supabase services:
+```bash
+docker compose logs supabase-*
+```
+
+#### Stop all Supabase services:
+```bash
+docker compose stop supabase-*
+```
+
+#### Start all Supabase services:
+```bash
+docker compose start supabase-*
+```
+
+#### Restart all Supabase services:
+```bash
+docker compose restart supabase-*
+```
+
+#### View logs for a specific Supabase service group:
+```bash
+# Database services
+docker compose logs supabase-db supabase-migrate
+
+# API services
+docker compose logs supabase-rest supabase-auth
+
+# UI services
+docker compose logs supabase-studio supabase-meta
+```
+
+### Label Structure
+
+Each Supabase service has two labels:
+- `com.usda-vision.service=supabase` - Identifies it as part of the Supabase namespace
+- `com.usda-vision.service.group=<group>` - Identifies the service's functional group
+
+### Benefits
+
+1. **Better Organization**: All Supabase services are visually grouped in the compose file
+2. **Easy Filtering**: Use labels to filter and manage related services
+3. **Clear Ownership**: Makes it obvious which services belong together
+4. **No Behavior Change**: Services still start by default with `docker compose up`
+5. **Flexible Management**: Can manage Supabase services separately when needed
+
+## Future Namespaces
+
+You can extend this pattern to other service groups:
+
+```yaml
+services:
+  api:
+    labels:
+      - "com.usda-vision.service=application"
+      - "com.usda-vision.service.group=api"
+  
+  web:
+    labels:
+      - "com.usda-vision.service=application"
+      - "com.usda-vision.service.group=frontend"
+```
+
+## Visual Organization in docker-compose.yml
+
+The Supabase services are also visually organized with a comment section:
+
+```yaml
+# ============================================================================
+# Supabase Services (Database & Backend)
+# ============================================================================
+```
+
+This makes it easy to find and understand the Supabase services section when viewing the file.
+
--- a/docs/DOCKER_RATE_LIMIT_FIX.md
+++ b/docs/DOCKER_RATE_LIMIT_FIX.md
@@ -0,0 +1,141 @@
+# Fixing Docker Hub Rate Limit Issues
+
+## Problem
+
+When pulling multiple Docker images, you may encounter:
+```
+Error response from daemon: toomanyrequests: Rate exceeded
+```
+
+This happens because Docker Hub limits the number of image pulls for anonymous users.
+
+## Solutions
+
+### Solution 1: Login to Docker Hub (Recommended)
+
+Logging in to Docker Hub increases your rate limit significantly:
+
+```bash
+docker login
+```
+
+Enter your Docker Hub username and password. This increases your rate limit from ~100 pulls per 6 hours to ~200 pulls per 6 hours.
+
+### Solution 2: Pull Images One at a Time
+
+Use the provided script to pull images with delays:
+
+```bash
+./scripts/pull-supabase-images.sh
+```
+
+This script:
+- Pulls images one at a time
+- Waits 10 seconds between pulls
+- Retries on rate limit errors
+- Shows progress and summary
+
+### Solution 3: Manual Pull with Delays
+
+Pull images manually with delays:
+
+```bash
+# Pull one image at a time with delays
+docker compose pull supabase-db
+sleep 10
+docker compose pull supabase-rest
+sleep 10
+docker compose pull supabase-auth
+sleep 10
+# ... continue for other services
+```
+
+### Solution 4: Wait and Retry
+
+If you hit the rate limit:
+
+1. **Wait 5-10 minutes** for the rate limit window to reset
+2. **Try again**:
+   ```bash
+   docker compose pull supabase-*
+   ```
+
+### Solution 5: Use Cached Images
+
+Check if images are already available locally:
+
+```bash
+# Check what Supabase images you have
+docker images | grep supabase
+
+# If images exist, you can start services without pulling
+docker compose up -d supabase-*
+```
+
+## Current Rate Limits
+
+- **Anonymous users**: ~100 pulls per 6 hours per IP
+- **Authenticated users**: ~200 pulls per 6 hours per account
+- **Paid plans**: Higher limits
+
+## Quick Fix for Supabase Services
+
+If you just need to start Supabase services and have some images cached:
+
+```bash
+# 1. Check what's already available
+docker images | grep -E "(supabase|postgres|inbucket)"
+
+# 2. Try starting services (will only pull missing images)
+docker compose up -d supabase-db supabase-rest supabase-auth supabase-realtime supabase-storage supabase-studio supabase-meta supabase-migrate supabase-inbucket
+
+# 3. If rate limited, wait 5 minutes and try pulling specific images:
+docker compose pull supabase-rest
+# Wait 10 seconds
+docker compose pull supabase-auth
+# Continue...
+```
+
+## Alternative: Use AWS ECR Directly
+
+Since Supabase images are on AWS ECR (`public.ecr.aws/supabase/`), you can pull directly:
+
+```bash
+# Pull directly from ECR (may have different rate limits)
+docker pull public.ecr.aws/supabase/postgrest:v12.2.12
+docker pull public.ecr.aws/supabase/gotrue:v2.177.0
+# ... etc
+```
+
+## Prevention
+
+1. **Always login**: `docker login` before pulling many images
+2. **Use local images**: Keep images locally when possible
+3. **Pull gradually**: Don't pull all images at once
+4. **Use image caching**: Docker caches layers, so subsequent pulls are faster
+
+## Verify Images Are Available
+
+After pulling, verify:
+
+```bash
+# List all Supabase-related images
+docker images | grep -E "(supabase|postgrest|gotrue|realtime|storage|studio|postgres-meta|inbucket)"
+
+# Check specific service images
+docker images public.ecr.aws/supabase/postgrest
+docker images public.ecr.aws/supabase/gotrue
+```
+
+## Start Services After Pulling
+
+Once images are pulled:
+
+```bash
+# Start all Supabase services
+docker compose up -d supabase-db supabase-rest supabase-auth supabase-realtime supabase-storage supabase-studio supabase-meta supabase-migrate supabase-inbucket
+
+# Or start everything
+docker compose up -d
+```
+
--- a/docs/RUNNING_SUPABASE_SERVICES.md
+++ b/docs/RUNNING_SUPABASE_SERVICES.md
@@ -0,0 +1,258 @@
+# Running Supabase Services with Docker Compose
+
+## Quick Start
+
+### Start All Supabase Services
+
+```bash
+# From project root
+docker compose up -d supabase-db supabase-rest supabase-auth supabase-realtime supabase-storage supabase-studio supabase-meta supabase-migrate supabase-inbucket
+```
+
+Or use the shorthand (all services starting with `supabase-`):
+
+```bash
+docker compose up -d $(docker compose config --services | grep supabase | tr '\n' ' ')
+```
+
+### Start All Services (Including Supabase)
+
+```bash
+# Start everything including Supabase
+docker compose up -d
+```
+
+## Individual Service Management
+
+### Start Specific Services
+
+```bash
+# Start just the database
+docker compose up -d supabase-db
+
+# Start database + API
+docker compose up -d supabase-db supabase-rest
+
+# Start all core services (db, rest, auth)
+docker compose up -d supabase-db supabase-rest supabase-auth
+```
+
+### Stop Supabase Services
+
+```bash
+# Stop all Supabase services
+docker compose stop supabase-*
+
+# Or stop specific services
+docker compose stop supabase-db supabase-rest
+```
+
+### Restart Supabase Services
+
+```bash
+# Restart all Supabase services
+docker compose restart supabase-*
+
+# Restart specific service
+docker compose restart supabase-db
+```
+
+## Checking Service Status
+
+### List All Supabase Services
+
+```bash
+# Using labels (recommended)
+docker compose ps --filter "label=com.usda-vision.service=supabase"
+
+# Using service name pattern
+docker compose ps supabase-*
+
+# Or just check all services
+docker compose ps
+```
+
+### View Logs
+
+```bash
+# All Supabase services
+docker compose logs -f supabase-*
+
+# Specific service
+docker compose logs -f supabase-db
+
+# Last 50 lines of a service
+docker compose logs --tail=50 supabase-migrate
+```
+
+### Check Service Health
+
+```bash
+# Check if database is healthy
+docker compose ps supabase-db
+
+# Check migration status
+docker compose logs supabase-migrate | tail -20
+
+# Check all Supabase services status
+docker compose ps --filter "label=com.usda-vision.service=supabase" --format "table {{.Name}}\t{{.Status}}\t{{.Ports}}"
+```
+
+## Service Dependencies
+
+Supabase services have dependencies. Start them in this order for best results:
+
+1. **supabase-db** - Database (must start first)
+2. **supabase-migrate** - Runs migrations (depends on db)
+3. **supabase-rest** - API (depends on db + migrate)
+4. **supabase-auth** - Authentication (depends on db)
+5. **supabase-meta** - Metadata service (depends on db)
+6. **supabase-realtime** - Realtime (depends on db + rest)
+7. **supabase-storage** - Storage (depends on db + rest)
+8. **supabase-studio** - Studio UI (depends on rest + auth)
+9. **supabase-inbucket** - Email testing (independent)
+
+**Note**: Docker Compose handles dependencies automatically via `depends_on`, so you can start them all at once and they'll start in the correct order.
+
+## Common Commands
+
+### Start Everything
+
+```bash
+docker compose up -d
+```
+
+### Start Only Supabase
+
+```bash
+docker compose up -d supabase-db supabase-rest supabase-auth supabase-realtime supabase-storage supabase-studio supabase-meta supabase-migrate supabase-inbucket
+```
+
+### Stop Everything
+
+```bash
+docker compose down
+```
+
+### Stop Only Supabase (Keep Other Services Running)
+
+```bash
+docker compose stop supabase-*
+```
+
+### View Real-time Logs
+
+```bash
+# All Supabase services
+docker compose logs -f supabase-*
+
+# Specific service
+docker compose logs -f supabase-db
+```
+
+### Reset Supabase (Remove Volumes)
+
+```bash
+# Stop and remove Supabase containers and volumes
+docker compose down -v supabase-*
+
+# Or use the reset script
+./scripts/docker-compose-reset.sh
+```
+
+## Troubleshooting
+
+### Rate Limit Errors
+
+If you see `toomanyrequests: Rate exceeded`, Docker Hub is rate-limiting you. Solutions:
+
+1. **Wait a few minutes** and try again
+2. **Login to Docker Hub** (increases rate limit):
+   ```bash
+   docker login
+   ```
+3. **Use cached images** if available:
+   ```bash
+   docker images | grep supabase
+   ```
+
+### Services Not Starting
+
+1. **Check logs**:
+   ```bash
+   docker compose logs supabase-db
+   ```
+
+2. **Check if ports are in use**:
+   ```bash
+   # Check if port 54322 (database) is in use
+   sudo lsof -i :54322
+   ```
+
+3. **Verify network exists**:
+   ```bash
+   docker network ls | grep usda-vision
+   ```
+
+### Migration Issues
+
+If migrations fail:
+
+1. **Check migration logs**:
+   ```bash
+   docker compose logs supabase-migrate
+   ```
+
+2. **Manually run migrations**:
+   ```bash
+   docker compose exec supabase-db psql -U postgres -d postgres -f /path/to/migration.sql
+   ```
+
+3. **Reset database** (⚠️ **WARNING**: This deletes all data):
+   ```bash
+   docker compose down -v supabase-db
+   docker compose up -d supabase-db
+   ```
+
+## Service URLs
+
+Once running, access services at:
+
+- **Supabase API**: http://localhost:54321
+- **Supabase Studio**: http://localhost:54323
+- **Database (direct)**: `postgresql://postgres:postgres@localhost:54322/postgres`
+- **Email Testing (Inbucket)**: http://localhost:54324
+- **Realtime**: ws://localhost:4000
+- **Storage API**: http://localhost:5000
+
+## Best Practices
+
+1. **Always start database first** (or let Compose handle dependencies)
+2. **Wait for migrations** before starting dependent services
+3. **Check logs** if services fail to start
+4. **Use labels** to filter and manage Supabase services
+5. **Keep volumes** for persistent data (don't use `-v` flag unless resetting)
+
+## Example: Full Startup Sequence
+
+```bash
+# 1. Start database
+docker compose up -d supabase-db
+
+# 2. Wait for database to be ready (check logs)
+docker compose logs -f supabase-db
+
+# 3. Start all other Supabase services
+docker compose up -d supabase-rest supabase-auth supabase-realtime supabase-storage supabase-studio supabase-meta supabase-migrate supabase-inbucket
+
+# 4. Verify all services are running
+docker compose ps --filter "label=com.usda-vision.service=supabase"
+```
+
+Or simply:
+
+```bash
+# Start everything at once (Compose handles dependencies)
+docker compose up -d
+```
+
--- a/docs/SUPABASE_DOCKER_COMPOSE.md
+++ b/docs/SUPABASE_DOCKER_COMPOSE.md
@@ -56,7 +56,7 @@ The default anon key for local development is:

 Migrations are automatically run on first startup via the `supabase-migrate` service. The service:
 1. Waits for the database to be ready
-2. Runs all migrations from `management-dashboard-web-app/supabase/migrations/` in alphabetical order
+2. Runs all migrations from `supabase/migrations/` in alphabetical order
 3. Runs seed files (`seed_01_users.sql` and `seed_02_phase2_experiments.sql`)

 If you need to re-run migrations, you can:
--- a/docs/SUPABASE_MIGRATION.md
+++ b/docs/SUPABASE_MIGRATION.md
@@ -0,0 +1,93 @@
+# Supabase Directory Migration
+
+## What Changed
+
+The Supabase configuration has been moved from `management-dashboard-web-app/supabase/` to the project root `supabase/` directory.
+
+## Why This Change?
+
+1. **Better Architecture**: Supabase is shared infrastructure, not specific to the web app
+2. **Monorepo Best Practice**: Infrastructure concerns should be at the root level
+3. **Easier Access**: Other services (APIs, scripts) can now easily reference the database
+4. **Clearer Ownership**: Makes it obvious that Supabase is a project-wide resource
+
+## Migration Steps
+
+### For Docker Compose Users
+
+No action needed! The `docker-compose.yml` has been updated to use the new paths. Just restart your containers:
+
+```bash
+docker compose down
+docker compose up -d
+```
+
+### For Supabase CLI Users
+
+**Before** (old way):
+```bash
+cd management-dashboard-web-app
+supabase start
+supabase db reset
+```
+
+**After** (new way):
+```bash
+# From project root - no need to cd!
+supabase start
+supabase db reset
+```
+
+The Supabase CLI automatically looks for the `supabase/` directory in the current working directory, so you can now run all Supabase commands from the project root.
+
+### Updating Your Workflow
+
+If you have scripts or documentation that reference the old path, update them:
+
+- ❌ `management-dashboard-web-app/supabase/migrations/`
+- ✅ `supabase/migrations/`
+
+- ❌ `management-dashboard-web-app/supabase/config.toml`
+- ✅ `supabase/config.toml`
+
+## Backward Compatibility
+
+The old directory (`management-dashboard-web-app/supabase/`) can be kept for reference, but it's no longer used by docker-compose or the Supabase CLI. You can safely remove it after verifying everything works:
+
+```bash
+# After verifying everything works with the new location
+rm -rf management-dashboard-web-app/supabase
+```
+
+## Verification
+
+To verify the migration worked:
+
+1. **Check docker-compose paths**:
+   ```bash
+   grep -r "supabase" docker-compose.yml
+   # Should show: ./supabase/ (not ./management-dashboard-web-app/supabase/)
+   ```
+
+2. **Test Supabase CLI**:
+   ```bash
+   # From project root
+   supabase status
+   # Should work without needing to cd into management-dashboard-web-app
+   ```
+
+3. **Test migrations**:
+   ```bash
+   docker compose up -d
+   docker compose logs supabase-migrate
+   # Should show migrations running successfully
+   ```
+
+## Benefits
+
+✅ Run Supabase commands from project root  
+✅ Clearer project structure  
+✅ Easier to share database across services  
+✅ Better alignment with monorepo best practices  
+✅ Infrastructure separated from application code  
+