MODEL/usda-vision
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove deprecated files and scripts to streamline the codebase

Browse Source 
- Deleted unused API test files, RTSP diagnostic scripts, and development utility scripts to reduce clutter.
- Removed outdated database schema and modularization proposal documents to maintain focus on current architecture.
- Cleaned up configuration files and logging scripts that are no longer in use, enhancing project maintainability.
This commit is contained in:
salirezav
2025-11-02 10:07:59 -05:00
parent f1a9cb0c1e
commit f6a37ca1ba
50 changed files with 7057 additions and 368 deletions
Download Patch File Download Diff File Expand all files Collapse all files

437
docs/CODE_QUALITY_IMPROVEMENTS.md Normal file
View File

@@ -0,0 +1,437 @@
# Code Quality Improvements - Simple & Safe Refactorings
## 📊 Current Codebase Analysis
### Largest Files (Focus Areas)
- `recorder.py`: 1,122 lines ⚠️
- `server.py`: 842 lines ⚠️
- `streamer.py`: 745 lines ⚠️
- `manager.py`: 614 lines
### Key Observations
1. **Large classes** with multiple responsibilities
2. **API routes** all in one method (`_setup_routes` - 700+ lines)
3. **Duplicate code** (camera error suppression, initialization)
4. **Long methods** (recording loop, streaming loop)
5. **Mixed concerns** (camera hardware + business logic)
---
## 🎯 Simple, Safe Refactorings (No Behavior Changes)
### 1. **Extract Duplicate Code** ⭐ Easy Win
**Problem**: `suppress_camera_errors()` appears in 3+ files
**Solution**: Move to shared utility
```python
# Before: Duplicate in recorder.py, streamer.py, monitor.py
@contextlib.contextmanager
def suppress_camera_errors():
# ... 20 lines duplicated
```
```python
# After: Single source in camera/utils.py
# camera/utils.py
@contextlib.contextmanager
def suppress_camera_errors():
"""Suppress camera SDK error output"""
# ... implementation
# Then import everywhere:
from .utils import suppress_camera_errors
```
**Impact**:
- ✅ Reduces duplication
- ✅ Single place to fix bugs
- ✅ No behavior change
- ⚡ 5 minutes, zero risk
---
### 2. **Split API Routes into Modules** ⭐ High Impact
**Problem**: `_setup_routes()` has 700+ lines, hard to navigate
**Current Structure**:
```python
# api/server.py
def _setup_routes(self):
@self.app.get("/cameras/...")
async def get_cameras(): ...
@self.app.post("/cameras/...")
async def start_recording(): ...
# ... 30+ more routes
```
**Solution**: Group routes by domain
```python
# api/server.py
def _setup_routes(self):
from .routes import camera_routes, recording_routes, system_routes
camera_routes.register(self.app, self.camera_manager)
recording_routes.register(self.app, self.camera_manager)
system_routes.register(self.app, self.state_manager, ...)
# api/routes/camera_routes.py
def register(app, camera_manager):
@app.get("/cameras")
async def get_cameras():
return camera_manager.get_all_camera_status()
# ... all camera routes
```
**Impact**:
- ✅ Much easier to find/edit routes
- ✅ Clearer organization
- ✅ Can split across files
- ⚡ 30 minutes, low risk
---
### 3. **Extract Camera Initialization** ⭐ Medium Impact
**Problem**: Camera initialization code duplicated in `recorder.py`, `streamer.py`, `monitor.py`
**Solution**: Create `CameraInitializer` class
```python
# camera/initializer.py
class CameraInitializer:
"""Handles camera initialization with consistent configuration"""
def __init__(self, camera_config: CameraConfig, device_info):
self.camera_config = camera_config
self.device_info = device_info
def initialize(self) -> tuple[int, Any, bool]:
"""Returns (hCamera, cap, monoCamera)"""
# Common initialization logic
# ...
return hCamera, cap, monoCamera
def configure_settings(self, hCamera):
"""Configure camera settings"""
# Common configuration
# ...
# Then use:
initializer = CameraInitializer(config, device_info)
self.hCamera, self.cap, self.monoCamera = initializer.initialize()
initializer.configure_settings(self.hCamera)
```
**Impact**:
- ✅ Removes ~200 lines of duplication
- ✅ Consistent initialization
- ✅ Easier to test
- ⚡ 1 hour, medium risk (test carefully)
---
### 4. **Break Down Large Methods** ⭐ Medium Impact
**Problem**: `_recording_loop()` is 100+ lines, `_streaming_loop()` is 80+ lines
**Solution**: Extract frame processing logic
```python
# Before: recorder.py
def _recording_loop(self, use_streamer_frames: bool):
while not self._stop_recording_event.is_set():
# Get frame (30 lines)
if use_streamer_frames:
# ... complex logic
else:
# ... complex logic
# Write frame (10 lines)
# Rate control (5 lines)
```
```python
# After: recorder.py
def _recording_loop(self, use_streamer_frames: bool):
frame_source = self._create_frame_source(use_streamer_frames)
while not self._stop_recording_event.is_set():
frame = frame_source.get_frame()
if frame:
self._write_frame(frame)
self._control_frame_rate()
def _create_frame_source(self, use_streamer_frames):
"""Create appropriate frame source"""
if use_streamer_frames:
return StreamerFrameSource(self.streamer)
return DirectCameraSource(self.hCamera, self.frame_buffer)
def _write_frame(self, frame):
"""Write frame to video"""
if self.video_writer:
self.video_writer.write(frame)
self.frame_count += 1
```
**Impact**:
- ✅ Methods are shorter, clearer
- ✅ Easier to test individual pieces
- ✅ Better readability
- ⚡ 2 hours, medium risk
---
### 5. **Type Hints & Documentation** ⭐ Easy, Incremental
**Problem**: Some methods lack type hints, unclear parameter meanings
**Solution**: Add type hints gradually (no behavior change)
```python
# Before
def start_recording(self, filename):
# ...
# After
def start_recording(self, filename: str) -> bool:
"""
Start video recording for the camera.
Args:
filename: Output filename (will be prefixed with timestamp)
Returns:
True if recording started successfully, False otherwise
Raises:
CameraException: If camera initialization fails
"""
```
**Impact**:
- ✅ Better IDE support
- ✅ Self-documenting code
- ✅ Catch errors earlier
- ⚡ Can do incrementally, zero risk
---
### 6. **Create Value Objects** ⭐ Medium Impact
**Problem**: Camera properties scattered across instance variables
**Current**:
```python
self.hCamera = ...
self.cap = ...
self.monoCamera = ...
self.frame_buffer = ...
self.frame_buffer_size = ...
```
**Better**:
```python
# camera/domain/camera_handle.py
@dataclass
class CameraHandle:
"""Represents a camera hardware connection"""
handle: int
capabilities: Any # Camera capability struct
is_monochrome: bool
frame_buffer: Any
frame_buffer_size: int
def is_valid(self) -> bool:
return self.handle is not None
# Usage
self.camera = CameraHandle(...)
if self.camera.is_valid():
# ...
```
**Impact**:
- ✅ Groups related data
- ✅ Easier to pass around
- ✅ Better encapsulation
- ⚡ 2 hours, low risk
---
### 7. **Extract Constants** ⭐ Easy Win
**Problem**: Magic numbers scattered throughout
```python
# Before
time.sleep(0.1)
timeout=200
maxsize=5
# After: camera/constants.py
STREAMING_LOOP_SLEEP = 0.1 # seconds
CAMERA_GET_BUFFER_TIMEOUT = 200 # milliseconds
FRAME_QUEUE_MAXSIZE = 5
```
**Impact**:
- ✅ Self-documenting
- ✅ Easy to tune
- ✅ No behavior change
- ⚡ 30 minutes, zero risk
---
## 🚀 Recommended Refactoring Order (Safe → Risky)
### Phase 1: **Quick Wins** (1-2 hours, zero risk)
1. ✅ Extract `suppress_camera_errors()` to shared utils
2. ✅ Extract constants to `constants.py`
3. ✅ Add type hints to public methods
### Phase 2: **Organization** (3-4 hours, low risk)
4. ✅ Split API routes into modules (`routes/camera_routes.py`, etc.)
5. ✅ Group related functions into utility modules
6. ✅ Improve docstrings
### Phase 3: **Structure** (6-8 hours, medium risk)
7. ✅ Extract `CameraInitializer` class
8. ✅ Break down large methods (`_recording_loop`, `_streaming_loop`)
9. ✅ Create value objects for camera handles
### Phase 4: **Advanced** (optional, higher risk)
10. ⚠️ Extract frame source abstractions
11. ⚠️ Create repository pattern for camera access
12. ⚠️ Dependency injection container
---
## 📋 Specific File Improvements
### `recorder.py` (1,122 lines)
**Quick wins**:
- Extract `suppress_camera_errors` → utils
- Extract constants (timeouts, buffer sizes)
- Split `_initialize_video_writer` (100+ lines) into smaller methods
**Medium refactoring**:
- Extract `_recording_loop` frame source logic
- Create `VideoWriterManager` class
- Extract camera configuration to separate class
### `server.py` (842 lines)
**Quick wins**:
- Split `_setup_routes` into route modules
- Extract WebSocket logic to separate file
- Group related endpoints
### `streamer.py` (745 lines)
**Quick wins**:
- Extract `suppress_camera_errors` → utils
- Extract RTSP FFmpeg command building
- Extract frame queue management
**Medium refactoring**:
- Extract `_streaming_loop` frame processing
- Create `FrameQueueManager` class
### `manager.py` (614 lines)
**Quick wins**:
- Extract camera discovery to separate class
- Split initialization methods
- Extract status reporting
---
## 🎯 "Good Enough" Approach
**Don't over-engineer!** Focus on:
1. **Readability**: Can a new developer understand it?
2. **Editability**: Can you change one thing without breaking others?
3. **Testability**: Can you test pieces in isolation?
**Avoid**:
- ❌ Premature abstraction
- ❌ Design patterns "just because"
- ❌ Perfect code (good enough > perfect)
- ❌ Breaking working code
---
## 💡 Minimal Viable Refactoring
**If you only do 3 things:**
1. **Extract duplicate code** (`suppress_camera_errors`, constants)
- 30 minutes, huge improvement
2. **Split API routes** (into route modules)
- 1 hour, makes API much more manageable
3. **Add type hints** (gradually, as you touch code)
- Ongoing, improves IDE support
**Result**: Much more maintainable code with minimal effort!
---
## 🔧 Tools to Help
### Code Quality Tools
```bash
# Linting
pip install ruff black mypy
ruff check .
black --check .
mypy camera-management-api/
# Complexity
pip install radon
radon cc camera-management-api/usda_vision_system -a
```
### Refactoring Safely
- ✅ Write tests first (if not present)
- ✅ Refactor in small steps
- ✅ Test after each change
- ✅ Use git branches
- ✅ Review diffs carefully
---
## 📊 Success Metrics
After refactoring, you should see:
- ✅ Lower cyclomatic complexity
- ✅ Smaller average method length
- ✅ Less duplicate code
- ✅ More consistent patterns
- ✅ Easier to add features
- ✅ Fewer bugs when making changes
---
## 🎓 Best Practices
1. **One refactoring per commit** - easier to review/rollback
2. **Don't refactor while adding features** - separate PRs
3. **Measure before/after** - use code metrics
4. **Document decisions** - why this structure?
5. **Keep it simple** - don't add complexity "for the future"
---
**Bottom Line**: Start with #1, #2, and #3 (duplicate extraction, route splitting, type hints). These give you 80% of the benefit with 20% of the effort, and they're completely safe!

427
docs/MODULARIZATION_PROPOSAL.md Normal file
View File

@@ -0,0 +1,427 @@
# Camera Management API - Modularization Proposal
## 📊 Current Architecture Analysis
### Current Structure
The `camera-management-api` is currently a **monolithic service** with the following components:
1. **API Server** (`api/server.py`)
- FastAPI REST endpoints
- WebSocket real-time updates
- Orchestrates all other components
2. **Camera Management** (`camera/`)
- Camera discovery & initialization
- Recording (CameraRecorder)
- Streaming (CameraStreamer) - MJPEG & RTSP
- Camera monitoring & recovery
3. **MQTT Client** (`mqtt/`)
- Machine state monitoring
- Event publishing
4. **Storage Manager** (`storage/`)
- File indexing
- Storage statistics
- Cleanup operations
5. **Auto Recording Manager** (`recording/`)
- Automated recording based on MQTT events
- Standalone auto-recorder
6. **Core Services**
- State Manager (in-memory state)
- Event System (pub/sub)
- Configuration management
7. **Video Services** (`video/`)
- Video streaming
- Metadata extraction
- Caching
### Current Service Separation
You already have:
- ✅ `media-api` - Video processing (thumbnails, transcoding)
- ✅ `mediamtx` - RTSP streaming server
- ✅ Microfrontend dashboard (shell + video-remote)
---
## 🎯 Modularization Strategies
### Strategy 1: **Modular Monolith** (Recommended to Start)
**Approach**: Keep as single service but improve internal structure with clear boundaries.
**Structure**:
```
camera-management-api/
├── core/ # Shared infrastructure
│ ├── state/
│ ├── events/
│ └── config/
├── camera/ # Camera hardware layer
├── recording/ # Recording logic
├── streaming/ # Streaming logic (separate from camera)
├── mqtt/ # MQTT integration
├── storage/ # Storage operations
└── api/ # API endpoints (orchestration layer)
```
**Pros**:
- ✅ Minimal disruption to working system
- ✅ Easier debugging (single process)
- ✅ Lower operational complexity
- ✅ Shared state remains simple
- ✅ No network latency between components
- ✅ Easier to maintain consistency
**Cons**:
- ❌ Can't scale components independently
- ❌ All-or-nothing deployment
- ❌ Single point of failure (mitigated by Docker)
**Best For**: Current state - proven system that works well together
---
### Strategy 2: **Strategic Microservices** (Hybrid Approach)
**Approach**: Split only high-value, independently scalable components.
**Services**:
#### Service 1: **camera-service** (Critical, Hardware-Dependent)
```
Responsibilities:
- Camera discovery & initialization
- Recording (CameraRecorder)
- Streaming (CameraStreamer) - MJPEG
- Camera monitoring & recovery
- Hardware state management
Port: 8001
Dependencies: Camera SDK, FFmpeg
Network: host (for camera access)
```
#### Service 2: **mqtt-service** (Stateless, Scalable)
```
Responsibilities:
- MQTT client & subscriptions
- Machine state monitoring
- Event publishing
Port: 8002
Dependencies: MQTT broker
Stateless: Yes
```
#### Service 3: **api-gateway** (Orchestration)
```
Responsibilities:
- REST API endpoints
- WebSocket server
- Request routing to services
- Aggregating responses
Port: 8000
Dependencies: camera-service, mqtt-service, state-manager
```
#### Service 4: **state-manager-service** (Optional - Shared State)
```
Responsibilities:
- Centralized state management
- Event bus/queue
- State persistence
Port: 8003
Database: Redis (recommended) or PostgreSQL
```
**Pros**:
- ✅ Camera service can be isolated/restarted independently
- ✅ MQTT service is stateless and scalable
- ✅ Clear separation of concerns
- ✅ Can scale MQTT service separately
- ✅ API gateway can handle load balancing
**Cons**:
- ❌ More complex deployment
- ❌ Network latency between services
- ❌ State synchronization challenges
- ❌ More containers to manage
- ❌ Service discovery needed
**Best For**: Future scaling needs, when you need:
- Multiple camera servers
- High MQTT message volume
- Different scaling requirements per component
---
### Strategy 3: **Full Microservices** (Advanced)
**Approach**: Split into granular services following domain boundaries.
**Services**:
1. `camera-service` - Hardware control
2. `recording-service` - Recording orchestration
3. `streaming-service` - MJPEG/RTSP streaming
4. `mqtt-service` - Machine state monitoring
5. `auto-recording-service` - Automated recording logic
6. `api-gateway` - API & routing
7. `state-service` - Centralized state
8. `storage-service` - File management
**Pros**:
- ✅ Maximum flexibility
- ✅ Independent scaling per service
- ✅ Technology diversity (can use different languages)
- ✅ Team autonomy
**Cons**:
- ❌ Very complex
- ❌ High operational overhead
- ❌ Distributed system challenges (consistency, latency)
- ❌ Overkill for current needs
**Best For**: Large team, complex requirements, need for maximum flexibility
---
## 🏆 Recommended Approach: **Incremental Modularization**
### Phase 1: **Internal Refactoring** (Current → 3 months)
**Goal**: Improve code organization without breaking changes
1. **Separate concerns within monolith**:
```
camera/
├── hardware/ # Camera SDK operations
├── recording/ # Recording logic
├── streaming/ # Streaming logic
└── monitoring/ # Health checks
```
2. **Use dependency injection**: Pass dependencies explicitly
3. **Clear interfaces**: Define contracts between modules
4. **Document boundaries**: Mark what can/can't be changed independently
**Outcome**: Cleaner code, easier to split later if needed
---
### Phase 2: **Extract MQTT Service** (3-6 months)
**Goal**: Split out stateless, independent component
**Why MQTT first?**
- ✅ Completely stateless
- ✅ No shared state with cameras
- ✅ Easy to scale
- ✅ Lower risk (doesn't affect camera operations)
**Implementation**:
- Move `mqtt/` to separate service
- Use Redis/RabbitMQ for event pub/sub
- API Gateway queries MQTT service for status
- MQTT service publishes to event bus
**Outcome**: First microservice, validates approach
---
### Phase 3: **Evaluate Further Splitting** (6+ months)
**Decision Point**: Based on actual needs
**If scaling cameras**:
- Extract `camera-service` to run on multiple machines
- Keep recording/streaming together (they're tightly coupled)
**If high API load**:
- Keep API gateway separate
- Scale gateway independently
**If complex state management**:
- Extract `state-service` with Redis/PostgreSQL
- Services query state service instead of in-memory
---
## 🔧 Implementation Details
### Shared Infrastructure (All Strategies)
#### 1. **Event Bus** (Essential for microservices)
```
Option A: Redis Pub/Sub (lightweight)
Option B: RabbitMQ (more features)
Option C: MQTT (you already have it!)
```
#### 2. **State Management**
```
Option A: Redis (fast, in-memory)
Option B: PostgreSQL (persistent, queryable)
Option C: Keep in-memory for now (simplest)
```
#### 3. **Service Discovery**
```
For microservices:
- Docker Compose service names (simple)
- Consul/Eureka (if needed)
- Kubernetes services (if migrating)
```
#### 4. **API Gateway Pattern**
```
nginx/Envoy: Route requests to services
FastAPI Gateway: Aggregate responses
GraphQL: Alternative aggregation layer
```
---
## 📋 Decision Matrix
| Factor | Modular Monolith | Strategic Split | Full Microservices |
|--------|------------------|----------------|-------------------|
| **Complexity** | ⭐ Low | ⭐⭐ Medium | ⭐⭐⭐ High |
| **Scalability** | ⭐ Limited | ⭐⭐ Good | ⭐⭐⭐ Excellent |
| **Development Speed** | ⭐⭐⭐ Fast | ⭐⭐ Medium | ⭐ Slow |
| **Operational Overhead** | ⭐ Low | ⭐⭐ Medium | ⭐⭐⭐ High |
| **Risk** | ⭐ Low | ⭐⭐ Medium | ⭐⭐⭐ High |
| **Cost** | ⭐ Low | ⭐⭐ Medium | ⭐⭐⭐ High |
| **Current Fit** | ⭐⭐⭐ Perfect | ⭐⭐ Good | ⭐ Overkill |
---
## 💡 My Recommendation
### **Start with Strategy 1: Modular Monolith + Internal Refactoring**
**Why?**
1. ✅ Your system is **already working well**
2. ✅ RTSP + Recording work concurrently (hard problem solved)
3. ✅ No immediate scaling needs identified
4. ✅ Single team managing it
5. ✅ Lower risk, faster improvements
**What to do now:**
1. **Refactor internal structure** (Phase 1)
- Separate camera, recording, streaming modules
- Clear interfaces between modules
- Dependency injection
2. **Add event bus infrastructure** (prepare for future)
- Set up Redis for events (even if monolith)
- Publish events through Redis pub/sub
- Services can subscribe when needed
3. **Monitor & Measure** (data-driven decisions)
- Track performance metrics
- Identify bottlenecks
- Measure actual scaling needs
4. **Extract when needed** (not before)
- Only split when you have concrete problems
- Start with MQTT service (safest first split)
- Then camera-service if scaling cameras
**Red Flags for Microservices** (when you DON'T need them):
- ❌ "We might need to scale" (YAGNI - You Ain't Gonna Need It)
- ❌ "Industry best practice" (without actual need)
- ❌ "Multiple teams" (you have one team)
- ❌ "Independent deployment" (current deployment is simple)
**Green Flags for Microservices** (when you DO need them):
- ✅ Actually scaling cameras to multiple servers
- ✅ High API load requiring independent scaling
- ✅ Need to update camera logic without touching MQTT
- ✅ Multiple teams working on different components
- ✅ Need different technology stacks per service
---
## 🚀 Quick Start: Internal Refactoring Plan
### Step 1: Create Module Boundaries
```
usda_vision_system/
├── camera/
│ ├── hardware/ # Camera SDK wrapper
│ │ ├── camera_sdk.py
│ │ └── device_discovery.py
│ ├── recording/ # Recording logic
│ │ ├── recorder.py
│ │ └── video_writer.py
│ ├── streaming/ # Streaming logic
│ │ ├── mjpeg_streamer.py
│ │ └── rtsp_streamer.py
│ └── monitoring/ # Health & recovery
│ └── health_check.py
```
### Step 2: Define Interfaces
```python
# camera/domain/interfaces.py
class ICameraHardware(ABC):
@abstractmethod
def initialize() -> bool
@abstractmethod
def capture_frame() -> Frame
class IRecorder(ABC):
@abstractmethod
def start_recording(filename: str) -> bool
@abstractmethod
def stop_recording() -> bool
```
### Step 3: Dependency Injection
```python
# Instead of direct instantiation
recorder = CameraRecorder(config, state_manager, event_system)
# Use factories/interfaces
recorder = RecorderFactory.create(config, dependencies)
```
---
## 📚 References & Further Reading
- **Modular Monolith**: https://www.kamilgrzybek.com/blog/posts/modular-monolith-primer
- **Microservices Patterns**: https://microservices.io/patterns/
- **When to Use Microservices**: https://martinfowler.com/articles/microservices.html
---
## ❓ Questions to Answer
Before deciding on microservices, ask:
1. **Do you need to scale components independently?**
- If no → Monolith is fine
2. **Do different teams work on different parts?**
- If no → Monolith is fine
3. **Are there actual performance bottlenecks?**
- If no → Don't optimize prematurely
4. **Can you deploy the monolith easily?**
- If yes → Monolith might be better
5. **Do you need different tech stacks per component?**
- If no → Monolith is fine
---
**Bottom Line**: Your system is working well. Focus on **improving code quality and organization** rather than splitting prematurely. Extract services when you have **concrete, measurable problems** that require it.

24
docs/README.md Normal file
View File

@@ -0,0 +1,24 @@
# Documentation
This directory contains project documentation and reference materials.
## Documentation Files
- **`CODE_QUALITY_IMPROVEMENTS.md`** - Analysis and suggestions for improving code quality in the camera-management-api
- **`MODULARIZATION_PROPOSAL.md`** - Different strategies for modularizing the API, including recommendations
- **`REFACTORING_PLAN.md`** - Step-by-step quick start guide for implementing code quality improvements
- **`REFACTORING_SUMMARY.md`** - Complete summary of the modularization and refactoring work performed (frontend microfrontends + backend improvements)
- **`database_schema.md`** - Database schema documentation
- **`rtsp_access_guide.md`** - Guide for accessing RTSP streams
- **`test_rtsp_working.md`** - Notes about RTSP testing
## Main Documentation
See the root `README.md` for project overview, setup instructions, and quick start guide.

249
docs/REFACTORING_PLAN.md Normal file
View File

@@ -0,0 +1,249 @@
# Quick Start: Code Quality Refactoring Plan
## 🎯 Priority Order (Safe → Risky)
### ✅ Task 1: Extract Duplicate Code (30 min, zero risk)
**File**: Create `camera-management-api/usda_vision_system/camera/utils.py`
Move `suppress_camera_errors()` from 3 files into one shared location.
**Files to update**:
- `camera/recorder.py`
- `camera/streamer.py`
- `camera/monitor.py`
**Benefit**: Single source of truth, easier to fix bugs
---
### ✅ Task 2: Extract Constants (30 min, zero risk)
**File**: Create `camera-management-api/usda_vision_system/camera/constants.py`
Extract magic numbers:
- Timeouts (200ms, 1000ms, etc.)
- Queue sizes (5, 10, 30)
- Sleep intervals (0.1s, etc.)
- FPS defaults (10.0, 15.0, 30.0)
**Benefit**: Self-documenting, easy to tune
---
### ✅ Task 3: Split API Routes (1-2 hours, low risk)
**Create**:
- `api/routes/__init__.py`
- `api/routes/camera_routes.py`
- `api/routes/recording_routes.py`
- `api/routes/system_routes.py`
- `api/routes/mqtt_routes.py`
- `api/routes/storage_routes.py`
**Move routes** from `api/server.py` to appropriate modules.
**Benefit**: Much easier to navigate 800+ line file
---
### ✅ Task 4: Add Type Hints (ongoing, zero risk)
Add type hints as you touch code:
- Start with public methods
- Use `Optional[...]` for nullable values
- Use `Dict[str, ...]` for dictionaries
**Benefit**: Better IDE support, catch errors early
---
## 📝 Detailed Steps for Task 1 (Start Here!)
### Step 1: Create utils file
```python
# camera-management-api/usda_vision_system/camera/utils.py
"""Shared utilities for camera operations"""
import contextlib
import os
@contextlib.contextmanager
def suppress_camera_errors():
"""Context manager to temporarily suppress camera SDK error output"""
original_stderr = os.dup(2)
original_stdout = os.dup(1)
try:
devnull = os.open(os.devnull, os.O_WRONLY)
os.dup2(devnull, 2) # stderr
os.dup2(devnull, 1) # stdout
os.close(devnull)
yield
finally:
os.dup2(original_stderr, 2)
os.dup2(original_stdout, 1)
os.close(original_stderr)
os.close(original_stdout)
```
### Step 2: Update imports in recorder.py
```python
# camera-management-api/usda_vision_system/camera/recorder.py
# Remove the duplicate function
# Change import to:
from .utils import suppress_camera_errors
```
### Step 3: Repeat for streamer.py and monitor.py
### Step 4: Test
```bash
docker compose restart api
# Verify everything still works
```
**That's it!** Single source of truth for camera error suppression.
---
## 📝 Detailed Steps for Task 2 (Constants)
### Step 1: Create constants file
```python
# camera-management-api/usda_vision_system/camera/constants.py
"""Constants for camera operations"""
# Timeouts (milliseconds)
CAMERA_GET_BUFFER_TIMEOUT = 200
CAMERA_INIT_TIMEOUT = 1000
CAMERA_TEST_CAPTURE_TIMEOUT = 1000
# Frame queue sizes
MJPEG_QUEUE_MAXSIZE = 5
RTSP_QUEUE_MAXSIZE = 10
RECORDING_QUEUE_MAXSIZE = 30
# Frame rates
PREVIEW_FPS = 10.0
RTSP_FPS = 15.0
DEFAULT_VIDEO_FPS = 30.0
# Sleep intervals (seconds)
STREAMING_LOOP_SLEEP = 0.1
FRAME_RATE_CONTROL_SLEEP_BASE = 0.01
# JPEG quality
PREVIEW_JPEG_QUALITY = 70
```
### Step 2: Update files to use constants
```python
# Before
pRawData, FrameHead = mvsdk.CameraGetImageBuffer(self.hCamera, 200)
# After
from .constants import CAMERA_GET_BUFFER_TIMEOUT
pRawData, FrameHead = mvsdk.CameraGetImageBuffer(self.hCamera, CAMERA_GET_BUFFER_TIMEOUT)
```
---
## 📝 Detailed Steps for Task 3 (Route Splitting)
### Step 1: Create routes directory structure
```
api/
├── __init__.py
├── server.py
├── models.py
└── routes/
├── __init__.py
├── camera_routes.py
├── recording_routes.py
├── system_routes.py
├── mqtt_routes.py
└── storage_routes.py
```
### Step 2: Example - camera_routes.py
```python
# api/routes/camera_routes.py
from fastapi import APIRouter, HTTPException
from typing import Dict
from ..models import CameraStatusResponse
def register_camera_routes(app, camera_manager, logger):
"""Register camera-related routes"""
@app.get("/cameras", response_model=Dict[str, CameraStatusResponse])
async def get_cameras():
"""Get all camera statuses"""
try:
return camera_manager.get_all_camera_status()
except Exception as e:
logger.error(f"Error getting cameras: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.get("/cameras/{camera_name}/status", response_model=CameraStatusResponse)
async def get_camera_status(camera_name: str):
"""Get specific camera status"""
# ... implementation
```
### Step 3: Update server.py
```python
# api/server.py
def _setup_routes(self):
from .routes import camera_routes, recording_routes, system_routes
# Register route groups
camera_routes.register_camera_routes(self.app, self.camera_manager, self.logger)
recording_routes.register_recording_routes(self.app, self.camera_manager, self.logger)
system_routes.register_system_routes(self.app, self.state_manager, self.logger)
# ... etc
```
---
## ✅ Testing After Each Refactoring
```bash
# 1. Restart API
docker compose restart api
# 2. Test key endpoints
curl http://localhost:8000/health
curl http://localhost:8000/system/status
curl http://localhost:8000/cameras
# 3. Test camera operations (if cameras connected)
curl http://localhost:8000/cameras/camera1/status
# 4. Check logs
docker compose logs api --tail 50
```
---
## 📊 Progress Tracking
- [ ] Task 1: Extract duplicate code (utils.py)
- [ ] Task 2: Extract constants (constants.py)
- [ ] Task 3: Split API routes (routes/ directory)
- [ ] Task 4: Add type hints (ongoing)
**Estimated Time**: 2-3 hours total for first 3 tasks
**Risk Level**: Very Low - All are structural changes with no behavior modification
---
**Start with Task 1 - it's the easiest and gives immediate benefit!**

358
docs/REFACTORING_SUMMARY.md Normal file
View File

@@ -0,0 +1,358 @@
# Code Quality Refactoring Summary
**Date**: November 2025
**Branch**: Modularization branch
**Status**: ✅ Completed and Verified
## Overview
This document summarizes the modularization and code quality refactoring work performed on the USDA Vision system. The work was done in two phases:
1. **Frontend Modularization** (React Dashboard) - Extracted features into microfrontends using Module Federation
2. **Backend Refactoring** (Camera Management API) - Improved code organization within the monolithic architecture
This document focuses primarily on Phase 2 (API refactoring), but provides context about the overall modularization strategy.
## Project Context: Two-Phase Modularization
### Phase 1: Frontend Modularization (React Dashboard)
**Status**: ✅ Completed
Before working on the API, we first modularized the React dashboard application into a microfrontend architecture:
- **Approach**: Used Vite Module Federation to create independently deployable frontend modules
- **First Module Extracted**: `video-remote` - The video library feature was extracted into its own microfrontend
- **Architecture**:
- Main dashboard acts as a "shell" application
- Remotely loads `video-remote` module when enabled via feature flags
- Supports gradual migration (local fallback components remain available)
- **Infrastructure Changes**:
- Created separate `media-api` container for video processing (thumbnails, transcoding)
- Added `mediamtx` container for RTSP/WebRTC streaming
- `video-remote` container runs independently and can be updated separately
- **Benefits Achieved**:
- Independent deployment of video library feature
- Better separation of concerns (media handling separate from main dashboard)
- Foundation for extracting more features (camera management, experiments, etc.)
### Phase 2: Backend Refactoring (Camera Management API)
**Status**: ✅ Completed
After successfully modularizing the frontend, we focused on improving the backend code quality. **Important**: We chose NOT to split the API into microservices, but rather to improve the organization within the existing monolithic architecture.
- **Approach**: Simple, low-risk refactorings within the monolithic structure
- **Philosophy**: "Simplest least destructive code refactorings that can significantly make the code more readable and manageable and editable"
- **Decision**: Keep monolithic architecture (no microservices) but improve internal organization
- **Why Monolithic**:
- Camera SDK and hardware interactions require tight coupling
- System is stable and working well
- Full microservices would be overkill and add complexity
- Focus on code quality over architectural changes
---
## Motivation
The API refactoring was requested to improve code quality and manageability within the existing monolithic architecture. The goal was to improve organization without resorting to full microservices architecture or breaking changes, following the successful pattern we established with the frontend modularization.
## Refactoring Tasks Completed
### 1. Extract Duplicate Code (`suppress_camera_errors`)
**Problem**: The `suppress_camera_errors()` context manager was duplicated in three files:
- `camera/recorder.py`
- `camera/streamer.py`
- `camera/monitor.py`
**Solution**:
- Created `camera/utils.py` with the centralized `suppress_camera_errors()` function
- Updated all three files to import from `utils` instead of defining locally
**Files Changed**:
- ✅ Created: `camera-management-api/usda_vision_system/camera/utils.py`
- ✅ Updated: `camera/recorder.py` - removed local definition, added import
- ✅ Updated: `camera/streamer.py` - removed local definition, added import
- ✅ Updated: `camera/monitor.py` - removed local definition, added import
**Benefits**:
- Single source of truth for error suppression logic
- Easier maintenance (fix bugs in one place)
- Consistent behavior across all camera modules
---
### 2. Extract Magic Numbers into Constants
**Problem**: Magic numbers scattered throughout camera code made it hard to understand intent and adjust settings:
- Queue sizes (5, 10, 30)
- Frame rates (10.0, 15.0, 30.0)
- Timeouts (200, 1000, 500 milliseconds)
- JPEG quality (70)
- Sleep intervals (0.1 seconds)
**Solution**:
- Created `camera/constants.py` with well-named constants
- Replaced all magic numbers with constant references
**Constants Defined**:
```python
# Queue sizes
MJPEG_QUEUE_MAXSIZE = 5
RTSP_QUEUE_MAXSIZE = 10
RECORDING_QUEUE_MAXSIZE = 30
# Frame rates
PREVIEW_FPS = 10.0
RTSP_FPS = 15.0
DEFAULT_VIDEO_FPS = 30.0
# JPEG quality
PREVIEW_JPEG_QUALITY = 70
# Timeouts (milliseconds)
CAMERA_GET_BUFFER_TIMEOUT = 200
CAMERA_INIT_TIMEOUT = 1000
CAMERA_TEST_CAPTURE_TIMEOUT = 500
# Sleep intervals (seconds)
STREAMING_LOOP_SLEEP = 0.1
BRIEF_PAUSE_SLEEP = 0.1
```
**Files Changed**:
- ✅ Created: `camera-management-api/usda_vision_system/camera/constants.py`
- ✅ Updated: `camera/recorder.py` - replaced magic numbers with constants
- ✅ Updated: `camera/streamer.py` - replaced magic numbers with constants
- ✅ Updated: `camera/manager.py` - replaced magic numbers with constants
- ✅ Updated: `camera/monitor.py` - added import for `CAMERA_TEST_CAPTURE_TIMEOUT`
**Benefits**:
- Self-documenting code (constants explain what values represent)
- Easy to adjust performance settings (change in one place)
- Reduced risk of inconsistent values across modules
- Better code readability
---
### 3. Split Monolithic API Routes into Domain Modules
**Problem**: `api/server.py` was 868 lines with all routes defined in a single `_setup_routes()` method, making it:
- Hard to navigate and find specific endpoints
- Difficult to maintain (one large file)
- Prone to merge conflicts
- Not following separation of concerns
**Solution**:
- Created `api/routes/` directory with domain-specific route modules
- Each module exports a `register_*_routes()` function
- Updated `server.py` to import and call these registration functions
**New File Structure**:
```
api/routes/
├── __init__.py # Exports all register functions
├── system_routes.py # /, /health, /system/status, /system/video-module
├── camera_routes.py # /cameras, /cameras/{name}/*, RTSP endpoints
├── recording_routes.py # /cameras/{name}/start-recording, stop-recording
├── mqtt_routes.py # /mqtt/status, /mqtt/events
├── storage_routes.py # /storage/stats, /storage/files, /storage/cleanup
├── auto_recording_routes.py # /cameras/{name}/auto-recording/*
└── recordings_routes.py # /recordings
```
**Files Changed**:
- ✅ Created: `api/routes/__init__.py`
- ✅ Created: `api/routes/system_routes.py` - 7 routes
- ✅ Created: `api/routes/camera_routes.py` - 14 routes
- ✅ Created: `api/routes/recording_routes.py` - 2 routes
- ✅ Created: `api/routes/mqtt_routes.py` - 2 routes
- ✅ Created: `api/routes/storage_routes.py` - 3 routes
- ✅ Created: `api/routes/auto_recording_routes.py` - 3 routes
- ✅ Created: `api/routes/recordings_routes.py` - 1 route
- ✅ Updated: `api/server.py` - reduced from 868 lines to 315 lines (63% reduction)
**Remaining in `server.py`**:
- WebSocket endpoint (`/ws`) - kept here as it's core to the server
- Debug endpoint (`/debug/camera-manager`) - utility endpoint
- Video module route integration - dynamic route inclusion
**Import Path Corrections Made**:
- Fixed all route modules to use correct relative imports:
- `from ...core.config` (three levels up from `api/routes/`)
- `from ..models` (one level up to `api/models`)
- Fixed `AutoRecordingManager` import path (was `auto_recording.manager`, corrected to `recording.auto_manager`)
- Added proper type hints to all registration functions
**Benefits**:
- **63% reduction** in `server.py` size (868 → 315 lines)
- Routes organized by domain (easy to find specific endpoints)
- Easier maintenance (smaller, focused files)
- Reduced merge conflicts (different developers work on different route modules)
- Better code organization following separation of concerns
- Easier to test (can test route modules independently)
---
## Key Design Decisions
### Why Keep WebSocket in `server.py`?
The WebSocket endpoint (`/ws`) was kept in `server.py` because:
- It's tightly coupled with the `WebSocketManager` class defined in `server.py`
- It's core functionality, not a domain-specific feature
- Moving it would require refactoring the manager class as well
### Why Use `register_*_routes()` Functions?
Each route module exports a function that takes dependencies (app, managers, logger) and registers routes. This pattern:
- Keeps route modules testable (can pass mock dependencies)
- Allows `server.py` to control dependency injection
- Makes it clear what dependencies each route module needs
### Why Not Move Debug Endpoint?
The `/debug/camera-manager` endpoint could be moved to `camera_routes.py`, but it was kept in `server.py` as a utility endpoint for debugging the server's internal state. This is a reasonable design choice for debug utilities.
---
## Verification
All refactoring changes were verified to work correctly:
✅ **API Starts Successfully**
- No import errors
- No syntax errors
- All route modules load correctly
✅ **Endpoints Function Correctly**
- `/health` - Returns healthy status
- `/system/status` - Returns system status with cameras, machines, recordings
- `/cameras` - Returns camera status (both cameras now show correct status)
- All other endpoints maintain functionality
✅ **No Regressions**
- Camera monitoring works correctly (camera1 shows "available" status)
- Constants are properly imported and used
- Utility functions work as expected
---
## Migration Notes for Future Developers
### Adding New Routes
1. **Identify the domain**: Which route module does your endpoint belong to?
- System/health → `system_routes.py`
- Camera operations → `camera_routes.py`
- Recording → `recording_routes.py`
- Storage → `storage_routes.py`
- MQTT → `mqtt_routes.py`
- Auto-recording → `auto_recording_routes.py`
- Recording sessions → `recordings_routes.py`
2. **Add route to appropriate module**: Use the existing pattern:
```python
@app.get("/your/endpoint")
async def your_endpoint():
# Implementation
```
3. **If creating a new domain**:
- Create new file: `api/routes/your_domain_routes.py`
- Export `register_your_domain_routes()` function
- Add import to `api/routes/__init__.py`
- Register in `server.py`'s `_setup_routes()` method
### Using Constants
When you need camera-related constants:
```python
from ...camera.constants import CAMERA_GET_BUFFER_TIMEOUT, PREVIEW_FPS
```
### Using Utility Functions
When you need camera error suppression:
```python
from ...camera.utils import suppress_camera_errors
with suppress_camera_errors():
# Camera operations
```
---
## Related Documentation
- `CODE_QUALITY_IMPROVEMENTS.md` - Original analysis and suggestions
- `REFACTORING_PLAN.md` - Step-by-step implementation guide
---
## Lessons Learned
1. **Start Small**: The refactoring started with the simplest tasks (extracting duplicates and constants) before tackling the larger route split.
2. **Verify as You Go**: Each task was verified before moving to the next, preventing cascading errors.
3. **Fix Imports Systematically**: When splitting routes, import paths needed careful correction. Using relative imports requires counting directory levels carefully.
4. **Maintain Type Safety**: Added type hints to all route registration functions for better IDE support and error detection.
5. **Test Endpoints**: Always test actual API endpoints after refactoring to ensure no functionality was broken.
---
## Future Improvement Opportunities
While not included in this refactoring, potential future improvements:
1. **Move Debug Endpoint**: Consider moving `/debug/camera-manager` to `camera_routes.py` for better organization
2. **Extract WebSocket Manager**: Could move `WebSocketManager` to a separate module if it grows
3. **Route Unit Tests**: Add unit tests for route modules to prevent regressions
4. **API Documentation**: Consider adding OpenAPI/Swagger tags to organize routes in API docs
5. **More Constants**: Consider extracting more magic numbers as the codebase evolves
---
## Overall Project Status
### Frontend (React Dashboard)
✅ **Microfrontend architecture implemented**
- `video-remote` module extracted and working
- Module Federation configured and tested
- Feature flags system in place for gradual rollout
- Foundation ready for extracting additional modules
### Backend (Camera Management API)
✅ **Monolithic refactoring completed**
- Code organization significantly improved
- Routes split into domain modules
- Constants and utilities extracted
- 63% reduction in main server file size
- **Decision**: Maintained monolithic architecture (not split into microservices)
- All functionality preserved and verified
---
## Summary
### Frontend Modularization
✅ **Microfrontend architecture established**
✅ **Video library extracted as first module**
✅ **Independent deployment pipeline ready**
✅ **Scalable pattern for future feature extraction**
### Backend Refactoring
✅ **3 refactoring tasks completed successfully**
✅ **Code quality significantly improved**
✅ **No functionality broken**
✅ **63% reduction in main server file size**
✅ **Better code organization and maintainability**
✅ **Maintained monolithic architecture (intentional decision)**
### Overall
✅ **Frontend: Microfrontends** (independent modules, Module Federation)
✅ **Backend: Improved Monolith** (better organization, maintainability, no microservices)
The system now has a modular frontend with improved backend code quality, all while maintaining full backward compatibility and system stability.

339
docs/database_schema.md Normal file
View File

@@ -0,0 +1,339 @@
# USDA Vision Database Schema
This document provides a comprehensive overview of the database schema for the USDA Vision pecan processing experiment management system.
## Database Overview
- **Database Type**: PostgreSQL (via Supabase)
- **Version**: PostgreSQL 17
- **Authentication**: Supabase Auth with Row Level Security (RLS)
- **Extensions**: uuid-ossp for UUID generation
## Core Tables
### 1. Authentication & User Management
#### `auth.users` (Supabase managed)
- Standard Supabase authentication table
- Contains user authentication data
#### `public.roles`
```sql
CREATE TABLE public.roles (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
name TEXT UNIQUE NOT NULL CHECK (name IN ('admin', 'conductor', 'analyst', 'data recorder')),
description TEXT NOT NULL,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
```
**Roles:**
- `admin`: Full system access with user management capabilities
- `conductor`: Operational access for conducting experiments and managing data
- `analyst`: Read-only access for data analysis and reporting
- `data recorder`: Data entry and recording capabilities
#### `public.user_profiles`
```sql
CREATE TABLE public.user_profiles (
id UUID PRIMARY KEY REFERENCES auth.users(id) ON DELETE CASCADE,
email TEXT NOT NULL,
role_id UUID REFERENCES public.roles(id), -- Nullable (legacy)
status TEXT DEFAULT 'active' CHECK (status IN ('active', 'disabled')),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
```
#### `public.user_roles` (Junction Table)
```sql
CREATE TABLE public.user_roles (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
user_id UUID NOT NULL REFERENCES public.user_profiles(id) ON DELETE CASCADE,
role_id UUID NOT NULL REFERENCES public.roles(id) ON DELETE CASCADE,
assigned_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
assigned_by UUID REFERENCES public.user_profiles(id),
UNIQUE(user_id, role_id)
);
```
### 2. Experiment Management
#### `public.experiment_phases`
```sql
CREATE TABLE public.experiment_phases (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
name TEXT NOT NULL UNIQUE,
description TEXT, -- Optional description for the phase
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_by UUID NOT NULL REFERENCES public.user_profiles(id)
);
```
**Purpose**: Groups experiments into logical phases for better organization and navigation.
#### `public.experiments`
```sql
CREATE TABLE public.experiments (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
experiment_number INTEGER UNIQUE NOT NULL,
reps_required INTEGER NOT NULL CHECK (reps_required > 0),
soaking_duration_hr FLOAT NOT NULL CHECK (soaking_duration_hr >= 0),
air_drying_time_min INTEGER NOT NULL CHECK (air_drying_time_min >= 0),
plate_contact_frequency_hz FLOAT NOT NULL CHECK (plate_contact_frequency_hz > 0),
throughput_rate_pecans_sec FLOAT NOT NULL CHECK (throughput_rate_pecans_sec > 0),
crush_amount_in FLOAT NOT NULL CHECK (crush_amount_in >= 0),
entry_exit_height_diff_in FLOAT NOT NULL,
results_status TEXT NOT NULL DEFAULT 'valid' CHECK (results_status IN ('valid', 'invalid')),
completion_status BOOLEAN NOT NULL DEFAULT false,
phase_id UUID REFERENCES public.experiment_phases(id) ON DELETE SET NULL,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_by UUID NOT NULL REFERENCES public.user_profiles(id)
);
```
**Purpose**: Experiment blueprints/templates that define the parameters for pecan processing experiments.
#### `public.experiment_repetitions`
```sql
CREATE TABLE public.experiment_repetitions (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
experiment_id UUID NOT NULL REFERENCES public.experiments(id) ON DELETE CASCADE,
repetition_number INTEGER NOT NULL CHECK (repetition_number > 0),
scheduled_date TIMESTAMP WITH TIME ZONE,
schedule_status TEXT NOT NULL DEFAULT 'pending schedule'
CHECK (schedule_status IN ('pending schedule', 'scheduled', 'canceled', 'aborted')),
completion_status BOOLEAN NOT NULL DEFAULT false,
is_locked BOOLEAN NOT NULL DEFAULT false,
locked_at TIMESTAMP WITH TIME ZONE,
locked_by UUID REFERENCES public.user_profiles(id),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_by UUID NOT NULL REFERENCES public.user_profiles(id),
CONSTRAINT unique_repetition_per_experiment UNIQUE (experiment_id, repetition_number)
);
```
**Purpose**: Individual repetitions of experiment blueprints that can be scheduled and executed.
### 3. Data Entry System
#### `public.experiment_phase_drafts`
```sql
CREATE TABLE public.experiment_phase_drafts (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
experiment_id UUID NOT NULL REFERENCES public.experiments(id) ON DELETE CASCADE,
repetition_id UUID NOT NULL REFERENCES public.experiment_repetitions(id) ON DELETE CASCADE,
user_id UUID NOT NULL REFERENCES public.user_profiles(id),
phase_name TEXT NOT NULL CHECK (phase_name IN ('pre-soaking', 'air-drying', 'cracking', 'shelling')),
status TEXT NOT NULL DEFAULT 'draft' CHECK (status IN ('draft', 'submitted', 'withdrawn')),
draft_name TEXT, -- Optional name for the draft
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
submitted_at TIMESTAMP WITH TIME ZONE,
withdrawn_at TIMESTAMP WITH TIME ZONE
);
```
**Purpose**: Phase-specific draft records for experiment data entry with status tracking.
#### `public.experiment_phase_data`
```sql
CREATE TABLE public.experiment_phase_data (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
phase_draft_id UUID NOT NULL REFERENCES public.experiment_phase_drafts(id) ON DELETE CASCADE,
phase_name TEXT NOT NULL CHECK (phase_name IN ('pre-soaking', 'air-drying', 'cracking', 'shelling')),
-- Pre-soaking phase data
batch_initial_weight_lbs FLOAT CHECK (batch_initial_weight_lbs >= 0),
initial_shell_moisture_pct FLOAT CHECK (initial_shell_moisture_pct >= 0 AND initial_shell_moisture_pct <= 100),
initial_kernel_moisture_pct FLOAT CHECK (initial_kernel_moisture_pct >= 0 AND initial_kernel_moisture_pct <= 100),
soaking_start_time TIMESTAMP WITH TIME ZONE,
-- Air-drying phase data
airdrying_start_time TIMESTAMP WITH TIME ZONE,
post_soak_weight_lbs FLOAT CHECK (post_soak_weight_lbs >= 0),
post_soak_kernel_moisture_pct FLOAT CHECK (post_soak_kernel_moisture_pct >= 0 AND post_soak_kernel_moisture_pct <= 100),
post_soak_shell_moisture_pct FLOAT CHECK (post_soak_shell_moisture_pct >= 0 AND post_soak_shell_moisture_pct <= 100),
avg_pecan_diameter_in FLOAT CHECK (avg_pecan_diameter_in >= 0),
-- Cracking phase data
cracking_start_time TIMESTAMP WITH TIME ZONE,
-- Shelling phase data
shelling_start_time TIMESTAMP WITH TIME ZONE,
bin_1_weight_lbs FLOAT CHECK (bin_1_weight_lbs >= 0),
bin_2_weight_lbs FLOAT CHECK (bin_2_weight_lbs >= 0),
bin_3_weight_lbs FLOAT CHECK (bin_3_weight_lbs >= 0),
discharge_bin_weight_lbs FLOAT CHECK (discharge_bin_weight_lbs >= 0),
bin_1_full_yield_oz FLOAT CHECK (bin_1_full_yield_oz >= 0),
bin_2_full_yield_oz FLOAT CHECK (bin_2_full_yield_oz >= 0),
bin_3_full_yield_oz FLOAT CHECK (bin_3_full_yield_oz >= 0),
bin_1_half_yield_oz FLOAT CHECK (bin_1_half_yield_oz >= 0),
bin_2_half_yield_oz FLOAT CHECK (bin_2_half_yield_oz >= 0),
bin_3_half_yield_oz FLOAT CHECK (bin_3_half_yield_oz >= 0),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
CONSTRAINT unique_phase_per_draft UNIQUE (phase_draft_id, phase_name)
);
```
**Purpose**: Phase-specific measurement data for experiments.
#### `public.pecan_diameter_measurements`
```sql
CREATE TABLE public.pecan_diameter_measurements (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
phase_data_id UUID NOT NULL REFERENCES public.experiment_phase_data(id) ON DELETE CASCADE,
measurement_number INTEGER NOT NULL CHECK (measurement_number >= 1 AND measurement_number <= 10),
diameter_in FLOAT NOT NULL CHECK (diameter_in >= 0),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
CONSTRAINT unique_measurement_per_phase UNIQUE (phase_data_id, measurement_number)
);
```
**Purpose**: Individual pecan diameter measurements (up to 10 per phase).
### 3. Conductor Availability Management
#### `public.conductor_availability`
```sql
CREATE TABLE public.conductor_availability (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
user_id UUID NOT NULL REFERENCES public.user_profiles(id) ON DELETE CASCADE,
available_from TIMESTAMP WITH TIME ZONE NOT NULL,
available_to TIMESTAMP WITH TIME ZONE NOT NULL,
notes TEXT, -- Optional notes about the availability
status TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('active', 'cancelled')),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_by UUID NOT NULL REFERENCES public.user_profiles(id),
-- Ensure available_to is after available_from
CONSTRAINT valid_time_range CHECK (available_to > available_from),
-- Ensure availability is in the future (can be modified if needed for past records)
CONSTRAINT future_availability CHECK (available_from >= NOW() - INTERVAL '1 day')
);
```
**Purpose**: Stores conductor availability windows for experiment scheduling with overlap prevention.
#### `public.experiment_phase_assignments`
```sql
CREATE TABLE public.experiment_phase_assignments (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
experiment_id UUID NOT NULL REFERENCES public.experiments(id) ON DELETE CASCADE,
repetition_id UUID NOT NULL REFERENCES public.experiment_repetitions(id) ON DELETE CASCADE,
conductor_id UUID NOT NULL REFERENCES public.user_profiles(id) ON DELETE CASCADE,
phase_name TEXT NOT NULL CHECK (phase_name IN ('pre-soaking', 'air-drying', 'cracking', 'shelling')),
scheduled_start_time TIMESTAMP WITH TIME ZONE NOT NULL,
scheduled_end_time TIMESTAMP WITH TIME ZONE NOT NULL,
status TEXT NOT NULL DEFAULT 'scheduled' CHECK (status IN ('scheduled', 'in-progress', 'completed', 'cancelled')),
notes TEXT, -- Optional notes about the assignment
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_by UUID NOT NULL REFERENCES public.user_profiles(id),
-- Ensure scheduled_end_time is after scheduled_start_time
CONSTRAINT valid_scheduled_time_range CHECK (scheduled_end_time > scheduled_start_time),
-- Ensure unique assignment per conductor per phase per repetition
CONSTRAINT unique_conductor_phase_assignment UNIQUE (repetition_id, conductor_id, phase_name)
);
```
**Purpose**: Assigns conductors to specific experiment repetition phases with scheduled times for future scheduling functionality.
## Key Functions
### Authentication & Authorization
- `public.get_user_role()`: Returns the first role for a user (backward compatibility)
- `public.get_user_roles()`: Returns all roles for a user as an array
- `public.is_admin()`: Checks if user has admin role
- `public.has_role(role_name TEXT)`: Checks if user has specific role
- `public.has_any_role(role_names TEXT[])`: Checks if user has any of the specified roles
- `public.can_manage_experiments()`: Checks if user can manage experiments (admin or conductor)
### User Management
- `public.generate_temp_password()`: Generates secure temporary passwords
- `public.create_user_with_roles(user_email, role_names, temp_password)`: Creates users with multiple roles
### Data Validation
- `validate_repetition_number()`: Ensures repetition numbers don't exceed experiment requirements
- `public.check_repetition_lock_before_withdrawal()`: Prevents withdrawal of locked repetitions
- `public.check_availability_overlap()`: Prevents overlapping availabilities for the same conductor
- `public.adjust_overlapping_availability()`: Automatically adjusts overlapping availabilities (alternative approach)
### Availability Management
- `public.get_available_conductors(start_time, end_time)`: Returns conductors available for a specific time range
- `public.is_conductor_available(conductor_id, start_time, end_time)`: Checks if a conductor is available for a specific time range
### Timestamp Management
- `public.handle_updated_at()`: Updates the updated_at timestamp
- `public.handle_phase_draft_status_change()`: Manages submitted_at and withdrawn_at timestamps
## Row Level Security (RLS) Policies
### Access Control Summary
- **Admin**: Full access to all tables and operations
- **Conductor**: Can manage experiments, experiment phases, and repetitions, view all data, manage their own availability
- **Analyst**: Read-only access to all data
- **Data Recorder**: Can create and manage their own phase drafts and data
### Key RLS Features
- All authenticated users can view experiments, experiment phases, and repetitions
- Admin and conductor roles can manage experiment phases
- Users can only modify their own phase drafts (unless admin)
- Users can only manage their own availability (unless admin)
- Conductors can view their own experiment phase assignments
- Locked repetitions prevent draft modifications
- Submitted drafts cannot be withdrawn if repetition is locked
- Role-based access control for user management
## Indexes
The database includes comprehensive indexing for performance:
- Primary key indexes on all tables
- Foreign key indexes for joins
- Status and date indexes for filtering
- Composite indexes for common query patterns
## Constraints
### Data Integrity
- Check constraints on numeric values (weights, percentages, etc.)
- Enum constraints on status fields
- Unique constraints to prevent duplicates
- Foreign key constraints for referential integrity
### Business Rules
- Repetition numbers cannot exceed experiment requirements
- Only one submitted draft per user per phase per repetition
- Moisture percentages must be between 0-100
- Weights and measurements must be non-negative
- Conductor availabilities cannot overlap for the same user
- Availability windows must have valid time ranges (end > start)
- Only one conductor assignment per phase per repetition
- Scheduled times must have valid ranges (end > start)
## Migration History
The schema has evolved through several migrations:
1. **RBAC Schema**: Initial role-based access control
2. **Multiple Roles Support**: Enhanced user role management
3. **Experiments Table**: Core experiment management
4. **Repetitions System**: Separated experiments from repetitions
5. **Data Entry System**: Phase-specific draft and data management
6. **Constraint Fixes**: Refined business rules and constraints
7. **Experiment Phases**: Added experiment grouping for better organization
8. **Conductor Availability**: Added availability management and experiment phase assignments for scheduling
This schema supports a comprehensive pecan processing experiment management system with robust security, data integrity, and flexible role-based access control.

72
docs/rtsp_access_guide.md Normal file
View File

@@ -0,0 +1,72 @@
# RTSP Streaming Access Guide (Tailscale)
## Your Tailscale IP
**100.93.40.84**
## Access URLs
Replace `exp-dash` with `100.93.40.84` in all URLs when accessing from outside the VM.
### 1. MediaMTX Web Interface (Best for Testing)
```
http://100.93.40.84:8889/static/
```
This shows all available streams with a built-in player.
### 2. Direct WebRTC Player for Camera1
```
http://100.93.40.84:8889/camera1/webrtc
```
This is a browser-based player that works without VLC.
### 3. RTSP URL (for VLC/ffplay)
```
rtsp://100.93.40.84:8554/camera1
```
**For VLC:**
- File → Open Network Stream
- URL: `rtsp://100.93.40.84:8554/camera1`
- Or use: `rtsp://100.93.40.84:8554/camera1?transport=tcp`
**For ffplay:**
```bash
ffplay -rtsp_transport tcp rtsp://100.93.40.84:8554/camera1
```
## Important Notes
1. **Stream Timeout**: MediaMTX closes streams after ~10 seconds if no one is watching. Make sure to:
- Start RTSP first: `curl -X POST http://exp-dash:8000/cameras/camera1/start-rtsp`
- Then quickly open the viewer within 10 seconds
2. **Ports Exposed**:
- 8554: RTSP
- 8889: WebRTC/HTTP API
- 8189: WebRTC UDP
3. **Network**: Tailscale creates a VPN, so you can access these URLs from anywhere, as long as your client device is also on the same Tailscale network.
## Testing Steps
1. **Start the stream:**
```bash
curl -X POST http://exp-dash:8000/cameras/camera1/start-rtsp
```
2. **Quickly open a viewer** (within 10 seconds):
- Open browser: `http://100.93.40.84:8889/static/`
- Or: `http://100.93.40.84:8889/camera1/webrtc`
3. **Check status:**
```bash
./check_rtsp_status.sh
```
## Troubleshooting
If you see "no stream is available":
- The stream might have timed out (waited >10 seconds without viewer)
- Restart the stream and quickly connect
- Check logs: `docker compose logs api --tail 20 | grep RTSP`

59
docs/test_rtsp_working.md Normal file
View File

@@ -0,0 +1,59 @@
# RTSP Streaming is Working! 🎉
**Status**: ✅ RTSP streaming is functional and VLC can view the stream!
## Current Status
- ✅ FFmpeg is encoding and publishing frames
- ✅ MediaMTX is receiving the RTSP stream
- ✅ VLC can successfully view the stream via RTSP
- ⚠️ WebRTC requires the stream to be active when accessed
## Access Methods
### 1. RTSP (Working - Use This!)
```bash
rtsp://100.93.40.84:8554/camera1
```
**For VLC:**
- File → Open Network Stream
- URL: `rtsp://100.93.40.84:8554/camera1`
- Or with TCP: `rtsp://100.93.40.84:8554/camera1?transport=tcp`
### 2. WebRTC (Browser Player)
The WebRTC player needs the stream to be active when you open it.
**To use WebRTC:**
1. First, make sure RTSP is running:
```bash
curl -X POST http://exp-dash:8000/cameras/camera1/start-rtsp
```
2. Then quickly open (within 10 seconds):
```
http://100.93.40.84:8889/camera1/webrtc
```
**Note**: WebRTC uses POST requests to `/camera1/whep`, so 404/405 errors are normal if the stream isn't active.
## Troubleshooting WebRTC
If you see "stream not found" in the browser:
- The RTSP stream may have timed out
- Restart RTSP and immediately open the WebRTC URL
- MediaMTX closes streams after ~10 seconds without active viewers
## Quick Test Commands
```bash
# Check if RTSP is running
curl -X POST http://exp-dash:8000/cameras/camera1/start-rtsp
# Check stream status
curl -s http://localhost:8889/v2/paths/get/camera1 | python3 -m json.tool
# Full diagnostic
./diagnose_rtsp.sh
```