Remove deprecated files and scripts to streamline the codebase
- Deleted unused API test files, RTSP diagnostic scripts, and development utility scripts to reduce clutter. - Removed outdated database schema and modularization proposal documents to maintain focus on current architecture. - Cleaned up configuration files and logging scripts that are no longer in use, enhancing project maintainability.
This commit is contained in:
salirezav
427
docs/MODULARIZATION_PROPOSAL.md
Normal file
427
docs/MODULARIZATION_PROPOSAL.md
Normal file
@@ -0,0 +1,427 @@
|
||||
# Camera Management API - Modularization Proposal
|
||||
|
||||
## 📊 Current Architecture Analysis
|
||||
|
||||
### Current Structure
|
||||
The `camera-management-api` is currently a **monolithic service** with the following components:
|
||||
|
||||
1. **API Server** (`api/server.py`)
|
||||
- FastAPI REST endpoints
|
||||
- WebSocket real-time updates
|
||||
- Orchestrates all other components
|
||||
|
||||
2. **Camera Management** (`camera/`)
|
||||
- Camera discovery & initialization
|
||||
- Recording (CameraRecorder)
|
||||
- Streaming (CameraStreamer) - MJPEG & RTSP
|
||||
- Camera monitoring & recovery
|
||||
|
||||
3. **MQTT Client** (`mqtt/`)
|
||||
- Machine state monitoring
|
||||
- Event publishing
|
||||
|
||||
4. **Storage Manager** (`storage/`)
|
||||
- File indexing
|
||||
- Storage statistics
|
||||
- Cleanup operations
|
||||
|
||||
5. **Auto Recording Manager** (`recording/`)
|
||||
- Automated recording based on MQTT events
|
||||
- Standalone auto-recorder
|
||||
|
||||
6. **Core Services**
|
||||
- State Manager (in-memory state)
|
||||
- Event System (pub/sub)
|
||||
- Configuration management
|
||||
|
||||
7. **Video Services** (`video/`)
|
||||
- Video streaming
|
||||
- Metadata extraction
|
||||
- Caching
|
||||
|
||||
### Current Service Separation
|
||||
You already have:
|
||||
- ✅ `media-api` - Video processing (thumbnails, transcoding)
|
||||
- ✅ `mediamtx` - RTSP streaming server
|
||||
- ✅ Microfrontend dashboard (shell + video-remote)
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Modularization Strategies
|
||||
|
||||
### Strategy 1: **Modular Monolith** (Recommended to Start)
|
||||
|
||||
**Approach**: Keep as single service but improve internal structure with clear boundaries.
|
||||
|
||||
**Structure**:
|
||||
```
|
||||
camera-management-api/
|
||||
├── core/ # Shared infrastructure
|
||||
│ ├── state/
|
||||
│ ├── events/
|
||||
│ └── config/
|
||||
├── camera/ # Camera hardware layer
|
||||
├── recording/ # Recording logic
|
||||
├── streaming/ # Streaming logic (separate from camera)
|
||||
├── mqtt/ # MQTT integration
|
||||
├── storage/ # Storage operations
|
||||
└── api/ # API endpoints (orchestration layer)
|
||||
```
|
||||
|
||||
**Pros**:
|
||||
- ✅ Minimal disruption to working system
|
||||
- ✅ Easier debugging (single process)
|
||||
- ✅ Lower operational complexity
|
||||
- ✅ Shared state remains simple
|
||||
- ✅ No network latency between components
|
||||
- ✅ Easier to maintain consistency
|
||||
|
||||
**Cons**:
|
||||
- ❌ Can't scale components independently
|
||||
- ❌ All-or-nothing deployment
|
||||
- ❌ Single point of failure (mitigated by Docker)
|
||||
|
||||
**Best For**: Current state - proven system that works well together
|
||||
|
||||
---
|
||||
|
||||
### Strategy 2: **Strategic Microservices** (Hybrid Approach)
|
||||
|
||||
**Approach**: Split only high-value, independently scalable components.
|
||||
|
||||
**Services**:
|
||||
|
||||
#### Service 1: **camera-service** (Critical, Hardware-Dependent)
|
||||
```
|
||||
Responsibilities:
|
||||
- Camera discovery & initialization
|
||||
- Recording (CameraRecorder)
|
||||
- Streaming (CameraStreamer) - MJPEG
|
||||
- Camera monitoring & recovery
|
||||
- Hardware state management
|
||||
|
||||
Port: 8001
|
||||
Dependencies: Camera SDK, FFmpeg
|
||||
Network: host (for camera access)
|
||||
```
|
||||
|
||||
#### Service 2: **mqtt-service** (Stateless, Scalable)
|
||||
```
|
||||
Responsibilities:
|
||||
- MQTT client & subscriptions
|
||||
- Machine state monitoring
|
||||
- Event publishing
|
||||
|
||||
Port: 8002
|
||||
Dependencies: MQTT broker
|
||||
Stateless: Yes
|
||||
```
|
||||
|
||||
#### Service 3: **api-gateway** (Orchestration)
|
||||
```
|
||||
Responsibilities:
|
||||
- REST API endpoints
|
||||
- WebSocket server
|
||||
- Request routing to services
|
||||
- Aggregating responses
|
||||
|
||||
Port: 8000
|
||||
Dependencies: camera-service, mqtt-service, state-manager
|
||||
```
|
||||
|
||||
#### Service 4: **state-manager-service** (Optional - Shared State)
|
||||
```
|
||||
Responsibilities:
|
||||
- Centralized state management
|
||||
- Event bus/queue
|
||||
- State persistence
|
||||
|
||||
Port: 8003
|
||||
Database: Redis (recommended) or PostgreSQL
|
||||
```
|
||||
|
||||
**Pros**:
|
||||
- ✅ Camera service can be isolated/restarted independently
|
||||
- ✅ MQTT service is stateless and scalable
|
||||
- ✅ Clear separation of concerns
|
||||
- ✅ Can scale MQTT service separately
|
||||
- ✅ API gateway can handle load balancing
|
||||
|
||||
**Cons**:
|
||||
- ❌ More complex deployment
|
||||
- ❌ Network latency between services
|
||||
- ❌ State synchronization challenges
|
||||
- ❌ More containers to manage
|
||||
- ❌ Service discovery needed
|
||||
|
||||
**Best For**: Future scaling needs, when you need:
|
||||
- Multiple camera servers
|
||||
- High MQTT message volume
|
||||
- Different scaling requirements per component
|
||||
|
||||
---
|
||||
|
||||
### Strategy 3: **Full Microservices** (Advanced)
|
||||
|
||||
**Approach**: Split into granular services following domain boundaries.
|
||||
|
||||
**Services**:
|
||||
1. `camera-service` - Hardware control
|
||||
2. `recording-service` - Recording orchestration
|
||||
3. `streaming-service` - MJPEG/RTSP streaming
|
||||
4. `mqtt-service` - Machine state monitoring
|
||||
5. `auto-recording-service` - Automated recording logic
|
||||
6. `api-gateway` - API & routing
|
||||
7. `state-service` - Centralized state
|
||||
8. `storage-service` - File management
|
||||
|
||||
**Pros**:
|
||||
- ✅ Maximum flexibility
|
||||
- ✅ Independent scaling per service
|
||||
- ✅ Technology diversity (can use different languages)
|
||||
- ✅ Team autonomy
|
||||
|
||||
**Cons**:
|
||||
- ❌ Very complex
|
||||
- ❌ High operational overhead
|
||||
- ❌ Distributed system challenges (consistency, latency)
|
||||
- ❌ Overkill for current needs
|
||||
|
||||
**Best For**: Large team, complex requirements, need for maximum flexibility
|
||||
|
||||
---
|
||||
|
||||
## 🏆 Recommended Approach: **Incremental Modularization**
|
||||
|
||||
### Phase 1: **Internal Refactoring** (Current → 3 months)
|
||||
**Goal**: Improve code organization without breaking changes
|
||||
|
||||
1. **Separate concerns within monolith**:
|
||||
```
|
||||
camera/
|
||||
├── hardware/ # Camera SDK operations
|
||||
├── recording/ # Recording logic
|
||||
├── streaming/ # Streaming logic
|
||||
└── monitoring/ # Health checks
|
||||
```
|
||||
|
||||
2. **Use dependency injection**: Pass dependencies explicitly
|
||||
3. **Clear interfaces**: Define contracts between modules
|
||||
4. **Document boundaries**: Mark what can/can't be changed independently
|
||||
|
||||
**Outcome**: Cleaner code, easier to split later if needed
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: **Extract MQTT Service** (3-6 months)
|
||||
**Goal**: Split out stateless, independent component
|
||||
|
||||
**Why MQTT first?**
|
||||
- ✅ Completely stateless
|
||||
- ✅ No shared state with cameras
|
||||
- ✅ Easy to scale
|
||||
- ✅ Lower risk (doesn't affect camera operations)
|
||||
|
||||
**Implementation**:
|
||||
- Move `mqtt/` to separate service
|
||||
- Use Redis/RabbitMQ for event pub/sub
|
||||
- API Gateway queries MQTT service for status
|
||||
- MQTT service publishes to event bus
|
||||
|
||||
**Outcome**: First microservice, validates approach
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: **Evaluate Further Splitting** (6+ months)
|
||||
**Decision Point**: Based on actual needs
|
||||
|
||||
**If scaling cameras**:
|
||||
- Extract `camera-service` to run on multiple machines
|
||||
- Keep recording/streaming together (they're tightly coupled)
|
||||
|
||||
**If high API load**:
|
||||
- Keep API gateway separate
|
||||
- Scale gateway independently
|
||||
|
||||
**If complex state management**:
|
||||
- Extract `state-service` with Redis/PostgreSQL
|
||||
- Services query state service instead of in-memory
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Implementation Details
|
||||
|
||||
### Shared Infrastructure (All Strategies)
|
||||
|
||||
#### 1. **Event Bus** (Essential for microservices)
|
||||
```
|
||||
Option A: Redis Pub/Sub (lightweight)
|
||||
Option B: RabbitMQ (more features)
|
||||
Option C: MQTT (you already have it!)
|
||||
```
|
||||
|
||||
#### 2. **State Management**
|
||||
```
|
||||
Option A: Redis (fast, in-memory)
|
||||
Option B: PostgreSQL (persistent, queryable)
|
||||
Option C: Keep in-memory for now (simplest)
|
||||
```
|
||||
|
||||
#### 3. **Service Discovery**
|
||||
```
|
||||
For microservices:
|
||||
- Docker Compose service names (simple)
|
||||
- Consul/Eureka (if needed)
|
||||
- Kubernetes services (if migrating)
|
||||
```
|
||||
|
||||
#### 4. **API Gateway Pattern**
|
||||
```
|
||||
nginx/Envoy: Route requests to services
|
||||
FastAPI Gateway: Aggregate responses
|
||||
GraphQL: Alternative aggregation layer
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 Decision Matrix
|
||||
|
||||
| Factor | Modular Monolith | Strategic Split | Full Microservices |
|
||||
|--------|------------------|----------------|-------------------|
|
||||
| **Complexity** | ⭐ Low | ⭐⭐ Medium | ⭐⭐⭐ High |
|
||||
| **Scalability** | ⭐ Limited | ⭐⭐ Good | ⭐⭐⭐ Excellent |
|
||||
| **Development Speed** | ⭐⭐⭐ Fast | ⭐⭐ Medium | ⭐ Slow |
|
||||
| **Operational Overhead** | ⭐ Low | ⭐⭐ Medium | ⭐⭐⭐ High |
|
||||
| **Risk** | ⭐ Low | ⭐⭐ Medium | ⭐⭐⭐ High |
|
||||
| **Cost** | ⭐ Low | ⭐⭐ Medium | ⭐⭐⭐ High |
|
||||
| **Current Fit** | ⭐⭐⭐ Perfect | ⭐⭐ Good | ⭐ Overkill |
|
||||
|
||||
---
|
||||
|
||||
## 💡 My Recommendation
|
||||
|
||||
### **Start with Strategy 1: Modular Monolith + Internal Refactoring**
|
||||
|
||||
**Why?**
|
||||
1. ✅ Your system is **already working well**
|
||||
2. ✅ RTSP + Recording work concurrently (hard problem solved)
|
||||
3. ✅ No immediate scaling needs identified
|
||||
4. ✅ Single team managing it
|
||||
5. ✅ Lower risk, faster improvements
|
||||
|
||||
**What to do now:**
|
||||
1. **Refactor internal structure** (Phase 1)
|
||||
- Separate camera, recording, streaming modules
|
||||
- Clear interfaces between modules
|
||||
- Dependency injection
|
||||
|
||||
2. **Add event bus infrastructure** (prepare for future)
|
||||
- Set up Redis for events (even if monolith)
|
||||
- Publish events through Redis pub/sub
|
||||
- Services can subscribe when needed
|
||||
|
||||
3. **Monitor & Measure** (data-driven decisions)
|
||||
- Track performance metrics
|
||||
- Identify bottlenecks
|
||||
- Measure actual scaling needs
|
||||
|
||||
4. **Extract when needed** (not before)
|
||||
- Only split when you have concrete problems
|
||||
- Start with MQTT service (safest first split)
|
||||
- Then camera-service if scaling cameras
|
||||
|
||||
**Red Flags for Microservices** (when you DON'T need them):
|
||||
- ❌ "We might need to scale" (YAGNI - You Ain't Gonna Need It)
|
||||
- ❌ "Industry best practice" (without actual need)
|
||||
- ❌ "Multiple teams" (you have one team)
|
||||
- ❌ "Independent deployment" (current deployment is simple)
|
||||
|
||||
**Green Flags for Microservices** (when you DO need them):
|
||||
- ✅ Actually scaling cameras to multiple servers
|
||||
- ✅ High API load requiring independent scaling
|
||||
- ✅ Need to update camera logic without touching MQTT
|
||||
- ✅ Multiple teams working on different components
|
||||
- ✅ Need different technology stacks per service
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start: Internal Refactoring Plan
|
||||
|
||||
### Step 1: Create Module Boundaries
|
||||
|
||||
```
|
||||
usda_vision_system/
|
||||
├── camera/
|
||||
│ ├── hardware/ # Camera SDK wrapper
|
||||
│ │ ├── camera_sdk.py
|
||||
│ │ └── device_discovery.py
|
||||
│ ├── recording/ # Recording logic
|
||||
│ │ ├── recorder.py
|
||||
│ │ └── video_writer.py
|
||||
│ ├── streaming/ # Streaming logic
|
||||
│ │ ├── mjpeg_streamer.py
|
||||
│ │ └── rtsp_streamer.py
|
||||
│ └── monitoring/ # Health & recovery
|
||||
│ └── health_check.py
|
||||
```
|
||||
|
||||
### Step 2: Define Interfaces
|
||||
|
||||
```python
|
||||
# camera/domain/interfaces.py
|
||||
class ICameraHardware(ABC):
|
||||
@abstractmethod
|
||||
def initialize() -> bool
|
||||
@abstractmethod
|
||||
def capture_frame() -> Frame
|
||||
|
||||
class IRecorder(ABC):
|
||||
@abstractmethod
|
||||
def start_recording(filename: str) -> bool
|
||||
@abstractmethod
|
||||
def stop_recording() -> bool
|
||||
```
|
||||
|
||||
### Step 3: Dependency Injection
|
||||
|
||||
```python
|
||||
# Instead of direct instantiation
|
||||
recorder = CameraRecorder(config, state_manager, event_system)
|
||||
|
||||
# Use factories/interfaces
|
||||
recorder = RecorderFactory.create(config, dependencies)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📚 References & Further Reading
|
||||
|
||||
- **Modular Monolith**: https://www.kamilgrzybek.com/blog/posts/modular-monolith-primer
|
||||
- **Microservices Patterns**: https://microservices.io/patterns/
|
||||
- **When to Use Microservices**: https://martinfowler.com/articles/microservices.html
|
||||
|
||||
---
|
||||
|
||||
## ❓ Questions to Answer
|
||||
|
||||
Before deciding on microservices, ask:
|
||||
|
||||
1. **Do you need to scale components independently?**
|
||||
- If no → Monolith is fine
|
||||
|
||||
2. **Do different teams work on different parts?**
|
||||
- If no → Monolith is fine
|
||||
|
||||
3. **Are there actual performance bottlenecks?**
|
||||
- If no → Don't optimize prematurely
|
||||
|
||||
4. **Can you deploy the monolith easily?**
|
||||
- If yes → Monolith might be better
|
||||
|
||||
5. **Do you need different tech stacks per component?**
|
||||
- If no → Monolith is fine
|
||||
|
||||
---
|
||||
|
||||
**Bottom Line**: Your system is working well. Focus on **improving code quality and organization** rather than splitting prematurely. Extract services when you have **concrete, measurable problems** that require it.
|
||||
|
||||
Reference in New Issue
Block a user