Compare commits

...
8 Commits
25 changed files with 1934 additions and 373 deletions

No files matched your search

+5
View File
@@ -0,0 +1,5 @@
# what the helly
DB_USER=smoke_admin
DB_PASSWORD=123456
DB_HOST=localhost
DB_NAME=smoke_detection
+1
View File
@@ -3,3 +3,4 @@
*.jsonl filter=lfs diff=lfs merge=lfs -text
*.png filter=lfs diff=lfs merge=lfs -text
*.jpg filter=lfs diff=lfs merge=lfs -text
*.db filter=lfs diff=lfs merge=lfs -text
+11 -1
View File
@@ -9,4 +9,14 @@ wheels/
# Virtual environments
.venv
Group_project_*
# Local sqlite
*.db*
# artifacts/logs i forgor what are the paths uhhhhhhh
alerts.jsonl
output.jsonl
output.mp4
static/alerts/
screenshots/
events/
logs/
+3
View File
@@ -0,0 +1,3 @@
[submodule "SmokeDetectionApp"]
path = SmokeDetectionApp
url = https://github.com/ldlda/SmokeDihtectionApp.git
+2 -2
View File
@@ -5,6 +5,6 @@
"**/*.rpymc": true,
"**/cache/": true
},
"pylint.args": ["--generated-members=cv2.*", "--extension-pkg-whitelist=cv2"],
// "editor.fontFamily": "'JetbrainsMono Nerd Font', Consolas, 'Courier New', monospace"
"pylint.args": ["--generated-members=cv2.*", "--extension-pkg-whitelist=cv2"]
// "editor.fontFamily": "'JetbrainsMono Nerd Font', Consolas, 'Courier New', monospace"
}
+128 -22
View File
@@ -7,19 +7,16 @@ A robust, multi-stage computer vision pipeline designed to detect smoking events
The pipeline operates in a "Fail-Fast" cascade with **early exit** optimization:
1. **Broad Phase (Person Detection)**:
- Detects humans in the frame.
- _Optimization_: If no people are found, processing stops for that frame.
2. **Narrow Phase (Cigarette Candidate Detection)**:
- Crops the detected person (with configurable margin for context).
- Runs a specialized YOLO detector to find small objects (cigarettes/lighters).
- _Early Exit_: If confidence ≥ 0.8, skips verification phase entirely.
- _Filter_: Rejects candidates smaller than 10x10 pixels.
3. **Verification Phase (Classification)**:
- Crops the candidate object with padding.
- Runs a dedicated Binary Classifier to verify "Smoking" vs "Not Smoking".
- _Output_: Only high-confidence events are flagged as alerts.
@@ -34,6 +31,8 @@ The pipeline operates in a "Fail-Fast" cascade with **early exit** optimization:
- **Active Mode (Max FPS)**: Switches to full speed when activity is detected.
- **Cooldown**: Stays active for 2 seconds after people leave frame.
- **Non-Maximum Suppression (NMS)**: Cleans up overlapping boxes per-class.
- **Weighted Boxes Fusion (WBF)**: Alternative to NMS that merges overlapping boxes for better localization.
- **Inference Budget Caps**: Limits expensive model calls (e.g., max 3 person crops, max 5 verifications).
- **Smart Event Recording**: Captures 2s before + 3s after each smoking event as separate clips.
- **Live Display**: Real-time visualization window with 'q' to quit.
- **Metrics Tracking**: FPS, latency, detection counts with summary on exit.
@@ -56,7 +55,52 @@ cd gp
uv sync
```
### Running the Pipeline
### Running the Server
The server runs the detection pipeline in the background while serving a REST API for the mobile app.
```bash
# Default: webcam 0, classic pipeline
uv run server.py
# Use stage-based pipeline with a specific camera
uv run server.py --staged --source 2
# RTSP camera
uv run server.py --source rtsp://admin:[email protected]:554/stream
# Custom host/port
uv run server.py --host 0.0.0.0 --port 9000
# Via uvicorn directly (uses env vars for config)
uvicorn server:app --host 0.0.0.0 --port 8000
```
On startup the server prints the LAN address for the mobile app:
```
==================================================
Server ready
Local: http://127.0.0.1:8000
Network: http://192.168.1.42:8000
Hostname: http://lda.local:8000
==================================================
```
Configuration can also be set via environment variables:
| Variable | Default | Description |
| :-------------- | :------------------------- | :------------------- |
| `CAMERA_SOURCE` | `0` | Webcam index or RTSP |
| `PERSON_MODEL` | `models/human-det-best.pt` | Person model path |
| `CIG_MODEL` | `models/cig-det-best.pt` | Cigarette model path |
| `VERIFY_MODEL` | `models/cig-cls-best.pt` | Classifier path |
| `STAGED` | `false` | Use stage pipeline |
| `USE_VLM` | `false` | Enable VLM stage |
| `VLM_API_KEY` | | API key for VLM |
| `VLM_PROVIDER` | `openai` | `openai`/`anthropic` |
### Running the Pipeline CLI (standalone)
```bash
# Run on a local video file
@@ -105,51 +149,50 @@ uv run -m tools.webcam_check
The system generates multiple artifacts during execution:
1. **Visual Debug (`output.mp4`)**:
- A video file with bounding boxes drawn for all stages.
- **Blue**: Detected Person.
- **Yellow**: Cigarette Candidate.
- **Red**: Confirmed Smoking Event.
2. **Frame Metadata (`output.jsonl`)**:
- A synchronized JSON Lines file containing raw detection data for every frame.
3. **Alert Log (`alerts.jsonl`)**:
- A structured log file containing **only** confirmed smoking events.
4. **Event Clips (`events/event_<timestamp>.mp4`)**:
- Smart clips containing only the smoking incidents.
- Includes 2 seconds of pre-event context and 3 seconds post-event.
- Automatically extends if smoking continues.
5. **Screenshots (`screenshots/alert_<frame>_<conf>.jpg`)**:
- Annotated JPG images saved when alerts fire.
- Useful for evidence/audit trails without full video.
6. **Application Logs (`logs/pipeline.log`)**:
- Rotating log files with detailed debug information.
- Includes thread names, function calls, and timing data.
7. **Metrics Summary** (printed on exit):
- Total frames, average FPS, latency, detection counts by label.
## 📂 Project Structure
```text
gp/
├── main.py # CLI Entry point
├── server.py # FastAPI server (runs pipeline + REST API)
├── main.py # CLI entry point (standalone pipeline)
├── pyproject.toml # Project config (uv/pip)
├── models/ # Directory for .pt model weights
├── logs/ # Rotating application logs
├── events/ # Smart event clips (auto-created)
├── screenshots/ # Alert evidence images
│
├── db/ # Database abstraction layer
│ ├── repository.py # Repository protocol + User/DetectionRecord
│ ├── sqlite.py # SQLiteRepository (zero-setup default)
│ └── mysql.py # MySQLRepository (optional)
│
├── pipeline/
│ ├── config.py # PipelineConfig dataclass (all thresholds)
│ ├── core.py # SmokingDetectionPipeline (classic, fixed stages)
@@ -166,6 +209,7 @@ gp/
│ │
│ ├── sinks/ # Output handlers
│ │ ├── alerts.py # FileAlertStore (JSONL)
│ │ ├── database.py # DatabaseSink (saves alerts to DB)
│ │ ├── display.py # WindowDisplaySink (live preview)
│ │ ├── events.py # SmartEventSink (clip recorder)
│ │ ├── metrics.py # MetricsSink (FPS, latency, counts)
@@ -179,12 +223,24 @@ gp/
│ │ └── api.py # APIPersonDetector, VLMVerifier (cloud stages)
│ │
│ └── ops/ # Algorithms & utilities
│ ├── nms.py # Non-Maximum Suppression
│ ├── nms.py # NMS + Weighted Boxes Fusion (WBF)
│ ├── selection.py # Budget-aware box selection (top-K, merge)
│ ├── geometry.py # Box expansion, zone filtering
│ └── drawing.py # Visualization helpers
│
└── tools/
└── webcam_check.py # Utility to list available webcams
├── tools/
│ └── webcam_check.py # Utility to list available webcams
│
└── SmokeDetectionApp/ # Android companion app
└── app/src/main/java/com/example/smokedetection/
├── LoginActivity # Auth screen + hidden server config
├── MainActivity # Dashboard with camera preview
├── HistoryActivity # Alert history list
├── StreamActivity # Live MJPEG viewer
├── ViewEvidenceActivity # Full-screen alert image
├── SmokeAlert # Foreground polling service
├── HistoryAdapter # RecyclerView adapter
└── ApiClient # Dynamic server URL + HTTP helpers
```
## ⚙️ Configuration
@@ -207,6 +263,12 @@ class PipelineConfig:
idle_interval: float = 0.5 # 2 FPS when idle
active_interval: float = 0.0 # Max FPS when active
cooldown_frames: int = 60 # Stay active after detection
# Inference budget caps
max_person_crops: int = 3 # Max cig model runs (0 = unlimited)
person_priority: str = "area" # "area", "confidence", "area_conf"
merge_people: bool = False # Merge overlapping people into regions
max_cig_verifications: int = 5 # Max verifier runs (0 = unlimited)
```
Pass a custom config to the pipeline:
@@ -222,6 +284,30 @@ config = PipelineConfig(
pipeline = SmokingDetectionPipeline(..., config=config)
```
## API Endpoints
The server (`server.py`) exposes these endpoints:
| Method | Endpoint | Description |
| :----- | :--------------- | :---------------------------------- |
| GET | `/ping` | Health check (verify connectivity) |
| POST | `/register` | Create a new user account |
| POST | `/login` | Authenticate and get user ID |
| GET | `/history` | Get detection records for a user |
| POST | `/clear_history` | Delete all records for a user |
| GET | `/snapshot` | Latest camera frame as JPEG |
| GET | `/video_feed` | MJPEG stream with detection overlay |
Interactive docs at **<http://localhost:8000/docs>** (Swagger UI).
## 📱 Android App
The companion app (`SmokeDetectionApp/`) connects to the server for auth, alert history, and live camera feed.
- **Dynamic server URL**: Tap the "Smoking Alert" title on the login screen **5 times** to configure the server address. The URL is persisted and verified via `/ping` before saving.
- **Background polling**: `SmokeAlert` foreground service checks `/history` every 5 seconds and pushes a notification on new detections.
- **Edge-to-edge**: All activities draw behind system bars with proper inset handling.
## 🔧 Performance Tuning
For RTSP streams, the `ThreadedStreamSource` uses a background thread to prevent frame buildup and handles reconnection automatically.
@@ -232,19 +318,39 @@ Key tuning parameters:
- `active_interval`: Set to 0.0 for max detection rate
- `cig_high_confidence`: Lower for faster alerts, higher for fewer false positives
### Inference Budget
When using lightweight (nano) models that produce many false positives, cap expensive downstream calls:
```python
config = PipelineConfig(
max_person_crops=3, # Only process 3 people max per frame
person_priority="area", # Prioritize largest (closest to camera)
merge_people=True, # Merge overlapping people into regions
merge_people_iou=0.3, # IoU threshold for merging
max_cig_verifications=5, # Only verify top 5 cigarette candidates
)
```
| Priority | Strategy |
| ------------ | --------------------------------------------------- |
| `area` | Largest boxes first (closer to camera, more detail) |
| `confidence` | Highest scoring boxes first |
| `area_conf` | Balanced: area × confidence |
## 🔌 Pluggable Stage System
The `--staged` flag enables the modular stage pipeline, allowing you to mix local models with cloud APIs:
### Stage Types
| Stage | Description | Use Case |
|:------|:------------|:---------|
| `LocalPersonDetector` | YOLO person detection | Default, GPU/CPU |
| `LocalCigaretteDetector` | YOLO cigarette detection | Default, GPU/CPU |
| `LocalVerifier` | YOLO classifier | Default, GPU/CPU |
| `APIPersonDetector` | Cloud person detection | No local GPU, scaling |
| `VLMVerifier` | GPT-4V/Claude Vision | High-stakes verification |
| Stage | Description | Use Case |
| :----------------------- | :----------------------- | :----------------------- |
| `LocalPersonDetector` | YOLO person detection | Default, GPU/CPU |
| `LocalCigaretteDetector` | YOLO cigarette detection | Default, GPU/CPU |
| `LocalVerifier` | YOLO classifier | Default, GPU/CPU |
| `APIPersonDetector` | Cloud person detection | No local GPU, scaling |
| `VLMVerifier` | GPT-4V/Claude Vision | High-stakes verification |
### VLM Verification Example
+1
Submodule SmokeDetectionApp added at 7ace087582.
+18
View File
@@ -0,0 +1,18 @@
"""Database abstraction layer.
Provides a Repository protocol and concrete implementations for SQLite and MySQL.
The server doesn't care which one it talks to.
"""
from db.repository import Repository
from db.sqlite import SQLiteRepository
__all__ = ["Repository", "SQLiteRepository"]
# Optional MySQL import (requires mysql-connector-python)
try:
from db.mysql import MySQLRepository
__all__.append("MySQLRepository")
except ImportError:
pass
+138
View File
@@ -0,0 +1,138 @@
"""MySQL implementation of the Repository."""
import os
from typing import Optional
import mysql.connector # type: ignore[import-untyped]
from db.repository import DetectionRecord, User
class MySQLRepository:
"""MySQL-backed repository. Needs a running MySQL server + .env config."""
def __init__(
self,
user: Optional[str] = None,
password: Optional[str] = None,
host: Optional[str] = None,
database: Optional[str] = None,
):
self.config = {
"user": user or os.getenv("DB_USER", "smoke_admin"),
"password": password or os.getenv("DB_PASSWORD", "123456"),
"host": host or os.getenv("DB_HOST", "localhost"),
"database": database or os.getenv("DB_NAME", "smoke_detection"),
}
def _connect(self):
return mysql.connector.connect(**self.config)
def initialize(self) -> None:
conn = self._connect()
cursor = conn.cursor()
cursor.execute(
"""CREATE TABLE IF NOT EXISTS users (
id INT AUTO_INCREMENT PRIMARY KEY,
username VARCHAR(50) UNIQUE NOT NULL,
password VARCHAR(255) NOT NULL
)"""
)
cursor.execute(
"""CREATE TABLE IF NOT EXISTS detections (
id INT AUTO_INCREMENT PRIMARY KEY,
user_id INT,
image_path VARCHAR(255) NOT NULL,
confidence FLOAT,
timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
camera_source VARCHAR(50) DEFAULT 'Laptop Webcam',
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
)"""
)
cursor.execute(
"INSERT IGNORE INTO users (username, password) VALUES ('admin', '123456')"
)
conn.commit()
conn.close()
# --- Users ---
def get_user_by_username(self, username: str) -> Optional[User]:
conn = self._connect()
cursor = conn.cursor(dictionary=True)
cursor.execute("SELECT * FROM users WHERE username = %s", (username,))
row = cursor.fetchone()
conn.close()
return User(**row) if row else None
def get_user_by_credentials(self, username: str, password: str) -> Optional[User]:
conn = self._connect()
cursor = conn.cursor(dictionary=True)
cursor.execute(
"SELECT * FROM users WHERE username = %s AND password = %s",
(username, password),
)
row = cursor.fetchone()
conn.close()
return User(**row) if row else None
def create_user(self, username: str, password: str) -> int:
conn = self._connect()
cursor = conn.cursor()
cursor.execute(
"INSERT INTO users (username, password) VALUES (%s, %s)",
(username, password),
)
conn.commit()
new_id = cursor.lastrowid
conn.close()
return new_id # type: ignore[return-value]
# --- Detections ---
def save_detection(
self,
user_id: int,
image_path: str,
confidence: float,
camera_source: str = "Laptop Webcam",
) -> int:
conn = self._connect()
cursor = conn.cursor()
cursor.execute(
"INSERT INTO detections (user_id, image_path, confidence, camera_source) VALUES (%s, %s, %s, %s)",
(user_id, image_path, confidence, camera_source),
)
conn.commit()
new_id = cursor.lastrowid
conn.close()
return new_id # type: ignore[return-value]
def get_detections(self, user_id: Optional[int] = None) -> list[DetectionRecord]:
conn = self._connect()
cursor = conn.cursor(dictionary=True)
if user_id is not None:
cursor.execute(
"SELECT * FROM detections WHERE user_id = %s ORDER BY timestamp DESC",
(user_id,),
)
else:
cursor.execute("SELECT * FROM detections ORDER BY timestamp DESC")
rows = cursor.fetchall()
conn.close()
return [DetectionRecord(**r) for r in rows]
def clear_detections(self, user_id: Optional[int] = None) -> int:
conn = self._connect()
cursor = conn.cursor()
if user_id is None:
cursor.execute("DELETE FROM detections")
else:
cursor.execute("DELETE FROM detections WHERE user_id = %s", (user_id,))
conn.commit()
count = cursor.rowcount
conn.close()
return count
def close(self) -> None:
pass # MySQL uses per-call connections
+68
View File
@@ -0,0 +1,68 @@
"""Repository protocol - the interface all DB backends must implement."""
from dataclasses import dataclass
from typing import Optional, Protocol, runtime_checkable
@dataclass
class User:
id: int
username: str
password: str # plaintext for now (legacy compat) - hash this later
@dataclass
class DetectionRecord:
id: int
user_id: int
image_path: str
confidence: float
timestamp: str
camera_source: str
@runtime_checkable
class Repository(Protocol):
"""Database operations for the smoking detection server."""
def initialize(self) -> None:
"""Create tables if they don't exist."""
...
# --- Users ---
def get_user_by_username(self, username: str) -> Optional[User]:
"""Find a user by username. Returns None if not found."""
...
def get_user_by_credentials(self, username: str, password: str) -> Optional[User]:
"""Find a user by username + password. Returns None if invalid."""
...
def create_user(self, username: str, password: str) -> int:
"""Create a new user. Returns the new user ID."""
...
# --- Detections ---
def save_detection(
self,
user_id: int,
image_path: str,
confidence: float,
camera_source: str = "Laptop Webcam",
) -> int:
"""Save a detection record. Returns the new record ID."""
...
def get_detections(self, user_id: Optional[int] = None) -> list[DetectionRecord]:
"""Get detection history, optionally filtered by user. Newest first."""
...
def clear_detections(self, user_id: Optional[int] = None) -> int:
"""Delete detections, optionally filtered by user. Returns rows deleted."""
...
def close(self) -> None:
"""Close the database connection."""
...
+126
View File
@@ -0,0 +1,126 @@
"""SQLite implementation of the Repository."""
import sqlite3
from pathlib import Path
from typing import Optional
from db.repository import DetectionRecord, User
# Schema that works with SQLite (no MySQL-specific syntax)
SCHEMA = """\
CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
username TEXT UNIQUE NOT NULL,
password TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS detections (
id INTEGER PRIMARY KEY AUTOINCREMENT,
user_id INTEGER,
image_path TEXT NOT NULL,
confidence REAL,
timestamp TEXT DEFAULT (datetime('now', 'localtime')),
camera_source TEXT DEFAULT 'Laptop Webcam',
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);
INSERT OR IGNORE INTO users (username, password) VALUES ('admin', '123456');
"""
class SQLiteRepository:
"""SQLite-backed repository. Zero setup, file-based."""
def __init__(self, db_path: str | Path = "data/smoke.db"):
self.db_path = Path(db_path)
self.db_path.parent.mkdir(parents=True, exist_ok=True)
self._conn: Optional[sqlite3.Connection] = None
def _get_conn(self) -> sqlite3.Connection:
if self._conn is None:
self._conn = sqlite3.connect(str(self.db_path), check_same_thread=False)
self._conn.row_factory = sqlite3.Row
self._conn.execute("PRAGMA journal_mode=WAL")
self._conn.execute("PRAGMA foreign_keys=ON")
return self._conn
def initialize(self) -> None:
conn = self._get_conn()
conn.executescript(SCHEMA)
conn.commit()
# --- Users ---
def get_user_by_username(self, username: str) -> Optional[User]:
row = (
self._get_conn()
.execute("SELECT * FROM users WHERE username = ?", (username,))
.fetchone()
)
return User(**dict(row)) if row else None
def get_user_by_credentials(self, username: str, password: str) -> Optional[User]:
row = (
self._get_conn()
.execute(
"SELECT * FROM users WHERE username = ? AND password = ?",
(username, password),
)
.fetchone()
)
return User(**dict(row)) if row else None
def create_user(self, username: str, password: str) -> int:
conn = self._get_conn()
cursor = conn.execute(
"INSERT INTO users (username, password) VALUES (?, ?)",
(username, password),
)
conn.commit()
return cursor.lastrowid # type: ignore[return-value]
# --- Detections ---
def save_detection(
self,
user_id: int,
image_path: str,
confidence: float,
camera_source: str = "Laptop Webcam",
) -> int:
conn = self._get_conn()
cursor = conn.execute(
"INSERT INTO detections (user_id, image_path, confidence, camera_source) VALUES (?, ?, ?, ?)",
(user_id, image_path, confidence, camera_source),
)
conn.commit()
return cursor.lastrowid # type: ignore[return-value]
def get_detections(self, user_id: Optional[int] = None) -> list[DetectionRecord]:
conn = self._get_conn()
if user_id is not None:
rows = conn.execute(
"SELECT * FROM detections WHERE user_id = ? ORDER BY timestamp DESC",
(user_id,),
).fetchall()
else:
rows = conn.execute(
"SELECT * FROM detections ORDER BY timestamp DESC"
).fetchall()
return [DetectionRecord(**dict(r)) for r in rows]
def clear_detections(self, user_id: Optional[int] = None) -> int:
conn = self._get_conn()
if user_id is None:
cursor = conn.execute("DELETE FROM detections")
else:
cursor = conn.execute(
"DELETE FROM detections WHERE user_id = ?", (user_id,)
)
conn.commit()
return cursor.rowcount
def close(self) -> None:
if self._conn:
self._conn.close()
self._conn = None
BIN
View File
Binary file not shown.
+18 -2
View File
@@ -2,6 +2,8 @@
from dataclasses import dataclass
from pipeline.ops.selection import SelectPriority
@dataclass
class PipelineConfig:
@@ -22,16 +24,30 @@ class PipelineConfig:
min_cig_size: int = 10 # Minimum cig box dimension
# Person crop expansion
person_margin: float = 0.2 # 20% expansion around person box
person_margin: float = 1.0 # 100% expansion around person box
# Verification crop padding
cig_crop_padding: float = 0.5 # 50% padding around cig for context
cig_crop_padding: float = 1.2 # 120% padding around cig for context
# Adaptive sampling
idle_interval: float = 0.5 # Seconds between inference when idle
active_interval: float = 0.0 # Seconds between inference when active
cooldown_frames: int = 60 # Frames to stay active after last detection
# === Inference Budget Caps ===
# Limits expensive model calls when there are too many detections
# Max person crops to run cig detection on (0 = unlimited)
max_person_crops: int = 3
# How to prioritize which people to process: "area", "confidence", "area_conf"
person_priority: SelectPriority = "area"
# Merge overlapping people into regions instead of processing separately
merge_people: bool = False
merge_people_iou: float = 0.3
# Max cigarette candidates to run verifier on (0 = unlimited)
max_cig_verifications: int = 5
# Default config instance
DEFAULT_CONFIG = PipelineConfig()
+13 -2
View File
@@ -17,12 +17,23 @@ from pipeline.ops.geometry import (
point_in_box,
point_in_polygon,
)
from pipeline.ops.nms import nms, nms_by_class
from pipeline.ops.nms import nms, nms_by_class, wbf, wbf_by_class
from pipeline.ops.selection import (
merge_overlapping,
select_for_verification,
select_top_k,
)
__all__ = [
# NMS
# NMS / Box Fusion
"nms",
"nms_by_class",
"wbf",
"wbf_by_class",
# Selection / Budget
"select_top_k",
"select_for_verification",
"merge_overlapping",
# Geometry
"expand_box",
"point_in_box",
+117 -1
View File
@@ -1,4 +1,4 @@
"""Non-Maximum Suppression algorithms for detection filtering."""
"""Box filtering algorithms: NMS and Weighted Boxes Fusion."""
from typing import List
@@ -59,3 +59,119 @@ def nms_by_class(
result.extend(nms(dets, iou_threshold))
return result
def wbf(
detections: List[Detection],
iou_threshold: float = 0.5,
skip_threshold: float = 0.0,
) -> List[Detection]:
"""
Weighted Boxes Fusion: Merge overlapping boxes instead of suppressing.
Unlike NMS which picks the best box and discards overlapping ones,
WBF fuses overlapping boxes by averaging their coordinates weighted
by confidence. This often gives better localization.
Args:
detections: List of Detection objects
iou_threshold: Boxes with IoU > threshold are fused together
skip_threshold: Ignore boxes with confidence below this
Returns:
Fused list of detections
Reference:
https://arxiv.org/abs/1910.13302
"""
if not detections:
return []
# Filter low confidence
dets = [d for d in detections if d.score >= skip_threshold]
if not dets:
return []
# Sort by confidence (highest first)
dets = sorted(dets, key=lambda d: d.score, reverse=True)
# Clusters of fused boxes
clusters: List[List[Detection]] = []
for det in dets:
matched_cluster = None
# Find a cluster this box belongs to
for cluster in clusters:
# Check IoU with the fused box (first element is always the fused result)
if cluster[0].box.iou(det.box) >= iou_threshold:
matched_cluster = cluster
break
if matched_cluster is not None:
# Add to existing cluster and recompute fused box
matched_cluster.append(det)
_update_fused_box(matched_cluster)
else:
# Start new cluster
clusters.append([det])
# Return the fused box from each cluster (first element)
return [cluster[0] for cluster in clusters]
def _update_fused_box(cluster: List[Detection]):
"""
Recompute the fused box for a cluster.
The fused box is stored as cluster[0].
"""
if len(cluster) == 1:
return
# Weighted average of coordinates
total_weight = sum(d.score for d in cluster)
x1 = sum(d.box.x1 * d.score for d in cluster) / total_weight
y1 = sum(d.box.y1 * d.score for d in cluster) / total_weight
x2 = sum(d.box.x2 * d.score for d in cluster) / total_weight
y2 = sum(d.box.y2 * d.score for d in cluster) / total_weight
# Fused confidence: average boosted by number of boxes
# More boxes agreeing = higher confidence (capped at 1.0)
avg_conf = total_weight / len(cluster)
fused_conf = min(1.0, avg_conf * (1 + 0.1 * (len(cluster) - 1)))
# Update the first element with fused values
fused = cluster[0]
fused.box.x1 = int(x1)
fused.box.y1 = int(y1)
fused.box.x2 = int(x2)
fused.box.y2 = int(y2)
fused.box.conf = fused_conf
fused.score = fused_conf
fused.metadata["wbf_count"] = len(cluster)
def wbf_by_class(
detections: List[Detection],
iou_threshold: float = 0.5,
skip_threshold: float = 0.0,
) -> List[Detection]:
"""
Run WBF separately for each class label.
This prevents boxes of different classes from being fused together.
"""
if not detections:
return []
# Group by label
by_label: dict[str, List[Detection]] = {}
for det in detections:
by_label.setdefault(det.label, []).append(det)
# Run WBF per class
result = []
for dets in by_label.values():
result.extend(wbf(dets, iou_threshold, skip_threshold))
return result
+174
View File
@@ -0,0 +1,174 @@
"""Budget-aware box selection for capping expensive inference calls."""
from typing import List, Literal
from pipeline.types import BoundingBox, Detection
type SelectPriority = Literal["area", "confidence", "area_conf"]
def select_top_k(
detections: List[Detection],
k: int,
diversity_iou: float = 0.5,
priority: SelectPriority = "area",
) -> List[Detection]:
"""
Select top K detections with spatial diversity.
Picks the K "best" boxes while ensuring they don't overlap too much.
This caps expensive downstream inference (e.g., max 3 person crops → cig model).
Args:
detections: Candidate detections
k: Maximum number to select
diversity_iou: Reject candidates overlapping selected boxes above this IoU
priority: How to rank candidates:
- "area": Larger boxes first (closer to camera, more detail)
- "confidence": Higher confidence first
- "area_conf": area * confidence (balanced)
Returns:
Up to K diverse, high-priority detections
"""
if not detections or k <= 0:
return []
# Score each detection
def score(det: Detection) -> float:
box = det.box
# area = (box.x2 - box.x1) * (box.y2 - box.y1)
area = box.area
if priority == "area":
return area
elif priority == "confidence":
return det.score
else: # area_conf
return area * det.score
# Sort by priority (highest first)
ranked = sorted(detections, key=score, reverse=True)
selected = []
for det in ranked:
if len(selected) >= k:
break
# Check diversity: don't select if overlaps too much with already selected
dominated = False
for sel in selected:
if det.box.iou(sel.box) > diversity_iou:
dominated = True
break
if not dominated:
selected.append(det)
return selected
def merge_overlapping(
detections: List[Detection],
iou_threshold: float = 0.3,
) -> List[Detection]:
"""
Merge overlapping boxes into larger regions.
Instead of processing each overlapping person separately, merge them
into one bigger crop. Good when people are clustered together.
Args:
detections: List of detections (e.g., people)
iou_threshold: Merge boxes with IoU above this
Returns:
Merged detections (fewer, larger boxes)
"""
if not detections:
return []
# Sort by confidence
dets = sorted(detections, key=lambda d: d.score, reverse=True)
clusters: List[List[Detection]] = []
for det in dets:
merged = False
for cluster in clusters:
# Check if this det overlaps with any box in cluster
for member in cluster:
if det.box.iou(member.box) >= iou_threshold:
cluster.append(det)
merged = True
break
if merged:
break
if not merged:
clusters.append([det])
# Create merged boxes from clusters
result = []
for cluster in clusters:
merged_det = _merge_cluster(cluster)
result.append(merged_det)
return result
def _merge_cluster(cluster: List[Detection]) -> Detection:
"""Merge a cluster of detections into one encompassing box."""
if len(cluster) == 1:
return cluster[0]
# Bounding box that contains all
x1 = min(d.box.x1 for d in cluster)
y1 = min(d.box.y1 for d in cluster)
x2 = max(d.box.x2 for d in cluster)
y2 = max(d.box.y2 for d in cluster)
# Confidence: max of cluster (could also use average)
conf = max(d.score for d in cluster)
# Use label from highest confidence detection
best = max(cluster, key=lambda d: d.score)
return Detection(
box=BoundingBox(x1=x1, y1=y1, x2=x2, y2=y2, conf=conf),
score=conf,
label=best.label,
metadata={
"merged_count": len(cluster),
"original_boxes": [
(d.box.x1, d.box.y1, d.box.x2, d.box.y2) for d in cluster
],
},
)
def select_for_verification(
detections: List[Detection],
k: int,
min_confidence: float = 0.0,
) -> List[Detection]:
"""
Select top K cigarette candidates for expensive verification.
Simple confidence-based selection for the verifier stage.
Higher confidence cigs are more likely real → verify those first.
Args:
detections: Cigarette candidates
k: Max verifications to run
min_confidence: Skip candidates below this
Returns:
Top K candidates by confidence
"""
if not detections or k <= 0:
return []
filtered = [d for d in detections if d.score >= min_confidence]
ranked = sorted(filtered, key=lambda d: d.score, reverse=True)
return ranked[:k]
+2
View File
@@ -1,6 +1,7 @@
"""Sinks for pipeline output handling."""
from pipeline.sinks.alerts import FileAlertStore
from pipeline.sinks.database import DatabaseSink
from pipeline.sinks.display import WindowDisplaySink
from pipeline.sinks.events import SmartEventSink
from pipeline.sinks.metrics import MetricsSink
@@ -8,6 +9,7 @@ from pipeline.sinks.screenshots import ScreenshotSink
from pipeline.sinks.video import VideoFileStore
__all__ = [
"DatabaseSink",
"FileAlertStore",
"MetricsSink",
"ScreenshotSink",
+74
View File
@@ -0,0 +1,74 @@
"""Database sink - saves detection alerts to the database."""
import logging
import time
from pathlib import Path
import cv2
from db.repository import Repository
from pipeline.ops.drawing import draw_alert_banner, draw_detections
from pipeline.types import Frame
logger = logging.getLogger(__name__)
class DatabaseSink:
"""
Pipeline sink that saves smoking alerts to a database.
On ticket (alert), saves an annotated screenshot and inserts a detection record.
"""
def __init__(
self,
db: Repository,
user_id: int = 1,
image_dir: str = "static/alerts",
camera_source: str = "Pipeline",
):
self.db = db
self.user_id = user_id
self.camera_source = camera_source
self.image_dir = Path(image_dir)
self.image_dir.mkdir(parents=True, exist_ok=True)
self.count = 0
logger.info("DatabaseSink initialized (user_id=%d, dir=%s)", user_id, image_dir)
def handle(self, frame: Frame):
if not frame.ticket:
return
self.count += 1
ticket = frame.ticket
# Save annotated image
filename = f"alert_{int(time.time())}_{self.count}.jpg"
filepath = self.image_dir / filename
img = frame.data.copy()
draw_detections(img, frame.detections)
draw_alert_banner(img, f"SMOKING {ticket.confidence:.0%}")
cv2.imwrite(str(filepath), img)
# Insert into database
web_path = f"/static/alerts/{filename}"
try:
self.db.save_detection(
user_id=self.user_id,
image_path=web_path,
confidence=ticket.confidence,
camera_source=self.camera_source,
)
logger.info(
"Alert #%d saved to DB: %s (%.2f)",
self.count,
filename,
ticket.confidence,
)
except Exception as e:
logger.error("Failed to save alert to DB: %s", e)
def close(self):
pass # DB connection is managed by the repository
+1
View File
@@ -149,6 +149,7 @@ class APIPersonDetector(APIStageBase):
return resp.json()
# haiku 4.5 is multimodal but i fear its going to ping 65 million images per second (we need cache control? who are we? why cant we upload video?)
class VLMVerifier(APIStageBase):
"""
Vision-Language Model verification stage.
+56 -6
View File
@@ -5,7 +5,12 @@ import time
from typing import List, Optional
from pipeline.config import DEFAULT_CONFIG, PipelineConfig
from pipeline.ops import nms_by_class
from pipeline.ops import (
merge_overlapping,
nms_by_class,
select_for_verification,
select_top_k,
)
from pipeline.protocols import Model, Verifier
from pipeline.stages.protocol import StageContext, StageResult
from pipeline.types import BoundingBox, Detection, Ticket
@@ -66,10 +71,39 @@ class LocalCigaretteDetector:
cfg = self.cfg
h_img, w_img = ctx.frame_data.shape[:2]
# === Budget: Limit how many people we process ===
people_to_process = ctx.people
if cfg.merge_people:
# Merge overlapping people into regions
people_to_process = merge_overlapping(
people_to_process, cfg.merge_people_iou
)
if len(people_to_process) < len(ctx.people):
logger.debug(
"Merged %d people into %d regions",
len(ctx.people),
len(people_to_process),
)
if cfg.max_person_crops > 0 and len(people_to_process) > cfg.max_person_crops:
# Select top K diverse people by priority
people_to_process = select_top_k(
people_to_process,
k=cfg.max_person_crops,
priority=cfg.person_priority,
)
logger.debug(
"Budget: Processing %d/%d people (priority=%s)",
len(people_to_process),
len(ctx.people),
cfg.person_priority,
)
all_cigs: List[Detection] = []
early_hit = None
for person in ctx.people:
for person in people_to_process:
# Expand person box for crop
crop_box = self._expand_box(person.box, cfg.person_margin, w_img, h_img)
ex1, ey1, ex2, ey2 = crop_box
@@ -195,12 +229,28 @@ class LocalVerifier:
cfg = self.cfg
h_img, w_img = ctx.frame_data.shape[:2]
# === Budget: Limit how many cigs we verify ===
cigs_to_verify = ctx.cigarettes
if (
cfg.max_cig_verifications > 0
and len(cigs_to_verify) > cfg.max_cig_verifications
):
# Select top K by confidence (most likely real → verify first)
cigs_to_verify = select_for_verification(
cigs_to_verify,
k=cfg.max_cig_verifications,
min_confidence=cfg.cig_min_confidence,
)
logger.debug(
"Budget: Verifying %d/%d cigarettes",
len(cigs_to_verify),
len(ctx.cigarettes),
)
confirmed: list[tuple[Detection, float]] = []
for cig_det in ctx.cigarettes:
offset = cig_det.metadata.get("crop_offset", (0, 0))
ex1, ey1 = offset
for cig_det in cigs_to_verify:
# Extract padded crop for classifier
crop = self._extract_cig_crop(ctx.frame_data, cig_det, w_img, h_img)
if crop.size == 0:
+1 -1
View File
@@ -60,7 +60,7 @@ class StagePipeline:
stage.name,
ctx.stop_reason,
)
continue
continue # break?
try:
result = stage.process(ctx)
+1 -1
View File
@@ -10,7 +10,7 @@ class BoundingBox:
y1: int
x2: int
y2: int
conf: float = 0.0
conf: float = 0.0 # legacy
@property
def area(self):
+6
View File
@@ -5,11 +5,17 @@ description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = [
"dotenv>=0.9.9",
"fastapi>=0.133.0",
"mysql-connector-python>=9.6.0",
"numpy>=2.4.0",
"opencv-python>=4.11.0.86",
"python-dotenv>=1.2.1",
"python-multipart>=0.0.22",
"torch>=2.9.1",
"torchvision>=0.24.1",
"ultralytics>=8.3.248",
"uvicorn>=0.41.0",
]
[[tool.uv.index]]
+385
View File
@@ -0,0 +1,385 @@
"""
Smoking Detection Server
========================
FastAPI server that runs the full detection pipeline on a live webcam feed
and exposes REST endpoints for the frontend (auth, history, video stream).
Usage:
uvicorn server:app --host 0.0.0.0 --port 8000
python server.py # dev mode
python server.py --staged # use stage-based pipeline
python server.py --source rtsp://... # RTSP camera
"""
import argparse
import logging
import os
import socket
import threading
import time
from contextlib import asynccontextmanager
from pathlib import Path
import cv2
from fastapi import FastAPI, Form, Response
from fastapi.responses import StreamingResponse
from fastapi.staticfiles import StaticFiles
from db import SQLiteRepository
from db.repository import Repository
from pipeline.config import PipelineConfig
from pipeline.core import SmokingDetectionPipeline
from pipeline.models import YoloCigaretteDetector, YoloClassifier, YoloPersonDetector
from pipeline.protocols import Sink
from pipeline.sinks import FileAlertStore, MetricsSink, SmartEventSink
from pipeline.sources.stream import ThreadedStreamSource
from pipeline.staged_core import StageBasedPipeline
from pipeline.types import Frame
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Global state (set during lifespan)
# ---------------------------------------------------------------------------
db: Repository = SQLiteRepository() # default: data/smoke.db
pipeline = None # set at startup
source: ThreadedStreamSource | None = None # set at startup
live_frame: Frame | None = None # latest frame for MJPEG / snapshot
# ---------------------------------------------------------------------------
# Sink that exposes frames to the web layer
# ---------------------------------------------------------------------------
class LiveFrameSink(Sink):
"""Captures the latest frame so API endpoints can stream it."""
def handle(self, frame: Frame):
global live_frame
live_frame = frame
def close(self):
pass
class DatabaseSmartEventSink(SmartEventSink):
"""Records event clips and stores clip path in DB as alert evidence."""
def __init__(
self,
db_repo: Repository,
fps: float,
user_id: int = 1,
camera_source: str = "Pipeline",
output_dir: str = "static/events",
):
super().__init__(output_dir=output_dir, fps=fps)
self.db_repo = db_repo
self.user_id = user_id
self.camera_source = camera_source
def _trigger_event(self, trigger_frame: Frame):
was_recording = self.recording
super()._trigger_event(trigger_frame)
# Save one DB row per newly started event clip
if was_recording or not self.recording or self.current_event_id is None:
return
ticket = trigger_frame.ticket
if not ticket:
return
filename = f"event_{self.current_event_id}.mp4"
web_path = f"/static/events/{filename}"
try:
self.db_repo.save_detection(
user_id=self.user_id,
image_path=web_path,
confidence=ticket.confidence,
camera_source=self.camera_source,
)
logger.info("Saved event alert to DB: %s", web_path)
except Exception as e:
logger.error("Failed to save event alert to DB: %s", e)
def _get_lan_address() -> tuple[str, str]:
"""Return (ip, hostname) for the primary LAN interface."""
ip = "127.0.0.1"
try:
# Open a UDP socket to an external IP (doesn't actually send anything)
# to figure out which local NIC the OS would route through.
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("8.8.8.8", 80))
ip = s.getsockname()[0]
s.close()
except Exception:
pass
hostname = socket.getfqdn()
return ip, hostname
# ---------------------------------------------------------------------------
# Pipeline runner (background thread)
# ---------------------------------------------------------------------------
def _run_pipeline():
"""Blocking call: runs in a daemon thread."""
assert pipeline is not None and source is not None
try:
pipeline.process_stream(source)
except Exception: # noqa: BLE001
logger.exception("Pipeline crashed")
# ---------------------------------------------------------------------------
# CLI args (parsed before uvicorn when run as __main__)
# ---------------------------------------------------------------------------
def parse_args():
p = argparse.ArgumentParser(description="Smoking Detection Server")
p.add_argument(
"--source", type=str, default="0", help="Webcam index or RTSP URL (default: 0)"
)
p.add_argument("--person-model", default="models/human-det-best.pt")
p.add_argument("--cig-model", default="models/cig-det-best.pt")
p.add_argument("--verify-model", default="models/cig-cls-best.pt")
p.add_argument("--staged", action="store_true", help="Use stage-based pipeline")
p.add_argument("--vlm", action="store_true")
p.add_argument("--vlm-key", default=None)
p.add_argument("--vlm-provider", default="openai", choices=["openai", "anthropic"])
p.add_argument("--vlm-model", default="gpt-4-vision-preview")
p.add_argument("--host", default="0.0.0.0")
p.add_argument("--port", type=int, default=8000)
p.add_argument("--debug", action="store_true")
return p.parse_args()
# Module-level defaults (overridden by __main__ block or env)
_args = argparse.Namespace(
source=os.environ.get("CAMERA_SOURCE", "0"),
person_model=os.environ.get("PERSON_MODEL", "models/human-det-best.pt"),
cig_model=os.environ.get("CIG_MODEL", "models/cig-det-best.pt"),
verify_model=os.environ.get("VERIFY_MODEL", "models/cig-cls-best.pt"),
staged=os.environ.get("STAGED", "").lower() in ("1", "true", "yes"),
vlm=os.environ.get("USE_VLM", "").lower() in ("1", "true", "yes"),
vlm_key=os.environ.get("VLM_API_KEY"),
vlm_provider=os.environ.get("VLM_PROVIDER", "openai"),
vlm_model=os.environ.get("VLM_MODEL", "gpt-4-vision-preview"),
debug=False,
)
# ---------------------------------------------------------------------------
# Lifespan - startup & shutdown
# ---------------------------------------------------------------------------
@asynccontextmanager
async def app_lifespan(_app: FastAPI):
global pipeline, source # noqa: PLW0603
logging.basicConfig(
level=logging.DEBUG if _args.debug else logging.INFO,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
datefmt="%H:%M:%S",
)
logging.getLogger("ultralytics").setLevel(logging.WARNING)
# 1. Database
logger.info("Initializing database...")
db.initialize()
# 2. Models
logger.info("Loading models...")
person_model = YoloPersonDetector(_args.person_model)
cig_model = YoloCigaretteDetector(_args.cig_model)
verifier = YoloClassifier(_args.verify_model)
logger.info("Models loaded.")
# 3. Source (webcam / RTSP)
cam = int(_args.source) if _args.source.isdigit() else _args.source
source = ThreadedStreamSource(cam)
logger.info("Camera source ready: %s (%.1f FPS)", _args.source, source.fps)
# 4. Sinks
sinks: list[Sink] = [
LiveFrameSink(),
DatabaseSmartEventSink(db, fps=source.fps, output_dir="static/events"),
MetricsSink(log_interval=200),
FileAlertStore("alerts.jsonl"),
]
# 5. Pipeline config
config = PipelineConfig(
idle_interval=0.5,
active_interval=0.0,
cooldown_frames=int(source.fps * 2),
)
# 6. Build pipeline
if _args.staged or _args.vlm:
vlm_key = _args.vlm_key or os.environ.get(
"OPENAI_API_KEY" if _args.vlm_provider == "openai" else "ANTHROPIC_API_KEY"
)
pipeline = StageBasedPipeline(
person_model=person_model,
cigarette_model=cig_model,
verifier_model=verifier,
vlm_api_key=vlm_key if _args.vlm else None,
vlm_provider=_args.vlm_provider,
vlm_model=_args.vlm_model,
use_vlm=_args.vlm,
sinks=sinks,
config=config,
)
else:
pipeline = SmokingDetectionPipeline(
person_model=person_model,
cigarette_model=cig_model,
verifier_model=verifier,
sinks=sinks,
config=config,
)
# 7. Launch pipeline in background thread
t = threading.Thread(target=_run_pipeline, daemon=True, name="pipeline")
t.start()
logger.info("Pipeline running in background.")
# Show real addresses
ip, hostname = _get_lan_address()
port = _args.port if hasattr(_args, "port") else 8000
logger.info("=" * 50)
logger.info(" Server ready")
logger.info(" Local: http://127.0.0.1:%d", port)
logger.info(" Network: http://%s:%d", ip, port)
if hostname and hostname != ip:
logger.info(" Hostname: http://%s:%d", hostname, port)
logger.info("=" * 50)
yield # --- app serves requests ---
# Shutdown
logger.info("Shutting down...")
source.stop()
for s in sinks:
s.close()
db.close()
logger.info("Goodbye.")
# ---------------------------------------------------------------------------
# FastAPI app
# ---------------------------------------------------------------------------
app = FastAPI(title="Smoking Detection", lifespan=app_lifespan) # type: ignore
# Serve static assets (alert images, CSS/JS, etc.)
Path("static/alerts").mkdir(parents=True, exist_ok=True)
Path("static/events").mkdir(parents=True, exist_ok=True)
app.mount("/static", StaticFiles(directory="static"), name="static")
# -- Auth ------------------------------------------------------------------
@app.get("/ping")
def ping():
"""Health check for the mobile app to verify connectivity."""
return {"status": "ok", "name": "SmokingDetection"}
@app.post("/register")
def register(username: str = Form(...), password: str = Form(...)):
try:
if db.get_user_by_username(username):
return {"status": "fail", "message": "Username already taken"}
new_id = db.create_user(username, password)
return {"status": "success", "user_id": new_id, "message": "Account created"}
except Exception as e:
return {"status": "error", "message": str(e)}
@app.post("/login")
def login(username: str = Form(...), password: str = Form(...)):
try:
user = db.get_user_by_credentials(username, password)
if user:
return {"status": "success", "user_id": user.id, "username": user.username}
return {"status": "fail", "message": "Invalid username or password"}
except Exception as e:
return {"status": "error", "message": str(e)}
# -- Detection history -----------------------------------------------------
@app.get("/history")
def get_history(user_id: int | None = None):
try:
records = db.get_detections(user_id)
return [
{
"id": r.id,
"user_id": r.user_id,
"image_path": r.image_path,
"confidence": r.confidence,
"timestamp": r.timestamp,
"camera_source": r.camera_source,
}
for r in records
]
except Exception:
return []
@app.post("/clear_history")
def clear_history():
try:
deleted = db.clear_detections()
logger.info("Cleared %d records", deleted)
return {"status": "success", "message": "History cleared"}
except Exception as e:
return {"status": "error", "message": str(e)}
# -- Video / Snapshot ------------------------------------------------------
@app.get("/snapshot")
def snapshot():
"""Return the latest frame as a JPEG."""
if live_frame is not None:
_, jpg = cv2.imencode(".jpg", live_frame.data)
return Response(content=jpg.tobytes(), media_type="image/jpeg")
return Response(status_code=404)
@app.get("/video_feed")
def video_feed():
"""MJPEG stream - drop into an <img> tag."""
def generate():
while True:
if live_frame is not None:
from pipeline.ops.drawing import draw_detections
img = live_frame.data.copy()
draw_detections(img, live_frame.detections)
_, jpg = cv2.imencode(".jpg", img)
yield (
b"--frame\r\n"
b"Content-Type: image/jpeg\r\n\r\n" + jpg.tobytes() + b"\r\n"
)
time.sleep(0.033) # ~30 FPS cap
return StreamingResponse(
generate(), media_type="multipart/x-mixed-replace;boundary=frame"
)
# ---------------------------------------------------------------------------
# Entry-point
# ---------------------------------------------------------------------------
if __name__ == "__main__":
import uvicorn
_args = parse_args()
uvicorn.run(app, host=_args.host, port=_args.port)
Generated
+583 -333
View File
File diff suppressed because it is too large. Load diff