Smoking Detection Pipeline

A robust, multi-stage computer vision pipeline designed to detect smoking events in video files, webcams, or real-time RTSP streams. The system uses a cascade of YOLO models with early-exit optimization for high precision and low latency.

🏗 Architecture

The pipeline operates in a "Fail-Fast" cascade with early exit optimization:

1. Broad Phase (Person Detection):

   • Detects humans in the frame.
   • Optimization: If no people are found, processing stops for that frame.

2. Narrow Phase (Cigarette Candidate Detection):

   • Crops the detected person (with configurable margin for context).
   • Runs a specialized YOLO detector to find small objects (cigarettes/lighters).
   • Early Exit: If confidence ≥ 0.8, skips verification phase entirely.
   • Filter: Rejects candidates smaller than 10x10 pixels.

3. Verification Phase (Classification):

   • Crops the candidate object with padding.
   • Runs a dedicated Binary Classifier to verify "Smoking" vs "Not Smoking".
   • Output: Only high-confidence events are flagged as alerts.

Key Features

• GPU Acceleration: Auto-detects CUDA and runs inference on GPU when available.
• Async Inference: Detection runs in a background thread; video output never lags.
• Early Exit: High-confidence detections skip the verification stage for faster alerts.
• Adaptive Sampling:
  • Idle Mode (2 FPS): When no people are detected, saves compute.
  • Active Mode (Max FPS): Switches to full speed when activity is detected.
  • Cooldown: Stays active for 2 seconds after people leave frame.
• Non-Maximum Suppression (NMS): Cleans up overlapping boxes per-class.
• Weighted Boxes Fusion (WBF): Alternative to NMS that merges overlapping boxes for better localization.
• Inference Budget Caps: Limits expensive model calls (e.g., max 3 person crops, max 5 verifications).
• Smart Event Recording: Captures 2s before + 3s after each smoking event as separate clips.
• Live Display: Real-time visualization window with 'q' to quit.
• Metrics Tracking: FPS, latency, detection counts with summary on exit.

🚀 Usage

Prerequisites

• Python 3.13+
• uv package manager (recommended)
• CUDA 13.0+ (optional, for GPU acceleration)

Installation

# Clone and enter directory
cd gp

# Install dependencies with uv
uv sync

Running the Server

The server runs the detection pipeline in the background while serving a REST API for the mobile app.

# Default: webcam 0, classic pipeline
uv run server.py

# Package entrypoint (same server)
uv run -m server

# Use stage-based pipeline with a specific camera
uv run server.py --staged --source 2

# RTSP camera
uv run server.py --source rtsp://admin:[email protected]:554/stream

# Custom host/port
uv run server.py --host 0.0.0.0 --port 9000

# Via uvicorn directly (uses env vars for config)
uvicorn server:app --host 0.0.0.0 --port 8000

Live + Clip Control (Server Runtime)

The server now runs a lifecycle manager for the live pipeline and a queued clip-job worker:

• Live manager: start/stop/pause/resume/switch source at runtime.
• Clip jobs: submit files to a queue, process one-at-a-time, track status.
• Resource policy: clip jobs can temporarily pause live processing and resume after job completion.

On startup the server prints the LAN address for the mobile app:

==================================================
  Server ready
  Local:    http://127.0.0.1:8000
  Network:  http://192.168.1.42:8000
  Hostname: http://lda.local:8000
==================================================

Configuration can also be set via environment variables:

Variable	Default	Description
CAMERA_SOURCE	0	Webcam index or RTSP
PERSON_MODEL	models/human-det-best.pt	Person model path
CIG_MODEL	models/cig-det-best.pt	Cigarette model path
VERIFY_MODEL	models/cig-cls-best.pt	Classifier path
STAGED	false	Use stage pipeline
USE_VLM	false	Enable VLM stage
VLM_API_KEY		API key for VLM
VLM_PROVIDER	openai	openai/anthropic

Running the Pipeline CLI (standalone)

# Run on a local video file
uv run main.py --source videos/test_footage.mp4

# Run on webcam (index 0)
uv run main.py --source 0 --display

# Run on webcam with live display, no file output
uv run main.py --source 0 --display --no-save

# Run on an RTSP stream (auto-detected)
uv run main.py --source rtsp://admin:[email protected]:554/stream

# Enable debug logging
uv run main.py --source videos/test.mp4 --debug

Check Available Webcams

uv run -m scripts.webcam_check

CLI Arguments

Argument	Default	Description
--source	Required	Video file, webcam index (0, 1), or RTSP URL.
--rtsp	False	Force RTSP mode (auto-detected for rtsp:// URLs).
--output	output.mp4	Path for the annotated output video.
--display	False	Show live preview window (press 'q' to quit).
--no-save	False	Skip video/event output (useful with --display).
--person-model	models/human-det-best.pt	Path to the Person Detection YOLO model.
--cig-model	models/cig-det-best.pt	Path to the Cigarette Detection YOLO model.
--verify-model	models/cig-cls-best.pt	Path to the Verification/Classifier model.
--debug	False	Enable verbose logging.
--staged	False	Use stage-based pipeline (modular, supports VLM).
--vlm	False	Enable VLM verification (GPT-4V, Claude Vision).
--vlm-key	env var	API key for VLM (or set OPENAI_API_KEY env var).
--vlm-provider	openai	VLM provider: openai or anthropic.
--vlm-model	gpt-4-vision-preview	VLM model name.

📊 Outputs

The system generates multiple artifacts during execution:

1. Visual Debug (output.mp4):

   • A video file with bounding boxes drawn for all stages.
   • Blue: Detected Person.
   • Yellow: Cigarette Candidate.
   • Red: Confirmed Smoking Event.

2. Frame Metadata (output.jsonl):

   • A synchronized JSON Lines file containing raw detection data for every frame.

3. Alert Log (alerts.jsonl):

   • A structured log file containing only confirmed smoking events.

4. Event Clips (events/event_<timestamp>.mp4):

   • Smart clips containing only the smoking incidents.
   • Includes 2 seconds of pre-event context and 3 seconds post-event.
   • Automatically extends if smoking continues.

5. Screenshots (screenshots/alert_<frame>_<conf>.jpg):

   • Annotated JPG images saved when alerts fire.
   • Useful for evidence/audit trails without full video.

6. Application Logs (logs/pipeline.log):

   • Rotating log files with detailed debug information.
   • Includes thread names, function calls, and timing data.

7. Metrics Summary (printed on exit):

   • Total frames, average FPS, latency, detection counts by label.

📂 Project Structure

gp/
├── server.py                # FastAPI server (runs pipeline + REST API)
├── server/                  # Modular server runtime + routes
│   ├── runtime.py           # PipelineManager + ClipJobManager
│   ├── routes.py            # API routers (/live/*, /clips/*, auth/history)
│   └── __main__.py          # `uv run -m server` entrypoint
├── main.py                  # CLI entry point (standalone pipeline)
├── pyproject.toml           # Project config (uv/pip)
├── models/                  # Directory for .pt model weights
├── logs/                    # Rotating application logs
├── events/                  # Smart event clips (auto-created)
├── screenshots/             # Alert evidence images
│
├── db/                      # Database abstraction layer
│   ├── repository.py        # Repository protocol + User/DetectionRecord
│   ├── sqlite.py            # SQLiteRepository (zero-setup default)
│   └── mysql.py             # MySQLRepository (optional)
│
├── pipeline/
│   ├── config.py            # PipelineConfig dataclass (all thresholds)
│   ├── core.py              # SmokingDetectionPipeline (classic, fixed stages)
│   ├── staged_core.py       # StageBasedPipeline (modular, VLM support)
│   ├── protocols.py         # Interfaces (Model, Verifier, Sink, StreamSource)
│   ├── types.py             # Data classes (Frame, Detection, Ticket, BoundingBox)
│   │
│   ├── models/              # ML model wrappers
│   │   └── yolo.py          # YOLO person/cig/classifier (GPU-enabled)
│   │
│   ├── sources/             # Stream inputs
│   │   ├── file.py          # FileStreamSource (video files, image sequences)
│   │   └── stream.py        # ThreadedStreamSource (RTSP, webcam)
│   │
│   ├── sinks/               # Output handlers
│   │   ├── alerts.py        # FileAlertStore (JSONL)
│   │   ├── database.py      # DatabaseSink (saves alerts to DB)
│   │   ├── display.py       # WindowDisplaySink (live preview)
│   │   ├── events.py        # SmartEventSink (clip recorder)
│   │   ├── metrics.py       # MetricsSink (FPS, latency, counts)
│   │   ├── screenshots.py   # ScreenshotSink (evidence images)
│   │   └── video.py         # VideoFileStore (async video writer)
│   │
│   ├── stages/              # Pluggable stage system
│   │   ├── protocol.py      # Stage protocol, StageContext, StageResult
│   │   ├── runner.py        # StagePipeline (executes stage chain)
│   │   ├── local.py         # LocalPersonDetector, LocalCigaretteDetector, LocalVerifier
│   │   └── api.py           # APIPersonDetector, VLMVerifier (cloud stages)
│   │
│   └── ops/                 # Algorithms & utilities
│       ├── nms.py           # NMS + Weighted Boxes Fusion (WBF)
│       ├── selection.py     # Budget-aware box selection (top-K, merge)
│       ├── geometry.py      # Box expansion, zone filtering
│       └── drawing.py       # Visualization helpers
│
├── scripts/
│   ├── webcam_check.py      # Utility to list available webcams
│   └── live_stress_test.py  # Stress test for /live/* and queue responsiveness
│
└── SmokeDetectionApp/       # Android companion app
    └── app/src/main/java/com/example/smokedetection/
        ├── LoginActivity     # Auth screen + hidden server config
        ├── MainActivity      # Dashboard with camera preview
        ├── HistoryActivity   # Alert history list
        ├── StreamActivity    # Live MJPEG viewer
        ├── ViewEvidenceActivity # Full-screen alert image
        ├── SmokeAlert        # Foreground polling service
        ├── HistoryAdapter    # RecyclerView adapter
        └── ApiClient         # Dynamic server URL + HTTP helpers

⚙️ Configuration

All pipeline thresholds are centralized in pipeline/config.py:

@dataclass
class PipelineConfig:
    # Detection thresholds
    person_nms_iou: float = 0.7
    cig_nms_iou: float = 0.5

    # Confidence thresholds
    cig_min_confidence: float = 0.2   # Minimum to consider
    cig_high_confidence: float = 0.8  # Skip verifier if above
    verify_threshold: float = 0.8     # Verifier must exceed

    # Adaptive sampling
    idle_interval: float = 0.5   # 2 FPS when idle
    active_interval: float = 0.0 # Max FPS when active
    cooldown_frames: int = 60    # Stay active after detection

    # Inference budget caps
    max_person_crops: int = 3       # Max cig model runs (0 = unlimited)
    person_priority: str = "area"   # "area", "confidence", "area_conf"
    merge_people: bool = False      # Merge overlapping people into regions
    max_cig_verifications: int = 5  # Max verifier runs (0 = unlimited)

Pass a custom config to the pipeline:

from pipeline.config import PipelineConfig

config = PipelineConfig(
    cig_high_confidence=0.9,  # Stricter early exit
    idle_interval=1.0,        # 1 FPS when idle
)

pipeline = SmokingDetectionPipeline(..., config=config)

API Endpoints

The server (server.py) exposes these endpoints:

Method	Endpoint	Description
GET	/ping	Health check (verify connectivity)
POST	/register	Create a new user account
POST	/login	Authenticate and get user ID
GET	/history	Get detection records (optionally by user_id)
POST	/clear_history	Delete detection records (optional user_id)
GET	/snapshot	Latest camera frame as JPEG
GET	/video_feed	MJPEG stream with detection overlay
GET	/live/status	Live pipeline state (running/paused/source)
POST	/live/start	Start live pipeline (optional source)
POST	/live/stop	Stop live pipeline
POST	/live/pause	Pause live pipeline
POST	/live/resume	Resume paused live pipeline
POST	/live/switch_source	Switch active webcam/RTSP source
POST	/clips/submit	Queue a clip job (upload file or local path)
GET	/clips	List recent clip jobs + queue stats
GET	/clips/{job_id}	Get status/details for a clip job

Interactive docs at http://localhost:8000/docs (Swagger UI).

Quick Test Calls

PowerShell examples:

$base='http://127.0.0.1:8000'

# Live controls
Invoke-WebRequest -UseBasicParsing -Uri ($base+'/live/status')
Invoke-WebRequest -UseBasicParsing -Method Post -Uri ($base+'/live/switch_source') -Body @{source='2'}
Invoke-WebRequest -UseBasicParsing -Method Post -Uri ($base+'/live/pause')
Invoke-WebRequest -UseBasicParsing -Method Post -Uri ($base+'/live/resume')

# Queue clip from local path
Invoke-WebRequest -UseBasicParsing -Method Post -Uri ($base+'/clips/submit') -Body @{path='test_media/139474-772542450_medium.mp4'}

# List jobs and inspect one job
Invoke-WebRequest -UseBasicParsing -Uri ($base+'/clips')
Invoke-WebRequest -UseBasicParsing -Uri ($base+'/clips/<job_id>')

curl examples:

curl http://127.0.0.1:8000/live/status
curl -X POST -F "source=2" http://127.0.0.1:8000/live/switch_source
curl -X POST -F "path=test_media/139474-772542450_medium.mp4" http://127.0.0.1:8000/clips/submit
curl http://127.0.0.1:8000/clips

📱 Android App

The companion app (SmokeDetectionApp/) connects to the server for auth, alert history, and live camera feed.

• Dynamic server URL: Tap the "Smoking Alert" title on the login screen 5 times to configure the server address. The URL is persisted and verified via /ping before saving.
• Background polling: SmokeAlert foreground service checks /history every 5 seconds and pushes a notification on new detections.
• Edge-to-edge: All activities draw behind system bars with proper inset handling.

🔧 Performance Tuning

For RTSP streams, the ThreadedStreamSource uses a background thread to prevent frame buildup and handles reconnection automatically.

Key tuning parameters:

• idle_interval: Increase to save GPU when scene is empty
• active_interval: Set to 0.0 for max detection rate
• cig_high_confidence: Lower for faster alerts, higher for fewer false positives

Inference Budget

When using lightweight (nano) models that produce many false positives, cap expensive downstream calls:

config = PipelineConfig(
    max_person_crops=3,        # Only process 3 people max per frame
    person_priority="area",    # Prioritize largest (closest to camera)
    merge_people=True,         # Merge overlapping people into regions
    merge_people_iou=0.3,      # IoU threshold for merging
    max_cig_verifications=5,   # Only verify top 5 cigarette candidates
)

Priority	Strategy
area	Largest boxes first (closer to camera, more detail)
confidence	Highest scoring boxes first
area_conf	Balanced: area × confidence

🔌 Pluggable Stage System

The --staged flag enables the modular stage pipeline, allowing you to mix local models with cloud APIs:

Stage Types

Stage	Description	Use Case
LocalPersonDetector	YOLO person detection	Default, GPU/CPU
LocalCigaretteDetector	YOLO cigarette detection	Default, GPU/CPU
LocalVerifier	YOLO classifier	Default, GPU/CPU
APIPersonDetector	Cloud person detection	No local GPU, scaling
VLMVerifier	GPT-4V/Claude Vision	High-stakes verification

VLM Verification Example

# Use GPT-4V for final verification
export OPENAI_API_KEY="sk-..."
uv run main.py --source 0 --display --vlm

# Use Claude Vision instead
export ANTHROPIC_API_KEY="sk-ant-..."
uv run main.py --source video.mp4 --vlm --vlm-provider anthropic --vlm-model claude-sonnet-4-20250514

Custom Stage Configuration

from pipeline.stages import (
    LocalPersonDetector,
    LocalCigaretteDetector,
    VLMVerifier,
    StagePipeline,
)

# Build custom stage chain
stages = [
    LocalPersonDetector(person_model, config),
    LocalCigaretteDetector(cig_model, config),
    # Skip local verifier, go straight to VLM
    VLMVerifier(api_key="sk-...", provider="openai"),
]

pipeline = StagePipeline(stages, config)
ctx = pipeline.run(frame)

if ctx.ticket:
    print(f"Smoking detected! Confidence: {ctx.ticket.confidence}")