A robust, multi-stage computer vision pipeline designed to detect smoking events in video files, webcams, or real-time RTSP streams. The system uses a cascade of YOLO models with early-exit optimization for high precision and low latency.

🏗 Architecture

The pipeline operates in a "Fail-Fast" cascade with early exit optimization:

Broad Phase (Person Detection):
- Detects humans in the frame.
- Optimization: If no people are found, processing stops for that frame.
Narrow Phase (Cigarette Candidate Detection):
- Crops the detected person (with configurable margin for context).
- Runs a specialized YOLO detector to find small objects (cigarettes/lighters).
- Early Exit: If confidence ≥ 0.8, skips verification phase entirely.
- Filter: Rejects candidates smaller than 10x10 pixels.
Verification Phase (Classification):
- Crops the candidate object with padding.
- Runs a dedicated Binary Classifier to verify "Smoking" vs "Not Smoking".
- Output: Only high-confidence events are flagged as alerts.

Key Features

GPU Acceleration: Auto-detects CUDA and runs inference on GPU when available.
Async Inference: Detection runs in a background thread; video output never lags.
Early Exit: High-confidence detections skip the verification stage for faster alerts.
Adaptive Sampling:
- Idle Mode (2 FPS): When no people are detected, saves compute.
- Active Mode (Max FPS): Switches to full speed when activity is detected.
- Cooldown: Stays active for 2 seconds after people leave frame.
Non-Maximum Suppression (NMS): Cleans up overlapping boxes per-class.
Weighted Boxes Fusion (WBF): Alternative to NMS that merges overlapping boxes for better localization.
Inference Budget Caps: Limits expensive model calls (e.g., max 3 person crops, max 5 verifications).
Smart Event Recording: Captures 2s before + 3s after each smoking event as separate clips.
Live Display: Real-time visualization window with 'q' to quit.
Metrics Tracking: FPS, latency, detection counts with summary on exit.

🚀 Usage

Prerequisites

Python 3.13+
uv package manager (recommended)
CUDA 13.0+ (optional, for GPU acceleration)

Installation

# Clone and enter directory
cd gp

# Install dependencies with uv
uv sync

Running the Pipeline

# Run on a local video file
uv run main.py --source videos/test_footage.mp4

# Run on webcam (index 0)
uv run main.py --source 0 --display

# Run on webcam with live display, no file output
uv run main.py --source 0 --display --no-save

# Run on an RTSP stream (auto-detected)
uv run main.py --source rtsp://admin:[email protected]:554/stream

# Enable debug logging
uv run main.py --source videos/test.mp4 --debug

Check Available Webcams

uv run -m tools.webcam_check

CLI Arguments

Argument	Default	Description
`--source`	Required	Video file, webcam index (0, 1), or RTSP URL.
`--rtsp`	`False`	Force RTSP mode (auto-detected for `rtsp://` URLs).
`--output`	`output.mp4`	Path for the annotated output video.
`--display`	`False`	Show live preview window (press 'q' to quit).
`--no-save`	`False`	Skip video/event output (useful with `--display`).
`--person-model`	`models/human-det-best.pt`	Path to the Person Detection YOLO model.
`--cig-model`	`models/cig-det-best.pt`	Path to the Cigarette Detection YOLO model.
`--verify-model`	`models/cig-cls-best.pt`	Path to the Verification/Classifier model.
`--debug`	`False`	Enable verbose logging.
`--staged`	`False`	Use stage-based pipeline (modular, supports VLM).
`--vlm`	`False`	Enable VLM verification (GPT-4V, Claude Vision).
`--vlm-key`	env var	API key for VLM (or set `OPENAI_API_KEY` env var).
`--vlm-provider`	`openai`	VLM provider: `openai` or `anthropic`.
`--vlm-model`	`gpt-4-vision-preview`	VLM model name.

📊 Outputs

The system generates multiple artifacts during execution:

Visual Debug (output.mp4):
- A video file with bounding boxes drawn for all stages.
- Blue: Detected Person.
- Yellow: Cigarette Candidate.
- Red: Confirmed Smoking Event.
Frame Metadata (output.jsonl):
- A synchronized JSON Lines file containing raw detection data for every frame.
Alert Log (alerts.jsonl):
- A structured log file containing only confirmed smoking events.
Event Clips (events/event_<timestamp>.mp4):
- Smart clips containing only the smoking incidents.
- Includes 2 seconds of pre-event context and 3 seconds post-event.
- Automatically extends if smoking continues.
Screenshots (screenshots/alert_<frame>_<conf>.jpg):
- Annotated JPG images saved when alerts fire.
- Useful for evidence/audit trails without full video.
Application Logs (logs/pipeline.log):
- Rotating log files with detailed debug information.
- Includes thread names, function calls, and timing data.
Metrics Summary (printed on exit):
- Total frames, average FPS, latency, detection counts by label.

📂 Project Structure

gp/
├── main.py                  # CLI Entry point
├── pyproject.toml           # Project config (uv/pip)
├── models/                  # Directory for .pt model weights
├── logs/                    # Rotating application logs
├── events/                  # Smart event clips (auto-created)
├── screenshots/             # Alert evidence images
│
├── pipeline/
│   ├── config.py            # PipelineConfig dataclass (all thresholds)
│   ├── core.py              # SmokingDetectionPipeline (classic, fixed stages)
│   ├── staged_core.py       # StageBasedPipeline (modular, VLM support)
│   ├── protocols.py         # Interfaces (Model, Verifier, Sink, StreamSource)
│   ├── types.py             # Data classes (Frame, Detection, Ticket, BoundingBox)
│   │
│   ├── models/              # ML model wrappers
│   │   └── yolo.py          # YOLO person/cig/classifier (GPU-enabled)
│   │
│   ├── sources/             # Stream inputs
│   │   ├── file.py          # FileStreamSource (video files, image sequences)
│   │   └── stream.py        # ThreadedStreamSource (RTSP, webcam)
│   │
│   ├── sinks/               # Output handlers
│   │   ├── alerts.py        # FileAlertStore (JSONL)
│   │   ├── display.py       # WindowDisplaySink (live preview)
│   │   ├── events.py        # SmartEventSink (clip recorder)
│   │   ├── metrics.py       # MetricsSink (FPS, latency, counts)
│   │   ├── screenshots.py   # ScreenshotSink (evidence images)
│   │   └── video.py         # VideoFileStore (async video writer)
│   │
│   ├── stages/              # Pluggable stage system
│   │   ├── protocol.py      # Stage protocol, StageContext, StageResult
│   │   ├── runner.py        # StagePipeline (executes stage chain)
│   │   ├── local.py         # LocalPersonDetector, LocalCigaretteDetector, LocalVerifier
│   │   └── api.py           # APIPersonDetector, VLMVerifier (cloud stages)
│   │
│   └── ops/                 # Algorithms & utilities
│       ├── nms.py           # NMS + Weighted Boxes Fusion (WBF)
│       ├── selection.py     # Budget-aware box selection (top-K, merge)
│       ├── geometry.py      # Box expansion, zone filtering
│       └── drawing.py       # Visualization helpers
│
└── tools/
    └── webcam_check.py      # Utility to list available webcams

⚙️ Configuration

All pipeline thresholds are centralized in pipeline/config.py:

@dataclass
class PipelineConfig:
    # Detection thresholds
    person_nms_iou: float = 0.7
    cig_nms_iou: float = 0.5

    # Confidence thresholds
    cig_min_confidence: float = 0.2   # Minimum to consider
    cig_high_confidence: float = 0.8  # Skip verifier if above
    verify_threshold: float = 0.8     # Verifier must exceed

    # Adaptive sampling
    idle_interval: float = 0.5   # 2 FPS when idle
    active_interval: float = 0.0 # Max FPS when active
    cooldown_frames: int = 60    # Stay active after detection

    # Inference budget caps
    max_person_crops: int = 3       # Max cig model runs (0 = unlimited)
    person_priority: str = "area"   # "area", "confidence", "area_conf"
    merge_people: bool = False      # Merge overlapping people into regions
    max_cig_verifications: int = 5  # Max verifier runs (0 = unlimited)

Pass a custom config to the pipeline:

from pipeline.config import PipelineConfig

config = PipelineConfig(
    cig_high_confidence=0.9,  # Stricter early exit
    idle_interval=1.0,        # 1 FPS when idle
)

pipeline = SmokingDetectionPipeline(..., config=config)

🔧 Performance Tuning

For RTSP streams, the ThreadedStreamSource uses a background thread to prevent frame buildup and handles reconnection automatically.

Key tuning parameters:

idle_interval: Increase to save GPU when scene is empty
active_interval: Set to 0.0 for max detection rate
cig_high_confidence: Lower for faster alerts, higher for fewer false positives

Inference Budget

When using lightweight (nano) models that produce many false positives, cap expensive downstream calls:

config = PipelineConfig(
    max_person_crops=3,        # Only process 3 people max per frame
    person_priority="area",    # Prioritize largest (closest to camera)
    merge_people=True,         # Merge overlapping people into regions
    merge_people_iou=0.3,      # IoU threshold for merging
    max_cig_verifications=5,   # Only verify top 5 cigarette candidates
)

Priority	Strategy
`area`	Largest boxes first (closer to camera, more detail)
`confidence`	Highest scoring boxes first
`area_conf`	Balanced: area × confidence

🔌 Pluggable Stage System

The --staged flag enables the modular stage pipeline, allowing you to mix local models with cloud APIs:

Stage Types

Stage	Description	Use Case
`LocalPersonDetector`	YOLO person detection	Default, GPU/CPU
`LocalCigaretteDetector`	YOLO cigarette detection	Default, GPU/CPU
`LocalVerifier`	YOLO classifier	Default, GPU/CPU
`APIPersonDetector`	Cloud person detection	No local GPU, scaling
`VLMVerifier`	GPT-4V/Claude Vision	High-stakes verification

VLM Verification Example

# Use GPT-4V for final verification
export OPENAI_API_KEY="sk-..."
uv run main.py --source 0 --display --vlm

# Use Claude Vision instead
export ANTHROPIC_API_KEY="sk-ant-..."
uv run main.py --source video.mp4 --vlm --vlm-provider anthropic --vlm-model claude-sonnet-4-20250514

Custom Stage Configuration

from pipeline.stages import (
    LocalPersonDetector,
    LocalCigaretteDetector,
    VLMVerifier,
    StagePipeline,
)

# Build custom stage chain
stages = [
    LocalPersonDetector(person_model, config),
    LocalCigaretteDetector(cig_model, config),
    # Skip local verifier, go straight to VLM
    VLMVerifier(api_key="sk-...", provider="openai"),
]

pipeline = StagePipeline(stages, config)
ctx = pipeline.run(frame)

if ctx.ticket:
    print(f"Smoking detected! Confidence: {ctx.ticket.confidence}")

README.md Unescape Escape

Smoking Detection Pipeline