Smoking Detection Pipeline

A robust, multi-stage computer vision pipeline designed to detect smoking events in video files, webcams, or real-time RTSP streams. The system uses a cascade of YOLO models with early-exit optimization for high precision and low latency.

🏗 Architecture

The pipeline operates in a "Fail-Fast" cascade with early exit optimization:

1. Broad Phase (Person Detection):

   • Detects humans in the frame.
   • Optimization: If no people are found, processing stops for that frame.

2. Narrow Phase (Cigarette Candidate Detection):

   • Crops the detected person (with configurable margin for context).
   • Runs a specialized YOLO detector to find small objects (cigarettes/lighters).
   • Early Exit: If confidence ≥ 0.8, skips verification phase entirely.
   • Filter: Rejects candidates smaller than 10x10 pixels.

3. Verification Phase (Classification):

   • Crops the candidate object with padding.
   • Runs a dedicated Binary Classifier to verify "Smoking" vs "Not Smoking".
   • Output: Only high-confidence events are flagged as alerts.

Key Features

• GPU Acceleration: Auto-detects CUDA and runs inference on GPU when available.
• Async Inference: Detection runs in a background thread; video output never lags.
• Early Exit: High-confidence detections skip the verification stage for faster alerts.
• Adaptive Sampling:
  • Idle Mode (2 FPS): When no people are detected, saves compute.
  • Active Mode (Max FPS): Switches to full speed when activity is detected.
  • Cooldown: Stays active for 2 seconds after people leave frame.
• Non-Maximum Suppression (NMS): Cleans up overlapping boxes per-class.
• Smart Event Recording: Captures 2s before + 3s after each smoking event as separate clips.
• Live Display: Real-time visualization window with 'q' to quit.
• Metrics Tracking: FPS, latency, detection counts with summary on exit.

🚀 Usage

Prerequisites

• Python 3.13+
• uv package manager (recommended)
• CUDA 13.0+ (optional, for GPU acceleration)

Installation

# Clone and enter directory
cd gp

# Install dependencies with uv
uv sync

Running the Pipeline

# Run on a local video file
uv run main.py --source videos/test_footage.mp4

# Run on webcam (index 0)
uv run main.py --source 0 --display

# Run on webcam with live display, no file output
uv run main.py --source 0 --display --no-save

# Run on an RTSP stream (auto-detected)
uv run main.py --source rtsp://admin:[email protected]:554/stream

# Enable debug logging
uv run main.py --source videos/test.mp4 --debug

Check Available Webcams

uv run -m tools.webcam_check

CLI Arguments

Argument	Default	Description
--source	Required	Video file, webcam index (0, 1), or RTSP URL.
--rtsp	False	Force RTSP mode (auto-detected for rtsp:// URLs).
--output	output.mp4	Path for the annotated output video.
--display	False	Show live preview window (press 'q' to quit).
--no-save	False	Skip video/event output (useful with --display).
--person-model	models/human-det-best.pt	Path to the Person Detection YOLO model.
--cig-model	models/cig-det-best.pt	Path to the Cigarette Detection YOLO model.
--verify-model	models/cig-cls-best.pt	Path to the Verification/Classifier model.
--debug	False	Enable verbose logging.
--staged	False	Use stage-based pipeline (modular, supports VLM).
--vlm	False	Enable VLM verification (GPT-4V, Claude Vision).
--vlm-key	env var	API key for VLM (or set OPENAI_API_KEY env var).
--vlm-provider	openai	VLM provider: openai or anthropic.
--vlm-model	gpt-4-vision-preview	VLM model name.

📊 Outputs

The system generates multiple artifacts during execution:

1. Visual Debug (output.mp4):

   • A video file with bounding boxes drawn for all stages.
   • Blue: Detected Person.
   • Yellow: Cigarette Candidate.
   • Red: Confirmed Smoking Event.

2. Frame Metadata (output.jsonl):

   • A synchronized JSON Lines file containing raw detection data for every frame.

3. Alert Log (alerts.jsonl):

   • A structured log file containing only confirmed smoking events.

4. Event Clips (events/event_<timestamp>.mp4):

   • Smart clips containing only the smoking incidents.
   • Includes 2 seconds of pre-event context and 3 seconds post-event.
   • Automatically extends if smoking continues.

5. Screenshots (screenshots/alert_<frame>_<conf>.jpg):

   • Annotated JPG images saved when alerts fire.
   • Useful for evidence/audit trails without full video.

6. Application Logs (logs/pipeline.log):

   • Rotating log files with detailed debug information.
   • Includes thread names, function calls, and timing data.

7. Metrics Summary (printed on exit):

   • Total frames, average FPS, latency, detection counts by label.

📂 Project Structure

gp/
├── main.py                  # CLI Entry point
├── pyproject.toml           # Project config (uv/pip)
├── models/                  # Directory for .pt model weights
├── logs/                    # Rotating application logs
├── events/                  # Smart event clips (auto-created)
├── screenshots/             # Alert evidence images
│
├── pipeline/
│   ├── config.py            # PipelineConfig dataclass (all thresholds)
│   ├── core.py              # SmokingDetectionPipeline (classic, fixed stages)
│   ├── staged_core.py       # StageBasedPipeline (modular, VLM support)
│   ├── protocols.py         # Interfaces (Model, Verifier, Sink, StreamSource)
│   ├── types.py             # Data classes (Frame, Detection, Ticket, BoundingBox)
│   │
│   ├── models/              # ML model wrappers
│   │   └── yolo.py          # YOLO person/cig/classifier (GPU-enabled)
│   │
│   ├── sources/             # Stream inputs
│   │   ├── file.py          # FileStreamSource (video files, image sequences)
│   │   └── stream.py        # ThreadedStreamSource (RTSP, webcam)
│   │
│   ├── sinks/               # Output handlers
│   │   ├── alerts.py        # FileAlertStore (JSONL)
│   │   ├── display.py       # WindowDisplaySink (live preview)
│   │   ├── events.py        # SmartEventSink (clip recorder)
│   │   ├── metrics.py       # MetricsSink (FPS, latency, counts)
│   │   ├── screenshots.py   # ScreenshotSink (evidence images)
│   │   └── video.py         # VideoFileStore (async video writer)
│   │
│   ├── stages/              # Pluggable stage system
│   │   ├── protocol.py      # Stage protocol, StageContext, StageResult
│   │   ├── runner.py        # StagePipeline (executes stage chain)
│   │   ├── local.py         # LocalPersonDetector, LocalCigaretteDetector, LocalVerifier
│   │   └── api.py           # APIPersonDetector, VLMVerifier (cloud stages)
│   │
│   └── ops/                 # Algorithms & utilities
│       ├── nms.py           # Non-Maximum Suppression
│       ├── geometry.py      # Box expansion, zone filtering
│       └── drawing.py       # Visualization helpers
│
└── tools/
    └── webcam_check.py      # Utility to list available webcams

⚙️ Configuration

All pipeline thresholds are centralized in pipeline/config.py:

@dataclass
class PipelineConfig:
    # Detection thresholds
    person_nms_iou: float = 0.7
    cig_nms_iou: float = 0.5

    # Confidence thresholds
    cig_min_confidence: float = 0.2   # Minimum to consider
    cig_high_confidence: float = 0.8  # Skip verifier if above
    verify_threshold: float = 0.8     # Verifier must exceed

    # Adaptive sampling
    idle_interval: float = 0.5   # 2 FPS when idle
    active_interval: float = 0.0 # Max FPS when active
    cooldown_frames: int = 60    # Stay active after detection

Pass a custom config to the pipeline:

from pipeline.config import PipelineConfig

config = PipelineConfig(
    cig_high_confidence=0.9,  # Stricter early exit
    idle_interval=1.0,        # 1 FPS when idle
)

pipeline = SmokingDetectionPipeline(..., config=config)

🔧 Performance Tuning

For RTSP streams, the ThreadedStreamSource uses a background thread to prevent frame buildup and handles reconnection automatically.

Key tuning parameters:

• idle_interval: Increase to save GPU when scene is empty
• active_interval: Set to 0.0 for max detection rate
• cig_high_confidence: Lower for faster alerts, higher for fewer false positives

🔌 Pluggable Stage System

The --staged flag enables the modular stage pipeline, allowing you to mix local models with cloud APIs:

Stage Types

Stage	Description	Use Case
LocalPersonDetector	YOLO person detection	Default, GPU/CPU
LocalCigaretteDetector	YOLO cigarette detection	Default, GPU/CPU
LocalVerifier	YOLO classifier	Default, GPU/CPU
APIPersonDetector	Cloud person detection	No local GPU, scaling
VLMVerifier	GPT-4V/Claude Vision	High-stakes verification

VLM Verification Example

# Use GPT-4V for final verification
export OPENAI_API_KEY="sk-..."
uv run main.py --source 0 --display --vlm

# Use Claude Vision instead
export ANTHROPIC_API_KEY="sk-ant-..."
uv run main.py --source video.mp4 --vlm --vlm-provider anthropic --vlm-model claude-sonnet-4-20250514

Custom Stage Configuration

from pipeline.stages import (
    LocalPersonDetector,
    LocalCigaretteDetector,
    VLMVerifier,
    StagePipeline,
)

# Build custom stage chain
stages = [
    LocalPersonDetector(person_model, config),
    LocalCigaretteDetector(cig_model, config),
    # Skip local verifier, go straight to VLM
    VLMVerifier(api_key="sk-...", provider="openai"),
]

pipeline = StagePipeline(stages, config)
ctx = pipeline.run(frame)

if ctx.ticket:
    print(f"Smoking detected! Confidence: {ctx.ticket.confidence}")