A robust, multi-stage computer vision pipeline designed to detect smoking events in video files, webcams, or real-time RTSP streams. The system uses a cascade of YOLO models with early-exit optimization for high precision and low latency.

🏗 Architecture

The pipeline operates in a "Fail-Fast" cascade with early exit optimization:

Broad Phase (Person Detection):
- Detects humans in the frame.
- Optimization: If no people are found, processing stops for that frame.
Narrow Phase (Cigarette Candidate Detection):
- Crops the detected person (with configurable margin for context).
- Runs a specialized YOLO detector to find small objects (cigarettes/lighters).
- Early Exit: If confidence ≥ 0.8, skips verification phase entirely.
- Filter: Rejects candidates smaller than 10x10 pixels.
Verification Phase (Classification):
- Crops the candidate object with padding.
- Runs a dedicated Binary Classifier to verify "Smoking" vs "Not Smoking".
- Output: Only high-confidence events are flagged as alerts.

Key Features

GPU Acceleration: Auto-detects CUDA and runs inference on GPU when available.
Async Inference: Detection runs in a background thread; video output never lags.
Early Exit: High-confidence detections skip the verification stage for faster alerts.
Adaptive Sampling:
- Idle Mode (2 FPS): When no people are detected, saves compute.
- Active Mode (Max FPS): Switches to full speed when activity is detected.
- Cooldown: Stays active for 2 seconds after people leave frame.
Non-Maximum Suppression (NMS): Cleans up overlapping boxes per-class.
Smart Event Recording: Captures 2s before + 3s after each smoking event as separate clips.
Live Display: Real-time visualization window with 'q' to quit.
Metrics Tracking: FPS, latency, detection counts with summary on exit.

🚀 Usage

Prerequisites

Python 3.13+
uv package manager (recommended)
CUDA 13.0+ (optional, for GPU acceleration)

Installation

# Clone and enter directory
cd gp

# Install dependencies with uv
uv sync

Running the Pipeline

# Run on a local video file
uv run main.py --source videos/test_footage.mp4

# Run on webcam (index 0)
uv run main.py --source 0 --display

# Run on webcam with live display, no file output
uv run main.py --source 0 --display --no-save

# Run on an RTSP stream (auto-detected)
uv run main.py --source rtsp://admin:[email protected]:554/stream

# Enable debug logging
uv run main.py --source videos/test.mp4 --debug

Check Available Webcams

uv run -m tools.webcam_check

CLI Arguments

Argument	Default	Description
`--source`	Required	Video file, webcam index (0, 1), or RTSP URL.
`--rtsp`	`False`	Force RTSP mode (auto-detected for `rtsp://` URLs).
`--output`	`output.mp4`	Path for the annotated output video.
`--display`	`False`	Show live preview window (press 'q' to quit).
`--no-save`	`False`	Skip video/event output (useful with `--display`).
`--person-model`	`models/human-det-best.pt`	Path to the Person Detection YOLO model.
`--cig-model`	`models/cig-det-best.pt`	Path to the Cigarette Detection YOLO model.
`--verify-model`	`models/cig-cls-best.pt`	Path to the Verification/Classifier model.
`--debug`	`False`	Enable verbose logging.

📊 Outputs

The system generates multiple artifacts during execution:

Visual Debug (output.mp4):
- A video file with bounding boxes drawn for all stages.
- Blue: Detected Person.
- Yellow: Cigarette Candidate.
- Red: Confirmed Smoking Event.
Frame Metadata (output.jsonl):
- A synchronized JSON Lines file containing raw detection data for every frame.
Alert Log (alerts.jsonl):
- A structured log file containing only confirmed smoking events.
Event Clips (events/event_<timestamp>.mp4):
- Smart clips containing only the smoking incidents.
- Includes 2 seconds of pre-event context and 3 seconds post-event.
- Automatically extends if smoking continues.
Screenshots (screenshots/alert_<frame>_<conf>.jpg):
- Annotated JPG images saved when alerts fire.
- Useful for evidence/audit trails without full video.
Application Logs (logs/pipeline.log):
- Rotating log files with detailed debug information.
- Includes thread names, function calls, and timing data.
Metrics Summary (printed on exit):
- Total frames, average FPS, latency, detection counts by label.

📂 Project Structure

gp/
├── main.py                  # CLI Entry point
├── pyproject.toml           # Project config (uv/pip)
├── models/                  # Directory for .pt model weights
├── logs/                    # Rotating application logs
├── events/                  # Smart event clips (auto-created)
├── screenshots/             # Alert evidence images
│
├── pipeline/
│   ├── config.py            # PipelineConfig dataclass (all thresholds)
│   ├── core.py              # SmokingDetectionPipeline (async inference)
│   ├── protocols.py         # Interfaces (Model, Verifier, Sink, StreamSource)
│   ├── types.py             # Data classes (Frame, Detection, Ticket, BoundingBox)
│   │
│   ├── models/              # ML model wrappers
│   │   └── yolo.py          # YOLO person/cig/classifier (GPU-enabled)
│   │
│   ├── sources/             # Stream inputs
│   │   ├── file.py          # FileStreamSource (video files, image sequences)
│   │   └── stream.py        # ThreadedStreamSource (RTSP, webcam)
│   │
│   ├── sinks/               # Output handlers
│   │   ├── alerts.py        # FileAlertStore (JSONL)
│   │   ├── display.py       # WindowDisplaySink (live preview)
│   │   ├── events.py        # SmartEventSink (clip recorder)
│   │   ├── metrics.py       # MetricsSink (FPS, latency, counts)
│   │   ├── screenshots.py   # ScreenshotSink (evidence images)
│   │   └── video.py         # VideoFileStore (async video writer)
│   │
│   └── ops/                 # Algorithms & utilities
│       ├── nms.py           # Non-Maximum Suppression
│       ├── geometry.py      # Box expansion, zone filtering
│       └── drawing.py       # Visualization helpers
│
└── tools/
    └── webcam_check.py      # Utility to list available webcams

⚙️ Configuration

All pipeline thresholds are centralized in pipeline/config.py:

@dataclass
class PipelineConfig:
    # Detection thresholds
    person_nms_iou: float = 0.7
    cig_nms_iou: float = 0.5

    # Confidence thresholds
    cig_min_confidence: float = 0.2   # Minimum to consider
    cig_high_confidence: float = 0.8  # Skip verifier if above
    verify_threshold: float = 0.8     # Verifier must exceed

    # Adaptive sampling
    idle_interval: float = 0.5   # 2 FPS when idle
    active_interval: float = 0.0 # Max FPS when active
    cooldown_frames: int = 60    # Stay active after detection

Pass a custom config to the pipeline:

from pipeline.config import PipelineConfig

config = PipelineConfig(
    cig_high_confidence=0.9,  # Stricter early exit
    idle_interval=1.0,        # 1 FPS when idle
)

pipeline = SmokingDetectionPipeline(..., config=config)

🔧 Performance Tuning

For RTSP streams, the ThreadedStreamSource uses a background thread to prevent frame buildup and handles reconnection automatically.

Key tuning parameters:

idle_interval: Increase to save GPU when scene is empty
active_interval: Set to 0.0 for max detection rate
cig_high_confidence: Lower for faster alerts, higher for fewer false positives