hotcoco¶

Perception evaluation for Python, written in Rust.

hotcoco evaluates perception models, starting with detection: boxes, masks, keypoints, and oriented boxes on the COCO, LVIS, and Open Images protocols. It's a drop-in replacement for pycocotools — same numbers to double precision, up to 36× faster — and it includes the analysis you'd otherwise need separate tools for: TIDE error analysis, confusion matrices, calibration, model comparison, and a dataset browser.

Get Started API Reference Open Notebook

Quick start¶

pip install hotcoco

PythonDrop-in replacementCLIRust

from hotcoco import COCO, COCOeval

coco_gt = COCO("instances_val2017.json")
coco_dt = coco_gt.load_res("detections.json")

ev = COCOeval(coco_gt, coco_dt, "bbox")
ev.run()

from hotcoco import init_as_pycocotools
init_as_pycocotools()

# All pycocotools imports now resolve to hotcoco
from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval

coco eval --gt instances_val2017.json --dt detections.json --iou-type bbox

use hotcoco::{COCO, COCOeval};
use hotcoco::params::IouType;
use std::path::Path;

let coco_gt = COCO::new(Path::new("instances_val2017.json"))?;
let coco_dt = coco_gt.load_res(Path::new("detections.json"))?;

let mut ev = COCOeval::new(coco_gt, coco_dt, IouType::Bbox);
ev.evaluate();
ev.accumulate();
ev.summarize();

Evaluate

COCO, LVIS, and Open Images protocols over boxes, masks, keypoints, and oriented boxes. init_as_pycocotools() patches existing pycocotools imports in place — no code changes.

Diagnose

TIDE error analysis, confusion matrices, confidence calibration, and per-image label-error detection — see what's actually costing you AP.

Explore your data

Browse any COCO dataset in a local web UI with annotation overlays, run a dataset healthcheck, and convert between COCO, YOLO, Pascal VOC, CVAT, DOTA, and Open Images CSV.

Compose

IoU kernels, matchers, and metric functions are public and work on plain numpy arrays. Every evaluation returns the same report shape, and each number is marked as standard or a hotcoco extension.

Performance¶

Bbox evaluation on COCO val2017 takes 0.14s; pycocotools takes 5.11s. Every COCO metric matches pycocotools to the limit of double precision.

Grouped bar chart of evaluation wall clock for bbox, segm and keypoints across the three libraries — COCO val2017, 36,781 detections.

Full tables, hardware, memory, and parity verification are in Benchmarks.

Error analysis¶

mAP tells you that your model misses; it doesn't tell you what to fix. hotcoco computes the breakdowns from the same evaluation pass: which error types cost the most AP (TIDE), which categories get confused with each other, whether the confidence scores are calibrated, and which images the model does worst on.

Row-normalized confusion matrix showing which COCO categories get mistaken for each other — Which categories the model confuses, and how much of the loss is background rather than a mix-up. See the confusion matrix guide.

Per-category AP as horizontal bars, best to worst — The 0.70 mean AP spans 0.96 (bed) to 0.14 (sports ball). See per-category AP.

The dataset browser¶

coco.browse() opens a local web UI that shows every image with its annotations overlaid, one color per category. Filter by category, zoom in, and scan a split for labeling problems without opening files one by one. Pass eval= and the same UI adds an interactive dashboard — PR curves, confusion matrix, TIDE errors, and per-image scores next to the images they come from.

The hotcoco dataset browser: a category filter sidebar beside a grid of thumbnails with colored annotation overlays — Every thumbnail is drawn with its annotations already overlaid. See the dataset browser guide.

Use the metrics directly¶

The metric functions don't require an evaluator — they're plain functions over arrays:

import numpy as np
from hotcoco import metrics

scores  = np.array([0.9, 0.8, 0.7, 0.6])
matched = np.array([True, False, True, True])

metrics.average_precision(scores, matched, num_gt=4)  # 0.6287
metrics.calibration_error(scores, matched)            # (ece, mce)

COCOeval calls these same functions internally, so numbers you compute by hand match what summarize() prints.

Panoptic and tracking are next, on the same engine — see the roadmap.