Benchmarks¶

Feature comparison¶

Five libraries evaluate COCO-format detections from Python; three of them are Rust engines. Versions: pycocotools 2.0.11, faster-coco-eval 1.8.0, ultrafast-pycocotools 0.1.13, vernier 0.5.4, hotcoco main (1.3 development).

Speed benchmarks¶

Hardware: Apple M1 MacBook Air — 8 cores (4 performance + 4 efficiency), 8 GB RAM Dataset: COCO val2017 — 5,000 images Detections: 36,781 synthetic — see Methodology Timing: Wall clock time — per-cell median of 3 runs, each in a fresh process, at both 1× and 10×, captured back to back in one session (2026-10-07). Absolute times drift between captures on this machine, so compare them only within a table; the speedup ratios are not affected. Peak memory is each process's own resident set. Versions: pycocotools 2.0.11, faster-coco-eval 1.8.0, ultrafast-pycocotools 0.1.13, vernier 0.5.4, hotcoco 1.2.0.

Results (1x detections)¶

Speedups in parentheses are vs pycocotools.

Peak resident memory, same runs (MB):

The three Rust engines are 11–113× ahead of pycocotools, and hotcoco is fastest on every row. On memory it sits below vernier and within about 1.2× of ultrafast-pycocotools, which parses detections straight into columnar arrays while hotcoco materializes one annotation record per detection.

Results (10x detections)¶

Scaling detections by 10x (~368,000) to test behavior under higher load:

Peak resident memory, same runs (MB):

The results file is read a few megabytes at a time, so at this scale the peak is set by the evaluation, not the load: hotcoco holds about 210 bytes per detection with no slack around it, the evaluator shares the datasets rather than copying them, and the peak is within 1.2× of ultrafast-pycocotools' on bbox and segm and below it on keypoints. On keypoints, the 325 MB results file no longer sits in memory beside its records.

Absolute times stay under 0.3s at 368,000 detections, and hotcoco is fastest on every row.

StreamingEval holds less still. On Objects365 val with 1M synthetic detections, matching each batch as it arrives and keeping only the lean per-pair records peaks at 270 MB above the loaded ground truth, against 739 MB for collecting every prediction and evaluating at the end. The numbers are identical either way, whatever the batch size.

Where the time goes¶

The preceding end-to-end numbers blend two very different phases: load (JSON parsing and index building — COCO() + loadRes) and eval (evaluate + accumulate + summarize). Splitting them shows where each library spends its time (same protocol, same synthetic detections as the 1× table):

hotcoco's evaluation phase is 186–378× faster than pycocotools'; the end-to-end headline is lower because loading, which no library can speed up by much more than 10×, is a large share of hotcoco's total.

The bbox-only GT row strips the polygon segmentation the official instances files carry on every annotation — about two-thirds of the file bytes, which bbox evaluation never reads. It represents datasets that never had masks (custom bbox datasets, YOLO conversions, Objects365), where the load column is the one you'll actually see.

Objects365 scale benchmark¶

Hardware: Windows 11, AMD Ryzen 5 5600X — 6 cores / 12 threads, 16 GB RAM + swap Dataset: Objects365 val — 80,000 images, 1.2M annotations, 365 categories Detections: ~1.2M synthetic bbox (capped at 100/image, seed=42) Timing: Wall clock time, single run Versions: pycocotools 2.0.11, faster-coco-eval 1.7.2, hotcoco 0.3.0 — this run predates the 1.0 loading work, so the hotcoco figure is conservative

Peak RAM is the peak working set (physical memory). Committed includes swap — both pycocotools and faster-coco-eval exceeded physical RAM and relied heavily on the pagefile, which significantly inflated their wall clock times. hotcoco completed within physical memory with minimal swap.

Metric parity¶

Reference: pycocotools 2.0.11, numpy 2.4.3. Ground truth: COCO val2017 — 5,000 images. Detections: the published instances_val2017_fake*_results.json files from ppwwyyxx/cocoapi, the same inputs pycocotools uses in its own tests.

The precision, recall, and scores arrays are bit-identical to pycocotools' on all three evaluation types — every cell, checked with an integer compare of the raw f64 bits. The summary metrics below are bit-identical too: they are means over those identical arrays, and hotcoco sums them in the order numpy's mean does — pairwise, over the same flattened elements. The diffs in the following tables are raw measured differences, not rounded.

Bounding box¶

The threshold grids are constructed to match numpy.linspace bit-for-bit, which removed the last systematic source of divergence here.

Segmentation¶

Exact. The residual segmentation once carried (AP ~1e-5) came from polygon rasterization, where the reference's C compiler contracts s*t+ys into a single fused multiply-add on arm64 but not on x86-64, whose PyPI wheels target a baseline without the instruction. hotcoco mirrors that choice per architecture, so the comparison is exact on either kind of machine. Masks can differ by a boundary pixel between the two architectures — as pycocotools' own do. The other two Rust evaluators do not mirror the contraction: on Apple Silicon, ultrafast-pycocotools 0.1.11 and vernier 0.5.4 differ from the arm64 pycocotools wheel in 283 cells of the segm precision tensor on these detections, by up to 0.72 per cell and 1e-5 in AP (measured 2026-09-25); on x86-64 all three agree.

Keypoints¶

Keypoint metrics are exact. Keypoint evaluation reports 10 metrics — see Keypoint evaluation.

TIDE¶

Reference: tidecv, on COCO val2017 at pos_thr=0.5. On the 4,589 images without a crowd region, every ΔAP agrees within ±0.005 and every error count matches. On all 5,000 images, Loc reads about 0.01 lower here and Miss about 0.007 higher; the other types still agree within ±0.005:

The ranking — which error type is costing you the most AP — is the same, and that is what the metric is for. The difference comes from crowd handling: hotcoco builds TIDE on the same COCO-convention matching as its AP (so tide_errors() and ev.stats always agree about which detections exist), while tidecv removes crowd regions from matching entirely — its false-positive counts run higher and its Miss runs lower as a result. scripts/parity_tide.py runs both comparisons.

Open Images¶

Reference: the TensorFlow Object Detection API — the implementation the official protocol page points to.

Open Images evaluation is compared over 70 cases covering group-of absorption, IoA containment at and around the 0.5 boundary, undetected group-of boxes, overlapping group-of boxes, and randomized multi-class scenes. Both mAP and per-class AP are compared, and every case agrees to within one ulp (worst difference 1.11e-16). It runs in CI on every commit.

Two things that comparison does not cover, and why provenance still reports "extension" for Open Images runs:

• Non-exhaustive image-level labels. The challenge ignores detections of a class not verified on an image, and counts detections of a negatively-labeled class as false positives. hotcoco does not implement this — it needs per-image label data that COCO-format JSON cannot carry. A real challenge submission would score differently.
• Hierarchy expansion is applied to annotations before evaluation rather than inside it, so it sits outside the compared surface.

Panoptic¶

Reference: panopticapi at commit 7bb4655, numpy 2.4.3. Ground truth: COCO panoptic val2017 — 5,000 images, 133 categories. Predictions: the ground truth perturbed deterministically by scripts/download_panoptic.py — segments dropped, relabeled, eroded, shifted, and merged, plus spurious segments — so that every matching rule fires and PQ sits well inside (0, 1).

Below the averages, every per-category TP, FP, and FN count is identical and the summed IoUs agree to 9.1e-12. The averages:

The differences that are not zero are summation order: panopticapi splits the images across worker processes and adds the per-worker sums, so its own last bits depend on the core count. The gate is 1e-9.

On the same files, same machine as the tables above, panopticapi's pq_compute takes 16.1 s across its 8 worker processes and hotcoco 6.0 s (medians of 3 fresh processes, one session); PNG decoding is most of hotcoco's time. The synthetic version of this comparison — 50 random cases and 5 hand-built ones, counts exact, plus the PNG-free mask path against the PNG path — runs in CI on every commit.

Semantic¶

References: mmsegmentation IoUMetric at v1.2.2 (c685fe6), torchmetrics 1.9.0 multiclass_jaccard_index, detectron2 SemSegEvaluator at 1e3e13b, and the Boundary IoU authors' boundary-iou-api mask_to_boundary at 37d2558, each run from its pinned source by scripts/gen_semantic_fixtures.py and frozen to tests/fixtures/semantic_expected.json. Inputs: 67 synthetic cases — 60 random ones of 2 to 9 classes, 1 to 4 label maps each, ignored pixels in half of them, reduce_zero_label in a fifth, classes present on one side only; four hand-built corners (a perfect prediction, an all-ignored split, a class never predicted, a class never in the ground truth); and three maps of 60 × 80 to 90 × 120 built from rectangles, so the boundary band is two or three pixels wide and most pixels are interior.

Against mmseg, the reference hotcoco follows: every per-class IoU, Acc, and Dice, mIoU, mAcc, aAcc, and mDice within 1e-9 on all 67 cases (the differences are 0 or one float64 ulp). Against torchmetrics, per-class IoU and mIoU within 1e-6 — it reduces in float32, and sits about 1e-8 from mmseg. Against detectron2, fwIoU, mAcc, and aAcc within 1e-9, and mIoU identical on the cases with no predicted-but-absent class and different on all 30 that have one, which is the documented convention choice being observed. Against the authors' kernel, every per-class boundary IoU and mBIoU within 1e-9; against detectron2's label-map erosion, mBIoU differs on 58 of the 67 cases, which is the second documented choice being observed. The confusion-matrix metrics are pure functions of the matrix and the band a pure function of the maps, so this synthetic comparison covers them; there is no real-dataset run for the family. Both suites run on every commit.

Verify it yourself¶

You do not have to take these numbers on faith, and you should not have to clone the repo to check them. Install both libraries and run your own ground truth and detections through each:

import contextlib, io
import numpy as np
from pycocotools.coco import COCO as PyCOCO
from pycocotools.cocoeval import COCOeval as PyCOCOeval
import hotcoco

GT, DT, IOU_TYPE = "instances_val2017.json", "my_detections.json", "bbox"

def run(coco_cls, eval_cls):
    with contextlib.redirect_stdout(io.StringIO()):   # both print a lot
        gt = coco_cls(GT)
        dt = gt.loadRes(DT)
        e = eval_cls(gt, dt, IOU_TYPE)
        e.evaluate(); e.accumulate(); e.summarize()
    return np.asarray(e.stats)

ref = run(PyCOCO, PyCOCOeval)
got = run(hotcoco.COCO, hotcoco.COCOeval)

for i, (a, b) in enumerate(zip(ref, got)):
    print(f"[{i:2}] pycocotools={a:.8f}  hotcoco={b:.8f}  diff={abs(a - b):.2e}")
print("max diff:", np.abs(ref - got).max())

On val2017 the project's own parity gate requires exact equality, so any difference at all is a bug. On your own data, anything above ~1e-12 is worth opening an issue.

The same shape works for hotcoco.mask against pycocotools.mask, operation by operation.

Beyond val2017, a hypothesis-based fuzzer checks ~10,000 generated datasets — including degenerate zero-area boxes and other edge cases hand-written tests miss — against pycocotools at a 1e-10 tolerance.

Methodology¶

• Wall clock time includes file I/O, evaluation, and accumulation. Excludes Python import time.
• Core count affects the ratio. hotcoco evaluates in parallel; pycocotools is single-threaded. Speedups therefore scale with the cores available, and the numbers here come from an 8-core machine — a 4-core laptop sees less, a 32-core server more. Run the suite on your own hardware for a figure that describes it.
• Detections are synthetic — generated from GT annotations with a fixed seed (seed=42), so AP scores are meaningless but detection count and format are representative of real model output. Fixed seed means results are identical across runs.
• Only detections are scaled for the 10x benchmark — ground truth annotations are unchanged.

Reproducing the benchmarks¶

These run from a repo checkout — see CONTRIBUTING for the build. One command fetches the annotations and generates the synthetic detection files:

just download-coco   # ~240 MB — val2017 annotations + parity result files

That produces:

data/
├── annotations/
│   ├── instances_val2017.json
│   └── person_keypoints_val2017.json
├── bbox_val2017_results.json
├── segm_val2017_results.json
└── kpt_val2017_results.json

With that in place:

just bench                                  # speed benchmark (1x)
uv run python scripts/bench.py --phases     # load/eval phase breakdown
uv run python scripts/bench.py --scale 10   # 10x stress test
just parity                                 # metric parity vs pycocotools
just parity-tide                            # TIDE vs tidecv
just download-panoptic                      # 35 MB — panoptic val2017 + perturbed predictions
just parity-panoptic                        # PQ/SQ/RQ vs panopticapi
uv run pytest tests/test_mask_parity.py     # hotcoco.mask vs pycocotools.mask, bit for bit
uv run pytest tests/test_parity_oid.py      # Open Images vs the TF Object Detection API
uv run pytest tests/test_parity_semantic.py # mIoU, mDice vs mmseg, torchmetrics, detectron2; boundary IoU vs its authors' kernel (frozen)
just fuzz                                   # hypothesis fuzzer, ~10,000 generated datasets

The Objects365 benchmark needs a separate download:

uv pip install polars
just download-o365                              # ~220 MB — O365 val annotations
uv run python scripts/bench_objects365.py