Benchmarks¶

Feature comparison¶

Speed benchmarks¶

Hardware: Apple M1 MacBook Air — 8 cores (4 performance + 4 efficiency), 8 GB RAM Dataset: COCO val2017 — 5,000 images Detections: 36,781 synthetic — see Methodology Timing: Wall clock time — per-cell median of 3 runs, at both 1× and 10×. The two scales were captured separately and the machine was busier during the 10× capture, so absolute times are comparable only within a table; the speedup ratios are not affected. Versions: pycocotools 2.0.11, faster-coco-eval 1.7.2, hotcoco 1.0.0

Results (1x detections)¶

Speedups in parentheses are vs pycocotools.

Results (10x detections)¶

Scaling detections by 10x (~368,000) to test behavior under higher load:

Absolute times stay under 2s at 368,000 detections. hotcoco's relative advantage is narrower here than in the 1× table — 8–33× rather than 19–36× — because per-call overhead, where it gains most, is a smaller share of the total once there is this much work to do.

Where the time goes¶

The preceding end-to-end numbers blend two very different phases: load (JSON parsing and index building — COCO() + loadRes) and eval (evaluate + accumulate + summarize). Splitting them shows where each library spends its time (single run, same synthetic detections as the 1× table):

The evaluation engine itself is 35–74× faster than pycocotools; the end-to-end headline is lower because JSON parsing is a much larger share of hotcoco's total than of anyone else's.

The bbox-only GT row strips the polygon segmentation the official instances files carry on every annotation — about two-thirds of the file bytes, which bbox evaluation never reads. It represents datasets that never had masks (custom bbox datasets, YOLO conversions, Objects365), where the load column is the one you'll actually see.

Objects365 scale benchmark¶

Hardware: Windows 11, AMD Ryzen 5 5600X — 6 cores / 12 threads, 16 GB RAM + swap Dataset: Objects365 val — 80,000 images, 1.2M annotations, 365 categories Detections: ~1.2M synthetic bbox (capped at 100/image, seed=42) Timing: Wall clock time, single run Versions: pycocotools 2.0.11, faster-coco-eval 1.7.2, hotcoco 0.3.0 — this run predates the 1.0 loading work, so the hotcoco figure is conservative

Peak RAM is the peak working set (physical memory). Committed includes swap — both pycocotools and faster-coco-eval exceeded physical RAM and relied heavily on the pagefile, which significantly inflated their wall clock times. hotcoco completed within physical memory with minimal swap.

Metric parity¶

Reference: pycocotools 2.0.11, numpy 2.4.3. Ground truth: COCO val2017 — 5,000 images. Detections: the published instances_val2017_fake*_results.json files from ppwwyyxx/cocoapi, the same inputs pycocotools uses in its own tests.

Every metric agrees to within 3.7e-14 — floating-point noise, the last few bits a f64 can represent. The diffs in the following tables are raw measured differences, not rounded.

Bounding box¶

The threshold grids are constructed to match numpy.linspace bit-for-bit, which removed the last systematic source of divergence here.

Segmentation¶

Exact. The residual segmentation once carried (AP ~1e-5) came from polygon rasterization, where the reference's C compiler contracts s*t+ys into a single fused multiply-add on arm64 but not on x86-64, whose PyPI wheels target a baseline without the instruction. hotcoco mirrors that choice per architecture, so the comparison is exact on either kind of machine. Masks can differ by a boundary pixel between the two architectures — as pycocotools' own do.

Keypoints¶

Keypoint metrics are exact. Keypoint evaluation reports 10 metrics — see Keypoint evaluation.

TIDE¶

Reference: tidecv, on COCO val2017 at pos_thr=0.5. Expect the five false-positive types to agree closely and Miss to read higher here:

The ranking — which error type is costing you the most AP — is the same, and that is what the metric is for. The difference comes from crowd handling: hotcoco builds TIDE on the same COCO-convention matching as its AP (so tide_errors() and ev.stats always agree about which detections exist), while tidecv removes crowd regions from matching entirely — its false-positive counts run higher and its Miss runs lower as a result.

Open Images¶

Reference: the TensorFlow Object Detection API — the implementation the official protocol page points to.

Open Images evaluation is compared over 70 cases covering group-of absorption, IoA containment at and around the 0.5 boundary, undetected group-of boxes, overlapping group-of boxes, and randomized multi-class scenes. Both mAP and per-class AP are compared, and every case agrees to within one ulp (worst difference 1.11e-16). It runs in CI on every commit.

Two things that comparison does not cover, and why provenance still reports "extension" for Open Images runs:

• Non-exhaustive image-level labels. The challenge ignores detections of a class not verified on an image, and counts detections of a negatively-labeled class as false positives. hotcoco does not implement this — it needs per-image label data that COCO-format JSON cannot carry. A real challenge submission would score differently.
• Hierarchy expansion is applied to annotations before evaluation rather than inside it, so it sits outside the compared surface.

Verify it yourself¶

You do not have to take these numbers on faith, and you should not have to clone the repo to check them. Install both libraries and run your own ground truth and detections through each:

import contextlib, io
import numpy as np
from pycocotools.coco import COCO as PyCOCO
from pycocotools.cocoeval import COCOeval as PyCOCOeval
import hotcoco

GT, DT, IOU_TYPE = "instances_val2017.json", "my_detections.json", "bbox"

def run(coco_cls, eval_cls):
    with contextlib.redirect_stdout(io.StringIO()):   # both print a lot
        gt = coco_cls(GT)
        dt = gt.loadRes(DT)
        e = eval_cls(gt, dt, IOU_TYPE)
        e.evaluate(); e.accumulate(); e.summarize()
    return np.asarray(e.stats)

ref = run(PyCOCO, PyCOCOeval)
got = run(hotcoco.COCO, hotcoco.COCOeval)

for i, (a, b) in enumerate(zip(ref, got)):
    print(f"[{i:2}] pycocotools={a:.8f}  hotcoco={b:.8f}  diff={abs(a - b):.2e}")
print("max diff:", np.abs(ref - got).max())

Anything above ~1e-12 on your data is worth opening an issue — that is the threshold the project's own parity gate uses.

The same shape works for hotcoco.mask against pycocotools.mask, operation by operation.

Beyond val2017, a hypothesis-based fuzzer checks ~10,000 generated datasets — including degenerate zero-area boxes and other edge cases hand-written tests miss — against pycocotools at a 1e-10 tolerance.

Methodology¶

• Wall clock time includes file I/O, evaluation, and accumulation. Excludes Python import time.
• Core count affects the ratio. hotcoco evaluates in parallel; pycocotools is single-threaded. Speedups therefore scale with the cores available, and the numbers here come from an 8-core machine — a 4-core laptop sees less, a 32-core server more. Run the suite on your own hardware for a figure that describes it.
• Detections are synthetic — generated from GT annotations with a fixed seed (seed=42), so AP scores are meaningless but detection count and format are representative of real model output. Fixed seed means results are identical across runs.
• Only detections are scaled for the 10x benchmark — ground truth annotations are unchanged.

Reproducing the benchmarks¶

These run from a repo checkout — see CONTRIBUTING for the build. One command fetches the annotations and generates the synthetic detection files:

just download-coco   # ~240 MB — val2017 annotations + parity result files

That produces:

data/
├── annotations/
│   ├── instances_val2017.json
│   └── person_keypoints_val2017.json
├── bbox_val2017_results.json
├── segm_val2017_results.json
└── kpt_val2017_results.json

With that in place:

just bench                                  # speed benchmark (1x)
uv run python scripts/bench.py --phases     # load/eval phase breakdown
uv run python scripts/bench.py --scale 10   # 10x stress test
just parity                                 # metric parity vs pycocotools
just parity-mask                            # hotcoco.mask vs pycocotools.mask, operation by operation
just parity-tide                            # TIDE vs tidecv
just parity-oid                             # Open Images vs the TF Object Detection API
just fuzz                                   # hypothesis fuzzer, ~10,000 generated datasets

The Objects365 benchmark needs a separate download:

uv pip install polars
just download-o365                              # ~220 MB — O365 val annotations
uv run python scripts/bench_objects365.py