COCOeval¶

Run COCO evaluation to compute AP/AR metrics.

from hotcoco import COCO, COCOeval

coco_gt = COCO("instances_val2017.json")
coco_dt = coco_gt.load_res("detections.json")

ev = COCOeval(coco_gt, coco_dt, "bbox")
ev.evaluate()
ev.accumulate()
ev.summarize()

use hotcoco::{COCO, COCOeval};
use hotcoco::params::IouType;
use std::path::Path;

let coco_gt = COCO::new(Path::new("instances_val2017.json"))?;
let coco_dt = coco_gt.load_res(Path::new("detections.json"))?;

let mut ev = COCOeval::new(coco_gt, coco_dt, IouType::Bbox);
ev.evaluate()?;
ev.accumulate();
ev.summarize();

Constructor¶

COCOeval(
    coco_gt: COCO,
    coco_dt: COCO,
    iou_type: str,
    *,
    lvis_style: bool = False,
    oid_style: bool = False,
    hierarchy: Hierarchy | None = None,
)

// Standard COCO
COCOeval::new(coco_gt: impl Into<Arc<COCO>>, coco_dt: impl Into<Arc<COCO>>, iou_type: IouType) -> Self

// LVIS federated
COCOeval::new_lvis(coco_gt: impl Into<Arc<COCO>>, coco_dt: impl Into<Arc<COCO>>, iou_type: IouType) -> Self

// Open Images
COCOeval::new_oid(coco_gt: impl Into<Arc<COCO>>, coco_dt: impl Into<Arc<COCO>>, hierarchy: Option<Hierarchy>) -> Self

The same classes are also reachable under hotcoco.detection (from hotcoco.detection import COCOeval), an explicit namespace for the detection metric family. Both spellings return the same objects.

Properties¶

params¶

params: Params

pub params: Params

See Params for every configurable field and an example of changing them.

stats¶

stats: np.ndarray | list[float]

ev.summarize()
print(f"AP: {ev.stats[0]:.3f}")
print(f"AP50: {ev.stats[1]:.3f}")

fn stats(&self) -> Option<&[f64]>

ev.summarize();
if let Some(stats) = ev.stats() {
    println!("AP: {:.3}", stats[0]);
    println!("AP50: {:.3}", stats[1]);
}

eval_imgs¶

Per-image, per-category, per-area-range evaluation results, populated after evaluate(). Entries are None where an image has neither ground truth nor detections for that category, as in pycocotools.

eval_imgs: list[dict | None]

fn eval_imgs(&self) -> &[Option<EvalImg>]

Each entry is a dict whose keys match pycocotools' evalImgs — image_id and category_id in snake_case, everything else camelCase (there are no snake_case variants of the camelCase keys):

For a worked example, see Per-image evaluation results.

ious¶

pycocotools' ious attribute: the IoU (or OKS) matrix of every image and category evaluate() ran over, keyed by (image_id, category_id). Empty before evaluate(), and fixed at that call as pycocotools' is: params edited afterwards for accumulate() do not move it.

ious: dict[tuple[int, int], np.ndarray | list]

fn iou_matrices(&self) -> Vec<((u64, u64), Option<IouMatrix>)>

eval¶

Accumulated precision/recall arrays, populated after accumulate().

eval: dict | None

fn accumulated(&self) -> Option<&AccumulatedEval>

precision and scores have shape [T x R x K x A x M]; recall has shape [T x K x A x M]:

A value of -1 means no data — for example, no ground truth annotations for that category and area combination.

For worked examples, see Precision and recall arrays and Extracting per-category AP.

Methods¶

evaluate¶

evaluate() -> None

Run per-image evaluation. Matches detections to ground truth annotations using greedy matching sorted by confidence. Must be called before accumulate().

Populates eval_imgs.

accumulate¶

accumulate() -> None

Accumulate per-image results into precision/recall curves using interpolated precision at 101 recall thresholds.

Populates eval.

summarize¶

summarize() -> None

Compute and print the standard COCO metrics. Populates stats.

The table goes through Python's sys.stdout, so contextlib.redirect_stdout captures or silences it, and it shows in a notebook cell. To skip printing altogether, call summary_lines instead. A library that embeds hotcoco and wants neither the table nor the non-default-parameter warnings should use it, and read what the warnings would have said from reference_deviations.

Prints 12 lines for bbox/segm (10 for keypoints):

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.382
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.584
 ...

run¶

run() -> None

Run the full pipeline in one call: evaluate() → accumulate() → summarize(). Primarily used with LVIS pipelines (Detectron2, MMDetection) that expect a single run() call.

It behaves as those three calls do: the table goes through sys.stdout, and the same ReferenceDeviationWarnings as summarize are emitted.

metric_keys¶

metric_keys() -> list[str]

Return metric names in canonical display order for the current evaluation mode. This is the authoritative ordering — the same list that drives summarize() and get_results().

ev = COCOeval(gt, dt, "bbox")
ev.metric_keys()
# ['AP', 'AP50', 'AP75', 'APs', 'APm', 'APl', 'AR1', 'AR10', 'AR100', 'ARs', 'ARm', 'ARl']

metric_defs¶

metric_defs() -> list[dict]

Return the metric catalog as structured data, one dict per metric in metric_keys() order: name (str), ap (bool — AP vs AR), iou_thr (float or None for the full 0.50:0.05:0.95 sweep), area (str), max_det (int), and freq_group ("rare" / "common" / "frequent" or None; LVIS only).

ev.metric_defs()[1]
# {'name': 'AP50', 'ap': True, 'iou_thr': 0.5, 'area': 'all', 'max_det': 100, 'freq_group': None}

This exists so renderers read a metric's axes instead of parsing them back out of its name — "AR10" is ambiguous between a detection cap of 10 and an IoU of 0.10, and only the catalog knows which. hotcoco's own PDF report and dashboard consume it.

get_results¶

get_results(prefix: str | None = None, per_class: bool = False) -> dict[str, float]

Return the summary metrics as a dict.

Standard bbox/segm keys: AP, AP50, AP75, APs, APm, APl, AR1, AR10, AR100, ARs, ARm, ARl.

Keypoint keys: AP, AP50, AP75, APm, APl, AR, AR50, AR75, ARm, ARl.

LVIS keys: see The 13 LVIS metrics.

ev.run()
ev.get_results()   # {"AP": 0.578, "AP50": 0.861, ...}

For prefixed and per-class keys and how they feed an experiment tracker, see Logging metrics.

print_results¶

print_results() -> None

Print a formatted results table to stdout. For LVIS, matches the lvis-api print_results() style.

Like summarize, it prints through sys.stdout, so contextlib.redirect_stdout, pytest's capsys, and notebook cells capture it. Called before summarize(), it prints nothing and emits a CallOrderWarning. In Rust, print_results_lines() returns the same lines without printing.

summary_lines¶

summary_lines() -> list[str]

The same lines summarize() prints, returned instead of written to stdout — one string per metric, already formatted. Use it to route the summary into a logger, a report, or a test assertion.

It populates stats as summarize() does and prints nothing, but it also skips the ReferenceDeviationWarnings that summarize() emits. That makes it the quiet path for code that embeds hotcoco, and it means nothing tells you the run is not comparable to the reference: call reference_deviations, or read provenance from report, when that matters.

virtual_cat_names¶

virtual_cat_names: list[str]   # property

Category names added by Open Images hierarchy expansion — ancestor categories that exist in the hierarchy but not in the dataset's own taxonomy. Empty when not in OID mode, when no hierarchy expansion occurred, or before evaluate(). Use it to distinguish expanded ancestor categories from the model's native classes:

ev = COCOeval(gt, dt, "bbox", oid_style=True, hierarchy=h)
ev.evaluate()
ev.virtual_cat_names   # for example, ['Carnivore', 'Mammal'] — no parentheses; it's a property

slice_by¶

slice_by(slices: dict[str, list[int]] | Callable[[dict], str]) -> dict[str, Any]

Re-accumulate metrics for named subsets of images without recomputing IoU, and return one metrics dict per slice. Pass either an explicit {name: [image_ids]} mapping or a function that takes an image dict and returns a slice name.

ev.run()
by_light = ev.slice_by({"day": day_ids, "night": night_ids})
by_light["night"]["AP"]

Requires evaluate() to have run. Matching is done once and reused for every slice, so slicing a dozen ways costs barely more than slicing one way. See sliced evaluation.

report¶

report() -> dict

Return a full evaluation report.

This is the shape every hotcoco metric family reports in, so code that renders a detection report renders a panoptic or tracking one unchanged.

Returns a dict with:

ev.run()
report = ev.report()

report["metrics"]["AP"]              # 0.377
report["per_class"]["person"]["AP"]  # 0.521

Provenance values¶

What each value covers and why is in Check provenance before you publish a number. For plotting curves, see Plotting precision-recall curves.

provenance¶

provenance() -> str

Return the same value as report()["provenance"], read from the configuration alone. See Checking before you evaluate.

is_benchmark_standard¶

is_benchmark_standard() -> bool

Return True exactly when provenance() is "parity_verified". Any other value — including a variant added in a future release — reads as False, so a renderer that checks this predicate caveats unknown provenance rather than passing it through. See Checking before you evaluate.

reference_deviations¶

reference_deviations() -> list[str]

Return one human-readable sentence per way this run departs from the reference configuration, empty exactly when provenance() is "parity_verified". See Checking before you evaluate.

results¶

results(per_class: bool = False) -> dict

Return evaluation results as a serializable dict.

Returns a dict with:

ev.run()
ev.results(per_class=True)   # {"hotcoco_version": ..., "metrics": {...}, "per_class": {...}}

Serialized, the dict looks like this:

{
  "hotcoco_version": "1.0.0",
  "provenance": "parity_verified",
  "params": {
    "iou_type": "bbox",
    "eval_mode": "coco",
    "iou_thresholds": [0.5, 0.55, ...],
    "recall_thresholds": [0.0, 0.01, ...],
    "area_ranges": {"all": [0, 10000000000.0], "small": [0, 1024.0], ...},
    "max_dets": [1, 10, 100],
    "use_cats": true,
    "kpt_oks_sigmas": [0.026, 0.025, ...],
    "reference_deviations": []
  },
  "metrics": {
    "AP": 0.378, "AP50": 0.584, "AP75": 0.412, ...
  },
  "per_class": {
    "person": 0.58, "car": 0.41, ...
  }
}

For a worked example, see Saving results to JSON.

save_results¶

save_results(path: str, per_class: bool = False) -> None

Save evaluation results to a JSON file. Raises IOError if the file cannot be written.

ev.run()
ev.save_results("results.json", per_class=True)

The file holds the dict returned by results().

confusion_matrix¶

confusion_matrix(
    iou_thr: float = 0.5,
    max_det: int | None = None,
    min_score: float | None = None,
) -> dict

Compute a per-category confusion matrix. Unlike evaluate(), this method compares all detections in an image against all ground truth boxes regardless of category, enabling cross-category confusion analysis.

This method is standalone — no evaluate() call is needed first.

Parameters:

Returns a dict with:

cm = ev.confusion_matrix(iou_thr=0.5, max_det=100)
cm["matrix"]   # np.ndarray uint64, shape (K+1, K+1)

For the cell layout and a full walkthrough, see Confusion matrix in the diagnostics guide.

tide_errors¶

tide_errors(
    pos_thr: float = 0.5,
    bg_thr: float = 0.1,
) -> dict

Decompose detection errors into six TIDE error types (Bolya et al., ECCV 2020) and compute ΔAP — the AP gain from eliminating each error type.

Requires evaluate() to have been called first.

Parameters:

Returns a dict with:

result = ev.tide_errors(pos_thr=0.5, bg_thr=0.1)
result["delta_ap"]   # {"Cls": ..., "Loc": ..., "Both": ..., "Dupe": ..., "Bkg": ..., "Miss": ..., "FP": ..., "FN": ...}

For the error taxonomy and a full walkthrough, see TIDE error analysis in the diagnostics guide.

calibration¶

calibration(
    n_bins: int = 10,
    iou_threshold: float = 0.5,
) -> dict

Compute confidence calibration metrics — how well confidence scores predict actual detection accuracy.

Requires evaluate() to have been called first. Bins all non-ignored detections by confidence score and compares the mean confidence in each bin to the fraction of true positives.

Parameters:

Returns a dict with:

cal = ev.calibration(n_bins=10, iou_threshold=0.5)
cal["ece"], cal["mce"]   # (0.0412, 0.1873)

For interpretation and a full walkthrough, see Confidence calibration in the diagnostics guide.

f_scores¶

f_scores(beta: float = 1.0) -> dict[str, float]

Compute F-beta scores after accumulate() (or run()).

For each (IoU threshold, category), finds the confidence operating point that maximizes F-beta, then averages across categories — analogous to how mAP averages precision. Returns three metrics mirroring AP/AP50/AP75.

Returns a dict with three keys:

Key names reflect beta, formatted with no trailing zeros: "F0.5", "F0.5_50", "F0.5_75" for beta=0.5; "F2", "F2_50", "F2_75" for beta=2.0.

Returns an empty dict if accumulate() has not been called.

ev.run()
ev.f_scores()   # {"F1": 0.523, "F1_50": 0.712, "F1_75": 0.581}

For a full walkthrough, see F-scores in the diagnostics guide.

image_diagnostics¶

image_diagnostics(
    iou_thr: float = 0.5,
    score_thr: float = 0.5,
) -> dict

Per-image diagnostics: annotation TP/FP/FN index, per-image F1 and AP scores, error profiles, and label error candidates.

Requires evaluate() to have been called first.

Parameters:

Returns a dict with:

Each image summary dict contains:

Each label error dict contains:

diag = ev.image_diagnostics(iou_thr=0.5, score_thr=0.5)
diag["img_summary"][42]   # {"tp": 3, "fp": 1, "fn": 0, "f1": 0.857, "ap": 0.75, "error_profile": "fp_heavy"}

For a full walkthrough, see Per-image diagnostics and label error detection in the diagnostics guide.

LVIS evaluation¶

These three names are drop-in replacements for the lvis-api package, so Detectron2 and MMDetection pipelines run unchanged. For the federated protocol, see LVIS evaluation; for redirecting an existing from lvis import ... pipeline, see LVIS-based pipelines.

LVISeval¶

hotcoco.LVISeval(gt: COCO, dt: COCO, iou_type: str = "segm") -> COCOeval

Returns a COCOeval configured for federated evaluation — equivalent to COCOeval(gt, dt, iou_type, lvis_style=True). Supports run(), print_results(), and get_results(), which is what those pipelines call. The default iou_type is "segm", matching lvis-api.

LVISEval (capital E) is an alias: that is the spelling lvis-api exports and the one from lvis import LVISEval expects.

ev = LVISeval(lvis_gt, lvis_dt, "segm")
ev.run()
ev.get_results()   # {"AP": ..., "APr": ..., "APc": ..., "APf": ..., "AR@300": ..., ...}

The metric set is listed in The 13 LVIS metrics.

LVIS¶

An alias for COCO, provided because lvis-api names its dataset class LVIS. Loading is identical.

LVISResults¶

hotcoco.LVISResults(lvis_gt: COCO, results, max_dets: int | None = 300) -> COCO

Returns a COCO detections object: lvis_gt.load_res(results) capped the way lvis-api's LVISResults caps it — each image's max_dets highest-scoring detections across every category, ties in results-file order. max_dets=-1 or None keeps every detection, and an integral float such as 300.0 is that integer. The cap is COCO.cap_detections_per_image(max_dets), which you can call directly (None keeps every detection).

LVISeval evaluates an LVISResults result as is, as lvis-api's LVISEval takes an LVISResults object unchanged, so max_dets=1000 or -1 holds through evaluation. Detections loaded with plain load_res() get lvis-api's default instead: LVISeval caps them at 300 the same way when constructed, as lvis-api caps a results path or list it loads itself. The 300 does not follow params.max_dets, which lvis-api's LVISEval does not pass to LVISResults either; StreamingEval with lvis_style=True applies the same 300.

StreamingEval¶

Evaluate detector batches as they come and get a COCOeval back. Each update() call matches its images right away; finalize() assembles every image seen into an evaluator ready for accumulate(), summarize(), and report(). The results are identical to a batch COCOeval over the same annotations, however the images were batched. For the training loop this is built for, see Streaming evaluation.

StreamingEval(
    categories: list[dict],
    iou_type: str = "bbox",
    lvis_style: bool = False,
    params: Params | None = None,
)

StreamingEval::new(params: Params, eval_mode: EvalMode, categories: Vec<Category>) -> Result<Self>

update¶

se.update(
    images: list[dict],
    gt_anns: list[dict],
    dt_anns: list[dict] | ndarray,
    *,
    segmentation: list[dict | list[list[float]]] | None = None,
) -> None

Match a batch of images' ground truth against their detections now. images are COCO image dicts with at least id — plus neg_category_ids and not_exhaustive_category_ids in LVIS mode, where they apply. A batch of one is fine; passing the detector's whole batch amortizes the per-call setup. gt_anns are annotation dicts in the shape COCO(dict) accepts; id is assigned and a missing area is derived (the mask's pixel count, or the box's w × h for an annotation without a mask), so targets as a data loader yields them — image_id, category_id, bbox, iscrowd — are enough. For segm, every image holding a polygon or box annotation in an evaluated category needs height and width, or the call raises ValueError naming the images. dt_anns are raw predictions in the shape load_res() accepts — image_id, category_id, bbox (or segmentation/keypoints), and score — and are loaded the same way: ids assigned, area derived, iscrowd cleared. Within an image, detections with tied scores rank in the order given, as they do in a results file, so keep a batch's predictions in the order the detector emitted them. An image seen again in a later call replaces its earlier result.

dt_anns can also be the numpy array load_res() accepts, shape (N, 7) with columns [image_id, x, y, w, h, score, category_id], which skips building a dict per detection. An (N, 6) array has no category column and puts every row in category 1, as load_res() does. An id that is NaN, negative, or not a whole number raises ValueError. For segm, segmentation is a list of N RLE or polygon entries, one per row; it goes only with an array, since a dict carries its own. Every row has a box, so each detection's area is the box's w × h, as load_res() gives a result that has a bbox, and that area decides the small, medium, and large buckets. For mask area, as torchmetrics computes it, pass dicts that carry segmentation and no bbox.

Loading and matching run without the GIL; converting the dicts does not. The guide has what update() costs.

A ground truth or detection whose category_id is not in categories raises KeyError naming every such id, as update_anns does for an unknown annotation id. Without it the label would drop out of every metric, because the category list is fixed when the StreamingEval is built. The batch is rejected whole: the evaluator is as it was before the call, so a caller that catches the error and carries on gets honest numbers. A batch COCOeval does not check, and drops such an annotation silently. Only the categories listed count: params.cat_ids narrows what is evaluated without making the rest unknown, and with use_cats false no category is checked, because every annotation pools into one.

A NaN score raises the same ValueError as load_res(). Calling this after finalize() raises RuntimeError.

merge¶

StreamingEval.merge(evaluators: Iterable[StreamingEval]) -> StreamingEval

A new StreamingEval holding every image the given evaluators have seen, as if one evaluator had received all their update() calls. This is how a run split across processes comes back together: each rank streams its shard of the images, one rank merges them all, and finalize() gives exactly what one stream over every image gives. No matching is redone. evaluators is any iterable, read one at a time, so a generator over from_bytes keeps one shard in memory beside the merged result rather than every rank's at once. An image seen by more than one evaluator keeps the result of the last, the rule update() applies to an image seen again, so an image that DistributedSampler repeats to even out the shards counts once. The evaluators passed in are left unchanged and usable.

Every evaluator must be built the same way: the same categories, iou_type, lvis_style, and params, img_ids included — leave img_ids empty on every rank rather than setting it to each rank's shard. Categories are compared by id, name, and LVIS frequency, in any order. Otherwise merge raises ValueError naming the first evaluator that differs and the field it differs in. An empty iterable raises ValueError, a single StreamingEval instead of an iterable of them raises TypeError, and a finalized evaluator raises RuntimeError.

to_bytes and from_bytes¶

se.to_bytes() -> bytes
StreamingEval.from_bytes(data: bytes) -> StreamingEval

Save the state and restore it, in this process or another: for example, all_gather each rank's to_bytes(), then from_bytes() and merge() them on one rank. Only what finalize() reads is saved, so an image replaced by a later update() costs nothing, and the size follows the images seen. The restored evaluator finalizes to the same numbers, keeps accepting update(), and merges like any other.

pickle, copy.copy, and copy.deepcopy use the same state, so a metric object that holds a StreamingEval can be checkpointed or copied. The bytes carry a format version, and from_bytes raises ValueError for bytes that are truncated, damaged, or from a version it does not read. Treat them as hotcoco's own format, not an interchange format: a release that changes it says so in the changelog.

finalize¶

se.finalize() -> COCOeval

Assemble every image seen so far into a COCOeval. Consumes the StreamingEval: update() or finalize() afterwards raises RuntimeError.

The returned evaluator holds the per-cell match results and the category list, not the annotations, which is what keeps finalize() cheap. It supports accumulate(), summarize(), report(), results(), get_results(), slice_by(), and compare(). Its eval_imgs is empty, and the analyses that rebuild full per-image records from the datasets — confusion_matrix(), tide_errors(), calibration(), f_scores(), and image_diagnostics() — see no cells. Build a batch COCOeval for those.

Module-level functions¶

__version__¶

hotcoco.__version__ -> str

The version of the compiled extension that is loaded, for example "1.1.0". For an installed wheel it equals importlib.metadata.version("hotcoco"). Use it to gate a workaround on a release or to put in a bug report.

compare¶

hotcoco.compare(
    eval_a: COCOeval,
    eval_b: COCOeval,
    n_bootstrap: int = 0,
    seed: int = 42,
    confidence: float = 0.95,
) -> dict

Pairwise model comparison. Both evaluators must have had evaluate() called and use the same eval_mode, iou_type, evaluation grid, and category axis — mismatched iou_thrs, rec_thrs, max_dets, area ranges, use_cats, or set of cat_ids raise ValueError rather than summarizing one run under the other's catalog. The same categories in another order compare fine: the per-category table pairs by id. Accumulation and summarization are performed internally on the shared image set.

Returns a dict with:

result = hotcoco.compare(ev_a, ev_b, n_bootstrap=1000)
result["deltas"]["AP"], result["ci"]["AP"]   # (0.033, {"lower": 0.01, "upper": 0.05, ...})

For bootstrap interpretation and a full walkthrough, see Model comparison in the diagnostics guide.