COCOeval¶

Run COCO evaluation to compute AP/AR metrics.

from hotcoco import COCO, COCOeval

coco_gt = COCO("instances_val2017.json")
coco_dt = coco_gt.load_res("detections.json")

ev = COCOeval(coco_gt, coco_dt, "bbox")
ev.evaluate()
ev.accumulate()
ev.summarize()

use hotcoco::{COCO, COCOeval};
use hotcoco::params::IouType;
use std::path::Path;

let coco_gt = COCO::new(Path::new("instances_val2017.json"))?;
let coco_dt = coco_gt.load_res(Path::new("detections.json"))?;

let mut ev = COCOeval::new(coco_gt, coco_dt, IouType::Bbox);
ev.evaluate();
ev.accumulate();
ev.summarize();

Constructor¶

COCOeval(
    coco_gt: COCO,
    coco_dt: COCO,
    iou_type: str,
    *,
    lvis_style: bool = False,
    oid_style: bool = False,
    hierarchy: Hierarchy | None = None,
)

// Standard COCO
COCOeval::new(coco_gt: COCO, coco_dt: COCO, iou_type: IouType) -> Self

// LVIS federated
COCOeval::new_lvis(coco_gt: COCO, coco_dt: COCO, iou_type: IouType) -> Self

// Open Images
COCOeval::new_oid(coco_gt: COCO, coco_dt: COCO, hierarchy: Option<Hierarchy>) -> Self

The same classes are also reachable under hotcoco.detection (from hotcoco.detection import COCOeval), an explicit namespace for the detection metric family. Both spellings return the same objects.

Properties¶

params¶

params: Params

pub params: Params

See Params for every configurable field and an example of changing them.

stats¶

stats: np.ndarray | list[float]

ev.summarize()
print(f"AP: {ev.stats[0]:.3f}")
print(f"AP50: {ev.stats[1]:.3f}")

fn stats(&self) -> Option<&[f64]>

ev.summarize();
if let Some(stats) = ev.stats() {
    println!("AP: {:.3}", stats[0]);
    println!("AP50: {:.3}", stats[1]);
}

eval_imgs¶

Per-image, per-category, per-area-range evaluation results, populated after evaluate(). Entries are None where an image has neither ground truth nor detections for that category, as in pycocotools.

eval_imgs: list[dict | None]

fn eval_imgs(&self) -> &[Option<EvalImg>]

Each entry is a dict whose keys match pycocotools' evalImgs — image_id and category_id in snake_case, everything else camelCase (there are no snake_case variants of the camelCase keys):

For a worked example, see Per-image evaluation results.

eval¶

Accumulated precision/recall arrays, populated after accumulate().

eval: dict | None

fn accumulated(&self) -> Option<&AccumulatedEval>

precision and scores have shape [T x R x K x A x M]; recall has shape [T x K x A x M]:

A value of -1 means no data — for example, no ground truth annotations for that category and area combination.

For worked examples, see Precision and recall arrays and Extracting per-category AP.

Methods¶

evaluate¶

evaluate() -> None

Run per-image evaluation. Matches detections to ground truth annotations using greedy matching sorted by confidence. Must be called before accumulate().

Populates eval_imgs.

accumulate¶

accumulate() -> None

Accumulate per-image results into precision/recall curves using interpolated precision at 101 recall thresholds.

Populates eval.

summarize¶

summarize() -> None

Compute and print the standard COCO metrics. Populates stats.

The table goes through Python's sys.stdout, so contextlib.redirect_stdout captures or silences it, and it shows in a notebook cell. To skip printing altogether, call summary_lines instead.

Prints 12 lines for bbox/segm (10 for keypoints):

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.382
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.584
 ...

run¶

run() -> None

Run the full pipeline in one call: evaluate() → accumulate() → summarize(). Primarily used with LVIS pipelines (Detectron2, MMDetection) that expect a single run() call.

metric_keys¶

metric_keys() -> list[str]

Return metric names in canonical display order for the current evaluation mode. This is the authoritative ordering — the same list that drives summarize() and get_results().

ev = COCOeval(gt, dt, "bbox")
ev.metric_keys()
# ['AP', 'AP50', 'AP75', 'APs', 'APm', 'APl', 'AR1', 'AR10', 'AR100', 'ARs', 'ARm', 'ARl']

metric_defs¶

metric_defs() -> list[dict]

Return the metric catalog as structured data, one dict per metric in metric_keys() order: name (str), ap (bool — AP vs AR), iou_thr (float or None for the full 0.50:0.05:0.95 sweep), area (str), max_det (int), and freq_group ("rare" / "common" / "frequent" or None; LVIS only).

ev.metric_defs()[1]
# {'name': 'AP50', 'ap': True, 'iou_thr': 0.5, 'area': 'all', 'max_det': 100, 'freq_group': None}

This exists so renderers read a metric's axes instead of parsing them back out of its name — "AR10" is ambiguous between a detection cap of 10 and an IoU of 0.10, and only the catalog knows which. hotcoco's own PDF report and dashboard consume it.

get_results¶

get_results(prefix: str | None = None, per_class: bool = False) -> dict[str, float]

Return the summary metrics as a dict.

Standard bbox/segm keys: AP, AP50, AP75, APs, APm, APl, AR1, AR10, AR100, ARs, ARm, ARl.

Keypoint keys: AP, AP50, AP75, APm, APl, AR, AR50, AR75, ARm, ARl.

LVIS keys: see The 13 LVIS metrics.

ev.run()
ev.get_results()   # {"AP": 0.578, "AP50": 0.861, ...}

For prefixed and per-class keys and how they feed an experiment tracker, see Logging metrics.

print_results¶

print_results() -> None

Print a formatted results table to stdout. For LVIS, matches the lvis-api print_results() style.

summary_lines¶

summary_lines() -> list[str]

The same lines summarize() prints, returned instead of written to stdout — one string per metric, already formatted. Use it to route the summary into a logger, a report, or a test assertion.

virtual_cat_names¶

virtual_cat_names: list[str]   # property

Category names added by Open Images hierarchy expansion — ancestor categories that exist in the hierarchy but not in the dataset's own taxonomy. Empty when not in OID mode, when no hierarchy expansion occurred, or before evaluate(). Use it to distinguish expanded ancestor categories from the model's native classes:

ev = COCOeval(gt, dt, "bbox", oid_style=True, hierarchy=h)
ev.evaluate()
ev.virtual_cat_names   # for example, ['Carnivore', 'Mammal'] — no parentheses; it's a property

slice_by¶

slice_by(slices: dict[str, list[int]] | Callable[[dict], str]) -> dict[str, Any]

Re-accumulate metrics for named subsets of images without recomputing IoU, and return one metrics dict per slice. Pass either an explicit {name: [image_ids]} mapping or a function that takes an image dict and returns a slice name.

ev.run()
by_light = ev.slice_by({"day": day_ids, "night": night_ids})
by_light["night"]["AP"]

Requires evaluate() to have run. Matching is done once and reused for every slice, so slicing a dozen ways costs barely more than slicing one way. See sliced evaluation.

report¶

report() -> dict

Return a full evaluation report.

This is the shape every hotcoco metric family reports in, so code that renders a detection report renders a panoptic or tracking one unchanged.

Returns a dict with:

ev.run()
report = ev.report()

report["metrics"]["AP"]              # 0.377
report["per_class"]["person"]["AP"]  # 0.521

Provenance values¶

What each value covers and why is in Check provenance before you publish a number. For plotting curves, see Plotting precision-recall curves.

provenance¶

provenance() -> str

Return the same value as report()["provenance"], read from the configuration alone. See Checking before you evaluate.

is_benchmark_standard¶

is_benchmark_standard() -> bool

Return True exactly when provenance() is "parity_verified". Any other value — including a variant added in a future release — reads as False, so a renderer that checks this predicate caveats unknown provenance rather than passing it through. See Checking before you evaluate.

reference_deviations¶

reference_deviations() -> list[str]

Return one human-readable sentence per way this run departs from the reference configuration, empty exactly when provenance() is "parity_verified". See Checking before you evaluate.

results¶

results(per_class: bool = False) -> dict

Return evaluation results as a serializable dict.

Returns a dict with:

ev.run()
ev.results(per_class=True)   # {"hotcoco_version": ..., "metrics": {...}, "per_class": {...}}

Serialized, the dict looks like this:

{
  "hotcoco_version": "1.0.0",
  "provenance": "parity_verified",
  "params": {
    "iou_type": "bbox",
    "eval_mode": "coco",
    "iou_thresholds": [0.5, 0.55, ...],
    "recall_thresholds": [0.0, 0.01, ...],
    "area_ranges": {"all": [0, 10000000000.0], "small": [0, 1024.0], ...},
    "max_dets": [1, 10, 100],
    "use_cats": true,
    "kpt_oks_sigmas": [0.026, 0.025, ...],
    "reference_deviations": []
  },
  "metrics": {
    "AP": 0.378, "AP50": 0.584, "AP75": 0.412, ...
  },
  "per_class": {
    "person": 0.58, "car": 0.41, ...
  }
}

For a worked example, see Saving results to JSON.

save_results¶

save_results(path: str, per_class: bool = False) -> None

Save evaluation results to a JSON file. Raises IOError if the file cannot be written.

ev.run()
ev.save_results("results.json", per_class=True)

The file holds the dict returned by results().

confusion_matrix¶

confusion_matrix(
    iou_thr: float = 0.5,
    max_det: int | None = None,
    min_score: float | None = None,
) -> dict

Compute a per-category confusion matrix. Unlike evaluate(), this method compares all detections in an image against all ground truth boxes regardless of category, enabling cross-category confusion analysis.

This method is standalone — no evaluate() call is needed first.

Parameters:

Returns a dict with:

cm = ev.confusion_matrix(iou_thr=0.5, max_det=100)
cm["matrix"]   # np.ndarray uint64, shape (K+1, K+1)

For the cell layout and a full walkthrough, see Confusion matrix in the diagnostics guide.

tide_errors¶

tide_errors(
    pos_thr: float = 0.5,
    bg_thr: float = 0.1,
) -> dict

Decompose detection errors into six TIDE error types (Bolya et al., ECCV 2020) and compute ΔAP — the AP gain from eliminating each error type.

Requires evaluate() to have been called first.

Parameters:

Returns a dict with:

result = ev.tide_errors(pos_thr=0.5, bg_thr=0.1)
result["delta_ap"]   # {"Cls": ..., "Loc": ..., "Both": ..., "Dupe": ..., "Bkg": ..., "Miss": ..., "FP": ..., "FN": ...}

For the error taxonomy and a full walkthrough, see TIDE error analysis in the diagnostics guide.

calibration¶

calibration(
    n_bins: int = 10,
    iou_threshold: float = 0.5,
) -> dict

Compute confidence calibration metrics — how well confidence scores predict actual detection accuracy.

Requires evaluate() to have been called first. Bins all non-ignored detections by confidence score and compares the mean confidence in each bin to the fraction of true positives.

Parameters:

Returns a dict with:

cal = ev.calibration(n_bins=10, iou_threshold=0.5)
cal["ece"], cal["mce"]   # (0.0412, 0.1873)

For interpretation and a full walkthrough, see Confidence calibration in the diagnostics guide.

f_scores¶

f_scores(beta: float = 1.0) -> dict[str, float]

Compute F-beta scores after accumulate() (or run()).

For each (IoU threshold, category), finds the confidence operating point that maximizes F-beta, then averages across categories — analogous to how mAP averages precision. Returns three metrics mirroring AP/AP50/AP75.

Returns a dict with three keys:

Key names reflect beta, formatted with no trailing zeros: "F0.5", "F0.5_50", "F0.5_75" for beta=0.5; "F2", "F2_50", "F2_75" for beta=2.0.

Returns an empty dict if accumulate() has not been called.

ev.run()
ev.f_scores()   # {"F1": 0.523, "F1_50": 0.712, "F1_75": 0.581}

For a full walkthrough, see F-scores in the diagnostics guide.

image_diagnostics¶

image_diagnostics(
    iou_thr: float = 0.5,
    score_thr: float = 0.5,
) -> dict

Per-image diagnostics: annotation TP/FP/FN index, per-image F1 and AP scores, error profiles, and label error candidates.

Requires evaluate() to have been called first.

Parameters:

Returns a dict with:

Each image summary dict contains:

Each label error dict contains:

diag = ev.image_diagnostics(iou_thr=0.5, score_thr=0.5)
diag["img_summary"][42]   # {"tp": 3, "fp": 1, "fn": 0, "f1": 0.857, "ap": 0.75, "error_profile": "fp_heavy"}

For a full walkthrough, see Per-image diagnostics and label error detection in the diagnostics guide.

LVIS evaluation¶

These three names are drop-in replacements for the lvis-api package, so Detectron2 and MMDetection pipelines run unchanged. For the federated protocol, see LVIS evaluation; for redirecting an existing from lvis import ... pipeline, see LVIS-based pipelines.

LVISeval¶

hotcoco.LVISeval(gt: COCO, dt: COCO, iou_type: str = "segm") -> COCOeval

Returns a COCOeval configured for federated evaluation — equivalent to COCOeval(gt, dt, iou_type, lvis_style=True). Supports run(), print_results(), and get_results(), which is what those pipelines call. The default iou_type is "segm", matching lvis-api.

LVISEval (capital E) is an alias: that is the spelling lvis-api exports and the one from lvis import LVISEval expects.

ev = LVISeval(lvis_gt, lvis_dt, "segm")
ev.run()
ev.get_results()   # {"AP": ..., "APr": ..., "APc": ..., "APf": ..., "AR@300": ..., ...}

The metric set is listed in The 13 LVIS metrics.

LVIS¶

An alias for COCO, provided because lvis-api names its dataset class LVIS. Loading is identical.

LVISResults¶

hotcoco.LVISResults(lvis_gt: COCO, results, max_dets: int = 300) -> COCO

Returns a COCO detections object, equivalent to lvis_gt.load_res(results). max_dets is accepted for API compatibility but not applied here — the 300-detection cap is a Params setting that LVISeval already configures.

Module-level functions¶

compare¶

hotcoco.compare(
    eval_a: COCOeval,
    eval_b: COCOeval,
    n_bootstrap: int = 0,
    seed: int = 42,
    confidence: float = 0.95,
) -> dict

Pairwise model comparison. Both evaluators must have had evaluate() called and use the same eval_mode, iou_type, and evaluation grid — mismatched iou_thrs, rec_thrs, max_dets, or area ranges raise ValueError rather than summarizing one run under the other's catalog. Accumulation and summarization are performed internally on the shared image set.

Returns a dict with:

result = hotcoco.compare(ev_a, ev_b, n_bootstrap=1000)
result["deltas"]["AP"], result["ci"]["AP"]   # (0.033, {"lower": 0.01, "upper": 0.05, ...})

For bootstrap interpretation and a full walkthrough, see Model comparison in the diagnostics guide.