COCOeval¶

Run COCO evaluation to compute AP/AR metrics.

from hotcoco import COCO, COCOeval

coco_gt = COCO("instances_val2017.json")
coco_dt = coco_gt.load_res("detections.json")

ev = COCOeval(coco_gt, coco_dt, "bbox")
ev.evaluate()
ev.accumulate()
ev.summarize()

use hotcoco::{COCO, COCOeval};
use hotcoco::params::IouType;
use std::path::Path;

let coco_gt = COCO::new(Path::new("instances_val2017.json"))?;
let coco_dt = coco_gt.load_res(Path::new("detections.json"))?;

let mut ev = COCOeval::new(coco_gt, coco_dt, IouType::Bbox);
ev.evaluate();
ev.accumulate();
ev.summarize();

Constructor¶

COCOeval(
    coco_gt: COCO,
    coco_dt: COCO,
    iou_type: str,
    *,
    lvis_style: bool = False,
    oid_style: bool = False,
    hierarchy: Hierarchy | None = None,
)

// Standard COCO
COCOeval::new(coco_gt: COCO, coco_dt: COCO, iou_type: IouType) -> Self

// LVIS federated
COCOeval::new_lvis(coco_gt: COCO, coco_dt: COCO, iou_type: IouType) -> Self

// Open Images
COCOeval::new_oid(coco_gt: COCO, coco_dt: COCO, hierarchy: Option<Hierarchy>) -> Self

Properties¶

params¶

params: Params

ev = COCOeval(coco_gt, coco_dt, "bbox")
ev.params.cat_ids = [1, 2, 3]
ev.params.max_dets = [1, 10, 100]

pub params: Params

let mut ev = COCOeval::new(coco_gt, coco_dt, IouType::Bbox);
ev.params.cat_ids = vec![1, 2, 3];
ev.params.max_dets = vec![1, 10, 100];

See Params for all configurable fields.

stats¶

stats: list[float] | None

ev.summarize()
print(f"AP: {ev.stats[0]:.3f}")
print(f"AP50: {ev.stats[1]:.3f}")

fn stats(&self) -> Option<&[f64]>

ev.summarize();
if let Some(stats) = ev.stats() {
    println!("AP: {:.3}", stats[0]);
    println!("AP50: {:.3}", stats[1]);
}

eval_imgs¶

Per-image evaluation results, populated after evaluate(). See Working with Results for details.

eval_imgs: list[dict | None]

fn eval_imgs(&self) -> &[Option<EvalImg>]

eval¶

Accumulated precision/recall arrays, populated after accumulate(). See Working with Results for details.

eval: dict | None

fn accumulated(&self) -> Option<&AccumulatedEval>

Methods¶

evaluate¶

evaluate() -> None

Run per-image evaluation. Matches detections to ground truth annotations using greedy matching sorted by confidence. Must be called before accumulate().

Populates eval_imgs.

accumulate¶

accumulate() -> None

Accumulate per-image results into precision/recall curves using interpolated precision at 101 recall thresholds.

Populates eval.

summarize¶

summarize() -> None

Compute and print the standard COCO metrics. Populates stats.

Prints 12 lines for bbox/segm (10 for keypoints):

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.382
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.584
 ...

run¶

run() -> None

Run the full pipeline in one call: evaluate() → accumulate() → summarize(). Primarily used with LVIS pipelines (Detectron2, MMDetection) that expect a single run() call.

metric_keys¶

metric_keys() -> list[str]

Return metric names in canonical display order for the current evaluation mode. This is the authoritative ordering — the same list that drives summarize() and get_results().

ev = COCOeval(gt, dt, "bbox")
ev.metric_keys()
# ['AP', 'AP50', 'AP75', 'APs', 'APm', 'APl', 'AR1', 'AR10', 'AR100', 'ARs', 'ARm', 'ARl']

Does not require evaluate() or run() — only depends on the evaluation mode and IoU type.

get_results¶

get_results(prefix: str | None = None, per_class: bool = False) -> dict[str, float]

Return the summary metrics as a dict. Must be called after summarize() (or run()). Returns an empty dict if summarize() has not been called.

Standard bbox/segm keys: AP, AP50, AP75, APs, APm, APl, AR1, AR10, AR100, ARs, ARm, ARl.

Keypoint keys: AP, AP50, AP75, APm, APl, AR, AR50, AR75, ARm, ARl.

LVIS keys: AP, AP50, AP75, APs, APm, APl, APr, APc, APf, AR@300, ARs@300, ARm@300, ARl@300.

ev.run()

# Basic usage (unchanged)
results = ev.get_results()
print(f"AP: {results['AP']:.3f}, AP50: {results['AP50']:.3f}")

# Prefixed keys — ready for any logger
results = ev.get_results(prefix="val/bbox")
# {"val/bbox/AP": 0.578, "val/bbox/AP50": 0.861, ...}

# With per-class AP
results = ev.get_results(prefix="val/bbox", per_class=True)
# {"val/bbox/AP": 0.578, ..., "val/bbox/AP/person": 0.82, ...}

print_results¶

print_results() -> None

Print a formatted results table to stdout. For LVIS, matches the lvis-api print_results() style. Must be called after summarize() (or run()).

results¶

results(per_class: bool = False) -> dict

Return evaluation results as a serializable dict. Must be called after summarize() (or run()). Raises RuntimeError if summarize() has not been called.

Returns a dict with:

ev.run()
r = ev.results()
print(r["metrics"]["AP"])

# With per-category breakdown
r = ev.results(per_class=True)
print(r["per_class"]["person"])

save_results¶

save_results(path: str, per_class: bool = False) -> None

Save evaluation results to a JSON file. Must be called after summarize() (or run()). Raises RuntimeError if summarize() has not been called, or IOError if the file cannot be written.

ev = COCOeval(coco_gt, coco_dt, "bbox")
ev.run()
ev.save_results("results.json")

# With per-category AP
ev.save_results("results_per_class.json", per_class=True)

The JSON structure matches the dict returned by results().

confusion_matrix¶

confusion_matrix(
    iou_thr: float = 0.5,
    max_det: int | None = None,
    min_score: float | None = None,
) -> dict

Compute a per-category confusion matrix. Unlike evaluate(), this method compares all detections in an image against all ground truth boxes regardless of category, enabling cross-category confusion analysis.

This method is standalone — no evaluate() call is needed first.

Parameters:

Returns a dict with:

Matrix layout (rows = GT, cols = predicted):

• matrix[i][j] where i ≠ K, j ≠ K — GT category i matched to predicted category j. On-diagonal = TP; off-diagonal = class confusion.
• matrix[i][K] — GT category i unmatched (false negative).
• matrix[K][j] — Predicted category j unmatched (false positive).

ev = COCOeval(coco_gt, coco_dt, "bbox")
cm = ev.confusion_matrix(iou_thr=0.5, max_det=100)

matrix = cm["matrix"]
cat_ids = cm["cat_ids"]

# True positives per category
tp = matrix.diagonal()[:-1]

# False negatives per category
fn = matrix[:-1, -1]

# False positives per category
fp = matrix[-1, :-1]

# Normalized view
print(cm["normalized"])

See Confusion Matrix in the evaluation guide for a full walkthrough.

tide_errors¶

tide_errors(
    pos_thr: float = 0.5,
    bg_thr: float = 0.1,
) -> dict

Decompose detection errors into six TIDE error types (Bolya et al., ECCV 2020) and compute ΔAP — the AP gain from eliminating each error type.

Requires evaluate() to have been called first.

Parameters:

Returns a dict with:

ev = COCOeval(coco_gt, coco_dt, "bbox")
ev.evaluate()
result = ev.tide_errors(pos_thr=0.5, bg_thr=0.1)

print(f"ap_base: {result['ap_base']:.3f}")
for k, v in sorted(result["delta_ap"].items(), key=lambda x: -x[1]):
    if k not in ("FP", "FN"):
        print(f"  {k}: ΔAP={v:.4f}  n={result['counts'].get(k, '—')}")

See TIDE Error Analysis in the evaluation guide for a detailed walkthrough.

calibration¶

calibration(
    n_bins: int = 10,
    iou_threshold: float = 0.5,
) -> dict

Compute confidence calibration metrics — how well confidence scores predict actual detection accuracy.

Requires evaluate() to have been called first. Bins all non-ignored detections by confidence score and compares the mean confidence in each bin to the fraction of true positives.

Parameters:

Returns a dict with:

ev = COCOeval(coco_gt, coco_dt, "bbox")
ev.evaluate()

cal = ev.calibration(n_bins=10, iou_threshold=0.5)
print(f"ECE: {cal['ece']:.4f}, MCE: {cal['mce']:.4f}")

# Per-category breakdown
for name, ece in sorted(cal["per_category"].items(), key=lambda x: -x[1])[:5]:
    print(f"  {name}: ECE={ece:.4f}")

See Confidence Calibration in the evaluation guide for a full walkthrough.

f_scores¶

f_scores(beta: float = 1.0) -> dict[str, float]

Compute F-beta scores after accumulate() (or run()).

For each (IoU threshold, category), finds the confidence operating point that maximizes F-beta, then averages across categories — analogous to how mAP averages precision. Returns three metrics mirroring AP/AP50/AP75.

Returns a dict with three keys:

Key names reflect beta: "F0.5", "F0.550", "F0.575" for beta=0.5, etc.

Returns an empty dict if accumulate() has not been called.

ev = COCOeval(coco_gt, coco_dt, "bbox")
ev.run()

# F1 (default)
scores = ev.f_scores()
print(f"F1: {scores['F1']:.3f}, F1@50: {scores['F150']:.3f}")

# Precision-weighted
print(ev.f_scores(beta=0.5))   # {"F0.5": ..., "F0.550": ..., "F0.575": ...}

# Recall-weighted
print(ev.f_scores(beta=2.0))   # {"F2.0": ..., "F2.050": ..., "F2.075": ...}

image_diagnostics¶

image_diagnostics(
    iou_thr: float = 0.5,
    score_thr: float = 0.5,
) -> dict

Per-image diagnostics: annotation TP/FP/FN index, per-image F1 and AP scores, error profiles, and label error candidates.

Requires evaluate() to have been called first.

Parameters:

Returns a dict with:

Each image summary dict contains:

Each label error dict contains:

ev = COCOeval(coco_gt, coco_dt, "bbox")
ev.evaluate()

diag = ev.image_diagnostics(iou_thr=0.5, score_thr=0.5)

# Worst images by F1
worst = sorted(diag["img_summary"].items(), key=lambda x: x[1]["f1"])[:5]

# Label errors
for le in diag["label_errors"]:
    print(f"{le['type']}: {le['dt_category']}→{le.get('gt_category', 'N/A')}")

See Per-image diagnostics in the evaluation guide.

Module-level functions¶

compare¶

hotcoco.compare(
    eval_a: COCOeval,
    eval_b: COCOeval,
    n_bootstrap: int = 0,
    seed: int = 42,
    confidence: float = 0.95,
) -> dict

Pairwise model comparison. Both evaluators must have had evaluate() called and use the same eval_mode and iou_type. Accumulation and summarization are performed internally on the shared image set.

Returns a dict with:

import hotcoco

gt = hotcoco.COCO("annotations.json")
ev_a = hotcoco.COCOeval(gt, gt.load_res("baseline.json"), "bbox")
ev_a.evaluate()
ev_b = hotcoco.COCOeval(gt, gt.load_res("improved.json"), "bbox")
ev_b.evaluate()

# Without bootstrap
result = hotcoco.compare(ev_a, ev_b)
print(result["deltas"]["AP"])  # e.g. +0.033

# With bootstrap CIs
result = hotcoco.compare(ev_a, ev_b, n_bootstrap=1000)
ci = result["ci"]["AP"]
print(f"[{ci['lower']:+.3f}, {ci['upper']:+.3f}]")  # e.g. [+0.01, +0.05]