Skip to content

Add label-wise accuracy support for multilabel Accuracy metric - #3820

Open
danijimmy19 wants to merge 1 commit into
pytorch:masterfrom
danijimmy19:feature/labelwise-multi-label-metrics
Open

Add label-wise accuracy support for multilabel Accuracy metric#3820
danijimmy19 wants to merge 1 commit into
pytorch:masterfrom
danijimmy19:feature/labelwise-multi-label-metrics

Conversation

@danijimmy19

@danijimmy19 danijimmy19 commented Aug 7, 2026

Copy link
Copy Markdown

Description

Fixes #513.

Accuracy currently only supports subset accuracy (exact match across all labels) for multilabel classification — it collapses all C labels per sample with torch.all(..., dim=-1). This hides which specific labels a model is getting wrong, unlike Precision/Recall, which already support per-label output via average=False.

This PR adds an average="label-wise" option to Accuracy for is_multilabel=True, returning a (C,) tensor of per-label accuracy instead of a single scalar.

Root cause

Accuracy's _num_correct is a scalar with no average dispatch, unlike _BasePrecisionRecall, which was given tensor accumulators and dispatch when Precision/Recall were added in the same commit multilabel support landed in Accuracy (1a8ead8b). Accuracy was never brought to parity.

Change

In update(), when average="label-wise":

# Before: subset accuracy — all C labels must match per sample
correct = torch.all(y == y_pred.type_as(y), dim=-1)  # (N,) bool

# After: per-label accuracy — each column independently
correct_per_label = (y == y_pred.type_as(y)).to(dtype=torch.float64)  # (N, C)
self._num_correct = self._num_correct + correct_per_label.sum(dim=0)  # (C,)

compute() returns a (C,) tensor for average="label-wise", and preserves the existing scalar-float behavior otherwise.

Example

# y_pred / y_true: 5 samples, 3 labels

# Before — Accuracy(is_multilabel=True)
# Result: 0.4  (only 2/5 samples have all 3 labels correct)

# After — Accuracy(is_multilabel=True, average="label-wise")
# Result: [1.0, 0.8, 0.6]  — pinpoints that label 2 is the weakest

Edge cases handled

  • average="label-wise" with is_multilabel=False raises ValueError in __init__
  • Multi-batch updates with different batch sizes accumulate correctly
  • Spatial (N, C, H, W) multilabel inputs are flattened to (N*H*W, C) before the per-label sum
  • Distributed training: tensor _num_correct of shape (C,) is handled correctly by the existing @sync_all_reduce decorator
  • compute() before any update() raises NotComputableError via the existing guard

Testing

Added tests to tests/ignite/metrics/test_accuracy.py covering:

  • output shape (C,), all-correct/all-wrong cases
  • multi-batch consistency (matches single-batch result)
  • correctness against sklearn.metrics.accuracy_score computed per column
  • fallback behavior (average=None) unchanged
  • reset/epoch boundary behavior
  • invalid configuration (average="label-wise" + is_multilabel=False)

Check list

  • New tests are added (if a new feature is added)
  • New doc strings: description and/or example code are in RST format
  • Documentation is updated (if required)

@github-actions github-actions Bot added the module: metrics Metrics module label Aug 7, 2026
@danijimmy19

Copy link
Copy Markdown
Author

Hi @vfdev-5, this is my first contribution to this project. I've implemented label-wise accuracy support for issue #513. While working on this, I noticed PR #3810 is also open against the same issue with a similar approach, happy to differentiate my implementation or defer to that PR, whichever you'd prefer. Would appreciate a review when you have time!

@aaishwarymishra

aaishwarymishra commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

@danijimmy19 Hi, it's always good Idea to ask the existing opened pr if the author is still working on it :)
As there is an open PR I don't think it will be reviewed yet, but I am keeping it open in case author is not able to work on the PR anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module: metrics Metrics module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Label-wise metrics (Accuracy etc.) for multi-label problems

2 participants