Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .ai/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Strive to write code as simple and explicit as possible.
- **Pipelines** — see [pipelines.md](pipelines.md) for pipeline conventions, patterns, and gotchas.
- **Modular pipelines** — see [modular.md](modular.md) for modular pipeline conventions, patterns, and gotchas.
- **Tests** — see [testing.md](testing.md) for test conventions: required test layers, tester mixins, and dummy-component rules.
- **Reporting** — see [reporting.md](reporting.md) for how to write bug reports (what a reproduction is) and performance claims (end-to-end numbers first).

## Skills

Expand Down
59 changes: 59 additions & 0 deletions .ai/reporting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Reporting bugs and performance claims

For issues and PR descriptions on this repo, from humans and agents alike.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
For issues and PR descriptions on this repo, from humans and agents alike.
Use this guide when writing bug reports or performance claims in issues and pull requests.


## Bug reports

A reproduction is **what you were doing when it broke, with everything unnecessary removed** — start from the real failing situation and delete, don't build a clean synthetic case from scratch. The test: can someone paste it into a terminal and see the same failure, without first accepting your theory of the cause? A script that demonstrates your theory is not a reproduction — it can pass while the real bug is still there.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We won't probably be pasting the code into the terminal but rather create a Python file out of the code snippet and run it?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A reproduction is **what you were doing when it broke, with everything unnecessary removed** — start from the real failing situation and delete, don't build a clean synthetic case from scratch. The test: can someone paste it into a terminal and see the same failure, without first accepting your theory of the cause? A script that demonstrates your theory is not a reproduction — it can pass while the real bug is still there.
A reproduction is the smallest version of the real workflow that still fails. Start with what you were doing when the failure occurred, then remove anything unrelated. Someone should be able to run it and see the same failure without first accepting your theory about its cause. A script that tests your theory is not a reproduction. It may pass even when the real workflow still fails.


- Keep the real model, settings, and dtype. A repro that downloads weights and takes two minutes is worth more than a fast synthetic one.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we should also ask it to strive to be minimal even in that regard. For example, just using the bare minimum dependencies, and no externalities at all (like invoking diffusers from a servlet for example). Thoughts?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Keep the real model, settings, and dtype. A repro that downloads weights and takes two minutes is worth more than a fast synthetic one.
- Keep the real model, settings, and dtype. A slower repro is better than a faster one that changes the behavior.

- If you truly can't produce one (gated model, 8 GPUs), say so at the top and give the closest thing you have. Don't silently substitute something smaller.

✓ A reproduction — the failing call itself, trimmed (from [#14518](https://github.com/huggingface/diffusers/issues/14518): Krea-2 OOMs a 16GB GPU under 4-bit quantization):

```python
import torch
from diffusers import Krea2Pipeline

pipe = Krea2Pipeline.from_pretrained(...)
pipe.to("cuda")

image = pipe(...).images[0]

print(f"peak allocated: {torch.cuda.max_memory_allocated() / 2**30:.2f} GiB") # 19.94 GiB — OOM on 16GB
```

✗ Not a reproduction — a synthetic benchmark built to demonstrate the suspected cause (paraphrased from an earlier draft of the same report):

```python
# "enable_gqa + attn_mask falls back to the math backend and materializes an S x S score matrix"
q = torch.randn(1, 24, 8100, 128, device="cuda", dtype=torch.bfloat16)
k = v = torch.randn(1, 6, 8100, 128, device="cuda", dtype=torch.bfloat16)
mask = torch.ones(1, 1, 8100, 8100, dtype=torch.bool, device="cuda")
F.scaled_dot_product_attention(q, k, v, attn_mask=mask, enable_gqa=True)
print(torch.cuda.max_memory_allocated())
```

The difference: a maintainer understands the first script at a glance — it's an ordinary pipeline call — while the second is not, and it only demonstrates the reporter's theory: it can go green while the real pipeline still dies on a 24GB card.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The difference: a maintainer understands the first script at a glance — it's an ordinary pipeline call — while the second is not, and it only demonstrates the reporter's theory: it can go green while the real pipeline still dies on a 24GB card.
The first script exposes the real pipeline call. The second only tests the proposed mechanism, so it may pass even when the pipeline still fails. Keep the hypothesis separate from the reproduction.


For structure, follow the [bug report template](../.github/ISSUE_TEMPLATE/bug-report.yml).

## Performance claims

Report **the end-to-end number only**: wall clock for the full pipeline call, before vs. after, on the same hardware, dtype, and seed. That's the number we decide with — if we want op-level detail, we'll ask.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Report **the end-to-end number only**: wall clock for the full pipeline call, before vs. after, on the same hardware, dtype, and seed. That's the number we decide with — if we want op-level detail, we'll ask.
Lead with the end-to-end result: measure the full pipeline call before and after the change on the same hardware, dtype, and seed. Only include lower-level measurements when requested.


Attach the script you measured with, please keep it as simple as possible (see `benchmarks/benchmarking_utils.py`):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Attach the script you measured with, please keep it as simple as possible (see `benchmarks/benchmarking_utils.py`):
Attach the measurement script and keep it as simple as possible. See `benchmarks/benchmarking_utils.py` for the repository's benchmarking helper.


```python
import torch.utils.benchmark as benchmark

def benchmark_fn(f, *args, **kwargs):
t0 = benchmark.Timer(stmt="f(*args, **kwargs)", globals={"args": args, "kwargs": kwargs, "f": f}, num_threads=1)
return f"{t0.blocked_autorange().mean:.3f}s"

pipe(**call_kwargs) # warmup
print(benchmark_fn(pipe, **call_kwargs))
print(f"peak memory: {torch.cuda.max_memory_allocated() / 1024**3:.2f}GB")
```

State the setup (GPU, dtype, torch/diffusers versions, attention backend) and the exact call (model id, resolution/duration, steps, batch size). One-shot timings are noise.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
State the setup (GPU, dtype, torch/diffusers versions, attention backend) and the exact call (model id, resolution/duration, steps, batch size). One-shot timings are noise.
State the setup (GPU, dtype, torch/diffusers versions, attention backend) and the exact call (model id, resolution/duration, steps, batch size). One-shot timings are noisy.

Loading