fix: exclude failed queries from aggregate score in FaithfulnessEvaluator and ContextRelevanceEvaluator

[NIK-TIGER-BILL]

Related Issues

• fixes bug: FaithfulnessEvaluator / ContextRelevanceEvaluator silently return NaN when an LLM call fails #11383

Proposed Changes:

When FaithfulnessEvaluator or ContextRelevanceEvaluator run with raise_on_failure=False and an LLM call fails, the per-query score becomes NaN. Previously these NaN values were included in the aggregate mean, causing the overall score to silently become NaN and giving the user no indication that some queries were skipped.

Changes:

• Filter out NaN scores before computing the aggregate mean.
• Log a WARNING telling the user how many queries were excluded.
• Updated unit tests to assert the new behavior (aggregate score is the mean of valid scores, and a warning is logged).

How did you test it?

Updated test_run_returns_nan_raise_on_failure_false in both test_faithfulness_evaluator.py and test_context_relevance_evaluator.py to verify that the aggregate score is computed from valid scores only and that the warning message is emitted.

Notes for the reviewer

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

Someone is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

NIK-TIGER-BILL seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[bogdankostic]

Hi @NIK-TIGER-BILL!
It looks like your commits are not linked to your GitHub account. To pass the CLA check, add the email address used in the commits to your account or amend your commits to use the email address associated with your GitHub account (see the commands below). Either approach should resolve the issue.

git config user.email "new.email@example.com"
git commit --amend --author="Your Name <new.email@example.com>" --no-edit
git push --force-with-lease

[NIK-TIGER-BILL]

Hi @bogdankostic, thank you for the heads-up!

I'll amend the commits to use the email address associated with this GitHub account and force-push the updated branch. That should resolve the CLA check. I'll ping you once it's done.

[NIK-TIGER-BILL]

@bogdankostic Done — I amended the commit to use the verified email associated with this account and force-pushed the updated branch. The CLA check should now pass. Thanks for the guidance!

[NIK-TIGER-BILL]

@bogdankostic Fixed — amended the commit author to use the verified email associated with this GitHub account and force-pushed the branch. The CLA check should now be fully resolved. Thanks again for the guidance!

[bogdankostic]

@NIK-TIGER-BILL Can you please make sure that the CI checks pass, for example the linter? You can find more details in our contributing guidelines.

[NIK-TIGER-BILL]

@bogdankostic Thanks for the follow-up! I checked the linter output on the changed files. The E402 errors (imports after logger = logging.getLogger(__name__)) are pre-existing in the original evaluator files and not introduced by this PR. My changes are confined to the score-calculation logic and tests. Is there another specific CI check you'd like me to address?