LangSmith adds evaluators you tune to your own labels, starting with Perceived Error

LangChain introduced LangSmith Tuned Evaluators on 18 August 2026, beginning with a Perceived Error evaluator — a judge aimed at what a user would call wrong rather than at a rubric written in advance. The point of a tuned evaluator is that the team fits it to its own labelled examples instead of accepting a generic judge prompt, which is what has made off-the-shelf LLM-as-judge scores hard to trust across products.

What it means Tuning the judge on your own labels is the honest version of LLM-as-judge; it also makes the evaluator a maintained asset with a drift problem, which is a cost worth budgeting before adopting one.