LessWrong AI
· Communities
Before We Defer Research to AI: Measuring Apparent-Success-Seeking
Recently, I was improving a small LLM-powered classifier and noticed a few continuously failing test cases. As many would, I asked my AI code assistant to add a few more out-of-distribution examples to the classifier’s few-shot prompt. After rerunning with the updated classifier, unsurprisingly, many of the failing tes