r/LocalLLaMA
· Communities
Revision Prompting: Trades slow (decoded) output tokens for cheap (prefilled) input tokens.
TL;DR: If you re-run the same prompt whenever the input changes, try sending the old input/output plus a diff of the input, and ask the model for a patch to the output. You generate ~2-10x fewer output tokens, and the untouched parts of the output stay byte-identical. This works, since in LLMs, tg is the bottleneck and