X · @teortaxesTex
· X / Twitter
wow did they discover speculative decoding or something? margins go up again!
wow did they discover speculative decoding or something?margins go up again!Stephanie Palazzolo: OpenAI engineers earlier this month developed an optimization that cut inference costs in half for models it was applied to.After the optimization was applied to logged-out ChatGPT traffic, it reduced the number of GPUs nee