X · @togethercompute
· X / Twitter
The first thing we learned building autoscaling for dedicated inference is that CPU-style metrics don't tell the whole picture. A GPU can read 60% bus…
The first thing we learned building autoscaling for dedicated inference is that CPU-style metrics don't tell the whole picture. A GPU can read 60% busy while the engine's queue is already backing up, because utilization measures arithmetic intensity, not pressure.@soyoung_park's deep dive covers what to do about it:->