X · @teortaxesTex
· X / Twitter
RT mr-r0b0t: "Trained on only 15B tokens, HydraHead improves long-context performance by 69%+ on NIAH at 512K context, approaching the performance of …
RT mr-r0b0t"Trained on only 15B tokens, HydraHead improves long-context performance by 69%+ on NIAH at 512K context, approaching the performance of Qwen3.5."NIAH = Needle In A HaystackBasically the model finds a single out of place string in a 512k ctx dataset.69% improvement 👀Tongyi Lab: HydraHead combines:• an interp