r/LocalLLaMA
· Communities
Softmax for People Trying To Run Models on FPGAs
My weekend sideproject was implementing the paper 'Approximating Softmax for FPGAs with Taylor Series and Pade Approximants' The paper’s motivation is the hardware constraints limiting exponential operations on FPGAs. I coded the paper in Python to verify the formulas just before getting it on my card. The authors find