r/MachineLearning
· Communities
High Dimensional, Dynamic Rotary Positional Embedding [P]
At the end of my last post, I presented an idea: what if I used the core of my last project, the cumulative matrix product, and repurposed it as a positional embedding? I just finished fleshing out the math behind HDD-RoPE and training a model with this positional embedding algorithm, and the results are excellent. Whe