r/LocalLLaMA
· Communities
Got a 27B model running locally on a Jetson Orin NX 16GB (1-bit). still kind of amazed it works
Disclosure: this is my own repo — all numbers below are measured on my own board. I've had a Jetson Orin NX 16GB sitting on my desk for a while and finally got around to seeing how far I could push it. Ended up with PrismML's Bonsai 27B running fully offline on it, and honestly I'm still a little surprised it works at