Talked with a dev in Austin who made me question my whole AI workflow

I was at a meetup last week and this guy from a small startup showed me how he uses local LLMs for code refactoring instead of paying for API calls. Said he saved $400 a month just by switching to a quantized model on his laptop. Made me realize I've been overspending on cloud AI tools for simple tasks. Anyone else started running more stuff locally and seen a difference?

3 comments

3 Comments

fionamurphy1d ago

too realistic dice rolls" wait what that's actually insane how is that even possible

brian_hart20h ago

The D&D thing is actually a perfect example of why local models can be better. @fionamurphy your friend is right, quantized models have this weird thing where their randomness is actually more consistent because its not being filtered through some corporate API safety layer. The dice roll thing is real - I tried a local llama model for some game logic and it gave me a natural 1 that felt way more brutal than what ChatGPT would generate. The devs setting up these local setups know what theyre doing with temperature settings and stuff.

gracej991d ago

Local models are wild honestly my coworker downloaded one to help him write D&D campaigns and now he's mad because the dice rolls are too realistic lmao