Can we have Datacentre AI... No, we have AI at Home. - Printable Version +- Drunkard's Walk Forums (http://www.accessdenied-rms.net/forums) +-- Forum: General (http://www.accessdenied-rms.net/forums/forumdisplay.php?fid=1) +--- Forum: General Chatter (http://www.accessdenied-rms.net/forums/forumdisplay.php?fid=2) +--- Thread: Can we have Datacentre AI... No, we have AI at Home. (/showthread.php?tid=15050)

Can we have Datacentre AI... No, we have AI at Home. - Dartz - 09-09-2026 The "AI" we have at home This is Colibri: https://github.com/JustVugg/colibri It's some open source thing that lets you run big large language models offline on 'ordinary' hardware. It works by only holding the parts of the model that're actually being triggered in RAM, rather than all 300GB. The rest it streams from your disk. The default model it runs with is GLM which is also open-source (Unless Nvidia gets dickish with Hugginface) It took 3 hours to generate a script that took Gemini 3 minutes to do - running at a blistering .24 tokens a second. But it ran - on a 5 year old laptop with 32GB of DDR4 RAM. And the script it spat out worked, and did exactly what I told it to do, logic errors and all. It'll run on 16 - potentially. But 32 is best. Also - it won't kill the SSD because reading is harmless. Meanwhile, I know someone whose been able to justify buying a fully loaded Mac Studio to their manager to run these models locally because they were burning so many tokens just using these things to automate software testing and data generation. One imagines once these things start getting unchained from the big datacentre model - then the big datacentres are going to be a little less valuable.  I haven't given it  the ability to read or write files yet - I don't know how to do that. But it says a lot that it's gotten to the point where I can get it to work - and it's been a curious thing to play with and the first 'computer' thing that's amused me since I last ran Xubuntu.