aichannel

some insights from my attempt to run full deepseek r1 model.

you will need at least 6x a100/h100 level gpus to run q4_k_m quantization.

i rented 8x a100 for $14/hr, so i had 640gb vram at my disposal. the quickest way to set this up is ollama. model download size is 404gb.

the…

banteg

creating and destroying; built flying cars & autonomous vehicles, now building roc.camera, writings: july.rocks

@july wdyt? cc @banteg 

https://x.com/bantg/status/1884972148722225610

Complete hardware + software setup for running Deepseek-R1 locally. The actual model, no distillations, and Q8 quantization for full quality. Total cost, $6,000. All download and part links below:

Matthew Carrigan

Working on Farcaster and Warpcast. Longer thoughts at https://dwr.email

https://x.com/carrigmat/status/1884244369907278106

yeah this was a solid thread

also ggml + cpu does seem like such a nice overall solution as there are so many more CPUs than GPUs