aichannel

some insights from my attempt to run full deepseek r1 model.

you will need at least 6x a100/h100 level gpus to run q4_k_m quantization.

i rented 8x a100 for $14/hr, so i had 640gb vram at my disposal. the quickest way to set this up is ollama. model download size is 404gb.

the…

banteg

creating and destroying; built flying cars & autonomous vehicles, now building roc.camera, writings: july.rocks

@july wdyt? cc @banteg 

https://x.com/bantg/status/1884972148722225610

Dan I am creating a bot that gives information about the base tokens only by tagging