gm8xx8

The push for smaller, cost-effective language models (SLMs) is accelerating! 

LLaMa3.1 has been pruned to 4B parameters. Nvidia cooked! 

Width pruning:
- MMLU accuracy: 60.5%
- GSM8K accuracy: 41.24%
- Speedup: ~1.7x
Depth pruning:
- MMLU accuracy: 58.7%
- GSM8K accuracy: 16.8%
- Speedup: ~2.7x
- depth pruning affects reasoning ability, consistent with recent findings (arxiv.org/abs/2407.16286).

Pruning and Distillation Benefits:

- MMLU scores improve by 16% compared to training from scratch.
- Training tokens are reduced by up to 40x, needing ~100B tokens.
- Compute cost for training is up to 1.8x lower.
- Performance is comparable to larger models trained on up to 15T tokens.

read more: https://developer.nvidia.com/blog/how-to-prune-and-distill-llama-3-1-8b-to-an-nvidia-llama-3-1-minitron-4b-model/

Llama-3.1-Minitron 4B—Nvidia cooked! 

- better quantization
- better model merging

don’t sleep.