𝚐𝔪𝟾𝚡𝚡𝟾

gm8xx8

Google releases Gemma 2 27B and 9B models.

benchmarks:

- 9B model leads its category, outperforming LLAMA 3 70B and Qwen 72B.
- trained on significantly fewer tokens, the 27B model uses 13 trillion and the 9B uses 8 trillion, both featuring advanced context and attention mechanisms for enhanced performance.

key updates include:

- knowledge distillation to optimize smaller models.
- hybrid attention layers for stable, efficient processing in long contexts.
- controlled attention expansion to ensure stable training.
- enhanced performance merging techniques like EMA, SLERP, and LITI.
- group query attention for quicker inference speeds.

https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315 

https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf

the 27B Gemma 2 model efficiently runs full-precision inference on a single Google Cloud TPU host, NVIDIA A100 80GB Tensor Core GPU, or NVIDIA H100 Tensor Core GPU.
-
word on the street is we’ll get an updated 2b soon.
🍻