Zora

Nvidia | nGPT: Normalized Transformer with Representation Learning on the Hypersphere

This paper propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere. 

> This research show that nGPT learns much faster, reducing the number of training steps required to achieve the same accuracy by a factor of 4 to 20, depending on the sequence length.

Source: https://arxiv.org/abs/2410.01131

Nous Research Announces an Open Source Implementation of NVIDIA’s nGPT Paper.

Their research shows that nGPT learns much faster and achieves comparable performance to GPT with significantly fewer training steps.

https://github.com/JoeLi12345/nGPT

Source: https://x.com/nousresearch/status/1898073676433551630