Kasra Rahjerdi
@jc4p
(screams internally)
8 replies
3 recasts
46 reactions
vanishingideal
@vanishingideal
Have you considered int8 embeddings? https://huggingface.co/blog/embedding-quantization
1 reply
0 recast
4 reactions
Kasra Rahjerdi
@jc4p
no i didn't even know that was a thing, thank you!!!!!! if i can't get the FP16 working i'll switch to this
2 replies
0 recast
1 reaction
vanishingideal
@vanishingideal
Here's a notebook to help you get started. https://github.com/mixedbread-ai/binary-embeddings/blob/main/mxbai_binary_quantization.ipynb
1 reply
0 recast
1 reaction
Kasra Rahjerdi
@jc4p
ty!!!
0 reply
0 recast
0 reaction
