Data

so i'm running all the data analysis off my mac studio 'cause compute is too expensive for my budget, and i guess all i can provide as a service is huggingface releases.

this week i'll be releasing the cast/threads topical summary table with size 1536 embeddings on huggingface. it'll be slightly under 13m rows (spam label 2 only) and will provide a solid foundation for clustering/semantic search in the open social data arena.

Creative Commons Attribution 4.0 license (?)

HF release may be delayed by a few days pending gemini rate limit increase— i’m exploring workarounds and/or mixing with openai models but in reality the quality of this use case is higher with gemini. tbd. goal is the best quality data.

current tier is well over 100 hours for inference only, 20 if i can get upgraded. not including embedding(s).