might've under estimated how long it'd take me to process 157 million casts, but in 6 hours i'll have authoritative rankings on who said what words first!!

sneak preview, shout out to my homegirl @keccers.eth

The Farcaster formerly known as Kchamp. Books, beats, bodybuilding, bright ideas and beautiful things.

katherinechampagne.com

🔍 ensdata.net ✦ 🖼️ pugson.net/rfe ✦  🌐 metadata.vision ✦ 𝙽𝙾𝚆: design engineer @family ✦ 𝙿𝚁𝙴𝚅: @rainbow 🌈

i’m sure i could’ve used a GPU to do this faster but im in too deep now to try another approach

anything that requires tons of storage or a GPU —>  https://lambdalabs.com/ (eg pulling the data from the hub)

anything that just requires CPU cycles —> locally

do you rent a machine or do it all locally?

this rn is lemmatization + tokenizing but for your case you def need embeddings, can generate those embeddings pretty cheaply on a local GPU using the hf dataset and https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 tho!

@operator & /operator | BCI, bioinformatics, search, chaos, decentralized AI

ooh nice i was thinking of building this into our agent but decided it was too much work

is it exact text/ngram or embedding based?

basically i wanted to be able to answer if someone asked "i saw this cast by ___ recently, are there other people who discussed similar things before?"

Sr. Product/UX Designer in automotive. Dad. /starcaster Design Resources: www.uiresource.design building /rumours rumourcast.xyz

So.. Who was the first one to mention a duck? :)

Im the guy that posted all those factory memes once

lmfao i’ll try to have them done by end of week

Are you doing embeddings because it was literally gonna take me like 20 days to generate them on my older Mac LMAO

hell yeah, have been waiting for someone to build this one

Impressive dedication to research and analysis