fam in ~18 hours we'll have 157 million casts with embeddings, pray for me it doesn't crash overnight

not sure if it will help or not but here's mine + 8bit quantization after on f16. i rerun this every 5 min

not sure if it will help or not but here's mine + 8bit quantization after on f16. i rerun this every 5 min

https://gist.github.com/alexpaden/b99668307e6e16c18e5ce581c8d719b8

prev founded rocketr.net, swish.id now exploring aiXcrypto identity ai. expired sprint car driver & energy data engineer.

^ it's cpu optimized for mac studio not gpu

love it, thank you! i’m planning on doing monthly ingestion of casts + monthly embeddings of those so hopefullyyyy this will be easier after i finish this giant backfill

yeah only 1 failure isn't bad at all it took me like a week because i was trying to figure out optimizations which i am still missing a bunch on i think

lol i casted about my learnings before (idk where) but the code prob says better. basically preallocating memory buffers and in my case using mps

i think on avg i was at ~10ktextspersecond but i think if correctly optimized could prob break 50-100k. gpu i think is faster inference idk this code records the time taken per major section which was helpful for me

silly me i thought the hard part would be writing a super optimized gRPC client 😂🙈