hey if you're wondering where the public data dump for April is, I need help figuring out the implications of my decisions with the Snapchain setup -- would love any ideas.

Short of it: Snapchain prefers a 24/7 running instance, can't afford to run a heavy box 24/7, stuck on filtering 90GB of data on 8GB of RAM

here's the old code for doing this on a beefy server capable of loading everything in RAM:

here's the old code for doing this on a beefy server capable of loading everything in RAM: https://github.com/jc4p/fast-hub-client/blob/9db0f923b6ad0f1fb943e45e3caaa1e1e7746d9d/HubClient/scripts/export_non_removed_reactions.sql

oh i messed up this chart the snapchain flow does NOT include shut down server at the end

it’s a full data dump of casts, likes, recasts, and follows — so anyone can do analysis on whatever they want

it’s a full data dump of casts, likes, recasts, and follows — so anyone can do analysis on whatever they want https://huggingface.co/datasets/jc4p/farcaster-casts

what’s this for? what is hugging face doing there?

@christopher has a great solution with this (where you can like, do extra filtering at the redis consumption layer) https://github.com/officialunofficial/waypoint -- my main issue rn stems from doing lifetime dumps instead of incremental dumps, if i had a queue of the last day's likes or something it would be a lot easier

future proof dev. joying life of bugs, phd on hot hot fix in prod

have you thought about Redis to store what you to access under a ttl renewed by any criteria?

🐟 Snappa | picosub.artlu.xyz | SassyHash.artlu.xyz

Find me with my same PFP on GH

not a dumb question at all, basically i am repeating code i wrote for bulk retrieval (all time) instead of writing new code to only pull the latest data, so i need something a lot more performant than shuttle -- last time i benchmarked my grpc client was 30-40x faster

stoopid question (forgive me for asking)--what's keeping you from using Shuttle?

https://github.com/farcasterxyz/hub-monorepo/tree/main/packages/shuttle

so the snapchain itself is an event driven system and then i retrieve data from it in bulk, i could switch over to a secondary event driven system tho and filter on demand then save

do you have to load everything into ram? can't you just run an event-driven system?

LONE Explorer 👨🏽‍🚀..Tech/crypto crunch, Gamer, music and art.exploring. 
Yours truly _86ed  /nightcrawlers

oh god i don’t understand it either tbh i tried setting it up manually then kept failing and ended up having claude code do it

Yo bro quick question though, I don't really understand how to migrate to the Snapchain

Can you perhaps help? I think my warpcast reward stuck cos of it🤔

I'm eth newbie..just learning
(working on @farcaster)

terminally.online

ty! best idea i have so far is doing the grpc client stuff on a temp server with lots of RAM but i have ONLY tested the client hitting localhost and don't want another optimization rabbit hole

tagging @sanjay and @dynemyte who might have ideas. (Sanjay is on vacation this week though)

I don't know much about Snapchain and basically stopped reading after parquet, but have you tried Clickhouse? I know it compresses the data and it's extremely fast

My casts are immutable, but my opinions might get a soft fork.

Followed by @woj.eth 🎖️

yeahhhh that is a valid idea but god i don't want to setup even more infrastructure haha, might be the best option though i could chunk into AWS then use a serverless function to combine

Would an intermittent data snapshot system help? Could store compressed chunks and process only active datasets on-demand, leveraging cloud compute for the heaviest bits?