Farcaster Devs

what is the fastest way to get all the links for a list of fids assuming i have direct access to a hub (and potentially the rocks db)?

- the hub rest api is slow and can only do a max pageSize of 1000, but is reliable
- grpc is much faster (no page size limit) but starts falling over when you blast it with requests

i'm considering trying to read directly from rocksdb at this stage

any ideas?

I'm eth newbie..just learning
(working on @farcaster)

terminally.online

Farcaster Development Boutique @ https://dTech.vision 💜 in love with tech 🤓decentralization, fitness and oldschool rap 😎

hm i see the rpc only returns linkAdd messages. don't see any filtering happening in these methods. will have a deeper look later into how deletes are handled by the link store and if there's any pruning that happens with linkAdds otherwise this may get tricky

SQLQuery against rocks db sounds solid + adaptable though be aware looking for all links you need to check for link delete aka unfollow messages too if you want to get the current state of what the FID intended

data science @ coinbase                                


t0ny.base.eth

would like to avoid replicating data to keep it simple, but will look into this thanks!

Getting rocks replicating to a graphdb like Neptune or neo4j might be the best for this type of thing, haven't found an open source or free option yet though