Content pfp
Content
@
0 reply
0 recast
0 reaction

Samuel pfp
Samuel
@samuellhuber
What issue do you currently have indexing Farcaster data? e.g. using Replicator, Shuttle or APIs?
1 reply
0 recast
5 reactions

Jason Goldberg ๐Ÿณ๏ธโ€๐ŸŒˆ pfp
Jason Goldberg ๐Ÿณ๏ธโ€๐ŸŒˆ
@betashop.eth
Shuttle
1 reply
0 recast
2 reactions

Samuel pfp
Samuel
@samuellhuber
What do you wish shuttle had ? Howโ€™s your experience running it?
1 reply
0 recast
0 reaction

ฮžric Juta pfp
ฮžric Juta
@ericjuta
forked shuttle, made the message type processing concurrent hard to estimate how many concurrent workers to allow/configure + hardware requirements for a backfill let alone how long it is too also replicator schema hadn't been fully ported to shuttle schema but ya I guess materialised tables are personal decisions
1 reply
0 recast
2 reactions

Samuel pfp
Samuel
@samuellhuber
Thought about just writing it in Rust? If youโ€™re already making it concurrent and add parallelism why stay with nodejs?
1 reply
0 recast
1 reaction

ฮžric Juta pfp
ฮžric Juta
@ericjuta
meh just ran it using bun, intermediary trying to gauge ROI on backfilling the 1TB of data will return to it later shame there's no public process atm on just preloading shuttle Message schema with a downloadable snapshot especially since the replicator > shuttle schema changed cc @sidshekhar
1 reply
0 recast
1 reaction

Samuel pfp
Samuel
@samuellhuber
So a public PSQL being synced using shuttle to then initialize from?
1 reply
0 recast
0 reaction

ฮžric Juta pfp
ฮžric Juta
@ericjuta
ya basically skip the backfill process via shuttle
1 reply
0 recast
1 reaction

Samuel pfp
Samuel
@samuellhuber
Do you have specific indexing requirements? Currently exploring if it is worth the effort to build shovel but for hubs or if shuttle is enough for now
1 reply
0 recast
0 reaction

ฮžric Juta pfp
ฮžric Juta
@ericjuta
nothing in particular atm just something flexible to be ready for decentralised channels and make graph queries on + possibly embeddings in the future
1 reply
0 recast
2 reactions

Samuel pfp
Samuel
@samuellhuber
@whyshock get in here. Do you think indexer will enable this? Then we build it @ericjuta do you think having to index everything (shuttle has no filters) is too much?
1 reply
0 recast
0 reaction