Content
@
0 reply
0 recast
0 reaction
Joe Petrich 🟪
@jpetrich
An engineer on my team has been working on a migration of some indexed blockchain data to postgres and took our queries from an initial average latency of almost 2 seconds to under 100ms. The process to get there was fascinating, imo. Would anyone else be interested in a writeup of the optimization techniques he used?
11 replies
2 recasts
28 reactions
Royal
@royalaid.eth
I am super curious about it and how it compares to the tech that @notnotstorm is using with Cryo
1 reply
0 recast
2 reactions
Joe Petrich 🟪
@jpetrich
What is Cryo? Have a link?
1 reply
0 recast
1 reaction
Royal
@royalaid.eth
https://github.com/paradigmxyz/cryo
1 reply
0 recast
2 reactions
Joe Petrich 🟪
@jpetrich
Ok interesting, I don't think it could be more performant than what I've written since it uses the same RPC calls. What we were optimizing was our queries of that data after we've indexed it in Postgres. I don't think there's a way, if using RPC, to improve much on the initial query latency.
3 replies
0 recast
1 reaction
Eddy Lazzarin ðŸŸ
@eddy
The reason why Shovel https://indexsupply.com/shovel/ is so fast is that it skips the RPC stage altogether.
2 replies
0 recast
1 reaction
Joe Petrich 🟪
@jpetrich
Shovel does look incredible, but it also looks like it uses eth_getLogs and eth_getBlockByNumber from whatever node URL you provide. There's some discussion in their GitHub of using a local node for low latency and it's got it's own jrpc client implementation which is probably more efficient than geth's?
1 reply
0 recast
3 reactions
Joe Petrich 🟪
@jpetrich
@indexsupply - just noticed you're here 🙂 - E2PG seems to be able to use some hybrid of RPC or a local node or something else custom you wrote? I'm in the middle of diving into your GitHub but very interested in learning more.
1 reply
0 recast
2 reactions
