Nar pfp
Nar
@nar
Feel like this is super common for me 1. Oh look a shiny API 2. But it’s rate limited 3. Let me just write a script to periodically query the API and persist the data to a DB I can freely query from Is there a better pattern? Otherwise doing something like this for ERC-20 and NFT indexers this week
11 replies
0 recast
2 reactions
Stas pfp
Stas
@stas
Hmm, not sure what the API does, but if it's a critical data source, I'd rather look into @typedarray.eth Ponder or deploy something like sugar if it's your protocol https://github.com/velodrome-finance/sugar
1 reply
0 recast
0 reaction
Shashank pfp
Shashank
@0xshash
For static data, this pattern could work. But for time sensitive data you'd have to do additional work to keep the data fresh and ensure you're not serving stale data And say the API adds a new feature, managing backfills etc can get nasty with low rate limits
1 reply
0 recast
0 reaction
Jem pfp
Jem
@jem
I effectively did this with Graph Protocol subgraphs. Its GraphQL API is slow, rate-limited and has paging limitations, which would prevent querying it for large amounts of data. So now we have a script that figures out the schema, pulls new data and persists to BigQuery. 🤓 Costs a few $/month.
1 reply
0 recast
0 reaction
Elie pfp
Elie
@elie
I will index the indexer 🤓 If you pay you have better limits
2 replies
0 recast
0 reaction
Joe Petrich 🟪 pfp
Joe Petrich 🟪
@jpetrich
You could outsource the caching to something else, but no, I don't think there's a better pattern.
0 reply
0 recast
0 reaction
Gabriel Ayuso pfp
Gabriel Ayuso
@gabrielayuso.eth
I do this. Just make sure you follow the API's terms of use.
0 reply
0 recast
0 reaction
Liang @ degencast.wtf 🎩 pfp
Liang @ degencast.wtf 🎩
@degencast.eth
we could p2p the data on ipfs. but how about the streams?
0 reply
0 recast
0 reaction
Liang @ degencast.wtf 🎩 pfp
Liang @ degencast.wtf 🎩
@degencast.eth
but how do you pay the devs to make this sustainable?
0 reply
0 recast
0 reaction
Liang @ degencast.wtf 🎩 pfp
Liang @ degencast.wtf 🎩
@degencast.eth
open & structured data updated with live stream seems to be a better pattern
0 reply
0 recast
0 reaction
Andrew pfp
Andrew
@andrewgoldsky
Check out Goldsky, they have a product called Mirror that directly streams onchain data / blocks / events into your existing postgres / rockset / elasticsearch db. No more infinite API calls, just sync the data directly and don't worry about maintenance. https://goldsky.com/
0 reply
0 recast
1 reaction
Nickolas Tazes | Watchen.xyz pfp
Nickolas Tazes | Watchen.xyz
@tazes
I’d love to find NFT smart contract deployer wallets. If the contract has a lot of transactions, most APIs hit rate limits, and it’s super annoying
0 reply
0 recast
0 reaction