Sanjay

@sanjay

229 Following

2981 Followers

Sanjay pfp

Just released Snapchain v0.4.0 to support this change. It will take effect on July 16th 17:00 UTC. Please upgrade your nodes before then. As part of this, we're also extending the existing storage units expiry by one year (no user action required). Snapchain has enough headroom that we don't need to expire yet.

2 replies

1 recast

9 reactions

Sanjay pfp

Released Hubble 0.19.3 which pushes version expiry to May 15. This will be the last hubble update. If you are still running hubs, please switch to snapchain instead of upgrading. Please reach out if you run into any issues with the migration.

1 reply

2 recasts

25 reactions

Sanjay pfp

Thinking about the upgrade process for Snapchain and looking for feedback. There are going to be two kinds of changes: a) Soft forks (bug fixes, minor improvements and non-consensus breaking changes) b). Hard forks (breaking changes, typically FIP implementations or other major changes that affects consensus)

2 replies

3 recasts

33 reactions

Sanjay pfp

Warpcast is currently being powered solely by snapchain. If you're publishing only to hubs, make sure you're also publishing to snapchain. Let us know if you run into issues with missing messages or with migrating to snapchain. https://snapchain.farcaster.xyz/guides/migrating-to-snapchain

6 replies

5 recasts

38 reactions

Sanjay pfp

Both @neynar and @warpcast are reading/writing to Snapchain now. If you’re using Hubble, you should migrate to use snapchain exclusively and let us know if you run into any issues. https://docs.farcaster.xyz/hubble/migrating https://warpcast.notion.site/Snapchain-Mainnet-Public-1b96a6c0c101809493cfda3998a65c7a?pvs=4

15 replies

32 recasts

182 reactions

Sanjay pfp

Probably the most challenging and fun project I’ve ever worked on. Incredible work by @dynemyte @suurkivi and @cassie. Special shoutout to the Informal Systems team for building Malachite, a rock solid rust Tendermint library which powers snapchain.

8 replies

19 recasts

112 reactions

Sanjay pfp

We've been so focused on snapchain, I forgot to do the Hub protocol release for Nov 27. Just released Hubble 1.17. Please upgrade before current version expires at Dec 11 midnight UTC. Apologies for the short notice!

0 reply

11 recasts

77 reactions

Sanjay pfp

Considering buying a house, and came across this very cool and very legal covenant (from 1946) for the land in the disclosures. Can't say I wasn't tempted to buy just to stick it to them.

7 replies

4 recasts

28 reactions

Sanjay pfp

I was originally leaning towards account ordering. But happy with where we ended up. The biggest issue with blockchains is managing block state growth. Thanks to @cassie for inspiring the solution on how to handle this in snapchains. Also, special thanks to @vrypan.eth for the App Ordering idea.

3 replies

96 recasts

155 reactions

Sanjay pfp

If you enjoy coming up with novel distributed systems algorithms, we have just the challenge for you.

3 replies

16 recasts

98 reactions

Sanjay pfp

Please upgrade Hubble to 1.14.4 which implements storage changes defined in https://github.com/farcasterxyz/protocol/discussions/191 We'll min version on Monday at the latest so all hubs will be ready for Aug 28 when storage units were originally scheduled to expire. If you are manually calculating storage, you'll need to update your logic. You may find the new helper functions implemented in the hub-nodejs package useful https://github.com/farcasterxyz/hub-monorepo/blob/main/packages/core/src/limits.ts

1 reply

3 recasts

19 reactions

Sanjay pfp

Since this day, we're at 50x the message count, 180x the peer count, ~40x the db size. Perf metrics are harder to compare since it was during an incident but currently p95 merge latency is ~30ms and p95 gossip delay is <1s (vs 2000ms and ~2.5hrs during the incident)

12 replies

39 recasts

136 reactions

Sanjay pfp

Hub message disruption today was caused by our hub losing gossip connectivity to all other hubs (unclear why exactly due to a logging bug). It was unable to regain connection due a bad interaction with a libp2p upgrade. Released 1.14.2 with a fix.

0 reply

1 recast

13 reactions

Sanjay pfp

Hubble 1.14 is out. It includes a bunch of fixes around follows (consistency issues and large compaction events breaking event streams). If you are using shuttle, make sure you’re on the latest version before upgrading hubs, there’s a breaking api change for events.

4 replies

23 recasts

130 reactions

Sanjay pfp

Message processing was broken today because some events exceeded grpc client default size. If you're using shuttle, please upgrade to 0.4.1 to get the fix. If you are constructing hub clients manually and listening to events, make sure to pass in the following param

3 replies

9 recasts

50 reactions

Sanjay pfp

This one was tricky to track down. There were a lot of dead ends. Special thanks to @wazzymandias.eth for basically fixing it last night and not telling the rest of us 😂 @cassie and @sds for some deep libp2p and tcp tuning magic, which we thankfully didn't need. And finally to my good friend Claude, who pointed me to the `node --prof` command which is able to profile worker threads, would've been much more difficult to narrow down the root cause without it.

13 replies

12 recasts

46 reactions

Sanjay pfp

Released Hubble 1.13.2 with a crash fix (thanks @cassie). Please upgrade for improved stability. We're also planning to min version to 1.13.1 early next week so all hubs support long casts. We're noticing some sync issues due to older hubs that don't support it.

10 replies

4 recasts

39 reactions

Sanjay pfp

Reminder that the replicator is deprecated by shuttle. We’re going to remove the replicator from the hub codebase by end of next week to avoid any confusion. If you’re still using the replicator, please migrate to the shuttle package. Let me know if you have any questions around migration.

17 replies

115 recasts

767 reactions

Sanjay pfp

We’re seeing message processing delays again. The team is working on scaling our systems to be able to handle it.

0 reply

2 recasts

14 reactions

Sanjay pfp

alpha version of the package is out https://github.com/farcasterxyz/hub-monorepo/tree/main/packages/hub-shuttle

2 replies

3 recasts

21 reactions