Content
@
0 reply
0 recast
0 reaction
Varun Srinivasan
@v
Please upgrade your hubs to v1.13.4 We've fixed the issue with delayed messages and plan to min version this release. Hubs that are not upgraded by 9am PT tomorrow will disconnect from the network.
14 replies
35 recasts
177 reactions
Varun Srinivasan
@v
Sync has been degraded for a few days, and most hubs are missing some messages. The network will automatically heal over the next week as hubs sync with each other and fill the gaps. We're also working with neynar and airstack to trigger faster syncs with them to speed this up. If you are running a hub and affected by this, DM e.
1 reply
1 recast
44 reactions
Varun Srinivasan
@v
The root cause was an expensive iteration. When a hub gets a message from a peer, it iterates over its entire peer store to track some stats. As the number of peers grew the iteration took longer and the number of iterations needed kept increasing. Eventually, the iteration couldn't be completed before the next one was triggered. This caused hubs to start crashing slowly. The fix was effectively a 1-line change that moved the iteration out of the critical path.
11 replies
3 recasts
38 reactions
Varun Srinivasan
@v
Huge shoutout to everyone who worked on this @sanjay @sds @wazzymandias.eth @cassie
3 replies
0 recast
32 reactions
downshift
@downshift.eth
amazing work you all! 👏
0 reply
0 recast
2 reactions
joshisdead.eth
@joshisdead.eth
Thank you guys!
0 reply
0 recast
0 reaction
baubergo
@baubergo-
Thanks everyone for fixing our purple app 💜
0 reply
0 recast
0 reaction
