Content pfp
Content
@
0 reply
0 recast
0 reaction

Varun Srinivasan pfp
Varun Srinivasan
@v
Please upgrade your hubs to v1.13.4 We've fixed the issue with delayed messages and plan to min version this release. Hubs that are not upgraded by 9am PT tomorrow will disconnect from the network.
14 replies
35 recasts
177 reactions

Varun Srinivasan pfp
Varun Srinivasan
@v
Sync has been degraded for a few days, and most hubs are missing some messages. The network will automatically heal over the next week as hubs sync with each other and fill the gaps. We're also working with neynar and airstack to trigger faster syncs with them to speed this up. If you are running a hub and affected by this, DM e.
1 reply
1 recast
44 reactions

Varun Srinivasan pfp
Varun Srinivasan
@v
The root cause was an expensive iteration. When a hub gets a message from a peer, it iterates over its entire peer store to track some stats. As the number of peers grew the iteration took longer and the number of iterations needed kept increasing. Eventually, the iteration couldn't be completed before the next one was triggered. This caused hubs to start crashing slowly. The fix was effectively a 1-line change that moved the iteration out of the critical path.
11 replies
3 recasts
38 reactions

Varun Srinivasan pfp
Varun Srinivasan
@v
Huge shoutout to everyone who worked on this @sanjay @sds @wazzymandias.eth @cassie
3 replies
0 recast
32 reactions

Hector pfp
Hector
@noctis
How many peers do you think until the previous implementation started lagging ?
1 reply
0 recast
0 reaction

Jacob pfp
Jacob
@jrf
must feel good
0 reply
0 recast
0 reaction

Tim Mustafin pfp
Tim Mustafin
@mustafin.eth
Thanks for sharing the root cause. Actually, looking at the metrics post-factum, it makes perfect sense there was some software-level limitation (CPU & Network saturation look quite flat)
0 reply
0 recast
0 reaction

BradGao pfp
BradGao
@bradgao
Per as I know, iterating ALL peers is not a good idea in a peer-to-peer protocol. The iteration count should have a max value, no matter how many peers existing in the network
0 reply
0 recast
1 reaction

wake pfp
wake
@wake.eth
well done
0 reply
0 recast
1 reaction

K pfp
K
@kijijij
If you do not need 'exact' no of peers you can let it run async. `statsd().gauge("peer_store_count", this.gossipNode.peerStoreCount());` * No "await" so it would be running in async * Releases current thread to work on other async tasks * You can still get peer count without obstructing main flow Assuming * Does not run every ms or second, this operation runs hourly / on peer joining, Otherwise a running count will be a good idea * Number of peers are not very important * resolves to accurate value
0 reply
0 recast
0 reaction

joshisdead.eth pfp
joshisdead.eth
@joshisdead.eth
Thanks for the update, V! And thanks to everyone who helped 🀩πŸ₯°
0 reply
0 recast
0 reaction

Penguin pfp
Penguin
@sleepypenguin.eth
Thank you and appreciate the work @v!! Question though on delayed tips. If it was delayed and just synced now, will it be deducted on today’s allowance rather that the day it was posted?
0 reply
0 recast
0 reaction

Todd pfp
Todd
@tthebc01
πŸŽ‰
0 reply
0 recast
0 reaction

SchmRdty.eth 🎩 pfp
SchmRdty.eth 🎩
@schmidtiest.eth
πŸ€” I need more… something.
0 reply
0 recast
0 reaction