fc-updates

we're looking into an issue where some casts are being delayed a minute or two between apps. thanks @woj.eth for reporting. 

it's probably a hub <> hub sync issue, and won't show up on status.warpcast.com which tracks hub<>warpcast  sync. 

if anyone has examples of delayed casts, please reply to this thread.

we had a few more issues through yesterday that may have caused delays. 

this morning we've landed some significant improvements. there's still work to be done, and this is the team's top priority. 

if you see any instances of delayed or missing casts, please let us know.

Technowatermelon. Elder Millenial. Building Farcaster. 

nf.td/varun

if you're interested in the technical details we had two overlapping issues: 

1. our hubs were restarting frequently for unknown reasons. this was causing 1-2 minute delays as they came back up. 

2. warpcast backend saw increased load and a mini-outage for a few minutes last night. team found and fixes the issues soon after.

hmmm any leads on why hubs were failing to start? could be similar root cause for our hubs failing back in the summer

👨‍💻 /uno and /outside, @unofficial

It appears that some sequence of rpc calls were making started causing them to OOM a few days ago. 

They restarted fine, it just happened very often so message queues backed up

ahh yes that might be my issue. we run exactly 24gb available, so when it upgrades it runs OOM to resync up, but restarting flushes temp and gives it enough breathing room on next sync