A key hub metric we are laser focussed on improving is sync health. 

We have about 800-1000 running hubs at any given time. Sync health is the % of these hubs that are fully caught up with the network. 

Our goal is to get to 99% sync health, and @wazzymandias.eth is leading this effort.

Technowatermelon. Elder Millenial. Building Farcaster. 

nf.td/varun

Our focus is now on debugging the last 5%. 

A likely culprit is sync with Ethereum. It's possible that some hubs are running into rate limits when talking to RPC providers. 

We're tracking this down and trying to move the sync rate up as quickly as possible.

Hubs fall behind for a variety of reasons - slow hardware, machine restarts, rate limits when talking to ethereum nodes etc. 

Last week we shipped an improvement to sync that let hubs catch up much more quickly. This improved sync health from ~85% to 95%.

I believe we do this daily cc @adityapk to confirm. 

With the recent DAU spike, I think 2-3M messages per day is expected.

How often are snapshots taken? Recent hub outage spun up at 97% about 3M messages behind

My crude golang hub experiment might poison your numbers.

You probably can just kick start a snapshot sync right after the snapshot updates and it’ll probably kick these servers back up to 98% at least - as a step 1

are y'all sharing the ongoing costs of running a hub, i.e. daily/monthly storage footprint?