horsefacts
@horsefacts.eth
Quick postmortem: looks like this was an outbound DNS query cache miss to our service provider discovery service. We read through to the backup service provider catalog cluster, but high latency to our Australian datacenter meant the request timed out and landed in the service provider catalog cluster request retry queue. Once this was processed we were able to resolve the current PfpHaloMutexService leader and request a write lock. Worked fine from there.
10 replies
1 recast
21 reactions
Cassie Heart
@cassie
This all could have been avoided if we had a PfpHaloMutexFactoryFactoryFactory
0 reply
0 recast
9 reactions
Jört User
@jort-user.eth
Amazing work. I would have just banned Australians from the app.
0 reply
0 recast
3 reactions
christopher
@christopher
It’s always DNS?
0 reply
0 recast
2 reactions
L3MBDA
@l3mbda
next time maybe, give Australia a heads-up before the timeout party starts
0 reply
0 recast
1 reaction
Harris
@harris-
This is cool but also really funny considering jort's stream of consciousness posting about it while it was probably being debugged Horse: "DNS cache backup cluster" Jört: "IM CORTANA IM LITERALLY MASTER CHIEF"
0 reply
0 recast
1 reaction
Steve
@stevedylandev.eth
Cloudflare postmortem ain’t got nothing on this
0 reply
0 recast
1 reaction
divya
@divyav.eth
got mine in 5 mins. thank you :)
0 reply
0 recast
1 reaction
juliettemeon 🎩
@juliettemeon.eth
I got, thank u ser 💜
0 reply
0 recast
0 reaction
Jört User
@jort-user.eth
In all seriousness though, very impressive!
0 reply
0 recast
0 reaction
tricil
@tricil
halofacts.eth
0 reply
0 recast
1 reaction
