Content
@
0 reply
0 recast
2 reactions
jtgi
@jtgi
fave oncall-ish story: me and @martinamps were sitting beside each tailing logs at a packed bill graham auditorium in san francisco. It was Twilio's big annual conf and our product was *the* keynote. Our vp product was halfway through the live demo when I turned from the logs to martin: "oh fuck."
1 reply
0 recast
5 reactions
jtgi
@jtgi
For some context, this product tied together all of twilio's major business units (voice, video, messaging) and we'd been putting in crazy hours all month. The first time we could properly test it end to end was 8 hours before it launched. And so we didn't sleep. I was on 2 red bulls, too many coffees and delirious.
1 reply
0 recast
0 reaction
jtgi
@jtgi
We started seeing exceptions come through the logs. You prob know the kind, pages of stack trace. Connections were dropping and the live demo was degrading in real time. I can only describe the feeling as absolute defeat and utter helplessness.
1 reply
0 recast
0 reaction
jtgi
@jtgi
It was at that moment we remembered, "the epoch!". At around 3am that morning, I had the bright idea to set an epoch to automatically transition the system from beta to public. I set a buffer of ~1 hour past the keynote. But the keynote ran late, it kicked in mid demo and set off a chain of split brain behaviour.
2 replies
0 recast
0 reaction
martin
@martinamps
Btw props… the epoch was actually a great idea and we may not have been ready in the AM without it
0 reply
0 recast
0 reaction
