Content
@
0 reply
0 recast
2 reactions
jtgi
@jtgi
Welcoming another friend @martinamps. We worked together @twilio, the best dude and one of the most talented engineers I've ever worked with. Welcome. And got a favorite oncall story?
1 reply
0 recast
7 reactions
martin
@martinamps
just trying to be as half a good a dude as you my man! hard to pick! one top oncall story might be a time when like 10% of calls powered by twimlbin were experiencing several seconds of post dial delay (dead air after answering) and nobody could figure out why…
1 reply
0 recast
1 reaction
martin
@martinamps
I got summoned in the early hours as an SME, very confused by the symptoms i hopped on a box and ran strace and kept finding open() calls hanging to w3.org … turns out the <?xml schema=“https://w3.org/…..xsd”> was being called by the xml validator and we’d been DDoSing them for years…
1 reply
0 recast
0 reaction
martin
@martinamps
Sure enough some googling revealed a mailing list discussion where somebody had the (actually!) bright idea to sleep(N) on some x% of requests initially to try get some attention. It worked!
1 reply
0 recast
0 reaction
jtgi
@jtgi
wild. who's attention did you get though, did w3.org respond in some way?
1 reply
0 recast
0 reaction
martin
@martinamps
no we just set a flag to stop calling that file for no reason 😀 what’s your fave?
0 reply
0 recast
0 reaction
