[ URLs have been changed to protect the identities of the hilarious ]
As any sysadmin knows, it’s favorable to stay a few steps removed from the unwashed masses we call “users”. Escalation procedures exist for a reason, and I’ll be damned if I am going to talk to a customer every time they forget a password. Sometimes, however, a problem comes along requiring the unique insight that only a SA has.
One such problem was presented to me just the other day in the form of a conversation in our internal chat room. It went something like this:
Alice: Hey, I have a customer on the line who is saying she can’t change her website. Any ideas?
Bob: Well did you confirm the URL?
Alice: Yep, same problem on our end.
Charlie: Have you logged into our server to make sure everything is stable?
Alice: Uh huh. Running just fine.
[ snip 20 minutes of throwing ideas around and none of them working ]
Zachary: Check the flux capacitor!
At this point I decided I should step in before someone got hurt. I asked for a ticket number so I could check out the issue myself, and here’s what was recorded:
“Ticket #13376047 – User unable to edit her web page at http://www.example.com/squigglykim ”
Not much to go off, but I like a challenge. I look up which of our servers is hosting her site and SSH in. Sure enough, everything is working fine. Other users are FTPing in without any problems. Web pages are being served and life is great. So what’s the issue?
Maybe it lies deeper. It’s a VM so maybe there is a limit on the number of connections you ca- No, that’s all good. Possibly an issue with permissi- nope, 755 like all the others. Ah ha! Certainly the user tripped our IPS and got blacklis- all clean.
At this point I started doing what I always do when confronted with a problem I can’t fix: Rubber Duck Debugging. Except instead of a rubber duck, I have a plush goat. Don’t judge me. So I start talking to the goat:
“The problem is that a user cannot edit her web page. Her URL is http://www.example.com/squigglykim. I’ve verified that the site exists and works in a browser. Her FTP credentials are right. She can log in as kim with her password. The user kim is configured the same as any other user. Kim just can’t edit her squigglykim pa-”
If ever there were a moment in all human history – recorded and not – where a man needed to hit his head upon a wall, this was it.