Thursday, May 24, 2007

Human factors

I said in my previous post that my #1 priority resulted in choosing Banyan VINES as our network operating system. In rolling out the Banyan VINES network over the next year or so I learned a lot about e-mail, directories and directory synchronization. This on the job experience proved to be invaluable in understanding the complexities and human-factors side of these technological problems...

  • Our office in Kenya would regularly lose its connection to the X.25 network. It happened the same day every month. Turns out the technician for the Kenyan telephone company would disconnect us, go and sit in the bar across the street and wait for one of us to come over and buy him a beer and slip him a little something.

  • I sat outside the Indian communications agency's managing director's office for an afternoon because no one in his office would believe that their modems were all configured incorrectly - across the country! Turns out they were set to 7-bit operation and not 8-bit. No one seemed to care they couldn't transfer a binary file over their network.

  • One of the operations staff was reading the newspaper on a Sunday afternoon in the computer room. His feet slipped off the console, landed on a server that was on wheels, the server skated over to the open tile on the raised floor and fell a couple of feet. It continued to run. He picked the server up, replaced the tile and went back to work. A few days later we had a complete server failure.

  • We had just installed fiber optic connections to our servers in Singapore. A cleaner is mopping the floors and accidentally pushes a server back towards the wall and snaps off the fiber connections. Remote diagnostics of that one from Ottawa was a lot of fun. No one thought to get the cleaner on the call.

  • I'm standing on an overlook on Highway 1 on the coast of California and my cell phone rings. One of the managers says our new VINES server isn't working - we had just finished putting our first "VINES on Unix" server in production. He says "We're going to turn it off and turn it back on". I scream "Noooooooooooooooo" into the phone as I hear him click the switch. It's not a good idea to turn the power off to a Unix server - it takes the team three days to get it restarted.
I'm sure we all have stories like these. They're funny when you think about them now but they've left scars. I still won't buy a server that's on wheels...

No comments: