Wednesday, July 12, 2006

Reality returns with a thud

It's definitely one of those days. I got a 4:47am wakeup call from the station: it seems that both of the Dunamis playout servers had stopped working overnight. (Dunamis is a system I designed to automatically generate digital graphics over program, squeezing the program video back if necessary. If you watch WSYR-TV and see school closings, tickers during the news or time and temperature, you're seeing Dunamis. It's a combination of generic and custom hardware, driven by enough custom software to fill several binders.)

Turns out that these servers did an automatic Windows Update at 3am, which forced them to reboot. That shouldn't be a problem; the computers are set to automatically log back onto the network and restart the necessary applications, so at most the system should only be down for a couple of minutes.

Not so today. Our folks at corporate headquarters decided that everyone needed to be reminded that the computers and email are the property of the corporation and that they reserve the right to monitor activity. Fine, not a problem... except that the reminder takes the form of a pop-up window with an OK button that must be clicked before the machine will finish booting up. Worse, the remote access software that lets me fix this kind of problem from home, now doesn't start until this OK button is clicked.

So now we have a number of computers critical to various on-air functions that shut down at the whim of Microsoft, stay down thanks to the whim of some lawyers, and require a special personal visit in the wee hours.

As Steve, a master control operator here twenty years ago, used to say: when the phone rings at three in the morning, it ain't Publishers Clearing House calling to say you've won.

Maybe I can set up automatic call forwarding to one of our lawyers... that ought to get the problem fixed pretty quickly.



Update... the forces of goodness and light have prevailed, and the popup has been banished — at least for now. Apparently the popup took down air-critical equipment at radio and television stations all across the country, and the resulting firestorm convinced the right folks that staying on the air is more important, after all.