San Francisco WAN Lockout - Pointing Fingers at Everyone Responsible

The San Francisco WAN Lockout incident is already written in the annals of IT history. I followed the development and the comments, and today i stumbled on a text Who is really to blame for the San Fran network lockout?. It does touch important issues, but leaves the white gloves on. So let's remove the gloves and point some fingers:

What was the situation?

  1. Apparently, Mr. Childs was the only person with unrestricted administrative right to manage the network, supposedly because of the incompetence of the other members of the team.
  2. The network is used to transport and manage all kinds of official documentation - including jail bookings and other law enforcement documents, payroll files, and e-mails
  3. He created an authentication scheme where only he had administrative access on the network.
  4. Apparently, the situation in points 1 to 3 was well know to the users and management, and was accepted as such.
  5. Mr. Childs clashed with the new Security Manager on the subject of authentication and control, which led to poor formal review
  6. The poor performance review and other undocumented power struggles led to the dismissal of Terry Childs and his subsequent arrest after he refused to relinquish the administrative passwords

Who's responsible?

  • Terry Childs
    • He played god and isolated all other network engineers from the network - thus preventing them from any chance to learn how to manage the network.
    • He created and to date is enforcing the actual lockout that is the reason for all this ruckus.
  • Terry Childs' direct line manager and the one level above
    • They knew that Terry Childs had absolute control over the network and permitted that - If they were uninformed of the situation, they should be fired for gross incompetence.
    • They did not create conditions for knowledge distribution and reduction of dependency on a single person (Terry Childs could have fallen ill or gotten in a car accident - they still need another engineer).
    • They did not identify that there is a potential superiority problem with Terry Childs. This superiority problem usually manifests in insubordination when the control is taken away from a person.
    • Poor human resource management - if all other network admins were so incompetent that administrative authority couldn't be given to them, why did they hire them?
  • Top management
    • They delayed or avoided implementing a security policy which Terry Childs would have had to obey.
    • They did not create no single point of failure strategy for their personnel.
  • Security Officer
    • He did not identify a risk that the employee may cause serious problems and did not propose alternative workarounds - for instance - hire the equipment manufacturer professional services to regain control and lock-out Terry Childs.
  • Entire line management
    • Poor problem management - Once it became clear that it will be difficult to regain control over the LAN, they fired Mr Childs and called the cops. This only worsened the problem, since the cat is out of the bag, and the problem is still unresolved.

So, someone in the great City of San Francisco should now go around and actually look into the work of all named here, because incident caused by Terry Childs is just the effect, not the root cause!

Talkback and comments are most welcome

4 comments:

Anonymous said...

"1. Apparently, Mr. Childs was the only person with unrestricted administrative right to manage the network, supposedly because of the incompetence of the other members of the team."

Terry Childs was the only person directed and paid by the City of San Francisco to implement, manage and secure the FiberWAN network. Whether or not anyone else was competent is irrelevant to this particular point. The city did not have a single other employee on payroll whose job it was to be a secondary administrator or co-admin of the FiberWAN network.

With no co-admin and in the absence of documented policy stating how someone else could go about requesting administrative access, or who could request it and who had authority to authorize anyone else to have it, Terry Childs was and had been left in the position of being the sole city employee to make such decisions. It was his job.

Anonymous said...

"3. He created an authentication scheme where only he had administrative access on the network."

It was his and solely his job to do so. No other employee of the city was being paid to administer, configure, secure or set passwords on those routers. He was specifically being paid to do so, no one else. It was his job.

Anonymous said...

"5. Mr. Childs clashed with the new Security Manager on the subject of authentication and control, which led to poor formal review"

In early June, Terry Childs sent repeated complaints of incompetency regarding a supervisor (Herb Tong) to that supervisor's superiors. When nothing was done about the informal complaints, Terry Childs filed a formal complaint regarding the supervisor (Herb Tong.) It was several weeks later, on the 20th of June that the reported clash with the new (position created and filled just this year) Security Manager (Jeana Pieralde) occurred.

The Security Manager position was new. Jeana Pieralde was promoted from a prior position within DTIS to the Security Manager position. Jeana Pieralde no longer worked in the same offices with Terry Childs. He returned to those offices on the evening of June 20th, 2008 after normal office hours (which end at 5 P.M.) to find Jeana Pieralde removing a hard drive from someone else's office. She claimed to be performing an unannounced audit.

Jeana Pieralde is the author of a proposed security policy for the city which is still waiting for committee review. That security policy, if accepted, may one day give Jeana Pieralde specific authority to perform audits and perhaps even to have administrative control over city communications networks.

Please dig deeper into this story.

Bozidar Spirovski said...

On Items 1 and 3 (first two comments): Which employee without good reason would accept such a responsibility without asking for a replacement, and after a month or two insisting on another person as his backup. But nevertheless, on those two counts, Childs acted to protect his "turf" once it was assigned to him, and it is higher management fault that they didn't remedy the situation
On item 5 - there will always be incompetent or inexperienced managers, at many companies. This doesn't mean that they don't think highly of themselves, and the reality is that if you step on their toes, they'll fight back, much worse then a competent manager

Designed by Posicionamiento Web