This topic comes up for me not necessarily as a rant, but as I was going through some old notes before tossing them away, and this issue came up. How much damage can an organization do to themselves by hiring discount, unqualified programming resources?

The answer: A hell of a lot.

If you own a small software development shop, do yourself a huge favor and keep reading.

Stupidity cost us about $250,000 over a 9 month period. Another $250,000 was lost for us in other means.

A previous company that I worked at had a habit of hiring low-cost coding staff to handle production quality servers- Not just any production machines… but the very hardware which was responsible for 100% of the revenue being generated for said company. This makes no sense as their client side programmers were generally top-notch, and among the more creative and competent that I have had the pleasure to work with.

Management also had a habit of protecting these junior staff members voraciously- for various reasons, but mainly economic. A single server-side programmer could cost two or three times more in salary to maintain. Also, some in the management felt that this staff member was a ‘worthy project’ that could become a superstar programmer someday. I mean, it worked for Darko Milicic during his stint with the Pistons, didn’t it?

I calculated that my previous employer definitely cost themselves $250,000 in lost cashmoney over a 9 month period for the mishaps they had in retaining ONE very junior server-side engineering resource. This was absolutely staggering to me. Another $250,000 could easily be accounted for in other means. I’ll get to that in a moment.

The other sources for my numbers are the assumptions that:

1. Yes, this programmer was causing problems nearly EVERY SINGLE DAY.
2. Five client/server development resources were utilized, at $20/hr in wage costs each time a bug needed to be fixed.
3. Eight QA staff were utilized, at $15/hr in wage costs to confirm said bug fix.
4. This underqualified developer introduced one bug per day, which took an average of six hours per bug to fix.
5. This individual worked for a period of 9 months, before finally being axed for causing a systemwide outage of all revenue generating products for a significant part of a business day.
6. Not entirely relevant, but said individual produced on average 15 lines of code per day, with the assumption of 21 working days per month. Vacation time is included in this count because this individual only really took one, and worked some weekends as well – so it all balances out.

The numbers I am leaving out are:

1. Costs to management salaries. The reason for this is twofold- one is that I don’t know how many staff members were involved in back-room decisionmaking each time we suffered downtime events. This ranges from one to four. Who knows…
2. Costs to my productivity. I was a systems admin, so it was my job to support the engineering staff. I had the unique experience of being able to read/understand code, but full well knew that I could not hold a candle to the proper engineers. In a nutshell, I could talk shop with them, and help diagnose problems. I was a bit faster than a plain old sysadmin at doing this. I was a good cheerleader at times. That’s about it.
3. Costs of contractors: During most of this 9 month period, we had consulting/contracting staff on board who had to shift gears to help fix problems on a near-regular basis. I simply did not accurately keep track of how often they had to drop whatever they were doing and scramble to help us.
4. The damage caused by spaghetti code. This is utterly incalculable. Each time new server-side programming staff were added they faced an absolute nightmare of a learning curve.
5. The amount of time lost by production server uptime – According to my notes we were down (and by that I mean not being able to process subscription material) for a period of nine total business days during that 9 month period. To this day I still don’t exactly know how many dollars per hour in transactions were being handled by those servers, so I can’t guess there. I won’t even bother to put a price tag on it, but it is the elephant in the room.

If I really do have to hazard a guess at the total cost of damage over 9 months, then I’ll say that stupidity cost us about USD $500,000 total. There is also the untold damage done to the reputation of the company, as subscribers probably grew tiresome of outages and took their business somewhere else.

How much damage can one incompetent programmer do to an organization? A hell of a lot.

In the end it isn’t my intention to throw anyone under the bus for the mistakes that were made. Guilty parties all know who they are, and to some degree or another, they all paid for it already, so there’s no need to flog that dead horse all over again.

The underqualified individual who boasted of general programming expertise which he did not have faced the prospect of never finding work in a proper engineering environment again. Management who felt they were saving money by having this resource around because of cost-effectiveness are facing the prospect of their entire company going down the tubes because the server-side codebase is nearly unmaintainable, among other things. Those of us who stuck around trying to fix the situation paid for it with a few sleepless nights, and the agony of ‘Heisenbug’ code which was nigh near unpossible to debug or maintain effectively.

I learned many things myself. I used to never document very well. Because of the insanity I was forced to deal with, I feel I have come a long way in that regard. I enjoy documenting now. It’s saved my ass a number of times since then.

Over a year after the staff member was let go, the server programmers were still fixing bugs.

Also, a young and promising programmer who was forced to help debug and manage the fix processes on a regular basis learned a hell of a lot from these failures, and it will serve him tremendously in his career as time goes on. This is the type of education you just can’t get in college. Over a year after the staff member was let go, he was still fixing bugs. The thing is, he’s already bailed on this company and isn’t looking back. If there was someone worth focusing on and making into your future team nucleus… this guy was the one.

What’s done is done, and hopefully dear reader, if you’re in management or if you own a small company, you won’t make the same mistakes that they did. I implore you to hire competent people. Don’t cheap out in that regard. It will cost you a whole lot more than you think.

Warning- this may not be really all that funny unless you are a database administrator or a geek with the right knowledge. This rather silly product I am working with stores its data in the backend via a Microsoft SQL Server database. There’s nothing wrong with that… but on further examination of the schema I found a pretty ridiculous jewel of a table setup. Click on the picture to see the bloody details.

For those not technically inclined- this is precisely NOT how you would utilize a relational database server. It’s poor design, and rather an extreme and laughable example.

It seems that the processed food industry has a lot in common between these two countries. Or perhaps its just better to say that corruption and greed have no language barriers.

The 2008 melamine tainted milk scandal in China has a surprising number of similarities with the Peanut Company salmonella scandal that is currently unfolding in the USA.

Both scandals were rooted in greed as the primary cause. SanLu, the Chinese company implicated in the scandal, did not want to lose any more money from increasing dairy production overhead. Likewise, the CEO of The Peanut Corporation, Stewart Parnell was motivated by similar intentions. He did not want to lose out on the value of recalled stock.

Both scandals were also fueled by lack of proper regulatory instruments. The FDA has been exposed as a paper tiger, and in China’s case local administrators urged the continued shipment of contaminated products. They were either paid off or were simply not equipped to do their jobs properly.

The punishment stage of the peanut scandal is not yet underway, so we will see how that goes. It is highly unlikely that Stewart Parnell will be executed by the state like Zhang Yujun and Geng Jinping have been sentenced.

One key difference in how things unfolded is the behavior of the press. While the typical ‘PANIC’ response within US media made more people aware of the situation, the Chinese sought to suppress information as to prevent rioting or other civil unrest.

Currently, Stewart Parnell is being grilled by a congressional subpoena, and more details have yet to be fully unraveled. He has taken the Fifth Amendment to avoid self incrimination on the sheer majority of questions asked. When confronted by Congressman Greg Walden (R), of Oregon, who held up a jar of peanut butter, an asked on a dare if he would eat his own product, Stewart again took the Fifth.

The lesson of the day if you are Stewart- If you’re under oath and a congressman dares you to eat something. YOU EAT IT!

I have found myself having to try and explain the relationship that Taiwan and China share, and usually to other Americans. Of course I am oversimplifying the issue, but the scenario I have come up with is this:

1. Imagine that the Confederacy (CSA) had won the American Civil War.
2. Now imagine that the government of the United States of America had fled to Cuba and set up shop there (kicking out most of the native Cubans), with the new capital being in Havana.
3. Lastly, imagine in this crazy situation that both governments to this day still exist, but have  continued strains on their relationships.

Does this make sense yet? Let’s do some substitutions so it does…

• The Confederacy in this case would be akin to the Communist government of the People’s Republic of China.
• Likewise, the USA government would be Taiwan, who fled their with a ragtag collection of military expats, businessmen, and intellectuals.
• Lastly, the Cuban natives who were largely displaced would be the various indigenous tribes that inhabited Taiwan before the “mainlanders” showed up.

So now this makes a lot more sense right? Of course, I realize that this is not a 100% correct analogy, but it holds up well for ‘elevator speech’ and ‘raging bar conversation’, and now ‘totally random web search’ material. Your mileage may vary, of course.

I guess there isn’t too much to say. Good riddance to quite possibly the worst president in American history.

• The newly hired provost Joe Westphal left his position after roughly three months on the job. Mr. Bob Kerrey, the university president announced that he would assume the role of interim provost until one could be found.
• The full-time  faculty held a meeting and voted no confidence in Mr. Kerrey along with James Murtha, the executive VP.
• A flurry of communications between the two parties, mostly through third party media worsened and escalated the situation to some degree.
• Mr. Kerrey then met the faculty face to face in a meeting that was off limits to students (presumably for the sake of airing out grievances on a straightforward manner and not to involve student issues).
• Some students then occupied the cafeteria and neighboring study facility at the  “GF” building on 65 Fifth Avenue.
• A video was released of a small fracas between one member of TNS security and a student.  This happened around noon on 18 Dec 2008, into the second day of the student occupation.
• Bob Kerrey canceled the student senate meeting which was due to have met in that very building on 8PM that night.
• Students occupied some parts of the ‘main’ building at 55 West 13th Street for a period of several hours starting from roughly 10PM on 18 Dec 2008.
• At some point near midnight, Bob Kerrey left the 55 West 13th street building and was chased down the street by a large group of student protesters to his apartment building a few blocks away.
• The students who occupied the 55 West 13th eventually left early in the morning of the 19th after some of their demands were met. The students holed up in 65 Fifth Avenue are still there.

If you’re a parent and you’re reading this, I just wanted to express some of my opinions about the situation so far. If you’re a parent of a New School student, thank you for putting your trust in the school and its staff.  I’m not a faculty member, and what I do for the university is akin to that of a ‘digital janitor’. There are a lot of dedicated and hard working staff trying to make sure that students get the best services and support that we can provide.

Even though in my job I don’t work with students directly, to see them thriving in their studies is one of the reasons I really love my job and don’t dread coming in to work every day. Working for their benefit is a hell of a lot better than in some previous jobs I have held in the corporate world. However, seeing things unfolding the way they are concerns me quite a bit. We seem to be headed down a darker path. For now since my involvement in this situation is very much peripheral, I’ll only comment on things that I know about.

• To the parents in particular- The 65 Fifth Avenue building is not a run-down rat pit. It is a bit dated in appearance perhaps, but is not at risk of falling apart and reports of that being the case are just overblown. On the flip side, it is due to be demolished and replaced with a bigger building. To me, the student’s cause regarding this building would be more than appropriate if the building was due to be replaced by a smaller one.
• The manner in how Mr. Kerrey was chased down the street was simply not cool. It devalues and cheapens to some degree the initial student cause. Try not to do that in the future, guys and gals.
• Canceling the student senate meeting despite the fact it was being held in the same building that the occupation was happening in was probably a bad idea. The administration let a very good opportunity to engage in impromptu dialog slide on that one.
• At the same time, when students refuse to even talk to Mr. Kerrey unless he resigns. It’s setting an unexpectedly difficult obstacle into the situation and to me is acting in bad faith. It seems that this and the above issue are riding hand-in-hand.
• The security staff are not a bunch of thugs. They are full time staff with a vested interest in good relations from all sides. My job gives me a unique angle in what they do, and some of them are pretty upset at how things are going down. Read: Some staff are sympathetic to some of the student demands but feel that things are not being handled in the best manner, in particular the lack of direct dialogue.
• Take another look at the video of the security guard/student fracas. I think both sides are at fault here. Both sides instigated each other, as some students got some ‘soccer diving for the referee’ in while the security staff should have kept his cool. Emotions running high, being spit on, or otherwise is not a good excuse for what happened.
• Mr. Kerrey is being scapegoated here, and rather unfairly so. If I were a student, I would be rather concerned if he resigned as the school would be not only out a provost, but a president as well. If there is anything to be learned from history, a vacuum at the top of leadership does no good to anyone.
• Lastly, there seems to be some animosity between some of the individual schools. I’ve only been here for little over a year, but have noticed some tension/rivalry between Lang and Parsons students. If this is what it takes to get the student body to act as one community unit, then so be it.

I hope that both sides can cool off and reach a point where they can talk directly to one another without having to resort to press releases or manifesto updates to communicate.

Some Links to Feed On

The New School -  President Bob Kerrey’s Blog

“The New School in Exile” official blog

“The New School in Exile” – a coalition of student and faculty – official site

Gawker.com Article