What Software Crisis?

The Crisis
I hear the term software crisis from time to time, and it has always confused me. The software industry continues to produce and improve software at an astronomical rate. Users continue to consume it at the same rate. In addition, for all intents and purposes, this appears to be a totally sustainable trend. So where is the crisis?

As I understand it, those that use the term refer to the common occurrence of software projects not meeting time, cost or quality estimates. Certainly excessive time, cost or a lack of quality is a concern, but estimates and reality are not equivalent. However, other types of projects do not have the same difficulty in making accurate predications, so something must be wrong. Right?

I don’t think so. There is an alternative explanation to why software project estimates are so often wrong. It is not that the software industry is worse, but that it is better. No, this is not a tie-in to the worse-is-better philosophy. This relates to a fourth element in time, cost and quality matrix. It is called efficiency. In these terms, it is the rate at which cost and time are converted into quality.

This quality, efficiency, is the root cause for all of the unpredictability in software project estimates. The reason, is that efficiency among software development, varies greatly. This may initially seem a cause for concern, based upon the assumption that if some projects have high efficiency, and others do not, there must be something wrong with those that do not. The problem is that the differences in efficiency are not due to flaws in software project management, but due to an inherent difference in the value of quality.

For example, the difference between having the worst or best programmers has far more effect on efficiency than the difference between the worst and best in almost any other profession. Is there a much greater gap between the intelligence of the best and worst among programmers and other professions? I don’t think so. There may be a greater gap today than 10 or 20 years ago, but the effects of the “software crisis” was alive back then as well. In addition, 10 or 20 years ago, it would be easy to say that the gap between intelligence of the best and worst was less than most professions, since only very smart people were involved.

The Multiplier
No, the difference is not in the size of the gap, but in the importance of it. For example, if you took two people from another profession, where one person was 25% more efficient than the other, and retrained both as programmers, all other things being equal, I think you would find that the gap in efficiency would be far greater in the when acting as programmers. It might be something more alike to a 200% or 500% difference.

The actual multiplier also varies far more from task to task, then other industries. Coding a data entry form might have a relatively low multiplier when compared to debugging a resource leak in a distributed application.

Knowing this, we could try to decrease the variance by only using the “best”. But this is obviously impractical, since that is a limited resource, and the demand for software is far greater than is producible from that resource.

So how do we cope? My advice would be “just live with it”. Focus on raising the average efficiency, rather than attempting to standardize it. The only way to standardize efficiency is to bring the best down, and the effect of this would have a dramatic negative effect on average efficiency, and overall work output. This is not a suggestion to abandon all attempts to measure efficiency and predict results. These practices are still as valid as in any profession, perhaps even more so. The point is that we must accept that as long as the value multiplier remains high (and we want it to be) we will never be as successful at it as other industries.

Final Thoughts
One insight that we can gain from this realization is that for the time being, our focus should be upon measurement and prediction, rather than control. Control only becomes effective when you have sufficient quality information. If you exercise too much control without quality information, you are likely to do more harm than good. The process generally works as is, so until you know how to fix it.. don’t.

Another insight is that we should focus more upon developing efficiency through common sense methods. We should invest more in tools, benefits, and assisting developers, rather than cutting costs and restricting freedom.