It strikes me that the explanation between the notable success in infrastructure and the disappointment in applications may have something to do with the identity of the user. When open source programmers build a search engine component such as Lucene, the users are other developers. They are essentially building for themselves. This means they understand the requirements perfectly and the result is some of the best software available. However, while developers writing database software know what other developers using database software really need and want, this is not the case for applications. Engineers don't instinctively know that aliasing is a crucial feature in OpenOffice. The nuances in understanding how a sales lead should get handled in a CRM system may as well be expressed to the typical engineer as if in a foreign language. Since open source is essentially about doing it yourself with a community of like-minded others, the lack of engineer-business hybrids means we'll never see open source apps achieve the same level of success as their counterparts.

There are one or two successful open source applications, the Firefox browser being perhaps the best known today. The other is Linux, which is a split personality story dominating the data center as the preferred server platform, but nowhere on the desktop.

These exceptions, however, reinforce the general principle about programmers developing software for themselves versus for other people. Engineers looking for an operating system can handle searching for, and loading up, a dozen different device drivers for their laptops (and possibly even writing one). Ordinary users can't, and as long as that remains the case, Linux will never progress much on the desktop. Firefox is not so much of an application as an application environment, with its behavior tightly constrained by the W3C standards. While developers go through considerable anguish debating the merits of IE, Firefox, Opera, Safari, and now Chrome, most ordinary users see simply a window with a website inside it and some controls for back, forward, print, and home. The real functional differences between browsers are more related to their ease of programming web apps, standards compliance, and browser extensions. Tabs alone are not a revolution.

How can we transfer the success of open source infrastructure projects to open source applications? I believe that a role needs to be made for the "non-contributing users." Too often, open source project bulletin boards respond to a "hey, this doesn't work" with a "hey, why don't you fix it." This might be a valid point if the response is to another techie's complaint about UTF-8 support, but in the case of a business user who took the time to point out how the app should work, this form of dialogue is definitely not helpful.

Understanding user requirements is not an easy task in commercial organizations either. Users typically don't think through their requirements with the same depth and logic as developers do. They also have no idea as to what is technologically possible. What this means in practice is that product development is better described as a process of trial and error, where successive approximations to the right functional fit are developed and delivered. This is a model that could work within the open source framework, but has yet to do so. I'm skeptical it ever will.

Open source is fundamentally based upon equality and meritocracy. Everyone gets a say, everyone can change the code, and your status is dependent on the quality and quantity of your contributions. Users, though, complicate the picture. They can't code, only point out what's needed. Since most users only know what they want when they see it, most feedback is a disheartening "this is wrong, that is wrong..." It gets worse when the feature they clamored for last month that was delivered in the early hours of Sunday morning goes unused, and instead now some other feature becomes the "most critical." It then starts to feel like indentured servitude - precisely what most programmers are trying to escape in their day jobs.

There is a successful model of collectivizing user input though, and that's SaaS.

With SaaS, rather like open source, you have not one, but many voices pointing out the numerous ways the software can be improved. Of course, you will hear good and bad ideas, but since we're in a market of ideas, the good ones win out. So over time, a SaaS solution begins to represent a consolidated set of best practices. The SaaS model provides the mechanism to select the best ideas and providers.

This same dynamic acts as a driver that improves software quality, simply because SaaS vendors can't operate for very long with a serious bug in their application. Not one, but hundreds or thousands of customers will be affected; as a SaaS vendor, you have to fix the issue immediately. Otherwise you will put your renewal business in jeopardy.

Because SaaS only operates in a single environment, the need to support a multiplicity of different configurations is absent. That means that engineering is free to focus their time on incorporating new functionality, using the constant stream of customer feedback that a large customer base will bring. In a way, SaaS customers are therefore not just subscribing to a service, they are leveraging the collected, concentrated, and distilled best practices of a whole market.

Of course there is a material difference - money. But then that's perhaps less of a difference than you might think. Open source is increasingly becoming a source of support and customization revenue for many organizations. SaaS is not much different in that regard.

I see open source continuing to dominate infrastructure development, and SaaS making ever bigger strides into the applications space. It's not too much of a stretch to consider the analogy of open source infrastructure development being akin to the open scientific community, and the provision of commercial SaaS applications as being like engineering commercial spin-offs. They can and do live in harmony.