All the Perl that's Practical to Extract and Report

Back to the innovation argument

ziggy on 2003-11-27T17:16:17

I'm working on an essay refuting the classic Microsoft argument that «open source is a huge black hole of productivity because thousands of developers are obsessed with reimplementing a 30-year old operating system». One of the issues I need to address is the assertion that open source can only copy, it can't innovate.

This argument has been on my mind for many months now. One unsatisfying response deals with what you mean by "innovation" -- true innovation is a rare occurrance, and proprietary software has very little innovation, so the apparent lack of innovation in open source is only because the necessary pre-conditions have been around for a little more than a decade.

I'm still trying to find areas where open source software preceded proprietary software. Here's what I've come up with:

  • graphical web browsers: NCSA Mosaic was open source, and preceeded all(?) proprietary web browsers
  • web servers: NCSA HTTPD was the first cross-platform web server
  • anti-spam engines: the first anti-spam engines (spambayes? spamassassin?) were open source
  • infrastructure: bind and sendmail. 'nuff said
  • secure services?: are qmail and djbdns the first instances of network services written using reduced privileges?
  • networking: Berkeley Sockets - used everywhere because it's open source and liberally licensed
  • remote GUIs: X supported separation of the display and the app years before VNC, Windows Terminal Server, Apple's Remote Desktop, and the like
  • intrusion detection: satan, snort, netstumbler, sniff and the like are open source. similar tools tend to be written as open source instead of proprietary software.
Then there's this small list of up-and-coming projects:
  • Manditory Access Control: SELinux and TrustedBSD are the two primary platforms for NSA research on MAC. Why? Because they're open source, and their research will be available to all to examine and commoditize. (Anyone know where Trusted Solaris falls into this mix?)
  • Parrot: is it just another VM? Or is it the first extensible VM?
Are these lists accurate? Is anything obvious missing? There are a few more things on my list of maybes: is CUPS just another print spooler, or does it bring something unique into the mix?

Internet

bart on 2003-11-28T10:03:21

Actually, the whole of Internet is (originally) based on open source tools. All of it.

Re:Internet

ziggy on 2003-11-28T17:26:01

I agree, but that position is arguable.

Opponents of open source like to cite that most things we like to consider open source are actually the fruits of government funded research programs. Take away UCBerkeley, DARPA, and it's uncertain whether projects like BSD, Berkeley Sockets, bind, sendmail and the whole rest of the internet could have started. That is to say, it's unclear whether an amorphous group of hackers (like, say, Linux kernel developers) could develop something new and innovative in an open source project without outside investment on what they were building. Sure, Linux can acquire really interesting features, like scalability from embeddable devices to mainframes to 64-way SMP, but it's unclear whether that kind of development could happen without the likes of IBM pitching in. (It's also unclear whether the basis of that question is even relevant, but that's a different story.)

That's why projects like snort, satan, stumbler, djbdns, qmail, spamassassin are slightly more interesting. They are open source and exhibit innovative characteristics and yet are not directly funded.

define innovative

brev on 2003-11-29T20:30:43

The common sense definition: if you thought of it, you might cry out 'eureka'! It's defined relative to the current state of the art.

Whereas vendors, MS among them, often use the term to mean product innovation. Well-understood ideas, packaged and integrated to match customer needs.

Anyway, with eureka innovation, OSS comes up short, but so does almost everything else. Brilliant new ideas are rarely productized soon enough to count as eureka-innovative.

But -- examples of eureka-innovation in OSS:
rsync
ReiserFS
ZODB and Zope
AceDB
Debian package management
qmail (almost every aspect is sui generis)
Perl scalar datatype; context coercion
python indenting

Product innovation in OSS: hard to think of examples because unix tends to be designed as a toolkit, with many-layered integration. The quantity of fundamental innovations (innovatons?) in any one thing is quite low.

The innovation argument may be irreconcilable. Businesses like the idea of innovation because they feel it's necessary to stay competitive.

But they want to *purchase* innovation. Whether this can even be done is debatable. But MS allows them to believe that they can.

OSS gives the end user the ability to innovate, but most organizations don't feel they are capable of that (and they may be right).

Re:define innovative

ziggy on 2003-11-29T21:29:36

Thank you very much. I've been thinking about this on and off since March. Most people who I've raised the issue with see the difference between "big" innovation and "little" innovation. No one has made the distinction as clearly: eureka-innovation vs. product-innovation.

Thanks!

Product innovation in OSS: hard to think of examples because unix tends to be designed as a toolkit, with many-layered integration. The quantity of fundamental innovations (innovatons?) in any one thing is quite low.
Hm. Good point. I never thought of Unix as something so close to perfection that it doesn't need leaps and bounds of innovation to stay relevant. It also accomodates those new ideas as they appear.

That also helps to explain part of why Microsoft is so hell-bent on Avalon heralding the next wave of computing. Start with a house of cards, and the need for lots of breakthrough innovation is quite pronounced.

Re:define innovative

brev on 2003-11-30T23:26:47

I never thought of Unix as something so close to perfection that it doesn't need leaps and bounds of innovation to stay relevant.

Erk, did it sound like I was saying Unix was close to perfection? Unix sucks, but it does seem to be more evolvable.

Start with a house of cards, and the need for lots of breakthrough innovation is quite pronounced.

Heh. But you know, if you were Bill Gates, frantically changing everything every three years has been confirmed as a profitable strategy. And by this time everyone in MS-land must think that's what innovation looks like.

Re:define innovative

ziggy on 2003-12-01T03:17:35

Erk, did it sound like I was saying Unix was close to perfection? Unix sucks, but it does seem to be more evolvable.
Whoops. Didn't mean to put words in your mouth.

Nevertheless, Unix is one of the best OS frameworks we've seen in 40 years. It's got a huge number of rough spots, but it's hard to point to something that's 10x better, or identify an area that needs a tenfold improvement in the basic Unix architecture. That's got to amount to something.

Um..

pdcawley on 2003-11-30T20:02:18

I believe the CERN HTTPD was cross platform first, but it may depend on what you mean by cross platform.