Chris, I just finished reading the introduction to OpenSources that you, Sam Ockman, and Mark Stone wrote. I'd like to direct your attention to the paragraphs about Lucid. One of the problems I see with the open source movement is its tendency to make up facts and then draw conclusions from them (more on this later). This is especially discouraging when I agree with the conclusions and have been fighting to have those conclusions generally believed for 20 years. The main problem with your story about Lucid is that almost every single "fact" in the story is simply incorrect. I *do* believe there are important lessons to be learned from the encounter Lucid had with free software, but none of them came to light in your treatment.
First, Lucid was not founded "to exploit and develop a streamlined version of the popular programmer's editor Emacs." Being the founder and either President or Chairman of the Board for Lucid's entire 10-year life (and its CTO the whole time), I believe I'm in a pretty good position to know things about the company.
Lucid was founded in 1984 to write and OEM a Common Lisp system for the delivery of AI systems on stock hardware. From 1984 until 1988 or 1989, we were the premier provider of Common Lisp systems and nothing else. We had OEM deals with essentially every computer maker in Europe, Japan, and the US. The software we wrote is still in use under the name Liquid Common Lisp, having been acquired by Harlequin in an assets sale—Harlequin itself was recently bought by Global Graphics.
Second is the treatment of the Lucid/Emacs interactions.
It is true that a subsequent Lucid product line used Emacs. This second product line is the one of interest to open source. This second line was centered around the idea of producing a Lisp-quality programming environment for C and C++. At the time, HTML was not entirely visible in the computing world, but the idea of hypertext was of great interest. Our basic idea was to use a compiler to locate within source text all the linguistically interesting elements and to have those elements and their relations to other elements be held in a repository for use by all the tools the programmer would use. Additionally, other tools would similarly "annotate" the source, and the programmer or programming team would operate on this richly annotated source. The central repository was the item that Lucid would sell. The idea was that Lucid would publish a set of protocols for compilers, debuggers, editors, and graphers to use which would enable other tools vendors to produce tools that could connect up to the server and thus be "Energize" tools (the name of our product).
These tools would hook up via a wire protocol so that the surrounding UI type tools could be distributed, both logically and geographically. For example, we designed a thing called an "extent" which was a file descriptor and a pair of indexes within the text of that file, along with an associated target which would link that text with the target (hypertext). The link was typed so the server would know what actions the user could take. In Emacs, such annotations would be marked or highlighted, and when a user clicked on one, the server would be notified (according to our protocol) and a popup menu constructed by Emacs to present the user with the operations.
The idea was that by defining a generally useful protocol for Emacs, Emacs could be made to appear very knowledgeable about the programming language when in fact a compiler for the language would have created all the annotations during its analysis. So, for instance, if a set of files went into creating the source text for a class definition, the server would know about that and the programmer could observe it by looking at the annotations. Well, on and on about the clevernesses of the idea.
But, back to Emacs. Because I had been a spokesman for the League for Programming Freedom for most of the 1980's and believed strongly in Emacs and what Stallman was doing (this is my interpretation of my own motives), I linked up with Richard and asked him how Lucid and FSF could work together. I forget who came up with the idea, but we concluded that if there were a generally useful set of features that an Emacs user and Emacs programmers could get to which could be defined as a wire protocol, and if all the changes could be coordinated through Stallman and given away freely using GPL, then if a commercial company provided proprietary software for sale that connected to Emacs via the protocol, that would be OK. We also agreed to publish all the specs for all the protocols and endeavor to modify GCC and GDB so that someone could provide a free version of the server Lucid was going to do.
To work with Stallman, Richard suggested we get JA as a consultant, who was an FSF person, and to contract with Cygnus for the GCC and GDB changes. Lucid agreed. As I recall, the then-current version of Emacs was 18 and JA had been working on version 19, which would work with X and other windowing systems and would be the scrollable, menuable version of Emacs. The Lucid extensions would start with the version 19 code.
Over time, I put about 5 developers working on the design and implementation of the changes along with JA.
Problems started to occur when two things happened. One was that JA turned out to be not such a good developer and designer, and the other was that Stallman wanted to generalize and rewrite all the code we produced. The problem with the latter from my perspective was that Stallman believed that there would be dozens of annotations per file and we believed that would be hundreds or thousands. His design was too slow, ours was too specialized. Plus, RMS wanted to move slowly and we wanted to move fast.
Also, working with Stallman became a real pain after a while. So, I finally put Jamie Zawinski on it full time and he became famous as a result. You can read his interesting observations about working with JA and Stallman somewhere or other.
On the failings of JA-written Emacs 19 code: The redisplay code was totally broken—not only real slow but incorrect and incomprehensible. Redoing that started my plunge into putting huge manpower on Lucid Emacs.
I should note that one of the ideas behind Energize was to have the UI interaction be pleasant and modern. Emacs of the era was mostly neither (though it was the editor I always used and still use—it's a hacker's editor). So in addition to extents we put in (variable width) fonts, colors, scroll bars, a menubar, and popup windows.
The key to Energize's success would be having a good quality compiler which could tell the server everything about the code (in fact, the annotations the compiler would produce were general enough that Lisp, Fortran, Ada, and Cobol could be annotated with a common set of markers—as I said, real clever). We had modified GCC mostly by paying Cygnus to do it, but we found that in the early 1990's, companies were generally too nervous to use an FSF (open source) compiler for mission-critical items, plus they wanted top performance. So Lucid bought a high-end compiler company (Peritus) and made that the primary compiler to use for Energize. Our main advertised advantage was that Energize could do function-level incremental compilation (and dynamic updating of running images), and for that we needed to be able to dump efficiently a representation of compiler state. Our belief was that the richness of information placed in one or only a few windows would prove the true benefit, but it was difficult to prove it the way incremental compilation was.
We also learned something else disturbing: Emacs was not very popular with the bulk of our market. This was a bit of a shock to us. We had explored very thoroughly the customer space for Energize (we called it Cadillac, and our informal motto was "an A++ environment for a C++ programmer")—talking to a variety of groups and companies for over a year in great detail. Emacs never seemed to be an issue. But we quickly found that the bulk of C and C++ programmers used and loved vi. Well, first we tried all the vi modes for Emacs we could find. No go—people were used to firing up a vi at the command line, hacking, then exiting before compiling. Starting and stopping Emacs didn't make sense at that time. We then tried to do the same thing with vi we did with Emacs, but we could not find anything we could use that corresponded to the official source. Finally we cobbled together a version of Energize that would examine the "damage" vi did to files to find out which ones to recompile.
However, the market did ok. During the last 3 quarters of Lucid's life, Energize sales grew at 35% per quarter. The Lisp business was by itself profitable. Lucid failed because of a loan situation the CEO got us into—the story can be found in my book "Patterns of Software," the section called "Into the Ground."
Lessons to be learned for open source have to do with the interactions we had with Stallman and FSF. Early on we had pretty complete agreement, though knowing RMS as well as I did, I suspected he never fully trusted a company. His dedication to Emacs was thorough, but his pace was way too slow for us and his design center (a dozen annotations per page) was aimed at human-entered hypertext not machine-generated annotations.
Eventually, Stallman put his foot down and said that he wanted to carefully (re)design our changes so they would work on character terminals. We were willing to help out with this project, but we needed to get things done fast. It wasn't that we did things more haphazardly, but we had up to 5 people available to work 12 hours a day on Emacs, and so a couple hours of Stallman a day except when he wanted to clog dance was not that impressive to us.
Our first choice was always overwhelmingly to have our changes be to the official Emacs 19 branch. Our second choice was a fork—even though JA proved to be incapable of moving the official fork forward and we thought RMS knew this as well. We never tried seriously to push our version of Emacs over the official one (because what would have been the sense in that?), but we did try hard to push ours as a 100% compatible one in that any Elisp code that worked in Emacs 18 would work in Lucid Emacs—so, same init files would work. This is important to understand: When RMS balked at putting our changes into the official source fork, that was a big business blow to us. To then have turned around and touted our version over RMS's would have been detrimental to our business. But, JWZ complicated this issue.
As I recall, RMS started to believe we were out to take over Emacs. This was reinforced by JWZ's growing hatred of RMS and JWZ's stated personal goal of burying RMS. However, when this started to happen, the game had already shifted to vi. We positioned Lucid Emacs to vi-based customers as a fancy source browser (which happened to be a mondo editor).
As a side note, I *believe* almost all our non-extents code made it from Lucid Emacs into Xemacs. The extents idea was obsoleted by HTML once it became visible. Anyhow, the trees merged well enough eventually.
Odds and ends:
"...and sell [Lucid Emacs] to the development community as a replacement for the original Emacs...."
We never sold (for money) Lucid Emacs, nor did we sell (as in "push") it as a standalone product or program, and we never tried to position it as a replacement for the original Emacs. We always sold our full C/C++ development environment and tried to "sell" Lucid Emacs as a legitimate, fully compatible version of Emacs of particular value because of its connection to Energize. Our strategy was for Emacs 19 to *be* Lucid Emacs, and that was working well until dealing with JA and Stallman overcame us.
"When the Lucid team went to pitch the Xemacs solution to various companies, they found that they could not draw enough of a distinction in the results produced by Xemacs versus Emacs."
We never pitched the Xemacs solution to editing or development. Some could argue that when we tried to pitch our *environment*—the compiler, the server, the modified GDB debugger, vi, and the Lucid Emacs browser/editor—some companies thought it was priced too high for the differences in productivity (getting incremental compilation to really provide a difference for real C++ code was very hard, and we never completely solved it). We priced Energize similarly to ParcPlace's existing market-leading product in the area. We put them out of that business, as I learned when I joined ParcPlace a few years later.
"...Lucid GPLed its Xemacs code before it went out of business."
This is on its face correct, but misleading. Lucid Emacs was GPLed from the start. Initially we were working with Stallman as best we could and with his minions, and for that period, Lucid Emacs was Emacs 19. When working with them became too difficult, the already GPLed Lucid Emacs remained GPLed. We posted our sources at MIT and Stanford on their FSF directories. We scored a lot of downloads over the years.
For me, the lessons for open source are as follows:
Now back to making up facts. I read "The Magic Cauldron" recently. There is a tone to it I find unappealing, but the initial flaw in might have long-term bad effects. There appears to be a big deal made of criticizing companies for basing the sale price of their software on the development cost of producing it, thus making companies "insane". I'm not sure whether there was ever a time when the development cost figured much into pricing decisions, but it never has in my experience as big cheese at Lucid for 10 years, as VP of Development at ParcPlace Systems for 4 years, or as an affiliate of Sun for 3 years—at each place I have been in the very middle of pricing decisions.
In all the cases I know of, pricing happens like this, piecemeal: First, you describe the product and the likely market. You look at a variety of price points and then estimate revenue over some time period—generally, there is a standard set of "psychologically determined" distinct viable price points. You pick the price that maximizes revenue and see whether that shows profit considering all development and other cost-of-sales expenses over the same time period. Then you do the development and repeat the exercise, but this time you set the price. The cost of development *does* figure in, but at about the same place that the cost of renting the building does.
Figuring out expected revenues at a price point requires a lot of research and knowledge about buying behavior, network effects, etc.
I've speculated that if there ever was a pricing methodology like the one Raymond posits, it ended with Turbo Pascal. As best I know, Turbo Pascal was a compiler developed by a small Danish company and was priced at the then-usual $500 for high-end sub-workstation-class compilers. Well, the company made a deal with Borland and then Borland hired its developer, Anders Hejlsberg. The cost of developing the software, then, for Borland was miniscule—perhaps as little as a few hundred thousand dollars. So they priced it like they priced everything at around $50 and it took off. Had the Danish company priced it that way (and had it the market presence and channels of Borland), it would have made the same revenue, but of course it couldn't have done that. Ever since, the methodology has been what I outlined above.
If Magic Cauldron is to have an impact—or perhaps if open source is to continue to command attention—it must not be full of "facts" like Raymond's pricing theories which causes great yukking when any businessman reads them. I say this even though I believe in his conclusions. Tribal storyteller he may be, neophyte cultural anthropologist he might be, economic and business theorist he is not.
If you do a second edition to OpenSources, I would really appreciate the factual part of the Lucid story to be corrected. Thanks.