Agile is a path (to trust (among other things))

I think I figured out why Steve Yegge was so inclined to bash Agile yesterday. It’s actually rather obvious when you think about it. I don’t think it’s some recruiting scam, like some do. No, there’s a much simpler explanation. Steve likes Google’s practices, and is trying to warn other Google employees away from Agile. It has nothing to do with the rest of the world, unless of course you’re lucky enough to have practices like Google, which work.

He went astray by thinking what’s bad for a group of Google newbies is bad for everyone. That’s an easy mistake to make since he probably spent at least 20 times more time with those young Google employees, and other Google employees then with other programmers, at least recently. I don’t really know anything more about the Google process than what Steve told us, but it sounds like it has promise. More importantly it’s working. It might be better than Agile. I say might because I don’t really know. To be honest, I’ve never had the opportunity to participate in an “Agile” project, but I’ve employed bits and pieces in different places over the years. And from those experiences one thing has been clear: anything is better than waterfall.

I can understand how, given his pre-existing skepticism, and observing a few Googlbies having problems with Agile, he might lash out. But Steve is wrong. Those Googlbies might actually be ditching a great process for a merely good process, but for the majority of us out here, the question is completely different. For us, for right now, Agile is the way to go. It’s not easy to convince management to try it, but I think it might be easier than asking to become Google. For example, if you wanted to be more like Google, the first thing you would do is fire 90% of management. That might not be such a bad thing because most bad agile is the result of bad managers who can’t adapt to the principles of agile, who keep trying to force old principles and values into the system. Principles like overwork, whipping and unrealistic date-driven development. But managers have most of the power at most companies, and it seems unlikely they’ll sign on for that plan.

The next items on the checklist are basically developer freedoms, which are hard to do if you skipped the first step and still have bad managers hovering around. Maybe you could get away with stripping those managers of their authority, so that hovering isn’t so destructive. But, I have a feeling they won’t sign up for that plan either. I mean the majority of the bad ones became managers because they crave authority, so they aren’t going to give it up easy.

Like many people have mentioned, these are people problems. What Agile is good at doing is creating an environment where management actually starts to understand what developers do. After that they’ll start to delegate more authority to the developers. Maybe one day everyone will end up like Google, but that’s not an easy change, and before anything like that happens management will have to first learn to trust development. Google probably works because management IS development. And I don’t just mean the guys at the top; I mean almost all of it.

However, the point behind all of it is trust. And trust is a tricky thing. A lot of people forget trust has two parts. The one most people focus upon, is a belief in honesty or integrity. But there is another part, a belief in capability. This division is the source of many arguments. People take it very personally if you say you don’t trust them. Much more seriously than if you simply said you didn’t think they were capable. It’s not very flattering to be told you’re not capable, but it’s a lot less insulting then to be told you’re not trusted, even though usually they mean the same thing.

In the workplace, you’ll hear people say they trust so and so, but only in the most confidence will you ever hear someone mention who they don’t trust, no matter which type of trust is involved. This is unfortunate, because when it’s a lack of trust in capability, people should know so they can respond rationally, either by proving their worthy of trust, or by improving. But trying to do this is like walking through a minefield. You never know when someone is going to blow up. Even if there is a pot of gold on the other side, is it really worth the risk?

Another part of the problem is that management often does not trust themselves, though they won’t easily admit this. They don’t trust themselves to spot the bad apples, or the incompetent. To deal with bad apples, they resort to paranoia. In their defense, part of the problem is that they are hindered by legalities put in place to protect minorities, disabled and other protected groups, they have a secondary effect of making it much more difficult to fire (or at least not promote) the incompetent. That’s a tough issue because at least for now, those groups do need some protection, and no one has found a better way of insuring it. Maybe sometime in the distant future we’ll fix that problem, but for now it’s a bit harder than any of our development quibbles.

To bring this back on track, Agile development is intended to encourage communication that promotes trust. It’s intended not only to help the developers understand the customer’s needs better, but also to help the customer understand the developer’s role better. Waterfall on the other hand attempts to squeeze every ounce of risk out of any decision. Unfortunately it makes the mistake of forgetting that by avoiding a decision, you have made a decision; a decision to stall.

And stall is what happens to project after project under a waterfall development methodology. By the time a decision is made, there may be no chance of misstep, but you’ve taken a considerable penalty for that piece of mind. Even worse you often find that things change so rapidly, that this point is unreachable. Organizations try to make up for lost time by cramming the follow up to the decision in unreasonable ways, and well.. we know what happens then, don’t we?

This has been quite a ramble, even for me, but overall what did I say? We need trust, Agile is a path from the here and now to trust, once we have mutual trust we can tackle the next stage, and according to Steve, Google already has done all of this. Wow, can that really be true?

Try your best Agile, and why we can't all be Google (yet).

Steve Yegge is definitely one of my favorite writers, because he makes some really good points, and in interesting ways. But I have to admit it’s usually full of a lot of blind spots. If he were asking me (he’s not) whether he should mellow out, I’d say no because in despite the blind spots, his rants are probably the most effective route from point a to b, for him. I think that’s because he’s not a zealot, and is willing to flip-flop his opinions several times while trying to find the right solution. Either way, when you see of these blind spots, you gotta yell “car!”

So why am I writing all this crap about someone else’s writing? I mean, in all likeliness I’m probably just as bad or worse in some other way, or maybe the same way. Well today, it’s because of his most recent post, Good Agile, Bad Agile. On the good side, he reminds anyone who forgot that there is a bad version of almost everything. He also points out many of the more risky Agile practices, like abuse of iterations as a recurring death march tool, or as an excuse to leave refactoring for time boxed iterations.

XP actually cover both worries, with the principles “Sustainable Pace“, and “Design Improvement”. Of course, those can be ignored, and according to Steve often are. He goes on to compare Agile to Google’s process, which certainly sounds very interesting. But Google’s not your average company, and just like Agile itself, there is no guarantee that if someone tried to export those practices to another company, that company wouldn’t screw it all up. There are several parts of those practices, at least today, are impossible, highly difficult or outrageously risky for any organization not in Google’s class.

Take the work queue he wraps up the description of good Agile with. Sounds cool, but I don’t see it on the market, and until it is, I don’t see any company without lots of “hand-wavy” math types putting a similar internal app into service. And I think that piece is critical to the whole system to provide some balance to the chaos factor of the other freedoms.

It sounds a lot better than index cards, but index cards in many ways area better than the project management software on the market today. I’ve seen a number of these and amazingly, they haven’t even tackled what should have been the first task: making it easy to get data into the system. It takes like a whole hour to enter in a few tasks for the week, and even then, it’s an inaccurate jumbled mess.

Beyond that visibility is generally very low, so that only the managers have a clue what’s going on. Or at least they think they do, because they don’t realize how bad the data is. Beyond being wrong, it leaves out huge gaping holes; and since non-managers never see the data, they never have cause to point out the deficiencies. And so managers make decisions as if that non-measured data didn’t even exist.

The index cards ideas are just a simple way to try to resolve some of these problems, like visibility and ease of data collection. If you resolved them in software then you’d gain the benefit of being able to use all that hand-wavy math, but today the rest of us get the worst of both worlds: no math, bad visibility, and painful data entry.

So, I’ve made a case for “all the bad stuff he described is the fault of the team's execution of the process”, so I might as well totally ignore Steve’s caution and move on to "all the good stuff he described is really Agile". To be truthful, not all of it is, but I have made the case that at least some of it would be very hard for any non Google class company to do alone. And a lot of the rest just might not work without those other pieces, or without the natural enthusiasm Google has from being able to be very selective, very rich, and very famous. It might work, but it’s hardly proven.

A lot of it is Agile, however. And Agile is potentially good. And, yes, if 90% it screws up it’s bad.. unless of course your comparing it to a process that screws up 95% of the time. In that case, it’s the most amazing thing since sliced bread. It does however mean that you need to pay attention to what you’re doing, don’t take things for granted, and always try to improve your process. And as far as bad Agile goes, well I’d say even the most questionable practices, like pair programming have value in moderation.

Even if taken to obsessive extremes they’re better than the other extreme. It might seem wasteful to tell half your staff “thou shalt not code”, but it’s a lot better than telling half your staff, “thou shalt analyze”, and the other half “thou shalt copy badly written documentation into badly written code, with no understanding of the real problem.”

So, for most organizations, it’s not a choice of good Agile vs. bad Agile, but bad waterfall vs. try your best Agile.

I can identify with Steve’s desire to separate out some of the technology aspects, like unit testing, continuous integration, refactoring, out from the process part of Agile. In many cases they are a lot more difficult to screw up so it’s nice to show that distinction, but a lot of the process practices are intended to help adoption of those practices, and to keep them in place, so you can’t really get the Agile people to drop em. At best, you could come up with a new group, like “The Workable Code Movement™”, that’s a subset of practices if that’s what you want.

Update: I wrote two more follow up posts on the same topic. Over the top? You decide.
See "Agile is a path (to trust (among other things))", and the more liked "Good Ideas taken Badly"

Closures for C++?

This one surprised me. I totally saw the Java closure thing coming, but somehow I assumed that the C++ standards group would be too stubborn to even consider such a change. But according to an article, the ISO are considering lambda syntax for C++.

It's not very well linked in the original article, but I believe the lambda proposal being considered is from Valentin Samko. What's most surprising is that it appears to me as if they are considering full closure capability.

What's next? C? or maybe COBOL. Or maybe LISP.. err hold on.

Anyhow, interesting stuff.

LCD Price Changes

I knew the day would one day come. Anyhow, 20" LCDs have really plummeted in price in the last 2 months. It used to be a 19" LCD was $200-350 depending on brand and milliseconds. The cheapest 20" LCD was about $500, which was a pretty steep jump.

Well, no longer. 20" LCD's today have dropped under $300, and the bargains like this ViewSonic are about $250. So now my upgrade recommendation must upgrade. 20" monitors are the right thing to buy for desktops (two is best of course).

Notably, not far above there is one outlier in the 22" range. Most 22's are $400 or more (which is still far better than than $700-$1000 of not long ago), but one Acer (AL2216WBD) goes for $330. That's a good sign, although I'd still recommend the 20 because a wide 22 is less than 7% more space for a 32% price hike. Widescreens aren't IMHO as good for business stuff either, unless you only get one, which you shouldn't ;p Besides that when you look at the space provided remember that a 20" 3:4 monitor will be about 14% larger than a 20" 16:9 monitor. Consumers are starting to wise up to this and now widescreens are selling at less than squares for the same diagonal, rather than a reverse premium.

The 19" to 20" jump at 3:4 is about 12% more space, for about 25% price hike. Considering the relative cheapness of both, I'd say 20" is the right choice.

I should caveat all of this by stating these recommendations for getting average work done. Graphics artists might find more preference to wide, and gamers are going to be more about the size of their one monitor (and the refresh rate), than about how many they can afford.

Making change work for you - Code Change Analysis

These days, the best developers, and development organizations realize maintainability is one of the most key aspects of design. Applications, libraries and frameworks rarely if ever remain static. These developments evolve over time. This is an obvious concept, but it has been ignored more often than one would think possible. How many times have projects placed “get it out the door” over the characteristics that will contribute to a maintainable system?

Maybe it’s just me, but I’m seeing progress in this area, and I’m hoping it’s a sign of a shift in thinking, or a better connection between developers and management. Of course, this progress might just be the most recent saga in a long, slow and inexorable process that has been progressing for longer than I have personal experience. Regardless, greater acceptance of techniques like unit testing, test-driven development, refactoring, static code analysis and two-way modeling & design are excellent examples of progress. Also, the evolution and standardization of tools like defect tracking and source code control are notable signs. As well, process is getting better as it is realized that trying to make all decisions upfront, and then “just code it”, just doesn’t work.

The concepts behind these improvements have been around for quite some time already. The newest is barely less 20 years old. Mainstream acceptance and integration are very important but once that’s done, what’s next? Well we need new ideas and new goals to follow.

One opportunity that I think we are missing is the ability to leverage code change itself. One of the main purposes of unit testing is to help insure behavior, other than what you intend to change, remains constant. With good tests, you can refactor with greater freedom and success. By running tests before and after change, you can determine, within some limits, how much behavioral change you have caused.

But unit tests only cover the behavior we have defined through tests, and we find it difficult to cover every conceivable possibility. But behavior isn’t the only thing changing in this scenario. Our code is changing as well, so why is it we don’t leverage that element of change more? Sure, our source control tools allow us to diff the files, and review them that way. And we can have code reviews that focus on the changes made. But all of this is a lot of work, and so there are limits to how thorough time will allow us to be. And besides, if we could find a way to make those activities less necessary, we’d be able to focus on other issues. Or we’d just do those activities in a more focused and effective manner.

So why not have automated tools to analyze code change itself? Such tools could highlight troublesome types of changes, much like static analysis tools do for static code. But one of the biggest problems with static analysis tools is they create too many false positives. On a large project, with strict settings a static analysis tool is likely to return 1000’s of warnings. If you had a more convenient way to see just those warnings that were related to change, the signal to noise ratio would be far better.

Beyond that, code change analysis could go much further. Did you introduce side-effects to a side-effect free method? If that happened, it would be very nice not only to be reminded of that change, but also to be reminded of any instances where that method is used within a loop. Have you introduced new dependencies? Wouldn’t it be nice to see a list of those for review? Have you opened up previously unreachable code paths?

There are opportunities that go beyond analyzing the code too. Automated analysis of code profiling histories would tell us when performance degraded in a certain area. Maybe we don’t want to address it right away, but later on it may help immensely to know these trends when we go back to perform optimization.

The point is, what we’re trying to manage is change. To do that, we need to make change work for us, at least some of the time. Just as unit testing embraces the concept that what we are searching for in testing is not simply correctness, but often unexpected change, we should also embrace that the search for change within other tools as well.

Java closures continues to chug along

The proposal to add closures to Java continue to move along. Neal Gafter has posted an update on the closure proposal.

Open Writing

Over the past few days, I’ve run into a number of very good posts about zealotry and perceptions. It’s something I’ve written about before, in The Opposing Viewpoint, and offhandedly mentioned others. It’s also a topic I feel strongly about. But it’s a hard point to make because zealots don’t really see themselves that way. It’s not much different from how the above average syndrome. You could even call it the more rational than average syndrome, if you care, though I think there are better and more common indicators. But regardless it still occurs, most likely because statistically, zealots are likely to already be more emotional, and thus find it extremely easy to miss or ignore those indicators.

And if they have trouble seeing this normally, calling them a zealot only makes the problem worse. That’s one reason I tried to leave mention of any specific group out of my post. The original spark in that case actually had surprisingly little to do with computers, though I could certainly apply it to things like language advocacy, like Mark-Jason Dominus does, or to technocentrism in general, like Brian Kuhn or Jeff Atwood do.

But really, the problem is a lot bigger than that, these are just those examples that are most familiar to us. The underlying message is keep an open mind. Also, you can have strong beliefs without being a zealot, but it requires you to take the time to listen to, and observe the arguments of the other side.

But these days, it seems people favor the loud and obnoxious approach more and more. The worst is the point by point retort. In this approach, a writer finds a point of view he disagrees with, a representative article on the topic and attempts to refute every sentence, no matter how weak the arguments are. Even if the original premise had a core of truth, the result is a quagmire of half-truths, exaggerations and just plain wrong bits.

People who write (or talk) this way, may think being loud helps to be heard better, but quite often it does not, at least not in a meaningful way. It will get a lot of attention, but usually that attention will be from those that already agreed with your original premise. Of those, some will be turned off by the mixed in idiocy. What you’re left with is a bunch of people who will believe anything you say that attacks position B. Maybe that’s what you’re looking for, to be surrounded by idolaters, but then your motives are most likely bad.

The writing I enjoy the most is that which introduces me to something new. Writing that gives a new perspective, new information or new concepts. I also think this is the most valuable type of writing. The value, at least in terms of accomplishment, however is connected to who the message reaches, not just the content. Being right is of course critical, but if you write for the choir, with fire and brimstone, you’ll never change opinion for the better. Most likely, you will introduce a divide between one side and the other. And this will lead to some real idiocy, either through flawed arguments that accompany your primary premise, or through the natural course of group mentality.

Don’t mistake an open mind and individualism for weakness. In the long run it will make your arguments stronger, because they will become more correct. As an added bonus, it will relieve a lot of stress when you stop thinking of things as us vs. them. It will also help you accomplish your original goal, though you may find along the way, that you were originally half right (or more optimistically 90%). Or you may find compromises that are far more valuable than a protracted disagreement. Regardless, the choice remains up to you, and the important part is that people are listening, and so are you.

Who is running the marketing department at AT&T?

Ok, a bit off topic, but I just could not believe this one. I got an email from AT&T that suggested I upgrade my Internet service to new and faster service. Sounded fine to me so I went to check it out. Well as with several other mistakes in marketing, it turns out the email directs you to a web page that asks you for you email address. (Don't they know this?). After entering my number in, I'm am told that I'm not eligible for an upgrade (Didn't they know this before they sent me an email?)

I am going to guess that this has now become standard far for AT&T customers, but what really topped it off for me was the way that AT&T politely informed me of that their offer was not valid.

The message read:

AT&T Yahoo! High Speed Internet Request for ZZZ-XXX-YYYY.

Our records indicate that you have a service that is not renewable and not eligible for an upgrade offer; however, you are welcome to keep your current service and rate by taking no further acrtion.
If you would like more information, please contact us at 1-877-722-3755.

I have to say, I'm awfully glad that I'm welcome to keep my current service. In fact I think I may write AT&T to thank them for this. The problem is I'm not sure if I should take any further "acrtion". I know they tell me I don't need to, but I kind of want to.

Think I should submit this one to Daily WTF? It's not like a misprint, or a improperly targeted email, or a rather insulting message is too surprising. But all three at once? WTF?

Advanced IDE - Language Workbenches

A few weeks ago, I was writing about the importance of metadata. In another recent post, I talked about a common pattern from A Pattern Language, and some manifestations of it. I ended by mentioning an essay of Martin Fowler’s that tuned into what I’ve been thinking about for a while, but haven’t quite so eloquently written.

Actually, this post was originally part of that post, but I felt it deserved separation from the somewhat JVM/CLR debate that emerged in the other post. I didn’t want to attach such an important topic to a post that touched on a topic with so much ideology attached.

To get to the point, Martin Fowlers work, talks about “language workbenches”. He defines a workbench as an environment where:

• Users can freely define new languages, which are fully integrated with each other.
• The primary source of information is a persistent abstract representation.
• Language designers define a DSL in three main parts: schema, editor(s), and generator(s).
• Language users manipulate a DSL through a projectional editor.
• A language workbench can persist incomplete or contradictory information in its abstract representation.

Since that might not make too much sense out of context, I’ll sum it up. The idea is the workbench maintains a central abstract representation of code. Editors, instead of being simple text editors interact with this representation. The overall user experience could be similar or the same, but the underlying mechanics totally different.

This is another example of the alcove pattern, I mentioned in the previous post, where the workbench is defining a high ceiling abstract representation, but multiple editors and sublanguages can be used to satisfy the needs of individual domains. This takes the inter project cross language compatibility established by JVM/CLR and extends it to intra project activities.

Advantages

Beyond the large and numerous advantages Martin mentions, what interests me is this is making the code data, in a different way than traditionally thought of. The focus so far has been upon the runtime qualities of the code-data relationship, but this has been limiting. The ability to rewrite your software at runtime has its uses, but by far the most common use of the code-data relationship has been macros, and definition of domain specific languages.

Workbenches would fulfill those abilities without requiring languages to include them. Languages could still include those abilities, and integrate them with the capabilities of a workbench. But, even languages without those features could interact with DSLs and be manipulated by macros.

Here are some other possible advantages I thought of:

Documentation

With a workbench, you could take the concept of documentation beyond the XML and annotation syntax available today. These are excellent, but current implementations have reached the limit of allowable complexity. Documentation needs, however, are still not fully met, so a new approach is necessary.

Feedback & Navigation

Improvements in editor feedback have become extremely valuable (statement completion, color coding, syntactical validation). The possibilities for more feedback and greater control over it are much greater when the code natively is abstract, rather than coerced through background compilation. Speed differences alone open a huge number of possibilities.

In Patterns of Software, Richard Gabriel says:

What is important is that it be easy for programmers to come up to speed with the code, to be able to navigate through it effectively, to be able to understand what changes to make, and to be able to make them safely and correctly.

A tool optimized for reading can do a vastly better job of providing navigation and feedback than one optimized for writing. But obviously, developers still need tools for writing/editing code. Language workbenches would provide more opportunities for tools optimized for reading. They could for example, allow inline function expansion, so that it’s possible to see what sub-calls are doing without jumping back and forth between files.

Editors could be better as well. Expanding upon the sub-call display concept, it could be allowed to edit sub-calls inline. Even better, extract method refactorings often require tweaks to the internal and external elements of the caller and callee. This interface could be allow these to happen without jumping back and forth and provide a more seamless conversion from one method to two methods.

Trivial translation

You could display language elements in C#, VB or even Java syntax. There are certainly limits to this. These examples work well because they are structurally very similar. In theory, the same is possible for more complex differences, but at some point, it would begin to look like prose automatically translated from English to Japanese. Italian to Spanish automatic translations, however are far more readable because they are structurally similar. Translation might still be useful from the point of view of reading code, but certainly you would never want two people editing the same code, one from a C perspective, and one from a LISP perspective.

Complex Problem Spaces

DSL’s are used for complex problem spaces too. Tools like the Web Service Software Factory or Smart Client Software Factory generate code which builds a framework for further development. But a difficulty is in establishing the barriers between what the DSL tool will need to modify later, and what is setup for the developer. In addition, a lot of this code is boilerplate, and while it’s great that developers don’t have to type it in any longer, it’s still makes reading the code difficult.

A workbench based design would allow this code to be better separated and provide a more abstract view of it, more amenable to maintenance, and better integrated into the development cycle than the wizards of today’s tools.

Problems

There certainly are issues that need to be resolved as well. Martin mentions a few good ones. I touched on one, in describing translation, the multiple editor problem. It’s important that you retain the same formatting capability (tabs, spaces, line breaks, etc.) that you have in editors today, but this is difficult when multiple editors are touching the data. The simple answer is you only use one editor, and the rest are views, but this would seriously cripple some of the advantages.

Most likely, we will see a pragmatic usage view emerge, where development teams choose certain restrictions to place upon practice. It’s silly to suggest that the workbench designers need to solve every single possible problem before they are usable. I recall when Microsoft first started talking about multiple language support there was a lot of worrying about developers needing to learn multiple languages. Besides the fact that the languages are almost identical, the problem was resolved inside teams by making a choice per project/team/organization that made sense.

Next Steps

What workbench designers do need to do is provide a high ceiling, in the form of a central storage format that meets the needs of all the sublanguages. Complexity here is unavoidable, but luckily its impact is restricted to tool designers, rather than global to all developers and users. This format could be a textual language, but that is probably more effort than it is worth because aesthetics and readability for this format are less important than capabilities, and maintaining a high level of capability with good aesthetics and readability is difficult.

Also possible are binary formats, but these are not well suited because only the most naïve designer would assume that future expansion would not be necessary. What’s best is probably a format like XML which already establishes aesthetic and readability quality levels (however poor), and is inherently extensible.

Despite this bias, I think the best place to start is with IL and bytecode. They may be binary formats, but they already accomplish 50% of the problem statement, so starting from scratch is abandoning a big opportunity for incremental development. Perhaps a mapping between IL and XML would be the right way to begin. A very notable advantage of this design is compilation then becomes a matter of removing the extra developer only elements (extra whitespace, comments, etc.)

Incremental development is also an important tool in insuring that these tools do not become the CASE tools of the future. Unfortunately most of the projects Martin mentions are taking the approach of one big leap, most specifically Intentional Software, which is almost secretive.

Connections, Shared Space and Autonomy

Patterns

It’s interesting how simple patterns reappear in the most unexpected places, and how they often the similarity goes unnoticed. For example, take this simple pattern condensed from A Pattern Language.

179. Alcoves**

. . . many large rooms are not complete unless they have smaller rooms and alcoves opening off them. . .

* * *

No homogeneous room, of homogeneous height, can serve a group of people well. To give a group a chance to be together, as a group, a room must also give them the chance to be alone, in one’s and two’s in the same space.

This problem is felt most acutely in the common rooms of a house—the kitchen, the family room, the living room. In fact, it is so critical there, that the house can drive the family apart when it remains unsolved. . . .

In modern life, the main function of a family is emotional; it is a source of security and love. But these qualities will only come into existence if the members of the house are physically able to be together as a family.

This is often difficult. The various members of the family come and go at different times of day; even when they are in the house, each has his own private interests. . . . In many houses, these interests force people to go off to their own rooms, away from the family. This happens for two reasons. First, in a normal family room, one person can easily be disturbed by what the others are doing. . . .Second, the family room does not usually have any space where people can leave things and not have them disturbed. . . .

To solve the problem, there must be some way in which the members of the family can be together, even when they are doing different things.

Therefore:

Make small places at the edge of any common room, usually no more than 6 feet wide and 3 to 6 feet deep and possibly much smaller. These alcoves should be large enough for two people to sit, chat, or play and sometimes large enough to contain a desk or table.

Give the alcove a ceiling, which is markedly lower than the ceiling height in the main room. . . . (Alexander 1977b)

CLR – Multiple Languages

Reading this for a second time invoked a number of thoughts. The first was to relate this pattern to virtual machines, like the .NET CLR. In this first example, I was thinking of how they serve as a nexus for different programming languages. The CLR (or JVM) itself is much like the room at the center, and the languages the alcoves. Much like height of ceilings, no one language can meet the needs of all developers or development tasks. Different developers require languages of different complexity. More importantly, different development tasks require languages of different complexity (and possibly developers of different complexity too).

But developers are best as a community to avoid solving the same problems more times than necessary. Without a central (intermediary) language, it’s not possible to share libraries seamlessly, and communication is more difficult. The CLR serves as a common center. IL establishes sets a level of maximum complexity, and each language retains some autonomy while enjoying the benefits of the CLR.

It’s worth noting that the CLR must be more complex than all other languages by nature of having to support the primitives of those languages. In principle, languages using the CLR could be more complex, but they would have to dumb down their behaviors and fake that complexity within IL. But by doing so, that complexity would become inaccessible to other language consumers. Ideally, you want the IL to support the same or greater level of complexity.

I use the CLR as the example here, because Microsoft was first to popularize the concept, but the JVM is just as capable of this. There are some minor differences, but I won’t waste time (today) trying to discover which is better. Instead, I’d prefer to move on to a similar example that the JVM popularized.

JVM - WORA

One goal of the JVM was to unite multiple operating systems. The concept was Write Once Run Anywhere (WORA). You can see the similarity, where the JVM acts as a center. Operating systems would share applications, not just libraries, and communication would be less difficult. However, like language interoperability you’d want the central room to have the highest ceiling, because otherwise you would be limited by the JVM.

But whereas writing a high level, nonuser language like Java bytecode or IL relaxed many constraints of design, due to not having to worry much about the user, multiple operating system support did not have the same advantage.

For simple actions that were already common to operating systems, the process worked well. For more complicated areas, like user interfaces, where different operating systems had significantly different implementations, it did not. Java has tried very hard to address this, but while Java 6 is set to address many outstanding UI issues, Microsoft is set to release a new UI framework. Beyond this, what about the unique UI features of the OS X, both today and tomorrow? In order to be the center for all operating systems, Java would need to be an operating system, bigger and more complex than all others.

Of course, a more practical approach is to rely upon WORA only in application domains that rely mostly upon simple common features. In these areas, the JVM serves portability needs very well.

It’s interesting to note that such portability is as possible with the CLR. Mono for example runs most .NET applications that don’t make use of user interface libraries tightly coupled to the Windows operating system. But cross platform user interface libraries are just as possible under the CLR as under the JVM. GTK# is an excellent example of one. But of course, the same difficulties with taking advantage of OS specific features arise, but if portability is more important than the latest features then this is an option.

It’s also interesting to note that other operating systems can have direct support of their own, like with Cocoa#. The Java community has generally discouraged this type of development, but it is just as possible under the JVM.

Integrated Development Environments – Language Workbenches

A few weeks ago, I was writing about the importance of metadata. Serendipitously, a reader of another post linked me into an essay of Martin Fowler’s that tuned into what I’ve been thinking about for a while, but haven’t quite so eloquently written. To sum it up, Martin talks about “language workbenches”. He defines a workbench as:

• Users can freely define new languages, which are fully integrated with each other.
• The primary source of information is a persistent abstract representation.
• Language designers define a DSL in three main parts: schema, editor(s), and generator(s).
• Language users manipulate a DSL through a projectional editor.
• A language workbench can persist incomplete or contradictory information in its abstract representation.

Since that might not make too much sense out of context, I’ll sum it up. The idea is the workbench maintains a central abstract representation of code. Editors, instead of being simple text editors interact with this representation. The overall user experience could be similar or the same, but the underlying mechanics totally different.

This is however, another example of the alcove pattern, where the workbench is defining a high ceiling abstract representation, but you can use multiple editors and sublanguages to satisfy the needs of individual domains. This takes the inter project cross language compatibility established by the JVM/CLR and extends it to intra project activities.

There’s more to this topic however, which I will cover in another post.

Other examples

There are plenty of other examples of this pattern. Operating Systems themselves are a good example. Applications need their own process space, but they also need a standard system for communication with hardware, other software, and the user.

Pieces of networks are good examples as well. The network itself isn’t a very good example, but anywhere that users have both a private and shared space, such as sites like digg and reddit are good examples.

This is an important concept to remember in designing network applications because it’s important not only that you give users a place to communicate, but also that you allow them enough isolation that they don’t wander off to their own rooms, or feel constrained within the one room.

Communities and thinking of others

My recent post A LISPie day. Data, Code, Context and Hints, garnered some interesting comments.

For example, I knew someone must have remapped parentheses ;p

Seriously though one commenter points out the rigorous body of conventions are a replacement of syntax. But, while naming conventions are good, and fulfill most of the purposes of syntax, they aren't syntax.

One of the differences is developers can step outside of those rules. In some cases this is good, but in most cases it is a bigger problem than help. My impression is LISP’s current users don’t notice this because the community is drawn from the elite of programming.

That's not to say LISP programmers are the elite, there are probably far more great Java programmers, though most of those probably know something about a lot of languages, LISP included. The thing is, there just aren't many dummies that stick with LISP.

The result is that conventions are followed, and you get the impression that they are actually sufficient, when they are only sufficient for a language with a limited community of that type. I actually understand the reaction from hardcore LISPers because they are happy with things as they are. They just wish people would stop badmouthing their language of choice.

But what I'm thinking about is how to bring many or all of the benefits of LISP capabilities to the wider world. That will require some compromises. It doesn’t mean LISP as it is today has to die, but it probably does mean that it won’t become more than the niche language it is today.

However, even this would be a great boon to existing LISPers as now they would have more in common with mainstream development. More importantly however is it would benefit the 99% of developers who don’t use LISP today. For many of these developers LISP is the wrong language because they don’t have the self control to obey/interpret those nice conventions. I wouldn’t say this group is a majority, but they are a large minority. Then there are even more developers who are working on projects where it would be wrong to exclude the first group, either because they need them, or it’s just too hard to weed them out.

So, I apologize to any of you that take this type of criticism the wrong way. It doesn’t mean you’re doing things the wrong way. It’s just sometimes you need to think about people other than yourself, and what their challenges are. If no one did this, then we’d only have a million or so computers worldwide, and my Mom would have no idea what the Internet was. Even now, she may not really know what it is, or at least how it works, but at least she knows enough to send/receive email, check bank accounts and book travel arrangements. This is because some developer who knew a lot more about these things was thinking of people like her when they created their software.

Eventually I think you’ll see a lot more people who are not professional programmers do some programming. I don’t want to see my development tools dumbed down, just to let these users in, but I’d still like to see some commonality between my tools and theirs, so if one of them asks a question I won’t have to learn a completely new tool to answer it.