Tuesday, December 14, 2004

Problem Hunting with Identify Software's AppSight

I was recently lucky enough to be invited to a demonstration of Identify Software's AppSight J2EE Application Support System. I was first made aware of their product at JavaOne where I stumbled across their stand and was intrigued by what they had to offer.

The problem that AppSight addresses is how to track down, diagnose and resolve problems in production environments. In my experience when a customer reports a problem it can be very difficult to reproduce the problem in the development environment. Log files rarely contain the detail necessary to paint the picture, due to the runtime overhead that such detail entails. Even very basic diagnosis (is the problem user error? a configuration problem? a software defect?) can be difficult for anything but the most trivial system. Throw into the mix various in-licensed components provided by multiple vendors and the picture gets very murky indeed.

AppSight's solution is based on the software equivalent of an aircraft's black box recorder. This is a software module deployed on the application server that records the behaviour of the application at multiple levels. Using this information the actual steps performed by the user can be reproduced. Identify's own web site provides lots more information, which I don't plan to reproduce, but there are a couple of points which I think are of particular interest:

  • I was immediately suspicious of the run-time cost of deploying such a module. However according to Identify themselves, there is a 2-4% overhead on the server and 1-2% overhead on the client.
  • You can define alerts based on particular events (say database connection pool usage reaching 90%). That in itself isn't so interesting. But what I think is really neat is that you can specify that the level of information gathering be increased based on a particular alert. That is, if there is a problem, automatically start gathering more information, rather than wait for it to become a real issue.

There are a couple of things that occurred to me after the demo that are worth considering:
  • Normally when I get a problem report, I like to use it as the basis for one or more test cases to add to my test suite. Is there any way this could be automated by AppSight?
  • On a more abstract level, is there a risk of some kind of Schrodinger's cat phenomenon? That is, could the presence of the AppSight module affect the behaviour under observation?

After the demo I was left with the feeling that I had seen a product that I would like to use in anger. Just need to find the opportunity now...

[I should point out that I have no connection with Identify Software nor have I actually worked with AppSight in a production environment.]

Thursday, December 09, 2004

Book Review: Beyond Software Architecture by Luke Hohmann

Beyond Software Architecture is a book about some of the business issues that need to be considered when developing a software system. It provides practical advice and insight into how software needs to be considered part of an overall business solution if it is to be successful. The emphasis of the book is on software products (mainly enterprise level) but there are many useful lessons to be learnt for project-based systems.

One of the key concepts introduced at an early stage in the book is the distinction between "marketecture" (marketing architecture) and "tarchitecture" (technical architecture). Under this categorization the traditional software architect (aka tarchitect) is responsible for tarchitecture and the business manager or program manager (aka marketect) is responsible for marketecture. This isn't a dry academic exercise in defining terminology: these two roles pervade the book. The underlying message is that the marketect and tarchitect need to be acutely aware of their own and each other's roles; the tarchitect needs to understand the business forces driving the product, and the marketect needs to appreciate the technical basis for the solution. Having the marketect and tarchitect roles defined is a necessary but not sufficient condition for creating winning solutions.

The book goes on to consider various issues which have implications for both marketecture and tarchitecture. For instance, which kind or product licensing model is suitable for the intended business model; what are the implications of using in-licensed technology in the solution; how should the product be branded; and so on. The point is that these are issues that the tarchitect needs to understand, be aware of and in some cases be involved in decision making for. However these aren't the kind of thing one normally reads about in books on software architecture.

For software architects working with software products, this book is essential reading; certainly in my own case, when I previously worked as architect on a product I would have found a lot of the information in the book directly relevant to my daily work. For those working on software projects, there are still some interesting practical issues such as installation, configuration and release management. However some of the other areas covered are purely related to products, so these can be skipped.

My only minor criticism of the book is that I would have enjoyed more real-world anecdotes; there are a number of sidebars in the book presenting the author's experience of a specific case supporting the more general message in the text. These were very interesting and I would have liked to see more of these.

Thursday, December 02, 2004

Don't mix software and The Economist

I started reading The Economist recently and have found it to be an interesting weekly look at what is happening in the world, providing a slightly broader view than Time or Newsweek. As a rule it tends to stick to politics and economics, though occasionally subjects on the fringe slip through. For instance a few weeks back there was a special article about outsourcing that was very interesting.

When I got my issue this week I noticed there was a special article about the software development industry entitled "Managing Complexity". I went straight to it, intrigued to see what they had to say about my "back yard". Alas the normal erudition and clarity that I have come to expect from The Economist were notable by their absence in this article.

The article began by making the point that a large proportion of software projects go seriously wrong and according to a recent study 60% are considered failures by the organizations that initiated them. So far so good. In the light of the recent Child Support Agency fiasco it is also something that I have been thinking about.

Beyond this point things start to go wrong. According to the article the problem can be summed up as follows
"The Culprit: poor design"
That's it. The article proceeds to present a peculiarly US-centric survey of trends in software development such as open-source, agile development etc. The quotes provided are from US software tool vendors, so it is unsurprising that they are in agreement that the solution to the problem is to buy the newest tools they have to offer.

I could point out the inaccuracies and fallacies in the article in great detail (and I'm sure that this would provide some catharsis for me - if you hadn't already guessed. the article touched a nerve). However I think it would be more interesting to speculate on some of the issues that the article ought to have delved into.

If I had a magic answer I would of course have become rich from this knowledge; however I think there are some clear areas that cause problems:
  • Poor management on the procurement side When an organization procures a software solution, there is some onus on them to know what they want. This may sound obvious, but I have worked on projects where a high-level management decision has been made that a group of users should use a new system that is to be developed, but when it comes down to it no one actually knows what the system is supposed to do. Large organizations often have a budget=power culture, so once the budget has been approved the procuring manager might not hang around very long, instead climbing further up the greasy pole.
  • Poor management on the vendor side Software project management is an area strewn with lots of literature (good and bad). I don't intend to repeat any of that here. However one area often overlooked is the need to manage the expectations of the procuring organization. The vendor management needs to be intimately aware of the maturity level of the procuring organization and factor this into planning. Related to this, a less mature procurer will often come up with new requirements as their understanding of what technology can offer improves during the course of the project. Vendor management has the responsibility to explain to the procurer that expanding the scope of the project by introducing the new requirements has consequences for deadlines and budgets. Note also that increasing the budget to accommodate such an expansion will inevitably be reported by the press as a budget overrun.
  • Overhyped Technology The one thing that unites otherwise warring software vendors is the claim that their new product will solve all of your development problems. This despite all previous experience to the contrary. Sometimes this overhype can get totally out of control which leads to inappropriate technology choices for a project. Witness the recent backlash against the use of EJBs.
These are the ones that occur to me straight off the top of my head. As I think of others I will add them to this list.

Wednesday, November 10, 2004

Teaching Design Patterns

I have a long standing lurking suspicion that design patterns are inherently difficult to teach; my own experience both teaching and using design patterns indicates that they are only really appreciated by students who are able to relate the design pattern in question directly to a program they have developed. Otherwise not only do students have difficulty understanding the design pattern, but it becomes very difficult for them to apply it as they have difficulty recognizing the appropriate situation.

Wednesday, October 20, 2004

Houdini - The Most Stupid Dog in The World

Years ago Yvonne and I lived next door to a dog called Houdini. He was a beautiful looking boxer, just a puppy when we first made his acquaintance. Unfortunately despite his charm and good looks, Houdini was undoubtedly the most stupid dog in the world.

The reason for this statement (and let's face it, dogs aren't renowned for their intelligence so it isn't a title bestowed lightly) is an incident that occurred about a year after we moved into the house next to his. It was his habit to lean his front paws on the shared fence in our back garden and peer over (he was very nosey). One day while in this position, he noticed something flapping around from the corner of his eye. Eventually he couldn't control himself, had a nip at it and then emitted a huge yelp - it was his own tail that had been wagging away. His owner Pam took him to the vet where he was bandaged up and sent home suitably chastened (or so we thought).

The next day, in the same position in the back garden he once more noticed something flapping round, so he again took a nip at it and emitted an even louder yelp; once again his own tail was the victim of his exuberance. Pam took him to the vet again, who as well as patching him up, mounted a conical collar on him so that he wouldn't be able to see his tail.

The following day, pleased as punch Houdini was in the garden again. This time he was not troubled by objects flapping in the corner of his eye. Unfortunately while he was in the garden there was a sudden downpour. Houdini, being a uniquely gifted dog decided to stare straight up into the sky during this downpour. The result of this was that his collar rapidly filled up with water and would undoubtedly have led to his drowning if an alert Pam had not charged out of her house at high speed and knocked Houdini's head forward causing the collar to empty.

After that Houdini was not allowed out in the rain on his own until his tail healed.

I challenge anyone to come up with a more stupid dog than Houdini.

Thursday, September 30, 2004

Policy-Based Design

I recently had the pleasure of reading an article written by one of my Systematic colleagues, Jan Reher, entitled "Policy-Based Design in the Real World" published in the C++ Users Journal. The article shows an example from a real application of the use of policy-based design, as described in Alexandrescu's book Modern C++ Design.

Policy-based design allows functionality to be parametrized in a number of independent dimensions. The textbook solution is an elegant application of C++ templates and multiple inheritance. The point of Jan's article was to demonstrate that the textbook solution can actually be used in a real application.

I found the article very interesting for a number of reasons. One of Jan's concerns was the real-world utility of policy-based design. I see this as a broader issue: many of us are familiar with the situation where an elegant textbook solution evolves into an unmanageable mess when applied in practice, so it is always good to get real validation of such a solution. On a practical note, since I have worked mainly with Java for the last few years, it was interesting to try to recast Jan's solution in Java (pre J2SE 5.0's generics). What became rapidly apparent was that the solution would be significantly more clunky in Java. Actually, even with generics, the Java solution would be less concise than the C++ one due to the use of multiple inheritance in the C++ solution.

Sunday, September 19, 2004

JXTA at FWJUG

Attended the monthly meeting of the Fort Worth Java Users Group last thursday where we learnt about JXTA and P2P solutions from Daniel Brookshier who is part of the JXTA development team and has written a book on the subject. The talk was very good and for me there were a couple of things that I really took away from the meeting:

- JXTA is not a Java specific technology. It is a protocol for P2P communication, and there are already implementations in several languages.

- P2P is really a paradigm shift in distributed computing, compared to a traditional client-server architecture. This is really blowing my mind a bit, and I still don't think I have totally got my head round it.

One of the examples that Daniel referred to several times was 312inc.com's product lean on me that performs backup using a P2P approach. This was a good example of where a bit more thought up front yields a massive saving in initial investment since a traditional approach would involve investing in backup servers etc.

Sunday, September 12, 2004

The difference between Britain, Denmark and the US

Being a Brit who has spent the last five years living in Denmark and now lives in the US, I have a unique perspective on the differences between these countries. Of course when you get down into the details, there are a huge number of contrasts that can be drawn. However I think I have been able to distil the the fundamental difference into a simple concept: how the different countries deal with problems.

In the US, if a problem arises, the solution is to throw more people at it. In Denmark, if a problem arises, a new tax is created to solve it. In Britain if a problem arises, people go to the pub, grumble about it over a beer, then stagger home solutionless but happy.

Readers are most welcome to provide their own national approach to problem solving :-)

Sunday, September 05, 2004

Book Review: IT Architectures and Middleware

IT Architectures and Middleware Second Edition
Chris Britton & Peter Bye, Addison Wesley 2004.

I have been reading a lot of books about architecture recently, so when I spotted this one at my local Borders, I bought it without looking too closely at it. If I had paid more attention then I would have noticed that even its title hints that it isn't a standard book about n-tier architecture and/or UML modelling. First, the book is about IT architecture not software architecture, and second middleware is a major driver for the book.

In fact I have to say this is one of the most refreshing books I have read recently. It has the subtitle "Strategies for Building Large, Integrated Systems"; though this is undoubtedly true, I would be more inclined to call it "Everything you wanted to know about architecture but daren't ask"! Certainly for me it filled in some gaps in my knowledge and in general drew together a number of topics that aren't always treated in architecture books, and gave them a nice coherent thread.

The approach the book takes is that making decisions about middleware are critical to an IT architecture, and the choice of middleware really drives the architecture. Practical issues that arise when creating an architecture are described at a level of detail which highlights the critical issues without getting bogged down in technology solutions. The policy throughout the book is to be vendor neutral; that's not to say that particular technologies are not discussed such as J2EE and .Net, but there is no rabid claim of unconditional superiority for any specific technology. That said, there are a number of examples (more of these below) that reflect that the authors have extensive experience with Unisys.
Avoiding specific technologies also means that the reader doesn't have to wade through pages of listings! On the other hand, the authors' experience is very useful when highlighting potential gotchas throughout the text.

On the negative side, the book is very concise (337 pages including index). Though I welcome a change from the standard 600+ page treatise, there is a fine line between being concise and being dry. I found towards the end that it was harder and harder to take in all of the information in the text, and I suspect that I will have to reread it to pick up the things I missed first time round. I was therefore surprised that there are a number of examples described in the final chapter of all places. I would much rather have seen these examples (and many more examples for that matter) earlier on, driving and motivating the material. This would inevitably have increased the length of the book, but I think it would have made it much kinder to the reader. As it is, I think any reader of the book needs to have some background in architecture; there is too much for a total novice to take in.

All that said, I would thoroughly recommend this book for anyone creating IT architectures who wants a solid, vendor neutral overview of the different issues that need to be considered. I have just started reading Beyond Software Architecture by Luke Hohmann which in its own way also provides an alternative take on software architecture. More of that later.


Saturday, September 04, 2004

Instanceof Code Smell

I have noticed several times over the last few years that Java developers misunderstand when to use instanceof. In particular I have noticed an instance of java.lang.Object interrogated using instanceof through a sequence of if statements. Each time I have seen this I have explained why this is a bad idea and have tried to find a reference that summarizes this code smell, but to no avail, so I have chosen to write something myself.

First of all, there are legitimate uses of instanceof. A good example is overriding the equals method in a class you are designing yourself. However too often instanceof is used as a solution when the underlying problem is poor class structure. To illustrate this, I have created a toy example. Suppose we have two entity classes: Book and Car. The Book class is shown below.
package example1;


public class Book {
private String title;
private String author;
private String isbn;

public Book(String title, String author, String isbn) {
this.title = title;
this.author = author;
this.isbn = isbn;
}

public String getAuthor() {
return author;
}

public String getIsbn() {
return isbn;
}

public String getTitle() {
return title;
}
}
There is nothing special about this class. Similarly with the Car class:
package example1;


import java.awt.Color;


public class Car {
private String manufacturer;
private String model;
private Color colour;

public Car(String manufacturer, String model, Color colour) {
this.manufacturer = manufacturer;
this.model = model;
this.colour = colour;
}

public Color getColour() {
return colour;
}

public String getManufacturer() {
return manufacturer;
}

public String getModel() {
return model;
}
}
The important thing to notice with these two classes is that from a business point of view they are unrelated, so there is no natural class hierarchy for them. Their only common superclass is java.lang.Object.

Suppose now that we want to be able to generate an XML representation for a list that can contain either Book or Car instances. Using instanceof we could do this as follows:
package example1;


import java.util.Iterator;
import java.util.List;

public class XMLGenerator {
public String generateXML(List items){
StringBuffer result = new StringBuffer();
Iterator iterator = items.iterator();
while (iterator.hasNext()){
result.append("<item>");
Object obj = iterator.next();
if (obj instanceof Book){
result.append(generateXML((Book) obj));
} else if (obj instanceof Car){
result.append(generateXML((Car) obj));
} else {
throw new IllegalArgumentException("Unexpected object: " + obj);
}
result.append("</item>");
}
return result.toString();
}

private String generateXML(Car car) {
StringBuffer result = new StringBuffer();
result.append("<car>");
result.append("<manufacturer>");
result.append(car.getManufacturer());
result.append("</manufacturer>");
result.append("<model>");
result.append(car.getModel());
result.append("</model>");
result.append("<colour>");
result.append(car.getColour());
result.append("</colour>");
result.append("</car>");
return result.toString();
}

private String generateXML(Book book) {
StringBuffer result = new StringBuffer();
result.append("<book>");
result.append("<title>");
result.append(book.getTitle());
result.append("</title>");
result.append("<author>");
result.append(book.getAuthor());
result.append("</author>");
result.append("<isbn>");
result.append(book.getIsbn());
result.append("</isbn>");
result.append("</book>");
return result.toString();
}
}
The part of this class that I really object to is
    Object obj = iterator.next();

if (obj instanceof Book){
result.append(generateXML((Book) obj));
} else if (obj instanceof Car){
result.append(generateXML((Car) obj));
} else {
throw new IllegalArgumentException("Unexpected object: " + obj);
}
The point is that we know from our design that all of the objects in this list should be capable of having XML generated. However by using instances of java.lang.Object we are throwing away this information and then recovering it using instanceof. That is, we are using features from the language to help us overcome our poor design.As an aside, I can also point out that using method overloading in the way this class does, is another symptom of this code smell.

How can we resolve this? The issue is to in some way pass responsibility for generating the XML onto the objects themselves. A first step could be to define an IGenerateXML interface
package example2;


public interface IGenerateXML {
public String generateXML();
}
Book and Car will then implement this interface:
package example2;


public class Book implements IGenerateXML {
private String title;
private String author;
private String isbn;

public Book(String title, String author, String isbn) {
this.title = title;
this.author = author;
this.isbn = isbn;
}

public String getAuthor() {
return author;
}

public String getIsbn() {
return isbn;
}

public String getTitle() {
return title;
}

public String generateXML() {
StringBuffer result = new StringBuffer();
result.append("<book>");
result.append("<title>");
result.append(getTitle());
result.append("</title>");
result.append("<author>");
result.append(getAuthor());
result.append("</author>");
result.append("<isbn>");
result.append(getIsbn());
result.append("</isbn>");
result.append("</book>");
return result.toString();
}
}

package example2;

import java.awt.Color;


public class Car implements IGenerateXML {
private String manufacturer;
private String model;
private Color colour;

public Car(String manufacturer, String model, Color colour) {
this.manufacturer = manufacturer;
this.model = model;
this.colour = colour;
}

public Color getColour() {
return colour;
}

public String getManufacturer() {
return manufacturer;
}

public String getModel() {
return model;
}

public String generateXML() {
StringBuffer result = new StringBuffer();
result.append("<car>");
result.append("<manufacturer>");
result.append(getManufacturer());
result.append("</manufacturer>");
result.append("<model>");
result.append(getModel());
result.append("</model>");
result.append("<colour>");
result.append(getColour());
result.append("</colour>");
result.append("</car>");
return result.toString();
}
}
The XMLGenerator class is now greatly simplified:
package example2;


import java.util.Iterator;
import java.util.List;


public class XMLGenerator {
public String generateXML(List items){
StringBuffer result = new StringBuffer();
Iterator iterator = items.iterator();
while (iterator.hasNext()){
result.append("<item>");
IGenerateXML obj = (IGenerateXML) iterator.next();
result.append(obj.generateXML());
result.append("</item>");
}
return result.toString();
}
}
That's it! No messing around with java.lang.Object, instanceof or erroneously overloaded methods.

Friday, September 03, 2004

nVidia and Mandrake 10

My PC at home runs Mandrake 10.0 and has done for the last couple of months since I upgraded from Mandrake 9.1. In general I have been very pleased with the distribution. My first brush with Linux was nearly 10 years ago when I installed Slackware off 4 floppy disks on a 386 machine which had previously run Windows 3.1. I was amazed by the transformation in the previously feeble machine; whereas under Windows 3.1 it felt like the machine was constantly dragging itself through mud, with Slackware it just flew. However even I have to admit that that installation wasn't very user friendly. In comparison modern distributions such as Mandrake have made quantum leaps.

However since upgrading I have had a problem with periodic random freezes. There seemed to be no consistent cause, but the result was always the same: the screen froze and would not accept any input (not even ctrl-alt-f1) and I couldn't even log in remotely and restart X.

Recently this seems to have been happening more frequently, culminating this evening when it seemed to freeze within 5 minutes of booting. I was sufficiently frustrated to be bothered to do something about it. Given that the machine previously was stable, I guessed the problem was a driver. Experimentation revealed that disabling the onboard nVidia network card (I have an Asus A7N8X motherboard) fixed the problem. Further investigation revealed that Mandrake 10 had automatically installed the open-source forcedeth nVidia driver. I downloaded a driver from nVidia's website and sure enough that fixed the problem. I have at least been running for over half an hour now without a freeze, so fingers crossed!

Friday, August 27, 2004

OpenOffice and cvs

I use OpenOffice at home for my word processing and spreadsheet needs. I often like to keep such documents under version control using cvs. One of the features of cvs that I have found very useful in the past is keyword substitution: version control information can be included in the document under version control so that for example the version number can be seen in the actual document. It works by including special text (for the version number the text is '$Revision: $') in the document. When the document is checked in cvs automatically inserts the relevant value after the ':'. That's fine and dandy but unfortunately it doesn't work with binary files whereas OpenOffice uses binary.

Closer examination of the OpenOffice format reveals that a text document is in fact saved as a zipped collection of xml files. Moreover OpenOffice's word processor supports user-defined properties; from File choose Properties and select the user-defined tab. You can then put the text $Revision: $ in one of these properties as shown below:




(Actually any text could be used, but I chose $Revision: $ to be consistent with normal keyword substitution in cvs.) You can then use this property in your document by selecting Insert | Fields | Other and expand the User-Defined item to choose the one containing $Revision: $.

This still leaves the problem of how to update this value. The solution I came up with is a Python script that you invoke when checking in instead of the normal cvs checkin. You can download this script here.When you run it something like the following should appear:


[pmu@mblinux python]$ sxw_checkin.py testdoc2.sxw "Eighth version"
+++> unzip -o testdoc2.sxw meta.xml
Archive: testdoc2.sxw
inflating: meta.xml
+++> zip testdoc2.sxw meta.xml
updating: meta.xml (deflated 56%)
+++> cvs ci -m"Eighth version" testdoc2.sxw
Checking in testdoc2.sxw;
/software/cvsroot/scripts/python/testdoc2.sxw,v <-- testdoc2.sxw new revision: 1.8; previous revision: 1.7 done


And hey presto! the property in the OpenOffice document has been updated (in this example it would now have the value "1.8").

Small print: I have only used the script on OpenOffice text documents so I can't say whether it works for the other tools that OpenOffice provides; please let me know if you try it!

Thursday, August 12, 2004

Trip to Britain

Just returned from a very pleasant trip to Britain catching up with friends. Stayed for a few days with our friends John and Claire where John regaled me with his new joke: on learning that presidential candidate John Kerry's wife was from the Heinz family, he asked an unimpressed Claire "Do you think he will be able to ketch-up".

Turns out the joke was on me anyway: Yvonne was captivated by John and Claire's Aibo ("Milton") so that is next on our shopping list. She has even chosen a name ("Darwin").

Monday, July 19, 2004

Real Audio on a homepod

Picked up a HomePod at JavaOne a few weeks back, but I am only now finding the time to play with it. One of the things I want to do with it is to be able to listen to Real audio streams, but this is not supported in the current version of the firmware. I have a longer term goal of writing a codec for the HomePod for this format, but just now I don't have the time to delve into it so I needed a workaround.

I tried various things with some open-source audio players such as xmms and MPlayer before finally hitting on something that works, albeit in a roundabout way. I have pointed winamp at the Real audio stream and at the same time I have installed a shoutcast server and the shoutcast broadcast plugin for winamp. Winamp then redirects the Real audio stream to my shoutcast server which then makes it available on my home network as streaming mp3 - a format supported by the HomePod. Not the most elegant solution, but it works. If anyone has other ideas I'm open to suggestions.

For more details, see my home page.