The Software Value Chain

I have read some interesting articles recently: "How Google Works" gives a fascinating overview of Google's pioneering approach to providing a high-performance service using commodity hardware. Similarly Amazon's Dynamo storage system provides an innovative approach to the problem of storing resilient persistent data without using an ACID database. What is striking in both cases is how industry norms are not just being ignored, but are being turned on their head. For many organisations the cost and risk of developing and maintaining a proprietary persistence layer would outweigh the performance benefits gained.

At first I thought that this was an issue of businesses whose core competence is not software choosing to effectively outsource these elements of their value chain e.g. by using a J2EE app server and an Oracle database. However thinking more about this there are lots of business who have a highly sophisticated approach to software who do not dare go against these industry norms. Why then do Google and Amazon innovate where others fear to tread? I see two reasons for this:

• By developing a custom infrastructure they maximise their competitive advantage since the infrastructure is tuned to precisely what the business needs, no more and no less. Amazon's Dynamo has been designed to exactly fit in with their business processes (e.g. a shopping cart) giving a performant and lean solution.
  
• The custom infrastructure is an enabler for business services through which they gain competitive advantage - how long would it take to serve up a standard Google query if the data was stored in an Oracle database?

Is this the way forward then? The reality is that most IT engineers are not as smart as those that designed Dynamo or worked out how to provide Google with a resilient infrastructure using commodity hardware. Standard platforms such as .Net and J2EE enable average engineers to deliver reliable solutions. Don't rush out to design a custom persistence layer just yet.