Getting Real, Release It!, and The Perpetual Beta: Modern Web Apps

I've been thinking an awful lot lately about web application architecture and release strategies. This has a lot to do with a project I am working on professionally, but it ties into everything I've been thinking about regarding architecture in general.

Here are some links for reading more about these topics and insights from some of today's top developers, including 37signals.com, the creators of the Ruby-on-Rails framework and developers of http://www.Meetup.com

Getting Real, by 37Signals

Learn about 37Signals by googling them. They are the creators of the Ruby-on-Rails framework that has pushed a lot of other people to higher levels of quality. Only a true Microsoft fanboy would not know that tons of what is inside of Microsoft's forthcoming ASP.NET MVC framework comes straight out of the Ruby-on-Rails framework.

Getting Real is a book by the 37Signals crew about how they do Agile development.

Read it here: http://gettingreal.37signals.com/index.php

Here is a key excerpt from chapter 2 in the essay entitled Fix Time and Budget Flex Scope

"Launching something great that's a little smaller in scope than planned is better than launching something mediocre and full of holes because you had to hit some magical time, budget, and scope window. Leave the magic to Houdini. You've got a real business to run and a real product to deliver.

Here are the benefits of fixing time and budget, and keeping scope flexible:

• Prioritization

  You have to figure out what's really important. What's going to make it into this initial release? This forces a constraint on you which will push you to make tough decisions instead of hemming and hawing.

• Reality

  Setting expectations is key. If you try to fix time, budget, and scope, you won't be able to deliver at a high level of quality. Sure, you can probably deliver something, but is "something" what you really want to deliver?

• Flexibility

  The ability to change is key. Having everything fixed makes it tough to change. Injecting scope flexibility will introduce options based on your real experience building the product. Flexibility is your friend."

Comments
I really like these ideas with regard to web applications. These principles become extremely important when you are migrating a system from an older technology to a new technology, especially when the benefit you seek is primarily for the application infrastructure as opposed to for the end-user benefit. You have to ask yourself questions like:

• How does this migration actually benefit the user?
  
• Is is tested well enough to replace the existing system such that users do not notice the change?
• Is there a way to minimize any possible damage the new infrastructure could cause to profitability should it not work as hoped?
  

As a consultant, you have to make clients aware of these questions. Your job is to inform them about possible problems and strategies for risk mitigation. Ultimately, they may select to do something you either agree with or don't, but you have to due your diligence when you recommend a migration strategy.

The situation is more complex when you are implementing not just a back-end migration, but also adding brand new features that you want to introduce to actually improve the user experience.

Suppose you have this scenario:

1. You have a client that operates a popular shopping web site
   
2. The existing application is running ASP with COM objects written in Visual Basic 6.0
3. The existing application works well and has been tested through real-world use for more than five years
4. The existing application continues in increase in value, leading to higher and higher ROI each year
5. Your client wants to migrate to Visual Basic.NET and ASP.NET 3.5
6. Your client wants to add new features to the system that will increase the usability and utility for the system's users such as Ajax-enhanced search pages and RESTful URLs that offer strong Search-Engine-Optimization benefits
   

You have to carefully weigh all of these demands and criteria. Ask questions like:

• How important is time-to-market?
  
• How important is it that users do not have any interruptions in service?
• What are the performance and scalability requirements?
  

Yes, Yes, and Yes
In my experience, most clients will want it as soon as possible and with as few interruptions as possible and with as good or better performance as the existing system. This is just a given.

But, you really cannot deliver all three of those concurrently. You have to make some trade-offs.

In this case, I believe the best migration strategy is what is called a vertical migration strategy. Read more about this in the Dr. Dobbs link below.

1. Create the new foundational architecture in Visual Basic.NET to support the system
2. Create Interop assemblies on top of the COM objects
3. Create the new, value-added functionality first
4. Bring up the system in beta side-by-side to the existing system so that the value-added features can be delivered to the users and so that the client realizes early return-on-investment (ROI) and gets early user feedback.
5. Monitor the system's performance and refactor any of the bottleneck areas caused by interop by implementing them in pure .NET code.
6. Add more features to the beta to slowly replace the existing application, getting user feedback and important performance and scalability information all the while to help direct your refactorings.

This is different from a horizontal migration strategy. In a horizontal strategy, you would select an entire layer of system, such as the UI, the Business Logic, or the Persistence Layer. A horizontal strategy is typically more complex and time-consuming and requires more testing.

However, you can use a very similar risk mitigation strategy to what you do in a vertical migration. You can bring up the new system side-by-side with the existing one and allow users to alpha and beta test it while you measure the usability, performance, and scalability and refactor as needed before it replaces the existing system.

You can read much more about the various approaches to ASP.NET application migration in this Dr. Dobbs online article: http://www.ddj.com/windows/184406077


Figure 1: Vertical Migration Strategy












Figure 2: Horizontal Migration Strategy



Release It!: Design and Deploy Production-Ready Software

Another book that I have eyed on shelves of late is called Release It!: Design and Deploy Production-Ready Software by Michael Nygard. Here is an interview with the author on InfoQ about the book and his lessons learned:

http://www.infoq.com/articles/nygard-release-it

Michael Nygard: First off, there's quite a bit of variation in what people mean by "feature complete". Even at best, it just means that all the specified functionality for a release has passed functional testing. For an agile team, it should also mean that all the acceptance tests pass. In some cases, though, all it means is that the developers finished their first draft of the code and threw it over the wall to the testers.

"Production ready" is orthogonal to "feature complete". Whether the acceptance tests pass or the testers give it a green check mark tells me nothing about how well the system as a whole is going to hold up under the stresses of real-world, every day use. Could be horrible, could be great.

For example, does it have a memory leak? Nobody actually runs a test server in the QA environment for a week or a month at a time, under realistic loads. We're lucky to get a week of testing, total, let alone a week just for longevity testing. So, passing QA doesn't tell me anything about memory leaks. It's very easy for memory leaks to get into production. Well, now that creates an operational problem, because the applications will have to be restarted regularly. Every memory leak I've seen is based on traffic, so the more traffic you get, the faster you leak memory. That means that you can't even predict when you'll have to restart the applications. It might be the middle of the busiest hour on your busiest day. Actually, it's pretty likely to happen during the busiest (i.e., the worst) times.

This is crucially important. Memory leaks come from third-party vendors just as often as they come from your own internal code. There is nothing like having to log in remotely to a web server when you're trying to have fun and the third-party component is causing your web server to hang. These are things that people rarely think about up front because they typically are problems revealed only by real system usage. I'll give a real world example:

Suppose you have a Visual Basic 6.0 COM object that use ADO internally. It may keep a RecordSet open to allow for consuming code to rewind the cursor and start over from the beginning. Well, .NET uses non-deterministic finalization, so you have to take care to call System.Interop.Marshal.ReleaseCOMObject to inform the runtime that it should destroy the COM object when you are finished with it. If you do not do this, you could end up with long-standing blocks against your database until the garbage collector frees the COM object.

I have run into this problem, and luckily was able to refactor my wrapper class in one single place to alleviate the problem. In the web application, we never rewind the collection, so it was safe for us to free the object after the while loop in the IEnumerable.GetEnumerator() completed.

As for this book, you can read the full table of contents and excerpts from the book in this PDF extracted from the book:

http://media.pragprog.com/titles/mnee/mnee-patterns.pdf

After looking at the TOC, I know this is a book I want to read.

Web 2.0 Applications and Agile Methodologies
This brings us into the territory of web 2.0 applications and the topic of the agile methodology.

The wikipedia entry for Web 2.0 defines the following key characteristics for a web 2.0 application:

"The sometimes complex and continually evolving technology infrastructure of Web 2.0 includes server-software, content-syndication, messaging-protocols, standards-oriented browsers with plugins and extensions, and various client-applications. The differing, yet complementary approaches of such elements provide Web 2.0 sites with information-storage, creation, and dissemination challenges and capabilities that go beyond what the public formerly expected in the environment of the so-called "Web 1.0".

Web 2.0 websites typically include some of the following features/techniques:

• Cascading Style Sheets to aid in the separation of presentation and content
• Folksonomies (collaborative tagging, social classification, social indexing, and social tagging)
• Microformats extending pages with additional semantics
• REST and/or XML- and/or JSON-based APIs
• Rich Internet application techniques, often Ajax-based
• Semantically valid XHTML and HTML markup
• Syndication, aggregation and notification of data in RSS or Atom feeds
• mashups, merging content from different sources, client- and server-side
• Weblog-publishing tools
• wiki or forum software, etc., to support user-generated content
• Internet privacy, the extended power of users to manage their own privacy in cloaking or deleting their own user content or profiles."

Suppose your task is to migrate a web 1.0 application to a web 2.0 application such that it increasingly resembles these features. How do you do that with maximal risk mitigation and protection of the existing system's ROI?

Taking Cue's from Yahoo and Google's Lead
First, from a process methodology standpoint, both Yahoo and Google have adopted an agile process. In particular, they have adopted a Scrum-based development methodology. You can watch the following videos to learn more about that:

• Scrum Tuning: Lessons Learned from Scrum Implementation: http://video.google.com/videoplay?docid=8795214308797356840
• Scrum et al: http://video.google.com/videoplay?docid=-7230144396191025011

Just to summarize, however, here is what Scrum looks like visually:


Verbally, the wikipedia article describes it as:

Scrum is a process skeleton that includes a set of practices and predefined roles. The main roles in scrum are the ScrumMaster who maintains the processes and works similar to a project manager, the Product Owner who represents the stakeholders, and the Team which includes the developers.

During each sprint, a 15-30 day period (length decided by the team), the team creates an increment of potential shippable (usable) software. The set of features that go into each sprint come from the product backlog, which is a prioritized set of high level requirements of work to be done. What backlog items go into the sprint is determined during the sprint planning meeting. During this meeting the Product Owner informs the team of the items in the product backlog that he wants completed. The team then determines how much of this they can commit to complete during the next sprint.^[4] During the sprint, no one is able to change the sprint backlog, which means that the requirements are frozen for sprint.

There are several good implementations of systems for managing the Scrum process and the "sprints" while others prefer to use yellow stickers and white-boards. One of Scrum's biggest advantages is that it is very easy to learn and requires little effort to start using.

Why have both Google and Yahoo adopted scrum? Well, just listen to what someone inside of Yahoo had to say about this in the Scrum Mailing List.

http://shmula.com/159/scrum-at-yahoo

"What the Times doesn’t say is that Yahoo! is now 18 month into its adoption of Scrum, and has upwards of 500 people (and steadily growing) using Scrum in the US, Europe, and India. Scrum is being used successfully for projects ranging from new product development Yahoo! Podcasts, which won a webby 6 months after launch, was built start-to-finish in distributed Scrum between the US and India) to heavy-duty infrastructure work on Yahoo! Mail (which serves north of a hundred million users each month). Most (but not all) of the teams using Scrum at Yahoo! are doing it by the book, with active support from inside and outside coaches (both of which in my opinion are necessary for best results).

Pete Deemer Chief Product Officer, Yahoo! Bangalore / CSM"

Microsoft also uses Scrum-based methodologies for building systems. See this eWeek article here for details:

http://www.eweek.com/c/a/IT-Management/Microsoft-Lauds-Scrum-Method-for-Software-Projects/

David Treadwell, corporate vice president of the .Net Developer Platform group at Microsoft, said that while Microsoft welcomes the use of methodologies like Scrum, "were not mandating them, but were encouraging them. So Scrum is one process—the idea that teams meet once a day for half an hour, figure out what theyre going to do then go off and do their work very quickly.

"Its most important to mandate levels of quality. You have to give teams some flexibility to achieve those results as is most effective for those teams."

Indeed, "There are a lot of things going on," at Microsoft on this task, Treadwell said. "We have realized inside Microsoft over the years that software practices we used in the mid-90s dont scale to the size of problems that were tackling today.

"And we made some assumptions around the turn of the century that those processes would scale up and result in certain time frames that we would be able to ship software.

"And what happened is as the projects got larger and larger, we introduced too many complex interdependencies on early software, more so than we could really digest throughout the system," said Treadwell.

"And that was super-challenging for us, given the scale of these projects. But now were being much more precise about where we take those interdependencies."

Microsoft ASP.NET and MVC: A new direction
We see further changes in Microsoft's approach to development in the way they are releasing upgrades to ASP.NET and the MVC framework. Take a look at http://www.codeplex.com/aspnet. This is the new home for the ASP.NET framework, where Microsoft makes "often and early" releases to the developer community. They are still a few steps short of going fully open-source, but I think they will get there sooner rather than later. At least, I hope they do if they hope to survive in the competitive market.

The Perpetual Beta: A way forward that ensures quality

Finally, arrive at the idea of a perpetual beta. This is something that Tim O'Reilly discussed a few years back about the nature of a web 2.0 system. Read more about his comments here:

http://en.wikipedia.org/wiki/Perpetual_beta

and here:

http://www.oreillynet.com/pub/a/oreilly/tim/news/2005/09/30/what-is-web-20.html?page=4

His key point about the concept of a perpetual beta is this:

"Users must be treated as co-developers, in a reflection of open source development practices (even if the software in question is unlikely to be released under an open source license.) The open source dictum, "release early and release often" in fact has morphed into an even more radical position, "the perpetual beta," in which the product is developed in the open, with new features slipstreamed in on a monthly, weekly, or even daily basis. It's no accident that services such as Gmail, Google Maps, Flickr, del.icio.us, and the like may be expected to bear a "Beta" logo for years at a time."^[1]

The Perpetual Beta Deployment Model in Practice

In practice, you could implement the Perpetual Beta model as follows:

Web Farm: production.samplesite.com
[ Production Server #1 - App v1.0 ] [ Production Server #2 - App v1.0]

Beta Server: beta.samplesite.com
[ Beta Server #1 - App v1.x ]

To follow Google and Yahoo's lead here, you would deploy release candidate builds to the Beta Server and allow users to select to use that version of the system, but they could always navigate to the production URL instead of the beta.

This provides you the following benefits:

• Users become part of your testing process and help you develop and guide the evolution of the system
• Your team becomes far less stressed and annoyed because they have a true metric of user satisfaction and system stability to gauge before changes actually go into the true production environment
• You decrease your time-to-market by having a system that is continually improving

Cautions and Caveats to this process:

• You have to take care in testing the beta build well in advance of pushing it to the beta server. Make sure there are no database concurrency issues or blocking processes that could cause the beta and the production system to conflict.
• You should be exercising automated tests and using regression test tools well in advance of the beta deployment.
• You will still not catch all problems before it goes into the real production environment. This is just the way development is.

Conclusion
These are just some of my notes and thoughts on the way forward for web application development. I know I'm already behind on most of this stuff, but some of my readers are further behind :-)

Is there a silver lining to all of this change and fast pace?

I believe there is. The silver lining is HTTP and URI.

If you build your URIs to last, last they will. That is the fundamental change in mindset that is taking place for most of the successful players right now. They are realizing that they can construct their services using RESTful designs that allow both applications and users to repurpose content and data in ways that nobody thought possible before.

If you don't believe me, just head on over to http://pipes.yahoo.com or read up on Google Base http://www.google.com/base.

URI stands for Uniform Resource Identifier. It's about time we started treating it like one. We've got to stop reinventing the wheel and start driving the car we have with the wheel we got already.

Check out my previous posts on REST or just read Roy Fielding's dissertation to learn more.

Related Resources

• Orbitz.com Lead Architect Brian Zimmer on Challenges on a Large Scale Project: http://www.infoq.com/interviews/Architecture-Brian-Zimmer
  
• James Shore, author of "The Art of Agile Development" discusses his book: http://www.infoq.com/interviews/The-Art-of-Agile-Development-James-Shore