Showing posts with label agile. Show all posts
Showing posts with label agile. Show all posts

Sunday, August 3, 2008

Certified Scrum Master Training from Innovel

I completed my Certified Scrum Master training Friday from Innovel. You can learn more about the training at http://www.scrumtraining.com.

My instructor was Chris Doss. He was a great instructor. It was a very good class and I met a lot of great people in the class as well. I think I was one of the younger folks in the class, so I was able to learn a lot from everyone there just through "osmosis".

Of course, the CSM course is not something that makes one really a well-practiced Scrum implementer. I think it's more of a way to get your feet wet. The progression for training in Scrum is diagrammed here: http://www.scrumalliance.org/training/

The course covers a lot of ground at a high level about Agile, Lean, and Scrum, with a touch of XP. I've been conducting independent study on these subjects and documenting the links and videos I've read and watched on the ATL ALT.NET wiki at http://atlalt.net/screwturn/High-Performance%20Teams.ashx.

Aside from watching those videos from Ken Schwaber and Jeff Sutherland, I read two Scrum books to prepare for the training:

Agile Software Development With Scrum:
http://www.amazon.com/Agile-Software-Development-SCRUM/dp/0130676349

Agile Project Management with Scrum:
http://www.amazon.com/Agile-Software-Development-SCRUM/dp/0130676349

I'm currently reading Scrum and XP From the Trenches from InfoQ Press by Henrik Kniberg:
http://www.infoq.com/minibooks/scrum-xp-from-the-trenches

Future Reading
I am now planning to read Head First Software Development from O'Reilly:
http://www.amazon.com/First-Software-Development-Brain-Friendly-Guides/dp/0596527357

This book is not overtly about Scrum, but the authors thank Henrik Kniberg's InfoQ book and it is heavily geared toward scrum from that I can tell so far.

I love the Head First concept and am currently reading Head First Design Patterns to freshen up on my understanding of design patterns, which is not as good as it should be.

Monday, June 2, 2008

Scrum Link List

As I've been talking about the Scrum software development process framework and its associated terms and phrases like agile, velocity, productivity, reduced defect rate, I have collected some great links.

Getting up to Speed on Scrum

After reading the definition on Wikipedia, I recommend watching Jeff Sutherland's presentation at: http://www.infoq.com/presentations/AgileEnterpriseRealWorldExperience

It presents real industry data and case studies that demonstrate how teams have transformed themselves by introducing agile practices using Scrum. It will definitely make you want to learn more.

Scrum Definition:
Concise definition and links:
http://en.wikipedia.org/wiki/Scrum_(development)

Hirotaka Takeuchi, coined the term Scrum in 1986 after studying how Japanese companies increased their productivity and competitiveness:
http://en.wikipedia.org/wiki/Hirotaka_Takeuchi

Hirotake Takeuchi's book on Knowledge Management:
http://www.amazon.com/dp/0470820748/

Presentations about Scrum:
"The Agile Enterprise: Real World Experience in Creating Agile Companies", by Jeff Sutherland:
http://www.infoq.com/presentations/AgileEnterpriseRealWorldExperience

"Scrum, et al", by Ken Scwaeber, co-inventor of the scrum method for software development, presented at Google:
http://video.google.com/videoplay?docid=-7230144396191025011&hl=en

"Scrum Tuning: Lessons Learned from Scrum implementation at Google", by Jeff Sutherland, co-inventor of the scrum method for software development:
http://video.google.com/videoplay?docid=8795214308797356840

"The Roots of Scrum", by Jeff Sutherland. A presentation with lots of empirical data and examples. He discusses the roots of Scrum within Japanese businesses like Toyota, and about his own experience with medical companies using Scrum:
http://www.infoq.com/presentations/The-Roots-of-Scrum

Audio Lessons:

These three introductions to Scrum by Ken Schwaber are excellent. They give you very clean and clear definitions of what the major aspects of Scrum are. Listen while you jog, listen while you drive, but just listen:

Part 1: http://blogs.conchango.com/howardvanrooijen/archive/2005/10/25/2296.aspx
Part 2: http://blogs.conchango.com/howardvanrooijen/archive/2005/10/31/2323.aspx
Part 3: http://blogs.conchango.com/howardvanrooijen/archive/2005/11/05/2362.aspx

Books:
Agile Software Development with Scrum, by Ken Schwaber and Mike Beedle:
http://www.amazon.com/Agile-Software-Development-SCRUM/dp/0130676349

Scrum and XP from the Trenches, a 90 Page Experience Report, by Henrik Kniberg:
http://www.infoq.com/minibooks/scrum-xp-from-the-trenches

Previous Posts by Me:
I have written a few posts recently about scrum and agile. You can find all those by clicking here.

Enjoy the links!

Monday, May 26, 2008

Scrum at Google: Ken Schwaber Talk

Ken Schwaber addresses Google about Google's Scrum implementation:




In this video he discusses the history of scrum coming out of Japan when Japanese firms were stressed to compete at higher levels. He also talks about how Google's competitor Yahoo uses scrum.

Related Resources

Agile podcast and interviews about Scrum with Ken Schwaber

These three introductions to Scrum by Ken are excellent. They give you very clean and clear definitions of what the major aspects of Scrum are. Highly reocmmended.

Part 1: http://blogs.conchango.com/howardvanrooijen/archive/2005/10/25/2296.aspx
Part 2: http://blogs.conchango.com/howardvanrooijen/archive/2005/10/31/2323.aspx
Part 3: http://blogs.conchango.com/howardvanrooijen/archive/2005/11/05/2362.aspx

Ken Schwaber's book:

Agile Project Management with Scrum, by Microsoft Press: http://www.amazon.com/Agile-Project-Management-Microsoft-Professional/dp/073561993X

Scrum Masters 2: funny video


Review: Scrum and XP from the Trenches



I am reading the book Scrum and XP from the Trenches by Henrik Kniberg. It is about his personal experience successfully implementing a Scrum-based methodology in his development team of 40 people in Stockholm, Sweeden.

You can read the book as a PDF on InfoQ's web site here: http://www.infoq.com/minibooks/scrum-xp-from-the-trenches

Or, you can purchase the book at LuLu here:
http://www.lulu.com/content/899349

Review
Here are my comments on each section of the book:

Foreward by Jeff Sutherland

Jeff co-created Scrum. In his foreward he makes these points:

JSG paraphase:

Direct excerpts:
  • Iterations must have fixed time boxes and be less than six weeks
    long.
  • Code at the end of the iteration must be tested by QA and be
    working properly.
  • A Scrum team must have a Product Owner and know who that
    person is.
  • The Product Owner must have a Product Backlog with estimates
    created by the team.
  • The team must have a Burndown Chart and know their velocity.
  • There must be no one outside a team interfering with the team
    during a Sprint.

Foreward by Mike Cohn

Mike is a founding member of the Scrum Alliance. He has written books, articles, and speaks regularly. Learn more at http://www.scrumalliance.org/profiles/8-mike-cohn.

Mike makes these points:

  • Scrum and XP are both practical, results-oriented approaches. They are about Getting Things Done
  • Prototype early and often rather than documenting requirements at an exhaustive level
  • Avoid excess modeling, prefer prototyping instead
  • Work on things that have potential to become part of the actual working solution
  • Refer to other resources for the theory behind Scrum. It is out there, but this book focuses on implementing Scrum successfully
Preface
  • He was skeptical going in, but was convinced shortly thereafter starting
  • Scrum worked for the author's 40 person team
  • Team's quality was way below acceptable, but implementing Scrum solved their problems
  • He will use Scrum by default for new projects in the future unless there is a specific reason not to use it

1. Introduction

  • Scrum is not a magic bullet
  • You tailor it to your context
  • His team was fighting fires all the time
  • Quality was low
  • Overtime was up
  • Deadlines were missed
  • "Scrum" was just a buzzword to most people
  • Implemented across teams of 3 to 12 people
  • Learned about different ways of managing backlog (Excel, Jira, index cards, etc)
  • Experimented w/different sprint sizes (2 to 6 weeks)
  • Used other XP practices like pair programming
  • Used continuous integration
  • Used TDD
  • Took Ken's certification course
  • Most useful info came from real "War Stories" and case studies of people actually solving problems with Scrum

2. How we do product backlogs

  • Backlog is hear and soul of Scrum
  • Backlog contains customer's desired features, prioritized by most critical
  • Call the backlog items User Stories or just Stories
  • ID, Name, Importance, Initial Estimate, How-To-Demo, Notes
  • 6 fields were most often used
  • Kept in Excel spreadsheet on shared drive with Multiple Editing allowed
  • Additional: Track, Components, Requestor, Bug Tracking ID if needed
  • Keep the product backlog at the Business Level not a technical level
  • Let the team figure out the How-To Technical Level
  • Ask Why as many teams as needed to get to the underlying intent if the Product Owner does state it in technical terms, and move technical language to Notes field.

3. How we prepare for Sprint planning

  • Product Backlog MUST exist first in ship-shape form
  • One Product Backlog and One Product Owner per Product
  • All items should have a unique Importance level
  • Leave large gaps. 100 does not mean 5 times more important than 20. Easier than making 20.5 if you did 20 and 21 instead when item C comes up in the middle
  • Product Owner should understand the intent of each story
  • Others can add stories, but only Product Owner can assign importance
  • Only the team can add an estimate
  • Tried using Jira for keeping the backlog, but too many clicks for Product Owner
  • Have not yet tried VersionOne or other Scrum tools

4. How we do Sprint planning

TODO page 25

5. How we communicate Sprints

6. How we do Sprint backlogs

7. How we arrange the team room

8. How we do daily Scrum

9. How we do Spring demos

10.How we do Spring retrospectives

11.Slack time between Sprints

12.How we do release planning and fixed priced contracts

13.How we combine Scrum with XP

14.How we do testing

15.How we handle multiple Scrum teams

16.How we handle geographically distributed teams

17.Scrum master checklist

18.Parting words

Recommended reading


Saturday, May 24, 2008

Getting Real, Release It!, and The Perpetual Beta: Modern Web Apps

I've been thinking an awful lot lately about web application architecture and release strategies. This has a lot to do with a project I am working on professionally, but it ties into everything I've been thinking about regarding architecture in general.

Here are some links for reading more about these topics and insights from some of today's top developers, including 37signals.com, the creators of the Ruby-on-Rails framework and developers of http://www.Meetup.com

Getting Real, by 37Signals

Learn about 37Signals by googling them. They are the creators of the Ruby-on-Rails framework that has pushed a lot of other people to higher levels of quality. Only a true Microsoft fanboy would not know that tons of what is inside of Microsoft's forthcoming ASP.NET MVC framework comes straight out of the Ruby-on-Rails framework.

Getting Real is a book by the 37Signals crew about how they do Agile development.

Read it here: http://gettingreal.37signals.com/index.php

Here is a key excerpt from chapter 2 in the essay entitled Fix Time and Budget Flex Scope

"Launching something great that's a little smaller in scope than planned is better than launching something mediocre and full of holes because you had to hit some magical time, budget, and scope window. Leave the magic to Houdini. You've got a real business to run and a real product to deliver.

Here are the benefits of fixing time and budget, and keeping scope flexible:

  • Prioritization

    You have to figure out what's really important. What's going to make it into this initial release? This forces a constraint on you which will push you to make tough decisions instead of hemming and hawing.
  • Reality

    Setting expectations is key. If you try to fix time, budget, and scope, you won't be able to deliver at a high level of quality. Sure, you can probably deliver something, but is "something" what you really want to deliver?
  • Flexibility

    The ability to change is key. Having everything fixed makes it tough to change. Injecting scope flexibility will introduce options based on your real experience building the product. Flexibility is your friend."
Comments
I really like these ideas with regard to web applications. These principles become extremely important when you are migrating a system from an older technology to a new technology, especially when the benefit you seek is primarily for the application infrastructure as opposed to for the end-user benefit. You have to ask yourself questions like:
  • How does this migration actually benefit the user?
  • Is is tested well enough to replace the existing system such that users do not notice the change?
  • Is there a way to minimize any possible damage the new infrastructure could cause to profitability should it not work as hoped?
As a consultant, you have to make clients aware of these questions. Your job is to inform them about possible problems and strategies for risk mitigation. Ultimately, they may select to do something you either agree with or don't, but you have to due your diligence when you recommend a migration strategy.

The situation is more complex when you are implementing not just a back-end migration, but also adding brand new features that you want to introduce to actually improve the user experience.

Suppose you have this scenario:
  1. You have a client that operates a popular shopping web site
  2. The existing application is running ASP with COM objects written in Visual Basic 6.0
  3. The existing application works well and has been tested through real-world use for more than five years
  4. The existing application continues in increase in value, leading to higher and higher ROI each year
  5. Your client wants to migrate to Visual Basic.NET and ASP.NET 3.5
  6. Your client wants to add new features to the system that will increase the usability and utility for the system's users such as Ajax-enhanced search pages and RESTful URLs that offer strong Search-Engine-Optimization benefits
You have to carefully weigh all of these demands and criteria. Ask questions like:
  • How important is time-to-market?
  • How important is it that users do not have any interruptions in service?
  • What are the performance and scalability requirements?
Yes, Yes, and Yes
In my experience, most clients will want it as soon as possible and with as few interruptions as possible and with as good or better performance as the existing system. This is just a given.

But, you really cannot deliver all three of those concurrently. You have to make some trade-offs.

In this case, I believe the best migration strategy is what is called a vertical migration strategy. Read more about this in the Dr. Dobbs link below.
  1. Create the new foundational architecture in Visual Basic.NET to support the system
  2. Create Interop assemblies on top of the COM objects
  3. Create the new, value-added functionality first
  4. Bring up the system in beta side-by-side to the existing system so that the value-added features can be delivered to the users and so that the client realizes early return-on-investment (ROI) and gets early user feedback.
  5. Monitor the system's performance and refactor any of the bottleneck areas caused by interop by implementing them in pure .NET code.
  6. Add more features to the beta to slowly replace the existing application, getting user feedback and important performance and scalability information all the while to help direct your refactorings.
This is different from a horizontal migration strategy. In a horizontal strategy, you would select an entire layer of system, such as the UI, the Business Logic, or the Persistence Layer. A horizontal strategy is typically more complex and time-consuming and requires more testing.

However, you can use a very similar risk mitigation strategy to what you do in a vertical migration. You can bring up the new system side-by-side with the existing one and allow users to alpha and beta test it while you measure the usability, performance, and scalability and refactor as needed before it replaces the existing system.

You can read much more about the various approaches to ASP.NET application migration in this Dr. Dobbs online article: http://www.ddj.com/windows/184406077


Figure 1: Vertical Migration Strategy












Figure 2: Horizontal Migration Strategy



Release It!: Design and Deploy Production-Ready Software


Another book that I have eyed on shelves of late is called Release It!: Design and Deploy Production-Ready Software by Michael Nygard. Here is an interview with the author on InfoQ about the book and his lessons learned:

http://www.infoq.com/articles/nygard-release-it

Michael Nygard: First off, there's quite a bit of variation in what people mean by "feature complete". Even at best, it just means that all the specified functionality for a release has passed functional testing. For an agile team, it should also mean that all the acceptance tests pass. In some cases, though, all it means is that the developers finished their first draft of the code and threw it over the wall to the testers.

"Production ready" is orthogonal to "feature complete". Whether the acceptance tests pass or the testers give it a green check mark tells me nothing about how well the system as a whole is going to hold up under the stresses of real-world, every day use. Could be horrible, could be great.

For example, does it have a memory leak? Nobody actually runs a test server in the QA environment for a week or a month at a time, under realistic loads. We're lucky to get a week of testing, total, let alone a week just for longevity testing. So, passing QA doesn't tell me anything about memory leaks. It's very easy for memory leaks to get into production. Well, now that creates an operational problem, because the applications will have to be restarted regularly. Every memory leak I've seen is based on traffic, so the more traffic you get, the faster you leak memory. That means that you can't even predict when you'll have to restart the applications. It might be the middle of the busiest hour on your busiest day. Actually, it's pretty likely to happen during the busiest (i.e., the worst) times.

This is crucially important. Memory leaks come from third-party vendors just as often as they come from your own internal code. There is nothing like having to log in remotely to a web server when you're trying to have fun and the third-party component is causing your web server to hang. These are things that people rarely think about up front because they typically are problems revealed only by real system usage. I'll give a real world example:

Suppose you have a Visual Basic 6.0 COM object that use ADO internally. It may keep a RecordSet open to allow for consuming code to rewind the cursor and start over from the beginning. Well, .NET uses non-deterministic finalization, so you have to take care to call System.Interop.Marshal.ReleaseCOMObject to inform the runtime that it should destroy the COM object when you are finished with it. If you do not do this, you could end up with long-standing blocks against your database until the garbage collector frees the COM object.

I have run into this problem, and luckily was able to refactor my wrapper class in one single place to alleviate the problem. In the web application, we never rewind the collection, so it was safe for us to free the object after the while loop in the IEnumerable.GetEnumerator() completed.

As for this book, you can read the full table of contents and excerpts from the book in this PDF extracted from the book:

http://media.pragprog.com/titles/mnee/mnee-patterns.pdf

After looking at the TOC, I know this is a book I want to read.

Web 2.0 Applications and Agile Methodologies
This brings us into the territory of web 2.0 applications and the topic of the agile methodology.

The wikipedia entry for Web 2.0 defines the following key characteristics for a web 2.0 application:

"The sometimes complex and continually evolving technology infrastructure of Web 2.0 includes server-software, content-syndication, messaging-protocols, standards-oriented browsers with plugins and extensions, and various client-applications. The differing, yet complementary approaches of such elements provide Web 2.0 sites with information-storage, creation, and dissemination challenges and capabilities that go beyond what the public formerly expected in the environment of the so-called "Web 1.0".

Web 2.0 websites typically include some of the following features/techniques:

Suppose your task is to migrate a web 1.0 application to a web 2.0 application such that it increasingly resembles these features. How do you do that with maximal risk mitigation and protection of the existing system's ROI?

Taking Cue's from Yahoo and Google's Lead
First, from a process methodology standpoint, both Yahoo and Google have adopted an agile process. In particular, they have adopted a Scrum-based development methodology. You can watch the following videos to learn more about that:
Just to summarize, however, here is what Scrum looks like visually:


Verbally, the wikipedia article describes it as:
Scrum is a process skeleton that includes a set of practices and predefined roles. The main roles in scrum are the ScrumMaster who maintains the processes and works similar to a project manager, the Product Owner who represents the stakeholders, and the Team which includes the developers.

During each sprint, a 15-30 day period (length decided by the team), the team creates an increment of potential shippable (usable) software. The set of features that go into each sprint come from the product backlog, which is a prioritized set of high level requirements of work to be done. What backlog items go into the sprint is determined during the sprint planning meeting. During this meeting the Product Owner informs the team of the items in the product backlog that he wants completed. The team then determines how much of this they can commit to complete during the next sprint.[4] During the sprint, no one is able to change the sprint backlog, which means that the requirements are frozen for sprint.

There are several good implementations of systems for managing the Scrum process and the "sprints" while others prefer to use yellow stickers and white-boards. One of Scrum's biggest advantages is that it is very easy to learn and requires little effort to start using.

Why have both Google and Yahoo adopted scrum? Well, just listen to what someone inside of Yahoo had to say about this in the Scrum Mailing List.

http://shmula.com/159/scrum-at-yahoo

"What the Times doesn’t say is that Yahoo! is now 18 month into its adoption of Scrum, and has upwards of 500 people (and steadily growing) using Scrum in the US, Europe, and India. Scrum is being used successfully for projects ranging from new product development Yahoo! Podcasts, which won a webby 6 months after launch, was built start-to-finish in distributed Scrum between the US and India) to heavy-duty infrastructure work on Yahoo! Mail (which serves north of a hundred million users each month). Most (but not all) of the teams using Scrum at Yahoo! are doing it by the book, with active support from inside and outside coaches (both of which in my opinion are necessary for best results).

Pete Deemer Chief Product Officer, Yahoo! Bangalore / CSM"

Microsoft also uses Scrum-based methodologies for building systems. See this eWeek article here for details:

http://www.eweek.com/c/a/IT-Management/Microsoft-Lauds-Scrum-Method-for-Software-Projects/

Microsoft ASP.NET and MVC: A new direction
We see further changes in Microsoft's approach to development in the way they are releasing upgrades to ASP.NET and the MVC framework. Take a look at http://www.codeplex.com/aspnet. This is the new home for the ASP.NET framework, where Microsoft makes "often and early" releases to the developer community. They are still a few steps short of going fully open-source, but I think they will get there sooner rather than later. At least, I hope they do if they hope to survive in the competitive market.

The Perpetual Beta: A way forward that ensures quality

Finally, arrive at the idea of a perpetual beta. This is something that Tim O'Reilly discussed a few years back about the nature of a web 2.0 system. Read more about his comments here:

http://en.wikipedia.org/wiki/Perpetual_beta

and here:

http://www.oreillynet.com/pub/a/oreilly/tim/news/2005/09/30/what-is-web-20.html?page=4

His key point about the concept of a perpetual beta is this:
"Users must be treated as co-developers, in a reflection of open source development practices (even if the software in question is unlikely to be released under an open source license.) The open source dictum, "release early and release often" in fact has morphed into an even more radical position, "the perpetual beta," in which the product is developed in the open, with new features slipstreamed in on a monthly, weekly, or even daily basis. It's no accident that services such as Gmail, Google Maps, Flickr, del.icio.us, and the like may be expected to bear a "Beta" logo for years at a time."[1]
The Perpetual Beta Deployment Model in Practice

In practice, you could implement the Perpetual Beta model as follows:

Web Farm: production.samplesite.com
[ Production Server #1 - App v1.0 ] [ Production Server #2 - App v1.0]

Beta Server: beta.samplesite.com
[ Beta Server #1 - App v1.x ]

To follow Google and Yahoo's lead here, you would deploy release candidate builds to the Beta Server and allow users to select to use that version of the system, but they could always navigate to the production URL instead of the beta.

This provides you the following benefits:
  • Users become part of your testing process and help you develop and guide the evolution of the system
  • Your team becomes far less stressed and annoyed because they have a true metric of user satisfaction and system stability to gauge before changes actually go into the true production environment
  • You decrease your time-to-market by having a system that is continually improving
Cautions and Caveats to this process:
  • You have to take care in testing the beta build well in advance of pushing it to the beta server. Make sure there are no database concurrency issues or blocking processes that could cause the beta and the production system to conflict.
  • You should be exercising automated tests and using regression test tools well in advance of the beta deployment.
  • You will still not catch all problems before it goes into the real production environment. This is just the way development is.
Conclusion
These are just some of my notes and thoughts on the way forward for web application development. I know I'm already behind on most of this stuff, but some of my readers are further behind :-)

Is there a silver lining to all of this change and fast pace?

I believe there is. The silver lining is HTTP and URI.

If you build your URIs to last, last they will. That is the fundamental change in mindset that is taking place for most of the successful players right now. They are realizing that they can construct their services using RESTful designs that allow both applications and users to repurpose content and data in ways that nobody thought possible before.

If you don't believe me, just head on over to http://pipes.yahoo.com or read up on Google Base http://www.google.com/base.

URI stands for Uniform Resource Identifier. It's about time we started treating it like one. We've got to stop reinventing the wheel and start driving the car we have with the wheel we got already.

Check out my previous posts on REST or just read Roy Fielding's dissertation to learn more.

Related Resources