The first lean tendency

[This is part 3 of  The big elephant in the scrum meeting room] 

One of the things that you can do now and pay for later, is to bind resources to work on items before they are ready for the work to begin. This early binding didn’t accomplish anything in reality, and either limited, or at best imposed more hurdles for the team to jump over should things need to change. One of the things that we were looking for when we decided to look at agile processes was to be more flexible, thereby requiring empowerment of the team, which ultimately allows the team to get more done.

With our standard scrum framework, we are used to having a sprint planning meeting and selecting what stories we will include in the sprint. If the team has extra cycles, you may or may not like the idea of augmenting the sprint with extra work items. And, if production issues develop that take of 50% of your resources for 2-3 weeks, you could just declare the sprint a failure. Both of these circumstances aren’t rare, in fact they have actually been known to happen (with some regularity).

One of the reasons that these issues remain problematic, is due to the way that it has been mitigated. Scrum attempts to address this by defining a short cycle time (the sprint of 2-3 weeks) and therefore minimizing the window in which these types of things can happen. While that sounded like a good idea on the surface, a 2 week cycle time is 2 weeks longer than the 5 seconds it takes to have a serious production outage. The change, in fact, needs to be in the way work is managed, so that it is queued in just in time fashion. One way we can handle this is by implementing scrumban. This is what I did, and it was an important first step towards achieving true team empowerment, and realizing some of the greatest efficiencies of any team in a very large organization.

With scrumban, like scrum, we still work with velocity calculations. There are just so many man hours of work that can be performed in our time box allocated towards each development cycle. (iteration, sprint, release....). Weather you measure things in story points (if you use ‘user stories’), man hours, or something else – the effect is the same. There’s so much work that can be expected to occur in this period. As mentioned above, with our scrum framework we have already allocated our velocity towards stories to include in this scrum, and if we have mis-estimated anything, or external events occur to upset our plans, we have to decide how to handle this. Since these things are more common in the typical fast paced business environment than not --- it’s clearly better to have a framework that accommodates these issues. That is what managers focus on, is it not? We intend to manage our business with processes that support it.

With scrumban, we aren’t going to allocate our velocity for the sprint at our sprint planning meeting. All we really need to do is to ensure that the product owner and scrum master have more than enough stories developed and in queue to cover what the team could ever possibly accomplish. With work in queue and not bound to a resource, it’s a simple process manifestation to have the team pull from the top of the queue when they are ready to being their next task. This queue, we call the WIP queue.

One of the benefits of not binding tasks to people early on in the process is how we can be more accommodating of events that occur as the sprint progresses. Production issue? Just put it at the top of the WIP queue. Why would we let production issues cause us to do something inefficient like always leave a 25% resource under utilization to accommodate for these types of events should they occur? To make the rest of our processes work, we need to carve out headcount and set it aside to handle 'just-in-case' events? Your managing this? --- really? Or maybe we fail the sprint so that we can deal with these issues, which is even worse. It is sad, but a large percentage of teams that I talk to actually do this.

As leaders in our business endeavors, we facilitate process improvements through retrospectives to accommodate the business issues that we deal with. The process changes to suit our business environment --- THAT is scrum.This is one of the reasons we empowered our teams with the scrum framework to begin with…. Isn’t it? To fail to make this leap is failure itself.

I believe that moving to scrumban was one necessary step in a process evolution that allowed for some of the greatest process improvements. It is not a simple change to make, and your team will have to think through how to work with the product owner to have sufficient, prioritized, meaningful WIP queued and ready. If you are using kanban cards and a board, you will probably end up adding a few columns to your board. One common change would be to pull work from the WIP queue to a “ready” queue. I may come back to this later and define what we had implemented at this point, but you will need to utilize your teams to evaluate the alternatives and develop the best method for your company.

One other thing that has happened, which is pretty important, is that we have implemented a pull system. The developers are pulling work through the system as it as needed. They are doing so “just in time”.  They aren’t doing it too early and it isn’t happening too late. This… is a massive improvement. This is just the beginning of our lean improvements.

Improving Scrum with lean techniques

[This is part 2 of The big elephant in the scrum meeting room ]

Last time we talked about how Scrum has issues out of the box for the typical organization where the development team supports production issues. That posting is entitled "The big elephant in the scrum meeting room." It’s a pretty key problem that most any organization claiming to be using scrum have dealt with in one way or another, and it’s also the main indicator that the framework was developed without a default method of managing or accounting for central real life business problem(s).

Looking around you may have heard of “lean” techniques in software development. This took me a while to work through, as I associate these techniques primarily with manufacturing or assembly. The basic problem of delivering the correct parts at the correct time so they can be assembled into equipment as its ordered both streamlines production, and minimizes waste (overages, etc.) producing the right amount of components at the right time. It’s hard if you don’t have a technical background in software development to see how this applies to the development of complex software applications. But the more you look at it; you realize that the problem is exactly the same.

For manufacturing you might have a cog machine, a widget machine, a sprocket machine, and a lab where you assemble the components into your product. Your goal is to control the supply chain so that excess time isn’t wasted on any of the machines leaving you with too many parts, or conversely – to few parts, and to have them arrive on the assembly table about the same time as the order arrives.

With software development, it isn’t really any different. I have C programmers, Java programmers, GUI programmers, Android developers and in the Scrum or planning meetings the staff tends to accept tasks primarily along these lines. We all know how it’s going to work out before it gets very far. Each of these “development machines” needs to organize and plan their work so that all of the components are ready when it comes time for “assembly” (AKA Integration testing).

We get into this only because we are going to talk about one technique that better accommodates the real life business problem --- we want to continue to produce software for future releases, but in real life we are still going to have to support our existing production environments. There are things that will indeed come back from production that will consume cycles.

Before we can get to the final implementation, we have to work through an intermediate method – Scrumban. The superior concept of WIP (Work In Progress) and just in time are ,if you think about it, clearly more agile in nature than planning work for a sprint at the onset.

The next posting covers a previous implementation of Scrumban and is called The first lean tendency

The big elephant in the scrum meeting room.

It's almost impossible to talk with other groups that are using scrum without coming across the same conversations with relative frequency.

"How does your team accommodate for investigating, working and managing production issues?"

Most organizations employ their development teams as support resources for the production environment. This makes sense as the group that engineered the solution would be well suited to support it. If that is not the case, it would be a financial conundrum to maintain a separate team of individuals to support the deployed software on top of the development groups.

How do different groups handle this problem? Typically, I hear:

• We reserve a certain percent of our velocity measurement aside to deal with production issues.
• We allocate a certain percent of our headcount to deal with production issues.
• We maintain a separate backlog of bugs/issues/defects, and we employ one of the methods above to integrate these changes into (or around) sprints.

Here is what is wrong with any of those choices:

• We reserve a certain percent of our velocity measurement aside to deal with production issues.
• And therefore, we never achieve our true potential, because we plan for less work than the team is proven to be capable of. To try to remedy this, we then we employ methods X,Y, and Z to deal with those effects…..
• We allocate a certain percent of our headcount to deal with production issues.
• And therefore, you have created an ad-hoc team to deal with production issues that operates outside of the development team, and you are less efficient than possible. You now have a separate team, and probably inherit some complexities of people moving in and out of the two teams, and worst case you even employed new management for this team.
• We maintain a separate backlog of bugs/issues/defects, and we employ one of the methods above to integrate these changes into sprints.  
• I think this is the worst of all choices, because now you are extending the poor efficiency of the above two items with a new inefficiency of keeping another backlog or list, that the team and product owner needs to spend time grooming, analyzing, etc.

The real issue here is that scrum, for all the good that it does, doesn't seem to take into account the reality of the typical development life cycle of software that most of us deal with. Most groups engineer, develop and deploy applications, and then in the end --support them.

I solved this and a few other issues by building in lean techniques to my scrum framework, and will detail those out over the next series of posts.

Next is Improving Scrum with lean techniques.

See First Posting

Good Resources to learn about Agile and Lean

Here are what I think are some of the best resources for Agile and Lean software development.


One of the main things that you need to get right if your managing a scrum team is the retrospective. For a process framework where to goal is to build a self improving, accountable team -- the retrospective is the main tenant. After all, a team of competent individuals can start off making all the mistakes you could imagine, but with a productive set of retrospectives all of the issues would be corrected and incremental improvement can then be realized. I still feel like the majority of people I have talked to about agile or Scrum don't get it. Team development is key to agile process (and to lean as well), and this book is I think the best resource available today:

Agile Retrospectives: Making Good Teams Great , by Esther Derby & Diana Larsen


Continuous improvement is the main tenant of kanban. (This sounds familiar to those who truly have implemented Scrum...) and I think the topics should be viewed as properly intertwined. Lean scrum is well suited to my current work environment, and this book gives a great overview of kanban:



Kanban , by David J Anderson


 Roy --

See First Post
See Previous Posting

The true power of Scrum

Process improvement initiatives have always been a driver of constant business change. Over time, there have been numerous ways to approach the issue of implementing business improvements, but it feels like Scrum as a process improvement tool has been to a large extent, overlooked or at least miscategorized.

Scrum has generated plenty of excitement that executives at companies associate with producing software faster, better, or more reliably. This has successfully helped to kick-off many Scrum initiatives. Unfortunately, it seems a lot of implementations fall woefully short of what I feel is the true gist of the framework: A process improvement initiative. More succinctly, Scrum is a process improvement initiative implemented by an empowered, highly performing Scrum team. By empowered we mean the team has the ability to alter its process without management approval, or at least without to do so without too much bureaucratic overhead. 

The Scrum framework is a fairly loose framework, which enables it to easily be adapted to varying business scenarios. In fact, the Scrum framework only defines 4 meetings. Arguably, the most important meeting of the four is the “Sprint Retrospective”. This meeting is designed to discuss the course that has been taken, and to recommend continuous process improvements. By ensuring that the Scrum team follows this step, and that they have the power to implement changes, any chaos or variance in the project, process, or team can be corrected.

Having talked to several organizations using Scrum, its concerning how many don’t actually have empowered Scrum teams. This is unfortunate, and I will argue that it is in fact the whole point of the framework. Without a properly empowered Scrum team who practices the retrospective and implements improvements, key benefits of Scrum remain un-realized.

If you are a manager with a Scrum team,  what are some of the symptoms that your Scrum team isn’t providing your organization with the full benefits offered by Scrum?

• Does the team need your approval to modify its processes? If so - to what extent?
• Do you attend the retrospective?
• Do you approve changes to the teams Scrum process or framework?
• Do you approve their estimates?
• Do you question your team’s estimates? Have you ever worked to get an estimate lowered?
• Have you ever assigned a task to team members?

If so, your team is neither empowered nor is your company realizing the process improvement benefits of Scrum. If retrospectives aren't conducted properly or your Scrum team is not empowered, you have simply missed the boat.

A good read:

Agile Retrospectives: Making Good Teams Great

See Previous Posting
See Next Posting

Agile Solutions: Improving financial integrity

In many large corporations managing waterfall projects, the first hurdle is typically passing a capital rationing process. This often amounts to project justification through a business case, comparison of ROI, or other such metrics, against those of other projects competing for the same capital allocations. The process usually culminates with allocation of funds to complete the chosen strategic projects.

There are many shortcomings of this method and the unacknowledged compromises made during this process are many. In order to get an initial budgetary allocation, rough estimates must be made as to the cost and timeline of the project – quite often without gathering complete (or any) requirements for the project. While this may be useful for making rough comparisons with other projects competing for funds, it isn’t historically as accurate as it should be. Accordingly, with the requirements neither clearly nor accurately defined at this stage, these estimates are highly prone to error.  After the project is funded, earnest requirements gathering and assessment begins, and project manager(s) begin to determine the best path to “finish” the project in the promised time frame without going over budget. With the bulk of requirements definition, collection, and interpretation occurring after budget allocation, projects ultimately must make the compromise to produce the most effective features possible with the budget that was pre-determined before the project began. 

Project management’s approach is often what can be considered a “make the best of what we have” approach. Financial controls are usually centered on not going into an overspend situation, while balancing the fulfillment of enough of the requirements to accomplish the mission and therefore keep the customer(s) satisfied. Project management techniques, such as earned value analysis, have done a good job with controlling projects to ensure that the agreed upon deliverables are on track and will be met. Far too many project managers, however, fail to incorporate EVA into their projects.  While this credits the project management profession with developing effective techniques to maintain expenditures against projections (controlling), it hasn’t ensured that the bridge between the sales pitches made to get projects approved and the actual implementation are executed in a manner that encourages the elimination of unnecessary features. Typically, once projects are funded, the gap between what was promised in the sales pitch, what the customer expected, and what is actually delivered can be substantial.

It is somewhat intriguing as to why such a broken, error prone process like this is in use at so many companies to date.

Agile management practices

Corporate rationing is a necessary process. Companies need to be able to weigh the benefits of one project vs. another, and be able to make decisions on which project deserves capital investment – whether for financial or strategic reasons.  The capital rationing justification process, however, does not have necessarily need to be integrated with the capital allocation process.

With a great percentage of projects starting off with highly inaccurate estimates of the work involved, and a hefty percentage of projects failing during the first 1/3 of the projects life, it is no surprise that dealing with the trepidation of management in order to gain allocation of funds for these initiatives is not always a simple task. Often, the allocation of the entire budget for the project is made in one lump sum, which isn’t a prudent financial control.

With agile projects going through release planning as well as iteration (or sprint) planning, each release or even iteration should be able to stand on its own merit for budgetary allocation. With an agile methodology, accurate LOE’s are determined at the start of each iteration, and it isn’t until this point in time where we can say with any accuracy what each specific feature (user story) requested will actually cost. At what point is the aggregation of many features for a specific cost a better control that paying the actual cost of only features that are needed?

With Scrum, for example, a potential supplement  to the iteration or sprint planning meeting would be to present the features delivered in the next iteration as well as the cost. With the justification for the project out of the way during the capital rationalization process, this allocation is a point in time where company financial controllers can assess the benefits of each expenditure, as well as intervene if the last one (or more) iterative releases haven’t delivered as expected. This earlier view into project issues could lead to great savings for the large number of projects that fail or simply overrun.

An iteration rationalization meeting with your financial controller would also have a few other benefits:

• The product owner, as well as the entire team, has to at least note the cost of each feature. For example, if user story 34 is to create an operational report (call it 10-B), and the cost of this report is $10,000 - it may bring more attention to whether this story is an actual requirement or simply a “wish list” item. When waterfall projects are under-funded due to bad estimation, these are the things that are typically trimmed out because of lack of overall funding. Using this as an example, there could be many more cases where these items aren’t trimmed out because the funding is present.
• The product owner and the agile team can take the action to eliminate report 10-B if the data is available from other reports, for example, and money saved from requirements reduction should be a performance indicator for the team. With the financial control officer actually joining the product owner and Scrum team on a persistent basis, the Scrum team now has more control and visibility to financial accountability issues.
• Integration of a responsible party from finance, who could conceivably be the product owner or other person already on the Scrum team, increases the overall agility of the product organization as a whole, increases financial controls for projects, and if implemented correctly can bring great savings to the typical organization. The elimination of several layers of bureaucracy is implied.

 ·

The symptoms of a faulty process

Traditional projects run by a project manager tend to focus to managing the project so that it doesn’t go over budget, and I haven’t seen many projects where each line item (user story, feature, or functional requirement) were evaluated for worthiness. 

A Scrum or agile management approach puts the financial, business/product, and engineering teams working together to determine the effectiveness of each feature vs. the broad generalizations that were made during rationalization. With everyone on the team realizing the overall constraints, a good team will be more likely to reject unnecessary features. Estimates of capital spent on un-necessary features are very large. Estimates that 45% of features are never used highlight the broken financial and project controls that most companies employ blindly today. Reinforcing the study of how the pareto principle applies to software development, the Standish group has found repeatedly that in addition to 45% of features developed are “never used”, 19% are “rarely used”, and 16% of features are “used sometimes”.  With 20% of features being used “often” and “always” there seems to be little action or progress on quelling expenditures on un-necessary software features. How valuable is the $10,000 10-B report if it is “never used”? If it is “rarely used”??

An agile financial control mechanism.

An ideal environment would have an expedient process whereby releases or iterations of a project would be approved on a case by case basis. This presents additional challenges, but gives monetary control to the decision makers regarding continuation of projects based on feature value of future releases (I.E. Are we really going to spend another 100K for another sprint/iteration to get these 2 reports?).

Breaking up the capital rationalization process into two parts: the justification process and the iterative allocation process could be an effective way to deal with a variety of issues. While this presents new process issues for almost any business, it does have a few key benefits. 

• There is a smaller commitment by capital committees for each project instance. Supplemental increments are on a case by case basis. Financially prudent managers may find a smaller allocation more copasetic, which limits the liability of bad decisions made by proceeding with projects that are inundated with bad data.
• This process would be more in line with ensuring that features are of value for the cost.
• Hasty requirements gathering executed early on before the project actually initiates could be discerned sooner, as each iteration stands alone for delivery of features and the cost of doing so.
• This would allow proposed projects to actually compete with each other. Two strategic projects competing for funds in the justification process could both be initiated for a set amount of iterations to determine which is more likely to succeed. The superior project continues, while the other is held back and evaluated further.

 

This process mimics most real life situations whereby goods or services are delivered to any business. Whether it’s building a house or adding a swimming pool, payment is made at the end of a distinct “phase”. In the case of capital rationing for a business that is growing to be more agile, payment is made for progress that has been proven.

See Next Posting
See Previous Posting

-Roy

Uncertainty is certain: A case for agile development practices

It sounds simple: “You tell us what to build, and we will build it.” Since 1970 the “waterfall” model of software development has been used on the vast majority of software development projects. From an engineering perspective, this seems to be a fairly direct and natural approach to solving a problem. First we will gather all of the information, and then we can begin construction. Once the product owner has conveyed everything that needs to be known, the team should have the skills to build it. This implicitly lays the problem squarely at the product owner’s feet: Just make sure that you tell us exactly what you need, and make sure that we understand it. (and P.S. Make sure you get it right the first time.)

Any engineer can tell you a tale where the gist of a ‘problem’ begins when mid-way through the construction of the project, the product owner(s) either develops new requirements or clarifies existing ones.  Waterfall projects run through just a few phases.  The areas we are discussing are the requirements phase, and the development or construction phase. With the requirements phase, the engineering team needs to gain a thorough understanding as to what is required. With the subsequent construction or development phase, the team builds the application(s) according to their understanding. Several other steps may be involved depending upon organizational structure and requirements, but these two steps are the major focus areas addressed by our software development methodology.  

Due to budgetary constraints, each of these phases or steps will ultimately be “timeboxed”, meaning you have a set period of time to develop requirements, and a set period of time to get the work completed. Often this is an unspoken compromise, and this in effect often constrains the project to meeting as many of the requirements as possible in the specified period of time. Typically, corporate budgeting and capital rationing require determining the cost of the project ahead of the requirements gathering, which in itself is a conundrum. With the cost set, and the functional requirements not completely thought through, all of these projects inherit some amount of un-quantified risk from the outset.

One of the key differentiation points with agile methodologies that may not be immediately apparent is the amount of time the product owner and engineering teams spend analyzing requirements. The engineering team spends the entirety of the requirements phase developing an understanding of the requirements, and also gets to clarify and develop their understanding through the construction phase as well. One key point is that the product owners had a much shorter time limit to develop their requirements, ideas and solutions : they were “timeboxed” into the requirements phase. One of the key benefits of agile, which I believe isn’t always expressed, is that the product owner will have the same opportunity to analyze, digest and understand the requirements of the platform as the engineering teams – for the entirety of the project. The idea embedded into the SCRUM approach specifically, works to enhance the job of partnering the requirements owner with the engineering teams for the life of the project. Working together throughout the course of the project on requirements refinement becomes the proficiency of the entire team.

Over the years numerous techniques have been developed to address the deficiencies inherit in the waterfall method. Some commonly seen deficiencies by phase are:

For requirements engineering:

·         The product owner may fail to enumerate requirements

·         The engineering team may fail to fully develop or understand requirements

For the development cycle:

·         The development team may introduce coding error

·         The development team may not meet established schedules

·         The development team mot create a solution that meets the requirements

While having participated in projects where the engineering team failed to fully understand the requirements of the product owner is certainly not a rare occurrence, there have been great improvements in this area through team-centric process improvement. 

Addressing shortcomings

Waterfall projects traditionally develop many of the same ad-hoc remedies to deal with issues imposed by the timeboxing of the requirements and development, errors in requirements definition or understanding, and even to accommodate for coding errors. We often perform one or more follow on releases to address these shortcomings. These can be in the form of “Phase 2” releases, “enhancement” releases or just subsequent versions of the project.  In doing this, these projects naturally follow the same waterfall process. This is repeated until everyone is satisfied, or we no long have resources available to do so. In our attempt to timebox different phases of the project, we planned on getting everything right the first time, without error. When we can accomplish this, it is a great success. Too often, however, product owners are left waiting for “Phase 2”.

Remediating the issue

One alternative approach is to simply remove the divide between the product owner and the engineering team. What if, through the entirety of the project, the product owner was an integral part of the engineering team? Project managers work towards getting the entire team motivated, headed in the same direction, and more importantly interested in the success of the project. If everyone were tied to the overall success of the project, rather than the deliverable for a particular phase of the project, would this lead to greater commitment from everyone for the duration of the project? Modern methodologies such as Scrum do just this. Instead of attempting to gather and digest all of the requirements at once, the team can now take them a piece at a time. Anything that is not yet  fully understood sits on the shelf while the team works through issues that are understood and can be managed. This accomplishes a few things:

• It allows understood features to be built and delivered. Whether the end user can make use of parts of the new functionality, or simply by getting features into testing sooner, progress is more easily discernable and monitored.
• It allows the product owner and engineering team to work through issues together. Immersing the product owner into the cycle, keeps visibility on issues that need to be ironed out with more detail in the requirements (user stories, perhaps), and just as importantly it helps the product owner to work through requirements issues that he may not be able to articulate in the first month of the project. Increasing the time allowed to articulate requirements from a beginning phase of the project to the life of the project will undoubtedly increase accuracy and overall understanding.
• It gets the whole team into a cycle of producing requirements, developing changes, and implementing them. With this repetition comes competency. Greater efficiencies will be developed in working through tough issues a piece at a time.

The difference that matters

With the waterfall method, we are laying the project out into distinct phases, where we time box requirements and development. We work to get everything right the first time, and when we don’t -- we followup this process with either subsequent phases or enhancement releases until the project requirements are satisfied. In this case, we are planning on no requirements changing over the course of the project and getting everything correct the first time.  With an agile methodology, such as Scrum, we are planning on things changing, and the complexities of the project are allowed to dissolve over the course of the project. The latter method simply manages what the first did not, change. By planning on requirements evolving over the course of the project, our methodology better reflects what we expect to happen and makes agile methodologies more consistent with our goal to manage the effort effectively.

-Roy

See Next Posting

(Moved 2010-12-20)