Notes from reading of “Agile Software Development with Scrum” by Ken Schwaber and Mike Beedle.
Scrum is a popular software development methodology today. The concept is simple – you have a backlog of work prioritized by the product owner. Development is done in sprints, and in each sprint, developers commit to doing a chunk of the backlog. Daily meetings are held to review what got done, what is being worked on next, and identify any roadblocks. During the sprint, no new work can be introduced into the sprint – it must go into the backlog and wait for the next sprint planning meeting. At the end of the sprint, the result is demoed to management, and the next sprint is planned.
Scrum is a simple methodology, yet it has some important characteristics:
- adds a cadence to development, which provides some structure – similar to how weeks do in our lives.
- very defined iteration cycles, which provide logical for feedback
- sets boundaries on roles – product owners are not dictating how implementation is done, and the engineering team is not trying to define the product
- what everyone is doing is very public. Transparency is powerful!
- emphasizes focus – define what needs done, and then focus on that and nothing else until the next sprint. This helps engineers keep from getting pulled in a dozen different directions.
- emphasizes commitment – the engineering teams states what they can do in the next sprint, and are expected to complete it.
Throughout the book, the “defined” and “empirical” process control models are contrasted.
- Defined process control model is appropriate for simple systems with very little noise.
- Empirical process control model must be used to manage and control complex (high noise) processes.
Scrum was created before the age of Github, Devops, and the widespread open-source development workflows we know today. So, there may be better ways to run projects today with modern tools. However, I think the book captures very well the problems common in software development organizations. The best solutions are all based on fundamental truths – good things happen when you have good people, some structure, freedom, transparency, commitment, focus, and regular user feedback.
Quotes
General
- “Work can and should be an ennobling experience.” So beings Scrum – Agile Software Development, one of the sanest and most practical books on agile software processes.
- Scrum represents a new, more accurate way of doing software development that is based on the assumption, that software is a new product every time that it is written or composed.
- Control is provided in daily scrums, and sprint review meetings.
- Scrum challenges practices and structures that get in the way of focused work.
- Scrum is a harness on complex processes, a constraint on complexity.
- Scrum demands the liberal application of common sense.
- Establishing an open, honest relationship with the customer is the most important aspect of Scrum; Scrum makes everything visible; it’s real hard to cover up incorrect expectations that you established with the customer.
- (regarding the words/colors on the front cover) If you’re like most people, your first impulse was to read “red, yellow, green…,” rather than the colors that the words are printed in - ‘blue, green, red…’ Noise has just interfered with your perception. When you look at one of the words, you see both its color and its meaning. If those two pieces of evidence are in conflict, you have to make a choice. Because experience has taught you that the meaning of a word is more important than the color of the word, interference occurs when you try to pay attention only to the ink color. The interference effect suggests you’re not always in complete control of what you pay attention to. Noise interferes.
- Noise in systems development is a function of the three vectors of requirements, technology, and people.
- Scrum implements an empirical approach based in process control theory. The empirical approach reintroduces flexibility, adaptability, and productivity into systems development. We say “reintroduces” because much has been lost over the past twenty years.
- I’ve found some things that improved my life, like always using the best engineers, forming cross-functional teams, and facilitating design sessions around white boards. These tactics all help, but without Scrum these projects were all eventually overwhelmed by the complexity inherent in systems development projects.
- Some people read about Scrum and think that it’s a cop out. They think: “Since everything is empirical, teams can reduce functionality or increase costs in order to meet goals! The system isn’t even a fixed thing since the Product Backlog keeps emerging and evolving. How do I know what to expect, how do I stop the project from slipping and going out of control? How do I stop the team from slacking off, from letting the organization and the customer down? I want to be able to set a date and a cost. Then I’ll hold the team responsible for delivering. If the team can’t deliver, I’ll contract to someone who can! This Scrum stuff doesn’t provide me with adequate controls!”
- These fears arise when someone doesn’t know Scrum controls a project through active management involvement. The traditional approach to systems development begins by defining the system vision and overall requirements. The development organization (or external contractor) estimates the cost, and the expense of the system is budgeted. The system is developed and implemented. The customer expects the system to deliver the business value he or she envisioned, but often this expectation can’t be verified for months after implementation. This is the “over the wall” approach to system development because there is little interaction between customer and development team.
- (The Psychological View of Scrum) I have said much about people interactions, but I haven’t explained what happens to people working in a Scrum team from the inside out. Scrum has different effects on people. For most, belonging to a Scrum team gets them to be highly focused, effective, cooperative and committed over time. However, there are also the scant few that don’t like Scrum, and that’s because Scrum always tells the truth about everyone.
- However, for the majority that do benefit from Scrum, their state of consciousness can be better explained by the concept of “flow”. Mihaly Csikszentmihalyi from the University of Chicago defines “flow” as the state of an individual having the following characteristics:
- They are working to accomplish clear goals. Both the Scrum Planning meetings, and the Daily Scrum meetings help define clear goals.
- They get immediate feedback about their progress toward these goals. The daily Scrum meetings give this feedback.
- They must use significant skill to achieve their goals. Scrum requires a balance of individuals with at least 50% of them to be experts, but Scrum also promotes fast learning, so skills transfer rapidly and effectively among the team members.
- They are in control of the work and have it in their power to accomplish it. Scrum assigns work from the Product Backlog and creates a Sprint Backlog that controls the work at all times. Also, Scrum allows the elevation and resolution of issues through the daily Scrum meetings. removing any impediments from the way of the developers.
- They can concentrate on the goals without being distracted. Scrum provides a comfort shell for developers where the Scrum Master acts as a firewall. They become deeply involved in the work. Scrum drives individuals to focus, commit and excel. They focus on the work and lose concern for themselves.
- They experience an altered sense of time.
- They consistently produce at high levels of accomplishment. Scrum allows developers to concentrate most of their time in developing software, and by doing so developers enter “flow” state.
- Scrum Values
- Commitment: Be willing to commit to a goal. Scrum provides people all the authority they need to meet their commitments.
- Focus: Do your job. focus all of your efforts and skills on doing the work that you’ve committed to doing. Don’t worry about anything else.
- One of the finest Scrum Masters I’ve ever worked with had a saying: “What’s that got to do with code?”
- Openness: Scrum keeps everything about a project visible to everyone.
- it is better to produce something than it is to pursue many alternatives, please everyone, and produce nothing.
- Respect: Individuals are shaped by their background and their experiences. It is important to respect the different people who comprise a team.
- Courage: Have the courage to commit, to act, to be open, and to expect respect.
- Although empowerment is a trendy word, most organizations are still hierarchical and authoritarian.
- I have proposed above that software development is the result of creating new products, and that by necessity it requires research and creativity. However, the commonality among research and creativity is knowledge creation.
- Scrum isn’t for everyone. But it is for those who need to wrestle working systems from the complexity of emerging requirements and unstable technology.
- First and foremost, the Scrum team values allow diverse team members to share information, trust each other, collaborate in tasks, and truly commit to deliver the system.
- Risk management and predictability view of Scrum:
- Risk of not pleasing the customer
- Risk of not completing all functionality
- Risk of poor estimating and planning
- Risk of not resolving issues promptly
- Risk of not being able to complete the development cycle
- Risk of taking too much work and changing expectations.
- However, to really understand the dynamics of self-organization, we must delve into Complexity Science.
- Daily Scrum meetings. Each and every day, the team reflects where it is, what issues it has and where it is going, and adapts accordingly. Scrum meetings ensure that everyone knows what everyone else is doing, promoting opportunities for mentoring or collaboration.
- Demo after Sprint. The Demo after the Sprint allows the team to give feedback to the customer on what the team accomplished, and for the customer to give feedback to the team. This allows the team to adapt according to the customer feedback and for the customer to adapt according to the work presented.
- End of Sprint and Sprint Planning meetings. Every Sprint the team reflects on what it has accomplished at the Sprint in the End of Sprint meeting, and reconfigures in the Sprint Planning meeting to develop new functionality.
Efficiency
- By having every member of the team see every day what every other team member was doing, we began to get comments from one developer that if he changed a few lines of code, he could eliminate days of work for another developer. This effect was so dramatic that the project accelerated to the point where it had to be slowed down. This hyper productive state was seen in several subsequent Scrums but never so dramatically as the one at Easel. It was a combination of the skill of the team, the flexibility of Smalltalk, and way we approached production prototypes that evolved into deliverable product.
- Furthermore, in the hyper productive state, the initial Scrum entered the “zone”. No matter what happened or what problems arose, the response of the team always was far better than the response of any individual. It reminded me of the stories about the Celtics basketball team at their peak, where they could do no wrong. The impact of entering the “zone” was not just hyper productivity. The personal lives of the people were changed. People said they would never forget working on such a project and they would always be looking for another experience like it. It induced open, team-oriented, fun-loving behavior in unexpected persons and eliminated those who were not productive from the team through peer embarrassment.
- Emergence is another way of saying that the whole is greater than the sum of the parts.
Theory
- I wanted to understand the reason why my customers’ methodologies didn’t work for my company, so I brought several systems development methodologies to process theory experts at the DuPont Experimental Station in 1995. These experts, led by Babatunde “Tunde” Ogannaike, are the most highly respected theorists in industrial process control. They know process control inside and out. Some of them even taught the subject at major universities. They had all been brought in by DuPont to automate the entire product flow, from forecasts and orders to product delivery.
- They inspected the systems development processes that I brought them.** I have rarely provided a group with so much laughter. They were amazed and appalled that my industry, systems development, was trying to do its work using a completely inappropriate process control model.** They said systems development had so much complexity and unpredictability that it had to be managed by a process control model they referred to as “empirical.” They said this was nothing new, and all complex processes that weren’t completely understood required the empirical model. They helped me go through a book that is the Bible of industrial process control theory, Process Dynamics, Modeling and Control [Tunde] to understand why I was off track.
- He was particularly amused that the tasks were linked together with dependencies, as though they could predictably start and finish just like a well defined industrial process.
- During my visit to DuPont, I experienced a true epiphany. Suddenly, something in me clicked and I realized why everyone in my industry had such problems building systems. I realized why the industry was in such trouble and had such a poor reputation. We were wasting our time trying to control our work by thinking we had an assembly line when the only proper control was frequent and first-hand inspection, followed by immediate adjustments.
- Adaptation and Natural Selection
- It is interesting to see what researchers in Complexity Science say about organizations that live close to the edge of chaos [Kauffman93]:
- There is an optimum of adaptability in systems that self-organize right at the edge of chaos.
- Natural selection chooses configurations that are more apt to adapt. Co-evolving systems (ecosystems) whose members have tuned their structures to live close to the edge of chaos live longer.
- Since Scrum teams live closer to the edge of chaos this means that:
- Scrum teams are more adaptable than traditional teams that organize with defined organizations, defined processes and defined roles.
- Scrum teams are able to survive longer i.e. stay in stable configurations for longer periods of time, because adaptability selects them to live longer.
- Scrum teams co-evolve better and longer with other teams that have similar structures. For example, if in addition to the application development teams the business organization uses Scrum and other adaptive techniques to organize itself, the whole ecosystem will co-evolve and live longer. This explains why Scrum works so well with business organizations that use adaptive methods, and with software teams that use agile methods like XP.
- In fact, all living systems are self-organizing systems but not all self-organizing systems are live systems. In that sense, a software development team is described as being more agile or more “alive” as its ability to adapt increases. Scrum rates very high in adaptability. Why? Throughout this book we have learned that Scrum doesn’t use very many hierarchical controls of management, processes or even role definitions. The organization pecking order inside a Scrum team is really based on knowledge relationships: whoever knows more about a subject takes a higher pecking order in a discussion. The development tasks are reorganized dynamically, minute-by-minute in developers’ collaborations, day-by-day in Scrum meetings, and Sprint-by-Sprint in Sprint planning meetings, so there aren’t any static process definitions. Instead tasks, like organizations, are arranged dynamically using short feedback cycles. Finally, the roles of developers on a Scrum team also adapt in short time cycles. Where more traditional organizations call for the definition of static roles, Scrum developers can change hats dynamically, often passing as analysts, testers, coders, designers, architects and integrators in a single day. Simply said, everyone does everything that is in his or her power to deliver the system.
Sprint Meetings
- (Sprint meetings) First, only the team members were allowed to talk. No one else could talk. If you weren’t on the team and you wanted to attend, you had to stay quiet. Second, the team was only allowed to talk about three things - what it had done since the last meeting, what it was planning on doing before the next meeting, and what was impeding its work.
- Team members should keep their responses to these questions brief and to the point. They shouldn’t elaborate or describe how the work was done or will be done unless they want to highlight help they may need. For instance, a team member may report that he or she intends to complete implementing a feature in a module, but he or she is having difficulty understanding how a specific algorithm works. Or the team member may report that he or she is going to check in some code but can’t get the source code management system to work without crashing.
- The Daily Scrum is not a design session and should not turn into a working session. Don’t discuss design or start to solve a problem. There isn’t enough time or flexibility in the Daily Scrum to begin working through issues of this magnitude. By limiting the meeting’s scope, the Scrum Master can keep the duration in check and constant. If the scope of the Daily Scrum expands, no one will know how much time to allocate to the meeting.
- If any discussion is needed other than the status provided by answering the three questions, a follow-up meeting may be requested. After a team member gives his or her status, another team member can interject, “I’d like to address this more after the Scrum. Anyone who’s interested should hang around afterward.” More than one team member may want to address the topic in more depth.
- Scrum meetings do much more for a team than just capturing information. They don’t only make everyone capture what they did, but it makes everyone say it in front of everybody else. That way everyone listens to what others are doing and they can offer to help them later. They don’t only make everyone say what the issues are, but it makes everyone say it face to face to their management. This forces everyone to have courage and to be honest, and gives everyone a tool to put pressure on management about resolving issues. It also makes everyone promise in front of everyone else what you will be working on next, so it puts everyone’s credibility and trust to the test. Scrum is about deep social interactions that build trust among team members.
- At the Daily Scrum, the team is asked if there are any impediments to performing its work. This question is not regularly asked in most development projects, and these projects accumulate impediments like a ship accumulates barnacles, until there are eventually so many impediments on the project that it is hard for the project to make any headway. The most formal impediment identification mechanism I’d heard of before Scrum was called the “project post mortem.” At the post mortem, after the damage is already done, management met with the team to identify what could have been done differently. Too little, too late.
Engineering Team
- The size of the team should be seven people, plus or minus two [Miller]. Teams as small as three can benefit, but the small size limits the amount of interaction that can occur and reduces productivity gains. Teams larger than eight don’t work out well.
- There are no titles on teams. Teams self-organize to turn the requirements and technology into product functionality. This type of stateless, ego-less, development team is flexible to address any work that arises. Scrum avoids people who refuse to code because they are systems architects, or designers. Everyone chips in and does his or her best, doing or learning how to do what is needed. Scrum Team members don’t have job descriptions other than doing the best possible. No titles, no exceptions.
- If the team doesn’t report any problems with the daily build at the Daily Scrum, I inquire why not. Daily builds always have problems. I sometimes find out that there is no daily build process. I view the absence of a daily build as dangerous. Without it, the team isn’t required to synchronize its code. Without the daily build, the team might not know that the code doesn’t compile cleanly. Without the daily build, there is no way to test the product daily.
- The team-based “All-at-Once” model was based on the Japanese approach to new product development, Sashimi and Scrum. We were already using production prototyping to build software. It was implemented in slices (Sashimi) where an entire piece of fully integrated functionality worked at the end of an iteration.
- As a Scrum Master, I’m often tempted to help a team resolve its internal problems. Experience has taught me not to. The team has committed to a goal. When I help them resolve differences, I’m taking some of their responsibility away. The team committed to the goal; the team gets to figure out how to meet the goal, as best they can.
- … tasks are the detailed pieces of work needed to convert the Product Backlog into working software. Tasks should have enough detail so that each task takes roughly four to sixteen hours to finish.
- A team is let loose for the thirty day Sprint. The team has committed to the goal and accepted the responsibility of building a product increment that meets the goal. It has the authority to act as it sees fit.
- The team is required to deliver a product increment at the end of the Sprint. Daily product builds are an excellent way for the team to measure its progress. Prior to the build, the team should update the test suite and follow each product build with a smoke, or regression, test. Performing code check-ins for the builds is also a good idea, as it improves team communication and coordination.
- Scrum asks people to try to wrest a predictable product from unpredictable complexity. Some people can’t handle this type of assignment. During the Sprint, they may decide that they want out. Other people relish the chance to build something that requires their best effort. Those who succeed at Scrum are the individuals that will form the core of an organization. Scrum helps identify these people.
- Reusability
- However, don’t forget that your primary concern in the first application is to deploy it to production. You can always refactor and repackage components for a higher level of reuse later after the release to production. So don’t get too hung up in trying to make everything reusable from scratch it just doesn’t work very well to foresee everything. However, do use every opportunity you can to set things up so that reuse will be easier this should always be a secondary concern in your first application.
- Very many developers and managers that work with new technologies touted as “reusable technologies” are accustomed to always working in the first release of an application. They tend to think that this is the most interesting part of development. The contrary is true - the interesting part of “reusability” starts to happen in the release of a second application and beyond.
- My advice is simply: don’t try to reuse anything that is not stable enough.
- First and foremost, don’t try to get multiple units to do parallel development from day one. Parallel development, as said earlier, is difficult to accomplish, and while it apparently accelerates development, it can also bring many problems on its own. Instead, first develop a minimal application that cuts through all the layers of the architecture using the techniques shown in this book, even if some of these layers are compromised.
- Don’t even try to do parallel development if the configuration management, testing, and release processes are not in place.
- Stop worrying about documentation. Document the system after you go to production.
Review/Management
- Sprint review: During the meeting, everyone visualizes the demonstrated product functionality working in the customer or user environment. As this is visualized, consider what functionality might be added in the next Sprint. The product increment is the focal point for brainstorming. For example, someone could suggest the following after seeing the product increment demonstrated: “If we did “controlled patient costs” manually, we could use this right now in registration!” or “This would solve the problems that we’re having tracking inventory in the districts. What would we have to do to make this work off the inventory database?” As the team demonstrates the product increment, it helps the attendees understand the weaknesses and strengths of the product increments, and the difficulties and successes it experienced pulling it together.
- No one should prepare extensively for the meeting. In order to enforce this rule, PowerPoint presentations and their ilk are forbidden. If the team feels that it has to spend more than two hours preparing for the meeting, then it usually has less to show for the Sprint than it had hoped, and it is trying to obscure this fact with a fancy presentation.
- (regarding the review meeting) The Steering Committee is now starting to understand empirical management. It is seeing what is really happening every month and making appropriate decisions. The Steering Committee is able to guide the project month-by-month based on results, not promises.
- At Sprint Reviews, the Scrum Team demonstrates to management what it has been able to produce. The team demonstrates the real, executable product functionality that it has built. Maybe the team has also developed models, designs, architectures, and use cases. These artifacts are only useful to the degree that they’ve guided team thinking. What really matters is the product functionality delivered. You can’t ship a design.
- Management’s job is to manage the four variables of cost, date, functionality, and quality as development proceeds. Management helps the customer tradeoff one variable against another, while still meeting their objectives. Sometimes management and the team can deliver on all four variables. More often, management has to intelligently and openly negotiate and make tradeoffs between the four variables with the customers.
- The Steering Committee is now starting to understand empirical management. It is seeing what is really happening every month and making appropriate decisions. The Steering Committee is able to guide the project month-by-month based on results, not promises.
- These comments reflect a dilemma in the software industry. We systems professionals know that tradeoffs have to be made, that everything can’t be known in advance. Our management wants predictability. We usually wind up telling our management what they want to hear. Is that so bad? Management doesn’t know, and it pretty much gets what we said, unless a real catastrophe occurs. What we lose is our management’s participation, knowledge, guidance, and wisdom. If management understood that we were making tradeoffs, it could participate and collaborate. There would be no surprises. By telling management that we can deliver exactly what we say, we’re setting management up for the big surprise.
- The customer is paying for the product. The product isn’t a fixed entity. It’s a tradeoff between the money the customer wants to spend, the business value that they want to get for the money, the date on which they feel they need the product, and the expected quality. Anyone who has had a house built for them knows this type of negotiation. They know that the negotiation doesn’t just happen at the start of the job, it happens throughout the job. Building software is a lot more unpredictable than building a house.
- The Scrum Master had never worked anywhere that didn’t provide either inexpensive or free coffee to its developers, so the Scrum Master was baffled by this oversight. The Scrum Master figured he would remedy this immediately, because it was a no-brainer, a slam-dunk, something with impact and little effort. The Scrum Master called the facilities manager and told her that the team needed coffee, and what would she recommend? She indicated that the company had coffee previously, but everyone griped about it. She had gotten tired of their complaints, and now there was no free coffee. The Scrum Master was aghast, and protested that if the team wanted to take caffeine to stay awake and write alert code, the least the company could do was provide the caffeine for free. The facilities manager retorted that this was a family company and it didn’t want to encourage indulgence in drugs that kept the engineers away from their families. Then she went ballistic and informed senior management that Scrum was undercutting company values!
- Whew, that was a workout just to get coffee. Eventually, coffee worked its way back into the organization, but each time with reference to when the Scrum Master got his head handed to him over a cup of coffee.
- But how does “inventory management” relate to software development? Well, any resource can be seen the same way including “developer’s time”, because whether companies use it appropriately or not, they have to pay either salaries or consulting fees. The virtues of Scrum from this perspective, are that Scrum provides very many feedback cycles at different levels of scale: Measuring all resource inventory levels constantly and changing fast in small amounts how developers use their time.
- Simply said, Scrum is the solution of the beer game applied to software development because Scrum makes it impossible for developer’s time to go to waste or for issues to impede development. Here developer’s time can be seen as “inventory” and issues impeding development can be seen as lack of inventory of another resource. For example, if a testing environment is not available, this can be seen as not having enough “testing environment inventory.”
- Software as well as business organization have been seen in this light before. For example, Jerry Weinberg, one of the greatest software gurus, bases most of his analysis in system dynamics archetypes [Weinberg). Peter Senge, from MIT’s Sloan school has also developed notation and techniques to document people interactions through system dynamics techniques [Senge].
- One of the most important responsibilities of managers is making sure that productivity remains as high as possible. Scrum provides management with a daily glimpse at productivity levels and the factors that are affecting them. It gives them a sort of managerial cheat sheet: a list of ways in which management can increase productivity. What an opportunity! Here is a chance for management to both remove the impedances forever and to help the project succeed. Every time managers hear the word “impediments,” they should think “opportunity.”
- The New New Product Development Game
- referenced in Agile Software Development with Scrum
- Nonaka and Takeuchi, in their famous Harvard Business Review article [Takeuchi and Nonaka], show how innovative companies create new products. In their analysis of ten of the most competitive and innovative companies on the planet they found that they require:
- Built-in instability
- Self-organizing project teams.
- Overlapping development phases
- Multi-learning
- Subtle control
- Transfer of learning
Comparison to other systems
- The book Wicked Problems, Righteous Solutions reviewed the reasons why the waterfall approach to software development does not work today. Requirements are not fully understood before the project begins. The user knows what they want only after they see an initial version of the software. Requirements change during the software construction process. New tools and technologies make implementation s strategies unpredictable.
- The control process for defined processes is the pert chart. For a pert chart to work, work hours must be consistently estimated and measured. Otherwise, every process has noise and the cumulative noise and inaccuracy sinks the whole project plan.
- I single out pert charts as a significant problem. Pert charts are useful to think through and model a sequence of activities. They are disastrous when used to control a complex project. Pert chart-based project control has led to systems development schizophrenia. Management thinks the pert chart models work in complex projects. It asks for progress reports based on the pert chart model. But the real work has very little to do with the pert chart. The work doesn’t follow the pert chart. The work goes its own way, rapidly evolving after the first task to adapt to the complex problems at hand.
- Throw out the pert charts, because Scrum requires much less tedious but much more involved management action!
- Guilt arose, and with guilt came apathy. Workers who do their best, but consistently fail to live up to their own expectations, eventually stop trying to do their best. It never seems to be good enough or appreciated.