Around that time I got a new contract with one of Sweden’s biggest travel companies as a “BDD coach”, which basically turned out to mean XP coaching and Scrum mastering together. It has been great fun and will continue into 2012. This assignment does not entail much programming but still lots of challenges. I am quite thankful for the switch and especially the timing.

This post is a summary of some of the things that I have observed and learned during this last year. Some of the things I knew before, but have been confirmed again. Other insights were new to me. Most of the things I mention are not technical in nature, because I think that would be less entertaining to read about.

It is next to impossible to transform a silo organisation from the bottom-up

This will come as no great surprise to seasoned agile coaches, but it may be worth repeating. You may certainly have a lasting, positive effect on the developers and teams, which is great, but on a higher level or on the bottom line you are actually not accomplishing much.

My attempts these last years ended up with something similar to “water-scrum-fall”. Requirements came from people in other departments, who were reluctant to participate. Operations were handled by an outsourced company, which gave them very little incentive to do much work at all. They were difficult to talk to and every delivery had to be scripted in minute details and with ridiculous lead times.

Not all large organisations are the same

I have a tendency to lump all large organisations into the same bucket (marked “dysfunctional”), but I’m not sure it has to be that way. Some large organisations have more distributed power, more collaboration and are generally more pleasant to work in than others. Of course, all large corporations contain dysfunctions and systemic wastes, but some seem to handle it better than others.

One of my customers has sister companies in all nordic countries, but there is no web czar to manage it all. Instead, there is a web council with regular meetings and coordination. And when there is a crises or an important decision facing them, they call a large meeting where multiple views and perspectives are heard until the tough decision is obvious to everyone. Much better.

Hiring people for their attitude opposes improvement

Most companies claim they hire people based mainly on competence and some on their social skills. This year I have encountered a company that does the opposite, they hire people mostly based on their attitude and expect to train them on the job. The upside of this is that the staff is generally wonderful, easygoing people. Each problem is viewed as an opportunity. In a crises, when they come together, they can move mountains.

One possible downside with this, though, is that everybody is a relations builder. This means that people will spend their days being nice and forthcoming to each other, which unfortunately also means that they will often hold back criticism, feedback and challenges. But that part is important at times. Where everybody is a “yes man”, there is also difficulty to challenge the status quo.

Recruiting managers internally is not all good

I always encourage recruiting people internally. I think there is just a bit of an advantage to have managers that actually know something about the work they oversee (strange, huh?). On the flip side, though, you may end up with people in positions where they are in over their heads. This year I have met architect astronauts who did not know the first thing about software development and never actually went to the IT department physically. I also encountered a project expert who had never heard that multiple deliveries during a project might actually be a good idea. It is both painful and embarrassing to watch these people trying to stay afloat.

Never get too comfortable with your position

Experience is invaluable but it can also be catastrophic. When you’re experienced consultant there is always a risk of underestimating situations. You have seen it before. You know what will happen. You count entirely on your personal abilities that have taken you through so many difficult situations before. You feel there is very little need to change. You know your thing and it always works.

Until it doesn’t. When things change quickly under your feet and you are busy talking and joking instead of listening and reflecting. Suddenly, it is clear to everybody else that you can’t cut it in this new world and you’re out on the street. This happened to one very experienced project manager I know.

Office seating is a radioactive topic

For some reason, changing the furniture or rearranging office seating seems to be the most risky decision any manager can take and to be avoided at all costs. I heard of one office seating rearrangement this year, where some people actually screamed insults at the people responsible for it. Other people simply refused to work, like little children. I have also failed on two occasions to get organisations to replace their furniture; unsuitable for agile practices like pairing.

The best excuse I heard was that the office furnishing was outsourced and, hence, could not be changed. Outsourcing office furniture seems reasonable since these products are basically commodities, but I think this conflates two separate things; the manufacturing of office furniture and office furnishing. The latter, being a context-dependent, creative service is not easily outsourced. By outsourcing furnishing, it became more important that all departments look and stay exactly the same instead of staff having facilities suitable for their type of work process.

Agile project management is basically an oxymoron

Agile project management is a very difficult and sensitive topic. Yes, most projects still need a project manager, even agile development projects. There may be valid reasons for this, for example the coordination of marketing efforts on a major release, but the two main reasons I see out there are not valid nor insurmountable.

The first reason is the way work itself is organised – in projects. The project model is powerful but is riddled with waste, sub-optimisation, and command-and-control thinking. Secondly, agile projects today typically live in non-agile contexts, which leads to a lot of synchronisation and negotiation between the two models. A truly lean/agile company would not suffer any of these flaws and, hence, need very little of project management.

Of course, there will always be a need for coordination, stewardship, and leadership, as in any effort, but the project managers I have met rarely understand their role in an agile environment.

The scope of budgetary madness is endless

I keep returning to the madness of budgets as a way of controlling spending in an organisation. Most consultancies are quickly aware of the importance of company budgets. I was positive, before the year, that we would see less of this, thanks to modern management movements such as Radical management and Beyond budgeting. However, from my point of view, it seems to be getting worse.

This year I heard of one instance, where budget constraints for next year were “met” by asking suppliers to invoice early, before the end of this year. Right, that’s 5% “saved”. I have been told horror stories of end-of-year budget corrections, where managers and employees spend much of their time during the last months of the year adjusting time reports by shifting hours from overdrawn projects to projects that still have “money left”, thereby removing the chance of actually learning something from all that time reporting. It’s just a game that everybody plays. I have seen one project suddenly put down because of “lack of budget” in a company that made a 949 Million EUR profit just the quarter before. On and on. When will the madness stop?

CQRS and Event sourcing

Technically, this year has been less rewarding. I re-started my Clojure effort, but failed to continue once again. I think I need a real assignment to really get into it. The TDD in Clojure course with Brian Marick was good, though. TDD in C# with NUnit and Specification by Example with SpecFlow are two areas where I have learned a lot, not having worked with .Net technology that much before. Most rewarding has probably been learning more about CQRS and Event sourcing, mostly from creative design conversations with project colleagues.

Personal failure: Avoiding giving feedback

Reading my work journal from 2011 it became clear to me just how I have avoided giving people negative feedback in delicate situations. It seems my basic strategy is just to put my head in the sand and hope for the best. I am not alone in avoiding these talks like the plague, but in my book that’s not good enough for the kind of consultant I wish to be. Feedback must be swift, private, and non-threatening. I resolve to practice this at least three times the coming six months.

I wish my readers all the best for 2012!

Let’s say that you are the new configuration manager at a small software house. The managers feel that their release management is conducted a bit “ad hoc” and they have brought you in to restore order.  You, on the other hand, Mr Big Shot CM, have been trained for years on how to perform professional release management at the big enterprise with lots of engineers. Real engineers, mind you, not fakey information ones. After you have assessed the situation you dictate that releases may only occur at most twice a year with a two-month stabilisation phase each. The developers shake their heads, but you are the professional so hopefully you know what you’re doing.

Let’s think about it, was this a wise move? Yes, this probably makes releases more regular and dependent (for a while at least), but is that really something the customers have requested? They will get their software later. The organisation will also have to take on the challenges of coordinating big batches of work and a lot more work going on. This is unnecessary work and costs money. So maybe the only thing you managed to create with your “professional” release process was a loss of customer and business value?

From both these examples we may draw a more general conclusion: Release problems occur when you misjudge the maturity of your development and operations process. You may think it’s less mature than it actually is, and mandate an unnecessarily long release period, which will lead to ceremony and unneeded work, or you may think it’s more mature than it really is, and mandate a unrealistically short release period, which will lead to instability and emergencies.

* * *

Instead, the length of your release phase has to be in synch with the maturity of your process. And the maturity, in turn, is correlated to your context. In a small, informal, risk-free environment, say an prototyping experiment, your process can and should be very “immature”. There is no point having a mature process. In a complex, formal, life-and-death context, say software for pace makers, your process has to be very mature. The problem is that people typically don’t know what mature looks like.

This can be illustrated as a graph of Ideal Release Period Lengths vs Process Maturity/Context Complexity below:

Let me explain. On the low end of the scale, you have hacking. There are lots of bugs but you fix them continuously, right there, in production. Very immature (maturity M is zero), but remember, complexity C is also zero. This is a small and unimportant setting and it’s fine. The ideal release period length (IRPL) is zero.

As things get more complicated, your process needs to mature too. More stakeholders and more servers demand it. More people are involved and you have to agree on how to do things. Both C and M increases (we move to the right). The IRPL gets longer. Of course, this is not only a good thing from a customer perspective, they have to wait longer for their software, but most companies accept this as if it were a natural law.

Some companies, in highly complex environments, will go to extremely bureaucratic procedures and in that situation, for their maturity, very long release periods are actually ideal (companies in the pharmaceutical arena come to mind). Some are even proud of having very long release phases, their judgement clouded by words like “professionalism” and “best practices”. This is the peak of the graph, which I hereby dub “The peak of release mediocrity”. It is mediocre because in reality it pays maximum bureaucracy for medium maturity.

These companies don’t realise that they are mediocre, because what is on the other side is protected by a sort of wall. It’s a wall of tradition, culture, management practices, and, most of all, of mindset. It takes real effort to jump over it.

At some point, in some organisations, teams do manage to break down the wall. What I think happens at this point is that these organisational heroes stop looking out for their own self-interest (long release periods is a kind of protection) and start looking honestly at what is best for the customer. It’s just that they start to view the world differently. They are determined to improve everything, everything from their understanding of user needs to the release process. And they do. They gradually improve. Process maturity M increases again. C, though, cannot get any higher and stays the same on the entire right hand side of the graph.

One day soon they realize that “Hey, we’re pretty good. We can take advantage of that and release more often without risking anything.” They are again finding that their IRPL has decreased. When they fully understand the benefits of this capability they get addicted to it. They fight like crazed cats to get that cycle time between releases down. As mentioned above, some manage to take it all the way down to minutes between releases. In effect, the same times as for hacking in production.

So, don’t kid yourself. Acknowledge where you are on the graph and strive to set your release periods accordingly. Of course, if you want to be great, strive to move past the wall of release mediocrity. This is where the disciplines of lean and agile development fit like a glove.

You are the lead developer of a team writing a new, enterprise-wide web application and you are finally getting close to release. The project manager has done what he could to push those last features in. There is just one week left for acceptance testing with stakeholders and the week after that, you’re supposed to be live. It sounds tight, but doable.

But there are problems up ahead. (You knew that, right?) Annoying bugs keep appearing. As the acceptance period starts, change requests arrive. Lots of them, actually. By Wednesday you have so many RFCs that you actually feel it would have been a great idea to store them in TFS instead of on index cards. (In case you don’t know TFS, that basically means you’re panicking.) Bug fixing is frustratingly slow. Some fixes seem to create new defects. By Friday, you taste the bitter fruit and delay the release “indefinitely”.

As you go up to the business guys the Monday after, you realise, by their frosty stares, that you are now persona non grata. They really hate you. Who can blame them, really? They bet their careers on you and promised their bosses that this new thing would redefine “awesome”. But it didn’t and now you’re done for. You think to yourself, “It’s not fair!” You just wanted to help. You did your best. How on earth did you end up in this mess?

Most developers recognise some variation on that tune (though perhaps not as dramatic). It’s not that uncommon. Many have been in optimistically short “stabilisation” phases that need to be extended for months and everybody’s frustrated. Let’s back up a bit and see if we can understand what’s happening.

If you haven’t noticed, agile development is really hot right now and one of the core tenets of agile ways is to be able to release new stuff really quickly. It is actually mandatory for software development agility, if you think about it. Imagine the opposite: If you couldn’t release fast, you wouldn’t be able to release new stuff very often, which means you would have batches and you couldn’t change your mind very often on what to develop, which would mean you’re not very agile. Hence, it’s mandatory. So, most agile teams try decrease the time between releases, including the time period from “code freeze” to production. Can we get it down to two weeks? One week? Some teams grow quite advanced, releasing new stuff every day, or even faster.

Of course, to savvy business people, releasing often sounds really appetising. The all-important “time-to-market” factor has been transformed from a threat to an opportunity. Even better, this buys more time to work on valuable functionality instead of doing release work. Any person with a business sense would pressure the development teams to have really short release cycles. Who wouldn’t? It makes business sense and, hey, that’s being agile, right?

"You want agile?"

What is often not considered is that to be able to release quickly you also have to be able to release safely. In other words, the probability of something being wrong with your software, or going wrong in the release process itself, has to be very low. Typically, this sophistication takes months or even years of effort to accomplish, depending on where you start. It implies great product leadership, direct line of sight to users, a high quality design, thoroughly tested code, and a highly sophisticated automation of the release process itself, including detection of malfunctioning versions with safe rollback. It’s quite the feat, actually, what these teams perform. It takes a lot of disciplined, focused work and a supporting environment to pull it off. There are no shortcuts.

Release problems, like those that appeared in the story above, occur when you really want to be agile, but you are not ready to pay the price for it; when you reduce the time period dedicated to release work to what you want, instead of what you are truly capable of. If the team is wrestling with untested, legacy code; if you haven’t validated your assumptions on how the software should work; or if your release procedures are manual and complicated, you simply need more time.

We can state this slightly more succinctly: You won’t be agile just by wishing you were. It’s not a gift, something you can get for Christmas. It takes dedication, discipline, and directed collaboration over long periods of time to succeed.

If you really want to go this way, and I think you should, I want to give you a mental tool to help you. Imagine that every morning, as you walk through the doors of the team room, Miss Grant, from the 80s show Fame, sits there with a defiant look. She just sits there waiting, until the noise dies down. All eyes are on her. Beautiful. Deadly. Suddenly she speaks to you with intense passion and precision:

“You’ve got big dreams? You want agile? Well, agile costs. And right here is where you start paying – in sweat.”

The first day of the LESS 2011 conference is over. I am sorry to report I am not ecstatic after the first day. There were some nice talks and some not-so-nice ones, which is to be expected, but generally I felt that the speakers spent too much time on discussing models instead of working hard on making their talk come alive and be interesting and understandable for their audience. As J. B. Rainsberger (@jbrains), a Canadian master developer, put it one of tweets during the day: “Speakers, please tell me stories. They entertain me more, and they help me understand more”.

There is much to say about the topics of the conference (Organisational Transformation, Lean & Agile Product Development, and Complexity & Systems Thinking), but tonight I am just going to settle on reflecting a bit about something that the last speaker of the day, James Sutton, author and software-systems architect, said in passing during his keynote. He mentioned that from what he had heard about management and company culture in the Scandinavian countries, they were much further along than in other countries. Basically, what he implied is that the these countries have a more sophisticated style of management. Less command and control and more people-orientation.

Are Scandinavian Bosses Better?

Is it true what James Sutton said? The answer is of course not that clear-cut. Yes, in Sweden there is a long running tradition in corporate life to achieve consensus for major decisions, but to what extent that is true today nobody really knows. Swedish managers have been ridiculed for this trait for decades now and I believe that if I were a Swedish manager I would feel the need to be a bit more “assertive”. Finnish managers were never really into consensus, I believe. And there are many silly bosses in Sweden, just like everywhere.

The other thing that normally comes up when talking about this is what is known in Sweden as the “high ceiling”. This means that it is fine to say whatever you like, even to your boss. In Sweden it is against the law to be fired for telling your boss she’s crazy. So yes, openness and transparency go hand in hand with Lean leadership style. However, there are many companies in Sweden today where this is just a faint memory. You have your boss. You see him or her occasionally. He doesn’t work with you and he doesn’t know your job. There is a distance between you both and you hesitate to criticise him and the company. This is certainly not the high ceiling culture and not uncommon in Sweden today.

And yes, collaboration and teamwork is big in Sweden, but also process and working efficiently. With all these management consultants toolheads doing “Lean” transformations, process is coming back “in a big fucking way”, to borrow a term from US pop culture. I feel the word “collaboration” has gotten a funky 70s smell in Sweden now. “We don’t have time for teamwork – results come first.” would probably not be a weird thing to say today.

Management Is Culture

Management literature, as well as other cultural influences, have long been geared towards the United States. American authors, actors, thinkers etc are very highly regarded in Sweden. If you invite an american to speak publicly, even if they have never heard of the guy, Swedes turn up in crowds. Most Swedes even try to speak English with a slight american accent. The language you speak is a clear indication of which culture is also the most highly regarded.

Given the above, I would think that it is still likely that the trends in management is to move towards a more western, traditional style of management. This implies more rules, more control, more security, more formality, and a bigger distance between the co-worker and the manager. I see a lot of this in the companies I visit or consult. I have seen very few companies that actually strive to simplify and get rid of strangling bureaucracy. And the EU sure isn’t helping.

I think is a vital to counter this movement now. Scandinavia is looking steadily towards the western shores, when we should have turned our heads towards the east some decades ago. I feel that Lean and Agile ways have an important role to play here.

I have been doing morning standups for many years now. For the most part I really like them. At their best, a team can find a peaceful place to talk honestly about the work, get a quick and accurate feeling of where they are and a feeling of being in the same boat. On the other hand, at their worst they can feel corny, pointless, boring, like reporting to a manager or just something we are supposed to do.

Why does it work sometimes and other times not? Mostly I think it is about the people, their relationships, and the company culture. A broken team conducts broken meetings. The facilitator is another important factor. For example, if she cannot guide the team in finding a fruitful path between skimming over problems and irrelevant detail, the standups will feel pointless or dull, respectively.

Finally, I do believe that the format of the meeting plays a significant role as well. Think of it this way: If you knew there was a format that would increase the likelihood of that great experience and minimise the time spent in the bad one, would you try to find it? I really want to urge people to experiment more with your standups.

Common Formats

Most people are familiar with the standard Scrum format. You go around the circle and each person in the team answers three questions: 1. What have you done since last time? 2. What are you going to do today? 3. Do you have any impediments to progress?

The great thing about this format is that it gets everybody talking, which builds team spirit. Also, it is easy to spot if someone is having a problem. The downside is that it’s more about the people than about the work. And it can get pretty time-consuming and repetitive for a larger team.

Another format, often called Walk the Board, has grown popular these last years. It moves focus from the people over to the actual work. It is adapted from Lean manufacturing and was designed to handle work covering many teams or larger groups of people.

This is the way it works: Instead of taking turns, the team comments on the stories that are currently active. For each story, the people working on the story, or someone taking responsibility of the story, helps everybody understand what is happening with it. It is especially important to focus on stuff that it is disrupting a rapid conclusion.

This way is much faster because the number of active stories should typically be much fewer than the number of people. If you are working really effectively, there should only be one or two active stories. Also, the conversation focuses on the work process and not whether Matt has a headache. The downside is that not everybody gets to speak except the most engaged or dominant people in the team. Smart, quiet people are left out. This way also makes it hard to spot if someone needs extra help or support to do their work.

A Form Experiment

I thought a bit about this and devised a new format – a combo format, if you like. My objective was to keep most of the good stuff and get rid of the bad parts. Here it is:

1. Walk the board. For each story, collaboratively talk about what has happened since last time. Unearth any snags, blocks or other impediments. Then, set our common targets for today.
2. For each one who has not spoken, let them tell us what they have been up to and what they are aiming for today. Look for ways to include their work on the board.
3. Ask: “Any additional information from anybody, which we would all benefit to hear?” Here, I may remind them about a coaching session today. Or someone shares there personal disasters of the morning and asks to be excused in advance for any foul temper.

Experiences

We have been using the format for about seven weeks now and it seems to work out pretty sweet. We talk about the work, everybody gets to speak, good information is spread around and it all takes 10 minutes or less for 6-7 people. Really, no meeting has been longer than 10 minutes.

The 2nd point has an additional purpose: It gives people like product leaders, operations guys, and representatives from collaborator teams a chance to talk at our meeting. In our current context, they are not part of the team full time and their work is not always visible on the board. This is not ideal, but if that is our reality right now, we need to adapt.

Any downsides? One thing I am not perfectly happy with is that question number 2 often turns into number 3, i.e. just information. It can be hard to see if these guys are making progress. Also, very few impediments are being reported, which I think is a bit of a bad smell. But we’ll keep trying and experimenting to see if we can iron out these wrinkles.

Please go ahead and try the format if you like. I’d love to hear from you if you do and what you thoughts are. But perhaps my main point is this: Don’t get stuck in a rut. Experiment with your standups until they feel just right – for you.

While I was at the conference I took notes during the talks so, for what it’s worth, you’ll find them below. I feel it is appropriate to apologize in advance for possible errors in this post; information flows very fast at a conference and it is easy to misunderstand.

Day 1

Tom Preston-Warner: GitHub Flavored Ruby

Gist: Some useful techniques we use at GitHub that may be useful to others.

Readme Driven Development is like User-Guide Driven Development for libraries. The basic idea is to write the README file first. To not confuse other users, call it “spec” or similar to start with. As soon as a new feature is done, copy that part of the spec to README.

TomDoc is not like RDoc or YARD. For humans, not machines. Read more: tomdoc.org

Semantic Versioning is a great way to number versions for public APIs. Numbering scheme: major.minor.patch.

• Major level: For changes that are backwards incompatible
• Minor level: For new functionality, big internal change, bug fixes (must be backwards compatible)
• Patch level: For bug fixes only

~>2.4 Pessimistic operator: Use any compatible version bigger than or same as 2.4 Read more: semver.org

Modularisation. Work as you would open source any part of your system.

Anthony Eden: API Design Matters

Gist: We should all think about APIs from the client perspective. You are an API designer.

5Cs for API design:

• Consistent (with itself or your other APIs), “economy of concepts”
• Clear in naming, descriptions, intent
• Convenient, easier to use than rewrite
• Concise, brief
• Complete

Bad example: HTTP::Net. Good example: Typheous, open-uri (uses File.open with block)

Book tip: “The little manual of API Design” by Jasmin Blanchette

Thorben Schröder: Bridging the Gap

Gist: Ruby and JavaScript are two great worlds that need to work together.

Problem: Integration will sometimes lead to duplication, e.g. validation code. Solution: Exctact common parts. Use at both places. Either JS from Ruby or vice versa.

For examples, to use JS from Ruby, check out:

• The Ruby Racer: To embed a JS interpreter into Ruby.
• CommonJS: To modularise JS
• modulr: To use CommonJS modules in Ruby
• rack-modulr: To use CommonJS modules in your Rack/Rails applications
• include_js: Use CommonJS modules in Ruby using Ruby Racer

Joseph Wilk: Limited Red Society

Gist: Red = Spec/Test not working = Work in Progress is a necessary state but we should avoid staying in that state for very long.

Being “in the red” is necessary but has bad implications. You can’t integrate, deliver etc. Spending longish times there is not good. Think of it as batch size. Therefore, we want to discover when and why we do this and learn how to improve.

Process: Limit Red -> Measure Red -> Visualise Red -> Learn

Limited Red. Tool posting data from test runs automatically to web site.

Coders Dojo: Katas with simple visualisations of red state included.

There are patterns for limiting red state, check out Industrial Logic, Joshua Kerievsky.

Recommended refactoring books:

• “Refactoring in Ruby” by Wake, Rutherford
• “Refactoring – Ruby Edition” by Fields, Harvie, Fowler

Joshua Wehner: Must It Always Be About Sex?

Gist: We have several diversity problems and we should fix that. Otherwise we will otherwise miss out on lots of customers, employees and simply human capacity in general.

Fact: 18% of CS salary to women (2009), down from 37% (1986). 1,5% of commiter in OS are women. Not only gender, also racial and age.

Project Implicit: Test to reveal biases.

Imposter Syndrome: Feeling of being revealed as a “faker”, often felt by people in minority.

Language matters, can introduce a bias just by wording.

The “No Asshole” Rule: People who do extraordinary things but demean others should be considered incompetent. They should not be fired, they should be fixed.

Nice rule: “Fight as if you’re right. Listen as if you’re wrong.”

People deemed to be “lucky” consistently introduce variety in their lives. Unlucky people do the same things every day. Is it luck then? Experiment: Count photos in newspaper. Unlucky people never saw add that state how many photos there were. They even missed big ads with monetary reward.

Recommendations:

• Reprogramme the “assholes”
• Anonymise applicant profiles
• Step outside your comfort zone, every day
• Be weird

Randall Thomas: Infinite Data, Finite Solutions

Gist: Bayesian statistics are preferrable to regular statistics for real-world problems. Statistics assume infinity and other unrealistic conditions. Bayesian statistics start with a guess and improves guesses based on empirical data.

A Bayesian Belief Network (BBN) is Bayes + DAG.

Weka is a project to create a BBN from data, e.g. a CSV file. Downloadable JAR, use with JRuby.

Joe O’Brian: Taking Back Education

Gist: University training in CS is broken. We need to find new ways to train people. Companies should do it themselves, because they are better at it.

Quote: “First book I give to apprentices is “Pragmatic Thinking and Learnning”.”

Most things we learn in education is never used. We are “business developers”.

Idea: Code School. Based on apprenticeship, older students mentorings younger.

Quote: “When was the last time you inspired someone?”

Day 2

Paul Campbell: In Search of “Mé Féin”

Gist: “Mé Féin” means myself. Make the world better by just doing good stuff. Only you know where you want to go. Don’t let the external world affect you.

This talk was basically a story on Pauls life, discovered through the lense of his last week. Each day was a part of his life. Full of fun references, for example:

• Awesome People Hanging Out Together
• Sleep sort
• Dear Photograph

Quote: “You never know when inspiration is goint to strike. Just be ready for it.”

Elise Huard: Actors on Stage

Gist: The actor model is a promising model for solving the concurrency problem.

Completely stateless, message passing between actors. An actor can send and respond to messages as well as create new actors.

The Ruby MRI does not have good support for it and Matz seems reluctant to develop it.

Elise made cunning use of photographs of silent movie stars interleaved with slides with quotes to create a beautiful presentation.

Jakob Mattsson: Beyond Ruby

Gist: Everything is dynamic – except Ruby.

According to Jakob, Ruby is not as moldable as it could be. Everything is not an object. Jakob wants to make a DSL in Ruby that looks like JavaScript or LISP. He calls it Blub.

Ryan Smith: Fewer Constraints, More Concurrency

Gist: To increase concurrency you may have to loosen some of your fundamental assumptions.

How much concurrency do I need? Amdahl’s Law: S = 1 / ((1 - P) + P/n). Speed increase is determined by how much of the computation must be performed sequentially (e.g. network calls, database calls). Number of processors (n) is not as important. Adding more processors to a 90% parallelizable process does not create a great speedup. We need 100% (or close).

What can I trade? Discovery in example application: Trade FIFO property of work queue. It was not vital for this application. Found: “Fuzzy FIFO”: Take any job above the top boundary. By doing this, more workers could work concurrently.

What does concurrent code look like? There are two different paths though code depending on state. In this case it was the NOWAIT part of an pgSQL query. It requires thinking but it’s not impossible.

Aaron Patterson: Mountain Dew and My Trail of Tears

Gist: Legacy code can make you cry but it doesn’t have to.

Pro tip: Use “ruby -w”, verbose turned on, warnings will be displayed.

Aaron is the only person both on Ruby core team and Rails core team.

Legacy Code contains knowledge and it solves something, but is often painful to work with. Aaron presented his own quasi-math “formula of tears”, to show how much he has wept when working with legacy code. For a single week it turned out to be two large bottles!

Techniques mentioned:

• Liskov Substitution Principle (LSP)
• Single Responsibility Principle (SRP)
• Method Extraction (or Extract Method)
• Object#extend: Insert module before ancestors
• Extract Object: attack huge classes, find groups of methods and fields
• Pass in delegate in constructor

Seams: Ways to modify execution without modifying the code, for example ruby -I to modify the load path.

Chad Fowler: “Legacy”

Gist: A legacy should be something we strive to achieve, not something generally loathed, but in software very little actually survives.

1st def of Legacy is inheritance, bequest, something beautiful. Examples: Beethoven, John Coltrane have left musical legacies, still noticable in music today. Kurt Vonnegut books, Gaudí architecture, Comme des Garçon clothing, Miro paintings.

Software projects often fail and have very low life-expectancy (like 5 yrs). Tough quote: “No-one will remember your work when you die.”

How do you create legacy software? Idea: Look at biology. Species survive because they are the fittest, most adaptive. Richard P. Gabriel has researched systems with trillions lines of code. Biology uses homeostasis, cell’s ability to regulate internal conditions using feedback loops. Small components and systems seem in common for oldest surving software as well. Maybe we should focus on building cells and systems consisting of cells?

Suggestions:

• Tiny components, as small as possible (cells)
• Kill and replace cells regularly, static means death
• Force heterogeneity, the environment will change
• Simple Interfaces, e.g. Unix pipes

Keavy McMinn: Must. Try. Harder

Gist: Really applying yourself is worth all the hard work – no matter for what.

1. The goal has to come from your heart. The French cheer with “Avec courage!”. The word “courage” actually means “from the heart”. You need to put it all out there. You will need the support of others.
2. Develop a process (and follow it). Set (flexible) goals. Break it down. Apply some stress – then recover. Reflect.
3. Practice. Don’t ignore the fear, try understanding it. Celebrate your achievements. Try to control what you can control and ignore the rest. Think positively. Practice consistently.

At the XP2011 conference in Madrid 2011, Laurent Bossavit gave a talk that I found very interesting, outlining some historic perspective on the decade-long agile movement. As one example, Laurent’s report that the conference proceedings from the NATO conference on software in Garmisch-Partenkirchen (in 1968) were censored was simply astonishing. This was the historic place where the term “software crises” was coined, which in turn lead to the creation of the, so called, Software Engineering research field. The censored paper was a brilliant satire of the whole idea and apparently it was simply removed from the resulting conference proceedings. But why? Well, gradually it was revealed that the hidden motive of the conference organisers was to convince NATO to fund research in this new field of science, which they obviously should lead. In their glorious small-mindedness, they must have felt threatened by having a paper in the proceedings mocking their beautiful idea. I guess I should not be surprised, given the human nature, but scientists? Censoring!? Since then, Software Engineering has entrenched itself at the universities of the world, producing massive amounts of papers but very little useful results.

At the end of the presentation, Laurent presented a call to action, where his focus was on science as a means to strengthen the agile movement. As I understood it, Laurent is convinced that science is needed to give us the hard evidence we need to make progress. For example, would it not be very nice to just say that research has proven, beyond doubt, that pair programming actually is better? Then we would not have to debate anymore. As much as I sympathise with Laurent’s goals I remain unconvinced as to the utility of science for this. My first concern is about repeatability.

Science Requires Repeatability

Science is about theory building. We construct more and more refined theories and models, based on observation and experiments. A theory is not valid until it has been corroborated by researchers in other places, which means that repeatability of experiments is a must. The important parameters must remain equal and produce equal or similar results. I will argue that this is simply not possible in the field of software development.

In software development, the following can be considered an axiom: You cannot complete the same work twice. You really can’t. If you use the same people but replace the assignment with a similiar assignment you have changed the nature of the work. If you use toy projects, intentionally created to be of the approximately same nature, then your tasks will be unrealistic. If you instead replace the people doing the work you will have a completely different set of experience, attitudes, skills, etc in action. OK, but what if you use exactly the same people and develop exactly the same software? Well, you can’t. The people who were in that first project are not the same when they take on the same work again. They have learned an incredible amount of useful stuff while developing the first project.

Software development is about learning and that inevitable fact prevents us from repeating experiments with the necessary scientific rigor. Dan North and Elizabeth Keogh have talked a lot recently about what they call deliberate discovery. In their reasoning, all indications point towards our own ignorance being the single biggest bottleneck in all development work. In fact, we often suffer from 2nd order ignorance, i.e. we don’t even know what we are ignorant of. Increasing the speed with which we learn about the problem and how to design the most useful solution to that problem within the boundaries set forth must therefore be vital. One common symptom of this is that developers, when asked about how long it would take them to redo the assignment they just did, typicallly answer somewhere between 20 and 40% of the original time. The rest of the time was “spent” on learning. You cannot step into the same river twice, as the saying goes, and people are about the same. You cannot use the same developers on the same assignment twice.

Agile has made a very valid and important point to our field, that the people on the teams are the most important factor for success. Alistair Cockburn has a lovely phrase for this: “People are a parameter of the first order in software development“. Since we will never have the same people twice then, consequently, we cannot keep that parameter constant. This will introduct a strong element of randomness into any study. My second concern here is context.

Context Is King

I believe that there is simply too much that is contextual in software development to enable us to draw strong, general conclusions. Our field is characterised by having surprisingly few general laws. Certainly, this is true for the people and social issues. For example, all teams have different problems. But even the “hard” practices, things that agile evangelists preach like refactoring or TDD, are absolutely not free from context. If you develop a one-off for a few days, for example a campaign site, would you bother doing strict TDD? For a product prototype? This proves that you can certainly do without TDD in some instances.

TDD is not “good” in itself even though I find it incredibly useful in most situations. To really prove the point, not even the projects that claim they use TDD exclusively don’t TDD everything. Behavior like logging, transactions, thread-safety, randomness are almost never designed using TDD since the test-driving does not affect the design. How could science then “prove” that “doing TDD” is beneficial?

Science or Evolution

So what could science give us? I believe that we should never attempt to find or believe hard data presented to us. As one example, what if a research study showed this: “Using planning poker increases accuracy of estimates of 23%”. Could you use that study to strengthen your agile adoption? I find that conclusion laughable. A similar study in some other place may come up with the exact opposite numbers (and they often do). I once looked at research on TDD and found a very fractured image. Some study claimed it increased quality but productivity went down. Another study claimed productivity went up but more errors were introduced. A 3rd one showed that productivity was about the same, but developers ran their tests more often. This is clearly ridiculous. This is research of little utility but for the graduate student.

We could perhaps have more human-oriented studies, like the ones in social sciences. We could have studies that observe and reflect on human behaviours, telling stories about human behaviour and outcomes. This would probably be interesting but I believe this is not what Laurent means when he calls for more research. Why? Because that kind of material will never convince the pointy-haired manager saying “Show me the research or I won’t condone agile practices here”.

But in there’s exactly the thing. I believe that trying to indulge these people, the laggards, by giving them what they want, is truly just a waste of time. Is convincing laggards really what we want to spend time doing? What would be the point? I mean, if we truly believe that agile and lean ways are more successful then, in time, the companies that use them will prevail and the other companies will eventually die off. If you don’t have the patience to wait for your management to “get it”, then change your job.

If we have enough trust and patience, time will tell who is right. I am afraid science will not help much but I have high hopes for evolution.

What Is the V model?

The V model (also explained here) is an extension of the sequential (“waterfall”) model of software development. Instead of showing the sequence of work tasks as a straight line as in the waterfall model, both ends of the process chain are bent upwards to signify higher levels of abstraction, different roles, and a correspondence between requirements, designs and tests. Looks pretty logical, right?

The V model

What Is Wrong with the V Model?

Historically, the V model was made with the waterfall model of software development in mind. It assumes long time-frames with phases progressing in an ordered fashion from requirements analysis to system design to programmering, and finally to testing. All descriptions of the V model talk about each node as a “phase”. Phases have more or less been removed from software development today, displaced by iterative or continuous models.

Philosophically, the V model assumes any problem can be solved by breaking it down into smaller and smaller pieces. This is the analytic view of any engineer. When the V model was created, most programmers were engineers and software engineering was in vogue. This idea is sadly flawed. First of all, not all problems can or should be solved this way. For example, any form of modelling or design job is really an iterative process of gradually discovering better (more useful) solutions. There is really no right or wrong. A top-down, divide-and-conquer attack is great for mathematics and engineering; less so for design. Secondly, only a minority of programmers are engineers today and they solve problems very far from engineering.

The V model implies an ordered sequence of steps, which is hard to reconcile with the daily realities of software development. Obviously, the person who designed the V model wanted development to occur that way because it appeals to a mind who likes order. But saying it is so does not change reality – it only misleads. Studies have shown how experts switch quickly between understanding the problem (analysis) and fleshing out a solution (design).

The V model divides responsibility strictly into roles, which creates waste. For example, the requirements analysis is performed by a business analyst or similar, the system design by an architect, and so on. This increases the divide between people instead of promoting collaboration. These last decades, there has been an increased understanding that to get good results, we have to work together. By grooming specialists with narrow responsibilities we will get more hand-overs, more misunderstandings, and more rework.

The V model is document-driven. It dictates that each development phase to the left result in some form of description, which can be used to design verification tests, which are then performed at the end, on the right-hand side of the model. We have gained the understanding that written communication is not always best. It has its advantages, but when you really want to spread a common understanding, face-to-face communication is superior.

The V model puts way too much time between specification and verification. In the V model, testing that requirements have been satisfied occurs quite some time after the requirements were fixed. Even unit testing is separated from programming of the unit, typically performed by another person than the developer. However, the sooner we can verify something, the faster we get feedback. In a process using feedback, that means quicker response times and quicker corrections. The last 15 years software developers have even managed to move the feedback before the module being developed and in that ingenious step merged verification and specification into one.

The V model puts far too much weight on verification testing (finding and removing defects at the end) and not nearly enough weight (none, to be frank) on preventing errors in the first place. Quality studies have shown that teams with great quality skills avoid many of the defects that end up costing a lot more if found at the verification stage. Examples of practices that promote this are pairing, specification-by-example, and test-driven development. There is a quote from Edwards Deming related to this:

“As Harold S. Dodge said many years ago, ‘You cannot inspect quality into a product.’ The quality is there or it isn’t by the time it’s inspected.”

Finally, the V model is too limited in scope. It assumes there are some omniscient people on one end with a perfect view of what the customers will need and those assumptions are never verified. We only verify that the requirements, as described, are satisfied, which is clearly not the same thing. The growing awareness of the role of the customer (or user) in product development has shown us that leaving them out of the loop is not a clever idea. We have realised, from user-centred design and user experience fields, that business value appears when someone actually uses a feature, never before that. From Toyota Production System we have the “pull” model, where customers are though of as pulling new products out of the factory. There is even talk among management consultants that we are entering a new era of “consumer capitalism“. The buyer is in control and companies that delight the customers will be successful. To summarize, we need continuous feedback from our customers and stakeholders.

Re-interpretation of the V Model

Now, there are probably readers who feel I have been too harsh on the V model in this article. They will claim that a modern interpretation of the V model is to read the length of one ‘V’ as one iteration instead of a project. Then, surely, the V model would make sense even in agile projects? My response to this would be an emphatic “NO“. Agile iterations are not small waterfalls. Activities of different type happen in parallel, dynamically, when needed. Furthermore, most of the other objections above would still hold.

But what if we narrowed it down even further? What if one ‘V’ just represented development of a single feature? A few days work, maybe. Would that solve the problem? Sadly, still no. There are still no feedback loops. The model still communicates an unrealistic degree of order. It still emphasises inspecting quality after-the-fact. It is still limited and heavy on work documentation. It still promotes specialists with a limited view of the whole process.

Let’s face it: The V model is not suitable for product development and it never really was. Why try to cram modern thinking into an obsolete model? Why not use some other model instead? It is time to put the V model to rest.

What Would a Better Model Look Like?

First of all, a new model would be contextual, meaning it depends on the situation. Like many have said before me, there are no “best practices” in product development. Therefore, I cannot give you a model that will work for you. Only you can do that. But what I can do is present some ideas, which I think could be identifiable in a process development model in harmony with current thinking.

I believe a more useful product development model would:

• …include users and other stakeholders, starting from their needs all the way to actual use of features and back again.
• …focus on optimizing flow of value through the process instead of having defined roles and steps.
• …de-emphasize phases, instead have work cycles with feedback to gradually refine the solution.
• …de-emphasize specialist roles and boundaries between them, instead focusing on teams working to optimise the whole.
• …de-emphasize documentation as a means of communication, instead relying on natural collaboration between team members.
• …remove the artificial boundary between requirements and tests. Specification by example shows us how to specify software in a human-friendly yet automatable manner, using examples to guide us.
• ..remove the artificial boundary between software design and tests. TDD shows us how to design software incrementally, with quality built-in, using tests to guide us.

På mitt jobb har vi komplexa affärsregler. Regler, sofistikerat utformade under årtionden för att avlägsna alla möjligheter för kunder att att göra några som helst relevanta jämförelser mellan olika leverantörer. Ingen förstår riktigt alla regler. Verksamhetskillarna med de snajdiga frisyrerna och kostymerna brukar komma ned till oss då och då när de behöver ha reda på hur något verkligen fungerar. Varför? Därför att reglerna för affärerna är inkodade i mjukvara. Någon programmerare, någon gång, har tvingats att förstå hur det borde fungera.

Idag realiseras verksamheten i mjukvara. Är det någon som flyttar papper på sitt kontor längre? När jag började jobba var frasen att IT-system var “verksamhetsstödjande”. Det stämmer oftast inte längre. Det är som att säga att en pacemaker stödjer verksamheten att hålla igång ett hjärta. Det är sant i någon mån, men redskapet för verksamheten är så väsentligt för verksamheten att det svårt att se vad som är vad. Verksamheten är formligen indränkt i mjukvara. Den sker inbäddad i system.

Ändå strävar nästan alla stora organisationer att göra tydlig åtskillnad mellan det som kallas “verksamhet”" och det som kallas “IT”. Det är här någonstans som den intresserade läsaren bör höja på ögonbrynen. Men vänta nu här…

För att resonera vidare, vad betyder det här för processen att utveckla verksamheten? Givetvis att den i många fall är samma sak, eller måste ske i synk med, vidareutveckling eller förändringar i mjukvara. Förändringar i affärsprocesser innebär oftast förändringar i mjukvara också. Omvänt, förändringar i IT-system är alltid verksamhetsförändring.

Vi har delat in vår värld i verksamhet och IT, men det verkar ju onekligen som att de mer och mer håller på att smälta samman. Borde då inte dessa båda sidor av samma mynt jobba ihop? Än värre, vad händer när vi beslutar oss för att utkontraktera hela IT-förvaltningen till ett låglöneland? Vad betyder det för vår verksamhetsutveckling?

Känns inte hela den här uppdelningen i “verksamheten” och “IT” märklig och föråldrad idag? Den uppstod i en tid när man var tvungen att ha ett fint universitetsbetyg i matematik för att komma nära en “elektronhjärna”. Självklart kan vi inte ha de där märkliga personerna i vita rockar springandes i korridorerna – vad skulle kunderna säga? Så vi gömde dem och lämnade dem till sitt. Åren gick och vi lät dem i stort sett förbli där på sin avdelning. Men världen har förändrats.

Jag tycker inte att det borde heta “verksamhet” och “IT” därför att IT är också verksamhet, vare sig du vill det eller inte. David Joyce, en leankonsult på Thoughtworks, sa nyligen i ett tal jag hörde: “Think of the IT staff of Lehman Brothers, I guess it took them until the day of the bankruptcy to realise they were part of the business too”. Det här språkbruket befäster en gräns mellan arbete som inte gagnar digitaliserade verksamheter.

Vi ser inte IT som en del av verksamheten idag och de flesta företag är inte organiserade på det sättet, men betyder det att vi inte kan tänka oss bättre sätt? Peter Tallungs, verksamhetsarkitekt på IRM, brukar jämföra med ekonomiavdelningen. “Tänk om vi sa att ekonomiavdelningen inte var en del av verksamheten. De skulle bli oerhört förolämpade. En del av dem brukar ju till och med tro att de leder verksamheten.” Jag vet inte hur det är med er, men jag börjar känna mig rätt förolämpad själv.

To Boldly Go…

One perspective on leading is that it is like convincing a group of people to go places where they have never been. It could even be the case that nobody has been there before.

In order to do this the following conditions are really useful:

• a clear idea of where you going
• a general plot on how to get there, the routes you will take
• a fairly accurate way of determining where you are
• an unstoppable determination to arrive

It is obvious that you cannot do this alone. You need to collaborate with other people to succeed. But so what? These collaborators are the exact same people you are bringing along to the trip and engaging them will just make them more committed. Some clever leaders even manage to set things up so that only some slight guidance is needed.

A Thinking Model

Imagine you are the captain of a grand ship. Perhaps it is an elegant wooden schooner requiring a crew of many people. The crew may smell a bit, but they are skillful and loyal. The sails are flapping and you can feel the wind messing with your hair. It is a strong ship, yet speedy and agile.

But this is no pleasure trip; you have a mission. Your job is to transport some fine passengers to their destination. The passengers are nice enough, but they seem very confused, which does not stop them from being very vocal about the conditions on board.

To achieve your mission, here are your most important resources:
1. The North Star This is the product vision, the business goals, the effects you want to achieve. What will the world look like when you have reached your destination? This tells everyone where we are going and will help you navigate. It helps achieve that clear distinction of what the product will and will not do. The goals may be adjusted during the way, but the general objectives will remain. Like a friendly star to gaze onto.
2. The Chart Plot An overview plan of the journey is useful as long as we accept that it is volatile. What are the major legs we need to travel? In other words, what intermediary goals should we seek? We may drift off course or we may deliberately choose to take another route so we must regularly re-plot to keep a good overview. The Chart Plot will keep you on course and determine your speed. It will help you avoid the reefs and getting caught in doldrums.
3. The Rolling Wave The waves of the sea will gradually roll you to new locations. As a traveller, you will see more and learn more, which you incorporate into your strategic maneuvering. You will need to gradually refine the features and plans for your product. What use is scouting for waves that are beyond the horizon? The Rolling Waves will carry you forward and you will gradually see more detail of previously unchartered waters.
4. The Crew The crew is your team of specialists. All of them have special abilities; yet, each one of them is a sailor. They know every inch of the ship and will cajole it into movement in the right direction, always keeping a look out for obstacles and reporting the position as we go along. One additional chore they have is the obligation to keep the ship clean and trimmed. The Crew will get the ship moving and keeping it so. They will steer it as best they can after your orders and report your position.
5. The Passengers. Your voyage will only be fortunate if you listen to your customers. Only through the users of your product can value materialise. They are your market and you should strive to delight them. Sure, they may feel like a nuisance sometimes, but they are the true enablers of the voyage. The faster you get them where they want to go, the happier they will be. The Passengers will help you understand if you are on the right course towards gratifying their wants and needs.

The unstoppable determination to arrive must come from yourself. It is significative of any fearless captain.

The Bold and the Beautiful

Finally, some advice for budding captains: Remember, if you push your crew too hard they will dislike it. They will feel gradually more fatigued and less motivated – even if you do energise them with the occasional bottle of rum. They are proud sailors! If you do not bend they may rebel against you. Or, more probably, simply swim to another ship. Furthermore, the maintenance of the ship will suffer and the wood will slowly deteriorate into a rotten, festering mush. This will slow down the top speed of the vessel and repairs will take time. It this is allowed to go on, the ship will capsize. So remember, when we reach harbour and we have had a few with the friendly inhabitants, we will probably need to make another trip with our trusty craft. It is a good idea to keep it ship shape.

The passengers, on the other hand, will only help you if you engage them. Otherwise, they will simply sit comfortably in their cabins, doing what they always do to make time pass. You must engage them, although they are not of your kind. They may have opinions on how you run this ship and you should not listen. They may quarrel with you about the insolence of the rugged sailors and you should politely defend your people. But when you dreamingly admire the stars together you should commit everything to memory.

Bon Voyage!