Why You Should Avoid Using Projects in Software

Summary

Organizing work around projects does not fit any form of work that is creative, open-ended, and patterns-based, relying on human ingenuity and judgement to create value. This is most notable in research, creative jobs, and development in all its forms; be it organizational change, personal growth or software development.

Projects in Open-ended Endeavours

What would the 15-year-old Jack Nicklaus, the legendary golfer, have responded if his coach had asked him for the probability that he’d meet milestone 3 for project Winning British Open as defined in the project plan as per 7th of March?

Golf ball 3

Yes it’s kind of silly to imagine this happening – yet in software this is just a part of normal life. Every day, thousands of developers and teams are asked to compare their progress against The Plan and estimate how much is left.

Now, I agree that writing software is not the same as developing your golf skills, but considering the level of craftsmanship and inherent variability in software it’s more appropriate and valuable to me to compare it to other kinds of patterns-based, open-ended activities than with production lines chugging out the same item over and over again.In these endeavours having a plan is actually less important that making steady progress and creating value.

A painting is finished when the artist says so, based on skill and experience gathered over a lifetime. But from the customer’s perspective it’s finished when it is finally destroyed. So who’s right? Both, I guess. A defined finish seems to not exist here. The same thinking can be applied to, for example, a research paper or a software program I think the common element here is that we are creating — and anything created has a life cycle, which resists being put into a project management framework.

Just to be clear, when I write project here I don’t mean a batch of work that belongs together. The latter are naturally occurring “lumps of work”. I use the term initiatives or epics for those. When I talk a about projects I mean a way of organizing work following some kind of formal or informal project model, typically lead by a project manager, with the expressed goal of delivering everything in scope on time and within budget. An initiative doesn’t need any of that.

But What About Agile Projects?

People still ask me: How should we work in agile projects? To start with, anything prefixed with  “agile” is cause for concern. Anything that requires modulation cannot possess that characteristic in itself, by nature, but must be adjusted to fit. That’s why we have “agile requirements”, “agile documentation”, “agile contracts”, “agile planning tools”, “agile processes”, “agile management”, and… “agile projects”.

So we know that projects need to be modified to fit an agile organization. And many people have written about that to explain how you should adjust your project model to work in an agile fashion. That’s fine, but that’s not my point. My point is this: Why bother?

Asking about how to do agile projects is the wrong question I feel, since it assumes that projects are the best (only?) way to work with software. A more interesting questions to me is this: How should we organize our work to create maximum value (learning) over time? And to the this question I strongly believe that the correct answer cannot be “in projects”.

Why Projects Aren’t Suitable In Software

Here’s a list of reasons I’ve compiled over the years, use it as you see fit…

  1. Projects encourages important thinking upfront, investigating problems and drawing up solutions, but… we know that we want to harness the learning built into development. Why make so many decisions right at the start, when we know the least?
  2. Projects by definition have a defined start and ending, after the project the results must be handed over to some other type of organization, but… software has a life cycle. Software is created, developed, actively maintained, passively maintained, and finally, decommissioned. Software and projects operate on different timescales.
  3. Project results are typically handed over to another group for “maintenance”, but… we know that handovers are costly and excellent quality requires ownership and continuity.
  4. To reach project goals, especially on dates and costs, quality is pushed to the background. Some projects even result in so-called “death marches”, where delivering something maintainable is not in scope, but… in software, quality is a primary success factor… because without excellent intrinsic quality, your product will be slow and expensive to maintain and have a high life cycle cost.
  5. Projects encourage us to create batches of work, being a big batch in itself, but… we know that a steady flow of value is way more effective (feedback + learning).
  6. Projects (and PMs) often entertain a narrow view of success, i.e. what’s best for the project, but… wouldn’t we rather optimize for the whole company? A project is a target to a PM and, as W. Edwards Deming famously said: “Give a manager a target and he will reach it, even if he has to destroy the company in the process“. Especially large projects tend to behave a bit like the old schoolyard bullies, taking what they feel like without asking.
  7. To acquire funding for their projects, PMs are often tempted or forced to overstate the expected positive outcomes of their projects, otherwise it won’t get funding, ending up in a never-ending spiral of silly overstatements, but… most people are decent and prefer to work in an environment where they don’t need to deceive others to do their job.
  8. Projects typically require setup, allocating people, finding resources like office space, writing presentations, holding lift-offs, agreeing on vision, mission, and how to work etc. In most settings you pull people to the project and not the opposite, but… we can do away with all that by simply letting teams already in full operation pull work to themselves instead.
  9. Projects use budgetary mechanisms to give attention to ideas and allocate funding to them, but… having funding is a very poor raison-d’être while creating happy customers is a good one.
  10. For psychological reasons (for example, ego and careerism) it is tempting to create massive projects for bold ideas, but… we know that large projects are associated with stunningly high risks and low success rates, close to zero after some point.

These observations have nothing to do with agile methods or any other school of software development. In fact, it has nothing to do with software development at all. It’s at the heart of all creative, open-ended work, including arts, craft, writing, architecture or organizational change. This also explains why having an agile “rollout” plan is silly. Change is not a project and it’s not as easy to simply “implement” it (I wish it was).

Now I don’t want to imply that if a client comes up with a project plan you should laugh in their face. Of course not. There are situations where projects can be suitable, e.g. when working with a new contractor team outside your organization. You can use the project to get to know each other and the defined end date to call things off if it didn’t work out. Just be aware of the risks that projects bring and work to mitigate them.

The Alternative

The alternative to not working in projects can be outlined like this:

  • Create and grow long-term, cross-functional development teams that create their own, smooth development process.
  • Give each team clear ownership of a product, part of a product or a few products, depending on size. Cultivate a long-term, product outlook on development.
  • Task each team with delivering value to customers (or learnings if something is unsuccessful) continuously, as often as the market can handle it, while expending less and less energy to do it.
  • Help the teams create more value to their customers using sound prioritization schemes and pull systems for work.
  • If suitable, batch larger work into initiatives, lasting a couple of months (enough to create milestone challenges to the teams), but keep scope high-level and stay open to changes and complete turnarounds as the learnings start pouring in.

You can of course persist and keep trying to make agile projects work, but seriously, why bother if you don’t have to?

When the manager’s out, the coaches dance on the table

I work as a software consultant and often in the role of a so-called “team coach” or “agile coach”. That’s basically just a hyped-up term for helping teams and departments to improve, to be more effective, and guide them on their way amongst a myriad of options. So coaching is great and many coaches I know are great, but there’s something bothering me. My opinion is that much of what I do is a kind of management – not of the people but of the work, and I’m getting more and more concerned about this. Should this really be the coach’s job?

Continue reading

Honey, let’s start over: A tale of business kaikaku

When Ian told them what he’d done, they couldn’t believe their ears. He had what!? He had erased every document in the company database concerning hiring and retaining staff. Joan’s deep thoughts on “talent management” simply gone. Gone! “Hey, what about the backups?”, Helen asked. Ian just smiled, “Those too”.

Continue reading

Recently, I retweeted an item from Neil Killick and I was surprised to see a response from Tom Gilb, legendary software methodologist. The tweets can be viewed in the picture below (read from bottom up).

Two tweets, my retweet and Tom Gilb's reply

Talking to Tom

Apart from the honor of getting some of his attention, I was a bit concerned with the response. I seemed simplistic and categorical to me. Quite frankly, it sounded like something out of a schoolbook in software engineering from the 80s. I find myself in disagreement with it. To disagree with Tom Gilb is scary so I needed to write down why.

Continue reading

Functional Scope Design: Why requirements must go and design for use should take its place

The Kanban Drama (a summary)

For those of you who don’t have time to follow along with every twist and turn in the ongoing struggle between Kanban connoisseurs and the rest of the software development world in general I would like to take this opportunity to summarize my impressions of what has happened over the last five years. It takes the shape of a short play. It’s a summary so I have taken the liberty of slightly shortening and abstracting history. Only slightly. Enjoy!

Continue reading

A User Story Template for Effect Maps

Introduction

If you are into User Experience (UX), you may be using effect maps, also known as impact maps or goal maps, to guide software development towards the intended effects. These maps are used to describe how your business goals will be achieved by helping important target groups fulfill their goals of using your product or service.

If you are using agile software development, you are probably using user stories to describe your features for development using some common template. In this post I will suggest a new user story template, specifically designed to be combined with effect maps.

Continue reading

Breaking through the Wall of Release Mediocrity

In my last post I discussed the problem of overestimating the quality of your software development and release process. In that scenario the time period allotted for a release is simply too short for a successful result. Of course, you can go wrong the other way as well.

Let’s say that you are the new configuration manager at a small software house. The managers feel that their release management is conducted a bit “ad hoc” and they have brought you in to restore order.  You, on the other hand, Mr Big Shot CM, have been trained for years on how to perform professional release management at the big enterprise with lots of engineers. Real engineers, mind you, not fakey information ones. After you have assessed the situation you dictate that releases may only occur at most twice a year with a two-month stabilisation phase each. The developers shake their heads, but you are the professional so hopefully you know what you’re doing.

Let’s think about it, was this a wise move? Yes, this probably makes releases more regular and dependent (for a while at least), but is that really something the customers have requested? They will get their software later. The organisation will also have to take on the challenges of coordinating big batches of work and a lot more work going on. This is unnecessary work and costs money. So maybe the only thing you managed to create with your “professional” release process was a loss of customer and business value?

Continue reading