One of the common struggles software companies face is the continuous battle between product and engineering teams over predictable delivery and estimation.

On one side, product managers, the board, customers, and other various stakeholders need to know when things are going to get done. And what is to blame? They need to know when that new feature will get to production so that they can set expectations, prepare a launch, and in general do their part of the job.

On the other side, engineers are facing the big challenge of providing effort estimates and delivery dates while diving into the unknown: Code written by others, hacks and dragons in the codebase, bugs that takes hours or days to fix, or even slow processes in place. So, they push to the other side.

Both sides have their merits, but truth is, time management is the only constant in those struggles. Time is a variable that cannot be altered, manipulated or bought, and the way it is spent, can determine the success of a company.

The scope of this post is to explore why long-term planning, estimation, predictability, and commitment is a game lost, pouring precious time down the drain. Then, we will follow up with how we can make better use of our time and boost productivity on all sides, by taking an alternate approach. 

Let’s go.

Expectation
Countdown to New Year's Eve has begun! Team leaders are meeting to prepare for the year ahead. They've segmented the goals into quarters and laid out a concrete plan of what to accomplish and when. The board of directors is all ears for the presentation, and the leadership team is filled with anticipation.

Meanwhile, the product team is hoping that everything will go smoothly, goals will be achieved, and bonuses will come rolling in. Commitments have been made. It's time to roll up our sleeves and get to work!

Reality
In the first few weeks of the year, everything seems to be going according to plan. However, things quickly become chaotic when the sales team demands that a new feature be added to the roadmap in order to close a deal with a demanding prospect. Meanwhile, a customer threatens to leave unless a reported bug is fixed or a certain feature is given top priority.

To make matters worse, the engineering team realizes that their initial time estimate for the new feature was way off, forcing them to sacrifice quality or cut corners just to meet the deadline. The feature is eventually deployed to production, only to be quickly pulled back due to customer complaints or lack of use, leaving everyone wondering what went wrong.

Expectation
The product team calls for a meeting with the engineering team to discuss the feasibility of feature X and obtain initial estimates on the development timeline. After several sessions, the engineers provide their prediction and commit to a delivery date. The pressure is on, and if they fail to meet the deadline, blaming fingers are pointed out, as they are seen to have failed to deliver as promised.

Reality
The product and engineering teams are in discussion about the feasibility and complexity of a new feature. The engineers are struggling to provide accurate estimates, as they are asked to estimate the unknown. They request a prototype to be made so they can have tangible data to work with, but the team is already busy building a feature from the existing plan. This lack of time for investigation or prototyping causes the whole plan to fall apart.

Expectation
The company has implemented story pointing as a means of estimating the effort needed to complete a user story. To achieve this, large user stories are broken down, and their estimates are given in working days.

Management monitors velocity, which is seen as a key performance metric. The expectation is that velocity should increase over time, as it represents the amount of work completed within a given time frame.

Reality
The engineering team struggles to equate story points to man-days, leading to an inaccurate estimation of velocity as a performance metric. To create the illusion of increasing velocity, the team starts to inflate story point estimations over time, even though the actual value delivered remains the same or even worse.

Expectation
The plan is set, the project is underway, with tasks assigned to each engineer. They begin working towards the delivery deadline, and any delays are attributed to them. If engineers fail to meet their deadlines, it is seen as either a failure in providing accurate effort estimates or simply being lazy.

Reality
It's Monday, and the engineers have started working on their tasks. However, the company has scheduled various activities, including performance reviews, 1:1s with managers, and meetings with the product team for input or new estimations. Additionally, the customer support team needs assistance troubleshooting a reported bug, and there may be incidents with production systems.

The engineers are finding it difficult to find focus time and get into the zone, resulting in only a few hours of productive work per day. No-meeting days or Slack-quiet days are highly valued and result in significant progress being made on assigned tasks.

I hope you get my points why long-term planning and predictable delivery is a hard problem to solve and requires a significant amount of energy, time, and effort to get it done. But does it worth it? What can we do instead?

After going through many years of transformation and maturity steps on a successful SaaS company, I came to realize that when releasing fast, everyone is happy.

By creating fast delivery pipelines and parallel value delivery streams, all of a sudden, product was happy, sales were happy, customer success was happy. And precise prediction and tedious planning made less and less sense, when the engineering team came to a point to release software much faster than the other (non-engineering) teams could consume. And that was a turning point.

But how can we get there? 

Here follows a 12 step guide:

Let's elaborate.

Grounded into reality, we should treat roadmaps as a direction rather than a fixed plan. It's important to plan for the short term and set long-term goals, but we should be open to adjusting our course as new information, opportunities, and market shifts arise.

After all, life is unpredictable and full of surprises. Just like we can't plan every aspect of our personal lives for the whole year, we can't plan every detail of a business strategy either. Instead, we should embrace uncertainty as a path to success. In nature, the species that adapt to changing conditions are the ones that survive, and companies are no different.

We have previously discussed why precise estimations are futile, consuming both time and energy and failing to deliver most of the time.

At the SaaS company where I work, we estimate effort in terms of weeks, months, or quarters. This approach enables the product team to obtain a rough estimate of the project's size and make informed decisions or adjust the scope accordingly.

Based on our appetite for a project, we can decide whether to proceed with its implementation. For instance, suppose the engineering team has estimated feature X to be a monthly project. In that case, we can skip it if the project's return on investment (ROI) is low and defer it to a later time. On the other hand, if the effort required is equivalent to a week's worth of work, it could be an easy win.

Long backlogs can be a heavy burden for teams to carry. If tickets remain in the backlog for over six months, it is a clear indication that they are not a priority. It's time to ditch them. This can be so liberating for our teams and encourage forward-thinking.

To streamline the backlog management process, consider adding automation to your backlog management software. Automating the closure of old items will save both time and energy, and boost morale. With this new approach, teams can focus on the most important and high-priority items, and move forward with a lighter load.

Folks at 37Signals employ the Shape-up agile framework, which emphasizes the use of pitches to frame project ideas. This approach has a significant impact, as it allows the product and engineering teams to collaborate and tackle the challenging task of shaping a solution for an idea. This results in reduced uncertainty, optimized estimations, and a more secure delivery process.

To minimize the risk of setbacks in a project, it's advisable to narrow down its scope. This can be done by dividing it into smaller parts, even if it means deferring some parts to future projects. Such an approach can lead to a more successful implementation and timely delivery of each part of the project.

Utilizing small teams of 2-3 people and reducing the scope of features can help minimize management and communication complexity. By breaking down projects into smaller parts, each team member can focus on specific tasks and work in parallel on different aspects of the project such as frontend, backend, or devops. This approach enables the team to integrate their work into a working feature along the way, leading to better collaboration and a more efficient workflow.

Maintaining focused and efficient work requires a considerable amount of mental effort. Unfortunately, this effort is easily disrupted by frequent interruptions, leading to inaccurate estimations and delayed delivery times. To prevent such disruptions, managers must prioritize their team's focus time and limit unnecessary meetings. They should also promote an asynchronous work culture, where team members can work at their own pace without interruptions.

Prioritize simplicity over complexity, both in design and coding. Junior developers often feel the need to prove themselves by demonstrating their ability to handle complex tasks, such as abstractions and patterns. However, in reality, this can slow down the development process. It's better to focus on just coding the necessary function initially. If it's needed elsewhere, it can be copied. If it's needed in multiple places, then it's time to refactor and abstract it into reusable utility code. However, this should be done later, not sooner. Too often, I've seen and written utility functions that end up being used in only one place, indefinitely.

Avoiding over-analysis and excessive delegation of decision-making can prevent slow progress. Instead, focus on implementing the design and collaborate to make changes as needed. For example, if the frontend developers encounter an issue with a corner case while working on UI mockups created by the design team, it's important to take action. Don't wait for the design team to come back, but instead make a decision and get creative to find a solution. Deploy the changes and iterate as necessary, without getting bogged down by someone else's authority or blocking a release.

DevOps can significantly improve a team's performance. The key is to establish streamlined processes that allow developers to deploy quickly and easily, multiple times a day, with minimal friction. DevOps empowers developers to safely release features and provides a secure environment where rollbacks are possible, and testing is automated to ensure confidence. Microservices can eliminate single points of failure. To assess your team's DevOps maturity, take the Google DORA test and identify areas for improvement. Don't settle for a low score - take action to improve!

One of my favorite tips is to avoid deploying big projects and instead deploy frequently, unfinished work. Why? Deploying progress to production provides your team with a sense of accomplishment. Not everything needs to provide value to the user just yet. For example, if you have added a new API endpoint, deploy it and mark the milestone. Then add a new hidden page and deploy it, marking another milestone. Once the team is confident, announce it to the masses using feature flags. This approach can significantly boost team morale. Trust me on that!

One powerful tip, shamelessly stolen from the book "No Rules Rules: Netflix and the Culture of Reinvention," is to always acknowledge the outcome of your hard work. If your project is a success, recognize it and celebrate your team's effort. Show them that their hard work has paid off. On the other hand, if your project fails, own up to it and share the lessons learned. This will not only help others avoid the same mistakes, but also show that your team is willing to take risks and learn from them. In both cases, it's a win-win for everyone involved.

You made this it far! Congratulations. You are ready to transform your company’s culture and performance for the better, by following those 12 steps.

Good luck!

- Nikos Vasileiou