How Acumen tackles the Mt. Everest of engineering challenges

Estimation of software projects is one of the most painful subjects for any tech company. (Look no further than the comments in this Hacker News post to see some of the strong emotions that this challenge provokes.)

Software is extremely complex, so accurately predicting how much time a project will take can feel nearly impossible. This leads many developers and teams to not take it seriously or even give up and drop time estimation completely – an obvious issue when the business relies on deadlines to coordinate launches, close sales, or drive renewals.

For those teams that haven’t given up on estimation completely, one common approach is to use various time units (such as Story Points, Hours/Days, or T-shirt sizing). But this approach often suffers from various challenges:

1. Unknown unknowns. The difficulty of predicting challenges and issues in the coming task/project makes it hard to allocate the right “buffer” when estimating. Case in point: based on unknown unknowns (e.g., will a certain project require unexpected refactoring of another part of the code base?), the same task might vary between a few hours and a few weeks. This “blowup factor” is supported by statistical research suggesting that developers generally have a good intuition for the median time of a task, but struggle with the average – resulting in major estimation inaccuracies in projects that involve  internal dependencies. 
2. Differing estimates. Based on their own subjective experiences and levels of productivity, different developers give divergent estimations. And even when they give the same estimate, they’re not really the same: Developer A’s 5 hours is not the same estimation as Developer B’s 5 hours.
3. The efficiency/predictability tradeoff. Giving a cushy time estimate for a task helps a developer be on time and thus predictable – but there’s always the possibility she could have done it more quickly with less time allotted. On the flip side: giving an aggressively short time estimate provides incentive to be as efficient as possible, but in many cases a developer will exceed his estimation, hurting her predictability.

Acumen’s vision is to ultimately solve for all of these challenges with traditional estimation approaches. (Full disclosure: we’re still working to crack the nut on unknown unknowns.) But based on feedback from our customers, we believe we’ve developed a novel solution to the second and third obstacles: creating a standard “currency” of estimation across devs and resolving tensions around efficiency vs. predictability in time estimation. 

How do we do this? Acumen has access to the engineering team’s data, and uses machine learning to infer the Maker Time (the actual developer coding time) spent on each task throughout the history of a team. Knowing Maker Time equips Acumen to help teams improve estimation accuracy throughout three stages of the software development lifecycle: planning, execution, and retrospectives. 

Planning:

Planning in Acumen involves standardizing individual time estimates to a common denominator, benchmarking the total capacity of the team and individual developers, and then forecasting workload on the individual level.  

1. Standardize. The starting point is to translate the estimations of all team members into a unified “currency” with a common denominator. So for example, Acumen translates one hour of each developer to team points:
   

2. Benchmark. The second stage is to understand the team’s velocity/capacity, by running the average amount of points a team is capable of doing in one iteration (sprint, week, etc.) – and also the number of points each team member is capable of. For a team with a total capacity of 95 points, the breakdown might look something like this:

3. Forecast. The third stage is to use standardized individual time estimates to estimate points at the individual level. Comparing an individual’s point capacity to his or her planned points might highlight areas of risk, where estimates are unrealistic and scope must be reduced. 

Execution

‍Acumen automatically identifies “lookalike” projects to benchmark progress and flag estimation risks throughout the execution of a software iteration. 

1. When the team starts working on a task (Task Z), Acumen identifies the similar tasks from the previous work of the team (a similar task means a task of the same type, done by the same developer, given similar estimation, handling similar components and files, etc.).
2. Once Acumen finds a set of similar tasks, we can compare Task Z Maker Time with the similar task’s Maker Time – and produce an indication whether the task is on track or exceeding the estimation (estimation by Points relevant to the specific developer). 
   

Acumen’s comparisons are done at the individual level, with the context of each developer’s patterns and point capacity. So if Developer A worked on Task Z for 4 hours, for example, Acumen might show the task as On Track. But if developer B worked on the same task for the same amount of time – 4 hours – Acumen might show it exceeding the estimation.

Retrospective

The same principle of “lookalike” projects applies to completed work. Once a team has finished a project, Acumen can look back at the tasks the team finished and show if a task was fast or slow compared to similar tasks. 

The bottom line

Planning is a notoriously challenging task for many dev teams that often feels more like guesswork than a data-driven exercise. 

Acumen’s approach to inferring Maker Time from the engineering team’s data – and surfacing relevant insights throughout the development lifecycle – can help teams improve the accuracy and value of estimation.

Learn more about Acumen's approach to unifying and analyzing engineering data.