Comparison of Overall Project Risk Assessments Using Expert Judgment Versus Monte Carlo Simulations

In the Fifth Edition of PMI’s PMBOK (Project Management Body of Knowledge) Guide (© 2013 PMI^®, Inc.) there is a good comparison (see Figure 11-13 on page 336 and Figure 11-17 on page 340) of the differences between using “Expert Judgment” and “Monte Carlo Simulations” to arrive at the range of overall Project Costs for a relatively simple project with just three (3) WBS (Work Breakdown Structure) elements.

If you would like to test this methodology out for yourself, below is the information needed.  You can enter this data directly into Chrono™.

The Chrono™ Gantt chart shown below has all the detailed project set-up data, including the WBS elements, task interdependencies, Most-Likely task durations, “Best-Case” & “Worst-Case” task durations, and the “Most-Likely” project finish date.

In that comparison, the “Most Likely” costs add up to $41,000,000.00. 

The “Worst-Case” costs add up to $68,000,000.00 and the “Best-Case” to $31,000,000.00. 

A Monte Carlo Simulation (run by combining the three (3) Triangular probability distributions, made up by those same numbers) shows that the “Most Likely” cost of $41,000,000.00 has just a 12% chance of being met (or an 88% risk of being missed), and the “Best-Case” of $31,000,000.00 has no chance of being met, while the $68,000,000.00 “Worst-Case” is significantly beyond the 100% confidence point. 

These are striking differences and should be understood if you are compelled to convey overall project risks in the most appropriate way (e.g., to establish a competitive advantage for winning a business proposal). 

Both approaches might seem valid, but why do they produce such different results, and which provides the most “right” answer?

Although both analyses are influenced by “estimation accuracy,” the Monte Carlo simulation approach is the most accurate, since simply adding Best-Case and Worst-Case estimates is naturally flawed. 

Let’s think about it. 

What are the odds of every task completion adhering to either the Best-Case or Worst-Case estimates, especially if they were provided by different people within the organization? 

In creating greater alignment – what are the odds of rolling just ten six-sided dice and having all of them settle on the same number? 

These are almost impossible odds which can be easily computed (0.000001654% or 1 out of 60,449,492 attempts) – and that answer can be closely replicated via a Monte Carlo simulation.  

That is why we do not use the aggregate of all Best-Case and Worst-Case numbers if we are trying to determine relatively realistic expectations for project cost and schedule risks.

To help prove this point, we performed a similar (and more complex) comparison to demonstrate the differences obtained for Project Schedule risk analysis using the two methodologies. 

The differences between comparative results are more pronounced due to the inherent complexity of project schedules, especially relative to merging of schedule paths. 

When analyzing project costs, you do not have to worry about merging, only additions.  But even project cost differences diverge significantly when project complexity increases (i.e., when you have a very expansive WBS).

The project schedule case study results are presented in the above chart. 

Here are the key take-aways:

1. Most project schedules (especially for complex projects) have more risks than opportunities (i.e., the difference between the Worst-Case duration estimates and the “Most Likely” are greater than the difference between the “Most Likely” duration estimates and the “Best-Case”) thus, the task level schedule percent confidence is usually less than 50%.  This means that the “Most Likely” end dates, in aggregate, result in a relatively low overall delivery schedule confidence level (typically less than 1%).  Yet most people assume the “Most Likely” dates are 50:50 (or 50% confident) – a major misconception.  In this relatively simple schedule example, the “Most Likely” date (1/1/2021) has a <1% confidence of being met.
2. When we add up all the Best-Case durations, we get a result that has basically no way of being achieved. Refer to the dice roll discussion above. 
3. When we add up all the Worst-Case durations, we get another result that is not practical. You basically cannot miss that date, and if you use it for a competitive proposal bid, it will likely put your bid out of contention.
4. Using a Monte Carlo simulation replicates reality the best. Other methods, like PERT could be used, but the results are not as precise, and that methodology cannot determine percentage confidence in meeting different project completion dates or costs. If the Monte Carlo simulation is readily available and easy to use, why would you not use it?  Estimates might be off (e.g., miscalculated or overly difficult to determine) and the use of all Triangular probability distribution functions, as used in this example, might not be applicable for all tasks (e.g., some task distributions might be more appropriately constructed from discrete values or a collection of discrete inputs from past actual data), but Monte Carlo simulations, like Chrono™, have provisions for those other probability distribution types as well.  In addition, to produce the most accurate results possible, your IMS (Integrated Master Schedule) must be valid – Chrono™ employs rules like the DCMA (Defense Contractor Management Agency) 14-Point Metrics that were established to ensure IMS validity (Chrono™ also provides a table listing how the IMS fairs against the 14-Point Metrics for those interested).
5. Using the Most-Likely estimates as aggressive targets might prove beneficial in another way – after all, you are using the most likely estimates provided by the various estimators, so they (especially those who are signing up to perform the work) will be okay with working to those estimates.  Since “work expands to meet the time allotted” (per Parkinson’s Law), we have found that having the project team work to the more aggressive “targets” is a sensible thing to do if you want to have a better chance of meeting an earlier commitment date or a lower total cost – a good way to potentially earn contract award fees.
6. Establishing a 70% confidence-level commitment date gives the team more of a fighting chance. We advocate use of that percentage, for beyond that point the “S Curve” (or cumulative probability curve, or percent confidence curve) starts to asymptote, yielding diminishing returns. Also, in our experience we have always been successful when using the 70% completion date as a commitment (as have other PMs we know) – but only if they have the team work to the original Most-Likely “target” schedule dates – so real-life empirical data supporting this approach does exist.  By the way, there is no other way to determine dates based on “percent confidence” without using a proper Monte Carlo simulation.
7. Most importantly, Monte Carlo simulation outputs facilitate pro-active risk management. There are many ways to attack project risk and bring the commitment date in and/or budget commitment down, while maintaining an achievable commitment confidence level. A Monte Carlo simulation tool is worth every penny, and if utilized properly, can bolster the performance of all your project portfolio management teams. You can go to RTConfidence.com and download a free 1-month trial today.  It is basically an add-in to Microsoft Project™.

If you would like to test this methodology out for yourself, below is the information needed.  You can enter this data directly into Chrono™. 

The Chrono™ Gantt chart shown below has all the detailed project set-up data, including the WBS elements, task interdependencies, Most-Likely task durations, “Best-Case” & “Worst-Case” task durations, and the “Most-Likely” project finish date.