Probability, Statistics and Other Frightening Stuff

Alan Jones

• Publisher: Routledge
• Year: 2018
• ISBN:
  9781351661379
• Edition: 1
• Topic: Business
• Subtopic: Forecasting
• Index: Business

Probability, Statistics and Other Frightening Stuff (Volume II of the Working Guides to Estimating & Forecasting series) considers many of the commonly used Descriptive Statistics in the world of estimating and forecasting. It considers values that are representative of the 'middle ground' (Measures of Central Tendency), and the degree of data scatter (Measures of Dispersion and Shape) around the 'middle ground' values.

A number of Probability Distributions and where they might be used are discussed, along with some fascinating and useful 'rules of thumb' or short-cut properties that estimators and forecasters can exploit in plying their trade. With the help of a 'Correlation Chicken', the concept of partial correlation is explained, including how the estimator or forecaster can exploit this in reflecting varying levels of independence and imperfect dependence between an output or predicted value (such as cost) and an input or predictor variable such as size.

Under the guise of 'Tails of the unexpected' the book concludes with two chapters devoted to Hypothesis Testing (or knowing when to accept or reject the validity of an assumed estimating relationship), and a number of statistically-based tests to help the estimator to decide whether to include or exclude a data point as an 'outlier', one that appears not to be representative of that which the estimator is tasked to produce. This is a valuable resource for estimators, engineers, accountants, project risk specialists as well as students of cost engineering.

1 Introduction and objectives

This series of books aspires to be a practical reference guide to a range of numerical techniques and models that an estimator might wish to consider in analysing historical data in order to forecast the future. Many of the examples and techniques discussed relate to cost estimating in some way, as the term estimator is frequently used synonymously to mean cost estimator. However, many of these numerical or quantitative techniques can be applied in other areas other than cost where estimating is required, such as scheduling, or in determining a forecast of physical characteristic, such as weight, length or some other technical parameter.

The original intention was to drip-feed the mathematics and statistics in the various volumes on an ‘as and when’ basis but this proved to be impractical and potentially generated a lot of repetition across the five volumes. A decision was taken to bring all the interesting/scary (delete as appropriate) stuff together as a single volume. As we have now commenced that journey together, we might safely assume that we have selected the ‘interesting’ rather than ‘scary’ option. As Volume I did refer to material in this volume, then there is a strong case that that logically this volume should have been Volume I, not Volume II. However, perhaps this would have put more people off than attract them

Despite the fact that this volume concentrates on the statistical analysis and probability, it should not be construed that estimators need to be out-and-out statisticians or ‘mathe-magicians’ who can miraculously conjure up an estimate out of nothing. However, the ability to juggle numbers is a pre-requisite skill of an estimator, but out of nothing? That does sound like guessing or picking a random number! (That said, we can use random numbers to generate an estimate, but there is still a structure to it; we deal with that in Volume V.)

Final thought on this volume: there is a strong analogy between statistical analysis and estimating. Estimates are predictions that have been based on a known or assumed context, described in a Basis of Estimate. Inferential Statistics are also predictions of what may occur, based on one or more Descriptive Statistics, or a pattern of behaviour in the data.

1.1 Why write this book? Who might find it useful? Why five volumes?

1.1.1 Why write this series? Who might find it useful?

The intended audience is quite broad, ranging from the relative ‘novice’ who is embarking on a career as a professional estimator, to those already seasoned in the science and dark arts of estimating. Somewhere between these two extremes of experience, there will be some who just want to know what tips and techniques they can use, to those who really want to understand the theory of why some things work and other things don’t. As a consequence, the style of this book is aimed to attract and provide signposts to both (and all those in between).

This series of books is not just aimed at cost estimators (although there is a natural bias there.) There may be some useful tips and techniques for other number jugglers, in which we might include other professionals like engineers or accountants who estimate but do not consider themselves to be estimators per se. Also, in using the term ‘estimator’, we should not constrain our thinking to those whose estimate’s output currency is cost or hours, but also those who estimate in different ‘currencies’, such as time and physical dimensions or some other technical characteristics.

Finally, in the process of writing this series of guides, it has been a personal voyage of discovery, cathartic even, reminding me of some of the things I once knew but seem to have forgotten or mislaid somewhere along the way. Also, in researching the content, I have discovered many things that I didn’t know and now wish I had known years ago when I started on my career, having fallen into it, rather than chosen it (does that sound familiar to other estimators?).

1.1.2 Why five volumes?

There are two reasons:

Size . . . there was too much material for the single printed volume that was originally planned . . . and that might have made it too much of a heavy reading so to speak. That brings out another point, the attempt at humour will remain around that level throughout.

Cost . . . even if it had been produced as a single volume (printed or electronic), the cost may have proved to be prohibitive without a mortgage, and the project would then have been unviable.

So, a decision was made to offer it as a set of five volumes, such that each volume could be purchased and read independently of the others. There is cross-referencing between the volumes, just in case any of us want to dig a little deeper, but by and large the five volumes can be read independently of each other. There is a common Glossary of terms across the five volumes which covers terminology that is defined and assumed throughout. This was considered to be essential in setting the right context, as there are many different interpretations of some words in common use in estimating circles. Regrettably, there is a lack of common understanding by what these terms mean, so the glossary clarifies what is meant in this series of volumes.

1.2 Features you'll find in this book and others in this series

People’s appetites for practical knowledge varies from the ‘How do I?’ to the ‘Why does that work?’ This book will attempt to cater for all tastes.

Many text books are written quite formally, using the third person which can give a feeling of remoteness. In this book, the style used is in first person plural, ‘we’ and ‘us’. Hopefully this will give the sense that this is a journey on which we are embarking together, and that you, the reader, are not alone, especially when it gets to the tricky bits! On that point, let’s look at some of the features in this series of Working Guides to Estimating and Forecasting . . .

1.2.1 Chapter context

Perhaps unsurprisingly, each chapter commences with a very short dialogue about what we are trying to achieve or the purpose of that chapter, and sometimes we might include an outline of a scenario or problem we are trying to address.

1.2.2 The lighter side (humour)

There are some who think that an estimator with a sense of humour is an oxymoron. (Not true, it’s what keeps us sane.) Experience gleaned from developing and delivering training for estimators has highlighted that people learn better if they are enjoying themselves. We will discover little ‘asides’ here and there, sometimes at random but usually in italics, to try and keep the attention levels up. (You’re not falling asleep already, are you?) In other cases, the humour, sometimes visual, is used as an aide memoire. Those of us who were hoping for a high level of razor-sharp wit, should prepare themselves for a level of disappointment!

1.2.3 Quotations

Here we take the old adage ‘A word to the wise . . .’ and give it a slight twist so that we can draw on the wisdom of those far wiser and more experienced in life than I have. We call these little interjections ‘A word (or two) from the wise?’ You will spot them easily by the rounded shadow boxes. In this one

A word (or two) from the wise?

'Mathematical theorems are true; statistical methods are sometimes effective when used with skill'.

David S Moore American statistician (Moore & Cobb, 2000)

David Moore (2000) emphasises the need for us to be able to interpret statistical methods or techniques . . . they will not give us a definitive numerical relationship. The estimator still needs to interpret the analysis and use judgement.

1.2.4 Definitions

Estimating is not just about numbers but requires the context of an estimate to be expressed in words. There are some words that have very precise meanings; there are others that mean different things to different people (estimators often fall into this latter group). To avoid confusion, we proffer definitions of key words and phrases so that we have a common understanding within the confines of this series of working guides. Where possible we have highlighted where we think that words may be interpreted differently in some sectors, which regrettably, is all too often. I am under no illusion that back in the safety of the real world we will continue to refer to them as they are understood in those sectors, areas and environments.

As this volume is all about the probability and statistics that underpin estimating, and certain other number juggling professions, perhaps we should clarify what we mean by the term ‘statistics’.

Definition 1.1 Statistics

1. The science or practice relating to the collection and interpretation of numerical and categorical data for the purposes of describing or inferring representative values of the whole data population from incomplete samples.
2. The numerical values, measures and context that have been generated as outputs from the above practice.

Statistics are very much like estimates, they need to have an accompanying context to have relevance and meaning, otherwise they may as well be random numbers (oh yes, we will be deal with random numbers in Volume V.)

I dare say that some of the definitions given may be controversial with some of us. However, the important point is that they are discussed and considered, and understood in the context of this book, so that everyone accessing these books has the same interpretation; we don’t have to agree with the ones given here forevermore – what estimator ever did that? The key point here is that we are able to appreciate that not everyone has the same interpretation of these terms. In some cases, we will defer to the Oxford English Dictionary (Stevenson and Waite, 2011) as the arbiter.

1.2.5 Discussions and explanations with a mathematical slant for Formula-philes

These sections are where we define the formulae that underpin many of the techniques in this book. They are boxed off with a header indicative of the dark side to warn off the faint hearted. We will, within reason, provide justification for the definitions and techniques used. For example:

For the Formula-philes: Definition of the Geometric Mean

1.2.6 Discussions and explanations without a mathematical slant for Formula-phobes

For those less geeky than me, who don’t get a buzz from knowing why a formula works (yes, it’s true, there are some estimators like that), there are the Formula-phobe sections with a suitable less sinister header to give you more of a warm comforting feeling. These are usually wordier with pictorial justifications, and with specific particular examples where it helps the understanding and acceptance.

For the Formula-phobes: One way logic is like a dead lobster

An analogy I remember coming across reading as a fledgling teenage mathematician, but for which sadly I can no longer recall its creator, relates to the fate of lobsters. It has stuck with me, and I recreate it here with my respects to whoever taught it to me.

Sad though it may be to talk of the untimely death of crustaceans, the truth is that all boiled lobsters are dead! However, we cannot say that the reverse is true – not all dead lobsters have been boiled!

One-way logic is a response to many-to-one relationship in which there are many circumstances that lead to a single outcome, but from that outcome we cannot stipulate what was the circumstance that led to it.

Please note that no real lobsters were harmed in the making of this analogy.

1.2.7 Caveat augur

Based on the fairly well-known warning to shoppers: ‘Caveat emptor’ (let the buyer beware) these call-out sections provide warnings to all soothsayers (or e...