Internal Controls Design website by Matthew Leitch (tutor, researcher, author, & consultant)
 New website, new perspective: www.WorkingInUncertainty.co.uk - Related articles - All articles - The author - Services

 Alternative risk listsResults of an online survey

by Matthew Leitch, 18 September 2007

Summary
Design of the survey
Survey respondents
Understanding different beliefs
Practical implications

## Thank you

First, thank you to everyone who responded to this survey, particularly if risk isn't particularly your speciality or you provided useful information in your comments, as many did.

## Summary

The survey probed basic beliefs connected with risk and revealed a surprise. Although a lot has been written about risk analysis as if the risks already exist and need only be discovered, over 80% of respondents believe that there are alternative ways to list risks that are not just more or less complete versions of a true list, or the same risks with different names or in a different order. Most people believe this for a variety of reasons, including different models/perspectives, different ways to break down outcomes into risks, and different knowledge. Also, most people believe that some lists are more useful than others even when equally valid and complete.

The same high degree of belief in alternative risk lists was apparent in respondents from all three sources and did not differ greatly between risk specialists and others, or between people with some mathematical skill and people with little or no mathematical skill.

This article describes the study and its results, then describes alternative views so that you can understand better why people think as they do. It also points out some practical implications of these findings for the way we talk about risk analysis and the topics that need to be covered in any guidance on how to analyse risk.

## Design of the survey

Respondents were invited to participate using postings to three professional discussion lists. The invitation contained a link to a very short online survey that posed the following questions:

 What country are you living in now? Do you consider yourself a risk or risk management specialist or expert? Yes        No Do you think you could solve the following equation in less than 20 seconds?  (x - 2)(x + 4) = 0 Yes        No Suppose a risk analysis is needed to create a list of risks for a given application (e.g. the risks of a project). Which of these statements do you agree with most? There is only one valid list of risks. (Though people might give different names to risks, get more of the total list of risks, or put them in different orders.)   There are alternative valid lists of risks. (This goes beyond just alternative names, orders, or degrees of completeness.) IF you think there are alternative valid risk lists, why might that be? Check all that could apply, e.g. when listing risks for a project. Different perspectives / different models of the project and its environment.   Different ways to split causes or outcomes into risks.   Different knowledge about the project and its environment.   Other. IF you think there are alternative valid risk lists, which do you agree with most? Any equally valid and complete list is equally useful.   Some lists may be more useful than others even when equally valid and complete. Are there any comments or explanations you would like to make? (Include your email address if you would like a reply.)

### Sources of respondents

To ensure a range of expertise in risk assessment among respondents they were recruited from three discussion lists to one of three identical versions of the survey. This meant that responses could be roughly separated between the more and less risk oriented respondents based on source, as well as by looking at survey responses. Here are the discussion lists involved:

1. RISKANAL: A discussion list about risk analysis that has a large membership, mainly in the USA, and includes many academics and practitioners.

2. AuditNet: A list mainly for internal auditors.

3. PMA Forum: A list about performance measurement, with many members from the UK.

## Survey respondents

The volunteers from each source can be summarised as follows:

 List Respondents Risk specialists Equation confident RISKANAL 78 79% 78% AuditNet 31 35% 65% PMA Forum 34 18% 71% All 143 55% 73%

Judging by their confidence in solving the equation in 20 seconds it looks as if RISKANAL members are either slightly more mathematically oriented than members of the other lists, or are younger. The equation used in the question is simple if you have just done an examination in mathematics at school but baffling if you did that years ago and haven't had to think about algebra since.

Asking questions about beliefs concerning abstract ideas like "risk" is extremely difficult. The comments from some respondents show something of the range of unexpected interpretations of words in the survey. However, the main results are clear. Most people think that alternative lists of risks are possible and vary in usefulness. The reasons for alternative lists go beyond different orders, descriptions, or degrees of completeness, and beyond different objectives or points in time.

 List Believe alternative risk lists possible Because of different perspective / models* Because of different splits* Because of different knowledge* Because of other reason* Alternative lists can have different usefulness* RISKANAL 88% 87% 61% 74% 28% 80% AuditNet 84% 88% 50% 77% 19% 96% PMA Forum 85% 83% 69% 66% 21% 90% All 87% 86% 60% 73% 24% 86%

* This is the percentage of the respondents believing alternative lists to be possible. These follow up questions were not applicable for respondents believing only one valid and complete list can exist.

It seems to make no consistent difference whether respondents consider themselves to be risk or risk management specialists. It also makes no consistent difference whether respondents can do simple algebra. In the table below "Risk" means the respondent is a risk specialist and "Math" means the respondent is able to do simple algebra.

 List % for Alternative lists: not Risk, not Math % for Alternative lists: not Risk, Math % for Alternative lists: Risk, not Math % for Alternative lists: Risk, Math RISKANAL 100% 78% 80% 90% AuditNet 86% 77% 75% 100% PMA Forum 100% 72% No Data% 100% All 96% 75% 79% 92%

## Understanding different beliefs

Obviously this survey does not give a detailed understanding of each respondent's rationale. However, while designing the survey I was aware of three main reasons why alternative, valid risk lists might be considered possible, and more have come to light from comments made by respondents.

### Perspectives considered in designing the survey

Here are the three main reasons I started with.

[In addition, different lists might result from different objectives/interests and from creating the risk lists at different times, though arguably these are part of the circumstances and not reasons in principle for alternative lists within the same circumstances.]

Risks as derived from models

Some people derive risk lists from models of the system, activity, or whatever they are making a risk list for. For example, an accountant might build a financial model of a project and then use a Monte Carlo simulation tool such as @RISK to represent uncertain inputs as probability distributions, and compute the implications for output variables and intermediate variables.

Each of these variables can be seen as a “risk” and a risk list could include all these, plus risk items for model uncertainty and a variety of other things.

The same can happen with non-quantified models and variation in risk lists results from the choice of model and the choice of method of deriving risks from the model.

With this perspective the risk list depends on what model you start with, and alternative models are common. Some risk lists are likely to be more useful than others in a given situation.

In the survey the reason "Different perspectives / different models of the project and its environment" was referring to this view, though respondents will not necessarily have read it that way.

Risks as potential events

In the usual textbook introduction to probability theory “events” are defined as sets of outcomes. For example, the outcomes from throwing a six sided die can be represented by the numbers 1, 2, 3, 4, 5, and 6 but an “event” is a sub-set of these, such as “less than 4”, “an odd number” or simply “six” (because sets can have just one member).

Clearly there are alternative ways to split the outcomes into events.

The same thinking can apply to “risks” on a risk register if you think they are the same as the mathematician’s events. Risks can usually (perhaps always) be seen as sets of potential outcomes. For example, “Losses from vehicles this year” might appear on a company’s risk register, but if they were more interested in these losses they might have captured the same outcomes within a larger number of “risks” perhaps for losses of different types of object, or even losses from individual vehicles, or for shorter periods of time.

Some risks are effectively infinite sets of outcomes. For example, the risk item “loss of market share” could refer to any extent of lost market share up to total loss, which is an interval of a continuous variable that mathematicians would usually regard as having infinitely many members.

Listing every outcome is not a practical possibility and again there are alternative ways to split down the total set of potential outcomes. Some splits are likely to be more useful than others in a given situation.

In the survey the reason "Different ways to split causes or outcomes into risks" was an attempt to refer to this view.

Knowledge

In many views of probability knowledge is crucial. New information leads to revisions of probabilities.

Applied to risk listing this gives another reason for recognising alternative lists. For example, suppose some situation with uncertain outcomes has already taken place. Some risk analysts already know the outcome but some do not. The analysts who know the outcome cannot have it as a risk any more because there is no uncertainty. However, the analysts still in ignorance can have it as a risk because, for them, the outcome is still uncertain.

In the survey the reason "Different knowledge about the project and its environment" was referring to this view.

### Further beliefs inferred from respondents' comments

Comments by respondents pointed towards yet more reasons for believing that alternative, valid risk lists are possible.

Personal perceptions

The idea seems to be that people have different perceptions of situations and these can be equally valid.

Levels of detail

Risks need to be addressed with different risk lists at different levels in an organisation because otherwise low level lists are too detailed for high level people and high level lists are too broad to be useful to low level people.

Practical limitations

Lists of risks are never complete because of the sheer number of possible risks and the difficulty of understanding the future.

Some respondents gave this explanation but also commented that they thought there was, in theory an ultimate and complete list reflecting all perspectives but in practice this was unobtainable and too complex to use. Some respondents with this view said alternative lists were possible and some said they were not.

Decision analysis perspective

The validity and usefulness of a risk lists depends on the decision questions being addressed.

Depends on the audience

The risk list provided depends on who the audience is. A list for the public might be different from one for private use.

Risk analysis is an art

Risk analysis is an art that depends as much on the analyst as on the facts.

## Practical implications

The high proportion of respondents who believe that alternative risk lists are possible was a surprise – to me at least – because the traditional language of risk management and content of the best known guidance on risk management suggests that alternative risk lists are not possible.

If you think alternative risk lists are possible then you may be interested in the following practical implications.

### Guidance

Although it is increasingly common for guidance to acknowledge alternative processes for arriving at a risk list it is rare to see explicit discussion of alternative lists.

Usually there is considerable scope to give more advice on:

• Why alternative lists are possible.

• What characteristics make some lists more useful than others in a given situation.

• When choices arise, even within a particular process.

• How to make those choices e.g. what to consider.

• How to change lists to meet new requirements and circumstances.

### Language

One way that the traditional language of risk management tends to suggest a unique risk list stands out in particular. It is the use of the phrase "Risk Identification."

If you are any kind of risk expert you probably don't see the problem with the phrase "Risk Identification" because we seem to have become so used to it that the conflict is invisible. However, consider the view from outside the risk world. Some uses of the word "identification" make sense but some do not. The following examples illustrate important points about this word.

 Situation Appropriate use The police discover the name of a suspicious man seen near a crime. "The police identified the suspect." A commuter solves a Sudoku puzzle. "The commuter identified the numbers that solved the puzzle." A bird watcher sees a bird fly between two trees. "The bird watcher identified the bird as a jay." A manager is faced with a difficult choice. "The manager identified the best option." Situation Inappropriate use An artist paints a picture. "The artist identified the picture." An architect designs a new building. "The architect identified the new building." A team of software developers creates a new graphics program. "The team identified the new software."

"Identification" is appropriate when a name is being put to something that exists, or when a limited range of possibilities is studied to pick the one that meets some criterion e.g. "the best". The thing to be identified exists already.

"Identified" is not appropriate when something is being created, invented, or developed. In other words, when the thing involved does not exist already.

The more it seems that developing a risk list involves choices and creates things ("risks") that did not exist before the analysis began, the more appropriate it is to use words like "develop", "analyse", "define", and "create" instead of "identify." More appropriate terms to replace "Risk Identification" include "Risk Set Definition", "Risk Analysis", "Risk Analysis Development", "Risk List Development", "Risk List Creation", "Risk Hierarchy Development", and so on.

Other language habits that subtly conflict with the idea of alternative lists include these:

• Using words that are similar to "identify" e.g. "We need to find risks related to this objective."

• Talking about "the risks" without reference to a particular analysis, as if "the risks" existed before any thinking was done.

• Talking about sets of risks as if they exist separately from any analysis e.g. "Strategic risks are risks that ..." said without reference to analysis so that is seems that strategic risks exist beforehand.

• Giving the name of a risk with no definition in a situation where the definition hasn't already been established, and expecting people to know what it is. For example, "Hey, how about technology acceleration risk?"

These are the comments by respondents related to beliefs about risk lists: