Results of a survey on internal control and risk management recommendations

by Matthew Leitch, 25 August 2004

Thank you

First, thank you to everyone who responded to this survey. It was a long and tough one that needed thought. The payoff is that we have some interesting results with profound implications for many, especially people in internal audit roles.

Summary

The results of this research strongly suggest that auditors, risk managers, and others who make recommendations for improving internal control and risk management can benefit from giving more attention to recommendations beyond the usual repertoire of sign offs, documentation, segregation of duties, and reconciliations. This implies that there could be great value in changes to audit approach and audit training and education.

In this online internet survey respondents were asked to consider eight hypothetical reviews of business activities and consider for each review a list of five potential recommendations for improvement. Respondents were asked:

• if, in their experience of organisations, the recommendation would probably already be in place;

• if the recommendation was probably a good one, assuming it wasn't already in place; and

• if they thought they were probably expected to make such recommendations. (If the respondent did not make recommendations in their job they were asked to consider if they would expect such recommendations to be made.)

This combination of questions revealed that certain types of recommendation were as likely to be good recommendations as others but were much less likely to be in place already. They were also less likely to be "expected."

Making these types of recommendation requires greater knowledge of risk and uncertainty but someone who has that knowledge would be able to use it often because the controls in question are rarely in place.

The survey showed no differences between the public and private sectors, except that public sector respondents thought a much wider range of recommendations was expected of them. This finding contrasts with opinions expressed to me by some, who speculated that the public sector was not interested in risk taking or risk management. In fact, the public sector respondents felt they were expected to make more wide reaching and sophisticated recommendations than did the private sector respondents.

Survey design

The survey form presented respondents with eight imaginary reviews (i.e. the scenarios) in the order shown below:

Each scenario was followed by five potential recommendations for improvement, displayed in a random order that differed for each respondent.

Respondents were asked three questions about each recommendation using the following instructions:

"Imagine that internal auditors or other risk management or internal control specialists have been doing some reviews of activities in an imaginary organisation. Each review found problems and some recommendations are under consideration."

"For each recommendation consider the following:"

• "Probably already in place": In your experience of organisations would you expect the recommended action to have been taken already? Click the first checkbox on each line if you think that most organisations would not need the recommendation because they would already have done what is being recommended.

• "Probably a good recommendation": Click the second checkbox if you think the idea would probably be a good one, assuming the action has not already been taken. Don't get picky about the details of the recommendation or its wording. The basic idea being suggested is what matters. Assume there are appropriate details that go along with this simple summary.

• "Probably expected": For the third checkbox it depends if you make recommendations on risk management / internal control in your job.

  • If you do make recommendations about risk management/internal control in your job click the checkbox if you think you are expected to suggest ideas like this.

  • If you do not make recommendations on these topics in your job click the box if you would expect auditors and other risk management specialists to suggest the idea.

  "All the recommendations should be taken individually. They are not intended to be linked. Also don't worry about the fact that the imaginary organisation seems to do a lot of unrelated things! The examples are drawn from life but not in the same organisation."

  Recommendation types

  In all there were 40 recommendations across 8 scenarios for respondents to consider. The recommendations were designed to be realistic and were not generated according to any particular scheme. However, for the purposes of analysis and summary they have been put into groups.

  The most interesting recommendations are those that (1) are good ones, and (2) are least likely to be in place already. A simple indicator of how interesting each recommendation is can be calculated as the number of respondents who thought the recommendation probably a good one (assuming it was not in place already) minus the number of respondents who thought it would probably be in place already. On this basis the most recommendations, in descending order of interest, were as shown in the table below. (See below for an explanation of the "Concept group" column.)

  Rank	Recommendation	Good - In place	Good	In place	Expected	Scenario	Concept group
  2	Include in each iteration of the product development process a step where the current areas of uncertainty/risk relevant to the idea under development are quickly listed and their impact is considered before actions to reduce the uncertainties or manage the risks are decided.	38	45	7	19	prod dev	Uncertainty
  19	The project management team should consider some form of training to develop their ability to talk openly about risk and uncertainty on the project and encourage others to report progress and risks honestly and completely.	37	44	7	24	project	Educate
  4	Provide education/training for product developers on how risk and uncertainty affect their work and how identifying uncertainties that matter can guide their research efficiently.	36	43	7	26	prod dev	Educate
  38	Risk management procedures should be revised to encourage people to revisit risks and responses much more often, to stay up to date, and to focus on things that are more specific and topical.	36	45	9	25	overall	Uncertainty
  39	More effort should be made to incorporate risk/uncertainty awareness into strategic decision making and not just routine clerical procedures.	35	46	11	30	overall	Uncertainty
  37	The way risk and uncertainty are quantified should be improved so that more numerical modelling and empirical support are used where appropriate.	34	38	4	23	overall	Quantify
  32	The computer model underlying the financial projections should be independently reviewed to ensure that it is correctly programmed.	33	44	11	25	oil	Sign.
  35	The risk management approach should also address the personal risk/uncertainty awareness, skills, and attitudes of staff, particularly managers at all levels.	33	41	8	26	overall	Educate
  36	Documentary evidence of internal controls / risk management should be enhanced so that any failure to carry out agreed controls is highlighted promptly.	33	44	11	31	overall	Doc.
  14	Projections about how much money the back-billing project will eventual raise should be expressed as ranges with probabilities rather than as a spuriously accurate 'best guess' number. For example, say the range of recoveries that is now 80% probable.	32	40	8	20	backbill	Quantify
  0	Write a policy on risk management for product development in the business unit.	31	40	9	31	prod dev	Doc.
  11	Predictions about how much money the back-billing project will eventually raise should be reviewed independently before being used in revenue forecasts.	30	39	9	22	backbill	Sign.
  20	Risks to improving service should be identified, documented, and assigned owners.	30	43	13	28	service	Risk mgmt
  7	The spreadsheet model estimating the financial results of a proposed conference should treat attendance as an uncertain variable with a probability distribution, and show the projected financial result as a distribution. From that it would be possible to see the estimated risk of, for example, making a loss. The number crunching can be done easily using widely available Excel add-ins.	29	42	13	20	confer	Quantify
  9	The conferences team should take time out to consider the range of outcomes from past conferences, how predictable they really are, and to think of ways they can manage conferences more flexibly and gain more information about visitors that will help in selecting conference topics, venues, and dates.	28	46	18	27	confer	Learning
  33	The source of all evidence used in making estimates should be stated, even if it is just to point out the name of the person whose gut feel it is.	28	40	12	23	oil	Uncertainty
  18	As far as possible without creating inefficiency the project should be divided into short term deliveries to stakeholders, not just internal deliveries within the project. This would accelerate business benefits, reduce committed resource before benefit delivery, and increase learning from experience.	27	43	16	22	project	Structure
  21	More short term indicators of progress should be sought, as the existing indicators are too long term to be used alone.	27	41	14	24	service	Learning
  22	Since the results from the new ideas are not certain the ideas should be trialled rapidly and revised as necessary as they are rolled out more widely. It is very important to learn as much as possible from experience.	27	41	14	23	service	Learning
  13	A small set of back charges should be taken through to bills and attempted recovery of money from customers as soon as possible to learn more about what it will take to do this on a larger scale.	26	38	12	23	backbill	Learning
  16	The project steering committee and project management team should set a good example by being open about uncertainties, communicating them, and showing that they expect others to be open with them.	26	45	19	25	project	Uncertainty
  17	The project plan should be reviewed to see if the dependencies can be reduced to improve the risk profile of the project.	26	45	19	24	project	Structure
  31	There are various uncertain variables in the projection and these should be modelled using probability distributions to explicitly represent the uncertainty and avoid the flaw of averages.	26	41	15	23	oil	Quantify
  30	The financial model needs to reflect the fact that decisions about whether to proceed further and how will be taken at various points in the proposed exploration. These options should be valued.	25	44	19	24	oil	Quantify
  3	Formal proposals for new products to be approved should include a section listing risks and how, if at all, they can be managed.	24	45	21	27	prod dev	Risk mgmt
  23	Priorities should be revised regularly - probably more often while ideas are still relatively untried.	22	38	16	19	service	Adaptability
  5	Evidence relating to likely attendance, revenues, and costs for proposed conferences should be documented in a written business case.	18	44	26	22	confer	Doc.
  29	Changes to resource allocations between products should be authorised appropriately and in writing.	18	43	25	26	prod mgmt	Sign.
  8	More should be found out about interest in potential conferences, for example by using surveys and looking at the readership of related magazines and journals.	17	42	25	22	confer	Learning
  15	A risk management process should be put in place to identify significant risks to the project, plan responses, and track progress.	16	45	29	32	project	Risk mgmt
  28	Alternative promotional strategies should be tried to find out which work best in each category.	16	38	22	19	prod mgmt	Learning
  6	Approval to proceed with a conference should be given by a committee of suitable managers and their approval should be evidenced in writing, for example in the minutes of the committee's meetings.	15	42	27	27	confer	Sign.
  12	The project should have a formally agreed scope and definition document, and a project plan.	14	41	27	27	backbill	Doc.
  26	The products should be managed as a portfolio during the year, with new products that go well being given extra resources to develop, while disappointing products get less.	14	39	25	19	prod mgmt	Adaptability
  10	Where it is discovered that past bills to a customer have been incomplete but the back charges are to be waived this should be authorised appropriately and documented.	13	44	31	29	backbill	Sign.
  34	A formal risk assessment exercise should be carried out.	13	45	32	28	oil	Risk mgmt
  1	Ensure that product development approvals are given in writing and signed off by suitable officials in the organisation.	11	44	33	29	prod dev	Sign.
  24	The service improvement plan should be authorised in writing at a high level.	11	42	31	24	service	Sign.
  25	Clear revenue, growth, and profit targets should be agreed for each product annually.	7	41	34	24	prod mgmt	Targets
  27	Products should be managed tightly to ensure that each product meets its annual targets.	0	31	31	22	prod mgmt	Targets

  Here is a scatter graph showing how the concept groups are distributed:

  The most interesting types of recommendation are towards the bottom right of the graph.

  The recommendations in the Targets group were brought down by the context used. The two recommendations on using targets were the least popular of all, though still thought to be good by most respondents. The recommendation is to set targets for individual products and ensure that each product meets its target.

  These were the only items in the survey I considered bad recommendations and it is comforting to know that one marketing specialist who responded to the survey (and identified himself to me) agreed.

  In other contexts the idea of setting fixed targets and managing towards them would have got higher approval, but in the product portfolio case more people are aware of the problems.

  The choice of this context for the Targets recommendations probably brought down the Targets group more than most would have done.

  In the table above the concept groups have been classified as "New" or "Old" according to my perception of which ones are traditional audit mainstays, based on over 10 years of audit experience. The final summary is of the figures for new and old recommendations:

  	In Place	Good	Expected
  Old	51.2%	90.9%	58.1%
  New	30.2%	90.6%	50.1%

  Public sector versus private sector

  Public and private sector responses did not differ noticeably in any way except one. The proportion of recommendations public sector respondents felt they were expected to make was dramatically higher than for the private sector respondents, and this difference was greatest for recommendations in the "new" concept groups.

  Only 16 of the 46 respondents were from the public sector, so to give an idea of how statistically significant the difference is I have calculated that if there was in fact no difference between the public and private sectors the probability of getting the result I got or something more extreme is only 12.4%, using a 1 tailed T test. In other words, it is very likely that this is a real difference, despite the sample size.

  Respondent roles

  The results seemed pretty much the same regardless of whether respondents were auditors, risk managers, performance managers or held some other role. This was a bit of a surprise as I had thought risk managers would be expected to make more varied recommendations than auditors.

  Differences between scenarios

  Different scenarios produced different responses, but I could see no meaning in the pattern. The low figure for "In place" for the overall review is perhaps because respondents may have had difficulty seeing a specific control within the recommendation.

  Scenario	In Place	Good	Expected
  prod dev	33.5%	94.3%	57.4%
  confer	47.4%	93.9%	51.3%
  backbill	37.8%	87.8%	52.6%
  project	39.1%	96.5%	55.2%
  service	38.3%	89.1%	51.3%
  prod mgmt	59.6%	83.5%	47.8%
  oil	38.7%	93%	53.5%
  overall	18.7%	93%	58.7%

  Comments by respondents

  The survey asked if the respondent had any other comments they would like to make about the survey or about recommendations. Excluding comments purely about the survey, the respondent comments were:

  "Wow, these questions made my head hurt.  Any time you wandered into recommendations regarding modelling and statistics, I backed off.  I haven't thought about those matters since college Econometric courses some 20 years ago.
  As internal auditors are primarily accountants first, and operational observers secondly, we would not be expected by my current organization to make comments on statistical models or product marketing recommendations.  Because of the detailed nature of these comments, I'm much more comfortable suggesting a topic for strategic direction than a specific management model.  The point is to get management to clarify their aims and directions.  Internal audit attempts to help them formalize this and then audit to their stated objectives.  We are not subject matter experts.  But we can comment on the effectiveness of processes based on outcomes."

  "There is a lot of work to be done, especially in the government sector in incorporating risk management (information/operations risk). I would like to see more concrete work on how to integrate or rather align specific business goals/missions with the IT part of the business. To date, it still seems to me there is a big gap.  The push from C-level executives to figure out ROI/ROSI (Return on Investment/Return on Security Investment)are still rather "fuzzy" numbers. How do we really get from here to a fully integrated risk management organization where risk is just business as usual and as much as possible, fully automated and dynamic?"

  "The "probably expected" box I interpreted to mean "probably expected within my current role". Working on a Sarbox project, I am planning to raise business issues but am not generally expected to do so. Some of the survey's suggested recommendations are also somewhat outside management's own expectations of a process or touching on areas that management are not expecting me to review - e.g. strategic marketing decisions and are not included on those grounds."

  "Risk management itself is a risk process - the degree of risk evaluation/ management depends on many factors, and the answers to the questions above, of necessity, cannot assess all the factors that would apply, especially between e.g. SMEs and multinationals."

  "Quantification doesn't help if there is no suitable information to quantify - which is often the case."

  "The oil case encourages the use of real options methodologies. However, virtually all the cases would benefit from this perspective."

  "Much of what's 'recommended' above is (or should be) standard public sector practice, given the push to formal project management (It's all in PRINCE). Risk management (Per HM Treasury) and OGC Gateway reviews."

  Respondent profile

  Respondents were invited to participate using professional discussion lists on the Internet and by some personal e-mails, but only where I was confident the person would not be predisposed to answer in a particular way. For example, if someone wrote to me about how they were interested in evolutionary project management I would not invite them to participate.

  Most respondents were from the USA and UK. The countries of respondents are shown on this graph:

  Most were internal auditors and risk managers. The roles of respondents are shown on this graphic:

  Limitations of the survey

  Besides the obvious limitations of a study where respondents represent a tiny section of a large population and are self-selected, and where the number of respondents is fairly small, there are some other imperfections in the design that should be borne in mind.

  The recommendations were nearly all intended to be good ones, so that some respondents may have been led into a pattern of answering "Good" to all of them. If there had been more bad recommendations to consider then the proportion of Good responses to the good recommendations might have been lower.

  Some respondents had technical difficulties with their browser's rendition of the survey and one reported that he had been unable to amend answers to some questions. This was not part of the design. The survey worked perfectly when tested on Internet Explorer.

  Although the evidence suggests that the new types of recommendation are likely to be very useful to auditors and others who make recommendations on controls there is a missing step in the chain of reasoning. The survey does not show how often in practice the newer types of control are appropriate. It only shows that there are good recommendations for controls that are unlikely to be in place already.

  Finally, the simple binary responses asked for do not tell us how confident respondents were that particular controls would already be in place or how good they thought the recommendations were. We only know what proportion of respondents thought the controls would probably be in place and what proportion of respondents thought the recommendations were probably good. There is a difference, though one would expect strength of feeling and volume of support to be correlated to some extent.

  Further information

  A good explanation of the rationale for this study is given in one of my articles for IRMI. The article is "Embedded risk management: the auditor's role".

  If you would like to analyse the original data yourself I can provide a matrix of the ratings given. The information will not allow you to identify respondents or their organisations.

  Copies of the original survey are also available. Please contact me at matthew@internalcontrolsdesign.co.uk.

  
  
  
  

  About the author: Matthew Leitch is an independent consultant and researcher specialising in internal control and risk management. He is a Chartered Accountant with a degree in psychology whose past career includes software development, marketing, auditing, accounting, and consulting. He spent 7 years as a controls specialist with PricewaterhouseCoopers, where he pioneered new methods for designing internal control systems for large scale business and financial processes, through projects for internationally known clients.

  Words © 2004 Matthew Leitch
  
  

  Home / more articles - The author - Contact on your terms - Feedback - Ask a question - Links - Services