This is the second blog post covering the content of our October 2019 ASQ Audit Division conference presentation sessions: "Revitalizing Your Audit Program Using Engaging Audit Reporting Practices".
In Part 1 we introduced several factors that often have limited visibility yet impact the effectiveness of audit reporting. Part 2 focuses on types of findings and visual representation in audit reports to better engage leadership.
Problem 1: Counting the number of nonconformities without understanding what the auditors are using as the "bar". When the bar is minimally meeting the requirements (Level 1), the management system can be very weak, yet look good to leadership due to low number of issues raised. The figure below shows this light switch mode of thinking of either right or wrong. Individual auditors or the defined audit programs may use a higher bar (Level 2-4). Without clarity of the level being used as the bar, the auditees and leadership are frustrated and confused by the inconsistencies in interpretations.
Problem 2: Leadership think no news is good news. No nonconformities must mean our systems are good, right? Well, every audit is a sampling of the processes and systems. The charts below are three (3) ways to show the same audit results. The pie charts show the count of the types of findings. The left chart is limited to bad news, Only the nonconformities and opportunities for improvement are shown. Reporting this way can build a culture by auditees that the auditors will never be satisfied. It encourages behavior to show less information to the auditors. The middle chart includes balanced view of audit scope. Conformity and strengths are also included. The grey color distinguishes minimally meeting requirements from fully conforming without crossing the line to identify more opportunities for improvement. This gives the departmental leadership a heads up and can encourage them to decide whether to put attention there or not.
The right chart adds what was NOT reviewed. This chart indicates what we have no news about. It encourages leadership to want more information about those processes and systems, especially if the audit reports show good practices. This type of reporting can encourage support of the audit program to ensure wider audit coverage is represented.
The chart below is the same data and shows the types of findings by the section of the ISO standard. It also shows what was not audited. This chart style helps encourage robust audit program coverage and reporting. Inquiring minds want to know feedback on the whole management system.
Now, let's switch from right and wrong thinking to evaluating maturity of the management systems. The table below is an older example from ISO 9004:2011 that is easy to understand. It shows the gradient from management focus, starting with limited focus, then widening with each maturity level.
Audit programs that use maturity level "scoring" or indicators can be powerful to give visual strengths and blind spots. Additionally, this format is excellent to show progress over time (or backsliding).
See ISO 9004:2018 for many other examples of maturity levels related to ISO 9001:2015 requirements.
The table below is a generic example of definitions for maturity levels. This approach is useful tool to help transition from right/wrong thinking without learning/teaching specific definitions for each requirement.
The slide below shows an excerpt of an integrated audit report where each criteria was rated on the maturity level scale above. There are three (3) segments shown to illustrate how one audit report may cover two (2) or three (3) sections of the company requirements. Imagine being the company leadership and getting three (3) different reports as shown in the three (3) groups of results. They each give a different perspective.
The chart below provides a holistic view of the entire integrated management system (quality, environmental, health and safety, security, product stewardship). Imagine having this type of report to use for your strategic planning and management review! Now we can have meaningful discussion and identify most appropriate next steps to achieve our performance goals and manage risk.
In summary, the slide below describes four (4) levels of criteria. If you are limited to right/wrong thinking in your audit program, start off by clarifying which bar or yardstick your organization wants to use to decide between conformity and nonconformity. This can be a step in the right direction to calibrate mindset within your auditor pool and the leadership team.
To generate excellent feedback about your organization's management systems, there are four (4) key factors described below. They are the overall audit program, the proficiency of the auditors, the reporting format, and the level of empowerment by the leadership to bring back candid and representative feedback.
As a company, Innovation Training & Consulting, Inc., follows this framework for auditing. Step 1. Understanding the level of commitment within the leadership team, or helping the team clarify what level they are willing to support, is a key foundation. Step 2. This cornerstone provides some boundaries for determining required resources including the audit program, auditors, protocols and any software required. Step 3. Empowerment of the auditors for robust investigation, candid feedback and understandable reports are next in the process. Step 4. Now we investigate identified issues to understand them and take action. We also celebrate our strengths and learnings and use our systems to generate great performance results.
Part 3 in this blog series goes behind the scenes to show how some of the above charts are generated. It will show simple tips for how to set up a spreadsheet to create pivot tables and charts.