Media centre

System Auditing for Compliance

Peter Boyd
Manager, Compliance Practices & Procedures

The following speech outlines CASA's new approach to the surveillance of airlines.

System auditing has been progressively introduced by CASA since mid 1999, in conjunction with new surveillance planning and a safety trend indicator.

This new approach to surveillance has the goal of encouraging airlines to take responsibility for their own safety management systems while maintaining CASA's monitoring and evaluation of compliance.

CASA's changes to surveillance and compliance will take another three years to be fully introduced to all sectors of the aviation industry.

Results from the new approach to surveillance have already proved strong, with a marked change in the airline sector.

Australia's commercial aviation environment has tended to reflect the country's unique geography - a land with large population centres linked by commercial activity, but also a land of huge distances and geographically isolated communities.

Australia has historically embraced aviation as one of the solutions to addressing the tyranny of distance.

The aviation industry has been regulated by government since the early days of aviation, through many stages of departmental evolution.

Currently, the Civil Aviation Safety Authority (CASA) regulates Australia's aviation industry through:

• drafting the Civil Aviation Regulations (The Rules"),
• safety education
• surveillance
• enforcement processes.

The Civil Aviation Act prescribes the above functions.

Before an organisation can legally offer commercial aviation services in Australia, it must obtain an Air Operator Certificate from CASA and a Certificate of Approval must be obtained if an organisation wishes to carry out maintenance on aircraft and components. CASA's sole concern is the safety of the operation when considering an operator for certification. Our regulations deal with the aspects of the flight that will bring the aircraft safely to its destination - Occupational health and safety issues do not come under the jurisdiction of CASA. The Australian aviation industry currently comprises approximately 900 Air Operator Certificates and 800 Certificates of approval.

We are in the midst of a fundamental change in the Authority's approach to the safety surveillance of the aviation industry. This is being undertaken in conjunction with a major review of the Civil Aviation Regulations.

This paper outlines the changed approach to surveillance and describes the planning processes that have been developed for CASA's surveillance program for the aviation industry, including the use of risk assessment techniques to focus CASA's surveillance efforts.

CASA has undertaken a restructuring process in the past two years where separate Divisions are now responsible for the distinct aspects of the Authority's responsibilities. The Compliance Division comprises approximately 350 staff located in offices throughout Australia and is tasked with the compliance and enforcement functions for the Authority, functions which are designed to ensure that operators comply with the Regulations.

To carry out this oversight of the industry the Compliance Division has been structured into two separate operational areas that are tasked with the surveillance of:

• Airline Operations and
• General Aviation Operations

The formation of the two offices has allowed for separate development of safety management system assessment processes for the two areas, aimed at the differing surveillance demands of the sectors of the aviation industry. There are obvious organisational size difference between airline and general aviation operators and the airline operators tend to have more extensive, formal management systems in place and can accept more stringent requirements for these systems.

The need for change

Aviation authorities around the world have been faced with a dramatic increase in air traffic in the last twenty years. The overall safety of flight has also increased markedly over this period, largely through advances in technology, such as collision avoidance systems and ground proximity warning devices. The combination of the huge increase in activity and improvements to the technology has led to a plateau in the number of serious accidents, in a global sense. The need to decrease the accident rate has been most keenly felt in the United States of America, where Congress has tasked the Federal Aviation Administration (FAA) with reducing the USA accident rate to 17% of the 1995 level by the year 2007. The USA and other International safety experts have been searching for a way to drive the accident rate still lower and have generally concluded that advances in technology will not deliver the desired step change.

At the same time, studies into the causes of airline accidents were beginning to conclude that a significant proportion of aviation accidents could be contributed to system deficiencies within organisations (the so-called "organisational accident"), whereas, in the past, they may have been simply attributed to pilot error. The FAA has embarked on a program to increase their airline operators' safety through an emphasis on operator management systems and a surveillance structure based on these systems. Other Civil Aviation Authorities have also launched similar initiatives. Whilst the overall thrust of the safety initiatives is similar across the globe, each country has tended to differ in their approach to the detail. This should be seen as a healthy sign, as management systems rely heavily on the interactions of the people working within the system and stand or fall according to the response of the people to the system in which they work. Similarly, the approach to surveillance of the safety management systems should also be adapted to the particular culture involved.

The change process - from final inspection to system audit

Airline Operator Surveillance

After the mid 1990s and in line with the global developments, CASA had started to introduce a system audit approach to surveillance. Quality system audit techniques were taught in training courses and attempts were made to integrate these techniques into the surveillance system, but there tended to be a widespread belief that the quality systems and the system approach was all very interesting, but where did it fit in with my real job?

At that time, CASA's surveillance program was structured around individual inspections by the Authority's operational staff (Flying Operations Inspectors, with a commercial flying background and Airworthiness Inspectors, with a maintenance engineering background). The system of surveillance was well defined and documented and can be summarised as:

• Inspections of the airline operator's flight-(an inspector would travel on the flight and observe the actions of the flight crew - called an en-route inspection)
• Inspections of the airline operator's facilities at ports
• Desk-top checks of sections of the operator's operations manual

The number of inspections required was determined by the different types of aircraft and the number of revenue hours flown by the airline operator. The inspections were recorded on checklists. The result of each inspection would be notified verbally to the immediate manager of the airline involved (usually the Captain of the flight). If there were breaches of the legislation noted during the inspection, these were communicated to the immediate manager and a non compliance notice issued.

In early 1999 the Australian National Audit Office carried out an audit of CASA's surveillance and enforcement activities. The audit report is a matter of public record and pointed to a number of shortcomings with the surveillance system, including a serious shortfall in meeting the inspection targets set by the system.

Clearly, changes had to be made in the system of surveillance of the aviation industry.

The system was characterised by the following:

• The system was structured around the technical disciplines
• The system was task-focused
• The focus was on the end product of the systems (the flight)
• Any identified problems tended to be fixed by "patches"
• There was considerable repetition of tasks
• The planning system was inflexible
• The inspections were carried out by individual inspectors
• The system was checklist-based
• Reporting was at a low level and the safety messages did not usually reach the accountable managers

The new approach to surveillance is characterised by the following:

• The system is based on each operator and industry sector
• Focuses on the systems that produce safe outcomes
• Required fixes are based on the systems needed to produce consistent results
• The planning system is organisation-based and uses risk assessment techniques
• Uses team-based audit techniques, where practical
• Records include full audit notes; standard checklists are not used
• Appropriate reporting structures are established

The new surveillance system was established in June, 1999 after a review of the system and was designed to incorporate the best-known practices in management systems.

The goal of the surveillance review was established as:

"To encourage and achieve industry development of management responsibility and effective safety management processes whilst efficiently monitoring and evaluating industry compliance."

The current aviation regulations do not have a requirement for airline operators to establish a management system to control their processes for safety and this was one of the reasons why implementing a systems-based surveillance system had been difficult to establish previously. In order to audit a system effectively, an appropriate framework for a safety management system must first be established.

For surveillance purposes, a safety management system was established for airlines based on the management system model

The safety system has been designed to allow an integrated management system approach for airline operators. In order for safety management system to be effective, it must be an integral part of the business, not an additional requirement. The safety management system achieves this through a similar approach to models for quality and environmental systems.

Safety management system elements

The CASA surveillance regime is based on a safety management system model, which is structured around four management system attributes. Each attribute has a number of elements that are selected for a particular audit.

The elements are:

Management Responsibility

Management Representative; Responsibility and Authority; Safety Policy; Objectives and Safety Planning; Review of Safety Management; Change Management; Risk Management and Hazard Identification; Emergency Response Planning; Internal Communication/ Consultation, Control of Documents; Control of Records;

Infrastructure

Personnel(Training and Training and Checking); facilities and equipment; Information; Human Factors;

Processes

Process Elements (Line Operations; Aircraft Performance; Load Control; Flight Planning and Dispatch; Rostering; Routes and Ports; Ground Handling; Maintenance Control); Service Development Processes; Purchasing/Subcontracting; Handling and Storage; Measuring Equipment Calibration;

Monitoring, Feedback, Corrective and Preventive Action

Incident and Accident Recording and Investigation; Remedial, Corrective and Preventive Action; Internal audit.

In general, CASA's inspection staff have experience in assessing airline operators in the management system attributes of "process" and "Infrastructure (training)".

In the early stages of safety management system surveillance development, airline operators are being audited in these areas of the management system only. This is to allow the airline operators time to fully implement safety management systems and to enable CASA staff to become more familiar with system audit techniques. These areas of the safety management system also have adequate current regulatory coverage.

The new regulations, which are due to be introduced in 2002, incorporate the full requirements for the safety management system for airline operators.

General methods

Each system audit has the following process:

• A scope for the audit is selected from the management system elements.
• The audit team prepares audit questions from the airline operator's documented procedures
• Pre- audit team meetings and briefings are held
• An entry meeting with the airline operator's management team is conducted
• Sufficient practices and "products of the system" are sampled during the audit to assure the auditors that the procedures are adequate and implemented
• Team meetings are held throughout the audit to compare notes
• Findings are documented on individual Requests for Corrective Action
• The findings are presented at an exit meeting with the airline operator's management team
• The audit findings are presented in a comprehensive audit report, complete with executive and element summaries and a compilation of the required Requests for Corrective Action for the airline operator
• A surveillance database is used to plan the audits, monitor the completion of the audits and is used to track responses to the audit findings

Surveillance planning

The surveillance regime is based on a rigorous entry assessment and a three-year certificate renewal cycle. A new entrant is issued a certificate for one year and subsequently a three-year certificate, if found satisfactory after the initial one-year audit and certificate renewal. Generally, the larger airline operators are held to a six-month surveillance frequency, although there is some scope to vary the frequency of surveillance visits for smaller airline operators based on the results of a risk assessment process - the safety trend indicator process. At the end of the three-year period a full audit is conducted before the certificate can be reissued.

Safety trend indicator

A safety trend indicator has been developed which is designed to quantify the stressors that are applicable to an airline operator that may lead to an increased likelihood of an unsafe situation.

The Indicator is used to target CASA surveillance resources at a local office planning level.

The Indicator comprises thirty questions that are answered by the inspector (or group of inspectors) who have the best knowledge of the airline operator and is completed at six-monthly intervals.

The following are typical samples from the thirty questions:

"Has there been a significant change to organisational structure or areas of responsibility in the preceding twelve months?"

"Has the organisation introduced new aircraft, or new routes, or made significant changes to procedures or processes, within the last twelve months?"

"Has the organisation been subject to takeover or change of ownership in the last twelve months?"

"Are there any indicators that the organisation is suffering financial stress?'

"Has the organisation been subject to significant expansion or contraction within the last 12 months?"

"Taken as a whole, does the organisation operate under more difficult conditions than other operators?"

General aviation office surveillance regime

The general aviation office has surveillance responsibility for a larger number of smaller certificates and the resources available for surveillance are different from the airline office.

As for the airline office, the regime is based on a rigorous entry assessment and a three-year certificate renewal cycle. A new entrant is issued a certificate for one year and subsequently a three-year certificate, if found satisfactory after the initial one-year audit and certificate renewal.

A risk analysis of the General Aviation Industry was carried out and the industry divided into sectors

The Safety Trend Indicator is completed every six months and is used to determine if an operator is to be visited by an audit team. At the end of the three-year period a full audit is conducted (depending on sub-sector) before the certificate can be reissued. A range of sample checks are also undertaken across the GA industry to ensure that industry compliance is maintained.

Results

CASA's emphasis on responsible management of safety in the industry has seen a marked change in the airline sector in the last two years. The major industry organisations have readily embraced the concepts and tangible results have resulted.

Status - Where are we now?

The change process has been estimated to be a five-year process; we are two years into the change.

CASA is committed to the change and is expending significant resources in training and internal system development.

Australia has an enviable aviation safety record and is currently one of the safest aviation environments in the world. CASA is working hard to even further improve this enviable record through the introduction of safety management concepts.

Our vision is "Safe Skies for All"

Media contact
Peter Gibson
ph 02 6217 1015
mobile 0419 296 446