Regulator Performance Framework self assessment report 2019-20

Executive summary

Self-assessment

The Australian Maritime Safety Authority (AMSA) undertook its fifth annual self-assessment of its performance against the Regulator Performance Framework (RPF) in August 2020.

The self-assessment was informed by:

• the draft 2019–20 AMSA annual performance statements
• the annual results of AMSA’s RPF customer survey; and
• the professional knowledge and experience of AMSA’s Executive team.

The 2019–20 self-assessment results broadly indicate that AMSA management:

• believes it has made tangible progress over the past year on the opportunities for improvement identified from previous assessments (2016 to 2018).
• is aware that there is room for improvement across the range of RPF KPIs, particularly in relation to the domestic commercial vessel (DCV) industry.

The sheer volume of AMSA-customer interactions (174,937: including phone, online and face-to-face interactions), as demonstrated by consistently high call numbers to our call centre AMSA Connect, provides AMSA with significant opportunity to positively impact our industry.

Just over 89 percent (89.3%) of customers were satisfied with the service delivered by AMSA Connect (see measure R.1.1: National System Customer Service), well above the call centre industry average of 70%, but not quite reaching our stretch target of 90%.

AMSA continues to be fortunate that the daily interactions of its staff with stakeholders and its long established reputation positively contributes, arguably contributing a ‘buffer’ of goodwill. However, AMSA management is very aware that such goodwill will not continue indefinitely in the absence of continued effort by management to improve our regulatory performance.

1.1 Self-assessment validation

In previous years, the AMSA self-assessment has been validated by the AMSA Advisory Committee (AAC) – an Australian peak maritime representative body. Due to the disbandment of the AAC in 2020, we were unable to secure a validation body in time for this year’s survey.

Background

1.2 Purpose

Commencing in July 2015, the purpose of the Regulator Performance Framework (RPF) is to encourage regulators to undertake their functions with the minimum impact necessary to achieve regulatory objectives, and to effect positive ongoing and lasting cultural change.

The RPF consists of six outcomes-based key performance indicators (KPIs) which set the Government’s overarching expectations of performance:

1. regulators do not unnecessarily impede the efficient operation of regulated entities
2. communication with regulated entities is clear, targeted and effective
3. actions undertaken by regulators are proportionate to the risk being managed
4. compliance and monitoring approaches are streamlined and coordinated
5. regulators are open and transparent in their dealings with regulated entities; and
6. regulators actively contribute to the continuous improvement of regulatory frameworks.

More information on the RPF is available on the Department of Prime Minister and Cabinet website.

1.3 Requirement

Regulators must self-assess their performance against the RPF annually. The results of the self-assessment must be:

• validated by an approved external stakeholder body—see comments above regarding validation
• certified by AMSA’s accountable authority—the AMSA Board; and
• provided to AMSA’s portfolio Minister and published no later than 31 December each year.

Method

1.4 Evidence

Where possible AMSA leveraged existing processes for data collection and analysis. The primary sources of evidence used for the self-assessment and validation were:

• the draft 2019–20 AMSA annual performance statements—available as part of AMSA’s 2019–20 annual report.
• the annual results of AMSA’s RPF customer online and phone surveys

AMSA self-assessment

Over the period 27 July – 7 August 2020 the AMSA Executive completed a self-assessment survey of AMSA’s performance against the RPF. In addition to exercising their professional judgement, the self-assessment was informed by documents listed at section 1.4.

The self-assessment survey consisted of six key statements/questions aligned to the RPF key performance indicators:

• Q1: AMSA helps vessel owners and seafarers safely operate or work on a vessel without getting in the way
• Q2: Our communications with those we regulate are clear and useful
• Q3: Given the risks involved in the industries AMSA regulates, the level of regulation is about right
• Q4: AMSA’s compliance and monitoring arrangements are well organised and efficient
• Q5: AMSA explains its regulatory decisions well, and
• Q6: AMSA is always trying to improve maritime regulations to create a safer and more efficient industry.

The response options were:

This year’s survey also included questions on AMSA’s progress against the opportunities for improvement identified in 2016–17 and 2017–18 self-assessments:

• 2016–17: how much progress has been made towards:
  • improving engagement and communication with a wide and diverse group of stakeholders – notably the DCV industry?
• 2017–18: how much progress has been made towards:
  • looking for more efficient ways of doing business with the DCV industry?
  • providing standardised technical/legislative advice to the DCV industry, and minimising the influence of individual interpretation by regulatory officers?

The response options for these questions were:

• no or little progress
• some progress; and
• significant progress.

If the response was ‘no or little progress’, respondents were asked to explain. If the response was ‘some’ or ‘significant’ progress, respondents were asked for examples.

Respondents were also asked to identify three opportunities for AMSA’s delivery of regulation in the future (see 1.8 for results).

Results

1.5 Self-assessment against RPF KPIs

The summary results of the 2019–20 self-assessment against each RPF KPI question are shown in Figure One (solid purple bars), including the results of the RPF customer survey (patterned purple bars). For comparative purposes Figure One also shows the:

• 2016–17 (solid blue bars), 2017–18 (solid orange bars) and 2018–19 (solid green bars) self-assessment results
• 2016–17, 2017–18, and 2018-19 annual results of the RPF customer survey (patterned blue, orange and green bars respectively).

The results and comments for each self-assessment KPI question are at Attachment 1.

1.6 Progress against 2016–17 opportunities.

Progress has been made. Table 2 details the evidence provided by the AMSA Executive in support of their assessment.

Table 2: Progress assessment – evidence

1.7 Progress against 2017–18 opportunities

Progress has been made. Table 3 details the evidence provided by the AMSA Executive in support of their assessment.

Table 3: Progress assessment—evidence

1.8 2019–20 Opportunities

Respondents were asked to consider how AMSA might deliver regulation in the future. The opportunities identified were:

• Amendments to the National Law
• Resolve some of the legislative impediments (noting the challenges)
• Digitisation of services
• Review administrative processes
• Promote stronger engagement from industry
• Continue to focus on customer satisfaction and engagement

Analysis

1.9 Self-assessment against RPF KPIs

Overall, there has been a slight increase since the 2018–19 self-assessment, with an average 4.96 ‘agree’ (4.43 in 2018–19).

There have been small movements across all the individual RPF KPIs, but given the small sample size of the AMSA Executive (n=4), these movements are immaterial.

1.10 Self-assessment against RPF customer survey

There has been a slight decline in the RPF customer survey results. It should be noted, however, that the number of survey respondents was much lower in 2019–20 than previous years. 

The most significant gap is for the KPI ‘AMSA explains its regulatory decisions well’ with the Executive scoring this question at 5.25 and the stakeholder participants scoring this question at 3.9.

Stakeholder respondents scored question 4: ‘Compliance and monitoring arrangements are well organised and efficient’ the lowest, only scoring 3.4, however this is a slight improvement from the 2018–19 results (3.22).

1.11 Impact of domestic commercial vessel participants

While the sample size is small, the RPF customer survey results provide a useful insight. The survey is available both online and as an opt-in phone survey for those who contact AMSA Connect. It is separate to AMSA’s complaints handling process, however, where specific issues are raised the feedback is passed to the operational area to address.

The DCV industry accounted for almost half of the RPF customer survey participants (note: participants who responded via the telephone survey were not provided the opportunity to select their industry sector. As these were all directed via a contact with AMSA Connect, it is assumed that these respondents were from the DCV industry).

The survey results indicate that DCV, as a single nationally regulated industry group, continues to be ambivalent towards AMSA. The DCV industry is now AMSA’s largest stakeholder group, and will therefore play a central role in gauging AMSA’s long-term success.

Those who identified as being from the DCV industry demonstrated the lowest levels of satisfaction with AMSA’s performance.

Conclusion

The 2019–20 self-assessment results broadly indicate that AMSA management:

• believes it has made tangible progress over the past year
• is very aware that there is room for improvement across the range of RPF KPIs, particularly in relation to the DCV industry.

AMSA continues to be fortunate that the daily interactions of its staff with stakeholders and its strong broader reputation contributes to a positive overall perception and a buffer of goodwill. However, AMSA management is very aware that ongoing goodwill is not guaranteed.

Attachment 1: AMSA 2019-20 RPF self-assessment detailed responses