Evaluations—STCW certificate of competency/proficiency and short courses—MT04

The purpose of this procedure is to:  

• Ensure courses presented by Registered training organisations comply with our approval requirements.
• Ensure the standards of training leading to the issue of certificates satisfy STCW requirements for the issue and continued endorsement of a Certificate of approval.
• Determine the effectiveness of the Registered training organisations quality system.
• Provide the Registered training organisations with an opportunity to improve its system of quality control.
• Provide an effective and efficient evaluation record, the data from which may be used to benefit maritime safety, education and training nationally.

Background

We as the Flag State designated approving Authority must make sure that approved courses meet the appropriate STCW Convention and national standards by a formal evaluation process using industry accepted evaluation standards and processes as required by STCW Regulation l/8.

The national standards recognised by us for our approved Registered training organisations are:

• Standards for Registered Training Organisations (RTOs) 2015
• Higher Education Standards Framework (Threshold Standards) 2015

We will conduct evaluations of our approved Registered training organisations to ensure compliance with ASQA and TEQSA Standards and will focus on STCW Regulations l/6, l/8 and l/12 and compliance of standards included in these Regulations. Our approved Registered training organisations must also incorporate these Regulations in compliance with these Standards.

Scope

This procedure details the process which our personnel authorised to conduct evaluations should follow when evaluating maritime training programs offered by a Registered training organisation to those who are seeking to gain a maritime certificate. It also includes information how to gain and maintain a certificate of course approval from us.

Definitions

Read Definitions for Registered training organisations delivering STCW AMSA approved courses—MT01.

Associated documents

• Australian Skills Quality Authority
• Higher Education Standards Framework (Threshold Standards) 2015
• International Maritime Organization (IMO) Model Courses
• International Convention on Standards of Training, Certification and Watchkeeping for Seafarers, 1978, as amended (STCW)
• Marine Order 70 (Seafarer certification) 2014
• Marine Order 71 (Master and deck officers) 2023
• Marine Order 72 (Engineer officers) 2014
• Marine Order 73 (Ratings) 2014
• Marine Order 74 (Masters and deck officers – yachts) 2014
• Maritime Training Package MAR (MTP) published by Australian Industry Skills (AIS)
• Standards for Registered Training Organisations (RTOs) 2015
• Tertiary Education Quality Standards Agency (TESQA)

In the case of the Australian Maritime College (AMC), reference should be made to the accredited curriculum documents.

Evaluation processes

1. Initial evaluation and issue of a Certificate of approval to Registered training organisations 

• Prior to the issue of a Certificate of approval, our evaluator may conduct an initial evaluation of the Registered training organisations in accordance with this procedure which may include a site visit or may conduct a desk top document review of the information submitted by the Registered training organisation requesting the course approval. Once a review has been completed and there are no outstanding non-conformances, on receiving a recommendation from the assessing Principal Advisor Nautical Certification (PANC) or Principal Advisor Engineering Certification (PAEC), the Manager may issue the Certificate of approval. A copy will be retained in the appropriate HPERM file.  The issue of the Certificate will allow the Registered training organisation to commence teaching the course.
• Subject to availability, we will endeavour to conduct a sampling of the course delivery as soon as practicable after approval. We may waive this requirement for existing Registered training organisations who have a good record with us.

2. Certificate of approval

A Certificate of approval may be issued for a maximum of five years. The Certificate will be issued subject to such conditions as determined by us. These conditions will include but are not limited to:

The Registered training organisation maintains all requirements for initial issue of the certificate.

Surveillance evaluations are allowed to be conducted.

The Registered training organisation must notify us of any changes to the system of quality control, training programs, change of lecturers, facilities and any event that may affect the quality of the program.

There will be a maximum of twelve participants for practical firefighting and survival sessions, unless otherwise authorised by us.

Course instructors are to be approved by us.

3. Surveillance and renewal evaluations

Surveillance and renewal evaluations will take the form as outlined below. We will conduct unscheduled evaluations when we consider it necessary to verify the course standard.

Our approved Registered training organisations which deliver the Diploma, Advanced Diploma or Degree level STCW courses will be evaluated annually. All other approved Registered training organisations which deliver STCW short courses will be evaluated biennially using a risk-based approach. These evaluations must be conducted within three months either side of the anniversary date of the issue date of your Certificate of approval.

If a renewal evaluation is conducted within three months before the expiry date of the certificate, then a new certificate may be issued for a maximum of five years from the expiry date of the existing certificate.

Applications for renewal evaluations must be in writing and must be received by us no later than 90 days before the Certificate expires.

If a Certificate of approval becomes invalid, you will not have approval by us to conduct the maritime training course(s).

4. Evaluation process

We will, where possible, conduct combined evaluations with the ASQA auditors to minimise disruption from the evaluation process. Our evaluator(s) must ensure that the Registered training organisation under evaluation meets all our requirements. The evaluation will be conducted within the requirements of the ASQA standard.

Where our approved Registered training organisation conducts engineer and deck training, both the PANC and PAEC should, where possible, conduct the evaluation; one will be appointed as Lead evaluator.

Where a Registered training organisation conducts specific training, then only the appropriate Principal should conduct the evaluation.

In circumstances where the appropriate Principal is not available, our appointed evaluator approved by the Manager may conduct the evaluation.

It is our standard practice to conduct annual evaluations of Registered training organisations who provide STCW Certificate of Competency courses and biennial evaluations of those Registered training organisations providing STCW short courses. If there are major non-conformances identified or we have concerns regarding the standard of a course(s) then the approval period may be reduced as determined by the Manager.

Other factors which may lead to an increase in evaluation frequency include:

• The number and type of non-conformances identified at the evaluation.
• An identified historical trend of non-conformances relating to a specific area.
• Misunderstanding or lack of knowledge of shipboard safety, operational and environmental subjects identified by our examiners when conducting oral examinations.
• A high failure rate in the oral examination.
• Serious complaints from industry or individuals concerning the quality of the training.

If the Manager determines that circumstances require a non-scheduled evaluation to be conducted, the Head of school (or similar) of the Registered training organisation will be contacted and informed of the reasons for the additional evaluation and the date when it will occur.

5. Evaluation preparation

Prior to starting an evaluation, the Lead evaluator must complete the following tasks:

• In consultation with the Registered training organisation, schedule a date for the evaluation.
• Estimate the time required to carry out the evaluation, considering the availability of the teaching and administrative staff and their workloads.
• Once the evaluation dates are determined, formally advise the Registered training organisation of the evaluation dates, the names of the evaluators attending, the times for the opening and closing meetings, and areas which our evaluators intend to concentrate on during the evaluation.
• Prior to the evaluation, send to the Registered training organisation a copy of our lecturer approval spreadsheet (HPRM D18/56867) for review.
• Minimise the disruption to lectures. (As the Registered training organisation is aware of the lecturers’ teaching commitments it is advisable for the Lead evaluator to allow the Registered training organisation to assign the evaluation schedule which should include time for the evaluator to talk to students and lecturers).
• Review recent changes to any regulatory requirements.
• Review the standards applicable to the activity to be evaluated.
  • From our appropriate HPERM files obtain copies of the previous evaluation reports, including copies of observation notes, non-conformance findings and the Certificate of approval including the addendum listing all the approved courses.
• Prepare a checklist and/or questions for the evaluation by taking copies of relevant documentation and/or selecting appropriate documentation for use during the evaluation.
• Review our local oral examination statistics to obtain information and trends of where candidates have failed.
• Request copies of audit and evaluation reports which the Registered training organisation has undergone since a previous AMSA evaluation to avoid re-evaluating areas unnecessarily.

6. Conduct of the evaluation

On arrival at the Registered training organisation the Lead evaluator must arrange a meeting with the Head of School (or similar) prior to the commencement of the evaluation to make formal contact.

The Lead evaluator will conduct an opening and closing meeting with relevant Registered training organisation staff members.

Evaluators are to collect information through interviews, examination of documents, observation of work being performed and work conditions in the area being evaluated. Evaluators are to note non-conformances and observations when they are identified.

The scope and depth of the evaluation is to be sufficient to determine adequate compliance with regulatory requirements, relevant industry and academic standards and the Registered training organisations own quality system.

During the evaluation, evaluators will:

• use the evaluation checklist and/or prepared questions
• record notes and times of opening and closing meetings
• collect relevant evidence to support findings
• review college or Registered training organisation working documents and procedures
• take photographs where this will support evidence of findings
• review lecturer qualifications and update our records.

7. Opening meeting

The opening meeting should commence at the agreed time, and where practicable, with all persons involved in the evaluation present. During the opening meeting the Lead evaluator will give an introductory talk and cover the following items:

• Introduction of the evaluation team.
• Confirm the purpose and scope of the evaluation.
• The areas the evaluators will concentrate on.
• A short summary of the evaluation methods and procedures (sampling procedures).
• Review and confirm the evaluation timetable and agenda.
• Ask for any questions concerning the evaluation process and clarify any ambiguities.
• State how non-conformances and observations will be addressed.
• Arrange for provision of suitable facilities to conduct interviews.
• Where applicable arrange for accompanying guides for onsite inspections.
• Advise that all information obtained during the evaluation will be treated in confidence.

8. Non-conformance findings and observation notes

Non-conformance findings

The evaluator is to classify any clearly identified irregularity against a specific requirement as a non-conformance and is to be stated as a non-conformance finding verbally at the closing meeting. The non-conformance is to be formally issued to the Registered training organisation in the evaluation report. All non-conformance findings are to be recorded in a clear and concise manner, supported by objective evidence, and recorded against a procedural requirement – there must be no ambiguity.

Examples of irregularities which may reflect a non-conformance finding are:

• the Registered training organisations approval arrangements
• education and training standards
• the availability or standard of equipment
• AMSA legislation requirements
• the Registered training organisations system of quality control
• Incomplete action plans.

If the evaluator finds a significant number of non-conformances, the Lead evaluator will inform the Manager and discuss the issues. The evaluation may be terminated, and the Registered training organisation instructed to take remedial action. Depending on the level of non-compliance, the Certificate of Approval may be suspended for a period of time or withdrawn and reissued as a short-term approval.

Observation

The evaluator is to record items where they consider a quality system improvement is warranted but is not a non-conformance as an observation. These are to be mentioned verbally at the closing meeting and are to be formally recorded in the evaluation report.

9. Closing meeting

Following completion of interviews and inspections the evaluators should allow time for discussion and recording of any observation notes or non-conformance findings.

Prior to the closing meeting the Lead evaluator will brief the Manager on the evaluation and identify both positive and negative outcomes.

At the allotted time the closing meeting should take place. Those in attendance should be the AMSA Lead evaluator and evaluators, relevant Registered training organisation staff, including the Head of School (or similar) and the document controller.

The main purpose of the meeting is to report on the results of the evaluation and to discuss all non-conformances and observations recorded during the evaluation.

At the meeting, discussion may occur regarding possible rectification action. However, all decisions on the method to close out non-conformances are the responsibility of the Registered training organisation.

At the closing meeting the Lead evaluator will thank the Registered training organisation and the staff for their co-operation during the evaluation and reassure them that all information gained from the evaluation will be treated in confidence. If the evaluation is not a certificate renewal evaluation, the Lead evaluator will sign and stamp the Certificate of approval providing the evaluation has been satisfactory.

10. Evaluation report

The Lead evaluator should complete the evaluation report and forward it to the Registered training organisation Head of School (or similar) within one calendar month of completion of the closing meeting.

The report is to address, as a minimum:

• the scope of the evaluation
• evaluation dates and identification of the section of the Registered training organisation identification of any referenced documentation
• identification of non-conformance findings and observations provided at the closing meeting
• the evaluator's assessment of the degree of compliance of the Registered training organisation with respect to the quality system documentation; all applicable regulatory requirements and our specified conditions and limitations associated with the Certificate of Approval
• identification of the evaluators and college staff involved with the evaluation
• the agreed timeframe to correct any non-conformance finding
• if considered necessary, the agreed date on which a re-evaluation may be performed to confirm that the non-conformance findings have been satisfactorily rectified.

The lead evaluator is also to submit a copy of the evaluation report to the Manager and the other evaluators if applicable and file in HPERM.

11. Evaluation follow-up

If the Registered training organisation has not contacted us and submitted the completed evaluation rectification action plan within the agreed time frame the Lead evaluator will contact the Head of School (or similar) of the Registered training organisation to follow up on the outstanding non-conformance findings;

The lead evaluator will notify the Manager of the situation and continue the follow-up process to achieve a satisfactory outcome;

The follow-up may consist of one or more of the following:

• signing off the non-conformances based upon evidence supplied in writing
• our evaluator/examiner conducting a supplementary evaluation
• a principal conducting a supplementary evaluation.

If outstanding non-conformances are not resolved within the agreed time frame, the Manager may withdraw the Certificate of approval from the Registered training organisation or issue a short-term Certificate of Approval. The Registered training organisation will be advised of the Manager’s decision.

12. Evaluation records

The Lead evaluator will place the following documents on the appropriate AMSA Registered training organisation HPERM file:

• evaluation report
• evaluation rectification action plan
• copy of the Certificate of Approval, together with the addendum listing all approved courses
• any other supporting documentary evidence
• update of lecturer qualifications (if necessary).