Evaluation of Maritime Cyber Security (MarCy) Training Programme

Abstract

The prevalence of digital technologies is growing in the maritime industry, as in other sectors. Consequently, concerns regarding cyber risks are also escalating. Incidents have occurred in the industry, and findings from academic studies further validate these concerns. While technical measures are being taken against cyber threats, the human element remains another crucial aspect that requires strengthening. To effectively combat cyber threats and vulnerabilities, it is imperative to enhance individuals' awareness through education and training. In order to address the cyber security training needs of maritime professionals and students, we have developed an approach called the Maritime Cyber Security (MarCy) training programme. In this study, we evaluate all stages of the proposed programme through four conducted training sessions involving different learner groups. As a result, the MarCy programme was improved based on the findings obtained during the training sessions and the feedback from the learners. This study validates that the MarCy programme is an effective approach to meet the cyber security training needs of various groups in the maritime domain.

1. Introduction

Sea transport handles over 80% of the global merchandise trade volume [42], and as a result, the maritime industry has become a vital sector with millions of professionals working in it. The number of seafarers alone is approximately 1.9 million [2]. The Standards of Training Certification and Watchkeeping (STCW) Code currently lacks any requirements or recommendations for enhancing the cyber awareness of seafarers. Therefore, seafarers, beyond the training requirements of the International Maritime Organization (IMO), typically receive cyber security training during their professional careers. However, the possibility of this situation changing is likely. The Republic of Korea submitted a proposal in December 2021 to discuss the importance of integrating cyber security training into the STCW [18]. The Sub-committee on Human Element Training and Watchkeeping at the IMO was invited to deliberate on relevant provisions concerning training for seafarers in the field of cyber security.

We developed the MarCy training programme by incorporating the insights and opinions of experts, with the aim of its application in maritime cyber security training courses. The purpose of this study is to evaluate the applicability and effectiveness of the MarCy training programme. This work is a continuation of our previous work proposed by Oruc, Chowdhury, and Gkioulos [30]. The contribution of our study can be summarized as: evaluation of the MarCy training programme, evaluation of training sessions, and sharing observations and participants' feedback. In this study, the MarCy programme was implemented to develop cyber security training courses for students and professionals. Before the training courses, a pre-requisite survey was conducted by holding a meeting with the leaders of partner organizations. Subsequently, partners sent another pre-requisite survey to invitees to gather their training expectations.

While the MarCy programme has the potential for expansion with additional modules to cater to various stakeholders in the maritime industry, the focus of the original paper is specifically on the implementation of cyber security training for seafarers and office staff in shipping companies. This study demonstrates the application of the MarCy programme in the training of students from different departments (both maritime and non-maritime), office employees in shipping companies, and technical staff from class societies. Even though this study does not cover the training of all professional groups, it is possible to implement the MarCy programme to provide cyber security training for these groups.

2. Background

Over the years, several models have been put forth to guide the design and development of training programs. One such model is the Critical Events Model (CEM) introduced by Leonard Nadler in 1982, which has acquired recognition as a well-established and extensively described approach to training design. The CEM provides a comprehensive framework for designing training courses. It is not limited to formal education settings, but can also address the training requirements of various organizations. Notably, the CEM is well-suited for industries characterized by rapid changes, offering a flexible and adaptable approach [25]. We made three modifications to CEM with the MarCy programme: adding a modular approach, tailoring it specifically for maritime cyber security, and replacing the original evaluation phase with our own evaluation approach relying on internal stakeholder feedback.

The modular training approach involves offering learners relevant components of a training program tailored to their individual training needs. This approach has been implemented for various training requirements and has proven particularly effective in vocational training [16, 11]. By providing attendees with only the necessary knowledge aligned with their learning needs, this approach minimizes disruptions to their professional lives. Consequently, modular training is a cost-effective method that can be delivered online, offering flexibility to both the training designer and the learner [39].

We employed the Delphi method to evaluate the MarCy training programme. The Delphi method is widely recognized in the literature as a suitable research instrument for gathering judgments and opinions on topics where knowledge may be incomplete [40]. This method involves an iterative process that aims to collect feedback from a selected panel of experts who provide their insights anonymously. We chose the Delphi method as the validation technique for our programme because it enables us to gather weighted feedback from the panel of experts and facilitates open debate without the need for practical evaluation methods such as experimentation.

3. Related Work

Erstad et al. [14] explore the application of a Human-Centered Design (HCD) approach for maritime cyber resilience training. The authors propose using HCD for the development of tailored maritime cyber resilience training, including simulator-based team training. By engaging with end-users and incorporating learning theories, the training becomes realistic and relevant to the learners' needs.

Canepa et al. [7] examine the challenges of maritime cyber security training and the use of a cyber range as a solution. The paper presents the Cyber-MAR project [10], which implements a federated cyber range solution as part of a cyber security training platform tailored to the maritime sector. The methodology includes qualitative analysis through literature review and analysis of target groups, as well as quantitative analysis of the results from the initial LMS training.

Potamos et al. [37] present a novel training curriculum and offer design guidelines for creating activities on a maritime cyber range, with the aim of strengthening defenses against ransomware attacks. The curriculum emphasizes structured walkthrough practice, which encourages active learning and enhances the practicality and memorability of the educational experience.

La Vallée et al. [21] discuss maritime-specific training conducted through a federation of cyber ranges. The training covers various aspects of defensive actions, focusing on the impact of ECDIS console infection on the navigation system.

The CYMET research project by the IAMU aims to enhance cyber awareness in the maritime industry through education and training [1]. The project evaluated the training needs of seafarers and provided recommendations for maritime education and training. A web-based training solution was proposed using Moodle [24] and itsLearning [38] platforms.

Chowdhury and Gkioulos [8] performed a study to develop and evaluate two participant-centered cyber security training exercises using the Personalized Learning Theory (PLT)-based training framework. Evaluation of both exercises showed that participants' engagement and motivation increased as they were involved in exercise development, leading them to use the training tools independently.

4. Methodology

In this study, the MarCy programme is implemented and evaluated, with a particular focus on evaluating leaders', learners', and invitees' engagement and feedback. To this end, the following process was pursued to make quantitative and qualitative analyses: identification of partners, performing pre-requisite surveys for leaders and invitees, analysis of training expectations, development of materials such as presentations and post-assessment surveys, performing training and post-assessment surveys, analysis of exams, evaluations, and feedback, and improvement of the MarCy training programme.

For the purpose of evaluating the effectiveness of the MarCy training programme, we decided to conduct training sessions. To facilitate this process, we actively sought out partners who would collaborate with us in organizing the training. In selecting our partners, we placed importance on ensuring a diverse range of potential learner profiles. Ultimately, we successfully established partnerships with four organizations: a class society, a student club affiliated with a university, a maritime faculty, and a maritime training company.

After determining our partners, we prepared two different pre-requisite surveys to understand the training needs and expectations. One of these surveys was designed for the leaders of our partners, while the other was aimed at invitees. Firstly, we conducted separate meetings with the leaders of our partners to perform pre-requisite surveys, each lasting 2.5 hours. After analyzing the training expectations, the training sessions were planned and post-assessment surveys were developed based on potential learner profiles and the training modules to be provided.

Through the post-assessment survey, information about the learners' background was collected and the questions in the Quiz were answered by learners prior to the training. Then, the training modules were delivered by the instructors. After each module, learners were asked to evaluate the module. After all modules were covered, learners answered the questions in the Test. Finally, the overall training was evaluated by the learners. The conducted training sessions were analyzed, considering the observations of the instructors, evaluations, feedback, and exam results, and the MarCy programme was improved based on the findings obtained.

5. Evaluation of the Programme

Initially, pre-requisite surveys were conducted with the leaders of all partners. It was observed during these meetings that the leaders lacked sufficient knowledge in the field of maritime cyber security. Therefore, it was necessary to provide them with detailed explanations regarding the questions included in the pre-requisite survey. The average Quiz scores in the conducted training sessions range from 43 to 51, while the Test scores range from 61 to 78. The training sessions resulted in an average score increase of 52%. It has been observed that all Test scores are higher than the Quiz scores, indicating that the conducted modules provide benefits to the learners.

In the identify the needs of the organization step, we found that the stated needs were already covered by the content we recommended in the MarCy programme. We observed that the leaders faced challenges in identifying their training needs because of their limited knowledge of maritime cyber security. We confirmed the modules specified in the MarCy programme met the needs of our partners, except for the Student Club, leading to the development of the M12 Autonomous Ships module.

In the specify job performance step, the roles and responsibilities of potential learners were examined. The study showed that employees might have combined roles such as "DPA & CSO", and that the responsibilities of professionals might also be a combination based on roles. The programme's recommendations were refined based on these findings, including adding the "safe manning of ships" responsibility.

In the identify learner needs step, modules were evaluated by learners according to their responsibilities. The evaluations provided by the learners after the training confirmed the validity of the MarCy programme's recommendation that training should be tailored to individuals' roles and responsibilities. The case study method was particularly preferred by learners across all partners.

In the build curriculum and select instructional strategies steps, partners' leaders were consulted. We developed a curriculum for the M12 Autonomous Ships module. Lectures, case studies, and discussions were implemented as instructional methods. All learners found lecture, discussion, and case study methods effective. Issues with training material delivery, human resources, and physical resources were observed and recommendations were made.

The training sessions were organized in online (live), on-site, and hybrid formats, allowing the MarCy programme to be verified for all three implementations. Training duration ranged from 1.5 to 3 hours, with approximately 20-25 minutes allocated per module. Half of the Class Society learners found the training duration insufficient. The MarCy programme recommends allocating at least two hours per training module.

This study verified that the MarCy programme can be used for the training of students, professionals working in ship operators, and technical personnel in class societies. It was confirmed that a new training module can be created using the programme, and that it can be used for hybrid, online, and on-site training programs. In future studies, training programs for other dimensions of the maritime domain such as naval forces and maritime authorities can be developed and evaluated using the MarCy programme.

6. Conclusion

This study demonstrated that background knowledge is crucial in understanding the cyber risks of critical ship systems. Therefore, when forming learning groups, it is beneficial to group individuals with close knowledge levels. Our study showed that regardless of the initial differences in learners' knowledge levels in cyber security, they converged towards each other after the training. It was found that at least 31% of maritime companies have previously experienced cyber attacks, while only 25% of office employees have received maritime cyber security training.

Based on the exam results before and after the training, this study demonstrates that the MarCy programme increased learners' scores by 27% to 81%. According to the learners' opinions, it is emphasized that a mandatory course should be developed by following the MarCy programme to address the cyber security training needs of cadets. This research contributes to bridging the gap between academia and industry in maritime cyber security.

This paper has received funding from the Research Council of Norway through the Maritime Cyber Resilience (MarCy, project number 295077) project and the NORCICS centre (project number 310105). The content reflects only the authors' views.

References

Full reference list (46 sources) available in the original PDF.