Ship Anomalous Behavior Detection Using Clustering and Deep Recurrent Neural Network

Abstract

In this study, we propose a real-time ship anomaly detection method driven by Automatic Identification System (AIS) data. The method uses ship trajectory clustering classes as a normal model and a deep learning algorithm as an anomaly detection tool. The method is divided into three main steps: (1) quality maintenance of the original AIS data, (2) extraction of normal ship trajectory clusters using Density-Based Spatial Clustering of Applications with Noise (DBSCAN), in which a segmented improved Dynamic Time Warping (DTW) algorithm is used to measure the degree of trajectory similarity, (3) the clustering results are used as a normative model to train a Bi-directional Gated Recurrent Unit (BiGRU) recurrent neural network, which is used as a trajectory predictor to achieve real-time ship anomaly detection. Experiments were conducted using real AIS data from the port of Tianjin, China. The experimental results are manifold. Firstly, the data pre-processing process effectively improves the quality of raw AIS data. Secondly, the ship trajectory clustering model can accurately classify the traffic flow of different modes in the sea area. Moreover, the trajectory prediction result of the BiGRU model has the smallest error with the actual ship trajectory and has a better trajectory prediction performance compared with the Long Short-Term Memory Network (LSTM) and Gated Recurrent Unit (GRU) models. In the final anomaly detection experiment, the detection accuracy and timeliness of the BiGRU model are also higher than LSTM and GRU. Therefore, the proposed method can achieve effective and timely detection of ship anomalous behaviors in terms of position, heading and speed during ship navigation, which provides insight to enhance the intelligence of marine traffic supervision and improve the safety of marine navigation.

1.1. Intelligent Maritime Traffic Supervision

As the density of marine vessels increases, maritime accidents occur frequently and marine traffic safety faces challenges. Traditional maritime monitoring is conducted by manual monitoring of Vessel Traffic Service (VTS), which is limited by complex and heterogeneous data, information overload, personnel fatigue, inattention, and cannot achieve a good monitoring effect. Intelligent maritime traffic supervision based on maritime data and computer technology can provide valuable information on the traffic situation in the sea area and assist in hazard supervision, so as to discover potential abnormal behavior of ships in time, reduce the occurrence of maritime accidents and violations, and guarantee the safety and economy of maritime navigation.

The Automatic Identification System (AIS) provides navigation information to ships and other receivers in real time, including dynamic, static and voyage-related information. The SOLAS Convention requires merchant ships of 300 GT or more on international routes to be equipped with AIS, with use in smaller ships and lifeboats gradually becoming more common. With intelligent marine traffic supervision, more AIS base stations, VTS and monitoring networks cover global coastal areas, and maritime traffic data collection grows exponentially from gigabytes to terabytes. Researching the maritime traffic patterns hidden in AIS data, discovering maritime traffic abnormalities in time, and improving supervision efficiency is an inevitable developing trend in the era of big data.

1.2. Abnormal Behavior of Ship

The abnormal behavior of a ship can be summarized as actions that do not conform to the predefined normal behavior model of a ship in the sea area, for example, a ship changing speed for abnormal reasons, changing course, deviating from the route, or appearing in an inappropriate area. These behaviors may also often be related to maritime accidents, illegal smuggling, pirate hijacking, or terrorist activities. Iphar believes several types of anomalies can be distinguished: point, contextual and collective anomalies. Roy defines the specific manifestations of abnormal ship behavior as position anomalies and motion anomalies, subdivided into 16 types of anomalous behavior. On this basis, Riveiro et al. summarize anomalous behavior related to situational context such as drug smuggling, human smuggling and terrorism, while Wolsing defines five general abnormal behaviors derived from the ship's trajectory: deviation from the standard course, unexpected AIS activity, unexpected port arrival, coming too close to other ships, and prohibited area entry.

It can be seen from the above research that the definitions of ship abnormal behavior differ according to different research emphases. However, essentially these anomalous behaviors will be identified in the ship's location, speed and course. This study therefore monitors three aspects — position, speed and course — as the observable signature of anomalous navigation.

1.3. Ship Anomaly Detection

In anomaly detection, a normative model should be defined based on the dataset, generated by predictable repeated events, with non-matching anomalies. Data-driven anomalous ship behavior detection consists of two steps: (1) learning from historical ship trajectory data to build a model representing normal ship behavior, generally represented as a statistical distribution or clusters of behavioral features; (2) anomaly detection by matching and comparing the deviation between the normal behavior model and the ship's motion data to be detected, recognized as anomalous if the deviation exceeds a threshold. Methods divide into statistical analysis, machine learning, and prediction-based anomaly detection. Statistical pioneers include Holst's grid and velocity-vector model with two-dimensional Gaussian distributions (improved by Laxhammar with EM and Gaussian Mixture Models, though requiring Gaussian-distributed data), Ristic's Kernel Density Estimation of motion patterns, Xiao's statistical description of AIS dynamics, and Rong's nonparametric Bayesian Gaussian-process trajectory prediction model with iterative updating from new AIS data.

Machine learning approaches learn a normal navigation model from historical tracks and build an optimal model through training: Vespe's unsupervised incremental learning of motion patterns adapting to environmental change; Mascaro's two-layer dynamic and static Bayesian networks with spatio-temporal context; Lei's space-sequence-behavior features with MT-MAD evaluation; Souza's Hidden Markov Model with data mining and multilayer filtering for fishing vessels; and Toloue's HMM on speed, position and course exceeding 96% accuracy. Prediction-based methods build a trajectory predictor and compare actual versus predicted values: Laxhammar's conformal predictor with Hausdorff trajectory similarity; Bomberger's unsupervised associative neural network for future normal-trajectory prediction; Zhao's in-water trajectory clustering feeding an LSTM predictor for real-time detection; and Tang's LSTM prediction combined with grid-based anomaly detection.

Statistical methods are easy to implement and accurate when historical data are sufficient, but threshold definition is complex, methods are problem-specific, and they fail when anomalies are evenly dispersed. Machine learning faces insufficient high-dimensional feature spaces and sample sizes, overfitting, local optima, and poor interpretability. Deep neural networks offer strong autonomous learning and feature extraction, and RNN variants improve efficiency and accuracy — yet studies applying deep recurrent networks to maritime anomaly detection remain scarce. Combining machine learning's modeling power with adaptive prediction will greatly improve detector applicability at sea — the gap this study addresses.

1.4. Motivation and Overview

The motivation of this research is to explore innovative research methods for ship AIS data quality maintenance, ship normal trajectory modeling, and ship trajectory anomaly detection based on data science and deep learning technology — dedicated to improving maritime intelligent supervision efficiency and maritime traffic safety, and providing scientific support to enhance maritime intelligent supervision capability. To achieve this goal, the paper proposes a real-time ship anomalous behavior detection method driven by AIS trajectory data: trajectory clustering extracts a normal-trajectory model within the water area, and a Bi-directional Gated Recurrent Unit (BiGRU) deep neural network serves as the anomaly detection tool.

Firstly, a set of data cleaning and repair methods is proposed for data quality problems in the original AIS data, effectively improving experimental data quality. Secondly, in DBSCAN ship-trajectory clustering, a segmented improved Dynamic Time Warping (DTW) distance measures trajectory similarity, fixing DTW's insensitivity to shape features, start/stop point positions, and trajectory direction. Finally, BiGRU is used as a constant predictor for real-time ship navigation-status detection: rather than merely comparing against the normal model, it autonomously recognizes the normal trajectory state from the deep network's own learned characteristics, achieving more accurate anomaly detection. Section 2 describes normal trajectory modeling (data quality maintenance plus DBSCAN with segmented-improved DTW); Section 3 the BiGRU prediction model and anomaly detection method; Section 4 experiments and results; Section 5 conclusions and future work.

2.1. Data Preprocessing

Original AIS data suffer anomalies through generation, encapsulation, broadcasting, transmission, decoding and storage, with variable noisy data; additionally most AIS messages are externally timestamped when accepted by the receiving station, so timestamp quality depends on the station rather than the true position-report time. Preprocessing is therefore necessary. Related work and observation categorize original AIS data problems into three aspects: insufficient trajectory data integrity (missing segments, unmatchable dynamic/static information, too few valid points — such data are excluded, and anchoring-state trajectories containing multiple voyages are split); insufficient trajectory data accuracy (field errors and movement-logic errors); and too much meaningless redundant data.

Accuracy repair follows the NMEA0183 protocol: MMSI (first three digits flag country code, last six the vessel identifier), longitude within [-180, 180], latitude within [-90, 90], and course within [0, 360) degrees — non-conforming points are eliminated. Motion-logic errors such as abnormal speed or position jumps are detected via average inter-point speed and kinematic constraints (forward/reverse acceleration bounds; estimated-position checks), with wrong speed values replaced by the two-point average and bad positions replaced by estimates. Redundancy is then reduced: with port-area traffic in the thousands daily and 2–30 s broadcast intervals, millions of slow-moving points accumulate, so the Douglas–Peucker algorithm extracts location and course feature points, plus a speed-change-rate metric retaining points whose rate exceeds 0.1 kn/s — preserving trajectory features while cutting storage and computation costs.

2.2. Trajectory Clustering Method

This section introduces the trajectory clustering model built on preprocessed AIS data, dividing sea-area traffic flow into distinct regular movement patterns. DBSCAN is a simple, effective density-based clustering algorithm with noise handling: it finds arbitrarily shaped clusters in noisy spatial data without presetting the cluster count and resolves spatially disjoint similar trajectories — ideal for unlabeled ship trajectories. Its two parameters are MinTrs (minimum points for a dense region) and eps (neighborhood distance); starting from an arbitrary unvisited object, all objects within eps (at least MinTrs) join one cluster, expanded recursively through neighborhood computation, as diagrammed in the paper's DBSCAN flowchart.

The key question is how to measure similarity between trajectories: AIS sampling timestamps and counts differ, forming unaligned discrete time series, so the study adopts Dynamic Time Warping (DTW), which locally stretches time to align sequences and finds the minimum-distance point matching with the shortest path through the Euclidean distance matrix. But classic DTW has three shortcomings for ship tracks: it cannot distinguish local trajectory characteristics, poorly distinguishes trajectory direction, and ignores start/end positions. The proposed segmented improved DTW converts point-to-point matching into sub-segment matching (point-to-segment distances averaged over segment length, plus angular distance between segments weighted with linear distance at 0.5 each), and doubles the weights of the first and last sub-segments to sensitize DTW to start/end positions — solving all three problems.

3.1. Bidirectional Gated Recurrent Unit Model

Deep neural networks map inputs to outputs through layered nonlinear mappings with powerful extraction and abstraction. The Recurrent Neural Network (RNN) suits spatiotemporal sequential data but suffers rapid deterioration of neuron-node memory; the Long Short-Term Memory (LSTM) model fixes this with autocorrelated core cells and three gates (input, output, forget) controlling information flow. The Gated Recurrent Unit (GRU) — a faster LSTM variant with fewer tensors and parameters and simpler training — merges the forget and input gates into a single update gate zt (with reset gate rt): the candidate state blends reset-modulated history with current input through tanh, and one gate simultaneously forgets (weight z) and remembers (weight 1−z), solving ordinary RNN gradient disappearance. Weight matrices Wz, Wr and candidate weights complete the formulation.

In LSTM and GRU, states propagate unidirectionally, so outputs depend only on past information. This study instead adopts the Bi-directional GRU (BiGRU): each training sequence passes through forward and backward networks joined to the same output, so the network adjusts not only from past context but also from future context via feedback — improving weight-update efficiency and trajectory prediction. This contextual awareness is why BiGRU serves as the study's real-time trajectory predictor and anomaly detector.

3.2. Method for Detecting Abnormal Ship Behavior

The basic principle: train the RNN on DBSCAN clustering results so that, from previous trajectory points and historical traffic patterns, the RNN acts as a real-time predictor of the ship's normal trajectory; deviation of the actual trajectory from the predicted normal trajectory signals abnormal sailing. The advantage is autonomous normal-state recognition from the network's own learned characteristics rather than mere comparison with a stored normal model — making the detector highly sensitive to prediction failure on trajectories that do not fit historical movement patterns, and differentiating predictions by anomaly degree. The pipeline runs in three steps. Step 1: collect historical AIS data in the monitored sea area, preprocess it, and extract the normal trajectory model with DBSCAN.

Step 2: the normal trajectory clusters from DBSCAN train the RNN to predict the ship's next-moment normal state. Each trajectory feature at time t is Y(t) = {lon, lat, sog, cog} (longitude, latitude, speed over ground, course over ground); a sliding window generates input/output sets, and the normal-state prediction model outputs Y(t+1)..Y(t+m) from the previous n steps. Step 3: predict the next-stage normal navigation status from past status and water-area history. Since most ship anomalies manifest in position, speed and course, the anomaly criterion is the deviation value combining geographic position distance with absolute speed and course deviations. Users set alarm thresholds per actual demand and training performance — with high and low thresholds for position (the most direct sailing-status indicator): the model alerts supervisors when real-time data exceed the high level or persistently exceed low levels, while accurate predictions mean normal sailing.

4.1. Experimental Data and Preprocessing

The study uses two months of 2019 AIS data from China's Tianjin Port area, selecting cargo ships and tankers. The raw data comprised 1,059,741 dynamic records, 376,333 static/voyage records, 3,884 ships (2,556 with additional information), and 712,214 track points. After preprocessing, 11,357 trajectories from 2,608 ships were retained: 30.3% of data had insufficient integrity (16.00% information mismatch, 14.30% insufficient spatial integrity), 1.76% of points had field errors (18,910 points), 0.67% had motion-logic errors (5,452 location, 1,736 speed errors), and compression achieved a 62.46% reduction to 267,365 key feature points. Before/after comparison shows messy trajectories, land crossings and trajectory mutations eliminated while original features are retained.

4.2. Clustering Analysis

To verify the segmented improved DTW, a clustering comparison of improved DTW, classic DTW and Hausdorff distance ran on 334 labeled ship trajectories with clear shape, direction and start/end differences, pre-marked as 8 clusters. With adaptive parameter selection (eps = 0.3 nm, MinTrs = 3), the improved DTW produced 8 clusters matching the labels (error rate 0.0058, i.e. 99.42% accuracy) — correctly separating clusters differing in local shape (clusters 1, 2, 4) and in start/end positions (2 vs 3 vs 6; 7 vs 8). Classic DTW merged some of these into 6 clusters (8.7% error), and Hausdorff managed only 4 clusters (17.9% error), missing opposite-direction and large local-shape differences. Environment: Intel i7-6700 CPU, 8 GB RAM, Windows 10, TensorFlow 2.0, Python.

4.3. Ship Abnormal Behavior Detection Experiment

4.3.1. Ship Navigation State Prediction

DBSCAN clusters were split 75%/25% into training and test sets to train BiGRU, with trajectory point intervals regularized to 2 minutes and all values normalized. With input time-step 5 and output step 1 (the next state predicted from five previous points), position, speed and course deviations evaluate the model. On two real test trajectories, predictions closely coincided with actual tracks, accurately capturing nonlinear steering and speed changes; BiGRU beat GRU and LSTM on average and maximum position, course and speed errors (e.g. trajectory 2 average position error 70.07 m for GRU vs 83.21 m for LSTM, with BiGRU lower still). Across iteration counts, LSTM and GRU errors fluctuated widely while BiGRU stayed stable with average position error under 90 m; overall BiGRU reduced average prediction error by 13.1% versus GRU and 21.3% versus LSTM.

4.3.2. Ship Abnormal Behavior Detection

Position low/high thresholds were set to 100 m and 250 m, heading and speed thresholds to ±3% of predicted normative values. Four real trajectories tested detection: No.1 (120 min: abnormal path, steering and shifting), No.2 (46 min: repeated abnormal channel-crossing steering), No.3 (30 min: sudden channel stop), No.4 (90 min: abnormal steering, sudden stop). Trajectory No.1 illustrates the detail: at the channel turning point the ship left the prescribed route into a reverse lane — at 12:32 heading changed to 292.3°, beyond the predicted normal course range, triggering a heading alert; 12:34–12:36 course deviation pulled position off the predicted track, triggering low-grade position Alert (1) until normal sailing resumed at 12:40. At 13:10 a sharp right turn sailed out of the channel while the predicted position stayed inside, triggering high-grade position Alert (2) through 13:22, with abnormal speed fluctuations also detected. At 13:40 speed abruptly rose 3.2 kn in 2 minutes above the predicted 12 kn — bad seamanship threatening channel traffic — triggering a speed alert, then position Alert (3) as the track diverged; at 13:50 speed plummeted 3.6 kn, triggering Alert (4).

Three metrics evaluate results: detection accuracy rate AR (detected abnormal points over actual abnormal points), false rate FR (misidentified normal points over normal points), and timeliness (timely recognition plus computing time). Across the four trajectories, BiGRU averaged 96.35% accuracy (93.75%, 91.66%, 100%, 100%) with 1.67% false rate and 9/9 timely recognitions (36.2–9.4 ms per track); GRU averaged 93.23% accuracy with 6.46% false rate and 8/9 timely; LSTM averaged 88.15% with 9.79% false rate and 8/9 timely — BiGRU misidentified only three points in Track No.1, and its computing time grows least with trajectory size, giving the best real-time performance. The method thus detects abnormal position, speed and heading behavior timely and effectively.

5. Conclusions

Intelligent maritime traffic monitoring overcomes traditional limits of supervisor inexperience and physical fatigue, raising supervision intelligence, informing traffic situations, assisting hazard supervision, and reducing accidents, violations and traffic-flow inefficiency. This study takes AIS broadcast trajectories as its object and proposes trajectory anomaly detection driven by AIS data, with trajectory clusters as the normal model and a deep network as the detection tool. Its threefold contribution: (1) AIS data-quality analysis across integrity, accuracy and conciseness, with a complete preprocessing suite (trajectory segmentation, base/position/speed abnormality repair, spatio-temporal feature extraction preserving features at 62.46% compression); (2) DBSCAN-based high-quality normative traffic models with a segmented improved DTW sensitive to direction, local shape and start/stop positions, reaching 99.42% clustering accuracy far above classic DTW and Hausdorff; (3) a BiGRU trajectory predictor solving RNN vanishing/exploding gradients with bidirectional context, achieving 96.35% anomaly accuracy at 1.67% false alarms with timely warnings, outperforming LSTM and GRU.

Limitations point to future work: the method needs high-quality sample trajectory models, and currently discards data with unmatchable dynamic/static information or too few valid points — repairing such data could optimize AIS mining. The normal model covers only cargo ships and tankers; shorter vessels such as fishing boats should be included for specific waters, and adaptive normal-model updating with new data should be strengthened for practice. Ship behavior also depends on spatio-temporal context — nearby vessels, weather, sea state — so correlating navigational-environment semantic analysis with behavior anomaly detection is another future priority.