A Review on Dropout Regularization Approaches for Deep Neural Networks within the Scholarly Domain

Abstract

Dropout is one of the most popular regularization methods in the scholarly domain for preventing a neural network model from overfitting in the training phase. Developing an effective dropout regularization technique that complies with the model architecture is crucial in deep learning-related tasks because various neural network architectures have been proposed, including convolutional neural networks (CNNs) and recurrent neural networks (RNNs), and they have exhibited reasonable performance in their specialized areas. In this paper, we provide a comprehensive and novel review of the state-of-the-art (SOTA) in dropout regularization. We explain various dropout methods, from standard random dropout to AutoDrop dropout (from the original to the advanced), and also discuss their performance and experimental capabilities. This paper provides a summary of the latest research on various dropout regularization techniques for achieving improved performance through "Internal Structure Changes", "Data Augmentation", and "Input Information". We can see that proper regularization with respect to structural constraints of network architecture is a critical factor to facilitate overfitting avoidance. We discuss the strengths and limitations of the methods presented in this work, which can serve as valuable references for future research and the development of new approaches. We also pay attention to the scholarly domain in the discussion in order to meet the overwhelming increase of scientific research outcomes by providing an analysis of several important academic scholarly issues of neural networks. Keywords: dropout; regularization; overfitting; data augmentation; dropout benefits.

1. Introduction

In recent years, deep learning techniques have performed remarkably in artificial intelligence improvement and automation. One of the most critical concerns in deep learning research is the regularization method used to prevent overfitting during training. Dropout [1], which was proposed in 2012, is a well-known regularization method that outperforms other regularization processes during optimization. It is used in various neural network tasks, such as image segmentation, data augmentation, image generation, machine translation, and image classification to reduce overfitting during training. The concept of dropout is relatively easy to implement compared to other regularization techniques. Initially, this technique was primarily applied to fully connected models where all neurons in one layer are connected to all neurons in the next layer. However, over time, numerous variations have been implemented for different perspectives and solutions to various problems. Those variations also include a number of modifications to encourage the regularization performances of existing neural network architectures.

For example, standard dropout [2], DropConnect [3], DropBlock [4], Skipout [5], MaxDropout [6], DropPath [7], selective CNN dropout, cutout [8], adversarial dropout methods [9], and spectral dropout [10] have attempted to explore the regularization issue from several aspects. This review paper provides an overview of previous and ongoing research on dropout regularization methods. While not exhaustive, we delve into specific dropout methods that are particularly relevant or beneficial for specific fields of research. We collect major dropout methods that are widely used in deep feedforward neural networks as well as convolutional neural networks [2, 11–22] due to their efficiency and effectiveness. Furthermore, our paper presents a comprehensive review, where we thoroughly explore the diverse perspectives and contributions of these methods in neural networks.

2. Background

In neural networks, regularization is a modification technique to the learning algorithm, which reduces its generalization error with the balance of its training error. This technique is particularly useful in neural networks, where the primary goal is to minimize overfitting during training. The benefits and drawbacks of each technique vary, depending on the specific problem and dataset. In this section, we specifically focus on regularization, with a particular emphasis on dropout regularization. To ensure a comprehensive explanation, we divide this discussion into two subsections.

Basic regularization methods are L1 and L2 regularization methods [38]. L1 regularization applies a constraint proportional to the sum of the absolute values of weights while L2 regularization applies a constraint proportional to the sum of the squares of weights. By doing so, the model is encouraged to learn smaller weight values, which in turn helps to prevent overfitting. In recent years, regularization algorithms for deep learning models, like feedforward and convolutional neural networks, have developed quickly. Extended methods of L1 and L2 regularization include weight decay [39], early stopping [40], and entropy regularization [41]. One popular regularization method is early stopping. This approach monitors the validation loss during the training process and stops it early before the model overfits the training data [40]. The final model is selected based on the lowest validation loss. Batch normalization [42, 43] is a method that normalizes the input to each layer of a neural network. It reduces the internal covariate shift and improves the convergence rate by smoothing the optimization landscape. It can also serve as a type of regularization by adding noise to the input of each layer. Data augmentation [44] is a technique used to increase the size of the training dataset by creating modified versions of the original data. The methods of creating modified versions include flipping, rotating, cropping images, etc. The original dropout [1] method randomly drops out units during the training phase. Dropout methods as regularization algorithms have seen rapid development in deep learning models, like feedforward neural networks (FNNs) and convolutional neural networks (CNNs).

Deep neural networks with more trainable parameters are generally desirable for extracting robust features from data. The networks with a large number of trainable parameters are prone to overfitting. To address this issue, in the standard dropout, the network switches off randomly chosen neurons during training [1]. On the one hand, EDropout, an energy-based dropout method, has been employed in [45] to perform the dropout of visible and hidden units in deep neural networks. EDropout introduces an energy-based measure to identify neurons that contribute the least to the overall energy of the network. These neurons are removed during training and testing to reduce the network's complexity and improve its efficiency. One possible way to enhance the performance and robustness of a CNN model is through the regularization dropout technique. The max-pooling dropout [46] technique randomly drops out some of the max-pooled values during the training of the convolution network. In terms of performance, max-pooling dropout is similar to the original dropout, which selects activation neurons at random [46]. To obtain accurate and robust models before testing, the researchers suggested using probabilistic weighted pooling instead of max-pooling for each feature map. The use of probabilistic weighted pooling has been shown to be effective based on empirical analysis. In the field of language modeling, Merity et al. introduced several regularization techniques, including dropout, to analyze their impact on LSTM language models [47]. In their study, the authors explored different regularization techniques including weight-dropped LSTM, along with different optimization methods, such as Adagrad and Adam. Through their experimentation, they aimed to analyze the impacts of these techniques on LSTM language models and determine their effectiveness in improving model performance.

3. Theoretical Concept of Dropout

In recent times, deep learning architectures have significantly evolved and expanded leading to improvements in various tasks, such as classification, object detection, and segmentation, etc. But, the utilization of larger and more complex deep learning architectures comes with the downside of increased risks of overfitting during training. In order to tackle the challenges in deep learning, researchers have put forward various regularization techniques with dropout being particularly prominent among them. The standard dropout [1] method can be used to relieve the overfitting issue, which randomly drops neurons from the neural network during training. For example, consider a neural network with an individual hidden linear layer of N units. Then the activation function Softmax is identical, taking the geometric arithmetic mean of the final output of the 2^N workable networks under standard Softmax [2]. This method is closer to a machine learning approach, such as the bagging method [48], which has trained the instance separately and the output inference has used the arithmetic mean.

Let us consider a single linear layer within a neural network. This layer is commonly referred to as a linear layer because it incorporates the behavior of a linear activation function, f(x) = x. In this (Figure 1) layer, the final neuron (which serves as the output of the layer) is obtained by calculating the weighted sum of all the inputs. While this simplified mathematical explanation holds empirically for certain non-linear networks, it is important to note that the estimation of the model involves minimizing a loss function. The ordinary least square (OLS) loss, Ln = 1/2 (t − Σ wi·Ii)^2 (1), represents the loss function used in general neural networks, while LD = 1/2 (t − Σ δi·wi·Ii)^2 (2) corresponds to the loss function specific to dropout networks. In the dropout network, the dropout rate δ follows a Bernoulli distribution with a keep-probability parameter. The analysis shows dropout approximates implicit ensembling of an exponential number of weight-sharing sub-networks, explaining its superior regularization power.

4. Contribution of Dropout Regularization Methods

In this section, we discuss the contribution of several commonly used dropout implements based on their performance under the dropout operation. The advantages of dropout regularization methods play a vital role in consolidating knowledge, highlighting key benefits and inspiring further research in the field of deep learning. The primary aim of this section is to highlight the advantages of dropout regularization methods and provide a comprehensive and consolidated overview of the benefits. In the following section, we provide a concise overview of the benefits that these methods offer.

On effectiveness improving, dropout techniques have significantly improved model effectiveness in utilizing training and promoting data across procedures: • Data augmentation. These methods [8, 41, 50, 51] augment available data by introducing input noise, generating additional training samples and enhancing model effectiveness during training — each training epoch employing dropout to present a new dataset by randomly dropping training data. • Preventing overfitting. Most dropout approaches [52, 53] serve regularization to mitigate overfitting — dropping neurons during training reduces interdependence between overfitting prevention and neurons, since random dropping obscures individual-neuron-to-overlearning relationships; dropout is widely used in DNNs to improve generalization. • Enhancing data representation. These methodologies [54–57] enhance data representation in pre-training — some generate random masks applied to input-data segments, using unmasked portions to predict masked ones, increasing data representation. • Preventing over-smoothing. Per research analysis, dropout in graph convolutional networks prevents over-smoothing [58] — overly deep GCNs make every node's representation identical, losing information; several studies find dropout effective mitigation.

On efficiency improving, dropout techniques improve both model effectiveness and efficiency across processes: • Model compression. Several dropout methods serve model compression [25, 59, 60] — random neuron dropout eases later structural compression such as neural pruning, and summarization methods cut parameter counts for better training efficiency and less overfitting. • Model uncertainty estimation. Well-known uncertainty-estimation techniques [15–17, 61] include interpreting dropout as Bayesian learning in Monte Carlo dropout — a Bayesian approximation of a deep Gaussian process — though it relies on grid search, nearly unusable in deeper models and reinforcement learning for its heavy time and resource cost. • Accelerate GCN training. Dropout in graph networks accelerates training by reducing the compute load of propagation over large neighborhoods while preserving representation quality.

5. Categorization of Dropout Regularization Approaches

During the period from 2012 to 2022, we categorized each dropout regularization approach based on its year of introduction and development. The timeline of these dropout regularization approaches is visually depicted in Figure 2. The graphical representation depicts the temporal continuity of dropout regularization techniques, providing a clear overview of their scenario over the years. Figure 3 illustrates the taxonomy of dropout approaches, categorized into three main categories: internal structure change, data augmentation, and input information. Each category further consists of several subcategories. By organizing the dropout approaches in this taxonomy, we aim to provide a clear and systematic overview of the different types of dropout techniques based on the changes they induce in the internal structure, their usage in data augmentation, and their impact on input information. This taxonomy serves as a valuable reference for understanding the various dropout approaches employed in deep learning. The following section provides an in-depth analysis of the taxonomic classification.

The timeline starts with standard dropout (Hinton et al.) in 2012 [1], then Maxout (Goodfellow et al.), DropConnect (Wan et al.), and fast dropout (Wang et al.) in 2013, multiplicative Gaussian noise (Srivastava et al.), annealed dropout (Rennie et al.), and empirical dropout analysis (Warde-Farley et al.) in 2014, Standout (Ba et al.), variational dropout (Kingma et al.), RNNdrop (Moon et al.), max-pooling dropout (Wu et al.), and spatial dropout in 2015, evolutionary dropout (Li et al.), variational RNN dropout (Gal et al.), Monte Carlo dropout (Gal et al.), selective CNN dropout (Park et al.), Swapout (Singh et al.), stochastic depth (Huang et al.), guided dropout (Keshar et al.), targeted dropout (Gomez et al.), information dropout (Achille et al.), and fraternal dropout (Żołna et al.) in 2016, and DropBlock (Ghiasi et al.) plus adversarial dropout methods (Park et al., Saito et al.) in 2018 — tracing the field's maturation from single-neuron dropping to structural, data, and information dropping.

6. Dropout Based on Internal Structure Changes

Different approaches have been proposed in addition to dropout for regularization in deep learning. In our review paper, we defined internal dropout-based regularization, which involves changing the weights and kernels during the training period without making any changes to the input values. Individual neuron dropout [1, 22, 65], as mentioned earlier, involves randomly dropping or deactivating individual neurons during the training phase. Each neuron has a probability of being dropped out, which means its output is set to zero. This helps prevent overfitting by reducing the reliance on specific neurons and encourages the network to learn more robust and generalizable representations. On the other hand, group neuron dropout takes a different approach. Instead of dropping individual neurons, it drops entire groups or subsets of neurons simultaneously. These groups can be predefined based on various factors, such as their spatial location or functional role within the network. In the following subsection, we provide a detailed discussion of some popular approaches.

In this section, we explore various popular neuron dropout approaches based on both feedforward neural networks (FNNs) and convolutional neural networks (CNNs): first, DropBlock [4] — deep networks generally perform best when parameters are over-optimized and trained with significant noise and regularization like weight decay and dropout. DropBlock is a form of structured dropout removing all units within a contiguous region of a feature map (tensor): dropped-region size is set by a hyperparameter, randomly selected each training iteration (Figure 4). Moreover, gradually increasing discarded units during training raises accuracy and desensitizes the model to hyperparameter variations such as learning rate and batch size. Image classification with AmoebaNet-B and ResNet-50 showed improved accuracy over other dropout variations: on ImageNet, cutout and AutoAugment beat the 2% ResNet-50 baseline, and AmoebaNet-B gained about 0.3% accuracy.

Second, MaxDropout [6]: while standard dropout [1] randomly drops neurons from training, MaxDropout deactivates neurons' computation from their activations — tensor values are first normalized, then outputs zeroed with threshold p selecting higher values for deactivation (Figure 5). The logic is that the highest, most decision-dominant activations deserve dropping to push the network toward more robust alternative features — competitive on CIFAR-10/100, especially combined with cutout as the performance table shows.

On dropout with recurrent neural networks (RNNs), a key application challenge is that RNNs have recurrent connections — the same weights at each time step — so dropping different units per step causes inconsistent, unstable behavior harming RNN performance. Yarin Gal and Zoubin Ghahramani illustrated a robust technique for applying dropout regularization to RNNs [15]: the key idea is showing how dropout can be applied to RNNs in a theoretically grounded manner, deriving a mathematical formulation from adding noise to hidden units. V. Pham et al. [70] demonstrate how dropout regularization improves RNN models for handwriting recognition — applying dropout only to feedforward connections, not recurrent ones, implemented in practice as a separate layer passing through input values except at dropped locations. "Recurrent dropout" [71] is another RNN dropout preserving important hidden-state information — introducing a "masking" tensor applied to the hidden state before the activation function, randomly generated each training iteration with probabilities from a dropout-rate hyperparameter.

7. Dropout Based on Data Augmentation

Data augmentation is the process of modifying data to reduce variance by making slight modifications to images and is used to generate more training data. Some dropout methods can serve as a form of data augmentation allowing for improved generalization without requiring specific domain knowledge [50]. In this section, we discuss the effectiveness of dropout based on data augmentation as a technique for improving model performance. While focusing on high-performance dropout based on data augmentation, we explore its capabilities and demonstrate its effectiveness in augmenting data for improved model performance.

The Cutout method [8] is a straightforward but promising data augmentation technique: during training it serves image augmentation and regularization by masking out square regions of the input data individually. In [8], the researcher analyzed optimal dropped-part sizes on CIFAR-100 and CIFAR-10: the optimum varied with per-class instance counts computed from the dataset, and cutout enhanced CNN strength and performance — for best results, CIFAR-10 dropped a 16 × 16 region while CIFAR-100's best was 8 × 8 (Figure 6 illustrates the workflow). Random erasing [51] is among the most common and comprehensive augmentation methods (Figure 7), operating dropout-like: it randomly selects an image area and replaces its pixels with random values, "erasing" that portion — used in CNN training to reduce overfitting and harden the model. Best performance used several architectures on Fashion-MNIST, CIFAR-10, and CIFAR-100, though it works best for WRN-28-10 and ResNeXt-8-64. Per grayscale analysis, random erasing improved WRN-28-10 from 4.01% to 3.65% top-1 error on Fashion-MNIST.

From a different perspective, demystifying dropout [33] offers another view: developed from theoretical and mathematical concepts covering forward and backward passes, it improves both standard dropout [1] and data augmentation [8]. The key point is that it randomly drops features during the forward pass but drops no features nor backpropagates errors during the backward pass; conversely, in backward dropout, all features are kept in the forward pass even when dropped, and only dropped-feature gradients are zeroed in the backward pass — a novel perspective on augmented dropout. SmoothMix [72] is a state-of-the-art regularization method preventing overfitting and improving data augmentation — mitigating the strong-edge problem via smooth boundary change between two images: the authors introduced kernel filters and injected noise to create an augmented input between two images. For evaluation, they compared ResNet-50 and PyramidNet-200 results on ImageNet and CIFAR-100.

8. Dropout Based on Input Information

Some dropout methods directly discard a portion of input information during training, serving various purposes in different scenarios. These purposes may include regularization, data augmentation, or enhancing data representation during the pretraining stage. In 2016, Sennrich et al. [73] introduced WordDropout as a technique in machine translation, which involves dropping out words from the input data. Another notable technique, proposed by Ghazvinine et al. [74] in 2019, is Mask-Predict. While most machine translation systems generate text from left to right, Mask-Predict adopts a masking approach to train the model. It begins by predicting all target words and then iteratively masking and regenerating a subset of words for which the model exhibits the least confidence. Current data enhancement methods can be broadly categorized into three main groups: spatial transformation, color distortion, and information dropping. Within the realm of information dropping, the act of discarding two-dimensional input information is commonly considered a data enhancement technique. In 2019, Verma et al. [75] introduced manifold mixup as a novel approach. Manifold mixup extends the concept of mixup to the feature level. The underlying idea is that features contain higher-order semantic information, and performing interpolation at the feature level can yield more meaningful samples. By applying manifold mixup, the model can benefit from the diverse and informative characteristics of different features. In 2019, Yun et al. [76] introduced CutMix as an advancement over the techniques of mixup and cutout. While cutout fills specific regions of an image with meaningless content, which may hinder the model's ability to utilize the complete training data, mixup employs linear interpolation but generates images that may not resemble natural images. In contrast, CutMix addresses these limitations by randomly selecting rectangular regions from one image (Xa) and replacing them with regions from another image while proportionally adjusting labels — achieving superior classification and localization performance.

9. Dropout with Transformer

In deep learning, the transformer is a type of neural network architecture that was introduced by Vaswani [79] in 2017. Transformers are designed for sequential input data, such as natural language sentences or time series data. Transformers leverage a self-attention mechanism that enables them to process the input sequence in parallel, unlike conventional recurrent neural networks (RNN) that process data sequentially. A. Fan et al. [80] introduced a new type of dropout technique called "structured dropout", which allows for the efficient reduction in the number of layers in a transformer model at the inference time. At inference time, the structured dropout technique is used to drop the non-important layers according to the learned mask. This allows for the efficient reduction in the depth of the transformer model, which can save computation time and memory.

Wu et al. [81] proposed a cutting-edge transformer-based dropout regularization method named UniDrop, which amalgamates three dropout modalities: feature dropout, structure dropout, and data dropout. PatchDropout [82] is a dropout regularization technique designed for vision transformer models. Unlike traditional dropout techniques that drop out individual tokens or features, PatchDropout randomly drops out patches of the input image during training. This reduces the computational and memory requirements of vision transformers while improving their generalization and robustness.

10. Materials for Experiments

In our review paper, we provide a direct comparison of each dropout method by analyzing the experimental results and performance. In this section, we provide a discussion of the dataset and a basic explanation of the experimental architecture. To facilitate a more transparent comparison, we divided our experimental materials into parallel subsections.

Training is one of the most crucial parts of a neural network. It relies heavily on both the baseline dataset and the data that are used for training, evaluation, and validation. These data sources are critical for ensuring the accuracy and effectiveness of the network's performance. The experiment on the dropout method utilized a majority of the benchmark dataset to showcase the algorithm's performance on various benchmark datasets, including CIFAR-10, CIFAR-100, and ImageNet. CIFAR (Canadian Institute For Advanced Research) is a well-renowned dataset, which consists of 80M images. However, for this experiment, most researchers have used two other subsets frequently: (i) CIFAR-10 and (ii) CIFAR-100. The CIFAR-10 subset is made with 60,000 image instances, each with a size of 32 × 32; it is divided into 50,000 instances for training and the remaining 10,000 instances for testing and validation. This dataset is compounded into 10 classes, representing various animals and objects. The CIFAR-100 subset is made with 60,000 image instances, each with a size of 32 × 32; it is divided into 50,000 instances for training and the remaining 10,000 instances for testing and validation. However, this dataset is compounded into 100 classes, representing animals and objects, which are classified; this is the counterpart-updated version. The ImageNet dataset is developed by the ImageNet project, which is developed for artificial intelligence tasks, such as image classification, image detection, etc. This dataset has been introduced from the "2012 ImageNet Large Scale Visual Recognition Challenge" (ILSVRC). This dataset comprises 1,240,000 instances with 224 × 224 images for training and 50,000 for testing/validation purposes. It contains 1000 classes, which are bigger than the CIFAR subsets.

Architectures: ResNet [83]. The residual network is a CNN architecture, the oldest architecture for regularization works. This ResNet group is among the most commonly used convolutional neural network architectures. Basically, this ResNet uses residual connections, concatenating the previous layer's output with further transformations. Different experiments use two ResNet variants differing by network depth: ResNet-18 [8] and ResNet-50 [4] are the most popular ResNet-family subvariants — ResNet-18 built on a 72-layer architecture with 18 deep layers, trained on the ImageNet database with over a million images; ResNet-50 with 50 layers trained on ImageNet. ViT-B/16 [84] models focus on the vision transformer: the visual transformer partitions an image into fixed patch sizes, then embeds each including a positional embedding system — showing superior computational efficiency and accuracy over CNNs. The wide residual network WRN [66] is another comprehensive regularization-work architecture with the same concept as ResNet but different residual connections and structural differences. PyramidNet [85], the most common neural network architecture in our experiment table and widely used for regularization analysis, increases the feature map in the network layer and moderately increases dimensionality — improving classification-task results.

11. Performance Comparison with Dropout Experiment Results

Convolutional neural networks (CNNs) and transformers are generally designed to perform the best optimal outcome in image processing. Dropout is an effective regularization method that can improve the performance and accuracy of neural networks by reducing overfitting and improving the smoothness of the model's output. Regularization methods, along with other components, can be easily incorporated into further experiments. Table 1 presents the results of individual CNN and transformer architectures on the CIFAR-100, CIFAR-10, and ImageNet datasets: ResNet-50 [4] at 23.49% error on ImageNet, improved by AutoDrop + RA [67] to 19.7%, AutoDrop alone [67] to 21.3%, RA [86] to 22.4%, FixRes [87] to 17.5%, DropBlock [4] to 21.65%, Mixup [87] to 22.1%, Fast AA [88] to 22.37%, BA [89] to 23.14%, AA [4] to 22.4%, cutout [4] to 23.48%, dropout [4] to 23.20%, LocalDrop [90] (5.3 / 26.2 / 21.1), ShakeDrop [91] (25.26 on CIFAR-100), Bag of Tricks [92] to 21.67, and GradAug [93] to 20.33.

On ViT-B/16 [84] (7.36 on CIFAR-100, 16.03 on ImageNet), RD [94] improves it to 6.71 and 15.62, and ViT-L/16 + RD [94] to 6.15 and 14.43. On ResNet-18 [8] (4.72 on CIFAR-10, 22.46 on CIFAR-100), cutout [8] improves to 3.99 and 21.96, MaxDropout [6] to 4.66 and 21.93, MaxDropout + cutout [6] to 3.76 and 21.82, TargetDrop [95] to 4.41 and 21.37, TargetDrop + cutout [95] to 3.67 and 21.25, and LocalDrop [90] to 4.3 and 22.2. On wide residual networks [66] (4.00 / 19.25 / 21.9), dropout [66] improves to 3.89 and 18.85, MaxDropout [6] to 3.84 and 18.81, TargetDrop + cutout [8] to 3.08 and 18.41, AutoDrop [67] to 3.1, AutoDrop + RE [67] to 2.1, and dropout + RA [86] to 2.7 and 16.7. On PyramidNet [85] (3.48 / 17.01 / 19.2), GradAug [93] improves CIFAR-100 to 13.76 and ShakeDrop + RA [86] CIFAR-10 to 1.5 — particularly advantageous for datasets with limited training samples facing generalization challenges.

12. Strengths and Limitations

In this section, we focus on recent and major dropout approaches, highlight their strengths, and attempt to identify their limitations based on previous research in Table 2. Some dropout approaches have extended their methods to overcome limitations while others still work within these limitations to improve their effectiveness: Dropout [1] reduces overfitting in the training layer, including supervised learning, speech recognition, and document classification — though for graph neural networks, convolutional operations and recurrent network performance may be relatively lower. DropBlock [4] creates a structure for dropped neurons — but the structure is fixed and randomly created. AutoDrop [67] has an automatic neuron-dropping pattern — but demonstrated improved performance for transformer and text-processing analyses only. Cutout [8] needs no domain knowledge to generate the mask — but for complex architectures and noisy image datasets, performance was not that high.

R-Drop [94] is careful about dropped-unit inference during training — but only outperforms in transformer and text-processing analyses. Fraternal dropout [28] reduces the sub-model gap and minimizes the loss function — but with training/inference inconsistency. ELD [96] regularizes the sub-model gap during inference — but may require more compute than standard dropout. Skipout [5] needs no extra hyperparameter tuning, easy to implement and use — but layer-by-layer training is critical for complex architectures. DropGNN [97] smooths any graph neural network's performance by reducing complexity — but dropping whole nodes or edges can lose significant information, harming performance. MaxDropout [6] deactivates the highest-valued activations to drive alternative features — but is sensitive to threshold p choice and may drop useful signals in noisy data.

13. Discussion and Open Issue

In our study, we conducted a comprehensive examination of the well-known regularization dropout method, a popular and effective technique for improving training convergence and final performance compared to other regularization approaches. Our performance table compares the performances of various dropout methods and baseline architectures using different benchmark datasets. As illustrated in Table 1, the ResNet-50 architecture with the ImageNet dataset exhibits superior performance when implementing AutoDrop + RA and FixRes regularization techniques, as evidenced by the lower error percentage compared to other methods. For the ResNet18 architecture trained on the CIFAR-10 dataset, similar results were observed, showing better performance compared to other methods such as MaxDropout + cutout and cutout. The dropout technique is effective at reducing overfitting and speeding up training for deep neural networks with large and complex computational depths.

In Figure 8, we present an overview of the overall characteristics of dropout methods [1, 3–6, 28, 67, 94, 96, 97] in a statistical view. Chart 1 specifically focuses on the percentages of dropout methods that meet the three main criteria: domain knowledge, automatic implementation, and random dropping. This chart provides insights into the extent to which dropout methods incorporate these criteria. Additionally, Chart 2 depicts the statistics for dropout methods that do not fulfill all three criteria. This chart highlights the prevalence of dropout methods that may lack one or more of the specified characteristics. By utilizing these two charts, we provide a comprehensive understanding of the distribution and prevalence of dropout methods based on their adherence to the defined criteria. Analyzing both charts, we observe that a significant majority of dropout methods possess domain knowledge — i.e. they are based on prior understanding or expertise in a specific domain. Notably, balancing the three criteria determines a method's general applicability versus domain specialization — an open issue guiding method selection and new-method design.

14. Conclusions

In this study, we observed more than forty regularization methods based on dropout, which can be categorized into several major groups (i.e., feedforward, convolutional, recurrent neural network, and transformer) with dropout operation and performance. Throughout our discussion, we explored various strategies, applications, interconnections, and contributions of these methods in the realm of neural networks. It is important to note that the selection of the most appropriate dropout method depends on the specific problem and architecture, and further research is needed to explore their full potential. The utilization of dropout in deep learning models has proven to be remarkably effective in enhancing performance and addressing overfitting. By randomly deactivating a portion of neurons during training, dropout encourages the network to learn more robust and generalized features, ultimately leading to improved performance on unseen test data.

This ability enhances model generalization and effectively mitigates the risk of overfitting, resulting in more robust and reliable neural networks. With its simplicity and effectiveness, dropout has become a widely adopted technique in the deep learning community, empowering researchers and practitioners to build more reliable and high-performing neural networks. To further strengthen dropout robustness in the future, researchers can delve into exploration and development per our review discussion — from automatic adaptive dropout, to integration with transformers and graph networks, to efficient Bayesian uncertainty-estimating methods — paths that will define the next generation of regularization techniques.