A Historical Survey of Advances in Transformer Architectures

Abstract

In recent times, transformer-based deep learning models have risen in prominence in the field of machine learning for a variety of tasks such as computer vision and text generation. Given this increased interest, a historical outlook at the development and rapid progression of transformer-based models becomes imperative in order to gain an understanding of the rise of this key architecture. This paper presents a survey of key works related to the early development and implementation of transformer models in various domains such as generative deep learning and as backbones of large language models. Previous works are classified based on their historical approaches, followed by key works in the domain of text-based applications, image-based applications, and miscellaneous applications. A quantitative and qualitative analysis of the various approaches is presented. Additionally, recent directions of transformer-related research such as those in the biomedical and timeseries domains are discussed. Finally, future research opportunities, especially regarding the multi-modality and optimization of the transformer training process, are identified. Keywords: transformers; deep learning; generative deep learning; large language models; GPT; computer vision.

1. Introduction

Ever since the introduction of the transformer model in June 2017 by Vaswani et al. [1], the world of deep learning has seen a rapid adaptation of the model in pushing the state of the art in a number of previously challenging tasks. Due to its prowess in sequence modeling and machine translation, the transformer architecture was initially widely implemented and indeed emerged as the predominant deep learning model for natural language processing (NLP) and generative deep-learning tasks [2]. Indeed, the introduction of transformers has been a key factor in the development of large language models such as GPT-3 and GPT-4, which are the basis of culturally significant tools such as ChatGPT [3].

However, inspired by the revolutionary self-attention mechanism in transformers, the architecture has since been implemented in various application domains such as that of images, audio, and time series data [4]. Indeed, in recent times, transformers have been touted as being a potential replacement for Convolutional Neural Networks (CNNs) for vision applications [5], with the introduction of the Vision Transformer (ViT) opening a new realm of architectures which build upon it. Considering the rapid increase in interest in transformer architecture, it becomes pertinent to examine in detail the architecture of the transformer as well as its historical progression from being introduced as an alternative to RNN-like architectures for sequence-to-sequence mapping to being one of the most impactful architectures in the current realm of deep learning.

Prior to the introduction of transformers to the deep learning space, the established state of the art in sequence modeling had long been Long Short-Term Memory networks (LSTMs) [6] and other forms of Recurrent Neural Networks (RNNs) [7], which were especially prevalent for transduction problems such as language modeling and machine translation due to their recurrence, which allows recent information to be accounted for. These models, however, suffered from limited parallelizability and vanishing gradients over long sequences — limitations the transformer addressed by abandoning recurrence entirely in favour of self-attention.

1.1. Transformers

In order to take a deeper look and investigate the success seen by the transformer model, it is imperative to examine, in detail, the architecture and workings of the solution proposed by Vaswani et al. [1]. Unlike previously proposed sequence transduction models like [11] and [12], transformers maintain the encoder–decoder structure but discard the recurrence and convolution aspects. This is made possible thanks to the novel multi-head attention mechanism proposed in addition to the point-wise feedforward networks ingrained in the transformer model.

The encoder consists of stacked layers, each containing multi-head self-attention followed by a feedforward network, with residual connections and layer normalization around each module. The decoder adds an encoder–decoder attention layer that lets it focus on relevant input positions. The following sections describe the various blocks contributing to this architecture in further detail.

1.2. Self-Attention

The first and most important component of transformer architecture is the self-attention mechanism which allows the model to learn the relationships between the elements of a sequence [13]. In the context of an LLM such as BERT, this would mean that in a sentence such as "The bank of the river is overflowing", the model would use self-attention to conclude that the "bank" in this case refers to the side of a river as opposed to a financial organization. The inputs to this layer in the encoder consist of queries and keys, and the attention function is applied as in Equation (1): Attention(Q, K, V) = softmax(QK^T / sqrt(d_k))V, where Q is the matrix of queries, K the matrix of keys, and V the matrix of values.

The dot product between the query and the key is first computed to form the scores S = QK^T, which determine how much attention is given to other words when encoding the word at the current position. These scores are then normalized to ensure gradient stability and enhance training, and the softmax function is applied to translate them into probabilities that are applied to the value matrix, so that vectors with larger probabilities receive greater focus from subsequent layers [5].

In transformers, a multi-head attention system is used wherein the original queries, keys, and values are projected into H different sets of learned projections. For each projection, the attention equation from (1) is applied to formulate the output, and the outputs across the H projections are concatenated to form the multi-head output as formulated in Equation (2). This design lets the model jointly attend to different representational aspects of different positions.

1.3. Feedforward Networks

Another important component in the functioning of transformers is the feedforward network which is applied after the self-attention layers in the encoder and decoder. This network consists of two linear transformations and a non-linear ReLU activation function which is applied to each position separately and identically. This ensures the same treatment across all positions in the input, meaning the token is processed in isolation, and allows the model to learn the complex transformations of the data at each position.

Going back to the example mentioned in the previous section, the self-attention would help give context and recognize the bank is not a financial organization, while the feedforward network would capture the nuance about the bank being slippery due to it being close to water, by adding additional layers of abstraction and complexity to the embeddings. The formulation for this network can be seen in Equation (3): FFN(x) = max(0, xW1 + b1)W2 + b2.

1.4. Residual Connections

The transformer also implements residual connections [15] around each module followed by layer normalization [16] which applies normalization layer by layer. This helps mitigate the vanishing gradient problem by allowing gradients to flow directly, bypassing several layers. Each transformer block can therefore be represented using the formulation seen in Equation (4), where the self-attention output is added element-wise to its input (the '+' operator refers to element-wise addition) before normalization and the feedforward network.

This residual connection boosts the flow of data by relaying the information forward and therefore serves to enhance the model's performance. In the context of the running example, these residual connections make sure essential characteristics of the word "bank" are not lost in the depth of the model's layers.

1.5. Position Encodings

As the self-attention process of the transformer discards the sequential way in which RNNs or LSTMs handle input embeddings and instead treats all inputs simultaneously and identically, the self-attention layer is not able to account for the position of words in a sentence. However, since words are sequential, a mechanism is needed which maintains the positions of the words within the encoded information, and therefore the transformer model makes use of position encodings which are added to the input embedding.

The formulation for the added embeddings is seen in Equation (5) with sine and cosine functions: PE(pos,2i) = sin(pos/10000^(2i/d_model)) and PE(pos,2i+1) = cos(pos/10000^(2i/d_model)), where pos is the position of a word, d_model the dimension and i the current dimension of the encoding. Each element of the positional encoding corresponds to a sinusoid, allowing the model to learn to attend based on relative positions and to extrapolate to longer sequences. These encodings have been a focal point of follow-up research proposing modifications such as learned encodings [17, 18] or relative position encodings [19].

2. Survey Methodology

The search for sources for this work was done following the PRISMA checklist [20]. Impactful works to be added to the survey were identified by searching online databases and scanning through the reference lists within the main papers. The search was applied mainly to Google Scholar, OpenAI, Papers with Code, and arXiv, as the majority of transformer works were found to be published through arXiv. As the survey is historical, the search was not limited by year, though works were found only from 2017 to the present, and the last search was conducted on 29 September 2023.

The following search terms were used across all databases: Transformers, State-of-The-Art Transformers, Key Transformer Architectures, Transformer Deep Learning, Transformer Vision, Transformer NLP, and BERT. Works were first shortlisted by impact factor and citation count, then further filtered by usefulness to the survey's subject. A data-extraction Excel spreadsheet was created with columns for paper name, author, date, proposed model, datasets, benchmarked models, results, and key notes, linked by serial number to a Word document of summarized key points.

3.1. Early Transformer Implementations

Since the introduction of the transformer model in 2017, a vast array of works have aimed to build upon its novel architecture to optimize performance across domains. The paper proposing the transformer has been cited more than 90,500 times as of 29 September 2023 according to Google Scholar [21]. Among the thousands of consequential works, a few notable models pushed the state of the art and established themselves as standards in their fields; Figure 3 displays a timeline of these works arranged chronologically and coded by application domain. Commonly used benchmark datasets include BookCorpus [48], WMT 2014 [49], Wikipedia [50], C4 [22], ImageNet [51], and COCO [52].

An early work building upon the transformer was that of Shaw et al. [19], which extended self-attention to efficiently consider relative position representations between sequence elements by modeling the input as a labeled, fully connected graph, reporting improvements on EN-DE and EN-FR tasks over the vanilla transformer. Another early and highly consequential work was that of Radford et al. [23], who proposed the famous Generative Pre-Training (GPT) approach, establishing unsupervised pre-training followed by task-specific fine-tuning.

This was followed by the BERT model of Devlin et al. in 2018 [17], which introduced deep bidirectional pre-training via masked language modeling and set records on GLUE and SQuAD. Improvements followed rapidly: RoBERTa [26] with optimized pre-training, Transformer-XL for long-range dependencies, XLNet with permutation ordering, GPT-2 with scaled-up size, DistilBERT and ALBERT with compression, and T5 [22] unifying all NLP tasks in a text-to-text framework.

In vision, Dosovitskiy et al. introduced the Vision Transformer (ViT) in 2020 [32], splitting images into patches processed as a sequence, followed by Touvron et al.'s DeiT [35] with data-efficient distillation. Lightweight efforts such as the Lite Transformer [29] tackled quadratic complexity, depth-scaling works such as DeepNet reached 1000 layers, efficient variants such as Longformer, Reformer, Linformer, Performer and OPT appeared, culminating in large-scale GPT-3 and GPT-4.

3.2. Text-Based Applications

Transformers have revolutionized text-based applications and NLP by solving problems such as text classification, question answering, text summarization, machine translation, and text generation [66]. The first prevalent model in text applications is the BERT model of Devlin et al. in 2018 [17], which remains relevant to this day because of how groundbreaking it was. BERT's NLP transformer has been the base for various other prevalent models such as RoBERTa [26] in 2019, which achieved excellent results with a BERT variant, ETC [67] in 2020, which reported high performance building upon BERT with RoBERTa weights, and Big Bird [68] in 2021 as a BERT variant for longer sequences.

Another notable transformer implementation for text is TENER [69], proposed in 2019 for named entity recognition — finding the start and end of an entity in a sentence and assigning it a class. This is especially useful in applications such as question generation [70], relation extraction [71], and coreference resolution [72]. The model adapts the transformer encoder to model both character-level and word-level features.

Overall, the pre-train-then-fine-tune methodology has dominated text applications, with architectures split between encoder-only models (BERT and family) for understanding, decoder-only models (GPT and family) for generation, and encoder–decoder models (T5) for general sequence transduction.

3.3. Image-Based Applications

In image-based applications, an early implementation was the Image Transformer of Parmar et al. in 2018 [73], which restricted transformer self-attention to local neighborhoods. In the image domain, however, one model reigns supreme: the Vision Transformer introduced by Dosovitskiy et al. in 2020 [32]. Numerous consequential works derive from it, including that of Touvron et al. [35], which outperformed ViT via distillation.

An alternative ViT-based framework is the Feature Fusion Vision Transformer (FFVT) of Wang et al. [74] in 2021, which adopts the ViT patch-generation process but modifies it to avoid overlap. An extremely recent vision solution is the Unsupervised Semantic Segmentation Transformer (STEGO) of Hamilton et al. in March 2022 [75], which localizes semantically meaningful categories within image corpora without any annotation, using a novel loss function that encourages features to form compact clusters while preserving their relationships across corpora.

These developments show transformers moving from a purely linguistic architecture to a backbone of computer vision, rivaling CNNs in classification, detection, and segmentation, with the advantage of capturing long-range dependencies across the whole image.

3.4. Miscellaneous Applications

In addition to the previously identified domains, two miscellaneous sets of approaches are discussed. The first is audio classification, for which a number of audio transformers have been proposed over the years. The first was proposed by Dong et al. in 2018 [76] with a two-dimensional attention block in an audio transformer model. A consequential model for audio captioning was the Audio Captioning Transformer (TRACKE) of Koizumi et al. [77] in 2020: TRACKE estimates keywords — a word set corresponding to audio events/scenes in the input audio — and generates the caption while referring to the estimated keywords to reduce word-selection indeterminacy. In 2021, Gong et al. proposed the Audio Spectrogram Transformer [78] as a convolution-free, purely attention-based model for audio classification.

The second miscellaneous set comprises time series modeling, first introduced by Liu et al. in 2021 [79] by adding gating to the vanilla transformer in an approach they call Gated Transformer Networks. Another 2021 model for time series forecasting was the Frequency Enhanced Decomposed Transformer (FEDformer) of Zhou et al. [80]. An interesting time-series implementation is TranAD of Tuli et al. in 2022 for anomaly detection in time series data [81], which uses focused score-based self-conditioning for robust multi-modal feature extraction and adversarial training for stability.

These applications demonstrate the flexibility of the transformer architecture and its transferability across modalities — from text and images to audio and time series — with relatively minor architectural adjustments preserving the essence of the attention mechanism.

3.5. Recent Directions

The recent increase in the relevance of transformers and the work exploring their uses has made them more accessible for real-world applications. One application area with heavy transformer use is medical image analysis [82]. While numerous earlier works applied AI algorithms to key medical problems such as COVID-19 detection [83] and fetal electrocardiogram extraction and detection [84, 85], the introduction of transformers for vision unlocked techniques such as image synthesis/reconstruction, registration, segmentation, detection, and diagnosis. As Li et al. [86] discuss, the ability of transformers to capture long-range dependencies and the scalability of self-attention enable their diverse usage in medicine.

Beyond medical imaging, Shamshad et al. [87] discuss transformer implementations in other medical applications such as leveraging text generation to produce medical reports and regression tasks such as survival outcome prediction. With increasing transformer depth and complexity, researchers have focused on the stability of extremely deep transformers; one scaling-based approach is DeepNet of Wang et al. [40], which introduces a new normalization function modifying residual connections with a theoretically derived initialization, successfully scaling transformers to 1000 layers.

Furthermore, wider transformer adoption has driven focus on lighter versions addressing compute and memory cost, through efficient models such as Longformer, Reformer, Linformer and Performer, and through distillation, quantization and pruning — paving the way for deployment on resource-constrained devices. The next section highlights and analyzes the results obtained by these models.

4.1. Historical Insight

Table 1 summarizes the historical works discussed in the previous section. Works are color-coded in the timeline: text/NLP-task works in blue and image-task works in orange. The timeline begins with "Attention Is All You Need" by Vaswani et al. in June 2017 [1], which achieved 28.4 BLEU on EN-DE and 41.8 BLEU on EN-FR with the big Transformer and has gathered over 90,500 citations.

It is followed by relative-position self-attention of Shaw et al. in March 2018 [19], generative pre-training of Radford et al. in June 2018 [23], and BERT of Devlin et al. in October 2018 [17] with an average GLUE test score of 82.1 for BERT Large and about 78,786 citations. For each work the table records the proposed model, datasets, benchmarked models, results, and citation counts, giving a quantitative picture of the development trajectory.

This historical view reveals a clear pattern: transformers began in machine translation, then dominated language understanding via bidirectional pre-training, expanded into large-scale generation (GPT-2, GPT-3), leapt into vision via ViT, and finally into multimodal and specialized applications — with the pace of new proposals accelerating year after year.

4.2. Application-Based Implementations

As seen from the model types used for these applications, an important aspect of text-based transformer implementation is the encoder and input encoding. The most widely used model has been BERT [17], which modifies input encodings to be bidirectional; RoBERTa [26] builds on it with an optimized pre-training process. Most other NLP-solving approaches involved input-encoding modifications such as TENER [69], ETC [67], and Big Bird [68], demonstrating the importance of encodings to the NLP process. Table 2 summarizes notable NLP transformer studies with results, such as an F1 of 94.6 on SQuAD 1.1 for RoBERTa.

In vision, applications revolved around adapting patch tokenization and training strategies (distillation in DeiT, feature fusion in FFVT, unsupervised clustering in STEGO), while audio and time-series applications focused on input representations (spectrograms, temporal embeddings) and specialized attention mechanisms (gating, frequency enhancement).

The quantitative and qualitative analysis shows that cross-domain success rests on two shared principles: large-scale pre-training followed by fine-tuning, and the ability of self-attention to model long-range dependencies regardless of the input signal's nature.

5. Gaps and Future Work

As the above discussion illustrates, transformer architectures are a field that has exploded with rapidly proposed new and existing works ever since the revolutionary publication of Vaswani et al. [1]. Figure 10 highlights the progression in architecture and complexity of transformer models since, visualizing notable implementations in order. Despite this rapid progression, certain gaps remain. One major gap in contemporary research is that transformers generally have quadratic computation and memory complexity due to modeling arbitrary long dependencies [91]. This has greatly hindered accessible transformer use and motivated a promising research avenue aimed at simplifying transformer training [92].

Indeed, the Lite Transformer [29] was introduced to address this very issue, as were implementations such as Longformer [93], Reformer [94], Linformer [95], Performer [96], and OPT [39]. These models are, however, only the start of a vast potential research space in optimizing transformer training procedures. This is pressing, as many state-of-the-art models simply aim to increase model size (GPT-4, for instance) [97], making such models impractical for many real-world applications.

Another interesting research issue is integrating all modalities without reshaping the architecture toward a single modality. Early implementations include models such as the Perceiver [98], which accepts all kinds of input but can only generate fixed outputs such as class probabilities, and Perceiver IO, with flexible inputs and outputs but still relying on modality specifics. A truly unified multimodal architecture remains an open challenge.

In conclusion, this survey identifies future research opportunities centered on multi-modality and optimization of the transformer training process: alleviating quadratic complexity, efficient scaling to thousands of layers, medical and temporal applications, and fusing modalities into a single architecture — trends that will define the next generation of transformer models.