小博客
RT-DETR: An abstract of how the model works. (en)

RT-DETR: An abstract of how the model works. (en)

itsnnka itsnnka 30 September 2026

Opening

What would happen if we could detect objects in real-time? RT-DETR (Real-Time Detection Transformer) answers that question. Unlike its predecessor Deformable DETR, RT-DETR offers an end-to-end approach without the post-processing that makes it slow. But, how does it actually work? What makes it fast yet accurate? In this study, I (the researcher) will dissect RT-DETR step by step along with its main components.

History and Theoretical Review

Sequential data is a type of data that has an important order or position, where each element in the data is interconnected based on time or a specific position. Before the advent of attention models, AI models for sequential data such as text and time series generally used RNNs (Recurrent Neural Networks). However, this model has a drawback, where RNNs perform computations or calculations sequentially, thus hindering the process of parallelization or data alignment during training. In this case, Long-Range Dependencies refer to the relationship or dependency between signals from two distant positions in an input or output sequence. Besides hindering data parallelization, RNN-based models also face challenges in learning long-range dependencies, because the number of operations required to connect two positions in a sequence increases as the distance between those positions grows. In this regard, the Transformer model architecture was introduced to overcome the above problems by relying on the self-attention mechanism a.k.a intra-attention—a mechanism that allows connecting various positions of a single sequence to compute a representation of that sequence, which facilitates parallelization and is effective in capturing long-range dependencies regardless of their distance in the sequence (Vaswani et al, 2017:2 & 6).

The main component of this architecture is Scaled Dot-Product Attention (SDPA)—that is, a function that maps a query (Q) and a set of Key (K) and Value (V) pairs into an output through the calculation of similarity scores of those values (Vaswani et al, 2017:3). The Query itself represents the vector currently being processed, for example, a word in a sentence. Meanwhile, the Key is a vector that represents another vector in the sequence that might be relevant to the Query. Then, the Value is a vector that represents the value or information of those vectors. The calculation process described earlier is done by computing the sum of dot products (dot product to measure similarity) between the Query and all Keys, the result of which is divided by a scaling factor in the form of the square root of the key dimension per-head or dk . Therefore, this scaling is a very important point because it makes the numbers in the dot product that grow large become small or normalized.

In this study, the focus on SDPA by the RT-DETR model is used to understand and learn the inter-patterns within an object/detail of an image. The attention process can be done simultaneously, this makes the model not only focus on 1 pattern but many patterns at once. To improve the model's capability, Multi-Head Attention (MHA) is used, which linearly projects the Query, Key, and Value by the number of heads (h) so that the SDPA process can be run in parallel on different representations. The results from all these heads are then concatenated and re-projected using an output projection matrix to produce the final representation.

Figure 1.1 Backbone model RT-DETR. Source: (Scientific Paper Competition by Researcher Documentation, 2026).

Before being analyzed by the AI model, the image obtained from the camera will be represented by converting it into an input in the form of an array of numbers with a mathematical notation as $x_{img} \in \mathbf{R}^{3 \times H_0 \times W_0}$ where $x_{img}$ is an image consisting of 3 dimensions of blocks of numbers that have different colors, namely Red, Green, Blue (RGB) with each dimension representing the height and width (height $\times$ width) of an image expressed in $H_0 \times W_0$ (Carion et al., 2020:6).

Then the AI model will process the data with the backbone, a component of the RT-DETR model—functioning to extract feature parts by the ResNet (Residual Network) model (see Figure 1.1). In this case, a feature is a representation of data produced by a process and simultaneously used as input for the next process. The backbone on the feature ximg will produce 3 features expressed as stages (S) with various different scales, where each $S_i$ is a real number consisting of an arrangement of tensors with different height and width dimensions (Zhu et al., 2020:4) so that the data is able to represent resolution down to the smallest level (pixel detail) based on the high value of the index of the $S_i$ value. From this, the result of the function $S_i=backbone(x_{img})$ is then called a feature map expressed as $S_i=\{S_3, S_4, S_5\}$ (Zhao et al., 2024:16968) where the meaning of feature map itself is a collection of feature data.

Figure 1.2 Mapping of block concepts in AI. Source: (Scientific Paper Competition by Researcher Documentation, 2026).

Then, so that the AI model is able to understand the contents of the image through the results of the feature map $S_i=\{S_3, S_4, S_5\}$, it will be processed in a block called the Efficient Hybrid Encoder which consists of 2 program functions (see Figure 1.2), namely the first is Attention-based Intra-scale Feature Interaction (AIFI) which consists of 2 main layers.


1.

Figure 1.3 Visual analysis of the attention vector space. Source: (Scientific Paper Competition by Researcher Documentation, 2026).

Multi-Head Attention (MHA) is a function of the concatenation of a set of Scaled Dot-Product Attention (SDPA) denoted as $head_h$, with $h>1$ as the number of heads—this makes the AI model able to learn different patterns or information in each vector space produced by each head simultaneously. Therefore, MHA is stated by Vaswani et al., (2017:4) in the form of the equation: $$MultiHead(Q,K,V)=Concat(head1(Q,K,V), ...... , head_h(Q,K,V))W_O$$ Where the head is stated as (Vaswani et al., 2017:5) $$ head_i(Q,K,V)=Softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$ As the researcher has written in the equation above, Query (Q) and Key (K) are represented as a set of arrows with different colors pointing to the sample point. Q refers to the red arrow and K as the green arrow—where each pair of Q and K arrows that are close to each other will form an angle $\theta$, the closer the angle between these arrows (the Q arrow is in the same direction as the K arrow), the greater the score in the dot product operation because of the greater similarity. The result of the $QK^T$ operation forms a vector score which will be normalized by $\frac{1}{\sqrt{d_k}}$ after which all vectors will be converted into probability form by the softmax activation function and perform a dot product operation with the value V so as to form the final representation of attention.

2. Feed-Forward Networks (FFN) which is a neural network component in the AI model that functions to enrich the tensor data representation of features through non-linear operations, this is because the attention mechanism is almost linear in nature. Therefore, this layer can be stated as

$$ FFN(x)=max(0, xW_1+b_1)W_2+b_2 $$ This is in line with the research conducted by Vaswani et al., (2017:5), where the activation function is stated in $max(0,x)$ which is used by the model to learn data patterns by removing features with negative values (changing them to 0) which means data that does not fit will be removed.


Before entering the next function, the model first produces a new feature, namely $F5$ through $S_5$ which is flattened so that the tensor dimension becomes a sequence through the equation $Q=K=V=Flatten(S_5)$ (Zhao et al., 2024:16969). This equation will produce 3 features with Q, K, V values to be used in the AIFI function and return the dimension size with the $Reshape(x)$ operation as a result of the flattening. Therefore, this process can be formulated as $$ F5=Reshape(AIFI(Q,K,V)) $$ Next, the second function used to combine features (fusion feature) from various different $S_i$ values and the previous $F_i$ (see Figure 1.2) is carried out with a function called CNN-based Cross-scale Feature Fusion (CCFF) (Zhao et al., 2024:16969) which is then formulated as $$ \begin{align} F4=CCFF({F5,S4}) \\ F3=CCFF({F4,S3}) \end{align} $$ Then the results of F3, F4, F5 will be concatenated and used as multi-scale feature input to the decoder, namely the block responsible for forming bounding boxes in the model's prediction—this block has a layer called Multi-Scale Deformable Attention (MSDeformAttn). This layer is similar to SDPA, but MSDeformAttn only looks for specific sampling points (K—Sampling points) on the object being searched for ($z_q$). In its notation, the F5 value basically already has objects that are clearly visible (semantic) in the image, this makes this value used as the selection of the most important object candidates in a process called query selection which produces $z_q$ and also the reference coordinates, namely the point on each object expressed in $p_q \in [0,1]^2, p_q=(x,y)$. Furthermore, the input parameters are stated as $\{x^l\}^L_{l=1}$ where $x^l$ consists of F3, F4, F5 so that the 3 inputs will be processed in parallel with more than 1 image scale.

Next, the output of the MSDeformAttn function in the form of a feature representation that has combined information from multiple resolution levels and searched for specific sampling points on the object will be passed to the next decoder block repeatedly until it produces the final object representation. This representation is then processed by the FFN Prediction Head layer to perform object coordinate regression so as to obtain a bounding box prediction expressed in the form of $b\in[0,1]^4$ with $b=\{b_x,b_y,b_w,b_h\}$ (Zhu et al., 2021:4).


Until the next page.

Thank you for reading, and for being here.

Source: (Houkago, Bokura wa Uchuu ni Madou, 2025).


Bibliography

  1. Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020). Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159.
  2. Zhao, Y., Lv, W., Xu, S., Wei, J., Wang, G., Dang, Q., ... & Chen, J. (2024). Detrs beat yolos on real-time object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16965-16974).
  3. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.

Special thanks to my LKTI friend a.k.a SRA

← Back