jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Review of chapter 2

Browse Source
This commit is contained in:
jfmcarreira
2018-07-30 15:38:33 +01:00
parent f3f362359e
commit 74109b2c4c
31 changed files with 751 additions and 352 deletions
Download Patch File Download Diff File Expand all files Collapse all files
021_VideoCoding/VideoCodingReview.tex
+159 -91
View File
@@ -89,26 +89,33 @@ Similar to the previous coding standards developed by JCT-VC, in \ac{hevc}~\cite
coding syntax and semantics are standardised.
The constrains and functionalities are defined through profiles and levels, while the semantic meaning of syntax elements define the decoding process, such that every decoder conforming to the standard generates the same output when given a complaint stream.
Such delimitation of scope in the standard allows maximum freedom for developers and manufactures to develop proprietary optimisations to cope with various applications’ requirements (\emph{e.g.}, balancing the compression ratio and the coding complexity).
\begin{figure}[tb]
\centering
% \includegraphics[width=0.75\columnwidth]{\FigsPath/HEVC/HEVCBlockDiagram}
\input{\FigsPath/HEVCBlockDiagram.tikz}
\caption{\HIGHLIGHT{Generic \ac{hevc} standard architecture.}}
\label{fig:HEVCBlockDiagram}
\end{figure}
\HIGHLIGHT{
Figure \ref{fig:HEVCBlockDiagram} illustrates the block diagram of the hybrid \ac{hevc} encoder. It follows the same overall architecture as its predecessors starting from H.261.
The encoder uses both intra- and inter-picture prediction to exploit spatial and temporal redundancies, respectively. Subsequently, the prediction residue transformed by linear spatial transform, which is then quantised, and entropy coded. This allows to reduce the bitstream redundancy to a minimum.
The resulting bitstream is combined with control information, which signals the prediction chosen, motion vectors, filtering operations and bitstream headers.
%The resulting bitstream is combined with control information, which signals the prediction chosen, motion vectors, filtering operations and bitstream headers.
}
\HIGHLIGHT{
As depicted in Figure \ref{fig:HEVCBlockDiagram}, each picture can be compressed with intra coding algorithms or inter prediction using previously encoded and reconstructed pictures kept in the \ac{dpb} as reference.
The encoding process follows a block-based approach as in previous standards with the larger units referred to as \ac{ctu} with a maximum size of $64\times64$. In order to improve the overall quality of the reconstructed pictures and smooth the coding artefacts, \ac{hevc} uses the deblocking and \ac{sao} filters.
The encoder has a built-in decoder to generate the reference pictures. In the decoding loop, the residual information is delivered to the inverse quantisation and transform modules. The finally reconstructed samples are obtained by adding the predicted signal and applying the filtering operations.
}
\begin{figure}[tb]
\centering
\includegraphics[width=\columnwidth]{021_VideoCoding/Figures/HEVCBlockDiagram}
%\input{\FigsPath/HEVCBlockDiagram.tikz}
\caption{\HIGHLIGHT{Block diagram of an \ac{hevc} standard encoder.}}
\label{fig:HEVCBlockDiagram}
\end{figure}
\glsreset{ctu}
\glsreset{cu}
\glsreset{pu}
\glsreset{tu}
\glsreset{nalu}
\glsreset{cabac}
\glsreset{rpset}
There are some novelties introduced in the \ac{hevc} standard. As depicted in Figure \ref{fig:HEVCBlockDiagram}, the encoding process follows a block-based approach, therefore each picture is partitioned into larger blocks, referred to as \acs{ctu}, with a maximum size of $64\times64$. Subsequently, a dynamic quad-tree partitioning is used to divide the picture into \acs{cu}. These units are introduced as the basic processing units for prediction, and for the transform and quantisation operations \acs{tu} are used.
In order to generate the prediction residue the \ac{hevc} may use one of 34 intra-prediction modes or take advantage of symmetric and asymmetric partitions for motion estimation and inter-prediction.
The motion estimation is performed in those pictures listed in the \ac{rpset} which was introduced in the \ac{hevc} to increase the error robustness of reference picture lists.
To improve the overall quality of the reconstructed pictures and smooth the coding artefacts, \ac{hevc} not only uses a deblocking filter but also a \ac{sao} filter. Finally, the control data, quantised coefficients, prediction modes, motion vectors and filter information are compressed using the \ac{cabac} entropy coder and generate a compressed bitstream composed of different \acs{nalu}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
@@ -226,7 +233,7 @@ This enables a more efficient prediction structure, while maintaining an access
However, it results in leading pictures which may not be decodable, which are explained in the next sub-section.
The introduction of \ac{cra} frames in \ac{hevc} was able to improve the coding efficiency by up to 6\% \cite{HEVCCRAPics2011}.
\HERE
%\HERE
The \ac{hevc} supports bitstream splicing, which means taking a particular bitstream starting from a \ac{rap} (both \ac{idr} and \ac{cra} frames) and insert it into another bitstream at a random-access point.
In case of bitstream splicing starting from an \ac{idr} frame the leading pictures does not have dependencies to frames prior the \ac{rap}, thus all frames are decodable.
@@ -258,21 +265,20 @@ Therefore, when a random-access is performed at the associated \ac{rap} frame, t
\subsubsection{Temporal sub-layer switching pictures}
\begin{figure}[tb]
\centering
% \includegraphics[width=0.75\columnwidth]{\FigsPath/HEVC/Pictures_TSA}
\input{\FigsPath/PictureTypes_TSA.tikz}
\caption{Temporal layers access pictures example.}
\label{Figure2:TSAExample}
\end{figure}
\HERE
A \ac{tsa} picture is a trailing picture that marks a temporal layer switching point.
When decoding a sub-set of the temporal layers and a \ac{tsa} picture is found from the temporal layer just above the maximum temporal layer currently being decoded, it is possible for the decoder to start decoding any number of additional temporal layers.
For example, frame $P_6$ in Figure~\ref{Figure2:TSAExample} is considered a \ac{tsa} picture since it only depends on a frame belonging to temporal layer 0 for prediction, as well as, any subsequent frame predicted from the \ac{tsa} frame $P_6$.
In this example, if the decoder was only decoding the temporal layer 0 it can after $P_6$ starts decoding both layer 0 and 1 or decode all the available three layers. Although theses frame allows higher flexibility in terms of layer switching, they severely constrain the prediction of frames following the \ac{tsa} picture.
\begin{figure}[tb]
\centering
% \includegraphics[width=0.75\columnwidth]{\FigsPath/HEVC/Pictures_TSA}
\input{\FigsPath/PictureTypes_TSA.tikz}
\caption{Temporal layers access pictures example.}
\label{Figure2:TSAExample}
\end{figure}
To reduce the constrains in frame prediction due to layer switching, \ac{hevc} introduces the \ac{stsa} picture.
These pictures have similar purpose as the \ac{tsa} frames but they only guarantee that frames in the same temporal layer as the \ac{stsa} frame in the decoding order are able to be correctly reconstructed. This is guaranteed by preventing any dependency from frames preceding the \ac{stsa} to be used as reference in its temporal layer.
One example of a \ac{stsa} frame is also shown in Figure~\ref{Figure2:TSAExample} in frame $P_2$.
@@ -366,10 +372,13 @@ Figure~\ref{Figure2:HEVCCodingStructure} illustrates an example of a \ac{ctu} pa
The \ac{cu} is the basic processing unit in \ac{hevc}. It is further partitioned into two types of units, specifically \ac{pu} and \ac{tu}, which are data blocks used for prediction and transform operations, respectively.
The decision whether to use intra-frame or inter-frame prediction modes is taken at \ac{cu} level. However, depending on the partitioning of the \ac{cu} into \acs{pu} different intra or inter coding modes can be used for each \ac{pu}.
\HERE
%The main key feature of \ac{hevc} is the new block partition structure, which achieves more flexibility and efficiency. Four different block concepts have been introduced in the standard: \ac{ctu}, \ac{cu}, \ac{pu}, and \ac{tu}, all of which have different roles. Figure \ref{fig:HEVCBlockStructure} illustrates an example of a CTU partitioning in HEVC, and it is compared to the \ac{h264} \ac{mb} structure. CTUs are defined as the high-level partition within each slice and can be as large as 64$\times$64, which can be recursively partitioned into four smaller blocks referred to as CUs. The CU is the basic units for the intra/inter prediction and can be as small as 8$\times$8. Subsequently, the CU are divided into up to four PUs which are used for inter prediction. For intra coding PU is not used. Finally, the CU is divided into a transform unit tree, which may include several TUs, in a quad-tree structure. Transform units are the basic units for the transform operations applied to the coding residue.
%
%Each CU may be split into two or four Pus resulting in either symmetric or asymmetric partitions. Asymmetric partitioning of the CU into two PUs enables more efficient motion estimation since it allows more flexible partitioning of the picture~\cite{BlockPartHEVC2012}. This scheme is able to improve the coding performance by up to $1\%$ when compared with rectangular symmetric partitioning.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -388,7 +397,7 @@ Intra predicted blocks are obtained using the previously encoded pixels around t
When the intra prediction mode is chosen for a \ac{cu}, the size of the \ac{pu} need to be the same as the \ac{cu}, except when the smaller \ac{cu} is selected and further splitting into four \acs{pu} is done. For example, when the smaller \ac{cu} size is $8\times8$, the encoder may decide to divide it into four \acs{pu} of $4\times4$ pixels, each one having its own intra prediction mode. This is signalled using a binary flag.
The use of small \acs{pu} is useful for complex regions that require fine-granularity prediction.
In \ac{hevc} three types of intra-frame prediction are defined: angular modes, Planar and DC~\cite{IntraCodingHEVC2012,HEVCIntraperformance2012}.
In \ac{hevc} there are 35 different of intra-frame prediction modes: Planar, DC and 33 angular modes as shown on the left of Figure~\ref{Figure2:IntraModesHEVC}~\cite{IntraCodingHEVC2012,HEVCIntraperformance2012}.
%Figure~\ref{Figure2:IntraModesHEVC} illustrates these intra prediction modes.
%As shown, the predictions are applied to square blocks only.
%The border around the square blocks corresponds to the reference pixels from which the prediction is applied (different colours correspond to different pixel values).
@@ -406,12 +415,12 @@ In order to overcome this issue and to allow all possible modes, \ac{hevc} uses
\begin{figure}[tb]
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{x{.6\columnwidth}x{.4\columnwidth}}
\begin{tabular}{x{.55\columnwidth}x{.45\columnwidth}}
\input{021_VideoCoding/Figures/IntraPredictions.tikz} &
\input{021_VideoCoding/Figures/IntraPredictionsExample.tikz} \\
\input{021_VideoCoding/Figures/IntraPredictionsDirectional.tikz} \\
\end{tabular}
}
\caption{List of directional intra prediction modes in \ac{hevc} (left) and an example of the intra prediction mode 32 for a $4\times4$ block.}
\caption{Directional intra prediction modes in \ac{hevc} (left) and an example of the intra prediction mode 32 for a $4\times4$ block.}
\label{Figure2:IntraModesHEVC}
\end{figure}
@@ -420,59 +429,89 @@ In order to overcome this issue and to allow all possible modes, \ac{hevc} uses
Directional intra prediction was first introduced in the \ac{h264} standard~\cite{H264OverviewJul2003}, to estimate regions with a structured texture or directional edges.
Directional prediction projects the reconstructed samples in the block neighbourhood (see reference pixels in Figure~\ref{Figure2:IntraModesHEVC}) along a specific direction. The \ac{hevc} increases the number of possible directions from 8 (in \ac{h264}) to 33~\cite{IntraCodingHEVC2012}. The direction of each projection is illustrated in the left-side of Figure~\ref{Figure2:IntraModesHEVC} by the directional modes 2 to 34.
%In the figure it is possible to notice that directional modes 2 to 10 only project the pixels from the left neighbouring border in a horizontal direction, modes 10 to 25 use both the left and top neighbouring pixels in a diagonal direction and modes 26 to 34 project the top neighbours in vertical direction.
As shown in the figure, the angular modes are designed to provide a dense coverage near the horizontal and vertical directions. Moreover, the angular directions are more coarse as it gets closer to the diagonals. This reflects the observed statistical prevalence of the angles and the prediction efficiency.
To improve the efficiency of angular prediction, when the reference samples need to be projected a bilinear interpolation is used from the two closest pixels using a $1/32$ accuracy. The dashed arrows in Figure~\ref{Figure2:IntraModesHEVC} correspond to a projection of interpolated pixels.
As shown in the figure, the angular modes are designed to provide a dense coverage near the horizontal (i.e., mode 10) and vertical (i.e., mode 26) directions. Moreover, the angular directions are more coarse as it gets closer to the diagonals. This reflects the observed statistical prevalence of the angles and the prediction efficiency.
To improve the efficiency of angular prediction, when the reference samples need to be projected a bilinear interpolation is used from the two closest pixels using a $1/32$ accuracy. The dashed arrows on right of Figure~\ref{Figure2:IntraModesHEVC} correspond to a projection of interpolated pixels.
\subsubsection{Planar and DC}
In contrast, with the angular mode which cover those regions with strong direction edges, \ac{hevc} also implements the Planar and DC modes targeting smooth regions. These modes are useful for smooth regions as they result in a linear or constant prediction.
\begin{figure}[tb]
\centering
\input{021_VideoCoding/Figures/IntraPredictionPlanar.tikz}
\caption{Example of Planar prediction for a $4\times4$ block.}
\label{Figure2:IntraModePlanar}
\end{figure}
The DC mode uses the average or the neighbouring reference samples to generate a constant prediction for the current \ac{pu}.
The Planar mode predicts the \ac{pu} through linear predictions from vertical and horizontal slopes derived from the neighbouring reference samples. As the Planar mode used linear surface for prediction it prevents the discontinuities at \ac{pu} boundaries. Both of these modes are also illustrated in Figure~\ref{Figure2:IntraModesHEVC}.
The Planar and DC modes are particularly efficient in the prediction of smooth regions.
The DC mode uses the average or the neighbouring reference samples to generate a constant prediction for the current \ac{pu} as a whole, i.e., all pixels are predicted from the same value.
The Planar mode predicts the \ac{pu} through linear interpolation from the four closest neighbouring reference pixels. Figure~\ref{Figure2:IntraModePlanar} shows the reference pixels (i.e., $p_1$, $p'_2$, $p_3$ and $p'_4$) and arrows pointing to the pixel being predicted.
Since the right and bottom neighbourhood is not available (not coded yet), the same pixel $p_2$ is repeated to form a column of reference pixels, marked with an ellipse in Figure~\ref{Figure2:IntraModePlanar}. A similar approach is also used to create a the bottom row of reference pixels from the pixel $p_4$ .
%As the Planar mode used linear surface for prediction it prevents the discontinuities at \ac{pu} boundaries.
\subsubsection{Sample smoothing}
In \ac{hevc} sample smoothing is applied at two distinct occasions two improve the overall performance of the prediction accuracy. Firstly, the reference samples are filtered and secondly the \ac{pu} is filtered after prediction. This improves the previous standard \ac{h264} where only reference samples were filtered.
In \ac{hevc} sample smoothing is applied in two distinct cases to improve the overall performance of the prediction accuracy.
Firstly, the reference samples are filtered and secondly the \ac{pu} is filtered after prediction.
%In comparison with previous standard \ac{h264}, where only reference samples were filtered.
In \ac{hevc} the reference sample smoothing is conditionally applied, based on the \ac{pu} and intra prediction mode. Similar to \ac{h264}, a three tap $\left[1\ 2\ 1\right]/4$ smoothing filter is applied, except for \ac{pu} sizes of $4\times4$. The prediction results from Planar mode is also filtered when the \ac{pu} size is greater than $8\times8$, and it is not used for the DC mode as it would not be useful.
In \ac{hevc}, the reference sample smoothing is conditionally applied, based on the \ac{pu} and intra prediction mode. Similar to \ac{h264}, a three tap smoothing filter (i.e., $\left[1\ 2\ 1\right]/4$) is applied, except for \ac{pu} sizes of $4\times4$. The prediction resulting from Planar mode is also filtered when the \ac{pu} size is greater than $8\times8$, and it is not used for the DC mode.
The second filter in \ac{hevc} is the boundary smoothing. This filter aims at removing the discontinuities along block boundaries due to intra prediction. The application of this filter is more exclusive, and it is only applied to the DC and angular mode 10 and 26 (exactly the horizontal and vertical directions, respectively). A two-tap filter $\left[3\ 1\right]/4$ fed by the original samples and the adjacent reference sample. For the DC mode this filter is applied to both the top row and the left-most column. In case of the angular mode 10 and 26 only the left-most column and the top row are filtered, respectively. Only the mentioned boundaries are filtered, as they are the most likely to introduce discontinuities. The result of the boundary smoothing can be noticed in Figure~\ref{Figure2:IntraModesHEVC} for the aforementioned modes.
The second filter in \ac{hevc} is the boundary smoothing. This filter aims at removing the discontinuities along block boundaries due to intra prediction.
This filter is only applied when the block size is smaller than $32\times32$ to the DC and angular modes 10 and 26 (exactly the horizontal and vertical directions, respectively).
For the DC mode a two-tap filter ($\left[3\ 1\right]/4$), fed by the original predicted value and the adjacent reference sample, is applied to both the top row and the left-most column of predicted pixels.
In the case of angular mode 10 (horizontal direction), only the predicted samples of the first column are changed by adding half of the difference between the adjacent reference sample and the top-left reference sample.
A similar filtering is done for angular mode 26 (vertical direction), but only for the first row
of predicted samples.
Only the mentioned boundaries are filtered, as they are the most likely to introduce discontinuities.
%\subsubsection{Rough Mode Decision}
%\cite{HEVCROughModeDecision}
\subsection{Inter-frame prediction}
The efficiency of video coding algorithms consistently relies on inter-frame prediction techniques used to reduce temporal redundancy. The underlying idea of inter-frame prediction is to estimate the current frame from one or more previously encoded reference frames using block-based motion compensated. The most common technique used for motion estimation in \ac{hevc}, as well as in \ac{h264}, is the boundary matching algorithm. In the \ac{hevc} standard an \ac{amvp} for efficient motion vector signalling, which defines the Merge Mode and differential coding of the motion information.
The efficiency of video coding algorithms consistently rely on inter-frame prediction techniques to reduce temporal redundancy.
The underlying idea of inter-frame prediction is to estimate the current frame from one or more previously encoded frames used as reference with block-based motion compensation. The most common technique used for motion estimation in \ac{hevc} is the block matching algorithm. In the \ac{hevc} standard an \ac{amvp} is used for efficient motion vector coding, which defines the Merge Mode and differential coding of the motion information.
\subsubsection{Motion estimation and compensation}
Motion estimation consists in searching for the current block with the highest similarity with the target block to be predicted (\ac{pu} in \ac{hevc}), using block-matching algorithms over the previously encoded frames, i.e. the reference frames. Typically, reference frames correspond to past or future temporal frames. The frame encoding order determines which reference frames are selected and whether future frames are available. Typically, video frame type P normally are predicted from the past as B frame are predicted from frames both in past and future.
Block-matching consist on subtracting the current block from one of equal size in the reference frame in a pre-defined window. The reference block that results in the lowest difference is selected for prediction.
In order to signal the selected block to be used for motion compensation the different between the target and the reference block positions are computed to obtain a two-dimensional vector, referred to as \ac{mv}. This vector is signalled to the decoder, in order to apply the same prediction.
Motion estimation consists in searching for a block with the highest similarity with the current block to be predicted (\ac{pu} in \ac{hevc}), using block-matching algorithms over previously encoded frames, i.e. the reference frames.
Typically, reference frames correspond to past or future temporally adjacent frames.
The frame encoding order determines which references are selected and which future frames are available.
The reference block that results in the lowest error is selected as the best block for prediction.
To identify the best matching block to be used for motion compensation, the different between the target and the reference block positions is encoded as a two-dimensional \glsfirst{mv}.
Similar to \ac{h264} the \ac{hevc} allows for either one or two \ac{mv} to be selected to compensate the target \ac{pu}. This results in uni-predictive and bi-predictive coding, respectively. \ac{hevc} also allows for weighted prediction, which consists in scaling and offset operations the predicted block using determined factors which are explicitly sent to the decoder.
The motion vectors are obtained from frames included in the reference frame list. Each frame is identified by an index, therefore to fully describe inter-frame prediction a combination of two parameters is required: \ac{mv} and reference index.
The \ac{hevc} allows for either one or two \acs{mv} to be used for each \ac{pu}. Video frames of type P are predicted from one single reference (one \ac{mv}) while frame of type B are predicted from two references (two \acs{mv}).
This results in uni-direction or bi-directional coding, respectively.
Furthermore, weighted prediction can also be used, which consists in scaling and offset operations performed on the prediction block to improve efficiency.
The \acs{mv} only refer to frames included in a reference frame list, where each frame is identified by an index. Therefore, to fully describe inter-frame prediction a combination of two parameters is required: \ac{mv} and reference frame index.
In recent standards motion compensation has been performed using a variable block size approach. In \ac{h264} block sizes ranging from $4\times4$ up to $16\times$ pixels were allowed. In \ac{hevc} a significant improvement was introduced by extending the block sizes up to $64\times64$ pixels. As mentioned before it also allows asymmetric rectangular block sizes.
By using larger prediction units, \ac{hevc} is able to save bits in the signalling of the selected \ac{mv}, because less \ac{mv} are required.
Since MPEG-2, motion estimation/compensation has been performed using blocks of variable size.
In \ac{h264} block sizes ranging from $4\times4$ up to $16\times$ pixels are allowed, while \ac{hevc} extends the possible block sizes up to $64\times64$ pixels, and also to asymmetric rectangular block sizes.
By using larger prediction units less \ac{mv} are required and fewer bits are spent in coding motion information.
In order to increase the efficiency of motion estimation using fractional \acs{mv} are used. This has been proven to improve accuracy of motion compensation. To accomplish this, frame interpolation is performed to generated non-integer pixels in the reference frame. Both \ac{h264} and \ac{hevc} standards use \acs{mv} up to quarter-pixel accuracy for luma component. For chroma samples, the \ac{mv} accuracy is determined according to the chroma sampling format, which for 4:2:0 results in eighth-pixel accuracy. In contrast, with the two-stage approach used in \ac{h264}, \ac{hevc} uses a single separable approach for the interpolation process without intermediate rounding operations. This improves the precision and simplifies the architecture of the fractional interpolation.
In order to increase the efficiency of motion estimation fractional \acs{mv} are used, which has been proven to improve motion compensation accuracy.
This is accomplished through interpolation to generate non-integer pixels in the reference frame, resulting in \acs{mv} up to quarter-pixel accuracy for luma samples.
For chroma samples, the \ac{mv} accuracy is determined according to the chroma sampling format, which for 4:2:0 results in eighth-pixel accuracy.
In contrast with the two-stage approach used in \ac{h264}, \ac{hevc} uses a single separable approach for the interpolation process without intermediate rounding operations. This improves the precision and simplifies the architecture of the fractional interpolation.
The block matching-based motion estimation is one of the most computationally complex procedures in the \ac{mv} encoder. Normally, the optimal solution for motion estimation is provided by the full-search algorithm, which tests all the possible candidate blocks in the search area. Although the full-search algorithm achieves the optimal performance, it requires the highest computational complexity. Therefore, sub-optimal algorithms have been proposed over the years to reduce the motion estimation complexity and decrease encoding time\cite{HEVCME2012,HEVCME2017,HEVCME2014}.
For example, the HM reference software implements a fast algorithm based on Test Zone Search ~\cite{HEVCME2012}.
Since motion estimation is a non-normative procedure, encoders may implement any alternative search algorithms that suits its needs (i.e., compression ratios or encoding time).
The block matching-based motion estimation is one of the most computationally intensive procedures in video encoders.
In general, the optimal solution for motion estimation is provided by the full-search algorithm, which tests all the possible candidate blocks in the search area. However, although the full-search algorithm achieves the optimal performance, it requires the highest computational complexity.
Therefore, sub-optimal algorithms have been proposed over the years to reduce the motion estimation complexity and decrease encoding time~\cite{HEVCME2012,HEVCME2017,HEVCME2014}.
For example, the HM reference software implements a fast algorithm based on Test Zone Search~\cite{HEVCME2012}.
Since motion estimation is a non-normative procedure, encoders may implement any alternative motion search algorithms.
\subsubsection{Reference picture management}
Another improvement in \ac{hevc} is related with the reference picture management scheme. \ac{hevc} introduces the \ac{rpset} concept~\cite{HEVCHighLevelReferenceDez2012}, which defines how previously decoded pictures are managed in the \ac{dpb}. Decoded frame in the \ac{dpb} are mark grouped in one of the following categories: (i) \emph{short-term reference}, (ii) \emph{long-term reference} and (iii) \emph{unused for reference}. Once a frame is marked as \emph{unused for reference} it is no longer used for motion compensation and it will be discarded from the \ac{dpb} after being displayed/outputted.
In \ac{h264} a sliding windows approach was used to implicitly manage the reference picture list. This process discards a given frame whenever the \ac{dpb} has the number of defined reference frames (which is signalled by the encoder). Moreover, \ac{h264} used an explicitly \ac{mmco} to send commands to the decoder and change the normal reference picture management process.
Another improvement in \ac{hevc} is related with the reference picture management scheme. \ac{hevc} introduces the \ac{rpset} concept~\cite{HEVCHighLevelReferenceDez2012}, which defines how previously decoded pictures are managed in the \ac{dpb}.
Decoded frames in the \ac{dpb} are grouped in one of the following categories: (i) \emph{short-term reference}, (ii) \emph{long-term reference} and (iii) \emph{unused for reference}.
Once a frame is marked as \emph{unused for reference} it is no longer used for motion compensation and it will be discarded from the \ac{dpb} after being sent to the displayed.
In \ac{h264} a sliding window approach is used to implicitly manage the reference picture list.
This process discards a given frame whenever the \ac{dpb} has the number of defined reference frames (which is controlled by the encoder). Moreover, \ac{h264} uses an explicitly \ac{mmco} to send commands to the decoder and change the normal reference picture management process.
\begin{table}[tb]
\centering
@@ -492,14 +531,21 @@ In \ac{h264} a sliding windows approach was used to implicitly manage the refere
\end{tabular}
\end{table}
In the \ac{hevc} standard instead of a signalling relative changes, the status of the \ac{dpb} is signalled for every slice.
The most fundamental difference with the \ac{rpset} concept compared to \ac{mmco}/sliding window of \ac{h264} is that for each particular slice a complete set of the reference frames that are used by the current or any subsequent frames are be provided.
Thus, a complete set of all frames that must be kept in the \ac{dpb} (in the present and future) are signalled.
This is different from the \ac{h264} scheme where only relative changes to the \ac{dpb} are signalled. With the \ac{rpset} concept, no information from earlier frames in decoding order is needed to maintain the correct status of the \ac{dpb}. Table~\ref{Table2:ReferencePictureListExample} shows the \ac{rpset} for some frames illustrated in Figure~\ref{Figure2:LeadingTralingPics}. As expected for the \ac{cra} frame ($I_2$) nothing is signalled. For the remaining frames, two types of signalled are used: frame number and a flag to mark if it is currently used. This is required because any frame not listed in a given \ac{rpset} will be discarded and subsequent frames cannot use it as reference.
In the \ac{hevc} standard the status of the \ac{dpb} is encoded for every slice, instead of the implicit managing used in the past.
The most fundamental difference in \ac{rpset} concept compared to \ac{mmco}/sliding window of \ac{h264} is that each particular frame has a complete set of the reference frames that are used or will be used (or will be used in subsequent frames).
Thus, a complete set of all frames that must be kept in the \ac{dpb} explicitly provided.
This is different from \ac{h264} where only relative changes to the \ac{dpb} encoded.
With the \ac{rpset} concept, no information from earlier frames in decoding order is needed to maintain the correct status of the \ac{dpb}.
Table~\ref{Table2:ReferencePictureListExample} shows the \ac{rpset} for some frames illustrated in Figure~\ref{Figure2:LeadingTralingPics}.
As expected for the \ac{cra} frame ($I_2$) no information is sent to the decoder because there is not inter-prediction and the \ac{dpb} should be empty.
For the remaining frames, two types of information are encoded: frame number and a binary flag to indicate whether it is currently used. This is required because any frame not listed in a given \ac{rpset} will be discarded by the decoder and then subsequent frames cannot use them as reference.
The \ac{rpset} concept in \ac{hevc} provides a basic level of error robustness to the reference picture management. In \ac{h264} whenever a reference frame is lost the decoder can look at the value of the
\emph{FrameNum} (incremental value associated with each reference frame) to detect the loss event. However, as gaps in \emph{FrameNum} are allowed the decoded might not be able to decide the missing frame was intentional or if it resulted from a loss event in the transmission. Moreover, even if the decoded if able to detect the loss event it might not be able to correctly compute the value the \ac{poc} associated with the concealed frame. Using the \ac{rpset} concept in the \ac{hevc} the decoder is always able to detect loss events and correctly identify the missing frames, preventing it from use wrong frames as reference.
The \ac{rpset} concept provides a basic level of error robustness to the reference picture management.
In \ac{h264} whenever a reference frame is lost the decoder can look at the value of the \emph{FrameNum} (incremental value associated with each reference frame) to detect the loss event.
However, as gaps in \emph{FrameNum} are allowed the decoder might not be able to decide whether the missing frame was intentional or resulted of a transmission loss event.
%Moreover, even if the decoded if able to detect the loss event it might not be able to correctly compute the value the \ac{poc} associated with the concealed frame.
Using the \ac{rpset} concept the decoder is always able to detect loss events and correctly identify the missing frames.
\subsubsection{Asymmetric block partitions}
@@ -514,15 +560,17 @@ The \ac{rpset} concept in \ac{hevc} provides a basic level of error robustness t
\end{figure}
When inter prediction mode is signalled, \ac{hevc} defines a higher range of partitioning schemes, allowing the \ac{cu} to be divided into one, two or four \acs{pu}.
The splitting into four \acs{pu} is only allowed when the \ac{cu} size is equal to the minimum. Otherwise, the splitting should be carried out at the \ac{cu} level which leads to the same partition tree.
Figure~\ref{Figure2:InterPuPartitions} illustrates the possible \ac{cu} partitioning into \acs{pu}. One should note that the partition type applied to the \ac{cu} is also applied to divide the \ac{cb} into \acs{pb}.
When inter prediction mode is selected a higher range of partitioning schemes is allowed, thus dividing the \ac{cu} into one, two or four \acs{pu}.
The splitting into four \acs{pu} is only allowed when the \ac{cu} size is equal to the minimum.
%Otherwise, the splitting should be carried out at the \ac{cu} level which leads to the same partition tree.
Figure~\ref{Figure2:InterPuPartitions} illustrates the possible \ac{cu} partitioning into \acs{pu}.
%One should note that the partition type applied to the \ac{cu} is also applied to divide the \ac{cb} into \acs{pb}.
Figure~\ref{Figure2:InterPuPartitions}~(a) shows the symmetric partitions.
The $N \times N$ corresponds to the case where the \ac{cu} is not further divided.
Then, the standard defines the horizontal and vertical division into two \acs{pu}, corresponding to the $\frac{N}{2}\times N$ and $N \times \frac{N}{2}$ cases, respectively.
Finally, the $\frac{N}{2} \times \frac{N}{2}$ case results in four \acs{pu}.
The partitions shown in Figure~\ref{Figure2:InterPuPartitions}~(b) correspond to the asymmetric partitioning, which is one of the main improvements introduced in \ac{hevc}, as it allows higher flexibility in motion estimation, leading to higher compression efficiency~\cite{HEVCBlock2012}.
These modes are only allowed for \acs{cu} with $16\times16$ pixels or larger.
These partitions types are only allowed for \acs{cu} with $16\times16$ pixels or larger.
\subsubsection{Merge mode}
@@ -534,21 +582,27 @@ These modes are only allowed for \acs{cu} with $16\times16$ pixels or larger.
\label{Figure2:MVCandidates}
\end{figure}
The Merge mode was introduced in \ac{hevc} standard, as a new technique to derive motion information from spatial and temporal neighbouring blocks~\cite{MotionCodingHEVC2013}. This technique extends the concepts of the direct and skip modes used in \ac{h264}~\cite{H264OverviewJul2003}, using a more sophisticated approach with two key differences.
Firstly, the Merge Mode allows for more \ac{mv} candidates. Thus, an index value is included to select one out of several \ac{mv} candidates. Secondly, it explicitly identifies the reference picture index used for the selected candidate, as well as, the reference picture list. This results in an improved flexibility, compared with the predefined values used in \ac{h264}. The \ac{mv} candidates allowed in the Merge Mode\cite{MotionCodingHEVC2013} are illustrated in Figure \ref{Figure2:MVCandidates}. The figure illustrates the spatial candidates' positions ($A_0$, $A_1$, $B_0$, $B_1$, $B_2$).
Moreover, it also uses a temporal motion vector candidate, derived from the co-located position on the temporal adjacent frame at positions $T_0$ and $T_1$. From this \ac{mv} candidate list the unavailable candidates are not accounted and duplicated candidates (i.e. with the same motion information) are removed.
In the Merge Mode, a $C-1$ number of candidates are used, selected from the aforementioned order and one temporal \ac{mv} candidate is selected to fulfil the total number of \ac{mv} ($C$). The value of $C$ is defined at the slice header and a flag controlling the use of temporal \ac{mv} candidates is sent at the \ac{pps}-level.
For B-slices, additional \ac{mv} candidates are generated by choosing two existing candidates according to a predefined
order for reference picture list 0 and list 1. A total number of twelve combination are specified in \ac{hevc} which are listed in ~\cite{HEVCOverviewDez2012}. For P-slices if the number of candidates is less than $C$ zero motion vectors are included to fill the remaining entries.
The Merge mode was introduced in \ac{hevc} as a new technique to derive motion information from spatial and temporal neighbouring blocks~\cite{MotionCodingHEVC2013}. This technique extends the concepts of the Direct and Skip modes used in \ac{h264}~\cite{H264OverviewJul2003}, using a more sophisticated approach with two key differences.
Firstly, the Merge Mode allows for more \ac{mv} candidates than the Skip in \ac{h264}.
Thus, an index is used to identify one out of several \ac{mv} candidates.
Secondly, the reference picture index is explicitly identified.
This results in improved flexibility, compared with the predefined values used in \ac{h264}.
The \ac{mv} candidates allowed in the Merge Mode\cite{MotionCodingHEVC2013} are illustrated in Figure \ref{Figure2:MVCandidates}.
The figure shows the positions of the spatial \ac{mv} candidates ($A_0$, $A_1$, $B_0$, $B_1$, $B_2$).
Moreover, it also uses a temporal \ac{mv} candidate, derived from the co-located position on the temporally adjacent frame at positions $T_0$ or $T_1$. From the \ac{mv} candidate list the unavailable candidates are not considered, and duplicated candidates (i.e. with the same motion information) are removed.
The Merge Mode uses a $C-1$ spatial \ac{mv} candidates, selected from the aforementioned order, and one temporal \ac{mv} candidate to fulfil the total number of \acs{mv} ($C$). The value of $C$ is defined at the slice header and a flag controlling the use of temporal \ac{mv} candidates is encoded at the \ac{pps}-level.
For B-slices, additional \ac{mv} candidates are considered by choosing two existing candidates according to a predefined order for reference picture list 0 and list 1. A total number of twelve combinations are specified in \ac{hevc}, which are listed in~\cite{HEVCOverviewDez2012}. For P-slices, if the number of candidates is less than $C$, then zero \acs{mv} are included to fill the remaining entries.
The Merge Mode allows for the signalling of a \ac{mv} candidate and the residual information. In \ac{hevc}, the Skip mode, i.e., transmitting only the \ac{mv} without residual information, is treated as a special case of the Merge Mode. In this case only a skip flag is enabled, and the corresponding merge index is transmitted.
The Merge Mode defines how to encode the \ac{mv} candidate and the residual information.
In \ac{hevc}, the Skip mode, i.e., transmitting only the \ac{mv} without residual information, is treated as a special case of the Merge Mode. In this case only a skip flag is enabled, and the corresponding merge index is transmitted.
\subsubsection{Motion vector prediction}
The \ac{hevc} also improves the performance of the motion vector coding for those cased where the Merge Mode is not used. \ac{hevc} always use a difference encoding approach for efficiently represents the motion information.
It considers the same \ac{mv} candidate of the Merge mode, illustrated in Figure~\ref{Figure2:MVCandidates}, to derive the motion vector predictors from which the difference vector is obtained.
From these \ac{mv} candidates only two are selected based on the following:
The \ac{hevc} also improves the performance of the \ac{mv} coding for those cases where the Merge Mode is not used.
\ac{hevc} always use a differential encoding approach for efficiently representing the motion information.
It considers the same \ac{mv} candidates of the Merge mode, illustrated in Figure~\ref{Figure2:MVCandidates}, to derive the \ac{mv} predictors from which the difference vector is obtained.
From these \ac{mv} candidates only the first two available candidates are selected based on the following order:
\begin{enumerate}
\item Left candidate: first available from $A_0$ and $A_1$.
\item Top candidate: first available from $B_0$, $B_1$ and $B_2$.
@@ -556,7 +610,8 @@ From these \ac{mv} candidates only two are selected based on the following:
\end{enumerate}
Therefore, the use of the temporal \ac{mv} candidate is limited to those cases where the spatial neighbours are not able to provide two valid candidates.
\ac{hevc} only allows a much lower number of candidates to be used in the motion vector prediction process, since it can still send a motion vector difference. Moreover, the encoder will perform motion estimation for each candidate, which increase the encoding complexity for every possibility tested. For those cases where the reference index is not the same in the current \ac{pu} and the neighbouring one, a scaling version of the \ac{mv} is used.
For \ac{mv} prediction only a much lower number of candidates is allowed, since the encoder can still send a \ac{mv} difference. Moreover, the encoder performs motion estimation for each candidate, which increases the encoding complexity for every possibility tested.
%For those cases where the reference index is not the same as the current \ac{pu} and the neighbouring one, a scaling version of the \ac{mv} is used.
%In the \ac{h264} standard, along different motion compensated coding techniques~\cite{H264OverviewJul2003}, the so-called P\_SKIP mode was introduced. In this mode, neither the quantised residual information, nor the \ac{mv} is transmitted. This mode is useful for large areas with no change or just constant motion, and it is only enabled for $16\times16$ blocks, \emph{i.e.}, the whole \ac{mb}. The \ac{mv} for the P\_SKIP mode is derived based on the surrounding neighbour MBs\cite{H264MotionVectorPrediction2003}. When no neighbours are available, or the left or top \acs{mv} are equal to zero, a zero \ac{mv} is used. Otherwise, the \ac{mv} is derived from the median value of the following three neighbours: left, above and above right, as shown in Figure \ref{SkipMergeMode}~(a) by $MV_A$, $MV_B$ and $MV_C$, respectively. Whenever the $MV_C$ is not available, the $MV_D$ is used instead.
%
@@ -570,24 +625,27 @@ Therefore, the use of the temporal \ac{mv} candidate is limited to those cases w
\label{Section2:TransformQuant}
\glsreset{tb}
\glsreset{tu}
Similarly, to the previous standards, \ac{hevc} uses a transform followed by a quantisation to compress the residual signal.
The residual block is partitioned into multiple square blocks, referred to as \ac{tb}.
Similarly, to the previous standards, \ac{hevc} uses a transform followed by quantisation to compress the residual signal.
The residual block is partitioned into multiple square blocks, referred to as \acs{tu}.
\subsubsection{Core transform}
A 2D transform of the residual information is obtained by applying 1D transforms in both the horizontal and vertical directions. The core transform matrices are obtained from the \ac{dct} basis functions to define a integer matrix of $32\times32$ points. Sub-sampled versions of this matrix are used to derive the remaining transform matrix sizes down to $4\times4$.
A special case is applied for the residual coding of a $4\times4$ intra coded block. In this case an alternative integer transform based on the \ac{dst}.
A special case is applied for coding the residual of $4\times4$ intra coded block.
In this case an alternative integer transform based on the \ac{dst} is used.
The motivation for \ac{dst} is related to the statistics of the residual signal of these blocks, which tend to present higher residue as the distance to the boundary used for prediction (i.e, top or left boundary) increases.
Experiments have shown that DST provides up to 1\% of bitrate savings for the compression of $4\times4$ intra predicted blocks, without significantly increase the computational complexity \cite{DCTDSTComp2013}.
Experiments have shown that \ac{dst} provides up to 1\% of bitrate savings for the compression of $4\times4$ intra predicted blocks, without significantly increase the computational complexity \cite{DCTDSTComp2013}.
\subsubsection{Quantisation}
\ac{hevc} standard follows the same principle introduced in \ac{h264} for the quantisation method. After obtaining the residual transform a quantisation is applied using a uniform-reconstruction quantiser scheme based on the \ac{qp}. The standard supports \ac{qp} values ranging from 0 to 51, and a increase of 6 in the \ac{qp} corresponds to doubling the quantisation step. This procedure is the main responsible to introduce the coding errors as it performs many-to-one mapping, which cannot be reversed.
\glsreset{qp}
The \ac{hevc} follows the same principle introduced previous standards for quantisation.
After applying the transform operation to prediction residue transform a quantisation is applied to the resulting coefficients using a uniform-reconstruction quantiser scheme based on a \ac{qp}.
The standard supports \ac{qp} values ranging from 0 to 51, and an increase of 6 in the \ac{qp} corresponds to doubling the quantisation step. This procedure is the main responsible for the coding distortion as it performs many-to-one mapping, which cannot be reversed.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -596,10 +654,13 @@ Experiments have shown that DST provides up to 1\% of bitrate savings for the c
\section{Entropy coding}
\label{Section2:Entropy}
In contrast to \ac{h264} standard, which allows two entropy coding methods, \ac{hevc} only uses the \ac{cabac} \cite{HEVCCabac2003}. \Gls{cabac} is an arithmetic coding method that only uses binary symbols, considering different probability models for each symbol. Entropy coding efficiency is closely linked with the context model selected. Therefore, this was carefully designed in \ac{hevc} standard, extending the functionality presented in \ac{h264}
\glsreset{cabac}
\ac{hevc} only uses the \ac{cabac}, which is an arithmetic coding method that only uses binary symbols, considering different probability models for each symbol~\cite{H264Cabac2003}.
Entropy coding efficiency is closely linked with the context model selected. Therefore, this was carefully designed in the \ac{hevc} standard, extending the functionality previously defined for \ac{h264}.
For example, the \ac{hevc} standard exploits the depth of the partition tree or residual transform tree in order to derive the context models for several syntax elements.
Regarding transform coefficient coding, \ac{cabac} uses a scanning method to firstly organise the coefficients and then transmits the position of the last non-zero transform coefficient.
Moreover, it transmits, a significance map, the sign bits and the levels for the transform coefficients.
Regarding transform coefficient coding, \ac{cabac} uses a scanning method to firstly organise the coefficients and then encodes the position of the last non-zero transform coefficient.
Moreover, a significance map is also encoded along the sign bits and the levels of the transform coefficients.
Three coefficient scanning methods are available: the diagonal up-right, the horizontal and the vertical scan.
The coefficient scanning is implicitly selected and always performed in $4\times4$ sub-blocks for all transform block sizes.
% Details about the context models selection and the compression of each syntax element of \ac{hevc} can be found in [1].
@@ -610,10 +671,15 @@ The coefficient scanning is implicitly selected and always performed in $4\times
\section{In-loop filters}
\label{Section2:Filters}
The \ac{hevc} standard uses two filtering procedures over the reconstructed pixels before writing them into the frame buffer, namely the Deblocking Filter and the \ac{sao} filter.
The main purpose of deblocking filter is to reduce blocking artefacts caused by the block-based coding tools, operating similarly to the deblocking filter of the \ac{h264} standard.
DBF is applied to the samples adjacent to the \ac{pu} or \ac{tu} boundaries, that are aligned on a $8\times8$ grid. Such grid-based restriction reduces the computational complexity and improves parallel-processing
implementation which is one of the main goals of \ac{hevc}. Three strength levels can be used by filter depending on the coded block characteristics.
\glsreset{dbf}
\glsreset{sao}
The \ac{hevc} standard uses two filtering procedures over the reconstructed pixels before writing them into the frame buffer, namely the \ac{dbf} and the \ac{sao} filter.
The main purpose of deblocking filter is to reduce blocking artefacts caused by the block-based coding tools.%, operating similarly to the deblocking filter of the \ac{h264} standard.
\ac{dbf} is applied to the samples that are spatially adjacent to the \ac{pu} or \ac{tu} boundaries, considering a $8\times8$ grid.
Such grid-based restriction reduces the computational complexity and improves parallel-processing
implementation which is one of the main goals of \ac{hevc}.
Three strength levels can be used by the \ac{dbf} depending on the coded block characteristics.
The \ac{sao} is a non-linear filter, that is adaptively applied to all samples of the image, after the deblocking filter. In its operation, the \ac{sao} filter modifies the samples
by adding an offset value, extracted from lookup tables transmitted by the encoder. For each \ac{ctu}, the encoder decides whether to apply the \ac{sao} filter or not, and if used, one of the two available filter types is applied, namely the band offset or the edge offset. In the band offset mode, the added offset value depends on the sample amplitude, while the edge offset uses the gradient to classify and derive the offset value.
@@ -626,13 +692,15 @@ by adding an offset value, extracted from lookup tables transmitted by the encod
\section{Summary}
\label{Section2:Summary}
This chapter described the main features of the \glsfirst{hevc} standard, giving a special focus on the high-level features, coding structures and newly introduced prediction modes.
This chapter was devoted to describe the main features of the \ac{hevc} standard, giving a special focus on the high-level features, coding structures and newly introduced prediction modes.
At the high-level \ac{hevc} introduces new \ac{nal} units types, not only for new signalling features but also for the new pictures types introduced. \ac{hevc} introduces new picture types to handle different types of random-access strategies, which allow more flexibility and higher coding efficiency, when compared to the random-access in previous standard such \ac{h264}.
At the high-level, \ac{hevc} introduces new \ac{nalu} types, and also new pictures types.
These allow the implementation of various random-access strategies, more flexibility and higher coding efficiency then its predecessor \ac{h264}.
Another major improvement comes from the inter-frame prediction. On the one hand new high-level reference picture management was introduced, which increase the error robustness, as it explicitly defines the active reference frame and the \ac{dpb} status. On the other hand it introduces the \ac{amvp} and the Merge mode which efficiently reduce the required bits or motion signalling.
Another major improvement comes from the inter-frame prediction. On the one hand new high-level reference picture management was introduced, which increase the error robustness, as it explicitly defines the active reference frames and the \ac{dpb} status. On the other hand it introduces the \ac{amvp} and the Merge mode which efficiently reduce the required bits for coding the motion information.
In general, \ac{hevc} achieves higher coding performance than its predecessors. Previous studies reported an average bitrate saving of approximately 50\% for the same subjective quality. Such amount of bitrate saving is normally related with increased prediction structures with higher dependencies. Such dependencies are more sensible to data loss, which will increase the error propagation to subsequent coding structures. Therefore, it is emergent to evaluate the impact of the transmission errors in the \ac{hevc} and develop efficient techniques to cope with data loss and reduce the error propagation.
The newly developed coding tools and prediction modes described in this chapter have resulted in approximately 50\% of bitrate savings for the same subjective quality in comparison with the previous standard.
However, such high coding efficiency implies higher data dependencies, which are more sensible to data loss and increase the error propagation to subsequent coding structures. Therefore, evaluating the impact of transmission errors and error propagation in \ac{hevc} is of most importance to develop efficient techniques to recover from data loss and reduce the error propagation.