jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Fix figures and add intro figure

Browse Source
This commit is contained in:
jfmcarreira
2018-08-01 03:13:39 +01:00
parent 74109b2c4c
commit cd670b8b51
51 changed files with 243 additions and 395 deletions
Download Patch File Download Diff File Expand all files Collapse all files
021_VideoCoding/VideoCodingReview.tex
+81 -81
View File
@@ -15,17 +15,17 @@
\glsunset{h264}
\glsreset{hevc}
This chapter describes the H.265~-~\ac{hevc}~\cite{H265StandardFeb2018} standard, focusing on the most relevant features of the encoding algorithm in the scope of the work developed in this thesis.
This chapter describes the H.265~-~\ac{hevc}~\cite{H265StandardFeb2018} standard, focusing on the most relevant features of the encoding algorithm in the scope of the work developed in this thesis.
In short, the \ac{hevc} standard essentially aims to support all existing applications of \ac{h264}~\cite{H264AVCStandard2003}, while extending two keys aspects: increased video resolution with higher compression ratio and also more efficient use of parallel processing architectures.
In order to increase the coding flexibility and higher video resolutions, the \ac{hevc} standard adopts a new block partition structure, as well as, new prediction modes.
In this chapter the new data structures coding techniques specifically introduced to increase the coding efficiency are described.
In order to increase the coding flexibility and higher video resolutions, the \ac{hevc} standard adopts a new block partition structure, as well as, new prediction modes.
In this chapter the new data structures coding techniques specifically introduced to increase the coding efficiency are described.
Particular emphasis is given to those which may have a significant impact in the error robustness characteristics of \ac{hevc} streams.
This chapter is organized as follows.
Section~\ref{Section2:GeneraVideoSystem} introduces the functional blocks of a generic hybrid video coding system, and Section~\ref{Section2:Overview} provides an overview of the \ac{hevc} standard.
This chapter is organized as follows.
Section~\ref{Section2:GeneraVideoSystem} introduces the functional blocks of a generic hybrid video coding system, and Section~\ref{Section2:Overview} provides an overview of the \ac{hevc} standard.
Then, Section~\ref{Section2:HighLevel} describes the high-level syntax elements which are used to carry relevant information in coded streams.
Section~\ref{Section2:CodingStructures} describes the data structures used to divide the video frames into smaller partitions and Section~\ref{Section2:PredictionModes} describes particular aspects of the main predictions techniques, namely intra and inter coding.
Finally, a brief description of the transform and quantisation, entropy coding and in-loop filters is provided in Sections~\ref{Section2:TransformQuant}, \ref{Section2:Entropy} and \ref{Section2:Filters}, respectively.
Section~\ref{Section2:CodingStructures} describes the data structures used to divide the video frames into smaller partitions and Section~\ref{Section2:PredictionModes} describes particular aspects of the main predictions techniques, namely intra and inter coding.
Finally, a brief description of the transform and quantisation, entropy coding and in-loop filters is provided in Sections~\ref{Section2:TransformQuant}, \ref{Section2:Entropy} and \ref{Section2:Filters}, respectively.
Section~\ref{Section2:Summary} concludes this chapter.
\clearpage
@@ -35,7 +35,7 @@ Section~\ref{Section2:Summary} concludes this chapter.
\section{General hybrid video coding}
\label{Section2:GeneraVideoSystem}
A digital video signal is a temporal succession of images, also referred to as pictures or frames, which usually presents a significant amount of temporal and spatial correlations.
A digital video signal is a temporal succession of images, also referred to as pictures or frames, which usually presents a significant amount of temporal and spatial correlations.
In general, video coding and distribution systems follow through a generic chain of processing and delivery stages: pre-processing, encoding, storage or transmission, decoding and post-processing.
The pre-processing stage is used adapt the raw video signal, either analogue or digital, to an appropriate format, such as RGB or YUV~\cite{DataCompression1997}.
Then, the video data is compressed by the respective encoding module, mostly reducing its redundancy and visual irrelevancy. The encoding process is responsible for representing the raw video in a highly compressed format, while maintaining a good trade-off between video quality and bitrate through high efficient algorithms and coding tools.
@@ -50,7 +50,7 @@ In the last stages of the chain, the compressed format is decoded video and pres
Since the early days of video compressing, hybrid video coding has been the basic structure for all video coding standards and recommendations of ITU-T \ac{vceg} and \ac{iso}/\ac{iec} \ac{mpeg}, e.g., from H.261 to \ac{hevc}.
Since the early days of video compressing, hybrid video coding has been the basic structure for all video coding standards and recommendations of ITU-T \ac{vceg} and \ac{iso}/\ac{iec} \ac{mpeg}, e.g., from H.261 to \ac{hevc}.
Figure~\ref{Figure2:HybridCodec} shows the block diagram of a generic hybrid video coding system~\cite{BookLucas2017}.
While the algorithmic structure has not been changed, the specific techniques represented by the building blocks have been refined and the applicable coding configuration has become more and more flexible over years through the use of an increasing variety of coding tools.
This is the reason why coding scheme is called hybrid, as it combines quiet different prediction modes between pictures with transform and entropy coding~\cite{VideoCodingFundamentals2015}.
@@ -69,12 +69,12 @@ On the one hand, the spatial redundancy is exploited through intra frame predict
This is possible due to a high correlation between neighbouring pixels within a frame. As result, in intra frame prediction, a block of pixels can be reconstructed from the blocks previously processed.
On the other hand, inter frame prediction is designed to exploit the temporal redundancy that exists between consecutive frames such as those temporally adjacent, e.g., the past frame or the future frame in regard to the current one
Temporal redundancy is normally referred to as the correlation between two adjacent frames since they usually have quiet similar visual content, such as visual objects and background.
Thus, the purpose of inter-frame prediction is to search for the best matching block for the current one in a set of reference frames.
Thus, the purpose of inter-frame prediction is to search for the best matching block for the current one in a set of reference frames.
After finding the best matching candidate for the current block, either by intra- or inter-frame prediction, only the difference between the original block and the predicted one needs to be encoded.
As shown in Figure~\ref{Figure2:HybridCodec}, these differences (also known as residual signal) are further processed and transformed (T) into the coefficients in the frequency domain.
As shown in Figure~\ref{Figure2:HybridCodec}, these differences (also known as residual signal) are further processed and transformed (T) into the coefficients in the frequency domain.
Such a transformation decorrelates the residue signal and concentrates the signal energy in few coefficients, mostly at low frequencies.
Subsequently, the transformed coefficients are quantized (Q) to reduce the representation accuracy of the signal, and consequently reduce the number of bits required for coding.
Subsequently, the transformed coefficients are quantized (Q) to reduce the representation accuracy of the signal, and consequently reduce the number of bits required for coding.
Finally, the quantized transform coefficients together with the prediction information are further compressed by the entropy encoder, forming an output stream suitable for transmission or storage, as shown in Figure~\ref{Figure2:HybridCodec}.
On the other side of the communication chain, the decoding stage performs the inverse operations on the stream to reconstruct the video signal. In the next section, the aforementioned blocks are described in the context of \ac{hevc} standard.
@@ -113,7 +113,7 @@ The encoder uses both intra- and inter-picture prediction to exploit spatial and
\glsreset{rpset}
There are some novelties introduced in the \ac{hevc} standard. As depicted in Figure \ref{fig:HEVCBlockDiagram}, the encoding process follows a block-based approach, therefore each picture is partitioned into larger blocks, referred to as \acs{ctu}, with a maximum size of $64\times64$. Subsequently, a dynamic quad-tree partitioning is used to divide the picture into \acs{cu}. These units are introduced as the basic processing units for prediction, and for the transform and quantisation operations \acs{tu} are used.
In order to generate the prediction residue the \ac{hevc} may use one of 34 intra-prediction modes or take advantage of symmetric and asymmetric partitions for motion estimation and inter-prediction.
In order to generate the prediction residue the \ac{hevc} may use one of 34 intra-prediction modes or take advantage of symmetric and asymmetric partitions for motion estimation and inter-prediction.
The motion estimation is performed in those pictures listed in the \ac{rpset} which was introduced in the \ac{hevc} to increase the error robustness of reference picture lists.
To improve the overall quality of the reconstructed pictures and smooth the coding artefacts, \ac{hevc} not only uses a deblocking filter but also a \ac{sao} filter. Finally, the control data, quantised coefficients, prediction modes, motion vectors and filter information are compressed using the \ac{cabac} entropy coder and generate a compressed bitstream composed of different \acs{nalu}.
@@ -144,7 +144,7 @@ Similar to \ac{h264}~\cite{H264AVCStandard2003}, the \ac{hevc} streams are based
\bitbox{#2}{#3}}
\begin{bytefield}[bitwidth=20pt]{16}
% \bitheader{0-15} \\
\colorbitbox{\TikZColorFill}{1}{F} & \colorbitbox{gray!60}{6}{NAL Type} & \colorbitbox{\TikZColorFill}{6}{Reserved} & \colorbitbox{gray!60}{3}{TIDP}
\colorbitbox{TikZColorFill}{1}{F} & \colorbitbox{gray!60}{6}{NAL Type} & \colorbitbox{TikZColorFill}{6}{Reserved} & \colorbitbox{gray!60}{3}{TIDP}
\end{bytefield}
\caption{Two bytes \ac{nal} unit header in \ac{hevc}.}
\label{Figure2:NalHeader}
@@ -209,7 +209,7 @@ The new set of parameters \ac{vps} contains information related to the different
\end{tabular}
\end{table}
The \ac{hevc} standard supports several pictures types as show in Table~\ref{Table2:PictureTypes}, which are divided in the following categories: (i) random-access points, (ii) leading pictures and (iii) temporal sub-layer access pictures~\cite{HEVCHighLevelReferenceDez2012,BookHEVC2014}.
The \ac{hevc} standard supports several pictures types as show in Table~\ref{Table2:PictureTypes}, which are divided in the following categories: (i) random-access points, (ii) leading pictures and (iii) temporal sub-layer access pictures~\cite{HEVCHighLevelReferenceDez2012,BookHEVC2014}.
The use of different types of pictures allows the \ac{hevc} stream to be more flexible and support a wider range of applications and a robust video communications.
They are important to perform random-access, which is a critical feature for channel switching, seek operations and dynamic streaming services.
Moreover, introducing random-access points into the bitstream increases its error robustness as they provide anchor points for a clean decoding after data loss or synchronisation problems.
@@ -219,25 +219,25 @@ Moreover, introducing random-access points into the bitstream increases its erro
\subsubsection{Random-Access Point}
The \ac{rap} pictures are used to provide points in the bitstream with no dependencies, where it is possible to start the decoding process. The standard defines that a conforming bitstream must start with a \ac{rap} picture.
These frames must belong to temporal sub-layer 0 and coded using intra coding techniques, i.e., must not using previous decoded frames as reference.
One should note that there may be frames solely compressed with solely intra prediction but not be marked as random-access points.
The \ac{rap} pictures are used to provide points in the bitstream with no dependencies, where it is possible to start the decoding process. The standard defines that a conforming bitstream must start with a \ac{rap} picture.
These frames must belong to temporal sub-layer 0 and coded using intra coding techniques, i.e., must not using previous decoded frames as reference.
One should note that there may be frames solely compressed with solely intra prediction but not be marked as random-access points.
It is always possible to start decoding from a \ac{rap} frame onwards, and to output any subsequent pictures in display order even if all pictures that precede the \ac{rap} in decoding order are discarded from the bitstream.
The \ac{idr} frame is an intra coded picture that completely refreshes the decoding process and starts a new conforming video stream. This means that neither the \ac{idr} nor any other subsequent frame in the decoding order have prior dependencies. \ac{idr} frames are allowed to have leading pictures, but they must be decodable, thus creating a true random-access point where all subsequent frames in the decoding order can be decoded and displayed.
Introducing \ac{rap} reduces the coding efficiency since no prior reference frames can be used.
In order to reduce this impact, the \ac{hevc} standard introduces a new \ac{rap} picture, referred to as \ac{cra} picture.
These frames do not refresh the decoding process and allow leading pictures to depend on frames that precede the \ac{cra} picture in the decoding order.
In order to reduce this impact, the \ac{hevc} standard introduces a new \ac{rap} picture, referred to as \ac{cra} picture.
These frames do not refresh the decoding process and allow leading pictures to depend on frames that precede the \ac{cra} picture in the decoding order.
This enables a more efficient prediction structure, while maintaining an access point to begin the decoding process.
However, it results in leading pictures which may not be decodable, which are explained in the next sub-section.
The introduction of \ac{cra} frames in \ac{hevc} was able to improve the coding efficiency by up to 6\% \cite{HEVCCRAPics2011}.
%\HERE
The \ac{hevc} supports bitstream splicing, which means taking a particular bitstream starting from a \ac{rap} (both \ac{idr} and \ac{cra} frames) and insert it into another bitstream at a random-access point.
The \ac{hevc} supports bitstream splicing, which means taking a particular bitstream starting from a \ac{rap} (both \ac{idr} and \ac{cra} frames) and insert it into another bitstream at a random-access point.
In case of bitstream splicing starting from an \ac{idr} frame the leading pictures does not have dependencies to frames prior the \ac{rap}, thus all frames are decodable.
However, whenever this is performed for a bitstream starting from a \ac{cra} frame, the leading associated frames might not be correctly decoded because some of their reference pictures are not present in the combined bitstream.
However, whenever this is performed for a bitstream starting from a \ac{cra} frame, the leading associated frames might not be correctly decoded because some of their reference pictures are not present in the combined bitstream.
Therefore, in order to make splicing operation straightforward the \acs{nalu} containing the \ac{cra} picture are changed to \acs{nalu} of type \ac{bla}. This way the decoder is aware that the leading frames should be discarded, namely the \ac{rasl} pictures.
@@ -260,13 +260,13 @@ Due to the introduction of more flexible random-access points using \ac{cra} pic
There are two types of leading pictures as shown in Table~\ref{Table2:PictureTypes}.
The \ac{radl} pictures which only depend on the associated \ac{rap} picture and do not have any prior dependencies to trailing pictures associated with previous \ac{rap} frame and can be correctly decoded.
On contrary, the \ac{rasl} pictures may have dependencies to a prior trailing pictures.
On contrary, the \ac{rasl} pictures may have dependencies to a prior trailing pictures.
Therefore, when a random-access is performed at the associated \ac{rap} frame, these frames cannot be correctly decoded, and should be skipped. The example of ~\ref{Figure2:LeadingTralingPics} the leading pictures represented correspond to \ac{rasl} pictures.
\subsubsection{Temporal sub-layer switching pictures}
A \ac{tsa} picture is a trailing picture that marks a temporal layer switching point.
A \ac{tsa} picture is a trailing picture that marks a temporal layer switching point.
When decoding a sub-set of the temporal layers and a \ac{tsa} picture is found from the temporal layer just above the maximum temporal layer currently being decoded, it is possible for the decoder to start decoding any number of additional temporal layers.
For example, frame $P_6$ in Figure~\ref{Figure2:TSAExample} is considered a \ac{tsa} picture since it only depends on a frame belonging to temporal layer 0 for prediction, as well as, any subsequent frame predicted from the \ac{tsa} frame $P_6$.
In this example, if the decoder was only decoding the temporal layer 0 it can after $P_6$ starts decoding both layer 0 and 1 or decode all the available three layers. Although theses frame allows higher flexibility in terms of layer switching, they severely constrain the prediction of frames following the \ac{tsa} picture.
@@ -279,10 +279,10 @@ In this example, if the decoder was only decoding the temporal layer 0 it can af
\label{Figure2:TSAExample}
\end{figure}
To reduce the constrains in frame prediction due to layer switching, \ac{hevc} introduces the \ac{stsa} picture.
To reduce the constrains in frame prediction due to layer switching, \ac{hevc} introduces the \ac{stsa} picture.
These pictures have similar purpose as the \ac{tsa} frames but they only guarantee that frames in the same temporal layer as the \ac{stsa} frame in the decoding order are able to be correctly reconstructed. This is guaranteed by preventing any dependency from frames preceding the \ac{stsa} to be used as reference in its temporal layer.
One example of a \ac{stsa} frame is also shown in Figure~\ref{Figure2:TSAExample} in frame $P_2$.
This frame can be used to switch to the layer 1 as it does not have dependencies to any prior frame in the same layer ($P_2$ only depends on $I_0$).
One example of a \ac{stsa} frame is also shown in Figure~\ref{Figure2:TSAExample} in frame $P_2$.
This frame can be used to switch to the layer 1 as it does not have dependencies to any prior frame in the same layer ($P_2$ only depends on $I_0$).
However, it cannot be classified as \ac{tsa} picture because $P_3$ has dependencies to prior frame in the decoding order to $P_1$.
Summarising, \ac{stsa} frames can be used to switch to a particular layer as the \ac{tsa} can be used to switch to any layer.
@@ -294,10 +294,10 @@ The high-level segmentation of a picture in \ac{hevc} is achieved through four t
\item[Error robustness:] partitioning the picture into smaller self-contained units in
order to gain error robustness by the ability to re-synchronize both the parsing and decoding
processes in case of data losses.
\item[Network adaptation:] adapt to the network constraint of \ac{mtu} size, found for example in \ac{ip} networks.
Such packetisation scheme restricts the maximum number of payload bits within a slice regardless of the size of the coded frame. To keep each slice within this limit and minimise the packetisation overhead, a variable number of coding units is used for each slice.
\item[Parallel processing:] partitioning the coded frame into processing data units which can be encoded in parallel. This is achieved by dividing the coding units in such way that they can be encoded and decoded independently of each other’s.
\end{description}
@@ -323,7 +323,7 @@ Another partitioning tool in \ac{hevc} standard is the dependent slices. These s
A new data partition has been included in the \ac{hevc} standard, referred to as Tiles\cite{HEVCTilesDec2013}. Tiles enable the partition of a coded frame into a smaller group of \acs{ctu} using horizontal and/or vertical boundaries. Figure~\ref{Figure2:HEVCTiles} shows two examples of slice partitioning using tiles. In both cases the frame is partitioned into three tiles by using two vertical boundaries. By only using boundaries to define tiles, signalling is achieved with a small overhead. These boundaries are defined in the \ac{pps} \acs{nalu} .
Although the \ac{hevc} standard enables the use of both slices and tiles at the same time, some constrains are imposed in order to reduce the implementation complexity.
Although the \ac{hevc} standard enables the use of both slices and tiles at the same time, some constrains are imposed in order to reduce the implementation complexity.
One of two cases must be used: (i) all \acs{ctu} within a tile must belong to the same slice, or (ii) all \acs{ctu} within a slice must belong to the same tile. These cases are illustrated in Figure~\ref{Figure2:HEVCTiles} (a) and (b), respectively.
The use of tiles provide several advantages. Specifically, they improve frame partitioning for parallel processing~\cite{WPPTilesParallelHEVC2017} when compared to slices, by reducing the required overhead. Moreover, they also improve the \ac{mtu} size matching and reduce the line buffer memory (by dividing the frame as in Figure~\ref{Figure2:HEVCTiles}). Finally, as tiles allow flexible partitioning, they enable the definition of \acs{roi} for asymmetric video coding.
@@ -363,13 +363,13 @@ In \ac{h264} each frame is divided into units so-called \acs{mb}, which are comp
\glsreset{ctu}
\glsreset{cu}
The basic data structures in \ac{hevc} are quite different from \ac{h264}. Each picture is divided into blocks named as \ac{ctu}~\cite{BlockPartHEVC2012}. The maximum \ac{ctu} size is an encoder configuration parameter and signalled to the decoder, and should be one of the following: $64\times64$, $32\times32$ and $16\times16$. The use of data structures with larger size in \ac{hevc} was adopted to achieve higher coding efficiency especially in high resolution video. Each \ac{ctu} can be recursively partitioned into four smaller units, referred to as \acs{cu}, until it reaches the minimum unit size, which cannot be smaller than to $8\times8$ pixels. Similar to the maximum \ac{ctu} size, the minimum size is also defined at the encoding time.
The basic data structures in \ac{hevc} are quite different from \ac{h264}. Each picture is divided into blocks named as \ac{ctu}~\cite{BlockPartHEVC2012}. The maximum \ac{ctu} size is an encoder configuration parameter and signalled to the decoder, and should be one of the following: $64\times64$, $32\times32$ and $16\times16$. The use of data structures with larger size in \ac{hevc} was adopted to achieve higher coding efficiency especially in high resolution video. Each \ac{ctu} can be recursively partitioned into four smaller units, referred to as \acs{cu}, until it reaches the minimum unit size, which cannot be smaller than to $8\times8$ pixels. Similar to the maximum \ac{ctu} size, the minimum size is also defined at the encoding time.
Figure~\ref{Figure2:HEVCCodingStructure} illustrates an example of a \ac{ctu} partitioning into \acs{cu}. The partitioning illustrated in the figure leads to the quad-tree structure represented on the left side with the corresponding coding order shown on the right side.
\glsreset{pu}
\glsreset{tu}
The \ac{cu} is the basic processing unit in \ac{hevc}. It is further partitioned into two types of units, specifically \ac{pu} and \ac{tu}, which are data blocks used for prediction and transform operations, respectively.
The \ac{cu} is the basic processing unit in \ac{hevc}. It is further partitioned into two types of units, specifically \ac{pu} and \ac{tu}, which are data blocks used for prediction and transform operations, respectively.
The decision whether to use intra-frame or inter-frame prediction modes is taken at \ac{cu} level. However, depending on the partitioning of the \ac{cu} into \acs{pu} different intra or inter coding modes can be used for each \ac{pu}.
\HERE
@@ -392,22 +392,22 @@ This section describes the prediction techniques used in the \ac{hevc} standard
\subsection{Intra-frame prediction}
Intra-frame prediction is used to efficiently reduce the spatial redundancy in video frames.
Intra predicted blocks are obtained using the previously encoded pixels around the current \ac{pu}.
Intra-frame prediction is used to efficiently reduce the spatial redundancy in video frames.
Intra predicted blocks are obtained using the previously encoded pixels around the current \ac{pu}.
When the intra prediction mode is chosen for a \ac{cu}, the size of the \ac{pu} need to be the same as the \ac{cu}, except when the smaller \ac{cu} is selected and further splitting into four \acs{pu} is done. For example, when the smaller \ac{cu} size is $8\times8$, the encoder may decide to divide it into four \acs{pu} of $4\times4$ pixels, each one having its own intra prediction mode. This is signalled using a binary flag.
The use of small \acs{pu} is useful for complex regions that require fine-granularity prediction.
In \ac{hevc} there are 35 different of intra-frame prediction modes: Planar, DC and 33 angular modes as shown on the left of Figure~\ref{Figure2:IntraModesHEVC}~\cite{IntraCodingHEVC2012,HEVCIntraperformance2012}.
%Figure~\ref{Figure2:IntraModesHEVC} illustrates these intra prediction modes.
%As shown, the predictions are applied to square blocks only.
In \ac{hevc} there are 35 different of intra-frame prediction modes: Planar, DC and 33 angular modes as shown on the left of Figure~\ref{Figure2:IntraModesHEVC}~\cite{IntraCodingHEVC2012,HEVCIntraperformance2012}.
%Figure~\ref{Figure2:IntraModesHEVC} illustrates these intra prediction modes.
%As shown, the predictions are applied to square blocks only.
%The border around the square blocks corresponds to the reference pixels from which the prediction is applied (different colours correspond to different pixel values).
%The example in the figure corresponds to a \ac{pu} size of $8\times8$ pixels, but block sizes ranging from $4\times4$ to $32\times32$ are also allowed.
The encoder chooses among all the available intra prediction modes and signals the best mode to the decoder, which must generate an identical prediction block, based on the previously reconstructed pixels.
The reconstructed reference pixels used in the prediction process belong to
the neighbouring blocks at left-down, left, top-left, top and top-right positions, as
can be seen in the example shown on the right in Figure~\ref{Figure2:IntraModesHEVC}.
can be seen in the example shown on the right in Figure~\ref{Figure2:IntraModesHEVC}.
Given a $N \times N$ \ac{pu}, intra prediction requires the top neighbouring row of $2N$ pixels, the left column of $2N$ pixels and the top-left neighbouring pixel.
Due to data unit boundaries, e.g., slice or tile boundaries, neighbouring pixels might not be all available to use intra prediction.
Due to data unit boundaries, e.g., slice or tile boundaries, neighbouring pixels might not be all available to use intra prediction.
In addition, when constrained intra prediction is enabled, i.e., reference pixels belonging to inter-predicted \acs{pu} are not used in order to remove error propagation from potentially erroneous reconstructed reference frames.
This may results in an incomplete set of neighbouring reference pixels.
In order to overcome this issue and to allow all possible modes, \ac{hevc} uses reference sample substitution to replace the unavailable reference pixels with the closest available ones, so that all intra prediction modes can be used.
@@ -427,7 +427,7 @@ In order to overcome this issue and to allow all possible modes, \ac{hevc} uses
\subsubsection{Angular modes}
Directional intra prediction was first introduced in the \ac{h264} standard~\cite{H264OverviewJul2003}, to estimate regions with a structured texture or directional edges.
Directional prediction projects the reconstructed samples in the block neighbourhood (see reference pixels in Figure~\ref{Figure2:IntraModesHEVC}) along a specific direction. The \ac{hevc} increases the number of possible directions from 8 (in \ac{h264}) to 33~\cite{IntraCodingHEVC2012}. The direction of each projection is illustrated in the left-side of Figure~\ref{Figure2:IntraModesHEVC} by the directional modes 2 to 34.
Directional prediction projects the reconstructed samples in the block neighbourhood (see reference pixels in Figure~\ref{Figure2:IntraModesHEVC}) along a specific direction. The \ac{hevc} increases the number of possible directions from 8 (in \ac{h264}) to 33~\cite{IntraCodingHEVC2012}. The direction of each projection is illustrated in the left-side of Figure~\ref{Figure2:IntraModesHEVC} by the directional modes 2 to 34.
%In the figure it is possible to notice that directional modes 2 to 10 only project the pixels from the left neighbouring border in a horizontal direction, modes 10 to 25 use both the left and top neighbouring pixels in a diagonal direction and modes 26 to 34 project the top neighbours in vertical direction.
As shown in the figure, the angular modes are designed to provide a dense coverage near the horizontal (i.e., mode 10) and vertical (i.e., mode 26) directions. Moreover, the angular directions are more coarse as it gets closer to the diagonals. This reflects the observed statistical prevalence of the angles and the prediction efficiency.
@@ -454,15 +454,15 @@ Since the right and bottom neighbourhood is not available (not coded yet), the s
\subsubsection{Sample smoothing}
In \ac{hevc} sample smoothing is applied in two distinct cases to improve the overall performance of the prediction accuracy.
Firstly, the reference samples are filtered and secondly the \ac{pu} is filtered after prediction.
In \ac{hevc} sample smoothing is applied in two distinct cases to improve the overall performance of the prediction accuracy.
Firstly, the reference samples are filtered and secondly the \ac{pu} is filtered after prediction.
%In comparison with previous standard \ac{h264}, where only reference samples were filtered.
In \ac{hevc}, the reference sample smoothing is conditionally applied, based on the \ac{pu} and intra prediction mode. Similar to \ac{h264}, a three tap smoothing filter (i.e., $\left[1\ 2\ 1\right]/4$) is applied, except for \ac{pu} sizes of $4\times4$. The prediction resulting from Planar mode is also filtered when the \ac{pu} size is greater than $8\times8$, and it is not used for the DC mode.
The second filter in \ac{hevc} is the boundary smoothing. This filter aims at removing the discontinuities along block boundaries due to intra prediction.
This filter is only applied when the block size is smaller than $32\times32$ to the DC and angular modes 10 and 26 (exactly the horizontal and vertical directions, respectively).
For the DC mode a two-tap filter ($\left[3\ 1\right]/4$), fed by the original predicted value and the adjacent reference sample, is applied to both the top row and the left-most column of predicted pixels.
This filter is only applied when the block size is smaller than $32\times32$ to the DC and angular modes 10 and 26 (exactly the horizontal and vertical directions, respectively).
For the DC mode a two-tap filter ($\left[3\ 1\right]/4$), fed by the original predicted value and the adjacent reference sample, is applied to both the top row and the left-most column of predicted pixels.
In the case of angular mode 10 (horizontal direction), only the predicted samples of the first column are changed by adding half of the difference between the adjacent reference sample and the top-left reference sample.
A similar filtering is done for angular mode 26 (vertical direction), but only for the first row
of predicted samples.
@@ -473,33 +473,33 @@ Only the mentioned boundaries are filtered, as they are the most likely to intro
\subsection{Inter-frame prediction}
The efficiency of video coding algorithms consistently rely on inter-frame prediction techniques to reduce temporal redundancy.
The efficiency of video coding algorithms consistently rely on inter-frame prediction techniques to reduce temporal redundancy.
The underlying idea of inter-frame prediction is to estimate the current frame from one or more previously encoded frames used as reference with block-based motion compensation. The most common technique used for motion estimation in \ac{hevc} is the block matching algorithm. In the \ac{hevc} standard an \ac{amvp} is used for efficient motion vector coding, which defines the Merge Mode and differential coding of the motion information.
\subsubsection{Motion estimation and compensation}
Motion estimation consists in searching for a block with the highest similarity with the current block to be predicted (\ac{pu} in \ac{hevc}), using block-matching algorithms over previously encoded frames, i.e. the reference frames.
Typically, reference frames correspond to past or future temporally adjacent frames.
The frame encoding order determines which references are selected and which future frames are available.
Motion estimation consists in searching for a block with the highest similarity with the current block to be predicted (\ac{pu} in \ac{hevc}), using block-matching algorithms over previously encoded frames, i.e. the reference frames.
Typically, reference frames correspond to past or future temporally adjacent frames.
The frame encoding order determines which references are selected and which future frames are available.
The reference block that results in the lowest error is selected as the best block for prediction.
To identify the best matching block to be used for motion compensation, the different between the target and the reference block positions is encoded as a two-dimensional \glsfirst{mv}.
The \ac{hevc} allows for either one or two \acs{mv} to be used for each \ac{pu}. Video frames of type P are predicted from one single reference (one \ac{mv}) while frame of type B are predicted from two references (two \acs{mv}).
This results in uni-direction or bi-directional coding, respectively.
This results in uni-direction or bi-directional coding, respectively.
Furthermore, weighted prediction can also be used, which consists in scaling and offset operations performed on the prediction block to improve efficiency.
The \acs{mv} only refer to frames included in a reference frame list, where each frame is identified by an index. Therefore, to fully describe inter-frame prediction a combination of two parameters is required: \ac{mv} and reference frame index.
Since MPEG-2, motion estimation/compensation has been performed using blocks of variable size.
Since MPEG-2, motion estimation/compensation has been performed using blocks of variable size.
In \ac{h264} block sizes ranging from $4\times4$ up to $16\times$ pixels are allowed, while \ac{hevc} extends the possible block sizes up to $64\times64$ pixels, and also to asymmetric rectangular block sizes.
By using larger prediction units less \ac{mv} are required and fewer bits are spent in coding motion information.
In order to increase the efficiency of motion estimation fractional \acs{mv} are used, which has been proven to improve motion compensation accuracy.
This is accomplished through interpolation to generate non-integer pixels in the reference frame, resulting in \acs{mv} up to quarter-pixel accuracy for luma samples.
For chroma samples, the \ac{mv} accuracy is determined according to the chroma sampling format, which for 4:2:0 results in eighth-pixel accuracy.
For chroma samples, the \ac{mv} accuracy is determined according to the chroma sampling format, which for 4:2:0 results in eighth-pixel accuracy.
In contrast with the two-stage approach used in \ac{h264}, \ac{hevc} uses a single separable approach for the interpolation process without intermediate rounding operations. This improves the precision and simplifies the architecture of the fractional interpolation.
The block matching-based motion estimation is one of the most computationally intensive procedures in video encoders.
In general, the optimal solution for motion estimation is provided by the full-search algorithm, which tests all the possible candidate blocks in the search area. However, although the full-search algorithm achieves the optimal performance, it requires the highest computational complexity.
In general, the optimal solution for motion estimation is provided by the full-search algorithm, which tests all the possible candidate blocks in the search area. However, although the full-search algorithm achieves the optimal performance, it requires the highest computational complexity.
Therefore, sub-optimal algorithms have been proposed over the years to reduce the motion estimation complexity and decrease encoding time~\cite{HEVCME2012,HEVCME2017,HEVCME2014}.
For example, the HM reference software implements a fast algorithm based on Test Zone Search~\cite{HEVCME2012}.
Since motion estimation is a non-normative procedure, encoders may implement any alternative motion search algorithms.
@@ -508,9 +508,9 @@ Since motion estimation is a non-normative procedure, encoders may implement any
\subsubsection{Reference picture management}
Another improvement in \ac{hevc} is related with the reference picture management scheme. \ac{hevc} introduces the \ac{rpset} concept~\cite{HEVCHighLevelReferenceDez2012}, which defines how previously decoded pictures are managed in the \ac{dpb}.
Decoded frames in the \ac{dpb} are grouped in one of the following categories: (i) \emph{short-term reference}, (ii) \emph{long-term reference} and (iii) \emph{unused for reference}.
Decoded frames in the \ac{dpb} are grouped in one of the following categories: (i) \emph{short-term reference}, (ii) \emph{long-term reference} and (iii) \emph{unused for reference}.
Once a frame is marked as \emph{unused for reference} it is no longer used for motion compensation and it will be discarded from the \ac{dpb} after being sent to the displayed.
In \ac{h264} a sliding window approach is used to implicitly manage the reference picture list.
In \ac{h264} a sliding window approach is used to implicitly manage the reference picture list.
This process discards a given frame whenever the \ac{dpb} has the number of defined reference frames (which is controlled by the encoder). Moreover, \ac{h264} uses an explicitly \ac{mmco} to send commands to the decoder and change the normal reference picture management process.
\begin{table}[tb]
@@ -534,17 +534,17 @@ This process discards a given frame whenever the \ac{dpb} has the number of defi
In the \ac{hevc} standard the status of the \ac{dpb} is encoded for every slice, instead of the implicit managing used in the past.
The most fundamental difference in \ac{rpset} concept compared to \ac{mmco}/sliding window of \ac{h264} is that each particular frame has a complete set of the reference frames that are used or will be used (or will be used in subsequent frames).
Thus, a complete set of all frames that must be kept in the \ac{dpb} explicitly provided.
This is different from \ac{h264} where only relative changes to the \ac{dpb} encoded.
With the \ac{rpset} concept, no information from earlier frames in decoding order is needed to maintain the correct status of the \ac{dpb}.
This is different from \ac{h264} where only relative changes to the \ac{dpb} encoded.
With the \ac{rpset} concept, no information from earlier frames in decoding order is needed to maintain the correct status of the \ac{dpb}.
Table~\ref{Table2:ReferencePictureListExample} shows the \ac{rpset} for some frames illustrated in Figure~\ref{Figure2:LeadingTralingPics}.
As expected for the \ac{cra} frame ($I_2$) no information is sent to the decoder because there is not inter-prediction and the \ac{dpb} should be empty.
As expected for the \ac{cra} frame ($I_2$) no information is sent to the decoder because there is not inter-prediction and the \ac{dpb} should be empty.
For the remaining frames, two types of information are encoded: frame number and a binary flag to indicate whether it is currently used. This is required because any frame not listed in a given \ac{rpset} will be discarded by the decoder and then subsequent frames cannot use them as reference.
The \ac{rpset} concept provides a basic level of error robustness to the reference picture management.
In \ac{h264} whenever a reference frame is lost the decoder can look at the value of the \emph{FrameNum} (incremental value associated with each reference frame) to detect the loss event.
The \ac{rpset} concept provides a basic level of error robustness to the reference picture management.
In \ac{h264} whenever a reference frame is lost the decoder can look at the value of the \emph{FrameNum} (incremental value associated with each reference frame) to detect the loss event.
However, as gaps in \emph{FrameNum} are allowed the decoder might not be able to decide whether the missing frame was intentional or resulted of a transmission loss event.
%Moreover, even if the decoded if able to detect the loss event it might not be able to correctly compute the value the \ac{poc} associated with the concealed frame.
%Moreover, even if the decoded if able to detect the loss event it might not be able to correctly compute the value the \ac{poc} associated with the concealed frame.
Using the \ac{rpset} concept the decoder is always able to detect loss events and correctly identify the missing frames.
\subsubsection{Asymmetric block partitions}
@@ -560,16 +560,16 @@ Using the \ac{rpset} concept the decoder is always able to detect loss events an
\end{figure}
When inter prediction mode is selected a higher range of partitioning schemes is allowed, thus dividing the \ac{cu} into one, two or four \acs{pu}.
The splitting into four \acs{pu} is only allowed when the \ac{cu} size is equal to the minimum.
%Otherwise, the splitting should be carried out at the \ac{cu} level which leads to the same partition tree.
Figure~\ref{Figure2:InterPuPartitions} illustrates the possible \ac{cu} partitioning into \acs{pu}.
When inter prediction mode is selected a higher range of partitioning schemes is allowed, thus dividing the \ac{cu} into one, two or four \acs{pu}.
The splitting into four \acs{pu} is only allowed when the \ac{cu} size is equal to the minimum.
%Otherwise, the splitting should be carried out at the \ac{cu} level which leads to the same partition tree.
Figure~\ref{Figure2:InterPuPartitions} illustrates the possible \ac{cu} partitioning into \acs{pu}.
%One should note that the partition type applied to the \ac{cu} is also applied to divide the \ac{cb} into \acs{pb}.
Figure~\ref{Figure2:InterPuPartitions}~(a) shows the symmetric partitions.
The $N \times N$ corresponds to the case where the \ac{cu} is not further divided.
Then, the standard defines the horizontal and vertical division into two \acs{pu}, corresponding to the $\frac{N}{2}\times N$ and $N \times \frac{N}{2}$ cases, respectively.
Figure~\ref{Figure2:InterPuPartitions}~(a) shows the symmetric partitions.
The $N \times N$ corresponds to the case where the \ac{cu} is not further divided.
Then, the standard defines the horizontal and vertical division into two \acs{pu}, corresponding to the $\frac{N}{2}\times N$ and $N \times \frac{N}{2}$ cases, respectively.
Finally, the $\frac{N}{2} \times \frac{N}{2}$ case results in four \acs{pu}.
The partitions shown in Figure~\ref{Figure2:InterPuPartitions}~(b) correspond to the asymmetric partitioning, which is one of the main improvements introduced in \ac{hevc}, as it allows higher flexibility in motion estimation, leading to higher compression efficiency~\cite{HEVCBlock2012}.
The partitions shown in Figure~\ref{Figure2:InterPuPartitions}~(b) correspond to the asymmetric partitioning, which is one of the main improvements introduced in \ac{hevc}, as it allows higher flexibility in motion estimation, leading to higher compression efficiency~\cite{HEVCBlock2012}.
These partitions types are only allowed for \acs{cu} with $16\times16$ pixels or larger.
\subsubsection{Merge mode}
@@ -583,23 +583,23 @@ These partitions types are only allowed for \acs{cu} with $16\times16$ pixels or
\end{figure}
The Merge mode was introduced in \ac{hevc} as a new technique to derive motion information from spatial and temporal neighbouring blocks~\cite{MotionCodingHEVC2013}. This technique extends the concepts of the Direct and Skip modes used in \ac{h264}~\cite{H264OverviewJul2003}, using a more sophisticated approach with two key differences.
Firstly, the Merge Mode allows for more \ac{mv} candidates than the Skip in \ac{h264}.
Thus, an index is used to identify one out of several \ac{mv} candidates.
Firstly, the Merge Mode allows for more \ac{mv} candidates than the Skip in \ac{h264}.
Thus, an index is used to identify one out of several \ac{mv} candidates.
Secondly, the reference picture index is explicitly identified.
This results in improved flexibility, compared with the predefined values used in \ac{h264}.
The \ac{mv} candidates allowed in the Merge Mode\cite{MotionCodingHEVC2013} are illustrated in Figure \ref{Figure2:MVCandidates}.
This results in improved flexibility, compared with the predefined values used in \ac{h264}.
The \ac{mv} candidates allowed in the Merge Mode\cite{MotionCodingHEVC2013} are illustrated in Figure \ref{Figure2:MVCandidates}.
The figure shows the positions of the spatial \ac{mv} candidates ($A_0$, $A_1$, $B_0$, $B_1$, $B_2$).
Moreover, it also uses a temporal \ac{mv} candidate, derived from the co-located position on the temporally adjacent frame at positions $T_0$ or $T_1$. From the \ac{mv} candidate list the unavailable candidates are not considered, and duplicated candidates (i.e. with the same motion information) are removed.
The Merge Mode uses a $C-1$ spatial \ac{mv} candidates, selected from the aforementioned order, and one temporal \ac{mv} candidate to fulfil the total number of \acs{mv} ($C$). The value of $C$ is defined at the slice header and a flag controlling the use of temporal \ac{mv} candidates is encoded at the \ac{pps}-level.
For B-slices, additional \ac{mv} candidates are considered by choosing two existing candidates according to a predefined order for reference picture list 0 and list 1. A total number of twelve combinations are specified in \ac{hevc}, which are listed in~\cite{HEVCOverviewDez2012}. For P-slices, if the number of candidates is less than $C$, then zero \acs{mv} are included to fill the remaining entries.
The Merge Mode defines how to encode the \ac{mv} candidate and the residual information.
The Merge Mode defines how to encode the \ac{mv} candidate and the residual information.
In \ac{hevc}, the Skip mode, i.e., transmitting only the \ac{mv} without residual information, is treated as a special case of the Merge Mode. In this case only a skip flag is enabled, and the corresponding merge index is transmitted.
\subsubsection{Motion vector prediction}
The \ac{hevc} also improves the performance of the \ac{mv} coding for those cases where the Merge Mode is not used.
The \ac{hevc} also improves the performance of the \ac{mv} coding for those cases where the Merge Mode is not used.
\ac{hevc} always use a differential encoding approach for efficiently representing the motion information.
It considers the same \ac{mv} candidates of the Merge mode, illustrated in Figure~\ref{Figure2:MVCandidates}, to derive the \ac{mv} predictors from which the difference vector is obtained.
From these \ac{mv} candidates only the first two available candidates are selected based on the following order:
@@ -610,7 +610,7 @@ From these \ac{mv} candidates only the first two available candidates are select
\end{enumerate}
Therefore, the use of the temporal \ac{mv} candidate is limited to those cases where the spatial neighbours are not able to provide two valid candidates.
For \ac{mv} prediction only a much lower number of candidates is allowed, since the encoder can still send a \ac{mv} difference. Moreover, the encoder performs motion estimation for each candidate, which increases the encoding complexity for every possibility tested.
For \ac{mv} prediction only a much lower number of candidates is allowed, since the encoder can still send a \ac{mv} difference. Moreover, the encoder performs motion estimation for each candidate, which increases the encoding complexity for every possibility tested.
%For those cases where the reference index is not the same as the current \ac{pu} and the neighbouring one, a scaling version of the \ac{mv} is used.
%In the \ac{h264} standard, along different motion compensated coding techniques~\cite{H264OverviewJul2003}, the so-called P\_SKIP mode was introduced. In this mode, neither the quantised residual information, nor the \ac{mv} is transmitted. This mode is useful for large areas with no change or just constant motion, and it is only enabled for $16\times16$ blocks, \emph{i.e.}, the whole \ac{mb}. The \ac{mv} for the P\_SKIP mode is derived based on the surrounding neighbour MBs\cite{H264MotionVectorPrediction2003}. When no neighbours are available, or the left or top \acs{mv} are equal to zero, a zero \ac{mv} is used. Otherwise, the \ac{mv} is derived from the median value of the following three neighbours: left, above and above right, as shown in Figure \ref{SkipMergeMode}~(a) by $MV_A$, $MV_B$ and $MV_C$, respectively. Whenever the $MV_C$ is not available, the $MV_D$ is used instead.
@@ -634,7 +634,7 @@ The residual block is partitioned into multiple square blocks, referred to as \a
A 2D transform of the residual information is obtained by applying 1D transforms in both the horizontal and vertical directions. The core transform matrices are obtained from the \ac{dct} basis functions to define a integer matrix of $32\times32$ points. Sub-sampled versions of this matrix are used to derive the remaining transform matrix sizes down to $4\times4$.
A special case is applied for coding the residual of $4\times4$ intra coded block.
A special case is applied for coding the residual of $4\times4$ intra coded block.
In this case an alternative integer transform based on the \ac{dst} is used.
The motivation for \ac{dst} is related to the statistics of the residual signal of these blocks, which tend to present higher residue as the distance to the boundary used for prediction (i.e, top or left boundary) increases.
Experiments have shown that \ac{dst} provides up to 1\% of bitrate savings for the compression of $4\times4$ intra predicted blocks, without significantly increase the computational complexity \cite{DCTDSTComp2013}.
@@ -643,8 +643,8 @@ Experiments have shown that \ac{dst} provides up to 1\% of bitrate savings for
\glsreset{qp}
The \ac{hevc} follows the same principle introduced previous standards for quantisation.
After applying the transform operation to prediction residue transform a quantisation is applied to the resulting coefficients using a uniform-reconstruction quantiser scheme based on a \ac{qp}.
The \ac{hevc} follows the same principle introduced previous standards for quantisation.
After applying the transform operation to prediction residue transform a quantisation is applied to the resulting coefficients using a uniform-reconstruction quantiser scheme based on a \ac{qp}.
The standard supports \ac{qp} values ranging from 0 to 51, and an increase of 6 in the \ac{qp} corresponds to doubling the quantisation step. This procedure is the main responsible for the coding distortion as it performs many-to-one mapping, which cannot be reversed.
@@ -656,7 +656,7 @@ The standard supports \ac{qp} values ranging from 0 to 51, and an increase of 6
\glsreset{cabac}
\ac{hevc} only uses the \ac{cabac}, which is an arithmetic coding method that only uses binary symbols, considering different probability models for each symbol~\cite{H264Cabac2003}.
\ac{hevc} only uses the \ac{cabac}, which is an arithmetic coding method that only uses binary symbols, considering different probability models for each symbol~\cite{H264Cabac2003}.
Entropy coding efficiency is closely linked with the context model selected. Therefore, this was carefully designed in the \ac{hevc} standard, extending the functionality previously defined for \ac{h264}.
For example, the \ac{hevc} standard exploits the depth of the partition tree or residual transform tree in order to derive the context models for several syntax elements.
Regarding transform coefficient coding, \ac{cabac} uses a scanning method to firstly organise the coefficients and then encodes the position of the last non-zero transform coefficient.
@@ -676,9 +676,9 @@ The coefficient scanning is implicitly selected and always performed in $4\times
The \ac{hevc} standard uses two filtering procedures over the reconstructed pixels before writing them into the frame buffer, namely the \ac{dbf} and the \ac{sao} filter.
The main purpose of deblocking filter is to reduce blocking artefacts caused by the block-based coding tools.%, operating similarly to the deblocking filter of the \ac{h264} standard.
\ac{dbf} is applied to the samples that are spatially adjacent to the \ac{pu} or \ac{tu} boundaries, considering a $8\times8$ grid.
\ac{dbf} is applied to the samples that are spatially adjacent to the \ac{pu} or \ac{tu} boundaries, considering a $8\times8$ grid.
Such grid-based restriction reduces the computational complexity and improves parallel-processing
implementation which is one of the main goals of \ac{hevc}.
implementation which is one of the main goals of \ac{hevc}.
Three strength levels can be used by the \ac{dbf} depending on the coded block characteristics.
The \ac{sao} is a non-linear filter, that is adaptively applied to all samples of the image, after the deblocking filter. In its operation, the \ac{sao} filter modifies the samples