jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Rename files and folders

Browse Source
This commit is contained in:
jfmcarreira
2018-12-14 02:39:48 +00:00
parent 4aea167b89
commit 61dd330c24
20 changed files with 114 additions and 145 deletions
Download Patch File Download Diff File Expand all files Collapse all files
021_VideoCoding/VideoCodingReview.tex
+37 -41
View File
@@ -1,8 +1,4 @@
% Figure folders
\renewcommand{\FigsPath}{./021_VideoCoding/Figures}
% \chapter{\HIGHLIGHT{The \ac{hevc} Standard}}
\chapter{The High Efficiency Video Coding standard \VERSION{3}}
\label{Chapter:HEVCReview}
@@ -40,7 +36,7 @@ In the last stages of the chain, the compressed format is decoded and presented
% \begin{figure}[tb]
% \centering
% \input{\FigsPath/GeneralVideoSystem.tikz}
% \input{021_VideoCoding/Figures/GeneralVideoSystem.tikz}
% \caption{Block diagram of a generic video coding system.}
% \label{FIG2:GeneralVideoSystem}
% \end{figure}
@@ -54,7 +50,7 @@ This is the reason why the coding scheme is called hybrid, as it combines quiet
\begin{figure}[t]
\centering
\input{\FigsPath/HybridCodingBlockDiagramPrePost.tikz}
\input{021_VideoCoding/Figures/HybridCodingBlockDiagramPrePost.tikz}
\caption{Block diagram of a generic video coding system, including the inner blocks of a hybrid encoding and decoding scheme (T and Q corresponds to the transform and quatisation operations, respectively).}
\label{Figure2:HybridCodec}
\end{figure}
@@ -63,7 +59,7 @@ This is the reason why the coding scheme is called hybrid, as it combines quiet
In general, there are two kinds of redundancy in video signal: spatial and temporal.
On the one hand, the spatial redundancy is exploited through intra frame prediction, where the underlying mechanism is to find the best matching block for the current one, by searching the neighbouring region.
This is possible due to the existence of high correlation among neighbouring pixels within a frame.
This is possible due to the existence of high correlation among neighbouring pixels within a frame.
As a result, when intra frame prediction is used, a block of pixels is reconstructed from the neighbouring blocks previously processed.
On the other hand, inter frame prediction is designed to exploit the temporal redundancy that exists among consecutive frames, such as those temporally adjacent, e.g., the past frame or the future frame in regard to the current one.
Temporal redundancy is normally referred to as the correlation between two adjacent frames since they usually have quiet similar visual content, such as foreground objects and background.
@@ -90,20 +86,20 @@ coding syntax and semantics are standardised.
The constrains and functionalities are defined through profiles and levels, while the semantic meaning of syntax elements define the decoding process, such that every decoder conforming to the standard generates the same output when given a conforming stream.
Such delimitation of scope in the standard concede maximum freedom for developers and manufactures to develop proprietary optimisations, allowing to cope with various applications’ requirements (\emph{e.g.}, balancing the compression ratio and the coding complexity).
Figure~\ref{fig:HEVCBlockDiagram} illustrates the block diagram of the \ac{hevc} encoder. It follows the same overall hybrid architecture as its predecessors starting from H.261.
The encoder uses both intra- and inter-picture prediction to exploit spatial and temporal redundancies, respectively.
The encoder uses both intra- and inter-picture prediction to exploit spatial and temporal redundancies, respectively.
Subsequently, the prediction residue is transformed by linear spatial transform and the resulting coefficients are then quantised and entropy coded, further reducing the bitstream redundancy.
%The resulting bitstream is combined with control information, which signals the prediction chosen, motion vectors, filtering operations and bitstream headers.
\begin{figure}[tb]
\centering
\includegraphics[width=\columnwidth]{021_VideoCoding/Figures/HEVCBlockDiagram}
%\input{\FigsPath/HEVCBlockDiagram.tikz}
%\input{021_VideoCoding/Figures/HEVCBlockDiagram.tikz}
\caption{Block diagram of an HEVC standard encoder.}
\label{fig:HEVCBlockDiagram}
\end{figure}
However, there are some novelties introduced in the \ac{hevc} standard. As depicted in Figure~\ref{fig:HEVCBlockDiagram}, the encoding process follows a block-based approach, therefore each picture is partitioned into larger blocks, referred to as \acs{ctu}, with a maximum size of $64\times64$ pixels.
Subsequently, a dynamic quad-tree partitioning is used to divide the picture into \acs{cu}.
Subsequently, a dynamic quad-tree partitioning is used to divide the picture into \acs{cu}.
These units are introduced as the basic processing units for prediction, while for the transform and quantisation operations, transform units are used.
In order to generate the prediction residue, the \ac{hevc} may either use one of 34 intra-prediction modes or take advantage of symmetric and asymmetric partitions for motion estimation and inter-prediction.
The motion estimation is performed in the pictures listed in the reference picture set, which was introduced in the \ac{hevc} to increase the robustness to errors and data loss of reference picture management.
@@ -129,8 +125,8 @@ Similar to \ac{h264}~\cite{H264AVCStandard2003}, the \ac{hevc} standard defines
The \ac{nal} is used for stream partitioning and synchronization, which is useful to deal with lossy network environments~\cite{H264OverIPJul2003}.
Using this concept the video stream is composed of different \ac{nal} units. The \ac{nal} concept provides the ability to map the \ac{vcl} data that represents coded video slices (or frames) on various transport layers, such as \ac{rtp}, MPEG-2 Systems and MPEG DASH. The~\ac{nal} units are classified into two categories: (i) \ac{vcl} and (ii) non-\ac{vcl}, as they carry coded video or associated data, respectively.
The \ac{hevc} standard defines a two bytes header to describe the content of each \ac{nal} unit, which are transmitted using a fixed coding format length, i.e., without variable length codes.
This allow extra error robustness but increases the bit overhead.
The \ac{hevc} standard defines a two bytes header to describe the content of each \ac{nal} unit, which are transmitted using a fixed coding format length, i.e., without variable length codes.
This allow extra error robustness but increases the bit overhead.
Figure~\ref{Figure2:NalHeader} shows the structure of the \ac{hevc} \ac{nal} unit header. The first bit \textit{F}, referred to as \textit{forbiden\_bit}, is always zero in order to ensure that a start code can be detected across different protocol standards. This is followed by the \ac{nal} type, which identifies its content, enabling the decoder to either use it if required for a particular decoding purpose or discard it. Another syntax element introduced by the \ac{hevc} standard is the \textit{temporal\_id\_plus\_one} (TIDP), which defines the temporal layer where the \ac{nal} unit belongs. This allows implicit temporal scalability (layers ranging from 0 to 6) by immediately discarding the \ac{nal} unit if it belongs to a layer that is higher than the pre-defined one. In the header, six bits are reserved for future extensions, preventing the creation of extra \ac{nal} units for scalable and multiview extensions, as done in \ac{h264}.
\begin{figure}[tb]
@@ -155,9 +151,9 @@ In this approach, such set of parameters is multiplexed in the bitstream and rep
The \ac{hevc} includes three sets of parameters, \ac{sps}, \ac{pps} and the newly introduced \ac{vps}\cite{HEVCHighLevelReferenceDez2012}.
% \ac{sps} include sequence related information, that include the decoder operation point, flags to enable or disable certain tools within a given profile. \Gls{pps} include picture related information, that may change for every picture, e.g., \ac{qp} and tilling information.
The new set of parameters \ac{vps} contains information related to the different layers of the video signal, avoiding duplications like in the \ac{h264} and its extensions. \Gls{vps} aims to provide signalling for every layer, guaranteeing an extendible standard, capable of supporting multiple layers.
Moreover, \ac{vps} also carries information for session negotiation (\emph{e.g.} profile, level and tier).
The \ac{sps} contains information related to the whole sequence (\emph{i.e.}, should affect all coded slices) and related to the decoder operation point, as well as flags to control optional tools and scalability.
The new set of parameters \ac{vps} contains information related to the different layers of the video signal, avoiding duplications like in the \ac{h264} and its extensions. \Gls{vps} aims to provide signalling for every layer, guaranteeing an extendible standard, capable of supporting multiple layers.
Moreover, \ac{vps} also carries information for session negotiation (\emph{e.g.} profile, level and tier).
The \ac{sps} contains information related to the whole sequence (\emph{i.e.}, should affect all coded slices) and related to the decoder operation point, as well as flags to control optional tools and scalability.
The \ac{pps} is responsible to carry information that may change for every picture, such as, initial \ac{qp}, flags for picture related tools and tilling information.
@@ -187,14 +183,14 @@ Moreover, introducing random-access points into the bitstream increases its erro
BLA &
Broken link access &
Leading pictures cannot be decoded \\
\midrule \midrule
\multicolumn{3}{c}{\textbf{Leading pictures}} \\
\midrule
RADL & \multicolumn{2}{l}{Random access decodable picture}\\
\midrule
RASL & \multicolumn{2}{l}{Random access skipped picture}\\
\midrule\midrule
\multicolumn{3}{c}{\textbf{Temporal sub-layer access pictures}} \\
\midrule
@@ -245,7 +241,7 @@ Due to the introduction of more flexible random-access points using \ac{cra} pic
\begin{figure}[tb]
\centering
\input{\FigsPath/PictureTypes_Leading_Trailing.tikz}
\input{021_VideoCoding/Figures/PictureTypes_Leading_Trailing.tikz}
\caption{Leading and trailing pictures associated with the random-access point $I_{1}$ (letters corresponds to the coding type and the numbers to the coding order).}
\label{Figure2:LeadingTralingPics}
\end{figure}
@@ -253,7 +249,7 @@ Due to the introduction of more flexible random-access points using \ac{cra} pic
There are two types of leading pictures, as shown in Table~\ref{Table2:PictureTypes}.
The \ac{radl} pictures, which can be correctly decoded, since they only depend on the associated \ac{rap} picture and do not have any prior dependencies to trailing pictures associated with the previous \ac{rap} frame.
On the contrary, the \ac{rasl} pictures may have dependencies to prior trailing pictures.
When a random-access is performed at the associated \ac{rap} frame, these frames cannot be correctly decoded, therefore they have to be skipped.
When a random-access is performed at the associated \ac{rap} frame, these frames cannot be correctly decoded, therefore they have to be skipped.
In the example of Figure~\ref{Figure2:LeadingTralingPics} the leading pictures correspond to \ac{rasl} pictures.
\subsubsection{Temporal sub-layer switching pictures}
@@ -262,19 +258,19 @@ In the example of Figure~\ref{Figure2:LeadingTralingPics} the leading pictures c
A \ac{tsa} picture is a trailing picture that marks a temporal layer switching point.
When decoding a sub-set of temporal layers, if a \ac{tsa} picture is found in the temporal layer just above the maximum temporal layer currently being decoded, it is possible to start decoding any number of additional temporal layers.
For example, frame $P_6$ in Figure~\ref{Figure2:TSAExample} is considered a \ac{tsa} picture since it only depends on a frame belonging to temporal layer 0 for prediction, as well as, any subsequent frame predicted from the \ac{tsa} frame $P_6$.
In this example, if the decoder is only decoding the temporal layer 0, after decoding $P_6$ it can also decode layer 1, or decode all the available three layers.
In this example, if the decoder is only decoding the temporal layer 0, after decoding $P_6$ it can also decode layer 1, or decode all the available three layers.
Although these frames allow higher flexibility in terms of layer switching, they severely constrain the prediction of frames following the \ac{tsa} picture.
\begin{figure}[tb]
\centering
% \includegraphics[width=0.75\columnwidth]{\FigsPath/HEVC/Pictures_TSA}
\input{\FigsPath/PictureTypes_TSA.tikz}
% \includegraphics[width=0.75\columnwidth]{021_VideoCoding/Figures/HEVC/Pictures_TSA}
\input{021_VideoCoding/Figures/PictureTypes_TSA.tikz}
\caption{Temporal layers access pictures example with three layers ($T_0$ to $T_2$).}
\label{Figure2:TSAExample}
\end{figure}
To reduce the constrains in frame prediction due to layer switching, \ac{hevc} introduces the \ac{stsa} picture.
These pictures have similar purpose as the \ac{tsa} frames, but they only guarantee that frames in the same temporal layer as the \ac{stsa} frame are able to be correctly reconstructed.
These pictures have similar purpose as the \ac{tsa} frames, but they only guarantee that frames in the same temporal layer as the \ac{stsa} frame are able to be correctly reconstructed.
%This is guaranteed by avoiding any dependencies from frames preceding the \ac{stsa} to be used as reference in its temporal layer.
This is guaranteed by not using frames preceding the \ac{stsa} as reference in its temporal layer.
One example of a \ac{stsa} frame is also shown in Figure~\ref{Figure2:TSAExample} in frame $P_2$.
@@ -285,7 +281,7 @@ Summarising, \ac{stsa} frames can be used to switch to a particular layer, as \a
\subsection{Picture partitioning}
The high-level segmentation of a picture in \ac{hevc} is achieved by using four different approaches associated to different data structures: regular slices, dependent slices, tiles and \ac{wpp}.
The high-level segmentation of a picture in \ac{hevc} is achieved by using four different approaches associated to different data structures: regular slices, dependent slices, tiles and \ac{wpp}.
Picture partitioning normally serves one or more of the following three purposes:
\begin{description}
\item[Error robustness:] partitioning the picture into smaller self-contained units in
@@ -313,8 +309,8 @@ A new data partition has been included in the \ac{hevc} standard, referred to as
\begin{figure}[tb]
\centering
\begin{tabular}{x{.45\columnwidth}x{.45\columnwidth}}
\input{\FigsPath/Tiles_Slices_Example_1.tikz} &
\input{\FigsPath/Tiles_Slices_Example_2.tikz} \\
\input{021_VideoCoding/Figures/Tiles_Slices_Example_1.tikz} &
\input{021_VideoCoding/Figures/Tiles_Slices_Example_2.tikz} \\
(a) & (b) \\
\end{tabular}
\caption{Examples of the interaction between tiles and slices.}
@@ -338,7 +334,7 @@ Each square corresponds to a \ac{ctu} and the numbers correspond to the time ins
\begin{figure}[tb]
\centering
\input{\FigsPath/WPP.tikz}
\input{021_VideoCoding/Figures/WPP.tikz}
\caption{Illustration of the wavefront parallel processing (numbers corresponds to the time instant each \ac{ctu} is processed).}
\label{Figure2:WPP}
\end{figure}
@@ -350,7 +346,7 @@ Each square corresponds to a \ac{ctu} and the numbers correspond to the time ins
\label{Section2:CodingStructures}
Despite the fact that the \ac{hevc} standard follows a traditional hybrid video coding architecture, it introduces significant changes in the data structures when compared to the previous standards (e.g., \ac{h264}).
In \ac{h264} each frame is divided into units so-called \acs{mb}, which are composed by a $16\times16$ block of luma samples and two $8\times8$ blocks of chroma samples, assuming video data in YUV 4:2:0 chroma sub-sampling.
In \ac{h264} each frame is divided into units so-called \acs{mb}, which are composed by a $16\times16$ block of luma samples and two $8\times8$ blocks of chroma samples, assuming video data in YUV 4:2:0 chroma sub-sampling.
The \ac{mb} is considered the basic processing unit in the \ac{h264}, and the respective coding mode defines whether the corresponding pixels are coded as intra or intra prediction.
Each \ac{mb} could also be partitioned into sub-blocks, which should follow the same prediction mode.
@@ -358,18 +354,18 @@ Each \ac{mb} could also be partitioned into sub-blocks, which should follow the
The basic data structures in \ac{hevc} are quite different from \ac{h264}. Each picture is divided into blocks named as \ac{ctu}~\cite{BlockPartHEVC2012}. The maximum \ac{ctu} size is an encoder configuration parameter that is signalled to the decoder, and should be one of the following: $64\times64$, $32\times32$ and $16\times16$ pixels. The use of data structures with larger size in \ac{hevc} was adopted to achieve higher coding efficiency especially in high resolution video. Each \ac{ctu} can be recursively partitioned into four smaller units, referred to as \acs{cu}, until it reaches the minimum unit size, which cannot be smaller than $8\times8$ pixels.
Similar to the maximum \ac{ctu} size, the minimum size is also defined at the encoder.
For instance, in case of homogeneous regions, large \acs{cu} can be used to represent such regions by using a smaller number of symbols than in the case of using several small units.
Figure~\ref{Figure2:HEVCCodingStructure} illustrates an example of a \ac{ctu} partitioning into \acs{cu}.
Figure~\ref{Figure2:HEVCCodingStructure} illustrates an example of a \ac{ctu} partitioning into \acs{cu}.
The partitioning illustrated in the figure leads to the quad-tree structure represented on the left side with the corresponding coding order shown on the right side.
Data structures in \ac{hevc} provide a significant improvement in comparison with previous standards, by eliminating the distinction between \ac{mb} and sub-\ac{mb} and using only the \ac{cu} concept. This allows for the multilevel hierarchical quadtree structure to be specified in a simple and elegant way, with size-independent syntax representation.
\begin{figure}[t]
\centering
\input{\FigsPath/BlockPartitionsWithTree.tikz}
\input{021_VideoCoding/Figures/BlockPartitionsWithTree.tikz}
\caption{Representation of a CTU ($64\times64$) and its partitioning into CUs and PUs (left); corresponding quad-tree with coding order (right).}
\label{Figure2:HEVCCodingStructure}
\end{figure}
The \ac{cu} is the basic processing unit in \ac{hevc}.
The \ac{cu} is the basic processing unit in \ac{hevc}.
It is further partitioned into two types of units, specifically \ac{pu} and \ac{tu}, which are data blocks used for prediction and transform operations, respectively.
The decision whether to use intra-frame or inter-frame prediction modes is taken at the \ac{cu} level. However, depending on the partitioning of the \ac{cu} into \acs{pu}, different intra or inter coding modes can be used for each \ac{pu}.
@@ -419,8 +415,8 @@ Directional intra prediction was first introduced in the \ac{h264} standard~\cit
Directional prediction projects the reconstructed samples in the block neighbourhood (see reference pixels in Figure~\ref{Figure2:IntraModesHEVC}) along a specific direction. The \ac{hevc} increases the number of possible directions from 8 (in \ac{h264}) to 33~\cite{IntraCodingHEVC2012}. The direction of each projection is illustrated in the left-side of Figure~\ref{Figure2:IntraModesHEVC} by the directional modes 2 to 34.
%In the figure it is possible to notice that directional modes 2 to 10 only project the pixels from the left neighbouring border in a horizontal direction, modes 10 to 25 use both the left and top neighbouring pixels in a diagonal direction and modes 26 to 34 project the top neighbours in vertical direction.
As shown in Figure~\ref{Figure2:IntraModesHEVC}, the angular modes are designed to provide a dense coverage near the horizontal (i.e., mode 10) and vertical (i.e., mode 26) directions.
Moreover, the angular directions are coarser as it gets closer to the diagonals.
As shown in Figure~\ref{Figure2:IntraModesHEVC}, the angular modes are designed to provide a dense coverage near the horizontal (i.e., mode 10) and vertical (i.e., mode 26) directions.
Moreover, the angular directions are coarser as it gets closer to the diagonals.
This reflects the observed statistical prevalence of the angles and the prediction efficiency.
To improve the efficiency of angular prediction, when the reference samples need to be projected, a bilinear interpolation is used from the two closest pixels using a $1/32$ accuracy. The dashed arrows on the right of Figure~\ref{Figure2:IntraModesHEVC} correspond to a projection of interpolated pixels.
@@ -498,7 +494,7 @@ Another improvement in \ac{hevc} is related with the reference picture manageme
Decoded frames in the \ac{dpb} are grouped in one of the following categories: (i) \emph{short-term reference}, (ii) \emph{long-term reference} and (iii) \emph{unused for reference}.
Once a frame is marked as \emph{unused for reference} it is no longer used for motion compensation and it will be discarded from the \ac{dpb} after being displayed.
In \ac{h264} a sliding window approach is used to implicitly manage the reference picture list.
This process discards a given frame whenever the \ac{dpb} has stored the maximum number of reference frames (which is defined and controlled by the encoder).
This process discards a given frame whenever the \ac{dpb} has stored the maximum number of reference frames (which is defined and controlled by the encoder).
Moreover, \ac{h264} uses an explicitly \ac{mmco} to send commands to the decoder and change the normal reference picture management process.
In the \ac{hevc} standard the status of the \ac{dpb} is encoded for every slice, instead of the implicit managing used in the past.
@@ -553,8 +549,8 @@ These partitions types are only allowed for \acs{cu} with $16\times16$ pixels or
\begin{figure}[tb]
\centering
\begin{tabular}{x{.8\columnwidth}x{.1\columnwidth}}
\input{\FigsPath/PUSymetric.tikz} & (a)\\
\input{\FigsPath/PUAsymetric.tikz} & (b)\\
\input{021_VideoCoding/Figures/PUSymetric.tikz} & (a)\\
\input{021_VideoCoding/Figures/PUAsymetric.tikz} & (b)\\
\end{tabular}
\caption{Partition types of an inter-coded CU into PUs; in (b) the size (and position between parentheses) corresponds to the smaller partition~\cite{HEVCBlock2012}.}
\label{Figure2:InterPuPartitions}
@@ -576,8 +572,8 @@ For B-slices, additional \ac{mv} candidates are considered by choosing two exist
\begin{figure}[tb]
\centering
\input{\FigsPath/MergeModeHEVC.tikz}
%\input{\FigsPath/HEVC/MergeModeHEVCTwoFrames.tikz}
\input{021_VideoCoding/Figures/MergeModeHEVC.tikz}
%\input{021_VideoCoding/Figures/HEVC/MergeModeHEVCTwoFrames.tikz}
\caption{Illustration of the MV prediction candidates in the AMVP and Merge Mode mode.}
\label{Figure2:MVCandidates}
\end{figure}
@@ -626,7 +622,7 @@ Simulation studies have shown that \ac{dst} provides up to 1\% of bitrate savin
The \ac{hevc} follows the same principle as previous standards in terms of quantisation.
After applying the transform operation to the prediction residue, a quantisation is applied to the resulting coefficients using a uniform-reconstruction quantiser scheme based on a \ac{qp}.
The standard supports \ac{qp} values ranging from 0 to 51, where an increase of 6 in the \ac{qp} corresponds to doubling the quantisation step.
The standard supports \ac{qp} values ranging from 0 to 51, where an increase of 6 in the \ac{qp} corresponds to doubling the quantisation step.
This procedure is the main cause for the coding distortion, as it performs a many-to-one mapping, which cannot be reversed.
@@ -659,8 +655,8 @@ Such grid-based restriction reduces the computational complexity and facilitates
Three strength levels can be used by the \ac{dbf}, depending on the coded block characteristics.
The \ac{sao} is a non-linear filter, that is adaptively applied to all samples of the image, after the deblocking filter. In its operation, the \ac{sao} filter modifies the samples
by adding an offset value, extracted from lookup tables transmitted by the encoder.
For each \ac{ctu}, the encoder decides whether to apply the \ac{sao} filter or not, and if used, one of the two filter types is applied, namely the band offset or the edge offset.
by adding an offset value, extracted from lookup tables transmitted by the encoder.
For each \ac{ctu}, the encoder decides whether to apply the \ac{sao} filter or not, and if used, one of the two filter types is applied, namely the band offset or the edge offset.
In the band offset mode, the added offset value depends on the sample amplitude, while the edge offset uses the gradient to classify and derive the offset value.