diff --git a/000_Preamble/abstract.tex b/000_Preamble/abstract.tex index 5046fe6..e4b96c4 100644 --- a/000_Preamble/abstract.tex +++ b/000_Preamble/abstract.tex @@ -6,24 +6,44 @@ \vspace{-5mm} -This thesis investigates the problem of robust coding and Error Concealment (EC) in \HIGHLIGHT{highly efficient video coders}. -After a review of the current state-of-art, a simulation study about error robustness, revealed that the High Efficiency Video Coding (HEVC) has weak protection against transmission errors with significant impact on video quality degradation. -Based on this evidence, the first contribution of this work is a new method to reduce the temporal dependencies between \acs{mv}, by improving the decoded video quality without compromising the compression efficiency. +%This thesis investigates the problem of robust coding and Error Concealment (EC) in High Efficiency Video Coding (HEVC). +%After a review of the current state of the art, a simulation study about error robustness, revealed that the HEVC has weak protection against transmission errors with significant impact on video quality degradation. +%Based on this evidence, the first contribution of this work is a new method to reduce the temporal dependencies between Motion Vectors (MVs), by improving the decoded video quality without compromising the compression efficiency. +% +%The second contribution of this thesis is a two-stage approach for reducing the mismatch of temporal predictions in case of video streams received with errors or lost data. +%At the encoder stage, the reference pictures are dynamically distributed based on a constrained Lagrangian rate-distortion optimization to reduce the number of predictions from a single reference. +%At the streaming stage, a prioritization algorithm, based on spatial dependencies, selects a reduced set of MVs to be transmitted, as side information, to reduce mismatched MV predictions at the decoder. +% +%The problem of EC-aware video coding is also investigated to enhance the overall error robustness. +%A new approach based on scalable coding and optimally EC selection is proposed, where the optimal EC modes are found by simulating transmission losses, followed by a saliency-weighted optimisation. +%Moreover, EC residual information is encoded using a rate-controlled enhancement layer. +%Both are transmitted to the decoder to be used in case of data loss. +% +%Finally, an adaptive error resilience scheme is proposed to dynamically predict the video stream that achieves the highest decoded quality for a particular loss case. +%A neural network selects among the various video streams, encoded with different levels of compression efficiency and error protection, based on information from the video signal, the coded stream and the transmission network. +% +%Overall, the new robust video coding methods investigated in this thesis yield consistent quality gains in comparison with other existing methods and also the ones implemented in the HEVC reference software. +%Furthermore, the trade-off between coding efficiency and error robustness is also better in the proposed methods. + +%\vspace{2mm} +%\textbf{Keywords:} 3D video, frame loss error concealment, subjective quality assessment. + + +This thesis investigates the problem of robust coding and error concealment in High Efficiency Video Coding (HEVC). +After a review of the current state of the art, a simulation study about error robustness, revealed that the HEVC has weak protection against transmission errors with significant impact on video quality degradation. +Based on this evidence, the first contribution of this work is a new method to reduce the temporal dependencies between motion vectors, by improving the decoded video quality without compromising the compression efficiency. The second contribution of this thesis is a two-stage approach for reducing the mismatch of temporal predictions in case of video streams received with errors or lost data. -At the encoder stage, the reference pictures are dynamically distributed based on a constrained Lagrangian rate-distortion optimization to reduce the number of predictions from a single reference. -At the streaming stage, a prioritization algorithm, based on spatial dependencies, selects a reduced set of MVs to be transmitted, as side information, to reduce mismatched MV predictions at the decoder. +At the encoding stage, the reference pictures are dynamically distributed based on a constrained Lagrangian rate-distortion optimization to reduce the number of predictions from a single reference. +At the streaming stage, a prioritization algorithm, based on spatial dependencies, selects a reduced set of motion vectors to be transmitted, as side information, to reduce mismatched motion predictions at the decoder. -The problem of EC-aware video coding is also investigated to enhance the overall error robustness. -A new approach based on scalable coding and optimally EC selection is proposed, where the optimal EC modes are found by simulating transmission losses, followed by a saliency-weighted optimisation. -Moreover, EC residual information is encoded using a rate-controlled enhancement layer. +The problem of error concealment-aware video coding is also investigated to enhance the overall error robustness. +A new approach based on scalable coding and optimally error concealment selection is proposed, where the optimal error concealment modes are found by simulating transmission losses, followed by a saliency-weighted optimisation. +Moreover, recovery residual information is encoded using a rate-controlled enhancement layer. Both are transmitted to the decoder to be used in case of data loss. Finally, an adaptive error resilience scheme is proposed to dynamically predict the video stream that achieves the highest decoded quality for a particular loss case. -A pre-trained neural network selects among the various video streams, encoded with different levels of compression efficiency and error protection, based on information from the video signal, the coded stream and the transmission network. +A neural network selects among the various video streams, encoded with different levels of compression efficiency and error protection, based on information from the video signal, the coded stream and the transmission network. Overall, the new robust video coding methods investigated in this thesis yield consistent quality gains in comparison with other existing methods and also the ones implemented in the HEVC reference software. Furthermore, the trade-off between coding efficiency and error robustness is also better in the proposed methods. - -%\vspace{2mm} -%\textbf{Keywords:} 3D video, frame loss error concealment, subjective quality assessment. \ No newline at end of file diff --git a/000_Preamble/thanks.tex b/000_Preamble/thanks.tex index 481f87b..32c7132 100644 --- a/000_Preamble/thanks.tex +++ b/000_Preamble/thanks.tex @@ -1,13 +1,13 @@ \chapter*{\glsknowledgmentsname} \addcontentsline{toc}{chapter}{\glsknowledgmentsname} -Firstly, I would like to express my gratitude to my advisers, Dr. Pedro Ant\'{o}nio Amado Assun\c{c}\~{a}o, Dr. S\'{e}rgio Manuel Maciel de Faria, Dr Erhan Ekmekcioglu and Dr. Ahmet Kondoz for the tireless support of my PhD studies and related research, for their motivation, patience, and immense knowledge. I am thankful for their guidance and availability that were essential for the proper conduct of this research work. I would also like to acknowledge Dr. Varuna de Silva for the initial support on this research~work. +Firstly, I would like to express my gratitude to my advisers, Prof. Pedro Ant\'{o}nio Amado Assun\c{c}\~{a}o, Prof. S\'{e}rgio Manuel Maciel de Faria, Dr Erhan Ekmekcioglu and Prof. Ahmet Kondoz for the tireless support of my PhD studies and related research, for their motivation, patience, and immense knowledge. I am thankful for their guidance and availability that were essential for the proper conduct of this research work. I would also like to acknowledge Dr. Varuna de Silva for the initial support on this research~work. I would like to acknowledge Funda\c{c}\~{a}o para a Ci\^{e}ncia e a Tecnologia (FCT), which provided the financial support for this PhD, under the grant SFRH/BD/86368/2012. -I am also grateful to the Instituto de Telecomunica\c{c}\~{o}es and to the Institute for Digital Technologies, which provided the computer resources and laboratory facilities with excellent conditions for my research work. +I am also grateful to the Instituto de Telecomunica\c{c}\~{o}es and to the Institute for Digital Technologies of Loughborough University London, which provided the computer resources and laboratory facilities with excellent conditions for my research work. -I would like to thank my research colleagues, both in Portugal and United Kingdom, for the friendship and the wonderful work environment, for sharing new ideas, and also for the moments outside the laboratory. +I would like to thank my research colleagues, both in Portugal and the United Kingdom, for the friendship and the wonderful work environment, for sharing new ideas, and also for the moments outside the laboratory. I also thank all my friends, that accompanied me during this journey, for the distracting and social moments. I must thank all communities around the world that contribute for the development of many free and open-source tools used in this research work, namely the GNU/Linux, KDE software, Ubuntu and Gentoo communities. I also acknowledge the help of Lu\'{i}s Lucas and Jo\~{a}o Santos in the Calyp software, Remoterun scripts and general computer management, which improved our developing capabilities and ease the experimental evaluations. diff --git a/010_Introduction/Introduction.tex b/010_Introduction/Introduction.tex index d1a66b9..732b09b 100644 --- a/010_Introduction/Introduction.tex +++ b/010_Introduction/Introduction.tex @@ -20,17 +20,16 @@ To cope with such demanding requirements a new coding standard was developed by The \ac{hevc} is currently the main video coding framework to extend current services towards higher resolution formats delivered with better compression efficiency. Similar to its predecessor, \ac{hevc} is a block-based hybrid video codec using predictive, transform and entropy coding. The coding flexibility and efficiency of \ac{hevc} results from the use of new coding tools and block structures, which enables new block partitions\cite{HEVCBlock2012}, improved prediction modes \cite{IntraCodingHEVC2012,AMPCodingHEVCJun2012} and new high-level features~\cite{HEVCHighLevelReferenceDez2012}, such as explicit reference picture management and new parameter sets. -However, it is known that higher coding efficiency leads to lower robustness in the presence of transmission errors and data loss. -Therefore, new coding tools of \ac{hevc} not only bring high compression efficiency~\cite{HEVCComparisonDec2012}, but also some other disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and inherent reduction of error robustness~\cite{StudyHEVCErrorsDez2012}. -Earlier studies about the error robustness characteristics of \ac{hevc} have shown that, in general, the use of high efficiency coding tools results in poor error resilience performance, in comparison with previous standards~\cite{QoEHEVCMay2014,SubjectiveEvalSep2013}. +%However, it is known that higher coding efficiency leads to lower robustness in the presence of transmission errors and data loss. +However, new coding tools of \ac{hevc} not only bring high compression efficiency~\cite{HEVCComparisonDec2012}, but also some other disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and inherent reduction of error robustness~\cite{StudyHEVCErrorsDez2012}. +Earlier studies about the error robustness characteristics of \ac{hevc} have shown that, in general, the use of high efficiency coding tools results in poor error resilience performance (score \emph{slightly annoying} in the \ac{mos} scale), in comparison with previous standards~\cite{QoEHEVCMay2014,SubjectiveEvalSep2013}. Thus, when \ac{hevc} streams incur in transmission losses, this leads to significant degradation of both objective and subjective quality. This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes of \ac{hevc}, which assume \mbox{error-free} transmission~\cite{ROPE2007}. A possible approach to encode robust video streams is to increase the error resilience of \ac{hevc} by including \mbox{error-aware} mechanisms in the coding process. As a result, the effect of error propagation, due to prediction mismatch at the decoding side, will be limited. -\glsunset{rtp} -\glsunset{ts} -The widespread use of \ac{hevc} in broadcast and multimedia delivery services is based on transport technologies such as \ac{rtp}~\cite{HEVCStreamMay2012}, \ac{ts}~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016}. + +The widespread use of \ac{hevc} in broadcast and multimedia delivery services is based on transport technologies such as \ac{rtp}~\cite{HEVCStreamMay2012}, \ac{ts}~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as \ac{dash}~\cite{HEVCHTTPStreamAug2016}. Recent developments also enable combinations of different technologies, e.g, \mbox{HTTP-based} streaming and \ac{ts}, to increase the flexibility of over-the-top video services\cite{Broadcast2017}. For content delivery over heterogeneous networks, the \ac{mmt}~\cite{ATSCMMT2016} is another possible technology to be used with the \ac{hevc} standard. Besides the transport technologies, the overall error robustness of video transmission may also benefit from the increasing flexibility of multimedia systems to cope with heterogeneous networks, by using auxiliary streams that allow the video decoder to increase the quality of the recovered lost slices~\cite{ErrorResilienceHeterogenousNetworks2016,HeterogeneousStreaming2016}. Moreover, error resilience techniques may also take advantage of the multi-stream based approaches, where each stream allows for different levels of error robustness~\cite{HEVCHTTPStreamAug2016,PerfMediaTransport2017}. @@ -55,18 +54,18 @@ Thus, this work investigates novel techniques to increase the video quality of \ The following objectives were pursued in this work: \begin{enumerate} \item -\textbf{study and evaluation of error robustness in highly compressed video:} +\textbf{Study and evaluation of error robustness in highly compressed video:} The objective of this research study is to evaluate the robustness against transmission errors of highly efficiently coded video. -The study and its conclusions aim at finding relevant research directions to develop new methods for increasing the error robust of coded streams. +The study and its conclusions aim at finding relevant research directions to develop new methods for increasing the error robustness of coded streams. The results and conclusions of this study were published in conference paper (\ref{Conf:ErrorResilience}) and in journal paper (\ref{Journal:TwoStage}) (see Appendix~\ref{Appen:Papers}). \item -\textbf{robust video coding:} +\textbf{Robust video coding:} At the encoder-side, novel techniques to extend the existing error robustness of highly predictive video coding techniques are investigated in order to increase the received video quality over error-prone networks. This research topic resulted in a new method to reduce the error propagation in motion vector coding algorithms, which was published in conference paper (\ref{Conf:DynamicTMVP}). Moreover, a novel two-stage approach was devised to improve the error robustness of highly efficiently coded video at the encoder and streaming levels, resulting in two conference publications (\ref{Conf:MotionVectorRedundancies} and~\ref{Conf:Chkb}) and a journal publication (\ref{Journal:TwoStage}). -\item \textbf{robust video decoding:} error concealment techniques are used to increase the error robustness of video decoding and improve the perceived video quality. +\item \textbf{Robust video decoding:} error concealment techniques are used to increase the error robustness of video decoding and improve the perceived video quality. The paradigm of \ac{ec} is evaluated at the encoder-side, as part of the encoding process, in order to optimise the reconstruction process at the decoder-side. This research resulted in a novel error concealment-aware technique to optimise the error concealment process at the encoder-side, which was proposed for intra-frame only in a conference paper (\ref{Conf:ConcAware}) and for generic robust video transmission in the journal (\ref{Journal:ConcAware}). %\item \textbf{jointly error resilience and concealment:} the ultimate goal of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet higher quality demands for different video contents characteristics and network conditions. @@ -80,13 +79,13 @@ The paradigm of \ac{ec} is evaluated at the encoder-side, as part of the encodin This thesis is organised as follows. The current chapter introduces the context, motivation and objectives of this thesis, as well as the original contributions. -The first two chapters review the \mbox{state-of-the-art} concepts and previous work related with this research. Chapter~\ref{Chapter:HEVCReview} presents an overview of the \ac{hevc} standard, focusing on the concepts explored in this work, e.g., high-level features and temporal predictions. Chapter~\ref{Chapter:ErrorResilienceReview} reviews the most important work dealing with robust video transmission also providing relevant background knowledge to this work. +The first two chapters review the \mbox{state-of-the-art} concepts and previous work related to this research. Chapter~\ref{Chapter:HEVCReview} presents an overview of the \ac{hevc} standard, focusing on the concepts explored in this work, e.g., high-level features and temporal predictions. Chapter~\ref{Chapter:ErrorResilienceReview} reviews the most important work dealing with robust video transmission also providing relevant background knowledge to this work. Chapter~\ref{Chapter:HEVCErrorResilience} firstly presents a study about the error robustness of high efficiently coded video, using the \ac{hevc} as the underlying framework for various coding configurations. The study presents the relevant tools that mostly affect the error robustness. Secondly, a new method is proposed to decrease the vulnerability of temporal motion prediction and decrease the error propagation, leading to higher video quality under error-prone conditions. -Chapter~\ref{Chapter:ErrorResilienceTechniques} proposes a novel error resilience approach for high efficiently video codecs. +Chapter~\ref{Chapter:ErrorResilienceTechniques} proposes a novel error resilience approach for high efficient video codecs. A new method to distribute the use of reference frames is proposed to reduce the error propagation and improve the \ac{ec} performance. Moreover, to reduce the effect of mismatched motion predictions (shown in Chapter~\ref{Chapter:HEVCErrorResilience}), a method based on selective motion vector redundancies is proposed. This method is able to improve the decoded video quality using a small overhead. @@ -106,7 +105,7 @@ At the encoder-side various video streams are generated with different levels of Then, a pre-training \ac{nn} is used to predict which video stream is expected to achieve the highest decoded video quality for a particular scenario. The selection is based on different input parameters, which include information from the video signal, the coded stream and the transmission network. Finally, Chapter~\ref{Chapter:Conclusions} concludes this dissertation and presents some suggestions for future work. -Appendix~\ref{Appen:Sequences} illustrates the original test signals used in simulations throughout the thesis. This includes a partial set of the \ac{hevc} test material and other external test material. +Appendix~\ref{Appen:Sequences} illustrates the original test signals used in simulations throughout the thesis. This includes a partial set of the \ac{hevc} test materials, as well as, other external test sequences. Appendix~\ref{Appen:Papers} presents a summary of the published papers, describing the contributions of the research work of this thesis. diff --git a/021_VideoCoding/Figures/BlockPartitionsWithTree.tikz b/021_VideoCoding/Figures/BlockPartitionsWithTree.tikz index 89bf433..f22f2b2 100644 --- a/021_VideoCoding/Figures/BlockPartitionsWithTree.tikz +++ b/021_VideoCoding/Figures/BlockPartitionsWithTree.tikz @@ -89,22 +89,22 @@ matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} child{ node [node_cu] {4} } child{ node [node_cu] {5} } } - child{ node [node_cu] {5} } child{ node [node_cu] {6} } + child{ node [node_cu] {7} } } - child{ node [node_cu] {7} } + child{ node [node_cu] {8} } child{ node [node_split] {} - child{ node [node_cu] {8} } + child{ node [node_cu] {9} } child{ node [node_split] {} - child{ node [node_cu] {9} } child{ node [node_cu] {10} } child{ node [node_cu] {11} } - child{ node [node_cu] (CUD3) {12} } + child{ node [node_cu] {12} } + child{ node [node_cu] (CUD3) {13} } } - child{ node [node_cu] {13} } - child{ node [node_cu] (CUD2) {14} } + child{ node [node_cu] {14} } + child{ node [node_cu] (CUD2) {15} } } - child{ node [node_cu] (CUD1) {15} } + child{ node [node_cu] (CUD1) {16} } ; diff --git a/021_VideoCoding/Figures/HEVCBlockDiagram.pdf b/021_VideoCoding/Figures/HEVCBlockDiagram.pdf index 6ba5263..5be0a8b 100644 Binary files a/021_VideoCoding/Figures/HEVCBlockDiagram.pdf and b/021_VideoCoding/Figures/HEVCBlockDiagram.pdf differ diff --git a/021_VideoCoding/Figures/HEVCBlockDiagram.tikz b/021_VideoCoding/Figures/HEVCBlockDiagram.tikz index 65df3b8..33bce9c 100644 --- a/021_VideoCoding/Figures/HEVCBlockDiagram.tikz +++ b/021_VideoCoding/Figures/HEVCBlockDiagram.tikz @@ -51,11 +51,11 @@ \node[labels, text width=2.3cm, above=0 of FTu] {Transform Units \\quad-tree}; -\node[processBlock, text width=.9cm, right=0.5 of FTu] (Transf) {$T$}; -\node[processBlock, text width=.9cm, right=0.5 of Transf] (Quant) {$Q$}; +\node[processBlock, text width=.9cm, right=0.7 of FTu] (Transf) {$T$}; +\node[processBlock, text width=.9cm, right=0.3 of Transf] (Quant) {$Q$}; \node[processBlock, text width=3.3cm, above=1 of Transf, xshift=.5cm] (GCC) {Encoder control}; -\node[labels,right=1.8 of Quant] (QuantCoef) {Quantized Coefficients}; +\node[labels,right=1.3 of Quant] (QuantCoef) {Quantized Coefficients}; \node[labels,right=2.7 of GCC] (ControlData) {Control data}; \path[blockArrow] (Quant) -- (QuantCoef) node [point,pos=0.5] (pointQuant) {}; @@ -129,7 +129,7 @@ \begin{pgfonlayer}{background} \path[arrowControl] (GCC -| Quant.90) -- (Quant.90); -\path[arrowControl] (GCC -| IntraEstm.30) -- (IntraEstm.30); +\path[arrowControl] (GCC -| IntraEstm.40) -- (IntraEstm.40); \path[arrowControl] (GCC -| MotionSearch.60) -- (MotionSearch.60); \path[arrowOutput] (QuantCoef) -- (VLC.135); diff --git a/021_VideoCoding/Figures/PUAsymetric.tikz b/021_VideoCoding/Figures/PUAsymetric.tikz index 5aa518f..7095166 100644 --- a/021_VideoCoding/Figures/PUAsymetric.tikz +++ b/021_VideoCoding/Figures/PUAsymetric.tikz @@ -34,28 +34,32 @@ matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 \draw[CU] (0,0) rectangle ++(1,4); % 64x64 \draw[CU] (1,0) rectangle ++(3,4); % 64x64 - \node[TextStyle] at (2,-.8) {$\frac{3N}{4}\times N$ (L)}; +% \node[TextStyle] at (2,-.8) {$\frac{3N}{4}\times N$ (L)}; + \node[TextStyle] at (2,-.8) {$\frac{N}{2}\times 2N$ (Left)}; \end{scope} \begin{scope}[xshift=5cm] \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 \draw[CU] (0,0) rectangle ++(3,4); % 64x64 \draw[CU] (3,0) rectangle ++(1,4); % 64x64 - \node[TextStyle] at (2,-.8) {$\frac{3N}{4}\times N$ (R)}; +% \node[TextStyle] at (2,-.8) {$\frac{3N}{4}\times N$ (R)}; + \node[TextStyle] at (2,-.8) {$\frac{N}{2}\times 2N$ (Right)}; \end{scope} \begin{scope}[xshift=10cm] \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 \draw[CU] (0,0) rectangle ++(4,3); % 64x64 \draw[CU] (0,3) rectangle ++(4,1); % 64x64 - \node[TextStyle] at (2,-.8) {$N\times\frac{3N}{4}$ (T)}; +% \node[TextStyle] at (2,-.8) {$N\times\frac{3N}{4}$ (T)}; + \node[TextStyle] at (2,-.8) {$2N\times\frac{N}{2}$ (Top)}; \end{scope} \begin{scope}[xshift=15cm] \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 \draw[CU] (0,0) rectangle ++(4,1); % 64x64 \draw[CU] (0,1) rectangle ++(4,3); % 64x64 - \node[TextStyle] at (2,-.8) {$N\times\frac{3N}{4}$ (B)}; +% \node[TextStyle] at (2,-.8) {$N\times\frac{3N}{4}$ (B)}; + \node[TextStyle] at (2,-.8) {$2N\times\frac{N}{2}$ (Bottom)}; \end{scope} diff --git a/021_VideoCoding/Figures/PUSymetric.tikz b/021_VideoCoding/Figures/PUSymetric.tikz index a464f84..d598c6a 100644 --- a/021_VideoCoding/Figures/PUSymetric.tikz +++ b/021_VideoCoding/Figures/PUSymetric.tikz @@ -32,26 +32,30 @@ matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} \begin{scope}[xshift=0] \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 - \node[TextStyle] at (2,-.8) {$N\times N$}; +% \node[TextStyle] at (2,-.8) {$N\times N$}; + \node[TextStyle] at (2,-.8) {$2N\times 2N$}; \end{scope} \begin{scope}[xshift=10cm] \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 \draw[CU] (0,0) rectangle ++(2,4); % 64x64 - \node[TextStyle] at (2,-.8) {$\frac{N}{2}\times N$}; +% \node[TextStyle] at (2,-.8) {$\frac{N}{2}\times N$}; + \node[TextStyle] at (2,-.8) {$N\times 2N$}; \end{scope} \begin{scope}[xshift=5cm] \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 \draw[CU] (0,0) rectangle ++(4,2); % 64x64 - \node[TextStyle] at (2,-.8) {$N\times\frac{N}{2}$}; +% \node[TextStyle] at (2,-.8) {$N\times\frac{N}{2}$}; + \node[TextStyle] at (2,-.8) {$2N\times N$}; \end{scope} \begin{scope}[xshift=15cm] \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 \draw[CU] (0,0) rectangle ++(2,2); % 64x64 \draw[CU] (2,2) rectangle ++(2,2); % 64x64 - \node[TextStyle] at (2,-.8) {$\frac{N}{2}\times\frac{N}{2}$}; +% \node[TextStyle] at (2,-.8) {$\frac{N}{2}\times\frac{N}{2}$}; + \node[TextStyle] at (2,-.8) {$N \times N$}; \end{scope} diff --git a/021_VideoCoding/VideoCodingReview.tex b/021_VideoCoding/VideoCodingReview.tex index db69ca9..7464617 100644 --- a/021_VideoCoding/VideoCodingReview.tex +++ b/021_VideoCoding/VideoCodingReview.tex @@ -4,7 +4,6 @@ % \chapter{\HIGHLIGHT{The \ac{hevc} Standard}} \chapter{The High Efficiency Video Coding standard \VERSION{3}} -%\chapter{\HIGHLIGHT{The State-of-Art Video Coding}} \label{Chapter:HEVCReview} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -17,7 +16,7 @@ This chapter presents an overview of the \ac{hevc}~\cite{H265StandardFeb2018} st In short, the \ac{hevc} standard essentially aims to support all existing applications of \ac{h264}~\cite{H264AVCStandard2003}, while extending two keys aspects: increased video resolution with higher compression ratio and also more efficient use of parallel processing architectures. In order to increase the coding flexibility and higher video resolutions, the \ac{hevc} standard adopts a new block partition structure, as well as, new prediction modes. In this chapter the new data structures and coding techniques, specifically introduced to increase the coding efficiency, are described. -Particular emphasis is given to those which may have a significant impact in the error robustness characteristics of \ac{hevc} streams. +Particular emphasis is given to those which may have a significant impact on the error robustness characteristics of \ac{hevc} streams. The chapter is organised as follows. Section~\ref{Section2:GeneraVideoSystem} introduces the functional blocks of a generic hybrid video coding system and Section~\ref{Section2:Overview} provides an overview of the \ac{hevc} standard. @@ -34,9 +33,9 @@ Section~\ref{Section2:Summary} concludes the chapter. \label{Section2:GeneraVideoSystem} A digital video signal is a temporal succession of images, also referred to as pictures or frames, which usually presents a significant amount of temporal and spatial correlations. -In general, video coding and distribution systems follow through a generic chain of processing and delivery stage, comprising pre-processing, encoding, storage or transmission, decoding and post-processing, as shown in Figure~\ref{Figure2:HybridCodec}. -The pre-processing stage is used adapt the raw video signal, either analogue or digital, to an appropriate format, such as RGB or YUV~\cite{DataCompression1997}. -Then, the video data is compressed by the respective encoding module, mostly reducing its redundancy and visual irrelevancy. The encoding process is responsible for representing the raw video in a highly compressed format, while maintaining a good trade-off between video quality and bitrate through high efficient algorithms and coding tools. +In general, video coding and distribution systems follow through a generic chain of processing and delivery stages, comprising pre-processing, encoding, storage or transmission, decoding and post-processing, as shown in Figure~\ref{Figure2:HybridCodec}. +The pre-processing stage is used to adapt the raw video signal, either analogue or digital, to an appropriate format, such as RGB or YUV~\cite{DataCompression1997}. +Then, the video data is compressed by the respective encoding modules, mostly reducing its redundancy and visual irrelevancy. The encoding process is responsible for representing the raw video in a highly compressed format, while maintaining a good trade-off between video quality and bitrate through high efficient algorithms and coding tools. In the last stages of the chain, the compressed format is decoded and presented by displaying the signal content after adaptation to the consumer equipment through post-processing, when necessary. % \begin{figure}[tb] @@ -56,7 +55,7 @@ This is the reason why the coding scheme is called hybrid, as it combines quiet \begin{figure}[t] \centering \input{\FigsPath/HybridCodingBlockDiagramPrePost.tikz} -\caption{Block diagram of a generic video coding system, including the inner blocks of a hybrid encoding and decoding scheme.} +\caption{Block diagram of a generic video coding system, including the inner blocks of a hybrid encoding and decoding scheme (T and Q corresponds to the transform and quatisation operations, respectively).} \label{Figure2:HybridCodec} \end{figure} @@ -65,14 +64,14 @@ This is the reason why the coding scheme is called hybrid, as it combines quiet In general, there are two kinds of redundancy in video signal: spatial and temporal. On the one hand, the spatial redundancy is exploited through intra frame prediction, where the underlying mechanism is to find the best matching block for the current one, by searching the neighbouring region. This is possible due to the existence of high correlation among neighbouring pixels within a frame. -As result, when intra frame prediction is used, a block of pixels is reconstructed from the neighbouring blocks previously processed. +As a result, when intra frame prediction is used, a block of pixels is reconstructed from the neighbouring blocks previously processed. On the other hand, inter frame prediction is designed to exploit the temporal redundancy that exists among consecutive frames, such as those temporally adjacent, e.g., the past frame or the future frame in regard to the current one. Temporal redundancy is normally referred to as the correlation between two adjacent frames since they usually have quiet similar visual content, such as foreground objects and background. Thus, the purpose of inter-frame prediction is to search for the best matching block for the current one in a set of reference frames. After finding the best matching candidate for the current block, either by intra- or inter-frame prediction, only the difference between the original block and the predicted one needs to be encoded. As shown in Figure~\ref{Figure2:HybridCodec}, these differences (also known as residual signal) are further processed and transformed (T) into coefficients in the frequency domain. -Such a transformation decorrelates the residue signal and concentrates the signal energy in few coefficients, mostly at low frequencies. +Such a transformation decorrelates the residue signal and concentrates the signal energy in a few coefficients, mostly at low frequencies. Subsequently, the transformed coefficients are quantised (Q) to reduce the representation accuracy of the signal, and consequently to reduce the number of bits required for coding. Finally, the quantised transform coefficients together with the prediction information are further compressed by the entropy encoder, forming an output stream suitable for transmission or storage, as shown in Figure~\ref{Figure2:HybridCodec}. On the other side of the communication chain, the decoding stage performs the inverse operations on the stream to reconstruct the video signal. @@ -85,14 +84,6 @@ In the next section, the aforementioned blocks are described in the context of t \section{Overview the standard features} \label{Section2:Overview} -\glsreset{ctu} -\glsreset{cu} -\glsreset{pu} -\glsreset{tu} -\glsreset{nal} -%\glsreset{nalu} -\glsreset{cabac} -\glsreset{rpset} Similar to the previous coding standards developed by \ac{jctvc}, in \ac{hevc}~\cite{HEVCOverviewJul2012,HEVCOverviewDez2012,BookHEVC2014} only the coding syntax and semantics are standardised. @@ -113,9 +104,9 @@ Subsequently, the prediction residue is transformed by linear spatial transform However, there are some novelties introduced in the \ac{hevc} standard. As depicted in Figure~\ref{fig:HEVCBlockDiagram}, the encoding process follows a block-based approach, therefore each picture is partitioned into larger blocks, referred to as \acs{ctu}, with a maximum size of $64\times64$ pixels. Subsequently, a dynamic quad-tree partitioning is used to divide the picture into \acs{cu}. -These units are introduced as the basic processing units for prediction, while for the transform and quantisation operations, \acs{tu} are used. +These units are introduced as the basic processing units for prediction, while for the transform and quantisation operations, transform units are used. In order to generate the prediction residue, the \ac{hevc} may either use one of 34 intra-prediction modes or take advantage of symmetric and asymmetric partitions for motion estimation and inter-prediction. -The motion estimation is performed in the pictures listed in the \ac{rpset}, which was introduced in the \ac{hevc} to increase the robustness to errors and data loss of reference picture management. +The motion estimation is performed in the pictures listed in the reference picture set, which was introduced in the \ac{hevc} to increase the robustness to errors and data loss of reference picture management. To improve the overall quality of the reconstructed pictures and smooth the coding artefacts, \ac{hevc} not only uses a deblocking filter but also a \ac{sao} filter. Finally, the control data, quantised coefficients, prediction modes, motion vectors and filter information are compressed using the \ac{cabac}. The compressed bitstream is then allocated to various \ac{nal} units. @@ -126,17 +117,13 @@ To improve the overall quality of the reconstructed pictures and smooth the codi \section{High-level syntax} \label{Section2:HighLevel} -\glsreset{nal} -\glsreset{sei} -\glsreset{vcl} - This section describes the high-level components of the \ac{hevc} standard that provide support for signalling and stream partitioning, which are relevant elements in the context of robust video communications. \ac{hevc} inherits several high-level features from \ac{h264}, e.g., the \ac{nal}, parameter sets, the use of \ac{poc} and \ac{sei} messages for auxiliary data. However, some high-level features defined in \ac{h264} were not included, such as \ac{fmo}, redundant slices, arbitrary slice order, data partitioning and switching slices. -Futhermore, several new high-level features are introduced, such as the \ac{vps}, tiles and wavefront tools for parallel processing, dependent slices, for reduced delay, and a new reference picture management concept. -Moreover, new pictures types were introduced to increase the random-access flexibility and temporal sub-layer switching. +Furthermore, several new high-level features are introduced, such as the \ac{vps}, tiles and wavefront tools for parallel processing, dependent slices, for reduced delay, and a new reference picture management concept. +Moreover, new picture types were introduced to increase the random-access flexibility and temporal sub-layer switching. -\subsection{Network Abstraction Layer} +\subsection{Network abstraction layer} Similar to \ac{h264}~\cite{H264AVCStandard2003}, the \ac{hevc} standard defines an adaptation layer to cope with different transport protocols. The \ac{nal} is used for stream partitioning and synchronization, which is useful to deal with lossy network environments~\cite{H264OverIPJul2003}. @@ -163,8 +150,6 @@ Figure~\ref{Figure2:NalHeader} shows the structure of the \ac{hevc} \ac{nal} uni \subsection{Parameters set} -\glsreset{vps} - The encoder uses a set of signalling packets as a robust method to share data with the decoder, referred to as parameter set units. This information was introduced in previous standards to deal with vulnerabilities in the transport of signalling information between the encoder and decoder. In this approach, such set of parameters is multiplexed in the bitstream and repeated as many times as required by the application, e.g., streaming, storage among others. In \ac{hevc}, this structure is inherited from \ac{h264}\cite{H264OverIPJul2003} with modifications to cope with new coding tools. The \ac{hevc} includes three sets of parameters, \ac{sps}, \ac{pps} and the newly introduced \ac{vps}\cite{HEVCHighLevelReferenceDez2012}. @@ -172,7 +157,7 @@ The \ac{hevc} includes three sets of parameters, \ac{sps}, \ac{pps} and the newl The new set of parameters \ac{vps} contains information related to the different layers of the video signal, avoiding duplications like in the \ac{h264} and its extensions. \Gls{vps} aims to provide signalling for every layer, guaranteeing an extendible standard, capable of supporting multiple layers. Moreover, \ac{vps} also carries information for session negotiation (\emph{e.g.} profile, level and tier). -The \ac{sps} contains information related to the whole sequence (\emph{i.e.}, should affect all coded slices) and related with the decoder operation point, as well as flags to control optional tools and scalability. +The \ac{sps} contains information related to the whole sequence (\emph{i.e.}, should affect all coded slices) and related to the decoder operation point, as well as flags to control optional tools and scalability. The \ac{pps} is responsible to carry information that may change for every picture, such as, initial \ac{qp}, flags for picture related tools and tilling information. @@ -231,7 +216,7 @@ These frames must belong to temporal sub-layer 0 and should be coded using intra One should note that there may be frames compressed with intra prediction only, but not be marked as random-access points. It is always possible to start decoding from a \ac{rap} frame onwards, and to output any subsequent pictures in the display order, even if all pictures that precede the \ac{rap} in the decoding order are discarded from the bitstream. -The \ac{idr} frame is an intra coded picture that completely refreshes the decoding process, i.e., cleans the decoder picture buffer preventing predictions to previously coded frames, starting a new conforming video stream. This means that neither the \ac{idr} nor any other subsequent frame in the decoding order have prior dependencies. \ac{idr} frames are allowed to have leading pictures, i.e., frames that follow the \ac{rap} picture in the decoding order but precede it in the display order. However, they must be decodable, thus creating a true random-access point where all subsequent frames in the decoding order can be decoded and displayed. +The \ac{idr} frame is an intra-coded picture that completely refreshes the decoding process, i.e., cleans the decoder picture buffer preventing predictions to previously coded frames, starting a new conforming video stream. This means that neither the \ac{idr} nor any other subsequent frame in the decoding order have prior dependencies. \ac{idr} frames are allowed to have leading pictures, i.e., frames that follow the \ac{rap} picture in the decoding order but precede it in the display order. However, they must be decodable, thus creating a true random-access point where all subsequent frames in the decoding order can be decoded and displayed. Introducing \ac{rap} reduces the coding efficiency since no prior reference frames can be used. In order to reduce this impact, the \ac{hevc} standard introduces a new \ac{rap} picture, referred to as \ac{cra} picture. @@ -261,7 +246,7 @@ Due to the introduction of more flexible random-access points using \ac{cra} pic \begin{figure}[tb] \centering \input{\FigsPath/PictureTypes_Leading_Trailing.tikz} - \caption{Leading and trailing pictures associated with a random-access point.} + \caption{Leading and trailing pictures associated with the random-access point $I_{1}$ (letters corresponds to the coding type and the numbers to the coding order).} \label{Figure2:LeadingTralingPics} \end{figure} @@ -284,7 +269,7 @@ Although these frames allow higher flexibility in terms of layer switching, they \centering % \includegraphics[width=0.75\columnwidth]{\FigsPath/HEVC/Pictures_TSA} \input{\FigsPath/PictureTypes_TSA.tikz} - \caption{Temporal layers access pictures example.} + \caption{Temporal layers access pictures example with three layers ($T_0$ to $T_2$).} \label{Figure2:TSAExample} \end{figure} @@ -310,7 +295,7 @@ Picture partitioning normally serves one or more of the following three purposes \item[Network adaptation:] adapt to the network constraint of \ac{mtu} size, found for example in \ac{ip} networks. Such packetisation scheme restricts the maximum number of payload bits within a slice regardless the size of the coded frame. To keep each slice within this limit and minimise the packetisation overhead, a variable number of coding units is used for each slice. - \item[Parallel processing:] partitioning the coded frame into processing data units, which can be encoded in parallel. This is achieved by dividing the coding units such they can be encoded and decoded independently of each other’s. + \item[Parallel processing:] partitioning the coded frame into processing data units, which can be encoded in parallel. This is achieved by dividing the coding units such that they can be encoded and decoded independently of each other’s. \end{description} \subsubsection{Slices} @@ -345,9 +330,7 @@ The use of tiles provide several advantages. Specifically, they improve frame pa \subsubsection{Wavefront Parallel Processing} -\glsreset{wpp} - -As mentioned before, the use of either slices or tiles to process each frame using parallel threads require the use of different entropy coding contexts for each slice/tile, in order to make them independently decodable. As consequence, the compression efficiency decreases. In order to overcome this issue, the concept of \ac{wpp} was introduced, which enables coding using efficient parallel processing~\cite{HEVCParallelCSVT2016}. +As mentioned before, the use of either slices or tiles to process each frame using parallel threads requires the use of different entropy coding contexts for each slice/tile, in order to make them independently decodable. As a consequence, the compression efficiency decreases. In order to overcome this issue, the concept of \ac{wpp} was introduced, which enables coding using efficient parallel processing~\cite{HEVCParallelCSVT2016}. In \ac{wpp}, each slice (or tile) is divided into single rows of \acs{ctu} which can be processed in parallel, but entropy coding and prediction are enabled across \acs{ctu} of different rows, in order to avoid loss in coding efficiency. To re-use the same context in the whole slice (or tile) each row requires a delay of two \acs{ctu} in order to ensure that all necessary dependencies have already been encoded/decoded. Figure~\ref{Figure2:WPP} illustrates an example of \ac{wpp} using at least five different threads. @@ -356,7 +339,7 @@ Each square corresponds to a \ac{ctu} and the numbers correspond to the time ins \begin{figure}[tb] \centering \input{\FigsPath/WPP.tikz} - \caption{Illustration of the wavefront parallel processing.} + \caption{Illustration of the wavefront parallel processing (numbers corresponds to the time instant each \ac{ctu} is processed).} \label{Figure2:WPP} \end{figure} @@ -372,10 +355,7 @@ The \ac{mb} is considered the basic processing unit in the \ac{h264}, and the re Each \ac{mb} could also be partitioned into sub-blocks, which should follow the same prediction mode. -\glsreset{ctu} -\glsreset{cu} - -The basic data structures in \ac{hevc} are quite different from \ac{h264}. Each picture is divided into blocks named as \ac{ctu}~\cite{BlockPartHEVC2012}. The maximum \ac{ctu} size is an encoder configuration parameter that is signalled to the decoder, and should be one of the following: $64\times64$, $32\times32$ and $16\times16$. The use of data structures with larger size in \ac{hevc} was adopted to achieve higher coding efficiency especially in high resolution video. Each \ac{ctu} can be recursively partitioned into four smaller units, referred to as \acs{cu}, until it reaches the minimum unit size, which cannot be smaller than $8\times8$ pixels. +The basic data structures in \ac{hevc} are quite different from \ac{h264}. Each picture is divided into blocks named as \ac{ctu}~\cite{BlockPartHEVC2012}. The maximum \ac{ctu} size is an encoder configuration parameter that is signalled to the decoder, and should be one of the following: $64\times64$, $32\times32$ and $16\times16$ pixels. The use of data structures with larger size in \ac{hevc} was adopted to achieve higher coding efficiency especially in high resolution video. Each \ac{ctu} can be recursively partitioned into four smaller units, referred to as \acs{cu}, until it reaches the minimum unit size, which cannot be smaller than $8\times8$ pixels. Similar to the maximum \ac{ctu} size, the minimum size is also defined at the encoder. For instance, in case of homogeneous regions, large \acs{cu} can be used to represent such regions by using a smaller number of symbols than in the case of using several small units. Figure~\ref{Figure2:HEVCCodingStructure} illustrates an example of a \ac{ctu} partitioning into \acs{cu}. @@ -385,14 +365,12 @@ Data structures in \ac{hevc} provide a significant improvement in comparison wit \begin{figure}[t] \centering \input{\FigsPath/BlockPartitionsWithTree.tikz} - \caption{Representation of a CTU ($64\times64$) and its partitioning into CUs and PUs (left); corresponding quad-tree with coding order (right)} + \caption{Representation of a CTU ($64\times64$) and its partitioning into CUs and PUs (left); corresponding quad-tree with coding order (right).} \label{Figure2:HEVCCodingStructure} \end{figure} -\glsreset{pu} -\glsreset{tu} - -The \ac{cu} is the basic processing unit in \ac{hevc}. It is further partitioned into two types of units, specifically \ac{pu} and \ac{tu}, which are data blocks used for prediction and transform operations, respectively. +The \ac{cu} is the basic processing unit in \ac{hevc}. +It is further partitioned into two types of units, specifically \ac{pu} and \ac{tu}, which are data blocks used for prediction and transform operations, respectively. The decision whether to use intra-frame or inter-frame prediction modes is taken at the \ac{cu} level. However, depending on the partitioning of the \ac{cu} into \acs{pu}, different intra or inter coding modes can be used for each \ac{pu}. @@ -402,16 +380,16 @@ The decision whether to use intra-frame or inter-frame prediction modes is taken \section{Prediction modes} \label{Section2:PredictionModes} -Prediction techniques are responsible for an important role in image and video coding standards, due to their ability to reduce the signal redundancy based on the previously encoded pixels. -These techniques include the directional intra prediction, for efficiently reduce spatial redundancy, and inter-frame motion compensation to remove the temporal redundancies. -This section describes the prediction techniques used in the \ac{hevc} standard comprise different algorithms. +Prediction techniques play an important role in image and video coding standards, due to their ability to reduce the signal redundancy based on the previously encoded pixels. +These techniques include the directional intra prediction for efficiently reduce spatial redundancy, and inter-frame motion compensation to remove the temporal redundancies. +This section describes the prediction techniques used in the \ac{hevc} standard that comprises different algorithms. \subsection{Intra-frame prediction} Intra-frame prediction is used to efficiently reduce the spatial redundancy within video frames. Intra predicted blocks are obtained using the previously encoded pixels around the current \ac{pu}. -When the intra prediction mode is chosen for a \ac{cu}, the size of the \ac{pu} needs to be the same as the \ac{cu}, except when the smaller \ac{cu} is selected and further divided into four \acs{pu} is done. +When the intra prediction mode is chosen for a \ac{cu}, the size of the \ac{pu} needs to be the same as the \ac{cu}, except when the smallest \ac{cu} size is selected and further division into four \acs{pu} is allowed. For example, when the smaller \ac{cu} size is $8\times8$, the encoder may decide to divide it into four \acs{pu} of $4\times4$ pixels, each one having its own intra prediction mode. This is signalled using a binary flag. The use of small \acs{pu} is useful for regions with many details that require fine-granularity prediction. @@ -470,7 +448,7 @@ In \ac{hevc} sample smoothing is applied in two distinct cases to improve the ov Firstly, the reference samples are filtered and secondly the \ac{pu} is filtered after prediction. %In comparison with previous standard \ac{h264}, where only reference samples were filtered. -In \ac{hevc}, the reference sample smoothing is conditionally applied, based on the \ac{pu} and intra prediction mode. Similar to \ac{h264}, a three tap smoothing filter (i.e., $\left[1\ 2\ 1\right]/4$) is applied, except for \ac{pu} sizes of $4\times4$. The prediction resulting from Planar mode is also filtered when the \ac{pu} size is greater than $8\times8$, and it is not used for the DC mode. +In \ac{hevc}, the reference sample smoothing is conditionally applied, based on the \ac{pu} and intra prediction mode. Similar to \ac{h264}, a three-tap smoothing filter (i.e., $\left[1\ 2\ 1\right]/4$) is applied, except for \ac{pu} sizes of $4\times4$. The prediction resulting from Planar mode is also filtered when the \ac{pu} size is greater than $8\times8$, and it is not used for the DC mode. The second filter in \ac{hevc} is the boundary smoothing. This filter aims at removing the discontinuities along block boundaries due to intra prediction. This filter is only applied when the block size is smaller than $32\times32$ to the DC, and angular modes 10 and 26 (exactly the horizontal and vertical directions, respectively). @@ -478,42 +456,39 @@ For the DC mode a two-tap filter ($\left[3\ 1\right]/4$), fed by the original pr In the case of angular mode 10 (horizontal direction), only the predicted samples of the first column are changed by adding half of the difference between the adjacent reference sample and the top-left reference sample. A similar filtering is done for angular mode 26 (vertical direction), but only for the first row of predicted samples. -Only the mentioned boundaries are filtered, as they are the most likely to introduce discontinuities. - -%\subsubsection{Rough Mode Decision} -%\cite{HEVCROughModeDecision} +Only the mentioned boundaries are filtered, as they are the ones most likely to introduce discontinuities. \subsection{Inter-frame prediction} -The efficiency of video coding algorithms consistently rely on inter-frame prediction techniques to reduce temporal redundancy. +The efficiency of video coding algorithms consistently relies on inter-frame prediction techniques to reduce temporal redundancy. The underlying idea of inter-frame prediction is to estimate the current frame from one or more previously encoded frames used as reference with block-based motion compensation. The most common technique used for motion estimation in \ac{hevc} is the block matching algorithm. In the \ac{hevc} standard an \ac{amvp} is used for efficient motion vector coding, which defines the Merge Mode and differential coding of the motion information. \subsubsection{Motion estimation and compensation} -Motion estimation consists in searching for a block with the highest similarity with the current block to be predicted (\ac{pu} in \ac{hevc}), using block-matching algorithms over previously encoded frames, i.e. the reference frames. +Motion estimation consists of searching for a block with the highest similarity with the current block to be predicted (\ac{pu} in \ac{hevc}), using block-matching algorithms over previously encoded frames, i.e. the reference frames. Typically, reference frames correspond to past or future temporally adjacent frames. The frame encoding order determines which references are selected and which future frames are available. The reference block that results in the lowest error is selected as the best block for prediction. -To identify the best matching block to be used for motion compensation, the different between the target and the reference block positions is encoded as a two-dimensional \glsfirst{mv}. +To identify the best matching block to be used for motion compensation, the difference between the target and the reference block positions is encoded as a two-dimensional \ac{mv}. The \ac{hevc} allows for either one or two \acs{mv} to be used for each \ac{pu}. Video frames of type P are predicted from one single reference (one \ac{mv}), while frame of type B are predicted from two references (two \acs{mv}). This results in uni-direction or bi-directional coding, respectively. -Furthermore, weighted prediction can also be used, which consists in scaling and offset operations performed on the prediction block to improve efficiency. +Furthermore, weighted prediction can also be used, which consists of scaling and offset operations performed on the prediction block to improve efficiency. The \acs{mv} only refer to frames included in a reference frame list, where each frame is identified by an index. Therefore, to fully describe inter-frame prediction a combination of two parameters is required: \ac{mv} and reference frame index. Since MPEG-2, motion estimation/compensation has been performed using blocks of variable size. -In \ac{h264} block sizes ranging from $4\times4$ up to $16\times$ pixels are allowed, while \ac{hevc} extends the possible block sizes up to $64\times64$ pixels, and also to asymmetric rectangular block sizes. +In \ac{h264} block sizes ranging from $4\times4$ up to $16\times16$ pixels are allowed, while \ac{hevc} extends the possible block sizes up to $64\times64$ pixels, and also to asymmetric rectangular block sizes. By using larger prediction units less \acs{mv} are required, thus fewer bits are spent in coding motion information. In order to increase the efficiency of motion estimation fractional \acs{mv} are used, which has been proven to improve motion compensation accuracy. This is accomplished through interpolation to generate non-integer pixels in the reference frame, resulting in \acs{mv} up to quarter-pixel accuracy for luma samples. For chroma samples, the \ac{mv} accuracy is determined according to the chroma sampling format, which for 4:2:0 results in eighth-pixel accuracy. In contrast with the two-stage approach used in \ac{h264}, \ac{hevc} uses a single separable approach for the interpolation process without intermediate rounding operations. This improves the precision and simplifies the architecture of the fractional interpolation. -The partition size as well the \ac{mv} are selected using a \ac{rd} optimisation which considers both the number of bits required to code the motion information and the video distortion. +The partition size as well as the \ac{mv} are selected using a \ac{rd} optimisation which considers both the number of bits required to code the motion information and the video distortion. The block matching-based motion estimation is one of the most computationally intensive procedures in video encoders. In general, the optimal solution for motion estimation is provided by the full-search algorithm, which tests all the possible candidate blocks in the search area. However, although the full-search algorithm achieves the optimal performance, it requires the highest computational complexity. Therefore, sub-optimal algorithms have been proposed over the years to reduce the motion estimation complexity and decrease encoding time~\cite{HEVCME2012,HEVCME2017,HEVCME2014}. -For example, the HM reference software implements a fast algorithm based on Test Zone Search~\cite{HEVCME2012}. +For example, the \ac{hm} software implements a fast algorithm based on Test Zone Search~\cite{HEVCME2012}. Since motion estimation is a non-normative procedure, encoders may implement any alternative motion search algorithms. @@ -532,12 +507,12 @@ Thus, a complete set of all frames that must be kept in the \ac{dpb} is explicit This is different from \ac{h264} where only relative changes to the \ac{dpb} are encoded. With the \ac{rpset} concept, no information from earlier frames in decoding order is needed to maintain the correct status of the \ac{dpb}. Table~\ref{Table2:ReferencePictureListExample} shows the \ac{rpset} for some frames illustrated in Figure~\ref{Figure2:LeadingTralingPics}. -As expected, for the \ac{cra} frame ($I_1$) no information is sent to the decoder because there is not inter-prediction and the \ac{dpb} should be empty. +As expected, for the \ac{cra} frame ($I_1$) no information is sent to the decoder because there is no inter-prediction and the \ac{dpb} should be empty. For the remaining frames, two types of information are encoded: frame number and a binary flag to indicate whether it is currently used. This is required because any frame not listed in a given \ac{rpset} will be discarded by the decoder and then subsequent frames cannot use them as reference. \begin{table}[tb] \centering - \caption{\glsfirst{rpset} example.} + \caption{Reference picture set example.} \label{Table2:ReferencePictureListExample} \small \TABLESVSPACE @@ -581,7 +556,7 @@ These partitions types are only allowed for \acs{cu} with $16\times16$ pixels or \input{\FigsPath/PUSymetric.tikz} & (a)\\ \input{\FigsPath/PUAsymetric.tikz} & (b)\\ \end{tabular} - \caption{Partition types of an inter coded CU into PUs~\cite{HEVCBlock2012}.} + \caption{Partition types of an inter-coded CU into PUs; in (b) the size (and position between parentheses) corresponds to the smaller partition~\cite{HEVCBlock2012}.} \label{Figure2:InterPuPartitions} \end{figure} @@ -596,7 +571,7 @@ This results in improved flexibility, compared with the predefined values used i The \ac{mv} candidates allowed in the Merge Mode\cite{MotionCodingHEVC2013} are illustrated in Figure~\ref{Figure2:MVCandidates}. The figure shows the positions of the spatial \ac{mv} candidates ($A_0$, $A_1$, $B_0$, $B_1$, $B_2$). Moreover, it also uses a temporal \ac{mv} candidate, derived from the co-located position on the temporally adjacent frame, at positions $T_0$ or $T_1$. From the \ac{mv} candidate list the unavailable candidates are not considered, and duplicated candidates (i.e. with the same motion information) are removed. -The Merge Mode uses a $C-1$ spatial \ac{mv} candidates, selected from the aforementioned order, and one \ac{tmvp} to fulfil the total number of \acs{mv} ($C$). The value of $C$ is defined at the slice header and a flag controlling the use of \ac{tmvp} is encoded at the \ac{pps}-level. +The Merge Mode uses a $C-1$ spatial \ac{mv} candidates, selected from the aforementioned order, and one \ac{tmvp} to fulfil the total number of $C$ \acs{mv}. The value of $C$ is defined at the slice header and a flag controlling the use of \ac{tmvp} is encoded at the \ac{pps}-level. For B-slices, additional \ac{mv} candidates are considered by choosing two existing candidates, according to a predefined order for reference picture list 0 and list 1. A total number of twelve combinations are specified in \ac{hevc}, which are listed in~\cite{HEVCOverviewDez2012}. For P-slices, if the number of candidates is less than $C$, then zero \acs{mv} are included to fill the remaining entries. \begin{figure}[tb] @@ -617,7 +592,7 @@ In this case, only a skip flag is enabled and the corresponding merge index is t The \ac{hevc} also improves the performance of \ac{mv} coding for those cases where the Merge Mode is not used. A differential encoding approach is always used for efficiently representing the motion information. -It considers the same \ac{mv} candidates of the Merge mode, illustrated in Figure~\ref{Figure2:MVCandidates}, to derive the \ac{mv} predictors from which the difference vector is obtained. +It considers the same \ac{mv} candidates of the Merge mode, illustrated in Figure~\ref{Figure2:MVCandidates}, to derive the \ac{mv} predictors from which the differential vector is obtained. From these \ac{mv} candidates only the first two available candidates are selected, based on the following order: \begin{enumerate} \item Left candidate: first available from $A_0$ and $A_1$. @@ -627,12 +602,6 @@ From these \ac{mv} candidates only the first two available candidates are select Therefore, the use of the temporal \ac{mv} candidate is limited to those cases where the spatial neighbours are not able to provide two valid candidates. For \ac{mv} prediction only a much lower number of candidates is allowed, since the encoder can still send the \ac{mv} difference. Moreover, the encoder performs motion estimation for each candidate, which increases the encoding complexity for every possibility tested. -%For those cases where the reference index is not the same as the current \ac{pu} and the neighbouring one, a scaling version of the \ac{mv} is used. - -%In the \ac{h264} standard, along different motion compensated coding techniques~\cite{H264OverviewJul2003}, the so-called P\_SKIP mode was introduced. In this mode, neither the quantised residual information, nor the \ac{mv} is transmitted. This mode is useful for large areas with no change or just constant motion, and it is only enabled for $16\times16$ blocks, \emph{i.e.}, the whole \ac{mb}. The \ac{mv} for the P\_SKIP mode is derived based on the surrounding neighbour MBs\cite{H264MotionVectorPrediction2003}. When no neighbours are available, or the left or top \acs{mv} are equal to zero, a zero \ac{mv} is used. Otherwise, the \ac{mv} is derived from the median value of the following three neighbours: left, above and above right, as shown in Figure~\ref{SkipMergeMode}~(a) by $MV_A$, $MV_B$ and $MV_C$, respectively. Whenever the $MV_C$ is not available, the $MV_D$ is used instead. -% - -%In contrast with the \ac{h264} standard, the \ac{hevc} standard allows the usage of a temporal \ac{mv} predictor, as well as, more spatial candidates to predict the current motion vector. Despite the improvements in the flexibility and coding efficiency, the increased levels of dependencies between the \acs{mv} make the motion information more prone to errors. Especially the introduction of the temporal candidate in the set of possible \acs{mv} increases the dependencies between subsequent frames. This leads to higher quality degradation in the presence of transmission errors, when compared to \ac{h264}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% @@ -641,28 +610,24 @@ For \ac{mv} prediction only a much lower number of candidates is allowed, since \label{Section2:TransformQuant} -\glsreset{tu} - -Similarly, to the previous standards, \ac{hevc} uses a transform followed by quantisation to compress the residual signal. +Similarly to the previous standards, \ac{hevc} uses a transform followed by quantisation to compress the residual signal. The residual block is partitioned into multiple square blocks, referred to as \acs{tu}. \subsubsection{Core transform} A 2D transform of the residual information is obtained by applying 1D transforms in both the horizontal and vertical directions. The core transform matrices are obtained from the \ac{dct} basis functions to define a integer matrix of $32\times32$ points. Sub-sampled versions of this matrix are used to derive the remaining transform matrix sizes down to $4\times4$. -A special case is applied for coding the residual of $4\times4$ intra coded block. +A special case is applied for coding the residual of $4\times4$ intra-coded block. In this case an alternative integer transform based on the \ac{dst} is used. The motivation for \ac{dst} is related to the statistics of the residual signal of these blocks, which tend to present higher residue as the distance to the boundary used for prediction (i.e, top or left boundary) increases. Simulation studies have shown that \ac{dst} provides up to 1\% of bitrate savings for the compression of $4\times4$ intra predicted blocks, without significantly increasing the computational complexity \cite{DCTDSTComp2013}. \subsubsection{Quantisation} -\glsreset{qp} - The \ac{hevc} follows the same principle as previous standards in terms of quantisation. After applying the transform operation to the prediction residue, a quantisation is applied to the resulting coefficients using a uniform-reconstruction quantiser scheme based on a \ac{qp}. The standard supports \ac{qp} values ranging from 0 to 51, where an increase of 6 in the \ac{qp} corresponds to doubling the quantisation step. -This procedure is the main responsible for the coding distortion, as it performs a many-to-one mapping, which cannot be reversed. +This procedure is the main cause for the coding distortion, as it performs a many-to-one mapping, which cannot be reversed. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -671,8 +636,6 @@ This procedure is the main responsible for the coding distortion, as it performs \section{Entropy coding} \label{Section2:Entropy} -\glsreset{cabac} - \ac{hevc} only uses the \ac{cabac}, which is an arithmetic coding method that only uses binary symbols, considering different probability models for each symbol~\cite{H264Cabac2003}. Entropy coding efficiency is closely linked with the context model selected. Therefore, this was carefully designed in the \ac{hevc} standard, extending the functionality previously defined for \ac{h264}. For example, the \ac{hevc} standard exploits the depth of the partition tree or residual transform tree, in order to derive the context models for several syntax elements. @@ -688,19 +651,17 @@ The coefficient scanning is implicitly selected and always performed in $4\times \section{In-loop filters} \label{Section2:Filters} -\glsreset{dbf} -\glsreset{sao} - The \ac{hevc} standard uses two filtering procedures over the reconstructed pixels before writing them into the frame buffer, namely the \ac{dbf} and the \ac{sao} filter. The main purpose of the deblocking filter is to reduce blocking artefacts caused by the block-based coding tools. %, operating similarly to the deblocking filter of the \ac{h264} standard. \ac{dbf} is applied to the samples that are spatially adjacent to the \ac{pu} or \ac{tu} boundaries, considering a $8\times8$ grid. -Such grid-based restriction reduces the computational complexity and improves parallel-processing -implementation, which is one of the main goals of \ac{hevc} design. +Such grid-based restriction reduces the computational complexity and facilitates the parallel-processing, which is one of the main goals of \ac{hevc} design. Three strength levels can be used by the \ac{dbf}, depending on the coded block characteristics. The \ac{sao} is a non-linear filter, that is adaptively applied to all samples of the image, after the deblocking filter. In its operation, the \ac{sao} filter modifies the samples -by adding an offset value, extracted from lookup tables transmitted by the encoder. For each \ac{ctu}, the encoder decides whether to apply the \ac{sao} filter or not, and if used, one of the two available filter types is applied, namely the band offset or the edge offset. In the band offset mode, the added offset value depends on the sample amplitude, while the edge offset uses the gradient to classify and derive the offset value. +by adding an offset value, extracted from lookup tables transmitted by the encoder. +For each \ac{ctu}, the encoder decides whether to apply the \ac{sao} filter or not, and if used, one of the two filter types is applied, namely the band offset or the edge offset. +In the band offset mode, the added offset value depends on the sample amplitude, while the edge offset uses the gradient to classify and derive the offset value. @@ -713,7 +674,7 @@ by adding an offset value, extracted from lookup tables transmitted by the encod This chapter was devoted to describe the main features of the \ac{hevc} standard with special emphasis on the high-level features, coding structures and newly introduced prediction modes. At the high-level, \ac{hevc} introduces new \ac{nal} types and also new pictures types. These allow the implementation of various random-access strategies, more flexibility and higher coding efficiency than its predecessor \ac{h264}. -Another major improvement comes from the inter-frame prediction. On the one hand new high-level reference picture management was introduced, which increase the error robustness, as it explicitly defines the active reference frames and the \ac{dpb} status. On the other hand it introduces the \ac{amvp} and the Merge mode, which efficiently reduce the required bits for coding the motion information. +Another major improvement comes from the inter-frame prediction. On the one hand new high-level reference picture management was introduced, which increases the error robustness, as it explicitly defines the active reference frames and the \ac{dpb} status. On the other hand it introduces the \ac{amvp} and the Merge mode, which efficiently reduce the required bits for coding the motion information. The newly developed coding tools and prediction modes described in this chapter have resulted in approximately 50\% of bitrate savings for the same subjective quality in comparison with the previous standard. However, such high coding efficiency implies higher data dependencies, which are more sensible to data loss and increase the error propagation to subsequent coding structures. Therefore, evaluating the impact of transmission errors and error propagation in \ac{hevc} is of most importance to develop efficient techniques to recover from data loss and reduce the error propagation. diff --git a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex index c9dfc57..0888e2f 100644 --- a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex +++ b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex @@ -83,7 +83,7 @@ The percentage of inter-coded blocks and the bitrates are also shown in the tabl \input{031_ErrorResilienceEvaluation/TablesSequences.tex} -For the simulation carried out in this study the HM reference software, version $16.2$, was used \cite{HEVCReferenceSoftware162}. +For the simulation carried out in this study the \ac{hm} reference software, version $16.2$, was used \cite{HEVCReferenceSoftware162}. The sequences were encoded with an \ac{idr} period of $16$ frames and all the coding modes enabled, following two recommended configurations: \ac{ld}, i.e., P-frames with 4 references frames, and \ac{ra}, i.e., B-frames with 2 reference frames in each list~\cite{HEVCCommonTests2012}. These prediction structures cover the use of P-frames and hierarchical B-frames~\cite{H264ScalableOverviewSep2007}, which are commonly used in HEVC streaming~\cite{RealTimeHEVC2016}. In order to achieve high quality images for all coded sequences (i.e., \ac{psnr} around 40 dB) different bitrates were used, as listed in Table~\ref{TableTestSequences}. The filtering operations were disabled at slices boundaries, in order to keep the slices self-contained. diff --git a/050_Conclusions/Conclusion.tex b/050_Conclusions/Conclusion.tex index 00023c0..a341b3f 100644 --- a/050_Conclusions/Conclusion.tex +++ b/050_Conclusions/Conclusion.tex @@ -53,7 +53,7 @@ Therefore, an adaptive scheme based on six input parameters and a pre-trained \a Results revealed that using the proposed adaptive scheme, an efficient trade-off between coding efficiency and error resilience is achieved for the majority of test conditions. Using this architecture, the problem of robust video coding and decoding was addressed using a combination of different methods, which leads to a superior performance in comparison with individual techniques. -As a final remark, one may conclude that the use of the novel methods proposed in this thesis consistently contribute to increase the robustness of video communications using state-of-art high efficiency video encoding tools are error prone networks. +As a final remark, one may conclude that the use of the novel methods proposed in this thesis consistently contribute to increase the robustness of video communications using state of the art high efficiency video encoding tools are error prone networks. The best performance is achieved by combining robust video coding techniques with efficient error concealment algorithms optimised at the encoder, using an optimised dynamic approach. diff --git a/References.tex b/References.tex index bb7fa66..538bebf 100644 --- a/References.tex +++ b/References.tex @@ -1,2 +1,2 @@ \bibliographystyle{IEEEtran} -\bibliography{../../References//ResilienceDataPartitioning,../../References//ResilienceROPE,../../References//NeuralNetworks,../../References//Resilience,../../References//OpticalFlow,../../References//ResilienceRedundant,../../References//VideoStreaming,../../References//HEVC,../../References//HEVCStreamErrors,../../References//H264,../../References//ResilienceDistortion,../../References//ResilienceLocalisation,../../References//ResilienceIntraRefresh,../../References//Concealment,../../References//VideoTransmissionStandards,../../References//Personal,../../References//ResilienceRPS,../../References//Misc,../../References//HEVCErrorResilience,../../References//ResilienceLeaky,../../References//ConcealmentDriven,../../References//ResilienceUEP,../../References//ConcealmentHEVC} +\bibliography{../../Deliveries/References/Concealment,../../Deliveries/References/ResilienceUEP,../../Deliveries/References/ResilienceDistortion,../../Deliveries/References/ResilienceLocalisation,../../Deliveries/References/HEVCStreamErrors,../../Deliveries/References/ResilienceIntraRefresh,../../Deliveries/References/OpticalFlow,../../Deliveries/References/VideoStreaming,../../Deliveries/References/Resilience,../../Deliveries/References/HEVC,../../Deliveries/References/Misc,../../Deliveries/References/NeuralNetworks,../../Deliveries/References/VideoTransmissionStandards,../../Deliveries/References/ConcealmentHEVC,../../Deliveries/References/ResilienceROPE,../../Deliveries/References/ResilienceDataPartitioning,../../Deliveries/References/HEVCErrorResilience,../../Deliveries/References/ResilienceRedundant,../../Deliveries/References/ResilienceLeaky,../../Deliveries/References/ResilienceRPS,../../Deliveries/References/Personal,../../Deliveries/References/ConcealmentDriven,../../Deliveries/References/H264} diff --git a/carreira_phd_thesis.sty b/carreira_phd_thesis.sty index 436ca9b..d780a18 100644 --- a/carreira_phd_thesis.sty +++ b/carreira_phd_thesis.sty @@ -117,7 +117,7 @@ \usepackage{balance} % *** ABREVIATIONS *** -\usepackage[acronym,toc,nomain,nonumberlist]{glossaries} +\usepackage[acronym,toc,nomain]{glossaries} %\newcommand{\customnewacronym}[4][]{\newacronym[#1,description=#4]{#2}{#3}{\MakeLowercase{#4}}} \newcommand{\customnewacronym}[4][]{\newacronym[#1,description=#4]{#2}{\mbox{#3}}{#4}} \setglossarystyle{long} diff --git a/carreira_phd_thesis.tex b/carreira_phd_thesis.tex index c7bc0b6..adad725 100644 --- a/carreira_phd_thesis.tex +++ b/carreira_phd_thesis.tex @@ -71,16 +71,19 @@ \glsunset{itut} \glsunset{iso} \glsunset{iec} +\glsunset{rtp} +\glsunset{ts} +\glsunset{dash} \mainmatter \include{010_Introduction/Introduction} \include{021_VideoCoding/VideoCodingReview} -\include{022_ErrorResilience/RobustTransmissionReview} -\include{031_ErrorResilienceEvaluation/ErrorResilienceEvaluation} -\include{041_ImprovedErrorResilience/ImprovedErrorResilience} -\include{042_ConcealmentAwareEncoding/ConcealmentAwareEncoding} -\include{043_EndToEndRobustSystem/EndToEndRobustSystem} -\include{050_Conclusions/Conclusion} +%\include{022_ErrorResilience/RobustTransmissionReview} +%\include{031_ErrorResilienceEvaluation/ErrorResilienceEvaluation} +%\include{041_ImprovedErrorResilience/ImprovedErrorResilience} +%%\include{042_ConcealmentAwareEncoding/ConcealmentAwareEncoding} +%\include{043_EndToEndRobustSystem/EndToEndRobustSystem} +%\include{050_Conclusions/Conclusion} \appendix \include{060_Appendix/Sequences} \include{060_Appendix/Papers} diff --git a/carreira_phd_thesis_abr.sty b/carreira_phd_thesis_abr.sty index ccfe4bd..0d65c13 100644 --- a/carreira_phd_thesis_abr.sty +++ b/carreira_phd_thesis_abr.sty @@ -5,16 +5,17 @@ \customnewacronym{qoe}{QoE}{Quality of Experience} \customnewacronym{qos}{QoS}{Quality of Service} \customnewacronym[longplural={frames per second},description=Frames per Second]{fps}{FPS}{frames per second} +\customnewacronym{mos}{MOS}{Mean Opinion Score} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Network %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \customnewacronym{rtp}{RTP}{Real-time Transport Protocol} \customnewacronym{ip}{IP}{Internet Protocol} -\customnewacronym{lte}{LTE}{Long Term Evolution} +\customnewacronym{ltea}{LTE-A}{Long Term Evolution - Advanced} \customnewacronym{mtu}{MTU}{Maximum Transfer Unit} \customnewacronym{mmt}{MMT}{MPEG Media Transport} \customnewacronym{ts}{MPEG-2 TS}{MPEG-2 Transport Stream} - +\customnewacronym{dash}{MPEG-DASH}{MPEG Dynamic Adaptive Streaming over HTTP} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Video Coding %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -23,6 +24,7 @@ \customnewacronym{iec}{IEC}{International Electrotechnical Commission} \customnewacronym{mpeg}{MPEG}{Moving Picture Experts Group} + \customnewacronym{vceg}{VCEG}{Video Coding Experts Group} \customnewacronym{jctvc}{JCT-VC}{Joint Collaborative Team on Video Coding} @@ -63,6 +65,7 @@ %% HEVC %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \customnewacronym{hevc}{HEVC}{High Efficiency Video Coding} +\customnewacronym{hm}{HM}{HEVC Test Model} \customnewacronym{vps}{VPS}{Video Paramenter Set} \customnewacronym{sps}{SPS}{Sequence Paramenter Set}