diff --git a/000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf b/000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf index bbb684b..2cc866e 100644 Binary files a/000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf and b/000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf differ diff --git a/000_Preamble/LU/Certificate_Originality_Thesis_Access.docx b/000_Preamble/LU/Certificate_Originality_Thesis_Access_v1.docx similarity index 100% rename from 000_Preamble/LU/Certificate_Originality_Thesis_Access.docx rename to 000_Preamble/LU/Certificate_Originality_Thesis_Access_v1.docx diff --git a/000_Preamble/LU/Certificate_Originality_Thesis_Access_v2.docx b/000_Preamble/LU/Certificate_Originality_Thesis_Access_v2.docx new file mode 100644 index 0000000..af46fe3 Binary files /dev/null and b/000_Preamble/LU/Certificate_Originality_Thesis_Access_v2.docx differ diff --git a/000_Preamble/LU/access.pdf b/000_Preamble/LU/access.pdf deleted file mode 100644 index 2cb1ea0..0000000 Binary files a/000_Preamble/LU/access.pdf and /dev/null differ diff --git a/000_Preamble/LU/origin.pdf b/000_Preamble/LU/origin.pdf deleted file mode 100644 index 5b0ace5..0000000 Binary files a/000_Preamble/LU/origin.pdf and /dev/null differ diff --git a/000_Preamble/abstract.tex b/000_Preamble/abstract.tex index 2f3699a..b4ee5ad 100755 --- a/000_Preamble/abstract.tex +++ b/000_Preamble/abstract.tex @@ -2,27 +2,28 @@ \selectlanguage{english} \chapter*{Abstract \VERSION{0}} -\addcontentsline{toc}{chapter}{Abstract \VERSION{0}} +\addcontentsline{toc}{chapter}{Abstract \VERSION{3}} -\vspace{-6mm} +\vspace{-5mm} This thesis investigates the problem of robust coding and \ac{ec} in highly efficient video coders. -Firstly, a simulation study of the error robustness of the \ac{hevc} is presented, focusing on the new tools adopted. Experimental results reveal the lack of protection against errors, which leads to significant quality degradation. -Based on this evidence, a new method to reduce the temporal dependencies of \ac{mv} is proposed, improving the decoded video quality without compromising the compression efficiency. +After a review of the current state-of-art, a simulation study of the error robustness of the \ac{hevc} is presented. +Experimental results reveal the lack of protection against errors, which leads to significant quality degradation. +Based on this evidence, the first contribution of this work is a new method to reduce the temporal dependencies between \acs{mv}, by improving the decoded video quality without compromising the compression efficiency. -The second contribution of this thesis is a two-stage approach for reducing the mismatch temporal predictions. -At the encoder stage, the reference pictures are dynamically distributed based on a constrained Lagrangian optimization, thus reducing the number of predictions from a single reference. +The second contribution of this thesis is a two-stage approach for reducing the mismatch of temporal predictions in case of video streams received with errors or lost data. +At the encoder stage, the reference pictures are dynamically distributed based on a constrained Lagrangian \ac{rd} optimization to reduce the number of predictions from a single reference. At the streaming stage, a prioritization algorithm, based on spatial dependencies, selects a reduced set of MVs to be transmitted, as side information, to reduce mismatched MV predictions at the decoder. -Results show that this method achieves higher quality that the reference HEVC for low-packet-loss rates, also outperforms existing techniques. The problem of EC-aware video coding is also investigated to enhance the overall error robustness. -A new approach based on a scalable coding and optimally EC selection is proposed, where the optimal EC modes are found by simulating transmission losses, followed by a saliency-weighted optimisation of the signalling~rate/EC~distortion cost. +A new approach based on scalable coding and optimally EC selection is proposed, where the optimal EC modes are found by simulating transmission losses, followed by a saliency-weighted \ac{rd} optimisation. Moreover, EC residual information is encoded using a rate-controlled \ac{el}. Both are transmitted to the decoder to be used in case of data loss. -Simulation results show that the proposed method achieves consistent quality gains in comparison with reference methods, which can be increased by using the EL information. - Finally, an adaptive error resilience scheme is proposed to dynamically predict the video stream that achieves the highest decoded quality for a particular loss case. -A pre-trained neural network selects among the various video streams encoded with different levels of compression efficiency and error protection, based on different input parameters that include information from the video signal, the coded stream and the transmission network. -Results show the advantage of using a dynamic approach to achieve a better trade-off between coding efficiency and error robustness. +A pre-trained neural network selects among the various video streams, encoded with different levels of compression efficiency and error protection, based on different input parameters that include information from the video signal, the coded stream and the transmission network. +The new robust video coding method investigated in this thesis yield consistent quality gains in comparison with other existing methods and also the ones implemented in the \ac{hevc} reference software. +Furthermore, the trade-off between coding efficiency and error robustness is also better in the proposed methods. +%\vspace{2mm} +%\textbf{Keywords:} 3D video, frame loss error concealment, subjective quality assessment. \ No newline at end of file diff --git a/000_Preamble/dedic.tex b/000_Preamble/dedic.tex index 8b5ee78..7453f3b 100755 --- a/000_Preamble/dedic.tex +++ b/000_Preamble/dedic.tex @@ -4,7 +4,7 @@ Dedicated to my future wife, \textbf{Bianca} and my parents, \textbf{Antonio} and \textbf{Maria} -\vspace{15cm} +\vspace{12cm} Anyone who has never\\ made a mistake has never\\ tried anything new.\\ \vspace{0.5cm} diff --git a/000_Preamble/thanks.tex b/000_Preamble/thanks.tex index 0aa14e9..14a7c54 100755 --- a/000_Preamble/thanks.tex +++ b/000_Preamble/thanks.tex @@ -4,11 +4,11 @@ Firstly, I would like to express my gratitude to my advisors, Dr. Pedro Ant\'{o}nio Amado Assun\c{c}\~{a}o, Dr. S\'{e}rgio Manuel Maciel de Faria, Dr Erhan Ekmekcioglu and Dr. Ahmet Kondoz for the tireless support of my PhD studies and related research, for their motivation, patience, and immense knowledge. I am thankful for their guidance and availability that were essential for the proper conduct of this research work. I would also like to acknowledge Dr. Varuna de Silva for the initial support on this research~work. -I would like to thank Funda\c{c}\~{a}o para a Ci\^{e}ncia e a Tecnologia (FCT), which provided +I would like to acknowledge Funda\c{c}\~{a}o para a Ci\^{e}ncia e a Tecnologia (FCT), which provided the financial support for this PhD, under the PhD grant SFRH/BD/86368/2012. I am also grateful to the Instituto de Telecomunica\c{c}\~{o}es and to the Institute for Digital Technologies, which provided the computer resources and laboratory facilities with excellent conditions for my research work. -I would like to acknowledge my research colleagues, both in Portugal and United Kingdom, for the friendship and the wonderful work environment, for sharing new ideas, and also for the moments outside the laboratory. +I would like to thank my research colleagues, both in Portugal and United Kingdom, for the friendship and the wonderful work environment, for sharing new ideas, and also for the moments outside the laboratory. I also thank all my friends, that accompanied me during this journey, for the distracting and social moments. I must thank all communities around the world that contribute for the development of many free and open-source tools used in this research work, namely the GNU/Linux, KDE software, Ubuntu and Gentoo communities. I also acknowledge the help of Luis Lucas and Jo\~{a}o Santos in the Calyp software, Remoterun scripts and general computer management, which improved our developing capabilities and ease the experimental evaluations. diff --git a/010_Introduction/Introduction.tex b/010_Introduction/Introduction.tex index 096e0ad..fc3e3bf 100755 --- a/010_Introduction/Introduction.tex +++ b/010_Introduction/Introduction.tex @@ -1,19 +1,23 @@ \graphicspath{{010_Introduction}} \glsreset{hevc} -\chapter{Introduction \VERSION{2}} +\chapter{Introduction \VERSION{3}} \label{Chapter:Intro} \section{Context and motivation} \glsreset{h264} +\glsunset{itut} +\glsunset{iso} +\glsunset{iec} -Over the recent years new advances in video compression and networking technologies have enabled the development of many multimedia applications over the internet, such as video conferencing, video streaming, video publishing, among others. -The increasing diversity of services, the demands for high quality multimedia and the growing popularity of \ac{uhd} formats (\emph{e.g.}, $4k$ or $8k$ resolution and beyond) are pushing technologies forward and opening new research directions. -Moreover, the emerging new video formats with increased resolutions, such as, 360-degree video, have created stronger needs for higher coding efficiency, beyond the previous standards, i.e., the MPEG-4/\ac{h264}~\cite{H264AVCStandard2003,H264OverviewJul2003}. + +In recent years new advances in video compression and networking technologies have enabled the development of many multimedia applications over the internet, such as video conferencing, video streaming, video publishing, among others. +The increasing diversity of services, the demand for high quality multimedia and the introduction of \ac{uhd} formats (\emph{e.g.}, $4k$ or $8k$ resolution and beyond) are pushing forward existing technologies and opening new research directions. +Moreover, the emerging new video formats with increased resolutions have created stronger needs for higher coding efficiency, beyond the previous standards, i.e., the \ac{h264}~\cite{H264AVCStandard2003,H264OverviewJul2003}. Such need has been driven not only by higher resolutions but also by stereoscopic and multi-view video formats, which increase the number of viewpoints that need to be encoded and delivered simultaneously. -Moreover, the ever increasing traffic generated by mobile multimedia applications \cite{CiscoMobile1621_Feb2017} and different video services with ever increasing popularity, e.g., Netflix, impose new challenges to the existing networks. -To cope with such demanding requirements a new coding standard was developed by the ITU-T \ac{vceg} and the \ac{iso}/\ac{iec} \ac{mpeg} standardisation organisations, referred to as H.265 - \ac{hevc}~\cite{H265StandardFeb2018,HEVCOverviewDez2012}. +Moreover, the ever increasing traffic generated by mobile multimedia applications \cite{CiscoMobile1621_Feb2017} and different video services with growing popularity, e.g., Netflix, impose new challenges to the existing networks. +To cope with such demanding requirements a new coding standard was developed by the \ac{itut} \ac{vceg} and the \ac{iso}/\ac{iec} \ac{mpeg} standardisation organisations, referred to as H.265 - \ac{hevc}~\cite{H265StandardFeb2018,HEVCOverviewDez2012}. \glsunset{h264} @@ -21,64 +25,53 @@ The \ac{hevc} is currently the main video coding framework to extend current ser Similar to its predecessor, \ac{hevc} is a block-based hybrid video codec using predictive, transform and entropy coding. The coding flexibility and efficiency of \ac{hevc} results from the use of new coding tools and block structures, which enables new block partitions\cite{HEVCBlock2012}, improved prediction modes \cite{IntraCodingHEVC2012,AMPCodingHEVCJun2012} and new high-level features~\cite{HEVCHighLevelReferenceDez2012}, such as explicit reference picture management and new parameter sets. However, it is known that higher coding efficiency leads to lower robustness in the presence of transmission errors and data loss. -Therefore, the \ac{hevc} new coding tools bring not only high compression efficiency~\cite{HEVCComparisonDec2012}, but also some other disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and inherent reduction of error robustness. -%\cite{StudyHEVCErrorsDez2012} -Earlier studies about the error robustness characteristics of \ac{hevc} have shown that, in general, the use of high efficiency coding tools results in poor error resilience performance, in comparison with previous standards. -Thus, when \ac{hevc} streams incur in transmission losses, this leads to significant degradation of both objective and subjective quality~\cite{QoEHEVCMay2014,SubjectiveEvalSep2013}. +Therefore, new coding tools of \ac{hevc} not only bring high compression efficiency~\cite{HEVCComparisonDec2012}, but also some other disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and inherent reduction of error robustness~\cite{StudyHEVCErrorsDez2012}. +Earlier studies about the error robustness characteristics of \ac{hevc} have shown that, in general, the use of high efficiency coding tools results in poor error resilience performance, in comparison with previous standards~\cite{QoEHEVCMay2014,SubjectiveEvalSep2013}. +Thus, when \ac{hevc} streams incur in transmission losses, this leads to significant degradation of both objective and subjective quality. This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes of \ac{hevc}, which assume \mbox{error-free} transmission~\cite{ROPE2007}. A possible approach to encode robust video streams is to increase the error resilience of \ac{hevc} by including \mbox{error-aware} mechanisms in the coding process. As a result, the effect of error propagation, due to prediction mismatch at the decoding side, will be limited. -The widespread use of \ac{hevc} in broadcast and multimedia delivery services is based on transport technologies such as \ac{rtp}~\cite{HEVCStreamMay2012}, \mbox{MPEG-2 TS}~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016}. -Recent developments also enable combinations of different technologies, e.g, \mbox{HTTP-based} streaming and \mbox{MPEG-2 TS}, to increase the flexibility of over-the-top (OTT) video services\cite{Broadcast2017}. For content delivery over heterogeneous networks, the \ac{mmt}~\cite{ATSCMMT2016} is another possible technology to be used with the \ac{hevc} standard. Besides the transport technologies, -the overall error robustness of video transmission may also benefit from the increasing flexibility of multimedia systems to cope with heterogeneous networks, by using auxiliary streams that allow the video decoder to increase the quality of the recovered lost slices~\cite{ErrorResilienceHeterogenousNetworks2016,HeterogeneousStreaming2016}. Moreover, error resilience techniques may also take advantage of the multi-stream based approach, where each stream allows different levels of robustness to errors~\mbox{\cite{HEVCHTTPStreamAug2016,PerfMediaTransport2017}}. +\glsunset{rtp} +\glsunset{ts} + +The widespread use of \ac{hevc} in broadcast and multimedia delivery services is based on transport technologies such as \ac{rtp}~\cite{HEVCStreamMay2012}, \ac{ts}~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016}. +Recent developments also enable combinations of different technologies, e.g, \mbox{HTTP-based} streaming and \ac{ts}, to increase the flexibility of over-the-top video services\cite{Broadcast2017}. For content delivery over heterogeneous networks, the \ac{mmt}~\cite{ATSCMMT2016} is another possible technology to be used with the \ac{hevc} standard. Besides the transport technologies, +the overall error robustness of video transmission may also benefit from the increasing flexibility of multimedia systems to cope with heterogeneous networks, by using auxiliary streams that allow the video decoder to increase the quality of the recovered lost slices~\cite{ErrorResilienceHeterogenousNetworks2016,HeterogeneousStreaming2016}. +Moreover, error resilience techniques may also take advantage of the multi-stream based approaches, where each stream allows for different levels of error robustness~\cite{HEVCHTTPStreamAug2016,PerfMediaTransport2017}. -%Despite the different characteristics of the existing services and applications, the users' \ac{qoe} consistently depends on how the compressed streams are adapted to the diverse networking technologies, as well as on how the transmission errors and/or packet losses are handled at the application level ~\cite{HEVCStreamMay2012,HEVCLayerIntegrationDez2012,StudyHEVCErrorsDez2012}. As mentioned before, the \ac{hevc} standard is expected to replace the \ac{h264} in future video services. - - -%The system interface of \ac{hevc} is an important component -%in the media access chain, and a prerequisite for its successful -%deployment in various consumer electronics applications. We -%consider four broad categories of systems: broadcast delivery -%over digital television and Internet Protocol Television (IPTV) -%channels including video-on-demand (VoD), streaming over -%the Internet using the Hypertext Transport Protocol (HTTP), -%real-time communications such as video conferencing, and -%store-forward delivery based on a file or a physical memory -%delivered to the end user (such as: camcorded videos or Blu- -%ray Disks). To this end, different system technologies have -%been standardized. These technologies can be summarized as -%follows. - - - - -In summary, increasing the error robustness of the highly compressed video streams is a relevant research problem, whose solutions will be useful to increase the adaptability for transmission over diverse network conditions and to achieve robust communications, thus providing acceptable levels of \ac{qoe} to the end users at all times. -In general, increased error robustness in video transmission can be jointly achieved at three different stages: (i) using robust video encoding, by selecting robust coding modes, (ii) after encoding, by adding robust features to the compressed streams, as side-information, and (iii) using robust video decoding techniques, e.g., efficient \ac{ec}. This is exemplified in Figure~\ref{FIG1:RobustVideoTransmission} where at the encoder different layers are used to increase the error robustness, which results in a video stream composed by a \ac{vcl} and \ac{sei} layers. Finally, at the decoder two processing layers are used to achieve robust video decoding. +In summary, increasing the error robustness of the highly compressed video streams is a relevant research challenge, whose solutions will be useful to increase the adaptability for transmission over diverse network conditions and also cope with communication errors, providing increased levels of \ac{qoe} to the end users at all times. +In general, increased error robustness in video transmission can be jointly achieved at three different stages: (i) using robust video encoding, by selecting robust coding modes, (ii) after encoding, by adding robust features to the compressed streams, as side-information, and (iii) using robust video decoding techniques, e.g., efficient \ac{ec}. This is exemplified in Figure~\ref{FIG1:RobustVideoTransmission} where at the encoder different layers are used to increase the error robustness, which results in a video stream composed by a \ac{vcl} and \ac{sei} layers. At the decoder two processing layers are used to achieve robust video decoding. The \ac{hevc} reference codec is continuously being improved and extended by multiple proponents from around the world, which also address the problem of error resilience and robust delivered transmission. -Although different \ac{ec} methods have been developed during the past years for \ac{h264}~\cite{2DRecursiveRefinement2010} and recently for \ac{hevc}~\cite{HEVCMCBlockAug2013}, a combined and optimised approach involving the encoder and decoder is yet to be developed, which would improve the users’ \ac{qoe}, by exploiting shared information between the encoder and the decoder-sides. +Although different \ac{ec} methods have been developed during the past years for \ac{h264}~\cite{2DRecursiveRefinement2010} and more recently for \ac{hevc}~\cite{HEVCMCBlockAug2013}, a combined and optimised approach involving the encoder and decoder has never been fully exploited. This would improve the users’ \ac{qoe}, by exploiting shared information between the encoder and the decoder-sides. \begin{figure}[tb] \centering \input{./010_Introduction/RobustVideoSystem.tikz} - \caption{Generalised diagram of a robust video transmission system.} + \caption{Generic robust video transmission system.} \label{FIG1:RobustVideoTransmission} \end{figure} \section{Main objectives and contributions} -The research objectives of this thesis are focused in robust error resilience for efficient error concealment in highly efficiency video decoders. +The research objectives of this thesis are defined within the scope of robust error resilience for efficient error concealment in highly efficiency video decoders. Thus, this work investigates novel techniques to increase the video quality of \ac{hevc} bitstreams delivered through error-prone networks. The following objectives were pursued in this work: \begin{enumerate} \item -\textbf{evaluation and study of the error robustness:} The first objective of this research work is to evaluate the robustness against transmission errors of highly efficiently coded video. The study conclusions aimed at finding relevant research directions to increase the error robust of coded streams. The results and conclusions of this study were published in conference paper (\ref{Conf:ErrorResilience}) and in journal paper (\ref{Journal:TwoStage}) (see Appendix~\ref{Appen:Papers}). +\textbf{study and evaluation of error robustness in highly compressed video:} +The objective of this research study is to evaluate the robustness against transmission errors of highly efficiently coded video. +The study and its conclusions aim at finding relevant research directions to develop new methods for increasing the error robust of coded streams. +The results and conclusions of this study were published in conference paper (\ref{Conf:ErrorResilience}) and in journal paper (\ref{Journal:TwoStage}) (see Appendix~\ref{Appen:Papers}). \item -\textbf{robust video coding:} At the encoder-side, novel techniques to extend the existing error robustness of highly predictive video coding techniques are investigated in order to increase the received video quality over error-prone networks. This research topic resulted in a new method to reduce the error propagation in motion vector coding algorithms, which was published in conference paper (\ref{Conf:DynamicTMVP}). Moreover, a novel two-stage approach was devised to improve the error robustness of highly efficiently coded video at the encoder and streaming levels, resulting in two conference publications (\ref{Conf:MotionVectorRedundancies} and~\ref{Conf:Chkb}) and a journal publication (\ref{Journal:TwoStage}). +\textbf{robust video coding:} +At the encoder-side, novel techniques to extend the existing error robustness of highly predictive video coding techniques are investigated in order to increase the received video quality over error-prone networks. +This research topic resulted in a new method to reduce the error propagation in motion vector coding algorithms, which was published in conference paper (\ref{Conf:DynamicTMVP}). +Moreover, a novel two-stage approach was devised to improve the error robustness of highly efficiently coded video at the encoder and streaming levels, resulting in two conference publications (\ref{Conf:MotionVectorRedundancies} and~\ref{Conf:Chkb}) and a journal publication (\ref{Journal:TwoStage}). -\item \textbf{robust video decoding:} error concealment techniques are used to increase the error robustness of video decoder and improve the perceived video quality. -The paradigm of error concealment is evaluated at the encoder-side, as part of the encoding process, in order to optimise the reconstruction process at the decoder-side. This research resulted in a novel error concealment-aware technique to optimise the error concealment process at the encoder-side, which was proposed for intra-frame only in a conference paper (\ref{Conf:ConcAware}) and for a generic robust video transmission in the journal (\ref{Journal:ConcAware}). +\item \textbf{robust video decoding:} error concealment techniques are used to increase the error robustness of video decoding and improve the perceived video quality. +The paradigm of \ac{ec} is evaluated at the encoder-side, as part of the encoding process, in order to optimise the reconstruction process at the decoder-side. This research resulted in a novel error concealment-aware technique to optimise the error concealment process at the encoder-side, which was proposed for intra-frame only in a conference paper (\ref{Conf:ConcAware}) and for generic robust video transmission in the journal (\ref{Journal:ConcAware}). %\item \textbf{jointly error resilience and concealment:} the ultimate goal of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet higher quality demands for different video contents characteristics and network conditions. \end{enumerate} @@ -93,16 +86,16 @@ This thesis is organised as follows. The current chapter introduces the context, The first two chapters review the \mbox{state-of-the-art} concepts and previous work related with this research. Chapter~\ref{Chapter:HEVCReview} presents an overview of the \ac{hevc} standard, focusing on the concepts explored in this work, e.g., high-level features and temporal predictions. Chapter~\ref{Chapter:ErrorResilienceReview} reviews the most important work dealing with robust video transmission also providing relevant background knowledge to this work. -Chapter~\ref{Chapter:HEVCErrorResilience} firstly presents a study about the error robustness of high efficiently coded video, using the \ac{hevc} as the underlying framework for various coding configuration. +Chapter~\ref{Chapter:HEVCErrorResilience} firstly presents a study about the error robustness of high efficiently coded video, using the \ac{hevc} as the underlying framework for various coding configurations. The study presents the relevant tools that mostly affect the error robustness. Secondly, a new method is proposed to decrease the vulnerability of temporal motion prediction and decrease the error propagation, leading to higher video quality under error-prone conditions. Chapter~\ref{Chapter:ErrorResilienceTechniques} proposes a novel error resilience approach for high efficiently video codecs. A new method to distribute the use of reference frames is proposed to reduce the error propagation and improve the \ac{ec} performance. -Moreover, to reduce the effect of mismatch motion predictions (shown in Chapter~\ref{Chapter:HEVCErrorResilience}) a method based on selective motion vector redundancies is proposed. +Moreover, to reduce the effect of mismatched motion predictions (shown in Chapter~\ref{Chapter:HEVCErrorResilience}), a method based on selective motion vector redundancies is proposed. This method is able to improve the decoded video quality using a small overhead. -Chapter~\ref{Chapter:ConcealmentAwareTechniques} addresses the problem of robust \ac{ec} of highly efficiently video streams. +Chapter~\ref{Chapter:ConcealmentAwareTechniques} addresses the problem of robust \ac{ec} of high efficiently coded video streams. %This work was motivated by the requirements for more accurate \ac{ec} algorithms which are able to reduce the error propagation. Different \ac{ec} algorithms were taken into consideration in order to develop a concealment-aware coding scheme. The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled to the decoder. @@ -112,10 +105,11 @@ A generalised saliency-weighted \ac{rd} optimisation is used and the residue be %The simulation results show that the proposed method achieves consistent quality gains in comparison with other reference methods and previous works. -Chapter~\ref{Chapter:EndToEndRobustSystem} proposes a novel adaptive error robustness method, in order to combine different techniques proposed in this thesis. At the encoder-side various video streams are generated with different levels of compression efficiency and error robustness. Then, a pre-training \ac{nn} is used to predict which video stream will achieve the highest decoded video quality for a particular scenario. The selection is based on different input parameters which include information from the video signal, the coded stream and the transmission network. - -Chapter~\ref{Chapter:Conclusions} concludes this dissertation and presents some suggestions for future work. +Chapter~\ref{Chapter:EndToEndRobustSystem} proposes a novel adaptive error robustness method, which combines the different techniques proposed in this thesis. +At the encoder-side various video streams are generated with different levels of compression efficiency and error robustness. +Then, a pre-training \ac{nn} is used to predict which video stream is expected to achieve the highest decoded video quality for a particular scenario. The selection is based on different input parameters, which include information from the video signal, the coded stream and the transmission network. +Finally, Chapter~\ref{Chapter:Conclusions} concludes this dissertation and presents some suggestions for future work. Appendix~\ref{Appen:Sequences} illustrates the original test signals used in simulations throughout the thesis. This includes a partial set of the \ac{hevc} test material and other external test material. Appendix~\ref{Appen:Papers} presents a summary of the published papers, describing the contributions of the research work of this thesis. diff --git a/010_Introduction/RobustVideoSystem.tikz b/010_Introduction/RobustVideoSystem.tikz index f66fe83..c5da38b 100644 --- a/010_Introduction/RobustVideoSystem.tikz +++ b/010_Introduction/RobustVideoSystem.tikz @@ -53,8 +53,8 @@ row sep=-\pgflinewidth, column sep=-\pgflinewidth, nodes={rectangle,text width=1.5cm,align=center}, - text depth=1ex, - minimum height=2mm, + text depth=1mm, + minimum height=1mm, nodes in empty cells } } @@ -66,7 +66,7 @@ }; % The labels -\node[text centered, xshift=1cm] at (arrow-1-1) {\footnotesize SEI}; +\node[text centered, xshift=1cm] at (arrow-1-1) {\scriptsize SEI}; \node[text centered, xshift=1cm, text width=2.2cm] at (arrow-2-1.south) {\footnotesize VCL}; @@ -92,9 +92,9 @@ (last3.south) -- (last2.east) -- (last1.north) -- cycle; -\path[decoration={markings, mark=at position .5 with \node[transform shape] {\scriptsize Video};}, postaction={decorate}] +\path[decoration={markings, mark=at position .5 with \node[transform shape] {\tiny Video};}, postaction={decorate}] ( $ (last1.north west)!0.5!(last1.north) $ ) -- ( $ (last2.center)!0.5!(last2.east) $ ); -\path[decoration={markings, mark=at position .5 with \node[transform shape] {\scriptsize Stream};}, postaction={decorate}] +\path[decoration={markings, mark=at position .5 with \node[transform shape] {\tiny Stream};}, postaction={decorate}] ( $ (last2.center)!0.5!(last2.east) $ ) -- ( $ (last3.south west)!0.6!(last3.south) $ ); diff --git a/021_VideoCoding/Figures/PictureTypes_TSA.tikz b/021_VideoCoding/Figures/PictureTypes_TSA.tikz index 92defa8..cbe4ece 100644 --- a/021_VideoCoding/Figures/PictureTypes_TSA.tikz +++ b/021_VideoCoding/Figures/PictureTypes_TSA.tikz @@ -30,7 +30,7 @@ \path[dependency] (P0.south west)++(-1,-.4) -- ++(0,+4.2cm) node[midway, anchor=center,fill=white, rotate=90] {Layer}; \node[label, left=.1 of P0] (T0) {T$_0$}; -\node[label] at (T0 |- P1) {T$_1$}; -\node[label] at (T0 |- P2) {T$_2$}; +\node[label] at (T0 |- P2) {T$_1$}; +\node[label] at (T0 |- P1) {T$_2$}; \end{tikzpicture} diff --git a/021_VideoCoding/VideoCodingReview.tex b/021_VideoCoding/VideoCodingReview.tex index f7be503..dde72c2 100755 --- a/021_VideoCoding/VideoCodingReview.tex +++ b/021_VideoCoding/VideoCodingReview.tex @@ -3,7 +3,7 @@ \renewcommand{\FigsPath}{./021_VideoCoding/Figures} % \chapter{\HIGHLIGHT{The \ac{hevc} Standard}} -\chapter{The High Efficiency Video Coding Standard \VERSION{2}} +\chapter{The High Efficiency Video Coding standard \VERSION{3}} %\chapter{\HIGHLIGHT{The State-of-Art Video Coding}} \label{Chapter:HEVCReview} @@ -15,7 +15,7 @@ \glsunset{h264} \glsreset{hevc} -This chapter describes the H.265~-~\ac{hevc}~\cite{H265StandardFeb2018} standard, focusing on the most relevant features of the encoding algorithm, in the scope of the work developed in this thesis. +This chapter presents an overview of the H.265~-~\ac{hevc}~\cite{H265StandardFeb2018} standard, focusing on the most relevant features of the encoding algorithm, considering the scope of the work developed in this thesis. In short, the \ac{hevc} standard essentially aims to support all existing applications of \ac{h264}~\cite{H264AVCStandard2003}, while extending two keys aspects: increased video resolution with higher compression ratio and also more efficient use of parallel processing architectures. In order to increase the coding flexibility and higher video resolutions, the \ac{hevc} standard adopts a new block partition structure, as well as, new prediction modes. In this chapter the new data structures and coding techniques, specifically introduced to increase the coding efficiency, are described. @@ -23,7 +23,7 @@ Particular emphasis is given to those which may have a significant impact in the \glsunset{hevc} -This chapter is organised as follows. +The chapter is organised as follows. Section~\ref{Section2:GeneraVideoSystem} introduces the functional blocks of a generic hybrid video coding system and Section~\ref{Section2:Overview} provides an overview of the \ac{hevc} standard. Then, Section~\ref{Section2:HighLevel} describes the high-level syntax elements, which are used to carry relevant information in coded streams. Section~\ref{Section2:CodingStructures} describes the data structures used to divide the video frames into smaller partitions and Section~\ref{Section2:PredictionModes} describes particular aspects of the main predictions techniques, namely intra and inter coding. @@ -38,10 +38,10 @@ Section~\ref{Section2:Summary} concludes the chapter. \label{Section2:GeneraVideoSystem} A digital video signal is a temporal succession of images, also referred to as pictures or frames, which usually presents a significant amount of temporal and spatial correlations. -In general, video coding and distribution systems follow through a generic chain of processing and delivery stages: pre-processing, encoding, storage or transmission, decoding and post-processing. +In general, video coding and distribution systems follow through a generic chain of processing and delivery stage, comprising pre-processing, encoding, storage or transmission, decoding and post-processing, as shown in Figure~\ref{Figure2:HybridCodec}. The pre-processing stage is used adapt the raw video signal, either analogue or digital, to an appropriate format, such as RGB or YUV~\cite{DataCompression1997}. Then, the video data is compressed by the respective encoding module, mostly reducing its redundancy and visual irrelevancy. The encoding process is responsible for representing the raw video in a highly compressed format, while maintaining a good trade-off between video quality and bitrate through high efficient algorithms and coding tools. -In the last stages of the chain, the compressed format is decoded and presented by displaying the signal content after adaptation to the consumer equipment through post-processing. +In the last stages of the chain, the compressed format is decoded and presented by displaying the signal content after adaptation to the consumer equipment through post-processing, when necessary. % \begin{figure}[tb] % \centering @@ -53,8 +53,8 @@ In the last stages of the chain, the compressed format is decoded and presented Since the early days of video compressing, hybrid video coding has been the basic structure for all video coding standards and recommendations of ITU-T \ac{vceg} and \ac{iso}/\ac{iec} \ac{mpeg}, e.g., from H.261 to \ac{hevc}. -Figure~\ref{Figure2:HybridCodec} shows the block diagram of a generic hybrid video coding system~\cite{BookLucas2017}. -While the algorithmic structure has not been changed, the specific techniques represented by the building blocks have been refined and the applicable coding configuration has become more and more flexible over years, through the use of an increasing variety of coding tools. +Figure~\ref{Figure2:HybridCodec} also shows the block diagram of a generic hybrid video coding system~\cite{BookLucas2017}. +While the generic algorithmic structure has not been changed, the specific techniques represented by the building blocks have been refined and the applicable coding configuration has become more and more flexible over years, through the use of an increasing variety of coding tools. This is the reason why the coding scheme is called hybrid, as it combines quiet different prediction modes between pictures with transform and entropy coding~\cite{VideoCodingFundamentals2015}. \begin{figure}[t] @@ -68,7 +68,8 @@ This is the reason why the coding scheme is called hybrid, as it combines quiet In general, there are two kinds of redundancy in video signal: spatial and temporal. On the one hand, the spatial redundancy is exploited through intra frame prediction, where the underlying mechanism is to find the best matching block for the current one, by searching the neighbouring region. -This is possible due to a high correlation between neighbouring pixels within a frame. As result, in intra frame prediction, a block of pixels can be reconstructed from the blocks previously processed. +This is possible due to the existence of high correlation between neighbouring pixels within a frame. +As result, when intra frame prediction is used, a block of pixels is reconstructed from the neighbouring blocks previously processed. On the other hand, inter frame prediction is designed to exploit the temporal redundancy that exists between consecutive frames, such as those temporally adjacent, e.g., the past frame or the future frame in regard to the current one. Temporal redundancy is normally referred to as the correlation between two adjacent frames since they usually have quiet similar visual content, such as foreground objects and background. Thus, the purpose of inter-frame prediction is to search for the best matching block for the current one in a set of reference frames. @@ -92,16 +93,18 @@ In the next section, the aforementioned blocks are described in the context of t \glsreset{cu} \glsreset{pu} \glsreset{tu} -\glsreset{nalu} +\glsreset{nal} +%\glsreset{nalu} \glsreset{cabac} \glsreset{rpset} -Similar to the previous coding standards developed by JCT-VC, in \ac{hevc}~\cite{HEVCOverviewJul2012,HEVCOverviewDez2012,BookHEVC2014} only the +Similar to the previous coding standards developed by \ac{jctvc}, in \ac{hevc}~\cite{HEVCOverviewJul2012,HEVCOverviewDez2012,BookHEVC2014} only the coding syntax and semantics are standardised. The constrains and functionalities are defined through profiles and levels, while the semantic meaning of syntax elements define the decoding process, such that every decoder conforming to the standard generates the same output when given a complaint stream. -Such delimitation of scope in the standard allows maximum freedom for developers and manufactures to develop proprietary optimisations to cope with various applications’ requirements (\emph{e.g.}, balancing the compression ratio and the coding complexity). +Such delimitation of scope in the standard concede maximum freedom for developers and manufactures to develop proprietary optimisations, allowing to cope with various applications’ requirements (\emph{e.g.}, balancing the compression ratio and the coding complexity). Figure~\ref{fig:HEVCBlockDiagram} illustrates the block diagram of the \ac{hevc} encoder. It follows the same overall hybrid architecture as its predecessors starting from H.261. -The encoder uses both intra- and inter-picture prediction to exploit spatial and temporal redundancies, respectively. Subsequently, the prediction residue transformed by linear spatial transform, which is then quantised, and entropy coded, allowing to further reduce the bitstream redundancy. +The encoder uses both intra- and inter-picture prediction to exploit spatial and temporal redundancies, respectively. +Subsequently, the prediction residue is transformed by linear spatial transform and the resulting coefficients are then quantised and entropy coded, further reducing the bitstream redundancy. %The resulting bitstream is combined with control information, which signals the prediction chosen, motion vectors, filtering operations and bitstream headers. \begin{figure}[tb] @@ -113,8 +116,9 @@ The encoder uses both intra- and inter-picture prediction to exploit spatial and \end{figure} However, there are some novelties introduced in the \ac{hevc} standard. As depicted in Figure~\ref{fig:HEVCBlockDiagram}, the encoding process follows a block-based approach, therefore each picture is partitioned into larger blocks, referred to as \acs{ctu}, with a maximum size of $64\times64$ pixels. -Subsequently, a dynamic quad-tree partitioning is used to divide the picture into \acs{cu}. These units are introduced as the basic processing units for prediction, and for the transform and quantisation operations, \acs{tu} are used. -In order to generate the prediction residue the \ac{hevc} may use one of 34 intra-prediction modes or take advantage of symmetric and asymmetric partitions for motion estimation and inter-prediction. +Subsequently, a dynamic quad-tree partitioning is used to divide the picture into \acs{cu}. +These units are introduced as the basic processing units for prediction, while for the transform and quantisation operations, \acs{tu} are used. +In order to generate the prediction residue, the \ac{hevc} may use one of 34 intra-prediction modes or take advantage of symmetric and asymmetric partitions for motion estimation and inter-prediction. The motion estimation is performed in the pictures listed in the \ac{rpset}, which was introduced in the \ac{hevc} to increase the robustness to errors and data loss of reference picture management. To improve the overall quality of the reconstructed pictures and smooth the coding artefacts, \ac{hevc} not only uses a deblocking filter but also a \ac{sao} filter. Finally, the control data, quantised coefficients, prediction modes, motion vectors and filter information are compressed using the \ac{cabac}. The compressed bitstream is then allocated to various \ac{nal} units. @@ -131,17 +135,20 @@ To improve the overall quality of the reconstructed pictures and smooth the codi \glsreset{vcl} This section describes the high-level components of the \ac{hevc} standard that provide support for signalling and stream partitioning, which are relevant elements in the context of robust video communications. -\ac{hevc} inherits several high-level features from \ac{h264}, e.g., the network abstraction layer, parameter sets, the use of \ac{poc} and \ac{sei} messages for auxiliary data. +\ac{hevc} inherits several high-level features from \ac{h264}, e.g., the \ac{nal}, parameter sets, the use of \ac{poc} and \ac{sei} messages for auxiliary data. However, some high-level features defined in \ac{h264} were not included, such as \ac{fmo}, redundant slices, arbitrary slice order, data partitioning and switching slices. -However, several new high-level features are introduced, such as the \ac{vps}, tiles and wavefront tools for parallel processing, dependent slices, for reduced delay, and a new reference picture management concept. +Futhermore, several new high-level features are introduced, such as the \ac{vps}, tiles and wavefront tools for parallel processing, dependent slices, for reduced delay, and a new reference picture management concept. Moreover, new pictures types were introduced to increase the random-access flexibility and temporal sub-layer switching. \subsection{Network Abstraction Layer} -Similar to \ac{h264}~\cite{H264AVCStandard2003}, the \ac{hevc} streams are based on \glsfirst{nal} for stream partitioning and synchronization, which is useful to deal with lossy network environments~\cite{H264OverIPJul2003}. Using this concept the stream is composed of different \ac{nal} units. The \ac{nal} concept provides the ability to map the \ac{vcl} data that represents coded video slices (or frames) on various transport layers, such as \ac{rtp}, MPEG-2 Systems and MPEG DASH. The~\ac{nal} units are classified into two categories: (i) \ac{vcl} and (ii) non-\ac{vcl}, as they carry coded video or associated data, respectively. +Similar to \ac{h264}~\cite{H264AVCStandard2003}, the \ac{hevc} standard defines an adaptation layer to cope with different transport protocols. +The \ac{nal} is used for stream partitioning and synchronization, which is useful to deal with lossy network environments~\cite{H264OverIPJul2003}. +Using this concept the video stream is composed of different \ac{nal} units. The \ac{nal} concept provides the ability to map the \ac{vcl} data that represents coded video slices (or frames) on various transport layers, such as \ac{rtp}, MPEG-2 Systems and MPEG DASH. The~\ac{nal} units are classified into two categories: (i) \ac{vcl} and (ii) non-\ac{vcl}, as they carry coded video or associated data, respectively. -The \ac{hevc} standard defines two bytes to describe the content of each \ac{nal} unit. -These bytes are used to identify the \ac{nal} type are transmitted using a fixed coding format length, i.e., without variable length codes. This allow extra error robustness but increases the bit overhead. Figure~\ref{Figure2:NalHeader} shows the structure of the \ac{hevc} \ac{nal} unit header. The first bit \textit{F}, referred to as \textit{forbiden\_bit}, is always zero in order to ensure that a start code can be detected across different protocol standards. This is followed by the \ac{nal} type, which identifies its content, enabling the decoder to either use it if required for a particular decoding purpose or discard it. Another syntax element introduced by the \ac{hevc} standard is the \textit{temporal\_id\_plus\_one} (TIDP), which defines the temporal layer where the \ac{nal} unit belongs to. This allows implicit temporal scalability (layers ranging from 0 to 6) by immediately discarding the \ac{nal} unit if it belongs to a layer higher than the desired one. In the header, six bits are reserved for future extensions, preventing the creation of extra \ac{nal} units for scalable and multiview extensions, as done in \ac{h264}. +The \ac{hevc} standard defines a two bytes header to describe the content of each \ac{nal} unit, which are transmitted using a fixed coding format length, i.e., without variable length codes. +This allow extra error robustness but increases the bit overhead. +Figure~\ref{Figure2:NalHeader} shows the structure of the \ac{hevc} \ac{nal} unit header. The first bit \textit{F}, referred to as \textit{forbiden\_bit}, is always zero in order to ensure that a start code can be detected across different protocol standards. This is followed by the \ac{nal} type, which identifies its content, enabling the decoder to either use it if required for a particular decoding purpose or discard it. Another syntax element introduced by the \ac{hevc} standard is the \textit{temporal\_id\_plus\_one} (TIDP), which defines the temporal layer where the \ac{nal} unit belongs. This allows implicit temporal scalability (layers ranging from 0 to 6) by immediately discarding the \ac{nal} unit if it belongs to a layer that is higher than the pre-defined one. In the header, six bits are reserved for future extensions, preventing the creation of extra \ac{nal} units for scalable and multiview extensions, as done in \ac{h264}. \begin{figure}[tb] \centering @@ -151,10 +158,10 @@ These bytes are used to identify the \ac{nal} type are transmitted using a fixed \rlap{\bitbox{#2}{\color{#1}\rule{\width}{\height}}}% \bitbox{#2}{#3}} \begin{bytefield}[bitwidth=20pt]{16} - % \bitheader{0-15} \\ + \bitheader{0-15} \\ \colorbitbox{TikZColorFill}{1}{F} & \colorbitbox{gray!60}{6}{NAL Type} & \colorbitbox{TikZColorFill}{6}{Reserved} & \colorbitbox{gray!60}{3}{TIDP} \end{bytefield} - \caption{Two bytes \ac{nal} unit header in HEVC.} + \caption{\ac{nal} unit header format in HEVC.} \label{Figure2:NalHeader} \end{figure} @@ -163,11 +170,14 @@ These bytes are used to identify the \ac{nal} type are transmitted using a fixed \glsreset{vps} The encoder uses a set of signalling packets as a robust method to share data with the decoder, referred to as parameter set units. This information was introduced in previous standards to deal with vulnerabilities in the transport of signalling information between the encoder and decoder, which may compromise the streaming applications. -Within this approach such set of parameters is multiplexed in the bitstream and repeated as many times as required by the application, e.g., streaming, storage among others. In \ac{hevc}, this structure is inherited from \ac{h264}\cite{H264OverIPJul2003} with modifications to cope with new coding tools. +In this approach, such set of parameters is multiplexed in the bitstream and repeated as many times as required by the application, e.g., streaming, storage among others. In \ac{hevc}, this structure is inherited from \ac{h264}\cite{H264OverIPJul2003} with modifications to cope with new coding tools. The \ac{hevc} includes three sets of parameters, \ac{sps}, \ac{pps} and the newly introduced \ac{vps}\cite{HEVCHighLevelReferenceDez2012}. % \ac{sps} include sequence related information, that include the decoder operation point, flags to enable or disable certain tools within a given profile. \Gls{pps} include picture related information, that may change for every picture, e.g., \ac{qp} and tilling information. -The new set of parameters \ac{vps} contains information related to the different layers of the video signal, avoiding duplications like in the \ac{h264} and its extensions. \Gls{vps} aims to provide signalling for every layer, guaranteeing an extendible standard, capable of supporting multiple layers. Moreover, \ac{vps} carries information for session negotiation (\emph{e.g.} profile, level and tier). The \ac{sps} contains information also related to the whole sequence (\emph{i.e.}, should affect all coded slices), and also related with the decoder operation point, as well as flags to control optional tools and scalability. The \ac{pps} is responsible to carry information that may change for every picture, such as, initial \ac{qp}, flags for picture related tools and tilling information. +The new set of parameters \ac{vps} contains information related to the different layers of the video signal, avoiding duplications like in the \ac{h264} and its extensions. \Gls{vps} aims to provide signalling for every layer, guaranteeing an extendible standard, capable of supporting multiple layers. +Moreover, \ac{vps} also carries information for session negotiation (\emph{e.g.} profile, level and tier). +The \ac{sps} contains information related to the whole sequence (\emph{i.e.}, should affect all coded slices) and related with the decoder operation point, as well as flags to control optional tools and scalability. +The \ac{pps} is responsible to carry information that may change for every picture, such as, initial \ac{qp}, flags for picture related tools and tilling information. \subsection{Picture types} @@ -213,22 +223,20 @@ The new set of parameters \ac{vps} contains information related to the different \end{tabular} \end{table} -The \ac{hevc} standard supports several pictures types, as show in Table~\ref{Table2:PictureTypes}, which are divided in the following categories: (i) random-access points, (ii) leading pictures and (iii) temporal sub-layer access pictures~\cite{HEVCHighLevelReferenceDez2012,BookHEVC2014}. +The \ac{hevc} standard supports several pictures types, as show in Table~\ref{Table2:PictureTypes}, which are classified in the following categories: (i) random-access points, (ii) leading pictures and (iii) temporal sub-layer access pictures~\cite{HEVCHighLevelReferenceDez2012,BookHEVC2014}. The use of different types of pictures allows the \ac{hevc} stream to be more flexible, support a wider range of applications and achieve robust video communications. They are important to perform random-access, which is a critical feature for channel switching, seek operations and dynamic streaming services. Moreover, introducing random-access points into the bitstream increases its error robustness as they provide anchor points for a clean decoding after data loss or synchronisation problems. -%\HIGHLIGHT{adicionar texto mencionar robustez} - \subsubsection{Random-Access Point} -The \ac{rap} pictures are used to provide points in the bitstream with no dependencies, where it is possible to start the decoding process. The standard defines that a conforming bitstream must start with a \ac{rap} picture. +The \ac{rap} are pictures used to provide access points in the bitstream with no dependencies, where it is possible to start the decoding process. The standard defines that a conforming bitstream must start with a \ac{rap} picture. These frames must belong to temporal sub-layer 0 and should be coded using intra coding techniques, i.e., must not use previously coded frames as reference. One should note that there may be frames compressed with intra prediction only, but not be marked as random-access points. It is always possible to start decoding from a \ac{rap} frame onwards, and to output any subsequent pictures in the display order, even if all pictures that precede the \ac{rap} in the decoding order are discarded from the bitstream. -The \ac{idr} frame is an intra coded picture that completely refreshes the decoding process, i.e., cleans the decoder picture buffer preventing predictions to previously coded frames, starting a new conforming video stream. This means that neither the \ac{idr} nor any other subsequent frame in the decoding order have prior dependencies. \ac{idr} frames are allowed to have leading pictures, i.e., frames follow the \ac{rap} picture in the decoding order but precede it in the display order. However, they must be decodable, thus creating a true random-access point where all subsequent frames in the decoding order can be decoded and displayed. +The \ac{idr} frame is an intra coded picture that completely refreshes the decoding process, i.e., cleans the decoder picture buffer preventing predictions to previously coded frames, starting a new conforming video stream. This means that neither the \ac{idr} nor any other subsequent frame in the decoding order have prior dependencies. \ac{idr} frames are allowed to have leading pictures, i.e., frames that follow the \ac{rap} picture in the decoding order but precede it in the display order. However, they must be decodable, thus creating a true random-access point where all subsequent frames in the decoding order can be decoded and displayed. Introducing \ac{rap} reduces the coding efficiency since no prior reference frames can be used. In order to reduce this impact, the \ac{hevc} standard introduces a new \ac{rap} picture, referred to as \ac{cra} picture. @@ -236,13 +244,13 @@ These frames do not refresh the decoding process, allowing leading pictures to d This enables a more efficient prediction structure, while maintaining an access point to begin the decoding process. However, it results in leading pictures which may not be decodable, which are explained in the next sub-section. The introduction of \ac{cra} frames in \ac{hevc} was able to improve the coding efficiency by up to 6\% \cite{HEVCCRAPics2011}. +% TODO: Deves saber o porque desta diferença ^ -%\HERE The \ac{hevc} supports bitstream splicing, which means taking a particular bitstream from a \ac{rap} (both \ac{idr} and \ac{cra} frames) and inserting it into another bitstream at a random-access point. In case of bitstream splicing, starting from an \ac{idr} frame, the leading pictures do not have dependencies from frames prior the \ac{rap}, thus all frames are decodable. However, whenever this is performed for a bitstream starting from a \ac{cra} frame, the leading associated frames might not be correctly decoded, because some of their reference pictures are not present in the combined bitstream. -Therefore, in order to make splicing operation straightforward, the \ac{nal} units containing the \ac{cra} picture are changed to the type \ac{bla}. This way, the decoder is aware that the leading frames should be discarded, namely the \ac{rasl} pictures. +To make the splicing operation straightforward, the \ac{nal} units containing the \ac{cra} picture are changed to the type \ac{bla}. Therefore, the decoder is aware that the leading frames should be discarded, namely the \ac{rasl} pictures. \subsubsection{Leading and trailing pictures} @@ -263,9 +271,10 @@ Due to the introduction of more flexible random-access points using \ac{cra} pic \end{figure} There are two types of leading pictures, as shown in Table~\ref{Table2:PictureTypes}. -The \ac{radl} pictures, which can be correctly decoded since they only depend on the associated \ac{rap} picture and do not have any prior dependencies to trailing pictures associated with the previous \ac{rap} frame. -On contrary, the \ac{rasl} pictures may have dependencies to prior trailing pictures. -When a random-access is performed at the associated \ac{rap} frame, these frames cannot be correctly decoded, therefore they should be skipped. In the example of Figure~\ref{Figure2:LeadingTralingPics} the leading pictures represented correspond to \ac{rasl} pictures. +The \ac{radl} pictures, which can be correctly decoded, since they only depend on the associated \ac{rap} picture and do not have any prior dependencies to trailing pictures associated with the previous \ac{rap} frame. +On the contrary, the \ac{rasl} pictures may have dependencies to prior trailing pictures. +When a random-access is performed at the associated \ac{rap} frame, these frames cannot be correctly decoded, therefore they have to be skipped. +In the example of Figure~\ref{Figure2:LeadingTralingPics} the leading pictures correspond to \ac{rasl} pictures. \subsubsection{Temporal sub-layer switching pictures} @@ -273,7 +282,8 @@ When a random-access is performed at the associated \ac{rap} frame, these frames A \ac{tsa} picture is a trailing picture that marks a temporal layer switching point. When decoding a sub-set of temporal layers, if a \ac{tsa} picture is found in the temporal layer just above the maximum temporal layer currently being decoded, it is possible to start decoding any number of additional temporal layers. For example, frame $P_6$ in Figure~\ref{Figure2:TSAExample} is considered a \ac{tsa} picture since it only depends on a frame belonging to temporal layer 0 for prediction, as well as, any subsequent frame predicted from the \ac{tsa} frame $P_6$. -In this example, if the decoder was only decoding the temporal layer 0, it can after decoding $P_6$ it can also decode both layers 0 and 1, or decode all the available three layers. Although theses frame allows higher flexibility in terms of layer switching, they severely constrain the prediction of frames following the \ac{tsa} picture. +In this example, if the decoder is only decoding the temporal layer 0, after decoding $P_6$ it can also decode layer 1, or decode all the available three layers. +Although these frames allow higher flexibility in terms of layer switching, they severely constrain the prediction of frames following the \ac{tsa} picture. \begin{figure}[tb] \centering @@ -284,7 +294,9 @@ In this example, if the decoder was only decoding the temporal layer 0, it can a \end{figure} To reduce the constrains in frame prediction due to layer switching, \ac{hevc} introduces the \ac{stsa} picture. -These pictures have similar purpose as the \ac{tsa} frames, but they only guarantee that frames in the same temporal layer as the \ac{stsa} frame are able to be correctly reconstructed. This is guaranteed by avoiding any dependency from frames preceding the \ac{stsa} to be used as reference in its temporal layer. +These pictures have similar purpose as the \ac{tsa} frames, but they only guarantee that frames in the same temporal layer as the \ac{stsa} frame are able to be correctly reconstructed. +%This is guaranteed by avoiding any dependencies from frames preceding the \ac{stsa} to be used as reference in its temporal layer. +This is guaranteed by not using frames preceding the \ac{stsa} as reference in its temporal layer. One example of a \ac{stsa} frame is also shown in Figure~\ref{Figure2:TSAExample} in frame $P_2$. This frame can be used to switch to layer 1, as it does not have dependencies to any prior frame in the same layer ($P_2$ only depends on $P_0$). However, it cannot be classified as \ac{tsa} picture because $P_3$ has dependencies to a prior frame in the decoding order ($P_1$). @@ -293,10 +305,11 @@ Summarising, \ac{stsa} frames can be used to switch to a particular layer, as \a \subsection{Picture partitioning} -The high-level segmentation of a picture in \ac{hevc} is achieved through four techniques: regular slices, dependent slices, tiles and \glsfirst{wpp}. Picture partitioning normally serves one or more of the following three purposes: +The high-level segmentation of a picture in \ac{hevc} is achieved by using four different approaches associated to different data structures: regular slices, dependent slices, tiles and \ac{wpp}. +Picture partitioning normally serves one or more of the following three purposes: \begin{description} \item[Error robustness:] partitioning the picture into smaller self-contained units in - order to gain error robustness, allowing to re-synchronize both the parsing and decoding + order to increase error robustness, allowing to re-synchronize both the parsing and decoding processes in case of data losses. \item[Network adaptation:] adapt to the network constraint of \ac{mtu} size, found for example in \ac{ip} networks. @@ -307,11 +320,11 @@ The high-level segmentation of a picture in \ac{hevc} is achieved through four t \subsubsection{Slices} -The \ac{hevc} standard preserves the slice structure defined in the \ac{h264} standard. A slice may comprise either the entire frame or a section of it, and all the associated data (\emph{i.e.}, entropy symbols, prediction and residue information) can be independently decoded. Each slice is transported in a different \ac{nal} unit. +The \ac{hevc} standard preserves the slice structure previously defined in the \ac{h264} standard. A slice may comprise either the entire frame or a section of it, and all the associated data (\emph{i.e.}, entropy symbols, prediction and residue information) can be independently decoded. Each slice is transported in a different \ac{nal} unit. As some dependencies across slice boundaries are disabled, each slice can be independently reconstructed, regardless whether previous slices were lost or incorrectly decoded. Slice partitioning is the only tool that can be used for parallel processing in a virtually identical form as in \ac{h264}. As each slice is independent from each other, it is straightforward to process multiple slices in parallel, without inter-process communication (except for inter-frame prediction). Although this is a simple approach, it incurs in substantial coding overhead due to the higher number of slice headers, and reduction of causal neighbours for prediction (due to lack of predictions across slice boundaries). Another disadvantage of using slices for parallel processing is that they are also used to meet the \gls{mtu} size constrains. Therefore, it might be impossible to meet both requirements using only one technique. -Another partitioning tool in \ac{hevc} standard is the dependent slices. These slices allow the partitioning of a frame at the coding unit boundaries without breaking intra-frame predictions. Moreover, as they have a smaller slice header they are more efficient in terms of signalling overhead than regular slices. Dependent slices are normally used to reduce the end-to-end delay by allowing part of the slice to transmitted while the rest of its data is still being processed. +Another partitioning tool in \ac{hevc} standard is the dependent slices. These types of slices allow the partitioning of a frame at the coding unit boundaries without breaking intra-frame predictions. Moreover, as they have a smaller slice header they are more efficient in terms of signalling overhead than regular slices. Dependent slices are normally used to reduce the end-to-end delay by allowing part of the slice to be transmitted while the rest of its data is still being processed. \subsubsection{Tiles} @@ -337,11 +350,13 @@ The use of tiles provide several advantages. Specifically, they improve frame pa \subsubsection{Wavefront Parallel Processing} +\glsreset{wpp} + As mentioned before, the use of either slices or tiles to process each frame using parallel threads require the use of different entropy coding contexts for each slice/tile, in order to make them independently decodable. As consequence, the compression efficiency decreases. In order to overcome this issue, the concept of \ac{wpp} was introduced, which enables coding using efficient parallel processing~\cite{HEVCParallelCSVT2016}. In \ac{wpp}, each slice (or tile) is divided into single rows of \acs{ctu} which can be processed in parallel, but entropy coding and prediction are enabled across \acs{ctu} of different rows, in order to avoid loss in coding efficiency. To re-use the same context in the whole slice (or tile) each row requires a delay of two \acs{ctu} in order to ensure that all necessary dependencies have already been encoded/decoded. Figure~\ref{Figure2:WPP} illustrates an example of \ac{wpp} using at least five different threads. -Each square corresponds to a \ac{ctu} and the numbers correspond to the time instance when a \ac{ctu} starts being processed. Using this staggered start (which appears like a wavefront, as represented in Figure~\ref{Figure2:WPP}), the parallelisation may use as many threads as the number of \acs{ctu} rows available in the video frame. However, although the number of threads does not affect the coding efficiency, the required inter-process communication substantially increases. +Each square corresponds to a \ac{ctu} and the numbers correspond to the time instance when a \ac{ctu} starts being processed. Using this staggered start, which appears like a wavefront, as represented in Figure~\ref{Figure2:WPP}, the parallelisation may use as many threads as the number of \acs{ctu} rows available in the video frame. However, although the number of threads does not affect the coding efficiency, the required inter-process communication substantially increases. \begin{figure}[tb] \centering @@ -357,8 +372,9 @@ Each square corresponds to a \ac{ctu} and the numbers correspond to the time ins \label{Section2:CodingStructures} Despite the fact that the \ac{hevc} standard follows a traditional hybrid video coding architecture, it introduces significant changes in the data structures when compared to the previous standards (e.g., \ac{h264}). -In \ac{h264} each frame is divided into units so-called \acs{mb}, which are composed by a $16\times16$ block of luma samples and two $8\times8$ blocks of chroma samples, assuming video data in YUV 4:2:0 chroma sub-sampling. The \ac{mb} is considered the basic processing unit in the \ac{h264}, and the respective coding mode define whether the corresponding pixels are coded as intra or intra prediction. -Each \ac{mb} could also be partitioned into sub-blocks which should follow the same prediction mode. +In \ac{h264} each frame is divided into units so-called \acs{mb}, which are composed by a $16\times16$ block of luma samples and two $8\times8$ blocks of chroma samples, assuming video data in YUV 4:2:0 chroma sub-sampling. +The \ac{mb} is considered the basic processing unit in the \ac{h264}, and the respective coding mode defines whether the corresponding pixels are coded as intra or intra prediction. +Each \ac{mb} could also be partitioned into sub-blocks, which should follow the same prediction mode. \glsreset{ctu} @@ -366,9 +382,9 @@ Each \ac{mb} could also be partitioned into sub-blocks which should follow the s The basic data structures in \ac{hevc} are quite different from \ac{h264}. Each picture is divided into blocks named as \ac{ctu}~\cite{BlockPartHEVC2012}. The maximum \ac{ctu} size is an encoder configuration parameter that is signalled to the decoder, and should be one of the following: $64\times64$, $32\times32$ and $16\times16$. The use of data structures with larger size in \ac{hevc} was adopted to achieve higher coding efficiency especially in high resolution video. Each \ac{ctu} can be recursively partitioned into four smaller units, referred to as \acs{cu}, until it reaches the minimum unit size, which cannot be smaller than $8\times8$ pixels. Similar to the maximum \ac{ctu} size, the minimum size is also defined at the encoder. -For instance, in case of homogeneous regions, large \acs{cu} can be used to represent those regions -by using a smaller number of symbols than is the case of using several small units. -Figure~\ref{Figure2:HEVCCodingStructure} illustrates an example of a \ac{ctu} partitioning into \acs{cu}. The partitioning illustrated in the figure leads to the quad-tree structure represented on the left side with the corresponding coding order shown on the right side. +For instance, in case of homogeneous regions, large \acs{cu} can be used to represent such regions by using a smaller number of symbols than in the case of using several small units. +Figure~\ref{Figure2:HEVCCodingStructure} illustrates an example of a \ac{ctu} partitioning into \acs{cu}. +The partitioning illustrated in the figure leads to the quad-tree structure represented on the left side with the corresponding coding order shown on the right side. Data structures in \ac{hevc} provide a significant improvement in comparison with previous standards, by eliminating the distinction between \ac{mb} and sub-\ac{mb} and using only the \ac{cu} concept. This allows for the multilevel hierarchical quadtree structure to be specified in a simple and elegant way, with size-independent syntax representation. \begin{figure}[t] @@ -385,12 +401,6 @@ The \ac{cu} is the basic processing unit in \ac{hevc}. It is further partitioned The decision whether to use intra-frame or inter-frame prediction modes is taken at the \ac{cu} level. However, depending on the partitioning of the \ac{cu} into \acs{pu}, different intra or inter coding modes can be used for each \ac{pu}. - -%The main key feature of \ac{hevc} is the new block partition structure, which achieves more flexibility and efficiency. Four different block concepts have been introduced in the standard: \ac{ctu}, \ac{cu}, \ac{pu}, and \ac{tu}, all of which have different roles. Figure~\ref{fig:HEVCBlockStructure} illustrates an example of a CTU partitioning in HEVC, and it is compared to the \ac{h264} \ac{mb} structure. CTUs are defined as the high-level partition within each slice and can be as large as 64$\times$64, which can be recursively partitioned into four smaller blocks referred to as CUs. The CU is the basic units for the intra/inter prediction and can be as small as 8$\times$8. Subsequently, the CU are divided into up to four PUs which are used for inter prediction. For intra coding PU is not used. Finally, the CU is divided into a transform unit tree, which may include several TUs, in a quad-tree structure. Transform units are the basic units for the transform operations applied to the coding residue. -% -%Each CU may be split into two or four Pus resulting in either symmetric or asymmetric partitions. Asymmetric partitioning of the CU into two PUs enables more efficient motion estimation since it allows more flexible partitioning of the picture~\cite{BlockPartHEVC2012}. This scheme is able to improve the coding performance by up to $1\%$ when compared with rectangular symmetric partitioning. - - %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -406,21 +416,16 @@ This section describes the prediction techniques used in the \ac{hevc} standard Intra-frame prediction is used to efficiently reduce the spatial redundancy within video frames. Intra predicted blocks are obtained using the previously encoded pixels around the current \ac{pu}. -When the intra prediction mode is chosen for a \ac{cu}, the size of the \ac{pu} need to be the same as the \ac{cu}, except when the smaller \ac{cu} is selected and further divided into four \acs{pu} is done. +When the intra prediction mode is chosen for a \ac{cu}, the size of the \ac{pu} needs to be the same as the \ac{cu}, except when the smaller \ac{cu} is selected and further divided into four \acs{pu} is done. For example, when the smaller \ac{cu} size is $8\times8$, the encoder may decide to divide it into four \acs{pu} of $4\times4$ pixels, each one having its own intra prediction mode. This is signalled using a binary flag. The use of small \acs{pu} is useful for regions with many details that require fine-granularity prediction. In \ac{hevc} there are 35 intra-frame prediction modes: Planar, DC and 33 angular modes, as shown on the left of Figure~\ref{Figure2:IntraModesHEVC}~\cite{IntraCodingHEVC2012,HEVCIntraperformance2012}. -%Figure~\ref{Figure2:IntraModesHEVC} illustrates these intra prediction modes. -%As shown, the predictions are applied to square blocks only. -%The border around the square blocks corresponds to the reference pixels from which the prediction is applied (different colours correspond to different pixel values). -%The example in the figure corresponds to a \ac{pu} size of $8\times8$ pixels, but block sizes ranging from $4\times4$ to $32\times32$ are also allowed. For each \ac{pu}, the encoder chooses among all the available intra prediction modes the best mode to be used by the decoder, based on the previously reconstructed pixels also available for similar prediction at the decoder. -The reconstructed reference pixels used in the prediction process belong to -the neighbouring blocks at left-down, left, top-left, top and top-right positions, as +The reconstructed reference pixels used in the prediction process belong to the neighbouring blocks located at left-down, left, top-left, top and top-right positions, as can be seen in the example shown on the right in Figure~\ref{Figure2:IntraModesHEVC}. Given a $N \times N$ \ac{pu}, intra prediction requires the top neighbouring row of $2N$ pixels, the left column of $2N$ pixels and the top-left neighbouring pixel. -Due to data unit boundaries (e.g., slice or tile boundaries) and constrained intra prediction (i.e., reference pixels belonging to inter-predicted \acs{pu} are not used in order to remove error propagation from potentially erroneous reconstructed reference frames) neighbouring pixels might not be all available to use intra prediction, resulting in an incomplete set of neighbouring reference pixels. +Due to data unit boundaries (e.g., slice or tile boundaries) and constrained intra prediction (i.e., reference pixels belonging to inter-predicted \acs{pu} are not used in order to remove error propagation from potentially erroneous reconstructed reference frames), neighbouring pixels might not be all available to be used as reference for intra prediction, resulting in an incomplete set of neighbouring reference pixels. In order to overcome this issue and to allow all possible modes, \ac{hevc} uses reference sample substitution to replace the unavailable reference pixels with the closest available ones, so that all intra prediction modes can be used. \begin{figure}[tb] @@ -441,8 +446,10 @@ Directional intra prediction was first introduced in the \ac{h264} standard~\cit Directional prediction projects the reconstructed samples in the block neighbourhood (see reference pixels in Figure~\ref{Figure2:IntraModesHEVC}) along a specific direction. The \ac{hevc} increases the number of possible directions from 8 (in \ac{h264}) to 33~\cite{IntraCodingHEVC2012}. The direction of each projection is illustrated in the left-side of Figure~\ref{Figure2:IntraModesHEVC} by the directional modes 2 to 34. %In the figure it is possible to notice that directional modes 2 to 10 only project the pixels from the left neighbouring border in a horizontal direction, modes 10 to 25 use both the left and top neighbouring pixels in a diagonal direction and modes 26 to 34 project the top neighbours in vertical direction. -As shown in Figure~\ref{Figure2:IntraModesHEVC}, the angular modes are designed to provide a dense coverage near the horizontal (i.e., mode 10) and vertical (i.e., mode 26) directions. Moreover, the angular directions are more coarse as it gets closer to the diagonals. This reflects the observed statistical prevalence of the angles and the prediction efficiency. -To improve the efficiency of angular prediction, when the reference samples need to be projected a bilinear interpolation is used from the two closest pixels using a $1/32$ accuracy. The dashed arrows on the right of Figure~\ref{Figure2:IntraModesHEVC} correspond to a projection of interpolated pixels. +As shown in Figure~\ref{Figure2:IntraModesHEVC}, the angular modes are designed to provide a dense coverage near the horizontal (i.e., mode 10) and vertical (i.e., mode 26) directions. +Moreover, the angular directions are coarser as it gets closer to the diagonals. +This reflects the observed statistical prevalence of the angles and the prediction efficiency. +To improve the efficiency of angular prediction, when the reference samples need to be projected, a bilinear interpolation is used from the two closest pixels using a $1/32$ accuracy. The dashed arrows on the right of Figure~\ref{Figure2:IntraModesHEVC} correspond to a projection of interpolated pixels. @@ -521,7 +528,8 @@ Another improvement in \ac{hevc} is related with the reference picture manageme Decoded frames in the \ac{dpb} are grouped in one of the following categories: (i) \emph{short-term reference}, (ii) \emph{long-term reference} and (iii) \emph{unused for reference}. Once a frame is marked as \emph{unused for reference} it is no longer used for motion compensation and it will be discarded from the \ac{dpb} after being displayed. In \ac{h264} a sliding window approach is used to implicitly manage the reference picture list. -This process discards a given frame whenever the \ac{dpb} has the number of defined reference frames (which is controlled by the encoder). Moreover, \ac{h264} uses an explicitly \ac{mmco} to send commands to the decoder and change the normal reference picture management process. +This process discards a given frame whenever the \ac{dpb} has stored the maximum number of reference frames (which is defined and controlled by the encoder). +Moreover, \ac{h264} uses an explicitly \ac{mmco} to send commands to the decoder and change the normal reference picture management process. In the \ac{hevc} standard the status of the \ac{dpb} is encoded for every slice, instead of the implicit managing used in the past. The most fundamental difference in \ac{rpset} concept compared to \ac{mmco}/sliding window of \ac{h264} is that each particular frame has a complete set of the reference frames that are used or will be used. @@ -534,7 +542,7 @@ For the remaining frames, two types of information are encoded: frame number and \begin{table}[tb] \centering - \caption{Reference Picture Set example.} + \caption{\glsfirst{rpset} example.} \label{Table2:ReferencePictureListExample} \small \TABLESVSPACE @@ -560,7 +568,7 @@ Using the \ac{rpset} concept, the decoder is always able to detect loss events a \subsubsection{Asymmetric block partitions} -When inter prediction mode is selected a higher range of partitioning schemes is allowed, thus dividing the \ac{cu} into one, two or four \acs{pu}. +When inter prediction mode is selected, a higher range of partitioning schemes is allowed, thus dividing the \ac{cu} into one, two or four \acs{pu}. The splitting into four \acs{pu} is only allowed when the \ac{cu} size is equal to the minimum. %Otherwise, the splitting should be carried out at the \ac{cu} level which leads to the same partition tree. Figure~\ref{Figure2:InterPuPartitions} illustrates the possible \ac{cu} partitioning into \acs{pu}. @@ -596,22 +604,24 @@ Moreover, it also uses a temporal \ac{mv} candidate, derived from the co-locate The Merge Mode uses a $C-1$ spatial \ac{mv} candidates, selected from the aforementioned order, and one \ac{tmvp} to fulfil the total number of \acs{mv} ($C$). The value of $C$ is defined at the slice header and a flag controlling the use of \ac{tmvp} is encoded at the \ac{pps}-level. For B-slices, additional \ac{mv} candidates are considered by choosing two existing candidates, according to a predefined order for reference picture list 0 and list 1. A total number of twelve combinations are specified in \ac{hevc}, which are listed in~\cite{HEVCOverviewDez2012}. For P-slices, if the number of candidates is less than $C$, then zero \acs{mv} are included to fill the remaining entries. -The Merge Mode defines how to encode the \ac{mv} candidate and the residual information. -In \ac{hevc}, the Skip mode, i.e., transmitting only the \ac{mv} without residual information, is treated as a special case of the Merge Mode. -In this case, only a skip flag is enabled and the corresponding merge index is transmitted. - \begin{figure}[tb] \centering \input{\FigsPath/MergeModeHEVC.tikz} %\input{\FigsPath/HEVC/MergeModeHEVCTwoFrames.tikz} - \caption{Illustration of the MV prediction candidates in the AMVP mode.} + \caption{Illustration of the MV prediction candidates in the AMVP and Merge Mode mode.} \label{Figure2:MVCandidates} \end{figure} +The Merge Mode defines how to encode the \ac{mv} candidate and the residual information. +In \ac{hevc}, the Skip mode, i.e., transmitting only the \ac{mv} without residual information, is treated as a special case of the Merge Mode. +In this case, only a skip flag is enabled and the corresponding merge index is transmitted. + + + \subsubsection{Motion vector prediction} -The \ac{hevc} also improves the performance of the \ac{mv} coding for those cases where the Merge Mode is not used. -\ac{hevc} always use a differential encoding approach for efficiently representing the motion information. +The \ac{hevc} also improves the performance of \ac{mv} coding for those cases where the Merge Mode is not used. +A differential encoding approach is always used for efficiently representing the motion information. It considers the same \ac{mv} candidates of the Merge mode, illustrated in Figure~\ref{Figure2:MVCandidates}, to derive the \ac{mv} predictors from which the difference vector is obtained. From these \ac{mv} candidates only the first two available candidates are selected, based on the following order: \begin{enumerate} @@ -620,8 +630,8 @@ From these \ac{mv} candidates only the first two available candidates are select \item Temporal candidate: first available from $T_0$ and $T_1$. \end{enumerate} Therefore, the use of the temporal \ac{mv} candidate is limited to those cases where the spatial neighbours are not able to provide two valid candidates. +For \ac{mv} prediction only a much lower number of candidates is allowed, since the encoder can still send the \ac{mv} difference. Moreover, the encoder performs motion estimation for each candidate, which increases the encoding complexity for every possibility tested. -For \ac{mv} prediction only a much lower number of candidates is allowed, since the encoder can still send a \ac{mv} difference. Moreover, the encoder performs motion estimation for each candidate, which increases the encoding complexity for every possibility tested. %For those cases where the reference index is not the same as the current \ac{pu} and the neighbouring one, a scaling version of the \ac{mv} is used. %In the \ac{h264} standard, along different motion compensated coding techniques~\cite{H264OverviewJul2003}, the so-called P\_SKIP mode was introduced. In this mode, neither the quantised residual information, nor the \ac{mv} is transmitted. This mode is useful for large areas with no change or just constant motion, and it is only enabled for $16\times16$ blocks, \emph{i.e.}, the whole \ac{mb}. The \ac{mv} for the P\_SKIP mode is derived based on the surrounding neighbour MBs\cite{H264MotionVectorPrediction2003}. When no neighbours are available, or the left or top \acs{mv} are equal to zero, a zero \ac{mv} is used. Otherwise, the \ac{mv} is derived from the median value of the following three neighbours: left, above and above right, as shown in Figure~\ref{SkipMergeMode}~(a) by $MV_A$, $MV_B$ and $MV_C$, respectively. Whenever the $MV_C$ is not available, the $MV_D$ is used instead. @@ -648,7 +658,7 @@ A 2D transform of the residual information is obtained by applying 1D transforms A special case is applied for coding the residual of $4\times4$ intra coded block. In this case an alternative integer transform based on the \ac{dst} is used. The motivation for \ac{dst} is related to the statistics of the residual signal of these blocks, which tend to present higher residue as the distance to the boundary used for prediction (i.e, top or left boundary) increases. -Experiments have shown that \ac{dst} provides up to 1\% of bitrate savings for the compression of $4\times4$ intra predicted blocks, without significantly increasing the computational complexity \cite{DCTDSTComp2013}. +Simulation studies have shown that \ac{dst} provides up to 1\% of bitrate savings for the compression of $4\times4$ intra predicted blocks, without significantly increasing the computational complexity \cite{DCTDSTComp2013}. \subsubsection{Quantisation} @@ -656,7 +666,8 @@ Experiments have shown that \ac{dst} provides up to 1\% of bitrate savings for The \ac{hevc} follows the same principle as previous standards in terms of quantisation. After applying the transform operation to the prediction residue, a quantisation is applied to the resulting coefficients using a uniform-reconstruction quantiser scheme based on a \ac{qp}. -The standard supports \ac{qp} values ranging from 0 to 51, where an increase of 6 in the \ac{qp} corresponds to doubling the quantisation step. This procedure is the main responsible for the coding distortion as it performs a many-to-one mapping, which cannot be reversed. +The standard supports \ac{qp} values ranging from 0 to 51, where an increase of 6 in the \ac{qp} corresponds to doubling the quantisation step. +This procedure is the main responsible for the coding distortion, as it performs a many-to-one mapping, which cannot be reversed. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -704,7 +715,7 @@ by adding an offset value, extracted from lookup tables transmitted by the encod \section{Summary} \label{Section2:Summary} -This chapter was devoted to describe the main features of the \ac{hevc} standard, giving a special focus on the high-level features, coding structures and newly introduced prediction modes. +This chapter was devoted to describe the main features of the \ac{hevc} standard with special emphasis on the high-level features, coding structures and newly introduced prediction modes. At the high-level, \ac{hevc} introduces new \ac{nal} types and also new pictures types. These allow the implementation of various random-access strategies, more flexibility and higher coding efficiency than its predecessor \ac{h264}. Another major improvement comes from the inter-frame prediction. On the one hand new high-level reference picture management was introduced, which increase the error robustness, as it explicitly defines the active reference frames and the \ac{dpb} status. On the other hand it introduces the \ac{amvp} and the Merge mode, which efficiently reduce the required bits for coding the motion information. diff --git a/022_ErrorResilience/RobustTransmissionReview.tex b/022_ErrorResilience/RobustTransmissionReview.tex index 15f3c0b..de517c0 100755 --- a/022_ErrorResilience/RobustTransmissionReview.tex +++ b/022_ErrorResilience/RobustTransmissionReview.tex @@ -1,4 +1,4 @@ -\chapter{Robust Video Coding: A Review \VERSION{2}} +\chapter{Robust video coding: a review \VERSION{3}} \label{Chapter:ErrorResilienceReview} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -12,17 +12,18 @@ In general, digital communications through networks are prone to transmission errors or data loss. These errors can be handled in different ways, such as retransmission, error correction, and \ac{ec} \cite{ErrorResilience2000,PixelInterlacingIntraConcealment2011}. -This chapter reviews the main robust video coding techniques that can be applied at the \ac{vcl} to reduce the error propagation and increase the decoded video quality. +This chapter reviews the existing techniques for robust video coding that can be applied at the \ac{vcl} to reduce the error propagation and increase the decoded video quality. Several techniques have been proposed in the past to deal with the problem of robust video transmission, especially for video coding standards with high temporal dependencies, such as \ac{h264} and more recently \ac{hevc}. -These techniques are normally focused on the spatio-temporal dependencies in the coded streams, which are used to obtain high coding gains but lead to low error robustness~\cite{ROPE2007}. -In this chapter the error resilience and concealment schemes are divided in different categories, depending on their functionality and algorithms. +These techniques are normally focused on the spatio-temporal dependencies in the coded streams, which are the basis of high coding gains but lead to low error robustness~\cite{ROPE2007}. +In this chapter the error resilience and error concealment schemes are divided in different categories, depending on their functionality and algorithms. Although each approach is based on different techniques they all aim at striking the best trade-off between coding efficiency and decoded video quality under lossy conditions. -This chapter is organised as follows: Section~\ref{Section22:Localisation} describes the localisation techniques, which are able to reduce the temporal and spatial dependencies in coded bitstreams. +The chapter is organised as follows. +Section~\ref{Section22:Localisation} describes the localisation techniques, which are able to reduce the temporal and spatial dependencies in coded bitstreams. Section~\ref{Section22:DataPartitionRedundant} provides an overview on methods based on data partition, classification and bitstream redundancy. %These techniques ease the introduction of extra information in order to recover from lost data. Section~\ref{Section22:Concealment} covers the techniques used at the decoder-side, as post-processing, to reduce the effect of errors and improve visual quality and smoothness. -Section~\ref{Section22:ConcealmentAware} provides a description of methods that use \ac{ec} techniques to achieve improved error robustness. +Section~\ref{Section22:ConcealmentAware} provides a description of methods that use \ac{ec} techniques to cope with transmission errors and improve error robustness. Finally, Section~\ref{Section22:Summary} concludes the chapter. % by providing a overall comparison between the different approaches. @@ -52,12 +53,12 @@ In this chapter the robust video coding techniques are grouped into four categor \end{table} Table~\ref{tab:errorresilience} provides a summary of the techniques covered in this section, some examples and the key advantages associated with each group. -Localisation techniques mainly localise and break the predictive coding loop, reducing the spatial and temporal dependencies, so that if an error occurs it is not likely to affect other parts of the coded video~\cite{ErrorResilienceModeHEVC2016,ErrorResilienceModeHEVC2017}. -Data partitioning rely on grouping the coded data into different categories depending on their relative importance. -This ease the introduction of unequal error protection at the transmission layer. +Localisation techniques mainly localise and break the predictive coding loop, reducing the spatial and temporal dependencies, so that if an error occurs, then it is not likely to affect other parts of the coded video~\cite{ErrorResilienceModeHEVC2016,ErrorResilienceModeHEVC2017}. +Data partitioning relies on grouping the coded data into different categories depending on their relative importance. +This can be used to ease the introduction of unequal error protection at the transmission layer. Redundant coding consists in using additional data blocks to enable correction of transmission errors and to achieve robust decoding in error-prone conditions. In case of the post-processing \ac{ec} methods, the problem of robust transmission is addressed from the decoder-side. -Finally, concealment-aware techniques are used at the encoder, to either adapt the coding loop or introduce extra information, which is used to increase the performance of \ac{ec} algorithms and increase the reconstruction quality of lost data at the decoder \cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ConcealmentIntraDataHidng2009,ConcealmentSignalling2008}. +Finally, error concealment-aware techniques are used at the encoder, to either adapt the coding loop or introduce extra information, which is used to increase the performance of \ac{ec} algorithms and increase the reconstruction quality of lost data at the decoder \cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ConcealmentIntraDataHidng2009,ConcealmentSignalling2008}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -73,7 +74,7 @@ Since these techniques break the predictive coding, they have a negative impact This group of error resilience techniques are able to reduce the error propagation at two levels, i.e., temporal and spatial levels. Figure~\ref{fig2:ErrorPropagation} is used to demonstrate the principle of error propagation and how it can be reduced using localisation techniques. The top of the figure illustrates a lost block and the corresponding spatial and temporal error propagation on the left and right, respectively. Spatial localisation techniques are used to reduce the error propagation within a frame, where a row of refresh blocks can be introduced to stop the error propagation. -To address the temporal error propagation, also refresh blocks (e.g., intra coded blocks) can be randomly introduced to stop error propagation. +To address the temporal error propagation, also refresh blocks (e.g., intra coded blocks) can be randomly introduced to break temporal dependencies and stop error propagation. As shown in Figure~\ref{fig2:ErrorPropagation} the refresh blocks are not affected by error propagation. In both examples of figure, the regions affected by errors are clearly reduced. @@ -90,15 +91,16 @@ In both examples of figure, the regions affected by errors are clearly reduced. \glsreset{rd} -A common localization technique deals with the error resilience problem within the scope of the \ac{rd} optimisation process used in video coding. +Common localization techniques deal with the error resilience problem within the scope of the \ac{rd} optimisation process used in video coding. Normally, such optimisation is equivalent to the minimisation of the Lagrangian cost given by: \begin{equation} J = D + \lambda R, \end{equation} where $D$ is the distortion of all pixels, $R$ the bitrate and $\lambda$ is the Lagrangian multiplier~\cite{LagragianMultiplier1982}. Such optimisation only takes into account the coding distortion ($D$), which is only suitable for error-free environments. -However, some alternative approaches with error resilient \ac{rd} optimisation take into consideration possible network losses~\cite{DistortionTrackingH264_ICIP2004}. +However, some alternative approaches use error resilient \ac{rd} optimisation, by taking into consideration possible network losses~\cite{DistortionTrackingH264_ICIP2004}. Other advanced approaches can take advantage of a \ac{rope} of the end-to-end distortion at the encoder-side~\cite{ROPEJun2000}. +This may be formulated as follows. Let $f_t^i$ denote the original value of pixel $i$ in frame at instant $t$, $\hat f_t^i$ denote its reconstruction in the encoder and $\tilde f_t^i$ the reconstruction in the decoder after \ac{ec}. Since data loss may occur in the transmission channel, $\tilde f_t^i$ is modelled as a random variable at the encoder. The expected \ac{mse} of each pixel is given by: @@ -108,50 +110,58 @@ E\left\{d_t^i\right\} = E\left\{ \left( f_t^i - \tilde f_t^i\right)^2 \right\} The packet loss probability ($p$) is taken into consideration and used in the distortion measurement. In case of intra-coded pixels the following equations are used: \begin{equation}\label{EQ2:E2EDIntra} -E\left\{ \tilde f_t^i \right\} = \left(1-p\right) \hat f_t^i + p E\left\{ \tilde f_{n-1}^i \right\}, +E\left\{ \tilde f_t^i \right\} = \left(1-p\right) \hat f_t^i + p E\left\{ \tilde f_{t-1}^i \right\}, \end{equation} \begin{equation} -E\left\{\left( \tilde f_t^i \right)^2 \right\} = \left(1-p\right) \left( \hat f_t^i \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\}, +E\left\{\left( \tilde f_t^i \right)^2 \right\} = \left(1-p\right) \left( \hat f_t^i \right) + p E\left\{ \left( \tilde f_{t-1}^i \right)^2 \right\}, \end{equation} while in the case of inter-coded pixels: \begin{equation} -E\left\{ \tilde f_t^i \right\} = \left(1-p\right) \left( \hat e_t^i + E\left\{ \tilde f_{n-1}^{j} \right\} \right) + p E\left\{ \tilde f_{n-1}^i \right\}, +E\left\{ \tilde f_t^i \right\} = \left(1-p\right) \left( \hat e_t^i + E\left\{ \tilde f_{t-1}^{j} \right\} \right) + p E\left\{ \tilde f_{t-1}^i \right\}, \end{equation} \begin{equation}\label{EQ2:E2EDInter} -E\left\{\left( \tilde f_t^i \right)^2 \right\} = \left(1-p\right) \left( \left( \hat e_t^i \right)^2 + 2 \hat e_t^i E\left\{ \tilde f_{n-1}^{j} \right\} + E\left\{ \left( \tilde f_{n-1}^{j} \right)^2 \right\} \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\}, +E\left\{\left( \tilde f_t^i \right)^2 \right\} = \left(1-p\right) \left( \left( \hat e_t^i \right)^2 + 2 \hat e_t^i E\left\{ \tilde f_{t-1}^{j} \right\} + E\left\{ \left( \tilde f_{t-1}^{j} \right)^2 \right\} \right) + p E\left\{ \left( \tilde f_{t-1}^i \right)^2 \right\}, \end{equation} assuming that the pixel $i$ is predicted from pixel $j$ in the previous frame. The prediction error $e_t^i$ is quantised to the value $\hat e_t^i$, which is transmitted to the decoder along with the motion information. Equations~\ref{EQ2:E2EDIntra} to~\ref{EQ2:E2EDInter} provide a measure of the distortion, taking into consideration not only the coding distortion but also the possibility of frame loss and consequent \ac{ec} distortion. -This technique was used in \cite{ROPEJun2000} to switch between intra and inter coding modes with the aim of reducing the temporal dependencies, thus achieving enhanced error robustness in video transmission. -In \cite{AdvancesROPEHE2642007} this technique was optimised to quarter pixel estimation, and in \cite{ROPEBurst2009} the method was extended for burst losses, by introducing the burst loss probability into the original end-to-end estimation. +This method was used in \cite{ROPEJun2000} to switch between intra and inter coding modes with the aim of reducing the temporal dependencies, thus achieving enhanced error robustness in video transmission. +In \cite{AdvancesROPEHE2642007} this was optimised to quarter pixel estimation, and in \cite{ROPEBurst2009} the same method was extended for burst losses, by introducing the burst loss probability into the original end-to-end estimation. Results revealed that higher video quality was achieved by considering both probabilities. -In \cite{E2EDistortion2011} a temporal \ac{ec} is considered in the distortion estimation, which increases the performance of the proposed method, by avoiding frame-copy that provides a lower bound on reconstruction quality. +In \cite{E2EDistortion2011} temporal \ac{ec} is considered in the distortion estimation, which increases the performance of the proposed method, by avoiding frame-copy that provides a lower bound on reconstruction quality. More advanced temporal \ac{ec} methods were also considered in the work presented in~\cite{EndToEndDistortionConcealment2014}. -The \acs{mv} used for motion compensation reconstruction is obtained using boundary matching search, which is able to achieve lower distortion when compared with frame-copy. Using more advanced \ac{ec} algorithms leads to a more accurate estimation of the end-to-end distortion, thus increasing the error robustness of the transmitted video. -To take into account the delay introduced in the encoding loop due to hierarchical prediction structures, delay based \ac{rd} optimisation was proposed in \cite{DelayRD_TCVT2015} by estimating both the distortion and the end-to-end delay that is minimised along the required bitrate. Using this approach, the total delay can be controlled without severely compromising the coding efficiency. -In the context of \ac{hevc} end-to-end distortion estimation was also used in the \ac{rd} optimisation showing significant quality improvements when compared to with the reference implementation \cite{ErrorResilienceModeHEVC2016}. +The \acs{mv} used for motion compensation reconstruction are obtained using boundary matching search, which is able to achieve lower distortion when compared with frame-copy. Using more advanced \ac{ec} algorithms leads to a more accurate estimation of the end-to-end distortion, thus increasing the error robustness of the transmitted video. +To take into account the delay introduced in the encoding loop due to hierarchical prediction structures, delay-based \ac{rd} optimisation was proposed in \cite{DelayRD_TCVT2015} by estimating both the distortion and the end-to-end delay that is minimised along with the required bitrate. +Using this approach, the total delay can be controlled without severely compromising the coding efficiency. +In the context of \ac{hevc}, end-to-end distortion estimation was also used in the \ac{rd} optimisation showing significant quality improvements when compared to with the reference implementation \cite{ErrorResilienceModeHEVC2016}. -Different quality metrics can also be used for end-to-end distortion estimation. The method presented in \cite{SSIMModeling2015} proposes the distortion estimation using the \ac{ssim} \cite{QualitySSIMRef2002} in order to increase the correlation between the estimated distortion and the perceptual quality noticed by the human visual system~\cite{VisualQualityMetrics2011}. Moreover, the \ac{ssim} quality metric was also used in learning based \ac{rd} models~\cite{RateDistortionNN_VCIP2017}, using a \ac{cnn} to predict the end-to-end distortion and the required rate, showing good accuracy in the \ac{rd} cost estimation. +Different quality metrics can also be used for end-to-end distortion estimation. +The method presented in \cite{SSIMModeling2015} proposes to estimate the distortion by using the \ac{ssim} \cite{QualitySSIMRef2002} in order to increase the correlation between the estimated distortion and the perceptual quality noticed by the human visual system~\cite{VisualQualityMetrics2011}. Moreover, the \ac{ssim} quality metric was also used in learning-based \ac{rd} models~\cite{RateDistortionNN_VCIP2017}, using a \ac{cnn} to predict the end-to-end distortion and the required rate, showing good accuracy in the \ac{rd} cost estimation. \subsection{Intra-refreshing} -In order to reduce the effects of error propagation within a single, frame spatial and temporal dependencies can be limited by introducing refresh points into the bitstream, by forcing intra-coded blocks~\cite{IntraRefresh1999,IntraRefreshAttentionCM2007,IntraRefreshRefSelectionJan2009,IntraRefreshICIP2009,SandroIntraRefresh2011,JointIntraRefresh2014,IntraRefreshFeedback2015,IntraRefreshRPS2017}. +In order to reduce the effects of error propagation within a single frame, spatial and temporal dependencies can be limited by introducing refresh points into the bitstream, by forcing intra-coded blocks~\cite{IntraRefresh1999,IntraRefreshAttentionCM2007,IntraRefreshRefSelectionJan2009,IntraRefreshICIP2009,SandroIntraRefresh2011,JointIntraRefresh2014,IntraRefreshFeedback2015,IntraRefreshRPS2017}. Figure~\ref{FIG2:IntraRefresh} (a) shows an example where intra-coded block are forced at random locations (dark gray sqares) in order to stop error propagation. -This technique is fairly efficient to improve the error robustness of video streams, by avoiding long error propagation both within a single frame but also across several subsequent frames. +This technique is fairly efficient to improve the error robustness of video streams, by avoiding long error propagation not only within a single frame but also across several subsequent frames. The main characteristic of this group of techniques is to increase the number of intra-coded blocks as they have no dependencies to previous frames. -However, this is done at the expense of higher amount of bits~\cite{IntraRefreshAttentionCM2007}. -Therefore, different alternatives have been proposed, which optimise the number of intra-coded blocks, which can be introduced randomly, periodically~\cite{SandroIntraRefresh2011}, based on the motion information or rate-distortion optimisation~\cite{IntraRefreshICIP2008,HEVCIntraRefreshICIP2017}. -Moreover, whenever is possible the encoder can use a feedback channel to decide where refresh regions should be introduced~\cite{IntraRefresh1999,IntraRefreshFeedback2015}. +However, this is done at the expense of using higher amount of coded bits~\cite{IntraRefreshAttentionCM2007}. +Therefore, different alternatives have been proposed to optimise the number of intra-coded blocks, which can be introduced either randomly or periodically~\cite{SandroIntraRefresh2011}, based on the motion information or rate-distortion optimisation~\cite{IntraRefreshICIP2008,HEVCIntraRefreshICIP2017}. +Moreover, whenever possible, the encoder can use a feedback channel to decide where refresh regions should be dynamically introduced in the coded stream~\cite{IntraRefresh1999,IntraRefreshFeedback2015}. +\begin{figure}[tb] + \centering + \input{./022_ErrorResilience/Figures/IntraRefresh.tikz} + \caption{Examples of intra-refreshing methods.} + \label{FIG2:IntraRefresh} +\end{figure} -A non-normative method for random intra refreshing is implemented in the \ac{h264} reference software, which introduces cyclic refresh points using a random pattern. -Figure~\ref{FIG2:IntraRefresh} (a) shows an example of intra-coded blocks (dark gray squares) being randomly places into the coded frames to prevent error propagation. +A non-normative method for random intra refreshing is implemented in the H.264 reference software, which introduces cyclic refresh points using a random pattern. +Figure~\ref{FIG2:IntraRefresh} (a) shows an example of intra-coded blocks (dark gray squares) being randomly placed into the coded frames to prevent error propagation. This approach guarantees that all \acs{mb} are refreshed at some point, therefore errors do not propagate indefinitely. However, all \acs{mb} are refreshed exactly the same number of times without consideration for rate distortion optimisation neither the image region containing the \acs{mb}. -In \cite{IntraRefreshICIP2008} a rate control scheme is used to decide if a given block should be refreshed using a intra-coded prediction. +In \cite{IntraRefreshICIP2008} a rate control scheme is used to decide whether a given block should be refreshed using intra-coded prediction. The decision is performed based on the ratio between the Lagrangian cost of the best intra and inter modes. Whenever this ratio is lower than a certain predefined threshold an intra mode is selected, which guarantees that intra refreshing is only forced when the Lagrangian cost of the intra mode is close to the optimal coding method. This is further extended to adapt the intra refresh algorithm to the network conditions, i.e., the threshold value is obtained from the bitrate and the loss ratio~\cite{IntraRefreshICIP2009}. @@ -159,27 +169,22 @@ Therefore, the number of intra-coded block automatically increases as the bitrat In the context of \ac{hevc}, intra refresh is also being used to reduce error propagation. Soft-refresh points using joint intra-inter prediction were proposed for this purpose~\cite{HEVCIntraRefreshICIP2017}. Although the image regions cannot be correctly decoded in case of errors (as happens with intra-refresh), this method significantly reduces the bitrate increase due to extra intra-coded blocks. -When compared with traditional intra-refresh, bitrate savings of up to 12\% are achieved for a 5\% \gls{plr}. +When compared with traditional intra-refresh, bitrate savings of up to 12\% are achieved for a 5\% packet loss. -\begin{figure}[tb] - \centering - \input{./022_ErrorResilience/Figures/IntraRefresh.tikz} - \caption{Examples of intra-refreshing methods.} - \label{FIG2:IntraRefresh} -\end{figure} \glsreset{roi} A different approach is to introduce refresh points based on the video content. -In~\cite{IntraRefreshAttentionCM2007} the error resilience of video streams is increased by combining rate control and attention based models to define \acs{roi}. +In~\cite{IntraRefreshAttentionCM2007} the error resilience of video streams is increased by combining rate control and attention-based models to define \acs{roi}. A weighting factor is used in the Lagrangian optimization to increase the number of intra-coded blocks in the image regions determined by the attention models. Figure~\ref{FIG2:IntraRefresh} (b) shows an example with a \ac{roi} marked with a black line, which varies for every frame. In this case, the intra-coded blocks are only introduced inside the marked region, in contrast to what is shown in Figure~\ref{FIG2:IntraRefresh} (a). -Results show that using the aforementioned method, less refresh points are required to improve the error robustness of the \ac{roi}. When comparing with \cite{DistortionTrackingH264_ICIP2004}, approximately 4 dB of \ac{psnr} gains are achieved in the \ac{roi} without compromising the overall video quality. +Results show that by using the aforementioned method, less refresh points are required to improve the error robustness of the \ac{roi}. When comparing with \cite{DistortionTrackingH264_ICIP2004}, approximately 4 dB of \ac{psnr} gains are achieved in the \ac{roi} without compromising the overall video quality. A more recent approach was proposed using a perceptual noticeable difference model to estimate the noticeable distortion, and then using a visual attention model \cite{SaliencyItti1998} to decide which blocks should be used as intra refresh in the bitstream~\cite{IntraRefreshPerceptual2010}. This method is able to improve the results of a previous work \cite{IntraRefresh2005}, especially for high motion sequences. This is due to the fact that such previous approach only used an estimation of the error propagation without accounting for perceptual information. A similar approach was also exploited for \ac{hevc} in \cite{HEVCROIErrorResilience2015,HEVCRoiIntraRefresh_DSP2016}, by using a feedback channel to provide information about the network errors, which is then used by the encoder to introduce intra-coded blocks into the \ac{roi}. -The simulation results reveal an increasing on the achieved video quality when higher number of intra blocks are used. Moreover, applying the proposed method only to the \ac{roi}, the bitrate overhead introduced is lower, leading to higher \ac{rd} gains. +The simulation results reveal an increase on the video quality when higher number of intra blocks are used. +Moreover, applying the proposed method only to the \ac{roi}, the bitrate overhead introduced is lower, leading to higher \ac{rd} gains. \subsection{Leaky-predictions} @@ -188,13 +193,13 @@ The simulation results reveal an increasing on the achieved video quality when h Directly using reconstructed past frames as reference frames for predictive coding, may result in substantial error propagation, whenever the reference frames are either lost or reconstructed from other distorted frames. To reduce error propagation one can employ leaky predictions, which scales down reconstructed frames before being used as reference~\cite{LeakyPred2009,ROILeakyPred2009}. This leads to an exponential decay of error propagation, with direct impact on error robustness. -In this approach, the reconstructed reference frames are scaled down by applying the following: +In such approach, the reconstructed reference frames are scaled down by applying the following: \begin{equation}\label{EQ2:LeakyPredConstant} \check f_t^i = \alpha \hat f_t^i + (1- \alpha)C, \end{equation} -where $\check f_t^i$ is a reference frame, $\alpha$ is the leaky factor used to control the amount of information from the reconstructed frame present in the new frame $\check f_t^i$, and $C$ is a constant value. +where $\hat f_t^i$ is a reference frame, $\alpha$ is the leaky factor used to control the amount of information from the reconstructed frame present in the new frame $\check f_t^i$, and $C$ is a constant value. Leaky prediction has been used in the past for error control, especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in Equation~(\ref{EQ2:LeakyPredConstant}) is replaced by a lower layer error-free reconstructed frame. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate, as shown in \cite{LeakyPredConstant1973}. The $\alpha$ parameter allows to control the effect of the constant value, and thus the coding efficiency. -By decreasing $\alpha$, the coding efficiency increases, however, the exponential decay in error propagation becomes less significant. +By decreasing $\alpha$, the coding efficiency increases, but the exponential decay in error propagation becomes less significant. % MHMCP_TCSVT2006 @@ -202,10 +207,10 @@ In order to overcome the loss in coding efficiency, weighted prediction can achi Alternatively, the two previously encoded frames can be used to generate an interpolated frame ($ \check f_t^i $) to be used as reference, using: %In case of leaky prediction instead of applying motion compensation is applied once, but with an interpolated frame obtained as follow: \begin{equation}\label{EQ2:LeakyPredDual} - \tilde f_t^i = \alpha \hat f_t^i + (1- \alpha) \hat f_{n-1}^i. + \check f_t^i = \alpha \hat f_t^i + (1- \alpha) \hat f_{t-1}^i. \end{equation} -Therefore, it is possible to decrease the error propagation using $\tilde f_t^i$ as reference frame for prediction, if only one of the original encoded frames is lost (i.e., $\hat f_{n}^i$ or $\hat f_{n-1}^i$). This is due to the fact that $\tilde f_t^i$ contains information from both $\hat f_{n}^i$ and $\hat f_{n-1}^i$. -In Equation (\ref{EQ2:LeakyPredDual}) $\hat f_{n}^i$ and $\hat f_{n-1}^i$ always correspond to the encoded frames and the generated frames ($\tilde f_t^i $) are not used to interpolate subsequent frames. This corresponds to a finite impulse response filter. +Therefore, it is possible to decrease the error propagation using $\check f_t^i$ as reference frame for prediction, if only one of the original encoded frames is lost (i.e., $\hat f_{t}^i$ or $\hat f_{t-1}^i$). This is due to the fact that $\check f_t^i$ contains information from both $\hat f_{t}^i$ and $\hat f_{t-1}^i$. +In Equation~(\ref{EQ2:LeakyPredDual}) $\hat f_{n}^i$ and $\hat f_{n-1}^i$ always correspond to the encoded frames and the generated frames ($\check f_t^i $) are not used to interpolate subsequent frames. This corresponds to a finite impulse response filter. \begin{figure}[tb] @@ -220,54 +225,54 @@ In Equation (\ref{EQ2:LeakyPredDual}) $\hat f_{n}^i$ and $\hat f_{n-1}^i$ always \label{fig2:ErrorPropagationROI} \end{figure} - -In \cite{ROILeakyPred2009} a method is presented, where the leaky prediction is confined to a pre-defined \ac{roi}, in order to increase the error robustness with lower impact on the coding efficiency. +The method presented in~\cite{ROILeakyPred2009} uses leaky prediction confined to a pre-defined \ac{roi}, in order to increase the error robustness with lower impact on the coding efficiency. Figure~\ref{fig2:ErrorPropagationROI} shows an example of error propagation within two different regions: \ac{roi} and background. As show in this figure, since the \ac{roi} uses background regions for predictions, error propagation occurs across regions. To overcome this problem, the method in \cite{ROILeakyPred2009} uses a weighted prediction for the \ac{roi} region obtained from the following predictions: (i) a globally optimal prediction obtained by unrestricted motion estimation, (ii) a prediction obtained by restricting motion estimation within the \ac{roi} region. -This limits the impact of errors in the background region on the \ac{roi}. +This limits the impact of errors in the background region to affect the \ac{roi}. Moreover, since the leaky predictions are only confined to some specific regions, higher coding efficiency is achieved when compared with other methods. -The method proposed in \cite{ROPERefGenJan2010} extends the traditional leaky prediction expressed in Equation (\ref{EQ2:LeakyPredDual}) by replacing the previous encoded frame ($\hat f_{n-1}^i.$) with the previous interpolated frame ($\tilde f_t^i$). This results in the following: +The method proposed in \cite{ROPERefGenJan2010} extends the traditional leaky prediction expressed in Equation~(\ref{EQ2:LeakyPredDual}) by replacing the previous encoded frame ($\hat f_{t-1}^i.$) by the previous interpolated frame ($\check f_t^i$). This results in the following: \begin{equation}\label{EQ2:LeakyPredIIR} -\tilde f_t^i = \alpha \hat f_t^i + (1- \alpha) \tilde f_{n-1}^i. +\check f_t^i = \alpha \hat f_t^i + (1- \alpha) \check f_{t-1}^i. \end{equation} -This corresponds to infinite impulse response filter. -Comparing both equations for the same $\alpha$, the Equation (\ref{EQ2:LeakyPredIIR}) implies stronger filtering than Equation (\ref{EQ2:LeakyPredDual}). Therefore, reference frames generated with heavier filtering have higher resilience against error propagation, but less correlation with the original frame, which negatively impacts inter predictions and hence coding efficiency. -The idea of combining multiple predictions was also used in the \ac{hevc} standard by combining it with end-to-end distortion to select the optimal predictions~\cite{DualFrameME2015}. +The above equation corresponds to an infinite impulse response filter. +Comparing both Equations~\ref{EQ2:LeakyPredDual}~and~\ref{EQ2:LeakyPredIIR} for the same $\alpha$, the latter implies stronger filtering than the former. +Therefore, reference frames generated with heavier filtering have higher resilience against error propagation, but less correlation with the original frame, which negatively impacts inter predictions and hence coding efficiency. +The idea of combining multiple predictions was also applied in the \ac{hevc} standard by combining it with end-to-end distortion to select the optimal predictions~\cite{DualFrameME2015}. \subsection{Reference picture selection} -Reference picture selection has been widely used in the past to reduce the temporal dependencies and reduce the error propagation \cite{ResilienceDualFrameBuffer2004,ErrorResilienceRPSFeb2009,RPSErrorsFeb2009,MVConcatenationAug2011}. +\ac{rps} has been widely used in the past to reduce the temporal dependencies and reduce the error propagation \cite{ResilienceDualFrameBuffer2004,ErrorResilienceRPSFeb2009,RPSErrorsFeb2009,MVConcatenationAug2011}. In \cite{ResilienceDualFrameBuffer2004}, an optimisation algorithm was devised to select between short and long term references based on the \ac{rope} end-to-end distortion estimation method. This was used to increase the prediction distance and reduce error propagation. -The same method can also be used in combination with feedback information from the network to improve the references frame selection. + The same method can also be used in combination with feedback information from the network to improve the efficiency of \ac{rps}. In the decoder-side, an \ac{ec} method was later used to take advantage of such prediction structure \cite{ConcealmentDualFrameBuffer2005}. -A feedback information from the network was also used in \cite{ErrorResilienceRPSFeb2009}, to reduce error propagation at end-user decoders, by dynamically selecting the reference frame, only from those correctly received. +A network feedback mechanism was proposed in \cite{ErrorResilienceRPSFeb2009}, to reduce error propagation at end-user decoders, by dynamically selecting the reference frame, only among those correctly received. Figure~\ref{fig2:ReferencePictureSelection} shows an example of an error resilience transcoding scheme, where the original temporal predictions (represented by the solid arrows) are transcoded to use only correctly received frames as reference, i.e., prediction represented by the dashed arrow. This is done based on network feedback. A more advanced scheme may be considered, where only the regions affected by mismatch predictions are processed in the compressed domain, and subsequently transcoded to limit the dependencies from lost frames. To achieve low complexity, \ac{mv} concatenation can be used, in order to obtain new inter-predictions for error-free frames \cite{MVConcatenationAug2011}. +In \cite{ChannelAdaptiveVideoStreaming2002} network feedback is combined with path diversity to select the reference frames that are more likely to be received at the decoder. +In such approach, the feedback information is not directly used to select the reference frames. +Instead, the reference frames are selected through an end-to-end distortion estimation process, taking into account the coding distortion and distortion resulting from channel errors (\ac{ec} distortion is also taken into account). +In \cite{ErrorResilienceModeHEVC2017} a \ac{roi} is defined based on the moving objects to limit the regions where the \ac{rps} is applied. +Intra refresh is also used, based on the temporal distance to the previous intra-refresh frame. +This method not only achieves noticeable quality gains, when compared with the reference \ac{hevc} implementation, but also when compared with traditional reference selection schemes. +In~\cite{IntraRefreshRefSelectionJan2009} an algorithm to introduce intra refresh points at the \ac{mb} level was proposed. +This method takes into consideration the decoding distortion, assuming that frame-copy is applied for \ac{ec} when packet loss occurs. +The refresh blocks are encoded using either intra or inter prediction through a secondary algorithm to select the most reliable reference frame (i.e., frame with less total expected distortion). + \begin{figure}[tb] - \centering - \includegraphics[width=0.95\columnwidth]{./022_ErrorResilience/Figures/RefPicSelection.png} - \caption{Examples of dynamic reference picture selection used in error resilient video transcoding \cite{ErrorResilienceRPSFeb2009}.} - \label{fig2:ReferencePictureSelection} + \centering + \includegraphics[width=0.95\columnwidth]{./022_ErrorResilience/Figures/RefPicSelection.png} + \caption{Examples of dynamic reference picture selection used in error resilient video transcoding \cite{ErrorResilienceRPSFeb2009}.} + \label{fig2:ReferencePictureSelection} \end{figure} -In \cite{ChannelAdaptiveVideoStreaming2002} network feedback is combined with path diversity to select the reference frames that are more likely to be received at the decoder. -In the proposed approach, the feedback information is not directly used to select the reference frames. -Instead, the reference frames are selected through an end-to-end distortion estimation process, taking into account the coding distortion and distortion resulting from channel errors (\ac{ec} distortion is also taken into account). -In \cite{ErrorResilienceModeHEVC2017} a \ac{roi} is defined based on the motion objects to limit the regions where the mechanism of reference picture selection is applied. -Intra refresh is also used, based on the temporal distance to the previous intra-refresh frame. -This method not only achieves noticeable quality gains, when compared with the reference \ac{hevc} standard, but also when comparing with traditional reference selection schemes. -In~\cite{IntraRefreshRefSelectionJan2009} an algorithm to introduce intra refresh points at the \ac{mb} level was proposed. -This method takes into consideration the decoding distortion assuming that frame-copy concealment is applied for \ac{ec} when packet loss occurs. -The refresh blocks are encoded using either intra prediction or inter prediction through a secondary algorithm to select the most reliable reference frame (i.e., frame with less total expected distortion). - -In the methods mentioned above the dependencies between the frames are analysed and reduced in order to decrease the error propagation, in cases of data loss. +In the methods mentioned above, the dependencies between frames are analysed and reduced in order to decrease the error propagation, in case of errors or data loss. Since the existing video coding standards are strongly based on predictive coding, these methods present an efficient approach in highly erroneous conditions. % However, the method in~\cite{RDReferenceSelectionJul2012} uses more complex distortion estimations, and in ideal conditions multiple loop encoders are needed to select the reference frames optimally. However, although these closed-loop techniques may offer a reliable control of the decoder distortion, the feedback required from the receiver side and multipath networking are not always available neither feasible in real-time and unidirectional channels (e.g. broadcast). @@ -279,43 +284,46 @@ However, although these closed-loop techniques may offer a reliable control of t \section{Data partitioning and redundant coding} \label{Section22:DataPartitionRedundant} -In a compressed video streams different symbols have different relevance in terms of their impact on error propagation. -Therefore, there is an unequal contribution of different syntax elements to the decoding distortion by error propagation, depending on the lost symbols. +In compressed video streams different symbols have different relevance in terms of their impact on error propagation. +Therefore, there is an unequal contribution of different syntax elements to the decoding distortion and error propagation, depending on the symbols that are lost in each case. Data partition techniques have been developed to provide a mechanism to cope with the unequal importance of coded information by grouping different parts of a compressed bitstream, according to their importance to the decoded quality. Thus, some of these groups can be protected more efficiently than others using \ac{uep} schemes. These are normally associated with \ac{fec} algorithms, whereby multi-levels of \ac{fec} are applied to achieve different levels of error protection. This is presented and discussed with more detail in Sub-section~\ref{SEC22:UEP} -In a single channel transmission, the most important data or syntax elements (e.g., headers) can be protected with stronger channel codes than the remaining ones. +In single channel transmission, the most important data or syntax elements (e.g., headers) can be protected with stronger channel codes than the remaining ones. In case of multi-channel communications, the important data can be routed over the most reliable channels. -Moreover, to achieve higher quality in a multi-channel transmission one can use extra information to increase the error resilience of video streams, in order to increase the probability of being able to decode them in case of errors. -For instance, redundancy can be added either in an explicitly (e.g., redundant picture) or implicitly (e.g., multiple description coding). Section~\ref{SEC22:RedundantPics}~and~\ref{SEC22:MDC} present a detailed review of these two approaches, respectively. +Moreover, to achieve higher quality in multi-channel transmission, one can use extra information to increase the error resilience of video streams, in order to increase the probability of being able to decode them in case of errors. +For instance, redundancy can be added either using explicit (e.g., redundant picture) or implicit (e.g., multiple description coding) mechanism. Section~\ref{SEC22:RedundantPics}~and~\ref{SEC22:MDC} present a detailed review of these two approaches, respectively. -\subsection{Standard methods for data partitioning and flexible macroblock ordering} +\subsection{Methods for data partitioning and macroblock ordering} \glsreset{fmo} -The concept of data partitioning was firstly introduced by the H.263 coding standard to group the coded data into different categories~\cite{H264OverIPJul2003}. -It was further improved, allowing the symbols in each slice to be divided into three partitions: +The concept of data partitioning was firstly introduced in the H.263 coding standard to group the coded data into different categories~\cite{H264OverIPJul2003}. +Then, it was further improved in \ac{h264}, allowing the symbols in each slice to be divided into three partitions: \begin{itemize} \item Type A: headers, \ac{mb} types, quantisation parameters and motion information. \item Type B: intra-coded information, such as, intra blocks signalling and intra coefficients. This type cannot be used without the partitions of type A being present. \item Type C: inter-coded information and inter coefficients. This type also requires the information of type A, but can be decoded without intra-coded information (type B). \end{itemize} -These partitions have different levels of importance, being data partitions of type A the most important as they carry the header information that are required by the remaining partitions. Then, as partitions of type B carry intra-coded information they are more important than type C information, as it can stop temporal error propagation. +These partitions have different levels of importance, according to the type of information carried in each one. +Data partitions of type A are the most important as they carry the header information that is required to decode the remaining partitions in the video stream. +Then, since partitions of type B carry intra-coded information, they are more important than type C, because they can stop temporal error propagation. -The \ac{h264} introduced an error resilience tool named \ac{fmo} technique, which allows each \ac{mb} to be freely assigned to a specific slice group (i.e., a set of slices), organised in a mixed-up fashion~\cite{H264FMO2006}. +An error resilience tool named \ac{fmo} technique was introduced in \ac{h264}, allowing each \ac{mb} to be freely assigned to a specific slice group (i.e., a set of slices), organised in a mixed-up fashion~\cite{H264FMO2006}. It supports up to eight slice groups in each picture, and the \acs{mb} within a group can be assigned to several slices in the default raster scan order. The case where only one slice group is used corresponds to the case where the \ac{fmo} tool is disabled. \ac{fmo} increases the error resilience by partitioning the bitstream into several groups, which is able to reduce the error propagation. -Results show that the use of \ac{fmo} contributes to enhance the subjective and objective quality, especially for high packet loss rates at cost of an affordable overhead~\cite{H264FMO2006}. +Simulation results show that the use of \ac{fmo} contributes to enhance the subjective and objective quality, especially for high packet loss rates at cost of an affordable overhead~\cite{H264FMO2006}. Six types of slice groups are supported by \ac{fmo}. -Figure~\ref{fig2:FMOTypes} shows \ac{fmo} types 0 to 5, since the type 6 as no defined pattern. +Figure~\ref{fig2:FMOTypes} shows \ac{fmo} types 0 to 5, since the type 6 has no defined pattern. In the \ac{fmo} type 0 each slice group is defined by a maximum number of \acs{mb}, and a new slice group begins whenever his maximum value is reached. The same slice group can be cyclic repeated within a frame. \ac{fmo} type 1 corresponds to scattered slices, where the \acs{mb} are assigned to each slice group with a fixed formula. Figure~\ref{fig2:FMOTypes} represents the \ac{fmo} type 1 with two slice groups resulting in a checkerboard pattern. In the \ac{fmo} type 2, slice groups are defined by rectangular shapes bounded by the top-left and bottom-right corners. -Although the slice groups are allowed to overlap each other, each \ac{mb} may only belong to one slice group. \ac{fmo} type 3, 4 and 5 are special types, each one allowing the configuration to change on every frame. -Nevertheless, only two slice groups are allowed in \ac{fmo} type 3 to 5, in order to keep the signalling overhead affordable. -The last \ac{fmo} type is the most generic case, where the slice groups are explicitly defined using high level packets (i.e., signalling \ac{nal} units), allowing more flexibility at the cost of higher signalling overhead. +Although the slice groups are allowed to overlap each other, each \ac{mb} may only belong to one slice group. +Similar to \ac{fmo} type 0, in the types 3, 4 and 5, slice groups are defined by a maximum number of \acs{mb}. Nevertheless, these are special types, since it is allowed that the configuration to change on every frame. +In order to keep the signalling overhead affordable, only two slice groups are allowed in these types. +The \ac{fmo} type 6 is the most generic case, where the slice groups are explicitly defined using high level packets (i.e., signalling \ac{nal} units), allowing more flexibility at the cost of higher signalling overhead. \begin{figure}[tb] \centering @@ -327,17 +335,18 @@ The last \ac{fmo} type is the most generic case, where the slice groups are expl In the past, \ac{fmo} has been widely used to increase the error robustness of video transmissions in different communication scenarios \cite{FMOErrorResilience2007,FMOResilience2010,FMOResilience2011,FMOROI2012,MotionEnergyDataPart2014}. In ~\cite{FMOErrorResilience2007} \ac{fmo} is used to increase the \ac{ec} performance and thus reduce the error propagation in prioritised video transmission. -The method divides the \acs{mb} into the two slice groups based on an estimation of the \ac{ec} distortion using a two-pass approach. Then, the slice group which has the most impact on the \ac{ec} distortion is transmitted using a reliable channel, and it is assumed to be received error-free. +The method divides the \acs{mb} into the two slice groups based on an estimation of the \ac{ec} distortion using a two-pass approach. +Then, the slice group which has the most impact on the \ac{ec} distortion is transmitted using a reliable channel, and it is assumed to be received error-free. Results show that the proposed approach is able to outperform the dispersed mode (\ac{fmo} type 1). Slice classification can also be based on the weighted sum of the motion vector information and the rate-distortion cost~\cite{FMOResilience2010}. -As in \cite{FMOErrorResilience2007}, the \acs{mb} are ranked based on the importance and distributed in the two groups with the same number of blocks. +As in \cite{FMOErrorResilience2007}, the \acs{mb} are ranked based on their importance and distributed in the two groups with the same number of blocks. In order to reduce the mapping complexity of the \acs{mb} into groups, the same map can be repeated for several subsequent P-frames \cite{FMOResilience2011}. This reduces the computational complexity of the \ac{mb} grouping while achieving similar error resilience performance. The method in \cite{FMOROI2012} splits the video frames into two regions, which are then grouped in different slice groups. To classify the different \acs{mb}, spatial information is extracted from the \ac{mb} bit allocation and temporal information is extracted based on the estimation of the \ac{ec} distortion. This technique is able to reduce the number of erroneous \acs{mb} up to 82\%, achieving a quality gain of up to 3.97\%, in terms of \ac{psnr}, in comparison with the case where \ac{fmo} is not used. Motion energy was also used to classify \acs{mb} into different groups \cite{MotionEnergyDataPart2014}, by comparing the motion energy of a given \ac{mb} with the motion energy of its neighbours and the average motion energy of the entire frame. -This allowed the authors to obtain a dynamic threshold which is then used to determine the \ac{mb} importance. In comparison with other \ac{uep} error protection schemes, this method achieve good quality improvements under different \acs{plr}. +This allowed the authors to obtain a dynamic threshold which is then used to determine the \ac{mb} importance. In comparison with other \ac{uep} error protection schemes, this method achieves good quality improvements under different \acs{plr}. In \cite{FMORedudantSlices_TB2007}, a new \ac{fmo} type is proposed combining the slice groups of type 1 and 3 in order to obtain a checkerboard pattern in a box-out scanning order, with the objective of increasing the scattering of lost \acs{mb}. Results show that the proposed \ac{fmo} type is able to outperform the existing types under single and burst loss events. The method described in \cite{FMOTypeROI2014} introduces a new type in order to efficiently divide the video content of each frame in two regions with different levels of importance, based on the \ac{fmo}~type~6. @@ -351,12 +360,11 @@ By proposing a new \ac{fmo} type they are able to signal several \ac{roi} areas \glsreset{uep} \Gls{uep} techniques are used to protect different parts of a video stream against errors according to their importance for the decoding quality. \ac{uep} approaches take advantage of the video stream characteristics that result from different types of content, both at the temporal and spatial level. This allows categorisation of coded data into different groups, by exploiting the fact that different regions are affected differently by transmission errors. -This not only ensures that those regions will be delivered with higher quality, but also leads to efficient use of the total overhead required to achieve increased error robustness in a video transmission. +This not only ensures that more important regions will be delivered with higher quality, but also leads to efficient use of the total overhead required to achieve increased error robustness in a video transmission. A wide variety of \ac{uep} schemes have been presented in the literature to optimise the performance of video communication over networks where the probability of packet loss is not negligible. It has been observed that, within a \ac{gop}, different frame types (e.g. I, P, or B frames) have different levels of contribution to the reconstructed video quality, when transmission losses occur. This has been exploited for \ac{uep} in video transmission, as shown in~\cite{UEP_TM2005,UEPMotion_TM2006}. - In \cite{UEPLayered_TB1999} an evaluation of layered video was proposed to increase the error robustness, by using redundant packets for helping the recovery of lost ones. In order to achieve reduced overhead for error protection, each video layer has a different importance and is protected using different levels of redundancy, thus guaranteeing that packets carrying data of lower layers have higher probability of being decoded without errors. This is a simple approach, not depending on the network capabilities, neither requiring a deep knowledge of the video contents. @@ -372,7 +380,7 @@ This work was extended in~\cite{UEPMotion_ICSPCC2012,UEPMotion_ICSPCS2013} to us Another work dealing with \ac{uep} is described in~\cite{Zhang20131390}, where the transmission distortion is estimated at the packet level, to assess the impact of each packet in the error propagation. In this case, different priorities are applied to each packet, allowing different levels of error protection. This was also used to schedule the packet transmission through multi-path networks\cite{PacketScheduling2010}. -Furthermore, in order to improve the classification of different regions of a video stream a perceptual algorithm can be used. +Furthermore, in order to improve the classification of different regions of video streams, a perceptual algorithm can be used. This strategy increases the correlation between the classification and subjective impact of errors. In \cite{UEPSaliency_TCSVT2010} a method based on a combination of two parameters is described: a spatial parameter, based on a perceptual attention model, and a temporal parameter, based on the frame position within the \ac{gop}, which are used to improve the efficiency of \ac{fec} assignment. Experimental results show a significant quality improvement, both using objective metrics and subjective evaluation. @@ -391,7 +399,7 @@ In~\cite{UEPVideoDegradation_ISCAS2017} a quality degradation model is proposed simulating error events and measuring the quality using the \ac{psnr} metric. From empirical results, a model was devised to predict the quality under different events. Then, the model is used to control the amount of redundant codes to be applied at the network level. -The scalable extension of the \ac{hevc} standard was also used to support \gls{uep} for video storage in cloud environments~\cite{UEP_TM2018}, by combining the layer information (i.e., base or enhancement layer) and the picture type (i.e., I, P or B-frame) to maximise the performance of the \ac{fec} codes, both at the temporal and quality/resolution levels. +The scalable extension of the \ac{hevc} standard was also used to support \gls{uep} for video storage in cloud environments~\cite{UEP_TM2018}, by combining the layer information (i.e., base or enhancement layer) and the picture type to maximise the performance of the \ac{fec} codes, both at the temporal and quality/resolution levels. @@ -408,7 +416,7 @@ For example, the primary picture may be coded with fine quantisation, while the On the one hand, if the primary picture is received, the redundant information is discarded. On the other hand, if errors occur and the primary picture is lost, the redundant one is used to provide a lower level of reconstructed quality and limiting the error propagation. -Zhu \emph{et al.} investigated the paradigm of redundant picture coding~\cite{RedudantPicsJan2009}, considering different approaches to develop a mixed method based on both reference picture selection to obtain a clean decoding, without error propagation, and hierarchical allocation of the redundant pictures to increase error robustness. +Zhu \emph{et al.} investigated the paradigm of redundant picture coding~\cite{RedudantPicsJan2009}, considering different approaches to develop a mixed method based on both \ac{rps} to obtain a clean decoding, without error propagation, and hierarchical allocation of the redundant pictures to increase error robustness. In~\cite{VPxResiliencePCS2016} the key frames are encoded and transmitted at a lower bitrate to be used in case of error, to replace the missing frame. Redundant pictures are a good mechanism to improve performance under error prone networking conditions, since they introduce secondary pictures that can be decoded when the primary picture is not available. @@ -434,19 +442,18 @@ These approaches increase the overall system performance by presenting an effici In \ac{mdc} schemes two or more bitstreams, named descriptions, are encoded and transmitted through different channels, such that an acceptable level of quality is achieved when only some of them reach the decoder \cite{MDC1993,MDC2005}. Figure~\ref{fig2:MDCSystem} illustrates a basic architecture of a \ac{mdc} encoder and decoder using two descriptions. In this example, the encoder-side creates two descriptions, i.e., $R_1$ and $R_2$, which shall be transmitted using two different channels, reducing the probability of the same video data being lost at the same time in both descriptions. -As shown in this diagram, three different outputs may be produced at the decoder-side: both descriptions are correctly received and the central decoder is used($D_0$), or either one of the descriptions is not received and then one of the side decoders is used ($D_{1,1}$ and $D_{1,2}$). +As shown in this diagram, three different outputs may be produced at the decoder-side: both descriptions are correctly received and the central decoder is used ($D_0$) or, either one of the descriptions is not received and then only one side decoder is used ($D_{1,1}$ or $D_{1,2}$). When both bitstreams are received, the central decoder combines the information carried by each description achieving higher quality than any single side decoder. -In the \ac{mdc} encoder the redundancy among descriptions can be controlled by changing the correlation between them. Higher correlation leads to higher redundancy, but also higher quality in the output of the side decoders. -In contrast with the case of redundant pictures, the redundant \ac{mdc} streams do not carry the same information, which means that if both are received, then the decoded quality may be increased. +In \ac{mdc} encoding, the redundancy among descriptions can be controlled by changing the correlation between them. +Higher correlation leads to higher redundancy and also higher robustness, also leading to higher quality in the output of the side decoders. +In contrast, with the case of redundant pictures, the redundant \ac{mdc} streams do not carry the same information, which means that if both are received, then the decoded quality may be increased. However, a combination of \ac{mdc} and redundant pictures can also be used to achieve higher quality gains \cite{MDCRedundantPicsJan2010}. Different types of \ac{mdc} approaches \cite{MDCHierarBFeb2012,MDCCorreiaJun2012} can be used to improve the quality under transmission errors, either by taking advantage of hierarchical B-pictures~\cite{MDCHierarBFeb2012} or by introducing extra auxiliary information to limit the error propagation in case of single description decoding~\cite{MDCCorreiaJun2012}. Although these techniques can be very effective in multi-path networks, a significant amount of overhead is required to create the different descriptions~\cite{MDC_TCSVT2011}. % Double check this reference %\HERE -%\cite{MDCRedundantPicsJan2010} -%\cite{MDC_TCSVT2011} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% @@ -456,8 +463,8 @@ Although these techniques can be very effective in multi-path networks, a signif %\glsreset{ec} -In general, \ac{ec} functions are implemented as a post-processing step in the decoder and being the last module in the communication chain capable of handling errors~\cite{ConcealmentReview1998}. -Thus, reconstruction of missing data should take place within the video decoder to minimise the perceived effects of lost data. +In general, \ac{ec} functions are implemented as a post-processing step in the decoder, thus they are the last module in the communication chain capable of handling errors~\cite{ConcealmentReview1998}. +In this case, reconstruction of missing data should take place within the video decoder to minimise the perceived effects of lost data. %Overall, to enhance the video quality, the design of the transmission system must incorporate adequate \ac{ec} tools. This section reviews several \ac{ec} methods based on spatial and temporal redundancies, as these play an important role in the overall performance of robust communication systems. @@ -493,40 +500,13 @@ improve the estimation accuracy of the lost region, the edges and image transiti \label{FIG3:pixel_interpolation} \end{figure} -Directional interpolation improves the performance of the pixel recovery, by firstly estimating the lost edges, using either correctly received or previously recovered pixels. To obtain the best interpolation direction, edges are estimated using the image gradient, for example applying the Sobel operator \cite{SignalImageVideoProcessing2012}. -%This operator gives the first-order approximation of the gradient of the image. This is widely used in image processing to locate the image edges, and uses an horizontal and vertical mask given by: -%\begin{equation} -%G_x = -%\begin{bmatrix} -%-1 & 0 & 1 \\ -%-2 & 0 & 2 \\ -%-1 & 0 & 1 -%\end{bmatrix},\ \ -%G_y = -%\begin{bmatrix} -%-1 & -2 & -1 \\ -% 0 & 0 & 0 \\ -% 1 & 2 & 1 -%\end{bmatrix}. -%\end{equation} -%The convolution between the image pixels and the Sobel matrixes results in both horizontal ($g_x$) and vertical ($g_y$) gradients, from which the magnitude and the angular direction are computed using, -%\begin{equation} -%\begin{aligned} -%G &= \sqrt{g^2_x + g^2_y}, \\ -%\theta &= tan^{-1} \left( \frac{g_y}{g_x} \right).\\ -%\end{aligned} -%\end{equation} -The Sobel operator is an efficient algorithm to detect edges, as it can reach extreme values at the image edges. +Directional interpolation improves the accuracy of the pixel recovery, by firstly estimating the lost edges, using either correctly received or previously recovered pixels. To obtain the best interpolation direction, edges are estimated using the image gradient, for example applying the Sobel operator \cite{SignalImageVideoProcessing2012}. +This operator is an efficient algorithm to detect edges, as it can reach extreme values at the image edges. The usage of this operator for \ac{ec} was evaluated in \cite{IntraConc2010,FineDirectTCE2006,SpatialEdgeConsecutiveBlocks2004,SpatialEdgeModConsecutiveBlocks2005,SpatialInterpolation2008}, and it is characterised as a fast and efficient method due to its small kernel \cite{GradEstimation2009}. -% \abbrev{FC}{Frame-Copy} -% \abbrev{MC}{Motion-Copy} - \subsection{Temporal error concealment techniques} -%\subsubsection{Frame and motion copy} - -The conventional way to recover missing frames in the temporal domain are the so-called \ac{fc} and \ac{mc}. +The conventional methods used to recover missing frames in the temporal domain are the so-called \ac{fc} and \ac{mc}. These \ac{ec} methods were implemented for the \ac{h264} reference software in \cite{H264RefConc2006}. The \ac{fc} method consists in repeating (copying) the frame that has the shortest temporal distance, in order to substitute the missing one, into the time instant where the loss occurred. The \ac{mc} method uses the motion information of the closest frame to recover the missing one, thus the missing pixels are recovered through motion compensation. @@ -539,13 +519,15 @@ Results in \cite{BiDirMC2011} show quality gains up to $4.99$~dB (\ac{psnr}) whe In \cite{2DRecursiveRefinement2010} the \ac{mc} method performance was increased by using a refinement algorithm to improve the accuracy of the \acs{mv} obtained from the closest decoded frame. The \ac{mv} differences between the vectors of successive frame pairs are recursively calculated along with the next refinement, until the changes become negligibly small (e.g., close to zero). This approach has shown to improve the performance of the \ac{mc} method, with gains up to $0.47\ dB$ (\ac{psnr}) for a \ac{plr} of $20\%$. -Moreover, + \subsubsection{Boundary match algorithms} -% \abbrev{ICMV}{Interpolated Candidate Motion Vectors} -\Gls{bma} are used to recover a missing region based on the available neighbouring regions. By using the missing block neighbourhood, a new \ac{mv} can be estimated for the missing block. Then, recovery based on motion compensation can be achieved. -A low complexity approach may use the boundary pixels to search for the \ac{mv} based on a spatial smoothness constraint imposed on the boundaries of the lost block \cite{BMAAlgorithm2008}. Figure~\ref{FIG3:BMAPixelInformation} represents the boundary information used to estimate the \ac{mv} for the lost block from a set of different candidates. + +\Gls{bma} are used to recover a missing region based on the available neighbouring regions. +By using the missing block neighbourhood, a new \ac{mv} can be estimated for the missing block. +Then, the recovery of the missing pixels if accomplished based on motion compensation. +A low complexity approach may use the boundary pixels to search for the \ac{mv}, based on a spatial smoothness constraint imposed on the boundaries of the lost block \cite{BMAAlgorithm2008}. Figure~\ref{FIG3:BMAPixelInformation} represents the boundary information used to estimate the \ac{mv} for the lost block from a set of different candidates. The \ac{bma} uses the available \acs{mv} provided by the top (T), bottom (B), left (L) and right (R) blocks to simulate different reconstructions of the lost block. Then, the optimal reconstruction is selected according to the sum of differences, between the pixels on the internal boundary in the recovered block and the corresponding ones on the external boundary (see Figure~\ref{FIG3:BMAPixelInformation}). @@ -558,70 +540,35 @@ Then, the optimal reconstruction is selected according to the sum of differences \end{figure} An implementation of the \ac{bma} was proposed in \cite{BMAAlgorithm2006}, by analysing the change of magnitude and angle between the \acs{mv} of the neighbouring blocks and the co-located ones in the previous frames. The average change is applied to the \ac{mv} of the block co-located with the lost one to compute the \ac{mv} candidate for recovery. -An improvement to \ac{bma} was obtained with the outer boundary matching algorithm\cite{OBMAMethod2008}, also known as decoder \ac{mv} estimation algorithm in \cite{DMVE2000}. +An improvement to \ac{bma} was developed using with the outer boundary matching algorithm\cite{OBMAMethod2008}, also known as decoder \ac{mv} estimation algorithm in \cite{DMVE2000}. In this method the distortion is calculated by the difference of two outer boundaries of reconstructed block, instead of an internal and external boundary in \ac{bma}. Although \ac{bma} and outer \ac{bma} have the same design principle of minimising the mismatch distortion, in the latter the distortion is measured in the outer boundaries, \emph{i.e.}, external boundaries of the reconstructed block. It was observed in the experimental results of \cite{OBMAMethod2008} that the outer \ac{bma} has better performance in video recovery than conventional \ac{bma}. The performance improvements are explained by the distortion minimisation of the outer boundary, which incorporates the edge information located in the neighbourhood of the lost block in the decision process to select the most accurate \ac{mv}. -%Moreover, in the method presented in \cite{OBMAMethod2008} further \ac{mv} candidates were introduced, a refined local search is performed and multiple boundary layers, \emph{i.e.}, increase the overlapped region to measure the distortion and to select the final MV, were tested. The experimental results revealed that the usage of a single layer match gives better performance than multiple layers, while selective search gives excellent trade-off between the image recovery and computation complexity -% In \cite{DynamicBMA2010} a novel approach to measure the spatial distortion on boundaries of the reconstructed block is presented. Instead of using only horizontal and vertical directions, diagonal and anti-diagonal are also tested, and the best direction is chosen from the reference frame. Since block boundaries may change abruptly, instead of minimising the distortion function the proposed method aims to approximate this distortion to an optimal difference. This optimal difference are determined in the neighbourhood of the co-located block in the reference frame. In \cite{BMAAlgorithm2008} a novel distortion function was proposed, to choose among different candidate MVs. The distortion function exploits both spatial and temporal smooth constraints. The temporal function is based on the outer boundary distortion, as in \cite{OBMAMethod2008}. The spatial distortion aims to reduce the gradient variances in boundaries of the reconstructed block, so the image edges may be guaranteed. -Several experimental results were obtained for different loss events, covering from an isolated block loss to the loss of a complete row of blocks. The proposed method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method. Moreover, subjective analysis reveal that this method recover the edges in the lost image. However, the method uses complex mathematical operations, which may not be practical in real time applications. +%Several experimental results were obtained for different loss events, covering from an isolated block loss to the loss of a complete row of blocks. +This method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method. +However, although subjective analysis revealed that this method is able to recover the edges in the lost image, complex mathematical operations are required, which may not be practical in real time applications. \subsubsection{Motion vector extrapolation} -As mentioned before, the \ac{mc} method is a simple way to reconstruct an estimation of a missing frame using motion information from previously decoded frames. Although this method can provide an acceptable performance comparing with the \ac{fc}, as shown in \cite{H264RefConc2006}, more complex algorithms that use the assumption of motion continuity across consecutive frames are proposed to recover, not only missing blocks, but also whole frames in compressed video. - -% \abbrev{MMA}{Multi-frame Motion vector Averaging} - -% To recover the entire missing frame a multi-frame motion vector averaging (MMA) algorithm was proposed in \cite{MMATempConc2003}. The method uses \acs{mv} of a few past frames (from $t-1$ to $t-L$), in order to estimate the forward \ac{mv} for each pixel in the last received frame. Then, each pixel of the last frame is projected onto the missing one. More specifically, the method proposed in \cite{MMATempConc2003} recovers the missing frame at instant $t$ using the following steps: -% \begin{enumerate} -% \item Generation of a \ac{mv} history based on the previously decoded frames from $t-1$ to $t-L$, with $L \in \left[ 2,5 \right]$, in order to understand which pixels were used to predict the ones of the frame at $t-1$. -% \item The \ac{mv} history is used to obtain a set of forward MV, from the frame $t-1$ to the missing one (at instant $t$). For each pixel, the forward \ac{mv} is measured by averaging the \ac{mv} history. -% \item A spatial regularisation of the \ac{mv} field is performed, by applying a median filter with a kernel size of $12$, to approximate an \mbox{edge-preserving} regularisation \cite{MedianFilterRegul2000}. -% \item The reconstruction of the missing frame is performed, at a half-pixel level, through the previously obtained forward MVs. The overlapped pixels are averaged, in order to obtain the final pixel value. -% \item The unfilled regions are recovered, with the median value of a $7\times7$ window -% \item Finally, the fully reconstructed frame at a half-pixel level is filtered using a mean filter with a $2\times2$ kernel, and down sampled. -% \end{enumerate} - -% This method usually produces relatively high quality in the reconstructed frame, but in low motion cases it behaves worse than the \ac{fc} method under a few conditions, as indicated by the experimental results in \cite{MMATempConc2003}. This phenomenon happens because the oldest reference frames usually do not have \ac{mv} similar to the missing one. Moreover, if $L$ increases, the complexity of the proposed algorithm may not be useful to be used in \ac{ec}, due to complexity and memory requirements to determine the \ac{mv} history. +As mentioned before, the \ac{mc} method is a simple way to reconstruct an estimation of a missing frame using motion information from previously decoded frames. +Although this method can provide an acceptable performance comparing with the \ac{fc}, as shown in \cite{H264RefConc2006}, more complex algorithms that use the assumption of motion continuity across consecutive frames are used to recover the missing information. \glsreset{mve} - - -\ac{mve} \cite{BlockBasedMVE2002,BlockBasedMVEOverllaped2011} methods are quiet simple yet efficient techniques that reconstruct the \acs{mv} of a lost frame, which in turn are used for motion compensation. +\ac{mve} \cite{BlockBasedMVE2002,BlockBasedMVEOverllaped2011} methods are efficient techniques to reconstruct the \acs{mv} of a lost frame, which in turn are used for motion compensation. These \acs{mv} are extrapolated from the last decoded frame \mbox{(reference frame)} onto the missing one~\cite{BlockBasedMVE2002}. Each block in the reference frame is projected to the missing one with its own direction given by the corresponding extrapolated \ac{mv} ($mv'$). An example of \ac{mve} is shown in Figure~\ref{FIG3:MVEExtrapolation}, where the vectors of frame $t-1$ are projected onto the missing frame at instant $t$. Note that, in the reference frame ($f_{t-1}$) the \acs{mv} may point to different frames ($f_{t-2}$~and~$f_{t-3}$), previously decoded, so \ac{mve} must take into account the temporal distance between different frames. To improve the accuracy of \ac{mve}, the damaged region is divided smaller blocks and, according to the matching area between the extrapolated block and the damaged one, the best \ac{mv} is chosen. In order words, the \ac{mv} for the each block is obtained from the extrapolated block with the highest number of pixels matching the missing locations. -Note that the final motion information need to be aligned to a fixed sise grid, e.g., $4\times4$ grid, which leads to a misalignment of the extrapolated \acs{mv}. +Note that the final motion information need to be aligned to a fixed size grid, e.g., $4\times4$ grid, which leads to a misalignment of the extrapolated \acs{mv}. Alternatively, in \cite{PixelBasedMVE2004} several blocks are extrapolated to the missing frame, resulting in overlapped regions, which are recovered using a weighted average of the multiple pixels candidates. The weight coefficients assume a maximum value in the center of the extrapolated region, and a minimum value at the boundaries. -% -%Since the choice of the best candidate \ac{mv} relies on the overlapped area of the extrapolated block with the lost region, the method may give erroneously estimated \ac{mv} in large motion scenes. The example of Figure~\ref{FIG3:MVEPixelBased} shows a lost $8\times8$ block (in the centre of the figure) and four possible extrapolated \acs{pu} that can be used to estimate lost the motion information. As illustrated in the figure, although the largest overlapped area belongs to the PU$_2$, the overlapped pixels correspond to a small portion of the whole missing block. Moreover, the majority of the lost block's area was not covered by any extrapolated \ac{pu}. Thus, motion recovery using only the information available in PU$_2$ will not properly characterise the motion information for all lost pixels. -% -%\begin{figure}[t] -%\centering -%\input{./022_ErrorResilience/Figures/PixelBasedMVE} -%\caption{Pixel based motion vector extrapolation \cite{PixelBasedMVE2004}.} -%\label{FIG3:MVEPixelBased} -%\vspace{3mm} -%\end{figure} -% -%To overcome the previous limitation and to achieve smoother and finer results, a \ac{pmve} algorithm was proposed in \cite{PixelBasedMVE2004}. This motion vector extrapolation method is divided into two cases: -%\begin{enumerate} -% \item The pixels in the lost region that are covered by at least one extrapolated block, such as circle points in Figure~\ref{FIG3:MVEPixelBased}, are recovered using the average of the \acs{mv} of the overlapped block. -% \item Those pixels which are not covered by any extrapolation, such as the triangle point in Figure~\ref{FIG3:MVEPixelBased}, the \ac{mv} of the co-located pixel in the previous frame is used. -%\end{enumerate} -% -%Then, if the estimated \ac{mv} is $(MV_x,MV_y)$, then the estimation of the missing pixel, $p_m(x,y)$, is obtained by: -%\begin{equation}\label{EQ:MotionComp} -% p_m(x,y) = p_r(x+MV_x,y+MV_y), -%\end{equation} -%where $p_r$ refers to the pixels in the previous frame. This pixel-based \ac{mve} provides similar results than the block-based \ac{mve} algorithm in low motion scenarios. Nevertheless, it is able to overcome the previous method in large motion scenes. -An extension to the previous approach was considered in \HIGHLIGHT{\cite{PelBasedMVE_TCE2007}}, by using the \acs{mv} of subsequent available frames. Thus, to estimate the lost motion information, forward and backward extrapolations are performed, in order to obtain two candidates to each lost pixel. Then, the lost pixels are obtained through motion compensation using the \acs{mv} determined by the average of two \ac{mv} candidates. The experimental results revealed that the bi-directional \ac{pmve} outperforms the isolated use of forward or backward \ac{pmve}. Moreover, the bi-directional \ac{pmve} achieves gains up to $1.85$ dB and $1.32$ dB (\ac{psnr}) comparing with the work in \cite{MMATempConc2003} and \ac{mve} \cite{BlockBasedMVE2002}. +An extension to the previous approach was considered in \cite{PelBasedMVE_TCE2007}, by using the \acs{mv} of subsequent available frames. Thus, to estimate the lost motion information, forward and backward extrapolations are performed, in order to obtain two candidates to each lost pixel. Then, the lost pixels are obtained through motion compensation using the \acs{mv} determined by the average of two \ac{mv} candidates. The experimental results revealed that the bi-directional \ac{pmve} outperforms the isolated use of forward or backward \ac{pmve}. Moreover, the bi-directional \ac{pmve} achieves gains up to $1.85$ dB and $1.32$ dB (\ac{psnr}) comparing with the work in \cite{MMATempConc2003} and \ac{mve} \cite{BlockBasedMVE2002}. +% TODO: Check reference PelBasedMVE_TCE2007 + \begin{figure}[t] \centering @@ -631,12 +578,9 @@ An extension to the previous approach was considered in \HIGHLIGHT{\cite{PelBase \end{figure} In \cite{2DConcHybridMotion2010} a \ac{hmve} algorithm was proposed, to improve the accuracy of the \acs{mv} of the \ac{pmve} method, by using the extrapolated vectors at the pixel and block level. Moreover, the \acs{mv} that were wrongly extrapolated are discarded in order to obtain a more accurate MV. -In this \ac{ec} method, firstly a pixel based motion extrapolation is performed and the pixels are organised into three groups. -%Group A contains the pixels that are overlapped by at least one extrapolated block; group B includes pixels that are not covered by any extrapolated block, but belong to a block that is covered, \emph{e.g.}, in Figure~\ref{FIG3:MVEPixelBased} the triangle point was not covered by any extrapolated \ac{mb} but it belongs to $8\times8$ block that is covered by 4 blocks. -%The remaining pixels are included in group C (star pixels in the figure). +In this \ac{ec} method, firstly a pixel-based motion extrapolation is performed and the pixels are organised into three groups. Then, through block-based \ac{mve} two new \acs{mv} are determined: the first one corresponds to the \ac{mv} of the most overlapped block with the current one, and second one is obtained by the weighted average value of the \acs{mv} corresponding to different overlapped blocks. To find the more accurate \ac{mv} from the \ac{mv} set composed described above. To achieve this, the \ac{mv} set is further refined using the distance between each pair of \acs{mv}, in order to discard the wrongly extrapolated \acs{mv}. -% Finally, the pixels of the lost regions (belonging to the groups A and B) are obtained through motion compensation, using the average of the remaining vectors in the \ac{mv} set. The pixels of the group C, although the \ac{mv} set is empty, the same process is applied using the \ac{mv} of the co-located pixel in the previous frame. This algorithm is able to outperform the \ac{pmve} for different packet loss probabilities, as shown by the results in \cite{2DConcHybridMotion2010}. The method was used to recover both intra and inter frames in video sequences compressed with the \ac{h264}, achieving \ac{psnr} gains above $1.09$ dB in the erroneous frames. @@ -645,15 +589,15 @@ For instance, in~\cite{HEVCMCBlockAug2013} the block partitions of the neighbour The reconstructed frame keeps smooth boundaries, leading to higher objective and visual quality. Moreover, in order to select only the reliable motion information, the residue information can also be used to decide the optimal \ac{mv}, as shown in~\cite{HEVCMCResidueDec2013}. -Overall, the aforementioned methods rely on post-processing without any assistance from the encoder-side, which usually leads to inaccurate recovery of the lost information. Thus, in order to achieve higher reconstruction quality, further information need to be shared with the decoder to minimise the residual distortion of \ac{ec}. +Overall, the aforementioned methods rely on post-processing without any assistance from the encoder-side, which usually leads to inaccurate recovery of the lost information. Thus, in order to achieve higher reconstruction quality, further information needs to be shared with the decoder to minimise the residual distortion of \ac{ec}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\section{Concealment-aware error resilience} +\section{Error concealment-aware error resilience} \label{Section22:ConcealmentAware} -The last category of techniques for robust video transmission address those algorithms that help the \ac{ec} process at the decoder-side. Although such techniques might overlap with some of the concepts described above, such as, data partition or redundant information, they have the main goal of improving the \ac{ec} performance. This class of algorithms combine the error resilience capabilities with \ac{ec} methods to achieve an improved overall performance. +The last category of techniques for robust video transmission address those algorithms that help the \ac{ec} process at the decoder-side. Although such techniques might overlap with some of the concepts described above, such as, data partition or redundant information, they have the main goal of improving the \ac{ec} performance. This class of algorithms combine the error resilience capabilities with \ac{ec} methods to achieve an improved overall performance achieving better video quality than in individual \ac{ec} methods. One of the approaches that can be used to improve the \ac{ec} performance is to estimate, at the encoding side, the \ac{ec} distortion introduced by the decoder in case of data loss. In \cite{ConcealmentDrivenSliceOrdering2008} the \ac{ec} distortion estimation is used for slice classification and \ac{uep}. Such distortion is also used to optimise motion estimation~\cite{ConcealmentAwareME2006}, by including the distortion of the subsequent frame (assuming it is lost and recovered) in the \ac{rd} cost optimisation of the current frame. This was also used in the context of \ac{hevc} to improve its error robustness, as shown in \cite{ErrorResilienceModeHEVC2016}. This kind of approaches take advantage of location techniques and data partitioning to improve the reconstruction quality achieved by \ac{ec} algorithms. Normally, they use a single \ac{ec} method that is implemented both at the encoder and decoder, based on temporal algorithms, e.g., \ac{mc} and \ac{mve}. diff --git a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex index a64c531..59121bd 100644 --- a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex +++ b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex @@ -6,7 +6,7 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % CHAPTER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\chapter{Error robustness of HEVC - Evaluation and improvements \VERSION{1}} +\chapter{Error robustness of HEVC - evaluation and improvements \VERSION{2}} \label{Chapter:HEVCErrorResilience} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -14,9 +14,16 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Introduction} -The use of \ac{hevc} in different type of application and video services over networks that are prone to errors and packet losses, requires mechanisms to minimise the effect of such probabilistic events. This chapter starts by presenting an evaluation study about the error resilience of \ac{hevc} in Section~\ref{Section3:ErrorResilienceEvaluation} , with the aim of defining the weak elements of coded streams and the level of error robustness under different transmission loss conditions. The evaluation study covers different coding configuration to evaluate the individual impact of some new coding tools introduced, e.g., increased partition size, intra-refreshing, slice partitioning and advanced motion vector prediction. +\glsreset{mv} +\glsreset{ec} +\glsreset{cu} +\glsreset{ld} +\glsreset{ra} + +The use of \ac{hevc} in different type of application and video services over networks that are prone to errors and packet losses, requires mechanisms to minimise the effect of such probabilistic events. This chapter starts by presenting an evaluation study about the error resilience of \ac{hevc} in Section~\ref{Section3:ErrorResilienceEvaluation} , with the aim of defining the weak elements of coded streams and the level of error robustness under different transmission loss conditions. The evaluation study covers different coding configuration to evaluate the individual impact of some new coding tools introduced in this standard, e.g., increased partition size, intra-refreshing, slice partitioning and advanced motion vector prediction. -In Section~\ref{Section3:DynamicMVRefreshHEVC} a novel method to increase the error robustness of the \ac{hevc} \ac{mv} coding is presented and compared to existing techniques. The proposed method introduces refresh point in the motion prediction used in the \ac{hevc}, which significantly reduces the impact of mismatch \ac{mv} predictions. +Then, in Section~\ref{Section3:DynamicMVRefreshHEVC}, based on the results of the simulation study, a novel method to increase the error robustness of the \ac{mv} coding is presented and compared to existing techniques. +The proposed method introduces refresh points in the motion prediction used in the \ac{hevc}, which significantly reduces the impact of mismatch \ac{mv} predictions. Finally, Section~\ref{Section3:Summary} concludes the chapter. \clearpage @@ -28,22 +35,22 @@ Finally, Section~\ref{Section3:Summary} concludes the chapter. \subsection{Motivation and objectives} -As mentioned before the \ac{hevc} standard has improved its predecessor in terms of coding efficiency, -higher complexity and data dependencies, mostly resulting from advanced predictions structures. -Although the increased complexity may be compensated with more efficient algorithms and faster hardware technologies, the impact of higher coding efficiency on reducing error robustness strongly affects the video quality under packet loss. +As mentioned before, the \ac{hevc} standard has improved over its predecessor in terms of coding efficiency, at cost of +higher complexity and data dependencies, mostly resulting from advanced prediction structures. +Although the increased complexity may be compensated with more efficient algorithms and faster hardware technologies, the impact of higher coding efficiency on reducing error robustness strongly affects the video quality under transmission losses. A critical aspect related to the performance of robust video streaming methods is the error detection accuracy. Since \ac{hevc} streaming is expected to use known transport technologies, such as RTP~\cite{HEVCStreamMay2012}, MPEG-2 TS~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016} and MPEG Media Transport\cite{ATSCMMT2016}, the error detection capabilities of the underlying transport protocols have significant impact on the overall system performance. In general, the error detection operation at the transport layers is decoupled from the video coding layer. -To cope with transmission errors the encoder implements robust coding methods, while the efficient recovery of lost video data is implemented in the decoder, through error concealment algorithms. -Between them, signalling that cope with different protocol layers ensures that the decoder is given the necessary information to identify the lost slices/frames in the video stream. -Therefore, the error robustness of video streams can be investigated on its own, assuming any possible form of streaming technologies and error detection. +To cope with transmission errors, the encoder implements robust coding methods, while the efficient recovery of lost video data is implemented in the decoder, through \ac{ec} algorithms. +Interlayer communication must be used to cope with different protocol layers, ensuring that the video decoder is given the necessary information to identify the lost slices/frames in the video stream. +Therefore, the error robustness of video streams can be investigated on its own, assuming any possible form of streaming/transport technologies and error detection. This is the main focus of the study presented in this section. -There are some studies presented in the past regarding the error robustness of the \ac{hevc} standard, which are worth to be mention as they provide relevant insights for the research of carried out in this research work. In~\cite{StudyHEVCErrorsDez2012,HEVCWireless_2016} both, the coding efficiency and error robustness of the \ac{hevc}, are evaluated and compared against the \ac{h264}. Experimental results confirm that \ac{hevc} has reduced error robustness despite its increased coding efficiency. +There are some studies presented in the past regarding the error robustness of the \ac{hevc} standard, which are worth to be mentioned as they provide relevant insights for the research carried out in this work. In~\cite{StudyHEVCErrorsDez2012,HEVCWireless_2016} both, the coding efficiency and error robustness of the \ac{hevc} are evaluated and compared against the \ac{h264}. Experimental results confirm that \ac{hevc} has reduced error robustness despite its increased coding efficiency. A subjective evaluation study is presented in~\cite{SubjectiveEvalSep2013,QoEHEVCMay2014} revealing that packet loss ratios higher than $3\%$ have significant effect on the perceived video quality. -%In both works it is concluded that the robustness of \ac{hevc} decreases for sequences complex motion. -The vulnerability of \ac{mv} prediction was investigated and its impact on the error robustness was evaluated \cite{DynamicTMVP2011}. Moreover, a method to introduce \ac{mv} refresh points was proposed, achieving average \ac{psnr} gains of approximately 3~dB. +The vulnerability of \ac{mv} prediction was investigated and its impact on the error robustness was evaluated in \cite{DynamicTMVP2011}. +Moreover, a method to introduce \ac{mv} refresh points was proposed, achieving average \ac{psnr} gains of approximately 3~dB. Summarising, the decrease in error robustness is mostly due to the strong data dependencies imposed by the highly complex prediction modes of \ac{hevc}. Since these are selected according to a rate distortion optimisation criterion that assumes error-free transmission, the decoder faces different problems whenever errors or data loss occur in the transmission network \cite{ROPE2007}. The following sub-sections present an error robustness evaluation of \ac{hevc} focused on the impact of the novel coding techniques introduced in the standard, extending the generic studies presented in previous works. This study provides relevant insights for research directions and motivation to investigate solutions for the error robustness problem in \ac{hevc}. @@ -53,15 +60,15 @@ The following case-studies are covered in this experimental evaluation: % \item experiments with and without filter operations, i.e., deblocking filter and \ac{sao}, in order to evaluate their impact on decoding video quality; \item different distances between intra frames, to evaluate the impact of introducing different number of refresh points; \item different slice partitioning schemes; - \item different \ac{mv} coding schemes using different types of \ac{mv} candidates, with analysis at the spatial and temporal level. + \item different \ac{mv} coding schemes using different types of \ac{mv} candidates, with analysis at the spatial and temporal levels. \end{enumerate} \subsection{Characterisation of the experiments} \label{ExperimentalTestSetup} -\input{031_ErrorResilienceEvaluation/TablesSequences.tex} -The experimental setup used in this study to evaluate the error robustness of the \ac{hevc} is based on seventeen test sequences, which were chosen to cover different types of motion and texture complexity. + +The experimental setup used in this study to evaluate the error robustness of the \ac{hevc} was based on seventeen test sequences, which were chosen to cover different types of motion and texture complexity. % Each sequence comprises 240 frames and the whole set covers. Table~\ref{TableTestSequences} presents a summary of the main characteristics of the test sequences. Appendix~\ref{Appen:Sequences} shows a single frame of each test sequence. The spatial and temporal complexity of these video sequences was measured by using the following metrics \cite{ITUTP9102008}: @@ -75,14 +82,16 @@ The spatial and temporal complexity of these video sequences was measured by usi TI = max_{time}\Big\{ std_{space} \big( f_{t}(x,y) - f_{t-1}(x,y) \big) \Big\}. \end{equation} \end{itemize} -The values of \ac{si} and \ac{ti} calculated for the test sequences are shown in the Table~\ref{TableTestSequences} (see second column). +The values of \ac{si} and \ac{ti} calculated for the test sequences are shown in the Table~\ref{TableTestSequences} (second column). +The percentage of inter-coded blocks and the bitrates are also shown in the table and explained below. + +\input{031_ErrorResilienceEvaluation/TablesSequences.tex} For the simulation carried out in this study the HM reference software, version $16.2$, was used \cite{HEVCReferenceSoftware162}. The sequences were encoded with an \ac{idr} period of $16$ frames and all the coding modes enabled, following two recommended configurations: \ac{ld}, i.e., P-frames with 4 references frames, and \ac{ra}, i.e., B-frames with 2 reference frames in each list~\cite{HEVCCommonTests2012}. These prediction structures cover the use of P-frames and hierarchical B-frames~\cite{H264ScalableOverviewSep2007}, which are commonly used in HEVC streaming~\cite{RealTimeHEVC2016}. In order to achieve high quality images for all coded sequences (i.e., \ac{psnr} around 40 dB) different bitrates were used, as listed in Table~\ref{TableTestSequences}. The filtering operations were disabled at slices boundaries, in order to keep the slices self-contained. -Moreover, the ratio of inter-coded blocks of each coded sequence using the default coding parameters is also shown in the third column of the table to illustrate the predictions distribution. - +Moreover, the ratio of inter-coded \acs{cu} of each coded sequence using the default coding parameters is also shown in the third column of the table since this is an important parameter to analyse error propagation. %The following bitrates were selected to achieve high quality (i.e., around 40 dB): 1.0 Mbits/s (Kendo), 1.5 Mbits/s (Book Arrival), 4.0 Mbits/s (BQSquare, Park Scene and Race Horses) and 12.0 Mbits/s (People on Street). % At the decoder-side, error concealment is based on motion compensation from previously decoded frames. @@ -98,14 +107,17 @@ The remaining configurations have been kept at their default values, as given in \subsection{Evaluation of the impact of the partition units size} -%The \ac{hevc} standard introduced new partitioning schemes to allow coding improvements in high resolution videos. These partition structures, named \glsfirst{cu}, are allowed to be as large as $64\times64$ pixels. In the first set of experiments the impact of the maximum \ac{cu} size in the decoded video quality is investigated. In this experiment the maximum \ac{cu} size is set to three different sizes, i.e., 16$\times$16, 32$\times$32 and 64$\times$64 pixels. The lower size is targeting the \ac{mb} size used in the \ac{h264}. Table~\ref{Table3:Quality_BlockSize} shows the average \ac{psnr} obtained for the three test conditions and for four different packet loss ratios. Results for the error-free scenario shows a consistent quality increase as the maximum \ac{cu} size increases, % up to a maximum of 1.41~dB for the Tennis sequence when comparing the maximum \ac{cu} size of $64\times64$ with $16\times16$. -This confirms the principle used by the \ac{hevc} standard, that allowing larger coding units lead to higher coding efficiency, achieving a maximum of 1.41~dB for the Tennis sequence which has the higher \ac{ti} parameter (see Table~\ref{TableTestSequences}). Moreover, it is also noticeable that sequences with higher amount of inter-coded blocks take less advantage of the larger \acs{cu}, in the results of the BQSquare sequence (i.e., highest ratio of inter-coded blocks and lowest quality gain in the error-free case). -Overall, comparing the results for the error-free and data loss cases, it is noticeable that similar quality gains are obtained, which are achieved due to the superior coding efficiency of larger \acs{cu}. In most of the cases the quality gains slightly decrease for increasing \acs{plr}. +This confirms the principle used by the \ac{hevc} standard, whereby larger coding units lead to higher coding efficiency. +For the Tennis sequence, which has the higher \ac{ti} parameter (see Table~\ref{TableTestSequences}), a maximum of 1.41~dB is achieved. +Moreover, it is also noticeable that sequences with higher amount of inter-coded blocks take less advantage of the larger \acs{cu}. +This is noticeable in the results of the BQSquare sequence (i.e., highest ratio of inter-coded blocks and lowest quality gain in the error-free case). +Overall, comparing the results for the error-free and data loss cases, it is noticeable that similar quality gains are obtained,mostly due to the superior coding efficiency of larger \acs{cu}. +In most cases the quality gains slightly decrease for increasing \acs{plr}. Therefore, one may conclude that the error robustness of \ac{hevc} streams is not affected by limiting the maximum \ac{cu} size. %\todo{Complete discussion of partition unit size} @@ -114,32 +126,35 @@ Therefore, one may conclude that the error robustness of \ac{hevc} streams is no % %that larger coding units should be maintained. Even though larger frame partitions lead to larger coding dependencies, it will not affect the robustness to packet loss and error propagation. This is mainly due to the fact that error propagation occurs mostly in complex and moving regions, which normally are coded using smaller partition units. -\begin{table}[!t] +\begin{table}[tbp] \centering \caption{Average \ac{psnr} for different maximum CU size.} \label{Table3:Quality_BlockSize} \TABLESVSPACE \input{./031_ErrorResilienceEvaluation/QualityEvaluation/TableCUSizes.tex} +% \vspace{6mm} \end{table} +\begin{figure}[tbp] + \centering + \renewcommand{\FIGSIZE}{0.38\textwidth} + % \resizebox{\columnwidth}{!}{ + \begin{tabular}{y{.45\columnwidth}y{.45\columnwidth}} + \small + \includegraphics[width=\FIGSIZE]{\HEVCRobustnessEvaluation/Bask_Partitions_16.png} & + \includegraphics[width=\FIGSIZE]{\HEVCRobustnessEvaluation/Bask_Partitions_64.png} \\ + (a) $16\times16$ & (b) $64\times64$ \\ + \end{tabular} + % } + \caption{Block partition and prediction mode for a region of Basketball Drill.} + \label{Figure3:BlockPartitionBask} +\end{figure} + Figure~\ref{Figure3:BlockPartitionBask} illustrates an example of the frame partitions for a region of the Basketball Drill sequence for two cases: (i) maximum \ac{cu} size of $64\times64$ and (i) maximum \ac{cu} size of $16\times16$. -As shown, in the moving regions, e.g., moving player, the partitions obtained for both cases have a similar pattern, mainly composed of \acs{cu} with small size. Larger \acs{cu} are only used in the floor, which has a lower spatial complexity and suffers small changes between frames, therefore, easier to compress with $64\times64$ \acs{cu}. Moreover, those regions can be recovered more accurately using neighbouring information correctly decoded (spatial and temporal), thus not incurring in severe error propagation. -Therefore, the overall error robustness is not affected by reducing the maximum \ac{cu} size. One may conclude that the use of larger coding units is able to improve the overall \ac{hevc} efficiency in error-free and error prone applications. +As shown, in the moving regions, e.g., moving player, the partitions obtained for both cases have a similar pattern, mainly composed of \acs{cu} with small size. Larger \acs{cu} are only used in the floor, which has a lower spatial complexity and suffers small changes between frames, therefore, easier to compress with $64\times64$ \acs{cu}. +Moreover, those regions can be more accurately recovered, by using neighbouring information correctly decoded (spatial and temporal), thus not incurring in severe error propagation. +Therefore, the overall error robustness is not affected by reducing the maximum \ac{cu} size. One may conclude that the use of larger coding units is able to improve the overall \ac{hevc} efficiency in error-free and error prone communications. -\begin{figure}[!t] - \centering - \renewcommand{\FIGSIZE}{0.4\textwidth} - % \resizebox{\columnwidth}{!}{ - \begin{tabular}{y{.45\columnwidth}y{.45\columnwidth}} - \small - \includegraphics[width=\FIGSIZE]{\HEVCRobustnessEvaluation/Bask_Partitions_16.png} & - \includegraphics[width=\FIGSIZE]{\HEVCRobustnessEvaluation/Bask_Partitions_64.png} \\ - (a) $16\times16$ & (b) $64\times64$ \\ - \end{tabular} - % } - \caption{Block partition and prediction mode for a region of the Basketball Drill sequence.} - \label{Figure3:BlockPartitionBask} -\end{figure} % \subsection{Impact of filtering operation} % @@ -161,7 +176,7 @@ Therefore, the overall error robustness is not affected by reducing the maximum In this sub-section the relation between the temporal distance of consecutive intra-coded frames, i.e., period of the intra frames, and the video quality under transmission errors is evaluated. As the intra-coded frames have no dependencies to prior coded pictures, they introduce a refreshing point into the bitstream, which stops temporal error propagation along several frames or slices. -This allows the decoder to recover from frame/slice loss and continue error-free decoding seamlessly. +This allows the decoder to recover from frame/slice loss and continue error-free decoding in a seamless manner. \begin{figure}[!t] \centering @@ -171,46 +186,48 @@ This allows the decoder to recover from frame/slice loss and continue error-free \label{Figure3:QualityvsIntra} \end{figure} -Figure~\ref{Figure3:QualityvsIntra} shows the average decoded video quality (\ac{psnr}) for different distance of intra frames, covering \acs{plr} between 3\% and 5\%. +Figure~\ref{Figure3:QualityvsIntra} shows the average decoded video quality (\ac{psnr}) for different distances of intra frames, covering \acs{plr} between 3\% and 5\%. Results in the figure indicate that as the distance between intra frames increases, the decoded video quality significantly decreases. The quality decreasing is especially noticeable for intra periods higher than 32 frames, indicating that temporal distances greater than approximately one second are not suitable for video streaming under error-prone conditions. -When a high temporal distance between intra frames is used, the quality decreasing is more noticeable in the People On Street and BQSquare sequences, which have higher spatial complexity (see values for \ac{si}), leading to a reduction of 13 dB and 14 dB, respectively. As spatial complexity increases, the encoder relies more on temporal predictions, which increase the error propagation across temporal adjacent frames, reducing the video quality under errors. +When a high temporal distance between intra frames is used, the quality decreasing is more noticeable in the People On Street and BQSquare sequences, which have higher spatial complexity (see values for \ac{si}), leading to a reduction of 13 dB and 14 dB, respectively. +As spatial complexity increases, the prediction residue energy also increases, which leads to more erroneous information in case of data loss, reducing the video quality under errors. -%\todo{Complete discussion of intra refreshing} +One should also note from the results of Figure~\ref{Figure3:QualityvsIntra} that, using a small distance between intra frames is not always more efficient. +Since intra frames require higher amount of bits, the overall coding efficiency is reduced, resulting in lower overall quality in case of data loss, when comparing with other cases. +This is clearly noticeable for the Four People sequence, where replacing the intra period from 8 to 16 frames leads to a quality increase of 1 dB for 5\% of \ac{plr}. +For the remaining sequences it can be concluded that using sixteen frames (not higher than half a second) between refresh points (i.e., intra frames) yelds an acceptable efficiency under errors. This provides a good trade-off between compression efficiency and error robustness in case of data loss. -One should also note from the results of Figure~\ref{Figure3:QualityvsIntra} that using a small distance between intra frames is not always more efficient. -Since intra frames require higher amount of bits, the overall coding efficiency is reduced, resulting in an overall quality that decreases even in data loss conditions. -This is clear noticeable for the Four People sequence, where increasing the intra period from 8 to 16 leads to a quality increasing of 1 dB for 5\% of \ac{plr}. -For the remaining sequences it can be concluded that using sixteen frames (not higher than half a second) between refresh points (i.e., intra frames) achieves an acceptable efficiency under errors. This provides a good trade-off between compression efficiency and error robustness in case of data loss. +\subsection{Evaluation for different slice partitioning schemes} -\subsection{Evaluation for using different slice partitioning schemes} +The partitioning of the frame into different slices increase the flexibility of video stream packetisation, along packets of different sizes, thus providing adequate adaptation of coded data units to different transport layers and networks. -The partitioning of the frame into different slices increase the flexibility of video streams to be divided into different packets, thus adapting them to the video transmission system. -By partitioning the video frame into different slices spatial refresh points are introduced, limiting the error propagation across different \acs{ctu}. -In this sub-section the amount of bytes per slice is evaluated, in order to study the influence of different schemes on the error robustness of \ac{hevc} streams. -A comparison of the quality obtained for different modes is carried out, in order to find the most efficient mode for video transmission under data loss conditions. +By partitioning the video frame into different slices, spatial refresh points are introduced, limiting the error propagation across different \acs{ctu}. +In this sub-section the amount of bytes per slice is evaluated, in order to study the influence of different slice partitioning schemes on the error robustness of \ac{hevc} streams. +A comparison of the quality obtained for different modes is also carried out, in order to find the most efficient mode for video transmission under data loss conditions. \begin{figure}[!t] \centering \input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsSlices.tex} \caption{Decoded video quality for different slice partitioning modes.} - %\input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsSlices.tex} - %\caption{Decoded video quality for different slice partitioning schemes (Complementary results in Figure~\ref{Appendix:Figure3:SlicePartitioning}).} \label{Figure3:SlicePartitioning} \end{figure} Figure~\ref{Figure3:SlicePartitioning} shows the average \ac{psnr} for four difference slice partitioning modes and different \acs{plr}. The first three modes are characterised by a different amount of bytes per slice (B/S), namely 1200, 2400 and 4800 bytes. In these cases each frame has a variable number of slices. The last mode uses a single slice per frame (referred to as \textit{SS}), corresponding to the case where slice partitioning is disabled. -Results in Figure~\ref{Figure3:SlicePartitioning} reveals a clear impact of the slice partitioning mode on the error robustness of the \ac{hevc}, as the quality degradation for increasing \ac{plr} significantly changes for each mode. -Comparing the cases with fixed amount of bytes per slice, it is clearly noticeable that when higher amount of bytes are used the quality degradation increases. This is true for the results of most of the test sequences shown in Figure~\ref{Figure3:SlicePartitioning}. -Although the use of smaller slices increases the overhead used for slice headers, the superior error robustness achieved with 1200 B/S is able to overcome the inherent degradation on coding efficiency. -This is due to the lack of refresh points within a frame, resulting in errors that affects larger regions of the frame, which are propagated to the subsequent inter predicted frames. +The results shown in Figure~\ref{Figure3:SlicePartitioning} reveal a clear impact of the slice partitioning mode on the error robustness of \ac{hevc}, as the quality degradation for increasing \ac{plr} significantly changes for each mode. +Comparing the cases with fixed amount of bytes per slice, it is clearly noticeable that a higher amount of bytes also corresponds to a higher quality degradation. +This is true for the results of most test sequences shown in Figure~\ref{Figure3:SlicePartitioning}. +Although the use of smaller slices increases the overhead required for slice headers, the superior error robustness achieved with 1200 B/S is able to overcome the inherent degradation on coding efficiency. +This is due to the lack of refresh points within a frame, resulting in errors that affect larger regions of the frame, which are propagated to the subsequent inter predicted frames. -The results for of \textit{SS} (slice partitioning disabled) also reveals that this approach also leads to lower error robustness when comparing with those cases where the frames are divided into multiple slices. Although higher coding efficiency is achieved in this mode, the lower error robustness leads to an overall poor performance under frame or slice loss conditions. -Results for Kendo sequence does not follow the main trend. This can be explained by the low bitrate required to compress the video signal, which is the lowest when compared with the other test sequences (see Table~\ref{TableTestSequences}). Therefore, most of the inter-coded frames can be transmitted using a single slice, even with 1200 bytes per slice, resulting in a similar partitioning for all tested modes, and thus similar error robustness. In this case, the higher coding efficiency achieved by the \textit{SS} case consequently leads to higher quality in case of errors. +The results for of \textit{SS} (slice partitioning disabled) reveals that this approach also leads to lower error robustness when comparing with those cases where the frames are divided into multiple slices. +Although higher coding efficiency is achieved in this mode, the lower error robustness leads to an overall poor performance under frame or slice loss conditions. +It is worthwhile to notice that the results for Kendo sequence do not follow the main trend. +This can be explained by the low bitrate required to compress the video signal, which is the lowest when compared with the other test sequences (see Table~\ref{TableTestSequences}). +Therefore, most of the inter-coded frames can be transmitted using a single slice, even with 1200 bytes per slice, resulting in a similar partitioning for all tested modes, and thus similar error robustness. +In this case, the higher coding efficiency achieved by the \textit{SS} case also leads to higher quality in case of errors. -%\todo{Complete discussion of slice partitioning} \subsection{Motion vector coding impact on error propagation} @@ -230,11 +247,11 @@ Therefore, in this sub-section the impact of using spatial and temporal \ac{mv} The first set of experiments aimed to investigate the impact of the amount of spatial \ac{mv} candidates in the video quality. As mentioned before, \ac{hevc} allows up to five candidates to be used, which can be provided solely by spatial neighbours. In order to check the impact of the \ac{mv} candidates, different coding configurations were used, each one characterised by different amounts of spatial \ac{mv} candidates, i.e., 1, 3 and 5 \acs{mv}. -Figure~\ref{Figure3:AvgPSNRMergeCandidates} shows the average quality results for the Basketball Drill and People On Street sequences under different packet loss conditions. +Figure~\ref{Figure3:AvgPSNRMergeCandidates} shows the average quality results for the Basketball Drill and People On Street sequences, under different packet loss conditions. The results show that, besides the number of \ac{mv} candidates, similar quality decreasing is achieved as the \ac{plr} increases. Therefore, one may conclude that varying the number of available \ac{mv} candidate does not affect the error resilience of the \ac{hevc}. -The occurrence rate for each spatial \ac{mv} candidate was analysed in order to check whether they are uniformly used. -Table~\ref{Table3:MergeIdxRatio} shows the usage percentage of each \ac{mv} candidate when three and five candidates are available for the slice partitioning mode of 1200 bytes/slice. The results indicate that most predictions use the first candidate and minor differences occur by adding 2 or more \acs{mv}. +The occurrence rate for each spatial \ac{mv} candidate was also analysed in order to check whether they are uniformly used. +Table~\ref{Table3:MergeIdxRatio} shows the usage percentage of each \ac{mv} candidate when three and five candidates are available for the slice partitioning mode of 1200 B/S. The results indicate that most predictions use the first candidate and minor differences occur by adding 2 or more \acs{mv}. Since the mostly used \ac{mv} candidates are the initial ones, changing the number of available candidates does not significantly affects the error resilience. Moreover, the slice partitioning mode used only 1200 bytes per slice, which introduces refresh points in each frame, reducing the spatial error propagation due to mismatched \ac{mv} predictions. @@ -263,11 +280,10 @@ Moreover, the slice partitioning mode used only 1200 bytes per slice, which intr The second set of experiments aimed to find the influence of losing temporal motion predictors on the error resilience of \ac{hevc}. In order to perform this test, a single frame loss in enforced, and the error propagation is evaluated for two configurations: (i) with the temporal \ac{mv} candidate enabled and (ii) disabled. -Figure~\ref{Figure3:PSNRvsFrameErrorFreeConc} illustrates the error propagation, when a single frame is lost. For this particular experiment the effect of intra-refreshing period of 32 frames was used to evaluate a long error propagation period. +Figure~\ref{Figure3:PSNRvsFrameErrorFreeConc} illustrates the error propagation, when a single frame is lost. For this particular experiment an intra-refreshing period of 32 frames was used to evaluate the impact on long error propagation periods. The results show that when the \ac{tmvp} mode is enabled, the loss of Frame \#6 leads to significant reduction in the reconstruction quality of the subsequent frames. -% This is true even for sequences with low motion activity as per Table~\ref{tab:sequences}. -On the contrary, when the temporal \ac{mv} candidate is disabled, the motion information on subsequent frames is not affected by the loss of a single frame and significantly higher quality is achieved. This is the effect of breaking the temporal dependencies between \acs{mv}, which prevents erroneous \acs{mv} to propagate throughout the set of dependent frames. -%This is more noticeable in the \ac{ld} configuration because more error propagation occurs in P-frames and several frames are affected until an intra frame is decoded (refresh point). +On the contrary, when the temporal \ac{mv} candidate is disabled, the motion information on subsequent frames is not affected by the loss of a single frame and significantly higher quality is obtained. +This is the effect of breaking the temporal dependencies between \acs{mv}, which prevents erroneous \acs{mv} to propagate throughout the set of dependent frames. \begin{figure}[!t] \centering @@ -281,14 +297,13 @@ On the contrary, when the temporal \ac{mv} candidate is disabled, the motion inf In order to further evaluate the impact of the temporal \ac{mv} dependencies on the error propagation, the \ac{hevc} streams were subject to random packet losses. Table~\ref{Table3:AvgPSNRWithoutTMVP} presents the average quality results, measured using the \ac{psnr}. -As expected, the results indicate that in error-free conditions higher quality is achieved by enabling the \ac{tmvp} candidate, since higher compression efficiency is obtained because more \ac{mv} predictors are used. -This is true for all both test conditions, \emph{i.e.}, both \ac{ld} and \ac{ra}. +As expected, the results indicate that under error-free conditions higher quality is achieved by enabling the \ac{tmvp} candidate, since higher compression efficiency is obtained because more \ac{mv} predictors are used. +This is true for both test conditions, \emph{i.e.}, \ac{ld} and \ac{ra}. However, in the presence of errors, the video quality significantly decreases due to low error robustness when temporal \ac{mv} candidates are used. Alternatively, average quality up to 5.17 dB of \ac{psnr} is achieved when the temporal \ac{mv} predictors are disabled in the \ac{ld} configuration. The results also show that the negative impact of using temporal \ac{mv} dependencies is smaller when B-frames are used. For the Random-Access, the average gain of disabling the \ac{tmvp} is 0.78 dB. -%This also confirms the previous results of Figure~\ref{Figure3:PSNRvsFrameErrorFreeConc}. A higher negative impact is obtained when only P-frames are used (\ac{ld} configuration), since subsequent frames are temporally closer, leading to more accurate temporal \ac{mv} candidates. -Moreover, the results also indicate that for higher loss ratios the negative impact of using the \ac{tmvp} does not increase, as both configuration are subject to high quality degradation. +Moreover, these results also indicate that for higher \ac{plr} the negative impact of using the \ac{tmvp} does not increase, as both configurations are subject to high quality degradation. \begin{table}[!t] \centering @@ -298,17 +313,17 @@ Moreover, the results also indicate that for higher loss ratios the negative imp \input{./031_ErrorResilienceEvaluation/QualityEvaluation/TableMVPs.tex} \end{table} -\subsection{Remarks} +\subsection*{Remarks on error robustness of HEVC} -Overall, the results presented in the previous sub-sections show that \ac{hevc} has poor robustness against transmission errors, resulting in significant quality degradation, though depending on the video content and coding options. -As shown, some coding options, such as slice partition and intra refreshing period, have high correlation with the decoded video quality when transmission errors affect the compressed streams. -Therefore, these are important aspects to take into consideration in the design an implementation of broadcasting video services and applications. +Overall, the results presented in the previous sub-sections show that \ac{hevc} has poor robustness against transmission errors, resulting in significant quality degradation, though depending on the video content and coding configurations. +As shown, some coding configurations, such as slice partition and intra refreshing period, have high correlation with the decoded video quality when transmission errors affect the compressed streams. +Therefore, these are important aspects to take into consideration in the design and implementation of video delivery services and applications over error-prone networks. One of the main improvements of the \ac{hevc} standard is the support for larger coding units up to $64\times64$, which does not have a negative impact on the error robustness of coded video streams, indicating that they should be allowed to achieve higher coding performance. Regarding the use of neighbouring information for \ac{mv} coding, on the one hand the spatial \ac{mv} candidates do not significantly decrease the error robustness. On the other hand, the use of temporal \ac{mv} candidates significantly contribute to increase the temporal dependencies, leading to more severe quality degradation in the presence of packet loss. Based on these evidences, the next section presents an efficient method to improve the error robustness of \ac{hevc}. -The proposed method is based on the underlying principle of reducing the mismatch of \ac{mv} predictions, by selectively disabling some \ac{mv} predictions at the encoder-side, reducing the overall temporal dependencies. +The proposed method is based on the underlying principle of reducing the mismatch of \ac{mv} predictions, by selectively disabling some \ac{mv} predictions at the encoder-side in order to reduce the overall temporal dependencies. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -326,13 +341,14 @@ Based on this evidence, this section proposes a method to make intelligent use o \glsreset{rd} -The new approach to deal with temporal motion vector dependencies is accomplished by modifying the \ac{rd} optimisation that is used to select the best motion vector for inter-prediction. -The objective is to increase the number of \acs{mv} that can be independently decoded, in order to improve the error recovery capability after a frame loss event. +The new approach to deal with temporal \ac{mv} dependencies is accomplished by modifying the \ac{rd} optimisation that is used to select the best \ac{mv} for inter-prediction. +The objective is to increase the number of \acs{mv} that can be independently decoded, in order to improve the error recovery capability after an arbitrary frame loss event. -The proposed method also overcomes the limitation of the method presented in \cite{DynamicTMVP2011}, which break the motion information dependencies between frames. -Instead of disabling the temporal motion candidate at the frame level, the proposed method aims to remove the dependencies at the \ac{cu} level. -To achieve this goal, the temporal dependency of all \ac{mv} candidates is firstly analysed for each block, in order to select on the suitable candidates to be used as predictors for the current \ac{mv}. -Based on the analysis of the temporal dependency, some temporal candidates are marked as unsuitable predictors of the current \ac{mv}, and for this reason they are not selected in for \ac{mv} prediction. In the proposed approach, a temporal \ac{mv} candidate is marked as unsuitable, based on the following criteria: +The proposed method also overcomes the limitation of the method presented in \cite{DynamicTMVP2011}, which breaks the motion information dependencies between frames. +Instead of disabling the temporal motion candidates at the frame level, the proposed method aims to remove the dependencies at the \ac{cu} level. +To achieve this goal, the temporal dependency of all \ac{mv} candidates is firstly analysed for each block, in order to select the suitable candidates to be used as predictors for the current \ac{mv}. +Based on the analysis of the temporal dependency, some temporal candidates are marked as unsuitable predictors of the current \ac{mv}, and for this reason they are not selected for \ac{mv} prediction. +In the proposed approach, a temporal \ac{mv} candidate is marked as unsuitable, based on the following criteria: \begin{enumerate} \item if it is encoded based on another temporal \ac{mv} candidate from a previously encoded frame; \item if it is predicted using a spatial neighbour that was previously encoded using a temporal \ac{mv} candidate. @@ -348,8 +364,10 @@ Based on the analysis of the temporal dependency, some temporal candidates are m \end{figure} Figure~\ref{ProposedDynamicMethod} can be used to explain the proposed method, illustrating different blocks and different \ac{mv} dependencies represented with arrows. Note that the squares illustrate a generic block size; the representation of the temporal dependencies is simplified, and in some cases may not correspond to the exact co-located block in the previous frame. In frame $f_{t-1}$ there are several \acs{mv} encoded using spatial predictions and one \ac{mv} encoded using a temporal prediction (arrow pointing to $f_{t-2}$). In frame $f_{t}$, the \ac{mv} corresponding to the block (1) may use the temporal \ac{mv} candidate, since the co-located block in $f_{t-1}$ is not temporally dependent. -However, in the proposed scheme the \ac{mv} prediction marked with the arrow (2) is not allowed, thus preventing the propagation of temporal dependencies. -Moreover, the block corresponding to the arrow (3) cannot use the temporal \ac{mv} candidate since it already depends on a \ac{mv} that was previously encoded using the temporal candidate. The proposed method reduces the temporal dependencies of the \glspl{mv} at the block level, providing improved error robustness to the encoded streams. Using this approach, the dependencies are selectively removed, improving the error recovery, without fully disabling the temporal \gls{mv} candidates for a given frame. Therefore, coding efficiency is not significantly penalised. +However, in the proposed scheme the \ac{mv} prediction identified with the arrow (2) is not allowed, which prevents the propagation of temporal dependencies. +Moreover, the block corresponding to the arrow (3) cannot use the temporal \ac{mv} candidate since it already depends on a \ac{mv} that was previously encoded using the temporal candidate. The proposed method reduces the temporal dependencies of the \glspl{mv} at the block level, providing improved error robustness to the encoded streams. +Using this approach, the dependencies are selectively removed, improving the error recovery accuracy, without fully disabling the temporal \gls{mv} candidates for a given frame. +Therefore, coding efficiency is not significantly penalised. \subsection{Performance evaluation} @@ -371,8 +389,9 @@ The performance of the proposed scheme (\PROP) was evaluated against two other a Firstly, the performance of the proposed method was measured under a single loss event (i.e., only one frame is affected by errors). Figure~\ref{Figure3:ProposedMethodEvaluationSingle} shows the error propagation for the different methods. The results show a significant quality difference between the proposed method, the \TIDR method and the reference \ac{hevc}. -When the temporal \ac{mv} candidate is disabled (proposed and \TIDR method) a higher error robustness is achieved. -The results also show the relevance of using the proposed method, as the \TIDR method is only able to recover from the frame loss after the refreshing frame, \emph{i.e.}, frame \#8, especially for the BQSquare sequence. +When the temporal \ac{mv} candidate is disabled (proposed and \TIDR method) higher error robustness is attained. +The results also show the benefits of using the proposed method over the \TIDR method. +While the \TIDR method is only able to recover from the frame loss after the refreshing frame, \emph{i.e.}, frame \#8, especially for the BQSquare sequence, the proposed method is able to recover right after the frame loss. \subsubsection{Quality evaluation under random errors} @@ -387,9 +406,10 @@ The results also show the relevance of using the proposed method, as the \TIDR m Secondly, further simulations were performed to evaluate the effectiveness of the proposed error resiliency method under realistic packet loss patterns. %As in previous experiments, each coded frame is transmitted in one packet. Table~\ref{Table3:AverageStdPSNR} illustrates the average standard deviation of the PSNR, for three of the test sequences. -The results show that increasing the \ac{plr} leads to higher quality deviations and that the impact of errors in the subjective quality experience is higher. -However, reducing the temporal dependencies of the \acs{mv} using the proposed method decreases the quality differences variation, achieving the lowest value for all tested scenarios. The proposed method is able to reduce the quality difference variations up to 0.94 dB for Kendo sequence (3$\%$ of packet loss) compared to the~\TIDR~method. -Since the proposed method achieves the lowest \ac{psnr} standard deviation, one may conclude that it is able to reduce the quality variations leading to a better visual quality. +The results show that increasing the \ac{plr} leads to higher quality variations, thus the impact of errors in the subjective quality experience is higher. +However, reducing the temporal dependencies of the \acs{mv} using the proposed method decreases such quality variations, achieving the lowest value for all tested scenarios. +The proposed method is able to reduce the quality variations up to 0.94 dB for Kendo sequence (3$\%$ of packet loss) compared to the~\TIDR~method. +Therefore, since the proposed method achieves the lowest \ac{psnr} standard deviation, one may conclude that it is able to reduce the video quality variations leading to a better visual quality. %\HIGHLIGHT{vantagem do std} @@ -399,11 +419,12 @@ Since the proposed method achieves the lowest \ac{psnr} standard deviation, one \input{031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNR.tex} \end{table} -Table~\ref{Table3:AverageResultsProposed} shows the Bjontegaard's average \ac{psnr} increase \cite{BJResults2001}. The results are obtained taking as reference the standard \ac{hevc} encoder. +Table~\ref{Table3:AverageResultsProposed} shows the Bjontegaard's average \ac{psnr} increase \cite{BJResults2001}. +The results are obtained taking the standard \ac{hevc} encoder as reference. % The results are obtained taking as reference the standard \ac{hevc} encoder, in order to evaluate the quality difference when temporal \ac{mv} candidates are disabled in both error free and under error-prone conditions. -The results presented in Table~\ref{Table3:AverageResultsProposed} show that the proposed method presents practically the same rate-distortion performance when compared to \TIDR method. +The results presented in Table~\ref{Table3:AverageResultsProposed} show that the proposed method presents practically the same rate-distortion performance when compared to the \TIDR method. In general, the average quality, in an error-free scenario, only decreases 0.028 dB in comparison to the \TIDR method and 0.034 dB in comparison to the reference encoder. -However, the proposed method widely outperforms the reference \TIDR technique~\cite{DynamicTMVP2011} in terms of error resiliency performance, achieving in average higher overall quality under error-prone conditions. +However, the proposed method widely outperforms the reference \TIDR technique in terms of error resiliency performance, achieving in average higher overall quality under error-prone conditions. The effectiveness of the proposed method increases at higher packet loss rates (\emph{e.g.},~$5\%$), achieving an average gain of 0.64 dB ($1.34 - 0.70$), and a maximum gain of 1 dB ($1.44-0.43$) for the Book Arrival sequence , when compared to the \TIDR technique. Higher gains are achieved when compared with the reference \ac{hevc} encoder, as shown in Table~\ref{Table3:AverageResultsProposed}. @@ -416,7 +437,7 @@ The effectiveness of the proposed method increases at higher packet loss rates ( In this chapter, the error resilience performance of \ac{hevc} was studied in detail under error prone conditions. This study was partially discussed and published in the paper~(\ref{Conf:ErrorResilience}). The simulation results revealed that new coding tools introduced in \ac{hevc} to improve coding efficiency are partially responsible for higher error propagation. -This motivates further research on new techniques to be developed for highly efficient encoders and decoders, in order to increase error resiliency and to improve the overall performance of video broadcast. +This motivated further research on new techniques for highly efficient encoders and decoders, in order to increase error resiliency and to improve the overall performance of video delivery services and applications. Then, a new approach to deal with the weak robustness of \ac{mv} coding was proposed, addressing \ac{mv} dependencies at \ac{cu} level. The proposed method selectively disables the temporal \ac{mv} candidates, in order to break the dependencies across more than one frame. This method was proposed in the publication (\ref{Conf:DynamicTMVP}). The experimental evaluation was able to demonstrate that the proposed method increases the error robustness and outperforms existing state-of-the-art methods and the reference codec. diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP.tex b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP.tex index 9effa66..0022b82 100644 --- a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP.tex +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP.tex @@ -8,7 +8,7 @@ set xtics 0,2,0+31 set key width 1 #set title '\TitleFont Basketball Drill' -plot '-' using 1:2 title '\KeyFont TMVP Enable' w lp ls 1, '-' using 1:2 title '\KeyFont TMVP Disabled' w lp ls 2 +plot '-' using 1:2 title '\KeyFont TMVP Enabled' w lp ls 1, '-' using 1:2 title '\KeyFont TMVP Disabled' w lp ls 2 0 42.08 44.10 44.86 1 42.08 44.10 44.86 2 40.24 43.19 44.01 diff --git a/031_ErrorResilienceEvaluation/TablesSequences.tex b/031_ErrorResilienceEvaluation/TablesSequences.tex index 0a0455d..dc7e36f 100644 --- a/031_ErrorResilienceEvaluation/TablesSequences.tex +++ b/031_ErrorResilienceEvaluation/TablesSequences.tex @@ -29,7 +29,7 @@ \begin{tabular}{m{3.1cm}|x{1.7cm}|x{.9cm}|x{1.3cm}|m{6.6cm}} \toprule \centering\let\newline\\\arraybackslash\hspace{0pt}\textbf{Sequence} & \textbf{SI / TI} & \textbf{Inter (\%)} & \textbf{Bitrate (Mbps)} & \centering\let\newline\\\arraybackslash\hspace{0pt}\textbf{Description} \\ \hline -{Basketball Drill \newline $832\times480@50Hz$} & 33.4 / 14.4 & 88.1 & 4.50 & High motion with several basket ball players captured from a high point \\ \hline +{Basketball Drill \newline $832\times480@50Hz$} & 33.4 / 14.4 & 88.1 & 4.50 & High motion with several basketball players captured from a high point \\ \hline {Basketball Drive \newline $1920\times1080@50Hz$} & 33.0 / 15.2 & 85.0 & 15.00 & Several basketball players passing a ball captured from a low point \\ \hline {Book Arrival \newline $1024\times768@30Hz$} & 28.4 / 21.7 & 90.7 & 1.50 & Moderate translational motion with two moving persons \\ \hline {Bosphorus \newline $3840\times2160@120Hz$} & 13.4 / 3.80 & 91.6 & 12.00 & Boat shipping at low motion with moderate complex background \\ \hline diff --git a/041_ImprovedErrorResilience/ImprovedErrorResilience.tex b/041_ImprovedErrorResilience/ImprovedErrorResilience.tex index b1fdd14..f58a7d7 100644 --- a/041_ImprovedErrorResilience/ImprovedErrorResilience.tex +++ b/041_ImprovedErrorResilience/ImprovedErrorResilience.tex @@ -3,7 +3,7 @@ \graphicspath{{\ImprovedErrorResilience/Results/}} -\chapter{A Two-stage Approach for Robust Coding and Streaming \VERSION{2}} +\chapter{A two-stage approach for robust coding and streaming \VERSION{2}} % \chapter{Error Resilience Techniques for \ac{hevc} Standard} %\chapter{Two-stage Robust Video Coding} \label{Chapter:ErrorResilienceTechniques} @@ -13,19 +13,15 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Introduction} -\glsreset{pu} -\glsreset{ctu} -\glsreset{mv} \glsreset{rd} +\glsreset{pu} +\glsreset{mv} -\glsreset{ra} -\glsreset{ld} +This chapter proposes a new two-stage approach to improve the error robustness of \ac{hevc} streaming, by reducing temporal error propagation in case of frame loss. +This is accomplished by reducing the prediction mismatch that occurs at the decoder after frame loss. +The proposed approach comprises the following two stages: (i) at the encoding stage, the reference pictures are dynamically selected based on constraining conditions and Lagrangian optimisation, which distributes the use of reference pictures throughout each frame, reducing the number of \ac{pu} that depend on a single reference; (ii) at the streaming stage, a \ac{mv} prioritisation algorithm, based on spatial dependencies, selects an optimal sub-set of \acs{mv} to be transmitted as side redundant information to reduce mismatched \ac{mv} predictions at the decoder. -%The increased compression ratios achieved by the \ac{hevc} standard lead to reduced robustness of coded streams, with increased susceptibility to network errors and consequent video quality degradation, as shown in Chapter~\ref{Chapter:HEVCErrorResilience}. -This chapter proposes a coding method, based on a two-stage approach, to improve the error robustness of \ac{hevc} streaming, by reducing temporal error propagation in case of frame loss. -This is accomplished by reducing the prediction mismatch that occurs at the decoder after frame loss, through the following two stages: (i) at the encoding stage, the reference pictures are dynamically selected based on constraining conditions and Lagrangian optimisation, which distributes the use of reference pictures throughout each frame, by reducing the number of \acs{pu} that depend on a single reference; (ii) at the streaming stage, a \ac{mv} prioritisation algorithm, based on spatial dependencies, selects an optimal sub-set of \acs{mv} to be transmitted, redundantly, as side information to reduce mismatched \ac{mv} predictions at the decoder. - -This chapter is organized as follows. Section~\ref{Section4:Problem} presents the problem definition and motivation for the method proposed in this chapter. Section~\ref{Section4:Architecture} provides an overview of the proposed method, describing the functional blocks that compose the two-stage approach for robust coding and streaming. Section~\ref{Section4:RefPicSel} describes the encoding stage of the proposed method, that is used to select the optimal reference picture for inter-prediction. Section~\ref{Section4:MVSelection} describes the streaming stage where a sub-set of \acs{mv} are selected to reduce the mismatch motion predictions. Section~\ref{Section4:Performance} provides a performance evaluation of the proposed method, illustrating the advantages of each stage and how they are combined to achieve superior error robustness in \ac{hevc} standard. Finally, Section~\ref{Section4:Summary} concludes the chapter. +This chapter is organized as follows. Section~\ref{Section4:Problem} presents the problem definition and motivation for the method proposed in the chapter. Section~\ref{Section4:Architecture} provides an overview of the proposed method, describing the functional blocks that compose the two-stage approach for robust coding and streaming. Section~\ref{Section4:RefPicSel} describes the encoding stage of the proposed method, that is used to select the optimal reference picture for inter-prediction. Section~\ref{Section4:MVSelection} describes the streaming stage where a sub-set of \acs{mv} are selected to reduce the mismatch motion predictions. Section~\ref{Section4:Performance} provides a performance evaluation of the proposed method, illustrating the advantages of each stage and how they are combined to achieve superior error robustness in \ac{hevc} standard. Finally, Section~\ref{Section4:Summary} concludes the chapter. \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -34,9 +30,18 @@ This chapter is organized as follows. Section~\ref{Section4:Problem} presents th \section{Problem definition} \label{Section4:Problem} +\glsreset{rd} +\glsreset{ec} +\glsreset{pu} +\glsreset{ctu} +\glsreset{cu} +\glsreset{mv} -As previously mentioned,\ac{hevc} bitstreams subject to transmission losses result in significant degradation of both objective and subjective quality. -This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes, which are selected according to a \ac{rd} optimisation criteria that assumes error-free transmission~\cite{ROPE2007}. +\glsreset{ra} +\glsreset{ld} + +As previously mentioned, when\ac{hevc} bitstreams are subjected to transmission losses, the decoded video signal incurs in significant degradation of both objective and subjective quality. +This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes, which are selected according to \ac{rd} optimisation criteria, which always assume error-free transmission~\cite{ROPE2007}. Therefore, high compression efficiency is achieved, but resilience to transmission errors is penalised. \begin{figure}[tb] @@ -46,21 +51,22 @@ Therefore, high compression efficiency is achieved, but resilience to transmissi \label{FIG4:ReferenceUsageRatio} \end{figure} -Highly efficient video encoders support the use of multiple reference frames for inter-frame prediction, which are selected based on a \ac{rd} cost that may lead to an unbalance distribution. -In order to evaluate the reference frame selection method attained by the \ac{rd} implemented in the reference \ac{hevc} encoder, different test sequences are encoded using two reference frames, referred to as $r_1$ and $r_2$, and its selection ratios are analysed. In this study, reference $r_1$ corresponds to the closest neighbouring frame and reference $r_2$ the second closest. +Highly efficient video encoders support the use of multiple reference frames for inter-frame prediction, which are selected based on minimisation of \ac{rd} cost that may lead to an unbalanced use of various reference frames. +In order to analyse the distribution of the reference frames selected by the \ac{rd} decision method used in the reference \ac{hevc} encoder, different test sequences were encoded using two reference frames, referred to as $r_1$ and $r_2$, and their selection ratios were analysed. +In this study, reference $r_1$ corresponds to the closest neighbouring frame and reference $r_2$ the second closest. Figure~\ref{FIG4:ReferenceUsageRatio} illustrates the average usage percentage of reference $r_1$ for different sequences. The remaining corresponds to those cases where reference $r_2$ is selected. These results show that, in general, reference $r_1$ is selected more often than $r_2$. This is mainly true for sequences with higher motion. For those sequences with lower motion, but high spatial complexity, e.g., BQSquare (see \ac{si} in Table~\ref{TableTestSequences}), both reference frames are equally (approximately) selected for temporal predictions, with $r_1$ being used by 58\% of cases and $r_2$ by 42\%. -This reveals that even with a configuration using only two references the encoder mainly uses the temporally closer one as the reference for inter-frame predictions. +This reveals that, even though two references are used, the encoder mainly uses the temporally closer one as the reference for inter-frame predictions. Therefore, if such reference frame is lost, a higher number of mismatched predictions will occur, inevitabely leading to higher error propagation. -Another technique that contributes to increased error propagation is the use of predictions in \ac{mv} coding~\cite{MotionCodingHEVC2011}, which contributes to increase the spatial and temporal dependencies. -Although the spatial \ac{mv} candidates are used more often, temporally predicted \acs{mv} are responsible for a greater impact on error propagation, as they propagate errors in the \ac{mv} decoding across different time instances. +Another coding technique that contributes to increase error propagation is the use of predictions in \ac{mv} coding~\cite{MotionCodingHEVC2011}, which also contributes to increase the total amount of spatial and temporal dependencies. +As shown in Chapter~\ref{Chapter:HEVCErrorResilience}, temporally predicted \acs{mv} are responsible for a greater impact on error propagation, as they propagate errors in the \ac{mv} decoding across different time instances. -To overcome these issues, in the next sections a two-fold approach is described to increase the error robustness in video streaming by jointly considering two different stages: (i) while encoding live video, by selecting robust coding modes and (ii) after encoding, by adding robust features to the compressed streams. While in the former this is accomplished by optimising coding parameters and decisions to increase robustness, in the latter some type of compressed domain processing is used to increase error robustness of pre-encoded streams, e.g., error resilient transcoding. Inevitably, in both cases this is achieved at the expense of some loss in coding efficiency, therefore the challenge is to find an optimum trade-off between robustness and bitrate overhead. -%In this chapter a two-fold approach is used to overcome the two problems mentioned above. +To overcome these issues a two-fold approach was devised to increase the error robustness in video streaming by jointly considering two different stages: (i) while encoding live video, by selecting robust coding modes and (ii) after encoding, by adding robust features to the compressed streams. While in the former this is accomplished by optimising coding parameters and decisions to increase robustness, in the latter some type of compressed domain processing is used to increase error robustness of pre-encoded streams, e.g., error resilient transcoding. Inevitably, in both cases this is achieved at the expense of some loss in coding efficiency, therefore the challenge is to find an optimum trade-off between robustness and bitrate overhead. +The proposed method is described in the remaining of this chapter. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% @@ -68,8 +74,8 @@ To overcome these issues, in the next sections a two-fold approach is described \section{Proposed two-stage architecture} \label{Section4:Architecture} -The goal of the proposed method is to reduce the error propagation in decoded video in case of frame loss. -The underlying idea is to constrain inter-frame prediction dependencies, in order to minimise the distortion $D_t$ given by +The goal of the proposed method is to reduce the error propagation in decoded video when the received stream was incurred in frame loss. +The underlying idea is to constrain inter-frame prediction dependencies, in order to minimise the frame distortion $D_t$ given by \begin{equation} D_t = \tilde f_t - \hat f_t, \text{when } \tilde f_{t-1} \neq \hat f_{t-1}, \label{eq1} @@ -91,23 +97,23 @@ Note that the streaming stage is an independent and complementary subsystem of t \end{figure} In case of frame loss, these two stages have the common objective of helping the decoder to recover from mismatched predictions at the expense of a small increase in bitrate. -This is equivalent to reduce the error propagation by using two techniques: (i) reducing the mismatch of inter-frame predictions, by encoding each \ac{pu} from a different reference frame and (ii) reducing the mismatch of \ac{mv} predictions by sending the most important \acs{mv} as side information. These are the two stages of the proposed method described in the following sections. +This is equivalent to reduce the error propagation by means of two different mechanisms: (i) reducing the mismatch of inter-frame predictions, by encoding each \ac{pu} from a different reference frame and (ii) reducing the mismatch of \ac{mv} predictions by sending the most important \acs{mv} as side information. These are the two stages of the proposed method described in the following sections. -\section{Encoding stage: Reference frame selection} +\section{Encoding stage: reference frame selection} \label{Section4:RefPicSel} The reference picture selection mechanism attempts to uniformly distribute the use of different reference pictures across the \ac{ctu} of a coded video frame. -In case of frame loss, the error propagation in the decoder is reduced because the number of \acs{ctu} predicted from the lost frame decreases and thus error concealment benefits as more data is correctly decoded from other reference frames. +In case of frame loss, the error propagation in the decoder is reduced because the number of \acs{ctu} predicted from the lost frame decreases and thus \ac{ec} benefits as more data is correctly decoded from other reference frames. The underlying principle of this method is to minimise the probability of using the same reference picture for neighbouring \acs{ctu}. -%For instance, + In a coding structure with two reference frames, as shown in Figure~\ref{fig:checkerboardRefSelection}, the reference frames $r_{1}$ and $r_{2}$ are used to predict the same number of \acs{ctu} in current frame~($C$), resulting in a checkerboard structure, where a different reference frame is used for predicting every other \ac{ctu} within the current frame. If each reference frame is used to predict half of the pixels in the current frame, when any of these references is lost, the subsequent predicted frame is only partially distorted by mismatched predictions. This is because only half of its predictions depend on the missing reference frame, which limits the overall error propagation. -Moreover, since the motion information of the predicted frame is encoded using as reference not only the missing frame but also the past ones, those \acs{mv} that cross over a missing reference frame, pointing to the previous ones, are still useful for reconstructing correct \acs{mv} and also for error concealment. +Moreover, since the motion information of the predicted frame is encoded using, not only the missing frame as reference but also the past ones, those \acs{mv} that cross over a missing reference frame, pointing to the previous ones, are still useful for reconstructing accurate \acs{mv} and also for \ac{ec}. In the limit, a straightforward implementation of the previous example leads to a simple interleaved uniform distribution of the reference frames over all \acs{ctu} in the current frame, forcing the checkerboard structure shown in Figure~\ref{fig:checkerboardRefSelection}. In the case of two reference frames, 50\% of the \acs{ctu} would use one reference frame, while the other 50\% would use the other reference. However, such solution would not take into consideration coding efficiency, since the selection process of the reference frame would not be \ac{rd} optimised for each \ac{pu} within a \ac{ctu}. @@ -215,7 +221,8 @@ Figure~\ref{fig:mvdependencies} illustrates how \acs{mv} have different importan In this figure, each block corresponds to a \ac{pu}, which has an associated \ac{mv}. The arrows point to the \ac{pu} containing the \ac{mv} used as predictor. In the current frame ($f_0$), two \acs{pu} have temporally dependent \acs{mv}. While the \ac{mv} of \ac{pu} \MVDEPA is used to predict a total of four other \acs{mv}, the \ac{mv} of \ac{pu} \MVDEPB is only used to predict one single \ac{mv}. Thus, in case of data loss (e.g., frame $f_{-1}$), the \ac{mv} of \ac{pu} \MVDEPA has more impact on error propagation to the frame $f_{0}$ and consequently quality degradation than the \ac{mv} of \ac{pu} \MVDEPB, because of its higher number of dependent \acs{mv}. -For this reason, in the \ac{mv} selection process, the proposed method assigns higher importance to the \ac{mv} of \ac{pu} \MVDEPA than \ac{pu} \MVDEPB. Moreover, in Figure~\ref{fig:mvdependencies}, it can be observed that although the number of \acs{mv} predicted from the \acs{mv} of \ac{pu} \MVDEPC is higher than \ac{pu} \MVDEPA (6 and 4 \acs{mv}, respectively), +For this reason, in the \ac{mv} selection process, the proposed method assigns higher importance to the \ac{mv} of \ac{pu} \MVDEPA than \ac{pu} \MVDEPB. +Moreover, in Figure~\ref{fig:mvdependencies} it can be observed that, although the number of \acs{mv} predicted from the \acs{mv} of \ac{pu} \MVDEPC is higher than \ac{pu} \MVDEPA (6 and 4 \acs{mv}, respectively), the image area that depends on the former is smaller that the latter. This is due to the variable size of \acs{pu}. Therefore, the image area covered by all dependent \acs{pu} is separately considered in the \ac{mv} selection process. @@ -231,12 +238,12 @@ Based on the evidence mentioned above, the best \acs{mv} to be encoded as side i \item the \acs{mv} with temporal dependencies are firstly selected as the most important set: $V$ = \{\emph{\ac{mv}: \ac{mv} is temporally dependent from others}\}. \item -the $MVs \in V$ are ranked according to the number of spatial dependencies associated to each one. +the sub-set of $MVs \in V$ are ranked according to the number of spatial dependencies associated to each one. \item the optimal sub-set $U \subset V$ is found as the best trade-off between the image area covered by the dependent \acs{pu} and bitrate overhead, i.e., the one that maximises the cost function in Equation (\ref{eq:MVSelectOpt}). \end{enumerate} -Considering $V$ the ordered set of \acs{mv} encoded for the current frame, i.e., $V=\{MV_1, MV_2, \cdots MV_s\}$ and $U \subset V$ always starting in the first vector, i.e., \mbox{$U_n=\{MV_1, MV_2, \cdots MV_n\}$} with $n\leq s$, the best set of \acs{mv} ($U_n^*)$ is selected based on the optimisation procedure given by Equation (\ref{eq:MVSelectOpt}). Note that any value of $n$ corresponds to a unique subset $U$. +This can be formulated as follows. Considering $V$ the ordered set of \acs{mv} encoded for the current frame, i.e., ${V=\{MV_1, MV_2, \cdots MV_s\}}$ and ${U \subset V}$ always starting in the first vector, i.e., ${U_n=\{MV_1, MV_2, \cdots MV_n\}}$ with $n\leq s$, the best set of \acs{mv} ($U_n^*)$ is selected based on the optimisation procedure given by Equation (\ref{eq:MVSelectOpt}). Note that any value of $n$ corresponds to a unique subset $U$. \begin{equation}\label{eq:MVSelectOpt} \newcommand{\argmax}{\operatornamewithlimits{arg\ max}} @@ -261,13 +268,13 @@ The subset of selected \acs{mv} is independently encoded, ensuring that no refer \label{Section4:Performance} -In this section the robustness of the proposed two-stage architecture is evaluated and discussed for two cases: (i) only using the encoding stage and (ii) using the streaming stage in addition to the encoding stage. In the former case, identified as ``\PROP'', only the dynamic reference picture selection method described in Section~\ref{Section4:RefPicSel} is evaluated. In the latter case, identified as ``\DYN'', besides the previous method of reference picture selection, redundant \acs{mv} are encoded as described in Section~\ref{Section4:MVSelection}, resulting in a combination of both stages. +In this section the performance of the proposed two-stage architecture is evaluated and discussed for two cases: (i) only using the encoding stage and (ii) using the streaming stage in addition to the encoding stage. In the former case, identified as ``\PROP'', only the dynamic reference picture selection method described in Section~\ref{Section4:RefPicSel} is evaluated. In the latter case, identified as ``\DYN'', besides the previous method of reference picture selection, redundant \acs{mv} are encoded as described in Section~\ref{Section4:MVSelection}, resulting in a combination of both stages. As references for comparison, three other methods are used: (i) a fixed checkerboard approach (\CHKB), (ii) the reference picture generation method based on leaky predictions proposed in \cite{ROPERefGenJan2010}, identified as ``\AVG'', and (iii) long-term reference frames (\LONG). The \LONG method only uses the key frames as reference, i.e., in the \ac{ra} configuration only one out of eight frames is used as reference for prediction, allowing a more robust transmission, as the reference frames are less likely to be hit by errors. The experimental setup used in this section is the same as the one previously described in Section~\ref{Section3:ErrorResilienceEvaluation}. Two test cases were considered for simulation: (i) in \emph{\LOSTALL} all frames may be equally affected by packet loss, including \ac{idr} frames; (ii) in \emph{\KEEPINTRA} \ac{idr} frames are assumed to be prioritised in the network and delivered error-free, or using some combined approach as recently proposed in \cite{ErrorResilienceHeterogenousNetworks2016}. -At the decoder-side, error concealment is based on motion compensation from previously decoded frames. The motion field of a missing frame is obtained applying \ac{mv} extrapolation from the two closest neighbours. Afterwards, the final \acs{mv} are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}. +At the decoder-side, \ac{ec} is based on motion compensation from previously decoded frames. The motion field of a missing frame is obtained applying \ac{mv} extrapolation from the two closest neighbours. Afterwards, the final \acs{mv} are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}. The evaluation of the encoding-stage is carried out separately from the streaming-stage. @@ -286,7 +293,7 @@ The evaluation of the encoding-stage is carried out separately from the streamin The performance of the encoding stage is evaluated on its own, in order to only determine the improvements achieved by the dynamic reference frame selection method. -In this case, the reference frames were reconstructed with distortion, as a result of non-perfect error concealment. +In this case, the reference frames were reconstructed with distortion, as a result of non-perfect recovery. Since the encoding stage does not deal with errors in temporal \ac{mv} predictions (this is handled by the streaming-stage), in this evaluation the temporal \ac{mv} predictor is disabled. This is necessary to avoid masking the results with the errors caused by wrong \ac{mv} predictions. @@ -318,8 +325,11 @@ This value was chosen to allow increasing the Lagrangian cost by about $50\%$, The error robustness achieved by the encoding-stage was firstly evaluated in regards to the error propagation, resulting from a single loss event in the \ac{ld} configuration. Figure~\ref{fig:RecoveredExample} shows the visual impact of error propagation on a region of the decoded Frame \#10 when Frame \#9 is affected by errors, for Basketball Drill and Race Horses sequences (the \ac{psnr} corresponds to the entire frame). The proposed method is compared with the method~in~\cite{ROPERefGenJan2010}, as both aim at reducing the error propagation. -The method~in~\cite{ROPERefGenJan2010} uses filtered frames as reference for predictions, therefore, in case of errors, the effects of mismatch predictions are significantly reduced by the filtering operation (see Sub-Section~\ref{SECTION2:LeakyPred}). -As shown in this figure, the proposed method is able to reduce the impact of mismatch decoding in comparison with the method~in~\cite{ROPERefGenJan2010}. Although both methods still reveal some predictions mismatch leading to inaccurate reconstructed of the objects contours, the proposed method is able to achieve a smoother reconstruction, which leads to higher quality in terms of \ac{psnr}. This is because, in frame \#10, the predictions depending on the missing frame \#9 are not used throughout the entire frame, thus, approximately half of the predictions are not affected by errors. In contrast, the method~in~\cite{ROPERefGenJan2010} requires several filtering operations to be applied (i.e, different reference frames need to be filtered) in order to reduce the effect of mismatch predictions. +The method~in~\cite{ROPERefGenJan2010} uses filtered frames as reference for predictions, therefore, in case of errors, the effects of mismatched predictions are significantly reduced by the filtering operation (see Sub-Section~\ref{SECTION2:LeakyPred}). +As shown in Figure~\ref{fig:RecoveredExample}, the proposed method is able to reduce the impact of mismatch decoding in comparison with the method~in~\cite{ROPERefGenJan2010}. +Although both methods still reveal some predictions mismatch leading to inaccurate reconstructed of the objects contours, the proposed method is able to achieve a smoother reconstruction, which leads to higher quality in terms of \ac{psnr}. +This is because, in frame \#10, these predictions that depend on the missing frame \#9 are not used throughout the entire frame, thus, approximately half of the predictions are not affected by errors. +In contrast, the method~in~\cite{ROPERefGenJan2010} requires several filtering operations to be applied (i.e, different reference frames need to be filtered) in order to reduce the effect of mismatched predictions. Summarising, the reconstructed frames illustrated in Figure~\ref{fig:RecoveredExample} reveal the higher efficiency achieved by distributing the use of the references frames. @@ -335,7 +345,7 @@ Summarising, the reconstructed frames illustrated in Figure~\ref{fig:RecoveredEx Figure~\ref{fig:PSNRvsFrameForRefPicSel}~shows the \ac{psnr} over a \ac{gop} with an \ac{idr} period of 32 frames, using the same constant bitrate for all streams. This figure shows that all methods are able to outperform the reference \ac{hevc} (\REF), gaining approximately $3$~dB in reconstruction quality. -The similar quality levels achieved by \CHKB and \PROP reveal that the dynamic selection of the reference frames used in \PROP is more robust because a better trade-off between coding efficiency and error resilience is attained. +The similar quality levels achieved by \CHKB and \PROP reveal that the dynamic selection of the reference frames used in \PROP is more robust, because a better trade-off between coding efficiency and error resilience is attained. Moreover, results in Figure~\ref{fig:PSNRvsFrameForRefPicSel} are in line with the reconstruction quality shown in Figure~\ref{fig:RecoveredExample}, confirming that the proposed method is able to outperform the reference frame generation technique in~\cite{ROPERefGenJan2010}. %Since the proposed method is more efficient than the others for error-free transmission (see~Table~\ref{tab:bd_error_free}), and it also provides better error robustness, it can be considered as an efficient solution to better deal with transmission errors. Overall, the method proposed for the encoding stage is able to increase the error robustness of \ac{hevc} without severely compromising the coding efficiency. @@ -352,7 +362,8 @@ In \cite{ROPERefGenJan2010}, the use of interpolated references in the encoding The proposed method is also able to outperform the \ac{hevc} reference software when an hierarchical coding structure is used (see results for \ac{ra} configuration in Table~\ref{tab:AverageQualityCHKBEXPRandomAccess}), which implicitly limits the error propagation. In comparison to the reference \ac{hevc}, quality improvements up to $1.42$~dB at PLR=$10\%$ are obtained for Kendo sequence.%, while the average gain across all test sequences is 1~dB. -When comparing results for both test cases, it is noticeable that the proposed method is able to outperform the reference methods for both cases, when \ac{idr} frame loss is allowed or not (\LOSTALL and \KEEPINTRA, respectively). As the \ac{plr} increases, the gains of the proposed method decrease in comparison with the case where the \ac{idr} frames are delivered error-free (i.e., case \KEEPINTRA), due to the higher error propagation of the missing \ac{idr} frame. Moreover, the quality gains are lower in the \LOSTALL case when comparing with \KEEPINTRA for the same \ac{plr}, for both the \ac{ld} and \ac{ra}. +When comparing results for both test cases, it is noticeable that the proposed method is able to outperform the reference methods for both cases, regardless whether \ac{idr} frame loss is allowed or not (\LOSTALL and \KEEPINTRA, respectively). +As the \ac{plr} increases, the gains of the proposed method decrease in comparison with the case where the \ac{idr} frames are delivered error-free (i.e., case \KEEPINTRA), due to the higher error propagation of the missing \ac{idr} frame. Moreover, the quality gains are lower in the \LOSTALL case when comparing with \KEEPINTRA for the same \ac{plr}, for both the \ac{ld} and \ac{ra}. Taking into account the above results and the spatio-temporal information shown in Table~\ref{TableTestSequences}, one can observe that the performance of the proposed method is lower for sequences with high spatial details and low temporal activity (e.g., BQSquare and Park Scene), which indicates higher effectiveness for video content with higher motion activity. %\HIGHLIGHT{complete with analysis of both cases} @@ -411,9 +422,11 @@ Table~\ref{tab:bd_error_free} shows the Bjontegaard's Delta PSNR (BD-PSNR) valu \end{table} The results in Table~\ref{tab:bd_error_free} indicate a small loss of coding efficiency of the proposed method, as well as, the reference methods tested. -The fixed checkerboard approach leads to an average quality reduction of $1.22$~dB for the \ac{ld} ($0.32$~dB for \ac{ra}), while the proposed method only loses $0.82$~dB and $0.24$~dB in the \ac{ld} and \ac{ra} configurations, respectively. This results clearly shows that dynamically selecting the reference frames is more efficient that using a fixed pattern, confirming the previous results for error-prone conditions. -Moreover, in comparison to \LONG and the method in~\cite{ROPERefGenJan2010}, the proposed method is still more efficient, consistently achieving better results. Comparing, the results for both error-free and error-prone conditions, one may conclude that the proposed method not only achieves higher quality gains, but also does reduces the impact on coding efficiency. -Overall, the loss in the quality compared to the reference \ac{hevc} is considered acceptable, given the increase in robustness obtained in lossy transmission demonstrated in the previous sub-sections. +The fixed checkerboard approach leads to an average quality reduction of $1.22$~dB for the \ac{ld} ($0.32$~dB for \ac{ra}), while the proposed method only loses $0.82$~dB and $0.24$~dB in the \ac{ld} and \ac{ra} configurations, respectively. +This results clearly show that dynamically selecting the reference frames is more efficient than using a fixed pattern, confirming the previous results for error-prone conditions. +Moreover, in comparison to \LONG and the method in~\cite{ROPERefGenJan2010}, the proposed method is still more efficient, because it consistently achieves better results. +Comparing, the results for both error-free and error-prone conditions, one may conclude that the proposed method not only achieves higher quality gains, but also reduces the impact on coding efficiency. +Overall, the loss in the quality compared to the reference \ac{hevc} is considered acceptable, given the increase in robustness obtained in lossy transmission. %As expected, this is due to the sub-optimal \ac{rd} optimisation of the proposed method, because the cost function also includes the cost associated to the choice of the reference frame, in addition to the rate and distortion. @@ -473,12 +486,12 @@ Results also show that for both approaches the decoded video quality increases a \includegraphics[width=\CurrFigSize,page=4]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf} \includegraphics[width=\CurrFigSize,page=5]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf} \includegraphics[width=\CurrFigSize,page=6]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf} - \caption{Decoded video quality for different amounts of redundancy, selected by using a fixed approach and the proposed method (\DYN).} + \caption{Decoded video quality for different amounts of redundancy, selected by using a fixed approach and the proposed dynamic method (\DYN).} \label{fig:QualityVsRedundancyMVSelection} \end{figure} It is noticeable in Figure~\ref{fig:QualityVsRedundancyMVSelection} that for higher \acs{plr}, the proposed method is able to achieve higher gains. -This reveals that when data loss increases it is more important optimise the amount of \acs{mv} selected per frame, in order to use more \acs{mv} in those frames that have a higher impact on the error propagation. +This reveals that when data loss increases it is more important to optimise the amount of \acs{mv} selected per frame, in order to use more \acs{mv} in those frames that have a higher impact on the error propagation. For instance, the proposed method is able to gain up to $1.5$~dB for the BQSquare sequence subject to a PLR=10\%. For lower \acs{plr}, since fewer frames are affected by data losses, the use of a fixed approach achieves similar performance as the proposed method (\DYN). @@ -489,9 +502,9 @@ Table~\ref{tab:AverageQualityRedundancies} presents the average \ac{psnr} of dec This study was also conducted for the two different cases, with and without \ac{idr} loss. It can be confirmed from the results of Table~\ref{tab:AverageQualityRedundancies} that, for each sequence, as $\alpha$ decreases more redundant \acs{mv} are used, leading to higher overhead rations. This results in higher objective video quality for all tested sequences. -This confirms that redundant \acs{mv} are able to minimise the error propagation, which is accomplished at a cost of an acceptable increase in the bitrate, since only a sub-set of \acs{mv} is transmitted. +These results also confirm that redundant \acs{mv} are able to minimise the error propagation, which is accomplished at the cost of an acceptable increase in the bitrate, since only a sub-set of \acs{mv} is transmitted. The use of redundant \acs{mv} yields better video quality for \ac{plr} as low as $1\%$, achieving higher quality gains when \ac{idr} frames are not lost (\KEEPINTRA). This is because of the higher quality degradation when \ac{idr} frames are lost, which is not due to mismatched \ac{mv} predictions, and thus cannot be reduced by transmitting extra \acs{mv} as side-information. -The use the proposed method also achieves higher quality for sequences with higher motion activity that makes use of inter-prediction more often, i.e., BQSquare and Traffic sequences (see Inter ratio in Table~\ref{TableTestSequences}), leading to a maximum gain of $3.93$~dB for \LOSTALL ($7.82$~dB for \KEEPINTRA). +The use the proposed method also achieves higher quality for sequences with higher motion activity that make use of inter-prediction more often, i.e., BQSquare and Traffic sequences (see Inter ratio in Table~\ref{TableTestSequences}), leading to a maximum gain of $3.93$~dB for \LOSTALL ($7.82$~dB for \KEEPINTRA). In summary, the proposed method is able to select the most relevant motion \ac{mv} information and when combined with a reference picture selection scheme the problem of mismatch \ac{mv} predictions is mitigated. Moreover, the dynamic approach devised to select the best \acs{mv} also contributes to increase the overall performance. @@ -511,7 +524,7 @@ Thus, using the proposed two-stage approach to recover temporal predictions cons In this chapter a two-stage approach was proposed to increase the error robustness of \ac{hevc} streaming and reduce the error propagation in case of packet loss. A constrained coding approach was devised to select reference frames and dynamically distribute temporal dependencies. -This is jointly used with a controlled amount of side information in coded streams, comprising a small set of the most relevant \acs{mv}, in order to minimise \ac{mv} mismatch at the decoder in the presence of frame loss. The use of \acs{mv} as side information was firstly proposed and evaluated in the publication~\ref{Conf:MotionVectorRedundancies} and the reference frame selection was further discussed in the publication~\ref{Conf:Chkb}. Subsequently, a wider analysis was carried out and published in \ref{Journal:TwoStage}. +This is jointly used with a controlled amount of side information in coded streams, comprising a small set of the most relevant \acs{mv}, in order to minimise \ac{mv} mismatch at the decoder in the presence of frame loss. The use of \acs{mv} as side information was firstly proposed and evaluated in the publication~(\ref{Conf:MotionVectorRedundancies}) and the reference frame selection was further discussed in the publication~(\ref{Conf:Chkb}). Subsequently, a wider analysis was carried out and published in (\ref{Journal:TwoStage}). As can be concluded from the results, this method contributes to reduce the temporal dependencies with consistent quality gains for different \ac{plr} and coding configurations, only incurring in a small drop of coding efficiency. Overall, the proposed approach is an effective method to cope with video transmission over error-prone networks. diff --git a/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex b/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex index 55026e1..e78e544 100644 --- a/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex +++ b/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex @@ -4,7 +4,7 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % CHAPTER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\chapter{Error Concealment-Aware Video Encoding \VERSION{2}} +\chapter[Error concealment-aware video encoding]{Error concealment-aware video \\ encoding \VERSION{2}} \label{Chapter:ConcealmentAwareTechniques} %\vspace{-6mm} %\clearpage @@ -14,17 +14,22 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Introduction} +\glsreset{rd} +\glsreset{mv} \glsreset{ec} +\glsreset{cu} +\glsreset{ld} +\glsreset{ra} \glsreset{ra} \glsreset{ld} -%\glsreset{rd} + In this chapter an error concealment-aware encoding scheme is proposed to improve the quality of decoded video in broadcast environments prone to transmission errors and data loss. The method aims to enhance the reconstruction of recovered lost frames by optimising the efficiency of the \ac{ec}. The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled through \ac{sei} messages. Such optimal \ac{ec} modes are found by simulating transmission losses followed by a Lagrangian optimisation of the \textit{signalling rate - \ac{ec} distortion} cost. A generalised saliency-weighted distortion is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. When data loss occurs the decoder uses the signalling information to improve the reconstruction quality. This chapter is organized as follows. Section~\ref{Section5:Problem} introduces the problem definition and motivation for the method proposed in this chapter. -Section~\ref{Section5:ECCandidates} presents an initial evaluation of different error concealment schemes in order to find the relevant ones to be used in the proposed error concealment-aware coding scheme. +Section~\ref{Section5:ECCandidates} presents an initial evaluation of different \ac{ec} schemes in order to find the relevant ones to be used in the proposed \ac{ec}-aware coding scheme. Section~\ref{Section5:Architecture} provides an overview of the proposed method architecture followed by a detailed description in Section~\ref{Section5:ProposedMethod}. Sections~\ref{Section5:ECSignallingPerformance}~and~\ref{Section5:OverallPerformance} present the performance results and discussion of the proposed \ac{ec} optimisation and signalling. Finally, Section~\ref{Section5:Summary} concludes the chapter. @@ -41,9 +46,10 @@ Finally, Section~\ref{Section5:Summary} concludes the chapter. %\glsunset{mb} In Chapter~\ref{Chapter:ErrorResilienceReview} different \ac{ec} algorithms that can effectively reconstruct lost video were described, though not all of them exhibit the same accuracy for every lost frame or slice. -Moreover, it was also shown that combining different \ac{ec} approaches at the \ac{mb}/\ac{pu} level to improve the reconstruction accuracy is not straightforward. -In the past, hybrid \ac{ec} schemes were proposed to improve the recovery quality of lost frames by either combining spatial and temporal \ac{ec} algorithms~\cite{HybridConcTB2008} or by optimising the \ac{mve} at the pixel and block levels~\cite{HybridConcealmentTB2008,2DConcHybridMotion2010}. -Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of motion-based recovery approaches by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}. Although such methods are able to reduce the reconstruction artefacts and improve the overall video quality, decoder-based decisions are still not fully accurate, as they only have limited information to be used as reference in the decision process. +Moreover, it was also shown that combining different \ac{ec} approaches at the block level to improve the reconstruction accuracy is not straightforward. +In the past, hybrid \ac{ec} schemes were proposed to improve the quality of recovered frames by either combining spatial and temporal \ac{ec} algorithms~\cite{HybridConcTB2008} or by optimising the \ac{mve} at the pixel and block levels~\cite{HybridConcealmentTB2008,2DConcHybridMotion2010}. +Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of motion-based recovery approaches by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}. +Although such methods are able to reduce the reconstruction artefacts and improve the overall video quality, decoder-based decisions are still not fully accurate, as they are based on limited information to be used as reference in the decision process. Therefore, higher reconstructed quality can be achieved by estimating the best \ac{ec} techniques at the encoding-side, based on the original video signal. %Such approach not only improves quality but also reduces the complexity of the video decoders. %In this chapter a robust coding driven by \ac{ec} methods is developed with the objective of enhancing the quality of reconstructed video at decoders receiving erroneous streams from the transmission network. The proposed approach combines error resilience tools at the encoder-side with \ac{ec} at the decoder-side. @@ -52,9 +58,10 @@ Extra information was also used to aid the \ac{ec} operation at the decoder-side Although these methods facilitate the \ac{ec} performance either through slice reordering or based on \acs{mv}, they always rely on the assumption that some fixed \ac{ec} method is used at the decoder. % The work presented in \cite{ConcealmentSignalling2008,SignallingRobustScalable2015} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder. Although such approaches achieves reasonably good performance, it relies on the spatial neighbouring information or multi-layer video, which may not be available in all cases. -To advance one step further, this chapter presents an investigation on concealment-aware video coding and proposes a new scheme to achieve high decoding flexibility in terms of the \ac{ec} approaches, leading to better video quality. -This is accomplished by selecting, at the encoder, among several \ac{ec} algorithms with different characteristics, the one that yields the lowest reconstruction distortion at the decoder and also minimises the \ac{rd} cost considering the necessary signalling rate. Thus, the proposed approach differs from similar previous methods \cite{ConcealmentAwareME2006,ErrorResilienceModeHEVC2016} by selecting the best error concealment method without affecting the encoder optimisation. -To further improve the quality of recovered frames, an enhancement layer is encoded with the mismatch residue between the transmitted and recovered frames. This is a dynamic optimisation process, which is more efficient than conventional approaches based on redundant pictures \cite{RedudantPicsJan2009,VPxResiliencePCS2016}, which transmit the whole frame as redundancy. +To advance one step further, this chapter presents an investigation on \ac{ec}-aware video coding and proposes a new scheme to achieve high decoding flexibility in terms of the \ac{ec} approaches, leading to better video quality. +This is accomplished by selecting, at the encoder, among several \ac{ec} algorithms with different characteristics, the one that yields the lowest reconstruction distortion at the decoder and also minimises the \ac{rd} cost considering the necessary signalling rate. Thus, the proposed approach differs from similar previous methods \cite{ConcealmentAwareME2006,ErrorResilienceModeHEVC2016} by selecting the best \ac{ec} method without affecting the encoder optimisation. +To further improve the quality of recovered frames, an enhancement layer is encoded with the mismatch residue computed as a difference between the transmitted and recovered frames. +This is a dynamic optimisation process, which is more efficient than conventional approaches based on redundant pictures \cite{RedudantPicsJan2009,VPxResiliencePCS2016}, which transmit the whole frame as redundancy. In contrast with the previous work presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}, which addressed the error resilience problem by optimising the coding dependencies, the approach followed in this chapter considers both, \ac{ec} performance and the coded rate required for signalling the best \ac{ec} mode to the decoder. @@ -65,7 +72,7 @@ In contrast with the previous work presented in Chapter~\ref{Chapter:ErrorResili \label{Section5:ECCandidates} -In order to propose a novel error concealment-aware technique different algorithms were taken into consideration. +In order to propose a novel \ac{ec}-aware technique different algorithms were taken into consideration. This section presents a description and evaluation of the most relevant \ac{ec} algorithms used in the proposed method. Firstly, a technical description of each method is provided followed by a performance comparison between them. @@ -78,7 +85,7 @@ This has been widely used in the past for \ac{h264} standard \cite{H264RefConc20 Let's consider the example of Figure~\ref{FIG5:MotionVectorExtrapolation}, where a missing frame ($f_0$) at time instant $t_0$ is reconstructed based on the information present in the temporally adjacent frames, $f_{-1}$ and $f_{-2}$. One of the common approaches is \ac{mc} where the missing frame is reconstructed through motion compensation directly using the co-located vectors in the closest neighbour frame ($f_{-1}$). The \acs{mv} used for the motion compensation ($v_0$) are obtained by copying the \acs{mv} $v_{-1}$ from the same spatial position in $f_{-1}$, as illustrated in the figure by the \ac{mv} $v^{1}_{0}$ which is obtained by copying $v^{1}_{-1}$. -An alternative approach is based on the reconstruction of the missing frame by using motion compensation through a extrapolation of the received \acs{mv}, of frame $f_{-1}$ at time instant $t_{-1}$, according to the following procedure: +An alternative approach is based on the reconstruction of the missing frame by using motion compensation through extrapolation of the received \acs{mv}, of frame $f_{-1}$ at time instant $t_{-1}$, according to the following procedure: \begin{enumerate} \item the \acs{mv} ($v_{-1}$) of the neighbouring frame ($f_{-1}$) are extrapolated to compute the set of \acs{mv} ($v_{0}$) for the missing frame as follows: @@ -94,8 +101,9 @@ x^{'} &= x - v_{0x}, \\ y^{'} &= y - v_{0y}. \end{align} \end{enumerate} -This method is illustrated in Figure~\ref{FIG5:MotionVectorExtrapolation} by the \ac{mv} $v^{2}_{0}$, which results from the extrapolation of the \ac{mv} $v^{2}_{-1}$, and is referred to as \glsfirst{mve}. -One should note that these methods may not result in a complete motion field for the entire frame, resulting in an incomplete reconstruction. For those regions the average of the three top-left neighbouring \acs{mv} are used to complete the motion field. +This method, referred to as \glsfirst{mve}, is illustrated in Figure~\ref{FIG5:MotionVectorExtrapolation}, where the \ac{mv} $v^{2}_{0}$ results from the extrapolation of $v^{2}_{-1}$. +One should note that these methods may not result in a complete motion field for the entire frame, resulting in an incomplete reconstruction. +For the regions without \acs{mv} the average of the three top-left neighbouring \acs{mv} are used to complete the motion field. %\todo{Grid-aligment! Incomplete!} \begin{figure}[tb] @@ -105,7 +113,8 @@ One should note that these methods may not result in a complete motion field for \label{FIG5:MotionVectorExtrapolation} \end{figure} -Finally, as a base for comparison, the default \ac{ec} in the reference \ac{hevc} implementation the \ac{fc} algorithm is also evaluated. This method consists in replacing the lost frame with a copy of the previous decoded one, which can also be described as a motion compensated reconstruction using only zero \acs{mv}. +Finally, as a base for comparison, the default \ac{ec} in the reference \ac{hevc} implementation the \ac{fc} algorithm is also evaluated. +This method consists in replacing the lost frame with a copy of the previously decoded one, which can also be described as a motion compensated reconstruction using only zero \acs{mv}. \subsection{Algorithms based on motion estimation} @@ -116,7 +125,8 @@ Typically, optical flow estimation algorithms are used to estimate a 2-D motion \begin{equation} E(x,y) = \sum_{x} \sum_{y} \bigg| f_{0}\big(x,y\big) - f_{1}\big( x + v_x(x,y), y+v_y(x,y) \big) \bigg|. \end{equation} -Besides a full search algorithm, where a block-based approach is used to search for the best \acs{mv} that minimises the sum of absolute errors, three other approaches were tested to evaluate their effectiveness in \ac{ec} algorithms. These three methods are the following: +Besides a full search algorithm, where a block-based approach is used to search for the best \acs{mv} that minimise the sum of absolute errors, three other approaches were tested to evaluate their effectiveness in \ac{ec} algorithms. +These three methods are the following: \begin{enumerate} \item a method based on the total variation using the minimization of the \mbox{L-1} norm~\cite{DualFlow2016}; \item a method which approximates the neighbourhood of each pixel by quadratic polynomials on both frames~\cite{PolyFlow2003}; @@ -125,13 +135,16 @@ Besides a full search algorithm, where a block-based approach is used to search \subsubsection{Motion estimation based on total variation and regularisation (\textit{DualFlow})} -In this method the motion field is estimated based on the minimization of the total variation obtained using the L-1 norm \cite{DualFlow2007,DualFlow2016}, taking into account the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}. This condition forces the estimated field to be regular across neighbouring pixels by penalising deviations in a quadratic sense. +In this method the motion field is estimated based on the minimization of the total variation obtained using the L-1 norm \cite{DualFlow2007,DualFlow2016}, taking into account the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}. +This condition forces the estimated field to be regular across neighbouring pixels by penalising deviations in a quadratic sense. Using this method, for each pixel $\mathbf{p}$, the motion field $v(\mathbf{p})$ is obtained by minimising the following energy function, \begin{equation}\label{EQ5:DualFlow} E(v) = \sum_{\mathbf{p} \in \Omega} \Big( \big| f_{-1}(\mathbf{p}) - f_{-2}( \mathbf{p} + v(\mathbf{p}))\big| + \big|\nabla v(\mathbf{p})\big| \Big)%d\mathbf{p},\ \mathbf{p} \in \Omega \end{equation} -where $f_{-1}$ and $f_{-2}$ are the two previous frames at time instants $t_{-1}$ and $t_{-2}$, as represented in Figure~\ref{FIG5:MotionVectorExtrapolation}. In Equation~\ref{EQ5:DualFlow}, $\Omega$ defines the sets of pixels of those frames, and $|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of the optical flow which effectively disallow discontinuities. This is based on the assumption that neighbouring regions belong to the same object, thus such regions should have similar motion. +where $f_{-1}$ and $f_{-2}$ are the two previous frames at time instants $t_{-1}$ and $t_{-2}$, as represented in Figure~\ref{FIG5:MotionVectorExtrapolation}. +In Equation~\ref{EQ5:DualFlow}, $\Omega$ defines the sets of pixels of those frames, and $|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of the optical flow which effectively disallow discontinuities. +This is based on the assumption that neighbouring regions belong to the same object, thus such regions should have similar motion. \subsubsection{Motion estimation based on polynomial expansion (\textit{PolyFlow})} @@ -143,7 +156,8 @@ f(\mathbf{p}) \sim \mathbf{p}\mathbf{A}_1\mathbf{p}^T+\mathbf{b}_1\mathbf{p}^T+C \end{equation} where $\mathbf{p} = [x\ y]$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ is a row vector and $C_1$ is a scalar. The coefficients are estimated using weighted least squares fit to the signal values in the neighbourhood. -Since the result of a polynomial expansion is such that each neighbourhood is approximated by a polynomial, it is analysed what happens if a polynomial undergoes an ideal translation. Let's consider the signal $f_1(\mathbf{p})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$, given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following: +Since the result of a polynomial expansion is such that each neighbourhood is approximated by a polynomial, it is necessary to analyse what happens if a polynomial undergoes an ideal translation. +Let's consider the signal $f_1(\mathbf{p})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$, given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following: \begin{align} f_2(\mathbf{p}) &= f_1(\mathbf{p}-\mathbf{d}) \nonumber \\ @@ -152,7 +166,7 @@ f_2(\mathbf{p}) &= \mathbf{p}\mathbf{A}_1\mathbf{p}^T + (\mathbf{b}_1-2\mathbf{d}\mathbf{A}_1)\mathbf{p}^T + \mathbf{d}\mathbf{A}_1\mathbf{d}^T - \mathbf{b}_1\mathbf{d}^T + C_1\\ &= \mathbf{p}A_2\mathbf{p}^T+\mathbf{b}_2\mathbf{p}^T+C_2, \nonumber \end{align} -which by equating the coefficients of the quadratic polynomials results in the following relations: +then, by equating the coefficients of the quadratic polynomials results in the following relations: \begin{align} \mathbf{A}_2 &= \mathbf{A}_1 \nonumber \\ \mathbf{b}_2 &= \mathbf{b}_1 - 2 \mathbf{d} \mathbf{A}_1, \\ @@ -165,25 +179,16 @@ This allows to find the displacement $\mathbf{d}$ for any signal dimensions by s %\todo{Completar o Deep flow} In this method the motion field is estimated based on extension of the total variation methods using descriptor (feature) matching. -This allows the optical flow estimation to accurately find large displacements as it is not bound by constrains, such as the Horn \& Such presented in Equation~(\ref{EQ5:DualFlow})\cite{LargeDisplacementOptFlow_TPAMI2011}. Moreover, the correspondences between descriptions are used to apply a coarse-to-fine wrapping, thus avoiding local minimum values. +This allows the optical flow estimation to accurately find large displacements, because it is not fully bound by any constrains, such as the Horn \& Such presented in Equation~(\ref{EQ5:DualFlow})\cite{LargeDisplacementOptFlow_TPAMI2011}. Moreover, the correspondences between descriptions are used to apply a coarse-to-fine wrapping, which avoids local minimum values. The combination of both approaches increases the matching accuracy. -This was also exploited in \cite{DeepFlow2013} with the following extensions: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layer optimisation. +This approach was also exploited in \cite{DeepFlow2013} with the following extensions: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layer optimisation. Figure~\ref{FIG5:DeepFlowDiagram} shows the blocks that are included in the description matching process. -The matching algorithm is built upon a multi-stage architecture with six layers with different patches sizes, interleaving convolutions and max-pooling, which is similar to deep convolutional networks \cite{GradientBasedConvNN_ProcIEEE1998}. -As shown in Figure~\ref{FIG5:DeepFlowDiagram}, a multi-size response levels are generated from the reference and target image using convolutions with different patches sizes. This method uses a bottom-up approach, i.e., starting at a fine level and moving towards the coarser levels (larger patches), which are built as an aggregation of responses of smaller patches. This results in a multi-size response pyramid. +The matching algorithm is built upon a multi-stage architecture using six layers with different patches sizes, interleaving convolutions and max-pooling, which is similar to deep convolutional networks \cite{GradientBasedConvNN_ProcIEEE1998}. +As shown in Figure~\ref{FIG5:DeepFlowDiagram}, multi-size response levels are generated from the reference and target images using convolutions with different patches sizes. This method uses a bottom-up approach, i.e., starting at a fine level and moving towards the coarser levels (larger patches), which are built as an aggregation of responses of smaller patches. This results in a multi-size response pyramid. Subsequently, a local maximum is extracted from each level to achieve a more accurate motion field (quasi-dense correspondence). -Finally, an initial motion field is extracted, referred to as $v'$ . - -\begin{figure}[tb] - \centering - \includegraphics[width=.95\columnwidth]{\ConcealmentAwareDir/Figures/DeepFlowBlockDiagram} - \caption{Block diagram of the deep matching algorithm \cite{DeepFlow2013}.} - \label{FIG5:DeepFlowDiagram} - \FIGSVSPACE -\end{figure} - -Subsequently, the final optical flow is obtained by minimising an energy function, which can be described by the following equation: +Then, an initial motion field is extracted, referred to as $v'$ . +The final optical flow is obtained by minimising an energy function, which can be described by the following equation: \begin{equation} E(v) = \sum_{\mathbf{p} \in \Omega} \Big( E_D(\mathbf{p}) + \alpha E_S(\mathbf{p}) + \beta E_M(\mathbf{p}) \Big), @@ -192,11 +197,19 @@ where $E_D$ is the data term, $E_S$ is a smoothing term and $E_M$ is the matchin The data term is obtained from the difference between matching pixels in both video frames. The smoothness term is a robust penalisation of the gradient of the flow, similarly to Equation (\ref{EQ5:DualFlow}), i.e., $E_S(\mathbf{p})=|\nabla v(\mathbf{p})|$. Finally, the matching term ($E_M$) is used to encourage the flow estimation to be similar to the precomputed motion field~($v'$). +\begin{figure}[t] + \centering + \includegraphics[width=.95\columnwidth]{\ConcealmentAwareDir/Figures/DeepFlowBlockDiagram} + \caption{Block diagram of the deep matching algorithm \cite{DeepFlow2013}.} + \label{FIG5:DeepFlowDiagram} + \FIGSVSPACE +\end{figure} \subsection{Experimental evaluation} -In this section a performance evaluation of the different \ac{ec} schemes previously described is carried out. This evaluation aims at finding the suitable methods to be used for efficiently reconstruct the missing frames in case of data loss. -The two \ac{ec} methods based on the correctly received \acs{mv} are tested, i.e., \ac{mc} and \ac{mve}, using the \ac{fc} method is used as reference for comparison. +This section presents a simulation study a performance evaluation of the different \ac{ec} schemes previously described. +This evaluation aims at finding the suitable methods to be used for efficient reconstruction of the missing frames in case of data loss. +The two \ac{ec} methods based on the correctly received \acs{mv} are tested, i.e., \ac{mc} and \ac{mve}, while the \ac{fc} method is used as reference for comparison. Moreover, the four different approaches to estimate the motion field for the missing frame are compared: (i) full search based on the minimisation of the sum of absolute errors (\FULLSEARCH), (ii) optical flow based on the duality between total variation and smoothness term (\DUALFLOW), (iii) optical flow based on polynomial approximations (\POLYFLOW) and (iv) optical flow based on deep matching (\DEEPFLOW). Table~\ref{TAB5:ECPerformaceComparison} shows the average quality (\ac{psnr}) obtained for the different methods under investigation. @@ -219,15 +232,15 @@ According to the results of Table~\ref{TAB5:ECPerformaceComparison} (columns 5 t Results for \FULLSEARCH method reveal that only minimising the sum of absolute errors between two consecutive frames might not always be directly related with objects' motions, which results in inaccurate motion field to be used in the reconstruction of lost frames. Finally, results from Table~\ref{TAB5:ECPerformaceComparison} reveal that the \DEEPFLOW algorithm leads to the most accurate motion field, in comparison with the other methods. It is able to achieve an average quality gain of 1.79 dB, and up to 3.39 dB for Book Arrival sequence, where \POLYFLOW only achieves 3.05 dB. -Since \POLYFLOW combines different techniques also used by other algorithms in study, e.g., total variation also used by \DUALFLOW, it is expected to achieve accurate estimations and consequently higher \ac{ec} reconstruction quality. Moreover, using descriptor matching in combination with smoothing conditions, the \DEEPFLOW algorithm is able to correctly estimate large displacements, which results in an improved performance for video signals with higher motion, e.g., Tennis sequence. +Since \POLYFLOW combines different techniques also used by other algorithms in study, e.g., total variation also used by \DUALFLOW, it is expected to achieve accurate estimations and consequently higher \ac{ec} reconstruction quality. +Moreover, using feature matching in combination with smoothing conditions, the \DEEPFLOW algorithm is able to correctly estimate large displacements, which results in improved performance for video signals with higher motion, e.g., Tennis sequence. \subsection*{Remarks} -Summarising, the analysis of the previous results indicates that all methods are able to cope with lost frames and manage to recover the missing data with acceptable quality. -Nevertheless, it is noticeable that lost frames recovered with an estimated motion field result in higher average quality, especially when using the \DEEPFLOW method. However, as the other methods tested are also able to achieve acceptable video quality, they shall not be neglected, as they provide alternative approaches for lost frame recovery with a lower computational complexity (i.e., they not require motion estimation). -Thus, in order to achieve higher reconstruction quality, different approaches shall be considered, aiming at balancing the overall complexity with the recovery quality. - -%\HERE +Summarising, the analysis of the previous results indicates that all methods are able to cope with lost frames and manage to recover the missing data with reasonable quality. +Nevertheless, it is noticeable that lost frames recovered with an estimated motion field result in higher average quality, especially when using the \DEEPFLOW method. +However, since the other methods are also able to achieve acceptable video quality, they shall not be neglected, as they provide alternative approaches for lost frame recovery with a lower computational complexity (i.e., they not require motion estimation). +Thus, in order to achieve higher reconstruction quality, different approaches shall be considered, aiming at balancing the overall complexity with the quality of the reconstructed frames. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% @@ -237,18 +250,19 @@ Thus, in order to achieve higher reconstruction quality, different approaches sh \glsreset{el} -This section presents the general architecture proposed for \ac{ec}-aware video encoding, describing its main functional blocks, and provides an introduction for the subsequent detailed descriptions. +This section presents the general architecture proposed for \ac{ec}-aware video encoding, describing its main functional blocks, and providing an introduction for the subsequent detailed descriptions. Figure~\ref{FIG5:BlockDiagram} illustrates the architecture of the proposed \ac{ec}-aware encoding scheme. After the video encoder, it includes data loss simulation at the \ac{nal} for every slice. Subsequently, \ac{ec} is applied to the simulated losses in order to find the best methods that should be used to reconstruct the lost slices in case of transmission errors. Saliency-weighted \ac{ec} mode decision is used to select the optimal \ac{ec} mode from the best trade-off between \ac{ec} distortion, using the original input video~($f$) as reference, and the number of bits required for signalling. -Then, the optimal \ac{ec} mode is sent to the decoder along with the residue formed between the error-free decoded frame and its error concealed counterpart. The scalable extension of the \ac{hevc} standard is used to encode this residue information. +Then, the optimal \ac{ec} mode is sent to the decoder along with the mismatch residue computed as a difference between the transmitted and recovered frames. +Finally, the scalable extension of the \ac{hevc} standard is used to encode this residue information. -For the \ac{ec} mode decision optimization the missing slices are reconstructed, as they would be in the decoder, i.e., recovered at the encoder by simulating the decoder \ac{ec}, but using different \ac{ec} methods based on motion estimation and extrapolation techniques, to selected the best one at the encoder-side. +For the \ac{ec} mode decision optimization the missing slices are reconstructed, as they would be in the decoder, i.e., they are recovered at the encoder by simulating the decoder \ac{ec}, but using several \ac{ec} methods, based on motion estimation and extrapolation techniques, to select the best one at the encoder-side. In this process, each \ac{ctu} is partitioned to form a quadtree structure of \ac{ecu} that are used to test the different \ac{ec} candidate modes. Then, the best trade-off between \ac{ecu} partition and signalling is found through \ac{rd} optimisation. In order to achieve an efficient use of the signalling overhead, a saliency-weighted procedure is embedded in the \ac{rd} optimisation based on a spatio-temporal saliency, where more signalling bits are allocated to regions with higher saliency values. -Finally, the signalling information is transmitted to the decoder by using the \ac{sei} \acs{nal} units~\cite{HEVCHighLevelReferenceDez2012}. -Since, in general, the optimal \ac{ec} mode is still not able to perfectly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el}. +Finally, the signalling information is transmitted to the decoder by using the \ac{sei} \ac{nal} units~\cite{HEVCHighLevelReferenceDez2012}. +Since, in general, the optimal \ac{ec} mode is still not able to perfectly reconstruct all the missing regions, extra residual information is multiplexed in the stream encoded as a scalable \ac{el}. This residual information contains the difference between the compressed ($\hat f_t$) and the \ac{ec} frames ($\tilde f$). \begin{figure}[tb] @@ -276,7 +290,7 @@ In this section the proposed saliency-weighted \ac{ec} mode decision algorithm i Firstly, the list of \ac{ec} candidates is provided based on the previously described methods. Then, the \ac{rd} optimisation is performed using a saliency region in each frame, in order to decrease the overall bitrate without compromising the reconstruction quality. The signalling coding strategy is also described. -Finally, the \ac{ec} residue is introduced as auxiliary information to improve the reconstruction quality is described. +Finally, coding the \ac{ec} residue, introduced as auxiliary information to improve the reconstruction quality, is described. \subsection{\ac{ec} candidates} @@ -306,7 +320,7 @@ The method M3 is based on a conventional \ac{mve}, and finally M4 reconstructs t \subsection{Optimal \ac{ec} mode selection} -As previously described in the sub-section~\ref{Section5:ECCandidates}, the proposed method allows to choose between four \ac{ec} candidates, which result in different recovered frames for each lost frame (simulated). +As previously described in sub-section~\ref{Section5:ECCandidates}, the proposed method allows to choose between four \ac{ec} candidates, which result in different recovered frames for each lost frame (simulated). The lost frame is partitioned into quadtree structures, each one forming a set of variable size \acs{ecu}. Then, the process of finding the optimal \ac{ec} method for each \ac{ecu} is formulated as a rate-distortion cost minimisation. To solve such problem, a Lagrangian optimisation is used to achieve the best trade-off between reconstruction quality and signalling overhead. Furthermore, image saliency driven by visual attention modelling is used in the optimisation process to weight the cost of the overhead information of each \ac{ecu}, according to the visual relevance of its content. @@ -340,13 +354,15 @@ m_{l,i}^* = \argmin\limits_{\footnotesize \substack{l\in\left\{0...3\right\} \\ where $R(m_{l,i})$ represents the number of overhead bits required to encode the \ac{ec} mode, and $D(m_{l,i})$ is the sum of absolute error of the \ac{ecu}. -In equation~\ref{EQ6:OptimalDepth}, besides the parameter $\lambda$ used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation. As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques}, this method uses the ratio between the temporal distance from the current frame ($t_C$) to the next refresh frame ($t_I$) and the period of refreshing frames ($T_{I}$). +In equation~\ref{EQ6:OptimalDepth}, besides the parameter $\lambda$, used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation. +As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques}, this method uses the ratio between the temporal distance from the current frame ($t_C$) to the next refresh frame ($t_I$) and the period of refreshing frames ($T_{I}$). %This weight varies between $\left[0;1 \right]$, achieving the lowest value for intra-coded frames and the maximum for the frame located immediately before an intra-coded one. -This behaviour follows the fact that when a lost frame is closer to the next intra-coded frame, error propagation is lower. Thus, it is not worthwhile to spend too many overhead bits to reduce it. This is accomplished by increasing the cost of such bits according to the temporal distance between the missing frame and the next I frame. -Moreover, a dynamic value for $\lambda$ is also used to adjust the amount of overhead within the frame, according to the visual relevance of each image region, determined using a saliency map. +This behaviour follows the fact that when a lost frame is closer to the next intra-coded frame, the error propagation is lower. +Thus, it is not worthwhile to spend too many overhead bits to reduce it. This is accomplished by increasing the cost of such bits according to the temporal distance between the missing frame and the next I frame. +Moreover, a dynamic value for $\lambda$ is also used to adjust the amount of overhead within the frame, according to the visual relevance of each image region, defined by a saliency map. The algorithm previously described results in \acs{ecu} with variable size, which is defined by the depth of the quad-tree structure, i.e., level $l$. The partition size is jointly optimised along with the best \ac{ec} candidate method using Equation~\ref{EQ6:OptimalDepth}. Alternatively, a fixed \ac{ecu} size can also be used by defining a fixed depth for all quad-tree structures, i.e., constant $l$ for the entire frame. This can be achieved by using Equation~\ref{EQ6:OptimalDepth} to solely optimise the \ac{ec} methods for each \ac{ecu}. -Although this reduces the overall complexity of the proposed method, it leads to a non-optimal solution which may required additional overhead. In Section~\ref{Section5:ECSignallingPerformance} a comparison between a dynamic and fixed approach is provided. +Although this reduces the overall complexity of the proposed method, it leads to a non-optimal solution which may require additional overhead. In Section~\ref{Section5:ECSignallingPerformance} a comparison between a dynamic and fixed approach is provided. \subsection{Saliency detection} @@ -414,7 +430,7 @@ Adaptive arithmetic coding~\cite{ArithmeticCodingJun1994} is used to efficiently As described earlier in this section, in order to allow further improvement of recovered slices, besides the best \ac{ec} mode, the proposed encoding scheme also allows encoding a controlled amount of \ac{ec} residue using the scalable extension of \ac{hevc} (see Figure~\ref{FIG5:BlockDiagram}). The enhancement layer module encodes the difference between the coded frame ($\hat f$) and its counterpart recovered by using the optimal \ac{ec} mode ($\tilde f$). Note that, the higher the accuracy of the best \ac{ec} mode, the lower the -amount of bits spent by the enhancement layer. +amount of bits carried by the enhancement layer. At the decoder-side, the reconstruction of a missing frame ($\tilde f$) follows three steps: \begin{enumerate} \item the missing frame is reconstructed using the optimal \ac{ec} methods for each \ac{ecu} $\tilde f'$ received from the encoder; @@ -428,7 +444,7 @@ At the decoder-side, the reconstruction of a missing frame ($\tilde f$) follows The additional residue decoded from the enhancement layer is particularly useful to improve the quality of those \acs{ctu} where the \ac{ec} strategies are not able to produce very accurate reconstruction. Moreover, since the enhancement layer only carries residual information the overhead is affordable. -Note that the generated bitstream is compliant with the standard, as the \ac{ec} signalling is encapsulated in \ac{sei} \acs{nal}, while the error concealment residue is carried in \acs{nal} related with the scalable extension. Then, an adapted decoder, capable of decoding both the \ac{sei} and the scalable \ac{nal} units is also able to take full benefit from the proposed method. Nevertheless, any standard \ac{hevc} decoder is also able to reconstruct the video, because it simply ignores the extra \ac{nal}. +Note that the generated bitstream is compliant with the standard, as the \ac{ec} signalling is encapsulated in \ac{sei} \acs{nal}, while the \ac{ec} residue is carried in \acs{nal} related with the scalable extension. Then, an adapted decoder, capable of decoding both the \ac{sei} and the scalable \ac{nal} units is also able to take full benefit from the proposed method. However, any standard \ac{hevc} decoder is also able to reconstruct the video, because it simply ignores the extra \ac{nal} units. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -441,6 +457,7 @@ This section presents the performance of the proposed \ac{ec}-aware optimisation Two different types of performance evaluation are presented. In the first one, only robust \ac{ec} of intra-frames is evaluated, while in the second one the proposed \ac{ec}-aware optimisation is applied to all frames, corresponding to a generic robust video transmission using optimal \ac{ec}. For comparison, it is used the \ac{psnr} obtained from the reference \ac{hevc} decoder using motion-copy (\MC) and frame-copy (\REF) for \ac{ec} of the lost frame. +Realistic conditions were used for transmission, by allowing independent losses for any type of \ac{nal} units, hence there is no guarantee that the decoder conditions are always equal to those at the encoder, which means that sub-optimal \ac{ec} may be used at the decoder. \subsection{Intra-frame EC optimisation} @@ -450,7 +467,7 @@ In this section the performance of the proposed method is evaluated only for int Since intra \acs{cu} result in higher amount of bits, they also require a higher number of packets, making them more prone to transmission errors and losses. Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets correspond to intra coded slices, which is a non-negligible probability of one of them being hit by transmission errors. Moreover, by limiting the proposed method to intra frames, the amount of required overhead is reduced. -In this case a constant $\lambda_0=10$ was considered without taking into consideration saliency estimation. +In this case a constant $\lambda_0=10$ was used, without taking into consideration saliency estimation. Since a relevant performance indicator is the error propagation over temporally predicted frames, this was evaluated using a single intra-frame loss. Figures~\ref{FIG5:singlelosslowdelay} and~\ref{FIG5:singlelossramdon} show the \ac{psnr} results when frame Intra-frame $\#16$ is missing, for \ac{ld} and \ac{ra} encoding configurations, respectively. @@ -524,12 +541,12 @@ It is also worthwhile to notice that the \ac{psnr} variance obtained with the %\subsubsection{Overhead analysis} Subsequently, the percentage of overhead bitrate introduced by the proposed method is compared against fixed \ac{ecu} sizes of $32\times32$ and $16\times16$, i.e., $Level=1$ and $Level=2$. -These results are presented in Table~\ref{TAB6:RedundancyIntra} and were obtained for a total bitrate, as shown in Table~\ref{TableTestSequences}. +These results are presented in Table~\ref{TAB6:RedundancyIntra} and were obtained for the total bitrate shown in Table~\ref{TableTestSequences}. It is noticeable from the results that reducing the \ac{ecu} size (see results for $32\times32$ and $16\times16$) increases the overhead ratios as more symbols are transmitted (one for each \ac{ecu}). While the maximum overhead of these fixed partitions is $1.81\%$ (Traffic), the proposed dynamic partitioning method is able to reduce the overhead to a maximum of $0.19\%$ (Kendo). -Moreover, comparing the results of Table~\ref{TAB6:RedundancyIntra} with the video quality results obtained in case of packet losses reveals in the proposed method only a small reduction in the reconstruction quality, even with a significantly lower overhead. +Moreover, comparing the results of Table~\ref{TAB6:RedundancyIntra} with the video quality obtained under packet loss reveals that the proposed method only incurs in a small reduction of the reconstruction quality, with a significantly lower overhead. This indicates that using dynamic partitioning allows to achieve a better trade-off between the amount of overhead and the \ac{ec} performance. -In summary, the proposed technique is able to enhance the performance of the standard decoding of intra coded frames by finding the best error concealment method at the encoder and sending such information with a small amount of signalling overhead. +In summary, the proposed technique is able to enhance the performance of the standard decoding of intra coded frames by finding the best \ac{ec} method to be used, for each \ac{ecu}, at the encoder using with a small amount of signalling overhead. \begin{table}[tbp] \newcommand{\somespaces}{} @@ -568,7 +585,8 @@ In summary, the proposed technique is able to enhance the performance of the sta \label{SEC4:OverallPerformance} In this section the proposed \ac{ec}-aware optimisation scheme is evaluated for a generalised robust video transmission of both intra and inter coded frames. -The experimental results presented in this section were obtained by applying the proposed technique to all video frames. +The experimental results presented in this section were obtained by applying the proposed technique to all video frames. +As in the previous subsection, a $\lambda_0=10$ was used. This section is organised in the following subsections: \begin{itemize} \item evaluation of the usage distribution of each \ac{ec} mode; @@ -620,8 +638,7 @@ The \ac{ec} candidate methods based on motion field estimation (i.e., M1, M2) ar The evaluation of the proposed saliency-weighted \ac{ec} mode decision (\DYNLAMBDA) was carried out under random loss events and it was compared with the reference methods. The video quality was evaluated by using two quality metrics: \ac{psnr} and a \ac{fsim}~\cite{FSIM_TIP2011}, in order to validate the performance of the proposed method in a consolidated manner. -Realistic conditions were used for transmission, by allowing independent losses for any type of \acs{nal}, hence there is no guarantee that the decoder conditions are always equal to those at the encoder, which means that sub-optimal \ac{ec} may be used at the decoder. -The results obtained without saliency weighting are also shown for comparison (\FIXEDLAMBDA). As in the previous subsection, a $\lambda_0=10$ was used. +The results obtained without saliency weighting are also shown for comparison (\FIXEDLAMBDA). Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM} show the average quality under different \acs{plr} for the \ac{ld} configuration. Table~\ref{TAB4:SignallingPSNR} shows the \ac{psnr} and weighted \ac{psnr} (\ac{wpsnr}) between parentheses. @@ -636,9 +653,9 @@ Moreover, the results of Table~\ref{TAB4:SignallingFSIM} confirm the quality imp Table~\ref{tab:redundancy_signalling} shows the percentage of overhead in the total rate with and without saliency-weighted optimisation, i.e., \FIXEDLAMBDA and \DYNLAMBDA, respectively. As shown in this table, the saliency-weighted optimisation reduces the amount of overhead by dynamically assigning unequal number of bits to image regions of different visual importance. These results show that using an adaptive value of $\lambda$, based on the saliency values, leads to an average overhead reduction of 28\% in comparison with the case where no saliency is used, i.e., constant $\lambda$. -It is worth to note that using the saliency-weighted optimisation to reduce the overhead does not compromise the video quality, as can be seen in the results shown in Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM}. +It is worthwhile to note that using the saliency-weighted optimisation to reduce the overhead does not compromise the video quality, as can be seen in the results shown in Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM}. The \ac{psnr} values for \FIXEDLAMBDA and \DYNLAMBDA are very close, but in terms of overhead savings the results are significant for most sequences. -The achieved \ac{wpsnr} and \ac{fsim} results further demonstrate the effectiveness of the saliency-based optimisation, because the lower overhead does not affect the most relevant image regions, i.e., those with higher saliency values. Thus, the proposed method using a adaptive $\lambda$ (i.e., using saliency) is able to not only achieve higher \ac{ec} quality than the reference methods, but also higher quality gains than using a constant $\lambda$ (i.e., without saliency). +The achieved \ac{wpsnr} and \ac{fsim} results further demonstrate the effectiveness of the saliency-based optimisation, because the lower overhead does not affect the most relevant image regions, i.e., those with higher saliency values. Thus, the proposed method, using a adaptive $\lambda$ (i.e., using saliency), is able to not only achieve higher \ac{ec} quality than the reference methods, but also higher quality gains than using a constant $\lambda$ (i.e., without saliency). This confirms the two-fold superiority of adaptive $\lambda$, i.e. not only the total overhead is reduced but also higher quality gains are achieved. %\todo{\HERE} @@ -733,14 +750,14 @@ R = \frac{\Delta PSNR(\ac{plr}=10\%)}{-\Delta PSNR(Error\text{-}free)}. The results shown in Table~\ref{tab:overall_gain_ratio}, demonstrate that the effectiveness of the \ac{ec}-aware coding scheme is higher than all other methods under comparison. In fact, not only the absolute \ac{psnr} difference (i.e., \ac{psnr} gain) is higher for the same \ac{plr}, but also the efficiency of the overhead is also higher. This means that the overhead bits spent to encode the signalling information and the enhancement residue carry more useful information than the other methods. For instance, in comparison with the method in \cite{DualFrameME2015} the proposed method leads to higher quality degradation in the error-free case. However, for PLR=10\% it clearly outperforms the method in \cite{DualFrameME2015} with a much higher value of $R$, indicating a far superior efficiency. -Results in the table~\ref{tab:overall_gain_ratio} also show that the small loss of coding efficiency in error-free transmission is clearly surpassed by the quality gains under error-prone transmission conditions. +Results in Table~\ref{tab:overall_gain_ratio} also show that the small loss of coding efficiency in error-free transmission is clearly surpassed by the quality gains under error-prone transmission conditions. The methods in \cite{ROPERefGenJan2010,DualFrameME2015} are based on modified predictions, while the method in \cite{VPxResiliencePCS2016} uses redundant data without any further optimisation. Both of these approaches require low additional computational complexity. However, as observed in the results shown in Table~\ref{tab:overall_gain_ratio}, the lower additional complexity also corresponds to lower quality gains in case of transmission errors. -The proposed method uses \ac{ec} techniques with different levels of computational complexity for each \ac{ecu}. Since an adaptive approach is used, the average complexity tends to be lower in comparison with other methods that rely on a fixed approach. Thus, less complex video decoding is obtained when using the proposed method when compared with other straightforward methods that always used the same \ac{ec} technique for all \acs{ctu}. +The proposed method uses \ac{ec} techniques with different levels of computational complexity for each \ac{ecu}. Since an adaptive approach is used, the average complexity tends to be lower in comparison with other methods that rely on a fixed approach. Thus, less complex video decoding is obtained when using the proposed method in comparison with other straightforward methods that always use the same \ac{ec} technique for all \acs{ctu}. \begin{table}[t] @@ -772,8 +789,8 @@ The proposed method uses \ac{ec} techniques with different levels of computati \section{Summary} \label{Section5:Summary} -In this chapter an \ac{ec}-aware encoding scheme is proposed by incorporating loss simulation at the encoder-side to enhance the EC efficiency at the decoder. +In this chapter a novel \ac{ec}-aware encoding scheme was proposed by incorporating loss simulation at the encoder-side to enhance the EC efficiency at the decoder. % The proposed method relies on loss simulation at the encoder-side to optimise the error concealment proposed. Then, the optimal error concealment is signalled to the decoder. -The novelty of this method is to include optimal \ac{ec} mode decision at the encoder, to be used by the decoder in case of packet loss, and also to allow encoding of the \ac{ec} residue using the enhancement layer of the \ac{hevc} standard. The use of \ac{ec} signalling was firstly proposed for the use on intra-coded frames in \ref{Conf:ConcAware}, and then improved and proposed for a generic robust video transmission, resulting in a more detailed analysis published in \ref{Journal:ConcAware}. +The novelty of this method is to include optimal \ac{ec} mode decision at the encoder, to be used by the decoder in case of packet loss, and also to allow encoding of the \ac{ec} residue using the enhancement layer of the \ac{hevc} standard. The use of \ac{ec} signalling was firstly proposed for the use on intra-coded frames in (\ref{Conf:ConcAware}), and then improved and proposed for a generic robust video transmission, resulting in a more detailed analysis published in (\ref{Journal:ConcAware}). The overall results demonstrate that the proposed method is able to efficiently reconstruct lost frames and reduce the error propagation compared with other reference methods. -A consistent quality improvement achieved at the cost of low overhead allows the use of this method in broadcasting video services and applications where packet loss probability is not negligible. +A consistent quality improvement achieved at the cost of low overhead allows the use of this method in broadcasting video services and network applications where packet loss probability is not negligible. diff --git a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex index 6502b3b..2f6ab09 100644 --- a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex +++ b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex @@ -1,7 +1,7 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % CHAPTER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\chapter{Adaptive Error Robustness for Highly Efficient Video Coding \VERSION{2}} +\chapter{Adaptive error robustness for highly efficient video coding \VERSION{3}} \label{Chapter:EndToEndRobustSystem} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -10,15 +10,17 @@ \section{Introduction} \glsunset{hevc} -\glsunset{ec} +\glsreset{ec} +\glsreset{rd} +\glsreset{el} \glsreset{nn} The effectiveness of different error resilience techniques is rather variable depending on the video content, the transmitted bitrate, network conditions, among others. -Therefore, it is not straightforward to select a single technique which achieves the highest efficiency for every possible scenario and also for the whole range of networking conditions. -This chapter addresses this problem by proposing an adaptive network that optimises the efficiency of the techniques previously described in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}. -At the encoder-side, a pre-trained neural network predicts the video stream that is able to achieve the highest decoded quality for a particular loss cases, among the various video streams encoded with different levels of compression efficiency and error robustness. -The selection is based on different input parameters that include information from the video signal, the coded stream and the transmission network. -Using this approach the overall performance of the proposed adaptive method significantly increases, yielding a better trade-off between compression efficiency and error robustness. +Therefore, it is not straightforward to select a single technique capable of guaranteeing the highest efficiency for every possible scenario and also for the whole range of networking conditions. +This chapter addresses this problem by proposing an adaptive error robustness scheme that optimises the efficiency of the techniques previously described in Chapters~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}. +At the encoder-side, a pre-trained neural network predicts the video stream that is able to achieve the highest decoded quality for particular loss cases, among various video streams encoded with different levels of compression efficiency and error robustness. +The stream selection is based on different input parameters that include information from the video signal, the coded stream and the transmission network. +Using this approach the overall performance of the proposed adaptive method significantly increases, achieving a better trade-off between compression efficiency and error robustness. This chapter is organised as follows. Section~\ref{Section6:Problem} presents the problem addressed in this chapter and experimental evidence, as motivation for the method investigated. @@ -33,23 +35,23 @@ Finally, Section~\ref{Section6:Summary} concludes the chapter. \section{The problem of adaptive error robustness} \label{Section6:Problem} -In the scope of this thesis different robust coding techniques were investigated to increase the overall performance of video transmission under error-prone conditions. +In the scope of this thesis different robust coding techniques were investigated and described in previous chapters to increase the overall performance of video transmission under error-prone conditions. Different algorithms were devised in the proposed methods, such as, reference frame selection, motion vector redundancies and error concealment signalling. -In previous chapters, these methods were evaluated against existing techniques, revealing superior performance and better efficiency to deal with error robustness problems in high efficient video coding. +In the research work described in Chapters~\ref{Chapter:ErrorResilienceReview}~to~\ref{Chapter:ConcealmentAwareTechniques}, these methods were evaluated against existing techniques, revealing superior performance and better efficiency to deal with error robustness problems in high efficient video coding. Overall, the performance results of these methods have demonstrated that they can efficiently increase the robustness of video streams against transmission errors and losses, leading to higher decoded video quality. However, each method on its own does not adapt to different conditions. -While the reference frame selection algorithm presented in Chapter~\ref{Chapter:ErrorResilienceTechniques} introduces low overhead in comparison with existing techniques, the method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} achieves higher quality gains at cost of slightly higher bitrate increase. +For instance, while the reference frame selection algorithm presented in Chapter~\ref{Chapter:ErrorResilienceTechniques} introduces low overhead in comparison with existing techniques, the method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} achieves higher quality gains at cost of slightly higher bitrate increase. Therefore, one may conclude that each method may be further optimised for different applications, video characteristics and network conditions. -This chapter advances one step forward , by investigating a joint robust coding method using different techniques to optimise the robust coder across different conditions, such as, the input content and the network information. -This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and can be selected based on the current conditions. +This is the motivation for the research presented in this chapter, which advances one step forward by investigating a joint robust coding method using different techniques to optimise the video coding process across different conditions, such as, the input content and the network information. +This scheme can be used in transmission systems where different video bitstreams can be made available at the encoder-side to be selected based on the dynamic conditions. \subsection{Comparison of the proposed methods} In order to address the above problem of adaptive robust coding, a simulation study was carried out to compare the error robustness performance of the previously proposed methods for different conditions. -This study extends the previous evaluations presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, as they target a wider range of test material and network conditions. -Moreover, it aims at comparing the performance of different techniques across a variety of video contents, bitrates and loss conditions, in order to find relevant insights which affect the decoded video quality. Consequently, these relevant insights may be used to develop an adaptive robust coding method. +This study extends the previous evaluations presented in Chapters~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, as they target a wider range of test material and network conditions. +Moreover, it aims at comparing the performance of different techniques across a variety of video content, bitrates and loss conditions, in order to find relevant insights about the most affect the decoded video quality. Consequently, these relevant insights may be used to develop an adaptive robust coding method. %In order to perform a wider evaluation different bitrate and \acs{plr} were used. Table~\ref{Table6:TestVariables} illustrates the selected configurations. As shown in the table a total of 15 test sequences were used encoder using 5 different target bitrates. One should note that the main bitrate varies based on the test sequence and it is shown in third column of Table~\ref{TableTestSequences}. Different network conditions were evaluated using a total of 13 different \acs{plr}, with higher special focus on lower loss ratios, which are more relevant to modern video streaming applications. The following methods were evaluated in the simulation study, namely: @@ -57,17 +59,17 @@ The following methods were evaluated in the simulation study, namely: \item the default \ac{hevc} decoder with frame-copy \ac{ec} (\REF), as reference method; \item the default \ac{hevc} encoder without robust coding-techniques and motion vector extrapolation \ac{ec} (\MEC); \item the proposed reference frame selection scheme (\MEXP); - \item the \MEXP method in combination with an \ac{ec} optimisation with 10\% of bitrate allocated for the scalable enhancement layer (\MEXPELTEN); + \item the method \MEXP in combination with an \ac{ec} optimisation with 10\% of bitrate allocated for the scalable enhancement layer (\MEXPELTEN); \item the same as above, with 20\% of bitrate allocated for the scalable enhancement layer (\MEXPELTWEN). \end{itemize} -These methods were selected as they are the main techniques proposed within this research work and cover two aspects of a robust video transmission: (i) robust video encoding by dynamically select the reference frames and (ii) robust video decoding by improving the error concealment performance. +These methods were selected because they are the main techniques proposed within this research work and cover two elements of a robust video communication chain: (i) robust video encoding by dynamically select the reference frames and (ii) robust video decoding by improving the \ac{ec} performance. \input{043_EndToEndRobustSystem/Results/AdaptiveSequenceList.tex} Table~\ref{Table6:TestMaterialCharacteristics} presents the list of test sequences used. These sequences were selected to cover a wide range of video content, with different levels of spatial and temporal complexity, as demonstrated by the values of \ac{si} and \ac{ti}~\cite{ITUTP9102008}, respectively. Moreover, from the bitstream encoded with the reference implementation of the \ac{hevc} encoder, the percentage of the inter-coded blocks and the average \ac{mv} magnitude was extracted and presented in the table (columns 4 and 5, respectively). -These parameters were considered as they might be related with the performance obtained with the proposed robust coding methods. +These parameters were considered due to their possible relation with the performance obtained by the proposed methods. The remaining configuration was kept as described in Section~\ref{Section3:ErrorResilienceEvaluation}. %The experimental methodology carried out in this section follows the same conditions described in Section~\ref{Section3:ErrorResilienceEvaluation}. @@ -81,9 +83,9 @@ The remaining configuration was kept as described in Section~\ref{Section3:Error \label{Figure6:AverageQualityMain} \end{figure} -Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} shows the average decoded video quality \ac{psnr} obtained from 50 trials, for different \acs{plr}. -Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate listed in Table~\ref{Table6:TestMaterialCharacteristics}, referred to as main bitrate. -Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained by reducing the main bitrate by 40\%. This is used to evaluate the impact of the network bitrate on the decoded video quality. The results shown in the figures corresponds to the same test sequences, in order to carry out an accurate comparison. +Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} show the average decoded video quality (\ac{psnr}) obtained for different \acs{plr}. +Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate listed in Table~\ref{Table6:TestMaterialCharacteristics}, referred to as the main bitrate. +Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained by reducing the main bitrate by 40\%. This is used to evaluate the impact of the network bandwidth on the decoded video quality. The results shown in the figures correspond to the same test sequences, in order to allow an accurate and fair comparison. @@ -95,17 +97,18 @@ Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained by reducin \label{Figure6:AverageQualityReduction} \end{figure} -Firstly, comparing the results for different bitrates, it is noticeable that lower quality is obtained for the lower bitrates, which results from the higher compression ratios used. It is also noticed that lower quality gains, when comparing with the reference \ac{hevc} encoder (\REF), are obtained for the video streams encoded with a 40\% reduction of bitrate. This indicates that for lower bitrates the performance of the proposed methods decreases, especially for the \MEXPELTWEN case, where 20\% of the bitrate is allocated for the enhancement layer. Therefore, for lower bitrates, using methods that have a lower impact on the coding efficiency (e.g., \MEXP) or even disable the robust coding techniques (\MEC) may lead to higher performance. +Firstly, comparing the results for different bitrates, it is noticeable that lower quality is obtained for the lower bitrates, which obviously results from the higher compression ratios used. It is also noticed that lower quality gains, when comparing with the reference \ac{hevc} encoder (\REF), are obtained for the video streams encoded with a 40\% reduction of bitrate. This indicates that for lower bitrates the performance of the proposed methods decreases, especially for the \MEXPELTWEN case, where 20\% of the bitrate is allocated for the enhancement layer. Therefore, for lower bitrates, using methods that have a lower impact on the coding efficiency (e.g., \MEXP) or even disable the robust coding techniques (\MEC) may lead to higher performance. Secondly, comparing the results across different video content, i.e., different test sequences, one can notice that the performance of the different methods is not consistent. -For instance, in the result of Figure~\ref{Figure6:AverageQualityMain}, for Tennis sequence the \MEXPELTWEN clearly outperforms the remaining methods, however, for the BQSquare sequence higher performance is achieved by using less bitrate (i.e., 10\%) allocated for the \ac{el} (\MEXPELTEN). -This is a consequence of the higher quality degradation due to higher motion in Tennis sequence (see results for TI and \ac{mv} size in Table~\ref{Table6:TestMaterialCharacteristics}) which required higher redundant information to increase the overall quality. +For instance, in the results of Figure~\ref{Figure6:AverageQualityMain}, for Tennis sequence the method \MEXPELTWEN clearly outperforms the remaining ones. +However for the BQSquare sequence, higher performance is achieved by using less bitrate (i.e., 10\%) allocated for the \ac{el} (\MEXPELTEN). +This is a consequence of the higher quality degradation due to higher motion in Tennis sequence (see results for TI and \ac{mv} size in Table~\ref{Table6:TestMaterialCharacteristics}) which requires higher redundant information to increase the overall quality. %\HIGHLIGHT{Mention SI!!} -Finally, comparing the results for all test sequences and both bitrates it can be clearly noticed that the \acs{plr} have a significant impact on the performance of the error robustness techniques. In case of the proposed reference frame selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions. For the \MEXPELTWEN case, higher \acs{plr} are required to be able to outperform both the reference \ac{hevc} and the \ac{mve} method. +Finally, by comparing the results for all test sequences and both bitrates it can be clearly noticed that the \ac{plr} has a significant impact on the performance of the error robustness techniques. In case of the proposed reference frame selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions. For method \MEXPELTWEN, higher \acs{plr} are required to be able to outperform both the reference \ac{hevc} and the \ac{mve} method. -Summarising, the performance of the proposed methods is not similar across different video content and bitrates, opening research to adaptively choose the best method that should be applied to achieve the highest decoded video quality. -In order to achieve increased efficiency and higher video quality, the best trade-off between auxiliary information and video bitrate can be found by selecting the optimal robust coding algorithm. +Summarising, the performance of the proposed methods does not exhibit a consistent behaviour across different video content and bitrates, opening a new research goal to adaptively choose the best method that should be applied to achieve the highest decoded video quality. +In order to achieve increased efficiency and higher video quality, the best trade-off between auxiliary information and video bitrate must be found by selecting the optimal robust coding algorithm. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% @@ -113,8 +116,8 @@ In order to achieve increased efficiency and higher video quality, the best trad \section{Architecture for adaptive error robustness} \label{Section6:ProposedMethod} -This section describes the proposed method, which aims at obtaining optimal error robustness for different streaming scenarios, characterised by different video content and network conditions. -The main objective of this robust coding scheme it to achieve an optimal trade-off between error robustness tools and coding efficiency. +This section describes the method proposed to attain the research goal defined above, which aims at obtaining optimal error robustness for different streaming scenarios, characterised by different video content and network conditions. +Figure~\ref{Figure6:BlockDiagram} illustrates the proposed architecture, whose main objective is to achieve an optimal trade-off between error robustness tools and coding efficiency. \begin{figure}[tb] \centering @@ -125,8 +128,8 @@ The main objective of this robust coding scheme it to achieve an optimal trade-o \FIGSVSPACE \end{figure} -Figure~\ref{Figure6:BlockDiagram} illustrates the proposed architecture. -Firstly, the input signals are coded using different levels of error protections. Then, an optimisation algorithm is used to decide which bitstream should be transmitted at any given instant, based on different parameters. The selected stream is expected to obtain the highest quality at the decoder-side for the given conditions. + +Firstly, the input signals are coded using different levels of error protection. Then, an optimisation algorithm is used to decide which bitstream should be transmitted at any given instant, based on different parameters. The selected stream is expected to obtain the highest quality at the decoder-side for the given conditions. A reference video coder is used to compress the video signal using highly efficient coding techniques, obtaining the best \ac{rd} performance for error-free environments. This is used as the reference bitstream. Subsequently, a transcoding operation is applied to this reference bitstream aided by the input video signal to efficiently select the reference frames used for inter prediction. @@ -140,12 +143,12 @@ The proposed approach results in four different bitstreams with different charac (iii) \SMEXPELTEN: robust video coding using selective reference frames and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue; (iv) \SMEXPELTWEN: same as the previous bitstream but with 20\% bitrate allocated for the scalable~\ac{el}. -Finally, the video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and feedback from the transmission channel. +The video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and feedback from the transmission channel. From the input video signal, relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, further information is also extracted from the coded bitstream using a stream parser. -Finally, the loss ratio of the transmission channel is also used. +Finally, the packet loss ratio of the transmission channel is also used. These parameters are fed into the optimisation block to select the best bitstream for transmission. The optimisation process is described in detail in the following section. - +% TODO: dizer que é assumido PLR com variações lentas para o atraso do feedback da rede nao ter influencia. The main purpose of this architecture is to allow dynamic stream switching during real time transmission. To accomplish this without causing drift at the decoder, the switching procedure should take place at refresh points, i.e., \ac{idr} and \ac{cra} frames, which correspond to clean random access points. In the proposed scheme, \ac{idr} frames are introduced every 16 frames, resulting in between 1 to 3 switching points every second, depending on the frame rate of the video signal. @@ -154,17 +157,17 @@ To accomplish this without causing drift at the decoder, the switching procedure %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\section{Error robustness selection method} +\section{Robust stream selection method} \label{Section6:OptimisationMethod} In this section the optimisation method used to select a robust video stream is presented. The selection method is based on a shallow neural network which predicts the optimal stream based on different parameters. -This section describes both the input parameters and the neural network used in the optimal stream selection process. Moreover, a description data set used to train and evaluate the neural network used is also provided. +This section describes both the input parameters and the neural network used in the optimal stream selection process. Moreover, a description data set used to train and evaluate the neural network is also provided. \subsection{Data set} %\def\NumCases{845\xspace} -\def\NumCases{910\xspace} +\def\NumCases{1092\xspace} The proposed method includes an optimisation stage that uses a neural network classifier, in order to select the optimal stream to be used for transmission. @@ -176,7 +179,7 @@ Therefore, several test cases were simulated in order to obtain results for a wi \item Packet loss ratio. \end{itemize} -To obtain the data set, 14 different test sequences with different scene content were used, as listed in Table~\ref{Table6:TestMaterialCharacteristics}. Those sequences were encoded with the main bitrate (see Table~\ref{Table6:TestMaterialCharacteristics}), as well as four different others, obtained by reducing the main bitrate by 10\%, 20\%, 30\% and 40\%. +To obtain the data set, 14 different test sequences with different scene content were used, as listed in Table~\ref{Table6:TestMaterialCharacteristics}. Those sequences were encoded with the main bitrate (see Table~\ref{Table6:TestMaterialCharacteristics}), four different others, obtained by reducing the main bitrate by 10\%, 20\%, 30\% and 40\%, and one with 10\% of bitrate increase. Then, the following \acs{plr} were used to cover a wide range of loss ratios giving more detail to low \acs{plr}: 0\%, 0.5\%, 1.0\%, 1.2\%, 1.6\%, 2.0\%, 2.2\%, 2.4\%, 2.6\%, 2.8\%, 3.0\%, 5.0\% and 10.0\%. This results in a total of \NumCases test conditions. @@ -186,23 +189,24 @@ This results in a total of \NumCases test conditions. \glsreset{ti} A set of different parameters is used to predict the optimal stream that achieves the highest quality in specific network conditions. -Based on these inputs the proposed scheme is able to select the appropriate robust coding scheme which should be used. -These parameters aim at characterise three different aspects of the video transmission system; (i) input video (two parameters), (ii) video stream (three parameters) and (iii) transmission network (one parameter). +Based on these inputs the proposed scheme should be able to select the appropriate robust coding stream that should be transmitted. +The parameter set characterises three different aspects of the video transmission system; (i) input video (two parameters), (ii) video stream (three parameters) and (iii) transmission network (one parameter). %These parameters aim at covering three different aspects of the video transmission: two parameters from the input video, three parameters from the video stream and one parameter from the transmission network. -From the input video the two parameters are measured in a frame by frame basis, while for the video stream information a stream parser is applied to the reference bitstream that was compressed only based optimal \ac{rd} optimisation without considering any robust coding techniques, as shown in Figure~\ref{Figure6:BlockDiagram}. Finally, the parameter related with the transmission network can be obtained using feedback information from the decoder. +From the input video the two parameters are measured in a frame by frame basis, while for the video stream information a stream parser is applied to the reference bitstream that was compressed only based optimal \ac{rd} optimisation without considering any robust coding techniques, as shown in Figure~\ref{Figure6:BlockDiagram}. Finally, the parameter related with the transmission network can be obtained using feedback information from the decoder or the network itself. %From the coded video streams complementary information to characterise the streaming content is also extracted. A stream parser is applied to the reference bitstream without error robustness techniques. Therefore, the prediction information extracted is related to a stream coded based on optimal \ac{rd} optimisation which was not affected by robust coding techniques. The following is extracted: The characterisation of these three aspects of the video transmission results in a total of six parameters: \begin{enumerate} -\item \textit{\ac{si}}: parameter expressing the amount of spatial detail, which consequently affects the error recovery operations, leading to higher quality degradation~\cite{ITUTP9102008}. +\item \textit{\ac{si}}: parameter expressing the amount of spatial detail, with impact on the error recovery performance~\cite{ITUTP9102008}. -\item \textit{\ac{ti}}: parameter expressing the amount of difference between temporally adjacent frames~\cite{ITUTP9102008}. This has a direct relation to the impact of errors, and thus the performance of the error resilience techniques, as shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, which were discussed in Section~\ref{Section6:Problem}. +\item \textit{\ac{ti}}: parameter expressing the amount of difference between temporally adjacent frames~\cite{ITUTP9102008}. +This has a direct relation in the impact of errors and performance of the error resilience techniques, as shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, which were discussed in Section~\ref{Section6:Problem}. \item \textit{Inter (percentage)}: the percentage of the image that is compressed using inter-frame prediction techniques. This is an indication of the distribution of spatial and temporal dependencies. -This is calculated as the ratio of inter-coded blocks over intra-coded ones. +This is calculated as the ratio between inter-coded blocks and intra-coded ones. Normally, coded streams with higher number of temporal dependencies are more prone to error propagation, thus higher number of bits should be allocated for error protection \cite{UEPScalable_TCSVT2010}. \item \textit{AVG\_MV}: for the inter-coded regions, the motion information is analysed and the average \ac{mv} magnitude is calculated to provide information about the motion intensity. @@ -211,7 +215,7 @@ As shown in previous studies, motion-based \ac{ec} algorithms are less efficient \item \textit{Bitrate}: information regarding the average bit allocation per frames is calculated, since it will affect the video quality and the amount of bits that can be available for auxiliary information. This value is expressed in Mbps, as illustrated in Table~\ref{Table6:TestMaterialCharacteristics}. \item \textit{\ac{plr}:} feedback information from the network to characterise the transmission losses, expressed by the percentage of packets loss. This is used since for higher loss ratio there is a clear increase in the quality degradation, which affects the performance of the robust coding techniques. -Transmission networks with higher loss ratios require a higher bit allocation for robust techniques, therefore the error resilience technique should adapt based on the value of the \ac{plr}. +Transmission networks with higher loss ratios require a higher bit allocation for robust techniques, therefore the error resilience technique should be able to adapt based on the value of the \ac{plr}. \end{enumerate} @@ -226,7 +230,8 @@ This aims at finding the correlation between the input parameters and the output In this study the streams are characterised by a number between 1 and 4. Table~\ref{Table6:Correlation} shows the absolute Person Correlation coefficient obtained across all sequences, bitrates and \acs{plr}. The table shows the results for the parameters previously described, as well as, a new parameter obtained from the average magnitude of the optical flow measured using the method in~\cite{DeepFlow2013} (\textit{OPT\_FLOW}). -Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable. As both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method, as it achieves a higher Person Correlation coefficient. +Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable. +Since both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method, due to its higher Person Correlation coefficient. \begin{table}[b] \centering @@ -262,7 +267,7 @@ In the past \acs{nn} have been widely used in video processing applications such In the proposed method a feed-forward network is used, which connects every neuron in a given layer to the neurons of the subsequent layer. Figure~\ref{FIG6:NeuralNetwork} shows the diagram of the \ac{nn} used in the proposed method. -The network contains and input and output layer, as well as, a single hidden layer. +The network contains an input and output layer, as well as, a single hidden layer. The input layer receives the six inputs, as described in the previous sub-section. These inputs are correlated with the optimisation variable being maximised, i.e., the output video quality. Subsequently, the network transforms the input data through a series of neurons in the hidden layer. The output layer comprises four neurons corresponding to each available video stream from \SMEC to \SMEXPELTWEN (different error resilience techniques). @@ -284,14 +289,15 @@ However, different network topologies with different number of neurons in each l \subsubsection{Training and validation} %As mentioned before a total of \NumCases test cases are used for the evaluation of the proposed method. -To effectively train and validate the \ac{nn}, the data set is randomly partitioned into three different groups: (i) 55\% of the test cases are used for training, (ii) 15\% for validation and (iii) 30\% for testing the \ac{nn} performance. -The \ac{nn} supervised training uses 591 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993}. -The remaining 254 conditions are used to evaluate the accuracy of the proposed method after training with different conditions. From this test one video sequence was left out, to be used in the performance evaluation of the proposed method. +To effectively train and validate the \ac{nn}, firstly the cases corresponding to three video sequence were left out, to be used in the performance evaluation of the proposed method. +The remaining data set was randomly partitioned into three different groups: (i) 65\% of the test cases are used for training, (ii) 15\% for validation and (iii) 20\% for testing the \ac{nn} performance. +The \ac{nn} supervised training uses 557 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993}. +Then, 129 test conditions were used for \ac{nn} validation. +Finally, 172 conditions are used to evaluate the accuracy of the proposed method after training with different conditions. - -Table~\ref{Table6:ConfusionTraining} shows the confusion matrix of the \ac{nn} training phase. -In this matrix each column corresponds to the target result, i.e., the cases where the coded streams, \SMEC to \SMEXPELTWEN, would lead to the highest decoded video quality. -Similarly, each row corresponds to the \ac{nn} selected by the trained networks. +Tables~\ref{Table6:ConfusionTraining}, \ref{Table6:ConfusionVal} and \ref{Table6:ConfusionTest} show the confusion matrices obtained for training, validation and test sets, respectively. +In these matrix each column corresponds to the target result, i.e., the cases where the coded streams, \SMEC to \SMEXPELTWEN, would lead to the highest decoded video quality. +Similarly, each row corresponds to the \ac{nn} selected by the trained network. Thus, those cases where the horizontal and vertical labels match, indicate that the network produced the correct output. The remaining positions correspond to the erroneous outputs. The last line shows the accuracy for each target output. @@ -299,7 +305,7 @@ These results correspond to the dataset used to train the \ac{nn}. \begin{table}[tb] \centering - \caption{Confusion matrix of the training and validation test cases.} + \caption{Confusion matrix of the training set.} \label{Table6:ConfusionTraining} \TABLESVSPACE \renewcommand{\arraystretch}{1.2} @@ -307,21 +313,54 @@ These results correspond to the dataset used to train the \ac{nn}. & & \multicolumn{4}{c}{Target} \\ \multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} & & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN \\ \hline - & \SMEC & 122 & 17 & 3 & 0 \\ - \cline{2-6} & \SMEXP & 13 & 80 & 19 & 2 \\ - \cline{2-6} & \SMEXPELTEN & 0 & 12 & 96 & 9 \\ - \cline{2-6} & \SMEXPELTWEN & 0 & 4 & 11 & 202 \\ - \hline - \multicolumn{2}{x{1.7cm}|}{Accuracy} & 90.3\% & 70.8\% & 74.4\% & 94.8\% \\ + & \SMEC & 132 & 32 & 6 & 6 \\ + \cline{2-6} & \SMEXP & 16 & 57 & 13 & 0 \\ + \cline{2-6} & \SMEXPELTEN & 4 & 10 & 69 & 18 \\ + \cline{2-6} & \SMEXPELTWEN & 9 & 10 & 17 & 158 \\ + \hline + \multicolumn{2}{x{1.7cm}|}{Accuracy} & 82.0\% & 52.3\% & 65.7\% & 86.8\% \\ \end{tabular} \end{table} Results in Table~\ref{Table6:ConfusionTraining} shows that streams \SMEC (i.e., without robust coding techniques) and \SMEXPELTWEN (i.e., using optimal \ac{ec} with 20\% of bitrate in the \ac{el}) lead to the highest quality in the majority of the test cases (are more often selected). -The \ac{nn} is able to achieve a high accuracy for \SMEC and \SMEXPELTWEN streams, correctly selecting the output stream in more than 90\% of cases. Regarding the remaining robust streams (i.e., \SMEXP and \SMEXPELTEN), an acceptable accuracy of 70.8\% and 74.4\% is achieved, respectively. This reveals that the \ac{nn} used in the proposed method is able to predict the coded stream which leads to the highest video quality across different conditions with acceptable accuracy. +The \ac{nn} is able to achieve a high accuracy for \SMEC and \SMEXPELTWEN streams, correctly selecting the output stream in more than 80\% of the cases. Regarding the remaining robust streams (i.e., \SMEXP and \SMEXPELTEN), an acceptable accuracy of 52.2\% and 60.6\% is achieved, respectively. +This reveals that the \ac{nn} used in the proposed method is able to predict the coded stream which leads to the highest video quality across different conditions with acceptable accuracy. +The results from the validation set, shown in Table~\ref{Table6:ConfusionVal}, reveal that the \ac{nn} training did not over-fit, since it shows similar accuracy as the results shown in Table~\ref{Table6:ConfusionTraining}. + + \begin{table}[tb] \centering - \caption{Confusion matrix of the test cases not used for testing.} + \caption{Confusion matrix of the validation set.} + \label{Table6:ConfusionVal} + \TABLESVSPACE + \renewcommand{\arraystretch}{1.2} + \begin{tabular}{x{.5cm}x{1cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}} + & & \multicolumn{4}{c}{Target} \\ + \multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} + & & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN \\ \hline + & \SMEC & 28 & 4 & 1 & 2 \\ + \cline{2-6} & \SMEXP & 4 & 12 & 6 & 0 \\ + \cline{2-6} & \SMEXPELTEN & 1 & 7 & 20 & 5 \\ + \cline{2-6} & \SMEXPELTWEN & 1 & 0 & 6 & 32 \\ + \hline + \multicolumn{2}{x{1.7cm}|}{Accuracy} & 82.4\% & 52.2\% & 60.6\% & 82.1\% \\ + %\bottomrule + \end{tabular} +\end{table} + + + +To further evaluate the performance of the \ac{nn}, the confusion matrix of the data set used for testing (test set) is shown in the Table~\ref{Table6:ConfusionTest}. +This table shows the same results as Table~\ref{Table6:ConfusionTraining}, but for different test cases, which were not used in the training phase. +% Therefore, the trained network is not aware of the test cases used for validation. +Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 50\% with a maximum value of 84.9\% when the \SMEXPELTWEN is the target stream. +Although a lower accuracy is obtained using the test set, the overall accuracy is still acceptable, since it allows to select the optimal robust coding technique (i.e., \SMEC to \SMEXPELTWEN) for the majority of the conditions. +In summary, the results presented in this sub-section reveal that the input parameters used in the proposed \ac{nn} are correlated with the decoded video quality, as they can effectively be used to perform data categorisation used in the selection of the optimal error robustness coding technique for the \ac{hevc} standard. + +\begin{table}[tb] + \centering + \caption{Confusion matrix of the test set.} \label{Table6:ConfusionTest} \TABLESVSPACE \renewcommand{\arraystretch}{1.2} @@ -329,31 +368,24 @@ The \ac{nn} is able to achieve a high accuracy for \SMEC and \SMEXPELTWEN stream & & \multicolumn{4}{c}{Target} \\ \multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} & & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN \\ \hline - & \SMEC & 56 & 7 & 2 & 0 \\ - \cline{2-6} & \SMEXP & 10 & 28 & 6 & 2 \\ - \cline{2-6} & \SMEXPELTEN & 1 & 5 & 31 & 8 \\ - \cline{2-6} & \SMEXPELTWEN & 0 & 5 & 7 & 86 \\ + & \SMEC & 41 & 10 & 5 & 1 \\ + \cline{2-6} & \SMEXP & 5 & 14 & 2 & 0 \\ + \cline{2-6} & \SMEXPELTEN & 2 & 0 & 23 & 7 \\ + \cline{2-6} & \SMEXPELTWEN & 3 & 4 & 10 & 45 \\ \hline - \multicolumn{2}{x{1.7cm}|}{Accuracy} & 83.6\% & 62.2\% & 67.4\% & 89.6\% \\ + \multicolumn{2}{x{1.7cm}|}{Accuracy} & 80.4\% & 50.0\% & 57.5\% & 84.9\% \\ %\bottomrule \end{tabular} \end{table} -To further evaluate the performance of the \ac{nn}, the confusion matrix of the data set used for testing is shown in the Table~\ref{Table6:ConfusionTest}. -This table shows the same results as Table~\ref{Table6:ConfusionTraining}, but for different test cases, which were not used in the training phase. -% Therefore, the trained network is not aware of the test cases used for validation. -Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 62.2\% with a maximum value of 89.6\% when the \SMEXPELTWEN is the target stream. Although a lower accuracy is obtained using the validation test cases, the overall accuracy is still acceptable, since it allows to select the optimal robust coding technique (i.e., \SMEC to \SMEXPELTWEN) for the majority of the conditions. -In summary, the results presented in this sub-section reveal that the input parameters used in the proposed \ac{nn} are correlated with the decoded video quality, as they can effectively be used to perform data categorisation used in the selection of the optimal error robustness coding technique for the \ac{hevc} standard. - \section{Performance evaluation} \label{Section6:ExperimentalResults} - In this section the performance of the proposed method is evaluated and discussed. The overall goal of the proposed method is to achieve improved quality using a combination of different robust coding techniques adaptively selected using a feed-forward \ac{nn}. The error resilience techniques comprising the proposed method are used in the context of the \ac{hevc} standard and compared with the reference implementation (\REF). -The quality obtained with the proposed optimal error robustness scheme (\PROP) is compared against the reference \ac{hevc} (\REF) and the methods used to generate the streams \SMEC to \SMEXPELTWEN. +The quality obtained with the proposed optimal error robustness scheme (\PROP) is compared against the reference \ac{hevc} (\REF) and the individual methods used to generate the streams \SMEC to \SMEXPELTWEN. \begin{figure}[!t] \centering @@ -366,11 +398,11 @@ The quality obtained with the proposed optimal error robustness scheme (\PROP) i Figure~\ref{Figure6:ProposedQuality} shows the quality obtained for the proposed method and the \REF case for the main target bitrate shown in Table~\ref{TableTestSequences}. Moreover, the quality obtained with the target stream (\BEST) is also shown. One should note that the \BEST case corresponds to the optimal selection, which can only be performed after quality evaluation at the decoder, therefore it is not suitable for real time applications. It is only used to establish the optimal case for comparison purposes. -The results in the figure allows to evaluate the impact in the decoded quality of the selection used in the proposed method and compare it with the optimal case. +The results in the figure allow the comparison between three relevant cases: proposed stream selection method, the optimal case and the reference method. As shown in Figure~\ref{Figure6:ProposedQuality}, the proposed method clearly outperforms the reference \ac{hevc} for all \acs{plr}. Moreover, in the error-free case (\ac{plr}=0), it leads to imperceptible quality reduction, since the proposed approach is able to select the optimal stream, thus disabling any robust coding technique when the transmission network does not have packet losses. -In case of data loss, the results show that the proposed method is able to match the quality obtained by the \BEST case in most cases, revealing that the proposed method is able to efficiently select the optimal stream for transmission. +In case of data loss, the results show that the proposed method is able to match the quality obtained by the \BEST case in most cases, revealing that the \ac{nn} is able to efficiently select the optimal stream for transmission. \begin{table}[tb] \centering @@ -382,59 +414,50 @@ In case of data loss, the results show that the proposed method is able to match \toprule \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\ & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN & \PROP \\ \hline -Basketball Drill & +1.61 & +2.66 & +2.48 & +2.44 & +2.75 \\ \hline -Book Arrival & +0.96 & +1.66 & +1.43 & +1.59 & +1.74 \\ \hline -BQSquare & +0.92 & +0.96 & +0.80 & +0.62 & +1.09 \\ \hline -Cactus & +1.28 & +2.04 & +1.89 & +1.96 & +2.09 \\ \hline -Four People & +0.28 & -0.23 & -0.88 & -0.99 & +0.25 \\ \hline -Kendo & +0.81 & +0.78 & +0.20 & +0.61 & +1.02 \\ \hline -Kimono & +1.65 & +3.04 & +3.26 & +3.27 & +3.41 \\ \hline -Kristen Sara & +0.36 & +0.15 & -0.33 & -0.50 & +0.38 \\ \hline -Park Scene & +1.64 & +2.18 & +1.88 & +2.02 & +2.25 \\ \hline -Party Scene & +0.21 & +0.20 & -0.07 & -0.24 & +0.29 \\ \hline -People on Street & +1.00 & +1.27 & +0.91 & +1.49 & +1.63 \\ \hline -Race Horses & +1.93 & +3.89 & +4.06 & +4.30 & +4.39 \\ \hline -Tennis & +1.15 & +2.24 & +2.17 & +3.12 & +3.18 \\ \hline -\textbf{Average} & \textbf{+1.06} & \textbf{+1.60} & \textbf{+1.37} & \textbf{+1.51} & \textbf{+1.88} \\ +Basketball Drill & +1.68 & +2.13 & +2.81 & +2.62 & +2.81 \\ \hline +Book Arrival & +1.00 & +1.51 & +1.79 & +1.74 & +1.87 \\ \hline +BQSquare & +0.98 & +1.14 & +1.06 & +0.73 & +1.20 \\ \hline +Cactus & +1.22 & +1.36 & +1.95 & +1.85 & +1.88 \\ \hline +Four People & +0.30 & -1.19 & -0.12 & -0.61 & +0.28 \\ \hline +Kimono & +1.55 & +2.68 & +2.86 & +3.06 & +3.07 \\ \hline +Kristen Sara & +0.38 & -0.52 & +0.20 & -0.38 & +0.30 \\ \hline +Park Scene & +1.65 & +1.73 & +2.24 & +2.10 & +1.83 \\ \hline +Party Scene & +0.21 & +0.11 & +0.17 & +0.44 & +0.30 \\ \hline +People on Street & +1.01 & +1.13 & +1.38 & +1.62 & +1.76 \\ \hline +Tennis & +1.18 & +2.11 & +2.43 & +3.28 & +3.28 \\ \hline +\textbf{Average} & \textbf{+1.01} & \textbf{+1.11} & \textbf{+1.52} & \textbf{+1.50} & \textbf{+1.69} \\ \bottomrule \end{tabular} \begin{tikzpicture}[overlay] -\draw[red, line width=1.2pt] (1.8,5.2) ellipse (3.5cm and 0.3cm); -\draw[red, line width=1.2pt] (3,3.6) ellipse (2.3cm and 0.3cm); -\draw[blue, line width=1.2pt] (6.7,1.45) ellipse (.85cm and 0.25cm); -\draw[blue, line width=1.2pt] (6.7,4.10) ellipse (.85cm and 0.25cm); +\draw[red, line width=1.2pt] (1.8,4.1) ellipse (3.5cm and 0.3cm); +\draw[red, line width=1.2pt] (1.8,3.05) ellipse (3.5cm and 0.3cm); +\draw[blue, line width=1.2pt] (6.7,.95) ellipse (.85cm and 0.25cm); +\draw[blue, line width=1.2pt] (6.7,3.6) ellipse (.85cm and 0.25cm); \end{tikzpicture} \end{table} -Subsequently, a comparison between the different methods is performed, in order to evaluate the quality gains obtained with the proposed method in comparison with the different techniques. +Subsequently, a comparison between the different methods is performed, in order to evaluate the quality gains obtained with the proposed method in comparison with each of the different techniques. %for every test case. -Table~\ref{Table6:AverageGains} shows the average quality gains in comparison with the reference \ac{hevc} obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams \SMEC to \SMEXPELTWEN. The average quality gains are measured across all test cases, i.e., different bitrates and different loss ratios. +Table~\ref{Table6:AverageGains} shows the average quality gains in comparison with the reference \ac{hevc} for the test set. +These were obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams \SMEC to \SMEXPELTWEN. The average quality gains are measured across all test cases, i.e., test set with different bitrates and different packet loss ratios. -Results in Table~\ref{Table6:AverageGains} shows that the robust coded streams \SMEC to \SMEXPELTWEN are able to achieve acceptable average quality gains up to 1.51 dB (see results for \SMEXP). -However, since the implementation of robust coding techniques decreases the coding efficiency, it also leads to lower quality than the reference \ac{hevc}. This is noticeable in the results obtained for the Four People sequence in the streams \SMEXP to \SMEXPELTWEN and for the Kristen and Sara sequence in the streams \SMEXPELTEN and \SMEXPELTWEN (see red ellipses). +Results in Table~\ref{Table6:AverageGains} show that the robust coded streams \SMEC to \SMEXPELTWEN are able to achieve acceptable average quality gains up to 1.50 dB (see results for \SMEXPELTEN). +However, since the implementation of robust coding techniques decreases the coding efficiency, it also leads to lower quality than the reference \ac{hevc}. +This is noticeable in the results obtained for the Four People sequence in the streams \SMEXP to \SMEXPELTWEN and for the Kristen and Sara sequence in the streams \SMEXPELTEN and \SMEXPELTWEN (see red ellipses). The overall performance of such methods reveals that they are not suitable for transmission conditions with low \acs{plr}, e.g., lower than 3\% (higher number of conditions have \acs{plr} lower than 3\%). -On the contrary, the proposed method is always able to outperform the reference \ac{hevc} and achieve quality gains of at least 0.25 dB for the Four People sequence. -Moreover, for the Kimono and Race Horses sequences, it is able to achieve the highest quality gains, of up 3.41 dB and 4.39 dB, respectively (see blue ellipses). Summarising, by adaptively selecting the video stream, and consequently the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness, achieving an average quality gain across different bitrates and \acs{plr} of 1.88 dB. +On the contrary, the proposed method is always able to outperform the reference \ac{hevc} and achieve quality gains of at least 0.28 dB for the Four People sequence. +For the Kimono and Tennis sequences, it is able to achieve the highest quality gains, of up 3.07 dB and 3.28 dB, respectively (see blue ellipses). +Summarising, by adaptively selecting the video stream, and implicitly the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness, achieving an average quality gain across different bitrates and \acs{plr} of 1.88 dB. -Finally, the performance evaluation of the proposed method is evaluated for a video sequence, which was not fed into the \ac{nn} training phase. This study aims at finding the accuracy of the optimal stream selection in case of a new video content. For this purpose the Basketball Drive sequence was randomly selected, and not used in previous simulations. -Table~\ref{Table6:AverageGainBasketBallDrive} shows the average quality results obtained with this sequence, similarly to Table~\ref{Table6:AverageGains}. -These results confirm that the proposed method is able to select the optimal robust stream to be used for transmission, even when new video content is transmitted. The proposed method achieves a quality gain of 3.55 dB in comparison with the reference \ac{hevc}, outperforming the quality achieved with the stream \SMEXPELTWEN (i.e., 3.45 dB). +Finally, the performance of the proposed method is evaluated for video sequences, which was not fed into the \ac{nn} training phase. +This study aims at finding the accuracy of the optimal stream selection in case of a new video content. +For this purpose the Basketball Drive sequence was randomly selected, and not used in previous simulations. +Table~\ref{Table6:AverageGainBasketBallDrive} shows the average quality results obtained with these sequences, similarly to Table~\ref{Table6:AverageGains}. +These results also confirm that the proposed method is able to select the optimal robust stream to be used for transmission, even when new video content is transmitted. +The proposed method achieves a quality gain up to 4.46 dB in comparison with the reference \ac{hevc}, outperforming the quality achieved with the stream \SMEXPELTWEN (i.e., 4.19 dB). This confirms the advantage of the proposed method to adaptively select among different streams with different levels of error protections. -\begin{table}[tb] - \centering - \caption{Average quality gain ($\Delta$PSNR) in comparison with the reference \ac{hevc} for the Basketball Drive sequence.} - \label{Table6:AverageGainBasketBallDrive} - \TABLESVSPACE - \newcommand{\colwidth}{1.7cm} - \begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} - \toprule - \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\ - & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN & \PROP \\ \hline - Basketball Drive & +2.10 & +2.68 & +3.36 & +3.45 & +3.55 \\ - \bottomrule - \end{tabular} -\end{table} + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% @@ -442,7 +465,24 @@ This confirms the advantage of the proposed method to adaptively select among di \section{Summary} \label{Section6:Summary} -In this chapter an adaptive error robustness method is proposed, based on reference frame selection and \ac{ec}-aware optimisation. By adaptively selecting a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the reference techniques. +\begin{table}[tb] + \centering + \caption{Average quality gain ($\Delta$PSNR) in comparison with the reference \ac{hevc} for video sequences not used in the NN training.} + \label{Table6:AverageGainBasketBallDrive} + \TABLESVSPACE + \newcommand{\colwidth}{1.7cm} + \begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} + \toprule + \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\ + & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN & \PROP \\ \hline + Basketball Drive & +2.10 & +2.68 & +3.36 & +3.45 & +3.55 \\ \hline + Kendo & +1.92 & +3.24 & +3.94 & +4.19 & +4.46 \\ \hline + Race Horses & +0.85 & +1.02 & +0.88 & +0.75 & +1.06 \\ + \bottomrule + \end{tabular} +\end{table} + +In this chapter an adaptive error robustness method is proposed, based on reference frame selection and \ac{ec}-aware optimisation. By adaptively selecting a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the individual reference techniques. %Moreover, the proposed method is able to outperform those cases where a single error resilience technique is used. The overall results demonstrate that the proposed method is able to increase the reconstruction video quality in case of errors. Also, by adapting the transmitted stream to the network conditions, a better trade-off between coding efficiency and error robustness can be achieved. diff --git a/050_Conclusions/Conclusion.tex b/050_Conclusions/Conclusion.tex index 2d3940e..9298f54 100755 --- a/050_Conclusions/Conclusion.tex +++ b/050_Conclusions/Conclusion.tex @@ -1,79 +1,83 @@ -\chapter{Conclusion and future work \VERSION{0}} +\chapter{Conclusion and future work \VERSION{3}} \label{Chapter:Conclusions} \glsresetall +\glsunset{hevc} \glsunset{h264} This thesis investigated new algorithms and techniques for efficient error robustness in highly efficiency video coders. -The importance of these techniques is justified by the spread of digital video contents, including high resolution formats, multi-view and 360-degree video formats, which significantly the network bandwidth requirements. Consequently, errors and data loss may occur along the transmission system. -Moreover, in order to meet the network bandwidth constrains video coding standard have been improved, by including advanced predictions techniques, which results in increased bitstream dependencies. -This increase the negative impact of errors, leading to higher quality degradation. -To overcome this problem, in this thesis different methods were proposed, addressing the problem of robust coding and efficient \ac{ec}. +The importance of this research work is justified by the widespread of digital video content, including high resolution formats, multi-view and 360-degree video formats, which significantly increase the network bandwidth requirements. +Consequently, errors and data loss may occur along the transmission system, despite the fact that video coding standards have been improved, in order to meet the network bandwidth constrains. +The use of advanced predictions techniques, results in increased bitstream dependencies with negative impact in the effect of errors, leading to higher quality degradation. +In this thesis different methods were proposed to address this problem, by investigating robust coding and efficient \ac{ec}. %Subsequently, for each chapter of this thesis, the main conclusions are presented. -Firstly, an introduction of the main motivation for this research work is presented, enumerating its main objectives and contributions. -Then, the main functional blocks of the state-of-the-art \ac{hevc} are described. +Firstly, an introduction of the main motivation for this research work was presented, enumerating its main objectives and contributions. +Then, the main functional blocks of the state-of-the-art \ac{hevc} were described. This description covered the high-level features, such as the new picture types and the slice partitioning schemes, the new block partitioning scheme, which allow multi-level hierarchical structures with size-independent representation, the main predictions techniques, including the newly introduced advanced motion vector prediction technique. -This aimed at introducing the novel techniques in \ac{hevc}, which may have a relevant impact on the error robustness of the coded streams. -The main algorithms proposed in the past to increase the error robustness of video streams were presented, describing techniques based on error resilience \ac{rd} optimisation, redundant based approaches and \ac{ec}-based methods. -It was concluded that, although the previous works provide efficient techniques to reduce the bitstream dependencies and ease the error recovery, the \ac{hevc} standard introduces new problems that could not be fully addressed with existing techniques. -Such problems range from new \ac{mv} coding techniques to the lack of robust coding techniques, such as \ac{fmo} or data partitioning. -Moreover, there was also open issues regarding robust \ac{ec}-based encoding, as existing method do not fully exploit the use of different methods with different levels of complexity, especially the use of advanced \ac{ec} techniques. +This provided the necessary knowledge about the novel techniques in \ac{hevc}, which may have a relevant impact on the error robustness of the coded streams. +The main algorithms proposed in the past to increase the error robustness of video streams were also presented, describing techniques based on error resilience \ac{rd} optimisation, redundant based approaches and \ac{ec}-based methods. +After a simulation study it was concluded that, although the previous works provide efficient techniques to reduce the bitstream dependencies and ease the error recovery, vulnerabilities of the \ac{hevc} standard could not be fully addressed with existing techniques. +The main problems range from new \ac{mv} coding techniques to the lack of robust coding mechanisms, such as \ac{fmo} or data partitioning. +Moreover, there were also open issues regarding robust \ac{ec}-based encoding, as existing methods do not fully exploit the use of different techniques with different levels of complexity, especially the use of advanced \ac{ec}-based approaches. -To investigate new techniques for robust video coding an evaluation study was firstly carried out, which aimed at finding the influence of the new coding tools on the error resilience of the \ac{hevc} streams. -Existing studies only performed generic comparisons between \ac{hevc} and previous standards (e.g., \ac{h264}). -This study extended existing ones by considering different coding configurations, which allowed to evaluate the impact of new coding tools. The study revealed that some features, such as increased block size, does not have significant impact of the error robustness performance. +To investigate new techniques for robust video coding an evaluation study was firstly carried out, with the aim of finding the influence of the new coding tools on the error resilience of the \ac{hevc} streams. +Existing studies were limited to generic comparisons between \ac{hevc} and previous standards (e.g., \ac{h264}). +This study extended existing ones by considering different coding configurations, which allowed to evaluate the impact of new coding tools. +The study revealed that some features, such as increased block size, do not have significant impact on the error robustness performance. However, slice partitioning and intra-refreshing have a noticeable impact on both, coding efficiency and error robustness. -Therefore, a small compromise in coding efficiency should be reasoned, in order to achieve an acceptable quality in case of errors. -Finally, one can also conclude that the use of temporal predictions in \ac{mv} coding significantly increases temporal dependencies, leading to increased quality degradation compromising the performance of an \ac{hevc}-based video transmission system. +This led to the insight that a small compromise in coding efficiency should be reasoned, in order to achieve an acceptable quality in case of errors. +Finally, one could also conclude that the use of temporal predictions in \ac{mv} coding significantly increases temporal dependencies, leading to higher quality degradation and compromising the performance of \ac{hevc}-based video transmission systems. -Based on the above evidences, a new \ac{mv} coding scheme was used in \ac{hevc}, aiming at reducing the temporal dependencies. +Based on the above evidences, a new \ac{mv} coding scheme was devised, aiming at reducing the temporal dependencies. In this scheme the use of the temporal \ac{mv} candidate is selectively used in some \acs{pu}, preventing temporal dependencies across several frames. -As this method significantly reduces the temporal dependencies, the number of mismatched \acs{mv} is reduced, consequently reducing the error propagation. -Moreover, as the temporal candidate is not fully disabled the impact on coding efficiency is reduced. -Experimental results showed a clear improvement on decoded video quality, across different video content, with a small bitrate increase. +As this method significantly reduces the temporal dependencies, the number of mismatched \acs{mv} is decreases. +Moreover, as the temporal candidate is not fully disabled the impact on coding efficiency was shown to be low. +Experimental results demonstrated that a clear improvement on decoded video quality is achieved, across different video content, with a small bitrate increase. - -The reference frame distribution in highly efficient video coders was investigated, as it may increase the number of mismatch predictions in case of erroneous reference frames. -Since the reference frame usage is consistently unbalanced, a new method based on constrained \ac{rd} optimisation was proposed to equally distribute the use of multiple reference frames across neighbouring \acs{pu}. -An extension using selective \ac{mv} redundancies was also investigated, in order to recover from erroneous motion predictions. +Another investigation carried out in this thesis was focused on the reference frame distribution in highly efficient video coders. +Since in general, the reference frame usage is unbalanced, a new method based on constrained \ac{rd} optimisation was proposed to equally distribute the use of multiple reference frames across neighbouring \acs{pu}. +An extension \HERE using selective \ac{mv} redundancies was also investigated, in order to recover from erroneous motion predictions. This resulted in a two-fold approach to reduce error propagation. -As part of the proposed method can be applied at the streaming-stage, using a low complexity parser, this method can be used to adapt existing streams to error-prone transmission channels. -The proposed method was compared against existing techniques, and the simulation results revealed a consistently superior performance of using the two-fold approach, making it suitable for video services where packet loss cannot be negligible. +As part of the proposed method can be applied at the streaming-stage, using a low complexity parser, this method can be used to adapt pre-encoded streams to error-prone transmission channels. +The proposed method was compared against existing techniques, and the simulation results revealed a consistently superior performance of using the two-fold approach for different packet loss ratios. -To address the problem of robust video decoding and increase the quality of the reconstructed frames, \ac{ec}-aware resilience techniques were investigated. -To extend existing techniques, a saliency-based \ac{ec} optimisation was proposed to optimally select the \ac{ec} which should be applied at the decoder in case of frame loss. This information is transmitted to the decoder as side-information. -Besides a saliency estimation, a coefficient based on the temporal duration of the error propagation was also considered, in order to keep the signalling overhead affordable. Finally, a small bit allocation was used to transmit an \ac{el}, which was used to improve the reconstruction quality, in those regions where the optimal \ac{ec} was not accurate. -Experimental results revealed that using different \ac{ec} algorithms increase the performance of the \ac{ec}-aware coding scheme and achieve superior performance when compared with existing methods. +To address the problem of robust video decoding and increase the quality of the reconstructed frames, \ac{ec}-aware resilience techniques were also investigated. +A saliency-based \ac{ec} optimisation was proposed to optimally select the \ac{ec} method that should be applied at the decoder in case of frame loss. +This information is transmitted to the decoder as side-information. +To keep the signalling overhead affordable, saliency estimation and a parameter based on the temporal duration of the error propagation +Then, a limited amount of overhead was used to encode a residual information in an \ac{el}, which is used to improve the reconstruction quality, in those regions where the optimal \ac{ec} was not accurate. +Experimental results revealed that using different \ac{ec} algorithms contributes to increase the performance of the \ac{ec}-aware coding scheme, achieving superior performance when compared with existing methods. -Finally, the problem of adaptive robust coding was investigated, in order to achieve superior quality with the error robustness techniques proposed in this research work. -Preliminary results showed that the robust coding technique which obtains the higher quality depends on different factors, such as video content, bitrate and network conditions. -Therefore, an adaptive scheme based on six input parameters and a pre-trained \ac{nn} was proposed, to predicts the video stream that is able to achieve the highest decoded quality. -Results revealed that using the proposed scheme, an efficient trade-off between coding efficiency and error resilience is achieved for the majority of test conditions. -Using this architecture, the problem of robust video coding and decoding is addressed using a combination of different methods, which leads to a superior performance when compared with individual use of a single technique. +Finally, the problem of adaptive robust coding was investigated, in order to achieve superior quality when different error robustness techniques are available and dynamic conditions are taken into account. +Simulation results showed that the robust coding technique which obtains the higher quality depends on different factors, such as video content, bitrate and network conditions. +Therefore, an adaptive scheme based on six input parameters and a pre-trained \ac{nn} was proposed, to predicts the video stream that is able to achieve the highest decoded quality, under transmission errors. +Results revealed that using the proposed adaptive scheme, an efficient trade-off between coding efficiency and error resilience is achieved for the majority of test conditions. +Using this architecture, the problem of robust video coding and decoding was addressed using a combination of different methods, which leads to a superior performance in comparison with individual techniques. -As a final remark, one may conclude that the use of novel methods, proposed in the context of this research work, are able to increase the robustness to transmission errors of the state-of-the-art video coding standard. -This was jointly achieved by achieving implementing robust video coding techniques with efficient error concealment algorithms optimised at the encoder. +As a final remark, one may conclude that the use of the novel methods proposed in this thesis consistently contribute to increase the robustness of video communications using state-of-art high efficiency video encoding tools are error prone networks. +The best performance is achieved by combining robust video coding techniques with efficient error concealment algorithms optimised at the encoder, using an optimised dynamic approach. \subsection*{Future work} -In this research work different methods were proposed to increase the error robustness of video streams and to enhance the quality achieved by \ac{ec} methods are the decoder. -In the context of this thesis there are also some suggestions for future research directions which can be used to strength the overall performance. +In the context of this thesis there are some open issues for future research directions which can be investigated to further strength the overall performance. -In the context of robust coding techniques, one can combine existing error resilience techniques, e.g., intra-refreshing constrains, in the Lagrangian optimisation used in the reference picture selection proposed in Chapter~\ref{Chapter:HEVCErrorResilience}. -This way it would be possible to further prevent error propagation by introducing extra refresh points. -Moreover, one can also use more advanced constrains, such as, end-to-end based optimisations, to achieve a more efficient distribution of the use of reference frames. +In the context of robust coding, one can further combine existing error resilience techniques, e.g., intra-refreshing constrains, in the Lagrangian optimisation used in the reference picture selection proposed in Chapter~\ref{Chapter:HEVCErrorResilience}. +This would possibly allow stronger limitation of error propagation by introducing extra refresh points. +Moreover, one can also use more advanced constrains, such as, end-to-end based optimisations, to devise more efficient methods for distribution of the use of reference frames. -Regarding the \ac{ec}-aware method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} there are also some suggestions of future research directions. Firstly, as the proposed does not constrain the \ac{ec} candidate, one can use more advanced algorithms, in order to improve the reconstruction accuracy. This can be adopted either by replacing the existing methods, but also extending beyond four candidates. -Alternatively, to reduce the computational complexity overhead at the encoder, instead of measuring the \ac{rd} cost for each method, pre-trained classification algorithms could be used to predict the optimal \ac{ec} mode for each block. -As the complexity of the decoder is also a relevant issue, due to the increasing of mobile and battery-based devices, the optimal \ac{ec} selection should also take into account the end devices, thus penalising high complexity algorithms (e.g., algorithms requiring an estimated motion field) in case of constraining devices. -The performance of the adaptive selection of the robust coding technique could also be improved by combining different classification algorithms, based on more complex neural network architectures. +Regarding the \ac{ec}-aware method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} there are also some possible improvements to be addressed in future research. +Firstly, more advanced algorithms ca be developed on a \ac{ctu} basis, in order to improve the reconstruction accuracy. +This can be implemented either by replacing the existing methods, or by extending beyond the four candidates that were used in this work. +Alternatively, to reduce the computational complexity overhead at the encoder, instead of measuring the \ac{rd} cost for each method, pre-trained classification algorithms can be investigated to predict the optimal \ac{ec} mode for each block. +As the complexity of the decoder is also a relevant issue, due to the increasing of mobile and battery-based devices, the optimal \ac{ec} selection should also take into account the end devices, thus penalising high complexity algorithms (e.g., algorithms requiring an estimated motion field) in case of constrained devices. +The performance of the adaptive selection of the robust coding technique can also be improved by combining different classification algorithms, based on more complex neural network architectures. -The error robustness schemes proposed in this research work could be extended to cope with robust transmission of multi-view video, where error propagate in the temporal and inter-layer domains. -Finally, as a new coding standard is currently being developed, targeting the deployment of higher-quality video services and emerging applications such as 360-degree omnidirectional immersive multimedia and high-dynamic-range video, it is emergent to study the impact of errors in such video applications and propose efficient techniques to provide high quality video to the end users. +Furthermore, the error robustness schemes proposed in this research work can be extended to cope with robust transmission of multi-view video, where errors propagate in the temporal and inter-layer domains. +Finally, as a new coding standard is currently being developed, targeting the deployment of higher-quality video services and emerging applications such as 360-degree multimedia and high-dynamic-range video, it is also an open direction for research the study of the impact of errors in such video applications and the development of novel efficient techniques to guarantee high quality visual experiences, to users of such emerging video formats. diff --git a/060_Appendix/Papers.tex b/060_Appendix/Papers.tex index e527b51..1e9b85c 100755 --- a/060_Appendix/Papers.tex +++ b/060_Appendix/Papers.tex @@ -9,9 +9,9 @@ Submitted papers are also presented. \begin{description} \item[\namedlabel{Journal:TwoStage}{J1}] -J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, ``A two-stage approach for robust HEVC coding and streaming,'' \emph{IEEE Transactions on Circuits and Systems for Video Technology}, vol. 28, no. 8, pp. 1960-1973, Aug. 2018 (Apr. 2017). +J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, ``A two-stage approach for robust HEVC coding and streaming,'' \emph{IEEE Transactions on Circuits and Systems for Video Technology}, vol. 28, no. 8, pp. 1960-1973, Aug. 2018 (available since Apr. 2017). \item[\namedlabel{Journal:ConcAware}{J2}] -J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, ``Error Concealment-Aware Encoding for Robust Video Transmission,'' \emph{IEEE Transactions on Broadcasting} (Aug. 2018). +J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, ``Error Concealment-Aware Encoding for Robust Video Transmission,'' \emph{IEEE Transactions on Broadcasting} (available since Aug. 2018). \end{description} \section{Conference publications} diff --git a/carreira_phd_thesis.tex b/carreira_phd_thesis.tex index 67572dc..ea58b1a 100644 --- a/carreira_phd_thesis.tex +++ b/carreira_phd_thesis.tex @@ -19,7 +19,7 @@ \newcommand{\HIGHLIGHT}[1]{\textcolor{red}{#1}} \newcommand{\HERE}{\textcolor{red}{CHECK HERE!}} \newcommand{\VERSION}[1]{\textcolor{red}{(r#1)}} - +% %\includeonly{000_Preamble/abstract} %\includeonly{010_Introduction/Introduction} %\includeonly{021_VideoCoding/VideoCodingReview} @@ -30,6 +30,8 @@ %\includeonly{043_EndToEndRobustSystem/EndToEndRobustSystem} %\includeonly{050_Conclusions/Conclusion} +\renewcommand{\baselinestretch}{1.3} + \begin{document} \title{Error Resilience and Concealment Techniques \\ for High Efficiency Video Coding} @@ -45,23 +47,20 @@ \titlefigs{\includegraphics[width=10cm]{lborolodonlogo.png}} -% make the title area -%\listoftodos - \maketitle -\frontmatter -% Certificate of Originality -%\clearpage -\includepdf[pages=1,pagecommand=]{000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf} -%\cleardoublepage -\includepdf[pages=2, pagecommand=]{000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf} +% Certificate of Originality +\cleardoublepage +\includepdf[pages={1},fitpaper=true,openright=false,pagecommand={}]{000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf} +\includepdf[pages={2},fitpaper=true,openright=false,pagecommand={}]{000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf} + + +\frontmatter \input{000_Preamble/dedic} \input{000_Preamble/thanks} \input{000_Preamble/abstract} -%\dominitoc \glsunsetall \tableofcontents \listoffigures @@ -69,6 +68,8 @@ \printglossary[title=List of Abbreviations] \glsresetall +\glsunset{ip} + \mainmatter \include{010_Introduction/Introduction} \include{021_VideoCoding/VideoCodingReview} @@ -81,10 +82,9 @@ \appendix \include{060_Appendix/Sequences} \include{060_Appendix/Papers} -%\def\extraresults{0} -%\include{060_Appendix/Results} \backmatter +%\renewcommand{\baselinestretch}{1.25} \input{References.tex} % that's all folks diff --git a/carreira_phd_thesis_abr.sty b/carreira_phd_thesis_abr.sty index 4dda5f8..ccfe4bd 100644 --- a/carreira_phd_thesis_abr.sty +++ b/carreira_phd_thesis_abr.sty @@ -2,33 +2,37 @@ %% Generic %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \customnewacronym{uhd}{UHD}{Ultra-HD} -\customnewacronym{qoe}{QoE}{Quality of Experience} \glsunset{qoe} -\customnewacronym{qos}{QoS}{Quality of Service} \glsunset{qos} +\customnewacronym{qoe}{QoE}{Quality of Experience} +\customnewacronym{qos}{QoS}{Quality of Service} \customnewacronym[longplural={frames per second},description=Frames per Second]{fps}{FPS}{frames per second} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Network %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\customnewacronym{rtp}{RTP}{Real-time Transport Protocol} \glsunset{rtp} -\customnewacronym{ip}{IP}{Internet Protocol} \glsunset{ip} +\customnewacronym{rtp}{RTP}{Real-time Transport Protocol} +\customnewacronym{ip}{IP}{Internet Protocol} \customnewacronym{lte}{LTE}{Long Term Evolution} \customnewacronym{mtu}{MTU}{Maximum Transfer Unit} \customnewacronym{mmt}{MMT}{MPEG Media Transport} +\customnewacronym{ts}{MPEG-2 TS}{MPEG-2 Transport Stream} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Video Coding %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\customnewacronym{mpeg}{MPEG}{Moving Picture Experts Group} \glsunset{mpeg} -\customnewacronym{vceg}{VCEG}{Video Coding Experts Group} \glsunset{vceg} +\customnewacronym{itut}{ITU-T}{International Telecommunication Union - Telecommunication} +\customnewacronym{iso}{ISO}{International Organization for Standardization} +\customnewacronym{iec}{IEC}{International Electrotechnical Commission} -\customnewacronym{iso}{ISO}{International Organization for Standardization} \glsunset{iso} -\customnewacronym{iec}{IEC}{International Electrotechnical Commission} \glsunset{iec} +\customnewacronym{mpeg}{MPEG}{Moving Picture Experts Group} +\customnewacronym{vceg}{VCEG}{Video Coding Experts Group} + +\customnewacronym{jctvc}{JCT-VC}{Joint Collaborative Team on Video Coding} \customnewacronym{gop}{GOP}{Group of Pictures} \customnewacronym{qp}{QP}{Quantized Parameter} \customnewacronym{rd}{R-D}{\mbox{Rate-Distortion}} \customnewacronym{nal}{NAL}{Network Abstraction Layer} -\customnewacronym{nalu}{NALU}{Network Abstraction Layer Unit} +%\customnewacronym{nalu}{NALU}{Network Abstraction Layer Unit} \customnewacronym{vcl}{VCL}{Video Coding Layer} \customnewacronym{poc}{POC}{Picture Order Count} \customnewacronym{sei}{SEI}{Supplemental Enhancement Information} @@ -50,7 +54,7 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% H.264 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\customnewacronym{h264}{H.264/AVC}{H.264 - Advanced Video Coding} \glsunset{h264} +\customnewacronym{h264}{H.264/AVC}{H.264 - Advanced Video Coding} \customnewacronym{mb}{MB}{Macroblock} \customnewacronym{fmo}{FMO}{Flexible Macroblock Ordering} \customnewacronym{mmco}{MMCO}{Memory Management Control Operation} @@ -122,7 +126,7 @@ \customnewacronym{plr}{PLR}{Packet Loss Ratio} \customnewacronym{mse}{MSE}{Mean Square Error} \customnewacronym{psnr}{PSNR}{Peak Signal-to-Noise Ratio} -\customnewacronym{wpsnr}{WPSNR}{Weighted Peak Signal-to-Noise Ratio} \glsunset{wpsnr} +\customnewacronym{wpsnr}{WPSNR}{Weighted Peak Signal-to-Noise Ratio} \customnewacronym{ssim}{SSIM}{Structural Similarity Index Metric} \customnewacronym{fsim}{FSIM}{Feature Similarity Index} @@ -190,7 +194,7 @@ \def\FIXED{\mbox{\textit{Fixed}}\xspace} %\def\DYN{\mbox{\textit{Prop-MV}}\xspace} \def\DYN{\mbox{\textit{Dyn}}\xspace} -\def\NOREDUND{\mbox{Without MVs}\xspace} +\def\NOREDUND{\mbox{\textit{Without MVs}}\xspace} \newcommand\REDUND[1]{\mbox{#1\%\ of MVs}\xspace} \newcommand\OPTREDUND[1]{\mbox{\DYN ($\alpha$=#1)}\xspace} @@ -201,8 +205,8 @@ %\def\MEXPELTEN{\textit{\mbox{Exp+EL10\%}}\xspace} %\def\MEXPELTWEN{\textit{\mbox{Exp+EL20\%}}\xspace} -\def\MEC{\textit{\mbox{M1}}} -\def\MEXP{\textit{\mbox{M2}}} +\def\MEC{\textit{\mbox{M1}}\xspace} +\def\MEXP{\textit{\mbox{M2}}\xspace} \def\MEXPELTEN{\textit{\mbox{M3}}\xspace} \def\MEXPELTWEN{\textit{\mbox{M4}}\xspace} diff --git a/estgleiria.cls b/estgleiria.cls index b891b6d..94bd6a7 100644 --- a/estgleiria.cls +++ b/estgleiria.cls @@ -59,8 +59,7 @@ \RequirePackage{ltxcmds} \RequirePackage{notoccite} % \RequirePackage{nomencl} -% \RequirePackage[a4paper,bindingoffset=0.0cm,vcentering=true,top=3cm,bottom=2.5cm,left=3.0cm,right=2cm]{geometry} -\RequirePackage[a4paper,bindingoffset=0.0cm,vcentering=true,top=3cm,bottom=2.5cm,left=3.5cm,right=2cm]{geometry} +\RequirePackage[a4paper,bindingoffset=0.0cm,vcentering=true,top=3cm,bottom=2.8cm,left=3.5cm,right=2cm]{geometry} % \RequirePackage[includeheadfoot,margin=3.5cm]{geometry} \def\EstgTeX{{\rm E\kern-.05em{\sc S\kern-.025em T\kern-.025em @@ -243,11 +242,12 @@ T\kern-.1667em\lower.5ex\hbox{E}\kern-.125emX\spacefactor1000} } \renewcommand\frontmatter{% - \insersecondtitle +% \insersecondtitle \cleardoublepage \setcounter{page}{1} \pagenumbering{roman} - \pagestyle{plain} +% \pagestyle{plain} + \properpagestyle \titleformat{\chapter}[display] {\normalfont\Large\bfseries} {\sc\chaptertitlename\ \thechapter}{20pt} @@ -348,8 +348,8 @@ T\kern-.1667em\lower.5ex\hbox{E}\kern-.125emX\spacefactor1000} \vspace*{\fill} \begin{center} % \local@cityname, \local@monthname\ de \number\year - \foreign@monthname\ \number\year \par - \vspace{2cm} +% \foreign@monthname\ \number\year \par + \vspace{3cm} \copyright\ \number\year\ by Jo\~{a}o Carreira \end{center} \end{titlepage} @@ -395,7 +395,7 @@ T\kern-.1667em\lower.5ex\hbox{E}\kern-.125emX\spacefactor1000} \newcommand\dedication[1]{ \gdef\@dedic{#1} - \cleardoublepage +% \cleardoublepage \thispagestyle{empty} \vspace*{\fill} \begin{flushright}