From 1c13a4e48ee8e3dba55b212519fd2b6693c8acd4 Mon Sep 17 00:00:00 2001 From: Joao Carreira Date: Thu, 6 Sep 2018 14:56:36 +0100 Subject: [PATCH] Fix some compound word --- 000_Preamble/abstract.tex | 6 +- 010_Introduction/Introduction.tex | 46 +-- 021_VideoCoding/VideoCodingReview.tex | 8 +- 022_ErrorResilience/Figures/IntraRefresh.tikz | 22 +- .../RobustTransmissionReview.tex | 332 ++++++++++-------- .../ErrorResilienceEvaluation.tex | 109 +++--- .../ImprovedErrorResilience.tex | 56 +-- .../EndToEndRobustSystem.tex | 6 +- carreira_phd_thesis_abr.sty | 2 +- 9 files changed, 309 insertions(+), 278 deletions(-) diff --git a/000_Preamble/abstract.tex b/000_Preamble/abstract.tex index 38caf22..4836fa8 100755 --- a/000_Preamble/abstract.tex +++ b/000_Preamble/abstract.tex @@ -17,11 +17,11 @@ % Then a subset of prioritised \acs{mv}selected with the previous is transmitted as redundancy, to reduce the use and propagation of mismatched \ac{mv} predictions in case of transmission errors or data loss. % The simulation results show that the proposed \acs{mv} selection method can effectively identify the most relevant motion field, achieving improved error robustness with a reduced redundancy overhead. % -% Further research will be pursed to improve the error robustness of the \ac{hevc} standard and develop novel concealment methods using combined info between the encoder and decoder side. +% Further research will be pursed to improve the error robustness of the \ac{hevc} standard and develop novel concealment methods using combined info between the encoder and decoder-side. %The increasing diversity of services and the demand for ever higher quality multimedia at wider resolutions, which leads to the growing popularity of Ultra-HD (UHD) format, are pushing technologies forward and opening new research directions. The recent developments in video coding techniques have resulted in a novel standard called \ac{hevc} that is the next breakthrough in multimedia compression after the globally adopted MPEG-4/\ac{h264} standard. However, the significantly improved levels of compression efficiency achieved by the \ac{hevc} standard lead to an inherent reduction in the error resilience performance under error prone conditions. It should be aimed at ensuring an adequate level of protection against network and transmission induced impairments (e.g., packet losses), without sacrificing the compression efficiency thanks to the novel block level operations done by \ac{hevc}. % -%My research work takes the \ac{hevc} standard to its core and builds new techniques on top of it to enhance its error resilience performance, and develop new processing techniques to improve decoder side error concealment performance. The first scope of this research work is to make an extensive analysis of various error resilience techniques developed for the \ac{hevc} standard, but also for the previous video coding standards (most notably \ac{h264}); identify their shortcomings and address those shortcomings. Up to date work has revealed that the \ac{mv} coding technique lacks robustness, which leads to quality degradation due to error propagation in lossy transmission environments. It has also been seen that the temporal dependency of motion information is comparatively higher than that in the \ac{h264} standard. Based on this evidence, a method was developed to make intelligent use of temporal \ac{mv} candidates during the motion estimation process, in order to decrease the temporal dependency, and improve the error resiliency without penalising the rate-distortion performance. The simulation results have shown that the proposed method improves the error resilience under tested conditions, compared to the reference method that always enables temporal \acs{mv} candidates. Another proposed method has exploited the prediction dependency of \glspl{mv}in order to select the most relevant ones for redundant coding with reduced overhead. Simulation results have shown that the proposed \gls{mv} selection method can effectively identify the most relevant motion fields, achieving improved error robustness with a significantly reduced redundancy overhead. +%My research work takes the \ac{hevc} standard to its core and builds new techniques on top of it to enhance its error resilience performance, and develop new processing techniques to improve decoder-side error concealment performance. The first scope of this research work is to make an extensive analysis of various error resilience techniques developed for the \ac{hevc} standard, but also for the previous video coding standards (most notably \ac{h264}); identify their shortcomings and address those shortcomings. Up to date work has revealed that the \ac{mv} coding technique lacks robustness, which leads to quality degradation due to error propagation in lossy transmission environments. It has also been seen that the temporal dependency of motion information is comparatively higher than that in the \ac{h264} standard. Based on this evidence, a method was developed to make intelligent use of temporal \ac{mv} candidates during the motion estimation process, in order to decrease the temporal dependency, and improve the error resiliency without penalising the rate-distortion performance. The simulation results have shown that the proposed method improves the error resilience under tested conditions, compared to the reference method that always enables temporal \acs{mv} candidates. Another proposed method has exploited the prediction dependency of \glspl{mv}in order to select the most relevant ones for redundant coding with reduced overhead. Simulation results have shown that the proposed \gls{mv} selection method can effectively identify the most relevant motion fields, achieving improved error robustness with a significantly reduced redundancy overhead. % -%The second scope of this research work is to develop new processing techniques for the concealment of the coding elements lost during the transmission and delivery system, using combined information between the encoder and decoder sides. Error concealment techniques mitigate the detrimental effects of network impairments on the perceptual multimedia quality and the performance these techniques determine the quality of a system directly. The third and the ultimate aim of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet a set of QoS/QoE requirements associated with current and future multimedia-based services (e.g., real-time multi-party video conferencing, Ultra-HD video on demand, etc.), such as bandwidth and delay. It is envisaged to produce a set of configuration profiles for chosen services based around the developed tools for \ac{hevc}. +%The second scope of this research work is to develop new processing techniques for the concealment of the coding elements lost during the transmission and delivery system, using combined information between the encoder and decoder-sides. Error concealment techniques mitigate the detrimental effects of network impairments on the perceptual multimedia quality and the performance these techniques determine the quality of a system directly. The third and the ultimate aim of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet a set of QoS/QoE requirements associated with current and future multimedia-based services (e.g., real-time multi-party video conferencing, Ultra-HD video on demand, etc.), such as bandwidth and delay. It is envisaged to produce a set of configuration profiles for chosen services based around the developed tools for \ac{hevc}. diff --git a/010_Introduction/Introduction.tex b/010_Introduction/Introduction.tex index d548683..ef9573d 100755 --- a/010_Introduction/Introduction.tex +++ b/010_Introduction/Introduction.tex @@ -8,24 +8,24 @@ \glsreset{h264} -Over the recent years new advances in video compression and networking technologies have enabled the development of many multimedia applications over the internet, such as video conferencing, video streaming, video publishing, among others. +Over the recent years new advances in video compression and networking technologies have enabled the development of many multimedia applications over the internet, such as video conferencing, video streaming, video publishing, among others. The increasing diversity of services, the demands for high quality multimedia and the growing popularity of \ac{uhd} formats (\emph{e.g.}, $4k$ or $8k$ resolution and beyond) are pushing technologies forward and opening new research directions. Moreover, the emerging new video formats with increased resolutions have created stronger needs for higher coding efficiency, beyond the previous standards, i.e., the MPEG-4/\ac{h264}~\cite{H264AVCStandard2003,H264OverviewJul2003}. -Such need has been driven not only by higher resolutions but also by stereoscopic and multi-view video formats, which increase the number of viewpoints that need to be encoded and delivered simultaneously. +Such need has been driven not only by higher resolutions but also by stereoscopic and multi-view video formats, which increase the number of viewpoints that need to be encoded and delivered simultaneously. Moreover, the ever increasing traffic generated by mobile multimedia applications \cite{CiscoMobile1621_Feb2017} and different video services with ever increasing popularity, e.g., Netflix, impose new challenges to the existing networks. To cope with such demanding requirements a new coding standard was developed by the ITU-T \ac{vceg} and the \ac{iso}/\ac{iec} \ac{mpeg} standardisation organisations, referred to as H.265 - \ac{hevc}~\cite{H265StandardFeb2018,HEVCOverviewDez2012}. \glsunset{h264} -The \ac{hevc} is currently the main video coding framework to extend current services towards higher resolution formats delivered with better compression efficiency. -Similar to its predecessor, \ac{hevc} is a block-based hybrid video codec using predictive, transform and entropy coding. -The coding flexibility and efficiency of \ac{hevc} results from the use of new coding tools and block structures, which enables new block partitions\cite{HEVCBlock2012}, improved prediction modes \cite{IntraCodingHEVC2012,AMPCodingHEVCJun2012} and new high-level features~\cite{HEVCHighLevelReferenceDez2012}, such as explicit reference picture management and new parameter sets. +The \ac{hevc} is currently the main video coding framework to extend current services towards higher resolution formats delivered with better compression efficiency. +Similar to its predecessor, \ac{hevc} is a block-based hybrid video codec using predictive, transform and entropy coding. +The coding flexibility and efficiency of \ac{hevc} results from the use of new coding tools and block structures, which enables new block partitions\cite{HEVCBlock2012}, improved prediction modes \cite{IntraCodingHEVC2012,AMPCodingHEVCJun2012} and new high-level features~\cite{HEVCHighLevelReferenceDez2012}, such as explicit reference picture management and new parameter sets. However, it is known that higher coding efficiency leads to lower robustness in the presence of transmission errors and data loss. Therefore, the \ac{hevc} new coding tools bring not only high compression efficiency~\cite{HEVCComparisonDec2012}, but also some other disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and inherent reduction of error robustness. %\cite{StudyHEVCErrorsDez2012} Earlier studies about the error robustness characteristics of \ac{hevc} have shown that, in general, the use of high efficiency coding tools results in poor error resilience performance, in comparison with previous standards. Thus, when \ac{hevc} streams incur in transmission losses, this leads to significant degradation of both objective and subjective quality~\cite{QoEHEVCMay2014,SubjectiveEvalSep2013}. -This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes of \ac{hevc}, which assume \mbox{error-free} transmission~\cite{ROPE2007}. +This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes of \ac{hevc}, which assume \mbox{error-free} transmission~\cite{ROPE2007}. A possible approach to encode robust video streams is to increase the error resilience of \ac{hevc} by including \mbox{error-aware} mechanisms in the coding process. As a result, the effect of error propagation, due to prediction mismatch at the decoding side, will be limited. The widespread use of \ac{hevc} in broadcast and multimedia delivery services is based on transport technologies such as \ac{rtp}~\cite{HEVCStreamMay2012}, \mbox{MPEG-2 TS}~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016}. @@ -56,7 +56,7 @@ the overall error robustness of video transmission may also benefit from the inc In summary, increasing the error robustness of the highly compressed video streams is a relevant research problem, whose solutions will be useful to increase the adaptability for transmission over diverse network conditions and to achieve robust communications, thus providing acceptable levels of \ac{qoe} to the end users at all times. In general, increased error robustness in video transmission can be jointly achieved at three different stages: (i) using robust video encoding, by selecting robust coding modes, (ii) after encoding, by adding robust features to the compressed streams, as side-information, and (iii) using robust video decoding techniques, e.g., efficient \ac{ec}. This is exemplified in Figure~\ref{FIG1:RobustVideoTransmission} where at the encoder different layers are used to increase the error robustness, which results in a video stream composed by a \ac{vcl} and \ac{sei} layers. Finally, at the decoder two processing layers are used to achieve robust video decoding. The \ac{hevc} reference codec is continuously being improved and extended by multiple proponents from around the world, which also address the problem of error resilience and robust delivered transmission. -Although different \ac{ec} methods have been developed during the past years for \ac{h264}~\cite{2DRecursiveRefinement2010} and recently for \ac{hevc}~\cite{HEVCMCBlockAug2013}, a combined and optimised approach involving the encoder and decoder is yet to be developed, which would improve the users’ \ac{qoe}, by exploiting shared information between the encoder and the decoder sides. +Although different \ac{ec} methods have been developed during the past years for \ac{h264}~\cite{2DRecursiveRefinement2010} and recently for \ac{hevc}~\cite{HEVCMCBlockAug2013}, a combined and optimised approach involving the encoder and decoder is yet to be developed, which would improve the users’ \ac{qoe}, by exploiting shared information between the encoder and the decoder-sides. \begin{figure}[tb] \centering @@ -71,18 +71,18 @@ Although different \ac{ec} methods have been developed during the past years for The research objectives of this thesis are focused in robust error resilience for efficient error concealment in highly efficiency video decoders. - -Thus, this work investigates novel techniques to increase the video quality of \ac{hevc} bitstreams delivered through error-prone networks. + +Thus, this work investigates novel techniques to increase the video quality of \ac{hevc} bitstreams delivered through error-prone networks. The following objectives were pursued in this work: \begin{enumerate} -\item +\item \textbf{evaluation and study of the error robustness:} The first objective of this research work is to evaluate the robustness against transmission errors of highly efficiently coded video. The study conclusions aimed at finding relevant research directions to increase the error robust of coded streams. The results and conclusions of this study were published in conference paper (\ref{Conf:ErrorResilience}) and in journal paper (\ref{Journal:TwoStage}) (see Appendix \ref{Appen:Papers}). -\item -\textbf{robust video coding:} At the encoder side, novel techniques to extend the existing error robustness of highly predictive video coding techniques are investigated in order to increase the received video quality over error-prone networks. This research topic resulted in a new method to reduce the error propagation in motion vector coding algorithms, which was published in conference paper (\ref{Conf:DynamicTMVP}). Moreover, a novel two-stage approach was devised to improve the error robustness of highly efficiently coded video at the encoder and streaming levels, resulting in two conference publications (\ref{Conf:MotionVectorRedundancies} and \ref{Conf:Chkb}) and a journal publication (\ref{Journal:TwoStage}). +\item +\textbf{robust video coding:} At the encoder-side, novel techniques to extend the existing error robustness of highly predictive video coding techniques are investigated in order to increase the received video quality over error-prone networks. This research topic resulted in a new method to reduce the error propagation in motion vector coding algorithms, which was published in conference paper (\ref{Conf:DynamicTMVP}). Moreover, a novel two-stage approach was devised to improve the error robustness of highly efficiently coded video at the encoder and streaming levels, resulting in two conference publications (\ref{Conf:MotionVectorRedundancies} and \ref{Conf:Chkb}) and a journal publication (\ref{Journal:TwoStage}). -\item \textbf{robust video decoding:} error concealment techniques are used to increase the error robustness of video decoder and improve the perceived video quality. -The paradigm of error concealment is evaluated at the encoder side, as part of the encoding process, in order to optimise the reconstruction process at the decoder side. This research resulted in a novel error concealment-aware technique to optimise the error concealment process at the encoder side, which was proposed for intra-frame only in a conference paper (\ref{Conf:ConcAware}) and for a generic robust video transmission in the journal (\ref{Journal:ConcAware}). +\item \textbf{robust video decoding:} error concealment techniques are used to increase the error robustness of video decoder and improve the perceived video quality. +The paradigm of error concealment is evaluated at the encoder-side, as part of the encoding process, in order to optimise the reconstruction process at the decoder-side. This research resulted in a novel error concealment-aware technique to optimise the error concealment process at the encoder-side, which was proposed for intra-frame only in a conference paper (\ref{Conf:ConcAware}) and for a generic robust video transmission in the journal (\ref{Journal:ConcAware}). %\item \textbf{jointly error resilience and concealment:} the ultimate goal of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet higher quality demands for different video contents characteristics and network conditions. \end{enumerate} @@ -97,26 +97,26 @@ This thesis is organised as follows. The current chapter introduces the context, The first two chapters review the \mbox{state-of-the-art} concepts and previous work related with this research. Chapter~\ref{Chapter:HEVCReview} presents an overview of the \ac{hevc} standard, focusing on the concepts explored in this work, e.g., high-level features and temporal predictions. Chapter~\ref{Chapter:ErrorResilienceReview} reviews the most important work dealing with robust video transmission also providing relevant background knowledge to this work. -Chapter~\ref{Chapter:HEVCErrorResilience} firstly presents a study about the error robustness of high efficiently coded video, using the \ac{hevc} as the underlying framework for various coding configuration. -The study presents the relevant tools that mostly affect the error robustness. +Chapter~\ref{Chapter:HEVCErrorResilience} firstly presents a study about the error robustness of high efficiently coded video, using the \ac{hevc} as the underlying framework for various coding configuration. +The study presents the relevant tools that mostly affect the error robustness. Secondly, a new method is proposed to decrease the vulnerability of temporal motion prediction and decrease the error propagation, leading to higher video quality under error-prone conditions. -Chapter~\ref{Chapter:ErrorResilienceTechniques} proposes a novel error resilience approach for high efficiently video codecs. +Chapter~\ref{Chapter:ErrorResilienceTechniques} proposes a novel error resilience approach for high efficiently video codecs. A new method to distribute the use of reference frames is proposed to reduce the error propagation and improve the \ac{ec} performance. -Moreover, to reduce the effect of mismatch motion predictions (shown in Chapter~\ref{Chapter:HEVCErrorResilience}) a method based on selective motion vector redundancies is proposed. +Moreover, to reduce the effect of mismatch motion predictions (shown in Chapter~\ref{Chapter:HEVCErrorResilience}) a method based on selective motion vector redundancies is proposed. This method is able to improve the decoded video quality using a small overhead. -Chapter~\ref{Chapter:ConcealmentAwareTechniques} addresses the problem of robust \ac{ec} of highly efficiently video streams. -%This work was motivated by the requirements for more accurate \ac{ec} algorithms which are able to reduce the error propagation. -Different \ac{ec} algorithms were taken into consideration in order to develop a concealment-aware coding scheme. +Chapter~\ref{Chapter:ConcealmentAwareTechniques} addresses the problem of robust \ac{ec} of highly efficiently video streams. +%This work was motivated by the requirements for more accurate \ac{ec} algorithms which are able to reduce the error propagation. +Different \ac{ec} algorithms were taken into consideration in order to develop a concealment-aware coding scheme. The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled to the decoder. %Such optimal EC\ac{ec} modes are found by simulating transmission losses followed by a \ac{rd} optimisation of the signalling rate and \ac{ec} distortion cost. -A generalised saliency-weighted \ac{rd} optimisation is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. +A generalised saliency-weighted \ac{rd} optimisation is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. %When data loss occurs the decoder uses the signalling information is used at the decoder, in case of data loss, to improve the reconstruction quality. %The simulation results show that the proposed method achieves consistent quality gains in comparison with other reference methods and previous works. -Chapter~\ref{Chapter:EndToEndRobustSystem} proposes a novel adaptive error robustness method, in order to combine different techniques proposed in this thesis. At the encoder side various video streams are generated with different levels of compression efficiency and error robustness. Then, a pre-training \ac{nn} is used to predict which video stream will achieve the highest decoded video quality for a particular scenario. The selection is based on different input parameters which include information from the video signal, the coded stream and the transmission network. +Chapter~\ref{Chapter:EndToEndRobustSystem} proposes a novel adaptive error robustness method, in order to combine different techniques proposed in this thesis. At the encoder-side various video streams are generated with different levels of compression efficiency and error robustness. Then, a pre-training \ac{nn} is used to predict which video stream will achieve the highest decoded video quality for a particular scenario. The selection is based on different input parameters which include information from the video signal, the coded stream and the transmission network. Chapter~\ref{Chapter:Conclusions} concludes this dissertation and presents some suggestions for future work. diff --git a/021_VideoCoding/VideoCodingReview.tex b/021_VideoCoding/VideoCodingReview.tex index 2b5726e..3eb6258 100755 --- a/021_VideoCoding/VideoCodingReview.tex +++ b/021_VideoCoding/VideoCodingReview.tex @@ -23,12 +23,12 @@ Particular emphasis is given to those which may have a significant impact in the \glsunset{hevc} -This chapter is organized as follows. +This chapter is organised as follows. Section~\ref{Section2:GeneraVideoSystem} introduces the functional blocks of a generic hybrid video coding system and Section~\ref{Section2:Overview} provides an overview of the \ac{hevc} standard. Then, Section~\ref{Section2:HighLevel} describes the high-level syntax elements, which are used to carry relevant information in coded streams. Section~\ref{Section2:CodingStructures} describes the data structures used to divide the video frames into smaller partitions and Section~\ref{Section2:PredictionModes} describes particular aspects of the main predictions techniques, namely intra and inter coding. Finally, a brief description of the transform and quantisation, entropy coding and in-loop filters is provided in Sections~\ref{Section2:TransformQuant}, \ref{Section2:Entropy} and \ref{Section2:Filters}, respectively. -Section~\ref{Section2:Summary} concludes this chapter. +Section~\ref{Section2:Summary} concludes the chapter. \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -76,8 +76,8 @@ Thus, the purpose of inter-frame prediction is to search for the best matching b After finding the best matching candidate for the current block, either by intra- or inter-frame prediction, only the difference between the original block and the predicted one needs to be encoded. As shown in Figure~\ref{Figure2:HybridCodec}, these differences (also known as residual signal) are further processed and transformed (T) into coefficients in the frequency domain. Such a transformation decorrelates the residue signal and concentrates the signal energy in few coefficients, mostly at low frequencies. -Subsequently, the transformed coefficients are quantized (Q) to reduce the representation accuracy of the signal, and consequently to reduce the number of bits required for coding. -Finally, the quantized transform coefficients together with the prediction information are further compressed by the entropy encoder, forming an output stream suitable for transmission or storage, as shown in Figure~\ref{Figure2:HybridCodec}. +Subsequently, the transformed coefficients are quantised (Q) to reduce the representation accuracy of the signal, and consequently to reduce the number of bits required for coding. +Finally, the quantised transform coefficients together with the prediction information are further compressed by the entropy encoder, forming an output stream suitable for transmission or storage, as shown in Figure~\ref{Figure2:HybridCodec}. On the other side of the communication chain, the decoding stage performs the inverse operations on the stream to reconstruct the video signal. In the next section, the aforementioned blocks are described in the context of the \ac{hevc} standard. diff --git a/022_ErrorResilience/Figures/IntraRefresh.tikz b/022_ErrorResilience/Figures/IntraRefresh.tikz index 00c6919..b703f3d 100644 --- a/022_ErrorResilience/Figures/IntraRefresh.tikz +++ b/022_ErrorResilience/Figures/IntraRefresh.tikz @@ -1,8 +1,12 @@ - \newcommand{\DrawFrame} { \draw[fill=TikZColorFill,fill opacity=1, thick] (0,0) rectangle (10,8); % Opacity } +\newcommand{\DrawFrameBorder} +{ +\draw[thick] (0,0) rectangle ++(10,8); +} + \newcommand{\DrawRandomBlocks} { \foreach \x in {1,2,...,14} @@ -39,20 +43,21 @@ \DrawRandomBlocks \coordinate (endPoint) at (9,0); + \DrawFrameBorder \end{scope} \begin{scope}[scale=\TikzFrameScale, yshift=2*\yfshift, xshift=2*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] \DrawFrame \coordinate (Center) at (5,3.5); \DrawRandomBlocks - + \DrawFrameBorder \end{scope} \node[FrameLabels] at (Center |- FrameLabel) { $f_{t+2}$ }; \begin{scope}[scale=\TikzFrameScale, yshift=1*\yfshift, xshift=1*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] \DrawFrame \coordinate (Center) at (5,3.5); \DrawRandomBlocks - + \DrawFrameBorder \end{scope} \node[FrameLabels] at (Center |- FrameLabel) { $f_{t+1}$ }; \begin{scope}[scale=\TikzFrameScale, yshift=0*\yfshift, xshift=0*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] @@ -62,12 +67,13 @@ \DrawRandomBlocks \coordinate (startPoint) at (0,0); + \DrawFrameBorder \end{scope} \node[FrameLabels] at (Center |- FrameLabel) { $f_{t}$ }; \path[draw, -latex] ($(FrameLabel)+(-8.8,-.4)$) -- ($(FrameLabel)+(1.2,-.4)$) node[below] {t} node[below, midway, yshift=-4] {(a) Random intra-refresh}; \end{scope} - \begin{scope}[yshift=-4.5cm] + \begin{scope}[yshift=-5cm] \begin{scope}[scale=\TikzFrameScale, yshift=3*\yfshift, xshift=3*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] \DrawFrame \coordinate (Center) at (5,3.5); @@ -87,6 +93,7 @@ \end{scope} \coordinate (endPoint) at (9,0); + \DrawFrameBorder \end{scope} \begin{scope}[scale=\TikzFrameScale, yshift=2*\yfshift, xshift=2*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] @@ -105,7 +112,7 @@ \begin{scope}[scale=.8, yshift=5.3cm, xshift=4.8cm] \draw[thick] (-3,0.5) .. (-2.5,2.5) .. (-.8,3.5).. (1.5,3).. (4.2,2.8).. (5,2.5).. (5,0.5) ..(2.5,-2).. (0,-0.5).. (-1,-.5).. (-3,0.5); \end{scope} - + \DrawFrameBorder \end{scope} \node[FrameLabels] at (Center |- FrameLabel) { $f_{t+2}$ }; @@ -125,7 +132,7 @@ \begin{scope}[scale=.8, yshift=4.3cm, xshift=4.8cm] \draw[thick] (-3,0.5) .. (-2.5,2.5) .. (-.8,3.5).. (1.5,3).. (4.2,2.8).. (5,2.5).. (5,0.5) ..(2.5,-2).. (0,-0.5).. (-1,-.5).. (-3,0.5); \end{scope} - + \DrawFrameBorder \end{scope} \node[FrameLabels] at (Center |- FrameLabel) { $f_{t+1}$ }; @@ -150,12 +157,13 @@ \coordinate (startPoint) at (0,0); + \DrawFrameBorder \end{scope} \node[FrameLabels] at (Center |- FrameLabel) { $f_{t}$ }; \path[draw, -latex] ($(FrameLabel)+(-8.8,-.4)$) -- ($(FrameLabel)+(1.2,-.4)$) node[below] {t} node[below, midway, yshift=-4] {(b) ROI based intra-refresh}; \end{scope} - \node[refreshBlock, anchor=center, minimum size=.5cm] (RefreshExample) at (0, -6.8) {}; + \node[refreshBlock, anchor=center, minimum size=.5cm] (RefreshExample) at (0, -7.5) {}; \node[labels, anchor=center, right=.1 of RefreshExample] {\small Intra-coded blocks}; diff --git a/022_ErrorResilience/RobustTransmissionReview.tex b/022_ErrorResilience/RobustTransmissionReview.tex index cc40ccb..d382fe4 100755 --- a/022_ErrorResilience/RobustTransmissionReview.tex +++ b/022_ErrorResilience/RobustTransmissionReview.tex @@ -1,4 +1,4 @@ -\chapter{Robust Video Coding: A Review \VERSION{1}} +\chapter{Robust Video Coding: A Review \VERSION{2}} \label{Chapter:ErrorResilienceReview} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -15,14 +15,16 @@ These errors can be handled in different ways, such as retransmission, error cor This chapter reviews the main robust video coding techniques that can be applied at the \ac{vcl} to reduce the error propagation and increase the decoded video quality. Several techniques have been proposed in the past to deal with the problem of robust video transmission, especially for video coding standards with high temporal dependencies, such as \ac{h264} and more recently \ac{hevc}. These techniques are normally focused on the spatio-temporal dependencies in the coded streams, which are used to obtain high coding gains but lead to low error robustness~\cite{ROPE2007}. -In this chapter the error resilience and concealment schemes are divided in different categories depending on their functionality and algorithms. -Although each approach is based on a different techniques they all aim at striking the most efficient balance between coding efficiency and decoded video quality under lossy conditions. +In this chapter the error resilience and concealment schemes are divided in different categories, depending on their functionality and algorithms. +Although each approach is based on different techniques they all aim at striking the best trade-off between coding efficiency and decoded video quality under lossy conditions. -This chapter is organised as follows: Section~\ref{Section22:Localisation} describes the localisation techniques, which reduce the temporal and spatial dependencies in coded bitstreams. Section~\ref{Section22:DataPartitionRedundant} provides an overview of methods based on data partition, classification and bitstream redundancy. +This chapter is organised as follows: Section~\ref{Section22:Localisation} describes the localisation techniques, which are able to reduce the temporal and spatial dependencies in coded bitstreams. +Section~\ref{Section22:DataPartitionRedundant} provides an overview on methods based on data partition, classification and bitstream redundancy. %These techniques ease the introduction of extra information in order to recover from lost data. -Section~\ref{Section22:Concealment} covers the techniques used at the decoder-side as post-processing to reduce the effect of errors and improve visual quality and smoothness. +Section~\ref{Section22:Concealment} covers the techniques used at the decoder-side, as post-processing, to reduce the effect of errors and improve visual quality and smoothness. Section~\ref{Section22:ConcealmentAware} provides a description of methods that use \ac{ec} techniques to achieve improved error robustness. -Finally, Section~\ref{Section22:Summary} concludes this chapter by providing a overall comparison between the different approaches. +Finally, Section~\ref{Section22:Summary} concludes the chapter. +% by providing a overall comparison between the different approaches. \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -31,10 +33,10 @@ Finally, Section~\ref{Section22:Summary} concludes this chapter by providing a o \section{Robust video coding techniques} In general, most of the compression efficiency is achieved by exploiting both the spatial and temporal redundancy of video signals. Since there is high correlation between neighbouring information, predictive coding schemes are very effective to achieve higher compression gains by exploiting such correlations. -However, the drawback of high predictive coding schemes is the strong dependencies between compressed symbols with great impact on error propagation. +However, the drawback of high predictive coding schemes is the strong dependencies between compressed symbols, which may have a great impact on error propagation. In order to increase the robustness of video transmission, several coding mechanisms can be employed at different elements of a transmission chain to mitigate the artefacts caused by errors and data loss. These mechanisms can be classified according to the functional characteristics and the type of problem they address~\cite{ErrorResilienceAug2005}. -In this chapter the robust video coding techniques ere grouped into four categories: +In this chapter the robust video coding techniques are grouped into four categories: \begin{itemize} \item Localization techniques; \item Data partitioning and redundant coding; @@ -55,7 +57,7 @@ Data partitioning rely on grouping the coded data into different categories depe This ease the introduction of unequal error protection at the transmission layer. Redundant coding consists in using additional data blocks to enable correction of transmission errors and to achieve robust decoding in error-prone conditions. In case of the post-processing \ac{ec} methods, the problem of robust transmission is addressed from the decoder-side. -Finally, concealment-aware techniques are used at the encoder, to either adapt the coding loop or introduce extra information which is used to increase the performance of \ac{ec} algorithms and increase the reconstruction quality of lost data at the decoder \cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ConcealmentIntraDataHidng2009,ConcealmentSignalling2008}. +Finally, concealment-aware techniques are used at the encoder, to either adapt the coding loop or introduce extra information, which is used to increase the performance of \ac{ec} algorithms and increase the reconstruction quality of lost data at the decoder \cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ConcealmentIntraDataHidng2009,ConcealmentSignalling2008}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -66,31 +68,24 @@ Finally, concealment-aware techniques are used at the encoder, to either adapt t The basic principle of localization techniques is to break the predictive coding loop at certain predictive points, in order to reduce the likelihood of errors affecting large regions of the video signal, considering both the spatial and temporal domains. -Since these techniques break the predictive coding, they have a negative impact in compression efficiency as expected. +Since these techniques break the predictive coding, they have a negative impact in compression efficiency, as expected. + +This group of error resilience techniques are able to reduce the error propagation at two levels, i.e., temporal and spatial levels. Figure~\ref{fig2:ErrorPropagation} is used to demonstrate the principle of error propagation and how it can be reduced using localisation techniques. +The top of the figure illustrates a lost block and the corresponding spatial and temporal error propagation on the left and right, respectively. +Spatial localisation techniques are used to reduce the error propagation within a frame, where a row of refresh blocks can be introduced to stop the error propagation. +To address the temporal error propagation, also refresh blocks (e.g., intra coded blocks) can be randomly introduced to stop error propagation. +As shown in Figure~\ref{fig2:ErrorPropagation} the refresh blocks are not affected by error propagation. +In both examples of figure, the regions affected by errors are clearly reduced. + \begin{figure}[tb] -\centering -\input{./022_ErrorResilience/Figures/LocalizationTechniques.tikz} -\caption{Examples of spatial and temporal error propagation and how it can be reduced with spatial and temporal error resilience techniques.} -\label{fig2:ErrorPropagation} -%\cite{ErrorResilienceAug2005} + \centering + \input{./022_ErrorResilience/Figures/LocalizationTechniques.tikz} + \caption{Examples of techniques for spatial and temporal error resilience.} + \label{fig2:ErrorPropagation} + %\cite{ErrorResilienceAug2005} \end{figure} -%\begin{figure}[tb] -% \centering -% \includegraphics[width=0.7\columnwidth]{./022_ErrorResilience/Figures/ErrorPropagation_1.png} -% \caption{Examples of spatial and temporal error propagation and how it can be reduce using localisation techniques \cite{ErrorResilienceAug2005}.} -% \label{fig2:ErrorPropagation} -%\end{figure} - -This group of error resilience techniques are able to reduce the error propagation at two levels, i.e., temporal and spatial levels. Figure~\ref{fig2:ErrorPropagation} is used to demonstrate the principle of error propagation and how it can be reduced using localisation techniques. The top of the figure illustrates a lost block and the correspondent spatial and temporal error propagation on the left and right, respectively. -Spatial localisation techniques -To reduce the error propagation within a frame a row of refresh blocks can be introduced to stop the error propagation. -To address the temporal error propagation refresh blocks (e.g., intra coded blocks) can be randomly introduce to stop error propagation. -As shown in the figure the refresh blocks are not affected error propagation. -In both examples of Figure~\ref{fig2:ErrorPropagation} the regions affected by errors are clearly reduced. - - \subsection{Error resilient rate-distortion optimisation} \glsreset{rd} @@ -102,56 +97,55 @@ Normally, such optimisation is equivalent to the minimisation of the Lagrangian \end{equation} where $D$ is the distortion of all pixels, $R$ the bitrate and $\lambda$ is the Lagrangian multiplier. Such optimisation only takes into account the coding distortion ($D$), which is only suitable for error-free environments. -Therefore, alternative approaches error resilient \ac{rd} optimisation, which take into consideration possible network losses~\cite{DistortionTrackingH264_ICIP2004}. -More advanced approaches can take advantage of a \ac{rope} of the end-to-end distortion at the encoder-side~\cite{ROPEJun2000}. +However, some alternative approaches with error resilient \ac{rd} optimisation take into consideration possible network losses~\cite{DistortionTrackingH264_ICIP2004}. +Other advanced approaches can take advantage of a \ac{rope} of the end-to-end distortion at the encoder-side~\cite{ROPEJun2000}. Let $f_t^i$ denote the original value of pixel $i$ in frame at instant $t$, $\hat f_t^i$ denote its reconstruction in the encoder and $\tilde f_t^i$ the reconstruction in the decoder after \ac{ec}. Since data loss may occur in the transmission channel, $\tilde f_t^i$ is modelled as a random variable at the encoder. The expected \ac{mse} of each pixel is given by: \begin{equation} -E\left\{d_t^i\right\} = E\left\{ \left( f_t^i - \tilde f_t^i\right)^2 \right\} = (f_t^i)^2 - 2 f_t^i E\left\{ \tilde f_t^i \right\} + E\left\{ \left(\tilde f_t^i\right)^2 \right\} +E\left\{d_t^i\right\} = E\left\{ \left( f_t^i - \tilde f_t^i\right)^2 \right\} = (f_t^i)^2 - 2 f_t^i E\left\{ \tilde f_t^i \right\} + E\left\{ \left(\tilde f_t^i\right)^2 \right\}. +\end{equation} +The packet loss probability ($p$) is taken into consideration and used in the distortion measurement. +In case of intra-coded pixels the following equations are used: +\begin{equation}\label{EQ2:E2EDIntra} +E\left\{ \tilde f_t^i \right\} = \left(1-p\right) \hat f_t^i + p E\left\{ \tilde f_{n-1}^i \right\}, +\end{equation} +\begin{equation} +E\left\{\left( \tilde f_t^i \right)^2 \right\} = \left(1-p\right) \left( \hat f_t^i \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\}, +\end{equation} +while in the case of inter-coded pixels: +\begin{equation} +E\left\{ \tilde f_t^i \right\} = \left(1-p\right) \left( \hat e_t^i + E\left\{ \tilde f_{n-1}^{j} \right\} \right) + p E\left\{ \tilde f_{n-1}^i \right\}, +\end{equation} +\begin{equation}\label{EQ2:E2EDInter} +E\left\{\left( \tilde f_t^i \right)^2 \right\} = \left(1-p\right) \left( \left( \hat e_t^i \right)^2 + 2 \hat e_t^i E\left\{ \tilde f_{n-1}^{j} \right\} + E\left\{ \left( \tilde f_{n-1}^{j} \right)^2 \right\} \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\}, \end{equation} -Then the packet loss probability ($p$) is taken into consideration and used in the distortion measurement. -In case of intra-coded pixels the following equation used: -\begin{align}\label{EQ2:E2EDIntra} -E\left\{ \tilde f_t^i \right\} &= \left(1-p\right) \hat f_t^i + p E\left\{ \tilde f_{n-1}^i \right\} \\ -E\left\{\left( \tilde f_t^i \right)^2 \right\} &= \left(1-p\right) \left( \hat f_t^i \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\}, -\end{align} -while in the case inter-coded pixels: -\begin{align}\label{EQ2:E2EDInter} -E\left\{ \tilde f_t^i \right\} &= \left(1-p\right) \left( \hat e_t^i + E\left\{ \tilde f_{n-1}^{j} \right\} \right) + p E\left\{ \tilde f_{n-1}^i \right\} \\ -E\left\{\left( \tilde f_t^i \right)^2 \right\} &= \left(1-p\right) \left( \left( \hat e_t^i \right)^2 + 2 \hat e_t^i E\left\{ \tilde f_{n-1}^{j} \right\} + E\left\{ \left( \tilde f_{n-1}^{j} \right)^2 \right\} \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\} -\end{align} assuming that the pixel $i$ is predicted from pixel $j$ in the previous frame. -The prediction error $e_t^i$, is quantized to the value $\hat e_t^i$, which is transmitted to the decoder along with the motion information. -Equations \ref{EQ2:E2EDIntra} and \ref{EQ2:E2EDInter} provide a measure of the distortion taking into consideration not only the coding distortion but also the possibility of frame loss and consequent \ac{ec} distortion. +The prediction error $e_t^i$ is quantised to the value $\hat e_t^i$, which is transmitted to the decoder along with the motion information. +Equations \ref{EQ2:E2EDIntra} to \ref{EQ2:E2EDInter} provide a measure of the distortion, taking into consideration not only the coding distortion but also the possibility of frame loss and consequent \ac{ec} distortion. -This technique was used in \cite{ROPEJun2000} to switch between intra and inter coding modes with the aim of reducing the temporal dependencies, and, thus achieving enhanced error robustness in video transmission. -In \cite{AdvancesROPEHE2642007} this technique was optimized to quarter pixel estimation, and then in \cite{ROPEBurst2009} the method was extended for burst losses, by introducing the burst loss probability into the original end-to-end estimation. +This technique was used in \cite{ROPEJun2000} to switch between intra and inter coding modes with the aim of reducing the temporal dependencies, thus achieving enhanced error robustness in video transmission. +In \cite{AdvancesROPEHE2642007} this technique was optimised to quarter pixel estimation, and in \cite{ROPEBurst2009} the method was extended for burst losses, by introducing the burst loss probability into the original end-to-end estimation. Results revealed that higher video quality was achieved by considering both probabilities. In \cite{E2EDistortion2011} a temporal \ac{ec} is considered in the distortion estimation, which increases the performance of the proposed method, by avoiding frame-copy that provides a lower bound on reconstruction quality. More advanced temporal \ac{ec} methods were also considered in the work presented in~\cite{EndToEndDistortionConcealment2014}. The \acs{mv} used for motion compensation reconstruction is obtained using boundary matching search, which is able to achieve lower distortion when compared with frame-copy. Using more advanced \ac{ec} algorithms leads to a more accurate estimation of the end-to-end distortion, thus increasing the error robustness of the transmitted video. -To take into account the delay introduced in the encoding loop due to hierarchical prediction structures, delay based \ac{rd} optimisation was proposed in \cite{DelayRD_TCVT2015} by estimating both the distortion and the end-to-end delay which is minimised along the required bitrate. Using this approach the total delay can be controlled without severely compromising the coding efficiency. -In the context of \ac{hevc} end-to-end distortion estimation was also used in the \ac{rd} optimisation \cite{ErrorResilienceModeHEVC2016} showing significant quality improvements comparing with the reference implementation. +To take into account the delay introduced in the encoding loop due to hierarchical prediction structures, delay based \ac{rd} optimisation was proposed in \cite{DelayRD_TCVT2015} by estimating both the distortion and the end-to-end delay that is minimised along the required bitrate. Using this approach, the total delay can be controlled without severely compromising the coding efficiency. +In the context of \ac{hevc} end-to-end distortion estimation was also used in the \ac{rd} optimisation showing significant quality improvements when compared to with the reference implementation \cite{ErrorResilienceModeHEVC2016}. -Different quality metrics can also be used for end-to-end distortion estimation. The method presented in \cite{SSIMModeling2015} proposes the distortion estimation using the \ac{ssim} \cite{QualitySSIMRef2002} in order to increase the correlation between the estimated distortion and the perceptual quality noticed by the human visual system~\cite{VisualQualityMetrics2011}. Moreover, the \ac{ssim} quality metric in learning based \ac{rd} modelling~\cite{RateDistortionNN_VCIP2017}, using a \acs{cnn} to predict both the distortion and the required rate, showing good accuracy in the \ac{rd} cost estimation. +Different quality metrics can also be used for end-to-end distortion estimation. The method presented in \cite{SSIMModeling2015} proposes the distortion estimation using the \ac{ssim} \cite{QualitySSIMRef2002} in order to increase the correlation between the estimated distortion and the perceptual quality noticed by the human visual system~\cite{VisualQualityMetrics2011}. Moreover, the \ac{ssim} quality metric was also used in learning based \ac{rd} models~\cite{RateDistortionNN_VCIP2017}, using a \ac{cnn} to predict the end-to-end distortion and the required rate, showing good accuracy in the \ac{rd} cost estimation. \subsection{Intra-refreshing} -In order to reduce the effects of error propagation within a single frame spatial and temporal dependencies can be limited by introducing refresh points into the bitstream by forcing intra-coded blocks\cite{IntraRefresh1999,IntraRefreshAttentionCM2007,IntraRefreshRefSelectionJan2009,IntraRefreshICIP2009,SandroIntraRefresh2011,JointIntraRefresh2014,IntraRefreshFeedback2015,IntraRefreshRPS2017}. +In order to reduce the effects of error propagation within a single, frame spatial and temporal dependencies can be limited by introducing refresh points into the bitstream, by forcing intra-coded blocks~\cite{IntraRefresh1999,IntraRefreshAttentionCM2007,IntraRefreshRefSelectionJan2009,IntraRefreshICIP2009,SandroIntraRefresh2011,JointIntraRefresh2014,IntraRefreshFeedback2015,IntraRefreshRPS2017}. +Figure~\ref{FIG2:IntraRefresh} (a) shows an example where intra-coded block are forced at random locations (dark gray sqares) in order to stop error propagation. This technique is fairly efficient to improve the error robustness of video streams, by avoiding long error propagation both within a single frame but also across several subsequent frames. The main characteristic of this group of techniques is to increase the number of intra-coded blocks as they have no dependencies to previous frames. However, this is done at the expense of higher amount of bits~\cite{IntraRefreshAttentionCM2007}. Therefore, different alternatives have been proposed, which optimise the number of intra-coded blocks, which can be introduced randomly, periodically~\cite{SandroIntraRefresh2011}, based on the motion information or rate-distortion optimisation~\cite{IntraRefreshICIP2008,HEVCIntraRefreshICIP2017}. Moreover, whenever is possible the encoder can use a feedback channel to decide where refresh regions should be introduced~\cite{IntraRefresh1999,IntraRefreshFeedback2015}. -\begin{figure}[tb] - \centering - \input{./022_ErrorResilience/Figures/IntraRefresh.tikz} - \caption{Examples of spatial and temporal error propagation and how it can be reduce using localisation techniques.} - \label{FIG2:IntraRefresh} -\end{figure} A non-normative method for random intra refreshing is implemented in the \ac{h264} reference software, which introduces cyclic refresh points using a random pattern. Figure~\ref{FIG2:IntraRefresh} (a) shows an example of intra-coded blocks (dark gray squares) being randomly places into the coded frames to prevent error propagation. @@ -162,17 +156,24 @@ The decision is performed based on the ratio between the Lagrangian cost of the Whenever this ratio is lower than a certain predefined threshold an intra mode is selected, which guarantees that intra refreshing is only forced when the Lagrangian cost of the intra mode is close to the optimal coding method. This is further extended to adapt the intra refresh algorithm to the network conditions, i.e., the threshold value is obtained from the bitrate and the loss ratio~\cite{IntraRefreshICIP2009}. Therefore, the number of intra-coded block automatically increases as the bitrate and the loss ratio increases. -In the context of \ac{hevc} intra refresh is also being used to reduce error propagation. +In the context of \ac{hevc}, intra refresh is also being used to reduce error propagation. Soft-refresh points using joint intra-inter prediction were proposed for this purpose~\cite{HEVCIntraRefreshICIP2017}. -Although the image regions cannot be correctly decoded in case of errors (as happens with intra-refresh), this method significantly decreases the bitrate increase due to extra intra-coded blocks. +Although the image regions cannot be correctly decoded in case of errors (as happens with intra-refresh), this method significantly reduces the bitrate increase due to extra intra-coded blocks. When compared with traditional intra-refresh, bitrate savings of up to 12\% are achieved for a 5\% \gls{plr}. +\begin{figure}[tb] + \centering + \input{./022_ErrorResilience/Figures/IntraRefresh.tikz} + \caption{Examples of intra-refreshing methods.} + \label{FIG2:IntraRefresh} +\end{figure} + \glsreset{roi} A different approach is to introduce refresh points based on the video content. In~\cite{IntraRefreshAttentionCM2007} the error resilience of video streams is increased by combining rate control and attention based models to define \acs{roi}. A weighting factor is used in the Lagrangian optimization to increase the number of intra-coded blocks in the image regions determined by the attention models. -Figure~\ref{FIG2:IntraRefresh} (b) shows an example with a \ac{roi} marked with a black line which varies for every frame. In this case the intra-coded blocks are only introduced inside the marked region, in contrast to what is shown in Figure~\ref{FIG2:IntraRefresh} (a). +Figure~\ref{FIG2:IntraRefresh} (b) shows an example with a \ac{roi} marked with a black line, which varies for every frame. In this case, the intra-coded blocks are only introduced inside the marked region, in contrast to what is shown in Figure~\ref{FIG2:IntraRefresh} (a). Results show that using the aforementioned method, less refresh points are required to improve the error robustness of the \ac{roi}. When comparing with \cite{DistortionTrackingH264_ICIP2004}, approximately 4 dB of \ac{psnr} gains are achieved in the \ac{roi} without compromising the overall video quality. A more recent approach was proposed using a perceptual noticeable difference model to estimate the noticeable distortion, and then using a visual attention model \cite{SaliencyItti1998} to decide which blocks should be used as intra refresh in the bitstream~\cite{IntraRefreshPerceptual2010}. This method is able to improve the results of a previous work \cite{IntraRefresh2005}, especially for high motion sequences. This is due to the fact that such previous approach only used an estimation of the error propagation without accounting for perceptual information. @@ -184,7 +185,7 @@ The simulation results reveal an increasing on the achieved video quality when h \subsection{Leaky-predictions} \label{SECTION2:LeakyPred} -Directly using reconstructed past frames as reference frames for predictive coding, may result in substantial error propagation whenever the reference frames are either lost or reconstructed from other distorted frames. +Directly using reconstructed past frames as reference frames for predictive coding, may result in substantial error propagation, whenever the reference frames are either lost or reconstructed from other distorted frames. To reduce error propagation one can employ leaky predictions, which scales down reconstructed frames before being used as reference~\cite{LeakyPred2009,ROILeakyPred2009}. This leads to an exponential decay of error propagation, with direct impact on error robustness. In this approach, the reconstructed reference frames are scaled down by applying the following: @@ -192,7 +193,8 @@ In this approach, the reconstructed reference frames are scaled down by applying \check f_t^i = \alpha \hat f_t^i + (1- \alpha)C, \end{equation} where $\check f_t^i$ is a reference frame, $\alpha$ is the leaky factor used to control the amount of information from the reconstructed frame present in the new frame $\check f_t^i$, and $C$ is a constant value. -Leaky prediction has been used in the past for error control, especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in Equation~(\ref{EQ2:LeakyPredConstant}) is replaced by a lower layer error-free reconstructed frame. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate as shown in \cite{LeakyPredConstant1973}. The $\alpha$ parameter allows to control the effect of the constant value, and thus the coding efficiency. By decreasing $\alpha$ the coding efficiency increases, however, the exponential decay in error propagation becomes less significant. +Leaky prediction has been used in the past for error control, especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in Equation~(\ref{EQ2:LeakyPredConstant}) is replaced by a lower layer error-free reconstructed frame. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate, as shown in \cite{LeakyPredConstant1973}. The $\alpha$ parameter allows to control the effect of the constant value, and thus the coding efficiency. +By decreasing $\alpha$, the coding efficiency increases, however, the exponential decay in error propagation becomes less significant. % MHMCP_TCSVT2006 @@ -219,13 +221,13 @@ In Equation (\ref{EQ2:LeakyPredDual}) $\hat f_{n}^i$ and $\hat f_{n-1}^i$ always \end{figure} -In \cite{ROILeakyPred2009} a method is presented where the leaky prediction is confined to a pre-defined \ac{roi}, in order to increase the error robustness with lower impact on the coding efficiency. -Figure \ref{fig2:ErrorPropagationROI} shows an example of error propagation within two different regions: \ac{roi} and background. As show in the figure, since the \ac{roi} uses background regions for predictions error propagation occurs across regions. +In \cite{ROILeakyPred2009} a method is presented, where the leaky prediction is confined to a pre-defined \ac{roi}, in order to increase the error robustness with lower impact on the coding efficiency. +Figure \ref{fig2:ErrorPropagationROI} shows an example of error propagation within two different regions: \ac{roi} and background. +As show in this figure, since the \ac{roi} uses background regions for predictions, error propagation occurs across regions. To overcome this problem, the method in \cite{ROILeakyPred2009} uses a weighted prediction for the \ac{roi} region obtained from the following predictions: (i) a globally optimal prediction obtained by unrestricted motion estimation, (ii) a prediction obtained by restricting motion estimation within the \ac{roi} region. This limits the impact of errors in the background region on the \ac{roi}. Moreover, since the leaky predictions are only confined to some specific regions, higher coding efficiency is achieved when compared with other methods. -%\HERE The method proposed in \cite{ROPERefGenJan2010} extends the traditional leaky prediction expressed in Equation (\ref{EQ2:LeakyPredDual}) by replacing the previous encoded frame ($\hat f_{n-1}^i.$) with the previous interpolated frame ($\tilde f_t^i$). This results in the following: \begin{equation}\label{EQ2:LeakyPredIIR} @@ -233,7 +235,7 @@ The method proposed in \cite{ROPERefGenJan2010} extends the traditional leaky pr \end{equation} This corresponds to infinite impulse response filter. Comparing both equations for the same $\alpha$, the Equation (\ref{EQ2:LeakyPredIIR}) implies stronger filtering than Equation (\ref{EQ2:LeakyPredDual}). Therefore, reference frames generated with heavier filtering have higher resilience against error propagation, but less correlation with the original frame, which negatively impacts inter predictions and hence coding efficiency. -The idea of combining multiple predictions was also been used in the \ac{hevc} standard by combining it with end-to-end distortion to select the optimal predictions~\cite{DualFrameME2015}. +The idea of combining multiple predictions was also used in the \ac{hevc} standard by combining it with end-to-end distortion to select the optimal predictions~\cite{DualFrameME2015}. \subsection{Reference picture selection} @@ -243,26 +245,31 @@ In \cite{ResilienceDualFrameBuffer2004}, an optimisation algorithm was devised t The same method can also be used in combination with feedback information from the network to improve the references frame selection. In the decoder-side, an \ac{ec} method was later used to take advantage of such prediction structure \cite{ConcealmentDualFrameBuffer2005}. -\begin{figure}[tb] -\centering -\includegraphics[width=0.95\columnwidth]{./022_ErrorResilience/Figures/RefPicSelection.png} -\caption{Examples of dynamic reference picture selection used in error resilient video transcoding \cite{ErrorResilienceRPSFeb2009}.} -\label{fig2:ReferencePictureSelection} -\end{figure} + A feedback information from the network was also used in \cite{ErrorResilienceRPSFeb2009}, to reduce error propagation at end-user decoders, by dynamically selecting the reference frame, only from those correctly received. Figure~\ref{fig2:ReferencePictureSelection} shows an example of an error resilience transcoding scheme, where the original temporal predictions (represented by the solid arrows) are transcoded to use only correctly received frames as reference, i.e., prediction represented by the dashed arrow. This is done based on network feedback. -A more advanced scheme may be considered, where only the regions affected by mismatch predictions are processed in the compressed domain, and subsequently transcoded to limit the dependencies from lost frames. To achieve low complexity \ac{mv} concatenation can be used, in order to obtain new inter-predictions for error-free frames \cite{MVConcatenationAug2011}. +A more advanced scheme may be considered, where only the regions affected by mismatch predictions are processed in the compressed domain, and subsequently transcoded to limit the dependencies from lost frames. +To achieve low complexity, \ac{mv} concatenation can be used, in order to obtain new inter-predictions for error-free frames \cite{MVConcatenationAug2011}. + +\begin{figure}[tb] + \centering + \includegraphics[width=0.95\columnwidth]{./022_ErrorResilience/Figures/RefPicSelection.png} + \caption{Examples of dynamic reference picture selection used in error resilient video transcoding \cite{ErrorResilienceRPSFeb2009}.} + \label{fig2:ReferencePictureSelection} +\end{figure} + In \cite{ChannelAdaptiveVideoStreaming2002} network feedback is combined with path diversity to select the reference frames that are more likely to be received at the decoder. In the proposed approach, the feedback information is not directly used to select the reference frames. Instead, the reference frames are selected through an end-to-end distortion estimation process, taking into account the coding distortion and distortion resulting from channel errors (\ac{ec} distortion is also taken into account). -In \cite{ErrorResilienceModeHEVC2017} a \ac{roi} is defined based on the motion objects to limit the regions where the mechanism of reference picture selection is applied. Intra refresh is also used based on the temporal distance to the previous intra-refresh frame. This method not only achieves significant quality gains when compared with the reference \ac{hevc} standard but also when comparing with tradition reference selection schemes. - +In \cite{ErrorResilienceModeHEVC2017} a \ac{roi} is defined based on the motion objects to limit the regions where the mechanism of reference picture selection is applied. +Intra refresh is also used, based on the temporal distance to the previous intra-refresh frame. +This method not only achieves noticeable quality gains, when compared with the reference \ac{hevc} standard, but also when comparing with traditional reference selection schemes. In~\cite{IntraRefreshRefSelectionJan2009} an algorithm to introduce intra refresh points at the \ac{mb} level was proposed. This method takes into consideration the decoding distortion assuming that frame-copy concealment is applied for \ac{ec} when packet loss occurs. The refresh blocks are encoded using either intra prediction or inter prediction through a secondary algorithm to select the most reliable reference frame (i.e., frame with less total expected distortion). -In the methods mentioned above the dependencies between the frames are analysed and reduced in order to decrease the error propagation in cases of data loss. +In the methods mentioned above the dependencies between the frames are analysed and reduced in order to decrease the error propagation, in cases of data loss. Since the existing video coding standards are strongly based on predictive coding, these methods present an efficient approach in highly erroneous conditions. % However, the method in~\cite{RDReferenceSelectionJul2012} uses more complex distortion estimations, and in ideal conditions multiple loop encoders are needed to select the reference frames optimally. However, although these closed-loop techniques may offer a reliable control of the decoder distortion, the feedback required from the receiver side and multipath networking are not always available neither feasible in real-time and unidirectional channels (e.g. broadcast). @@ -271,41 +278,43 @@ However, although these closed-loop techniques may offer a reliable control of t %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\section{Data partition and redundant coding} +\section{Data partitioning and redundant coding} \label{Section22:DataPartitionRedundant} In a compressed video streams different symbols have different relevance in terms of their impact on error propagation. Therefore, there is an unequal contribution of different syntax elements to the decoding distortion by error propagation, depending on the lost symbols. -Data partition techniques have been developed to provide a mechanism to cope with the unequal importance of coded information by grouping different parts of a compressed bitstream according to their importance to the decoded quality. -Then, some of these groups can be protected more efficiently than others using \ac{uep} schemes. These are normally associated with \ac{fec} algorithms, whereby multi-levels of \ac{fec} are applied to achieve different levels of error protection. This is presented and discussed with more detail in Sub-section~\ref{SEC22:UEP} +Data partition techniques have been developed to provide a mechanism to cope with the unequal importance of coded information by grouping different parts of a compressed bitstream, according to their importance to the decoded quality. +Thus, some of these groups can be protected more efficiently than others using \ac{uep} schemes. These are normally associated with \ac{fec} algorithms, whereby multi-levels of \ac{fec} are applied to achieve different levels of error protection. This is presented and discussed with more detail in Sub-section~\ref{SEC22:UEP} In a single channel transmission, the most important data or syntax elements (e.g., headers) can be protected with stronger channel codes than the remaining ones. In case of multi-channel communications, the important data can be routed over the most reliable channels. -Moreover, to achieve higher quality in multi-channel transmission one can use extra information to increase the error resilience of video streams, in order to increase the probability of being able to decode then in case of errors. -For instance, redundancy can be added either in an explicit form, e.g., redundant picture, or implicit, e.g., multiple description coding \cite{MDC1993}. Section~\ref{SEC22:RedundantPics}~and~\ref{SEC22:MDC} present a detailed review of these two approaches, respectively. +Moreover, to achieve higher quality in a multi-channel transmission one can use extra information to increase the error resilience of video streams, in order to increase the probability of being able to decode them in case of errors. +For instance, redundancy can be added either in an explicitly (e.g., redundant picture) or implicitly (e.g., multiple description coding). Section~\ref{SEC22:RedundantPics}~and~\ref{SEC22:MDC} present a detailed review of these two approaches, respectively. -\subsection{Data partitioning and flexible macroblock ordering} +\subsection{Standard methods for data partitioning and flexible macroblock ordering} \glsreset{fmo} -The concept of data partitioning was firstly introduced in the H.263 coding standard to split the coded data into different categories\cite{H264OverIPJul2003}. It was further improved in the next standard and it allows the symbols in each slice to be divided into three partitions: +The concept of data partitioning was firstly introduced by the H.263 coding standard to group the coded data into different categories~\cite{H264OverIPJul2003}. +It was further improved, allowing the symbols in each slice to be divided into three partitions: \begin{itemize} - \item Type A: headers, \ac{mb} types, quantization parameters and motion information. + \item Type A: headers, \ac{mb} types, quantisation parameters and motion information. \item Type B: intra-coded information, such as, intra blocks signalling and intra coefficients. This type cannot be used without the partitions of type A being present. - \item Type C: inter-coded information and inter coefficients. This type also requires the information of type A but can be decoder without intra-coded information (type B). + \item Type C: inter-coded information and inter coefficients. This type also requires the information of type A, but can be decoded without intra-coded information (type B). \end{itemize} -These partitions have different levels of importance, being data partitions of type A the most important as they carry the header information which are required by the remaining partitions. Then, as partitions of type B carry intra-coded information they are more important than type C information as it can stop temporal error propagation. +These partitions have different levels of importance, being data partitions of type A the most important as they carry the header information that are required by the remaining partitions. Then, as partitions of type B carry intra-coded information they are more important than type C information, as it can stop temporal error propagation. -The \ac{h264} introduced an error resilience tool named \ac{fmo} technique, which allows each \ac{mb} to be freely assigned to a specific slice group (i.e., a set of slices), organized in a mixed-up fashion~\cite{H264FMO2006}. -The standard supports up to eight slice groups in each picture, and the \acs{mb} within a group can be assigned to several slices in the default raster scan order. +The \ac{h264} introduced an error resilience tool named \ac{fmo} technique, which allows each \ac{mb} to be freely assigned to a specific slice group (i.e., a set of slices), organised in a mixed-up fashion~\cite{H264FMO2006}. +It supports up to eight slice groups in each picture, and the \acs{mb} within a group can be assigned to several slices in the default raster scan order. The case where only one slice group is used corresponds to the case where the \ac{fmo} tool is disabled. \ac{fmo} increases the error resilience by partitioning the bitstream into several groups, which is able to reduce the error propagation. Results show that the use of \ac{fmo} contributes to enhance the subjective and objective quality, especially for high packet loss rates at cost of an affordable overhead~\cite{H264FMO2006}. -Six types of slice groups are supported by \ac{fmo}. Figure~\ref{fig2:FMOTypes} shows \ac{fmo} types 0 to 5. -In the \ac{fmo} type 0 each slice group is defined by a maximum number of \acs{mb}, and a new slice group beings whenever this maximum value is reached. The same slice group can be cyclic repeated within a frame. +Six types of slice groups are supported by \ac{fmo}. +Figure~\ref{fig2:FMOTypes} shows \ac{fmo} types 0 to 5, since the type 6 as no defined pattern. +In the \ac{fmo} type 0 each slice group is defined by a maximum number of \acs{mb}, and a new slice group begins whenever his maximum value is reached. The same slice group can be cyclic repeated within a frame. \ac{fmo} type 1 corresponds to scattered slices, where the \acs{mb} are assigned to each slice group with a fixed formula. Figure~\ref{fig2:FMOTypes} represents the \ac{fmo} type 1 with two slice groups resulting in a checkerboard pattern. -In the \ac{fmo} type 2 slice groups are defined by rectangular shapes bounded by the top-left and bottom-right corners. +In the \ac{fmo} type 2, slice groups are defined by rectangular shapes bounded by the top-left and bottom-right corners. Although the slice groups are allowed to overlap each other, each \ac{mb} may only belong to one slice group. \ac{fmo} type 3, 4 and 5 are special types, each one allowing the configuration to change on every frame. Nevertheless, only two slice groups are allowed in \ac{fmo} type 3 to 5, in order to keep the signalling overhead affordable. The last \ac{fmo} type is the most generic case, where the slice groups are explicitly defined using high level packets (i.e., signalling \ac{nal} units), allowing more flexibility at the cost of higher signalling overhead. @@ -320,19 +329,22 @@ The last \ac{fmo} type is the most generic case, where the slice groups are expl In the past, \ac{fmo} has been widely used to increase the error robustness of video transmissions in different communication scenarios \cite{FMOErrorResilience2007,FMOResilience2010,FMOResilience2011,FMOROI2012,MotionEnergyDataPart2014}. In ~\cite{FMOErrorResilience2007} \ac{fmo} is used to increase the \ac{ec} performance and thus reduce the error propagation in prioritised video transmission. -The method divides the \acs{mb} into the two slice groups based on an estimation of the \ac{ec} distortion using a two-pass approach. Then, the slice group which has the most impact on the \ac{ec} distortion is transmitted using a reliable channel and it is assumed to be received error-free. +The method divides the \acs{mb} into the two slice groups based on an estimation of the \ac{ec} distortion using a two-pass approach. Then, the slice group which has the most impact on the \ac{ec} distortion is transmitted using a reliable channel, and it is assumed to be received error-free. Results show that the proposed approach is able to outperform the dispersed mode (\ac{fmo} type 1). Slice classification can also be based on the weighted sum of the motion vector information and the rate-distortion cost~\cite{FMOResilience2010}. -As in \cite{FMOErrorResilience2007}, the \acs{mb} are ranked based on the importance and distributed in the two groups with the same amount of blocks. In order to reduce the complexity of mapping the \acs{mb} into groups the same map can be repeated for several subsequent P-frames \cite{FMOResilience2011}. +As in \cite{FMOErrorResilience2007}, the \acs{mb} are ranked based on the importance and distributed in the two groups with the same number of blocks. +In order to reduce the mapping complexity of the \acs{mb} into groups, the same map can be repeated for several subsequent P-frames \cite{FMOResilience2011}. This reduces the computational complexity of the \ac{mb} grouping while achieving similar error resilience performance. -The method in \cite{FMOROI2012} splits the video frames into two regions, which are then grouped in different slice groups.To classify the different \acs{mb}, spatial information is extracted from the \ac{mb} bit allocation, and temporal information is also extracted based on the estimation of the \ac{ec} distortion. This technique is able to reduce the number of erroneous \acs{mb} up to 82\% achieving a quality gain of up to 3.97\%, in terms of \ac{psnr}, in comparison with the case where \ac{fmo} is not used. +The method in \cite{FMOROI2012} splits the video frames into two regions, which are then grouped in different slice groups. +To classify the different \acs{mb}, spatial information is extracted from the \ac{mb} bit allocation and temporal information is extracted based on the estimation of the \ac{ec} distortion. +This technique is able to reduce the number of erroneous \acs{mb} up to 82\%, achieving a quality gain of up to 3.97\%, in terms of \ac{psnr}, in comparison with the case where \ac{fmo} is not used. Motion energy was also used to classify \acs{mb} into different groups \cite{MotionEnergyDataPart2014}, by comparing the motion energy of a given \ac{mb} with the motion energy of its neighbours and the average motion energy of the entire frame. This allowed the authors to obtain a dynamic threshold which is then used to determine the \ac{mb} importance. In comparison with other \ac{uep} error protection schemes, this method achieve good quality improvements under different \acs{plr}. -In \cite{FMORedudantSlices_TB2007} a new \ac{fmo} type is proposed combining the slice groups type 1 and 3 in order to obtain a checkerboard pattern in a box-out scanning order, with the objective of increasing the scattering of lost \acs{mb}. Results show that the proposed \ac{fmo} type is able to outperform the existing types under single and burst loss events. -The method described in \cite{FMOTypeROI2014} introduces a new type in order to efficiently divide the video content of each frame in two regions with different levels of importance, based on the \ac{fmo} type 6. +In \cite{FMORedudantSlices_TB2007}, a new \ac{fmo} type is proposed combining the slice groups of type 1 and 3 in order to obtain a checkerboard pattern in a box-out scanning order, with the objective of increasing the scattering of lost \acs{mb}. Results show that the proposed \ac{fmo} type is able to outperform the existing types under single and burst loss events. +The method described in \cite{FMOTypeROI2014} introduces a new type in order to efficiently divide the video content of each frame in two regions with different levels of importance, based on the \ac{fmo}~type~6. %Then, a spiral protection is applied to scan the \acs{mb} grouping then in two groups. -By proposing a new \ac{fmo} type they are able to signal several \ac{roi} areas achieving lower overhead than using the original \ac{fmo} Type 6. +By proposing a new \ac{fmo} type they are able to signal several \ac{roi} areas achieving lower overhead than using the original \ac{fmo} type 6. \subsection{Unequal error protection} @@ -341,47 +353,47 @@ By proposing a new \ac{fmo} type they are able to signal several \ac{roi} areas \glsreset{uep} \Gls{uep} techniques are used to protect different parts of a video stream against errors according to their importance for the decoding quality. \ac{uep} approaches take advantage of the video stream characteristics that result from different types of content, both at the temporal and spatial level. This allows categorisation of coded data into different groups, by exploiting the fact that different regions are affected differently by transmission errors. -This not only ensures that those regions will be delivered with higher quality, but also leads to efficient use of the total overhead required to achieve increased the error robustness in video transmission. +This not only ensures that those regions will be delivered with higher quality, but also leads to efficient use of the total overhead required to achieve increased error robustness in a video transmission. -A wide variety of \ac{uep} schemes have been presented in the literature to optimize the performance of video communication over networks where the probability of packet loss is not negligible. -It has been observed that, within a \ac{gop}, different frame types (e.g. I, P, or B frames) have different levels of contribution to the reconstructed video quality when transmission losses occur. This has been exploited for \ac{uep} in video transmission as shown in~\cite{UEP_TM2005,UEPMotion_TM2006}. +A wide variety of \ac{uep} schemes have been presented in the literature to optimise the performance of video communication over networks where the probability of packet loss is not negligible. +It has been observed that, within a \ac{gop}, different frame types (e.g. I, P, or B frames) have different levels of contribution to the reconstructed video quality, when transmission losses occur. +This has been exploited for \ac{uep} in video transmission, as shown in~\cite{UEP_TM2005,UEPMotion_TM2006}. In \cite{UEPLayered_TB1999} an evaluation of layered video was proposed to increase the error robustness, by using redundant packets for helping the recovery of lost ones. In order to achieve reduced overhead for error protection, each video layer has a different importance and is protected using different levels of redundancy, thus guaranteeing that packets carrying data of lower layers have higher probability of being decoded without errors. -This is a simple approach, not dependent on the network capabilities, neither requiring a deep knowledge of the video contents. +This is a simple approach, not depending on the network capabilities, neither requiring a deep knowledge of the video contents. A lightweight model is used to assign packet priorities to be used by \ac{uep} algorithms~\cite{UEP_TCE2011}. Such model is based on a weighted sum of four components: (i) a slice type parameter giving higher importance to refresh and reference frames; (ii) a binary flag indicating whether a network packet transports video stream headers, i.e., \ac{nal} or slice header; (iii) a parameter indicating the distance in packets to the next slice; (iv) a parameter indicating the temporal distance to the next refresh frame. -These parameters can be obtained by parsing the \gls{nal} header, which makes it suitable to low-end encoders or even to prioritise each packet at a streaming level. +These parameters can be obtained by parsing the \gls{nal} header, which makes it suitable to low-end decoders or even to prioritise each packet at a streaming level. -However, \ac{uep} techniques usually require a deep video analysis which in general contributes to increase the encoding complexity. +However, \ac{uep} techniques usually require a deep video analysis, which in general contributes to increase the encoding complexity. Nevertheless, one can use information already needed for encoding, such as motion, to classify different frames or even different regions within a frame~\cite{UEPMotion_TM2006}. -The motion information is used to determine the motion energy which is given by the L2-norm of the \acs{mv} within a predictive regions. +The motion information is used to determine the motion energy, which is given by the L2-norm of the \acs{mv} within a predictive regions. For instance, in~\cite{UEPMotionEnergy_WCNC2013} the motion energy of each frame is computed and compared with neighbouring frames. The energy of co-located block in adjacent frame is compared, and higher importance is given to those frames that have higher motion energy differences. This method was compared with equal error protection, resulting in higher objective and subjective quality and revealing the relation between motion information and impact on errors. -This work was extended in ~\cite{UEPMotion_ICSPCC2012,UEPMotion_ICSPCS2013} to use a two levels \ac{uep} approach, where besides taking into consideration the \ac{mb} and frame importance, the combined importance of different frames is also used to determine the level of importance of each \ac{gop}. +This work was extended in~\cite{UEPMotion_ICSPCC2012,UEPMotion_ICSPCS2013} to use a two-level \ac{uep} approach, where besides taking into consideration the \ac{mb} and frame importance, the combined importance of different frames is also used to determine the level of importance of each \ac{gop}. Another work dealing with \ac{uep} is described in~\cite{Zhang20131390}, where the transmission distortion is estimated at the packet level, to assess the impact of each packet in the error propagation. -Then, different priorities are applied to each packet, allowing different levels of error protection. This was also used to schedule the packet transmission through multi-path networks\cite{PacketScheduling2010}. +In this case, different priorities are applied to each packet, allowing different levels of error protection. This was also used to schedule the packet transmission through multi-path networks\cite{PacketScheduling2010}. Furthermore, in order to improve the classification of different regions of a video stream a perceptual algorithm can be used. This strategy increases the correlation between the classification and subjective impact of errors. -In \cite{UEPSaliency_TCSVT2010} a method based on a combination of two parameters is described: a spatial parameter based on a perceptual attention model and a temporal parameter based on the frame position within the \ac{gop}. -These parameters are then used to improve the efficiency of \ac{fec} assignment, and experimental results show a significant quality improvement both using objective metrics and subjective evaluation. +In \cite{UEPSaliency_TCSVT2010} a method based on a combination of two parameters is described: a spatial parameter, based on a perceptual attention model, and a temporal parameter, based on the frame position within the \ac{gop}, which are used to improve the efficiency of \ac{fec} assignment. Experimental results show a significant quality improvement, both using objective metrics and subjective evaluation. \glsreset{el} -The scalable video coding framework provides implicit support for \ac{uep} through the layered structures defined with different levels of quality~\cite{UEPScalable_TCE2008,UEPScalable_TCSVT2010,UEPLayerVideo_TM2014, UEPScalable_ICSPICES2015}. -In~\cite{UEPScalable_TCSVT2010} an algorithm to recursively estimate the total expected distortion in the decoder at the picture-level for each layer is proposed. +The scalable video coding framework provides implicit support for \ac{uep} through the layered structures, defined with different levels of quality~\cite{UEPScalable_TCE2008,UEPScalable_TCSVT2010,UEPLayerVideo_TM2014, UEPScalable_ICSPICES2015}. +In~\cite{UEPScalable_TCSVT2010}, an algorithm to recursively estimate the total expected distortion in the decoder at the picture-level for each layer is proposed. This method takes into consideration \ac{el} truncation, i.e., a specific layer being discarded, and \ac{ec} to measure an accurate estimation of the end-to-end distortion. The estimation is carried out for every layer in the video stream, in order to apply an optimisation of different levels of error protection across several layers. Also based on scalable coding, the work in~\cite{UEPLayerVideo_TM2014} proposes a technique for finding the optimal \ac{fec} coding rates, which allow for a correct \ac{el} decoding when errors affect the base layer. This prevents discarding correctly received \acs{el} frames due to missing dependencies, increasing the overall video quality. -In the context of \ac{hevc} there are also methods to achieve \ac{uep}. -Since techniques like \ac{fmo} are not available, different approaches were investigated \cite{SliceImportance_ICHNRSR2015,HEVCUepLte_IWCMCC2015,UEPVideoDegradation_ISCAS2017,UEP_TM2018} +In the context of \ac{hevc}, there are also methods to achieve \ac{uep}. +Since techniques like \ac{fmo} are not available, different approaches were investigated \cite{SliceImportance_ICHNRSR2015,HEVCUepLte_IWCMCC2015,UEPVideoDegradation_ISCAS2017,UEP_TM2018}. The work in~\cite{SliceImportance_ICHNRSR2015} proposes a resource allocation scheme based on prioritisation at the slice level (i.e., \ac{nal} unit level) for \ac{lte} Advanced networks~\cite{HEVCUepLte_IWCMCC2015}. The prioritisation is performed by analysing the inter-frame predictions dependencies, giving higher rankings to slices having higher number of dependent pixels. In~\cite{UEPVideoDegradation_ISCAS2017} a quality degradation model is proposed for \ac{hevc} video, by simulating error events and measuring the quality using the \ac{psnr} metric. -From the empirical results a model was devised to predict the quality under different events. +From empirical results, a model was devised to predict the quality under different events. Then, the model is used to control the amount of redundant codes to be applied at the network level. -The scalable extension of the \ac{hevc} standard was also used to support \gls{uep} for video storage in cloud environments~\cite{UEP_TM2018}, by combining the layer information (i.e., base or enhancement layer) and the picture type (i.e., I, P or B-frame) to maximise the performance of the \ac{fec} codes both at the temporal and quality/resolution levels. +The scalable extension of the \ac{hevc} standard was also used to support \gls{uep} for video storage in cloud environments~\cite{UEP_TM2018}, by combining the layer information (i.e., base or enhancement layer) and the picture type (i.e., I, P or B-frame) to maximise the performance of the \ac{fec} codes, both at the temporal and quality/resolution levels. @@ -393,22 +405,24 @@ The scalable extension of the \ac{hevc} standard was also used to support \gls{u \subsection{Redundant pictures} \label{SEC22:RedundantPics} -Redundant pictures is a well-known technique adopted in the \ac{h264} standard \cite{RedudantPicsJan2009}, which allows for different representations of the same coded picture to be transmitted using different encoding parameters. -For example, the primary picture may be coded with fine quantization, while the redundant one with coarse quantisation at lower bitrate. -On the one hand if the primary picture is received, the redundant information is discarded. -On the other hand if errors occur and the primary picture is lost, the redundant one is used to provide a lower level of reconstructed quality and limiting the error propagation. +A well-known technique adopted in the \ac{h264} standard is the use of redundant picture, which allows for different representations of the same coded picture to be transmitted using different encoding parameters~\cite{RedudantPicsJan2009}. +For example, the primary picture may be coded with fine quantisation, while the redundant one may use coarse quantisation resulting in a lower bitrate. +On the one hand, if the primary picture is received, the redundant information is discarded. +On the other hand, if errors occur and the primary picture is lost, the redundant one is used to provide a lower level of reconstructed quality and limiting the error propagation. -Zhu \emph{et al.} investigated the paradigm of redundant picture coding~\cite{RedudantPicsJan2009}, considering different approaches in order to develop a mixed method based on both reference picture selection to obtain a clean decoding without error propagation and hierarchical allocation of the redundant pictures to increase error robustness. -In~\cite{VPxResiliencePCS2016} the key frames are encoded and transmitted at lower bitrate to be used in case of error to replace the missing frame. +Zhu \emph{et al.} investigated the paradigm of redundant picture coding~\cite{RedudantPicsJan2009}, considering different approaches to develop a mixed method based on both reference picture selection to obtain a clean decoding, without error propagation, and hierarchical allocation of the redundant pictures to increase error robustness. +In~\cite{VPxResiliencePCS2016} the key frames are encoded and transmitted at a lower bitrate to be used in case of error, to replace the missing frame. Redundant pictures are a good mechanism to improve performance under error prone networking conditions, since they introduce secondary pictures that can be decoded when the primary picture is not available. Moreover, redundant pictures might also work as localisation techniques when different reference frames are selected. As the number of redundant pictures increases, the amount of mismatch temporal predictions in case of data loss is reduced, since the redundant picture can replace the erroneous ones. This achieves a similar goal as reducing the video stream dependencies, i.e., localisation techniques. The main drawback of such approaches is the large amount of redundant data and overheads that are required for their efficient implementation. In~\cite{RedundantMVsJune2008} a newer approach was proposed to reduce the amount of information needed to encode redundant pictures, by introducing only redundant \acs{mv}. -This was also used in~\cite{VPxResilience2016}, where the motion information is used to recover the missing frame. +This was also used in~\cite{VPxResilience2016}, where the motion information was used to recover the missing frame. These approaches increase the overall system performance by presenting an efficient trade-off between the redundant bitrate and the error robustness. + + \subsection{Multiple description coding} \label{SEC22:MDC} @@ -419,21 +433,22 @@ These approaches increase the overall system performance by presenting an effici \label{fig2:MDCSystem} \end{figure} -In \ac{mdc} schemes two or more bitstreams, named descriptions, are encoded and transmitted through different channels, such that an acceptable level of quality is achieved when only some of them reach the decoder \cite{MDC1993}. +In \ac{mdc} schemes two or more bitstreams, named descriptions, are encoded and transmitted through different channels, such that an acceptable level of quality is achieved when only some of them reach the decoder \cite{MDC1993,MDC2005}. Figure \ref{fig2:MDCSystem} illustrates a basic architecture of a \ac{mdc} encoder and decoder using two descriptions. -The encoder-side creates two descriptions, i.e., $R_1$ and $R_2$, which shall be transmitted using two different channels, reducing the probability of the same video data being lost at the same time in both descriptions. -As shown in the figure three different outputs may be produced at the decoder-side: both descriptions are correctly received and the central decoder is used, or either one of the descriptions is not received and then one of the side decoders is used. +In this example, the encoder-side creates two descriptions, i.e., $R_1$ and $R_2$, which shall be transmitted using two different channels, reducing the probability of the same video data being lost at the same time in both descriptions. +As shown in this diagram, three different outputs may be produced at the decoder-side: both descriptions are correctly received and the central decoder is used($D_0$), or either one of the descriptions is not received and then one of the side decoders is used ($D_{1,1}$ and $D_{1,2}$). When both bitstreams are received, the central decoder combines the information carried by each description achieving higher quality than any single side decoder. -In the \ac{mdc} encoder the redundancy among descriptions can be controlled by changing the correlation between them. Higher correlation leads to higher redundancy but also higher quality in the output of the side decoders. -In contrast with the case of redundant pictures, the redundant \ac{mdc} streams do not carry the same information which means that if both are received, then the decoded quality is increased. However, a combination of \ac{mdc} and redundant pictures can also be used to achieve increase quality gains \cite{MDCRedundantPicsJan2010}. -Different types of \ac{mdc} approaches \cite{MDC2005,MDCHierarBFeb2012,MDCCorreiaJun2012} can be used to improve the quality under transmission errors, either by taking advantage of hierarchical B-pictures~\cite{MDCHierarBFeb2012} or by introducing extra auxiliary information to limit the error propagation in case of single description decoding~\cite{MDCCorreiaJun2012}. -Although these techniques can be very effective in multi-path networks, a significant amount of overhead is required to create the different descriptions\cite{MDCCorreiaJun2012}. - +In the \ac{mdc} encoder the redundancy among descriptions can be controlled by changing the correlation between them. Higher correlation leads to higher redundancy, but also higher quality in the output of the side decoders. +In contrast with the case of redundant pictures, the redundant \ac{mdc} streams do not carry the same information, which means that if both are received, then the decoded quality may be increased. +However, a combination of \ac{mdc} and redundant pictures can also be used to achieve higher quality gains \cite{MDCRedundantPicsJan2010}. +Different types of \ac{mdc} approaches \cite{MDCHierarBFeb2012,MDCCorreiaJun2012} can be used to improve the quality under transmission errors, either by taking advantage of hierarchical B-pictures~\cite{MDCHierarBFeb2012} or by introducing extra auxiliary information to limit the error propagation in case of single description decoding~\cite{MDCCorreiaJun2012}. +Although these techniques can be very effective in multi-path networks, a significant amount of overhead is required to create the different descriptions~\cite{MDC_TCSVT2011}. +% Double check this reference %\HERE -\cite{MDCRedundantPicsJan2010} -\cite{MDC_TCSVT2011} +%\cite{MDCRedundantPicsJan2010} +%\cite{MDC_TCSVT2011} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% @@ -443,7 +458,7 @@ Although these techniques can be very effective in multi-path networks, a signif %\glsreset{ec} -In general, \ac{ec} functions are implemented as post-processing step in the decoder and thus is the last module in the communication chain capable of handling errors~\cite{ConcealmentReview1998}. +In general, \ac{ec} functions are implemented as a post-processing step in the decoder and being the last module in the communication chain capable of handling errors~\cite{ConcealmentReview1998}. Thus, reconstruction of missing data should take place within the video decoder to minimise the perceived effects of lost data. %Overall, to enhance the video quality, the design of the transmission system must incorporate adequate \ac{ec} tools. This section reviews several \ac{ec} methods based on spatial and temporal redundancies, as these play an important role in the overall performance of robust communication systems. @@ -480,7 +495,7 @@ improve the estimation accuracy of the lost region, the edges and image transiti \label{FIG3:pixel_interpolation} \end{figure} -Directional interpolation improves the performance of the pixel recovery, by firstly estimating the lost edges using either correctly received or previously recovered pixels. In order to obtain the best direction to interpolate, edges are calculated using the image gradient, for example applying the Sobel operator \cite{SignalImageVideoProcessing2012}. +Directional interpolation improves the performance of the pixel recovery, by firstly estimating the lost edges, using either correctly received or previously recovered pixels. To obtain the best interpolation direction, edges are estimated using the image gradient, for example applying the Sobel operator \cite{SignalImageVideoProcessing2012}. %This operator gives the first-order approximation of the gradient of the image. This is widely used in image processing to locate the image edges, and uses an horizontal and vertical mask given by: %\begin{equation} %G_x = @@ -503,7 +518,8 @@ Directional interpolation improves the performance of the pixel recovery, by fir %\theta &= tan^{-1} \left( \frac{g_y}{g_x} \right).\\ %\end{aligned} %\end{equation} -The Sobel operator is an efficient algorithm to detect edges, as it can reach extreme values at the image edges, so it can be used in pixel interpolation. The usage of this operator for \ac{ec} was evaluated in \cite{IntraConc2010,FineDirectTCE2006,SpatialEdgeConsecutiveBlocks2004,SpatialEdgeModConsecutiveBlocks2005,SpatialInterpolation2008}, and it is characterised as a fast and efficient method due to its small kernel \cite{GradEstimation2009}. +The Sobel operator is an efficient algorithm to detect edges, as it can reach extreme values at the image edges. +The usage of this operator for \ac{ec} was evaluated in \cite{IntraConc2010,FineDirectTCE2006,SpatialEdgeConsecutiveBlocks2004,SpatialEdgeModConsecutiveBlocks2005,SpatialInterpolation2008}, and it is characterised as a fast and efficient method due to its small kernel \cite{GradEstimation2009}. % \abbrev{FC}{Frame-Copy} % \abbrev{MC}{Motion-Copy} @@ -514,14 +530,18 @@ The Sobel operator is an efficient algorithm to detect edges, as it can reach ex The conventional way to recover missing frames in the temporal domain are the so-called \ac{fc} and \ac{mc}. These \ac{ec} methods were implemented for the \ac{h264} reference software in \cite{H264RefConc2006}. -The \ac{fc} method consists in repeating (copying) the frame that has the shortest temporal distance in order to to the missing one, into the time instant where the loss occurred. +The \ac{fc} method consists in repeating (copying) the frame that has the shortest temporal distance, in order to substitute the missing one, into the time instant where the loss occurred. The \ac{mc} method uses the motion information of the closest frame to recover the missing one, thus the missing pixels are recovered through motion compensation. -Previous works and simulation results showed that the \ac{mc} method achieves higher quality in the reconstructed frames in most of the cases~\cite{2DRecursiveRefinement2010}. However, the complexity increases with the implementation of the \ac{mc} method, when compared with \ac{fc}. +Previous works and simulation results showed that the \ac{mc} method achieves higher quality in the reconstructed frames in most cases~\cite{BiDirMC2011,2DRecursiveRefinement2010}. However, the complexity increases with the implementation of the \ac{mc} method, when compared with \ac{fc}. -As an example, in \cite{2DRecursiveRefinement2010} a whole frame loss \ac{ec} algorithm was proposed to improve the performance of the \ac{mc} method, by using a refinement algorithm to improve the accuracy of the \acs{mv} obtained from the closest decoded frame. -Moreover, in this work the \acs{mv} between consecutive frames are assumed to be strongly correlated, so the motion information of several preceding frames is taken into account to estimate the lost one. Then, the \ac{mv} differences between the vectors of successive frame pairs are recursively calculated along with the next refinement, until the changes become negligibly small (e.g., close to zero). -This is shown to improve the performance of the \ac{mc} method, achieving gains up $0.47\ dB$ (\ac{psnr}) for a \ac{plr} of $20\%$. +In order to improve the \mbox{motion-copy} performance, two reference frames can be used as sources of \acs{mv} to reconstruct the missing one \cite{BiDirMC2011}. +The average value of two vectors is used to reconstruct the missing motion information. +Results in \cite{BiDirMC2011} show quality gains up to $4.99$~dB (\ac{psnr}) when compared with \ac{mc} method. +In \cite{2DRecursiveRefinement2010} the \ac{mc} method performance was increased by using a refinement algorithm to improve the accuracy of the \acs{mv} obtained from the closest decoded frame. +The \ac{mv} differences between the vectors of successive frame pairs are recursively calculated along with the next refinement, until the changes become negligibly small (e.g., close to zero). +This approach has shown to improve the performance of the \ac{mc} method, with gains up to $0.47\ dB$ (\ac{psnr}) for a \ac{plr} of $20\%$. +Moreover, \subsubsection{Boundary match algorithms} % \abbrev{ICMV}{Interpolated Candidate Motion Vectors} @@ -539,14 +559,16 @@ Then, the optimal reconstruction is selected according to the sum of differences \label{FIG3:BMAPixelInformation} \end{figure} -An implementation of the \ac{bma} was proposed in \cite{BMAAlgorithm2006}, by analysing the change of magnitude and angle between the \acs{mv} of the neighbouring blocks and the co-located ones in the previous frames. The average change is applied to the \ac{mv} of block co-located lost one to compute the \ac{mv} candidate for recovery. +An implementation of the \ac{bma} was proposed in \cite{BMAAlgorithm2006}, by analysing the change of magnitude and angle between the \acs{mv} of the neighbouring blocks and the co-located ones in the previous frames. The average change is applied to the \ac{mv} of the block co-located with the lost one to compute the \ac{mv} candidate for recovery. An improvement to \ac{bma} was obtained with the outer boundary matching algorithm\cite{OBMAMethod2008}, also known as decoder \ac{mv} estimation algorithm in \cite{DMVE2000}. -In this method the distortion is calculated by the difference of two outer boundaries of reconstructed block, instead of internal and external boundary in \ac{bma}. -Although \ac{bma} and outer \ac{bma} have the same design principle of minimising the mismatch distortion, in latter the distortion is measured in the outer boundaries, \emph{i.e.}, external boundaries of the reconstructed block. It was observed in the experimental results \cite{OBMAMethod2008} that outer \ac{bma} has better performance in video recovery than conventional \ac{bma}. The performance improvements are explained by the distortion minimisation of the outer boundary, which incorporates the edge information located in the neighbourhood of the lost block in the decision process to select the most accurate \ac{mv}. +In this method the distortion is calculated by the difference of two outer boundaries of reconstructed block, instead of an internal and external boundary in \ac{bma}. +Although \ac{bma} and outer \ac{bma} have the same design principle of minimising the mismatch distortion, in the latter the distortion is measured in the outer boundaries, \emph{i.e.}, external boundaries of the reconstructed block. +It was observed in the experimental results of \cite{OBMAMethod2008} that the outer \ac{bma} has better performance in video recovery than conventional \ac{bma}. The performance improvements are explained by the distortion minimisation of the outer boundary, which incorporates the edge information located in the neighbourhood of the lost block in the decision process to select the most accurate \ac{mv}. %Moreover, in the method presented in \cite{OBMAMethod2008} further \ac{mv} candidates were introduced, a refined local search is performed and multiple boundary layers, \emph{i.e.}, increase the overlapped region to measure the distortion and to select the final MV, were tested. The experimental results revealed that the usage of a single layer match gives better performance than multiple layers, while selective search gives excellent trade-off between the image recovery and computation complexity % In \cite{DynamicBMA2010} a novel approach to measure the spatial distortion on boundaries of the reconstructed block is presented. Instead of using only horizontal and vertical directions, diagonal and anti-diagonal are also tested, and the best direction is chosen from the reference frame. Since block boundaries may change abruptly, instead of minimising the distortion function the proposed method aims to approximate this distortion to an optimal difference. This optimal difference are determined in the neighbourhood of the co-located block in the reference frame. In \cite{BMAAlgorithm2008} a novel distortion function was proposed, to choose among different candidate MVs. The distortion function exploits both spatial and temporal smooth constraints. The temporal function is based on the outer boundary distortion, as in \cite{OBMAMethod2008}. -The spatial distortion aims to reduce the gradient variances in boundaries of the reconstructed block, so the image edges may be guaranteed. Several experimental results were obtained for different loss events, covering from isolated block lost to the loss of a complete row of blocks. The proposed method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method. Moreover, subjective analysis reveal that this method recover the edges in the lost image. However, the method uses complex mathematical operations, which may not be feasible in real time applications. +The spatial distortion aims to reduce the gradient variances in boundaries of the reconstructed block, so the image edges may be guaranteed. +Several experimental results were obtained for different loss events, covering from an isolated block loss to the loss of a complete row of blocks. The proposed method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method. Moreover, subjective analysis reveal that this method recover the edges in the lost image. However, the method uses complex mathematical operations, which may not be practical in real time applications. \subsubsection{Motion vector extrapolation} @@ -570,13 +592,13 @@ As mentioned before, the \ac{mc} method is a simple way to reconstruct an estima -\ac{mve} \cite{BlockBasedMVE2002,BlockBasedMVEOverllaped2011} methods are implemented quiet simple yet efficient techniques that reconstruct the \acs{mv} of a lost frame which in turn are used for motion compensation. +\ac{mve} \cite{BlockBasedMVE2002,BlockBasedMVEOverllaped2011} methods are quiet simple yet efficient techniques that reconstruct the \acs{mv} of a lost frame, which in turn are used for motion compensation. These \acs{mv} are extrapolated from the last decoded frame \mbox{(reference frame)} onto the missing one~\cite{BlockBasedMVE2002}. Each block in the reference frame is projected to the missing one with its own direction given by the corresponding extrapolated \ac{mv} ($mv'$). An example of \ac{mve} is shown in Figure~\ref{FIG3:MVEExtrapolation}, where the vectors of frame $t-1$ are projected onto the missing frame at instant $t$. Note that, in the reference frame ($f_{t-1}$) the \acs{mv} may point to different frames ($f_{t-2}$~and~$f_{t-3}$), previously decoded, so \ac{mve} must take into account the temporal distance between different frames. -To improve the accuracy of \ac{mve}, the damaged region is divided smaller blocks, and according to the matching area between the extrapolated block and the damaged one, the best \ac{mv} is chosen. +To improve the accuracy of \ac{mve}, the damaged region is divided smaller blocks and, according to the matching area between the extrapolated block and the damaged one, the best \ac{mv} is chosen. In order words, the \ac{mv} for the each block is obtained from the extrapolated block with the highest number of pixels matching the missing locations. -Note that the final motion information need to be aligned to a fixed size grid, e.g., $4\times4$ grid, which leads to a misalignment of the extrapolated \acs{mv}. +Note that the final motion information need to be aligned to a fixed sise grid, e.g., $4\times4$ grid, which leads to a misalignment of the extrapolated \acs{mv}. Alternatively, in \cite{PixelBasedMVE2004} several blocks are extrapolated to the missing frame, resulting in overlapped regions, which are recovered using a weighted average of the multiple pixels candidates. The weight coefficients assume a maximum value in the center of the extrapolated region, and a minimum value at the boundaries. % @@ -635,20 +657,20 @@ Overall, the aforementioned methods rely on post-processing without any assistan The last category of techniques for robust video transmission address those algorithms that help the \ac{ec} process at the decoder-side. Although such techniques might overlap with some of the concepts described above, such as, data partition or redundant information, they have the main goal of improving the \ac{ec} performance. This class of algorithms combine the error resilience capabilities with \ac{ec} methods to achieve an improved overall performance. -One of the approaches that can be used to improve the \ac{ec} performance is to estimate, at the encoding time, the \ac{ec} distortion introduced by the decoder in case of data loss. +One of the approaches that can be used to improve the \ac{ec} performance is to estimate, at the encoding side, the \ac{ec} distortion introduced by the decoder in case of data loss. In \cite{ConcealmentDrivenSliceOrdering2008} the \ac{ec} distortion estimation is used for slice classification and \ac{uep}. Such distortion is also used to optimise motion estimation~\cite{ConcealmentAwareME2006}, by including the distortion of the subsequent frame (assuming it is lost and recovered) in the \ac{rd} cost optimisation of the current frame. This was also used in the context of \ac{hevc} to improve its error robustness, as shown in \cite{ErrorResilienceModeHEVC2016}. This kind of approaches take advantage of location techniques and data partitioning to improve the reconstruction quality achieved by \ac{ec} algorithms. Normally, they use a single \ac{ec} method that is implemented both at the encoder and decoder, based on temporal algorithms, e.g., \ac{mc} and \ac{mve}. -Another approach is the use \ac{ec} \ac{mv} which are vectors that might be carried by intra-coded units for the purpose of error recovery. When an error occurs in a given coding unit, the \ac{ec} \ac{mv} from the neighbouring units can be used to form a prediction from previously received frames. This concept was exploited in \cite{ConcealmentIntraDataHidng2009} by using data-hiding to carry additional motion information in the \ac{dct} coefficients. This improves the reconstruction quality and also adds complexity from the decoder to the encoder. +Another approach is the use \ac{ec} \acs{mv}, which are vectors that might be carried by intra-coded units for the purpose of error recovery. When an error occurs in a given coding unit, the \ac{ec} \ac{mv} from the neighbouring units can be used to form a prediction from previously received frames. This concept was exploited in \cite{ConcealmentIntraDataHidng2009} by using data-hiding to carry additional motion information in the \ac{dct} coefficients. This improves the reconstruction quality, but transfers the complexity from the decoder to the encoder. A different approach to improve the \ac{ec} performance was presented in \cite{PixelInterlacingIntraConcealment2011}. Pixel interlacing is used to divide the frame into different packets in order to enhance the performance of intra-frame \ac{ec} by using linear interpolation. This technique reduces the probability of burst losses, therefore, the neighbouring pixels surrounding the lost regions are available to be used as reference for the concealment operation. -The above methods facilitate the \ac{ec} performance, either through slice reordering or based on \acs{mv}, but they always rely on the assumption that a fixed \ac{ec} method is used at the decoder. +These methods improve the \ac{ec} performance, either through slice reordering or based on \acs{mv}, but they always rely on the assumption that a fixed \ac{ec} method is used at the decoder. In \cite{ResilienceRECAP_ICASSP2009} a low-resolution video is transmitted alongside the main bitstream in order to be used as reference for \ac{ec}. Such redundant stream is used by the decoder to improve the estimation accuracy of motion information. This method was extended in \cite{SaliencyErrorConcealment_TM2013} by using asymmetric coding of the redundant information based on Itti's attention model~\cite{SaliencyItti1998}. -The work presented in \cite{ConcealmentSignalling2008,BurstConcealmentSignalling2008} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder, using \ac{roi} to identify the most sensitive image area. +The work presented in \cite{ConcealmentSignalling2008,BurstConcealmentSignalling2008} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder, using a \ac{roi} to identify the most sensitive image area. Although such an approach achieves reasonably good performance, it relies on the spatial neighbouring information, which may not be available in case of full frame loss. Moreover, it requires transmission of the \ac{roi} map to the decoder, which increases the overall bitrate. -More recently a \ac{ec} signalling methods was proposed for scalable video \cite{SignallingRobustScalable2015}, in order to select the best frame to replace the missing one. +More recently an \ac{ec} signalling methods was proposed for scalable video \cite{SignallingRobustScalable2015}, in order to select the best frame to replace the missing one. -Summarising, \ac{ec}-aware error techniques allow the combination of efficient algorithms, e.g., redundant pictures or \ac{ec}-aware motion estimation or embedded information, with algorithms at the decoder which take advantages of such extra information. Such techniques are able to achieve quality improvements in comparison with error recovery approaches at the decoder on its own. Although effective techniques have been proposed in the past, the exploration of dynamic techniques which explore the potential of different \ac{ec} techniques with adaptable decoder have not been fully exploited. +Summarising, \ac{ec}-aware error techniques allow the combination of efficient algorithms, e.g., redundant pictures, \ac{ec}-aware motion estimation or embedded information, with algorithms at the decoder that take advantages of this extra information. Such techniques are able to achieve quality improvements in comparison with error recovery approaches at the decoder on its own. Although effective techniques have been proposed in the past, the exploration of dynamic techniques, which explore the potential of different \ac{ec} techniques with adaptable decoder, have not been fully exploited. @@ -659,8 +681,8 @@ Summarising, \ac{ec}-aware error techniques allow the combination of efficient a \section{Summary} \label{Section22:Summary} -This chapter presented an overview of the different error resilience techniques proposed over the last years, based on their main characteristics. Different approaches were described covering techniques for robust video transmission, like dependency reduction at the encoder-side and \ac{ec} by post-processing at the decoder. The techniques described in this chapter provide relevant insights for the methods investigated in this work. -Different algorithms were considered and combined to increase the error robustness of highly efficient coded video and improve the reconstruction quality in case of errors. +This chapter presented an overview of the different error resilience techniques proposed over the last years, based on their main characteristics. Different approaches were described covering techniques for robust video transmission, like dependency reduction at the encoder-side and \ac{ec}, by post-processing at the decoder. The techniques described in this chapter provide relevant insights for the methods investigated in this work. +Different algorithms were considered and combined to increase the error robustness of highly efficient coded video and improve the reconstruction quality in case of transmission errors. diff --git a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex index 2f19ed1..0584c5b 100644 --- a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex +++ b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex @@ -14,9 +14,9 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Introduction} -The use of \ac{hevc} in different type of application and video services over networks that are prone to errors and packet losses, requires mechanisms to minimise the effect o such probabilistic events. This chapter starts by presenting an evaluation study about the error resilience of \ac{hevc} in Section~\ref{Section3:ErrorResilienceEvaluation} , with the aim of defining the weak elements of coded streams and level of error robustness under different transmission loss conditions. The evaluation study covers different coding configuration to evaluate the individual impact of some new coding tools introduced, e.g., increased partition size, intra-refreshing, slice partitioning and advanced motion vector prediction. +The use of \ac{hevc} in different type of application and video services over networks that are prone to errors and packet losses, requires mechanisms to minimise the effect o such probabilistic events. This chapter starts by presenting an evaluation study about the error resilience of \ac{hevc} in Section~\ref{Section3:ErrorResilienceEvaluation} , with the aim of defining the weak elements of coded streams and level of error robustness under different transmission loss conditions. The evaluation study covers different coding configuration to evaluate the individual impact of some new coding tools introduced, e.g., increased partition size, intra-refreshing, slice partitioning and advanced motion vector prediction. In Section~\ref{Section3:DynamicMVRefreshHEVC} a novel method to increase the error robustness of the \ac{hevc} \ac{mv} coding is presented and evaluated against existing techniques. The method proposed introduces refresh point in the motion prediction used in the \ac{hevc}, which significant reduces the impact of mismatch \ac{mv} predictions. -Finally, Section~\ref{Section3:Summary} concludes this chapter. +Finally, Section~\ref{Section3:Summary} concludes the chapter. \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -27,7 +27,7 @@ Finally, Section~\ref{Section3:Summary} concludes this chapter. \subsection{Motivation and objectives} -As mentioned before the \ac{hevc} standard has improved its predecessor in terms of coding efficiency, +As mentioned before the \ac{hevc} standard has improved its predecessor in terms of coding efficiency, higher complexity and data dependencies, mostly resulting from advanced predictions structures. Although the increased complexity may be compensated with more efficient algorithms and faster hardware technologies, the impact of higher coding efficiency on reducing error robustness strongly affects the video quality under packet loss. @@ -35,12 +35,12 @@ A critical aspect related to the performance of robust video streaming methods i Since \ac{hevc} streaming is expected to use known transport technologies, such as RTP~\cite{HEVCStreamMay2012}, MPEG-2 TS~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016} and MPEG Media Transport\cite{ATSCMMT2016}, the error detection capabilities of the underlying transport protocols have significant impact on the overall system performance. In general, the error detection operation at the transport layers is decoupled from the video coding layer. To cope with transmission errors the encoding side implements robust coding methods, while the efficient recovery of lost video data is implemented in the decoder through error concealment algorithms. -Between them, signalling between different protocol layers ensures that the decoder is given the necessary information to identify the lost slices/frames in the video stream. -Therefore, the error robustness of video streams can be investigated on its own, assuming any possible form of streaming technologies and error detection. +Between them, signalling between different protocol layers ensures that the decoder is given the necessary information to identify the lost slices/frames in the video stream. +Therefore, the error robustness of video streams can be investigated on its own, assuming any possible form of streaming technologies and error detection. This is the main focus of the study presented in this section. -There are some studies presented in the past regarding the error robustness of the \ac{hevc} standard which are worth to be mention as they provide relevant insights for the research of carried out in this research work. In~\cite{StudyHEVCErrorsDez2012,HEVCWireless_2016} both the coding efficiency and error robustness of the \ac{hevc} are evaluated and compared against the \ac{h264}. Experimental results confirm that \ac{hevc} has reduced error robustness despite its increased coding efficiency. -A subjective evaluation study is presented in~\cite{SubjectiveEvalSep2013,QoEHEVCMay2014} revealing that packet loss ratios higher than $3\%$ have significant effect on the perceived video quality. +There are some studies presented in the past regarding the error robustness of the \ac{hevc} standard which are worth to be mention as they provide relevant insights for the research of carried out in this research work. In~\cite{StudyHEVCErrorsDez2012,HEVCWireless_2016} both the coding efficiency and error robustness of the \ac{hevc} are evaluated and compared against the \ac{h264}. Experimental results confirm that \ac{hevc} has reduced error robustness despite its increased coding efficiency. +A subjective evaluation study is presented in~\cite{SubjectiveEvalSep2013,QoEHEVCMay2014} revealing that packet loss ratios higher than $3\%$ have significant effect on the perceived video quality. %In both works it is concluded that the robustness of \ac{hevc} decreases for sequences complex motion. The vulnerability of \ac{mv} prediction was investigated and its impact on the error robustness was evaluated \cite{DynamicTMVP2011}. Moreover, a method to introduce \ac{mv} refresh points was proposed, achieving average \ac{psnr} gains of approximately 3~dB. Summarising, the decrease in error robustness is mostly due to the strong data dependencies imposed by the highly complex prediction modes of \ac{hevc}. Since these are selected according to a rate distortion optimisation criterion, assuming error-free transmission, the decoder faces a different problem whenever errors or data loss occur in the transmission network \cite{ROPE2007}. @@ -79,12 +79,12 @@ The values of \ac{si} and \ac{ti} calculated for the test sequences are shown in For the simulation carried out in this study the HM reference software, version $16.2$, was used \cite{HEVCReferenceSoftware162}. The sequences were encoded with an \ac{idr} period of $16$ frames and all the coding modes enabled, following two recommended configurations: \ac{ld}, i.e., P-frames with 4 references frames, and \ac{ra}, i.e., B-frames with 2 reference frames in each list~\cite{HEVCCommonTests2012}. These prediction structures cover the use of P-frames and hierarchical B-frames~\cite{H264ScalableOverviewSep2007}, which are commonly used in HEVC streaming~\cite{RealTimeHEVC2016}. -In order to achieve high quality images for all coded sequences (i.e., \ac{psnr} around 40 dB) different bitrates were used, as listed in Table~\ref{TableTestSequences}. The filtering operations were disabled at slices boundaries in order to keep the slices self-contained. +In order to achieve high quality images for all coded sequences (i.e., \ac{psnr} around 40 dB) different bitrates were used, as listed in Table~\ref{TableTestSequences}. The filtering operations were disabled at slices boundaries in order to keep the slices self-contained. Moreover, the ratio of inter-coded blocks of each coded sequence using the default coding parameters is also shown in the third column of the table to illustrate the predictions distribution. %The following bitrates were selected to achieve high quality (i.e., around 40 dB): 1.0 Mbits/s (Kendo), 1.5 Mbits/s (Book Arrival), 4.0 Mbits/s (BQSquare, Park Scene and Race Horses) and 12.0 Mbits/s (People on Street). -% At the decoder side, error concealment is based on motion compensation from previously decoded frames. +% At the decoder-side, error concealment is based on motion compensation from previously decoded frames. % The motion field of a missing frame is obtained applying \ac{mv} extrapolation from the two closest neighbours. % Afterwards, the final \acs{mv} are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}. To simulate the lossy transmission environment, each frame is encoded into several slices, each one packetized into one \ac{nal} unit, which are then transmitted as the payload of independent packets. @@ -97,17 +97,17 @@ The filtering operations were disabled at slices boundaries in order to keep the \subsection{Evaluation of the impact of the partition units size} -%The \ac{hevc} standard introduced new partitioning schemes to allow coding improvements in high resolution videos. These partition structures, named \glsfirst{cu}, are allowed to be as large as $64\times64$ pixels. +%The \ac{hevc} standard introduced new partitioning schemes to allow coding improvements in high resolution videos. These partition structures, named \glsfirst{cu}, are allowed to be as large as $64\times64$ pixels. In the first set of experiments the impact of the maximum \ac{cu} size in the decoded video quality is investigated. In this experiment the maximum \ac{cu} size is set to three different sizes, i.e., 16$\times$16, 32$\times$32 and 64$\times$64 pixels. The lower size is targeting the \ac{mb} size used in the \ac{h264}. -Table~\ref{Table3:Quality_BlockSize} shows the average \ac{psnr} obtained for the three test conditions and for four different packet loss ratios. Results for the error-free scenario shows a consistent quality increase as the maximum \ac{cu} size increases, up to a maximum of 1.41 dB for the Tennis sequence when comparing the maximum \ac{cu} size of $64\times64$ with $16\times16$. +Table~\ref{Table3:Quality_BlockSize} shows the average \ac{psnr} obtained for the three test conditions and for four different packet loss ratios. Results for the error-free scenario shows a consistent quality increase as the maximum \ac{cu} size increases, up to a maximum of 1.41~dB for the Tennis sequence when comparing the maximum \ac{cu} size of $64\times64$ with $16\times16$. This confirms the principle used by the \ac{hevc} standard, that allowing larger coding units lead to higher coding efficiency, achieving a maximum of 1.41~dB for the Tennis sequence which has the higher \ac{ti} parameter (see Table~\ref{TableTestSequences}). Moreover, it is also noticeable that sequences with higher amount of inter-coded blocks take less advantage of the larger \acs{cu}, by the results of the BQSquare sequence (i.e., highest ratio of inter-coded blocks and lowest quality gain in the error-free case). -Overall, comparing the results for the error-free and data loss cases it is noticeable that similar quality gains are obtained, which are achieved due to the superior coding efficiency of larger \acs{cu}. In most of the cases the quality gains slightly decrease for increasing \acs{plr}. -Therefore, one may conclude that the error robustness of \ac{hevc} streams is not affected by limiting the maximum \ac{cu} size. +Overall, comparing the results for the error-free and data loss cases it is noticeable that similar quality gains are obtained, which are achieved due to the superior coding efficiency of larger \acs{cu}. In most of the cases the quality gains slightly decrease for increasing \acs{plr}. +Therefore, one may conclude that the error robustness of \ac{hevc} streams is not affected by limiting the maximum \ac{cu} size. %\todo{Complete discussion of partition unit size} -%Results for the case of data loss indicate that using larger coding units leads to higher video quality. +%Results for the case of data loss indicate that using larger coding units leads to higher video quality. % %that larger coding units should be maintained. Even though larger frame partitions lead to larger coding dependencies, it will not affect the robustness to packet loss and error propagation. This is mainly due to the fact that error propagation occurs mostly in complex and moving regions, which normally are coded using smaller partition units. @@ -119,7 +119,7 @@ Therefore, one may conclude that the error robustness of \ac{hevc} streams is no \input{./031_ErrorResilienceEvaluation/QualityEvaluation/TableCUSizes.tex} \end{table} -Figure~\ref{Figure3:BlockPartitionBask} illustrates an example of the frame partitions for a region of the Basketball Drill sequence for two cases: (i) maximum \ac{cu} size of $64\times64$ and (i) maximum \ac{cu} size of $16\times16$. +Figure~\ref{Figure3:BlockPartitionBask} illustrates an example of the frame partitions for a region of the Basketball Drill sequence for two cases: (i) maximum \ac{cu} size of $64\times64$ and (i) maximum \ac{cu} size of $16\times16$. As shown, in the moving regions, e.g., moving player, the partitions obtained for both cases have a similar pattern, mainly composed of \acs{cu} with small size. Larger \acs{cu} are only used in the floor which has a lower spatial complexity and suffers small changes between frames, therefore, easier to compress with $64\times64$ \acs{cu}. Moreover, those regions can be recovered more accurately using neighbouring information (spatial and temporal) correctly decoded, thus not incurring in severe error propagation. Therefore, the overall error robustness is not affected by reducing the maximum \ac{cu} size. One may conclude that the use of larger coding units is able to improve the overall \ac{hevc} efficiency in error-free and error prone applications. @@ -156,8 +156,8 @@ Therefore, the overall error robustness is not affected by reducing the maximum \subsection{Evaluation of the impact of intra refreshing} -In this sub-section the relation between the temporal distance of consecutive intra-coded frames, i.e., period of the intra frames, and the video quality under transmission errors is evaluated. -As the intra-coded frames have no dependencies to prior coded pictures, they introduce a refreshing point into the bitstream, which stops temporal error propagation along several frames or slices. +In this sub-section the relation between the temporal distance of consecutive intra-coded frames, i.e., period of the intra frames, and the video quality under transmission errors is evaluated. +As the intra-coded frames have no dependencies to prior coded pictures, they introduce a refreshing point into the bitstream, which stops temporal error propagation along several frames or slices. This allows the decoder to recover from frame/slice loss and continue error-free decoding seamlessly. \begin{figure}[!t] @@ -168,21 +168,21 @@ This allows the decoder to recover from frame/slice loss and continue error-free \label{Figure3:QualityvsIntra} \end{figure} -Figure~\ref{Figure3:QualityvsIntra} shows the average decoded video quality (\ac{psnr}) for different distance of intra frames, covering \acs{plr} between 3\% and 5\%. -Results in the figure indicate that as the distance between intra frames increases, the decoded video quality significantly decreases. +Figure~\ref{Figure3:QualityvsIntra} shows the average decoded video quality (\ac{psnr}) for different distance of intra frames, covering \acs{plr} between 3\% and 5\%. +Results in the figure indicate that as the distance between intra frames increases, the decoded video quality significantly decreases. The quality decreasing is especially noticeable for intra periods higher than 32 frames, indicating that temporal distances greater than approximately one second are not suitable for video streaming under error-prone conditions. -When a high temporal distance between intra frames is used, the quality decreasing is more noticeable in the People On Street and BQSquare sequences, which have higher spatial complexity (see values for \ac{si}), leading to a reduction of 13 dB and 14 dB, respectively. As spatial complexity increases the encoder rely more on temporal predictions making the bitstream more prone to error propagation. +When a high temporal distance between intra frames is used, the quality decreasing is more noticeable in the People On Street and BQSquare sequences, which have higher spatial complexity (see values for \ac{si}), leading to a reduction of 13 dB and 14 dB, respectively. As spatial complexity increases the encoder rely more on temporal predictions making the bitstream more prone to error propagation. %\todo{Complete discussion of intra refreshing} -One should also note from the results of Figure~\ref{Figure3:QualityvsIntra} that using a small distance between intra frames is not always more efficient. -Since intra frames require higher amount of bits, the overall coding efficiency is reduced, resulting in an overall quality that decreases even in data loss conditions. -This is clear noticeable for the Four People sequence, where increasing the intra period from 8 to 16 leads to a quality increasing of 1 dB for 5\% of \ac{plr}. +One should also note from the results of Figure~\ref{Figure3:QualityvsIntra} that using a small distance between intra frames is not always more efficient. +Since intra frames require higher amount of bits, the overall coding efficiency is reduced, resulting in an overall quality that decreases even in data loss conditions. +This is clear noticeable for the Four People sequence, where increasing the intra period from 8 to 16 leads to a quality increasing of 1 dB for 5\% of \ac{plr}. For the remaining sequences it can be concluded that using sixteen frames (not higher than half a second) between refresh points (i.e., intra frames) achieves an acceptable efficiency under errors. This provides an efficient trade-off between compression efficiency and error robustness in case of data loss. \subsection{Evaluation for using different slice partitioning schemes} -The partitioning of the \acs{ctu} into different slices increase the flexibility of video streams to be partitioned into different packets, thus adapting them to the video transmission system. -By partitioning the coded video frame into different slices spatial refresh points are introduced, limiting the error propagation across different \acs{ctu}. +The partitioning of the \acs{ctu} into different slices increase the flexibility of video streams to be partitioned into different packets, thus adapting them to the video transmission system. +By partitioning the coded video frame into different slices spatial refresh points are introduced, limiting the error propagation across different \acs{ctu}. In this sub-section the amount of bytes per slice is evaluated, in order to evaluate the influence of different schemes on the error robustness of \ac{hevc} streams. The quality obtained different mode is compared in order to find the most efficient that suits for video transmission under data loss conditions. \begin{figure}[!t] @@ -194,11 +194,11 @@ In this sub-section the amount of bytes per slice is evaluated, in order to eval \label{Figure3:SlicePartitioning} \end{figure} -Figure~\ref{Figure3:SlicePartitioning} shows the average \ac{psnr} for four difference slice partitioning modes and different \acs{plr}. -The first three modes are characterised by a different amount of bytes per slice (B/S), namely 1200, 2400 and 4800 bytes, respectively. In these cases each frame has a variable number of slices. -The last mode uses a single slice per frame (referred to as \textit{SS}), corresponding to the case where slice partitioning is disabled. +Figure~\ref{Figure3:SlicePartitioning} shows the average \ac{psnr} for four difference slice partitioning modes and different \acs{plr}. +The first three modes are characterised by a different amount of bytes per slice (B/S), namely 1200, 2400 and 4800 bytes, respectively. In these cases each frame has a variable number of slices. +The last mode uses a single slice per frame (referred to as \textit{SS}), corresponding to the case where slice partitioning is disabled. Results in the figure reveals a clear impact of the slice partitioning mode on the error robustness of the \ac{hevc}, as the quality degradation for increasing \ac{plr} significantly changes for each mode. -Comparing the cases with fixed amount of bytes per slice, it is clearly noticeable that when higher amount of bytes are used the quality degradation increases. This is true for the results of most of the test sequences shown in Figure~\ref{Figure3:SlicePartitioning}. +Comparing the cases with fixed amount of bytes per slice, it is clearly noticeable that when higher amount of bytes are used the quality degradation increases. This is true for the results of most of the test sequences shown in Figure~\ref{Figure3:SlicePartitioning}. Although the use of smaller slices increases the overhead used for slice headers, the superior error robustness of achieved with 1200 B/S is able to overcome the inherent degradation on coding efficiency. This is due to the lack of refresh points within a frame, resulting in errors that affects larger regions of the frame which are propagated to the subsequent inter predicted frames. @@ -210,7 +210,7 @@ Results for Kendo sequence does not follow the main trend. This can be explained \subsection{Motion vector coding impact on error propagation} -In \ac{hevc}, the new \ac{mv} prediction modes known as \ac{amvp} and Merge Mode~\cite{MotionCodingHEVC2011} may be significant contributors to poor performance under transmission errors. These modes are characterised by coding the motion information using previously encoded \acs{mv}, which obviously leads to prediction mismatches at the decoder side when such neighbouring information is missing due to network losses. +In \ac{hevc}, the new \ac{mv} prediction modes known as \ac{amvp} and Merge Mode~\cite{MotionCodingHEVC2011} may be significant contributors to poor performance under transmission errors. These modes are characterised by coding the motion information using previously encoded \acs{mv}, which obviously leads to prediction mismatches at the decoder-side when such neighbouring information is missing due to network losses. Therefore, in this sub-section the impact of using spatial and temporal \ac{mv} candidates on the decoded video quality under errors is evaluated. \subsubsection{Spatial \ac{mv} candidates} @@ -223,9 +223,9 @@ Therefore, in this sub-section the impact of using spatial and temporal \ac{mv} \label{Figure3:AvgPSNRMergeCandidates} \end{figure} -The first set of experiments aimed to investigate the impact of the amount of spatial \ac{mv} candidates in the video quality. +The first set of experiments aimed to investigate the impact of the amount of spatial \ac{mv} candidates in the video quality. As mentioned before, \ac{hevc} allows up to five candidates to be used, which can be provided solely by spatial neighbours. In order to check the impact of the \ac{mv} candidates, different coding configurations were used, each one characterised by different amounts of spatial \ac{mv} candidates, i.e., 1, 3 and 5 \acs{mv}. -Figure~\ref{Figure3:AvgPSNRMergeCandidates} shows the average quality results for the Basketball Drill and People On Street sequences under different packet loss conditions. +Figure~\ref{Figure3:AvgPSNRMergeCandidates} shows the average quality results for the Basketball Drill and People On Street sequences under different packet loss conditions. The results show that, besides the number of \ac{mv} candidates, similar quality decreasing is achieved as the \ac{plr} increases. Therefore, one may conclude that varying the number of available \ac{mv} candidate does not affect the error resilience of the \ac{hevc}. The occurrence rate for each spatial \ac{mv} candidate was analysed in order to check whether they are uniformly used. @@ -276,7 +276,7 @@ On the contrary, when the temporal \ac{mv} candidate is disabled, the motion inf In order to further evaluate the impact of the temporal \ac{mv} dependencies on the error propagation, the \ac{hevc} streams were subject to random packet losses. Table~\ref{Table3:AvgPSNRWithoutTMVP} presents the average quality results, measured using the \ac{psnr}. -As expected, the results indicate that in error-free conditions higher quality is achieved by enabling the \ac{tmvp} candidate, since higher compression efficiency is obtained because more \ac{mv} predictors are used. +As expected, the results indicate that in error-free conditions higher quality is achieved by enabling the \ac{tmvp} candidate, since higher compression efficiency is obtained because more \ac{mv} predictors are used. This is true for all both test conditions, \emph{i.e.}, both \ac{ld} and \ac{ra}. However, in the presence of errors, the video quality significantly decreases due to low error robustness when temporal \ac{mv} candidates are used. Alternatively, average quality up to 5.17 dB of \ac{psnr} when the temporal \ac{mv} predictors are disabled is achieved in the \ac{ld} configuration. The results also show that the negative impact of using temporal \ac{mv} dependencies is smaller when B-frames are used. For the Random-Access, the average gain of disabling the \ac{tmvp} is 0.78 dB. @@ -294,14 +294,14 @@ Moreover, the results also indicate that for higher loss ratios the negative imp \subsection{Remarks} -Overall, the results presented in the previous sub-sections show that \ac{hevc} has poor robustness against transmission errors, resulting in significant quality degradation, though dependent still on the video content and coding options. -As shown, some coding options, such as slice partition and intra refreshing period, have high correlation with the decoded video quality when transmission errors affect the compressed streams. +Overall, the results presented in the previous sub-sections show that \ac{hevc} has poor robustness against transmission errors, resulting in significant quality degradation, though dependent still on the video content and coding options. +As shown, some coding options, such as slice partition and intra refreshing period, have high correlation with the decoded video quality when transmission errors affect the compressed streams. Therefore, these are important aspects to take into consideration in the design an implementation of broadcasting video services and applications. -One of the main improvements of the \ac{hevc} standard is the support for larger coding units up to $64\times64$, which does not have a negative impact the error robustness of coded video streams, indicating that they should be allowed to achieve higher coding performance. +One of the main improvements of the \ac{hevc} standard is the support for larger coding units up to $64\times64$, which does not have a negative impact the error robustness of coded video streams, indicating that they should be allowed to achieve higher coding performance. Regarding the use of neighbouring information for \ac{mv} coding, on the one hand the spatial \ac{mv} candidates do not significantly decrease the error robustness. On the other hand, the use of temporal \ac{mv} candidates significantly contribute to increase the temporal dependencies, leading to more severe quality degradation in the presence of packet loss. -Based on these evidences, the next section presents an efficient method to improve the error robustness of \ac{hevc}. +Based on these evidences, the next section presents an efficient method to improve the error robustness of \ac{hevc}. The proposed method is based on the underlying principle of reducing the mismatch of \ac{mv} predictions, by selectively disable some \ac{mv} predictions at the encoder-side, reducing the overall temporal dependencies. @@ -313,19 +313,19 @@ The proposed method is based on the underlying principle of reducing the mismatc %\subsection{Problem formulation} -The aforementioned results revealed low error resilience in motion prediction because any error is easily propagated through several frames. +The aforementioned results revealed low error resilience in motion prediction because any error is easily propagated through several frames. Based on this evidence, this section proposes a method to make intelligent use of temporal \ac{mv} candidates during the motion estimation process, in order to decrease the temporal dependency, and improve the error resiliency without penalising the \ac{rd} performance. \subsection{Proposed method} \glsreset{rd} -The new approach to deal with temporal motion vector dependencies is accomplished by modifying the \ac{rd} optimisation that is used to select the best motion vector for inter-prediction. +The new approach to deal with temporal motion vector dependencies is accomplished by modifying the \ac{rd} optimisation that is used to select the best motion vector for inter-prediction. The objective is to increase the number of \acs{mv} that can be independently decoded to improve the error recovery capability after a frame loss event. The proposed method also overcomes the limitation of the method presented in \cite{DynamicTMVP2011}, which break the motion information dependencies between frames. -Instead of disabling the temporal motion candidate at the frame level, the proposed method aims to remove the dependencies at the \ac{cu} level. -To achieve this goal, the temporal dependency of all \ac{mv} candidates is firstly analysed for each block, in order to select on the suitable candidates to be used as predictors for the current \ac{mv}. +Instead of disabling the temporal motion candidate at the frame level, the proposed method aims to remove the dependencies at the \ac{cu} level. +To achieve this goal, the temporal dependency of all \ac{mv} candidates is firstly analysed for each block, in order to select on the suitable candidates to be used as predictors for the current \ac{mv}. Based on the analysis of the temporal dependency, some temporal candidates are marked as unsuitable predictors of the current \ac{mv}, and for this reason are not selected in the Merge mode. In the proposed approach, a temporal \ac{mv} candidate is marked as unsuitable based on the following criteria: \begin{enumerate} \item if it is encoded based on another temporal \ac{mv} candidate from a previously encoded frame; @@ -341,8 +341,8 @@ Based on the analysis of the temporal dependency, some temporal candidates are m \FIGSVSPACE \end{figure} -Figure \ref{ProposedDynamicMethod} can be used to explain the proposed method, illustrating different blocks and different \ac{mv} dependencies represented with arrows. Note that the squares illustrate a generic block size; the representation of the temporal dependencies is simplified, and in some cases may not correspond to the exact co-located block in the previous frame. In frame $f_{t-1}$ there are several \acs{mv} encoded using spatial predictions and one \ac{mv} encoded using a temporal prediction (arrow pointing to $f_{t-2}$). In frame $f_{t}$, the \ac{mv} corresponding to the block (1) may use the temporal \ac{mv} candidate, since the co-located block in $f_{t-1}$ is not temporally dependent. -However, in the proposed scheme the \ac{mv} prediction marked with the arrow (2) is not allowed, thus preventing the propagation of temporal dependencies. +Figure \ref{ProposedDynamicMethod} can be used to explain the proposed method, illustrating different blocks and different \ac{mv} dependencies represented with arrows. Note that the squares illustrate a generic block size; the representation of the temporal dependencies is simplified, and in some cases may not correspond to the exact co-located block in the previous frame. In frame $f_{t-1}$ there are several \acs{mv} encoded using spatial predictions and one \ac{mv} encoded using a temporal prediction (arrow pointing to $f_{t-2}$). In frame $f_{t}$, the \ac{mv} corresponding to the block (1) may use the temporal \ac{mv} candidate, since the co-located block in $f_{t-1}$ is not temporally dependent. +However, in the proposed scheme the \ac{mv} prediction marked with the arrow (2) is not allowed, thus preventing the propagation of temporal dependencies. Moreover, the block corresponding to the arrow (3) cannot use the temporal \ac{mv} candidate since it already depends on a \ac{mv} that was previously encoded using the temporal candidate. The proposed method reduces the temporal dependencies of the \glspl{mv} at the block level, providing improved error robustness to the encoded streams. Using this approach, the dependencies are selectively removed, improving the error recovery, without fully disabling the temporal \gls{mv} candidates for a given frame. Therefore, coding efficiency is not significantly penalised. @@ -361,8 +361,8 @@ The performance of the proposed scheme (\PROP) was evaluated against two other a \subsubsection{Error propagation under single loss events} -Firstly, the performance of the proposed method was measured under a single loss event (i.e., only one frame is affected by errors). -Figure \ref{Figure3:ProposedMethodEvaluationSingle} shows the error propagation for the different methods. +Firstly, the performance of the proposed method was measured under a single loss event (i.e., only one frame is affected by errors). +Figure \ref{Figure3:ProposedMethodEvaluationSingle} shows the error propagation for the different methods. The results show a significant quality difference between the proposed method, the \TIDR method, and the reference \ac{hevc}. When the temporal \ac{mv} candidate is disabled (proposed and \TIDR method) more error robustness can be achieved. The results also show the relevance of using the proposed method, as the \TIDR method is only able to recover from the frame loss after the refreshing frame, \emph{i.e.}, frame \#8, especially for the BQSquare sequence @@ -377,13 +377,14 @@ The results also show the relevance of using the proposed method, as the \TIDR m \input{./031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNRSTD.tex} \end{table} -Secondly, further simulations were performed to evaluate the effectiveness of the proposed error resiliency method under realistic packet loss patterns. +Secondly, further simulations were performed to evaluate the effectiveness of the proposed error resiliency method under realistic packet loss patterns. %As in previous experiments, each coded frame is transmitted in one packet. -Table~\ref{Table3:AverageStdPSNR} illustrates the average standard deviation of the PSNR, for three of the test sequences. -The results show that increasing the \ac{plr} leads to higher quality deviations, and the impact of errors in the subjective quality experience is higher. -However, reducing the temporal dependencies of the \acs{mv} using the proposed method decreases the quality differences variation, achieving the lowest value for all scenarios tested. The proposed method is able to reduce the quality difference variations up to 0.94 dB in Kendo sequence (3$\%$ of packet loss) compared to the~\TIDR~method. +Table~\ref{Table3:AverageStdPSNR} illustrates the average standard deviation of the PSNR, for three of the test sequences. +The results show that increasing the \ac{plr} leads to higher quality deviations, and the impact of errors in the subjective quality experience is higher. +However, reducing the temporal dependencies of the \acs{mv} using the proposed method decreases the quality differences variation, achieving the lowest value for all scenarios tested. The proposed method is able to reduce the quality difference variations up to 0.94 dB in Kendo sequence (3$\%$ of packet loss) compared to the~\TIDR~method. +Since the proposed method achieves the lowest \ac{psnr} standard deviation, one may conclude that it is able to reduce the quality variations leading to a better visual quality. -\HIGHLIGHT{vantagem do std} +%\HIGHLIGHT{vantagem do std} \begin{table}[tbp] \centering @@ -404,10 +405,10 @@ The effectiveness of the proposed method increases at higher packet loss rates ( \section{Summary} \label{Section3:Summary} -In this chapter, the error resilience performance of \ac{hevc} was studied in detail under error prone conditions. +In this chapter, the error resilience performance of \ac{hevc} was studied in detail under error prone conditions. The simulation results revealed that new coding tools introduced in \ac{hevc} to improve coding efficiency are also responsible for higher error propagation. This motivates further research on new techniques to be developed for highly efficient encoders and decoders, in order to increase error resiliency and to improve the overall performance of video broadcast. -Then, a new approach to deal with the weak robustness of \ac{mv} coding was proposed, addressing \ac{mv} dependencies at \ac{cu} level. -The proposed method selectively disables the temporal \ac{mv} candidates, in order to break the dependencies across more than one frame. The experimental evaluation was able to demonstrate that the proposed method increases the error robustness and outperforms existing state-of-the-art methods and the reference codec. -The results also show that the proposed method is able to provide a good trade-off between coding efficiency and error resilience. +Then, a new approach to deal with the weak robustness of \ac{mv} coding was proposed, addressing \ac{mv} dependencies at \ac{cu} level. +The proposed method selectively disables the temporal \ac{mv} candidates, in order to break the dependencies across more than one frame. The experimental evaluation was able to demonstrate that the proposed method increases the error robustness and outperforms existing state-of-the-art methods and the reference codec. +The results also show that the proposed method is able to provide a good trade-off between coding efficiency and error resilience. Overall. these new findings demonstrate that selective usage of temporal \ac{mv} candidates is an efficient approach to enhance error resilience in high efficiency video encoders. diff --git a/041_ImprovedErrorResilience/ImprovedErrorResilience.tex b/041_ImprovedErrorResilience/ImprovedErrorResilience.tex index 5e76c65..eb6c3ca 100644 --- a/041_ImprovedErrorResilience/ImprovedErrorResilience.tex +++ b/041_ImprovedErrorResilience/ImprovedErrorResilience.tex @@ -34,8 +34,8 @@ This chapter is organized as follows. Section~\ref{Section4:Problem} presents th As mentioned before \ac{hevc} bitstreams subject to transmission losses result in significant degradation of both objective and subjective quality. -This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes, which are selected according to a \ac{rd} optimisation criteria, assuming error-free transmission~\cite{ROPE2007}. -Therefore, high compression efficiency is achieved, but resilience to transmission errors is penalised. +This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes, which are selected according to a \ac{rd} optimisation criteria, assuming error-free transmission~\cite{ROPE2007}. +Therefore, high compression efficiency is achieved, but resilience to transmission errors is penalised. \begin{figure}[tb] \centering @@ -46,15 +46,15 @@ Therefore, high compression efficiency is achieved, but resilience to transmissi Highly efficient video encoders support the use of multiple reference frames for inter-frame prediction, which are selected based on a \ac{rd} cost which may lead to an unbalance distribution. In order to show evaluate the unbalanced is the reference frame selection attained by the \ac{rd} implemented in the reference \ac{hevc} encoder, different test sequences are encoded using two reference frames, referred to as $r_1$ and $r_2$, and its selection ratios are analysed. In this study, reference $r_1$ corresponds to the closest neighbouring frame and reference $r_2$ the second closest. -Figure~\ref{FIG4:ReferenceUsageRatio} illustrates the average usage percentage of reference $r_1$ for different sequences. -The remaining corresponds to those cases where reference $r_2$ were selected. +Figure~\ref{FIG4:ReferenceUsageRatio} illustrates the average usage percentage of reference $r_1$ for different sequences. +The remaining corresponds to those cases where reference $r_2$ were selected. These results show that, in general, reference $r_1$ is selected more often than $r_2$. -This is mainly true for sequences with higher motion. For those sequences with lower motion but high spatial complexity, e.g., BQSquare (see \ac{si} in Table~\ref{TableTestSequences}), both reference frames equally (approximately) selected for temporal predictions, with $r_1$ being used for 58\% of cases and $r_2$ for 42\%. +This is mainly true for sequences with higher motion. For those sequences with lower motion but high spatial complexity, e.g., BQSquare (see \ac{si} in Table~\ref{TableTestSequences}), both reference frames equally (approximately) selected for temporal predictions, with $r_1$ being used for 58\% of cases and $r_2$ for 42\%. This reveals that even with a configuration with two references the encoder will mainly use the temporally closer one as the reference for inter-frame predictions. Therefore, if such reference frame is lost, a higher number of mismatched predictions will occur, inevitabely leading to higher error propagation. -Another technique that contributes to increased error propagation is the use of predictions in \ac{mv} coding~\cite{MotionCodingHEVC2011}, which contributes to increase the spatial and temporal dependencies. +Another technique that contributes to increased error propagation is the use of predictions in \ac{mv} coding~\cite{MotionCodingHEVC2011}, which contributes to increase the spatial and temporal dependencies. Although the spatial \ac{mv} candidates are used more often, temporally predicted \acs{mv} are responsible for a greater impact on error propagation, as they propagate errors in the \ac{mv} decoding across different time instances. To overcome these issues, in the next sections a two-fold approach is described to increase the error robustness in video streaming by jointly considering two different stages: (i) while encoding live video, by selecting robust coding modes and (ii) after encoding, by adding robust features to the compressed streams. While in the former this is accomplished by optimising coding parameters and decisions to increase robustness, in the latter some type of compressed domain processing is used to increase error robustness of pre-encoded streams, e.g., error resilient transcoding. Inevitably, in both cases this is achieved at the expense of some loss in coding efficiency, therefore the challenge is to find an optimum trade-off between robustness and bitrate overhead. In this chapter a two-fold approach is used to overcome the two problems mentioned above. @@ -65,7 +65,7 @@ To overcome these issues, in the next sections a two-fold approach is described \section{Proposed two-stage architecture} \label{Section4:Architecture} -The goal of the proposed method is to reduce the error propagation in decoded video in case of frame loss. +The goal of the proposed method is to reduce the error propagation in decoded video in case of frame loss. The underlying idea is to constrain inter-frame prediction dependencies, in order to minimise the distortion $D_t$ given by \begin{equation} D_t = \tilde f_t - \hat f_t, \text{when } \tilde f_{t-1} \neq \hat f_{t-1}, @@ -95,11 +95,11 @@ This is equivalent to reducing the error propagation by using two techniques: (i \label{Section4:RefPicSel} -The reference picture selection mechanism attempts to uniformly distribute the use of different reference pictures across the \ac{ctu} of a coded video frame. -In case of frame loss, the error propagation in the decoder is reduced because the number of \acs{ctu} predicted from the lost frame decreases and thus error concealment benefits as more data is correctly decoded from other frames. +The reference picture selection mechanism attempts to uniformly distribute the use of different reference pictures across the \ac{ctu} of a coded video frame. +In case of frame loss, the error propagation in the decoder is reduced because the number of \acs{ctu} predicted from the lost frame decreases and thus error concealment benefits as more data is correctly decoded from other frames. The underlying principle of this method is to minimise the probability of using the same reference picture for neighbouring \acs{ctu}. -%For instance, +%For instance, In a coding structure with two reference frames, as shown in Figure~\ref{fig:checkerboardRefSelection}, the reference frames $r_{1}$ and $r_{2}$ are used to predict the same number of \acs{ctu} in current frame~($C$), resulting in a checkerboard structure, where a different reference frame is used for predicting every other \ac{ctu} within the current frame. If each reference frame is used to predict half of the pixels in the current frame, when any of these references is lost, the subsequent predicted frame is only partially distorted by mismatched predictions. This is because only half of its predictions depend on the missing reference frame, which limits the overall error propagation. @@ -187,8 +187,8 @@ Figure~\ref{fig:RefPicSelectionExample} illustrates the distribution of referenc It is noticeable in Figure~\ref{fig:RefPicSelectionExample} that for several \acs{ctu} the \PROP method is able to keep the same partitions and reference frames when compared to the optimal encoding (\REF). This can be observed, for example in the small partitions on the left, where the \PROP method uses the same reference frame as \REF. Thus, the proposed method is expected to achieve better coding efficiency than \CHKB, since more optimal coding modes may be chosen. % TID aqui? -Summarising, by using Equations~(\ref{eq:Lagrangian})~and~(\ref{eq:KFactor}), the cost of using a given reference frame increases as the number of \acs{pu} using it for prediction increases. -However, the reference frames are not hard-selected without taking into account the R-D cost. +Summarising, by using Equations~(\ref{eq:Lagrangian})~and~(\ref{eq:KFactor}), the cost of using a given reference frame increases as the number of \acs{pu} using it for prediction increases. +However, the reference frames are not hard-selected without taking into account the R-D cost. In general, it is more difficult to reach a uniform use of reference frames in complex sequences than in simpler ones. Note that standard compatibility is maintained, since the proposed optimisation method does not require any syntax change. Moreover, since the proposed approach is employed at the Lagrangian optimisation already implemented in the \ac{hevc} reference software, the complexity of the encoder is not significantly affected. @@ -198,7 +198,7 @@ Note that standard compatibility is maintained, since the proposed optimisation \glsreset{tmvp} -Since \ac{hevc} uses differential and predictive \ac{mv} coding, whenever a \ac{mv} is lost the subsequent ones will also be affected, until a refresh point (i.e., \ac{mv} coded on its own, without using \ac{mv} prediction) is reached to break the dependency chain. +Since \ac{hevc} uses differential and predictive \ac{mv} coding, whenever a \ac{mv} is lost the subsequent ones will also be affected, until a refresh point (i.e., \ac{mv} coded on its own, without using \ac{mv} prediction) is reached to break the dependency chain. Thus, the amount of error propagation strongly depends on whether the \ac{tmvp} is used. In this streaming stage, to increase the number of possible refresh points and improve error robustness, a small number of \acs{mv} are selected to be transmitted as side information. The selection of these redundant \acs{mv} is based on a trade-off between the image area covered by such \acs{mv} and the number of bits to encode them. The best tradeoff corresponds to the \acs{mv} that cover the largest image area using the lowest amount of encoding bits. @@ -209,7 +209,7 @@ The best tradeoff corresponds to the \acs{mv} that cover the largest image area Figure~\ref{fig:mvdependencies} illustrates how \acs{mv} have different importance in error propagation. In this figure, each block corresponds to a \ac{pu}, which has an associated \ac{mv}. The arrows point to the \ac{pu} containing the \ac{mv} used as predictor. In the current frame ($f_0$), two \acs{pu} have temporally dependent \acs{mv}. -While the \ac{mv} of \ac{pu} \MVDEPA is used to predict a total of four other \acs{mv}, the \ac{mv} of \ac{pu} \MVDEPB is only used to predict one single \ac{mv}. +While the \ac{mv} of \ac{pu} \MVDEPA is used to predict a total of four other \acs{mv}, the \ac{mv} of \ac{pu} \MVDEPB is only used to predict one single \ac{mv}. Thus, in case of data loss (e.g., frame $f_{-1}$), the \ac{mv} of \ac{pu} \MVDEPA has more impact on error propagation to the frame $f_{0}$ and consequently quality degradation than the \ac{mv} of \ac{pu} \MVDEPB, because of its higher number of dependent \acs{mv}. For this reason, in the \ac{mv} selection process, the proposed method assigns higher importance to the \ac{mv} of \ac{pu} \MVDEPA than \ac{pu} \MVDEPB. Moreover, in Figure~\ref{fig:mvdependencies}, it can be observed that although the number of \acs{mv} predicted from the \acs{mv} of \ac{pu} \MVDEPC is higher than \ac{pu} \MVDEPA (6 and 4 \acs{mv}, respectively), the image area that is dependent from the former is smaller that the latter. This is due to the variable size of \acs{pu}. @@ -261,9 +261,9 @@ In this section the robustness of the proposed two-stage architecture is evaluat As references for comparison, three other methods are used: (i) a fixed checkerboard approach (\CHKB), (ii) the reference picture generation method based on leaky predictions proposed in \cite{ROPERefGenJan2010}, identified as ``\AVG'', and (iii) long-term reference frames (\LONG). The \LONG method only uses the key frames as reference, i.e., in the \ac{ra} configuration only one out of eight frames is used as reference for prediction, allowing a more robust transmission, as the reference frames are less likely to be hit by errors. -The experimental setup used in this section is the same as the one previously described in Section~\ref{Section3:ErrorResilienceEvaluation}. +The experimental setup used in this section is the same as the one previously described in Section~\ref{Section3:ErrorResilienceEvaluation}. Two test cases were considered for simulation: (i) in \emph{\LOSTALL} all frames may be equally affected by packet loss, including \ac{idr} frames; (ii) in \emph{\KEEPINTRA} \ac{idr} frames are assumed to be prioritised in the network and delivered error-free, or using some combined approach as recently proposed in \cite{ErrorResilienceHeterogenousNetworks2016}. -At the decoder side, error concealment is based on motion compensation from previously decoded frames. The motion field of a missing frame is obtained applying \ac{mv} extrapolation from the two closest neighbours. Afterwards, the final \acs{mv} are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}. +At the decoder-side, error concealment is based on motion compensation from previously decoded frames. The motion field of a missing frame is obtained applying \ac{mv} extrapolation from the two closest neighbours. Afterwards, the final \acs{mv} are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}. The evaluation of the encoding-stage is carried out separately from the streaming-stage. @@ -272,7 +272,7 @@ The evaluation of the encoding-stage is carried out separately from the streamin %The sequences were encoded with all coding modes enabled and following two recommended prediction structures: Low-Delay (P-frames with 4 references frames) and \ac{ra} (B-frames with 2 reference frames in each list)~\cite{HEVCCommonTests2012}. These prediction structures cover the use of P-frames and hierarchical B-frames~\cite{H264ScalableOverviewSep2007}, which are commonly used in HEVC streaming~\cite{RealTimeHEVC2016}. %To simulate a broadcast transmission an \ac{idr} period of $32$ frames was used and the filtering operations were disabled at slices boundaries in order to keep the slices self-contained. The remaining configurations have been kept at their default values as given in the common test conditions~\cite{HEVCCommonTests2012}. %In order to achieve high quality images for all coded sequences (i.e., PSNR around 40 dB) different bitrates were used. -%At the decoder side, error concealment is based on motion compensation from previously decoded frames. The motion field of a missing frame is obtained applying MV extrapolation from the two closest neighbours. Afterwards, the final MVs are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}. +%At the decoder-side, error concealment is based on motion compensation from previously decoded frames. The motion field of a missing frame is obtained applying MV extrapolation from the two closest neighbours. Afterwards, the final MVs are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}. %To simulate the lossy transmission environment, each frame is encoded into several slices, each one packetized into one NAL unit, which are then transmitted as the payload of independent packets. % %In our experiments a whole packet is lost whenever an error occurs, originating a lost slice. Random packet loss was simulated using a two-state Markov model with the following average burst length 1.24, 1.47, 1.83 and 2.05, and a maximum burst length of 2, 3, 5 and 7, for packet loss ratios (PLR) of 1\%, 3\%, 5\%, 10\%, respectively. @@ -283,13 +283,13 @@ The evaluation of the encoding-stage is carried out separately from the streamin The performance of the encoding stage is evaluated on its own in order to only determine the improvements achieved by the dynamic reference frame selection method. In this case, the reference frames were reconstructed with distortion, as a result of non-perfect error concealment. -Since the encoding stage does not deal with errors in temporal \ac{mv} predictions (this is handled by the streaming-stage), in this evaluation the temporal \ac{mv} predictor is disabled. +Since the encoding stage does not deal with errors in temporal \ac{mv} predictions (this is handled by the streaming-stage), in this evaluation the temporal \ac{mv} predictor is disabled. This is necessary to avoid masking the results with the errors caused by wrong \ac{mv} predictions. -The performance of the encoding stage (\PROP) was evaluated using $\gamma = 5$. +The performance of the encoding stage (\PROP) was evaluated using $\gamma = 5$. This value was chosen to allow increasing the Lagrangian cost by about $50\%$, when a reference frame is used for prediction of more than three \acs{ctu} in comparison to the usage of the other reference frames (see Figure~\ref{fig:weightfct}). Nevertheless, further results for a wide range of $\gamma$ values are also presented. -%\HERE +%\HERE \subsubsection{Quality evaluation under error-prone conditions} \begin{figure}[tb] @@ -312,7 +312,7 @@ This value was chosen to allow increasing the Lagrangian cost by about $50\%$, \end{figure} The error robustness achieved by the encoding-stage was firstly evaluated in regards to the error propagation resulting from a single loss event in the \ac{ld} configuration. -Figure~\ref{fig:RecoveredExample} shows the visual impact of error propagation on a region of the decoded Frame \#10 when Frame \#9 is affected by errors, for Basketball Drill and Race Horses sequences (the \ac{psnr} corresponds to the entire frame). +Figure~\ref{fig:RecoveredExample} shows the visual impact of error propagation on a region of the decoded Frame \#10 when Frame \#9 is affected by errors, for Basketball Drill and Race Horses sequences (the \ac{psnr} corresponds to the entire frame). The proposed method is compared with the method~in~\cite{ROPERefGenJan2010} as both aim at reducing the error propagation. The method~in~\cite{ROPERefGenJan2010} uses filtered frames as reference for predictions, therefore, in case of errors, the effects of mismatch predictions are significantly reduced by the filtering operation (see Sub-Section~\ref{SECTION2:LeakyPred}). As shown in this figure, the proposed method is able to reduce the impact of mismatch decoding in comparison with the method~in~\cite{ROPERefGenJan2010}. Although both methods still reveal some predictions mismatch leading to inaccurate reconstructed of the objects contours, the proposed method is able to achieve a smoother reconstruction, which leads to higher quality in terms of \ac{psnr}. This is because, in frame \#10, the predictions depending on the missing frame \#9 are not used throughout the entire frame, thus, approximately half of the predictions are not affected by errors. In contrast, the method~in~\cite{ROPERefGenJan2010} requires several filtering operations to be applied (i.e, different reference frames need to be filtered) in order to reduce the effect of mismatch predictions. @@ -342,14 +342,14 @@ Further tests were run to evaluate the effectiveness of the proposed method unde The use of long-term reference pictures (\LONG) is only tested for the \ac{ra} configuration, where key frames are available, while method \cite{ROPERefGenJan2010} is tested for the \ac{ld} configuration. For each test condition, 50 trials were performed and the average quality (\ac{psnr}) obtained across all trials for both coding configurations are shown in Table~\ref{tab:AverageQualityCHKBEXPLowDelay}~and~\ref{tab:AverageQualityCHKBEXPRandomAccess}, for the \ac{ld} and \ac{ra}, respectively. In both tables the absolute \ac{psnr} is shown for the reference \ac{hevc} case (\REF), while the \ac{psnr} difference is presented for the other cases. -Results show that the proposed method is able to outperform other methods for the same loss conditions. +Results show that the proposed method is able to outperform other methods for the same loss conditions. An average quality gain up to $2.26$~dB is obtained for the \ac{ld} configuration (see Table~\ref{tab:AverageQualityCHKBEXPLowDelay}), in comparison to the \REF, while the fixed checkerboard pattern approach (\CHKB) only achieves up to $2.15$~dB. In \cite{ROPERefGenJan2010}, the use of interpolated references in the encoding loop helps to reduce error propagation, at a cost of poor predictions, thus degrading the coding efficiency. This is the main reason that justifies a lower overall performance of the method proposed in \cite{ROPERefGenJan2010} (see results for ``\AVG''). The proposed method is also able to outperform the \ac{hevc} reference software when an hierarchical coding structure is used (see results for \ac{ra} configuration in Table~\ref{tab:AverageQualityCHKBEXPRandomAccess}), which implicitly limits the error propagation. In comparison to the reference \ac{hevc}, quality improvements up to $1.42$~dB at PLR=$10\%$ are obtained for Kendo sequence.%, while the average gain across all test sequences is 1~dB. When comparing results for both test cases, it is noticeable that the proposed method is able to outperform the reference methods both when \ac{idr} frame loss is allowed or not (\LOSTALL and \KEEPINTRA, respectively). As the \ac{plr} increases, the gains of the proposed method decrease in comparison with the case where the \ac{idr} frames are delivered error-free (i.e., case \KEEPINTRA), due to the higher error propagation of the missing \ac{idr} frame. Moreover, the quality gains are lower in the \LOSTALL case when comparing with \KEEPINTRA for the same \ac{plr}, for both the \ac{ld} and \ac{ra}. -Taking into account the above results and the spatio-temporal information shown in Table~\ref{TableTestSequences}, one can observe that the performance of the proposed method is lower for sequences with high spatial details and low temporal activity (e.g., BQSquare and Park Scene), which indicates higher effectiveness for video content with higher motion activity. +Taking into account the above results and the spatio-temporal information shown in Table~\ref{TableTestSequences}, one can observe that the performance of the proposed method is lower for sequences with high spatial details and low temporal activity (e.g., BQSquare and Park Scene), which indicates higher effectiveness for video content with higher motion activity. %\HIGHLIGHT{complete with analysis of both cases} @@ -369,10 +369,10 @@ Taking into account the above results and the spatio-temporal information shown The influence of the $\gamma$ parameter used in Equation (\ref{eq:KFactor}) on the video quality was also evaluated, in order to check how the cost constrains affect the performance of the proposed method. Figure \ref{fig:QualityVsGamma} shows the average quality gains of the proposed method in comparison to the reference \ac{hevc} (\REF) across all test sequences, for different values of $\gamma$. -For the error-free case (PLR=0\%), the results in this figure show that increasing $\gamma$ leads to a decrease in the coding efficiency, especially in the \ac{ld} configuration. +For the error-free case (PLR=0\%), the results in this figure show that increasing $\gamma$ leads to a decrease in the coding efficiency, especially in the \ac{ld} configuration. The reason is that higher values of $\gamma$ lead to higher values of $W(r_{\phi})$, therefore the reference picture selection cost is severely constrained by the exponential weights. Since $T_{ID}$ is constant for all frames in the \ac{ld}, higher weights are used for all frames ($T_{ID}$ is used to reduce the weights in case of hierarchical coding) and the encoder is not able to select the best reference frame in \c{rd} sense. -In the case where packet loss occurs, higher values of $\gamma$ lead to improved error resilience. +In the case where packet loss occurs, higher values of $\gamma$ lead to improved error resilience. Since higher weights are used in the \ac{rd} cost, the encoder is constrained to distribute the use of the reference frames of different \acs{pu}, reducing the error propagation. For very high values (e.g., $\gamma=1000$) the encoder is forced to uniformly select the reference frames, resulting in a fixed checkerboard pattern. Overall, the $\gamma$ parameter is able to control the reference frame selection constrains, @@ -418,7 +418,7 @@ Overall, the loss in the quality compared to the reference \ac{hevc} is consider \subsubsection{Complexity overhead} The implementation of this method slightly increases the complexity of a standard non-robust encoder. Therefore, the computational complexity has also been considered as a performance metric in this research, which is measured as the encoding time in a controlled hardware platform. The reference used for comparison with other methods is the \ac{hevc} reference software encoding time, running on the same platform with the same coding configuration. -The relative complexity increase of the robust encoder using \PROP, \CHKB and the method presented in \cite{ROPERefGenJan2010} is shown in Table~\ref{tab:cpu_time}. +The relative complexity increase of the robust encoder using \PROP, \CHKB and the method presented in \cite{ROPERefGenJan2010} is shown in Table~\ref{tab:cpu_time}. As shown in this table, the method presented in \cite{ROPERefGenJan2010} is the most time consuming because it requires post-processing of the encoded frames in order to interpolate new references. @@ -446,8 +446,8 @@ This is due to the fact that in the \CHKB method there is no \ac{rd} optimisatio \subsection{Streaming stage} -In this section the coding performance that results from transmitting redundant \ac{mv} (streaming stage) in combination with the reference picture selection scheme (encoding stage) is presented. -In these experimental tests the temporal \ac{mv} candidates are enabled and \acs{mv} are dynamically added as side-information into the stream, as described in Section \ref{Section4:MVSelection}. +In this section the coding performance that results from transmitting redundant \ac{mv} (streaming stage) in combination with the reference picture selection scheme (encoding stage) is presented. +In these experimental tests the temporal \ac{mv} candidates are enabled and \acs{mv} are dynamically added as side-information into the stream, as described in Section \ref{Section4:MVSelection}. Different values of $\alpha$ are used to achieve different overhead ratios. All streams were encoded with the same total bitrate, including the redundant bits, in order to make a fair quality comparison. diff --git a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex index b8e5cc2..74d734a 100644 --- a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex +++ b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex @@ -186,13 +186,13 @@ However, different network topologies with different number of neurons in each l \section{Experimental evaluation} \label{Section6:ExperimentalResults} -\def\NumCases{910\xspace} +\def\NumCases{845\xspace} In this section the experimental results obtained with the proposed method are presented and discussed. The error resilience techniques used in the proposed method are used in the context of the \ac{hevc} standard and compared with the reference implementation (\REF). %The experimental results were obtained with the methodology described in Section~\ref{ExperimentalTestSetup}. To demonstrate the effectiveness of the proposed optimal error robustness scheme several test cases were used. -In this experimental evaluation 14 test sequences were used, encoded using at 5 different bitrates. +In this experimental evaluation 13 test sequences were used, encoded using at 5 different bitrates. Moreover, the following \acs{plr} were used to cover a wide range of loss rations giving more detail to low \acs{plr}: 0\%, 0.5\%, 1.0\%, 1.2\%, 1.6\%, 2.0\%, 2.2\%, 2.4\%, 2.6\%, 2.8\%, 3.0\%, 5.0\% and 10.0\%. This results in a total of \NumCases test conditions. @@ -231,7 +231,7 @@ In the latter, the quality obtained with the proposed optimal error robustness s %As mentioned before a total of \NumCases test cases are used for the evaluation of the proposed method. To effectively train and validate the \ac{nn} used in the proposed method the data set is randomly divided based on the following: 70\% of the test cases are used for training and 30\% for validation. -The \ac{nn} supervised training uses 637 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993}. +The \ac{nn} supervised training uses 591 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993}. The remaining 273 conditions are used to evaluate the accuracy of the proposed method after training using different conditions. diff --git a/carreira_phd_thesis_abr.sty b/carreira_phd_thesis_abr.sty index 45cd7ad..543fd8e 100644 --- a/carreira_phd_thesis_abr.sty +++ b/carreira_phd_thesis_abr.sty @@ -101,7 +101,7 @@ \customnewacronym{fec}{FEC}{Forward Error Correction} \customnewacronym{rir}{RIR}{Random Intra Refreshing} \customnewacronym[longplural={Regions of Interest}]{roi}{ROI}{Region of Interest} -\customnewacronym{rope}{ROPE}{Recursive Optimal Per-Pixel Estimation} +\customnewacronym{rope}{ROPE}{Recursive Optimal Per-pixel Estimation} \customnewacronym{rps}{RPS}{Reference Picture Selection} \customnewacronym{uep}{UEP}{Unequal Error Protection} \customnewacronym{mdc}{MDC}{Multiple Description Coding}