Review thesis after PA comments

This commit is contained in:
jfmcarreira committed 2018-09-17 23:11:37 +01:00
1 parent 2a97ab6c09
commit 9bdc2ee860
24 files changed
+798 -749

No files matched your search

+135 -191
View File
@@ -1,4 +1,4 @@
\chapter{Robust Video Coding: A Review \VERSION{2}}
\chapter{Robust video coding: a review \VERSION{3}}
\label{Chapter:ErrorResilienceReview}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -12,17 +12,18 @@
In general, digital communications through networks are prone to transmission errors or data loss.
These errors can be handled in different ways, such as retransmission, error correction, and \ac{ec} \cite{ErrorResilience2000,PixelInterlacingIntraConcealment2011}.
This chapter reviews the main robust video coding techniques that can be applied at the \ac{vcl} to reduce the error propagation and increase the decoded video quality.
This chapter reviews the existing techniques for robust video coding that can be applied at the \ac{vcl} to reduce the error propagation and increase the decoded video quality.
Several techniques have been proposed in the past to deal with the problem of robust video transmission, especially for video coding standards with high temporal dependencies, such as \ac{h264} and more recently \ac{hevc}.
These techniques are normally focused on the spatio-temporal dependencies in the coded streams, which are used to obtain high coding gains but lead to low error robustness~\cite{ROPE2007}.
In this chapter the error resilience and concealment schemes are divided in different categories, depending on their functionality and algorithms.
These techniques are normally focused on the spatio-temporal dependencies in the coded streams, which are the basis of high coding gains but lead to low error robustness~\cite{ROPE2007}.
In this chapter the error resilience and error concealment schemes are divided in different categories, depending on their functionality and algorithms.
Although each approach is based on different techniques they all aim at striking the best trade-off between coding efficiency and decoded video quality under lossy conditions.
This chapter is organised as follows: Section~\ref{Section22:Localisation} describes the localisation techniques, which are able to reduce the temporal and spatial dependencies in coded bitstreams.
The chapter is organised as follows.
Section~\ref{Section22:Localisation} describes the localisation techniques, which are able to reduce the temporal and spatial dependencies in coded bitstreams.
Section~\ref{Section22:DataPartitionRedundant} provides an overview on methods based on data partition, classification and bitstream redundancy.
%These techniques ease the introduction of extra information in order to recover from lost data.
Section~\ref{Section22:Concealment} covers the techniques used at the decoder-side, as post-processing, to reduce the effect of errors and improve visual quality and smoothness.
Section~\ref{Section22:ConcealmentAware} provides a description of methods that use \ac{ec} techniques to achieve improved error robustness.
Section~\ref{Section22:ConcealmentAware} provides a description of methods that use \ac{ec} techniques to cope with transmission errors and improve error robustness.
Finally, Section~\ref{Section22:Summary} concludes the chapter.
% by providing a overall comparison between the different approaches.
@@ -52,12 +53,12 @@ In this chapter the robust video coding techniques are grouped into four categor
\end{table}
Table~\ref{tab:errorresilience} provides a summary of the techniques covered in this section, some examples and the key advantages associated with each group.
Localisation techniques mainly localise and break the predictive coding loop, reducing the spatial and temporal dependencies, so that if an error occurs it is not likely to affect other parts of the coded video~\cite{ErrorResilienceModeHEVC2016,ErrorResilienceModeHEVC2017}.
Data partitioning rely on grouping the coded data into different categories depending on their relative importance.
This ease the introduction of unequal error protection at the transmission layer.
Localisation techniques mainly localise and break the predictive coding loop, reducing the spatial and temporal dependencies, so that if an error occurs, then it is not likely to affect other parts of the coded video~\cite{ErrorResilienceModeHEVC2016,ErrorResilienceModeHEVC2017}.
Data partitioning relies on grouping the coded data into different categories depending on their relative importance.
This can be used to ease the introduction of unequal error protection at the transmission layer.
Redundant coding consists in using additional data blocks to enable correction of transmission errors and to achieve robust decoding in error-prone conditions.
In case of the post-processing \ac{ec} methods, the problem of robust transmission is addressed from the decoder-side.
Finally, concealment-aware techniques are used at the encoder, to either adapt the coding loop or introduce extra information, which is used to increase the performance of \ac{ec} algorithms and increase the reconstruction quality of lost data at the decoder \cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ConcealmentIntraDataHidng2009,ConcealmentSignalling2008}.
Finally, error concealment-aware techniques are used at the encoder, to either adapt the coding loop or introduce extra information, which is used to increase the performance of \ac{ec} algorithms and increase the reconstruction quality of lost data at the decoder \cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ConcealmentIntraDataHidng2009,ConcealmentSignalling2008}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -73,7 +74,7 @@ Since these techniques break the predictive coding, they have a negative impact
This group of error resilience techniques are able to reduce the error propagation at two levels, i.e., temporal and spatial levels. Figure~\ref{fig2:ErrorPropagation} is used to demonstrate the principle of error propagation and how it can be reduced using localisation techniques.
The top of the figure illustrates a lost block and the corresponding spatial and temporal error propagation on the left and right, respectively.
Spatial localisation techniques are used to reduce the error propagation within a frame, where a row of refresh blocks can be introduced to stop the error propagation.
To address the temporal error propagation, also refresh blocks (e.g., intra coded blocks) can be randomly introduced to stop error propagation.
To address the temporal error propagation, also refresh blocks (e.g., intra coded blocks) can be randomly introduced to break temporal dependencies and stop error propagation.
As shown in Figure~\ref{fig2:ErrorPropagation} the refresh blocks are not affected by error propagation.
In both examples of figure, the regions affected by errors are clearly reduced.
@@ -90,15 +91,16 @@ In both examples of figure, the regions affected by errors are clearly reduced.
\glsreset{rd}
A common localization technique deals with the error resilience problem within the scope of the \ac{rd} optimisation process used in video coding.
Common localization techniques deal with the error resilience problem within the scope of the \ac{rd} optimisation process used in video coding.
Normally, such optimisation is equivalent to the minimisation of the Lagrangian cost given by:
\begin{equation}
J = D + \lambda R,
\end{equation}
where $D$ is the distortion of all pixels, $R$ the bitrate and $\lambda$ is the Lagrangian multiplier~\cite{LagragianMultiplier1982}.
Such optimisation only takes into account the coding distortion ($D$), which is only suitable for error-free environments.
However, some alternative approaches with error resilient \ac{rd} optimisation take into consideration possible network losses~\cite{DistortionTrackingH264_ICIP2004}.
However, some alternative approaches use error resilient \ac{rd} optimisation, by taking into consideration possible network losses~\cite{DistortionTrackingH264_ICIP2004}.
Other advanced approaches can take advantage of a \ac{rope} of the end-to-end distortion at the encoder-side~\cite{ROPEJun2000}.
This may be formulated as follows.
Let $f_t^i$ denote the original value of pixel $i$ in frame at instant $t$, $\hat f_t^i$ denote its reconstruction in the encoder and $\tilde f_t^i$ the reconstruction in the decoder after \ac{ec}.
Since data loss may occur in the transmission channel, $\tilde f_t^i$ is modelled as a random variable at the encoder.
The expected \ac{mse} of each pixel is given by:
@@ -108,50 +110,58 @@ E\left\{d_t^i\right\} = E\left\{ \left( f_t^i - \tilde f_t^i\right)^2 \right\}
The packet loss probability ($p$) is taken into consideration and used in the distortion measurement.
In case of intra-coded pixels the following equations are used:
\begin{equation}\label{EQ2:E2EDIntra}
E\left\{ \tilde f_t^i \right\} = \left(1-p\right) \hat f_t^i + p E\left\{ \tilde f_{n-1}^i \right\},
E\left\{ \tilde f_t^i \right\} = \left(1-p\right) \hat f_t^i + p E\left\{ \tilde f_{t-1}^i \right\},
\end{equation}
\begin{equation}
E\left\{\left( \tilde f_t^i \right)^2 \right\} = \left(1-p\right) \left( \hat f_t^i \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\},
E\left\{\left( \tilde f_t^i \right)^2 \right\} = \left(1-p\right) \left( \hat f_t^i \right) + p E\left\{ \left( \tilde f_{t-1}^i \right)^2 \right\},
\end{equation}
while in the case of inter-coded pixels:
\begin{equation}
E\left\{ \tilde f_t^i \right\} = \left(1-p\right) \left( \hat e_t^i + E\left\{ \tilde f_{n-1}^{j} \right\} \right) + p E\left\{ \tilde f_{n-1}^i \right\},
E\left\{ \tilde f_t^i \right\} = \left(1-p\right) \left( \hat e_t^i + E\left\{ \tilde f_{t-1}^{j} \right\} \right) + p E\left\{ \tilde f_{t-1}^i \right\},
\end{equation}
\begin{equation}\label{EQ2:E2EDInter}
E\left\{\left( \tilde f_t^i \right)^2 \right\} = \left(1-p\right) \left( \left( \hat e_t^i \right)^2 + 2 \hat e_t^i E\left\{ \tilde f_{n-1}^{j} \right\} + E\left\{ \left( \tilde f_{n-1}^{j} \right)^2 \right\} \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\},
E\left\{\left( \tilde f_t^i \right)^2 \right\} = \left(1-p\right) \left( \left( \hat e_t^i \right)^2 + 2 \hat e_t^i E\left\{ \tilde f_{t-1}^{j} \right\} + E\left\{ \left( \tilde f_{t-1}^{j} \right)^2 \right\} \right) + p E\left\{ \left( \tilde f_{t-1}^i \right)^2 \right\},
\end{equation}
assuming that the pixel $i$ is predicted from pixel $j$ in the previous frame.
The prediction error $e_t^i$ is quantised to the value $\hat e_t^i$, which is transmitted to the decoder along with the motion information.
Equations~\ref{EQ2:E2EDIntra} to~\ref{EQ2:E2EDInter} provide a measure of the distortion, taking into consideration not only the coding distortion but also the possibility of frame loss and consequent \ac{ec} distortion.
This technique was used in \cite{ROPEJun2000} to switch between intra and inter coding modes with the aim of reducing the temporal dependencies, thus achieving enhanced error robustness in video transmission.
In \cite{AdvancesROPEHE2642007} this technique was optimised to quarter pixel estimation, and in \cite{ROPEBurst2009} the method was extended for burst losses, by introducing the burst loss probability into the original end-to-end estimation.
This method was used in \cite{ROPEJun2000} to switch between intra and inter coding modes with the aim of reducing the temporal dependencies, thus achieving enhanced error robustness in video transmission.
In \cite{AdvancesROPEHE2642007} this was optimised to quarter pixel estimation, and in \cite{ROPEBurst2009} the same method was extended for burst losses, by introducing the burst loss probability into the original end-to-end estimation.
Results revealed that higher video quality was achieved by considering both probabilities.
In \cite{E2EDistortion2011} a temporal \ac{ec} is considered in the distortion estimation, which increases the performance of the proposed method, by avoiding frame-copy that provides a lower bound on reconstruction quality.
In \cite{E2EDistortion2011} temporal \ac{ec} is considered in the distortion estimation, which increases the performance of the proposed method, by avoiding frame-copy that provides a lower bound on reconstruction quality.
More advanced temporal \ac{ec} methods were also considered in the work presented in~\cite{EndToEndDistortionConcealment2014}.
The \acs{mv} used for motion compensation reconstruction is obtained using boundary matching search, which is able to achieve lower distortion when compared with frame-copy. Using more advanced \ac{ec} algorithms leads to a more accurate estimation of the end-to-end distortion, thus increasing the error robustness of the transmitted video.
To take into account the delay introduced in the encoding loop due to hierarchical prediction structures, delay based \ac{rd} optimisation was proposed in \cite{DelayRD_TCVT2015} by estimating both the distortion and the end-to-end delay that is minimised along the required bitrate. Using this approach, the total delay can be controlled without severely compromising the coding efficiency.
In the context of \ac{hevc} end-to-end distortion estimation was also used in the \ac{rd} optimisation showing significant quality improvements when compared to with the reference implementation \cite{ErrorResilienceModeHEVC2016}.
The \acs{mv} used for motion compensation reconstruction are obtained using boundary matching search, which is able to achieve lower distortion when compared with frame-copy. Using more advanced \ac{ec} algorithms leads to a more accurate estimation of the end-to-end distortion, thus increasing the error robustness of the transmitted video.
To take into account the delay introduced in the encoding loop due to hierarchical prediction structures, delay-based \ac{rd} optimisation was proposed in \cite{DelayRD_TCVT2015} by estimating both the distortion and the end-to-end delay that is minimised along with the required bitrate.
Using this approach, the total delay can be controlled without severely compromising the coding efficiency.
In the context of \ac{hevc}, end-to-end distortion estimation was also used in the \ac{rd} optimisation showing significant quality improvements when compared to with the reference implementation \cite{ErrorResilienceModeHEVC2016}.
Different quality metrics can also be used for end-to-end distortion estimation. The method presented in \cite{SSIMModeling2015} proposes the distortion estimation using the \ac{ssim} \cite{QualitySSIMRef2002} in order to increase the correlation between the estimated distortion and the perceptual quality noticed by the human visual system~\cite{VisualQualityMetrics2011}. Moreover, the \ac{ssim} quality metric was also used in learning based \ac{rd} models~\cite{RateDistortionNN_VCIP2017}, using a \ac{cnn} to predict the end-to-end distortion and the required rate, showing good accuracy in the \ac{rd} cost estimation.
Different quality metrics can also be used for end-to-end distortion estimation.
The method presented in \cite{SSIMModeling2015} proposes to estimate the distortion by using the \ac{ssim} \cite{QualitySSIMRef2002} in order to increase the correlation between the estimated distortion and the perceptual quality noticed by the human visual system~\cite{VisualQualityMetrics2011}. Moreover, the \ac{ssim} quality metric was also used in learning-based \ac{rd} models~\cite{RateDistortionNN_VCIP2017}, using a \ac{cnn} to predict the end-to-end distortion and the required rate, showing good accuracy in the \ac{rd} cost estimation.
\subsection{Intra-refreshing}
In order to reduce the effects of error propagation within a single, frame spatial and temporal dependencies can be limited by introducing refresh points into the bitstream, by forcing intra-coded blocks~\cite{IntraRefresh1999,IntraRefreshAttentionCM2007,IntraRefreshRefSelectionJan2009,IntraRefreshICIP2009,SandroIntraRefresh2011,JointIntraRefresh2014,IntraRefreshFeedback2015,IntraRefreshRPS2017}.
In order to reduce the effects of error propagation within a single frame, spatial and temporal dependencies can be limited by introducing refresh points into the bitstream, by forcing intra-coded blocks~\cite{IntraRefresh1999,IntraRefreshAttentionCM2007,IntraRefreshRefSelectionJan2009,IntraRefreshICIP2009,SandroIntraRefresh2011,JointIntraRefresh2014,IntraRefreshFeedback2015,IntraRefreshRPS2017}.
Figure~\ref{FIG2:IntraRefresh} (a) shows an example where intra-coded block are forced at random locations (dark gray sqares) in order to stop error propagation.
This technique is fairly efficient to improve the error robustness of video streams, by avoiding long error propagation both within a single frame but also across several subsequent frames.
This technique is fairly efficient to improve the error robustness of video streams, by avoiding long error propagation not only within a single frame but also across several subsequent frames.
The main characteristic of this group of techniques is to increase the number of intra-coded blocks as they have no dependencies to previous frames.
However, this is done at the expense of higher amount of bits~\cite{IntraRefreshAttentionCM2007}.
Therefore, different alternatives have been proposed, which optimise the number of intra-coded blocks, which can be introduced randomly, periodically~\cite{SandroIntraRefresh2011}, based on the motion information or rate-distortion optimisation~\cite{IntraRefreshICIP2008,HEVCIntraRefreshICIP2017}.
Moreover, whenever is possible the encoder can use a feedback channel to decide where refresh regions should be introduced~\cite{IntraRefresh1999,IntraRefreshFeedback2015}.
However, this is done at the expense of using higher amount of coded bits~\cite{IntraRefreshAttentionCM2007}.
Therefore, different alternatives have been proposed to optimise the number of intra-coded blocks, which can be introduced either randomly or periodically~\cite{SandroIntraRefresh2011}, based on the motion information or rate-distortion optimisation~\cite{IntraRefreshICIP2008,HEVCIntraRefreshICIP2017}.
Moreover, whenever possible, the encoder can use a feedback channel to decide where refresh regions should be dynamically introduced in the coded stream~\cite{IntraRefresh1999,IntraRefreshFeedback2015}.
\begin{figure}[tb]
\centering
\input{./022_ErrorResilience/Figures/IntraRefresh.tikz}
\caption{Examples of intra-refreshing methods.}
\label{FIG2:IntraRefresh}
\end{figure}
A non-normative method for random intra refreshing is implemented in the \ac{h264} reference software, which introduces cyclic refresh points using a random pattern.
Figure~\ref{FIG2:IntraRefresh} (a) shows an example of intra-coded blocks (dark gray squares) being randomly places into the coded frames to prevent error propagation.
A non-normative method for random intra refreshing is implemented in the H.264 reference software, which introduces cyclic refresh points using a random pattern.
Figure~\ref{FIG2:IntraRefresh} (a) shows an example of intra-coded blocks (dark gray squares) being randomly placed into the coded frames to prevent error propagation.
This approach guarantees that all \acs{mb} are refreshed at some point, therefore errors do not propagate indefinitely.
However, all \acs{mb} are refreshed exactly the same number of times without consideration for rate distortion optimisation neither the image region containing the \acs{mb}.
In \cite{IntraRefreshICIP2008} a rate control scheme is used to decide if a given block should be refreshed using a intra-coded prediction.
In \cite{IntraRefreshICIP2008} a rate control scheme is used to decide whether a given block should be refreshed using intra-coded prediction.
The decision is performed based on the ratio between the Lagrangian cost of the best intra and inter modes.
Whenever this ratio is lower than a certain predefined threshold an intra mode is selected, which guarantees that intra refreshing is only forced when the Lagrangian cost of the intra mode is close to the optimal coding method.
This is further extended to adapt the intra refresh algorithm to the network conditions, i.e., the threshold value is obtained from the bitrate and the loss ratio~\cite{IntraRefreshICIP2009}.
@@ -159,27 +169,22 @@ Therefore, the number of intra-coded block automatically increases as the bitrat
In the context of \ac{hevc}, intra refresh is also being used to reduce error propagation.
Soft-refresh points using joint intra-inter prediction were proposed for this purpose~\cite{HEVCIntraRefreshICIP2017}.
Although the image regions cannot be correctly decoded in case of errors (as happens with intra-refresh), this method significantly reduces the bitrate increase due to extra intra-coded blocks.
When compared with traditional intra-refresh, bitrate savings of up to 12\% are achieved for a 5\% \gls{plr}.
When compared with traditional intra-refresh, bitrate savings of up to 12\% are achieved for a 5\% packet loss.
\begin{figure}[tb]
\centering
\input{./022_ErrorResilience/Figures/IntraRefresh.tikz}
\caption{Examples of intra-refreshing methods.}
\label{FIG2:IntraRefresh}
\end{figure}
\glsreset{roi}
A different approach is to introduce refresh points based on the video content.
In~\cite{IntraRefreshAttentionCM2007} the error resilience of video streams is increased by combining rate control and attention based models to define \acs{roi}.
In~\cite{IntraRefreshAttentionCM2007} the error resilience of video streams is increased by combining rate control and attention-based models to define \acs{roi}.
A weighting factor is used in the Lagrangian optimization to increase the number of intra-coded blocks in the image regions determined by the attention models.
Figure~\ref{FIG2:IntraRefresh} (b) shows an example with a \ac{roi} marked with a black line, which varies for every frame. In this case, the intra-coded blocks are only introduced inside the marked region, in contrast to what is shown in Figure~\ref{FIG2:IntraRefresh} (a).
Results show that using the aforementioned method, less refresh points are required to improve the error robustness of the \ac{roi}. When comparing with \cite{DistortionTrackingH264_ICIP2004}, approximately 4 dB of \ac{psnr} gains are achieved in the \ac{roi} without compromising the overall video quality.
Results show that by using the aforementioned method, less refresh points are required to improve the error robustness of the \ac{roi}. When comparing with \cite{DistortionTrackingH264_ICIP2004}, approximately 4 dB of \ac{psnr} gains are achieved in the \ac{roi} without compromising the overall video quality.
A more recent approach was proposed using a perceptual noticeable difference model to estimate the noticeable distortion, and then using a visual attention model \cite{SaliencyItti1998} to decide which blocks should be used as intra refresh in the bitstream~\cite{IntraRefreshPerceptual2010}.
This method is able to improve the results of a previous work \cite{IntraRefresh2005}, especially for high motion sequences. This is due to the fact that such previous approach only used an estimation of the error propagation without accounting for perceptual information.
A similar approach was also exploited for \ac{hevc} in \cite{HEVCROIErrorResilience2015,HEVCRoiIntraRefresh_DSP2016}, by using a feedback channel to provide information about the network errors, which is then used by the encoder to introduce intra-coded blocks into the \ac{roi}.
The simulation results reveal an increasing on the achieved video quality when higher number of intra blocks are used. Moreover, applying the proposed method only to the \ac{roi}, the bitrate overhead introduced is lower, leading to higher \ac{rd} gains.
The simulation results reveal an increase on the video quality when higher number of intra blocks are used.
Moreover, applying the proposed method only to the \ac{roi}, the bitrate overhead introduced is lower, leading to higher \ac{rd} gains.
\subsection{Leaky-predictions}
@@ -188,13 +193,13 @@ The simulation results reveal an increasing on the achieved video quality when h
Directly using reconstructed past frames as reference frames for predictive coding, may result in substantial error propagation, whenever the reference frames are either lost or reconstructed from other distorted frames.
To reduce error propagation one can employ leaky predictions, which scales down reconstructed frames before being used as reference~\cite{LeakyPred2009,ROILeakyPred2009}.
This leads to an exponential decay of error propagation, with direct impact on error robustness.
In this approach, the reconstructed reference frames are scaled down by applying the following:
In such approach, the reconstructed reference frames are scaled down by applying the following:
\begin{equation}\label{EQ2:LeakyPredConstant}
\check f_t^i = \alpha \hat f_t^i + (1- \alpha)C,
\end{equation}
where $\check f_t^i$ is a reference frame, $\alpha$ is the leaky factor used to control the amount of information from the reconstructed frame present in the new frame $\check f_t^i$, and $C$ is a constant value.
where $\hat f_t^i$ is a reference frame, $\alpha$ is the leaky factor used to control the amount of information from the reconstructed frame present in the new frame $\check f_t^i$, and $C$ is a constant value.
Leaky prediction has been used in the past for error control, especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in Equation~(\ref{EQ2:LeakyPredConstant}) is replaced by a lower layer error-free reconstructed frame. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate, as shown in \cite{LeakyPredConstant1973}. The $\alpha$ parameter allows to control the effect of the constant value, and thus the coding efficiency.
By decreasing $\alpha$, the coding efficiency increases, however, the exponential decay in error propagation becomes less significant.
By decreasing $\alpha$, the coding efficiency increases, but the exponential decay in error propagation becomes less significant.
% MHMCP_TCSVT2006
@@ -202,10 +207,10 @@ In order to overcome the loss in coding efficiency, weighted prediction can achi
Alternatively, the two previously encoded frames can be used to generate an interpolated frame ($ \check f_t^i $) to be used as reference, using:
%In case of leaky prediction instead of applying motion compensation is applied once, but with an interpolated frame obtained as follow:
\begin{equation}\label{EQ2:LeakyPredDual}
\tilde f_t^i = \alpha \hat f_t^i + (1- \alpha) \hat f_{n-1}^i.
\check f_t^i = \alpha \hat f_t^i + (1- \alpha) \hat f_{t-1}^i.
\end{equation}
Therefore, it is possible to decrease the error propagation using $\tilde f_t^i$ as reference frame for prediction, if only one of the original encoded frames is lost (i.e., $\hat f_{n}^i$ or $\hat f_{n-1}^i$). This is due to the fact that $\tilde f_t^i$ contains information from both $\hat f_{n}^i$ and $\hat f_{n-1}^i$.
In Equation (\ref{EQ2:LeakyPredDual}) $\hat f_{n}^i$ and $\hat f_{n-1}^i$ always correspond to the encoded frames and the generated frames ($\tilde f_t^i $) are not used to interpolate subsequent frames. This corresponds to a finite impulse response filter.
Therefore, it is possible to decrease the error propagation using $\check f_t^i$ as reference frame for prediction, if only one of the original encoded frames is lost (i.e., $\hat f_{t}^i$ or $\hat f_{t-1}^i$). This is due to the fact that $\check f_t^i$ contains information from both $\hat f_{t}^i$ and $\hat f_{t-1}^i$.
In Equation~(\ref{EQ2:LeakyPredDual}) $\hat f_{n}^i$ and $\hat f_{n-1}^i$ always correspond to the encoded frames and the generated frames ($\check f_t^i $) are not used to interpolate subsequent frames. This corresponds to a finite impulse response filter.
\begin{figure}[tb]
@@ -220,54 +225,54 @@ In Equation (\ref{EQ2:LeakyPredDual}) $\hat f_{n}^i$ and $\hat f_{n-1}^i$ always
\label{fig2:ErrorPropagationROI}
\end{figure}
In \cite{ROILeakyPred2009} a method is presented, where the leaky prediction is confined to a pre-defined \ac{roi}, in order to increase the error robustness with lower impact on the coding efficiency.
The method presented in~\cite{ROILeakyPred2009} uses leaky prediction confined to a pre-defined \ac{roi}, in order to increase the error robustness with lower impact on the coding efficiency.
Figure~\ref{fig2:ErrorPropagationROI} shows an example of error propagation within two different regions: \ac{roi} and background.
As show in this figure, since the \ac{roi} uses background regions for predictions, error propagation occurs across regions.
To overcome this problem, the method in \cite{ROILeakyPred2009} uses a weighted prediction for the \ac{roi} region obtained from the following predictions: (i) a globally optimal prediction obtained by unrestricted motion estimation, (ii) a prediction obtained by restricting motion estimation within the \ac{roi} region.
This limits the impact of errors in the background region on the \ac{roi}.
This limits the impact of errors in the background region to affect the \ac{roi}.
Moreover, since the leaky predictions are only confined to some specific regions, higher coding efficiency is achieved when compared with other methods.
The method proposed in \cite{ROPERefGenJan2010} extends the traditional leaky prediction expressed in Equation (\ref{EQ2:LeakyPredDual}) by replacing the previous encoded frame ($\hat f_{n-1}^i.$) with the previous interpolated frame ($\tilde f_t^i$). This results in the following:
The method proposed in \cite{ROPERefGenJan2010} extends the traditional leaky prediction expressed in Equation~(\ref{EQ2:LeakyPredDual}) by replacing the previous encoded frame ($\hat f_{t-1}^i.$) by the previous interpolated frame ($\check f_t^i$). This results in the following:
\begin{equation}\label{EQ2:LeakyPredIIR}
\tilde f_t^i = \alpha \hat f_t^i + (1- \alpha) \tilde f_{n-1}^i.
\check f_t^i = \alpha \hat f_t^i + (1- \alpha) \check f_{t-1}^i.
\end{equation}
This corresponds to infinite impulse response filter.
Comparing both equations for the same $\alpha$, the Equation (\ref{EQ2:LeakyPredIIR}) implies stronger filtering than Equation (\ref{EQ2:LeakyPredDual}). Therefore, reference frames generated with heavier filtering have higher resilience against error propagation, but less correlation with the original frame, which negatively impacts inter predictions and hence coding efficiency.
The idea of combining multiple predictions was also used in the \ac{hevc} standard by combining it with end-to-end distortion to select the optimal predictions~\cite{DualFrameME2015}.
The above equation corresponds to an infinite impulse response filter.
Comparing both Equations~\ref{EQ2:LeakyPredDual}~and~\ref{EQ2:LeakyPredIIR} for the same $\alpha$, the latter implies stronger filtering than the former.
Therefore, reference frames generated with heavier filtering have higher resilience against error propagation, but less correlation with the original frame, which negatively impacts inter predictions and hence coding efficiency.
The idea of combining multiple predictions was also applied in the \ac{hevc} standard by combining it with end-to-end distortion to select the optimal predictions~\cite{DualFrameME2015}.
\subsection{Reference picture selection}
Reference picture selection has been widely used in the past to reduce the temporal dependencies and reduce the error propagation \cite{ResilienceDualFrameBuffer2004,ErrorResilienceRPSFeb2009,RPSErrorsFeb2009,MVConcatenationAug2011}.
\ac{rps} has been widely used in the past to reduce the temporal dependencies and reduce the error propagation \cite{ResilienceDualFrameBuffer2004,ErrorResilienceRPSFeb2009,RPSErrorsFeb2009,MVConcatenationAug2011}.
In \cite{ResilienceDualFrameBuffer2004}, an optimisation algorithm was devised to select between short and long term references based on the \ac{rope} end-to-end distortion estimation method. This was used to increase the prediction distance and reduce error propagation.
The same method can also be used in combination with feedback information from the network to improve the references frame selection.
The same method can also be used in combination with feedback information from the network to improve the efficiency of \ac{rps}.
In the decoder-side, an \ac{ec} method was later used to take advantage of such prediction structure \cite{ConcealmentDualFrameBuffer2005}.
A feedback information from the network was also used in \cite{ErrorResilienceRPSFeb2009}, to reduce error propagation at end-user decoders, by dynamically selecting the reference frame, only from those correctly received.
A network feedback mechanism was proposed in \cite{ErrorResilienceRPSFeb2009}, to reduce error propagation at end-user decoders, by dynamically selecting the reference frame, only among those correctly received.
Figure~\ref{fig2:ReferencePictureSelection} shows an example of an error resilience transcoding scheme, where the original temporal predictions (represented by the solid arrows) are transcoded to use only correctly received frames as reference, i.e., prediction represented by the dashed arrow. This is done based on network feedback.
A more advanced scheme may be considered, where only the regions affected by mismatch predictions are processed in the compressed domain, and subsequently transcoded to limit the dependencies from lost frames.
To achieve low complexity, \ac{mv} concatenation can be used, in order to obtain new inter-predictions for error-free frames \cite{MVConcatenationAug2011}.
In \cite{ChannelAdaptiveVideoStreaming2002} network feedback is combined with path diversity to select the reference frames that are more likely to be received at the decoder.
In such approach, the feedback information is not directly used to select the reference frames.
Instead, the reference frames are selected through an end-to-end distortion estimation process, taking into account the coding distortion and distortion resulting from channel errors (\ac{ec} distortion is also taken into account).
In \cite{ErrorResilienceModeHEVC2017} a \ac{roi} is defined based on the moving objects to limit the regions where the \ac{rps} is applied.
Intra refresh is also used, based on the temporal distance to the previous intra-refresh frame.
This method not only achieves noticeable quality gains, when compared with the reference \ac{hevc} implementation, but also when compared with traditional reference selection schemes.
In~\cite{IntraRefreshRefSelectionJan2009} an algorithm to introduce intra refresh points at the \ac{mb} level was proposed.
This method takes into consideration the decoding distortion, assuming that frame-copy is applied for \ac{ec} when packet loss occurs.
The refresh blocks are encoded using either intra or inter prediction through a secondary algorithm to select the most reliable reference frame (i.e., frame with less total expected distortion).
\begin{figure}[tb]
\centering
\includegraphics[width=0.95\columnwidth]{./022_ErrorResilience/Figures/RefPicSelection.png}
\caption{Examples of dynamic reference picture selection used in error resilient video transcoding \cite{ErrorResilienceRPSFeb2009}.}
\label{fig2:ReferencePictureSelection}
\centering
\includegraphics[width=0.95\columnwidth]{./022_ErrorResilience/Figures/RefPicSelection.png}
\caption{Examples of dynamic reference picture selection used in error resilient video transcoding \cite{ErrorResilienceRPSFeb2009}.}
\label{fig2:ReferencePictureSelection}
\end{figure}
In \cite{ChannelAdaptiveVideoStreaming2002} network feedback is combined with path diversity to select the reference frames that are more likely to be received at the decoder.
In the proposed approach, the feedback information is not directly used to select the reference frames.
Instead, the reference frames are selected through an end-to-end distortion estimation process, taking into account the coding distortion and distortion resulting from channel errors (\ac{ec} distortion is also taken into account).
In \cite{ErrorResilienceModeHEVC2017} a \ac{roi} is defined based on the motion objects to limit the regions where the mechanism of reference picture selection is applied.
Intra refresh is also used, based on the temporal distance to the previous intra-refresh frame.
This method not only achieves noticeable quality gains, when compared with the reference \ac{hevc} standard, but also when comparing with traditional reference selection schemes.
In~\cite{IntraRefreshRefSelectionJan2009} an algorithm to introduce intra refresh points at the \ac{mb} level was proposed.
This method takes into consideration the decoding distortion assuming that frame-copy concealment is applied for \ac{ec} when packet loss occurs.
The refresh blocks are encoded using either intra prediction or inter prediction through a secondary algorithm to select the most reliable reference frame (i.e., frame with less total expected distortion).
In the methods mentioned above the dependencies between the frames are analysed and reduced in order to decrease the error propagation, in cases of data loss.
In the methods mentioned above, the dependencies between frames are analysed and reduced in order to decrease the error propagation, in case of errors or data loss.
Since the existing video coding standards are strongly based on predictive coding, these methods present an efficient approach in highly erroneous conditions.
% However, the method in~\cite{RDReferenceSelectionJul2012} uses more complex distortion estimations, and in ideal conditions multiple loop encoders are needed to select the reference frames optimally.
However, although these closed-loop techniques may offer a reliable control of the decoder distortion, the feedback required from the receiver side and multipath networking are not always available neither feasible in real-time and unidirectional channels (e.g. broadcast).
@@ -279,43 +284,46 @@ However, although these closed-loop techniques may offer a reliable control of t
\section{Data partitioning and redundant coding}
\label{Section22:DataPartitionRedundant}
In a compressed video streams different symbols have different relevance in terms of their impact on error propagation.
Therefore, there is an unequal contribution of different syntax elements to the decoding distortion by error propagation, depending on the lost symbols.
In compressed video streams different symbols have different relevance in terms of their impact on error propagation.
Therefore, there is an unequal contribution of different syntax elements to the decoding distortion and error propagation, depending on the symbols that are lost in each case.
Data partition techniques have been developed to provide a mechanism to cope with the unequal importance of coded information by grouping different parts of a compressed bitstream, according to their importance to the decoded quality.
Thus, some of these groups can be protected more efficiently than others using \ac{uep} schemes. These are normally associated with \ac{fec} algorithms, whereby multi-levels of \ac{fec} are applied to achieve different levels of error protection. This is presented and discussed with more detail in Sub-section~\ref{SEC22:UEP}
In a single channel transmission, the most important data or syntax elements (e.g., headers) can be protected with stronger channel codes than the remaining ones.
In single channel transmission, the most important data or syntax elements (e.g., headers) can be protected with stronger channel codes than the remaining ones.
In case of multi-channel communications, the important data can be routed over the most reliable channels.
Moreover, to achieve higher quality in a multi-channel transmission one can use extra information to increase the error resilience of video streams, in order to increase the probability of being able to decode them in case of errors.
For instance, redundancy can be added either in an explicitly (e.g., redundant picture) or implicitly (e.g., multiple description coding). Section~\ref{SEC22:RedundantPics}~and~\ref{SEC22:MDC} present a detailed review of these two approaches, respectively.
Moreover, to achieve higher quality in multi-channel transmission, one can use extra information to increase the error resilience of video streams, in order to increase the probability of being able to decode them in case of errors.
For instance, redundancy can be added either using explicit (e.g., redundant picture) or implicit (e.g., multiple description coding) mechanism. Section~\ref{SEC22:RedundantPics}~and~\ref{SEC22:MDC} present a detailed review of these two approaches, respectively.
\subsection{Standard methods for data partitioning and flexible macroblock ordering}
\subsection{Methods for data partitioning and macroblock ordering}
\glsreset{fmo}
The concept of data partitioning was firstly introduced by the H.263 coding standard to group the coded data into different categories~\cite{H264OverIPJul2003}.
It was further improved, allowing the symbols in each slice to be divided into three partitions:
The concept of data partitioning was firstly introduced in the H.263 coding standard to group the coded data into different categories~\cite{H264OverIPJul2003}.
Then, it was further improved in \ac{h264}, allowing the symbols in each slice to be divided into three partitions:
\begin{itemize}
\item Type A: headers, \ac{mb} types, quantisation parameters and motion information.
\item Type B: intra-coded information, such as, intra blocks signalling and intra coefficients. This type cannot be used without the partitions of type A being present.
\item Type C: inter-coded information and inter coefficients. This type also requires the information of type A, but can be decoded without intra-coded information (type B).
\end{itemize}
These partitions have different levels of importance, being data partitions of type A the most important as they carry the header information that are required by the remaining partitions. Then, as partitions of type B carry intra-coded information they are more important than type C information, as it can stop temporal error propagation.
These partitions have different levels of importance, according to the type of information carried in each one.
Data partitions of type A are the most important as they carry the header information that is required to decode the remaining partitions in the video stream.
Then, since partitions of type B carry intra-coded information, they are more important than type C, because they can stop temporal error propagation.
The \ac{h264} introduced an error resilience tool named \ac{fmo} technique, which allows each \ac{mb} to be freely assigned to a specific slice group (i.e., a set of slices), organised in a mixed-up fashion~\cite{H264FMO2006}.
An error resilience tool named \ac{fmo} technique was introduced in \ac{h264}, allowing each \ac{mb} to be freely assigned to a specific slice group (i.e., a set of slices), organised in a mixed-up fashion~\cite{H264FMO2006}.
It supports up to eight slice groups in each picture, and the \acs{mb} within a group can be assigned to several slices in the default raster scan order.
The case where only one slice group is used corresponds to the case where the \ac{fmo} tool is disabled.
\ac{fmo} increases the error resilience by partitioning the bitstream into several groups, which is able to reduce the error propagation.
Results show that the use of \ac{fmo} contributes to enhance the subjective and objective quality, especially for high packet loss rates at cost of an affordable overhead~\cite{H264FMO2006}.
Simulation results show that the use of \ac{fmo} contributes to enhance the subjective and objective quality, especially for high packet loss rates at cost of an affordable overhead~\cite{H264FMO2006}.
Six types of slice groups are supported by \ac{fmo}.
Figure~\ref{fig2:FMOTypes} shows \ac{fmo} types 0 to 5, since the type 6 as no defined pattern.
Figure~\ref{fig2:FMOTypes} shows \ac{fmo} types 0 to 5, since the type 6 has no defined pattern.
In the \ac{fmo} type 0 each slice group is defined by a maximum number of \acs{mb}, and a new slice group begins whenever his maximum value is reached. The same slice group can be cyclic repeated within a frame.
\ac{fmo} type 1 corresponds to scattered slices, where the \acs{mb} are assigned to each slice group with a fixed formula. Figure~\ref{fig2:FMOTypes} represents the \ac{fmo} type 1 with two slice groups resulting in a checkerboard pattern.
In the \ac{fmo} type 2, slice groups are defined by rectangular shapes bounded by the top-left and bottom-right corners.
Although the slice groups are allowed to overlap each other, each \ac{mb} may only belong to one slice group. \ac{fmo} type 3, 4 and 5 are special types, each one allowing the configuration to change on every frame.
Nevertheless, only two slice groups are allowed in \ac{fmo} type 3 to 5, in order to keep the signalling overhead affordable.
The last \ac{fmo} type is the most generic case, where the slice groups are explicitly defined using high level packets (i.e., signalling \ac{nal} units), allowing more flexibility at the cost of higher signalling overhead.
Although the slice groups are allowed to overlap each other, each \ac{mb} may only belong to one slice group.
Similar to \ac{fmo} type 0, in the types 3, 4 and 5, slice groups are defined by a maximum number of \acs{mb}. Nevertheless, these are special types, since it is allowed that the configuration to change on every frame.
In order to keep the signalling overhead affordable, only two slice groups are allowed in these types.
The \ac{fmo} type 6 is the most generic case, where the slice groups are explicitly defined using high level packets (i.e., signalling \ac{nal} units), allowing more flexibility at the cost of higher signalling overhead.
\begin{figure}[tb]
\centering
@@ -327,17 +335,18 @@ The last \ac{fmo} type is the most generic case, where the slice groups are expl
In the past, \ac{fmo} has been widely used to increase the error robustness of video transmissions in different communication scenarios \cite{FMOErrorResilience2007,FMOResilience2010,FMOResilience2011,FMOROI2012,MotionEnergyDataPart2014}.
In ~\cite{FMOErrorResilience2007} \ac{fmo} is used to increase the \ac{ec} performance and thus reduce the error propagation in prioritised video transmission.
The method divides the \acs{mb} into the two slice groups based on an estimation of the \ac{ec} distortion using a two-pass approach. Then, the slice group which has the most impact on the \ac{ec} distortion is transmitted using a reliable channel, and it is assumed to be received error-free.
The method divides the \acs{mb} into the two slice groups based on an estimation of the \ac{ec} distortion using a two-pass approach.
Then, the slice group which has the most impact on the \ac{ec} distortion is transmitted using a reliable channel, and it is assumed to be received error-free.
Results show that the proposed approach is able to outperform the dispersed mode (\ac{fmo} type 1).
Slice classification can also be based on the weighted sum of the motion vector information and the rate-distortion cost~\cite{FMOResilience2010}.
As in \cite{FMOErrorResilience2007}, the \acs{mb} are ranked based on the importance and distributed in the two groups with the same number of blocks.
As in \cite{FMOErrorResilience2007}, the \acs{mb} are ranked based on their importance and distributed in the two groups with the same number of blocks.
In order to reduce the mapping complexity of the \acs{mb} into groups, the same map can be repeated for several subsequent P-frames \cite{FMOResilience2011}.
This reduces the computational complexity of the \ac{mb} grouping while achieving similar error resilience performance.
The method in \cite{FMOROI2012} splits the video frames into two regions, which are then grouped in different slice groups.
To classify the different \acs{mb}, spatial information is extracted from the \ac{mb} bit allocation and temporal information is extracted based on the estimation of the \ac{ec} distortion.
This technique is able to reduce the number of erroneous \acs{mb} up to 82\%, achieving a quality gain of up to 3.97\%, in terms of \ac{psnr}, in comparison with the case where \ac{fmo} is not used.
Motion energy was also used to classify \acs{mb} into different groups \cite{MotionEnergyDataPart2014}, by comparing the motion energy of a given \ac{mb} with the motion energy of its neighbours and the average motion energy of the entire frame.
This allowed the authors to obtain a dynamic threshold which is then used to determine the \ac{mb} importance. In comparison with other \ac{uep} error protection schemes, this method achieve good quality improvements under different \acs{plr}.
This allowed the authors to obtain a dynamic threshold which is then used to determine the \ac{mb} importance. In comparison with other \ac{uep} error protection schemes, this method achieves good quality improvements under different \acs{plr}.
In \cite{FMORedudantSlices_TB2007}, a new \ac{fmo} type is proposed combining the slice groups of type 1 and 3 in order to obtain a checkerboard pattern in a box-out scanning order, with the objective of increasing the scattering of lost \acs{mb}. Results show that the proposed \ac{fmo} type is able to outperform the existing types under single and burst loss events.
The method described in \cite{FMOTypeROI2014} introduces a new type in order to efficiently divide the video content of each frame in two regions with different levels of importance, based on the \ac{fmo}~type~6.
@@ -351,12 +360,11 @@ By proposing a new \ac{fmo} type they are able to signal several \ac{roi} areas
\glsreset{uep}
\Gls{uep} techniques are used to protect different parts of a video stream against errors according to their importance for the decoding quality. \ac{uep} approaches take advantage of the video stream characteristics that result from different types of content, both at the temporal and spatial level. This allows categorisation of coded data into different groups, by exploiting the fact that different regions are affected differently by transmission errors.
This not only ensures that those regions will be delivered with higher quality, but also leads to efficient use of the total overhead required to achieve increased error robustness in a video transmission.
This not only ensures that more important regions will be delivered with higher quality, but also leads to efficient use of the total overhead required to achieve increased error robustness in a video transmission.
A wide variety of \ac{uep} schemes have been presented in the literature to optimise the performance of video communication over networks where the probability of packet loss is not negligible.
It has been observed that, within a \ac{gop}, different frame types (e.g. I, P, or B frames) have different levels of contribution to the reconstructed video quality, when transmission losses occur.
This has been exploited for \ac{uep} in video transmission, as shown in~\cite{UEP_TM2005,UEPMotion_TM2006}.
In \cite{UEPLayered_TB1999} an evaluation of layered video was proposed to increase the error robustness, by using redundant packets for helping the recovery of lost ones.
In order to achieve reduced overhead for error protection, each video layer has a different importance and is protected using different levels of redundancy, thus guaranteeing that packets carrying data of lower layers have higher probability of being decoded without errors.
This is a simple approach, not depending on the network capabilities, neither requiring a deep knowledge of the video contents.
@@ -372,7 +380,7 @@ This work was extended in~\cite{UEPMotion_ICSPCC2012,UEPMotion_ICSPCS2013} to us
Another work dealing with \ac{uep} is described in~\cite{Zhang20131390}, where the transmission distortion is estimated at the packet level, to assess the impact of each packet in the error propagation.
In this case, different priorities are applied to each packet, allowing different levels of error protection. This was also used to schedule the packet transmission through multi-path networks\cite{PacketScheduling2010}.
Furthermore, in order to improve the classification of different regions of a video stream a perceptual algorithm can be used.
Furthermore, in order to improve the classification of different regions of video streams, a perceptual algorithm can be used.
This strategy increases the correlation between the classification and subjective impact of errors.
In \cite{UEPSaliency_TCSVT2010} a method based on a combination of two parameters is described: a spatial parameter, based on a perceptual attention model, and a temporal parameter, based on the frame position within the \ac{gop}, which are used to improve the efficiency of \ac{fec} assignment. Experimental results show a significant quality improvement, both using objective metrics and subjective evaluation.
@@ -391,7 +399,7 @@ In~\cite{UEPVideoDegradation_ISCAS2017} a quality degradation model is proposed
simulating error events and measuring the quality using the \ac{psnr} metric.
From empirical results, a model was devised to predict the quality under different events.
Then, the model is used to control the amount of redundant codes to be applied at the network level.
The scalable extension of the \ac{hevc} standard was also used to support \gls{uep} for video storage in cloud environments~\cite{UEP_TM2018}, by combining the layer information (i.e., base or enhancement layer) and the picture type (i.e., I, P or B-frame) to maximise the performance of the \ac{fec} codes, both at the temporal and quality/resolution levels.
The scalable extension of the \ac{hevc} standard was also used to support \gls{uep} for video storage in cloud environments~\cite{UEP_TM2018}, by combining the layer information (i.e., base or enhancement layer) and the picture type to maximise the performance of the \ac{fec} codes, both at the temporal and quality/resolution levels.
@@ -408,7 +416,7 @@ For example, the primary picture may be coded with fine quantisation, while the
On the one hand, if the primary picture is received, the redundant information is discarded.
On the other hand, if errors occur and the primary picture is lost, the redundant one is used to provide a lower level of reconstructed quality and limiting the error propagation.
Zhu \emph{et al.} investigated the paradigm of redundant picture coding~\cite{RedudantPicsJan2009}, considering different approaches to develop a mixed method based on both reference picture selection to obtain a clean decoding, without error propagation, and hierarchical allocation of the redundant pictures to increase error robustness.
Zhu \emph{et al.} investigated the paradigm of redundant picture coding~\cite{RedudantPicsJan2009}, considering different approaches to develop a mixed method based on both \ac{rps} to obtain a clean decoding, without error propagation, and hierarchical allocation of the redundant pictures to increase error robustness.
In~\cite{VPxResiliencePCS2016} the key frames are encoded and transmitted at a lower bitrate to be used in case of error, to replace the missing frame.
Redundant pictures are a good mechanism to improve performance under error prone networking conditions, since they introduce secondary pictures that can be decoded when the primary picture is not available.
@@ -434,19 +442,18 @@ These approaches increase the overall system performance by presenting an effici
In \ac{mdc} schemes two or more bitstreams, named descriptions, are encoded and transmitted through different channels, such that an acceptable level of quality is achieved when only some of them reach the decoder \cite{MDC1993,MDC2005}.
Figure~\ref{fig2:MDCSystem} illustrates a basic architecture of a \ac{mdc} encoder and decoder using two descriptions.
In this example, the encoder-side creates two descriptions, i.e., $R_1$ and $R_2$, which shall be transmitted using two different channels, reducing the probability of the same video data being lost at the same time in both descriptions.
As shown in this diagram, three different outputs may be produced at the decoder-side: both descriptions are correctly received and the central decoder is used($D_0$), or either one of the descriptions is not received and then one of the side decoders is used ($D_{1,1}$ and $D_{1,2}$).
As shown in this diagram, three different outputs may be produced at the decoder-side: both descriptions are correctly received and the central decoder is used ($D_0$) or, either one of the descriptions is not received and then only one side decoder is used ($D_{1,1}$ or $D_{1,2}$).
When both bitstreams are received, the central decoder combines the information carried by each description achieving higher quality than any single side decoder.
In the \ac{mdc} encoder the redundancy among descriptions can be controlled by changing the correlation between them. Higher correlation leads to higher redundancy, but also higher quality in the output of the side decoders.
In contrast with the case of redundant pictures, the redundant \ac{mdc} streams do not carry the same information, which means that if both are received, then the decoded quality may be increased.
In \ac{mdc} encoding, the redundancy among descriptions can be controlled by changing the correlation between them.
Higher correlation leads to higher redundancy and also higher robustness, also leading to higher quality in the output of the side decoders.
In contrast, with the case of redundant pictures, the redundant \ac{mdc} streams do not carry the same information, which means that if both are received, then the decoded quality may be increased.
However, a combination of \ac{mdc} and redundant pictures can also be used to achieve higher quality gains \cite{MDCRedundantPicsJan2010}.
Different types of \ac{mdc} approaches \cite{MDCHierarBFeb2012,MDCCorreiaJun2012} can be used to improve the quality under transmission errors, either by taking advantage of hierarchical B-pictures~\cite{MDCHierarBFeb2012} or by introducing extra auxiliary information to limit the error propagation in case of single description decoding~\cite{MDCCorreiaJun2012}.
Although these techniques can be very effective in multi-path networks, a significant amount of overhead is required to create the different descriptions~\cite{MDC_TCSVT2011}.
% Double check this reference
%\HERE
%\cite{MDCRedundantPicsJan2010}
%\cite{MDC_TCSVT2011}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
@@ -456,8 +463,8 @@ Although these techniques can be very effective in multi-path networks, a signif
%\glsreset{ec}
In general, \ac{ec} functions are implemented as a post-processing step in the decoder and being the last module in the communication chain capable of handling errors~\cite{ConcealmentReview1998}.
Thus, reconstruction of missing data should take place within the video decoder to minimise the perceived effects of lost data.
In general, \ac{ec} functions are implemented as a post-processing step in the decoder, thus they are the last module in the communication chain capable of handling errors~\cite{ConcealmentReview1998}.
In this case, reconstruction of missing data should take place within the video decoder to minimise the perceived effects of lost data.
%Overall, to enhance the video quality, the design of the transmission system must incorporate adequate \ac{ec} tools.
This section reviews several \ac{ec} methods based on spatial and temporal redundancies, as these play an important role in the overall performance of robust communication systems.
@@ -493,40 +500,13 @@ improve the estimation accuracy of the lost region, the edges and image transiti
\label{FIG3:pixel_interpolation}
\end{figure}
Directional interpolation improves the performance of the pixel recovery, by firstly estimating the lost edges, using either correctly received or previously recovered pixels. To obtain the best interpolation direction, edges are estimated using the image gradient, for example applying the Sobel operator \cite{SignalImageVideoProcessing2012}.
%This operator gives the first-order approximation of the gradient of the image. This is widely used in image processing to locate the image edges, and uses an horizontal and vertical mask given by:
%\begin{equation}
%G_x =
%\begin{bmatrix}
%-1 & 0 & 1 \\
%-2 & 0 & 2 \\
%-1 & 0 & 1
%\end{bmatrix},\ \
%G_y =
%\begin{bmatrix}
%-1 & -2 & -1 \\
% 0 & 0 & 0 \\
% 1 & 2 & 1
%\end{bmatrix}.
%\end{equation}
%The convolution between the image pixels and the Sobel matrixes results in both horizontal ($g_x$) and vertical ($g_y$) gradients, from which the magnitude and the angular direction are computed using,
%\begin{equation}
%\begin{aligned}
%G &= \sqrt{g^2_x + g^2_y}, \\
%\theta &= tan^{-1} \left( \frac{g_y}{g_x} \right).\\
%\end{aligned}
%\end{equation}
The Sobel operator is an efficient algorithm to detect edges, as it can reach extreme values at the image edges.
Directional interpolation improves the accuracy of the pixel recovery, by firstly estimating the lost edges, using either correctly received or previously recovered pixels. To obtain the best interpolation direction, edges are estimated using the image gradient, for example applying the Sobel operator \cite{SignalImageVideoProcessing2012}.
This operator is an efficient algorithm to detect edges, as it can reach extreme values at the image edges.
The usage of this operator for \ac{ec} was evaluated in \cite{IntraConc2010,FineDirectTCE2006,SpatialEdgeConsecutiveBlocks2004,SpatialEdgeModConsecutiveBlocks2005,SpatialInterpolation2008}, and it is characterised as a fast and efficient method due to its small kernel \cite{GradEstimation2009}.
% \abbrev{FC}{Frame-Copy}
% \abbrev{MC}{Motion-Copy}
\subsection{Temporal error concealment techniques}
%\subsubsection{Frame and motion copy}
The conventional way to recover missing frames in the temporal domain are the so-called \ac{fc} and \ac{mc}.
The conventional methods used to recover missing frames in the temporal domain are the so-called \ac{fc} and \ac{mc}.
These \ac{ec} methods were implemented for the \ac{h264} reference software in \cite{H264RefConc2006}.
The \ac{fc} method consists in repeating (copying) the frame that has the shortest temporal distance, in order to substitute the missing one, into the time instant where the loss occurred.
The \ac{mc} method uses the motion information of the closest frame to recover the missing one, thus the missing pixels are recovered through motion compensation.
@@ -539,13 +519,15 @@ Results in \cite{BiDirMC2011} show quality gains up to $4.99$~dB (\ac{psnr}) whe
In \cite{2DRecursiveRefinement2010} the \ac{mc} method performance was increased by using a refinement algorithm to improve the accuracy of the \acs{mv} obtained from the closest decoded frame.
The \ac{mv} differences between the vectors of successive frame pairs are recursively calculated along with the next refinement, until the changes become negligibly small (e.g., close to zero).
This approach has shown to improve the performance of the \ac{mc} method, with gains up to $0.47\ dB$ (\ac{psnr}) for a \ac{plr} of $20\%$.
Moreover,
\subsubsection{Boundary match algorithms}
% \abbrev{ICMV}{Interpolated Candidate Motion Vectors}
\Gls{bma} are used to recover a missing region based on the available neighbouring regions. By using the missing block neighbourhood, a new \ac{mv} can be estimated for the missing block. Then, recovery based on motion compensation can be achieved.
A low complexity approach may use the boundary pixels to search for the \ac{mv} based on a spatial smoothness constraint imposed on the boundaries of the lost block \cite{BMAAlgorithm2008}. Figure~\ref{FIG3:BMAPixelInformation} represents the boundary information used to estimate the \ac{mv} for the lost block from a set of different candidates.
\Gls{bma} are used to recover a missing region based on the available neighbouring regions.
By using the missing block neighbourhood, a new \ac{mv} can be estimated for the missing block.
Then, the recovery of the missing pixels if accomplished based on motion compensation.
A low complexity approach may use the boundary pixels to search for the \ac{mv}, based on a spatial smoothness constraint imposed on the boundaries of the lost block \cite{BMAAlgorithm2008}. Figure~\ref{FIG3:BMAPixelInformation} represents the boundary information used to estimate the \ac{mv} for the lost block from a set of different candidates.
The \ac{bma} uses the available \acs{mv} provided by the top (T), bottom (B), left (L) and right (R) blocks to simulate different reconstructions of the lost block.
Then, the optimal reconstruction is selected according to the sum of differences, between the pixels on the internal boundary in the recovered block and the corresponding ones on the external boundary (see Figure~\ref{FIG3:BMAPixelInformation}).
@@ -558,70 +540,35 @@ Then, the optimal reconstruction is selected according to the sum of differences
\end{figure}
An implementation of the \ac{bma} was proposed in \cite{BMAAlgorithm2006}, by analysing the change of magnitude and angle between the \acs{mv} of the neighbouring blocks and the co-located ones in the previous frames. The average change is applied to the \ac{mv} of the block co-located with the lost one to compute the \ac{mv} candidate for recovery.
An improvement to \ac{bma} was obtained with the outer boundary matching algorithm\cite{OBMAMethod2008}, also known as decoder \ac{mv} estimation algorithm in \cite{DMVE2000}.
An improvement to \ac{bma} was developed using with the outer boundary matching algorithm\cite{OBMAMethod2008}, also known as decoder \ac{mv} estimation algorithm in \cite{DMVE2000}.
In this method the distortion is calculated by the difference of two outer boundaries of reconstructed block, instead of an internal and external boundary in \ac{bma}.
Although \ac{bma} and outer \ac{bma} have the same design principle of minimising the mismatch distortion, in the latter the distortion is measured in the outer boundaries, \emph{i.e.}, external boundaries of the reconstructed block.
It was observed in the experimental results of \cite{OBMAMethod2008} that the outer \ac{bma} has better performance in video recovery than conventional \ac{bma}. The performance improvements are explained by the distortion minimisation of the outer boundary, which incorporates the edge information located in the neighbourhood of the lost block in the decision process to select the most accurate \ac{mv}.
%Moreover, in the method presented in \cite{OBMAMethod2008} further \ac{mv} candidates were introduced, a refined local search is performed and multiple boundary layers, \emph{i.e.}, increase the overlapped region to measure the distortion and to select the final MV, were tested. The experimental results revealed that the usage of a single layer match gives better performance than multiple layers, while selective search gives excellent trade-off between the image recovery and computation complexity
% In \cite{DynamicBMA2010} a novel approach to measure the spatial distortion on boundaries of the reconstructed block is presented. Instead of using only horizontal and vertical directions, diagonal and anti-diagonal are also tested, and the best direction is chosen from the reference frame. Since block boundaries may change abruptly, instead of minimising the distortion function the proposed method aims to approximate this distortion to an optimal difference. This optimal difference are determined in the neighbourhood of the co-located block in the reference frame.
In \cite{BMAAlgorithm2008} a novel distortion function was proposed, to choose among different candidate MVs. The distortion function exploits both spatial and temporal smooth constraints. The temporal function is based on the outer boundary distortion, as in \cite{OBMAMethod2008}.
The spatial distortion aims to reduce the gradient variances in boundaries of the reconstructed block, so the image edges may be guaranteed.
Several experimental results were obtained for different loss events, covering from an isolated block loss to the loss of a complete row of blocks. The proposed method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method. Moreover, subjective analysis reveal that this method recover the edges in the lost image. However, the method uses complex mathematical operations, which may not be practical in real time applications.
%Several experimental results were obtained for different loss events, covering from an isolated block loss to the loss of a complete row of blocks.
This method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method.
However, although subjective analysis revealed that this method is able to recover the edges in the lost image, complex mathematical operations are required, which may not be practical in real time applications.
\subsubsection{Motion vector extrapolation}
As mentioned before, the \ac{mc} method is a simple way to reconstruct an estimation of a missing frame using motion information from previously decoded frames. Although this method can provide an acceptable performance comparing with the \ac{fc}, as shown in \cite{H264RefConc2006}, more complex algorithms that use the assumption of motion continuity across consecutive frames are proposed to recover, not only missing blocks, but also whole frames in compressed video.
% \abbrev{MMA}{Multi-frame Motion vector Averaging}
% To recover the entire missing frame a multi-frame motion vector averaging (MMA) algorithm was proposed in \cite{MMATempConc2003}. The method uses \acs{mv} of a few past frames (from $t-1$ to $t-L$), in order to estimate the forward \ac{mv} for each pixel in the last received frame. Then, each pixel of the last frame is projected onto the missing one. More specifically, the method proposed in \cite{MMATempConc2003} recovers the missing frame at instant $t$ using the following steps:
% \begin{enumerate}
% \item Generation of a \ac{mv} history based on the previously decoded frames from $t-1$ to $t-L$, with $L \in \left[ 2,5 \right]$, in order to understand which pixels were used to predict the ones of the frame at $t-1$.
% \item The \ac{mv} history is used to obtain a set of forward MV, from the frame $t-1$ to the missing one (at instant $t$). For each pixel, the forward \ac{mv} is measured by averaging the \ac{mv} history.
% \item A spatial regularisation of the \ac{mv} field is performed, by applying a median filter with a kernel size of $12$, to approximate an \mbox{edge-preserving} regularisation \cite{MedianFilterRegul2000}.
% \item The reconstruction of the missing frame is performed, at a half-pixel level, through the previously obtained forward MVs. The overlapped pixels are averaged, in order to obtain the final pixel value.
% \item The unfilled regions are recovered, with the median value of a $7\times7$ window
% \item Finally, the fully reconstructed frame at a half-pixel level is filtered using a mean filter with a $2\times2$ kernel, and down sampled.
% \end{enumerate}
% This method usually produces relatively high quality in the reconstructed frame, but in low motion cases it behaves worse than the \ac{fc} method under a few conditions, as indicated by the experimental results in \cite{MMATempConc2003}. This phenomenon happens because the oldest reference frames usually do not have \ac{mv} similar to the missing one. Moreover, if $L$ increases, the complexity of the proposed algorithm may not be useful to be used in \ac{ec}, due to complexity and memory requirements to determine the \ac{mv} history.
As mentioned before, the \ac{mc} method is a simple way to reconstruct an estimation of a missing frame using motion information from previously decoded frames.
Although this method can provide an acceptable performance comparing with the \ac{fc}, as shown in \cite{H264RefConc2006}, more complex algorithms that use the assumption of motion continuity across consecutive frames are used to recover the missing information.
\glsreset{mve}
\ac{mve} \cite{BlockBasedMVE2002,BlockBasedMVEOverllaped2011} methods are quiet simple yet efficient techniques that reconstruct the \acs{mv} of a lost frame, which in turn are used for motion compensation.
\ac{mve} \cite{BlockBasedMVE2002,BlockBasedMVEOverllaped2011} methods are efficient techniques to reconstruct the \acs{mv} of a lost frame, which in turn are used for motion compensation.
These \acs{mv} are extrapolated from the last decoded frame \mbox{(reference frame)} onto the missing one~\cite{BlockBasedMVE2002}.
Each block in the reference frame is projected to the missing one with its own direction given by the corresponding extrapolated \ac{mv} ($mv'$).
An example of \ac{mve} is shown in Figure~\ref{FIG3:MVEExtrapolation}, where the vectors of frame $t-1$ are projected onto the missing frame at instant $t$. Note that, in the reference frame ($f_{t-1}$) the \acs{mv} may point to different frames ($f_{t-2}$~and~$f_{t-3}$), previously decoded, so \ac{mve} must take into account the temporal distance between different frames.
To improve the accuracy of \ac{mve}, the damaged region is divided smaller blocks and, according to the matching area between the extrapolated block and the damaged one, the best \ac{mv} is chosen.
In order words, the \ac{mv} for the each block is obtained from the extrapolated block with the highest number of pixels matching the missing locations.
Note that the final motion information need to be aligned to a fixed sise grid, e.g., $4\times4$ grid, which leads to a misalignment of the extrapolated \acs{mv}.
Note that the final motion information need to be aligned to a fixed size grid, e.g., $4\times4$ grid, which leads to a misalignment of the extrapolated \acs{mv}.
Alternatively, in \cite{PixelBasedMVE2004} several blocks are extrapolated to the missing frame, resulting in overlapped regions, which are recovered using a weighted average of the multiple pixels candidates.
The weight coefficients assume a maximum value in the center of the extrapolated region, and a minimum value at the boundaries.
%
%Since the choice of the best candidate \ac{mv} relies on the overlapped area of the extrapolated block with the lost region, the method may give erroneously estimated \ac{mv} in large motion scenes. The example of Figure~\ref{FIG3:MVEPixelBased} shows a lost $8\times8$ block (in the centre of the figure) and four possible extrapolated \acs{pu} that can be used to estimate lost the motion information. As illustrated in the figure, although the largest overlapped area belongs to the PU$_2$, the overlapped pixels correspond to a small portion of the whole missing block. Moreover, the majority of the lost block's area was not covered by any extrapolated \ac{pu}. Thus, motion recovery using only the information available in PU$_2$ will not properly characterise the motion information for all lost pixels.
%
%\begin{figure}[t]
%\centering
%\input{./022_ErrorResilience/Figures/PixelBasedMVE}
%\caption{Pixel based motion vector extrapolation \cite{PixelBasedMVE2004}.}
%\label{FIG3:MVEPixelBased}
%\vspace{3mm}
%\end{figure}
%
%To overcome the previous limitation and to achieve smoother and finer results, a \ac{pmve} algorithm was proposed in \cite{PixelBasedMVE2004}. This motion vector extrapolation method is divided into two cases:
%\begin{enumerate}
% \item The pixels in the lost region that are covered by at least one extrapolated block, such as circle points in Figure~\ref{FIG3:MVEPixelBased}, are recovered using the average of the \acs{mv} of the overlapped block.
% \item Those pixels which are not covered by any extrapolation, such as the triangle point in Figure~\ref{FIG3:MVEPixelBased}, the \ac{mv} of the co-located pixel in the previous frame is used.
%\end{enumerate}
%
%Then, if the estimated \ac{mv} is $(MV_x,MV_y)$, then the estimation of the missing pixel, $p_m(x,y)$, is obtained by:
%\begin{equation}\label{EQ:MotionComp}
% p_m(x,y) = p_r(x+MV_x,y+MV_y),
%\end{equation}
%where $p_r$ refers to the pixels in the previous frame. This pixel-based \ac{mve} provides similar results than the block-based \ac{mve} algorithm in low motion scenarios. Nevertheless, it is able to overcome the previous method in large motion scenes.
An extension to the previous approach was considered in \HIGHLIGHT{\cite{PelBasedMVE_TCE2007}}, by using the \acs{mv} of subsequent available frames. Thus, to estimate the lost motion information, forward and backward extrapolations are performed, in order to obtain two candidates to each lost pixel. Then, the lost pixels are obtained through motion compensation using the \acs{mv} determined by the average of two \ac{mv} candidates. The experimental results revealed that the bi-directional \ac{pmve} outperforms the isolated use of forward or backward \ac{pmve}. Moreover, the bi-directional \ac{pmve} achieves gains up to $1.85$ dB and $1.32$ dB (\ac{psnr}) comparing with the work in \cite{MMATempConc2003} and \ac{mve} \cite{BlockBasedMVE2002}.
An extension to the previous approach was considered in \cite{PelBasedMVE_TCE2007}, by using the \acs{mv} of subsequent available frames. Thus, to estimate the lost motion information, forward and backward extrapolations are performed, in order to obtain two candidates to each lost pixel. Then, the lost pixels are obtained through motion compensation using the \acs{mv} determined by the average of two \ac{mv} candidates. The experimental results revealed that the bi-directional \ac{pmve} outperforms the isolated use of forward or backward \ac{pmve}. Moreover, the bi-directional \ac{pmve} achieves gains up to $1.85$ dB and $1.32$ dB (\ac{psnr}) comparing with the work in \cite{MMATempConc2003} and \ac{mve} \cite{BlockBasedMVE2002}.
% TODO: Check reference PelBasedMVE_TCE2007
\begin{figure}[t]
\centering
@@ -631,12 +578,9 @@ An extension to the previous approach was considered in \HIGHLIGHT{\cite{PelBase
\end{figure}
In \cite{2DConcHybridMotion2010} a \ac{hmve} algorithm was proposed, to improve the accuracy of the \acs{mv} of the \ac{pmve} method, by using the extrapolated vectors at the pixel and block level. Moreover, the \acs{mv} that were wrongly extrapolated are discarded in order to obtain a more accurate MV.
In this \ac{ec} method, firstly a pixel based motion extrapolation is performed and the pixels are organised into three groups.
%Group A contains the pixels that are overlapped by at least one extrapolated block; group B includes pixels that are not covered by any extrapolated block, but belong to a block that is covered, \emph{e.g.}, in Figure~\ref{FIG3:MVEPixelBased} the triangle point was not covered by any extrapolated \ac{mb} but it belongs to $8\times8$ block that is covered by 4 blocks.
%The remaining pixels are included in group C (star pixels in the figure).
In this \ac{ec} method, firstly a pixel-based motion extrapolation is performed and the pixels are organised into three groups.
Then, through block-based \ac{mve} two new \acs{mv} are determined: the first one corresponds to the \ac{mv} of the most overlapped block with the current one, and second one is obtained by the weighted average value of the \acs{mv} corresponding to different overlapped blocks.
To find the more accurate \ac{mv} from the \ac{mv} set composed described above. To achieve this, the \ac{mv} set is further refined using the distance between each pair of \acs{mv}, in order to discard the wrongly extrapolated \acs{mv}.
% Finally, the pixels of the lost regions (belonging to the groups A and B) are obtained through motion compensation, using the average of the remaining vectors in the \ac{mv} set. The pixels of the group C, although the \ac{mv} set is empty, the same process is applied using the \ac{mv} of the co-located pixel in the previous frame.
This algorithm is able to outperform the \ac{pmve} for different packet loss probabilities, as shown by the results in \cite{2DConcHybridMotion2010}.
The method was used to recover both intra and inter frames in video sequences compressed with the \ac{h264}, achieving \ac{psnr} gains above $1.09$ dB in the erroneous frames.
@@ -645,15 +589,15 @@ For instance, in~\cite{HEVCMCBlockAug2013} the block partitions of the neighbour
The reconstructed frame keeps smooth boundaries, leading to higher objective and visual quality.
Moreover, in order to select only the reliable motion information, the residue information can also be used to decide the optimal \ac{mv}, as shown in~\cite{HEVCMCResidueDec2013}.
Overall, the aforementioned methods rely on post-processing without any assistance from the encoder-side, which usually leads to inaccurate recovery of the lost information. Thus, in order to achieve higher reconstruction quality, further information need to be shared with the decoder to minimise the residual distortion of \ac{ec}.
Overall, the aforementioned methods rely on post-processing without any assistance from the encoder-side, which usually leads to inaccurate recovery of the lost information. Thus, in order to achieve higher reconstruction quality, further information needs to be shared with the decoder to minimise the residual distortion of \ac{ec}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Concealment-aware error resilience}
\section{Error concealment-aware error resilience}
\label{Section22:ConcealmentAware}
The last category of techniques for robust video transmission address those algorithms that help the \ac{ec} process at the decoder-side. Although such techniques might overlap with some of the concepts described above, such as, data partition or redundant information, they have the main goal of improving the \ac{ec} performance. This class of algorithms combine the error resilience capabilities with \ac{ec} methods to achieve an improved overall performance.
The last category of techniques for robust video transmission address those algorithms that help the \ac{ec} process at the decoder-side. Although such techniques might overlap with some of the concepts described above, such as, data partition or redundant information, they have the main goal of improving the \ac{ec} performance. This class of algorithms combine the error resilience capabilities with \ac{ec} methods to achieve an improved overall performance achieving better video quality than in individual \ac{ec} methods.
One of the approaches that can be used to improve the \ac{ec} performance is to estimate, at the encoding side, the \ac{ec} distortion introduced by the decoder in case of data loss.
In \cite{ConcealmentDrivenSliceOrdering2008} the \ac{ec} distortion estimation is used for slice classification and \ac{uep}. Such distortion is also used to optimise motion estimation~\cite{ConcealmentAwareME2006}, by including the distortion of the subsequent frame (assuming it is lost and recovered) in the \ac{rd} cost optimisation of the current frame. This was also used in the context of \ac{hevc} to improve its error robustness, as shown in \cite{ErrorResilienceModeHEVC2016}. This kind of approaches take advantage of location techniques and data partitioning to improve the reconstruction quality achieved by \ac{ec} algorithms. Normally, they use a single \ac{ec} method that is implemented both at the encoder and decoder, based on temporal algorithms, e.g., \ac{mc} and \ac{mve}.