Final revised version
This commit is contained in:
1 parent
59d007205a
commit
3b63e37cc4
14 files changed
+119
-118
No files matched your search
@@ -7,14 +7,12 @@
|
||||
\vspace*{\fill}
|
||||
\section{Introduction}
|
||||
|
||||
|
||||
|
||||
In general, digital communications through networks are prone to errors or data loss.
|
||||
In general, digital communications through networks are prone to transmission errors and packet loss.
|
||||
These errors can be handled in different ways, such as retransmission, error correction, and \ac{ec} \cite{ErrorResilience2000,PixelInterlacingIntraConcealment2011}.
|
||||
This chapter reviews the existing techniques for robust video coding that can be applied at the \ac{vcl} to reduce the error propagation and increase the decoded video quality.
|
||||
Several techniques have been proposed in the past to deal with the problem of robust video transmission, especially for video coding standards with high temporal dependencies, such as \ac{h264} and more recently \ac{hevc}.
|
||||
These techniques are normally focused on the spatio-temporal dependencies in the coded streams, which are the basis of high coding gains but lead to low error robustness~\cite{ROPE2007}.
|
||||
In this chapter the error resilience and error concealment schemes are divided in different categories, depending on their functionality and algorithms.
|
||||
In this chapter the error resilience and error concealment schemes are classified in different categories, depending on their functionality and algorithms.
|
||||
Although each approach is based on different techniques they all aim at striking the best trade-off between coding efficiency and decoded video quality under lossy conditions.
|
||||
|
||||
The chapter is organised as follows.
|
||||
@@ -91,7 +89,7 @@ In both examples, the regions affected by errors are clearly reduced.
|
||||
|
||||
|
||||
Common localization techniques deal with the error resilience problem within the scope of the \ac{rd} optimisation process used in video coding.
|
||||
Normally, such optimisation is equivalent to the minimisation of the Lagrangian optimisation cost ($J$) \cite{RateDistortion_SPM1998} given by:
|
||||
Normally, such optimisation is equivalent to the minimisation of the Lagrangian cost ($J$) \cite{RateDistortion_SPM1998} given by:
|
||||
\begin{equation}\label{EQ2:LagragianCost}
|
||||
J = D + \lambda R,
|
||||
\end{equation}
|
||||
@@ -175,7 +173,7 @@ When compared with traditional intra-refresh, bitrate savings of up to 12\% are
|
||||
|
||||
A different approach is to introduce refresh points based on the video content.
|
||||
In~\cite{IntraRefreshAttentionCM2007} the error resilience of video streams is increased by combining rate control and attention-based models to define \acs{roi}.
|
||||
A weighting factor is used in the original Lagrangian optimization~\cite{RateDistortion_SPM1998}, as given by (\ref{EQ2:LagragianCost}), to increase the number of intra-coded blocks in the image regions determined by the attention models.
|
||||
A weighting factor is introduced in the conventional Lagrangian cost~\cite{RateDistortion_SPM1998}, to increase the number of intra-coded blocks in the image regions defined by the attention models.
|
||||
Figure~\ref{FIG2:IntraRefresh} (b) shows an example with a \ac{roi} marked with a black line, which varies for every frame. In this case, the intra-coded blocks are only introduced inside the marked region, in contrast to what is shown in Figure~\ref{FIG2:IntraRefresh} (a).
|
||||
Results show that by using the aforementioned method, less refresh points are required to improve the error robustness of the \ac{roi}. When comparing with \cite{DistortionTrackingH264_ICIP2004}, approximately 4 dB of \ac{psnr} gains are achieved in the \ac{roi} without compromising the overall video quality.
|
||||
A more recent approach was proposed using a perceptual noticeable difference model to estimate the noticeable distortion, and then using a visual attention model \cite{SaliencyItti1998} to decide which blocks should be used as intra refresh in the bitstream~\cite{IntraRefreshPerceptual2010}.
|
||||
@@ -198,7 +196,7 @@ In such approach, the reconstructed reference frames are scaled down by applying
|
||||
\end{equation}
|
||||
where $\hat f_t^i$ is a reference frame, $\alpha$ is the leaky factor used to control the amount of information from the reconstructed frame present in the new frame $\check f_t^i$, and $C$ is a constant value.
|
||||
Leaky prediction has been used in the past for error control, especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in~(\ref{EQ2:LeakyPredConstant}) is replaced by a lower layer error-free reconstructed frame. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate, as shown in \cite{LeakyPredConstant1973}. The $\alpha$ parameter allows to control the effect of the constant value, and thus the coding efficiency.
|
||||
By decreasing $\alpha$, the coding efficiency increases, but the exponential decay in error propagation becomes less significant. For the determination of the $\alpha$ parameter, one can use network information such as packet loss probability $p$ leading to $\alpha=(1-p)$. However, as can be seen in previous studies, empirical approaches using exhaustive can also be used to optimise the $\alpha$ value for each sequence~\cite{ROPERefGenJan2010}.
|
||||
\HIGHLIGHT{By decreasing $\alpha$, the coding efficiency increases, but the exponential decay in error propagation becomes less significant. To determine $\alpha$, one can use network feedback information, such as packet loss probability $p$, resulting in $\alpha=(1-p)$. Alternatively, empirical approaches using exhaustive search can also be used to find optimal values of $\alpha$ for each sequence~\cite{ROPERefGenJan2010}.}
|
||||
|
||||
% MHMCP_TCSVT2006
|
||||
|
||||
@@ -210,9 +208,7 @@ Alternatively, the two previously encoded frames can be used to generate an inte
|
||||
\end{equation}
|
||||
Therefore, it is possible to decrease the error propagation using $\check f_t^i$ as reference frame for prediction, if only one of the original encoded frames is lost (i.e., $\hat f_{t}^i$ or $\hat f_{t-1}^i$). This is due to the fact that $\check f_t^i$ contains information from both $\hat f_{t}^i$ and $\hat f_{t-1}^i$.
|
||||
In~(\ref{EQ2:LeakyPredDual}) $\hat f_{t}^i$ and $\hat f_{t-1}^i$ always correspond to the encoded frames and the generated frames ($\check f_t^i $) are not used to interpolate subsequent frames. This corresponds to a finite impulse response filter.
|
||||
Although the use of two reference frames for leaky prediction is an effective way to reduce the error propagation with a small penalty in coding efficiency, the use of an interpolated frame in (\ref{EQ2:LeakyPredDual}) still leads to a severe error propagation in case of burst frame loss.
|
||||
In this case both frames would be affect by packet loss, which would reduce the effectiveness of the method.
|
||||
|
||||
In the case of burst losses where both $ \hat f_t^i $ and $ \hat f_{t-1}^i $ are missing at the decoder, the interpolated frame ($\check f_t^i$) suffers from severe drift, which significantly decreases its quality, because $ \hat f_t^i $ and $ \hat f_{t-1}^i $ are recovered using error concealment algorithms, i.e., they are not correctly decoded.
|
||||
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
@@ -222,7 +218,7 @@ In this case both frames would be affect by packet loss, which would reduce the
|
||||
% \end{tabular}
|
||||
% \caption{Example of error propagation with a ROI (shaded area indicates error, and the arrows indicate temporal predictions to previous frame) \cite{ROILeakyPred2009}.}
|
||||
\input{./022_ErrorResilience/Figures/ROILeakyPrediction.tikz}
|
||||
\caption{Example of error propagation with two different regions; as shown errors can propagate from the background to the ROI.}
|
||||
\caption{Example of error propagation across two different regions: errors from background affect the ROI.}
|
||||
\label{fig2:ErrorPropagationROI}
|
||||
\end{figure}
|
||||
|
||||
@@ -252,8 +248,8 @@ In \cite{ResilienceDualFrameBuffer2004}, an optimisation algorithm was devised t
|
||||
In the decoder-side, an \ac{ec} method was later used to take advantage of such prediction structure \cite{ConcealmentDualFrameBuffer2005}.
|
||||
|
||||
A network feedback mechanism was proposed in \cite{ErrorResilienceRPSFeb2009} to reduce error propagation at end-user decoders, by dynamically selecting the reference frame only among those correctly received.
|
||||
Figure~\ref{fig2:ReferencePictureSelection} shows an example of an error resilience transcoding scheme, where the original temporal predictions (represented by the solid arrows) are transcoded to use only correctly received frames as reference, i.e., prediction represented by the dashed arrow. This is done based on network feedback.
|
||||
\HIGHLIGHT{As this technique requires transcoding operations, the complexity increase with the \ac{gop} length, as reported in~\cite{ErrorResilienceRPSFeb2009}. Typically, the \ac{gop} length is defined by taking into consideration the video application (i.e., synchronisation, random-access or maximum refresh period allowed) and not by the robust coding scheme.}
|
||||
Figure~\ref{fig2:ReferencePictureSelection} shows an example of an error resilience transcoding scheme, where the original temporal predictions (represented by the solid arrows) are transcoded to only use correctly received frames as reference, i.e., prediction represented by the dashed arrow. This is done based on network feedback.
|
||||
The length of the \ac{gop} has several implications on the performance of this transcoding scheme. For instance the complexity increases with the \ac{gop} length, because the number of frames that must be decoded also increases~\cite{ErrorResilienceRPSFeb2009}. An absolute optimal \ac{gop} length was not found in~\cite{ErrorResilienceRPSFeb2009}, since it mostly depends on the application requirements (i.e., random-access, maximum refresh period, etc).
|
||||
A more advanced scheme may be considered, where only the regions affected by mismatch predictions are processed in the compressed domain, and subsequently transcoded to limit the dependencies from lost frames.
|
||||
To achieve low complexity, \ac{mv} concatenation can be used, in order to obtain new inter-predictions for error-free frames \cite{MVConcatenationAug2011}.
|
||||
|
||||
@@ -290,8 +286,8 @@ In compressed video streams different symbols have different relevance in terms
|
||||
Therefore, there is an unequal contribution of different syntax elements to the decoding distortion and error propagation, depending on the symbols that are lost in each case.
|
||||
Data partition techniques have been developed to provide a mechanism to cope with the unequal importance of coded information by grouping different parts of a compressed bitstream, according to their importance to the decoded quality.
|
||||
Thus, some of these groups can be protected more efficiently than others using \ac{uep} schemes.
|
||||
These are normally associated with \ac{fec} algorithms, which are typically applied at the channel level which is not aware of the video content, leading to a sub-optimal error protection~\cite{UEPMotion_TM2006}.
|
||||
Using \ac{uep}, a source-channel optimisation is applied, whereby multi-levels of \ac{fec} are used to achieve different levels of error protection according to the importance of the coded information.
|
||||
\HIGHLIGHT{These are normally associated with \ac{fec} mechanisms, which are typically applied at the channel level without taking into consideration that not all video data is equally important. This leads to a sub-optimal error protection~\cite{UEPMotion_TM2006}.
|
||||
In general, when using \ac{uep}, source-channel optimisation is applied, whereby multi-levels of \ac{fec} are used to achieve different levels of error protection according to the importance of coded video data.}
|
||||
This is presented and discussed with more detail in Sub-section~\ref{SEC22:UEP}
|
||||
|
||||
In single channel transmission, the most important data or syntax elements (e.g., headers) can be protected with stronger channel codes than the remaining ones.
|
||||
@@ -327,7 +323,7 @@ In the \ac{fmo} type 0 each slice group is defined by a maximum number of \acs{m
|
||||
In the \ac{fmo} type 2, slice groups are defined by rectangular shapes bounded by the top-left and bottom-right corners.
|
||||
Although the slice groups are allowed to overlap each other, each \ac{mb} may only belong to one slice group.
|
||||
Similar to \ac{fmo} type 0, in the types 3, 4 and 5 in Figure~\ref{fig2:FMOTypes}, slice groups are defined by a maximum number of \acs{mb}.
|
||||
However, contrary of \ac{fmo} type 0, the configuration is allowed to change every frame for these types
|
||||
However, contrary to \ac{fmo} type 0, the slice groups are allowed to change every frame for types 3, 4 and 5.
|
||||
In order to keep the signalling overhead affordable, only two slice groups are allowed in these types.
|
||||
The \ac{fmo} type 6 is the most generic case, where the slice groups are explicitly defined using high level packets (i.e., signalling \ac{nal} units), allowing more flexibility at the cost of higher signalling overhead.
|
||||
|
||||
@@ -486,7 +482,8 @@ A typical approach to recover missing regions in an image is to estimate the los
|
||||
Figure~\ref{FIG3:pixel_interpolation} shows the estimation process of a lost pixel ($lp$), within the lost region ($R_L$), using the four closest pixels ($p_i$) as reference, and the distance between the lost pixel and the reference ones ($d_i$). The pixel value ($lp$) is recovered using a weighted average of $n$ neighbouring pixels, defined as follows:
|
||||
\begin{equation}
|
||||
\label{eq:pixel_interpolation}
|
||||
lp = \left. \sum_{i=1}^n{ \frac{p_i}{d_i} } \middle/ \sum_{i=1}^n{ \frac{1}{d_i} } \right. ,
|
||||
%lp = \left. \sum_{i=1}^n{ \frac{p_i}{d_i} } \middle/ \sum_{i=1}^n{ \frac{1}{d_i} } \right. ,
|
||||
lp = \frac{ \sum_{i=1}^n{ \frac{p_i}{d_i} } }{ \sum_{i=1}^n{ \frac{1}{d_i} } } ,
|
||||
\end{equation}
|
||||
%\begin{equation}
|
||||
%\label{eq:pixel_interpolation}
|
||||
@@ -524,7 +521,7 @@ The \ac{mv} differences between the vectors of successive frame pairs are recurs
|
||||
This approach has shown to improve the performance of the \ac{mc} method, with gains up to $0.47\ dB$ (\ac{psnr}) for a \ac{plr} of $20\%$.
|
||||
|
||||
|
||||
\subsubsection{Boundary match algorithms}
|
||||
\subsubsection{Boundary matching algorithms}
|
||||
|
||||
|
||||
\Gls{bma} are used to recover a missing region based on the available neighbouring regions.
|
||||
@@ -538,7 +535,7 @@ Then, the optimal reconstruction is selected according to the sum of differences
|
||||
\centering
|
||||
%\includegraphics[width=0.45\columnwidth]{./022_ErrorResilience/Figures/BMAAlgorithm}
|
||||
\input{./022_ErrorResilience/Figures/BMAAlgorithm}
|
||||
\caption{Illustration of the lost block and the neighbouring information used in boundary match algorithm.}
|
||||
\caption{Illustration of the lost block and the neighbouring information used in boundary matching algorithm.}
|
||||
\label{FIG3:BMAPixelInformation}
|
||||
\end{figure}
|
||||
|
||||
@@ -574,14 +571,14 @@ An extension to the previous approach was considered in \cite{PelBasedMVE_TCE200
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\includegraphics[width=0.9\columnwidth]{./022_ErrorResilience/Figures/MVE}
|
||||
\caption{Motion vector extrapolation with the corresponding extrapolated macroblock, based on the method presented in \cite{BlockBasedMVE2002}.}
|
||||
\caption{Motion vector extrapolation with the corresponding extrapolated block, based on the method presented in \cite{BlockBasedMVE2002}.}
|
||||
\label{FIG3:MVEExtrapolation}
|
||||
\end{figure}
|
||||
|
||||
In \cite{2DConcHybridMotion2010} a \ac{hmve} algorithm was proposed, to improve the accuracy of the \acs{mv} of the \ac{pmve} method, by using the extrapolated vectors at the pixel and block level. Moreover, the \acs{mv} that were wrongly extrapolated are discarded in order to obtain a more accurate MV.
|
||||
In this \ac{ec} method, firstly a pixel-based motion extrapolation is performed and the pixels are organised into three groups.
|
||||
Then, through block-based \ac{mve} two new \acs{mv} are determined: the first one corresponds to the \ac{mv} of the most overlapped block with the current one, and second one is obtained by the weighted average value of the \acs{mv} corresponding to different overlapped blocks.
|
||||
To discard the wrongly extrapolated \acs{mv} and finding the more accurate ones from the MV set described above, further refinement is applied based on the distance between each pair of \acs{mv}.
|
||||
\HIGHLIGHT{To discard the wrongly extrapolated \acs{mv} and finding the most accurate ones, further refinement is applied based on the distance between each pair of \acs{mv}.}
|
||||
This algorithm is able to outperform the \ac{pmve} for different packet loss probabilities, as shown by the results in \cite{2DConcHybridMotion2010}.
|
||||
The method was used to recover both intra and inter frames in video sequences compressed with the \ac{h264}, achieving \ac{psnr} gains above $1.09$ dB in the erroneous frames.
|
||||
|
||||
|
||||
Reference in new issue
Block a user