669 lines
75 KiB
TeX
Executable File
669 lines
75 KiB
TeX
Executable File
\chapter{Robust Video Coding: A Review \VERSION{1}}
|
|
\label{Chapter:ErrorResilienceReview}
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Introduction}
|
|
|
|
\glsreset{ec}
|
|
\glsreset{vcl}
|
|
|
|
|
|
In general, digital communications through networks are prone to transmission errors or data loss.
|
|
These errors can be handled in different ways, such as retransmission, error correction, and \ac{ec} \cite{ErrorResilience2000,PixelInterlacingIntraConcealment2011}.
|
|
This chapter reviews the main robust video coding techniques that can be applied at the \ac{vcl} to reduce the error propagation and increase the decoded video quality.
|
|
Several techniques have been proposed in the past to deal with the problem of robust video transmission, especially for video coding standards with high temporal dependencies, such as \ac{h264} and more recently \ac{hevc}.
|
|
These techniques are normally focused on the spatio-temporal dependencies in the coded streams, which are used to obtain high coding gains but lead to low error robustness~\cite{ROPE2007}.
|
|
In this chapter the error resilience and concealment schemes are divided in different categories depending on their functionality and algorithms.
|
|
Although each approach is based on a different techniques they all aim at striking the most efficient balance between coding efficiency and decoded video quality under lossy conditions.
|
|
|
|
This chapter is organised as follows: Section~\ref{Section22:Localisation} describes the localisation techniques, which reduce the temporal and spatial dependencies in coded bitstreams. Section~\ref{Section22:DataPartitionRedundant} provides an overview of methods based on data partition, classification and bitstream redundancy.
|
|
%These techniques ease the introduction of extra information in order to recover from lost data.
|
|
Section~\ref{Section22:Concealment} covers the techniques used at the decoder-side as post-processing to reduce the effect of errors and improve visual quality and smoothness.
|
|
Section~\ref{Section22:ConcealmentAware} provides a description of methods that use \ac{ec} techniques to achieve improved error robustness.
|
|
Finally, Section~\ref{Section22:Summary} concludes this chapter by providing a overall comparison between the different approaches.
|
|
|
|
\clearpage
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Robust video coding techniques}
|
|
|
|
In general, most of the compression efficiency is achieved by exploiting both the spatial and temporal redundancy of video signals. Since there is high correlation between neighbouring information, predictive coding schemes are very effective to achieve higher compression gains by exploiting such correlations.
|
|
However, the drawback of high predictive coding schemes is the strong dependencies between compressed symbols with great impact on error propagation.
|
|
In order to increase the robustness of video transmission, several coding mechanisms can be employed at different elements of a transmission chain to mitigate the artefacts caused by errors and data loss.
|
|
These mechanisms can be classified according to the functional characteristics and the type of problem they address~\cite{ErrorResilienceAug2005}.
|
|
In this chapter the robust video coding techniques ere grouped into four categories:
|
|
\begin{itemize}
|
|
\item Localization techniques;
|
|
\item Data partitioning and redundant coding;
|
|
\item Error concealment at the decoder;
|
|
\item Concealment-aware error resilience.
|
|
\end{itemize}
|
|
|
|
\begin{table}[!b]
|
|
\centering
|
|
\caption{Advantages of error robustness techniques by category.}
|
|
\label{tab:errorresilience}
|
|
\input{022_ErrorResilience/ErrorResilienceCategories.tex}
|
|
\end{table}
|
|
|
|
Table~\ref{tab:errorresilience} provides a summary of the techniques covered in this section, some examples and the key advantages associated with each group.
|
|
Localisation techniques mainly localise and break the predictive coding loop, reducing the spatial and temporal dependencies, so that if an error occurs it is not likely to affect other parts of the coded video~\cite{ErrorResilienceModeHEVC2016,ErrorResilienceModeHEVC2017}.
|
|
Data partitioning rely on grouping the coded data into different categories depending on their relative importance.
|
|
This ease the introduction of unequal error protection at the transmission layer.
|
|
Redundant coding consists in using additional data blocks to enable correction of transmission errors and to achieve robust decoding in error-prone conditions.
|
|
In case of the post-processing \ac{ec} methods, the problem of robust transmission is addressed from the decoder-side.
|
|
Finally, concealment-aware techniques are used at the encoder, to either adapt the coding loop or introduce extra information which is used to increase the performance of \ac{ec} algorithms and increase the reconstruction quality of lost data at the decoder \cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ConcealmentIntraDataHidng2009,ConcealmentSignalling2008}.
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Localisation techniques}
|
|
\label{Section22:Localisation}
|
|
|
|
|
|
The basic principle of localization techniques is to break the predictive coding loop at certain predictive points, in order to reduce the likelihood of errors affecting large regions of the video signal, considering both the spatial and temporal domains.
|
|
Since these techniques break the predictive coding, they have a negative impact in compression efficiency as expected.
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
\input{./022_ErrorResilience/Figures/LocalizationTechniques.tikz}
|
|
\caption{Examples of spatial and temporal error propagation and how it can be reduced with spatial and temporal error resilience techniques.}
|
|
\label{fig2:ErrorPropagation}
|
|
%\cite{ErrorResilienceAug2005}
|
|
\end{figure}
|
|
|
|
%\begin{figure}[tb]
|
|
% \centering
|
|
% \includegraphics[width=0.7\columnwidth]{./022_ErrorResilience/Figures/ErrorPropagation_1.png}
|
|
% \caption{Examples of spatial and temporal error propagation and how it can be reduce using localisation techniques \cite{ErrorResilienceAug2005}.}
|
|
% \label{fig2:ErrorPropagation}
|
|
%\end{figure}
|
|
|
|
This group of error resilience techniques are able to reduce the error propagation at two levels, i.e., temporal and spatial levels. Figure~\ref{fig2:ErrorPropagation} is used to demonstrate the principle of error propagation and how it can be reduced using localisation techniques. The top of the figure illustrates a lost block and the correspondent spatial and temporal error propagation on the left and right, respectively.
|
|
Spatial localisation techniques
|
|
To reduce the error propagation within a frame a row of refresh blocks can be introduced to stop the error propagation.
|
|
To address the temporal error propagation refresh blocks (e.g., intra coded blocks) can be randomly introduce to stop error propagation.
|
|
As shown in the figure the refresh blocks are not affected error propagation.
|
|
In both examples of Figure~\ref{fig2:ErrorPropagation} the regions affected by errors are clearly reduced.
|
|
|
|
|
|
\subsection{Error resilient rate-distortion optimisation}
|
|
|
|
\glsreset{rd}
|
|
|
|
A common localization technique deals with the error resilience problem within the scope of the \ac{rd} optimisation process used in video coding.
|
|
Normally, such optimisation is equivalent to the minimisation of the Lagrangian cost given by:
|
|
\begin{equation}
|
|
J = D + \lambda R,
|
|
\end{equation}
|
|
where $D$ is the distortion of all pixels, $R$ the bitrate and $\lambda$ is the Lagrangian multiplier.
|
|
Such optimisation only takes into account the coding distortion ($D$), which is only suitable for error-free environments.
|
|
Therefore, alternative approaches error resilient \ac{rd} optimisation, which take into consideration possible network losses~\cite{DistortionTrackingH264_ICIP2004}.
|
|
More advanced approaches can take advantage of a \ac{rope} of the end-to-end distortion at the encoder-side~\cite{ROPEJun2000}.
|
|
Let $f_t^i$ denote the original value of pixel $i$ in frame at instant $t$, $\hat f_t^i$ denote its reconstruction in the encoder and $\tilde f_t^i$ the reconstruction in the decoder after \ac{ec}.
|
|
Since data loss may occur in the transmission channel, $\tilde f_t^i$ is modelled as a random variable at the encoder.
|
|
The expected \ac{mse} of each pixel is given by:
|
|
\begin{equation}
|
|
E\left\{d_t^i\right\} = E\left\{ \left( f_t^i - \tilde f_t^i\right)^2 \right\} = (f_t^i)^2 - 2 f_t^i E\left\{ \tilde f_t^i \right\} + E\left\{ \left(\tilde f_t^i\right)^2 \right\}
|
|
\end{equation}
|
|
Then the packet loss probability ($p$) is taken into consideration and used in the distortion measurement.
|
|
In case of intra-coded pixels the following equation used:
|
|
\begin{align}\label{EQ2:E2EDIntra}
|
|
E\left\{ \tilde f_t^i \right\} &= \left(1-p\right) \hat f_t^i + p E\left\{ \tilde f_{n-1}^i \right\} \\
|
|
E\left\{\left( \tilde f_t^i \right)^2 \right\} &= \left(1-p\right) \left( \hat f_t^i \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\},
|
|
\end{align}
|
|
while in the case inter-coded pixels:
|
|
\begin{align}\label{EQ2:E2EDInter}
|
|
E\left\{ \tilde f_t^i \right\} &= \left(1-p\right) \left( \hat e_t^i + E\left\{ \tilde f_{n-1}^{j} \right\} \right) + p E\left\{ \tilde f_{n-1}^i \right\} \\
|
|
E\left\{\left( \tilde f_t^i \right)^2 \right\} &= \left(1-p\right) \left( \left( \hat e_t^i \right)^2 + 2 \hat e_t^i E\left\{ \tilde f_{n-1}^{j} \right\} + E\left\{ \left( \tilde f_{n-1}^{j} \right)^2 \right\} \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\}
|
|
\end{align}
|
|
assuming that the pixel $i$ is predicted from pixel $j$ in the previous frame.
|
|
The prediction error $e_t^i$, is quantized to the value $\hat e_t^i$, which is transmitted to the decoder along with the motion information.
|
|
Equations \ref{EQ2:E2EDIntra} and \ref{EQ2:E2EDInter} provide a measure of the distortion taking into consideration not only the coding distortion but also the possibility of frame loss and consequent \ac{ec} distortion.
|
|
|
|
This technique was used in \cite{ROPEJun2000} to switch between intra and inter coding modes with the aim of reducing the temporal dependencies, and, thus achieving enhanced error robustness in video transmission.
|
|
In \cite{AdvancesROPEHE2642007} this technique was optimized to quarter pixel estimation, and then in \cite{ROPEBurst2009} the method was extended for burst losses, by introducing the burst loss probability into the original end-to-end estimation.
|
|
Results revealed that higher video quality was achieved by considering both probabilities.
|
|
In \cite{E2EDistortion2011} a temporal \ac{ec} is considered in the distortion estimation, which increases the performance of the proposed method, by avoiding frame-copy that provides a lower bound on reconstruction quality.
|
|
More advanced temporal \ac{ec} methods were also considered in the work presented in~\cite{EndToEndDistortionConcealment2014}.
|
|
The \acs{mv} used for motion compensation reconstruction is obtained using boundary matching search, which is able to achieve lower distortion when compared with frame-copy. Using more advanced \ac{ec} algorithms leads to a more accurate estimation of the end-to-end distortion, thus increasing the error robustness of the transmitted video.
|
|
To take into account the delay introduced in the encoding loop due to hierarchical prediction structures, delay based \ac{rd} optimisation was proposed in \cite{DelayRD_TCVT2015} by estimating both the distortion and the end-to-end delay which is minimised along the required bitrate. Using this approach the total delay can be controlled without severely compromising the coding efficiency.
|
|
In the context of \ac{hevc} end-to-end distortion estimation was also used in the \ac{rd} optimisation \cite{ErrorResilienceModeHEVC2016} showing significant quality improvements comparing with the reference implementation.
|
|
|
|
Different quality metrics can also be used for end-to-end distortion estimation. The method presented in \cite{SSIMModeling2015} proposes the distortion estimation using the \ac{ssim} \cite{QualitySSIMRef2002} in order to increase the correlation between the estimated distortion and the perceptual quality noticed by the human visual system~\cite{VisualQualityMetrics2011}. Moreover, the \ac{ssim} quality metric in learning based \ac{rd} modelling~\cite{RateDistortionNN_VCIP2017}, using a \acs{cnn} to predict both the distortion and the required rate, showing good accuracy in the \ac{rd} cost estimation.
|
|
|
|
|
|
\subsection{Intra-refreshing}
|
|
|
|
In order to reduce the effects of error propagation within a single frame spatial and temporal dependencies can be limited by introducing refresh points into the bitstream by forcing intra-coded blocks\cite{IntraRefresh1999,IntraRefreshAttentionCM2007,IntraRefreshRefSelectionJan2009,IntraRefreshICIP2009,SandroIntraRefresh2011,JointIntraRefresh2014,IntraRefreshFeedback2015,IntraRefreshRPS2017}.
|
|
This technique is fairly efficient to improve the error robustness of video streams, by avoiding long error propagation both within a single frame but also across several subsequent frames.
|
|
The main characteristic of this group of techniques is to increase the number of intra-coded blocks as they have no dependencies to previous frames.
|
|
However, this is done at the expense of higher amount of bits~\cite{IntraRefreshAttentionCM2007}.
|
|
Therefore, different alternatives have been proposed, which optimise the number of intra-coded blocks, which can be introduced randomly, periodically~\cite{SandroIntraRefresh2011}, based on the motion information or rate-distortion optimisation~\cite{IntraRefreshICIP2008,HEVCIntraRefreshICIP2017}.
|
|
Moreover, whenever is possible the encoder can use a feedback channel to decide where refresh regions should be introduced~\cite{IntraRefresh1999,IntraRefreshFeedback2015}.
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
\input{./022_ErrorResilience/Figures/IntraRefresh.tikz}
|
|
\caption{Examples of spatial and temporal error propagation and how it can be reduce using localisation techniques.}
|
|
\label{FIG2:IntraRefresh}
|
|
\end{figure}
|
|
|
|
A non-normative method for random intra refreshing is implemented in the \ac{h264} reference software, which introduces cyclic refresh points using a random pattern.
|
|
Figure~\ref{FIG2:IntraRefresh} (a) shows an example of intra-coded blocks (dark gray squares) being randomly places into the coded frames to prevent error propagation.
|
|
This approach guarantees that all \acs{mb} are refreshed at some point, therefore errors do not propagate indefinitely.
|
|
However, all \acs{mb} are refreshed exactly the same number of times without consideration for rate distortion optimisation neither the image region containing the \acs{mb}.
|
|
In \cite{IntraRefreshICIP2008} a rate control scheme is used to decide if a given block should be refreshed using a intra-coded prediction.
|
|
The decision is performed based on the ratio between the Lagrangian cost of the best intra and inter modes.
|
|
Whenever this ratio is lower than a certain predefined threshold an intra mode is selected, which guarantees that intra refreshing is only forced when the Lagrangian cost of the intra mode is close to the optimal coding method.
|
|
This is further extended to adapt the intra refresh algorithm to the network conditions, i.e., the threshold value is obtained from the bitrate and the loss ratio~\cite{IntraRefreshICIP2009}.
|
|
Therefore, the number of intra-coded block automatically increases as the bitrate and the loss ratio increases.
|
|
In the context of \ac{hevc} intra refresh is also being used to reduce error propagation.
|
|
Soft-refresh points using joint intra-inter prediction were proposed for this purpose~\cite{HEVCIntraRefreshICIP2017}.
|
|
Although the image regions cannot be correctly decoded in case of errors (as happens with intra-refresh), this method significantly decreases the bitrate increase due to extra intra-coded blocks.
|
|
When compared with traditional intra-refresh, bitrate savings of up to 12\% are achieved for a 5\% \gls{plr}.
|
|
|
|
\glsreset{roi}
|
|
|
|
A different approach is to introduce refresh points based on the video content.
|
|
In~\cite{IntraRefreshAttentionCM2007} the error resilience of video streams is increased by combining rate control and attention based models to define \acs{roi}.
|
|
A weighting factor is used in the Lagrangian optimization to increase the number of intra-coded blocks in the image regions determined by the attention models.
|
|
Figure~\ref{FIG2:IntraRefresh} (b) shows an example with a \ac{roi} marked with a black line which varies for every frame. In this case the intra-coded blocks are only introduced inside the marked region, in contrast to what is shown in Figure~\ref{FIG2:IntraRefresh} (a).
|
|
Results show that using the aforementioned method, less refresh points are required to improve the error robustness of the \ac{roi}. When comparing with \cite{DistortionTrackingH264_ICIP2004}, approximately 4 dB of \ac{psnr} gains are achieved in the \ac{roi} without compromising the overall video quality.
|
|
A more recent approach was proposed using a perceptual noticeable difference model to estimate the noticeable distortion, and then using a visual attention model \cite{SaliencyItti1998} to decide which blocks should be used as intra refresh in the bitstream~\cite{IntraRefreshPerceptual2010}.
|
|
This method is able to improve the results of a previous work \cite{IntraRefresh2005}, especially for high motion sequences. This is due to the fact that such previous approach only used an estimation of the error propagation without accounting for perceptual information.
|
|
|
|
A similar approach was also exploited for \ac{hevc} in \cite{HEVCROIErrorResilience2015,HEVCRoiIntraRefresh_DSP2016}, by using a feedback channel to provide information about the network errors, which is then used by the encoder to introduce intra-coded blocks into the \ac{roi}.
|
|
The simulation results reveal an increasing on the achieved video quality when higher number of intra blocks are used. Moreover, applying the proposed method only to the \ac{roi}, the bitrate overhead introduced is lower, leading to higher \ac{rd} gains.
|
|
|
|
|
|
\subsection{Leaky-predictions}
|
|
|
|
Directly using reconstructed past frames as reference frames for predictive coding, may result in substantial error propagation whenever the reference frames are either lost or reconstructed from other distorted frames.
|
|
To reduce error propagation one can employ leaky predictions, which scales down reconstructed frames before being used as reference~\cite{LeakyPred2009,ROILeakyPred2009}.
|
|
This leads to an exponential decay of error propagation, with direct impact on error robustness.
|
|
In this approach, the reconstructed reference frames are scaled down by applying the following:
|
|
\begin{equation}\label{EQ2:LeakyPredConstant}
|
|
\check f_t^i = \alpha \hat f_t^i + (1- \alpha)C,
|
|
\end{equation}
|
|
where $\check f_t^i$ is a reference frame, $\alpha$ is the leaky factor used to control the amount of information from the reconstructed frame present in the new frame $\check f_t^i$, and $C$ is a constant value.
|
|
Leaky prediction has been used in the past for error control, especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in Equation~(\ref{EQ2:LeakyPredConstant}) is replaced by a lower layer error-free reconstructed frame. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate as shown in \cite{LeakyPredConstant1973}. The $\alpha$ parameter allows to control the effect of the constant value, and thus the coding efficiency. By decreasing $\alpha$ the coding efficiency increases, however, the exponential decay in error propagation becomes less significant.
|
|
|
|
% MHMCP_TCSVT2006
|
|
|
|
In order to overcome the loss in coding efficiency, weighted prediction can achieve similar advantages, by using motion compensation from two previously encoded frames to generate a single prediction for the current frame \cite{WeightedBiPred2005}.
|
|
Alternatively, the two previously encoded frames can be used to generate an interpolated frame ($ \check f_t^i $) to be used as reference, using:
|
|
%In case of leaky prediction instead of applying motion compensation is applied once, but with an interpolated frame obtained as follow:
|
|
\begin{equation}\label{EQ2:LeakyPredDual}
|
|
\tilde f_t^i = \alpha \hat f_t^i + (1- \alpha) \hat f_{n-1}^i.
|
|
\end{equation}
|
|
Therefore, it is possible to decrease the error propagation using $\tilde f_t^i$ as reference frame for prediction, if only one of the original encoded frames is lost (i.e., $\hat f_{n}^i$ or $\hat f_{n-1}^i$). This is due to the fact that $\tilde f_t^i$ contains information from both $\hat f_{n}^i$ and $\hat f_{n-1}^i$.
|
|
In Equation (\ref{EQ2:LeakyPredDual}) $\hat f_{n}^i$ and $\hat f_{n-1}^i$ always correspond to the encoded frames and the generated frames ($\tilde f_t^i $) are not used to interpolate subsequent frames. This corresponds to a finite impulse response filter.
|
|
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
% \begin{tabular}{x{4cm}x{4cm}x{4cm}}
|
|
% \multicolumn{3}{c}{\includegraphics[width=12cm]{./022_ErrorResilience/Figures/ROILeakyPrediction.png}} \\
|
|
% $f_{n-1}$ & $f_{n}$ & $f_{n+1}$ \\
|
|
% \end{tabular}
|
|
% \caption{Example of error propagation with a ROI (shaded area indicates error, and the arrows indicate temporal predictions to previous frame) \cite{ROILeakyPred2009}.}
|
|
\input{./022_ErrorResilience/Figures/ROILeakyPrediction.tikz}
|
|
\caption{Example of error propagation with two different regions; as shown errors can propagate from the background to the ROI.}
|
|
\label{fig2:ErrorPropagationROI}
|
|
\end{figure}
|
|
|
|
|
|
In \cite{ROILeakyPred2009} a method is presented where the leaky prediction is confined to a pre-defined \ac{roi}, in order to increase the error robustness with lower impact on the coding efficiency.
|
|
Figure \ref{fig2:ErrorPropagationROI} shows an example of error propagation within two different regions: \ac{roi} and background. As show in the figure, since the \ac{roi} uses background regions for predictions error propagation occurs across regions.
|
|
To overcome this problem, the method in \cite{ROILeakyPred2009} uses a weighted prediction for the \ac{roi} region obtained from the following predictions: (i) a globally optimal prediction obtained by unrestricted motion estimation, (ii) a prediction obtained by restricting motion estimation within the \ac{roi} region.
|
|
This limits the impact of errors in the background region on the \ac{roi}.
|
|
Moreover, since the leaky predictions are only confined to some specific regions, higher coding efficiency is achieved when compared with other methods.
|
|
|
|
%\HERE
|
|
|
|
The method proposed in \cite{ROPERefGenJan2010} extends the traditional leaky prediction expressed in Equation (\ref{EQ2:LeakyPredDual}) by replacing the previous encoded frame ($\hat f_{n-1}^i.$) with the previous interpolated frame ($\tilde f_t^i$). This results in the following:
|
|
\begin{equation}\label{EQ2:LeakyPredIIR}
|
|
\tilde f_t^i = \alpha \hat f_t^i + (1- \alpha) \tilde f_{n-1}^i.
|
|
\end{equation}
|
|
This corresponds to infinite impulse response filter.
|
|
Comparing both equations for the same $\alpha$, the Equation (\ref{EQ2:LeakyPredIIR}) implies stronger filtering than Equation (\ref{EQ2:LeakyPredDual}). Therefore, reference frames generated with heavier filtering have higher resilience against error propagation, but less correlation with the original frame, which negatively impacts inter predictions and hence coding efficiency.
|
|
The idea of combining multiple predictions was also been used in the \ac{hevc} standard by combining it with end-to-end distortion to select the optimal predictions~\cite{DualFrameME2015}.
|
|
|
|
|
|
\subsection{Reference picture selection}
|
|
|
|
Reference picture selection has been widely used in the past to reduce the temporal dependencies and reduce the error propagation \cite{ResilienceDualFrameBuffer2004,ErrorResilienceRPSFeb2009,RPSErrorsFeb2009,MVConcatenationAug2011}.
|
|
In \cite{ResilienceDualFrameBuffer2004}, an optimisation algorithm was devised to select between short and long term references based on the \ac{rope} end-to-end distortion estimation method. This was used to increase the prediction distance and reduce error propagation.
|
|
The same method can also be used in combination with feedback information from the network to improve the references frame selection.
|
|
In the decoder-side, an \ac{ec} method was later used to take advantage of such prediction structure \cite{ConcealmentDualFrameBuffer2005}.
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
\includegraphics[width=0.95\columnwidth]{./022_ErrorResilience/Figures/RefPicSelection.png}
|
|
\caption{Examples of dynamic reference picture selection used in error resilient video transcoding \cite{ErrorResilienceRPSFeb2009}.}
|
|
\label{fig2:ReferencePictureSelection}
|
|
\end{figure}
|
|
|
|
A feedback information from the network was also used in \cite{ErrorResilienceRPSFeb2009}, to reduce error propagation at end-user decoders, by dynamically selecting the reference frame, only from those correctly received.
|
|
Figure~\ref{fig2:ReferencePictureSelection} shows an example of an error resilience transcoding scheme, where the original temporal predictions (represented by the solid arrows) are transcoded to use only correctly received frames as reference, i.e., prediction represented by the dashed arrow. This is done based on network feedback.
|
|
A more advanced scheme may be considered, where only the regions affected by mismatch predictions are processed in the compressed domain, and subsequently transcoded to limit the dependencies from lost frames. To achieve low complexity \ac{mv} concatenation can be used, in order to obtain new inter-predictions for error-free frames \cite{MVConcatenationAug2011}.
|
|
In \cite{ChannelAdaptiveVideoStreaming2002} network feedback is combined with path diversity to select the reference frames that are more likely to be received at the decoder.
|
|
In the proposed approach, the feedback information is not directly used to select the reference frames.
|
|
Instead, the reference frames are selected through an end-to-end distortion estimation process, taking into account the coding distortion and distortion resulting from channel errors (\ac{ec} distortion is also taken into account).
|
|
In \cite{ErrorResilienceModeHEVC2017} a \ac{roi} is defined based on the motion objects to limit the regions where the mechanism of reference picture selection is applied. Intra refresh is also used based on the temporal distance to the previous intra-refresh frame. This method not only achieves significant quality gains when compared with the reference \ac{hevc} standard but also when comparing with tradition reference selection schemes.
|
|
|
|
In~\cite{IntraRefreshRefSelectionJan2009} an algorithm to introduce intra refresh points at the \ac{mb} level was proposed.
|
|
This method takes into consideration the decoding distortion assuming that frame-copy concealment is applied for \ac{ec} when packet loss occurs.
|
|
The refresh blocks are encoded using either intra prediction or inter prediction through a secondary algorithm to select the most reliable reference frame (i.e., frame with less total expected distortion).
|
|
|
|
In the methods mentioned above the dependencies between the frames are analysed and reduced in order to decrease the error propagation in cases of data loss.
|
|
Since the existing video coding standards are strongly based on predictive coding, these methods present an efficient approach in highly erroneous conditions.
|
|
% However, the method in~\cite{RDReferenceSelectionJul2012} uses more complex distortion estimations, and in ideal conditions multiple loop encoders are needed to select the reference frames optimally.
|
|
However, although these closed-loop techniques may offer a reliable control of the decoder distortion, the feedback required from the receiver side and multipath networking are not always available neither feasible in real-time and unidirectional channels (e.g. broadcast).
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Data partition and redundant coding}
|
|
\label{Section22:DataPartitionRedundant}
|
|
|
|
In a compressed video streams different symbols have different relevance in terms of their impact on error propagation.
|
|
Therefore, there is an unequal contribution of different syntax elements to the decoding distortion by error propagation, depending on the lost symbols.
|
|
Data partition techniques have been developed to provide a mechanism to cope with the unequal importance of coded information by grouping different parts of a compressed bitstream according to their importance to the decoded quality.
|
|
Then, some of these groups can be protected more efficiently than others using \ac{uep} schemes. These are normally associated with \ac{fec} algorithms, whereby multi-levels of \ac{fec} are applied to achieve different levels of error protection. This is presented and discussed with more detail in Sub-section~\ref{SEC22:UEP}
|
|
|
|
In a single channel transmission, the most important data or syntax elements (e.g., headers) can be protected with stronger channel codes than the remaining ones.
|
|
In case of multi-channel communications, the important data can be routed over the most reliable channels.
|
|
Moreover, to achieve higher quality in multi-channel transmission one can use extra information to increase the error resilience of video streams, in order to increase the probability of being able to decode then in case of errors.
|
|
For instance, redundancy can be added either in an explicit form, e.g., redundant picture, or implicit, e.g., multiple description coding \cite{MDC1993}. Section~\ref{SEC22:RedundantPics}~and~\ref{SEC22:MDC} present a detailed review of these two approaches, respectively.
|
|
|
|
|
|
\subsection{Data partitioning and flexible macroblock ordering}
|
|
|
|
\glsreset{fmo}
|
|
The concept of data partitioning was firstly introduced in the H.263 coding standard to split the coded data into different categories\cite{H264OverIPJul2003}. It was further improved in the next standard and it allows the symbols in each slice to be divided into three partitions:
|
|
\begin{itemize}
|
|
\item Type A: headers, \ac{mb} types, quantization parameters and motion information.
|
|
\item Type B: intra-coded information, such as, intra blocks signalling and intra coefficients. This type cannot be used without the partitions of type A being present.
|
|
\item Type C: inter-coded information and inter coefficients. This type also requires the information of type A but can be decoder without intra-coded information (type B).
|
|
\end{itemize}
|
|
These partitions have different levels of importance, being data partitions of type A the most important as they carry the header information which are required by the remaining partitions. Then, as partitions of type B carry intra-coded information they are more important than type C information as it can stop temporal error propagation.
|
|
|
|
The \ac{h264} introduced an error resilience tool named \ac{fmo} technique, which allows each \ac{mb} to be freely assigned to a specific slice group (i.e., a set of slices), organized in a mixed-up fashion~\cite{H264FMO2006}.
|
|
The standard supports up to eight slice groups in each picture, and the \acs{mb} within a group can be assigned to several slices in the default raster scan order.
|
|
The case where only one slice group is used corresponds to the case where the \ac{fmo} tool is disabled.
|
|
\ac{fmo} increases the error resilience by partitioning the bitstream into several groups, which is able to reduce the error propagation.
|
|
Results show that the use of \ac{fmo} contributes to enhance the subjective and objective quality, especially for high packet loss rates at cost of an affordable overhead~\cite{H264FMO2006}.
|
|
|
|
Six types of slice groups are supported by \ac{fmo}. Figure~\ref{fig2:FMOTypes} shows \ac{fmo} types 0 to 5.
|
|
In the \ac{fmo} type 0 each slice group is defined by a maximum number of \acs{mb}, and a new slice group beings whenever this maximum value is reached. The same slice group can be cyclic repeated within a frame.
|
|
\ac{fmo} type 1 corresponds to scattered slices, where the \acs{mb} are assigned to each slice group with a fixed formula. Figure~\ref{fig2:FMOTypes} represents the \ac{fmo} type 1 with two slice groups resulting in a checkerboard pattern.
|
|
In the \ac{fmo} type 2 slice groups are defined by rectangular shapes bounded by the top-left and bottom-right corners.
|
|
Although the slice groups are allowed to overlap each other, each \ac{mb} may only belong to one slice group. \ac{fmo} type 3, 4 and 5 are special types, each one allowing the configuration to change on every frame.
|
|
Nevertheless, only two slice groups are allowed in \ac{fmo} type 3 to 5, in order to keep the signalling overhead affordable.
|
|
The last \ac{fmo} type is the most generic case, where the slice groups are explicitly defined using high level packets (i.e., signalling \ac{nal} units), allowing more flexibility at the cost of higher signalling overhead.
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
\includegraphics[width=0.8\columnwidth]{./022_ErrorResilience/Figures/FMO_Slices_Type.png}
|
|
\caption{Different types of flexible macroblock reordering \cite{H264FMO2006}.}
|
|
\label{fig2:FMOTypes}
|
|
\end{figure}
|
|
|
|
|
|
In the past, \ac{fmo} has been widely used to increase the error robustness of video transmissions in different communication scenarios \cite{FMOErrorResilience2007,FMOResilience2010,FMOResilience2011,FMOROI2012,MotionEnergyDataPart2014}.
|
|
In ~\cite{FMOErrorResilience2007} \ac{fmo} is used to increase the \ac{ec} performance and thus reduce the error propagation in prioritised video transmission.
|
|
The method divides the \acs{mb} into the two slice groups based on an estimation of the \ac{ec} distortion using a two-pass approach. Then, the slice group which has the most impact on the \ac{ec} distortion is transmitted using a reliable channel and it is assumed to be received error-free.
|
|
Results show that the proposed approach is able to outperform the dispersed mode (\ac{fmo} type 1).
|
|
Slice classification can also be based on the weighted sum of the motion vector information and the rate-distortion cost~\cite{FMOResilience2010}.
|
|
As in \cite{FMOErrorResilience2007}, the \acs{mb} are ranked based on the importance and distributed in the two groups with the same amount of blocks. In order to reduce the complexity of mapping the \acs{mb} into groups the same map can be repeated for several subsequent P-frames \cite{FMOResilience2011}.
|
|
This reduces the computational complexity of the \ac{mb} grouping while achieving similar error resilience performance.
|
|
The method in \cite{FMOROI2012} splits the video frames into two regions, which are then grouped in different slice groups.To classify the different \acs{mb}, spatial information is extracted from the \ac{mb} bit allocation, and temporal information is also extracted based on the estimation of the \ac{ec} distortion. This technique is able to reduce the number of erroneous \acs{mb} up to 82\% achieving a quality gain of up to 3.97\%, in terms of \ac{psnr}, in comparison with the case where \ac{fmo} is not used.
|
|
Motion energy was also used to classify \acs{mb} into different groups \cite{MotionEnergyDataPart2014}, by comparing the motion energy of a given \ac{mb} with the motion energy of its neighbours and the average motion energy of the entire frame.
|
|
This allowed the authors to obtain a dynamic threshold which is then used to determine the \ac{mb} importance. In comparison with other \ac{uep} error protection schemes, this method achieve good quality improvements under different \acs{plr}.
|
|
|
|
In \cite{FMORedudantSlices_TB2007} a new \ac{fmo} type is proposed combining the slice groups type 1 and 3 in order to obtain a checkerboard pattern in a box-out scanning order, with the objective of increasing the scattering of lost \acs{mb}. Results show that the proposed \ac{fmo} type is able to outperform the existing types under single and burst loss events.
|
|
The method described in \cite{FMOTypeROI2014} introduces a new type in order to efficiently divide the video content of each frame in two regions with different levels of importance, based on the \ac{fmo} type 6.
|
|
%Then, a spiral protection is applied to scan the \acs{mb} grouping then in two groups.
|
|
By proposing a new \ac{fmo} type they are able to signal several \ac{roi} areas achieving lower overhead than using the original \ac{fmo} Type 6.
|
|
|
|
|
|
\subsection{Unequal error protection}
|
|
\label{SEC22:UEP}
|
|
|
|
\glsreset{uep}
|
|
|
|
\Gls{uep} techniques are used to protect different parts of a video stream against errors according to their importance for the decoding quality. \ac{uep} approaches take advantage of the video stream characteristics that result from different types of content, both at the temporal and spatial level. This allows categorisation of coded data into different groups, by exploiting the fact that different regions are affected differently by transmission errors.
|
|
This not only ensures that those regions will be delivered with higher quality, but also leads to efficient use of the total overhead required to achieve increased the error robustness in video transmission.
|
|
|
|
A wide variety of \ac{uep} schemes have been presented in the literature to optimize the performance of video communication over networks where the probability of packet loss is not negligible.
|
|
It has been observed that, within a \ac{gop}, different frame types (e.g. I, P, or B frames) have different levels of contribution to the reconstructed video quality when transmission losses occur. This has been exploited for \ac{uep} in video transmission as shown in~\cite{UEP_TM2005,UEPMotion_TM2006}.
|
|
|
|
In \cite{UEPLayered_TB1999} an evaluation of layered video was proposed to increase the error robustness, by using redundant packets for helping the recovery of lost ones.
|
|
In order to achieve reduced overhead for error protection, each video layer has a different importance and is protected using different levels of redundancy, thus guaranteeing that packets carrying data of lower layers have higher probability of being decoded without errors.
|
|
This is a simple approach, not dependent on the network capabilities, neither requiring a deep knowledge of the video contents.
|
|
A lightweight model is used to assign packet priorities to be used by \ac{uep} algorithms~\cite{UEP_TCE2011}. Such model is based on a weighted sum of four components: (i) a slice type parameter giving higher importance to refresh and reference frames; (ii) a binary flag indicating whether a network packet transports video stream headers, i.e., \ac{nal} or slice header; (iii) a parameter indicating the distance in packets to the next slice; (iv) a parameter indicating the temporal distance to the next refresh frame.
|
|
These parameters can be obtained by parsing the \gls{nal} header, which makes it suitable to low-end encoders or even to prioritise each packet at a streaming level.
|
|
|
|
However, \ac{uep} techniques usually require a deep video analysis which in general contributes to increase the encoding complexity.
|
|
Nevertheless, one can use information already needed for encoding, such as motion, to classify different frames or even different regions within a frame~\cite{UEPMotion_TM2006}.
|
|
The motion information is used to determine the motion energy which is given by the L2-norm of the \acs{mv} within a predictive regions.
|
|
For instance, in~\cite{UEPMotionEnergy_WCNC2013} the motion energy of each frame is computed and compared with neighbouring frames. The energy of co-located block in adjacent frame is compared, and higher importance is given to those frames that have higher motion energy differences.
|
|
This method was compared with equal error protection, resulting in higher objective and subjective quality and revealing the relation between motion information and impact on errors.
|
|
This work was extended in ~\cite{UEPMotion_ICSPCC2012,UEPMotion_ICSPCS2013} to use a two levels \ac{uep} approach, where besides taking into consideration the \ac{mb} and frame importance, the combined importance of different frames is also used to determine the level of importance of each \ac{gop}.
|
|
|
|
Another work dealing with \ac{uep} is described in~\cite{Zhang20131390}, where the transmission distortion is estimated at the packet level, to assess the impact of each packet in the error propagation.
|
|
Then, different priorities are applied to each packet, allowing different levels of error protection. This was also used to schedule the packet transmission through multi-path networks\cite{PacketScheduling2010}.
|
|
Furthermore, in order to improve the classification of different regions of a video stream a perceptual algorithm can be used.
|
|
This strategy increases the correlation between the classification and subjective impact of errors.
|
|
In \cite{UEPSaliency_TCSVT2010} a method based on a combination of two parameters is described: a spatial parameter based on a perceptual attention model and a temporal parameter based on the frame position within the \ac{gop}.
|
|
These parameters are then used to improve the efficiency of \ac{fec} assignment, and experimental results show a significant quality improvement both using objective metrics and subjective evaluation.
|
|
|
|
\glsreset{el}
|
|
|
|
The scalable video coding framework provides implicit support for \ac{uep} through the layered structures defined with different levels of quality~\cite{UEPScalable_TCE2008,UEPScalable_TCSVT2010,UEPLayerVideo_TM2014, UEPScalable_ICSPICES2015}.
|
|
In~\cite{UEPScalable_TCSVT2010} an algorithm to recursively estimate the total expected distortion in the decoder at the picture-level for each layer is proposed.
|
|
This method takes into consideration \ac{el} truncation, i.e., a specific layer being discarded, and \ac{ec} to measure an accurate estimation of the end-to-end distortion. The estimation is carried out for every layer in the video stream, in order to apply an optimisation of different levels of error protection across several layers.
|
|
Also based on scalable coding, the work in~\cite{UEPLayerVideo_TM2014} proposes a technique for finding the optimal \ac{fec} coding rates, which allow for a correct \ac{el} decoding when errors affect the base layer. This prevents discarding correctly received \acs{el} frames due to missing dependencies, increasing the overall video quality.
|
|
|
|
In the context of \ac{hevc} there are also methods to achieve \ac{uep}.
|
|
Since techniques like \ac{fmo} are not available, different approaches were investigated \cite{SliceImportance_ICHNRSR2015,HEVCUepLte_IWCMCC2015,UEPVideoDegradation_ISCAS2017,UEP_TM2018}
|
|
The work in~\cite{SliceImportance_ICHNRSR2015} proposes a resource allocation scheme based on prioritisation at the slice level (i.e., \ac{nal} unit level) for \ac{lte} Advanced networks~\cite{HEVCUepLte_IWCMCC2015}.
|
|
The prioritisation is performed by analysing the inter-frame predictions dependencies, giving higher rankings to slices having higher number of dependent pixels.
|
|
In~\cite{UEPVideoDegradation_ISCAS2017} a quality degradation model is proposed for \ac{hevc} video, by
|
|
simulating error events and measuring the quality using the \ac{psnr} metric.
|
|
From the empirical results a model was devised to predict the quality under different events.
|
|
Then, the model is used to control the amount of redundant codes to be applied at the network level.
|
|
The scalable extension of the \ac{hevc} standard was also used to support \gls{uep} for video storage in cloud environments~\cite{UEP_TM2018}, by combining the layer information (i.e., base or enhancement layer) and the picture type (i.e., I, P or B-frame) to maximise the performance of the \ac{fec} codes both at the temporal and quality/resolution levels.
|
|
|
|
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%\section{Redundant Coding}
|
|
\subsection{Redundant pictures}
|
|
\label{SEC22:RedundantPics}
|
|
|
|
Redundant pictures is a well-known technique adopted in the \ac{h264} standard \cite{RedudantPicsJan2009}, which allows for different representations of the same coded picture to be transmitted using different encoding parameters.
|
|
For example, the primary picture may be coded with fine quantization, while the redundant one with coarse quantisation at lower bitrate.
|
|
On the one hand if the primary picture is received, the redundant information is discarded.
|
|
On the other hand if errors occur and the primary picture is lost, the redundant one is used to provide a lower level of reconstructed quality and limiting the error propagation.
|
|
|
|
Zhu \emph{et al.} investigated the paradigm of redundant picture coding~\cite{RedudantPicsJan2009}, considering different approaches in order to develop a mixed method based on both reference picture selection to obtain a clean decoding without error propagation and hierarchical allocation of the redundant pictures to increase error robustness.
|
|
In~\cite{VPxResiliencePCS2016} the key frames are encoded and transmitted at lower bitrate to be used in case of error to replace the missing frame.
|
|
Redundant pictures are a good mechanism to improve performance under error prone networking conditions, since they introduce secondary pictures that can be decoded when the primary picture is not available.
|
|
|
|
Moreover, redundant pictures might also work as localisation techniques when different reference frames are selected. As the number of redundant pictures increases, the amount of mismatch temporal predictions in case of data loss is reduced, since the redundant picture can replace the erroneous ones.
|
|
This achieves a similar goal as reducing the video stream dependencies, i.e., localisation techniques.
|
|
The main drawback of such approaches is the large amount of redundant data and overheads that are required for their efficient implementation.
|
|
In~\cite{RedundantMVsJune2008} a newer approach was proposed to reduce the amount of information needed to encode redundant pictures, by introducing only redundant \acs{mv}.
|
|
This was also used in~\cite{VPxResilience2016}, where the motion information is used to recover the missing frame.
|
|
These approaches increase the overall system performance by presenting an efficient trade-off between the redundant bitrate and the error robustness.
|
|
|
|
\subsection{Multiple description coding}
|
|
\label{SEC22:MDC}
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
\includegraphics[width=0.7\columnwidth]{./022_ErrorResilience/Figures/MDC.png}
|
|
\caption{Architecture of multiple description system \cite{MDC2005}.}
|
|
\label{fig2:MDCSystem}
|
|
\end{figure}
|
|
|
|
In \ac{mdc} schemes two or more bitstreams, named descriptions, are encoded and transmitted through different channels, such that an acceptable level of quality is achieved when only some of them reach the decoder \cite{MDC1993}.
|
|
Figure \ref{fig2:MDCSystem} illustrates a basic architecture of a \ac{mdc} encoder and decoder using two descriptions.
|
|
The encoder-side creates two descriptions, i.e., $R_1$ and $R_2$, which shall be transmitted using two different channels, reducing the probability of the same video data being lost at the same time in both descriptions.
|
|
As shown in the figure three different outputs may be produced at the decoder-side: both descriptions are correctly received and the central decoder is used, or either one of the descriptions is not received and then one of the side decoders is used.
|
|
When both bitstreams are received, the central decoder combines the information carried by each description achieving higher quality than any single side decoder.
|
|
|
|
In the \ac{mdc} encoder the redundancy among descriptions can be controlled by changing the correlation between them. Higher correlation leads to higher redundancy but also higher quality in the output of the side decoders.
|
|
In contrast with the case of redundant pictures, the redundant \ac{mdc} streams do not carry the same information which means that if both are received, then the decoded quality is increased. However, a combination of \ac{mdc} and redundant pictures can also be used to achieve increase quality gains \cite{MDCRedundantPicsJan2010}.
|
|
Different types of \ac{mdc} approaches \cite{MDC2005,MDCHierarBFeb2012,MDCCorreiaJun2012} can be used to improve the quality under transmission errors, either by taking advantage of hierarchical B-pictures~\cite{MDCHierarBFeb2012} or by introducing extra auxiliary information to limit the error propagation in case of single description decoding~\cite{MDCCorreiaJun2012}.
|
|
Although these techniques can be very effective in multi-path networks, a significant amount of overhead is required to create the different descriptions\cite{MDCCorreiaJun2012}.
|
|
|
|
%\HERE
|
|
|
|
\cite{MDCRedundantPicsJan2010}
|
|
\cite{MDC_TCSVT2011}
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Error concealment by post-processing}
|
|
\label{Section22:Concealment}
|
|
|
|
%\glsreset{ec}
|
|
|
|
In general, \ac{ec} functions are implemented as post-processing step in the decoder and thus is the last module in the communication chain capable of handling errors~\cite{ConcealmentReview1998}.
|
|
Thus, reconstruction of missing data should take place within the video decoder to minimise the perceived effects of lost data.
|
|
%Overall, to enhance the video quality, the design of the transmission system must incorporate adequate \ac{ec} tools.
|
|
This section reviews several \ac{ec} methods based on spatial and temporal redundancies, as these play an important role in the overall performance of robust communication systems.
|
|
|
|
|
|
\subsection{Spatial error concealment techniques}
|
|
% \label{SEC3:SpatialConcealment}
|
|
%
|
|
% This section introduces several concealment techniques that exploit the spatial redundancy of images to recover the lost data. To recover lost information, the methods introduced in this section entirely rely on neighbouring image regions received without errors.
|
|
% Consequently, spatial error concealment is mostly useful in the case where information of the neighbouring frames are not available.
|
|
|
|
%\subsection{Pixel level concealment}
|
|
|
|
|
|
A typical approach to recover missing regions in an image is to estimate the lost regions based on linear interpolation of neighbouring pixels \cite{ErrorConcSpatial1995,ConcTCE2001}.
|
|
Figure~\ref{FIG3:pixel_interpolation} shows the estimation process of a lost pixel ($lp$), within the lost region ($R_L$), using the four closest pixels ($p_i$) as reference, and the distance between the lost pixel and the reference ones ($d_i$). The pixel value ($lp$) is recovered using a weighted average, as follows:
|
|
\begin{equation}
|
|
\label{eq:pixel_interpolation}
|
|
lp = \left. \sum_{i}^n{ \frac{p_i}{d_i} } \middle/ \sum_{i}{ \frac{1}{d_i} } \right. .
|
|
\end{equation}
|
|
%\begin{equation}
|
|
%\label{eq:pixel_interpolation}
|
|
%\normalsize lp = \frac{\sum_i{ \frac{p_i}{d_i} } }{ \sum_{i}{ \frac{1}{d_i} } }.
|
|
%\end{equation}
|
|
|
|
Liner interpolation is a simple yet effective method to recover lost regions in digital images. However, as each lost pixel is recovered by giving the same importance to each reference direction, the image discontinuities are not taken into account. Thus, some pixels used as reference may belong to distinct regions of the image, without significant correlation with the lost region itself.
|
|
This leads to a poor estimation of lost pixels, resulting in noticeable artefacts in the reconstructed image. For example, in Figure~\ref{FIG3:pixel_interpolation} the lost pixel ($lp$) and the pixel $p_4$ belong to different regions separated by an edge. Therefore, $p_4$ may not be suitable to recover the lost pixel value. So, to
|
|
improve the estimation accuracy of the lost region, the edges and image transitions must be considered, by selecting correct pixels to be used in the interpolation function. To overcome this problem, directional interpolation can be used, by recovering the lost pixels based on those within the same region, according to the edge directions.
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
\input{./022_ErrorResilience/Figures/LinearInterpolation}
|
|
\caption{Lost pixel interpolation from neighbouring error-free pixels. }
|
|
\label{FIG3:pixel_interpolation}
|
|
\end{figure}
|
|
|
|
Directional interpolation improves the performance of the pixel recovery, by firstly estimating the lost edges using either correctly received or previously recovered pixels. In order to obtain the best direction to interpolate, edges are calculated using the image gradient, for example applying the Sobel operator \cite{SignalImageVideoProcessing2012}.
|
|
%This operator gives the first-order approximation of the gradient of the image. This is widely used in image processing to locate the image edges, and uses an horizontal and vertical mask given by:
|
|
%\begin{equation}
|
|
%G_x =
|
|
%\begin{bmatrix}
|
|
%-1 & 0 & 1 \\
|
|
%-2 & 0 & 2 \\
|
|
%-1 & 0 & 1
|
|
%\end{bmatrix},\ \
|
|
%G_y =
|
|
%\begin{bmatrix}
|
|
%-1 & -2 & -1 \\
|
|
% 0 & 0 & 0 \\
|
|
% 1 & 2 & 1
|
|
%\end{bmatrix}.
|
|
%\end{equation}
|
|
%The convolution between the image pixels and the Sobel matrixes results in both horizontal ($g_x$) and vertical ($g_y$) gradients, from which the magnitude and the angular direction are computed using,
|
|
%\begin{equation}
|
|
%\begin{aligned}
|
|
%G &= \sqrt{g^2_x + g^2_y}, \\
|
|
%\theta &= tan^{-1} \left( \frac{g_y}{g_x} \right).\\
|
|
%\end{aligned}
|
|
%\end{equation}
|
|
The Sobel operator is an efficient algorithm to detect edges, as it can reach extreme values at the image edges, so it can be used in pixel interpolation. The usage of this operator for \ac{ec} was evaluated in \cite{IntraConc2010,FineDirectTCE2006,SpatialEdgeConsecutiveBlocks2004,SpatialEdgeModConsecutiveBlocks2005,SpatialInterpolation2008}, and it is characterised as a fast and efficient method due to its small kernel \cite{GradEstimation2009}.
|
|
|
|
% \abbrev{FC}{Frame-Copy}
|
|
% \abbrev{MC}{Motion-Copy}
|
|
|
|
\subsection{Temporal error concealment techniques}
|
|
|
|
%\subsubsection{Frame and motion copy}
|
|
|
|
The conventional way to recover missing frames in the temporal domain are the so-called \ac{fc} and \ac{mc}.
|
|
These \ac{ec} methods were implemented for the \ac{h264} reference software in \cite{H264RefConc2006}.
|
|
The \ac{fc} method consists in repeating (copying) the frame that has the shortest temporal distance in order to to the missing one, into the time instant where the loss occurred.
|
|
The \ac{mc} method uses the motion information of the closest frame to recover the missing one, thus the missing pixels are recovered through motion compensation.
|
|
Previous works and simulation results showed that the \ac{mc} method achieves higher quality in the reconstructed frames in most of the cases~\cite{2DRecursiveRefinement2010}. However, the complexity increases with the implementation of the \ac{mc} method, when compared with \ac{fc}.
|
|
|
|
As an example, in \cite{2DRecursiveRefinement2010} a whole frame loss \ac{ec} algorithm was proposed to improve the performance of the \ac{mc} method, by using a refinement algorithm to improve the accuracy of the \acs{mv} obtained from the closest decoded frame.
|
|
Moreover, in this work the \acs{mv} between consecutive frames are assumed to be strongly correlated, so the motion information of several preceding frames is taken into account to estimate the lost one. Then, the \ac{mv} differences between the vectors of successive frame pairs are recursively calculated along with the next refinement, until the changes become negligibly small (e.g., close to zero).
|
|
This is shown to improve the performance of the \ac{mc} method, achieving gains up $0.47\ dB$ (\ac{psnr}) for a \ac{plr} of $20\%$.
|
|
|
|
|
|
\subsubsection{Boundary match algorithms}
|
|
% \abbrev{ICMV}{Interpolated Candidate Motion Vectors}
|
|
|
|
\Gls{bma} are used to recover a missing region based on the available neighbouring regions. By using the missing block neighbourhood, a new \ac{mv} can be estimated for the missing block. Then, recovery based on motion compensation can be achieved.
|
|
A low complexity approach may use the boundary pixels to search for the \ac{mv} based on a spatial smoothness constraint imposed on the boundaries of the lost block \cite{BMAAlgorithm2008}. Figure~\ref{FIG3:BMAPixelInformation} represents the boundary information used to estimate the \ac{mv} for the lost block from a set of different candidates.
|
|
The \ac{bma} uses the available \acs{mv} provided by the top (T), bottom (B), left (L) and right (R) blocks to simulate different reconstructions of the lost block.
|
|
Then, the optimal reconstruction is selected according to the sum of differences, between the pixels on the internal boundary in the recovered block and the corresponding ones on the external boundary (see Figure~\ref{FIG3:BMAPixelInformation}).
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
%\includegraphics[width=0.45\columnwidth]{./022_ErrorResilience/Figures/BMAAlgorithm}
|
|
\input{./022_ErrorResilience/Figures/BMAAlgorithm}
|
|
\caption{Illustration of the boundary match algorithm.}
|
|
\label{FIG3:BMAPixelInformation}
|
|
\end{figure}
|
|
|
|
An implementation of the \ac{bma} was proposed in \cite{BMAAlgorithm2006}, by analysing the change of magnitude and angle between the \acs{mv} of the neighbouring blocks and the co-located ones in the previous frames. The average change is applied to the \ac{mv} of block co-located lost one to compute the \ac{mv} candidate for recovery.
|
|
An improvement to \ac{bma} was obtained with the outer boundary matching algorithm\cite{OBMAMethod2008}, also known as decoder \ac{mv} estimation algorithm in \cite{DMVE2000}.
|
|
In this method the distortion is calculated by the difference of two outer boundaries of reconstructed block, instead of internal and external boundary in \ac{bma}.
|
|
Although \ac{bma} and outer \ac{bma} have the same design principle of minimising the mismatch distortion, in latter the distortion is measured in the outer boundaries, \emph{i.e.}, external boundaries of the reconstructed block. It was observed in the experimental results \cite{OBMAMethod2008} that outer \ac{bma} has better performance in video recovery than conventional \ac{bma}. The performance improvements are explained by the distortion minimisation of the outer boundary, which incorporates the edge information located in the neighbourhood of the lost block in the decision process to select the most accurate \ac{mv}.
|
|
%Moreover, in the method presented in \cite{OBMAMethod2008} further \ac{mv} candidates were introduced, a refined local search is performed and multiple boundary layers, \emph{i.e.}, increase the overlapped region to measure the distortion and to select the final MV, were tested. The experimental results revealed that the usage of a single layer match gives better performance than multiple layers, while selective search gives excellent trade-off between the image recovery and computation complexity
|
|
% In \cite{DynamicBMA2010} a novel approach to measure the spatial distortion on boundaries of the reconstructed block is presented. Instead of using only horizontal and vertical directions, diagonal and anti-diagonal are also tested, and the best direction is chosen from the reference frame. Since block boundaries may change abruptly, instead of minimising the distortion function the proposed method aims to approximate this distortion to an optimal difference. This optimal difference are determined in the neighbourhood of the co-located block in the reference frame.
|
|
In \cite{BMAAlgorithm2008} a novel distortion function was proposed, to choose among different candidate MVs. The distortion function exploits both spatial and temporal smooth constraints. The temporal function is based on the outer boundary distortion, as in \cite{OBMAMethod2008}.
|
|
The spatial distortion aims to reduce the gradient variances in boundaries of the reconstructed block, so the image edges may be guaranteed. Several experimental results were obtained for different loss events, covering from isolated block lost to the loss of a complete row of blocks. The proposed method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method. Moreover, subjective analysis reveal that this method recover the edges in the lost image. However, the method uses complex mathematical operations, which may not be feasible in real time applications.
|
|
|
|
\subsubsection{Motion vector extrapolation}
|
|
|
|
As mentioned before, the \ac{mc} method is a simple way to reconstruct an estimation of a missing frame using motion information from previously decoded frames. Although this method can provide an acceptable performance comparing with the \ac{fc}, as shown in \cite{H264RefConc2006}, more complex algorithms that use the assumption of motion continuity across consecutive frames are proposed to recover, not only missing blocks, but also whole frames in compressed video.
|
|
|
|
% \abbrev{MMA}{Multi-frame Motion vector Averaging}
|
|
|
|
% To recover the entire missing frame a multi-frame motion vector averaging (MMA) algorithm was proposed in \cite{MMATempConc2003}. The method uses \acs{mv} of a few past frames (from $t-1$ to $t-L$), in order to estimate the forward \ac{mv} for each pixel in the last received frame. Then, each pixel of the last frame is projected onto the missing one. More specifically, the method proposed in \cite{MMATempConc2003} recovers the missing frame at instant $t$ using the following steps:
|
|
% \begin{enumerate}
|
|
% \item Generation of a \ac{mv} history based on the previously decoded frames from $t-1$ to $t-L$, with $L \in \left[ 2,5 \right]$, in order to understand which pixels were used to predict the ones of the frame at $t-1$.
|
|
% \item The \ac{mv} history is used to obtain a set of forward MV, from the frame $t-1$ to the missing one (at instant $t$). For each pixel, the forward \ac{mv} is measured by averaging the \ac{mv} history.
|
|
% \item A spatial regularisation of the \ac{mv} field is performed, by applying a median filter with a kernel size of $12$, to approximate an \mbox{edge-preserving} regularisation \cite{MedianFilterRegul2000}.
|
|
% \item The reconstruction of the missing frame is performed, at a half-pixel level, through the previously obtained forward MVs. The overlapped pixels are averaged, in order to obtain the final pixel value.
|
|
% \item The unfilled regions are recovered, with the median value of a $7\times7$ window
|
|
% \item Finally, the fully reconstructed frame at a half-pixel level is filtered using a mean filter with a $2\times2$ kernel, and down sampled.
|
|
% \end{enumerate}
|
|
|
|
% This method usually produces relatively high quality in the reconstructed frame, but in low motion cases it behaves worse than the \ac{fc} method under a few conditions, as indicated by the experimental results in \cite{MMATempConc2003}. This phenomenon happens because the oldest reference frames usually do not have \ac{mv} similar to the missing one. Moreover, if $L$ increases, the complexity of the proposed algorithm may not be useful to be used in \ac{ec}, due to complexity and memory requirements to determine the \ac{mv} history.
|
|
|
|
\glsreset{mve}
|
|
|
|
|
|
|
|
\ac{mve} \cite{BlockBasedMVE2002,BlockBasedMVEOverllaped2011} methods are implemented quiet simple yet efficient techniques that reconstruct the \acs{mv} of a lost frame which in turn are used for motion compensation.
|
|
These \acs{mv} are extrapolated from the last decoded frame \mbox{(reference frame)} onto the missing one~\cite{BlockBasedMVE2002}.
|
|
Each block in the reference frame is projected to the missing one with its own direction given by the corresponding extrapolated \ac{mv} ($mv'$).
|
|
An example of \ac{mve} is shown in Figure~\ref{FIG3:MVEExtrapolation}, where the vectors of frame $t-1$ are projected onto the missing frame at instant $t$. Note that, in the reference frame ($f_{t-1}$) the \acs{mv} may point to different frames ($f_{t-2}$~and~$f_{t-3}$), previously decoded, so \ac{mve} must take into account the temporal distance between different frames.
|
|
To improve the accuracy of \ac{mve}, the damaged region is divided smaller blocks, and according to the matching area between the extrapolated block and the damaged one, the best \ac{mv} is chosen.
|
|
In order words, the \ac{mv} for the each block is obtained from the extrapolated block with the highest number of pixels matching the missing locations.
|
|
Note that the final motion information need to be aligned to a fixed size grid, e.g., $4\times4$ grid, which leads to a misalignment of the extrapolated \acs{mv}.
|
|
Alternatively, in \cite{PixelBasedMVE2004} several blocks are extrapolated to the missing frame, resulting in overlapped regions, which are recovered using a weighted average of the multiple pixels candidates.
|
|
The weight coefficients assume a maximum value in the center of the extrapolated region, and a minimum value at the boundaries.
|
|
%
|
|
%Since the choice of the best candidate \ac{mv} relies on the overlapped area of the extrapolated block with the lost region, the method may give erroneously estimated \ac{mv} in large motion scenes. The example of Figure~\ref{FIG3:MVEPixelBased} shows a lost $8\times8$ block (in the centre of the figure) and four possible extrapolated \acs{pu} that can be used to estimate lost the motion information. As illustrated in the figure, although the largest overlapped area belongs to the PU$_2$, the overlapped pixels correspond to a small portion of the whole missing block. Moreover, the majority of the lost block's area was not covered by any extrapolated \ac{pu}. Thus, motion recovery using only the information available in PU$_2$ will not properly characterise the motion information for all lost pixels.
|
|
%
|
|
%\begin{figure}[t]
|
|
%\centering
|
|
%\input{./022_ErrorResilience/Figures/PixelBasedMVE}
|
|
%\caption{Pixel based motion vector extrapolation \cite{PixelBasedMVE2004}.}
|
|
%\label{FIG3:MVEPixelBased}
|
|
%\vspace{3mm}
|
|
%\end{figure}
|
|
%
|
|
%To overcome the previous limitation and to achieve smoother and finer results, a \ac{pmve} algorithm was proposed in \cite{PixelBasedMVE2004}. This motion vector extrapolation method is divided into two cases:
|
|
%\begin{enumerate}
|
|
% \item The pixels in the lost region that are covered by at least one extrapolated block, such as circle points in Figure~\ref{FIG3:MVEPixelBased}, are recovered using the average of the \acs{mv} of the overlapped block.
|
|
% \item Those pixels which are not covered by any extrapolation, such as the triangle point in Figure~\ref{FIG3:MVEPixelBased}, the \ac{mv} of the co-located pixel in the previous frame is used.
|
|
%\end{enumerate}
|
|
%
|
|
%Then, if the estimated \ac{mv} is $(MV_x,MV_y)$, then the estimation of the missing pixel, $p_m(x,y)$, is obtained by:
|
|
%\begin{equation}\label{EQ:MotionComp}
|
|
% p_m(x,y) = p_r(x+MV_x,y+MV_y),
|
|
%\end{equation}
|
|
%where $p_r$ refers to the pixels in the previous frame. This pixel-based \ac{mve} provides similar results than the block-based \ac{mve} algorithm in low motion scenarios. Nevertheless, it is able to overcome the previous method in large motion scenes.
|
|
An extension to the previous approach was considered in \HIGHLIGHT{\cite{PelBasedMVE_TCE2007}}, by using the \acs{mv} of subsequent available frames. Thus, to estimate the lost motion information, forward and backward extrapolations are performed, in order to obtain two candidates to each lost pixel. Then, the lost pixels are obtained through motion compensation using the \acs{mv} determined by the average of two \ac{mv} candidates. The experimental results revealed that the bi-directional \ac{pmve} outperforms the isolated use of forward or backward \ac{pmve}. Moreover, the bi-directional \ac{pmve} achieves gains up to $1.85$ dB and $1.32$ dB (\ac{psnr}) comparing with the work in \cite{MMATempConc2003} and \ac{mve} \cite{BlockBasedMVE2002}.
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\includegraphics[width=0.9\columnwidth]{./022_ErrorResilience/Figures/MVE}
|
|
\caption{Motion vector extrapolation with the corresponding extrapolated macroblock, based on the method presented in \cite{BlockBasedMVE2002}.}
|
|
\label{FIG3:MVEExtrapolation}
|
|
\end{figure}
|
|
|
|
In \cite{2DConcHybridMotion2010} a \ac{hmve} algorithm was proposed, to improve the accuracy of the \acs{mv} of the \ac{pmve} method, by using the extrapolated vectors at the pixel and block level. Moreover, the \acs{mv} that were wrongly extrapolated are discarded in order to obtain a more accurate MV.
|
|
In this \ac{ec} method, firstly a pixel based motion extrapolation is performed and the pixels are organised into three groups.
|
|
%Group A contains the pixels that are overlapped by at least one extrapolated block; group B includes pixels that are not covered by any extrapolated block, but belong to a block that is covered, \emph{e.g.}, in Figure~\ref{FIG3:MVEPixelBased} the triangle point was not covered by any extrapolated \ac{mb} but it belongs to $8\times8$ block that is covered by 4 blocks.
|
|
%The remaining pixels are included in group C (star pixels in the figure).
|
|
Then, through block-based \ac{mve} two new \acs{mv} are determined: the first one corresponds to the \ac{mv} of the most overlapped block with the current one, and second one is obtained by the weighted average value of the \acs{mv} corresponding to different overlapped blocks.
|
|
To find the more accurate \ac{mv} from the \ac{mv} set composed described above. To achieve this, the \ac{mv} set is further refined using the distance between each pair of \acs{mv}, in order to discard the wrongly extrapolated \acs{mv}.
|
|
% Finally, the pixels of the lost regions (belonging to the groups A and B) are obtained through motion compensation, using the average of the remaining vectors in the \ac{mv} set. The pixels of the group C, although the \ac{mv} set is empty, the same process is applied using the \ac{mv} of the co-located pixel in the previous frame.
|
|
This algorithm is able to outperform the \ac{pmve} for different packet loss probabilities, as shown by the results in \cite{2DConcHybridMotion2010}.
|
|
The method was used to recover both intra and inter frames in video sequences compressed with the \ac{h264}, achieving \ac{psnr} gains above $1.09$ dB in the erroneous frames.
|
|
|
|
Previous works presented for the \ac{hevc} standard are mainly based on temporal \ac{ec} approaches.
|
|
For instance, in~\cite{HEVCMCBlockAug2013} the block partitions of the neighbouring frames were used to improve the performance of \ac{mve}.
|
|
The reconstructed frame keeps smooth boundaries, leading to higher objective and visual quality.
|
|
Moreover, in order to select only the reliable motion information, the residue information can also be used to decide the optimal \ac{mv}, as shown in~\cite{HEVCMCResidueDec2013}.
|
|
|
|
Overall, the aforementioned methods rely on post-processing without any assistance from the encoder-side, which usually leads to inaccurate recovery of the lost information. Thus, in order to achieve higher reconstruction quality, further information need to be shared with the decoder to minimise the residual distortion of \ac{ec}.
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Concealment-aware error resilience}
|
|
\label{Section22:ConcealmentAware}
|
|
|
|
The last category of techniques for robust video transmission address those algorithms that help the \ac{ec} process at the decoder-side. Although such techniques might overlap with some of the concepts described above, such as, data partition or redundant information, they have the main goal of improving the \ac{ec} performance. This class of algorithms combine the error resilience capabilities with \ac{ec} methods to achieve an improved overall performance.
|
|
|
|
One of the approaches that can be used to improve the \ac{ec} performance is to estimate, at the encoding time, the \ac{ec} distortion introduced by the decoder in case of data loss.
|
|
In \cite{ConcealmentDrivenSliceOrdering2008} the \ac{ec} distortion estimation is used for slice classification and \ac{uep}. Such distortion is also used to optimise motion estimation~\cite{ConcealmentAwareME2006}, by including the distortion of the subsequent frame (assuming it is lost and recovered) in the \ac{rd} cost optimisation of the current frame. This was also used in the context of \ac{hevc} to improve its error robustness, as shown in \cite{ErrorResilienceModeHEVC2016}. This kind of approaches take advantage of location techniques and data partitioning to improve the reconstruction quality achieved by \ac{ec} algorithms. Normally, they use a single \ac{ec} method that is implemented both at the encoder and decoder, based on temporal algorithms, e.g., \ac{mc} and \ac{mve}.
|
|
|
|
Another approach is the use \ac{ec} \ac{mv} which are vectors that might be carried by intra-coded units for the purpose of error recovery. When an error occurs in a given coding unit, the \ac{ec} \ac{mv} from the neighbouring units can be used to form a prediction from previously received frames. This concept was exploited in \cite{ConcealmentIntraDataHidng2009} by using data-hiding to carry additional motion information in the \ac{dct} coefficients. This improves the reconstruction quality and also adds complexity from the decoder to the encoder.
|
|
|
|
A different approach to improve the \ac{ec} performance was presented in \cite{PixelInterlacingIntraConcealment2011}. Pixel interlacing is used to divide the frame into different packets in order to enhance the performance of intra-frame \ac{ec} by using linear interpolation. This technique reduces the probability of burst losses, therefore, the neighbouring pixels surrounding the lost regions are available to be used as reference for the concealment operation.
|
|
The above methods facilitate the \ac{ec} performance, either through slice reordering or based on \acs{mv}, but they always rely on the assumption that a fixed \ac{ec} method is used at the decoder.
|
|
In \cite{ResilienceRECAP_ICASSP2009} a low-resolution video is transmitted alongside the main bitstream in order to be used as reference for \ac{ec}. Such redundant stream is used by the decoder to improve the estimation accuracy of motion information. This method was extended in \cite{SaliencyErrorConcealment_TM2013} by using asymmetric coding of the redundant information based on Itti's attention model~\cite{SaliencyItti1998}.
|
|
|
|
The work presented in \cite{ConcealmentSignalling2008,BurstConcealmentSignalling2008} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder, using \ac{roi} to identify the most sensitive image area.
|
|
Although such an approach achieves reasonably good performance, it relies on the spatial neighbouring information, which may not be available in case of full frame loss. Moreover, it requires transmission of the \ac{roi} map to the decoder, which increases the overall bitrate.
|
|
More recently a \ac{ec} signalling methods was proposed for scalable video \cite{SignallingRobustScalable2015}, in order to select the best frame to replace the missing one.
|
|
|
|
Summarising, \ac{ec}-aware error techniques allow the combination of efficient algorithms, e.g., redundant pictures or \ac{ec}-aware motion estimation or embedded information, with algorithms at the decoder which take advantages of such extra information. Such techniques are able to achieve quality improvements in comparison with error recovery approaches at the decoder on its own. Although effective techniques have been proposed in the past, the exploration of dynamic techniques which explore the potential of different \ac{ec} techniques with adaptable decoder have not been fully exploited.
|
|
|
|
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
% Section
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Summary}
|
|
\label{Section22:Summary}
|
|
|
|
This chapter presented an overview of the different error resilience techniques proposed over the last years, based on their main characteristics. Different approaches were described covering techniques for robust video transmission, like dependency reduction at the encoder-side and \ac{ec} by post-processing at the decoder. The techniques described in this chapter provide relevant insights for the methods investigated in this work.
|
|
Different algorithms were considered and combined to increase the error robustness of highly efficient coded video and improve the reconstruction quality in case of errors.
|
|
|
|
|
|
|
|
|
|
|
|
|