534 lines
70 KiB
TeX
Executable File
534 lines
70 KiB
TeX
Executable File
% Figure folders
|
|
\newcommand{\ErrorResilienceReviewFigs}{./022_ErrorResilience/Figures}
|
|
\newcommand{\TransmissionResiliencefigs}{./022_ErrorResilience/Figures}
|
|
\newcommand{\ErrorConcealmentReviewFigs}{./022_ErrorResilience/Figures}
|
|
|
|
|
|
\chapter{Robust Video Coding: A Review}
|
|
\label{Chapter:ErrorResilienceReview}
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Introduction}
|
|
|
|
In general, digital communications through networks are prone to transmission errors and data losses.
|
|
These errors can be handled in different ways, such as retransmission, error correction, and \ac{ec} \cite{ErrorResilience2000,PixelInterlacingIntraConcealment2011}.
|
|
This chapter reviews the main robust video coding techniques that can be applied at the video encoder/decoder to reduce the error propagation and increase the decode video quality. Several techniques have been proposed in the past to deal with the problem of robust video transmission, especially for video coding standard with high temporal dependencies, such as \ac{h264} and more recently \ac{hevc}. These techniques normally address the spatio-temporal dependencies in video coders which provide higher coding gains~\cite{ROPE2007} but lead to lower error robustness. In this chapter the error resilience and concealment schemes are divided in different categories depending on they functionality and algorithms used. Although different each approach follows a different techniques they all aim at striking the most efficient balance between coding efficiency and decoded video quality.
|
|
|
|
This chapter is organised as follows: Section~\ref{Section22:Localisation} describes the localisation techniques, which reduce the temporal and spatial dependencies in coded bitstreams. Section~\ref{Section22:DataPartitionRedundant} provides a overview of methods based on data partition and classification and bitstream redundancy. These techniques ease the introduction of extra information in order to recover from lost data. Section~\ref{Section22:Concealment} covers the techniques used at the decoder-side as post-processing tools to reduce the effect of errors and improve visual quality and smoothness. Section~\ref{Section22:ConcealmentAware} provides a description of methods use the \ac{ec} techniques to provide improved error robustness at the encoder-side. Finally, Section~\ref{Section22:Summary} concludes this chapter by providing a overall comparison between the different approaches.
|
|
|
|
\clearpage
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Categories of robust video coding techniques}
|
|
|
|
%This chapter provides a review of the existing techniques used for robust video transmission under error-prone networks.
|
|
In order to increase the robustness of video transmission, robust coding mechanisms can be employed in a transmission chain to mitigate the artefacts caused by data loss. These techniques can be divided into several categories depending on the functional characteristics and the main problem they address~\cite{ErrorResilienceAug2005}. In this chapter we divide the robust video coding techniques into four categories:
|
|
\begin{itemize}
|
|
\item Localization techniques;
|
|
\item Data partitioning and redundant coding;
|
|
\item Error concealment by post-processing;
|
|
\item Concealment-aware error resilience.
|
|
\end{itemize}
|
|
|
|
\begin{table}[!b]
|
|
\centering
|
|
\caption{Advantages of error robustness techniques by category.}
|
|
\label{tab:errorresilience}
|
|
\input{022_ErrorResilience/ErrorResilienceCategories.tex}
|
|
\end{table}
|
|
|
|
Table~\ref{tab:errorresilience} provides a summary of the techniques covered in this section and the key advantages associated with each class of techniques.
|
|
Localisation techniques are able to reduce the spatial and temporal dependencies in the bitstreams~\cite{ErrorResilienceModeHEVC2016,ErrorResilienceModeHEVC2017}.
|
|
Data partitioning groups the coded data into different categories depending on their relative importance.
|
|
Redundant coding makes use of redundant blocks of data to correct the transmission errors and achieve robust decoding in error prone conditions.
|
|
The problem of robust transmission may also be addressed from the decoder side, by using post-processing \ac{ec} methods to ease the effects of data loss.
|
|
Finally, concealment-aware techniques might be adopted at the encoder to increase the performance of \ac{ec} algorithms and increase the reconstruction quality of lost data at the decoder~\cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ConcealmentIntraDataHidng2009,ConcealmentSignalling2008}.
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Localisation techniques}
|
|
\label{Section22:Localisation}
|
|
|
|
In general, video coding standard the compression efficiency is achieved by exploiting both the spatial and temporal redundancy of video sources. Since there is high correlation between neighbouring information, predictive coding schemes are very effective and obtain high compression gains exploiting such correlations.
|
|
The drawback of applying such techniques is the fact that the compressed symbols becomes very dependent on each other's which highly increase the error propagation.
|
|
Localization techniques break the predictive coding loop in order to reduce the likelihood of errors affect large regions of the video stream, both in the spatial and temporal domains.
|
|
As expected since localization techniques break the predictive coding, it reduces the efficiency of such methods. This leads to lower compression ratios and increasing bitrate.
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
\includegraphics[width=0.7\columnwidth]{\TransmissionResiliencefigs/ErrorPropagation_1.png}
|
|
\caption{Examples of spatial and temporal error propagation and how it can be reduce using localisation techniques \cite{ErrorResilienceAug2005}.}
|
|
\label{fig2:ErrorPropagation}
|
|
\end{figure}
|
|
|
|
This category of error resilience techniques are able to reduce the error propagation at two levels, i.e., temporal and spatial levels as shown in Figure~\ref{fig2:ErrorPropagation}.
|
|
As shown in the figure spatial techniques break the dependencies within a single frame, as temporal error propagation can be reduce by selectively breaking the dependencies across different frames.
|
|
|
|
\subsection{Error resilient rate-distortion optimisation}
|
|
|
|
A common localization technique addresses the error resilience problem of the rate-distortion optimisation on video encoders. Normally, such optimisation is equivalent to the minimisation of the Lagrangian cost given by:
|
|
\begin{equation}
|
|
J = D + \lambda R,
|
|
\end{equation}
|
|
where $D$ is the distortion of all pixels, $R$ the required bitrate, and $\lambda$ is the Lagrangian multiplier.
|
|
In \cite{ROPEJun2000,ROPE2007,ROPERefGenJan2010} a rate distortion optimisation for error resilience was presented.
|
|
In order to take into consideration possible network losses a recursive optimal per-pixel estimation (ROPE) of the end-to-end distortion is applied at the encoder-side.
|
|
Let $f_n^i$ denote the original value of pixel $i$ in frame $n$, $\hat f_n^i$ denote its encoder reconstruction and $tilde f_n^i$ the decoder reconstruction after error concealment if needed.
|
|
Since data loss may occur in the transmission channel, $tilde f_n^i$ is considered a random variable at the encoder.
|
|
The expected mean square error of each pixel is given by:
|
|
\begin{align}
|
|
E\left\{d_n^i\right\} &= E\left\{ \left( f_n^i - \tilde f_n^i\right)^2 \right\} \nonumber \\
|
|
&= (f_n^i)^2 - 2 f_n^i E\left\{ \tilde f_n^i \right\} + E\left\{ \left(\tilde f_n^i\right)^2 \right\}
|
|
\end{align}
|
|
Then to take into consideration the channel losses packet loss probability ($p$) must be used in the distortion measurement.
|
|
This leads to the following for intra-coded pixels:
|
|
\begin{align}\label{EQ2:E2EDIntra}
|
|
E\left\{ \tilde f_n^i \right\} &= \left(1-p\right) \hat f_n^i + p E\left\{ \tilde f_{n-1}^i \right\} \nonumber \\
|
|
E\left\{\left( \tilde f_n^i \right)^2 \right\} &= \left(1-p\right) \left( \hat f_n^i \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\},
|
|
\end{align}
|
|
and inter-coded pixels:
|
|
\begin{align}\label{EQ2:E2EDInter}
|
|
E\left\{ \tilde f_n^i \right\} &= \left(1-p\right) \left( \hat e_n^i + E\left\{ \tilde f_{n-1}^{j} \right\} \right) + p E\left\{ \tilde f_{n-1}^i \right\} \nonumber \\
|
|
E\left\{\left( \tilde f_n^i \right)^2 \right\} &= \left(1-p\right) \left( \left( \hat e_n^i \right)^2 + 2 \hat e_n^i E\left\{ \tilde f_{n-1}^{j} \right\} + E\left\{ \left( \tilde f_{n-1}^{j} \right)^2 \right\} \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\}
|
|
\end{align}
|
|
where the inter-coded pixel $i$ is predicted from pixel $j$ in the previous frame.
|
|
The prediction error $e_n^i$, is quantized to the value $\hat e_n^i$, which is transmitted to the decoder along with the motion information.
|
|
Equations \ref{EQ2:E2EDIntra} and \ref{EQ2:E2EDInter} allow to measure the distortion taking into consideration the compression error but also the possible of frame loss and consequent \ac{ec} distortion.
|
|
This technique was used in \cite{ROPEJun2000} to switch between intra and inter coding modes reducing the temporal dependencies, and, thus achieving superior error robustness in video transmission. In \cite{AdvancesROPEHE2642007} this technique was optimized to quarter pixel estimation in \ac{h264} standard. Moreover, in \cite{ROPEBurst2009} this method is extended for burst losses by introducing the probability of burst losses into the original end-to-end estimation. This shown to improve the quality under frame loss.
|
|
In \cite{EndToEndDistortionConcealment2014} the distortion estimation assumes a temporal \ac{ec} at the decoder-side.
|
|
This increases the performance of the proposed as it does not rely of frame-copy which leads to lower reconstruction quality. More advanced \ac{ec} method were also consider in the study presented in~\cite{EndToEndDistortionConcealment2014}.
|
|
|
|
In order to increase the error robustness of the \ac{hevc} standard distortion estimation was also used in the \ac{rd} optimisation \cite{ErrorResilienceModeHEVC2016} showing significant quality improvements comparing with the reference \ac{hevc} implementation.
|
|
The method in \cite{SSIMModeling2015} proposes the estimation of the end-to-end distortion using the \ac{ssim} \cite{QualitySSIMRef2002} due to its trade-off between complexity and efficiency to evaluate the perceptual quality \cite{VisualQualityMetrics2011}.
|
|
|
|
\subsection{Intra-refreshing}
|
|
|
|
In order to reduce the spatial dependencies one can introduce refresh points into the bitstream through intra-coded blocks \cite{IntraRefresh1999,IntraRefreshAttentionCM2007,IntraRefreshRefSelectionJan2009,IntraRefreshICIP2009,SandroIntraRefresh2011,JointIntraRefresh2014,IntraRefreshFeedback2015,IntraRefreshRPS2017}.
|
|
These techniques are fairly efficient to promote the error robustness of bitstreams, which avoid long error propagation at the temporal level.
|
|
The main feature of these techniques is to increase the number of intra-coded blocks as they have no dependencies to previous frames, however at the expense of higher amount of bits~\cite{IntraRefreshAttentionCM2007}.
|
|
Therefore, different alternatives have been proposed optimising the number of intra-coded blocks introduced by reducing it the refresh regions.
|
|
Such refresh regions can be introduced randomly, periodically, based on the motion information or on the end-to-end distortion estimation.
|
|
Moreover, whenever is possible the encoder can use a feedback channel to decide where to introduce refresh regions~\cite{IntraRefresh1999,IntraRefreshFeedback2015}.
|
|
|
|
In the \ac{h264} reference software it was introduced a non-normative technique for random intra refreshing which introduces cyclic refresh not in raster scan order. Instead a random pattern is applied. This technique guarantees that all \acs{mb} are refreshed at some point, therefore errors do not propagate indefinitely. However, this technique refresh all \ac{mb} exactly the same number of times without consideration for rate distortion optimisation neither the region containing the \acs{mb}.
|
|
In \cite{IntraRefreshICIP2008} a rate control scheme is used to decide if a given \ac{mb} should be refreshed using a intra-coded block. The decision is performed based on the ratio of the Lagrangian cost between the best intra and inter mode. Whenever this ration is lower than a threshold a intra mode is selected. Using this guarantees that intra refreshing only forces an intra prediction mode if its Lagrangian cost is closer to optimal coding method. In \cite{IntraRefreshICIP2009} this is further extended to adapt the intra refresh algorithm to the network conditions, i.e., the threshold value is obtained from the bitrate and the \ac{plr} value. This technique automatically increases the number of intra-coded block as the bitrate and the packet loss increases. In case of \ac{hevc} intra refresh is also being used to reduce error propagation \cite{HEVCIntraRefreshICIP2017}. It was proposed to use joint intra-inter prediction to introduce soft-refresh points in the bitstream. Although such regions are not correctly decoded in case of errors (as happens with intra-refresh), it significantly decreases the bitrate increasing due the amount of intra-coded blocks. When compared with traditional intra-refresh bitrate saving of up to 12\% are achieved for a 5\% \gls{plr}.
|
|
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
\includegraphics[width=0.7\columnwidth]{\TransmissionResiliencefigs/ROI_IntraRefresh_1.png}
|
|
\caption{Examples of spatial and temporal error propagation and how it can be reduce using localisation techniques \cite{IntraRefreshAttentionCM2007}.}
|
|
\label{fig2:IntraRefreshAttention}
|
|
\end{figure}
|
|
|
|
A different approach introduces refresh points based on the video contents. In~\cite{IntraRefreshAttentionCM2007} the error resilience of video streams is increased combining rate control and attention based models to define \acs{roi}. A weighting factor is used in the Lagrangian optimization to increase the number of intra blocks areas determined by the attention models. Using the aforementioned method lower refresh points are required to achieve high error resilience in the \ac{roi}.
|
|
A more recent approach was proposed in \cite{IntraRefreshPerceptual2010} using a perceptual noticeable difference model to determine an estimation of noticeable distortion. Then, using a visual attention model presented in [XX] intra refresh blocks are introduce in the bitstream. This method is able to improve the results of a previous work \cite{IntraRefresh2005}, especially for high motion sequences. This is due to the fact the previous approach only used an estimation of the error propagation without accounting for perceptual information.
|
|
|
|
This was be also exploited for the \ac{hevc} standard \cite{HEVCROIErrorResilience2015} in combination with a feedback channel. Based on the received information the encoder can introduce intra \acs{cu} in the \ac{roi} areas. Experimental results reveal an increasing on the achieved video quality when higher number of intra \acs{cu} are used. Moreover, applying the proposed method to the \ac{roi} exclusively lower bitrate overhead is introduced leading to higher rate distortion gains.
|
|
|
|
% \HIGHLIGHT{INTRA REFRESHING}
|
|
|
|
\subsection{Leaky-predictions}
|
|
|
|
Directly using reconstructed past frames as reference frames for predictive coding, may result in substantial error propagation whenever the reference frames are lost.
|
|
In order to reduce error propagation one can employ leaky predictions \cite{LeakyPred2009,ROILeakyPred2009}, which scales down reconstructed frames.
|
|
This leads to an exponential decay of error propagation, which leads to higher error robustness.
|
|
In order to generate such reference frames the following can be applied
|
|
\begin{equation}\label{EQ2:LeakyPredConstant}
|
|
\check f_n^i = \alpha \hat f_n^i + (1- \alpha)C,
|
|
\end{equation}
|
|
where $\check f_n^i$ is a reference frame pixel, $\alpha$ is the leak factor, and $C$ is a constant value.
|
|
The use of leaky predictions has been used in the past for error control especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in Equation~(\ref{EQ2:LeakyPredConstant}) is replace by a lower layer error-free reconstruction. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate as shown in \cite{LeakyPredConstant1973}.
|
|
|
|
In order to overcome the coding efficiency problem weighted prediction can be used with similar advantages. Since \ac{h264} motion compensation can be applied from two individual previously encoded frames and combined together to generate a single prediction for the current frame \cite{WeightedBiPred2005}.
|
|
In case of leaky prediction instead of applying motion compensation is applied once, but with an interpolated frame obtained as follow:
|
|
\begin{equation}\label{EQ2:LeakyPredDual}
|
|
\check f_n^i = \alpha \hat f_n^i + (1- \alpha) \hat f_{n-1}^i.
|
|
\end{equation}
|
|
Using $\check f_n^i$ as reference frame for prediction significantly decrease error propagation if only one of the original coded frames is lost (i.e., $\hat f_{n}^i$ or $\hat f_{n-1}^i$).
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
\begin{tabular}{x{4cm}x{4cm}x{4cm}}
|
|
\multicolumn{3}{c}{\includegraphics[width=12cm]{\TransmissionResiliencefigs/ROILeakyPrediction.png}} \\
|
|
$f_{n-1}$ & $f_{n}$ & $f_{n+1}$ \\
|
|
\end{tabular}
|
|
\caption{Example of error propagation with a \ac{roi} (shaded area indicates error, and the arrows indicate temporal predictions to previous frame) \cite{ROILeakyPred2009}.}
|
|
\label{fig2:ErrorPropagationROI}
|
|
\end{figure}
|
|
|
|
|
|
In \cite{ROILeakyPred2009} in order to increase the error robustness with lower impact on the coding efficiency leaky prediction is confined to a pre-defined \ac{roi}.
|
|
Figure \ref{fig2:ErrorPropagationROI} shows an example of error propagation within two different regions: \ac{roi} region and background.
|
|
As show in the figure, since the \ac{roi} regions have predictions from regions in the background error propagation across regions occurs.
|
|
Therefore, the method in \cite{ROILeakyPred2009} uses a weighted prediction for the \ac{roi} region obtained as follows: 1) a globally optimal prediction obtained by unrestricted motion estimation, 2) a prediction obtained by restricting motion estimation within the \ac{roi} region. This limits the impact on \ac{roi} from errors that affect background region. Moreover, since the leaky predictions are confined to a specific regions higher coding efficiency is achieved when compared with other methods.
|
|
Equation (\ref{EQ2:LeakyPredDual}) results in a finite impulse response based weighting of reference frames, as previously weighted frames are not considered. In \cite{ROPERefGenJan2010} an infinitive impulse response based scheme is proposed to increase the decay in error propagation. To accomplish that, in Equation (\ref{EQ2:LeakyPredDual}) the use of the previous encoded frame ($\hat f_{n-1}^i.$) is replaced by the previous interpolated frame ($\check f_n^i$).
|
|
The idea of combining multiple predictions was also been used in the \ac{hevc} standard by combining it with end-to-end distortion to select the optimal predictions.
|
|
|
|
\subsection{Reference picture selection}
|
|
|
|
In order to reduce the temporal dependencies and reduce the error propagation reference picture selection has been widely used in the past \cite{ResilienceDualFrameBuffer2004,ErrorResilienceRPSFeb2009,RPSErrorsFeb2009,MVConcatenationAug2011}.
|
|
In \cite{ResilienceDualFrameBuffer2004}, an optimisation algorithm was used to select between short and long term references based on the end-to-end distortion obtained using the \ac{rope} method, in order to increase the prediction distance and reduce error propagation.
|
|
This method can also be used in combination with feedback channels to improve the references frame selection.
|
|
In the decoder side, an \ac{ec} method was later used to take advantage of such prediction structure \cite{ConcealmentDualFrameBuffer2005}.
|
|
|
|
\begin{figure}[b]
|
|
\centering
|
|
\includegraphics[width=0.95\columnwidth]{\TransmissionResiliencefigs/RefPicSelection.png}
|
|
\caption{Examples of reference picture selection in error resilience video transcoding \cite{ErrorResilienceRPSFeb2009}.}
|
|
\label{fig2:ReferencePictureSelection}
|
|
\end{figure}
|
|
|
|
A feedback channel was also used in \cite{ErrorResilienceRPSFeb2009}, to reduce error propagation at end-user decoders, by dynamically selecting the correctly received frames as references. Figure~\ref{fig2:ReferencePictureSelection} shows an example of error resilience transcoding, where based on network feedback the original temporal predictions (represented by the solid arrows) are transcoded to use the correctly received frame, i.e., prediction represented by the dashed arrow.
|
|
A more advanced scheme may be considered, where only the regions affected by mismatch predictions are process in the compressed domain, and subsequently transcoded, thus limit the dependencies to lost frames.
|
|
This was further exploited in \cite{MVConcatenationAug2011}. To limit complexity motion vector concatenation is employed, in order to obtain new inter-predictions to error-free frames.
|
|
In \cite{ChannelAdaptiveVideoStreaming2002} feedback-based is combined with path diversity to select the reference frames that are more likely to be received at the decoder.
|
|
In the proposed approach, the feedback is not directly used to select the reference frames.
|
|
Instead, the reference frames are selected through an end-to-end distortion estimation, taking into account the source distortion and the channel distortion (concealment distortion is also taken into account in channel distortion).
|
|
In \cite{ErrorResilienceModeHEVC2017} a \ac{roi} is defined based on the motion objects is used to limit the reference picture selection based on a feedback channel.
|
|
Moreover, intra refresh is also used based on the temporal distance to the previous intra-refresh frame.
|
|
|
|
In~\cite{IntraRefreshRefSelectionJan2009} an algorithm to introduce intra refresh points at the macroblock level was proposed for the \ac{h264} standard. This method takes into consideration the distortion when no concealment is applied, the distortion when frame-copy concealment is applied and the packet loss conditions, in order to select the refresh blocks. Those refresh blocks are encoded using either intra prediction or inter prediction through a secondary algorithm to select the most reliable reference frame (i.e., frame with less total expected distortion).
|
|
|
|
In the methods mentioned above the dependencies between the frames are analysed and reduced in order to decrease the error propagation in cases of data loss. Since the existing video coding standards use intense predictive coding, these methods present an efficient approach in highly erroneous conditions.
|
|
% However, the method in~\cite{RDReferenceSelectionJul2012} uses more complex distortion estimations, and in ideal conditions multiple loop encoders are needed to select the reference frames optimally.
|
|
Although these closed-loop techniques may offer a reliable control of the decoder distortion, the feedback required from the receiver side and multipath networking are not always available neither feasible in real-time and unidirectional channels (e.g. broadcast).
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Data partition and redundant coding}
|
|
\label{Section22:DataPartitionRedundant}
|
|
|
|
In a compressed bitstream different symbols have different relevance in terms of error propagation, therefore the impact on the decoded quality is dependent on lost symbols. Therefore, data partition techniques have been developed to group different parts of the compressed bitstream according to their importance to the decoded quality. Then, some of those groups can be protected more efficiently than others using \ac{uep}. These techniques are normally associated with \ac{fec} algorithms. By applying multi-levels of \ac{fec}, different levels of error protection are achieved.
|
|
|
|
In a single channel transmission, the important groups can be protected with stronger channel codes than the remaining groups. In case of multi-channel the important groups can be routed over the reliable channels. Moreover, to achieve higher quality in multi-channel transmission one can use extra information to increase the error resilience of video streams, and be able to decode then in case of errors. Redundancy can be added in an explicit form, e.g., redundant picture, or it can be implicit, e.g., multiple description coding \cite{MDC2005}.
|
|
|
|
|
|
\subsection{Flexible Macroblock Ordering}
|
|
|
|
The \ac{h264}~\cite{H264OverviewJul2003} introduced an error resilience tool named \ac{fmo} technique~\cite{H264FMO2006}. This technique allows each \ac{mb} to be assigned freely to a specific slice group (i.e., a set of slices), organized in a mixed-up fashion. The standard supports up to eight slices groups in each picture, and \acs{mb} within a group can be assigned to a several slices in the default raster scan order.
|
|
The case where only one slice groups is used corresponds to the case where the \ac{fmo} tools is disabled.
|
|
\ac{fmo} increases the error resilience by partitioning the bitstream into several groups. Results show that by using \ac{fmo} it is possible to enhance the subjective and objective quality especially for high packet losses with at cost of an affordable overhead~\cite{H264FMO2006}.
|
|
|
|
\ac{h264} \ac{fmo} support six types of slice groups. Figure~\ref{fig2:FMOTypes} shows \ac{fmo} types 0 to 5. In the \ac{fmo} Type 0 each slice groups has a maximum number of \acs{mb} before the next slice group begins. However, the same slice groups can appear more than once in a frame. \ac{fmo} Type 1 corresponds to scattered slices, where the \acs{mb} are assigned to each slice groups with a fixed formula. In \ref{fig2:FMOTypes} it is represented the \ac{fmo} type 1 case with two slice groups resulting in a checkerboard pattern. In the \ac{fmo} Type 2 slice groups are defined by rectangular shapes defined by the top-left and bottom-right corners. Although the standard allows slice groups to overlap each other, each \ac{mb} may only belong to one slice group. \ac{fmo} Type 3, 4 and 5 are special types each allow the configuration to change on every frame. It only allows two slice groups, in order to keep the signalling overhead affordable. The last \ac{fmo} type is the most general case, where the slice groups are explicitly defined using high level packets (i.e., signalling \ac{nal} units), allow more flexibility at the cost of higher signalling overhead.
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
\includegraphics[width=0.8\columnwidth]{\TransmissionResiliencefigs/FMO_Slices_Type.png}
|
|
\caption{Different types of flexible macroblock reordering \cite{H264FMO2006}.}
|
|
\label{fig2:FMOTypes}
|
|
\end{figure}
|
|
|
|
|
|
In the past \ac{fmo} has been used widely used to increase the error robustness of video transmissions~ \cite{FMOErrorResilience2007,FMOResilience2010,FMOResilience2011,FMOROI2012,MotionEnergyDataPart2014} in \ac{h264} streams. In ~\cite{FMOErrorResilience2007} \ac{fmo} is used to increase the \ac{ec} performance and thus reduce the error propagation in H.264. The method divides the \acs{mb} into the two slice groups based on the \ac{ec} distortion using a two-pass approach. Then, the slice groups which has more impact of the \ac{ec} distortion is transmitted using a reliable channel, and it is assumed to be received error-free. Results show that the proposed approach is able to outperform the dispersed mode (\ac{fmo} type 1).
|
|
In \cite{FMOResilience2010} slice classification is also used based on the weighted sum of the motion vector information and the rate-distortion cost. As in the previous approach, the \acs{mb} are ranked based on the importance and distributed in the two groups until the same amount of \acs{mb} are presented in the two slice groups. This was further improved in \cite{FMOResilience2011} by applying a prediction based on the previous frame to estimate the mapping of the \acs{mb} into the slice groups. This reduces the computational complexity of the \ac{mb} grouping while achieving similar error resilience performance.
|
|
The method in \cite{FMOROI2012} divides the video into two regions which are then grouped in different slice groups. In order to classify the different \ac{mb} spatial information is devised from the \ac{mb} bit allocation, and temporal information is extracted based on the \ac{ec} distortion. This technique is able to reduce the number of erroneous \ac{mb} up to 82\% achieving quality of up to 3.97\%, measured with \ac{psnr}, when comparing to the case where \ac{fmo} is not used.
|
|
In \cite{MotionEnergyDataPart2014} a method based on motion energy is also used to classify \acs{mb} into different groups by .
|
|
In order to classify a given \ac{mb} into a groups its motion energy is compared with the motion energy of its neighbours and the average motion energy of the entire frame. This allows to obtain a dynamic threshold which is then used to determine the \ac{mb} importance. This method is compared with other \ac{uep} error protection schemes showing quality improvements under different \acs{plr}.
|
|
|
|
In \cite{FMORedudantSlices_TB2007} a new \ac{fmo} type is proposed combining the type 1 and 3 in order to obtain a checkerboard pattern in a box-out scanning order, thus increasing the scattering of lost \acs{mb}. Results show that the proposed \ac{fmo} type is able to outperform the existing types under single and burst loss events.
|
|
The method described in \cite{FMOTypeROI2014} introduces a new type in order to efficiently divide the video contents in two regions with different levels of importance, based on the \ac{fmo} Type 6. Then, a spiral protection is applied to scan the \acs{mb} grouping then in two groups. Moreover, by proposing a new \ac{fmo} type they are able to signal several \ac{roi} areas achieving lower overhead than using the original \ac{fmo} Type 6.
|
|
|
|
|
|
\subsection{Unequal error protection}
|
|
|
|
\glsreset{uep}
|
|
|
|
Video signals normally have different types of content, both at the temporal and spatial level. Therefore, there is the possibility of categorising it into different groups. Moreover, it is well-know that different regions of a video bitstream are affected differently by transmission errors.
|
|
This motivated the use of \Gls{uep} techniques in video streaming, which allows different parts of bitstream to be better protected against errors. This not only ensures that those regions will be delivered with higher quality, but also leads to a reduction of the total overhead required to increase the error robustness of video streams.
|
|
|
|
\ac{uep} has been used for a long time to increase the error robustness of video transmission. A wide variety of unequal loss protection schemes have been developed in the literature to optimize the performance of video communication over networks with packet loss. It has been observed that, within a \ac{gop}, different frame types (e.g. I, P, or B frames) have different levels of contribution to the reconstructed video quality. This has been exploited for unequal error protection in video transmission as shown in~\cite{UEP_TM2005,UEPMotion_TM2006}.
|
|
|
|
In \cite{UEPLayered_TB1999} an evaluation of layered video was described. In order to increase the error robustness, redundant packets are used which ease the recovering of lost ones. In order to reduce the overhead for error protection each video layer was given a different importance and protected using different levels of redundancy, thus guaranteeing that lower level packets have a higher probability of being decoded without errors. This is a simple approach not dependent on the network capabilities, neither requires a deep knowledge of the video signals. A lightweight model is used to assign packets priorities to be used by error protection algorithms~\cite{UEP_TCE2011}. This way an \ac{uep} can be achieved. The model is based on a weighted sum of four components: (i) a slice type parameter giving higher importance to refresh and reference frames; (ii) a binary flag indicating where a network packet transports video related headers, i.e., \ac{nal} or slice header; (iii) a parameter indicating the distance to the packet containing the next slice; (iv) a parameter indicating the distance to the next refresh frame. The encoder-side is able to extract these parameters based on the \gls{nal} header, which makes it suitable to low-end encoders or even to prioritise each packet at a streaming level.
|
|
|
|
However, \ac{uep} techniques usually require a deep video analysis which is difficult its integration in every transmission scenario.
|
|
For instance, one can use the motion information to classify different frames or even different regions within a frame, e.g., \ac{mb} in \ac{h264}\cite{UEPMotion_TM2006}. The motion information is used to determine the motion energy which is given by the L2-norm of the \acs{mv} within a determined region.
|
|
In~\cite{UEPMotionEnergy_WCNC2013} the motion energy of each frame is computed and compared with neighbouring frames. To achieve this the energy of co-located \acs{mb} in subsequent frames are compared, and higher importance is given to frames with higher motion energy differences. This method was compared with equal error protection revealing higher objective and subjective quality revealing the relation between motion information and impact of errors.
|
|
This work was extended in order to use a two levels unequal protection~\cite{UEPMotion_ICSPCC2012,UEPMotion_ICSPCS2013}. Not only \ac{mb} and frame importance is taken into consideration but also the combined importance of different frames are used to determine the importance of each \ac{gop}.
|
|
|
|
In \cite{Zhang20131390} the transmission distortion is estimated at the packet level, to assess the impact of each packet in the error propagation. Then, different priorities are applied to each packet allow different levels of error protection. This was also be used to schedule the packet transmission through multi path networks\cite{PacketScheduling2010}.
|
|
In order to improve the classification of different regions of a video stream a perceptual algorithm can be used. This increases the correlation between the classification and subjective impact of errors.
|
|
In \cite{UEPSaliency_TCSVT2010} an method based on a combination of two parameters: a spatial parameter based on a perceptual attention model and a temporal parameters based on the frame position within \ac{gop}. The parameter is then used for \ac{fec} assignment, and experimental results show a significant quality improvement both used objective metrics and subjective evaluation.
|
|
|
|
In scalable video coding scheme, the layered structure of the bit stream provides a natural framework for \ac{uep} \cite{UEPScalable_TCE2008,UEPScalable_TCSVT2010,UEPScalable_ICSPICES2015,UEP_TM2018}.
|
|
In~\cite{UEPScalable_TCSVT2010} an algorithm to estimate the overall distortion of reconstructed frame in the decoder is proposed for scalable video. At the encoder-side different enhancement layer truncation, error propagation and \ac{ec} at taken into account to perform the distortion estimation. The method performs a recursively computation of the total expected distortion in the decoder at the picture-level for each layer in the prediction hierarchy. This is used to perform a cross-layer optimisation and apply different levels of error protection.
|
|
The work in~\cite{UEPLayerVideo_TM2014} proposes a technique for finding the optimal coding rates for coded bit-streams at the encoder-side, in order to optimise the necessary \ac{fec} codes at the enhancement layer which aid the decoding of erroneous base-layer video. This not only improves the overall decoded video quality but also prevents that the correctly received enhancement layers are discarded due to missing dependencies.
|
|
|
|
In the context of \ac{hevc} there are also methods to achieve unequal error protection. Since techniques like \ac{fmo} are not available, different approaches were taken into account \cite{HEVCUepLte_IWCMCC2015,UEPVideoDegradation_ISCAS2017,UEP_TM2018}.The work in~\cite{SliceImportance_ICHNRSR2015} proposes a resource allocation scheme based on prioritisation for \ac{lte} Advanced networks~\cite{HEVCUepLte_IWCMCC2015}. The prioritisation is performed at the slice level (i.e., \ac{nal} unit level) based on the inter-frame predictions. Such predictions are analysed and a ranking is given based on the amount of pixels predicted based on the current slice. In~\cite{UEPVideoDegradation_ISCAS2017} a quality degradation model is proposed for video encoded using the \ac{hevc} standard. The quality degradation is obtained by simulation error events and measuring the quality using the \ac{psnr} metric. From the empirical results a model can be devised to predict the quality under different events. This model is used to control the amount of redundant codes are applied at the network level. The work in~\cite{UEP_TM2018} proposes the use of \gls{uep} for video storage in cloud environments in the context of the scalable extension of the \ac{hevc} standard. The proposed method not only takes into consideration the layer information, but also considers the picture type (i.e., I, P or B-frame) to achieve a cross-optimisation.
|
|
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%\section{Redundant Coding}
|
|
\subsection{Redundant pictures}
|
|
|
|
Redundant pictures is a well-known technique adopted in the \ac{h264} standard \cite{RedudantPicsJan2009}, which allows for different representations of the same picture to be transmitted using different encoding parameters.
|
|
For example, the primary picture may be coded with fine quantization, while the redundant one with coarse quantisation at lower bitrate.
|
|
One the one hand if the primary picture is received, the redundant information is discarded.
|
|
One the other hand if errors occur and the primary picture is lost, the redundant one is used to provide a lower level of reconstructed quality and contain the error propagation.
|
|
|
|
Zhu \emph{et al.} investigated the paradigm of redundant picture coding in \ac{h264} standard~\cite{RedudantPicsJan2009}.
|
|
Different approaches were investigated in order to achieve a mixed method based on both reference picture selection to obtain a clean decoding without error propagation and a hierarchical allocation of the redundant pictures to increase the error robustness.
|
|
Redundant pictures achieve good performance under error prone conditions, since they introduce secondary pictures that can be decoded when the primary picture is not available.
|
|
|
|
Moreover, redundant pictures might also work as localisation techniques when different reference frames are selected. However, such approaches generate large amount of redundant data and overheads. In~\cite{RedundantMVsJune2008} a newer approach was proposed to reduce the amount of information in the redundant pictures of the \ac{h264} standard, by introducing only redundant \acs{mv}. This approach increases the overall system performance by presenting an efficient trade-off between the resultant bitrate and the error robustness.
|
|
|
|
\subsection{Multiple description coding}
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
\includegraphics[width=0.7\columnwidth]{\TransmissionResiliencefigs/MDC.png}
|
|
\caption{Architecture of multiple description system \cite{MDC2005}.}
|
|
\label{fig2:MDCSystem}
|
|
\end{figure}
|
|
|
|
In a \ac{mdc} scheme different bitstreams, named descriptions, are encoded such that a acceptable level of quality is achieve with one of the bitstreams isolated~\cite{MDC2005}.
|
|
Figure \ref{fig2:MDCSystem} illustrates the a basic architecture of encoder and decoder used by the \ac{mdc} technique.
|
|
The encoder-side creates two descriptions, i.e., $R_1$ and $R_2$, which shall be transmitted using two different channels, reducing the probability of both being lost at the same time.
|
|
As shown in the figure three situation may happen at the decoder-side: both descriptions are correctly received and decoder using the central decoder, or either one of the descriptions are not received and then one of the side decoders is used.
|
|
When both bitstreams are received, the central decoder combines the information carried by each description achieving higher quality.
|
|
In the \ac{mdc} encoder the redundancy can be controlled by changing the correlation between descriptions. Higher correlation leads to higher redundancy but also higher quality in the output of the side decoders.
|
|
In contrast with redundant pictures the redundant streams do not carry the same information which means that if both are received the decoded quality is increased.
|
|
|
|
Different types of multiple description coding \cite{MDCHierarBFeb2012} can also be used to improve the quality under transmission errors in multiple path networks \cite{MDCCorreiaJun2012}. Although these techniques can be very effective in multi-path networks, a significant amount of overhead is required to create the different descriptions.
|
|
|
|
%\HERE
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Error concealment by post-processing}
|
|
\label{Section22:Concealment}
|
|
|
|
\glsreset{ec}
|
|
|
|
In general, the \ac{ec} operation is a post processing step in the decoder side and it is the last step to handle errors in the communication chain\cite{ConcealmentReview1998}. Thus, concealment of missing data should take place within the video decoder to minimise perceived effects of lost data. In summary, to enhance the 3D video quality and depth perception, the design of the transmission system must involve the development of adequate \ac{ec} tools. This section reviews several \ac{ec} methods based on spatial and temporal redundancies.
|
|
|
|
|
|
\subsection{Spatial error concealment techniques}
|
|
% \label{SEC3:SpatialConcealment}
|
|
%
|
|
% This section introduces several concealment techniques that exploit the spatial redundancy of images to recover the lost data. To recover lost information, the methods introduced in this section entirely rely on neighbouring image regions received without errors.
|
|
% Consequently, spatial error concealment is mostly useful in the case where information of the neighbouring frames are not available.
|
|
|
|
%\subsection{Pixel level concealment}
|
|
|
|
|
|
A typical approach to conceal missing regions in an image is based on linear interpolation of neighbouring pixels \cite{ErrorConcSpatial1995,ConcTCE2001}. Figure~\ref{FIG3:pixel_interpolation} shows the estimation process of a lost pixel ($lp$), within the lost region ($R_L$), using the four closest pixels ($p_i$) as reference, and the distance between the lost pixel and the reference ones ($d_i$). The pixel value ($lp$) is recovered using a weighted average, as follows:
|
|
\begin{equation}
|
|
\label{eq:pixel_interpolation}
|
|
lp = \left. \sum_{i}^n{ \frac{p_i}{d_i} } \middle/ \sum_{i}{ \frac{1}{d_i} } \right. .
|
|
\end{equation}
|
|
%\begin{equation}
|
|
%\label{eq:pixel_interpolation}
|
|
%\normalsize lp = \frac{\sum_i{ \frac{p_i}{d_i} } }{ \sum_{i}{ \frac{1}{d_i} } }.
|
|
%\end{equation}
|
|
|
|
|
|
Liner interpolation is a simple yet effective method to recover lost regions in digital images. However, as each lost pixel is recovered by giving the same importance to each reference, the image discontinuities are not taken into account. Thus, some pixels used as reference may belong to distinct regions of the image, without significant correlation with the lost region itself.
|
|
This leads to a poor estimation of lost pixels, resulting in noticeable artefacts in the reconstructed image. For example, in Figure~\ref{FIG3:pixel_interpolation} the lost pixel ($lp$) and the pixel $p_4$ are separated by an edge. Therefore, $p_4$ may not be suitable to recover the lost pixel value. So, to ensure a correct concealment of the lost region, the edges and image transitions must be considered, by selecting correct pixels to be used in the interpolation function. To overcome this problem, a directional interpolation can be performed, by recovering the lost pixels based on those within the same region, according to the edge directions.
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
\input{\ErrorConcealmentReviewFigs/LinearInterpolation}
|
|
\caption{ Lost pixel interpolation from neighbouring error-free pixels. }
|
|
\label{FIG3:pixel_interpolation}
|
|
\end{figure}
|
|
|
|
The directional interpolation improves the performance of the pixel recovery, by estimating the lost edges using either correctly received or previously recovered pixels. In order to obtain the best direction to interpolate, edges are calculated using the image gradient, for example applying the Sobel operator \cite{SignalImageVideoProcessing2012}. This operator gives the first-order approximation of the gradient of the image. This is widely used in image processing to locate the image edges, and uses an horizontal and vertical mask given by:
|
|
\begin{equation}
|
|
G_x =
|
|
\begin{bmatrix}
|
|
-1 & 0 & 1 \\
|
|
-2 & 0 & 2 \\
|
|
-1 & 0 & 1
|
|
\end{bmatrix},\ \
|
|
G_y =
|
|
\begin{bmatrix}
|
|
-1 & -2 & -1 \\
|
|
0 & 0 & 0 \\
|
|
1 & 2 & 1
|
|
\end{bmatrix}.
|
|
\end{equation}
|
|
The convolution between the image pixels and the Sobel matrixes results in both horizontal ($g_x$) and vertical ($g_y$) gradients, from which the magnitude and the angular direction are computed using,
|
|
\begin{equation}
|
|
\begin{aligned}
|
|
G &= \sqrt{g^2_x + g^2_y}, \\
|
|
\theta &= tan^{-1} \left( \frac{g_y}{g_x} \right).\\
|
|
\end{aligned}
|
|
\end{equation}
|
|
|
|
The Sobel operator is an efficient algorithm to detect edges, as it can reach extreme values at the image edges, so it can be used in pixel interpolation. The usage of this operator for \ac{ec} was evaluated in \cite{IntraConc2010,FineDirectTCE2006,SpatialEdgeConsecutiveBlocks2004,SpatialEdgeModConsecutiveBlocks2005,SpatialInterpolation2008}, and it is characterised as a fast and efficient method due to its small kernel \cite{GradEstimation2009}.
|
|
|
|
% \abbrev{FC}{Frame-Copy}
|
|
% \abbrev{MC}{Motion-Copy}
|
|
|
|
\subsection{Temporal error concealment techniques}
|
|
|
|
\subsubsection{Motion-copy}
|
|
|
|
The simplest way to recover missing frames in the temporal domain is using the \ac{fc} or the \ac{mc} method. These two basic concealment methods were proposed for the H.264 reference software in \cite{H264RefConc2006}. The \ac{fc} method consists in repeating (copying) the temporally closest frame in order to fill the gap of the missing frame. The \ac{mc} method uses the motion information of the closest frame to recover the missing one, thus the missing pixels are recovered through motion compensation. The experimental results showed that the \mbox{motion-copy} method achieves higher quality in the reconstructed frames in most of the cases. However, the complexity increases with the implementation of the \ac{mc} method, when compared with the \ac{fc} method.
|
|
|
|
In \cite{2DRecursiveRefinement2010} a whole frame loss error concealment algorithm was proposed to improve the performance of the \ac{mc} method, since no refinement is performed in the \acs{mv} obtained from the closest decoded frame. Moreover, in this work the \acs{mv} between consecutive frames are assumed to have a strong correlation, so the motion information of several preceding frames is taken into account to estimate the lost one. Then, the motion vector differences between the vectors of successive frame pairs are recursively calculated, as well as the next refinement area is determined, until it becomes unchanged. This is shown to improve the performance of the \ac{mc} algorithm, achieving gains up $0.47\ dB$ (\ac{psnr}) for a \ac{plr} of $20\%$.
|
|
|
|
|
|
\subsubsection{Boundary match algorithms}
|
|
% \abbrev{ICMV}{Interpolated Candidate Motion Vectors}
|
|
|
|
The recovery of missing blocks in video frames may rely on the available neighbouring regions, to find the suitable \ac{mv} to reconstruct the missing pixels.
|
|
A low complexity approach may use the boundary information of surrounding blocks to search for the motion vector based on a spatial smoothness constraint on boundaries of a lost \ac{mb} \cite{BMAAlgorithm2008}. Figure~\ref{FIG3:BMAPixelInformation} represents the boundary information used to estimate the \ac{mv} for the lost block from a set of different \acs{mv} candidates.
|
|
The \ac{bma} uses the available \acs{mv} provided by the top (T), bottom (B), left (L) and right (R) blocks (see Figure~\ref{FIG3:BMAPixelInformation}) to fetch the pixels from the previous decoded frame.
|
|
Then, the optimal \ac{mv} is selected according to the sum of differences, between pixels on the internal boundary in the recovered block and the corresponding ones on the external boundary.
|
|
These pixels are marked in the figure.
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
%\includegraphics[width=0.45\columnwidth]{\ErrorConcealmentReviewFigs/BMAAlgorithm}
|
|
\input{\ErrorConcealmentReviewFigs/BMAAlgorithm}
|
|
\caption{Illustration of the boundary match algorithm.}
|
|
\label{FIG3:BMAPixelInformation}
|
|
\end{figure}
|
|
|
|
In \cite{BMAAlgorithm2006}, error concealment based on the boundary matching algorithm is improved, by introducing the concept of the interpolated candidate motion vectors.
|
|
The candidate \acs{mv} are calculated with an interpolation of the motion vectors of the neighbouring block in the missing frame and the region around the corresponding block in the previous frame.
|
|
The interpolated candidate \acs{mv} are created based on the change of magnitude and angle of the top, bottom, left and right neighbouring \acs{mv} of the lost block from the previous frame to the current one.
|
|
Then, these modifications are applied to the \ac{mv} of the co-located block with the lost one in the previous frame.
|
|
A more generalised proposal is to apply such transformation not only to the closest neighbours but also to a wider area of blocks.
|
|
The concept of interpolated candidate \acs{mv} can be applied to any method based on \ac{bma}, improving its performance, by introducing more accurate candidates.
|
|
|
|
Another improvement to \ac{bma} was obtained with the outer boundary matching algorithm\cite{OBMAMethod2008}, also known as decoder \ac{mv} estimation algorithm in \cite{DMVE2000}.
|
|
In this method the distortion is calculated by the difference of two outer boundaries of reconstructed block, instead of internal and external boundary in \ac{bma}.
|
|
Although \ac{bma} and outer \ac{bma} have the same design principle of minimising the mismatch distortion, in later the distortion function is measured in the outer boundaries, \emph{i.e.}, external boundaries of the reconstructed block. It was observed in the experimental results that outer \ac{bma} has better performance in video recovery \cite{OBMAMethod2008} than conventional \ac{bma}. The performance improvements are explained by the usage of the outer boundary in the distortion functions, which incorporates the edge information within the neighbourhood of the lost block in the decision of the most accurate \ac{mv} to recover the missing pixels \cite{OBMAMethod2008}. Moreover, in the method presented in \cite{OBMAMethod2008} further \ac{mv} candidates were introduced, a refined local search is performed and multiple boundary layers, \emph{i.e.}, increase the overlapped region to measure the distortion and to select the final MV, were tested. The experimental results revealed that the usage of a single layer match gives better performance than multiple layers, while selective search gives excellent trade-off between the image recovery and computation complexity
|
|
|
|
|
|
% \abbrev{OBMA}{Outer Boundary Matching Algorithm}
|
|
% \abbrev{DMVE}{Decoder Motion Vector Estimation}
|
|
|
|
|
|
|
|
% In \cite{DynamicBMA2010} a novel approach to measure the spatial distortion on boundaries of the reconstructed block is presented. Instead of using only horizontal and vertical directions, diagonal and anti-diagonal are also tested, and the best direction is chosen from the reference frame. Since block boundaries may change abruptly, instead of minimising the distortion function the proposed method aims to approximate this distortion to an optimal difference. This optimal difference are determined in the neighbourhood of the co-located block in the reference frame.
|
|
% In \cite{BMAAlgorithm2008} a novel distortion function was proposed, to choose among different candidate MVs. The distortion function exploits both spatial and temporal smooth constraints. The temporal function is based on the outer boundary distortion, as in \cite{OBMAMethod2008}.
|
|
% The spatial distortion aims to reduce the gradient variances in boundaries of the reconstructed block, so the image edges may be guaranteed. Several experimental results were obtained for different loss events, covering from isolated block lost to the loss of a complete row of blocks. The proposed method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method. Moreover, subjective analysis reveal that this method recover the edges in the lost image. However, the method uses complex mathematical operations, which may not be feasible in real time applications.
|
|
|
|
\subsubsection{Motion vector extrapolation}
|
|
|
|
As mentioned before, the \ac{mc} method is a simple way of concealment a missing frame using motion information from previously decoded frames. Although this method can provide an acceptable performance comparing with the \ac{fc}, as shown in \cite{H264RefConc2006}, more complex algorithms that use the assumption of motion continuity across consecutive frames are proposed to conceal, not only missing blocks, but also whole frames in compressed video.
|
|
|
|
% \abbrev{MMA}{Multi-frame Motion vector Averaging}
|
|
|
|
% To recover the entire missing frame a multi-frame motion vector averaging (MMA) algorithm was proposed in \cite{MMATempConc2003}. The method uses \acs{mv} of a few past frames (from $t-1$ to $t-L$), in order to estimate the forward \ac{mv} for each pixel in the last received frame. Then, each pixel of the last frame is projected onto the missing one. More specifically, the method proposed in \cite{MMATempConc2003} recovers the missing frame at instant $t$ using the following steps:
|
|
% \begin{enumerate}
|
|
% \item Generation of a \ac{mv} history based on the previously decoded frames from $t-1$ to $t-L$, with $L \in \left[ 2,5 \right]$, in order to understand which pixels were used to predict the ones of the frame at $t-1$.
|
|
% \item The \ac{mv} history is used to obtain a set of forward MV, from the frame $t-1$ to the missing one (at instant $t$). For each pixel, the forward \ac{mv} is measured by averaging the \ac{mv} history.
|
|
% \item A spatial regularisation of the \ac{mv} field is performed, by applying a median filter with a kernel size of $12$, to approximate an \mbox{edge-preserving} regularisation \cite{MedianFilterRegul2000}.
|
|
% \item The reconstruction of the missing frame is performed, at a half-pixel level, through the previously obtained forward MVs. The overlapped pixels are averaged, in order to obtain the final pixel value.
|
|
% \item The unfilled regions are recovered, with the median value of a $7\times7$ window
|
|
% \item Finally, the fully reconstructed frame at a half-pixel level is filtered using a mean filter with a $2\times2$ kernel, and down sampled.
|
|
% \end{enumerate}
|
|
|
|
% This method usually produces relatively high quality in the reconstructed frame, but in low motion cases it behaves worse than the \ac{fc} method under a few conditions, as indicated by the experimental results in \cite{MMATempConc2003}. This phenomenon happens because the oldest reference frames usually do not have \ac{mv} similar to the missing one. Moreover, if $L$ increases, the complexity of the proposed algorithm may not be useful to be used in error concealment, due to complexity and memory requirements to determine the \ac{mv} history.
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\includegraphics[width=0.9\columnwidth]{\ErrorConcealmentReviewFigs/MVE}
|
|
\caption{\ac{mve} extrapolation with the corresponding extrapolated macroblock, based on the method presented in \cite{BlockBasedMVE2002}.}
|
|
\label{FIG3:MVEExtrapolation}
|
|
\end{figure}
|
|
|
|
\ac{mve} \cite{BlockBasedMVE2002,BlockBasedMVEOverllaped2011} methods are a simple but efficient way to reconstruct the \acs{mv} of a lost frame, and they are able to achieve good performance. In the method presented in \cite{BlockBasedMVE2002}, \acs{mv} are extrapolated from the last decoded frame \mbox{(reference frame)} onto the missing one. Firstly, each \ac{mb} in the reference frame is projected to the missing one with its own direction given by the corresponding extrapolated motion vector ($mv'$). An example of motion vector extrapolation is shown in Figure~\ref{FIG3:MVEExtrapolation}, where the vectors of the frame $t-1$ are projected onto the missing frame at instant $t$. Note that, in the reference frame ($f_{t-1}$) the motion vectors may point to different frames ($f_{t-2}$~and~$f_{t-3}$), previously decoded, so \ac{mv} extrapolation must take into account the difference between the original temporal distance and the projected one.
|
|
Secondly, to improve the accuracy of \ac{mve}, the damaged region is divided into $4$ blocks ($8\times8$ pixels). Then, according to the matching area between the extrapolated block and the damaged $8\times8$ region, the best \ac{mv} is chosen.
|
|
In order words, the \ac{mv} for the each block is obtained from the extrapolated block that achieves a higher number of overlapped pixels with the missing $8\times8$ region. Finally, the reconstructed \acs{mv} are used to fetch the missing pixels from the previously decoded frame ($f_{t-1}$). Although this method is able to overcome the disadvantage of incorrect \ac{mb} displacement, the choice of the best \ac{mv} is not very effective \cite{PixelBasedMVE2004}. In \cite{BlockBasedMVE2002} the \ac{mve} is used to extrapolate the \ac{mb} of the reference frame.
|
|
This results in overlapped regions, which are recovered using a weighted average of the multiple pixels candidates.
|
|
The weight coefficients assume a maximum value in the center of the extrapolated region, and a minimum value at the boundaries.
|
|
|
|
Since the choice of the best candidate \ac{mv} relies on the overlapped area of the extrapolated block with the lost region, the method may give erroneously estimated \ac{mv} in large motion scenes. The example of Figure~\ref{FIG3:MVEPixelBased} shows a lost $8\times8$ block (in the centre of the figure) and four possible extrapolated \acs{pu} that can be used to estimate lost the motion information. As illustrated in the figure, although the largest overlapped area belongs to the PU$_2$, the overlapped pixels correspond to a small portion of the whole missing block. Moreover, the majority of the lost block's area was not covered by any extrapolated \ac{pu}. Thus, motion recovery using only the information available in PU$_2$ will not properly characterise the motion information for all lost pixels.
|
|
|
|
To overcome the previous limitation and to achieve smoother and finer results, a \ac{pmve} algorithm was proposed in \cite{PixelBasedMVE2004}. This motion vector extrapolation method is divided into two cases:
|
|
\begin{enumerate}
|
|
\item The pixels in the lost region that are covered by at least one extrapolated block, such as circle points in Figure~\ref{FIG3:MVEPixelBased}, are recovered using the average of the \acs{mv} of the overlapped block.
|
|
\item Those pixels which are not covered by any extrapolation, such as the triangle point in Figure~\ref{FIG3:MVEPixelBased}, the \ac{mv} of the co-located pixel in the previous frame is used.
|
|
\end{enumerate}
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\input{\ErrorConcealmentReviewFigs/PixelBasedMVE}
|
|
\caption{Pixel based motion vector extrapolation \cite{PixelBasedMVE2004}.}
|
|
\label{FIG3:MVEPixelBased}
|
|
\vspace{3mm}
|
|
\end{figure}
|
|
|
|
Then, if the estimated \ac{mv} is $(MV_x,MV_y)$, then the estimation of the missing pixel, $p_m(x,y)$, is obtained by:
|
|
\begin{equation}\label{EQ:MotionComp}
|
|
p_m(x,y) = p_r(x+MV_x,y+MV_y),
|
|
\end{equation}
|
|
where $p_r$ refers to the pixels in the previous frame. This pixel-based \ac{mve} provides similar results than the block-based \ac{mve} algorithm in low motion scenarios. Nevertheless, it is able to overcome the previous method in large motion scenes.
|
|
An extension to the previous approach was considered in \cite{PixelBasedMVE2004}, by using the \acs{mv} of subsequent available frames. Thus, to estimate the lost motion information, forward and backward extrapolations are performed, in order to obtain two candidates to each lost pixel. Then, the lost pixel is estimated using the Equation~\ref{EQ:MotionComp}, using the \ac{mv} determined by the average of two \ac{mv} candidates. The experimental results revealed that the bi-directional \ac{pmve} outperforms the isolated use of forward or backward \ac{pmve}. Moreover, the bi-directional \ac{pmve} achieves gains up to $1.85$ dB and $1.32$ dB (\ac{psnr}) comparing with the work in \cite{MMATempConc2003} and \ac{mve} \cite{BlockBasedMVE2002}.
|
|
|
|
In \cite{2DConcHybridMotion2010} a \ac{hmve} algorithm was proposed, to provide more accuracy to the estimated motion information. The method aims to improve the accuracy of the \acs{mv} of the \ac{pmve} method, by using the extrapolated vectors at the pixel and block level. Moreover, the proposed method discards the \ac{mv} that were wrongly extrapolated in order to obtain an accurate MV. In this concealment method, firstly a pixel based motion extrapolation is performed and the pixels is organised into three groups. Group A contains the pixels that are overlapped by at least one extrapolated block (both circle and diamond points in Figure~\ref{FIG3:MVEPixelBased}).
|
|
Group B includes the pixels that are not covered by any extrapolated block, but belong to a block that is covered, \emph{e.g.}, in Figure~\ref{FIG3:MVEPixelBased} the triangle point was not covered by any extrapolated \ac{mb} but it belongs to $8\times8$ block that is covered by 4 blocks.
|
|
The remaining pixels are included in group C (star pixels in the figure). Then, through the block-based \ac{mve} method described in \cite{BlockBasedMVE2002} two new \acs{mv} are determined: the first one corresponds to the \ac{mv} of the most overlapped block with the current one, and second one is obtained by the weighted average value of the \acs{mv} corresponding to different overlapped blocks.
|
|
|
|
The \ac{hmve} method aims to find the more accurate \ac{mv} from the \ac{mv} set composed described above. To achieve this, the \ac{mv} set is further refined using the distance between each pair of \acs{mv}, in order to discard the wrongly extrapolated \acs{mv}. Finally, the pixels of the lost regions (belonging to the groups A and B) are obtained through motion compensation, using the average of the remaining vectors in the \ac{mv} set. The pixels of the group C, although the \ac{mv} set is empty, the same process is applied using the \ac{mv} of the co-located pixel in the previous frame.
|
|
This algorithm is able to outperform the \ac{pmve} for different packet loss probabilities, as shown by the results in \cite{2DConcHybridMotion2010}. The method is used to conceal both I (INTRA) and P frames in video sequence compressed with the \ac{h264}, achieving more $1.09$ dB (\ac{psnr}) in the erroneous frames.
|
|
|
|
%\HIGHLIGHT{HEVC}
|
|
In the case of HEVC, there are also a few works worth to be mentioned. The works presented for the \ac{hevc} standard are mainly based on temporal \ac{ec} approaches.
|
|
In~\cite{HEVCMCBlockAug2013} the block partitions of the neighbouring frames were used to improve the performance motion vector extrapolation. Using this the reconstructed frame keeps the boundaries smooth, leading to higher objective and visual quality.
|
|
Moreover, in order to select only the reliable motion information, the residue information can be exploited, as shown in~\cite{HEVCMCResidueDec2013}.
|
|
|
|
Overall, the aforementioned methods rely on post-processing without any assistance from the encoder-side, which may lead to inaccurate recovery of the lost information. Thus, in order to achieve higher reconstruction quality further information need to be shared with the decoder to minimise the residual distortion of \ac{ec}.
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Concealment-aware error resilience}
|
|
\label{Section22:ConcealmentAware}
|
|
|
|
The last category of techniques for robust video transmission address those algorithms that help the \ac{ec} process at the decoder-side. Although such techniques might overlap with concepts described above, such as, data partition or redundant information, they have the main goal of improving the \ac{ec} performance. This group of algorithms combine the error resilience capabilities with \ac{ec} methods to achieve an improved overall performance.
|
|
|
|
One of the approaches that can be used to improve the \ac{ec} performance is to estimate, at the encoding time, the \ac{ec} distortion introduced by the decoder in case of data loss.
|
|
In \cite{ConcealmentDrivenSliceOrdering2008} the \ac{ec} distortion estimation is used for slice classification and \ac{uep}. Such distortion is also used to optimise motion estimation~\cite{ConcealmentAwareME2006}, by including the distortion of the subsequent frame (assuming it is lost and recovered) in the \ac{rd} cost optimisation of the current frame. This was also used in the context of the \ac{hevc} to improve its error robustness as shown in \cite{ErrorResilienceModeHEVC2016}. This kinda of approaches take advantage of location techniques and data partitioning to improve the reconstruction quality achieved by the \ac{ec} algorithms. Normally, they use a single \ac{ec} method that is implemented both at the encoder and decoder, based on temporal algorithms, e.g., \ac{mc} and \ac{mve}.
|
|
|
|
Another approach is the use \ac{ec} motion vectors which are \acs{mv} that might be carried by intra-coded units for the purpose of concealing errors. When an error occurs in a given unit, the concealment motion vector from the neighbouring units can be used to form a prediction from previously received frames. This concept was exploited in \cite{ConcealmentIntraDataHidng2009} by using data-hiding to carry additional motion information in the \ac{dct} coefficients. This technique not only improves the reconstruction quality but also shift the algorithm complexity from the decoder to the encoder-side.
|
|
|
|
A different approach to improve the \ac{ec} performance was presented in \cite{PixelInterlacingIntraConcealment2011}. Pixel interlacing is used to divide the frame into different packets in order to enhance the performance of intra-frame \ac{ec} using linear interpolation. This technique reduces the probabilities of burst losses, therefore, the neighbouring pixels surrounding the lost regions are available to be used as reference for the concealment operation.
|
|
The above methods facilitate the \ac{ec} performance, either through slice reordering or based on \acs{mv}, they always rely on the assumption that a fixed \ac{ec} method is used at the decoder.
|
|
In \cite{ResilienceRECAP_ICASSP2009} a low-resolution video is transmitted alongside the main bitstream in order to be used as reference for concealment. The redundant stream is used by the decoder to improve the estimation accuracy of motion information. This method was extended in \cite{SaliencyErrorConcealment_TM2013} by using asymmetric coding of the redundant information based on Itti's attention model~\cite{SaliencyItti1998}.
|
|
|
|
The work presented in \cite{ConcealmentSignalling2008,BurstConcealmentSignalling2008} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder, using \ac{roi} to identify the most sensitive image area.
|
|
Although such an approach achieves reasonably good performance, it relies on the spatial neighbouring information, which may not be available in case of full frame loss. Moreover, it requires transmission of the \ac{roi} map to the decoder, which increases the overall bitrate.
|
|
More recently a EC signalling methods was proposed for scalable video \cite{SignallingRobustScalable2015}, in order to select the best frame to be used to replace the missing one.
|
|
|
|
Summarising, \ac{ec}-aware error techniques allow the combination of efficient algorithms, e.g., redundant pictures or error concealment-aware motion estimation or embedded information, with algorithms at the decoder which take advantages of such extra information. Such techniques are able to achieve quality improvements in comparison with the application of error recovery approaches a the decoder on its own. Although effective techniques have been proposed in the past, the exploration of dynamic techniques which explore the potential of different error concealment techniques with adaptable decoder have not been fully exploited.
|
|
|
|
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
% Section
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Summary}
|
|
\label{Section22:Summary}
|
|
|
|
This chapter presented an overview of the different error resilience techniques proposed over the last years, based on they key features. Different approaches were described covering techniques for robust video transmission from dependency reduction at the encoder-side to error concealment by post-processing at the decoder.
|
|
|
|
The techniques described in this chapter provide relevant insights for the methods proposed in this work. In order to achieve increase the error robustness of the \ac{hevc} bitstrems and improved reconstruction quality in case of errors the different algorithms presented before are considered and combined. In Chapter~\ref{Chapter:HEVCErrorResilience} localisation techniques are used to decrease motion vector dependencies in \ac{hevc} standard achieving higher quality at cost of small coding efficiency loss. Chapter~\ref{Chapter:ErrorResilienceTechniques} takes advantage of reference picture selection scheme in combination with motion vector redundancies to improve \ac{hevc} streaming.
|
|
Chapter~\ref{Chapter:ConcealmentAwareTechniques} proposes an error-concealment aware for increase reconstruction quality.
|
|
Finally, Chapter~\ref{Chapter:EndToEndRobustSystem} proposes an end-to-end robust video transmission scheme by combining the error resilience and error concealment based on content, network conditions and application.
|
|
%
|
|
|
|
|
|
|
|
|
|
|
|
|