Review all chapters and start conclusions
This commit is contained in:
1 parent
461b560b17
commit
8d3a4c72ff
20 files changed
+304
-427
No files matched your search
@@ -18,7 +18,7 @@ These techniques are normally focused on the spatio-temporal dependencies in the
|
||||
In this chapter the error resilience and concealment schemes are divided in different categories, depending on their functionality and algorithms.
|
||||
Although each approach is based on different techniques they all aim at striking the best trade-off between coding efficiency and decoded video quality under lossy conditions.
|
||||
|
||||
This chapter is organised as follows: Section~\ref{Section22:Localisation} describes the localisation techniques, which are able to reduce the temporal and spatial dependencies in coded bitstreams.
|
||||
This chapter is organised as follows: Section~\ref{Section22:Localisation} describes the localisation techniques, which are able to reduce the temporal and spatial dependencies in coded bitstreams.
|
||||
Section~\ref{Section22:DataPartitionRedundant} provides an overview on methods based on data partition, classification and bitstream redundancy.
|
||||
%These techniques ease the introduction of extra information in order to recover from lost data.
|
||||
Section~\ref{Section22:Concealment} covers the techniques used at the decoder-side, as post-processing, to reduce the effect of errors and improve visual quality and smoothness.
|
||||
@@ -70,7 +70,7 @@ Finally, concealment-aware techniques are used at the encoder, to either adapt t
|
||||
The basic principle of localization techniques is to break the predictive coding loop at certain predictive points, in order to reduce the likelihood of errors affecting large regions of the video signal, considering both the spatial and temporal domains.
|
||||
Since these techniques break the predictive coding, they have a negative impact in compression efficiency, as expected.
|
||||
|
||||
This group of error resilience techniques are able to reduce the error propagation at two levels, i.e., temporal and spatial levels. Figure~\ref{fig2:ErrorPropagation} is used to demonstrate the principle of error propagation and how it can be reduced using localisation techniques.
|
||||
This group of error resilience techniques are able to reduce the error propagation at two levels, i.e., temporal and spatial levels. Figure~\ref{fig2:ErrorPropagation} is used to demonstrate the principle of error propagation and how it can be reduced using localisation techniques.
|
||||
The top of the figure illustrates a lost block and the corresponding spatial and temporal error propagation on the left and right, respectively.
|
||||
Spatial localisation techniques are used to reduce the error propagation within a frame, where a row of refresh blocks can be introduced to stop the error propagation.
|
||||
To address the temporal error propagation, also refresh blocks (e.g., intra coded blocks) can be randomly introduced to stop error propagation.
|
||||
@@ -122,7 +122,7 @@ E\left\{\left( \tilde f_t^i \right)^2 \right\} = \left(1-p\right) \left( \left(
|
||||
\end{equation}
|
||||
assuming that the pixel $i$ is predicted from pixel $j$ in the previous frame.
|
||||
The prediction error $e_t^i$ is quantised to the value $\hat e_t^i$, which is transmitted to the decoder along with the motion information.
|
||||
Equations \ref{EQ2:E2EDIntra} to \ref{EQ2:E2EDInter} provide a measure of the distortion, taking into consideration not only the coding distortion but also the possibility of frame loss and consequent \ac{ec} distortion.
|
||||
Equations~\ref{EQ2:E2EDIntra} to~\ref{EQ2:E2EDInter} provide a measure of the distortion, taking into consideration not only the coding distortion but also the possibility of frame loss and consequent \ac{ec} distortion.
|
||||
|
||||
This technique was used in \cite{ROPEJun2000} to switch between intra and inter coding modes with the aim of reducing the temporal dependencies, thus achieving enhanced error robustness in video transmission.
|
||||
In \cite{AdvancesROPEHE2642007} this technique was optimised to quarter pixel estimation, and in \cite{ROPEBurst2009} the method was extended for burst losses, by introducing the burst loss probability into the original end-to-end estimation.
|
||||
@@ -193,7 +193,7 @@ In this approach, the reconstructed reference frames are scaled down by applying
|
||||
\check f_t^i = \alpha \hat f_t^i + (1- \alpha)C,
|
||||
\end{equation}
|
||||
where $\check f_t^i$ is a reference frame, $\alpha$ is the leaky factor used to control the amount of information from the reconstructed frame present in the new frame $\check f_t^i$, and $C$ is a constant value.
|
||||
Leaky prediction has been used in the past for error control, especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in Equation~(\ref{EQ2:LeakyPredConstant}) is replaced by a lower layer error-free reconstructed frame. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate, as shown in \cite{LeakyPredConstant1973}. The $\alpha$ parameter allows to control the effect of the constant value, and thus the coding efficiency.
|
||||
Leaky prediction has been used in the past for error control, especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in Equation~(\ref{EQ2:LeakyPredConstant}) is replaced by a lower layer error-free reconstructed frame. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate, as shown in \cite{LeakyPredConstant1973}. The $\alpha$ parameter allows to control the effect of the constant value, and thus the coding efficiency.
|
||||
By decreasing $\alpha$, the coding efficiency increases, however, the exponential decay in error propagation becomes less significant.
|
||||
|
||||
% MHMCP_TCSVT2006
|
||||
@@ -222,7 +222,7 @@ In Equation (\ref{EQ2:LeakyPredDual}) $\hat f_{n}^i$ and $\hat f_{n-1}^i$ always
|
||||
|
||||
|
||||
In \cite{ROILeakyPred2009} a method is presented, where the leaky prediction is confined to a pre-defined \ac{roi}, in order to increase the error robustness with lower impact on the coding efficiency.
|
||||
Figure \ref{fig2:ErrorPropagationROI} shows an example of error propagation within two different regions: \ac{roi} and background.
|
||||
Figure~\ref{fig2:ErrorPropagationROI} shows an example of error propagation within two different regions: \ac{roi} and background.
|
||||
As show in this figure, since the \ac{roi} uses background regions for predictions, error propagation occurs across regions.
|
||||
To overcome this problem, the method in \cite{ROILeakyPred2009} uses a weighted prediction for the \ac{roi} region obtained from the following predictions: (i) a globally optimal prediction obtained by unrestricted motion estimation, (ii) a prediction obtained by restricting motion estimation within the \ac{roi} region.
|
||||
This limits the impact of errors in the background region on the \ac{roi}.
|
||||
@@ -260,8 +260,8 @@ To achieve low complexity, \ac{mv} concatenation can be used, in order to obtain
|
||||
In \cite{ChannelAdaptiveVideoStreaming2002} network feedback is combined with path diversity to select the reference frames that are more likely to be received at the decoder.
|
||||
In the proposed approach, the feedback information is not directly used to select the reference frames.
|
||||
Instead, the reference frames are selected through an end-to-end distortion estimation process, taking into account the coding distortion and distortion resulting from channel errors (\ac{ec} distortion is also taken into account).
|
||||
In \cite{ErrorResilienceModeHEVC2017} a \ac{roi} is defined based on the motion objects to limit the regions where the mechanism of reference picture selection is applied.
|
||||
Intra refresh is also used, based on the temporal distance to the previous intra-refresh frame.
|
||||
In \cite{ErrorResilienceModeHEVC2017} a \ac{roi} is defined based on the motion objects to limit the regions where the mechanism of reference picture selection is applied.
|
||||
Intra refresh is also used, based on the temporal distance to the previous intra-refresh frame.
|
||||
This method not only achieves noticeable quality gains, when compared with the reference \ac{hevc} standard, but also when comparing with traditional reference selection schemes.
|
||||
In~\cite{IntraRefreshRefSelectionJan2009} an algorithm to introduce intra refresh points at the \ac{mb} level was proposed.
|
||||
This method takes into consideration the decoding distortion assuming that frame-copy concealment is applied for \ac{ec} when packet loss occurs.
|
||||
@@ -293,7 +293,7 @@ For instance, redundancy can be added either in an explicitly (e.g., redundant p
|
||||
\subsection{Standard methods for data partitioning and flexible macroblock ordering}
|
||||
|
||||
\glsreset{fmo}
|
||||
The concept of data partitioning was firstly introduced by the H.263 coding standard to group the coded data into different categories~\cite{H264OverIPJul2003}.
|
||||
The concept of data partitioning was firstly introduced by the H.263 coding standard to group the coded data into different categories~\cite{H264OverIPJul2003}.
|
||||
It was further improved, allowing the symbols in each slice to be divided into three partitions:
|
||||
\begin{itemize}
|
||||
\item Type A: headers, \ac{mb} types, quantisation parameters and motion information.
|
||||
@@ -308,7 +308,7 @@ The case where only one slice group is used corresponds to the case where the \a
|
||||
\ac{fmo} increases the error resilience by partitioning the bitstream into several groups, which is able to reduce the error propagation.
|
||||
Results show that the use of \ac{fmo} contributes to enhance the subjective and objective quality, especially for high packet loss rates at cost of an affordable overhead~\cite{H264FMO2006}.
|
||||
|
||||
Six types of slice groups are supported by \ac{fmo}.
|
||||
Six types of slice groups are supported by \ac{fmo}.
|
||||
Figure~\ref{fig2:FMOTypes} shows \ac{fmo} types 0 to 5, since the type 6 as no defined pattern.
|
||||
In the \ac{fmo} type 0 each slice group is defined by a maximum number of \acs{mb}, and a new slice group begins whenever his maximum value is reached. The same slice group can be cyclic repeated within a frame.
|
||||
\ac{fmo} type 1 corresponds to scattered slices, where the \acs{mb} are assigned to each slice group with a fixed formula. Figure~\ref{fig2:FMOTypes} represents the \ac{fmo} type 1 with two slice groups resulting in a checkerboard pattern.
|
||||
@@ -330,11 +330,11 @@ In ~\cite{FMOErrorResilience2007} \ac{fmo} is used to increase the \ac{ec} perfo
|
||||
The method divides the \acs{mb} into the two slice groups based on an estimation of the \ac{ec} distortion using a two-pass approach. Then, the slice group which has the most impact on the \ac{ec} distortion is transmitted using a reliable channel, and it is assumed to be received error-free.
|
||||
Results show that the proposed approach is able to outperform the dispersed mode (\ac{fmo} type 1).
|
||||
Slice classification can also be based on the weighted sum of the motion vector information and the rate-distortion cost~\cite{FMOResilience2010}.
|
||||
As in \cite{FMOErrorResilience2007}, the \acs{mb} are ranked based on the importance and distributed in the two groups with the same number of blocks.
|
||||
As in \cite{FMOErrorResilience2007}, the \acs{mb} are ranked based on the importance and distributed in the two groups with the same number of blocks.
|
||||
In order to reduce the mapping complexity of the \acs{mb} into groups, the same map can be repeated for several subsequent P-frames \cite{FMOResilience2011}.
|
||||
This reduces the computational complexity of the \ac{mb} grouping while achieving similar error resilience performance.
|
||||
The method in \cite{FMOROI2012} splits the video frames into two regions, which are then grouped in different slice groups.
|
||||
To classify the different \acs{mb}, spatial information is extracted from the \ac{mb} bit allocation and temporal information is extracted based on the estimation of the \ac{ec} distortion.
|
||||
To classify the different \acs{mb}, spatial information is extracted from the \ac{mb} bit allocation and temporal information is extracted based on the estimation of the \ac{ec} distortion.
|
||||
This technique is able to reduce the number of erroneous \acs{mb} up to 82\%, achieving a quality gain of up to 3.97\%, in terms of \ac{psnr}, in comparison with the case where \ac{fmo} is not used.
|
||||
Motion energy was also used to classify \acs{mb} into different groups \cite{MotionEnergyDataPart2014}, by comparing the motion energy of a given \ac{mb} with the motion energy of its neighbours and the average motion energy of the entire frame.
|
||||
This allowed the authors to obtain a dynamic threshold which is then used to determine the \ac{mb} importance. In comparison with other \ac{uep} error protection schemes, this method achieve good quality improvements under different \acs{plr}.
|
||||
@@ -354,7 +354,7 @@ By proposing a new \ac{fmo} type they are able to signal several \ac{roi} areas
|
||||
This not only ensures that those regions will be delivered with higher quality, but also leads to efficient use of the total overhead required to achieve increased error robustness in a video transmission.
|
||||
|
||||
A wide variety of \ac{uep} schemes have been presented in the literature to optimise the performance of video communication over networks where the probability of packet loss is not negligible.
|
||||
It has been observed that, within a \ac{gop}, different frame types (e.g. I, P, or B frames) have different levels of contribution to the reconstructed video quality, when transmission losses occur.
|
||||
It has been observed that, within a \ac{gop}, different frame types (e.g. I, P, or B frames) have different levels of contribution to the reconstructed video quality, when transmission losses occur.
|
||||
This has been exploited for \ac{uep} in video transmission, as shown in~\cite{UEP_TM2005,UEPMotion_TM2006}.
|
||||
|
||||
In \cite{UEPLayered_TB1999} an evaluation of layered video was proposed to increase the error robustness, by using redundant packets for helping the recovery of lost ones.
|
||||
@@ -432,13 +432,13 @@ These approaches increase the overall system performance by presenting an effici
|
||||
\end{figure}
|
||||
|
||||
In \ac{mdc} schemes two or more bitstreams, named descriptions, are encoded and transmitted through different channels, such that an acceptable level of quality is achieved when only some of them reach the decoder \cite{MDC1993,MDC2005}.
|
||||
Figure \ref{fig2:MDCSystem} illustrates a basic architecture of a \ac{mdc} encoder and decoder using two descriptions.
|
||||
Figure~\ref{fig2:MDCSystem} illustrates a basic architecture of a \ac{mdc} encoder and decoder using two descriptions.
|
||||
In this example, the encoder-side creates two descriptions, i.e., $R_1$ and $R_2$, which shall be transmitted using two different channels, reducing the probability of the same video data being lost at the same time in both descriptions.
|
||||
As shown in this diagram, three different outputs may be produced at the decoder-side: both descriptions are correctly received and the central decoder is used($D_0$), or either one of the descriptions is not received and then one of the side decoders is used ($D_{1,1}$ and $D_{1,2}$).
|
||||
When both bitstreams are received, the central decoder combines the information carried by each description achieving higher quality than any single side decoder.
|
||||
|
||||
In the \ac{mdc} encoder the redundancy among descriptions can be controlled by changing the correlation between them. Higher correlation leads to higher redundancy, but also higher quality in the output of the side decoders.
|
||||
In contrast with the case of redundant pictures, the redundant \ac{mdc} streams do not carry the same information, which means that if both are received, then the decoded quality may be increased.
|
||||
In contrast with the case of redundant pictures, the redundant \ac{mdc} streams do not carry the same information, which means that if both are received, then the decoded quality may be increased.
|
||||
However, a combination of \ac{mdc} and redundant pictures can also be used to achieve higher quality gains \cite{MDCRedundantPicsJan2010}.
|
||||
Different types of \ac{mdc} approaches \cite{MDCHierarBFeb2012,MDCCorreiaJun2012} can be used to improve the quality under transmission errors, either by taking advantage of hierarchical B-pictures~\cite{MDCHierarBFeb2012} or by introducing extra auxiliary information to limit the error propagation in case of single description decoding~\cite{MDCCorreiaJun2012}.
|
||||
Although these techniques can be very effective in multi-path networks, a significant amount of overhead is required to create the different descriptions~\cite{MDC_TCSVT2011}.
|
||||
@@ -516,7 +516,7 @@ Directional interpolation improves the performance of the pixel recovery, by fir
|
||||
%\theta &= tan^{-1} \left( \frac{g_y}{g_x} \right).\\
|
||||
%\end{aligned}
|
||||
%\end{equation}
|
||||
The Sobel operator is an efficient algorithm to detect edges, as it can reach extreme values at the image edges.
|
||||
The Sobel operator is an efficient algorithm to detect edges, as it can reach extreme values at the image edges.
|
||||
The usage of this operator for \ac{ec} was evaluated in \cite{IntraConc2010,FineDirectTCE2006,SpatialEdgeConsecutiveBlocks2004,SpatialEdgeModConsecutiveBlocks2005,SpatialInterpolation2008}, and it is characterised as a fast and efficient method due to its small kernel \cite{GradEstimation2009}.
|
||||
|
||||
% \abbrev{FC}{Frame-Copy}
|
||||
@@ -534,12 +534,12 @@ Previous works and simulation results showed that the \ac{mc} method achieves hi
|
||||
|
||||
|
||||
In order to improve the \mbox{motion-copy} performance, two reference frames can be used as sources of \acs{mv} to reconstruct the missing one \cite{BiDirMC2011}.
|
||||
The average value of two vectors is used to reconstruct the missing motion information.
|
||||
The average value of two vectors is used to reconstruct the missing motion information.
|
||||
Results in \cite{BiDirMC2011} show quality gains up to $4.99$~dB (\ac{psnr}) when compared with \ac{mc} method.
|
||||
In \cite{2DRecursiveRefinement2010} the \ac{mc} method performance was increased by using a refinement algorithm to improve the accuracy of the \acs{mv} obtained from the closest decoded frame.
|
||||
The \ac{mv} differences between the vectors of successive frame pairs are recursively calculated along with the next refinement, until the changes become negligibly small (e.g., close to zero).
|
||||
This approach has shown to improve the performance of the \ac{mc} method, with gains up to $0.47\ dB$ (\ac{psnr}) for a \ac{plr} of $20\%$.
|
||||
Moreover,
|
||||
Moreover,
|
||||
|
||||
\subsubsection{Boundary match algorithms}
|
||||
% \abbrev{ICMV}{Interpolated Candidate Motion Vectors}
|
||||
@@ -560,12 +560,12 @@ Then, the optimal reconstruction is selected according to the sum of differences
|
||||
An implementation of the \ac{bma} was proposed in \cite{BMAAlgorithm2006}, by analysing the change of magnitude and angle between the \acs{mv} of the neighbouring blocks and the co-located ones in the previous frames. The average change is applied to the \ac{mv} of the block co-located with the lost one to compute the \ac{mv} candidate for recovery.
|
||||
An improvement to \ac{bma} was obtained with the outer boundary matching algorithm\cite{OBMAMethod2008}, also known as decoder \ac{mv} estimation algorithm in \cite{DMVE2000}.
|
||||
In this method the distortion is calculated by the difference of two outer boundaries of reconstructed block, instead of an internal and external boundary in \ac{bma}.
|
||||
Although \ac{bma} and outer \ac{bma} have the same design principle of minimising the mismatch distortion, in the latter the distortion is measured in the outer boundaries, \emph{i.e.}, external boundaries of the reconstructed block.
|
||||
Although \ac{bma} and outer \ac{bma} have the same design principle of minimising the mismatch distortion, in the latter the distortion is measured in the outer boundaries, \emph{i.e.}, external boundaries of the reconstructed block.
|
||||
It was observed in the experimental results of \cite{OBMAMethod2008} that the outer \ac{bma} has better performance in video recovery than conventional \ac{bma}. The performance improvements are explained by the distortion minimisation of the outer boundary, which incorporates the edge information located in the neighbourhood of the lost block in the decision process to select the most accurate \ac{mv}.
|
||||
%Moreover, in the method presented in \cite{OBMAMethod2008} further \ac{mv} candidates were introduced, a refined local search is performed and multiple boundary layers, \emph{i.e.}, increase the overlapped region to measure the distortion and to select the final MV, were tested. The experimental results revealed that the usage of a single layer match gives better performance than multiple layers, while selective search gives excellent trade-off between the image recovery and computation complexity
|
||||
% In \cite{DynamicBMA2010} a novel approach to measure the spatial distortion on boundaries of the reconstructed block is presented. Instead of using only horizontal and vertical directions, diagonal and anti-diagonal are also tested, and the best direction is chosen from the reference frame. Since block boundaries may change abruptly, instead of minimising the distortion function the proposed method aims to approximate this distortion to an optimal difference. This optimal difference are determined in the neighbourhood of the co-located block in the reference frame.
|
||||
In \cite{BMAAlgorithm2008} a novel distortion function was proposed, to choose among different candidate MVs. The distortion function exploits both spatial and temporal smooth constraints. The temporal function is based on the outer boundary distortion, as in \cite{OBMAMethod2008}.
|
||||
The spatial distortion aims to reduce the gradient variances in boundaries of the reconstructed block, so the image edges may be guaranteed.
|
||||
The spatial distortion aims to reduce the gradient variances in boundaries of the reconstructed block, so the image edges may be guaranteed.
|
||||
Several experimental results were obtained for different loss events, covering from an isolated block loss to the loss of a complete row of blocks. The proposed method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method. Moreover, subjective analysis reveal that this method recover the edges in the lost image. However, the method uses complex mathematical operations, which may not be practical in real time applications.
|
||||
|
||||
\subsubsection{Motion vector extrapolation}
|
||||
@@ -679,7 +679,7 @@ Summarising, \ac{ec}-aware error techniques allow the combination of efficient a
|
||||
\section{Summary}
|
||||
\label{Section22:Summary}
|
||||
|
||||
This chapter presented an overview of the different error resilience techniques proposed over the last years, based on their main characteristics. Different approaches were described covering techniques for robust video transmission, like dependency reduction at the encoder-side and \ac{ec}, by post-processing at the decoder. The techniques described in this chapter provide relevant insights for the methods investigated in this work.
|
||||
This chapter presented an overview of the different error resilience techniques proposed over the last years, based on their main characteristics. Different approaches were described covering techniques for robust video transmission, like dependency reduction at the encoder-side and \ac{ec}, by post-processing at the decoder. The techniques described in this chapter provide relevant insights for the methods investigated in this work.
|
||||
Different algorithms were considered and combined to increase the error robustness of highly efficient coded video and improve the reconstruction quality in case of transmission errors.
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user