Review the use of transmission losses to network losses

This commit is contained in:
jfmcarreira committed 2018-12-15 01:42:24 +00:00
1 parent 77cfe2f79d
commit 57fdfd0863
9 files changed
+37 -60

No files matched your search

@@ -9,7 +9,7 @@
In general, digital communications through networks are prone to transmission errors or data loss.
In general, digital communications through networks are prone to errors or data loss.
These errors can be handled in different ways, such as retransmission, error correction, and \ac{ec} \cite{ErrorResilience2000,PixelInterlacingIntraConcealment2011}.
This chapter reviews the existing techniques for robust video coding that can be applied at the \ac{vcl} to reduce the error propagation and increase the decoded video quality.
Several techniques have been proposed in the past to deal with the problem of robust video transmission, especially for video coding standards with high temporal dependencies, such as \ac{h264} and more recently \ac{hevc}.
@@ -23,7 +23,7 @@ Section~\ref{Section22:Localisation} describes the localisation techniques, whic
Section~\ref{Section22:DataPartitionRedundant} provides an overview on methods based on data partition, classification and bitstream redundancy.
%These techniques ease the introduction of extra information in order to recover from lost data.
Section~\ref{Section22:Concealment} covers the techniques used at the decoder-side, as post-processing, to reduce the effect of errors and improve visual quality and smoothness.
Section~\ref{Section22:ConcealmentAware} provides a description of methods that use \ac{ec} techniques to cope with transmission errors and improve error robustness.
Section~\ref{Section22:ConcealmentAware} provides a description of methods that use \ac{ec} techniques to cope with packet loss and improve error robustness.
Finally, Section~\ref{Section22:Summary} concludes the chapter.
% by providing a overall comparison between the different approaches.
@@ -133,11 +133,11 @@ The results obtained in \cite{ROPEBurst2009} revealed that higher video quality
In \cite{E2EDistortion2011} temporal \ac{ec} is considered in the distortion estimation, which increases the performance of the proposed method, by avoiding frame-copy that provides a lower bound on reconstruction quality.
More advanced temporal \ac{ec} methods were also considered in the work presented in~\cite{EndToEndDistortionConcealment2014}.
The \acs{mv} used for motion compensation reconstruction are obtained using boundary matching search, which is able to achieve lower distortion when compared with frame-copy. Using more advanced \ac{ec} algorithms leads to a more accurate estimation of the end-to-end distortion, thus increasing the error robustness of the transmitted video.
To take into account the delay introduced in the encoding loop due to hierarchical prediction structures, delay-based \ac{rd} optimisation was proposed in \cite{DelayRD_TCVT2015} by estimating both the distortion and the end-to-end delay that is minimised along with the required bitrate.
To take into account the delay introduced in the encoding loop due to hierarchical prediction structures, delay-based \ac{rd} optimisation was proposed in \cite{DelayRD_TCVT2015} by estimating both the distortion and the end-to-end delay that is minimised along with the required bitrate.
Using this approach, the total delay can be controlled without severely compromising the coding efficiency.
In the context of \ac{hevc}, end-to-end distortion estimation was also used in the \ac{rd} optimisation showing significant quality improvements when compared to with the reference implementation \cite{ErrorResilienceModeHEVC2016}.
Different quality metrics can also be used for end-to-end distortion estimation.
Different quality metrics can also be used for end-to-end distortion estimation.
The method presented in \cite{SSIMModeling2015} proposes to estimate the distortion by using the \ac{ssim} \cite{QualitySSIMRef2002} in order to increase the correlation between the estimated distortion and the perceptual quality noticed by the human visual system~\cite{VisualQualityMetrics2011}. Moreover, the \ac{ssim} quality metric was also used in learning-based \ac{rd} models~\cite{RateDistortionNN_VCIP2017}, using a \ac{cnn} to predict the end-to-end distortion and the required rate, showing good accuracy in the \ac{rd} cost estimation.
@@ -182,7 +182,7 @@ A more recent approach was proposed using a perceptual noticeable difference mod
This method is able to improve the results of a previous work \cite{IntraRefresh2005}, especially for high motion sequences. This is due to the fact that such previous approach only used an estimation of the error propagation without accounting for perceptual information.
A similar approach was also exploited for \ac{hevc} in \cite{HEVCROIErrorResilience2015,HEVCRoiIntraRefresh_DSP2016}, by using a feedback channel to provide information about the network errors, which is then used by the encoder to introduce intra-coded blocks into the \ac{roi}.
The simulation results reveal an increase on the video quality when higher number of intra blocks are used.
The simulation results reveal an increase on the video quality when higher number of intra blocks are used.
Moreover, applying the proposed method only to the \ac{roi}, the bitrate overhead introduced is lower, leading to higher \ac{rd} gains.
@@ -209,8 +209,8 @@ Alternatively, the two previously encoded frames can be used to generate an inte
\check f_t^i = \alpha \hat f_t^i + (1- \alpha) \hat f_{t-1}^i.
\end{equation}
Therefore, it is possible to decrease the error propagation using $\check f_t^i$ as reference frame for prediction, if only one of the original encoded frames is lost (i.e., $\hat f_{t}^i$ or $\hat f_{t-1}^i$). This is due to the fact that $\check f_t^i$ contains information from both $\hat f_{t}^i$ and $\hat f_{t-1}^i$.
In~(\ref{EQ2:LeakyPredDual}) $\hat f_{t}^i$ and $\hat f_{t-1}^i$ always correspond to the encoded frames and the generated frames ($\check f_t^i $) are not used to interpolate subsequent frames. This corresponds to a finite impulse response filter.
Although the use of two reference frames for leaky prediction is an effective way to reduce the error propagation with a small penalty in coding efficiency, the use of interpolated frame obtained using (\ref{EQ2:LeakyPredDual}) still leads to a severe error propagation in case of burst frame loss.
In~(\ref{EQ2:LeakyPredDual}) $\hat f_{t}^i$ and $\hat f_{t-1}^i$ always correspond to the encoded frames and the generated frames ($\check f_t^i $) are not used to interpolate subsequent frames. This corresponds to a finite impulse response filter.
Although the use of two reference frames for leaky prediction is an effective way to reduce the error propagation with a small penalty in coding efficiency, the use of interpolated frame obtained using (\ref{EQ2:LeakyPredDual}) still leads to a severe error propagation in case of burst frame loss.
In this case both frames would be affect by packet loss, which would reduce the effectiveness of this method.
@@ -290,7 +290,7 @@ In compressed video streams different symbols have different relevance in terms
Therefore, there is an unequal contribution of different syntax elements to the decoding distortion and error propagation, depending on the symbols that are lost in each case.
Data partition techniques have been developed to provide a mechanism to cope with the unequal importance of coded information by grouping different parts of a compressed bitstream, according to their importance to the decoded quality.
Thus, some of these groups can be protected more efficiently than others using \ac{uep} schemes.
These are normally associated with \ac{fec} algorithms, which are typically applied at the channel level which is not aware of the video content, leading to a sub-optimal error protection~\cite{UEPMotion_TM2006}.
These are normally associated with \ac{fec} algorithms, which are typically applied at the channel level which is not aware of the video content, leading to a sub-optimal error protection~\cite{UEPMotion_TM2006}.
Using \ac{uep}, a source-channel optimisation is applied, whereby multi-levels of \ac{fec} are used to achieve different levels of error protection, overcoming this problem.
This is presented and discussed with more detail in Sub-section~\ref{SEC22:UEP}
@@ -311,7 +311,7 @@ Then, it was further improved in \ac{h264}, allowing the symbols in each slice t
\item Type C: inter-coded information and inter coefficients. This type also requires the information of type A, but can be decoded without intra-coded information (type B).
\end{itemize}
These partitions have different levels of importance, according to the type of information carried in each one.
Data partitions of type A are the most important as they carry the header information that is required to decode the remaining partitions in the video stream.
Data partitions of type A are the most important as they carry the header information that is required to decode the remaining partitions in the video stream.
Then, since partitions of type B carry intra-coded information, they are more important than type C, because they can stop temporal error propagation.
An error resilience tool named \ac{fmo} technique was introduced in \ac{h264}, allowing each \ac{mb} to be freely assigned to a specific slice group (i.e., a set of slices), organised in a mixed-up fashion~\cite{H264FMO2006}.
@@ -325,8 +325,8 @@ Figure~\ref{fig2:FMOTypes} shows \ac{fmo} types 0 to 5 only, since the type 6 ha
In the \ac{fmo} type 0 each slice group is defined by a maximum number of \acs{mb}, and a new slice group begins whenever his maximum value is reached. The same slice group can be cyclic repeated within a frame.
\ac{fmo} type 1 corresponds to scattered slices, where the \acs{mb} are assigned to each slice group with a fixed formula. Figure~\ref{fig2:FMOTypes} represents the \ac{fmo} type 1 with two slice groups resulting in a checkerboard pattern.
In the \ac{fmo} type 2, slice groups are defined by rectangular shapes bounded by the top-left and bottom-right corners.
Although the slice groups are allowed to overlap each other, each \ac{mb} may only belong to one slice group.
Similar to \ac{fmo} type 0, in the types 3, 4 and 5 in Figure~\ref{fig2:FMOTypes}, slice groups are defined by a maximum number of \acs{mb}.
Although the slice groups are allowed to overlap each other, each \ac{mb} may only belong to one slice group.
Similar to \ac{fmo} type 0, in the types 3, 4 and 5 in Figure~\ref{fig2:FMOTypes}, slice groups are defined by a maximum number of \acs{mb}.
However, for these types the configuration is allowed to change for every frame, contrary of \ac{fmo} type 0.
In order to keep the signalling overhead affordable, only two slice groups are allowed in these types.
The \ac{fmo} type 6 is the most generic case, where the slice groups are explicitly defined using high level packets (i.e., signalling \ac{nal} units), allowing more flexibility at the cost of higher signalling overhead.
@@ -341,7 +341,7 @@ The \ac{fmo} type 6 is the most generic case, where the slice groups are explici
In the past, \ac{fmo} has been widely used to increase the error robustness of video transmissions in different communication scenarios \cite{FMOErrorResilience2007,FMOResilience2010,FMOResilience2011,FMOROI2012,MotionEnergyDataPart2014}.
In~\cite{FMOErrorResilience2007} \ac{fmo} is used to increase the \ac{ec} performance and thus reduce the error propagation in prioritised video transmission.
The method divides the \acs{mb} into two slice groups based on an estimation of the \ac{ec} distortion using a two-pass approach.
The method divides the \acs{mb} into two slice groups based on an estimation of the \ac{ec} distortion using a two-pass approach.
Then, the slice group which has the most impact on the \ac{ec} distortion is transmitted using a reliable channel, and it is assumed to be received error-free.
The experimental results shows that the proposed approach is able to outperform the dispersed mode (\ac{fmo} type 1)~\cite{FMOErrorResilience2007}.
Slice classification can also be based on the weighted sum of the motion vector information and the rate-distortion cost~\cite{FMOResilience2010}.
@@ -450,7 +450,7 @@ In this example, the encoder-side creates two descriptions, i.e., $R_1$ and $R_2
As shown in this diagram, three different outputs may be produced at the decoder-side: both descriptions are correctly received and the central decoder is used ($D_0$) or, either one of the descriptions is not received and then only one side decoder is used ($D_{1,1}$ or $D_{1,2}$).
When both bitstreams are received, the central decoder combines the information carried by each description achieving higher quality than any single side decoder.
In \ac{mdc} encoding, the redundancy among descriptions can be controlled by changing the correlation between them.
In \ac{mdc} encoding, the redundancy among descriptions can be controlled by changing the correlation between them.
Higher correlation leads to higher redundancy and also higher robustness, also leading to higher quality in the output of the side decoders.
In contrast, with the case of redundant pictures, the redundant \ac{mdc} streams do not carry the same information, which means that if both are received, then the decoded quality may be increased.
However, a combination of \ac{mdc} and redundant pictures can also be used to achieve higher quality gains \cite{MDCRedundantPicsJan2010}.
@@ -527,8 +527,8 @@ This approach has shown to improve the performance of the \ac{mc} method, with g
\subsubsection{Boundary match algorithms}
\Gls{bma} are used to recover a missing region based on the available neighbouring regions.
By using the missing block neighbourhood, a new \ac{mv} can be estimated for the missing block.
\Gls{bma} are used to recover a missing region based on the available neighbouring regions.
By using the missing block neighbourhood, a new \ac{mv} can be estimated for the missing block.
Then, the recovery of the missing pixels is accomplished based on motion compensation.
A low complexity approach may use the boundary pixels to search for the \ac{mv}, based on a spatial smoothness constraint imposed on the boundaries of the lost block \cite{BMAAlgorithm2008}. Figure~\ref{FIG3:BMAPixelInformation} represents the boundary information used to estimate the \ac{mv} for the lost block from a set of different candidates.
The \ac{bma} uses the available \acs{mv} provided by the top (T), bottom (B), left (L) and right (R) blocks to simulate different reconstructions of the lost block.
@@ -549,13 +549,13 @@ Although \ac{bma} and outer \ac{bma} have the same design principle of minimisin
It was observed in the experimental results of \cite{OBMAMethod2008} that the outer \ac{bma} has better performance in video recovery than conventional \ac{bma}. The performance improvements are explained by the distortion minimisation of the outer boundary, which incorporates the edge information located in the neighbourhood of the lost block in the decision process to select the most accurate \ac{mv}.
In \cite{BMAAlgorithm2008} a novel distortion function was proposed, to choose among different candidate MVs. The distortion function exploits both spatial and temporal smooth constraints. The temporal function is based on the outer boundary distortion, as in \cite{OBMAMethod2008}.
The spatial distortion aims to reduce the gradient variances in boundaries of the reconstructed block, so the image edges may be guaranteed.
%Several experimental results were obtained for different loss events, covering from an isolated block loss to the loss of a complete row of blocks.
This method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method.
%Several experimental results were obtained for different loss events, covering from an isolated block loss to the loss of a complete row of blocks.
This method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method.
However, although subjective analysis revealed that this method is able to recover the edges in the lost image, complex mathematical operations are required, which may not be practical in real time applications.
\subsubsection{Motion vector extrapolation}
As mentioned before, the \ac{mc} method is a simple way to reconstruct an estimation of a missing frame using motion information from previously decoded frames.
As mentioned before, the \ac{mc} method is a simple way to reconstruct an estimation of a missing frame using motion information from previously decoded frames.
Although this method can provide an acceptable performance comparing with the \ac{fc}, as shown in \cite{H264RefConc2006}, more complex algorithms that use the assumption of motion continuity across consecutive frames are used to recover the missing information.
\ac{mve} \cite{BlockBasedMVE2002,BlockBasedMVEOverllaped2011} methods are efficient techniques to reconstruct the \acs{mv} of a lost frame, which in turn are used for motion compensation.