Review of chapter 2

This commit is contained in:
jfmcarreira committed 2018-07-30 15:38:33 +01:00
1 parent f3f362359e
commit 74109b2c4c
31 files changed
+751 -352

No files matched your search

@@ -22,7 +22,7 @@
\end{tabular}&
Unequal error protection, transport priority and multi-channel transmission \\
\midrule
Error concealment by post-processing &
Error concealment at the decoder &
\begin{tabular}{@{}l@{}}
$\bullet$ Linear interpolation \\
$\bullet$ Boundary matching\\
@@ -12,24 +12,39 @@
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
In general, digital communications through networks are prone to transmission errors and data losses.
These errors can be handled in different ways, such as retransmission, error correction, and \ac{ec} \cite{ErrorResilience2000,PixelInterlacingIntraConcealment2011}.
This chapter reviews the main robust video coding techniques that can be applied at the video encoder/decoder to reduce the error propagation and increase the decode video quality. Several techniques have been proposed in the past to deal with the problem of robust video transmission, especially for video coding standard with high temporal dependencies, such as \ac{h264} and more recently \ac{hevc}. These techniques normally address the spatio-temporal dependencies in video coders which provide higher coding gains~\cite{ROPE2007} but lead to lower error robustness. In this chapter the error resilience and concealment schemes are divided in different categories depending on they functionality and algorithms used. Although different each approach follows a different techniques they all aim at striking the most efficient balance between coding efficiency and decoded video quality.
\glsreset{ec}
\glsreset{vcl}
This chapter is organised as follows: Section~\ref{Section22:Localisation} describes the localisation techniques, which reduce the temporal and spatial dependencies in coded bitstreams. Section~\ref{Section22:DataPartitionRedundant} provides a overview of methods based on data partition and classification and bitstream redundancy. These techniques ease the introduction of extra information in order to recover from lost data. Section~\ref{Section22:Concealment} covers the techniques used at the decoder-side as post-processing tools to reduce the effect of errors and improve visual quality and smoothness. Section~\ref{Section22:ConcealmentAware} provides a description of methods use the \ac{ec} techniques to provide improved error robustness at the encoder-side. Finally, Section~\ref{Section22:Summary} concludes this chapter by providing a overall comparison between the different approaches.
In general, digital communications through networks are prone to transmission errors or data loss.
These errors can be handled in different ways, such as retransmission, error correction, and \ac{ec} \cite{ErrorResilience2000,PixelInterlacingIntraConcealment2011}.
This chapter reviews the main robust video coding techniques that can be applied at the \ac{vcl} to reduce the error propagation and increase the decoded video quality.
Several techniques have been proposed in the past to deal with the problem of robust video transmission, especially for video coding standards with high temporal dependencies, such as \ac{h264} and more recently \ac{hevc}.
These techniques are normally focused on the spatio-temporal dependencies in the coded streams, which are used to obtain high coding gains but lead to low error robustness~\cite{ROPE2007}.
In this chapter the error resilience and concealment schemes are divided in different categories depending on their functionality and algorithms.
Although each approach is based on a different techniques they all aim at striking the most efficient balance between coding efficiency and decoded video quality under lossy conditions.
This chapter is organised as follows: Section~\ref{Section22:Localisation} describes the localisation techniques, which reduce the temporal and spatial dependencies in coded bitstreams. Section~\ref{Section22:DataPartitionRedundant} provides an overview of methods based on data partition, classification and bitstream redundancy.
%These techniques ease the introduction of extra information in order to recover from lost data.
Section~\ref{Section22:Concealment} covers the techniques used at the decoder-side as post-processing to reduce the effect of errors and improve visual quality and smoothness.
Section~\ref{Section22:ConcealmentAware} provides a description of methods that use \ac{ec} techniques to achieve improved error robustness.
Finally, Section~\ref{Section22:Summary} concludes this chapter by providing a overall comparison between the different approaches.
\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Categories of robust video coding techniques}
\section{Robust video coding techniques}
%This chapter provides a review of the existing techniques used for robust video transmission under error-prone networks.
In order to increase the robustness of video transmission, robust coding mechanisms can be employed in a transmission chain to mitigate the artefacts caused by data loss. These techniques can be divided into several categories depending on the functional characteristics and the main problem they address~\cite{ErrorResilienceAug2005}. In this chapter we divide the robust video coding techniques into four categories:
In general, most of the compression efficiency is achieved by exploiting both the spatial and temporal redundancy of video signals. Since there is high correlation between neighbouring information, predictive coding schemes are very effective to achieve higher compression gains by exploiting such correlations.
However, the drawback of high predictive coding schemes is the strong dependencies between compressed symbols with great impact on error propagation.
In order to increase the robustness of video transmission, several coding mechanisms can be employed at different elements of a transmission chain to mitigate the artefacts caused by errors and data loss.
These mechanisms can be classified according to the functional characteristics and the type of problem they address~\cite{ErrorResilienceAug2005}.
In this chapter the robust video coding techniques ere grouped into four categories:
\begin{itemize}
\item Localization techniques;
\item Data partitioning and redundant coding;
\item Error concealment by post-processing;
\item Error concealment at the decoder;
\item Concealment-aware error resilience.
\end{itemize}
@@ -40,12 +55,13 @@ In order to increase the robustness of video transmission, robust coding mechani
\input{022_ErrorResilience/ErrorResilienceCategories.tex}
\end{table}
Table~\ref{tab:errorresilience} provides a summary of the techniques covered in this section and the key advantages associated with each class of techniques.
Localisation techniques are able to reduce the spatial and temporal dependencies in the bitstreams~\cite{ErrorResilienceModeHEVC2016,ErrorResilienceModeHEVC2017}.
Data partitioning groups the coded data into different categories depending on their relative importance.
Redundant coding makes use of redundant blocks of data to correct the transmission errors and achieve robust decoding in error prone conditions.
The problem of robust transmission may also be addressed from the decoder side, by using post-processing \ac{ec} methods to ease the effects of data loss.
Finally, concealment-aware techniques might be adopted at the encoder to increase the performance of \ac{ec} algorithms and increase the reconstruction quality of lost data at the decoder~\cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ConcealmentIntraDataHidng2009,ConcealmentSignalling2008}.
Table~\ref{tab:errorresilience} provides a summary of the techniques covered in this section, some examples and the key advantages associated with each group.
Localisation techniques mainly localise and break the predictive coding loop, reducing the spatial and temporal dependencies, so that if an error occurs it is not likely to affect other parts of the coded video~\cite{ErrorResilienceModeHEVC2016,ErrorResilienceModeHEVC2017}.
Data partitioning rely on grouping the coded data into different categories depending on their relative importance.
This ease the introduction of unequal error protection at the transmission layer.
Redundant coding consists in using additional data blocks to enable correction of transmission errors and to achieve robust decoding in error-prone conditions.
In case of the post-processing \ac{ec} methods, the problem of robust transmission is addressed from the decoder-side.
Finally, concealment-aware techniques are used at the encoder, to either adapt the coding loop or introduce extra information which is used to increase the performance of \ac{ec} algorithms and increase the reconstruction quality of lost data at the decoder \cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ConcealmentIntraDataHidng2009,ConcealmentSignalling2008}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -54,8 +70,7 @@ Finally, concealment-aware techniques might be adopted at the encoder to increas
\section{Localisation techniques}
\label{Section22:Localisation}
In general, video coding standard the compression efficiency is achieved by exploiting both the spatial and temporal redundancy of video sources. Since there is high correlation between neighbouring information, predictive coding schemes are very effective and obtain high compression gains exploiting such correlations.
The drawback of applying such techniques is the fact that the compressed symbols becomes very dependent on each other's which highly increase the error propagation.
Localization techniques break the predictive coding loop in order to reduce the likelihood of errors affect large regions of the video stream, both in the spatial and temporal domains.
As expected since localization techniques break the predictive coding, it reduces the efficiency of such methods. This leads to lower compression ratios and increasing bitrate.
@@ -77,8 +92,8 @@ A common localization technique addresses the error resilience problem of the ra
\end{equation}
where $D$ is the distortion of all pixels, $R$ the required bitrate, and $\lambda$ is the Lagrangian multiplier.
In \cite{ROPEJun2000,ROPE2007,ROPERefGenJan2010} a rate distortion optimisation for error resilience was presented.
In order to take into consideration possible network losses a recursive optimal per-pixel estimation (ROPE) of the end-to-end distortion is applied at the encoder-side.
Let $f_n^i$ denote the original value of pixel $i$ in frame $n$, $\hat f_n^i$ denote its encoder reconstruction and $tilde f_n^i$ the decoder reconstruction after error concealment if needed.
In order to take into consideration possible network losses a \ac{rope} of the end-to-end distortion is applied at the encoder-side.
Let $f_n^i$ denote the original value of pixel $i$ in frame $n$, $\hat f_n^i$ denote its encoder reconstruction and $\tilde f_n^i$ the decoder reconstruction after error concealment if needed.
Since data loss may occur in the transmission channel, $tilde f_n^i$ is considered a random variable at the encoder.
The expected mean square error of each pixel is given by:
\begin{align}
@@ -145,7 +160,7 @@ In order to generate such reference frames the following can be applied
where $\check f_n^i$ is a reference frame pixel, $\alpha$ is the leak factor, and $C$ is a constant value.
The use of leaky predictions has been used in the past for error control especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in Equation~(\ref{EQ2:LeakyPredConstant}) is replace by a lower layer error-free reconstruction. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate as shown in \cite{LeakyPredConstant1973}.
In order to overcome the coding efficiency problem weighted prediction can be used with similar advantages. Since \ac{h264} motion compensation can be applied from two individual previously encoded frames and combined together to generate a single prediction for the current frame \cite{WeightedBiPred2005}. In the case of \ac{hevc}, a method was proposed to force the use of dual inter-frame prediction~\ctie{DualFrameME2015} leading to lower error propagations.
In order to overcome the coding efficiency problem weighted prediction can be used with similar advantages. Since \ac{h264} motion compensation can be applied from two individual previously encoded frames and combined together to generate a single prediction for the current frame \cite{WeightedBiPred2005}. In the case of \ac{hevc}, a method was proposed to force the use of dual inter-frame prediction~\cite{DualFrameME2015} leading to lower error propagations.
In case of leaky prediction instead of applying motion compensation is applied once, but with an interpolated frame obtained as follow:
\begin{equation}\label{EQ2:LeakyPredDual}
\check f_n^i = \alpha \hat f_n^i + (1- \alpha) \hat f_{n-1}^i.
@@ -276,9 +291,9 @@ For example, the primary picture may be coded with fine quantization, while the
One the one hand if the primary picture is received, the redundant information is discarded.
One the other hand if errors occur and the primary picture is lost, the redundant one is used to provide a lower level of reconstructed quality and contain the error propagation.
Zhu \emph{et al.} investigated the paradigm of redundant picture coding in \ac{h264} standard~\cite{RedudantPicsJan2009}.
Different approaches were investigated in order to achieve a mixed method based on both reference picture selection to obtain a clean decoding without error propagation and a hierarchical allocation of the redundant pictures to increase the error robustness. In~\cite{VPxResiliencePCS2016} the key frames are encoded and transmitted at lower bitrate to be used in case of error to replace the missing frame.
Redundant pictures achieve good performance under error prone conditions, since they introduce secondary pictures that can be decoded when the primary picture is not available.
Zhu \emph{et al.} investigated the paradigm of redundant picture coding in \ac{h264} standard~\cite{RedudantPicsJan2009}.
Different approaches were investigated in order to achieve a mixed method based on both reference picture selection to obtain a clean decoding without error propagation and a hierarchical allocation of the redundant pictures to increase the error robustness. In~\cite{VPxResiliencePCS2016} the key frames are encoded and transmitted at lower bitrate to be used in case of error to replace the missing frame.
Redundant pictures achieve good performance under error prone conditions, since they introduce secondary pictures that can be decoded when the primary picture is not available.
Moreover, redundant pictures might also work as localisation techniques when different reference frames are selected. However, such approaches generate large amount of redundant data and overheads. In~\cite{RedundantMVsJune2008} a newer approach was proposed to reduce the amount of information in the redundant pictures of the \ac{h264} standard, by introducing only redundant \acs{mv}. This was also used in~\cite{VPxResilience2016}, where the motion information is used to recover the missing frame.
These approaches increases the overall system performance by presenting an efficient trade-off between the redundant bitrate and the error robustness.