jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Initial review of Chapter 6

Browse Source
This commit is contained in:
jfmcarreira
2018-12-14 00:57:02 +00:00
parent c0f90963e3
commit 4aea167b89
1 changed files with 18 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex
+18 -21
View File
@@ -13,7 +13,7 @@
\vspace*{\fill}
\section{Introduction}
In this chapter an error concealment-aware encoding scheme is proposed to improve the quality of decoded video in broadcast environments prone to transmission errors and data loss.
In this chapter an error concealment-aware encoding scheme is proposed to improve the quality of decoded video in networks prone to errors and data loss.
The method aims to enhance the reconstruction of recovered lost frames by optimising the efficiency of the \ac{ec}.
The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled through \ac{sei} messages. Such optimal \ac{ec} modes are found by simulating transmission losses followed by a Lagrangian optimisation of the \textit{signalling rate - \ac{ec} distortion} cost. A generalised saliency-weighted distortion is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. When data loss occurs the decoder uses the signalling information to improve the reconstruction quality.
@@ -31,23 +31,20 @@ Finally, Section~\ref{Section5:Summary} concludes the chapter.
\label{Section5:Problem}
In Chapter~\ref{Chapter:ErrorResilienceReview} different \ac{ec} algorithms that can effectively reconstruct lost video were described, though not all of them exhibit the same accuracy for every lost frame or slice.
Moreover, it was also shown that combining different \ac{ec} approaches at the block level to improve the reconstruction accuracy is not straightforward.
Moreover, it was also shown that combining different \ac{ec} approaches at the block level for improving the reconstruction accuracy is not straightforward.
In the past, hybrid \ac{ec} schemes were proposed to improve the quality of recovered frames by either combining spatial and temporal \ac{ec} algorithms~\cite{HybridConcTB2008} or by optimising the \ac{mve} at the pixel and block levels~\cite{HybridConcealmentTB2008,2DConcHybridMotion2010}.
Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of motion-based recovery approaches by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}.
Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of motion-based recovery approaches by using the information of correctly received neighbouring frames, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}.
Although such methods are able to reduce the reconstruction artefacts and improve the overall video quality, decoder-based decisions are still not fully accurate, as they are based on limited information to be used as reference in the decision process.
Therefore, higher reconstructed quality can be achieved by estimating the best \ac{ec} techniques at the encoding-side, based on the original video signal.
%Such approach not only improves quality but also reduces the complexity of the video decoders.
%In this chapter a robust coding driven by \ac{ec} methods is developed with the objective of enhancing the quality of reconstructed video at decoders receiving erroneous streams from the transmission network. The proposed approach combines error resilience tools at the encoder-side with \ac{ec} at the decoder-side.
Therefore, higher reconstructed quality can be achieved by estimating the best \ac{ec} techniques at the encoding-side, based on the original video signal..
Different approaches were proposed in the past to address the problem of \ac{ec}-aware video coding~\cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ErrorResilienceModeHEVC2016}.
Extra information was also used to aid the \ac{ec} operation at the decoder-side, either by sending redundant information \cite{ResilienceRECAP_ICASSP2009,SaliencyErrorConcealment_TM2013} or by embedding additional coded data into the bitstream \cite{ConcealmentIntraDataHidng2009}.
Although these methods facilitate the \ac{ec} performance either through slice reordering or based on \acs{mv}, they always rely on the assumption that some fixed \ac{ec} method is used at the decoder.
% The work presented in \cite{ConcealmentSignalling2008,SignallingRobustScalable2015} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder. Although such approaches achieves reasonably good performance, it relies on the spatial neighbouring information or multi-layer video, which may not be available in all cases.
Although these methods facilitate the \ac{ec} performance either through slice reordering or based on \acs{mv}, they always rely on the assumption that a fixed \ac{ec} method is used at the decoder.
To advance one step further, this chapter presents an investigation on \ac{ec}-aware video coding and proposes a new scheme to achieve high decoding flexibility in terms of the \ac{ec} approaches, leading to better video quality.
To advance one step further, this chapter presents an investigation of \ac{ec}-aware video coding and proposes a new scheme to achieve high decoding flexibility in terms of the \ac{ec} approaches, leading to better video quality.
This is accomplished by selecting, at the encoder, among several \ac{ec} algorithms with different characteristics, the one that yields the lowest reconstruction distortion at the decoder and also minimises the \ac{rd} cost considering the necessary signalling rate. Thus, the proposed approach differs from similar previous methods \cite{ConcealmentAwareME2006,ErrorResilienceModeHEVC2016} by selecting the best \ac{ec} method without affecting the encoder optimisation.
To further improve the quality of recovered frames, an enhancement layer is encoded with the mismatch residue computed as a difference between the transmitted and recovered frames.
This is a dynamic optimisation process, which is more efficient than conventional approaches based on redundant pictures \cite{RedudantPicsJan2009,VPxResiliencePCS2016}, which transmit the whole frame as redundancy.
In contrast with the previous work presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}, which addressed the error resilience problem by optimising the coding dependencies, the approach followed in this chapter considers both, \ac{ec} performance and the coded rate required for signalling the best \ac{ec} mode to the decoder.
In contrast with the previous work presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}, which addressed the error resilience problem by optimising the coding dependencies, the approach described in this chapter considers both, \ac{ec} performance and the coded rate required for signalling the best \ac{ec} mode to the decoder.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -108,19 +105,19 @@ Then, the recovered motion-field is used in combination with the \ac{mve} method
Typically, optical flow estimation algorithms are used to estimate a 2-D motion field, comprising one \ac{mv} $v=(v_x, v_y)$, for each pixel, which minimises the following energy function:
\begin{equation}
E(x,y) = \sum_{x} \sum_{y} \bigg| f_{0}\big(x,y\big) - f_{1}\big( x + v_x(x,y), y+v_y(x,y) \big) \bigg|.
E(x,y) = \sum_{x} \sum_{y} \bigg| f_{0}\big(x,y\big) - f_{-1}\big( x + v_x(x,y), y+v_y(x,y) \big) \bigg|.
\end{equation}
Besides a full search algorithm, where a block-based approach is used to search for the best \acs{mv} that minimise the sum of absolute errors, three other approaches were tested to evaluate their effectiveness in \ac{ec} algorithms.
These three methods are the following:
These three methods are as follows:
\begin{enumerate}
\item a method based on the total variation using the minimization of the \mbox{L-1} norm~\cite{DualFlow2016};
\item a method based on the total variation of the pixel values using the minimization of the \mbox{L-1} norm~\cite{DualFlow2016};
\item a method which approximates the neighbourhood of each pixel by quadratic polynomials on both frames~\cite{PolyFlow2003};
\item a method based on a combination of feature matching and total variation~\cite{DeepFlow2013}. %\todo{check here!}
\end{enumerate}
\subsubsection{Motion estimation based on total variation and regularisation (\textit{DualFlow})}
In this method the motion field is estimated based on the minimization of the total variation obtained using the L-1 norm \cite{DualFlow2007,DualFlow2016}, taking into account the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}.
In this method the motion field is estimated based on the minimization of the total variation of pixel values measured using the L-1 norm \cite{DualFlow2007,DualFlow2016}, taking into account the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}.
This condition forces the estimated field to be regular across neighbouring pixels by penalising deviations in a quadratic sense.
Using this method, for each pixel $\mathbf{p}$, the motion field $v(\mathbf{p})$ is obtained by minimising the following energy function,
\begin{equation}\label{EQ5:DualFlow}
@@ -128,7 +125,7 @@ E(v) =
\sum_{\mathbf{p} \in \Omega} \Big( \big| f_{-1}(\mathbf{p}) - f_{-2}( \mathbf{p} + v(\mathbf{p}))\big| + \big|\nabla v(\mathbf{p})\big| \Big)%d\mathbf{p},\ \mathbf{p} \in \Omega
\end{equation}
where $f_{-1}$ and $f_{-2}$ are the two previous frames at time instants $t_{-1}$ and $t_{-2}$, as represented in Figure~\ref{FIG5:MotionVectorExtrapolation}.
In~(\ref{EQ5:DualFlow}), $\Omega$ defines the sets of pixels of those frames, and $|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of the optical flow which effectively disallow discontinuities.
In~(\ref{EQ5:DualFlow}), $\Omega$ defines the sets of pixels of those frames, and $|\nabla v(\mathbf{p})|$ is a regularisation term that penalises high variations of the optical flow which effectively disallow discontinuities.
This is based on the assumption that neighbouring regions belong to the same object, thus such regions should have similar motion.
@@ -167,8 +164,8 @@ The combination of both approaches increases the matching accuracy.
This approach was also exploited in \cite{DeepFlow2013} with the following extensions: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layer optimisation.
Figure~\ref{FIG5:DeepFlowDiagram} shows the blocks that are included in the description matching process.
The matching algorithm is built upon a multi-stage architecture using six layers with different patches sizes, interleaving convolutions and max-pooling, which is similar to deep convolutional networks \cite{GradientBasedConvNN_ProcIEEE1998}.
As shown in Figure~\ref{FIG5:DeepFlowDiagram}, multi-size response levels are generated from the reference and target images using convolutions with different patches sizes. This method uses a bottom-up approach, i.e., starting at a fine level and moving towards the coarser levels (larger patches), which are built as an aggregation of responses of smaller patches. This results in a multi-size response pyramid.
The matching algorithm is built upon a multi-stage architecture using six layers with different patch sizes, interleaving convolutions and max-pooling, which is similar to deep convolutional networks \cite{GradientBasedConvNN_ProcIEEE1998}.
As shown in Figure~\ref{FIG5:DeepFlowDiagram}, multi-size response levels are generated from the reference and target images using convolutions with different patch sizes. This method uses a bottom-up approach, i.e., starting at a fine level and moving towards the coarser levels (larger patches), which are built as an aggregation of responses of smaller patches. This results in a multi-size response pyramid.
Subsequently, a local maximum is extracted from each level to achieve a more accurate motion field (quasi-dense correspondence).
Then, an initial motion field is extracted, referred to as $v'$ .
The final optical flow is obtained by minimising an energy function, which can be described by the following equation:
@@ -190,10 +187,10 @@ The smoothness term is a robust penalisation of the gradient of the flow, simila
\subsection{Experimental evaluation}
This section presents a simulation study a performance evaluation of the different \ac{ec} schemes previously described.
This section presents a simulation study of the different \ac{ec} schemes previously described.
This evaluation aims at finding the suitable methods to be used for efficient reconstruction of the missing frames in case of data loss.
The two \ac{ec} methods based on the correctly received \acs{mv} are tested, i.e., \ac{mc} and \ac{mve}, while the \ac{fc} method is used as reference for comparison.
Moreover, the four different approaches to estimate the motion field for the missing frame are compared: (i) full search based on the minimisation of the sum of absolute errors (\FULLSEARCH), (ii) optical flow based on the duality between total variation and smoothness term (\DUALFLOW), (iii) optical flow based on polynomial approximations (\POLYFLOW) and (iv) optical flow based on deep matching (\DEEPFLOW).
Moreover, the four different approaches to estimating the motion field for the missing frame are compared: (i) full search based on the minimisation of the sum of absolute errors (\FULLSEARCH), (ii) optical flow based on the duality between total variation and smoothness term (\DUALFLOW), (iii) optical flow based on polynomial approximations (\POLYFLOW) and (iv) optical flow based on deep matching (\DEEPFLOW).
Table~\ref{TAB5:ECPerformaceComparison} shows the average quality (\ac{psnr}) obtained for the different methods under investigation.
The absolute value of \ac{psnr} is shown for the \FC method, while the differential values are shown for the remaining methods.
@@ -203,7 +200,7 @@ This can be explained by the objects' trajectories of objects which are preserve
\begin{table}[tb]
\centering
\caption{Average quality (\ac{psnr}) using different error concealment algorithms}
\caption{Average decoded video quality (\ac{psnr} in dB) for seven error concealment algorithms under 10\% of random packet loss.}
\label{TAB5:ECPerformaceComparison}
\TABLESVSPACE
\input{\ConcealmentAwareDir/Results/PerformanceConcealmentMethods.tex}
@@ -219,7 +216,7 @@ Moreover, using feature matching in combination with smoothing conditions, the \
\subsection*{Remarks}
Summarising, the analysis of the previous results indicates that all methods are able to cope with lost frames and manage to recover the missing data with reasonable quality.
Summarising, the analysis of the previous results indicates that all \ac{ec} methods are able to cope with lost frames and manage to recover the missing data with reasonable quality.
Nevertheless, it is noticeable that lost frames recovered with an estimated motion field result in higher average quality, especially when using the \DEEPFLOW method.
However, since the other methods are also able to achieve acceptable video quality, they shall not be neglected, as they provide alternative approaches for lost frame recovery with a lower computational complexity (i.e., they not require motion estimation).
Thus, in order to achieve higher reconstruction quality, different approaches shall be considered, aiming at balancing the overall complexity with the quality of the reconstructed frames.