Fix figures and add intro figure
This commit is contained in:
@@ -18,14 +18,10 @@
|
||||
\glsreset{rd}
|
||||
|
||||
In this chapter an error concealment-aware encoding scheme is proposed to improve the quality of decoded video in broadcast environments prone to transmission errors and data loss. The method aims to improve the quality of recovered lost frames by optimising the efficiency of the \ac{ec} beforehand at the encoder-side.
|
||||
The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled to the decoder through \ac{sei} messages. Such optimal \ac{ec} modes are found by simulating transmission losses followed by a lagrangian optimisation of the \textit{signalling rate - \ac{ec} distortion} cost. A generalised saliency-weighted distortion is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. When data loss occurs the decoder uses the signalling information is used at the decoder, in case of data loss, to improve the reconstruction quality.
|
||||
The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled through \ac{sei} messages. Such optimal \ac{ec} modes are found by simulating transmission losses followed by a lagrangian optimisation of the \textit{signalling rate - \ac{ec} distortion} cost. A generalised saliency-weighted distortion is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. When data loss occurs the decoder uses the signalling information to improve the reconstruction quality.
|
||||
|
||||
%In this chapter a novel error concealment-aware video encoding scheme is proposed to increase the robustness of \ac{hevc} streams over transmission channels prone to errors or data loss, such as broadcasting networks.
|
||||
%The method aims to improve the quality of recovered lost frames by optimising the efficiency of the error concealment (\ac{ec}) beforehand at the encoder-side. The proposed approach is based on finding the optimal \ac{ec} mode, in \ac{rd} sense, to be used at the decoder side. The process includes data loss simulation and minimisation of a lagrangian cost based on saliency-weighted distortion and signalling rate. Furthermore the residue
|
||||
%between the coded frame and its error reconstructed version is also encoded using a rate-controlled enhancement scalable layer.
|
||||
|
||||
This chapter is organized as follows. Section~\ref{Section5:Problem} presents the problem formulation and motivation for the method proposed in this chapter. In Section~\ref{Section5:ECCandidates} does an initial evaluation of different error concealment schemes in order to evaluate the relevant ones to be used in the proposed error concealment-aware coding scheme.
|
||||
Section~\ref{Section5:OverviewProposedMethod} provides an overview of the architecture of the proposed method and Section~\ref{Section5:ProposedMethod} describe the proposed method in detail.
|
||||
This chapter is organized as follows. Section~\ref{Section5:Problem} presents the problem formulation and motivation for the method proposed in this chapter. In Section~\ref{Section5:ECCandidates} does an initial evaluation of different error concealment schemes in order to find the relevant ones to be used in the proposed error concealment-aware coding scheme.
|
||||
Section~\ref{Section5:OverviewProposedMethod} provides an overview of the architecture of the proposed method followed by a detailed description in Section~\ref{Section5:ProposedMethod}.
|
||||
Section~\ref{Section5:ECSignallingPerformance} presents a discussion of the performance of the proposed \ac{ec} optimisation and signalling, and Section~\ref{Section5:OverallPerformance} presents an overall performance of the proposed method.
|
||||
Finally, Section~\ref{Section5:Summary} concludes this chapter.
|
||||
|
||||
@@ -37,12 +33,17 @@ Finally, Section~\ref{Section5:Summary} concludes this chapter.
|
||||
\label{Section5:Problem}
|
||||
|
||||
\glsreset{mve}
|
||||
\glsunset{mb}
|
||||
|
||||
Chapter~\ref{Chapter:ErrorResilienceReview} described different \ac{ec} algorithms that can effectively reconstruct lost data, leading to lower error propagation. However, as mentioned before different algorithms might not present the same accuracy for every lost frame or slice. Moreover, selection at the \ac{mb}/\ac{pu} level is not straightforward. Approaches in the past proposed the combination of spatial and temporal \ac{ec} strategies to improve the reconstruction quality~\cite{HybridConcTB2008}. Regarding temporal \ac{ec}, there are hybrid algorithms to optimise the \ac{mve} at the pixel and block levels \cite{HybridConcealmentTB2008,2DConcHybridMotion2010}. Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of the motion-based recovery by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}. Although such methods are able to improve the reconstruction quality and reduce the artefacts, decoder-based decisions are still not fully accurate, as they do not have enough reference information to ease the decision process. Therefore, higher performance can be achieved by performing encoding-side estimations and decisions based on the error-free information. This approach not only improves quality but also reduces the complexity of the video decoders.
|
||||
In Chapter~\ref{Chapter:ErrorResilienceReview} different \ac{ec} algorithms that can effectively reconstruct lost data were described. However, different algorithms might not present the same accuracy for every lost frame or slice. Moreover, selection at the \ac{mb}/\ac{pu} level is not straightforward.
|
||||
In the past, hybrid \ac{ec} schemes were proposed to improve the reconstruction quality by either combining spatial and temporal \ac{ec} algorithms~\cite{HybridConcTB2008} or by optimising the \ac{mve} at the pixel and block levels~\cite{HybridConcealmentTB2008,2DConcHybridMotion2010}.
|
||||
Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of the motion-based recovery by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}. Although such methods are able to reduce the reconstruction artefacts and improve the overall quality, decoder-based decisions are still not fully accurate, as they do not have enough reference information to ease the decision process. Therefore, higher performance can be achieved by performing encoding-side estimation and decision based on the error-free information.
|
||||
%Such approach not only improves quality but also reduces the complexity of the video decoders.
|
||||
%In this chapter a robust coding driven by \ac{ec} methods is developed with the objective of enhancing the quality of reconstructed video at decoders receiving erroneous streams from the transmission network. The proposed approach combines error resilience tools at the encoder-side with \ac{ec} at the decoder-side.
|
||||
Different approaches were proposed in the past to address the problem of \ac{ec} at the encoder-side \cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ErrorResilienceModeHEVC2016}. Moreover, extra information was used to aid the \ac{ec} operation at the decoder-side, either by sending redundant data \cite{ResilienceRECAP_ICASSP2009,SaliencyErrorConcealment_TM2013} or by embedding data into the bitstream \cite{ConcealmentIntraDataHidng2009}. Although the above methods facilitate the \ac{ec} performance either through slice reordering or based on \acs{mv}, they always rely on the assumption that a fixed \ac{ec} method is used at the decoder.
|
||||
% The work presented in \cite{ConcealmentSignalling2008,SignallingRobustScalable2015} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder. Although such approaches achieves reasonably good performance, it relies on the spatial neighbouring information or multi-layer video, which may not be available in all cases.
|
||||
|
||||
In this chapter a robust coding driven by \ac{ec} methods with the objective of enhancing the quality of reconstructed video at decoders receiving erroneous streams from the transmission network. The proposed approach combines error resilience tools at the encoder-side with \ac{ec} at the decoder-side. Different approaches were proposed in the past to that address the problem of \ac{ec} at the encoder-side \cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ErrorResilienceModeHEVC2016}. Moreover, extra information was used to aid the \ac{ec} operation at the decoder-side, either by sending redundant data \cite{ResilienceRECAP_ICASSP2009,SaliencyErrorConcealment_TM2013} or by embedding data into the bitstream \cite{ConcealmentIntraDataHidng2009}. Although the above methods facilitate the EC performance either through slice reordering or based on \acs{mv}, they always rely on the assumption that a fixed \ac{ec} method is used at the decoder. The work presented in \cite{ConcealmentSignalling2008,SignallingRobustScalable2015} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder. Although such approaches achieves reasonably good performance, it relies on the spatial neighbouring information or multi-layer video, which may not be available in all cases.
|
||||
|
||||
To advance one step further, in this chapter an EC-aware encoding scheme is proposed to achieve higher flexibility in terms of the error concealment approaches which leads to higher recovered video quality.
|
||||
To advance one step further, in this chapter an concealment-aware encoding scheme is proposed to achieve higher flexibility in terms of the \ac{ec} approaches, leading to higher recovered video quality.
|
||||
This is accomplished by selecting at the encoder, among several \ac{ec} algorithms with different characteristics, the one that yields the lowest reconstruction distortion at the decoder and also minimises the \ac{rd} cost considering the necessary signalling rate. Thus, the proposed approach outperforms similar previous methods \cite{ConcealmentAwareME2006,ErrorResilienceModeHEVC2016} by selecting the best error concealment method without affecting the encoder optimisation.
|
||||
To further improve the quality of recovered frames, an enhancement layer is encoded with the mismatch residue between the transmitted and recovered frames. This is a dynamic optimisation process, which is more efficient than traditional approaches based on redundant pictures \cite{RedudantPicsJan2009,VPxResiliencePCS2016}, which transmit the whole frame as redundancy.
|
||||
In contrast with previous work presented in Chapter~\ref{Chapter:ErrorResilienceTechniques} that addressed the error resilience problem by optimising the coding dependencies, the approach followed in this work considers both \ac{ec} performance and the rate required for signalling the best \ac{ec} mode to the decoder.
|
||||
@@ -54,100 +55,106 @@ In contrast with previous work presented in Chapter~\ref{Chapter:ErrorResilience
|
||||
\section{Traditional error concealment algorithms}
|
||||
\label{Section5:ECCandidates}
|
||||
|
||||
In order to proposed a novel error concealment-aware technique different algorithms were taken into consideration. This section provides a description and evaluation of the error concealment candidates used in the proposed method. Firstly, a detailed description of each approach is provided followed by a performance comparison between each method.
|
||||
In order to proposed a novel error concealment-aware technique different algorithms were taken into consideration. This section provides a description and an evaluation of the error concealment candidates used in the proposed method.
|
||||
Firstly, a detailed description of each approach is provided followed by a performance comparison between each method.
|
||||
|
||||
\subsection{Motion-compensation algorithms}
|
||||
%\subsection{Motion-compensated reconstruction}
|
||||
|
||||
The most common algorithms to recover missing frames used the correctly received \acs{mv} from the neighbouring frames. This has been widely used in the past for \ac{h264} standard \cite{H264RefConc2006,BlockBasedMVE2002,BlockBasedMVEOverllaped2011} and also in the \ac{hevc} standard~\cite{HEVCMCBlockAug2013,HEVCMCResidueDec2013,DownhillHEVCCon_ICCE2017}.
|
||||
The most common algorithms to recover missing frames takes advantage of the correctly received \acs{mv} from the neighbouring frames.
|
||||
This has been widely used in the past for \ac{h264} standard \cite{H264RefConc2006,BlockBasedMVE2002,BlockBasedMVEOverllaped2011} and also in the \ac{hevc} standard~\cite{HEVCMCBlockAug2013,HEVCMCResidueDec2013,DownhillHEVCCon_ICCE2017}.
|
||||
|
||||
Let's consider the example of Figure~\ref{FIG5:MotionVectorExtrapolation}, where a missing frame ($f_0$) at time instance $t_0$ is reconstructed based the information present in the neighbouring frames, $f_{-1}$ and $f_{-2}$.
|
||||
One of the common approaches, i.e., motion-copy based algorithms reconstructs the missing frame through motion compensation directly using the co-located vectors in the closest neighbour frame ($f_{-1}$). The \acs{mv} for the motion compensation ($v_0$) are obtained by copying the \acs{mv} $v_{-1}$ from the same position in $f_{-1}$, as shown by the example (1) in the figure.
|
||||
|
||||
\begin{figure}[b]
|
||||
\centering
|
||||
\input{\ConcealmentAwareDir/Figures/MotionVectorExtrapolation.tikz}
|
||||
\caption{Representation of the video frame with associated original and extrapolated motion vectors represented at full and dashed arrows, respectively.}
|
||||
\label{FIG5:MotionVectorExtrapolation}
|
||||
\end{figure}
|
||||
|
||||
An alternative approach reconstructs the missing frame by using motion compensation through the correctly received \acs{mv}, of frame $f_{-1}$ at time instant $t_{-1}$, according to the following procedure:
|
||||
The \acs{mv} ($v_{-1}$) of the neighbouring frame ($f_{-1}$) are extrapolated to compute the set of \acs{mv} ($v_{0}$) for the missing frame as follows:
|
||||
\begin{enumerate}
|
||||
\item
|
||||
the \acs{mv} ($v_{-1}$) of the neighbouring frame ($f_{-1}$) are extrapolated to compute the set of \acs{mv} ($v_{0}$) for the missing frame as follows:
|
||||
\begin{equation}
|
||||
% v^\zeta_0 = \frac{ t_0 - t_{C} }{ t_{-1} - t_{R} } \times v^\zeta_{-1},\ \zeta=s,r
|
||||
v_0 = \frac{ t_0 - t_{-1} }{ t_{-1} - t_{R} } \times v_{-1},
|
||||
\end{equation}
|
||||
where $t_{R}$ is the time instant of the reference frame pointed by the original \acs{mv} $v_{-1}$. Then, the \ac{mv} associated to the block $b_{-1}(x,y)$ at spatial position $(x,y)$ in $f_{-1}$ may be used to recover the block $b^{'}_{{0}}(x^{'},y^{'})$ in $f_0$, at position $(x^{'},y^{'})$ obtained from $v_0$ components as follows:
|
||||
where $t_{R}$ is the time instant of the reference frame pointed by the original \acs{mv} $v_{-1}$.
|
||||
\item
|
||||
the \ac{mv} associated to the block $b_{-1}(x,y)$ at spatial position $(x,y)$ in $f_{-1}$ may be used to recover the block $b^{'}_{{0}}(x^{'},y^{'})$ in $f_0$, at position $(x^{'},y^{'})$ obtained from $v_0$ components as follows:
|
||||
\begin{align}
|
||||
x^{'} &= x - v_{0x} \\
|
||||
y^{'} &= y - v_{0y}
|
||||
x^{'} &= x - v_{0x}, \\
|
||||
y^{'} &= y - v_{0y}.
|
||||
\end{align}
|
||||
\end{enumerate}
|
||||
This is illustrated in Figure~\ref{FIG5:MotionVectorExtrapolation} by the example (2). This method is referred to as \glsfirst{mve}.
|
||||
|
||||
Finally, as a base for comparison the default \ac{ec} implemented in the reference \ac{hevc} implementation the \ac{fc} algorithm is also evaluated. This method consists in replacing the lost frame with a copy of the previous decoded one, which can also be described as a motion compensated reconstruction using only zero \acs{mv}.
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
\input{\ConcealmentAwareDir/Figures/MotionVectorExtrapolation.tikz}
|
||||
\caption{Representation of the video frame with associated original and extrapolated motion vectors represented using full and dashed arrows, respectively.}
|
||||
\label{FIG5:MotionVectorExtrapolation}
|
||||
\end{figure}
|
||||
|
||||
Finally, as a base for comparison the default \ac{ec} in the reference \ac{hevc} implementation the \ac{fc} algorithm is also evaluated. This method consists in replacing the lost frame with a copy of the previous decoded one, which can also be described as a motion compensated reconstruction using only zero \acs{mv}.
|
||||
|
||||
\subsection{Algorithms based on motion estimation}
|
||||
|
||||
In this sub-section describe different method that can be used to estimate a motion-field, which can be used to reconstruct the missing information. The obtained motion-field is used in combination with the \ac{mve} method to recover the lost frame.
|
||||
In this sub-section different method that can be used to estimate a motion-field are described, which can be used to reconstruct the missing motion information.
|
||||
The obtained motion-field is used in combination with the \ac{mve} algorithm to recover the lost frame.
|
||||
|
||||
Optical flow estimation algorithms are used to estimate a 2-D \ac{mv}, $v=(v_x, v_y)$ for each pixel, such as, the following energy is minimised:
|
||||
Typically, optical flow estimation algorithms are used to estimate a 2-D \ac{mv} field, $v=(v_x, v_y)$, for each pixel, such as the following energy function is minimised:
|
||||
\begin{equation}
|
||||
E(x,y) = \sum_{x} \sum_{y} \bigg| f_{0}\big(x,y\big) - f_{1}\big( x + v_x(x,y), y+v_y(x,y) \big) \bigg|^2,
|
||||
E(x,y) = \sum_{x} \sum_{y} \bigg| f_{0}\big(x,y\big) - f_{1}\big( x + v_x(x,y), y+v_y(x,y) \big) \bigg|^2.
|
||||
\end{equation}
|
||||
resulting an unique motion field given by $v$.
|
||||
Besides using a full search algorithm where all block-based approach is used to search for the best \acs{mv} based on the sum of absolute errors, in this work three different approaches were tested to evaluate they effectiveness to be used in \ac{ec} algorithms, namely:
|
||||
Besides using a full search algorithm where a block-based approach is used to search for the best \acs{mv} based on the sum of absolute errors, in this work three different approaches were tested to evaluate their effectiveness to be used in \ac{ec} algorithms (described below), namely:
|
||||
\begin{enumerate}
|
||||
\item a method based on the total variation based on the minimization of the \mbox{L-1} norm~\cite{DualFlow2016};
|
||||
\item a method which approximates the neighbouring information of a given pixel by quadratic polynomials on both frames~\cite{PolyFlow2003};
|
||||
\item a method based on a combination of descriptor matching and large displacement estimation~\cite{DeepFlow2013}.
|
||||
\end{enumerate}
|
||||
|
||||
|
||||
\subsubsection{Motion estimation based on total variation and regularisation (\textit{DualFlow})}
|
||||
|
||||
In this method the motion field is estimated based on the minimization of the total variation obtained using the L-1 norm \cite{DualFlow2007,DualFlow2016} and taking into account the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}. This condition forces the estimated field to be regular across neighbouring pixels by penalising deviations in a quadratic way.
|
||||
In this method the motion field is estimated based on the minimization of the total variation obtained using the L-1 norm \cite{DualFlow2007,DualFlow2016}, taking into account the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}. This condition forces the estimated field to be regular across neighbouring pixels by penalising deviations in a quadratic way.
|
||||
Using this method, for each pixel $\mathbf{p}$, the motion field $v(\mathbf{p})$ is obtained by minimising the following energy function,
|
||||
\begin{equation}\label{EQ5:DualFlow}
|
||||
E(v) =
|
||||
\sum_{\mathbf{p} \in \Omega} \Big( \big| f_{-1}(\mathbf{p}) - f_{-2}( \mathbf{p} + v(\mathbf{p}))\big| + \big|\nabla v(\mathbf{p})\big| \Big)%d\mathbf{p},\ \mathbf{p} \in \Omega
|
||||
\end{equation}
|
||||
where $f_{-1}$ and $f_{-2}$ are the two previous frames at time instances $t_{-1}$ and $t_{-2}$, as represented in Figure~\ref{FIG5:MotionVectorExtrapolation}. In the equation, $\Omega$ defines the sets of pixels of those frames.
|
||||
$|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variation of the optical flow which effectively disallow discontinuities. This is based on the assumption that neighbouring regions belong to the same object, thus such regions should have similar motion.
|
||||
$|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of the optical flow which effectively disallow discontinuities. This is based on the assumption that neighbouring regions belong to the same object, thus such regions should have similar motion.
|
||||
|
||||
|
||||
\subsubsection{Motion estimation based on polynomial expansion (\textit{PolyFlow})}
|
||||
|
||||
In this method the motion field is estimated based on a polynomial expansion a neighbourhood of a pixel~\cite{PolyFlow2003}.
|
||||
In this method only quadratic polynomials are considered, which gives the local signal model, expressed in a local coordinate system as
|
||||
In this method only quadratic polynomials are considered, expressed as:
|
||||
\begin{equation}\label{EQ5:PolyFlow}
|
||||
f(\mathbf{x}) \sim \mathbf{x}^TA_1\mathbf{x}+\mathbf{b}_1^T\mathbf{x}+C_1
|
||||
f(\mathbf{p}) \sim \mathbf{p}^TA_1\mathbf{p}+\mathbf{b}_1^T\mathbf{p}+C_1
|
||||
\end{equation}
|
||||
where $\mathbf{A}$ is a symmetric matrix, $\mathbf{B}$ a vector and $C$ a scalar. The coefficients are
|
||||
where $\mathbf{p} = (x,y)^T$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ a vector and $C_1$ a scalar. The coefficients are
|
||||
estimated from a weighted least squares fit to the signal values in the neighbourhood.
|
||||
|
||||
Since the result of polynomial expansion is that each neighbourhood is approximated by a polynomial, we start by analysing what happens if a polynomial undergoes an ideal translation. Let's consider the signal $f_1(\mathbf{x})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$ given by $\mathbf{d}$. This can be expressed by the following:
|
||||
Since the result of polynomial expansion is that each neighbourhood is approximated by a polynomial, we start by analysing what happens if a polynomial undergoes an ideal translation. Let's consider the signal $f_1(\mathbf{p})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$ given by $\mathbf{d}=(v_x,v_y)^T$. This can be expressed by the following:
|
||||
\begin{align}
|
||||
f_2(\mathbf{x})
|
||||
&= f_1(\mathbf{x}+\mathbf{d}) \nonumber \\
|
||||
&= (\mathbf{x}+\mathbf{d})^TA_1(\mathbf{x}+\mathbf{d})+\mathbf{b}_1^T(\mathbf{x}+\mathbf{d})+C_1 \\
|
||||
&= \mathbf{x}^TA_2\mathbf{x}+\mathbf{b}_2^T\mathbf{x}+C_2, \nonumber
|
||||
f_2(\mathbf{p})
|
||||
&= f_1(\mathbf{p}+\mathbf{d}) \nonumber \\
|
||||
&= (\mathbf{p}+\mathbf{d})^TA_1(\mathbf{p}+\mathbf{d})+\mathbf{b}_1^T(\mathbf{p}+\mathbf{d})+C_1 \\
|
||||
&= \mathbf{p}^TA_2\mathbf{p}+\mathbf{b}_2^T\mathbf{p}+C_2, \nonumber
|
||||
\end{align}
|
||||
which results in the following relations,
|
||||
\begin{align}
|
||||
\mathbf{A}_2 &= A_1 \nonumber \\
|
||||
\mathbf{b}_2 &= \mathbf{b} + 2 \mathbf{A}_1\mathbf{d}, \\
|
||||
\mathbf{A}_2 &= \mathbf{A}_1 \nonumber \\
|
||||
\mathbf{b}_2 &= \mathbf{b}_1 + 2 \mathbf{A}_1\mathbf{d}, \\
|
||||
C_2 &= \mathbf{d}^T\mathbf{A}_1\mathbf{d}+\mathbf{b}_1\mathbf{d}+C_1. \nonumber
|
||||
\end{align}
|
||||
This allows us to find the displacement $\mathbf{d}$ for any signal dimensions by solving the above equations. Detailed implementations of the polynomial expansion and solving can be found in~\cite{NormalizedDiffConvolution1993,farneback2002polynomial}.
|
||||
This allows to find the displacement $\mathbf{d}$ for any signal dimensions by solving the above equations. Detailed implementations of the polynomial expansion and solving can be found in~\cite{NormalizedDiffConvolution1993,farneback2002polynomial}.
|
||||
|
||||
\subsubsection{Large displacement optical flow with deep matching (\textit{DeepFlow})}
|
||||
|
||||
In this method the motion field is estimated based on the large displacement optical flow presented in \cite{LargeDisplacementOptFlow_TPAMI2011}, which extends the total variation methods, as the one expressed by Equation (\ref{EQ5:DualFlow}), including descriptor matching. The correspondences between descriptions are used to apply a coarse-to-fine wrapping, thus avoiding local minimum values.
|
||||
The matching algorithm is built upon a multi-stage architecture with six layers with different patches sizes, interleaving convolutions and max-pooling \cite{DeepFlow2013}. The combination of both approaches leads towards a better matching.
|
||||
In this method the motion field is estimated based on the large displacement optical flow presented in \cite{LargeDisplacementOptFlow_TPAMI2011}, which extends the total variation methods, e.g. the one expressed by Equation (\ref{EQ5:DualFlow}), by including descriptor matching.
|
||||
The correspondences between descriptions are used to apply a coarse-to-fine wrapping, thus avoiding local minimum values. The matching algorithm is built upon a multi-stage architecture with six layers with different patches sizes, interleaving convolutions and max-pooling \cite{DeepFlow2013}. The combination of both approaches increases the matching accuracy.
|
||||
|
||||
In \cite{DeepFlow2013} to improve the estimated motion field accuracy, the following extension to the method proposed in~\cite{LargeDisplacementOptFlow_TPAMI2011} were introduced: (i) description matching was used in the minimisation problem; (ii) a normalisation parameter was introduced to down-height the impact of regions with high spatial changes; (iii) different weights are used for each layers of optimisation. Figure~\ref{FIG5:DeepFlowDiagram} shows the blocks that are included in the estimation process. As shown in the figure a multi-size optimisation is applied by using different patches sizes. This is used to estimate an initial motion field, referred to as $v'$ .
|
||||
In \cite{DeepFlow2013} to improve the estimated motion field accuracy, the following extension to the method proposed in~\cite{LargeDisplacementOptFlow_TPAMI2011} were introduced: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layers of optimisation. Figure~\ref{FIG5:DeepFlowDiagram} shows the blocks that are included in the estimation process. As shown in the figure a multi-size optimisation is applied by using different patches sizes. This is used to estimate an initial motion field, referred to as $v'$ .
|
||||
|
||||
\begin{figure}[t]
|
||||
\begin{figure}[b]
|
||||
\centering
|
||||
\includegraphics[width=.95\columnwidth]{\ConcealmentAwareDir/Figures/DeepFlowBlockDiagram}
|
||||
\caption{Block diagram of the deep matching algorithm \cite{DeepFlow2013}.}
|
||||
@@ -155,7 +162,7 @@ In \cite{DeepFlow2013} to improve the estimated motion field accuracy, the follo
|
||||
\FIGSVSPACE
|
||||
\end{figure}
|
||||
|
||||
Subsequently, the optical flow is obtained by minimising an energy function, which can be described by the following:
|
||||
Subsequently, the final optical flow is obtained by minimising an energy function, which can be described by the following:
|
||||
\begin{equation}
|
||||
E(v) =
|
||||
\sum_{\mathbf{p} \in \Omega} \Big( E_D(\mathbf{p}) + \alpha E_S(\mathbf{p}) + \beta E_M(\mathbf{p}) \Big),
|
||||
@@ -167,10 +174,10 @@ The smoothness term is a robust penalisation of the gradient of the flow as used
|
||||
|
||||
\subsection{Experimental evaluation}
|
||||
|
||||
In this section a performance evaluation of the different \ac{ec} schemes previously described is carried out. This evaluation aims at finding the suitable methods to be used for efficiently reconstruct the missing frames in case of data loss. In this evaluation the default \ac{ec} method of the reference \ac{hevc} implementation is used as a reference for comparison. Then, two method based on the correctly received \acs{mv} are tested, i.e., \ac{mc} and \ac{mve}. Moreover, the four different approaches based on a estimated motion field are compared: (i) full search based on the minimisation of the sum of absolute errors (\FULLSEARCH), (ii) optical flow based a duality between total variation and smoothness term (\DUALFLOW), (iii) optical flow based on polynomial approximations (\POLYFLOW) and (iv) optical flow based on deep matching (\DEEPFLOW).
|
||||
In this section a performance evaluation of the different \ac{ec} schemes previously described is carried out. This evaluation aims at finding the suitable methods to be used for efficiently reconstruct the missing frames in case of data loss. In this evaluation the \ac{fc} method is used as reference for comparison. Then, two methods based on the correctly received \acs{mv} are tested, i.e., \ac{mc} and \ac{mve}. Moreover, the four different approaches based on a estimated motion field are compared: (i) full search based on the minimisation of the sum of absolute errors (\FULLSEARCH), (ii) optical flow based a duality between total variation and smoothness term (\DUALFLOW), (iii) optical flow based on polynomial approximations (\POLYFLOW) and (iv) optical flow based on deep matching (\DEEPFLOW).
|
||||
|
||||
Table~\ref{TAB5:ECPerformaceComparison} shows the average quality results obtained using the \ac{psnr} metric for different methods. The absolute value is shown for the frame-copy method (\FC), and differential values are shown for the remaining methods.
|
||||
Results reveal that the methods \MC and \MVE are able to overcome \FC which does not use any \acs{mv}, achieving average quality gains of up to 1.41 dB and 1.46 dB, respectively. Moreover, by extrapolating the \acs{mv} from the correctly receiving frame to the lost time instance (\MVE) is able to outperform simple motion-copy (\MC).
|
||||
Table~\ref{TAB5:ECPerformaceComparison} shows the average quality results obtained using the \ac{psnr} metric for different methods. The absolute value is shown for the \FC method, and differential values are shown for the remaining methods.
|
||||
Results reveal that the methods \MC and \MVE are able to overcome \FC which does not use any \acs{mv}, achieving average quality gains of up to 1.41 dB and 1.46 dB, respectively. Moreover, by extrapolating the \acs{mv} from the correctly receiving frame to the lost time instance (\MVE) is able to outperform straightforward motion-copy (\MC).
|
||||
|
||||
\begin{table}[tb]
|
||||
\centering
|
||||
@@ -180,12 +187,15 @@ Results reveal that the methods \MC and \MVE are able to overcome \FC which does
|
||||
\input{\ConcealmentAwareDir/Results/PerformanceConcealmentMethods.tex}
|
||||
\end{table}
|
||||
|
||||
Results for \ac{ec} methods based motion estimation (see columns 5 to 8 of Table~\ref{TAB5:ECPerformaceComparison}) reveal a overall superior performance when comparing with methods which rely of correctly received \acs{mv}.
|
||||
Results for \FULLSEARCH method reveals that only minimising the sum of absolute errors between two consecutive frames might not always be directly related with objects' motions. Thus, this may results in a inaccurate motion estimation to be used for missing frames reconstruction. Finally, results from Table~\ref{TAB5:ECPerformaceComparison} reveal that the \DEEPFLOW algorithm leads to the most accurate motion field to be used for \ac{ec}, in comparison with the other methods. It is able to achieve an average quality gain of 1.79 dB, and up to 3.39 dB for Book Arrival sequence, where \POLYFLOW only achieves 3.05 dB.
|
||||
Results for \ac{ec} methods based on motion estimation (see columns 5 to 8 of Table~\ref{TAB5:ECPerformaceComparison}) reveal an overall superior performance when comparing with methods which rely of correctly received \acs{mv}.
|
||||
Results for \FULLSEARCH method reveals that only minimising the sum of absolute errors between two consecutive frames might not always be directly related with objects' motions. Thus, this results in an inaccurate motion estimation to be used for the reconstruction of lost frames. Finally, results from Table~\ref{TAB5:ECPerformaceComparison} reveal that the \DEEPFLOW algorithm leads to the most accurate motion field to be used for \ac{ec}, in comparison with the other methods. It is able to achieve an average quality gain of 1.79 dB, and up to 3.39 dB for Book Arrival sequence, where \POLYFLOW only achieves 3.05 dB.
|
||||
|
||||
\subsection{Discussion and conclusions}
|
||||
\subsection{Conclusions}
|
||||
|
||||
Summarising, the analysis of the previous results indicates that all methods cope with lost frames and manage to recover the missing data with acceptable quality. Nevertheless, it is noticeable that the recovering lost frames with an estimated motion field leads to higher quality, especially using the \DEEPFLOW method. However, as the other methods tested are also able to achieve acceptable video quality they shall not be neglected. Thus, in order to achieve higher reconstruction quality different approaches shall be considered.
|
||||
Summarising, the analysis of the previous results indicates that all methods cope with lost frames and manage to recover the missing data with acceptable quality. Nevertheless, it is noticeable that the recovering lost frames with an estimated motion field leads to higher quality, especially using the \DEEPFLOW method. However, as the other methods tested are also able to achieve acceptable video quality they shall not be neglected, as they provide alternative approaches for lost frame recovery with lower computational complexity (i.e., they not require motion estimation).
|
||||
Thus, in order to achieve higher reconstruction quality different approaches shall be considered, aiming at balancing the overall complexity with the recovery quality.
|
||||
|
||||
%\HERE
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
%% SECTION %%
|
||||
@@ -216,7 +226,7 @@ For the \ac{ec} mode decision optimization the missing slices are reconstructed,
|
||||
In this process, each \ac{ctu} is partitioned to form a quadtree structure of \ac{ecu} that are used to test the different \ac{ec} candidate modes.
|
||||
Then, the best trade-off between \ac{ecu} partition and signalling is found through \ac{rd} optimisation.
|
||||
In order to achieve an efficient use of the signalling overhead a saliency-weighted procedure is embedded in the \ac{rd} optimisation based on a spatio-temporal saliency, where more signalling bits are allocated to regions with higher saliency values.
|
||||
Finally, the signalling information is transmitted to the decoder by using the \ac{sei} \ac{nalu} type~\cite{HEVCHighLevelReferenceDez2012}.
|
||||
Finally, the signalling information is transmitted to the decoder by using the \ac{sei} \acs{nalu} type~\cite{HEVCHighLevelReferenceDez2012}.
|
||||
Since in general, the optimal \ac{ec} mode is still not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el}.
|
||||
This residual information contains the difference between the compressed ($\hat f_t$) and the \ac{ec} frames ($\tilde f$).
|
||||
|
||||
@@ -255,7 +265,7 @@ A set of four \ac{ec} candidate modes (M1...M4) is used in the optimisation pro
|
||||
\FIGSVSPACE
|
||||
\end{figure}
|
||||
|
||||
The first two methods M1, M2 rely on an estimated motion field, using the method \DEEPFLOW \cite{DeepFlow2013}, to recover the lost frame. Then, it is extrapolated at the pixel level, i.e., one \ac{mv} is assigned to each pixel, and \ac{cu} level, i.e., one \ac{mv} is assigned to the entire block, at the missing frame instant. This results in two different reconstructions of the missing frame, where the former may increase the accuracy of the pixel extrapolation, but the latter has the advantage of keeping the block structure and generate a smoother reconstruction.
|
||||
The first two methods M1, M2 rely on an estimated motion field, using the method \DEEPFLOW \cite{DeepFlow2013}, to recover the lost frame. Then, it is extrapolated at the pixel level, i.e., one \ac{mv} is assigned to each pixel, and \ac{pu} level, i.e., one \ac{mv} is assigned to the entire block, at the missing frame instant. This results in two different reconstructions of the missing frame, where the former may increase the accuracy of the pixel extrapolation, but the latter has the advantage of keeping the block structure and generate a smoother reconstruction.
|
||||
|
||||
The method M3 is based on a traditional \glsfirst{mve}, and finally M4 reconstructs the missing frames using a \glsfirst{mc} algorithm.
|
||||
|
||||
@@ -263,7 +273,7 @@ The method M3 is based on a traditional \glsfirst{mve}, and finally M4 reconstru
|
||||
\subsection{Selection optimization}
|
||||
|
||||
As described above, in the sub-section~\ref{Section5:ECCandidates}, the proposed method includes four \ac{ec} candidates, which result in different recovered frames for each lost frame (simulated).
|
||||
The lost frame is partitioned into quadtree structures of \ac{ecu}, each one forming a set of variable size \acs{ecu}.
|
||||
The lost frame is partitioned into quadtree structures, each one forming a set of variable size \acs{ecu}.
|
||||
Then, finding the optimal \ac{ec} method for each \ac{ecu} is formulated as a rate-distortion cost minimisation. To solve the aforementioned problem, a Lagrangian optimisation is used~\cite{LagragianMultiplier1982} to achieve the best trade-off between reconstruction quality and signalling overhead.
|
||||
Furthermore, image saliency driven by visual attention modelling is used in the optimisation process to weight the cost of the overhead information of each \ac{ecu}, according to the visual relevance of its content.
|
||||
|
||||
@@ -387,11 +397,16 @@ Note that the generated bitstream is compliant with the standard, as the error c
|
||||
\section{Error concealment signalling evaluation}
|
||||
\label{Section5:ECSignallingPerformance}
|
||||
|
||||
This section presents the performance of the proposed \ac{ec}-aware optimisation scheme using the \ac{hevc} standard as the basis of the encoding framework.
|
||||
Two different evaluations are presented. In the former, only robust \ac{ec} of intra-frames is evaluated, while in the latter the proposed \ac{ec}-aware optimisation is applied to all frames corresponding to a generic robust video transmission using optimal \ac{ec}.
|
||||
|
||||
|
||||
\subsection{Intra-frame EC optimisation}
|
||||
\label{SEC4:IntraPerformance}
|
||||
|
||||
The evaluation of the proposed method is performed only for intra frames to study their impact on error propagation. Moreover, as intra coding units result in higher amount of bits, they also need a higher number of packets.
|
||||
Results using this setup indicate that for an intra-period of 16 frames, approximately 42\% of the packets corresponds to intra coded slices. Also, by limiting the proposed method to intra frames, the amount of required overhead is reduced.
|
||||
As mention before, in this section the evaluation of the proposed method is performed only for intra frames to study their impact on error propagation. As intra coding units result in higher amount of bits, they also need a higher number of packets, making them more prone to transmission errors and losses.
|
||||
Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets corresponds to intra coded slices.
|
||||
%Also, by limiting the proposed method to intra frames, the amount of required overhead is reduced.
|
||||
|
||||
% \begin{table}[tb]
|
||||
% \newcommand{\somespaces}{}
|
||||
@@ -528,7 +543,7 @@ The results confirm the superiority of the proposed method, outperforming the re
|
||||
It is also worthwhile to notice that the \ac{psnr} variance obtained with the \PROP method is lower, which indicates lower quality variations in the video segments affected by errors.
|
||||
|
||||
%The quality gains achieved by the proposed method are similar to those achieved by previous works~\cite{ConcealmentIntraDataHidng2009,ConcealmentSignalling2008} for the same \acs{plr}. However, these require a higher amount of overhead (approximately $3\%$), thus the proposed method achieves a better trade-off between quality and overhead.
|
||||
In summary, the proposed technique is able to enhance the performance of the standard decoder by finding the best error concealment method at the encoder and sending such information with a small amount of signalling overhead.
|
||||
In summary, the proposed technique is able to enhance the performance of the standard decoding of intra coded frames by finding the best error concealment method at the encoder and sending such information with a small amount of signalling overhead.
|
||||
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
@@ -538,7 +553,7 @@ In summary, the proposed technique is able to enhance the performance of the sta
|
||||
% \subsection{Overall performance evaluation}
|
||||
\label{SEC4:OverallPerformance}
|
||||
|
||||
The performance of the proposed \ac{ec}-aware optimisation scheme is evaluated using the \ac{hevc} standard as the basis of the encoding framework. This section is organised in the following subsections:
|
||||
In this section the proposed \ac{ec}-aware optimisation scheme is evaluated for a generalised robust video transmission of intra and inter coded frames. This section is organised in the following subsections:
|
||||
\begin{itemize}
|
||||
\item evaluation of the usage distribution of each \ac{ec} mode;
|
||||
\item performance evaluation of the \ac{ec} optimal mode decision (without using the enhancement layer);
|
||||
@@ -584,12 +599,12 @@ The results in Table \ref{tab:useratio} show that all \ac{ec} modes are effectiv
|
||||
The \ac{ec} candidate methods based on the estimated motion field (i.e., M1, M2) are selected more often because they provide more accurate representation of the scene's motion than the others (i.e., M3, M4).
|
||||
|
||||
|
||||
\subsubsection{Performance evaluation of optimal \ac{ec} mode decision}
|
||||
\subsubsection{Evaluation of optimal \ac{ec} mode decision}
|
||||
\label{sec:ECModeSelectionPerformance}
|
||||
|
||||
\input{\ConcealmentAwareDir/Results/TableSignallingPerformance.tex}
|
||||
|
||||
The performance of the proposed saliency-weighted \ac{ec} mode decision (\DYNLAMBDA) was evaluated under loss events and compared with the reference methods. The video quality was evaluated by using two quality metrics: \ac{psnr} and a \ac{fsim}~\cite{FSIM_TIP2011} in order to validate the performance of proposed method in a consolidated manner.
|
||||
The evaluation of the proposed saliency-weighted \ac{ec} mode decision (\DYNLAMBDA) was carried out under loss events and compared with the reference methods. The video quality was evaluated by using two quality metrics: \ac{psnr} and a \ac{fsim}~\cite{FSIM_TIP2011} in order to validate the performance of proposed method in a consolidated manner.
|
||||
Realistic conditions were used for transmission, by allowing independent losses for any of type of \acs{nal}, hence there is no guarantee that the decoder conditions are always equal to those at the encoder, which means that a signalling mismatch may occur.
|
||||
The results obtained without saliency weighting are also shown for comparison (\FIXEDLAMBDA). In this case a constant $\lambda_0=10$ is used.
|
||||
|
||||
@@ -606,7 +621,7 @@ As shown in the Table, the saliency-weighted optimisation reduces the amount of
|
||||
These results show that using an adaptive value of $\lambda$ based on the saliency values leads to an average overhead reduction of 28\% in comparison with the case where no saliency is used, i.e., constant $\lambda$.
|
||||
|
||||
It is worth to note that using the saliency-weighted optimisation to reduce the overhead does not compromise the video quality, as can be seen in the results shown in Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM}. The \ac{psnr} values for \FIXEDLAMBDA and \DYNLAMBDA are very close, but in overhead savings are significant for most sequences.
|
||||
The achieved WPSNR and \ac{fsim} results further demonstrate the effectiveness of the saliency-based optimisation, because the lower overhead does not affect the most relevant image regions, i.e., those with higher saliency values. Thus, the proposed method using a adaptive $\lambda$ (i.e., using saliency) is able to not only achieve higher \ac{ec} quality than the reference methods, but also higher quality gains than using a constant $\lambda$ (i.e., without saliency).
|
||||
The achieved \ac{wpsnr} and \ac{fsim} results further demonstrate the effectiveness of the saliency-based optimisation, because the lower overhead does not affect the most relevant image regions, i.e., those with higher saliency values. Thus, the proposed method using a adaptive $\lambda$ (i.e., using saliency) is able to not only achieve higher \ac{ec} quality than the reference methods, but also higher quality gains than using a constant $\lambda$ (i.e., without saliency).
|
||||
This confirms the two-fold superiority of adaptive $\lambda$, i.e. not only the total overhead is reduced but also higher quality gains are achieved.
|
||||
|
||||
|
||||
@@ -653,6 +668,7 @@ As in the previous simulations, the packets were randomly discarded to simulate
|
||||
The \ac{psnr} obtained with the proposed method is compared with the default method used by reference \ac{hevc} , i.e., frame-copy (\REF), motion-copy (\MC)~\cite{HEVCMCResidueDec2013}, and the method proposed in \cite{ROPERefGenJan2010}, which implements a robust scheme to minimise error propagation.
|
||||
The following results were obtained using the same constant bitrate for each test condition.
|
||||
|
||||
\subsection{Quality evaluation}
|
||||
|
||||
\input{\ConcealmentAwareDir/Results/TableOveralPerformance.tex}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user