jfmcarreira/doc_phd_thesis
1
0 Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Review thesis after PA comments

Browse Source
This commit is contained in:
jfmcarreira committed 2018-09-17 23:11:37 +01:00
1 parent 2a97ab6c09
commit 9bdc2ee860
24 files changed
+798 -749
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex
+86 -69
View File
@@ -4,7 +4,7 @@
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CHAPTER
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Error Concealment-Aware Video Encoding \VERSION{2}}
\chapter[Error concealment-aware video encoding]{Error concealment-aware video \\ encoding \VERSION{2}}
\label{Chapter:ConcealmentAwareTechniques}
%\vspace{-6mm}
%\clearpage
@@ -14,17 +14,22 @@
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
\glsreset{rd}
\glsreset{mv}
\glsreset{ec}
\glsreset{cu}
\glsreset{ld}
\glsreset{ra}
\glsreset{ra}
\glsreset{ld}
%\glsreset{rd}
In this chapter an error concealment-aware encoding scheme is proposed to improve the quality of decoded video in broadcast environments prone to transmission errors and data loss.
The method aims to enhance the reconstruction of recovered lost frames by optimising the efficiency of the \ac{ec}.
The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled through \ac{sei} messages. Such optimal \ac{ec} modes are found by simulating transmission losses followed by a Lagrangian optimisation of the \textit{signalling rate - \ac{ec} distortion} cost. A generalised saliency-weighted distortion is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. When data loss occurs the decoder uses the signalling information to improve the reconstruction quality.
This chapter is organized as follows. Section~\ref{Section5:Problem} introduces the problem definition and motivation for the method proposed in this chapter.
Section~\ref{Section5:ECCandidates} presents an initial evaluation of different error concealment schemes in order to find the relevant ones to be used in the proposed error concealment-aware coding scheme.
Section~\ref{Section5:ECCandidates} presents an initial evaluation of different \ac{ec} schemes in order to find the relevant ones to be used in the proposed \ac{ec}-aware coding scheme.
Section~\ref{Section5:Architecture} provides an overview of the proposed method architecture followed by a detailed description in Section~\ref{Section5:ProposedMethod}.
Sections~\ref{Section5:ECSignallingPerformance}~and~\ref{Section5:OverallPerformance} present the performance results and discussion of the proposed \ac{ec} optimisation and signalling.
Finally, Section~\ref{Section5:Summary} concludes the chapter.
@@ -41,9 +46,10 @@ Finally, Section~\ref{Section5:Summary} concludes the chapter.
%\glsunset{mb}
In Chapter~\ref{Chapter:ErrorResilienceReview} different \ac{ec} algorithms that can effectively reconstruct lost video were described, though not all of them exhibit the same accuracy for every lost frame or slice.
Moreover, it was also shown that combining different \ac{ec} approaches at the \ac{mb}/\ac{pu} level to improve the reconstruction accuracy is not straightforward.
In the past, hybrid \ac{ec} schemes were proposed to improve the recovery quality of lost frames by either combining spatial and temporal \ac{ec} algorithms~\cite{HybridConcTB2008} or by optimising the \ac{mve} at the pixel and block levels~\cite{HybridConcealmentTB2008,2DConcHybridMotion2010}.
Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of motion-based recovery approaches by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}. Although such methods are able to reduce the reconstruction artefacts and improve the overall video quality, decoder-based decisions are still not fully accurate, as they only have limited information to be used as reference in the decision process.
Moreover, it was also shown that combining different \ac{ec} approaches at the block level to improve the reconstruction accuracy is not straightforward.
In the past, hybrid \ac{ec} schemes were proposed to improve the quality of recovered frames by either combining spatial and temporal \ac{ec} algorithms~\cite{HybridConcTB2008} or by optimising the \ac{mve} at the pixel and block levels~\cite{HybridConcealmentTB2008,2DConcHybridMotion2010}.
Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of motion-based recovery approaches by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}.
Although such methods are able to reduce the reconstruction artefacts and improve the overall video quality, decoder-based decisions are still not fully accurate, as they are based on limited information to be used as reference in the decision process.
Therefore, higher reconstructed quality can be achieved by estimating the best \ac{ec} techniques at the encoding-side, based on the original video signal.
%Such approach not only improves quality but also reduces the complexity of the video decoders.
%In this chapter a robust coding driven by \ac{ec} methods is developed with the objective of enhancing the quality of reconstructed video at decoders receiving erroneous streams from the transmission network. The proposed approach combines error resilience tools at the encoder-side with \ac{ec} at the decoder-side.
@@ -52,9 +58,10 @@ Extra information was also used to aid the \ac{ec} operation at the decoder-side
Although these methods facilitate the \ac{ec} performance either through slice reordering or based on \acs{mv}, they always rely on the assumption that some fixed \ac{ec} method is used at the decoder.
% The work presented in \cite{ConcealmentSignalling2008,SignallingRobustScalable2015} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder. Although such approaches achieves reasonably good performance, it relies on the spatial neighbouring information or multi-layer video, which may not be available in all cases.
To advance one step further, this chapter presents an investigation on concealment-aware video coding and proposes a new scheme to achieve high decoding flexibility in terms of the \ac{ec} approaches, leading to better video quality.
This is accomplished by selecting, at the encoder, among several \ac{ec} algorithms with different characteristics, the one that yields the lowest reconstruction distortion at the decoder and also minimises the \ac{rd} cost considering the necessary signalling rate. Thus, the proposed approach differs from similar previous methods \cite{ConcealmentAwareME2006,ErrorResilienceModeHEVC2016} by selecting the best error concealment method without affecting the encoder optimisation.
To further improve the quality of recovered frames, an enhancement layer is encoded with the mismatch residue between the transmitted and recovered frames. This is a dynamic optimisation process, which is more efficient than conventional approaches based on redundant pictures \cite{RedudantPicsJan2009,VPxResiliencePCS2016}, which transmit the whole frame as redundancy.
To advance one step further, this chapter presents an investigation on \ac{ec}-aware video coding and proposes a new scheme to achieve high decoding flexibility in terms of the \ac{ec} approaches, leading to better video quality.
This is accomplished by selecting, at the encoder, among several \ac{ec} algorithms with different characteristics, the one that yields the lowest reconstruction distortion at the decoder and also minimises the \ac{rd} cost considering the necessary signalling rate. Thus, the proposed approach differs from similar previous methods \cite{ConcealmentAwareME2006,ErrorResilienceModeHEVC2016} by selecting the best \ac{ec} method without affecting the encoder optimisation.
To further improve the quality of recovered frames, an enhancement layer is encoded with the mismatch residue computed as a difference between the transmitted and recovered frames.
This is a dynamic optimisation process, which is more efficient than conventional approaches based on redundant pictures \cite{RedudantPicsJan2009,VPxResiliencePCS2016}, which transmit the whole frame as redundancy.
In contrast with the previous work presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}, which addressed the error resilience problem by optimising the coding dependencies, the approach followed in this chapter considers both, \ac{ec} performance and the coded rate required for signalling the best \ac{ec} mode to the decoder.
@@ -65,7 +72,7 @@ In contrast with the previous work presented in Chapter~\ref{Chapter:ErrorResili
\label{Section5:ECCandidates}
In order to propose a novel error concealment-aware technique different algorithms were taken into consideration.
In order to propose a novel \ac{ec}-aware technique different algorithms were taken into consideration.
This section presents a description and evaluation of the most relevant \ac{ec} algorithms used in the proposed method.
Firstly, a technical description of each method is provided followed by a performance comparison between them.
@@ -78,7 +85,7 @@ This has been widely used in the past for \ac{h264} standard \cite{H264RefConc20
Let's consider the example of Figure~\ref{FIG5:MotionVectorExtrapolation}, where a missing frame ($f_0$) at time instant $t_0$ is reconstructed based on the information present in the temporally adjacent frames, $f_{-1}$ and $f_{-2}$.
One of the common approaches is \ac{mc} where the missing frame is reconstructed through motion compensation directly using the co-located vectors in the closest neighbour frame ($f_{-1}$). The \acs{mv} used for the motion compensation ($v_0$) are obtained by copying the \acs{mv} $v_{-1}$ from the same spatial position in $f_{-1}$, as illustrated in the figure by the \ac{mv} $v^{1}_{0}$ which is obtained by copying $v^{1}_{-1}$.
An alternative approach is based on the reconstruction of the missing frame by using motion compensation through a extrapolation of the received \acs{mv}, of frame $f_{-1}$ at time instant $t_{-1}$, according to the following procedure:
An alternative approach is based on the reconstruction of the missing frame by using motion compensation through extrapolation of the received \acs{mv}, of frame $f_{-1}$ at time instant $t_{-1}$, according to the following procedure:
\begin{enumerate}
\item
the \acs{mv} ($v_{-1}$) of the neighbouring frame ($f_{-1}$) are extrapolated to compute the set of \acs{mv} ($v_{0}$) for the missing frame as follows:
@@ -94,8 +101,9 @@ x^{'} &= x - v_{0x}, \\
y^{'} &= y - v_{0y}.
\end{align}
\end{enumerate}
This method is illustrated in Figure~\ref{FIG5:MotionVectorExtrapolation} by the \ac{mv} $v^{2}_{0}$, which results from the extrapolation of the \ac{mv} $v^{2}_{-1}$, and is referred to as \glsfirst{mve}.
One should note that these methods may not result in a complete motion field for the entire frame, resulting in an incomplete reconstruction. For those regions the average of the three top-left neighbouring \acs{mv} are used to complete the motion field.
This method, referred to as \glsfirst{mve}, is illustrated in Figure~\ref{FIG5:MotionVectorExtrapolation}, where the \ac{mv} $v^{2}_{0}$ results from the extrapolation of $v^{2}_{-1}$.
One should note that these methods may not result in a complete motion field for the entire frame, resulting in an incomplete reconstruction.
For the regions without \acs{mv} the average of the three top-left neighbouring \acs{mv} are used to complete the motion field.
%\todo{Grid-aligment! Incomplete!}
\begin{figure}[tb]
@@ -105,7 +113,8 @@ One should note that these methods may not result in a complete motion field for
\label{FIG5:MotionVectorExtrapolation}
\end{figure}
Finally, as a base for comparison, the default \ac{ec} in the reference \ac{hevc} implementation the \ac{fc} algorithm is also evaluated. This method consists in replacing the lost frame with a copy of the previous decoded one, which can also be described as a motion compensated reconstruction using only zero \acs{mv}.
Finally, as a base for comparison, the default \ac{ec} in the reference \ac{hevc} implementation the \ac{fc} algorithm is also evaluated.
This method consists in replacing the lost frame with a copy of the previously decoded one, which can also be described as a motion compensated reconstruction using only zero \acs{mv}.
\subsection{Algorithms based on motion estimation}
@@ -116,7 +125,8 @@ Typically, optical flow estimation algorithms are used to estimate a 2-D motion
\begin{equation}
E(x,y) = \sum_{x} \sum_{y} \bigg| f_{0}\big(x,y\big) - f_{1}\big( x + v_x(x,y), y+v_y(x,y) \big) \bigg|.
\end{equation}
Besides a full search algorithm, where a block-based approach is used to search for the best \acs{mv} that minimises the sum of absolute errors, three other approaches were tested to evaluate their effectiveness in \ac{ec} algorithms. These three methods are the following:
Besides a full search algorithm, where a block-based approach is used to search for the best \acs{mv} that minimise the sum of absolute errors, three other approaches were tested to evaluate their effectiveness in \ac{ec} algorithms.
These three methods are the following:
\begin{enumerate}
\item a method based on the total variation using the minimization of the \mbox{L-1} norm~\cite{DualFlow2016};
\item a method which approximates the neighbourhood of each pixel by quadratic polynomials on both frames~\cite{PolyFlow2003};
@@ -125,13 +135,16 @@ Besides a full search algorithm, where a block-based approach is used to search
\subsubsection{Motion estimation based on total variation and regularisation (\textit{DualFlow})}
In this method the motion field is estimated based on the minimization of the total variation obtained using the L-1 norm \cite{DualFlow2007,DualFlow2016}, taking into account the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}. This condition forces the estimated field to be regular across neighbouring pixels by penalising deviations in a quadratic sense.
In this method the motion field is estimated based on the minimization of the total variation obtained using the L-1 norm \cite{DualFlow2007,DualFlow2016}, taking into account the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}.
This condition forces the estimated field to be regular across neighbouring pixels by penalising deviations in a quadratic sense.
Using this method, for each pixel $\mathbf{p}$, the motion field $v(\mathbf{p})$ is obtained by minimising the following energy function,
\begin{equation}\label{EQ5:DualFlow}
E(v) =
\sum_{\mathbf{p} \in \Omega} \Big( \big| f_{-1}(\mathbf{p}) - f_{-2}( \mathbf{p} + v(\mathbf{p}))\big| + \big|\nabla v(\mathbf{p})\big| \Big)%d\mathbf{p},\ \mathbf{p} \in \Omega
\end{equation}
where $f_{-1}$ and $f_{-2}$ are the two previous frames at time instants $t_{-1}$ and $t_{-2}$, as represented in Figure~\ref{FIG5:MotionVectorExtrapolation}. In Equation~\ref{EQ5:DualFlow}, $\Omega$ defines the sets of pixels of those frames, and $|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of the optical flow which effectively disallow discontinuities. This is based on the assumption that neighbouring regions belong to the same object, thus such regions should have similar motion.
where $f_{-1}$ and $f_{-2}$ are the two previous frames at time instants $t_{-1}$ and $t_{-2}$, as represented in Figure~\ref{FIG5:MotionVectorExtrapolation}.
In Equation~\ref{EQ5:DualFlow}, $\Omega$ defines the sets of pixels of those frames, and $|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of the optical flow which effectively disallow discontinuities.
This is based on the assumption that neighbouring regions belong to the same object, thus such regions should have similar motion.
\subsubsection{Motion estimation based on polynomial expansion (\textit{PolyFlow})}
@@ -143,7 +156,8 @@ f(\mathbf{p}) \sim \mathbf{p}\mathbf{A}_1\mathbf{p}^T+\mathbf{b}_1\mathbf{p}^T+C
\end{equation}
where $\mathbf{p} = [x\ y]$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ is a row vector and $C_1$ is a scalar. The coefficients are estimated using weighted least squares fit to the signal values in the neighbourhood.
Since the result of a polynomial expansion is such that each neighbourhood is approximated by a polynomial, it is analysed what happens if a polynomial undergoes an ideal translation. Let's consider the signal $f_1(\mathbf{p})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$, given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following:
Since the result of a polynomial expansion is such that each neighbourhood is approximated by a polynomial, it is necessary to analyse what happens if a polynomial undergoes an ideal translation.
Let's consider the signal $f_1(\mathbf{p})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$, given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following:
\begin{align}
f_2(\mathbf{p})
&= f_1(\mathbf{p}-\mathbf{d}) \nonumber \\
@@ -152,7 +166,7 @@ f_2(\mathbf{p})
&= \mathbf{p}\mathbf{A}_1\mathbf{p}^T + (\mathbf{b}_1-2\mathbf{d}\mathbf{A}_1)\mathbf{p}^T + \mathbf{d}\mathbf{A}_1\mathbf{d}^T - \mathbf{b}_1\mathbf{d}^T + C_1\\
&= \mathbf{p}A_2\mathbf{p}^T+\mathbf{b}_2\mathbf{p}^T+C_2, \nonumber
\end{align}
which by equating the coefficients of the quadratic polynomials results in the following relations:
then, by equating the coefficients of the quadratic polynomials results in the following relations:
\begin{align}
\mathbf{A}_2 &= \mathbf{A}_1 \nonumber \\
\mathbf{b}_2 &= \mathbf{b}_1 - 2 \mathbf{d} \mathbf{A}_1, \\
@@ -165,25 +179,16 @@ This allows to find the displacement $\mathbf{d}$ for any signal dimensions by s
%\todo{Completar o Deep flow}
In this method the motion field is estimated based on extension of the total variation methods using descriptor (feature) matching.
This allows the optical flow estimation to accurately find large displacements as it is not bound by constrains, such as the Horn \& Such presented in Equation~(\ref{EQ5:DualFlow})\cite{LargeDisplacementOptFlow_TPAMI2011}. Moreover, the correspondences between descriptions are used to apply a coarse-to-fine wrapping, thus avoiding local minimum values.
This allows the optical flow estimation to accurately find large displacements, because it is not fully bound by any constrains, such as the Horn \& Such presented in Equation~(\ref{EQ5:DualFlow})\cite{LargeDisplacementOptFlow_TPAMI2011}. Moreover, the correspondences between descriptions are used to apply a coarse-to-fine wrapping, which avoids local minimum values.
The combination of both approaches increases the matching accuracy.
This was also exploited in \cite{DeepFlow2013} with the following extensions: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layer optimisation.
This approach was also exploited in \cite{DeepFlow2013} with the following extensions: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layer optimisation.
Figure~\ref{FIG5:DeepFlowDiagram} shows the blocks that are included in the description matching process.
The matching algorithm is built upon a multi-stage architecture with six layers with different patches sizes, interleaving convolutions and max-pooling, which is similar to deep convolutional networks \cite{GradientBasedConvNN_ProcIEEE1998}.
As shown in Figure~\ref{FIG5:DeepFlowDiagram}, a multi-size response levels are generated from the reference and target image using convolutions with different patches sizes. This method uses a bottom-up approach, i.e., starting at a fine level and moving towards the coarser levels (larger patches), which are built as an aggregation of responses of smaller patches. This results in a multi-size response pyramid.
The matching algorithm is built upon a multi-stage architecture using six layers with different patches sizes, interleaving convolutions and max-pooling, which is similar to deep convolutional networks \cite{GradientBasedConvNN_ProcIEEE1998}.
As shown in Figure~\ref{FIG5:DeepFlowDiagram}, multi-size response levels are generated from the reference and target images using convolutions with different patches sizes. This method uses a bottom-up approach, i.e., starting at a fine level and moving towards the coarser levels (larger patches), which are built as an aggregation of responses of smaller patches. This results in a multi-size response pyramid.
Subsequently, a local maximum is extracted from each level to achieve a more accurate motion field (quasi-dense correspondence).
Finally, an initial motion field is extracted, referred to as $v'$ .
\begin{figure}[tb]
\centering
\includegraphics[width=.95\columnwidth]{\ConcealmentAwareDir/Figures/DeepFlowBlockDiagram}
\caption{Block diagram of the deep matching algorithm \cite{DeepFlow2013}.}
\label{FIG5:DeepFlowDiagram}
\FIGSVSPACE
\end{figure}
Subsequently, the final optical flow is obtained by minimising an energy function, which can be described by the following equation:
Then, an initial motion field is extracted, referred to as $v'$ .
The final optical flow is obtained by minimising an energy function, which can be described by the following equation:
\begin{equation}
E(v) =
\sum_{\mathbf{p} \in \Omega} \Big( E_D(\mathbf{p}) + \alpha E_S(\mathbf{p}) + \beta E_M(\mathbf{p}) \Big),
@@ -192,11 +197,19 @@ where $E_D$ is the data term, $E_S$ is a smoothing term and $E_M$ is the matchin
The data term is obtained from the difference between matching pixels in both video frames.
The smoothness term is a robust penalisation of the gradient of the flow, similarly to Equation (\ref{EQ5:DualFlow}), i.e., $E_S(\mathbf{p})=|\nabla v(\mathbf{p})|$. Finally, the matching term ($E_M$) is used to encourage the flow estimation to be similar to the precomputed motion field~($v'$).
\begin{figure}[t]
\centering
\includegraphics[width=.95\columnwidth]{\ConcealmentAwareDir/Figures/DeepFlowBlockDiagram}
\caption{Block diagram of the deep matching algorithm \cite{DeepFlow2013}.}
\label{FIG5:DeepFlowDiagram}
\FIGSVSPACE
\end{figure}
\subsection{Experimental evaluation}
In this section a performance evaluation of the different \ac{ec} schemes previously described is carried out. This evaluation aims at finding the suitable methods to be used for efficiently reconstruct the missing frames in case of data loss.
The two \ac{ec} methods based on the correctly received \acs{mv} are tested, i.e., \ac{mc} and \ac{mve}, using the \ac{fc} method is used as reference for comparison.
This section presents a simulation study a performance evaluation of the different \ac{ec} schemes previously described.
This evaluation aims at finding the suitable methods to be used for efficient reconstruction of the missing frames in case of data loss.
The two \ac{ec} methods based on the correctly received \acs{mv} are tested, i.e., \ac{mc} and \ac{mve}, while the \ac{fc} method is used as reference for comparison.
Moreover, the four different approaches to estimate the motion field for the missing frame are compared: (i) full search based on the minimisation of the sum of absolute errors (\FULLSEARCH), (ii) optical flow based on the duality between total variation and smoothness term (\DUALFLOW), (iii) optical flow based on polynomial approximations (\POLYFLOW) and (iv) optical flow based on deep matching (\DEEPFLOW).
Table~\ref{TAB5:ECPerformaceComparison} shows the average quality (\ac{psnr}) obtained for the different methods under investigation.
@@ -219,15 +232,15 @@ According to the results of Table~\ref{TAB5:ECPerformaceComparison} (columns 5 t
Results for \FULLSEARCH method reveal that only minimising the sum of absolute errors between two consecutive frames might not always be directly related with objects' motions, which results in inaccurate motion field to be used in the reconstruction of lost frames.
Finally, results from Table~\ref{TAB5:ECPerformaceComparison} reveal that the \DEEPFLOW algorithm leads to the most accurate motion field, in comparison with the other methods.
It is able to achieve an average quality gain of 1.79 dB, and up to 3.39 dB for Book Arrival sequence, where \POLYFLOW only achieves 3.05 dB.
Since \POLYFLOW combines different techniques also used by other algorithms in study, e.g., total variation also used by \DUALFLOW, it is expected to achieve accurate estimations and consequently higher \ac{ec} reconstruction quality. Moreover, using descriptor matching in combination with smoothing conditions, the \DEEPFLOW algorithm is able to correctly estimate large displacements, which results in an improved performance for video signals with higher motion, e.g., Tennis sequence.
Since \POLYFLOW combines different techniques also used by other algorithms in study, e.g., total variation also used by \DUALFLOW, it is expected to achieve accurate estimations and consequently higher \ac{ec} reconstruction quality.
Moreover, using feature matching in combination with smoothing conditions, the \DEEPFLOW algorithm is able to correctly estimate large displacements, which results in improved performance for video signals with higher motion, e.g., Tennis sequence.
\subsection*{Remarks}
Summarising, the analysis of the previous results indicates that all methods are able to cope with lost frames and manage to recover the missing data with acceptable quality.
Nevertheless, it is noticeable that lost frames recovered with an estimated motion field result in higher average quality, especially when using the \DEEPFLOW method. However, as the other methods tested are also able to achieve acceptable video quality, they shall not be neglected, as they provide alternative approaches for lost frame recovery with a lower computational complexity (i.e., they not require motion estimation).
Thus, in order to achieve higher reconstruction quality, different approaches shall be considered, aiming at balancing the overall complexity with the recovery quality.
%\HERE
Summarising, the analysis of the previous results indicates that all methods are able to cope with lost frames and manage to recover the missing data with reasonable quality.
Nevertheless, it is noticeable that lost frames recovered with an estimated motion field result in higher average quality, especially when using the \DEEPFLOW method.
However, since the other methods are also able to achieve acceptable video quality, they shall not be neglected, as they provide alternative approaches for lost frame recovery with a lower computational complexity (i.e., they not require motion estimation).
Thus, in order to achieve higher reconstruction quality, different approaches shall be considered, aiming at balancing the overall complexity with the quality of the reconstructed frames.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
@@ -237,18 +250,19 @@ Thus, in order to achieve higher reconstruction quality, different approaches sh
\glsreset{el}
This section presents the general architecture proposed for \ac{ec}-aware video encoding, describing its main functional blocks, and provides an introduction for the subsequent detailed descriptions.
This section presents the general architecture proposed for \ac{ec}-aware video encoding, describing its main functional blocks, and providing an introduction for the subsequent detailed descriptions.
Figure~\ref{FIG5:BlockDiagram} illustrates the architecture of the proposed \ac{ec}-aware encoding scheme.
After the video encoder, it includes data loss simulation at the \ac{nal} for every slice. Subsequently, \ac{ec} is applied to the simulated losses in order to find the best methods that should be used to reconstruct the lost slices in case of transmission errors.
Saliency-weighted \ac{ec} mode decision is used to select the optimal \ac{ec} mode from the best trade-off between \ac{ec} distortion, using the original input video~($f$) as reference, and the number of bits required for signalling.
Then, the optimal \ac{ec} mode is sent to the decoder along with the residue formed between the error-free decoded frame and its error concealed counterpart. The scalable extension of the \ac{hevc} standard is used to encode this residue information.
Then, the optimal \ac{ec} mode is sent to the decoder along with the mismatch residue computed as a difference between the transmitted and recovered frames.
Finally, the scalable extension of the \ac{hevc} standard is used to encode this residue information.
For the \ac{ec} mode decision optimization the missing slices are reconstructed, as they would be in the decoder, i.e., recovered at the encoder by simulating the decoder \ac{ec}, but using different \ac{ec} methods based on motion estimation and extrapolation techniques, to selected the best one at the encoder-side.
For the \ac{ec} mode decision optimization the missing slices are reconstructed, as they would be in the decoder, i.e., they are recovered at the encoder by simulating the decoder \ac{ec}, but using several \ac{ec} methods, based on motion estimation and extrapolation techniques, to select the best one at the encoder-side.
In this process, each \ac{ctu} is partitioned to form a quadtree structure of \ac{ecu} that are used to test the different \ac{ec} candidate modes.
Then, the best trade-off between \ac{ecu} partition and signalling is found through \ac{rd} optimisation.
In order to achieve an efficient use of the signalling overhead, a saliency-weighted procedure is embedded in the \ac{rd} optimisation based on a spatio-temporal saliency, where more signalling bits are allocated to regions with higher saliency values.
Finally, the signalling information is transmitted to the decoder by using the \ac{sei} \acs{nal} units~\cite{HEVCHighLevelReferenceDez2012}.
Since, in general, the optimal \ac{ec} mode is still not able to perfectly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el}.
Finally, the signalling information is transmitted to the decoder by using the \ac{sei} \ac{nal} units~\cite{HEVCHighLevelReferenceDez2012}.
Since, in general, the optimal \ac{ec} mode is still not able to perfectly reconstruct all the missing regions, extra residual information is multiplexed in the stream encoded as a scalable \ac{el}.
This residual information contains the difference between the compressed ($\hat f_t$) and the \ac{ec} frames ($\tilde f$).
\begin{figure}[tb]
@@ -276,7 +290,7 @@ In this section the proposed saliency-weighted \ac{ec} mode decision algorithm i
Firstly, the list of \ac{ec} candidates is provided based on the previously described methods.
Then, the \ac{rd} optimisation is performed using a saliency region in each frame, in order to decrease the overall bitrate without compromising the reconstruction quality.
The signalling coding strategy is also described.
Finally, the \ac{ec} residue is introduced as auxiliary information to improve the reconstruction quality is described.
Finally, coding the \ac{ec} residue, introduced as auxiliary information to improve the reconstruction quality, is described.
\subsection{\ac{ec} candidates}
@@ -306,7 +320,7 @@ The method M3 is based on a conventional \ac{mve}, and finally M4 reconstructs t
\subsection{Optimal \ac{ec} mode selection}
As previously described in the sub-section~\ref{Section5:ECCandidates}, the proposed method allows to choose between four \ac{ec} candidates, which result in different recovered frames for each lost frame (simulated).
As previously described in sub-section~\ref{Section5:ECCandidates}, the proposed method allows to choose between four \ac{ec} candidates, which result in different recovered frames for each lost frame (simulated).
The lost frame is partitioned into quadtree structures, each one forming a set of variable size \acs{ecu}.
Then, the process of finding the optimal \ac{ec} method for each \ac{ecu} is formulated as a rate-distortion cost minimisation. To solve such problem, a Lagrangian optimisation is used to achieve the best trade-off between reconstruction quality and signalling overhead.
Furthermore, image saliency driven by visual attention modelling is used in the optimisation process to weight the cost of the overhead information of each \ac{ecu}, according to the visual relevance of its content.
@@ -340,13 +354,15 @@ m_{l,i}^* = \argmin\limits_{\footnotesize \substack{l\in\left\{0...3\right\} \\
where $R(m_{l,i})$ represents the number of overhead bits required to encode the \ac{ec} mode, and $D(m_{l,i})$ is the sum of absolute error of the \ac{ecu}.
In equation~\ref{EQ6:OptimalDepth}, besides the parameter $\lambda$ used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation. As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques}, this method uses the ratio between the temporal distance from the current frame ($t_C$) to the next refresh frame ($t_I$) and the period of refreshing frames ($T_{I}$).
In equation~\ref{EQ6:OptimalDepth}, besides the parameter $\lambda$, used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation.
As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques}, this method uses the ratio between the temporal distance from the current frame ($t_C$) to the next refresh frame ($t_I$) and the period of refreshing frames ($T_{I}$).
%This weight varies between $\left[0;1 \right]$, achieving the lowest value for intra-coded frames and the maximum for the frame located immediately before an intra-coded one.
This behaviour follows the fact that when a lost frame is closer to the next intra-coded frame, error propagation is lower. Thus, it is not worthwhile to spend too many overhead bits to reduce it. This is accomplished by increasing the cost of such bits according to the temporal distance between the missing frame and the next I frame.
Moreover, a dynamic value for $\lambda$ is also used to adjust the amount of overhead within the frame, according to the visual relevance of each image region, determined using a saliency map.
This behaviour follows the fact that when a lost frame is closer to the next intra-coded frame, the error propagation is lower.
Thus, it is not worthwhile to spend too many overhead bits to reduce it. This is accomplished by increasing the cost of such bits according to the temporal distance between the missing frame and the next I frame.
Moreover, a dynamic value for $\lambda$ is also used to adjust the amount of overhead within the frame, according to the visual relevance of each image region, defined by a saliency map.
The algorithm previously described results in \acs{ecu} with variable size, which is defined by the depth of the quad-tree structure, i.e., level $l$. The partition size is jointly optimised along with the best \ac{ec} candidate method using Equation~\ref{EQ6:OptimalDepth}. Alternatively, a fixed \ac{ecu} size can also be used by defining a fixed depth for all quad-tree structures, i.e., constant $l$ for the entire frame. This can be achieved by using Equation~\ref{EQ6:OptimalDepth} to solely optimise the \ac{ec} methods for each \ac{ecu}.
Although this reduces the overall complexity of the proposed method, it leads to a non-optimal solution which may required additional overhead. In Section~\ref{Section5:ECSignallingPerformance} a comparison between a dynamic and fixed approach is provided.
Although this reduces the overall complexity of the proposed method, it leads to a non-optimal solution which may require additional overhead. In Section~\ref{Section5:ECSignallingPerformance} a comparison between a dynamic and fixed approach is provided.
\subsection{Saliency detection}
@@ -414,7 +430,7 @@ Adaptive arithmetic coding~\cite{ArithmeticCodingJun1994} is used to efficiently
As described earlier in this section, in order to allow further improvement of recovered slices, besides the best \ac{ec} mode, the proposed encoding scheme also allows encoding a controlled amount of \ac{ec} residue using the scalable extension of \ac{hevc} (see Figure~\ref{FIG5:BlockDiagram}).
The enhancement layer module encodes the difference between the coded frame ($\hat f$) and its counterpart recovered by using the optimal \ac{ec} mode ($\tilde f$).
Note that, the higher the accuracy of the best \ac{ec} mode, the lower the
amount of bits spent by the enhancement layer.
amount of bits carried by the enhancement layer.
At the decoder-side, the reconstruction of a missing frame ($\tilde f$) follows three steps:
\begin{enumerate}
\item the missing frame is reconstructed using the optimal \ac{ec} methods for each \ac{ecu} $\tilde f'$ received from the encoder;
@@ -428,7 +444,7 @@ At the decoder-side, the reconstruction of a missing frame ($\tilde f$) follows
The additional residue decoded from the enhancement layer is particularly useful to improve the quality of those \acs{ctu} where the \ac{ec} strategies are not able to produce very accurate reconstruction.
Moreover, since the enhancement layer only carries residual information the overhead is affordable.
Note that the generated bitstream is compliant with the standard, as the \ac{ec} signalling is encapsulated in \ac{sei} \acs{nal}, while the error concealment residue is carried in \acs{nal} related with the scalable extension. Then, an adapted decoder, capable of decoding both the \ac{sei} and the scalable \ac{nal} units is also able to take full benefit from the proposed method. Nevertheless, any standard \ac{hevc} decoder is also able to reconstruct the video, because it simply ignores the extra \ac{nal}.
Note that the generated bitstream is compliant with the standard, as the \ac{ec} signalling is encapsulated in \ac{sei} \acs{nal}, while the \ac{ec} residue is carried in \acs{nal} related with the scalable extension. Then, an adapted decoder, capable of decoding both the \ac{sei} and the scalable \ac{nal} units is also able to take full benefit from the proposed method. However, any standard \ac{hevc} decoder is also able to reconstruct the video, because it simply ignores the extra \ac{nal} units.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -441,6 +457,7 @@ This section presents the performance of the proposed \ac{ec}-aware optimisation
Two different types of performance evaluation are presented.
In the first one, only robust \ac{ec} of intra-frames is evaluated, while in the second one the proposed \ac{ec}-aware optimisation is applied to all frames, corresponding to a generic robust video transmission using optimal \ac{ec}.
For comparison, it is used the \ac{psnr} obtained from the reference \ac{hevc} decoder using motion-copy (\MC) and frame-copy (\REF) for \ac{ec} of the lost frame.
Realistic conditions were used for transmission, by allowing independent losses for any type of \ac{nal} units, hence there is no guarantee that the decoder conditions are always equal to those at the encoder, which means that sub-optimal \ac{ec} may be used at the decoder.
\subsection{Intra-frame EC optimisation}
@@ -450,7 +467,7 @@ In this section the performance of the proposed method is evaluated only for int
Since intra \acs{cu} result in higher amount of bits, they also require a higher number of packets, making them more prone to transmission errors and losses.
Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets correspond to intra coded slices, which is a non-negligible probability of one of them being hit by transmission errors.
Moreover, by limiting the proposed method to intra frames, the amount of required overhead is reduced.
In this case a constant $\lambda_0=10$ was considered without taking into consideration saliency estimation.
In this case a constant $\lambda_0=10$ was used, without taking into consideration saliency estimation.
Since a relevant performance indicator is the error propagation over temporally predicted frames, this was evaluated using a single intra-frame loss.
Figures~\ref{FIG5:singlelosslowdelay} and~\ref{FIG5:singlelossramdon} show the \ac{psnr} results when frame Intra-frame $\#16$ is missing, for \ac{ld} and \ac{ra} encoding configurations, respectively.
@@ -524,12 +541,12 @@ It is also worthwhile to notice that the \ac{psnr} variance obtained with the
%\subsubsection{Overhead analysis}
Subsequently, the percentage of overhead bitrate introduced by the proposed method is compared against fixed \ac{ecu} sizes of $32\times32$ and $16\times16$, i.e., $Level=1$ and $Level=2$.
These results are presented in Table~\ref{TAB6:RedundancyIntra} and were obtained for a total bitrate, as shown in Table~\ref{TableTestSequences}.
These results are presented in Table~\ref{TAB6:RedundancyIntra} and were obtained for the total bitrate shown in Table~\ref{TableTestSequences}.
It is noticeable from the results that reducing the \ac{ecu} size (see results for $32\times32$ and $16\times16$) increases the overhead ratios as more symbols are transmitted (one for each \ac{ecu}).
While the maximum overhead of these fixed partitions is $1.81\%$ (Traffic), the proposed dynamic partitioning method is able to reduce the overhead to a maximum of $0.19\%$ (Kendo).
Moreover, comparing the results of Table~\ref{TAB6:RedundancyIntra} with the video quality results obtained in case of packet losses reveals in the proposed method only a small reduction in the reconstruction quality, even with a significantly lower overhead.
Moreover, comparing the results of Table~\ref{TAB6:RedundancyIntra} with the video quality obtained under packet loss reveals that the proposed method only incurs in a small reduction of the reconstruction quality, with a significantly lower overhead.
This indicates that using dynamic partitioning allows to achieve a better trade-off between the amount of overhead and the \ac{ec} performance.
In summary, the proposed technique is able to enhance the performance of the standard decoding of intra coded frames by finding the best error concealment method at the encoder and sending such information with a small amount of signalling overhead.
In summary, the proposed technique is able to enhance the performance of the standard decoding of intra coded frames by finding the best \ac{ec} method to be used, for each \ac{ecu}, at the encoder using with a small amount of signalling overhead.
\begin{table}[tbp]
\newcommand{\somespaces}{}
@@ -568,7 +585,8 @@ In summary, the proposed technique is able to enhance the performance of the sta
\label{SEC4:OverallPerformance}
In this section the proposed \ac{ec}-aware optimisation scheme is evaluated for a generalised robust video transmission of both intra and inter coded frames.
The experimental results presented in this section were obtained by applying the proposed technique to all video frames.
The experimental results presented in this section were obtained by applying the proposed technique to all video frames.
As in the previous subsection, a $\lambda_0=10$ was used.
This section is organised in the following subsections:
\begin{itemize}
\item evaluation of the usage distribution of each \ac{ec} mode;
@@ -620,8 +638,7 @@ The \ac{ec} candidate methods based on motion field estimation (i.e., M1, M2) ar
The evaluation of the proposed saliency-weighted \ac{ec} mode decision (\DYNLAMBDA) was carried out under random loss events and it was compared with the reference methods.
The video quality was evaluated by using two quality metrics: \ac{psnr} and a \ac{fsim}~\cite{FSIM_TIP2011}, in order to validate the performance of the proposed method in a consolidated manner.
Realistic conditions were used for transmission, by allowing independent losses for any type of \acs{nal}, hence there is no guarantee that the decoder conditions are always equal to those at the encoder, which means that sub-optimal \ac{ec} may be used at the decoder.
The results obtained without saliency weighting are also shown for comparison (\FIXEDLAMBDA). As in the previous subsection, a $\lambda_0=10$ was used.
The results obtained without saliency weighting are also shown for comparison (\FIXEDLAMBDA).
Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM} show the average quality under different \acs{plr} for the \ac{ld} configuration.
Table~\ref{TAB4:SignallingPSNR} shows the \ac{psnr} and weighted \ac{psnr} (\ac{wpsnr}) between parentheses.
@@ -636,9 +653,9 @@ Moreover, the results of Table~\ref{TAB4:SignallingFSIM} confirm the quality imp
Table~\ref{tab:redundancy_signalling} shows the percentage of overhead in the total rate with and without saliency-weighted optimisation, i.e., \FIXEDLAMBDA and \DYNLAMBDA, respectively.
As shown in this table, the saliency-weighted optimisation reduces the amount of overhead by dynamically assigning unequal number of bits to image regions of different visual importance.
These results show that using an adaptive value of $\lambda$, based on the saliency values, leads to an average overhead reduction of 28\% in comparison with the case where no saliency is used, i.e., constant $\lambda$.
It is worth to note that using the saliency-weighted optimisation to reduce the overhead does not compromise the video quality, as can be seen in the results shown in Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM}.
It is worthwhile to note that using the saliency-weighted optimisation to reduce the overhead does not compromise the video quality, as can be seen in the results shown in Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM}.
The \ac{psnr} values for \FIXEDLAMBDA and \DYNLAMBDA are very close, but in terms of overhead savings the results are significant for most sequences.
The achieved \ac{wpsnr} and \ac{fsim} results further demonstrate the effectiveness of the saliency-based optimisation, because the lower overhead does not affect the most relevant image regions, i.e., those with higher saliency values. Thus, the proposed method using a adaptive $\lambda$ (i.e., using saliency) is able to not only achieve higher \ac{ec} quality than the reference methods, but also higher quality gains than using a constant $\lambda$ (i.e., without saliency).
The achieved \ac{wpsnr} and \ac{fsim} results further demonstrate the effectiveness of the saliency-based optimisation, because the lower overhead does not affect the most relevant image regions, i.e., those with higher saliency values. Thus, the proposed method, using a adaptive $\lambda$ (i.e., using saliency), is able to not only achieve higher \ac{ec} quality than the reference methods, but also higher quality gains than using a constant $\lambda$ (i.e., without saliency).
This confirms the two-fold superiority of adaptive $\lambda$, i.e. not only the total overhead is reduced but also higher quality gains are achieved.
%\todo{\HERE}
@@ -733,14 +750,14 @@ R = \frac{\Delta PSNR(\ac{plr}=10\%)}{-\Delta PSNR(Error\text{-}free)}.
The results shown in Table~\ref{tab:overall_gain_ratio}, demonstrate that the effectiveness of the \ac{ec}-aware coding scheme is higher than all other methods under comparison. In fact, not only the absolute \ac{psnr} difference (i.e., \ac{psnr} gain) is higher for the same \ac{plr}, but also the efficiency of the overhead is also higher.
This means that the overhead bits spent to encode the signalling information and the enhancement residue carry more useful information than the other methods.
For instance, in comparison with the method in \cite{DualFrameME2015} the proposed method leads to higher quality degradation in the error-free case. However, for PLR=10\% it clearly outperforms the method in \cite{DualFrameME2015} with a much higher value of $R$, indicating a far superior efficiency.
Results in the table~\ref{tab:overall_gain_ratio} also show that the small loss of coding efficiency in error-free transmission is clearly surpassed by the quality gains under error-prone transmission conditions.
Results in Table~\ref{tab:overall_gain_ratio} also show that the small loss of coding efficiency in error-free transmission is clearly surpassed by the quality gains under error-prone transmission conditions.
The methods in \cite{ROPERefGenJan2010,DualFrameME2015} are based on modified predictions, while the method in \cite{VPxResiliencePCS2016} uses redundant data without any further optimisation.
Both of these approaches require low additional computational complexity.
However, as observed in the results shown in Table~\ref{tab:overall_gain_ratio}, the lower additional complexity also corresponds to lower quality gains in case of transmission errors.
The proposed method uses \ac{ec} techniques with different levels of computational complexity for each \ac{ecu}. Since an adaptive approach is used, the average complexity tends to be lower in comparison with other methods that rely on a fixed approach. Thus, less complex video decoding is obtained when using the proposed method when compared with other straightforward methods that always used the same \ac{ec} technique for all \acs{ctu}.
The proposed method uses \ac{ec} techniques with different levels of computational complexity for each \ac{ecu}. Since an adaptive approach is used, the average complexity tends to be lower in comparison with other methods that rely on a fixed approach. Thus, less complex video decoding is obtained when using the proposed method in comparison with other straightforward methods that always use the same \ac{ec} technique for all \acs{ctu}.
\begin{table}[t]
@@ -772,8 +789,8 @@ The proposed method uses \ac{ec} techniques with different levels of computati
\section{Summary}
\label{Section5:Summary}
In this chapter an \ac{ec}-aware encoding scheme is proposed by incorporating loss simulation at the encoder-side to enhance the EC efficiency at the decoder.
In this chapter a novel \ac{ec}-aware encoding scheme was proposed by incorporating loss simulation at the encoder-side to enhance the EC efficiency at the decoder.
% The proposed method relies on loss simulation at the encoder-side to optimise the error concealment proposed. Then, the optimal error concealment is signalled to the decoder.
The novelty of this method is to include optimal \ac{ec} mode decision at the encoder, to be used by the decoder in case of packet loss, and also to allow encoding of the \ac{ec} residue using the enhancement layer of the \ac{hevc} standard. The use of \ac{ec} signalling was firstly proposed for the use on intra-coded frames in \ref{Conf:ConcAware}, and then improved and proposed for a generic robust video transmission, resulting in a more detailed analysis published in \ref{Journal:ConcAware}.
The novelty of this method is to include optimal \ac{ec} mode decision at the encoder, to be used by the decoder in case of packet loss, and also to allow encoding of the \ac{ec} residue using the enhancement layer of the \ac{hevc} standard. The use of \ac{ec} signalling was firstly proposed for the use on intra-coded frames in (\ref{Conf:ConcAware}), and then improved and proposed for a generic robust video transmission, resulting in a more detailed analysis published in (\ref{Journal:ConcAware}).
The overall results demonstrate that the proposed method is able to efficiently reconstruct lost frames and reduce the error propagation compared with other reference methods.
A consistent quality improvement achieved at the cost of low overhead allows the use of this method in broadcasting video services and applications where packet loss probability is not negligible.
A consistent quality improvement achieved at the cost of low overhead allows the use of this method in broadcasting video services and network applications where packet loss probability is not negligible.