jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Finish initial review of chapter 4 to 6

Browse Source
This commit is contained in:
jfmcarreira
2018-08-30 02:35:31 +01:00
parent ba6b505983
commit 8d252f0dbd
21 changed files with 567 additions and 493 deletions
Download Patch File Download Diff File Expand all files Collapse all files
042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex
+238 -213
View File
@@ -4,7 +4,7 @@
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CHAPTER
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Error Concealment-Aware for Robust Video Decoding}
\chapter{Error Concealment-Aware Video Encoding}
\label{Chapter:ConcealmentAwareTechniques}
%\vspace{-6mm}
%\clearpage
@@ -15,15 +15,18 @@
\section{Introduction}
\glsreset{ec}
\glsreset{rd}
\glsreset{ra}
\glsreset{ld}
%\glsreset{rd}
In this chapter an error concealment-aware encoding scheme is proposed to improve the quality of decoded video in broadcast environments prone to transmission errors and data loss. The method aims to improve the quality of recovered lost frames by optimising the efficiency of the \ac{ec} beforehand at the encoder-side.
The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled through \ac{sei} messages. Such optimal \ac{ec} modes are found by simulating transmission losses followed by a lagrangian optimisation of the \textit{signalling rate - \ac{ec} distortion} cost. A generalised saliency-weighted distortion is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. When data loss occurs the decoder uses the signalling information to improve the reconstruction quality.
In this chapter an error concealment-aware encoding scheme is proposed to improve the quality of decoded video in broadcast environments prone to transmission errors and data loss.
The method aims to enhance the reconstruction of recovered lost frames by optimising the efficiency of the \ac{ec}.
The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled through \ac{sei} messages. Such optimal \ac{ec} modes are found by simulating transmission losses followed by a Lagrangian optimisation of the \textit{signalling rate - \ac{ec} distortion} cost. A generalised saliency-weighted distortion is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. When data loss occurs the decoder uses the signalling information to improve the reconstruction quality.
This chapter is organized as follows. Section~\ref{Section5:Problem} presents the problem formulation and motivation for the method proposed in this chapter. In Section~\ref{Section5:ECCandidates} does an initial evaluation of different error concealment schemes in order to find the relevant ones to be used in the proposed error concealment-aware coding scheme.
Section~\ref{Section5:OverviewProposedMethod} provides an overview of the architecture of the proposed method followed by a detailed description in Section~\ref{Section5:ProposedMethod}.
Section~\ref{Section5:ECSignallingPerformance} presents a discussion of the performance of the proposed \ac{ec} optimisation and signalling, and Section~\ref{Section5:OverallPerformance} presents an overall performance of the proposed method.
Finally, Section~\ref{Section5:Summary} concludes this chapter.
Section~\ref{Section5:Architecture} provides an overview of the architecture of the proposed method followed by a detailed description in Section~\ref{Section5:ProposedMethod}.
Sections~\ref{Section5:ECSignallingPerformance}~and~\ref{Section5:OverallPerformance} present the performance results and discussion of the proposed \ac{ec} optimisation and signalling.
Finally, Section~\ref{Section5:Summary} concludes the chapter.
\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -33,41 +36,48 @@ Finally, Section~\ref{Section5:Summary} concludes this chapter.
\label{Section5:Problem}
\glsreset{mve}
\glsunset{mb}
\glsreset{mc}
%\glsunset{mb}
In Chapter~\ref{Chapter:ErrorResilienceReview} different \ac{ec} algorithms that can effectively reconstruct lost data were described. However, different algorithms might not present the same accuracy for every lost frame or slice. Moreover, selection at the \ac{mb}/\ac{pu} level is not straightforward.
In the past, hybrid \ac{ec} schemes were proposed to improve the reconstruction quality by either combining spatial and temporal \ac{ec} algorithms~\cite{HybridConcTB2008} or by optimising the \ac{mve} at the pixel and block levels~\cite{HybridConcealmentTB2008,2DConcHybridMotion2010}.
Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of the motion-based recovery by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}. Although such methods are able to reduce the reconstruction artefacts and improve the overall quality, decoder-based decisions are still not fully accurate, as they do not have enough reference information to ease the decision process. Therefore, higher performance can be achieved by performing encoding-side estimation and decision based on the error-free information.
In Chapter~\ref{Chapter:ErrorResilienceReview} different \ac{ec} algorithms that can effectively reconstruct lost video were described, though not all of them exhibit the same accuracy for every lost frame or slice.
Moreover, it was also shown that combining different \ac{ec} approaches at the \ac{mb}/\ac{pu} level to improve the reconstruction accuracy is not straightforward.
In the past, hybrid \ac{ec} schemes were proposed to improve the recovery quality of lost frames by either combining spatial and temporal \ac{ec} algorithms~\cite{HybridConcTB2008} or by optimising the \ac{mve} at the pixel and block levels~\cite{HybridConcealmentTB2008,2DConcHybridMotion2010}.
Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of motion-based recovery approaches by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}. Although such methods are able to reduce the reconstruction artefacts and improve the overall video quality, decoder-based decisions are still not fully accurate, as they only have limited information to be used as reference in the decision process.
Therefore, higher reconstructed quality can be achieved by estimating the best \ac{ec} techniques at the encoding-side based on the original video signal.
%Such approach not only improves quality but also reduces the complexity of the video decoders.
%In this chapter a robust coding driven by \ac{ec} methods is developed with the objective of enhancing the quality of reconstructed video at decoders receiving erroneous streams from the transmission network. The proposed approach combines error resilience tools at the encoder-side with \ac{ec} at the decoder-side.
Different approaches were proposed in the past to address the problem of \ac{ec} at the encoder-side \cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ErrorResilienceModeHEVC2016}. Moreover, extra information was used to aid the \ac{ec} operation at the decoder-side, either by sending redundant data \cite{ResilienceRECAP_ICASSP2009,SaliencyErrorConcealment_TM2013} or by embedding data into the bitstream \cite{ConcealmentIntraDataHidng2009}. Although the above methods facilitate the \ac{ec} performance either through slice reordering or based on \acs{mv}, they always rely on the assumption that a fixed \ac{ec} method is used at the decoder.
Different approaches were proposed in the past to address the problem of \ac{ec}-aware video coding~\cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ErrorResilienceModeHEVC2016}.
Extra information was also used to aid the \ac{ec} operation at the decoder-side, either by sending redundant information \cite{ResilienceRECAP_ICASSP2009,SaliencyErrorConcealment_TM2013} or by embedding additional coded data into the bitstream \cite{ConcealmentIntraDataHidng2009}.
Although these methods facilitate the \ac{ec} performance either through slice reordering or based on \acs{mv}, they always rely on the assumption that some fixed \ac{ec} method is used at the decoder.
% The work presented in \cite{ConcealmentSignalling2008,SignallingRobustScalable2015} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder. Although such approaches achieves reasonably good performance, it relies on the spatial neighbouring information or multi-layer video, which may not be available in all cases.
To advance one step further, in this chapter an concealment-aware encoding scheme is proposed to achieve higher flexibility in terms of the \ac{ec} approaches, leading to higher recovered video quality.
This is accomplished by selecting at the encoder, among several \ac{ec} algorithms with different characteristics, the one that yields the lowest reconstruction distortion at the decoder and also minimises the \ac{rd} cost considering the necessary signalling rate. Thus, the proposed approach outperforms similar previous methods \cite{ConcealmentAwareME2006,ErrorResilienceModeHEVC2016} by selecting the best error concealment method without affecting the encoder optimisation.
To further improve the quality of recovered frames, an enhancement layer is encoded with the mismatch residue between the transmitted and recovered frames. This is a dynamic optimisation process, which is more efficient than traditional approaches based on redundant pictures \cite{RedudantPicsJan2009,VPxResiliencePCS2016}, which transmit the whole frame as redundancy.
In contrast with previous work presented in Chapter~\ref{Chapter:ErrorResilienceTechniques} that addressed the error resilience problem by optimising the coding dependencies, the approach followed in this work considers both \ac{ec} performance and the rate required for signalling the best \ac{ec} mode to the decoder.
To advance one step further, this chapter presents n investigation on concealment-aware video coding and proposes a new scheme to achieve high decoding flexibility in terms of the \ac{ec} approaches, leading to better video quality.
This is accomplished by selecting, at the encoder, among several \ac{ec} algorithms with different characteristics, the one that yields the lowest reconstruction distortion at the decoder and also minimises the \ac{rd} cost considering the necessary signalling rate. Thus, the proposed approach differs from similar previous methods \cite{ConcealmentAwareME2006,ErrorResilienceModeHEVC2016} by selecting the best error concealment method without affecting the encoder optimisation.
To further improve the quality of recovered frames, an enhancement layer is encoded with the mismatch residue between the transmitted and recovered frames. This is a dynamic optimisation process, which is more efficient than conventional approaches based on redundant pictures \cite{RedudantPicsJan2009,VPxResiliencePCS2016}, which transmit the whole frame as redundancy.
In contrast with the previous work presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}, which addressed the error resilience problem by optimising the coding dependencies, the approach followed in this chapter considers both \ac{ec} performance and the coded rate required for signalling the best \ac{ec} mode to the decoder.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Traditional error concealment algorithms}
\section{Error concealment algorithms}
\label{Section5:ECCandidates}
In order to proposed a novel error concealment-aware technique different algorithms were taken into consideration. This section provides a description and an evaluation of the error concealment candidates used in the proposed method.
Firstly, a detailed description of each approach is provided followed by a performance comparison between each method.
\subsection{Motion-compensation algorithms}
In order to proposed a novel error concealment-aware technique different algorithms were taken into consideration.
This section presents a description and evaluation of the most relevant \ac{ec} algorithms used in the proposed method.
Firstly, a technical description of each method is provided followed by a performance comparison between them.
\subsection{EC algorithms based on motion-compensation}
%\subsection{Motion-compensated reconstruction}
The most common algorithms to recover missing frames takes advantage of the correctly received \acs{mv} from the neighbouring frames.
The most common algorithms to recover missing frames take advantage of the correctly received \acs{mv} from the neighbouring frames.
This has been widely used in the past for \ac{h264} standard \cite{H264RefConc2006,BlockBasedMVE2002,BlockBasedMVEOverllaped2011} and also in the \ac{hevc} standard~\cite{HEVCMCBlockAug2013,HEVCMCResidueDec2013,DownhillHEVCCon_ICCE2017}.
Let's consider the example of Figure~\ref{FIG5:MotionVectorExtrapolation}, where a missing frame ($f_0$) at time instance $t_0$ is reconstructed based the information present in the neighbouring frames, $f_{-1}$ and $f_{-2}$.
One of the common approaches, i.e., motion-copy based algorithms reconstructs the missing frame through motion compensation directly using the co-located vectors in the closest neighbour frame ($f_{-1}$). The \acs{mv} for the motion compensation ($v_0$) are obtained by copying the \acs{mv} $v_{-1}$ from the same position in $f_{-1}$, as shown by the example (1) in the figure.
Let's consider the example of Figure~\ref{FIG5:MotionVectorExtrapolation}, where a missing frame ($f_0$) at time instant $t_0$ is reconstructed based on the information present in the temporally adjacent frames, $f_{-1}$ and $f_{-2}$.
One of the common approaches is \ac{mc} where the missing frame is reconstructed through motion compensation directly using the co-located vectors in the closest neighbour frame ($f_{-1}$). The \acs{mv} used for the motion compensation ($v_0$) are obtained by copying the \acs{mv} $v_{-1}$ from the same spatial position in $f_{-1}$, as illustrated in the figure by the \ac{mv} $v^{1}_{0}$ which is obtained by copying $v^{1}_{-1}$.
An alternative approach reconstructs the missing frame by using motion compensation through the correctly received \acs{mv}, of frame $f_{-1}$ at time instant $t_{-1}$, according to the following procedure:
An alternative approach is based on the reconstruction of the missing frame by using motion compensation through a extrapolation of the received \acs{mv}, of frame $f_{-1}$ at time instant $t_{-1}$, according to the following procedure:
\begin{enumerate}
\item
the \acs{mv} ($v_{-1}$) of the neighbouring frame ($f_{-1}$) are extrapolated to compute the set of \acs{mv} ($v_{0}$) for the missing frame as follows:
@@ -75,7 +85,7 @@ the \acs{mv} ($v_{-1}$) of the neighbouring frame ($f_{-1}$) are extrapolated to
% v^\zeta_0 = \frac{ t_0 - t_{C} }{ t_{-1} - t_{R} } \times v^\zeta_{-1},\ \zeta=s,r
v_0 = \frac{ t_0 - t_{-1} }{ t_{-1} - t_{R} } \times v_{-1},
\end{equation}
where $t_{R}$ is the time instant of the reference frame pointed by the original \acs{mv} $v_{-1}$.
where $t_{R}$ is the time instant of the reference frame pointed by the original \acs{mv} $v_{-1}$, which in the example of Figure~\ref{FIG5:MotionVectorExtrapolation} corresponds to $t_{-2}$.
\item
the \ac{mv} associated to the block $b_{-1}(x,y)$ at spatial position $(x,y)$ in $f_{-1}$ may be used to recover the block $b^{'}_{{0}}(x^{'},y^{'})$ in $f_0$, at position $(x^{'},y^{'})$ obtained from $v_0$ components as follows:
\begin{align}
@@ -83,7 +93,9 @@ x^{'} &= x - v_{0x}, \\
y^{'} &= y - v_{0y}.
\end{align}
\end{enumerate}
This is illustrated in Figure~\ref{FIG5:MotionVectorExtrapolation} by the example (2). This method is referred to as \glsfirst{mve}.
This method is illustrated in Figure~\ref{FIG5:MotionVectorExtrapolation} by the \ac{mv} $v^{2}_{0}$ which results from the extrapolation of the \ac{mv} $v^{2}_{-1}$, and is referred to as \glsfirst{mve}.
One should note that these methods may not result in a complete motion field for the entire frame, resulting in an incomplete reconstruction. For those regions the average of the three top-left neighbouring \acs{mv} are used to complete the motion field.
%\todo{Grid-aligment! Incomplete!}
\begin{figure}[tb]
\centering
@@ -96,46 +108,45 @@ Finally, as a base for comparison the default \ac{ec} in the reference \ac{hevc}
\subsection{Algorithms based on motion estimation}
In this sub-section different method that can be used to estimate a motion-field are described, which can be used to reconstruct the missing motion information.
The obtained motion-field is used in combination with the \ac{mve} algorithm to recover the lost frame.
In this sub-section a different method to estimate a motion-field is described. This can be used to reconstruct the missing motion information of a lost frame.
Then, the recovered motion-field is used in combination with the \ac{mve} method to recover the lost frame.
Typically, optical flow estimation algorithms are used to estimate a 2-D \ac{mv} field, $v=(v_x, v_y)$, for each pixel, such as the following energy function is minimised:
Typically, optical flow estimation algorithms are used to estimate a 2-D motion field, comprising one \ac{mv} $v=(v_x, v_y)$, for each pixel, which minimises the following energy function:
\begin{equation}
E(x,y) = \sum_{x} \sum_{y} \bigg| f_{0}\big(x,y\big) - f_{1}\big( x + v_x(x,y), y+v_y(x,y) \big) \bigg|^2.
E(x,y) = \sum_{x} \sum_{y} \bigg| f_{0}\big(x,y\big) - f_{1}\big( x + v_x(x,y), y+v_y(x,y) \big) \bigg|.
\end{equation}
Besides using a full search algorithm where a block-based approach is used to search for the best \acs{mv} based on the sum of absolute errors, in this work three different approaches were tested to evaluate their effectiveness to be used in \ac{ec} algorithms (described below), namely:
Besides a full search algorithm where a block-based approach is used to search for the best \acs{mv} that minimises the sum of absolute errors, three other approaches were tested to evaluate their effectiveness in \ac{ec} algorithms. These three methods are the following:
\begin{enumerate}
\item a method based on the total variation based on the minimization of the \mbox{L-1} norm~\cite{DualFlow2016};
\item a method which approximates the neighbouring information of a given pixel by quadratic polynomials on both frames~\cite{PolyFlow2003};
\item a method based on a combination of descriptor matching and large displacement estimation~\cite{DeepFlow2013}.
\item a method based on the total variation using the minimization of the \mbox{L-1} norm~\cite{DualFlow2016};
\item a method which approximates the neighbourhood of each pixel by quadratic polynomials on both frames~\cite{PolyFlow2003};
\item a method based on a combination of descriptor matching and large displacement estimation~\cite{DeepFlow2013}. \HERE
\end{enumerate}
\subsubsection{Motion estimation based on total variation and regularisation (\textit{DualFlow})}
In this method the motion field is estimated based on the minimization of the total variation obtained using the L-1 norm \cite{DualFlow2007,DualFlow2016}, taking into account the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}. This condition forces the estimated field to be regular across neighbouring pixels by penalising deviations in a quadratic way.
In this method the motion field is estimated based on the minimization of the total variation obtained using the L-1 norm \cite{DualFlow2007,DualFlow2016}, taking into account the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}. This condition forces the estimated field to be regular across neighbouring pixels by penalising deviations in a quadratic sense.
Using this method, for each pixel $\mathbf{p}$, the motion field $v(\mathbf{p})$ is obtained by minimising the following energy function,
\begin{equation}\label{EQ5:DualFlow}
E(v) =
\sum_{\mathbf{p} \in \Omega} \Big( \big| f_{-1}(\mathbf{p}) - f_{-2}( \mathbf{p} + v(\mathbf{p}))\big| + \big|\nabla v(\mathbf{p})\big| \Big)%d\mathbf{p},\ \mathbf{p} \in \Omega
\end{equation}
where $f_{-1}$ and $f_{-2}$ are the two previous frames at time instances $t_{-1}$ and $t_{-2}$, as represented in Figure~\ref{FIG5:MotionVectorExtrapolation}. In the equation, $\Omega$ defines the sets of pixels of those frames.
$|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of the optical flow which effectively disallow discontinuities. This is based on the assumption that neighbouring regions belong to the same object, thus such regions should have similar motion.
where $f_{-1}$ and $f_{-2}$ are the two previous frames at time instants $t_{-1}$ and $t_{-2}$, as represented in Figure~\ref{FIG5:MotionVectorExtrapolation}. In Equation~\ref{EQ5:DualFlow}, $\Omega$ defines the sets of pixels of those frames, and $|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of the optical flow which effectively disallow discontinuities. This is based on the assumption that neighbouring regions belong to the same object, thus such regions should have similar motion.
\subsubsection{Motion estimation based on polynomial expansion (\textit{PolyFlow})}
In this method the motion field is estimated based on a polynomial expansion a neighbourhood of a pixel~\cite{PolyFlow2003}.
In this method the motion field is estimated based on a polynomial expansion of each pixel neighbourhood~\cite{PolyFlow2003}.
In this method only quadratic polynomials are considered, expressed as:
\begin{equation}\label{EQ5:PolyFlow}
f(\mathbf{p}) \sim \mathbf{p}A_1\mathbf{p}^T+\mathbf{b}_1\mathbf{p}^T+C_1
f(\mathbf{p}) \sim \mathbf{p}\mathbf{A}_1\mathbf{p}^T+\mathbf{b}_1\mathbf{p}^T+C_1
\end{equation}
where $\mathbf{p} = [x\ y]$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ is a row vector and $C_1$ is a scalar. The coefficients are estimated from a weighted least squares fit to the signal values in the neighbourhood.
where $\mathbf{p} = [x\ y]$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ is a row vector and $C_1$ is a scalar. The coefficients are estimated using weighted least squares fit to the signal values in the neighbourhood.
Since the result of polynomial expansion is that each neighbourhood is approximated by a polynomial, we start by analysing what happens if a polynomial undergoes an ideal translation. Let's consider the signal $f_1(\mathbf{p})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$ given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following:
\begin{align}
f_2(\mathbf{p})
&= f_1(\mathbf{p}-\mathbf{d}) \nonumber \\
&= (\mathbf{p}-\mathbf{d})A_1(\mathbf{p}-\mathbf{d})^T+\mathbf{b}_1(\mathbf{p}+\mathbf{d})^T+C_1 \\
&= (\mathbf{p}-\mathbf{d})\mathbf{A}_1(\mathbf{p}-\mathbf{d})^T+\mathbf{b}_1(\mathbf{p}-\mathbf{d})^T+C_1 \\
&= \mathbf{p}\mathbf{A}_1\mathbf{p}^T - 2\mathbf{d}\mathbf{A}_1\mathbf{p}^T+ \mathbf{d}\mathbf{A}_1\mathbf{d}^T+ \mathbf{b}_1\mathbf{p}^T-\mathbf{b}_1\mathbf{d}^T + C_1 \nonumber \\
&= \mathbf{p}\mathbf{A}_1\mathbf{p}^T + (\mathbf{b}_1-2\mathbf{d}\mathbf{A}_1)\mathbf{p}^T + \mathbf{d}\mathbf{A}_1\mathbf{d}^T - \mathbf{b}_1\mathbf{d}^T + C_1\\
&= \mathbf{p}A_2\mathbf{p}^T+\mathbf{b}_2\mathbf{p}^T+C_2, \nonumber
@@ -150,6 +161,8 @@ This allows to find the displacement $\mathbf{d}$ for any signal dimensions by s
\subsubsection{Large displacement optical flow with deep matching (\textit{DeepFlow})}
\todo{Completar o Deep flow}
In this method the motion field is estimated based on the large displacement optical flow presented in \cite{LargeDisplacementOptFlow_TPAMI2011}, which extends the total variation methods, e.g. the one expressed by Equation (\ref{EQ5:DualFlow}), by including descriptor matching.
The correspondences between descriptions are used to apply a coarse-to-fine wrapping, thus avoiding local minimum values. The matching algorithm is built upon a multi-stage architecture with six layers with different patches sizes, interleaving convolutions and max-pooling \cite{DeepFlow2013}. The combination of both approaches increases the matching accuracy.
@@ -175,10 +188,13 @@ The smoothness term is a robust penalisation of the gradient of the flow as used
\subsection{Experimental evaluation}
In this section a performance evaluation of the different \ac{ec} schemes previously described is carried out. This evaluation aims at finding the suitable methods to be used for efficiently reconstruct the missing frames in case of data loss. In this evaluation the \ac{fc} method is used as reference for comparison. Then, two methods based on the correctly received \acs{mv} are tested, i.e., \ac{mc} and \ac{mve}. Moreover, the four different approaches based on a estimated motion field are compared: (i) full search based on the minimisation of the sum of absolute errors (\FULLSEARCH), (ii) optical flow based a duality between total variation and smoothness term (\DUALFLOW), (iii) optical flow based on polynomial approximations (\POLYFLOW) and (iv) optical flow based on deep matching (\DEEPFLOW).
In this section a performance evaluation of the different \ac{ec} schemes previously described is carried out. This evaluation aims at finding the suitable methods to be used for efficiently reconstruct the missing frames in case of data loss.
The two \ac{ec} methods based on the correctly received \acs{mv} are tested, i.e., \ac{mc} and \ac{mve}, using the \ac{fc} method is used as reference for comparison.
Moreover, the four different approaches to estimate the motion field for the missing frame are compared: (i) full search based on the minimisation of the sum of absolute errors (\FULLSEARCH), (ii) optical flow based on the duality between total variation and smoothness term (\DUALFLOW), (iii) optical flow based on polynomial approximations (\POLYFLOW) and (iv) optical flow based on deep matching (\DEEPFLOW).
Table~\ref{TAB5:ECPerformaceComparison} shows the average quality results obtained using the \ac{psnr} metric for different methods. The absolute value is shown for the \FC method, and differential values are shown for the remaining methods.
Results reveal that the methods \MC and \MVE are able to overcome \FC which does not use any \acs{mv}, achieving average quality gains of up to 1.41 dB and 1.46 dB, respectively. Moreover, by extrapolating the \acs{mv} from the correctly receiving frame to the lost time instance (\MVE) is able to outperform straightforward motion-copy (\MC).
Table~\ref{TAB5:ECPerformaceComparison} shows the average quality (\ac{psnr}) obtained for the different methods under investigation.
The absolute value of \ac{psnr} is shown for the \FC method, while the differential values are shown for the remaining methods.
The results shown in the table reveal that methods \MC and \MVE are both able to overcome \FC, achieving average quality gains of up to 1.41 dB and 1.46 dB, respectively. Moreover, by extrapolating the \acs{mv} from the correctly receiving frame to the lost one (\MVE), it is also possible to outperform motion-copy (\MC).
\begin{table}[tb]
\centering
@@ -188,12 +204,15 @@ Results reveal that the methods \MC and \MVE are able to overcome \FC which does
\input{\ConcealmentAwareDir/Results/PerformanceConcealmentMethods.tex}
\end{table}
\HERE
Results for \ac{ec} methods based on motion estimation (see columns 5 to 8 of Table~\ref{TAB5:ECPerformaceComparison}) reveal an overall superior performance when comparing with methods which rely of correctly received \acs{mv}.
Results for \FULLSEARCH method reveals that only minimising the sum of absolute errors between two consecutive frames might not always be directly related with objects' motions. Thus, this results in an inaccurate motion estimation to be used for the reconstruction of lost frames. Finally, results from Table~\ref{TAB5:ECPerformaceComparison} reveal that the \DEEPFLOW algorithm leads to the most accurate motion field to be used for \ac{ec}, in comparison with the other methods. It is able to achieve an average quality gain of 1.79 dB, and up to 3.39 dB for Book Arrival sequence, where \POLYFLOW only achieves 3.05 dB.
\subsection{Conclusions}
\subsection*{Remarks}
Summarising, the analysis of the previous results indicates that all methods cope with lost frames and manage to recover the missing data with acceptable quality. Nevertheless, it is noticeable that the recovering lost frames with an estimated motion field leads to higher quality, especially using the \DEEPFLOW method. However, as the other methods tested are also able to achieve acceptable video quality they shall not be neglected, as they provide alternative approaches for lost frame recovery with lower computational complexity (i.e., they not require motion estimation).
Summarising, the analysis of the previous results indicates that all methods are able to cope with lost frames and manage to recover the missing data with acceptable quality.
Nevertheless, it is noticeable that lost frames recovered with an estimated motion field result in higher average quality, especially when using the \DEEPFLOW method. However, as the other methods tested are also able to achieve acceptable video quality, they shall not be neglected, as they provide alternative approaches for lost frame recovery with lower computational complexity (i.e., they not require motion estimation).
Thus, in order to achieve higher reconstruction quality different approaches shall be considered, aiming at balancing the overall complexity with the recovery quality.
%\HERE
@@ -201,17 +220,16 @@ Thus, in order to achieve higher reconstruction quality different approaches sha
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Proposed architecture}
\label{Section5:OverviewProposedMethod}
\section{Architecture for the EC-aware video encoding}
\label{Section5:Architecture}
\glsreset{ec}
%\glsreset{ec}
This section presents an overview of the proposed method architecture describing the functional blocks presented providing an introduction for the subsequent detailed descriptions.
Figure~\ref{FIG5:BlockDiagram} illustrates the architecture of the proposed encoding scheme.
After the video encoder, it includes data loss simulation at the \ac{nal} for every slice. Subsequently, error concealment is applied to find the best \ac{ec} methods that should be used to reconstruct the lost slices in case of transmission errors.
Saliency-weighted \ac{ec} mode decision is used to select the optimal \ac{ec} mode from the best trade-off between \ac{ec} distortion, using the original input video ($f$) as reference, and the number of bits required for signalling.
Then the optimal mode is sent to the decoder along with the residue formed between the error-free decoded frame and the error concealed one, using the scalable extension of the \ac{hevc} standard.
This section presents the general architecture proposed for \ac{ec}-aware video encoding and describes its main the functional blocks, also providing an introduction for the subsequent detailed descriptions.
Figure~\ref{FIG5:BlockDiagram} illustrates the architecture of the proposed \ac{ec}-aware encoding scheme.
After the video encoder, it includes data loss simulation at the \ac{nal} for every slice. Subsequently, \ac{ec} is applied to the simulated losses in order to find the best methods that should be used to reconstruct the lost slices in case of transmission errors.
Saliency-weighted \ac{ec} mode decision is used to select the optimal \ac{ec} mode from the best trade-off between \ac{ec} distortion, using the original input video~($f$) as reference, and the number of bits required for signalling.
Then, the optimal \ac{ec} mode is sent to the decoder along with the residue formed between the error-free decoded frame and its error concealed counterpart. The scalable extension of the \ac{hevc} standard is used to encode this residue information.
\begin{figure}[tb]
\centering
@@ -223,12 +241,12 @@ Then the optimal mode is sent to the decoder along with the residue formed betwe
\end{figure}
For the \ac{ec} mode decision optimization the missing slices are reconstructed, as they would be in the decoder, but using different \ac{ec} methods based on motion estimation and extrapolation techniques, to selected the best one at the encoder-side.
For the \ac{ec} mode decision optimization the missing slices are reconstructed, as they would be in the decoder, i.e., recovered at the encoder by simulating the decoder \ac{ec}, but using different \ac{ec} methods based on motion estimation and extrapolation techniques, to selected the best one at the encoder-side.
In this process, each \ac{ctu} is partitioned to form a quadtree structure of \ac{ecu} that are used to test the different \ac{ec} candidate modes.
Then, the best trade-off between \ac{ecu} partition and signalling is found through \ac{rd} optimisation.
In order to achieve an efficient use of the signalling overhead a saliency-weighted procedure is embedded in the \ac{rd} optimisation based on a spatio-temporal saliency, where more signalling bits are allocated to regions with higher saliency values.
In order to achieve an efficient use of the signalling overhead, a saliency-weighted procedure is embedded in the \ac{rd} optimisation based on a spatio-temporal saliency, where more signalling bits are allocated to regions with higher saliency values.
Finally, the signalling information is transmitted to the decoder by using the \ac{sei} \acs{nalu} type~\cite{HEVCHighLevelReferenceDez2012}.
Since in general, the optimal \ac{ec} mode is still not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el}.
Since in general, the optimal \ac{ec} mode is still not able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el}.
This residual information contains the difference between the compressed ($\hat f_t$) and the \ac{ec} frames ($\tilde f$).
The transmitted information is used to assist the \ac{ec} operation at the decoder to recover erroneous frames with high accuracy, by using the best \ac{ec} mode and also residual information to reduce error propagation.
@@ -239,18 +257,19 @@ The remaining of this section describes in detail the building blocks of the pro
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\section{Saliency-weighted error concealment mode decision}
\section{Error concealment mode decision}
\section{Saliency-weighted \ac{ec} mode decision}
%\section{Error concealment mode decision}
\label{Section5:ProposedMethod}
In this section the saliency-weighted \ac{ec} mode decision algorithm is described. Firstly, the list of \ac{ec} candidates from the previously described methods are enumerated. Then the \ac{rd} optimisation is proposed based on a saliency detection to decrease the overall bitrate without compromising the reconstruction quality. Moreover, the signalling coding strategy is also described. Finally, the introducing of \ac{ec} residue as auxiliary information to improve the reconstruction quality is described.
In this section the proposed saliency-weighted \ac{ec} mode decision algorithm is described. Firstly, the list of \ac{ec} candidates is provided based on the previously described methods.
Then, the \ac{rd} optimisation is proposed using a saliency region in each frame to decrease the overall bitrate without compromising the reconstruction quality.
The signalling coding strategy is also described.
Finally, the introducing of \ac{ec} residue as auxiliary information to improve the reconstruction quality is described.
\subsection{Error concealment candidates}
\subsection{\ac{ec} candidates}
A set of four \ac{ec} candidate modes (M1...M4) is used in the optimisation process that selects one of them as best for each particular \ac{ecu}. Each \ac{ec} mode corresponds to a different \ac{ec} method, as shown in Figure~\ref{FIG5:concealmentcandidates}, namely:
A set of four \ac{ec} candidate modes (M1...M4) is used in the optimisation process that selects one of them as the best for each particular \ac{ecu}. Each \ac{ec} mode corresponds to a different \ac{ec} method, as shown in Figure~\ref{FIG5:concealmentcandidates}, namely:
\begin{itemize}
\item M1, M2: These two methods are based on extrapolation of an estimated motion field;
\item M3: This method is based on \ac{mv} extrapolation of previously received \acs{mv};
@@ -266,16 +285,17 @@ A set of four \ac{ec} candidate modes (M1...M4) is used in the optimisation pro
\FIGSVSPACE
\end{figure}
The first two methods M1, M2 rely on an estimated motion field, using the method \DEEPFLOW \cite{DeepFlow2013}, to recover the lost frame. Then, it is extrapolated at the pixel level, i.e., one \ac{mv} is assigned to each pixel, and \ac{pu} level, i.e., one \ac{mv} is assigned to the entire block, at the missing frame instant. This results in two different reconstructions of the missing frame, where the former may increase the accuracy of the pixel extrapolation, but the latter has the advantage of keeping the block structure and generate a smoother reconstruction.
The first two methods M1, M2 rely on an estimated motion field, using the method \DEEPFLOW \cite{DeepFlow2013}, to recover the lost frame. Then, such motion field is extrapolated at the pixel level, i.e., one \ac{mv} is assigned to each pixel, and \ac{pu} level, i.e., one \ac{mv} is assigned to the entire unit.
This results in two different reconstructions of the missing frame, where the former may increase the accuracy of the pixel extrapolation, but the latter has the advantage of keeping the block structure and generate a smoother reconstruction.
The method M3 is based on a traditional \glsfirst{mve}, and finally M4 reconstructs the missing frames using a \glsfirst{mc} algorithm.
The method M3 is based on a convention \ac{mve}, and finally M4 reconstructs the missing frames using the \ac{mc} algorithm.
\subsection{Selection optimization}
\subsection{Optimal \ac{ec} mode selection}
As described above, in the sub-section~\ref{Section5:ECCandidates}, the proposed method includes four \ac{ec} candidates, which result in different recovered frames for each lost frame (simulated).
As previously described in the sub-section~\ref{Section5:ECCandidates}, the proposed method allows to choose between four \ac{ec} candidates, which result in different recovered frames for each lost frame (simulated).
The lost frame is partitioned into quadtree structures, each one forming a set of variable size \acs{ecu}.
Then, finding the optimal \ac{ec} method for each \ac{ecu} is formulated as a rate-distortion cost minimisation. To solve the aforementioned problem, a Lagrangian optimisation is used~\cite{LagragianMultiplier1982} to achieve the best trade-off between reconstruction quality and signalling overhead.
Then, the process of finding the optimal \ac{ec} method for each \ac{ecu} is formulated as a rate-distortion cost minimisation. To solve such problem, a Lagrangian optimisation is used~\cite{LagragianMultiplier1982} to achieve the best trade-off between reconstruction quality and signalling overhead.
Furthermore, image saliency driven by visual attention modelling is used in the optimisation process to weight the cost of the overhead information of each \ac{ecu}, according to the visual relevance of its content.
% \HIGHLIGHT{%
@@ -296,19 +316,25 @@ Furthermore, image saliency driven by visual attention modelling is used in the
% \label{eq:Lagrangian}
% \end{align}
The optimal \ac{ec} mode $m_{l,i}^*$, for the $i_{th}$ \ac{ecu} which size is defined by the quadtree partitioning level $l$, is obtained by minimising the following lagrangian cost:
\begin{equation}\label{eq:optdepth}
The optimal \ac{ec} mode $m_{l,i}^*$, for the $i_{th}$ \ac{ecu} of size defined by the quadtree partitioning level $l$, is obtained by minimising the following lagrangian cost:
\begin{equation}\label{EQ6:OptimalDepth}
\small
\newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
% (m^*,l^*) = \argmin_{m,l} \left\{ D(m,l) + \left(1+\frac{t_0-t_I}{T_{I}}\right) \lambda R(m,l) \right\},
% l^* = \argmin_{l=1..3} \left\{ D(l) + \lambda \left(1-\frac{t_I-t_c}{T_{\ac{ird}}}\right) R(l) \right\},
m_{l,i}^* = \argmin\limits_{\footnotesize \substack{l\in\left\{0...3\right\} \\ i\in\left\{1...4^l\right\}}}\left\{ D(m_{l,i}) + \left(1+\frac{t_0-t_I}{T_{I}}\right) \lambda R(m_{l,i}) \right\},
m_{l,i}^* = \argmin\limits_{\footnotesize \substack{l\in\left\{0...3\right\} \\ i\in\left\{1...4^l\right\}}}\left\{ D(m_{l,i}) + \left(1+\frac{t_C-t_I}{T_{I}}\right) \lambda R(m_{l,i}) \right\},
\end{equation}
where $R(m_{l,i})$ represents the number of overhead bits required to encode the \ac{ec} mode, and $D(m_{l,i})$ is sum of absolute error of the \ac{ecu}.
In equation \ref{eq:optdepth}, besides the parameter $\lambda$ used to trade-off the cost of distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$ according to the duration of the error propagation. This is given by the ratio between the temporal distance from the current frame ($t_0$) to the next refresh frame ($t_I$) and the period of refresh frames ($T_{I}$). This weight varies between $\left[0;1 \right]$, achieving the lowest value for intra-coded frames and the maximum for the frame located immediately before an intra-coded one.
Moreover, a dynamic value for $\lambda$ is used to adjust the amount of overhead across the frame, according to the visual relevance of each image region given by a saliency map.
In equation \ref{EQ6:OptimalDepth}, besides the parameter $\lambda$ used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$ according to the duration of the error propagation. As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques} this method uses the ratio between the temporal distance from the current frame ($t_C$) to the next refresh frame ($t_I$) and the period of refresh frames ($T_{I}$).
%This weight varies between $\left[0;1 \right]$, achieving the lowest value for intra-coded frames and the maximum for the frame located immediately before an intra-coded one.
This behaviour follows the fact that when a lost frame is closer to the next intra-coded frame, error propagation is lower. Thus, it is not worthwhile to spend too many overhead bits to reduce it. This is accomplished by increasing the cost of such bits according to the temporal distance between the missing frame and the next I frame.
Moreover, a dynamic value for $\lambda$ is also used to adjust the amount of overhead within the frame, according to the visual relevance of each image region given by a saliency map.
The algorithm described above results in \acs{ecu} with variable size, which is defined by the depth of the quad-tree structure, i.e., level $l$. The partition size is jointly optimised along with the best \ac{ec} candidate method using Equation~\ref{EQ6:OptimalDepth}. Alternatively, a fixed \ac{ecu} size can also be used by defining a fixed depth for all quad-tree structures, i.e., constant $l$ for the entire frame. This can be achieved by using Equation~\ref{EQ6:OptimalDepth} to solely optimise the \ac{ec} methods for each \ac{ecu}.
Although this reduces the overall complexity of the proposed method, it leads to a non-optimal solution which may required additional overhead. In Section~\ref{Section5:ECSignallingPerformance} a comparison between a dynamic and fixed approach is provided.
\subsection{Saliency detection}
@@ -325,8 +351,8 @@ On the one hand, higher values of saliency (i.e. regions with higher visual rele
% \lambda = \lambda_{0} + \frac{\Delta S}{S_{\ac{ctu}} - \mean S},
% \lambda = \lambda_{0} e^{\frac{\mean S - S_{\ac{ctu}}}{\Delta S}}, \Delta S > 0,
\lambda=\begin{cases}
\lambda_{0} e^{\frac{\mean S - s}{\Delta S}}, & \text{if $\Delta S > 0$}.\\
\lambda_{0}, & \text{no saliency used}.
\lambda_{0} e^{\frac{\mean S - s}{\Delta S}}, & \text{if $\Delta S > 0$},\\
\lambda_{0}, & if \Delta S=0 \text{ (no saliency used)},
\end{cases}
\end{equation}
where $s$ is the saliency value of the current \ac{ctu}, $\mean S$ is the average saliency value and $\Delta S$ is the difference between the maximum and minimum.
@@ -363,22 +389,22 @@ One should note that the method is not dependent on the existence of a saliency
Figure~\ref{FIG5:signalling} illustrates the structure of the signalling information for each \ac{ctu} comprising a quadtree of \acs{ecu}.
In this structure, one bit is used for the $split$ flag and two bits for the $ecm$ mode associated to each \ac{ecu}.
In this structure, one bit is used for the $split$ flag and two bits for the \ac{ec} mode associated to each \ac{ecu} ($ecm \in \{M1, M2, M3, M4\}$).
The $split$ flag indicates whether the \ac{ecu} is further split into the next~level~($split=1$) or \ac{ecu} partitioning stops at the current level ($split=0$) and the corresponding $ecm$ modes are inserted.
Since no split is necessary at the highest level, then $split=1$ followed by the four $ecm$ modes for each \ac{ecu}.
This structure allows dynamic selection of different levels within each frame.
This structure allows dynamic selection of different values for the level of the quad-tree partitioning ($l$) within each frame.
Adaptive arithmetic coding~\cite{ArithmeticCodingJun1994} is used to efficiently encode both the $split$ and $ecm$ symbols and further reduce the overhead. Note that the \ac{sei} information uses much less bitrate than the video signal, thus it is very unlikely to be hit during transmission.
\subsection{Enhancement-layer concealment residue}
\subsection{Enhancement-layer: \ac{ec} residue coding}
As described earlier, in order to allow further improvement of recovered slices, besides the best \ac{ec} mode, the proposed encoding scheme also encodes a controlled amount of \ac{ec} residue using the scalable extension of \ac{hevc} (see Figure~\ref{FIG5:BlockDiagram}).
As described earlier, in order to allow further improvement of recovered slices, besides the best \ac{ec} mode, the proposed encoding scheme also allows encoding a controlled amount of \ac{ec} residue using the scalable extension of \ac{hevc} (see Figure~\ref{FIG5:BlockDiagram}).
The enhancement layer encodes the difference between the coded frame ($\hat f$) and its counterpart recovered by using the optimal \ac{ec} mode ($\tilde f$).
Note that, the higher the accuracy of the best \ac{ec} mode, the lower the
amount of bits spent in the enhancement layer.
At the decoder-side, the reconstruction of a missing frame ($\tilde f$) follows three steps:
\begin{enumerate}
\item the missing frame is reconstructed using the optimal signalled \ac{ec} methods for each \ac{ecu} ($\tilde f'$);
\item the missing frame is reconstructed using the optimal \ac{ec} methods for each \ac{ecu} $\tilde f'$ received from the encoder;
\item the enhancement layer residue is decoded ($e^c$);
\item the recovered frame is obtained as follows:
\begin{equation}
@@ -386,10 +412,10 @@ At the decoder-side, the reconstruction of a missing frame ($\tilde f$) follows
\end{equation}
\end{enumerate}
The additional residue received from the enhancement layer is particularly useful to improve the quality of those \acs{ctu} where the \ac{ec} strategies are not able to produce very accurate reconstruction.
The additional residue decoded from the enhancement layer is particularly useful to improve the quality of those \acs{ctu} where the \ac{ec} strategies are not able to produce very accurate reconstruction.
Moreover, since the enhancement layer only carries residual information the overhead is affordable.
Note that the generated bitstream is compliant with the standard, as the error concealment signalling is encapsulated in \ac{sei} \ac{nalu} while the error concealment residue is carried in \acs{nalu} related with the scalable extension. Then, an adapted decoder, capable of decoding both the \ac{sei} \acs{nalu} and the scalable \acs{nalu} takes full benefit from the proposed method. Nevertheless, any standard \ac{hevc} decoder is also able to reconstruct the video, because it simply ignores the extra \acs{nalu}.
Note that the generated bitstream is compliant with the standard, as the \ac{ec} signalling is encapsulated in \ac{sei} \ac{nalu}, while the error concealment residue is carried in \acs{nalu} related with the scalable extension. Then, an adapted decoder, capable of decoding both the \ac{sei} \acs{nalu} and the scalable \acs{nalu} is also able to take full benefit from the proposed method. Nevertheless, any standard \ac{hevc} decoder is also able to reconstruct the video, because it simply ignores the extra \acs{nalu}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -398,16 +424,19 @@ Note that the generated bitstream is compliant with the standard, as the error c
\section{Error concealment signalling evaluation}
\label{Section5:ECSignallingPerformance}
This section presents the performance of the proposed \ac{ec}-aware optimisation scheme using the \ac{hevc} standard as the basis of the encoding framework.
Two different evaluations are presented. In the former, only robust \ac{ec} of intra-frames is evaluated, while in the latter the proposed \ac{ec}-aware optimisation is applied to all frames corresponding to a generic robust video transmission using optimal \ac{ec}.
This section presents the performance of the proposed \ac{ec}-aware optimisation scheme using the \ac{hevc} reference encoder as the basis of the encoding framework.
Two different types of performance evaluation are presented.
In the first one, only robust \ac{ec} of intra-frames is evaluated, while in the second one the proposed \ac{ec}-aware optimisation is applied to all frames corresponding to a generic robust video transmission using optimal \ac{ec}.
\subsection{Intra-frame EC optimisation}
\label{SEC4:IntraPerformance}
As mention before, in this section the evaluation of the proposed method is performed only for intra frames to study their impact on error propagation. As intra coding units result in higher amount of bits, they also need a higher number of packets, making them more prone to transmission errors and losses.
Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets corresponds to intra coded slices.
%Also, by limiting the proposed method to intra frames, the amount of required overhead is reduced.
In this section the performance of the proposed method is evaluated only for intra frames to demonstrate its efficiency in reducing error propagation.
Since intra \acs{cu} result in higher amount of bits, they also require a higher number of packets, making them more prone to transmission errors and losses.
Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets correspond to intra coded slices, which is a non-negligible probability of one of then being hit by transmission errors.
Moreover, by limiting the proposed method to intra frames, the amount of required overhead is reduced.
% \begin{table}[tb]
% \newcommand{\somespaces}{}
@@ -435,117 +464,113 @@ Results indicate that for an intra-period of 16 frames, approximately 42\% of th
% These results show that all error concealment candidates are used, which indicates that all of them are found relevant by the proposed method to recover the missing frames.
% The candidate methods based on the estimated motion field (i.e., M1, M2) are selected more often because they provide more accurate clues about the scene's motion than the others (i.e., M3, M4).
\subsubsection{Overhead analysis}
\begin{table}[tb]
\newcommand{\somespaces}{}
\centering
\caption{Percentage of overhead (\%)}
\label{tab:redundancy}
\TABLESVSPACE
\footnotesize
% \resizebox{\columnwidth}{!}{
\begin{tabular}{p{3.2cm}|x{2cm}|x{2cm}|x{3.5cm}}
\toprule
\multirow{2}{*}{\vbox{\centering \textbf{Sequence}}}
& $32\times32$ & $16\times16$ & \PROP \\
& \textbf{Level: 1} & \textbf{Level: 2} & \textbf{Level: dynamic} \\
\hline
% Basketball Drill & 0.08 \ 0.11 & 0.27 \ 0.40 & 0.08 \ 0.60 \\
Basketball Drill & 0.08 & 0.27 & 0.08 \\
\hline
% Race Horses & 0.16 & 0.65 & 0.08 \\
% \hline
Kendo & 0.27 & 1.05 & 0.19 \\
\hline
Park Scene & 0.42 & 1.72 & 0.05 \\
\hline
People On Street & 0.06 & 0.25 & 0.11\\
\hline
Traffic & 0.50 & 1.81 & 0.06 \\
%\hline
%Jockey & 0.38 & 1.53 & 0.24 \\
\bottomrule
\end{tabular}
% }
\end{table}
Table \ref{tab:redundancy} shows the overhead introduced by the proposed method and also for fixed size partitions of $32\times32$ and $16\times16$, i.e., $Depth=1$ and $Depth=2$. While the maximum overhead of these fixed partitions is \HIGHLIGHT{$1.81\%$} (Traffic) the proposed dynamic partitioning method is able to reduce the overhead to a maximum of \HIGHLIGHT{$0.19\%$} (Kendo).
% This is considered acceptable, given the gains obtained under lossy transmission.
\subsubsection{Quality evaluation}
%\subsubsection{Quality evaluation}
\begin{figure}[!t]
\centering
\input{./042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameLowDelay.tex}
\caption{Error propagation using Low-Delay for the Basketball Drill sequence.}
\label{FIG5:singlelosslowdelay}
\vspace{2mm}
\input{./042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameRandomAccess.tex}
\caption{Error propagation using Random-Access for the Park Scene sequence.}
\label{FIG5:singlelossramdon}
\FIGSVSPACE
\centering
\input{./042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameLowDelay.tex}
\caption{Error propagation using \ac{ld} for the Basketball Drill sequence.}
\label{FIG5:singlelosslowdelay}
\vspace{2mm}
\input{./042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameRandomAccess.tex}
\caption{Error propagation using \ac{ra} for the Park Scene sequence.}
\label{FIG5:singlelossramdon}
\FIGSVSPACE
\end{figure}
Since a relevant performance indicator is the error propagation over temporally predicted frames, this was evaluated using a single frame loss.
Figures \ref{FIG5:singlelosslowdelay} and \ref{FIG5:singlelossramdon} show the \ac{psnr} results when frame $\#16$ is missing, for Low-Delay and Random-Access encoding configurations, respectively. these Figures show the results for the proposed method (\PROP), the motion-copy method (\MC), and also for a fixed partition size ($16\times16$).
These results indicate that the proposed method (\PROP) outperforms the reference one and it is able to achieve quality gains up to \HIGHLIGHT{2 dB} for Low-Delay configuration (see Figure~\ref{FIG5:singlelosslowdelay}), decreasing the error propagation and increasing the quality of the affected frames up to \HIGHLIGHT{3 dB}.
The use of Random-Access configuration leads to higher quality degradation than LD, due the higher temporal distance of the reference frame used for concealment (frame $\#8$ is used to conceal lost frame~$\#16$). Note that when Frame \# 16 is lost, all frames within the range \#9 to \#31 (i.e., next I-frame) are affected by error propagation.
Since a relevant performance indicator is the error propagation over temporally predicted frames, this was evaluated using a single intra-frame loss.
Figures \ref{FIG5:singlelosslowdelay} and \ref{FIG5:singlelossramdon} show the \ac{psnr} results when frame Intra-frame $\#16$ is missing, for \ac{ld} and \ac{ra} encoding configurations, respectively. These Figures show the results for the proposed method (\PROP), the motion-copy method (\MC), and also an alternative proposed method using fixed \ac{ecu} size of $16\times16$ pixels.
These results indicate that the proposed method (\PROP) outperforms the reference one and it is able to achieve quality gains up to \HIGHLIGHT{2 dB} for \ac{ld} configuration (see Figure~\ref{FIG5:singlelosslowdelay}), decreasing the error propagation and increasing the quality of the frames affected from the loss of the intra frame, by up to \HIGHLIGHT{3 dB}.
The use of \ac{ra} configuration leads to higher quality degradation than LD, due the higher temporal distance of the reference frame used for \ac{ec} (Frame $\#8$ is used to recover the lost Frame~$\#16$). Note that when Intra-frame \#16 is lost, all frames within the range \#9 to \#31 (i.e., next I-frame) are affected by error propagation.
Nevertheless, the proposed method still achieves similar quality gains (approximately 2 dB).
Moreover, for both configurations, even when the overhead is reduced, by moving from fixed partition to dynamic partition size, the proposed method presents only a small reduction in the reconstruction quality. This indicates that using dynamic partitioning allows to achieve a better trade-off between the amount of overhead and the error concealment performance.
Further tests were performed to evaluate the effectiveness of the proposed method under different loss rates. In these experiments the packets corresponding to intra frames were randomly discarded to simulate loss events, at different \acs{plr} rates with an average burst length of 5 packets.
For each test condition 25 trials were performed and the average quality of the frames affected by errors (the lost frame and dependent ones) is presented in Table~\ref{tab:randomloss}. The table also shows the \ac{psnr} variance value in parentheses and the difference to the \ac{psnr} obtained with a reference \ac{hevc} decoder.
% The absolute \ac{psnr} value is shown for the reference case (\MC), while the \ac{psnr} difference is presented for the proposed method.
The results confirm the superiority of the proposed method, outperforming the reference \ac{hevc} for both low and high \ac{plr}. Using the \ac{ld} coding configuration, the proposed method is able to achieve quality gains of \HIGHLIGHT{0.88 dB} for 3\% of \ac{plr} and \HIGHLIGHT{1.50 dB} for 10\% of \ac{plr}. In case of RA, higher quality gains are achieved, leading to an average improvement of \HIGHLIGHT{3.44 dB} for 10\% of \ac{plr}.
It is also worthwhile to notice that the \ac{psnr} variance obtained with the \PROP method is lower, which indicates lower quality variations in the video segments affected by errors.
%The quality gains achieved by the proposed method are similar to those achieved by previous works~\cite{ConcealmentIntraDataHidng2009,ConcealmentSignalling2008} for the same \acs{plr}. However, these require a higher amount of overhead (approximately $3\%$), thus the proposed method achieves a better trade-off between quality and overhead.
\begin{table}[!t]
\newcommand{\somespaces}{}
\centering
% \caption{Average quality (\ac{psnr}) of the frames affected by errors and its variance under random packet loss}
\caption{Average \ac{psnr} and variance of affected frames}
\label{tab:randomloss}
\TABLESVSPACE
\small
\footnotesize
\newcommand{\colwidth}{1.6cm}
\begin{tabular}{p{2.8cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}}
\toprule
\multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{3}{c|}{\textbf{PLR=3\%}} & \multicolumn{2}{c}{\textbf{PLR=10\%}}\\
& \MC & \PROP & $\Delta$PSNR& \MC & \PROP & $\Delta$PSNR \\
\midrule
\multicolumn{7}{c}{Low-delay configuration} \\ [2pt]
\hline
Basketball Drill & 33.15 (0.42) & 34.26 (0.24) & \optbold{+1.11} & 32.97 (0.47) & 35.90 (0.27) & \optbold{+2.93}\\ \hline
% Race Horses & 30.78 (1.32) & 31.55 (0.94) & \optbold{+0.77} & 30.26 (1.55) & 31.12 (1.20) & \optbold{+0.86}\\ \hline
Kendo & 34.33 (1.28) & 36.57 (1.11) & \optbold{+2.24} & 33.84 (1.26) & 37.49 (1.05) & \optbold{+3.65}\\ \hline
Park Scene & 35.22 (0.12) & 35.33 (0.08) & \optbold{+0.11} & 34.75 (0.11) & 34.96 (0.09) & \optbold{+0.21}\\ \hline
People On Street & 34.36 (0.32) & 34.98 (0.22) & \optbold{+0.62} & 32.17 (0.65) & 33.45 (0.50) & \optbold{+1.28}\\ \hline
Traffic & 36.77 (0.48) & 36.88 (0.57) & \optbold{+0.11} & 36.04 (0.30) & 36.31 (0.27) & \optbold{+0.27}\\ \hline
Jockey & 36.00 (1.14) & 37.20 (1.05) & \optbold{+1.20} & 35.44 (1.21) & 36.74 (1.18) & \optbold{+1.30}\\ \hline
\textbf{Average} & \textbf{34.37 (0.73)} & \textbf{35.25 (0.60)} & \textbf{+0.88} & \textbf{33.64 (0.79)} & \textbf{35.14 (0.65)} & \textbf{+1.50} \\
% \midrule
% \multicolumn{7}{c}{Random-access configuration} \\ [2pt]
% \hline
% Basketball Drill & 30.22 (8.46) & 30.81 (7.66) & \optbold{+0.59} & 29.19 (7.60) & 29.84 (6.88) & \optbold{+0.65} \\ \hline
% % Race Horses & 27.35 (8.02) & 29.92 (5.89) & \optbold{+2.57} & 25.75 (7.76) & 28.61 (6.14) & \optbold{+2.86} \\ \hline
% Kendo & 27.45 (23.9) & 32.38 (18.6) & \optbold{+4.93} & 25.22 (28.3) & 36.98 (22.4) & \optbold{+11.8} \\ \hline
% Park Scene & 33.91 (1.40) & 34.46 (1.07) & \optbold{+0.55} & 31.38 (1.85) & 32.50 (1.48) & \optbold{+1.12} \\ \hline
% People On Street & 29.30 (5.12) & 31.44 (3.17) & \optbold{+2.14} & 24.89 (5.37) & 27.69 (4.09) & \optbold{+2.80} \\ \hline
% Traffic & 35.64 (1.34) & 38.12 (1.24) & \optbold{+2.48} & 32.85 (1.88) & 35.60 (1.40) & \optbold{+2.75} \\ \hline
% Jockey & 32.35 (12.7) & 34.06 (8.60) & \optbold{+1.71} & 29.96 (12.5) & 32.11 (8.90) & \optbold{+2.15} \\ \hline
% \textbf{Average} & \textbf{30.89 (5.64)} & \textbf{33.03 (2.14)} & \textbf{+2.14} & \textbf{28.46 (5.68)} & \textbf{31.90 (4.44)} & \textbf{+3.44}\\
\bottomrule
\end{tabular}
% }
\begin{table}[tbp]
\newcommand{\somespaces}{}
\centering
\caption{Average quality (\ac{psnr}) of the frames affected by errors and its variance under random packet loss.}
%\caption{Average \ac{psnr} and variance of affected frames}
\label{tab:randomloss}
\TABLESVSPACE
\small
\footnotesize
\newcommand{\colwidth}{1.6cm}
\begin{tabular}{p{2.8cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}}
\toprule
\multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{3}{c|}{\textbf{PLR=3\%}} & \multicolumn{2}{c}{\textbf{PLR=10\%}}\\
& \MC & \PROP & $\Delta$PSNR& \MC & \PROP & $\Delta$PSNR \\
\midrule
\multicolumn{7}{c}{Low-delay configuration} \\ [2pt]
\hline
Basketball Drill & 33.15 (0.42) & 34.26 (0.24) & \optbold{+1.11} & 32.97 (0.47) & 35.90 (0.27) & \optbold{+2.93}\\ \hline
Race Horses & 30.78 (1.32) & 31.55 (0.94) & \optbold{+0.77} & 30.26 (1.55) & 31.12 (1.20) & \optbold{+0.86}\\ \hline
Kendo & 34.33 (1.28) & 36.57 (1.11) & \optbold{+2.24} & 33.84 (1.26) & 37.49 (1.05) & \optbold{+3.65}\\ \hline
Park Scene & 35.22 (0.12) & 35.33 (0.08) & \optbold{+0.11} & 34.75 (0.11) & 34.96 (0.09) & \optbold{+0.21}\\ \hline
People On Street & 34.36 (0.32) & 34.98 (0.22) & \optbold{+0.62} & 32.17 (0.65) & 33.45 (0.50) & \optbold{+1.28}\\ \hline
Traffic & 36.77 (0.48) & 36.88 (0.57) & \optbold{+0.11} & 36.04 (0.30) & 36.31 (0.27) & \optbold{+0.27}\\ \hline
Jockey & 36.00 (1.14) & 37.20 (1.05) & \optbold{+1.20} & 35.44 (1.21) & 36.74 (1.18) & \optbold{+1.30}\\ \hline
\textbf{Average} & \textbf{34.37 (0.73)} & \textbf{35.25 (0.60)} & \textbf{+0.88} & \textbf{33.64 (0.79)} & \textbf{35.14 (0.65)} & \textbf{+1.50} \\
\midrule
\multicolumn{7}{c}{Random-access configuration} \\ [2pt]
\hline
Basketball Drill & 30.22 (8.46) & 30.81 (7.66) & \optbold{+0.59} & 29.19 (7.60) & 29.84 (6.88) & \optbold{+0.65} \\ \hline
Race Horses & 27.35 (8.02) & 29.92 (5.89) & \optbold{+2.57} & 25.75 (7.76) & 28.61 (6.14) & \optbold{+2.86} \\ \hline
Kendo & 27.45 (23.9) & 32.38 (18.6) & \optbold{+4.93} & 25.22 (28.3) & 36.98 (22.4) & \optbold{+11.8} \\ \hline
Park Scene & 33.91 (1.40) & 34.46 (1.07) & \optbold{+0.55} & 31.38 (1.85) & 32.50 (1.48) & \optbold{+1.12} \\ \hline
People On Street & 29.30 (5.12) & 31.44 (3.17) & \optbold{+2.14} & 24.89 (5.37) & 27.69 (4.09) & \optbold{+2.80} \\ \hline
Traffic & 35.64 (1.34) & 38.12 (1.24) & \optbold{+2.48} & 32.85 (1.88) & 35.60 (1.40) & \optbold{+2.75} \\ \hline
Jockey & 32.35 (12.7) & 34.06 (8.60) & \optbold{+1.71} & 29.96 (12.5) & 32.11 (8.90) & \optbold{+2.15} \\ \hline
\textbf{Average} & \textbf{30.89 (5.64)} & \textbf{33.03 (2.14)} & \textbf{+2.14} & \textbf{28.46 (5.68)} & \textbf{31.90 (4.44)} & \textbf{+3.44}\\
\bottomrule
\end{tabular}
% }
\end{table}
Further tests were performed to evaluate the effectiveness of the proposed method under more different loss rates. In the experiments the packets corresponding to intra frames were randomly discarded to simulate loss events, at different \acs{plr} rates with an average burst length of 5 packets.
For each test condition 25 trials were performed and the average quality of the frames affected by errors (the lost frame and dependent ones) is presented in Table~\ref{tab:randomloss}. The table also shows the \ac{psnr} variance value in parentheses and the difference to the \ac{psnr} obtained with a reference \ac{hevc} decoder
% The absolute \ac{psnr} value is shown for the reference case (\MC), while the \ac{psnr} difference is presented for the proposed method.
The results confirm the superiority of the proposed method, outperforming the reference \ac{hevc} for both low and high \ac{plr}. Using Low-Delay coding configuration, the proposed method is able to achieve quality gains of \HIGHLIGHT{0.88 dB} for 3\% of \ac{plr} and \HIGHLIGHT{1.50 dB} for 10\% of \ac{plr}. In case of RA, higher quality gains are achieved, leading to an average improvement of \HIGHLIGHT{3.44 dB} for 10\% of \ac{plr}.
It is also worthwhile to notice that the \ac{psnr} variance obtained with the \PROP method is lower, which indicates lower quality variations in the video segments affected by errors.
%\subsubsection{Overhead analysis}
%The quality gains achieved by the proposed method are similar to those achieved by previous works~\cite{ConcealmentIntraDataHidng2009,ConcealmentSignalling2008} for the same \acs{plr}. However, these require a higher amount of overhead (approximately $3\%$), thus the proposed method achieves a better trade-off between quality and overhead.
Subsequently, the percentage of overhead bitrate introduced by the proposed method is compared against fixed \ac{ecu} sizes of $32\times32$ and $16\times16$, i.e., $Level=1$ and $Level=2$.
These results are presented in Table \ref{TAB6:RedundancyIntra} and were obtained for a total bitrate as shown in Table~\ref{TableTestSequences}.
It is noticeable from the results that reducing the \ac{ecu} size (see results for $32\times32$ and $16\times16$) increases the overhead ratios as more symbols are transmitted (one for each \ac{ecu}).
While the maximum overhead of these fixed partitions is \HIGHLIGHT{$1.81\%$} (Traffic) the proposed dynamic partitioning method is able to reduce the overhead to a maximum of \HIGHLIGHT{$0.19\%$} (Kendo).
Moreover, comparing the results of Table \ref{TAB6:RedundancyIntra} with the video quality results obtained in case of packet losses reveals that, in the proposed method, only a small reduction in the reconstruction quality even with a significantly lower overhead.
This indicates that using dynamic partitioning allows to achieve a better trade-off between the amount of overhead and the \c{ec} performance.
In summary, the proposed technique is able to enhance the performance of the standard decoding of intra coded frames by finding the best error concealment method at the encoder and sending such information with a small amount of signalling overhead.
\begin{table}[tbp]
\newcommand{\somespaces}{}
\centering
\caption{Percentage of overhead (\%)}
\label{TAB6:RedundancyIntra}
\TABLESVSPACE
\footnotesize
% \resizebox{\columnwidth}{!}{
\begin{tabular}{p{3.2cm}|x{2cm}|x{2cm}|x{3.5cm}}
\toprule
\multirow{2}{*}{\vbox{\centering \textbf{Sequence}}}
& $32\times32$ & $16\times16$ & \PROP \\
& \textbf{Level: 1} & \textbf{Level: 2} & \textbf{Level: dynamic} \\
\hline
Basketball Drill & 0.08 & 0.27 & 0.08 \\ \hline
Race Horses & 0.16 & 0.65 & 0.08 \\ \hline
Kendo & 0.27 & 1.05 & 0.19 \\ \hline
Park Scene & 0.42 & 1.72 & 0.05 \\ \hline
People On Street & 0.06 & 0.25 & 0.11\\ \hline
Traffic & 0.50 & 1.81 & 0.06 \\ %\hline
%Jockey & 0.38 & 1.53 & 0.24 \\
\bottomrule
\end{tabular}
% }
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% SECTION
@@ -554,7 +579,9 @@ In summary, the proposed technique is able to enhance the performance of the sta
% \subsection{Overall performance evaluation}
\label{SEC4:OverallPerformance}
In this section the proposed \ac{ec}-aware optimisation scheme is evaluated for a generalised robust video transmission of intra and inter coded frames. This section is organised in the following subsections:
In this section the proposed \ac{ec}-aware optimisation scheme is evaluated for a generalised robust video transmission of both intra and inter coded frames.
The experimental results presented in this section were obtained by applying the proposed technique to all video frames.
This section is organised in the following subsections:
\begin{itemize}
\item evaluation of the usage distribution of each \ac{ec} mode;
\item performance evaluation of the \ac{ec} optimal mode decision (without using the enhancement layer);
@@ -569,7 +596,8 @@ In this section the proposed \ac{ec}-aware optimisation scheme is evaluated for
The \ac{ec} mode usage distribution is an indicator of the usefulness of each \ac{ec} mode.
Table \ref{tab:useratio} shows the average $Level$ selected for the \ac{ecu} quadtree and the usage percentage of each \ac{ec} mode.
It is noticeable from the results that the proposed method leads to low average $Level$ values (i.e., less than 1), which implicitly results in less signalling information. This also justifies the low overhead shown in Table~\ref{tab:redundancy_signalling}.
It is noticeable from these results that the proposed method leads to low average $Level$ values (lower than 1), resulting in larger \acs{ecu}, which requires a lower amount of \ac{ecu} to be transmitted than if higher values of $Level$ were chosen. This implicitly results in low signalling information.
This also justifies the low overhead shown in Table~\ref{tab:redundancy_signalling}.
\begin{table}[b]
\newcommand{\somespaces}{}
@@ -596,30 +624,31 @@ Tennis & 1.17 & 25.27 & 30.03 & 23.22 & 21.48 \\ %\hline
\end{tabular}
\end{table}
The results in Table \ref{tab:useratio} show that all \ac{ec} modes are effectively used, which indicates that all of them are found relevant by to recover the missing data.
The \ac{ec} candidate methods based on the estimated motion field (i.e., M1, M2) are selected more often because they provide more accurate representation of the scene's motion than the others (i.e., M3, M4).
The results in Table \ref{tab:useratio} also show that all \ac{ec} modes are effectively used, which indicates that all of them are found relevant to recover the missing data.
The \ac{ec} candidate methods based on motion field estimation (i.e., M1, M2) are selected more often because they provide more accurate representation of the scene's motion than the others (i.e., M3, M4).
\subsubsection{Evaluation of optimal \ac{ec} mode decision}
\label{sec:ECModeSelectionPerformance}
\input{\ConcealmentAwareDir/Results/TableSignallingPerformance.tex}
The evaluation of the proposed saliency-weighted \ac{ec} mode decision (\DYNLAMBDA) was carried out under loss events and compared with the reference methods. The video quality was evaluated by using two quality metrics: \ac{psnr} and a \ac{fsim}~\cite{FSIM_TIP2011} in order to validate the performance of proposed method in a consolidated manner.
Realistic conditions were used for transmission, by allowing independent losses for any of type of \acs{nal}, hence there is no guarantee that the decoder conditions are always equal to those at the encoder, which means that a signalling mismatch may occur.
The evaluation of the proposed saliency-weighted \ac{ec} mode decision (\DYNLAMBDA) was carried out under random loss events and compared with the reference methods. The video quality was evaluated by using two quality metrics: \ac{psnr} and a \ac{fsim}~\cite{FSIM_TIP2011} in order to validate the performance of the proposed method in a consolidated manner.
Realistic conditions were used for transmission, by allowing independent losses for any of type of \acs{nal}, hence there is no guarantee that the decoder conditions are always equal to those at the encoder, which means that sub-optimal \ac{ec} may be used at the decoder.
The results obtained without saliency weighting are also shown for comparison (\FIXEDLAMBDA). In this case a constant $\lambda_0=10$ is used.
Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM} show the average quality under different \acs{plr} for the Low-delay configuration.
Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM} show the average quality under different \acs{plr} for the \ac{ld} configuration.
Table~\ref{TAB4:SignallingPSNR} shows the \ac{psnr} and weighted \ac{psnr} (\ac{wpsnr}) between parentheses.
The \ac{mse} weights used in the \ac{wpsnr} are defined by the saliency values~\cite{SaliencyQualityMetric2011}.
Table~\ref{TAB4:SignallingFSIM} shows the results obtained for the \ac{fsim} quality metric.
Note that, the absolute value is shown for the reference case (\REF), while the differential values are shown for the remaining methods.
From the \ac{psnr} results shown in Table~\ref{TAB4:SignallingPSNR}, one may conclude that the proposed method (\DYNLAMBDA) consistently outperforms the reference ones for both low and high \acs{plr}. Average \ac{psnr} gains up to 2.95~dB for PLR=3\% and 2.58~dB for PLR=10\% are achieved. It is also worthwhile to notice that higher performance is obtained for sequences with high motion, such as Basketball Drill, which leads to \ac{psnr} gains up 4.82~dB Moreover, the results of Table~\ref{TAB4:SignallingFSIM} confirm the quality improvement achieved by the proposed method, leading to an increase of \ac{fsim} up to 4.32, which indicates that the proposed method is able to recover important image features with higher accuracy.
\input{\ConcealmentAwareDir/Results/TableSignallingPerformance.tex}
From the \ac{psnr} results shown in Table~\ref{TAB4:SignallingPSNR}, one may conclude that the proposed method (\DYNLAMBDA) consistently outperforms the reference ones for both low and high \acs{plr}. Average \ac{psnr} gains up to 2.95~dB for PLR=3\% and 2.58~dB for PLR=10\% are achieved. It is also worthwhile to notice that higher performance is obtained for sequences with high motion, such as Basketball Drill, which leads to \ac{psnr} gains up 4.82~dB.
Moreover, the results of Table~\ref{TAB4:SignallingFSIM} confirm the quality improvement achieved by the proposed method, leading to an increase of \ac{fsim} up to 4.32, which indicates that the proposed method is able to recover important image features with higher accuracy than the others.
Table~\ref{tab:redundancy_signalling} shows the percentage of overhead in the total rate with and without saliency-weighted optimisation, i.e., \FIXEDLAMBDA and \DYNLAMBDA, respectively.
As shown in the Table, the saliency-weighted optimisation reduces the amount of overhead by dynamically assigning unequal number of bits to image regions of different visual importance.
These results show that using an adaptive value of $\lambda$ based on the saliency values leads to an average overhead reduction of 28\% in comparison with the case where no saliency is used, i.e., constant $\lambda$.
These results show that using an adaptive value of $\lambda$, based on the saliency values, leads to an average overhead reduction of 28\% in comparison with the case where no saliency is used, i.e., constant $\lambda$.
It is worth to note that using the saliency-weighted optimisation to reduce the overhead does not compromise the video quality, as can be seen in the results shown in Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM}. The \ac{psnr} values for \FIXEDLAMBDA and \DYNLAMBDA are very close, but in overhead savings are significant for most sequences.
The achieved \ac{wpsnr} and \ac{fsim} results further demonstrate the effectiveness of the saliency-based optimisation, because the lower overhead does not affect the most relevant image regions, i.e., those with higher saliency values. Thus, the proposed method using a adaptive $\lambda$ (i.e., using saliency) is able to not only achieve higher \ac{ec} quality than the reference methods, but also higher quality gains than using a constant $\lambda$ (i.e., without saliency).
@@ -635,11 +664,7 @@ The figure shows the results obtained by decoding the stream using the proposed
\input{\ConcealmentAwareDir/Results/ECMQualitySingleloss.tex}
Figure \ref{FIG5:singlelossIntra} shows the \ac{psnr} for the same conditions but with a loss event on the intra-coded Frame \#16, in sequences Kimono and Tennis. These results reveal that the proposed method is able to improve the reconstruction of inter-coded frames and also outperforms the existing techniques whenever a intra-coded frame is missing.
%\HIGHLIGHT{Although in this frame loss event the recovery quality is lower for every method tested, the proposed approach is able to achieve lower error propagation and achieve quality gains up to 2 dB (see results for Tennis sequence) - ESTA FRASE NAO SE ENTENDE} <- Apenas com estes resultados esta frase nao faz sentido
These results indicate that the proposed method outperforms both reference methods since the time instant when the loss occurs. The lost frame itself is recovered with much higher quality than using the reference \ac{ec} methods. This is due to the use of the optimal \ac{ec} mode to recover each \ac{ecu}, following the \ac{sei} information received from the encoder.
These results also indicate that the proposed method outperforms both reference methods since the time instant when the loss occurs. The lost frame itself is recovered with much higher quality than using the reference \ac{ec} methods. This is due to the use of the optimal \ac{ec} mode to recover each \ac{ecu}, following the \ac{sei} information received from the encoder.
In this case, the proposed method is able to achieve quality gains up to \HIGHLIGHT{5~dB}, decreasing the error propagation and increasing the \ac{psnr} of affected frames by up to \HIGHLIGHT{6~dB}.
The visual impact of using the proposed \ac{ec}-aware encoding can be evaluated by observing the recovered frame (i.e., frame \#6) of Basketball Drill as an example. Figure~\ref{FIG5:RecoveredExample} shows that higher visual quality and less reconstruction artefacts are obtained in comparison with the \MC method. It is worthwhile to note that the proposed method is able to reconstruct the object edges with higher accuracy than the reference \MC.
@@ -666,8 +691,8 @@ The visual impact of using the proposed \ac{ec}-aware encoding can be evaluated
%\subsection{Quality evaluation}
As in the previous simulations, the packets were randomly discarded to simulate loss events, at different \acs{plr}.
The \ac{psnr} obtained with the proposed method is compared with the default method used by reference \ac{hevc} , i.e., frame-copy (\REF), motion-copy (\MC)~\cite{HEVCMCResidueDec2013}, and the method proposed in \cite{ROPERefGenJan2010}, which implements a robust scheme to minimise error propagation.
The following results were obtained using the same constant bitrate for each test condition.
The quality obtained with the proposed method is compared with the default method used by reference \ac{hevc} , i.e., frame-copy (\REF), motion-copy (\MC)~\cite{HEVCMCResidueDec2013}, and the method proposed in \cite{ROPERefGenJan2010}, which implements a robust scheme to minimise error propagation.
In this experiment both layers, i.e., video stream and enhancement layer with the EC residue, may be affected by errors.
\subsection{Quality evaluation}
@@ -675,8 +700,8 @@ The following results were obtained using the same constant bitrate for each tes
Tables~\ref{tab:concealmentResidueRandomLoss_PLR3}~and~\ref{tab:concealmentResidueRandomLoss_PLR10} present the average \ac{psnr} of decoded video for different \acs{plr} and different percentages of total overhead (i.e., signalling rate plus enhancement layer rate). The absolute \ac{psnr} value is shown for the reference case, while the \ac{psnr} difference is presented for the remaining methods.
These results show that transmitting the \ac{ec} residue in the enhancement layer further improves the reconstruction quality obtained when only the optimal \ac{ec} method is used (see previous section).
By only using 10\% of the total bit rate, the proposed \ac{ec}-aware coding scheme is able to outperform the reference \ac{hevc}, the \MC and the method proposed in \cite{ROPERefGenJan2010} for both Low-delay and Random-access configurations.
Using 20\% of the total bitrate the quality gains increase up to \HIGHLIGHT{6.90~dB} for the Basketball Drill sequence when comparing with the reference \ac{hevc}, achieving an average gain of \HIGHLIGHT{3.79~dB} for the Low-delay configuration (\HIGHLIGHT{1.33}~dB for the Random-access).
By only using 10\% of the total bit rate, the proposed \ac{ec}-aware coding scheme is able to outperform the reference \ac{hevc}, the \MC and the method proposed in \cite{ROPERefGenJan2010} for both \ac{ld} and \ac{ra} configurations.
Using 20\% of the total bitrate the quality gains increase up to \HIGHLIGHT{6.90~dB} for the Basketball Drill sequence when comparing with the reference \ac{hevc}, achieving an average gain of \HIGHLIGHT{3.79~dB} for the \ac{ld} configuration (\HIGHLIGHT{1.33}~dB for the \ac{ra}).
When comparing the results of Tables~\ref{tab:concealmentResidueRandomLoss_PLR3}~and~\ref{tab:concealmentResidueRandomLoss_PLR10} with the spatial and temporal information of each sequence (SI/TI in Table~\ref{TableTestSequences}) it is noticeable that higher quality gains are achieved for sequences with higher motion information, i.e., Basketball~Drill and Tennis sequences.
Moreover, for sequences with higher spatial information (e.g., BQSquare and People on Street) considerable quality gains are also achieved.
@@ -722,14 +747,14 @@ R = \abs{\frac{\Delta PSNR(\ac{plr}=10\%)}{\Delta PSNR(Error\text{-}free)}}.
The results shown in Table~\ref{tab:overall_gain_ratio}, demonstrate that the effectiveness of the \ac{ec}-aware coding scheme is higher than all other methods under comparison. In fact, not only the absolute \ac{psnr} difference (i.e., \ac{psnr} gain) is higher for the same \ac{plr}, but also the efficiency of the overhead is also higher. This means that the rate spent to encode the signalling information and the enhancement residue carry more useful information than the other methods.
For instance, in comparison with \textit{Ref \cite{DualFrameME2015}} the proposed method leads to higher quality degradation in the error-free case. However, for PLR=10\% it clearly outperforms \textit{Ref \cite{DualFrameME2015}} with a much higher value of $R$, indicating a far superior efficiency.
For instance, in comparison with the method in \cite{DualFrameME2015} the proposed method leads to higher quality degradation in the error-free case. However, for PLR=10\% it clearly outperforms the method in \cite{DualFrameME2015} with a much higher value of $R$, indicating a far superior efficiency.
Results in the table~\ref{tab:overall_gain_ratio} also show that the small loss of coding efficiency in error-free transmission is clearly surpassed by the quality gains under error-prone transmission conditions.
The methods in \cite{ROPERefGenJan2010,DualFrameME2015} are based on modified predictions, while the method in \cite{VPxResiliencePCS2016} uses redundant data without any further optimisation.
Both of these approaches require low additional computation complexity. However, as observed in the results shown in Table ~\ref{tab:overall_gain_ratio} the lower additional complexity also corresponds to lower quality gains in case of transmission errors.
The proposed method uses \ac{ec} techniques with different levels of computational complexity for each \ac{ecu}. Since an adaptive approach is used, the average complexity tends to be lower in comparison with other methods that rely on a fixed approach. Thus, less complex video decoding is obtained when using the proposed method when compared with other straightforward methods that always used the same \ac{ec} techniques for all \acs{ctu}.
The proposed method uses \ac{ec} techniques with different levels of computational complexity for each \ac{ecu}. Since an adaptive approach is used, the average complexity tends to be lower in comparison with other methods that rely on a fixed approach. Thus, less complex video decoding is obtained when using the proposed method when compared with other straightforward methods that always used the same \ac{ec} technique for all \acs{ctu}.
\begin{table}[t]
@@ -761,8 +786,8 @@ The proposed method uses \ac{ec} techniques with different levels of computati
\section{Summary}
\label{Section5:Summary}
In this chapter an error concealment-aware encoder is proposed by incorporating loss simulation at the encoder-side to enhance the error concealment efficiency at the decoder.
In this chapter an \ac{ec}-aware encoding scheme is proposed by incorporating loss simulation at the encoder-side to enhance the EC efficiency at the decoder.
% The proposed method relies on loss simulation at the encoder-side to optimise the error concealment proposed. Then, the optimal error concealment is signalled to the decoder.
The novelty of this method is to include optimal error concealment mode decision at the encoder, to be used by the decoder in case of data loss, and also to allow encoding of the error concealment residue using the enhancement layer of the \ac{hevc} standard.
The novelty of this method is to include optimal \ac{ec} mode decision at the encoder, to be used by the decoder in case of packet loss, and also to allow encoding of the \ac{ec} residue using the enhancement layer of the \ac{hevc} standard.
The overall results demonstrate that the proposed method is able to efficiently reconstruct lost frames and reduce the error propagation compared with other reference methods.
A consistent quality improvement is achieved at the cost of low overhead, which allows to use this method in broadcasting video services and applications where packet loss probability should not be ignored.
A consistent quality improvement achieved at the cost of low overhead, allows the use of this method in broadcasting video services and applications where packet loss probability is not negligible.
042_ConcealmentAwareEncoding/Figures/BlockDiagram.tikz
+4 -4
View File
@@ -45,7 +45,7 @@
%% NALU Loss and EC optimisation
\node [propBlock, above=2.2 of Encoder.west, anchor=south west, text width=3.8cm, minimum height=2.4cm] (ConcOpt) {Saliency-weighted EC Mode Decision};
\node [propBlock, above=2.2 of Encoder.west, anchor=south west, text width=3.8cm, minimum height=2.8cm] (ConcOpt) {Saliency-weighted EC Mode Decision};
\node [propBlock, below=.8 of ConcOpt.south east, anchor=north east, text width=1.7cm, minimum height=.9cm] (LossSim) {NALU Loss Simulation};
\path[arrow] (EncoderNAL -| LossSim.south) -- (LossSim) node[point] at (EncoderNAL -| LossSim) {};;
\path[arrow] (Decoder) -- (ConcOpt.south -| Decoder) node[pos=.6,coordinate] (p2) {} node[point] at (p2) {};
@@ -75,19 +75,19 @@
\path[arrow] (ConcOpt.east |- Signalling) -- (Signalling) node[above, pos=.45, yshift=-1] {\scriptsize M1...M4};
%% Input
\node[text width=.5cm, text centered] at (-1.7,4.5) (Input) {Input video};
\node[text width=.5cm, text centered] at (-1.7,4.9) (Input) {Input video};
\path[arrow] (Input) |- (Encoder) node[left,pos=.2] {$f$};
\path[arrow] (Input |- ConcOpt) -- (ConcOpt) node[point] at (Input |- ConcOpt) {};
%% TS Mux and output
\path (Encoder.south) -- (Encoder |- Signalling.north) node[midway] (middleRef) {}; % Find middle ref
\node [refBlock, right=8.5 of middleRef, text width=.7cm, minimum height=5.5cm] (NAL) {TS Mux};
\node [refBlock, right=8.5 of middleRef, text width=.7cm, minimum height=5.8cm] (NAL) {Mux};
\node [right of=NAL, node distance=1.5cm, coordinate] (Out) {};
\path[arrow] (EncoderNAL) -- (NAL.west |- EncoderNAL) node[above,pos=1, xshift=-.4cm] {VCL$_0$};
\path[arrow] (ScalableNAL) -- (NAL.west |- Scalable) node[above,pos=1, xshift=-.4cm] {VCL$_1$};
\path[arrow] (SignallingNAL) -- (NAL.west |- Signalling) node[above,pos=1, xshift=-.4cm] {SEI};
\path[arrow] (NAL) -- (Out) node[above,pos=1, xshift=-.2cm, text width=1.3cm] {Transport Stream};
\path[arrow] (NAL) -- (Out) node[above,pos=1, xshift=-.2cm, text width=1.3cm] {Video Stream};
\end{tikzpicture}
042_ConcealmentAwareEncoding/Figures/MotionVectorExtrapolation.tikz
+25 -60
View File
@@ -31,9 +31,10 @@
\tikzstyle{refblock}=[rectangle,draw=darkgray, thick]
\tikzstyle{projectedblock}=[rectangle,draw=black, thick,dashed]
\tikzstyle{refmv_linestyle}=[-, thick]
\tikzstyle{frame_label}=[anchor=center,scale=0.8]
\tikzstyle{refmv}=[draw=black, refmv_linestyle,decoration={markings,mark=at position 1 with {\arrow[ultra thick,scale=1]{latex}}},postaction={decorate}]
\tikzstyle{extrapolated_mv}=[refmv_linestyle,dashed, draw=blue, decoration={markings,mark=at position 1 with {\arrow[ultra thick,scale=1]{latex}}},postaction={decorate}]
\tikzstyle{extrapolated_mv}=[refmv_linestyle,dashed, draw, decoration={markings,mark=at position 1 with {\arrow[ultra thick,scale=1]{latex}}},postaction={decorate}]
\tikzstyle{arrow}=[draw, ->,>=stealth',thick,,decoration={markings,mark=at position 1 with {\arrow[ultra thick,scale=0.5]{stealth}}},postaction={decorate}]
@@ -66,42 +67,22 @@ style={yslant=\TikZyslant,xscale=\TikZxscale}]
\foreach \y in {1,3}
\draw[black, dotted] (\x,\y) +(0,0) rectangle ++(1,1);
\coordinate (TB1) at (1+0.7,4+0.8);
\draw[projectedblock] (TB1) rectangle ++(1,-1);
\coordinate (TB2) at (3,4);
\draw[projectedblock] (TB2) rectangle ++(1,-1);
\coordinate (TB1) at (1,4);
%\coordinate (TB2) at (3,4);
\coordinate (TB3) at (3+0.2/2,2-0.8/2);
\draw[projectedblock] (TB3) rectangle ++(1,-1);
\coordinate (TB4) at (2+0.9,2+0.6);
\coordinate (TB4) at (3,2);
\draw[projectedblock] (TB1) rectangle ++(1,-1);
\draw[projectedblock] (TB4) rectangle ++(1,-1);
% \coordinate (TB4) at (3,2);
% \draw[projectedblock] (TB4) rectangle ++(1,-1);
\end{scope}
\end{pgfonlayer}
% % Frame t+1
% \begin{pgfonlayer}{L0}
% \begin{scope}[ yshift=4*\yframeshift, xshift=4*\xframeshift,
% style={yslant=\TikZyslant,xscale=\TikZxscale}]
%
%
% \coordinate (TP1B1) at (1,4);
% \coordinate (TP1B2) at (3,4);
% \coordinate (TP1B3) at (1,2);
% \coordinate (TP1B4) at (3,2);
%
% %Draw a frame
% \DrawFrame{white}{4}
%
% % \draw[black,very thick] (TP1B2) +(0,0) rectangle ++(1,-1);
% \draw[black,very thick] (TP1B4) +(0,0) rectangle ++(1,-1);
%
% \end{scope}
% \end{pgfonlayer}
% Frame t-1
\begin{pgfonlayer}{L1}
\begin{scope}[ yshift=2*\TikZyfshift, xshift=2*\xframeshift,
@@ -110,32 +91,25 @@ style={yslant=\TikZyslant,xscale=\TikZxscale}]
\coordinate (T0B1) at (1,4);
\coordinate (T0B2) at (3,4);
\coordinate (T0B3) at (1,2);
\coordinate (T0B4) at (3,2);
\coordinate (T0B4) at (2,2);
\coordinate (TM1AuxB3) at (1,2);
\coordinate (Bottom_V1) at (2.5,0);
\draw[extrapolated_mv] (TB1) to (T0B1);
\coordinate (T0B1_aux) at (1-0.7,4-0.8);
% Draw Motion vectors
\coordinate (TM1B0) at (3+0.2,2-0.8);
\coordinate (TM1B0C) at (3+0.2/2,2-0.8/2);
% \draw[refmv] (TP1B4) to (TM1B0);
\draw[refblock] (TM1B0) rectangle ++(1,-1);
\draw[extrapolated_mv] (TB3) to (TM1B0);
\draw[refblock] (TM1B0C) rectangle ++(1,-1);
\draw[extrapolated_mv] (TB4) to (TM1B0C);
\path[extrapolated_mv] (TB1) -- (T0B1_aux) node[above,pos=.2] {\footnotesize $v^{1}_{0}$};;
\path[extrapolated_mv] (TB4) -- (T0B4) node[above,pos=.4] {\footnotesize $v^{2}_{0}$};;
\coordinate (TM1B1) at (3-0.6,4-0.5);
\draw[extrapolated_mv] (TB2) to (TM1B1);
\DrawFrame{white}{4}
\draw[black,very thick] (T0B1) +(0,0) rectangle ++(1,-1);
\draw[black,very thick] (T0B2) +(0,0) rectangle ++(1,-1);
\draw[projectedblock] (T0B1_aux) +(0,0) rectangle ++(1,-1);
\draw[black,very thick] (T0B4) +(0,0) rectangle ++(1,-1);
\end{scope}
\end{pgfonlayer}
@@ -148,10 +122,11 @@ style={yslant=\TikZyslant,xscale=\TikZxscale}]
%Define Points
\coordinate (TM2B1) at (1-0.7,4-0.8);
\coordinate (TM2B2) at (3-0.6,4-0.5);
\coordinate (TM2B4) at (2-0.9,2-0.6);
% Draw Motion vectors
\draw[refmv] (T0B1) to (TM2B1);
\draw[refmv] (T0B2) to (TM2B2);
\path[refmv] (T0B1) -- (TM2B1) node[above,pos=.2] {\footnotesize $v^{1}_{-1}$};
\path[refmv] (T0B4) -- (TM2B4) node[above,pos=.3] {\footnotesize $v^{2}_{-1}$};
\DrawFrame{white}{4}
@@ -159,19 +134,19 @@ style={yslant=\TikZyslant,xscale=\TikZxscale}]
% Draw block
\draw[refblock] (TM2B1) rectangle ++(1,-1);
\draw[refblock] (TM2B2) rectangle ++(1,-1);
\draw[refblock] (TM2B4) rectangle ++(1,-1);
\end{scope}
\end{pgfonlayer}
%MVs name
\node[scale=0.5] at (3*\xframeshift-1.0cm,4.9) {(2)};
\node[scale=0.5] at (3*\xframeshift-1.0cm,3) {(1)};
% \node[scale=0.5] at (3*\xframeshift-1.0cm,4.9) {(2)};
% \node[scale=0.5] at (3*\xframeshift-1.0cm,3) {(1)};
% \node[scale=0.5] at (3*\xframeshift-2.0cm,2) {(2)};
% \node[scale=0.5] at (3*\xframeshift-1.0cm,0.3) {(4)};
\tikzstyle{frame_label}=[anchor=center,scale=0.8]
% \node[frame_label,scale=0.95] (LostFrameLabel) at (4*\xframeshift-0.9cm,-1.8) {Lost frame };
% \path[draw,-latex,thick] (LostFrameLabel) edge[out=90, in=0] (3*\xframeshift+5cm*\TikZxscale,0cm);
@@ -193,15 +168,5 @@ style={yslant=\TikZyslant,xscale=\TikZxscale}]
% \node[frame_label] at (4*\xframeshift,\yframeshift*4) { $t_{1}$ };
\end{scope}
% \begin{scope}[yshift=-3.2cm,xshift=-0.5cm]
% \begin{scope}
% \draw[refmv] (0,0) to (1.5,0);
% \fill (2cm,0) node[frame_label,scale=0.95,right]{Original Motion Vector};
% \end{scope}
% \begin{scope}[xshift=14cm]
% \draw[extrapolated_mv] (0,0) to (1.5,0);
% \fill (2cm,0) node[frame_label,scale=0.95,right]{Extrapolated Motion Vector};
% \end{scope}
% \end{scope}
\end{tikzpicture}
042_ConcealmentAwareEncoding/Results/ECMQualitySingleloss.tex
+2
View File
@@ -19,6 +19,7 @@ set xtics 5,2,15
set ytics 4 right
set key width 6
set yrange[20:45]
set title '\TitleFont Basketball Drill'
plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \MC' w lp ls 2, '-' using 1:2 title '\KeyFont \DEEPFLOW' w lp ls 3, '-' using 1:2 title '\KeyFont \PROP' w lp ls 4
0 42.08 44.10 44.86
@@ -106,6 +107,7 @@ e
19 40.22 42.85 43.05
e
unset yrange
set title '\TitleFont Book Arrival'
plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \MC' w lp ls 2, '-' using 1:2 title '\KeyFont \DEEPFLOW' w lp ls 3, '-' using 1:2 title '\KeyFont \PROP' w lp ls 4
0 41.35 41.07 43.41
042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameLowDelay.tex
+1 -1
View File
@@ -7,7 +7,7 @@ set xrange[6:6+27]
set xtics 6,2,6+27
set key bottom left width 0
#set title '\TitleFont Basketball Drill'
plot '-' using 1:2 title '\KeyFont \MC' w lp ls 1, '-' using 1:2 title '\KeyFont \emph{16x16}' w lp ls 2, '-' using 1:2 title '\KeyFont \PROP' w lp ls 3
plot '-' using 1:2 title '\KeyFont \MC' w lp ls 1, '-' using 1:2 title '\KeyFont \emph{$16\times16$}' w lp ls 2, '-' using 1:2 title '\KeyFont \PROP' w lp ls 3
2 40.24 43.19 44.01
3 40.24 42.96 43.94
4 41.26 43.62 44.49
042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameRandomAccess.tex
+1 -1
View File
@@ -7,7 +7,7 @@ set xrange[6:6+27]
set xtics 6,2,6+27
set key bottom left width 0
#set title '\TitleFont Park Scene'
plot '-' using 1:2 title '\KeyFont \MC' w lp ls 1, '-' using 1:2 title '\KeyFont \emph{16x16}' w lp ls 2, '-' using 1:2 title '\KeyFont \PROP' w lp ls 3
plot '-' using 1:2 title '\KeyFont \MC' w lp ls 1, '-' using 1:2 title '\KeyFont \emph{$16\times16$}' w lp ls 2, '-' using 1:2 title '\KeyFont \PROP' w lp ls 3
2 39.63 41.81 43.47
3 39.22 41.77 43.39
4 39.84 41.86 43.47
042_ConcealmentAwareEncoding/Results/TableSignallingPerformance.tex
+1 -1
View File
@@ -91,7 +91,7 @@
\begin{sidewaystable}
\newcommand{\somespaces}{}
\centering
\caption{Average quality (\ac{psnr} and \ac{wpsnr}) of the affected frames for Low-Delay configuration.}
\caption{Average quality (PSNR and WPSNR) of the affected frames for Low-Delay configuration.}
\label{TAB4:SignallingPSNR}
\TABLESVSPACE
\small