jfmcarreira/doc_phd_thesis
1
0 Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Fix abreviations and equations referencing

Browse Source
This commit is contained in:
jfmcarreira committed 2018-12-10 19:19:32 +00:00
1 parent 935eb96ed4
commit 52d96c729a
11 files changed
+40 -144
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex
+6 -45
View File
@@ -13,16 +13,6 @@
\vspace*{\fill}
\section{Introduction}
\glsreset{rd}
\glsreset{mv}
\glsreset{ec}
\glsreset{cu}
\glsreset{ld}
\glsreset{ra}
\glsreset{ra}
\glsreset{ld}
In this chapter an error concealment-aware encoding scheme is proposed to improve the quality of decoded video in broadcast environments prone to transmission errors and data loss.
The method aims to enhance the reconstruction of recovered lost frames by optimising the efficiency of the \ac{ec}.
The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled through \ac{sei} messages. Such optimal \ac{ec} modes are found by simulating transmission losses followed by a Lagrangian optimisation of the \textit{signalling rate - \ac{ec} distortion} cost. A generalised saliency-weighted distortion is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. When data loss occurs the decoder uses the signalling information to improve the reconstruction quality.
@@ -40,10 +30,6 @@ Finally, Section~\ref{Section5:Summary} concludes the chapter.
\section{Problem definition}
\label{Section5:Problem}
\glsreset{mve}
\glsreset{mc}
%\glsunset{mb}
In Chapter~\ref{Chapter:ErrorResilienceReview} different \ac{ec} algorithms that can effectively reconstruct lost video were described, though not all of them exhibit the same accuracy for every lost frame or slice.
Moreover, it was also shown that combining different \ac{ec} approaches at the block level to improve the reconstruction accuracy is not straightforward.
In the past, hybrid \ac{ec} schemes were proposed to improve the quality of recovered frames by either combining spatial and temporal \ac{ec} algorithms~\cite{HybridConcTB2008} or by optimising the \ac{mve} at the pixel and block levels~\cite{HybridConcealmentTB2008,2DConcHybridMotion2010}.
@@ -142,7 +128,7 @@ E(v) =
\sum_{\mathbf{p} \in \Omega} \Big( \big| f_{-1}(\mathbf{p}) - f_{-2}( \mathbf{p} + v(\mathbf{p}))\big| + \big|\nabla v(\mathbf{p})\big| \Big)%d\mathbf{p},\ \mathbf{p} \in \Omega
\end{equation}
where $f_{-1}$ and $f_{-2}$ are the two previous frames at time instants $t_{-1}$ and $t_{-2}$, as represented in Figure~\ref{FIG5:MotionVectorExtrapolation}.
In Equation~\ref{EQ5:DualFlow}, $\Omega$ defines the sets of pixels of those frames, and $|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of the optical flow which effectively disallow discontinuities.
In~(\ref{EQ5:DualFlow}), $\Omega$ defines the sets of pixels of those frames, and $|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of the optical flow which effectively disallow discontinuities.
This is based on the assumption that neighbouring regions belong to the same object, thus such regions should have similar motion.
@@ -156,7 +142,7 @@ f(\mathbf{p}) \sim \mathbf{p}\mathbf{A}_1\mathbf{p}^T+\mathbf{b}_1\mathbf{p}^T+C
where $\mathbf{p} = [x\ y]$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ is a row vector and $C_1$ is a scalar. The coefficients are estimated using weighted least squares fit to the signal values in the neighbourhood.
Since the result of a polynomial expansion is such that each neighbourhood is approximated by a polynomial, it is necessary to analyse what happens if a polynomial undergoes an ideal translation.
Let's consider the signal $f_1(\mathbf{p})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$, given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following:
Let's consider the signal $f_1(\mathbf{p})$ represented by~(\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$, given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following:
\begin{align}
f_2(\mathbf{p})
&= f_1(\mathbf{p}-\mathbf{d}) \nonumber \\
@@ -175,10 +161,8 @@ This allows to find the displacement $\mathbf{d}$ for any signal dimensions by s
\subsubsection{Large displacement optical flow with deep matching (\textit{DeepFlow})}
%\todo{Completar o Deep flow}
In this method the motion field is estimated based on extension of the total variation methods using descriptor (feature) matching.
This allows the optical flow estimation to accurately find large displacements, because it is not fully bound by any constrains, such as the Horn \& Such presented in Equation~(\ref{EQ5:DualFlow})\cite{LargeDisplacementOptFlow_TPAMI2011}. Moreover, the correspondences between descriptions are used to apply a coarse-to-fine wrapping, which avoids local minimum values.
This allows the optical flow estimation to accurately find large displacements, because it is not fully bound by any constrains, such as the Horn \& Such presented in~(\ref{EQ5:DualFlow})\cite{LargeDisplacementOptFlow_TPAMI2011}. Moreover, the correspondences between descriptions are used to apply a coarse-to-fine wrapping, which avoids local minimum values.
The combination of both approaches increases the matching accuracy.
This approach was also exploited in \cite{DeepFlow2013} with the following extensions: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layer optimisation.
@@ -194,7 +178,7 @@ E(v) =
\end{equation}
where $E_D$ is the data term, $E_S$ is a smoothing term and $E_M$ is the matching term.
The data term is obtained from the difference between matching pixels in both video frames.
The smoothness term is a robust penalisation of the gradient of the flow, similarly to Equation (\ref{EQ5:DualFlow}), i.e., $E_S(\mathbf{p})=|\nabla v(\mathbf{p})|$. Finally, the matching term ($E_M$) is used to encourage the flow estimation to be similar to the precomputed motion field~($v'$).
The smoothness term is a robust penalisation of the gradient of the flow, similarly to~(\ref{EQ5:DualFlow}), i.e., $E_S(\mathbf{p})=|\nabla v(\mathbf{p})|$. Finally, the matching term ($E_M$) is used to encourage the flow estimation to be similar to the precomputed motion field~($v'$).
\begin{figure}[t]
\centering
@@ -215,7 +199,6 @@ Table~\ref{TAB5:ECPerformaceComparison} shows the average quality (\ac{psnr}) ob
The absolute value of \ac{psnr} is shown for the \FC method, while the differential values are shown for the remaining methods.
The results shown in the table reveal that methods \MC and \MVE are both able to overcome \FC, achieving average quality gains of up to 1.41 dB and 1.46 dB, respectively. Moreover, by extrapolating the \acs{mv} from the correctly receiving frame to the lost one (\MVE), it is also possible to outperform motion-copy (\MC).
This can be explained by the objects' trajectories of objects which are preserved in the \MVE method (assuming constant translational motion), in contrast with the \MC method, which assumes that neighbouring objects have parallel motion which may result in inaccurate estimations.
%\HERE
\begin{table}[tb]
@@ -247,8 +230,6 @@ Thus, in order to achieve higher reconstruction quality, different approaches sh
\section{Architecture for the EC-aware video encoding}
\label{Section5:Architecture}
\glsreset{el}
This section presents the general architecture proposed for \ac{ec}-aware video encoding, describing its main functional blocks, and providing an introduction for the subsequent detailed descriptions.
Figure~\ref{FIG5:BlockDiagram} illustrates the architecture of the proposed \ac{ec}-aware encoding scheme.
After the video encoder, it includes data loss simulation at the \ac{nal} for every slice. Subsequently, \ac{ec} is applied to the simulated losses in order to find the best methods that should be used to reconstruct the lost slices in case of transmission errors.
@@ -324,24 +305,6 @@ The lost frame is partitioned into quadtree structures, each one forming a set o
Then, the process of finding the optimal \ac{ec} method for each \ac{ecu} is formulated as a rate-distortion cost minimisation. To solve such problem, a Lagrangian optimisation is used to achieve the best trade-off between reconstruction quality and signalling overhead.
Furthermore, image saliency driven by visual attention modelling is used in the optimisation process to weight the cost of the overhead information of each \ac{ecu}, according to the visual relevance of its content.
% \HIGHLIGHT{%
% The optimal depth $Level$ ($l^*$) that determines the best \ac{ecu} size is obtained by minimising the following lagragian cost.}
% \begin{equation}\label{eq:optdepth}
% \newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
% l^* = \argmin_{l=0..3} \left\{ D(l) + \left(1+\frac{t_0-t_I}{T_{I}}\right) \lambda R(l) \right\},
% % l^* = \argmin_{l=1..3} \left\{ D(l) + \lambda \left(1-\frac{t_I-t_c}{T_{\ac{ird}}}\right) R(l) \right\},
% \end{equation}
% \HIGHLIGHT{where $R(l)$ represents the number of overhead bits required to encode \ac{ec} mode, and $D(l)$ is mean square error of the \ac{ctu} formed by $4^l$ ECUs.
% }
% \HIGHLIGHT{%
% In equation~\ref{eq:optdepth}, besides the parameter $\lambda$ used to trade-off the cost of distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(l)$ according to the duration of the error propagation decreases. This is given by the ratio between the temporal distance from the current frame ($t_0$) to the next refresh frame ($t_I$) and the period of refresh frames ($T_{I}$).
% }
% \begin{align}
% (ecm^*, l^*) &= arg \min_{\phi \in M} J(ecm, l), \\
% J(ecm,l) &= D(ecm,l) + \left(1+\frac{t_0-t_I}{T_{I}}\right) \lambda R(ecm,l),
% \label{eq:Lagrangian}
% \end{align}
The optimal \ac{ec} mode $m_{l,i}^*$, for the $i_{th}$ \ac{ecu} of size defined by the quadtree partitioning level $l$, is obtained by minimising the following Lagrangian cost:
\begin{equation}\label{EQ6:OptimalDepth}
\small
@@ -353,14 +316,14 @@ m_{l,i}^* = \argmin\limits_{\footnotesize \substack{l\in\left\{0...3\right\} \\
where $R(m_{l,i})$ represents the number of overhead bits required to encode the \ac{ec} mode, and $D(m_{l,i})$ is the sum of absolute error of the \ac{ecu}.
In equation~\ref{EQ6:OptimalDepth}, besides the parameter $\lambda$, used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation.
In~(\ref{EQ6:OptimalDepth}), besides the parameter $\lambda$, used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation.
As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques}, this method uses the ratio between the temporal distance from the current frame ($t_C$) to the \HIGHLIGHT{previous refresh frame} ($t_I$) and the period of refreshing frames ($T_{I}$).
%This weight varies between $\left[0;1 \right]$, achieving the lowest value for intra-coded frames and the maximum for the frame located immediately before an intra-coded one.
This behaviour follows the fact that when a lost frame is closer to the next intra-coded frame, the error propagation is lower.
Thus, it is not worthwhile to spend too many overhead bits to reduce it. This is accomplished by increasing the cost of such bits according to the temporal distance between the missing frame and the next I frame.
Moreover, a dynamic value for $\lambda$ is also used to adjust the amount of overhead within the frame, according to the visual relevance of each image region, defined by a saliency map.
The algorithm previously described results in \acs{ecu} with variable size, which is defined by the depth of the quad-tree structure, i.e., level $l$. The partition size is jointly optimised along with the best \ac{ec} candidate method using Equation~\ref{EQ6:OptimalDepth}. Alternatively, a fixed \ac{ecu} size can also be used by defining a fixed depth for all quad-tree structures, i.e., constant $l$ for the entire frame. This can be achieved by using Equation~\ref{EQ6:OptimalDepth} to solely optimise the \ac{ec} methods for each \ac{ecu}.
The algorithm previously described results in \acs{ecu} with variable size, which is defined by the depth of the quad-tree structure, i.e., level $l$. The partition size is jointly optimised along with the best \ac{ec} candidate method using~(\ref{EQ6:OptimalDepth}). Alternatively, a fixed \ac{ecu} size can also be used by defining a fixed depth for all quad-tree structures, i.e., constant $l$ for the entire frame. This can be achieved by using~(\ref{EQ6:OptimalDepth}) to solely optimise the \ac{ec} methods for each \ac{ecu}.
Although this reduces the overall complexity of the proposed method, it leads to a non-optimal solution which may require additional overhead. In Section~\ref{Section5:ECSignallingPerformance} a comparison between a dynamic and fixed approach is provided.
@@ -575,7 +538,6 @@ In summary, the proposed technique is able to enhance the performance of the sta
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% SECTION
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -685,7 +647,6 @@ The visual impact of using the proposed \ac{ec}-aware encoding can be evaluated
\section{Overall performance evaluation}
\label{Section5:OverallPerformance}
%\subsection{Quality evaluation}
As in the previous simulations, the packets were randomly discarded to simulate loss events, at different \acs{plr}.
The quality obtained with the proposed method is compared with the default method used by reference \ac{hevc} , i.e., frame-copy (\REF), motion-copy (\MC)~\cite{HEVCMCResidueDec2013}, and the method proposed in \cite{ROPERefGenJan2010}, which implements a robust scheme to minimise error propagation.