jfmcarreira/doc_phd_thesis
1
0 Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Commit before SF changes

Browse Source
This commit is contained in:
jfmcarreira committed 2018-09-03 02:03:59 +01:00
1 parent 594357af56
commit 8fb90680d2
10 files changed
+270 -280
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex
+32 -21
View File
@@ -4,7 +4,7 @@
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CHAPTER
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Error Concealment-Aware Video Encoding}
\chapter{Error Concealment-Aware Video Encoding \VERSION{1}}
\label{Chapter:ConcealmentAwareTechniques}
%\vspace{-6mm}
%\clearpage
@@ -119,7 +119,7 @@ Besides a full search algorithm where a block-based approach is used to search f
\begin{enumerate}
\item a method based on the total variation using the minimization of the \mbox{L-1} norm~\cite{DualFlow2016};
\item a method which approximates the neighbourhood of each pixel by quadratic polynomials on both frames~\cite{PolyFlow2003};
\item a method based on a combination of descriptor matching and large displacement estimation~\cite{DeepFlow2013}. \HERE
\item a method based on a combination of features matching and total variations~\cite{DeepFlow2013}. \HERE
\end{enumerate}
\subsubsection{Motion estimation based on total variation and regularisation (\textit{DualFlow})}
@@ -161,14 +161,21 @@ This allows to find the displacement $\mathbf{d}$ for any signal dimensions by s
\subsubsection{Large displacement optical flow with deep matching (\textit{DeepFlow})}
\todo{Completar o Deep flow}
%\todo{Completar o Deep flow}
In this method the motion field is estimated based on the large displacement optical flow presented in \cite{LargeDisplacementOptFlow_TPAMI2011}, which extends the total variation methods, e.g. the one expressed by Equation (\ref{EQ5:DualFlow}), by including descriptor matching.
The correspondences between descriptions are used to apply a coarse-to-fine wrapping, thus avoiding local minimum values. The matching algorithm is built upon a multi-stage architecture with six layers with different patches sizes, interleaving convolutions and max-pooling \cite{DeepFlow2013}. The combination of both approaches increases the matching accuracy.
In this method the motion field is estimated based on extension of the total variational methods using descriptor (feature) matching.
This allows the optical flow estimation to accurately find large displacements as it is not bound by constrains, such as the Horn \& Such presented in Equation~(\ref{EQ5:DualFlow})\cite{LargeDisplacementOptFlow_TPAMI2011}. Moreover, the correspondences between descriptions are used to apply a coarse-to-fine wrapping, thus avoiding local minimum values.
The combination of both approaches increases the matching accuracy.
In \cite{DeepFlow2013} to improve the estimated motion field accuracy, the following extension to the method proposed in~\cite{LargeDisplacementOptFlow_TPAMI2011} were introduced: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layers of optimisation. Figure~\ref{FIG5:DeepFlowDiagram} shows the blocks that are included in the estimation process. As shown in the figure a multi-size optimisation is applied by using different patches sizes. This is used to estimate an initial motion field, referred to as $v'$ .
This was further exploited by introducing deep convolutional networks~\cite{DeepFlow2013}.
This was also exploited in \cite{DeepFlow2013} with the following extensions: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layer optimisation.
Figure~\ref{FIG5:DeepFlowDiagram} shows the blocks that are included in the description matching process.
The matching algorithm is built upon a multi-stage architecture with six layers with different patches sizes, interleaving convolutions and max-pooling, which is similar to deep convolutional networks \cite{GradientBasedConvNN_ProcIEEE1998}.
As shown in the figure a multi-size response levels are generated from the reference and target image using convolutions with different patches sizes. This method uses bottom-up approach, i.e., starting at a fine level and moving towards the coarser levels (larger patches) which are built as an aggregation of responses of smaller patches. This results in a multi-size response pyramid.
Subsequently, a local maximum is extracted from each level to achieve a more accurate motion field (quasi-dense correspondence).
Finally, an initial motion field is extracted, referred to as $v'$ .
\begin{figure}[b]
\begin{figure}[tb]
\centering
\includegraphics[width=.95\columnwidth]{\ConcealmentAwareDir/Figures/DeepFlowBlockDiagram}
\caption{Block diagram of the deep matching algorithm \cite{DeepFlow2013}.}
@@ -194,7 +201,10 @@ Moreover, the four different approaches to estimate the motion field for the mis
Table~\ref{TAB5:ECPerformaceComparison} shows the average quality (\ac{psnr}) obtained for the different methods under investigation.
The absolute value of \ac{psnr} is shown for the \FC method, while the differential values are shown for the remaining methods.
The results shown in the table reveal that methods \MC and \MVE are both able to overcome \FC, achieving average quality gains of up to 1.41 dB and 1.46 dB, respectively. Moreover, by extrapolating the \acs{mv} from the correctly receiving frame to the lost one (\MVE), it is also possible to outperform motion-copy (\MC).
The results shown in the table reveal that methods \MC and \MVE are both able to overcome \FC, achieving average quality gains of up to 1.41 dB and 1.46 dB, respectively. Moreover, by extrapolating the \acs{mv} from the correctly receiving frame to the lost one (\MVE), it is also possible to outperform motion-copy (\MC).
This can be explained by the objects' trajectories of objects which are preserved in the \MVE method (assuming constant translational motion), in contrast with the \MC method, which assumes that neighbouring objects have parallel motion which may result in inaccurate estimations.
%\HERE
\begin{table}[tb]
\centering
@@ -204,10 +214,12 @@ The results shown in the table reveal that methods \MC and \MVE are both able to
\input{\ConcealmentAwareDir/Results/PerformanceConcealmentMethods.tex}
\end{table}
\HERE
Results for \ac{ec} methods based on motion estimation (see columns 5 to 8 of Table~\ref{TAB5:ECPerformaceComparison}) reveal an overall superior performance when comparing with methods which rely of correctly received \acs{mv}.
Results for \FULLSEARCH method reveals that only minimising the sum of absolute errors between two consecutive frames might not always be directly related with objects' motions. Thus, this results in an inaccurate motion estimation to be used for the reconstruction of lost frames. Finally, results from Table~\ref{TAB5:ECPerformaceComparison} reveal that the \DEEPFLOW algorithm leads to the most accurate motion field to be used for \ac{ec}, in comparison with the other methods. It is able to achieve an average quality gain of 1.79 dB, and up to 3.39 dB for Book Arrival sequence, where \POLYFLOW only achieves 3.05 dB.
According to the results of Table~\ref{TAB5:ECPerformaceComparison} (columns 5 to 8) the \ac{ec} methods based on motion estimation achieve an overall superior performance when compared with methods which rely of correctly received \acs{mv} (i.e., \MC and \MVE).
Results for \FULLSEARCH method reveal that only minimising the sum of absolute errors between two consecutive frames might not always be directly related with objects' motions, which results in inaccurate motion field to be used in the reconstruction of lost frames.
Finally, results from Table~\ref{TAB5:ECPerformaceComparison} reveal that the \DEEPFLOW algorithm leads to the most accurate motion field, in comparison with the other methods.
This is able to achieve an average quality gain of 1.79 dB, and up to 3.39 dB for Book Arrival sequence, where \POLYFLOW only achieves 3.05 dB.
Since \POLYFLOW combines different techniques also used by other algorithms in study, e.g., total variation also used by \DUALFLOW, it is expected to achieve accurate estimations and consequently higher \ac{ec} reconstruction quality. Moreover, using descriptor matching in combination with smoothing conditions, the \DEEPFLOW algorithm is able to correctly estimate large displacements, which results in an improved performance for video signals with higher motion, e.g., Tennis sequence.
\subsection*{Remarks}
@@ -231,16 +243,6 @@ After the video encoder, it includes data loss simulation at the \ac{nal} for ev
Saliency-weighted \ac{ec} mode decision is used to select the optimal \ac{ec} mode from the best trade-off between \ac{ec} distortion, using the original input video~($f$) as reference, and the number of bits required for signalling.
Then, the optimal \ac{ec} mode is sent to the decoder along with the residue formed between the error-free decoded frame and its error concealed counterpart. The scalable extension of the \ac{hevc} standard is used to encode this residue information.
\begin{figure}[tb]
\centering
%\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram}
\input{042_ConcealmentAwareEncoding/Figures/BlockDiagram.tikz}
\caption{Architecture of the error concealment-aware encoding scheme.}
\label{FIG5:BlockDiagram}
\FIGSVSPACE
\end{figure}
For the \ac{ec} mode decision optimization the missing slices are reconstructed, as they would be in the decoder, i.e., recovered at the encoder by simulating the decoder \ac{ec}, but using different \ac{ec} methods based on motion estimation and extrapolation techniques, to selected the best one at the encoder-side.
In this process, each \ac{ctu} is partitioned to form a quadtree structure of \ac{ecu} that are used to test the different \ac{ec} candidate modes.
Then, the best trade-off between \ac{ecu} partition and signalling is found through \ac{rd} optimisation.
@@ -249,6 +251,15 @@ Finally, the signalling information is transmitted to the decoder by using the \
Since in general, the optimal \ac{ec} mode is still not able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el}.
This residual information contains the difference between the compressed ($\hat f_t$) and the \ac{ec} frames ($\tilde f$).
\begin{figure}[tb]
\centering
%\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram}
\input{042_ConcealmentAwareEncoding/Figures/BlockDiagram.tikz}
\caption{Architecture of the error concealment-aware encoding scheme.}
\label{FIG5:BlockDiagram}
\FIGSVSPACE
\end{figure}
The transmitted information is used to assist the \ac{ec} operation at the decoder to recover erroneous frames with high accuracy, by using the best \ac{ec} mode and also residual information to reduce error propagation.
The remaining of this section describes in detail the building blocks of the proposed encoding architecture, i.e., the \ac{ec} candidate modes, the saliency-weighted optimal \ac{ec} mode decision, signalling and enhancement layer coding.
042_ConcealmentAwareEncoding/Results/TableSignallingPerformance.tex
+1 -1
View File
@@ -120,7 +120,7 @@ Tennis & 28.47 (30.79) & +1.66 (+1.27) & +2.
\begin{sidewaystable}
\newcommand{\somespaces}{}
\centering
\caption{Average \ac{fsim} of the entire video sequence in frame loss conditions (scaled between $[0-100]$).}
\caption{Average \ac{fsim} under frame loss conditions (scaled between $[0-100]$).}
\label{TAB4:SignallingFSIM}
\TABLESVSPACE
\small