jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Finish thesis review

Browse Source
This commit is contained in:
jfmcarreira
2018-12-19 15:27:35 +00:00
parent 1c50b4c036
commit 59d007205a
17 changed files with 131 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files
042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex
+16 -15
View File
@@ -107,14 +107,14 @@ E(x,y) = \sum_{x} \sum_{y} \bigg| f_{0}\big(x,y\big) - f_{-1}\big( x + v_x(x,y),
Besides a full search algorithm, where a block-based approach is used to search for the best \acs{mv} that minimise the sum of absolute errors, three other approaches were tested to evaluate their effectiveness in \ac{ec} algorithms.
These three methods are as follows:
\begin{enumerate}
\item a method based on the total variation of the pixel values using the minimization of the \mbox{L-1} norm~\cite{DualFlow2016};
\item a method based on the duality between total pixel difference and regularisation term~\cite{DualFlow2016};
\item a method which approximates the neighbourhood of each pixel by quadratic polynomials on both frames~\cite{PolyFlow2003};
\item a method based on a combination of feature matching and total variation~\cite{DeepFlow2013}. %\todo{check here!}
\item a method based on a combination of feature matching and total pixel difference~\cite{DeepFlow2013}. %\todo{check here!}
\end{enumerate}
\subsubsection{Motion estimation based on total variation and regularisation (\textit{DualFlow})}
\subsubsection{Motion estimation based on a regularisation term (\textit{DualFlow})}
In this method the motion field is estimated based on the minimization of the total variation of pixel values measured using the L-1 norm \cite{DualFlow2007,DualFlow2016}, taking into account the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}.
In this method the motion field is estimated based on the minimization of the total difference of pixel values measured using the L-1 norm \cite{DualFlow2007,DualFlow2016}, taking into account a regularisation term obtained from the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}.
This condition forces the estimated field to be regular across neighbouring pixels by penalising deviations in a quadratic sense.
Using this method, for each pixel $\mathbf{p}$, the motion field $v(\mathbf{p})$ is obtained by minimising the following energy function,
\begin{equation}\label{EQ5:DualFlow}
@@ -133,7 +133,7 @@ In this method only quadratic polynomials are considered, expressed as:
\begin{equation}\label{EQ5:PolyFlow}
f(\mathbf{p}) \sim \mathbf{p}\mathbf{A}_1\mathbf{p}^T+\mathbf{b}_1\mathbf{p}^T+C_1
\end{equation}
where $\mathbf{p} = [x\ y]$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ is a row vector and $C_1$ is a scalar. The coefficients are estimated using weighted least squares fit to the signal values in the neighbourhood. The use of quadratic polynomials was chosen primarily due to simplicity and to keep the computational complexity as low as possible. However, as stated in \cite{farneback2002polynomial} the extension to higher degree is straightforward.
where $\mathbf{p} = [x\ y]$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ is a row vector and $C_1$ is a scalar. The coefficients are estimated using weighted least squares fit to the signal values in the neighbourhood. The use of quadratic polynomials was chosen primarily due to simplicity and to keep the computational complexity as low as possible, accepting a small penalty in the estimation accuracy. However, as stated in \cite{farneback2002polynomial} the extension to higher degree is straightforward.
Since the result of a polynomial expansion is such that each neighbourhood is approximated by a polynomial, it is necessary to analyse what happens if a polynomial undergoes an ideal translation.
Let's consider the signal $f_1(\mathbf{p})$ represented by~(\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$, given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following:
@@ -155,7 +155,7 @@ This allows to find the displacement $\mathbf{d}$ for any signal dimensions by s
\subsubsection{Large displacement optical flow with deep matching (\textit{DeepFlow})}
In this method the motion field is estimated based on extension of the total variation methods using descriptor (feature) matching.
In this method the motion field is estimated based on extension of the total pixel difference methods using descriptor (feature) matching.
This allows the optical flow estimation to accurately find large displacements, because it is not fully bound by any constrains, such as the Horn \& Such presented in~(\ref{EQ5:DualFlow})\cite{LargeDisplacementOptFlow_TPAMI2011}. Moreover, the correspondences between descriptions are used to apply a coarse-to-fine wrapping, which avoids local minimum values.
The combination of both approaches increases the matching accuracy.
@@ -187,7 +187,7 @@ The smoothness term is a robust penalisation of the gradient of the flow, simila
This section presents a simulation study of the different \ac{ec} schemes previously described.
This evaluation aims at finding the suitable methods to be used for efficient reconstruction of the missing frames in case of data loss.
The two \ac{ec} methods based on the correctly received \acs{mv} are tested, i.e., \ac{mc} and \ac{mve}, while the \ac{fc} method is used as reference for comparison.
Moreover, the four different approaches to estimating the motion field for the missing frame are compared: (i) full search based on the minimisation of the sum of absolute errors (\FULLSEARCH), (ii) optical flow based on the duality between total variation and smoothness term (\DUALFLOW), (iii) optical flow based on polynomial approximations (\POLYFLOW) and (iv) optical flow based on deep matching (\DEEPFLOW).
Moreover, the four different approaches to estimating the motion field for the missing frame are compared: (i) full search based on the minimisation of the sum of absolute errors (\FULLSEARCH), (ii) optical flow based on the duality between total difference and regularisation term (\DUALFLOW), (iii) optical flow based on polynomial approximations (\POLYFLOW) and (iv) optical flow based on deep matching (\DEEPFLOW).
Table~\ref{TAB5:ECPerformaceComparison} shows the average quality (\ac{psnr}) obtained for the different methods under investigation.
The absolute value of \ac{psnr} is shown for the \FC method, while the differential values are shown for the remaining methods.
@@ -208,7 +208,7 @@ According to the results of Table~\ref{TAB5:ECPerformaceComparison} (columns 5 t
Results for \FULLSEARCH method reveal that only minimising the sum of absolute errors between two consecutive frames might not always be directly related with objects' motions, which results in inaccurate motion field to be used in the reconstruction of lost frames.
Finally, results from Table~\ref{TAB5:ECPerformaceComparison} reveal that the \DEEPFLOW algorithm leads to the most accurate motion field, in comparison with the other methods.
It is able to achieve an average quality gain of 1.79 dB, and up to 3.39 dB for Book Arrival sequence, where \POLYFLOW only achieves 3.05 dB.
Since \POLYFLOW combines different techniques also used by other algorithms in study, e.g., total variation also used by \DUALFLOW, it is expected to achieve accurate estimations and consequently higher \ac{ec} reconstruction quality.
Since \POLYFLOW combines different techniques also used by other algorithms in study, e.g., total difference of pixel values also used by \DUALFLOW, it is expected to achieve accurate estimations and consequently higher \ac{ec} reconstruction quality.
Moreover, using feature matching in combination with smoothing conditions, the \DEEPFLOW algorithm is able to correctly estimate large displacements, which results in improved performance for video signals with higher motion, e.g., Tennis sequence.
\subsection*{Remarks}
@@ -382,7 +382,7 @@ Adaptive arithmetic coding~\cite{ArithmeticCodingJun1994} is used to efficiently
\end{figure}
\subsection{Enhancement-layer: \ac{ec} residue coding}
\subsection{Enhancement-layer: EC residue coding}
As described earlier in this section, in order to allow further improvement of recovered slices, besides the best \ac{ec} mode, the proposed encoding scheme also allows encoding a controlled amount of \ac{ec} residue using the scalable extension of \ac{hevc} (see Figure~\ref{FIG5:BlockDiagram}).
The enhancement layer module encodes the difference between the coded frame ($\hat f$) and its counterpart recovered by using the optimal \ac{ec} mode ($\tilde f$).
@@ -424,7 +424,8 @@ In this section the performance of the proposed method is evaluated only for int
Since intra \acs{cu} result in higher amount of bits, they also require a higher number of packets, making them more prone to errors and data loss.
Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets correspond to intra coded slices, which is a non-negligible probability of one of them being hit by transmission errors which would lead to packet loss.
Moreover, by limiting the proposed method to intra frames, the amount of required overhead is reduced.
In this case a constant $\lambda_0=10$ was used, without taking into consideration saliency estimation. This value results from an exhaustive search targetting acceptable quality gains with a low overhead (i.e., average overhead under 5\%).
In this case a constant $\lambda_0=10$ was used, without taking into consideration saliency estimation.
This value results from an exhaustive search targetting acceptable quality gains with a low overhead (i.e., average overhead under 5\%).
Since a relevant performance indicator is the error propagation over temporally predicted frames, this was evaluated using a single intra-frame loss.
Figures~\ref{FIG5:singlelosslowdelay} and~\ref{FIG5:singlelossramdon} show the \ac{psnr} results when frame Intra-frame $\#16$ is missing, for \ac{ld} and \ac{ra} encoding configurations, respectively.
@@ -696,7 +697,7 @@ Since the proposed \ac{ec}-aware scheme is able to achieve higher quality withou
An advantage of using a feedback channel is the low extra complexity required to implement robust coding because the encoder receives specific information about transmission errors. This allows straightforward spatio-temporal localization of the lost data without computational effort.
Table~\ref{tab:overall_gain_ratio} shows the average quality differences (\textbf{$\mathbf{\Delta}$ PSNR})between the reference method (\REF) and the proposed one plus three other previous works. Two different test conditions are shown in the table: error-free and PLR=10\%. Moreover, an efficiency measure $R$ is defined, as the ratio between the quality gains obtained under packet loss and the quality decrease in the error-free case, i.e.,
Table~\ref{tab:overall_gain_ratio} shows the average quality differences (\textbf{$\mathbf{\Delta}$ PSNR})between the reference method (\REF) and the proposed one plus three other previous works. Two different test conditions are shown in the table: \ac{plf} and PLR=10\%. Moreover, an efficiency measure $R$ is defined, as the ratio between the quality gains obtained under packet loss and the quality decrease in the \ac{plf} case, i.e.,
\begin{equation}
R = \frac{\Delta PSNR(\ac{plr}=10\%)}{-\Delta PSNR(Error\text{-}free)}.
\end{equation}
@@ -704,8 +705,8 @@ R = \frac{\Delta PSNR(\ac{plr}=10\%)}{-\Delta PSNR(Error\text{-}free)}.
The results shown in Table~\ref{tab:overall_gain_ratio}, demonstrate that the effectiveness of the \ac{ec}-aware coding scheme is higher than all other methods under comparison. In fact, not only the absolute \ac{psnr} difference (i.e., \ac{psnr} gain) is higher for the same \ac{plr}, but also the efficiency of the overhead is also higher.
This means that the overhead bits spent to encode the signalling information and the enhancement residue carry more useful information than the other methods.
For instance, in comparison with the method in \cite{DualFrameME2015} the proposed method leads to higher quality degradation in the error-free case. However, for PLR=10\% it clearly outperforms the method in \cite{DualFrameME2015}, achieving an higher $R$, indicating a superior efficiency.
Results in Table~\ref{tab:overall_gain_ratio} also show that the small loss of coding efficiency in error-free transmission is clearly surpassed by the quality gains under error-prone transmission conditions.
For instance, in comparison with the method in \cite{DualFrameME2015} the proposed method leads to higher quality degradation in those cases without errors. However, for PLR=10\% it clearly outperforms the method in \cite{DualFrameME2015}, achieving an higher $R$, indicating a superior efficiency.
Results in Table~\ref{tab:overall_gain_ratio} also show that the small loss of coding efficiency in packet loss free transmission is clearly surpassed by the quality gains under error-prone transmission conditions.
The methods in \cite{ROPERefGenJan2010,DualFrameME2015} are based on modified predictions, while the method in \cite{VPxResiliencePCS2016} uses redundant data without any further optimisation.
Both of these approaches require low additional computational complexity.
However, as observed in the results shown in Table~\ref{tab:overall_gain_ratio}, the lower additional complexity also corresponds to lower quality gains in case of packet loss.
@@ -718,14 +719,14 @@ The proposed method uses \ac{ec} techniques with different levels of computati
\begin{table}[t]
\newcommand{\somespaces}{}
\centering
\caption{Comparison of the proposed EC-aware robust coding architecture against existing in error-free and packet loss environments.}\label{tab:overall_gain_ratio}
\caption{Comparison of the proposed EC-aware robust coding architecture against existing techniques in environments with and without packet loss.}\label{tab:overall_gain_ratio}
\footnotesize
% \resizebox{\columnwidth}{!}{
\begin{tabular}{p{2.4cm}x{2.4cm}x{2.4cm}x{2cm}}
% \begin{tabular}{l|c|c|c}
\toprule
\multirow{2}{*}{\vbox{\textbf{Method}}} & \multicolumn{2}{c}{\textbf{$\mathbf{\Delta}$ PSNR (dB)}} & \multirow{2}{*}{\centering \vbox{$\mathbf{R}$ }} \\
& {Error-free} & {PLR=10\%} & \\
& {\NOLOSS} & {PLR=10\%} & \\
% \hline
\midrule
\AVG & -0.88 & 1.64 & 1.9 \\ \midrule %\hline