jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Revise Chapter 6

Browse Source
This commit is contained in:
jfmcarreira
2018-12-14 22:07:37 +00:00
parent 61dd330c24
commit b577e76e0d
3 changed files with 22 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex
+18 -17
View File
@@ -133,7 +133,7 @@ In this method only quadratic polynomials are considered, expressed as:
\begin{equation}\label{EQ5:PolyFlow}
f(\mathbf{p}) \sim \mathbf{p}\mathbf{A}_1\mathbf{p}^T+\mathbf{b}_1\mathbf{p}^T+C_1
\end{equation}
where $\mathbf{p} = [x\ y]$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ is a row vector and $C_1$ is a scalar. The coefficients are estimated using weighted least squares fit to the signal values in the neighbourhood.
where $\mathbf{p} = [x\ y]$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ is a row vector and $C_1$ is a scalar. The coefficients are estimated using weighted least squares fit to the signal values in the neighbourhood. The use of quadratic polynomials was chosen primarily due to simplicity and to keep the computational complexity as low as possible. However, as stated in \cite{farneback2002polynomial} the extension to higher degree is straightforward.
Since the result of a polynomial expansion is such that each neighbourhood is approximated by a polynomial, it is necessary to analyse what happens if a polynomial undergoes an ideal translation.
Let's consider the signal $f_1(\mathbf{p})$ represented by~(\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$, given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following:
@@ -310,25 +310,26 @@ m_{l,i}^* = \argmin\limits_{\footnotesize \substack{l\in\left\{0...3\right\} \\
where $R(m_{l,i})$ represents the number of overhead bits required to encode the \ac{ec} mode, and $D(m_{l,i})$ is the sum of absolute error of the \ac{ecu}.
In~(\ref{EQ6:OptimalDepth}), besides the parameter $\lambda$, used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation.
As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques}, this method uses the ratio between the temporal distance from the current frame ($t_C$) to the \HIGHLIGHT{previous refresh frame} ($t_I$) and the period of refreshing frames ($T_{I}$).
In~(\ref{EQ6:OptimalDepth}), besides the parameter $\lambda$, which is used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation.
As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques}, this method uses the ratio between the temporal distance from the current frame ($t_C$) to the previous refresh frame ($t_I$) and the period of refreshing frames ($T_{I}$).
%This weight varies between $\left[0;1 \right]$, achieving the lowest value for intra-coded frames and the maximum for the frame located immediately before an intra-coded one.
This behaviour follows the fact that when a lost frame is closer to the next intra-coded frame, the error propagation is lower.
Thus, it is not worthwhile to spend too many overhead bits to reduce it. This is accomplished by increasing the cost of such bits according to the temporal distance between the missing frame and the next I frame.
Moreover, a dynamic value for $\lambda$ is also used to adjust the amount of overhead within the frame, according to the visual relevance of each image region, defined by a saliency map.
Moreover, a dynamic value for $\lambda$ is also used to adjust the amount of overhead within the frame, according to the visual relevance of each image region, defined by a saliency map (see Section~\ref{Section6:SaliencyDetection}).
The algorithm previously described results in \acs{ecu} with variable size, which is defined by the depth of the quad-tree structure, i.e., level $l$. The partition size is jointly optimised along with the best \ac{ec} candidate method using~(\ref{EQ6:OptimalDepth}). Alternatively, a fixed \ac{ecu} size can also be used by defining a fixed depth for all quad-tree structures, i.e., constant $l$ for the entire frame. This can be achieved by using~(\ref{EQ6:OptimalDepth}) to solely optimise the \ac{ec} methods for each \ac{ecu}.
Although this reduces the overall complexity of the proposed method, it leads to a non-optimal solution which may require additional overhead. In Section~\ref{Section5:ECSignallingPerformance} a comparison between a dynamic and fixed approach is provided.
\subsection{Saliency detection}
\label{Section6:SaliencyDetection}
The saliency maps are computed based on the well-known Itti's attention model \cite{SaliencyItti1998} that applies a hierarchical decomposition, based on a set of spatial features, which are combined with a center-surround weighting to obtain the saliency map.
Since the Itti's model only produces saliency maps for still images, in this work it is further combined with a temporal feature. This is obtained by estimating the optical flow between two consecutive frames, based on the algorithm presented in \cite{HornSchunckOpticalFlow1981}. The final saliency map is achieved by combining the spatial and temporal components, $S_{s}$ and $S_{t}$ respectively, as follows,
\begin{equation}\label{eq:saliency}
S = \rho S_{t} + (1-\rho) S_{s}
\end{equation}
The parameter $\rho$ is used to give higher relevance to the temporal component, since motion is correlated with error propagation, producing longer artefacts and consequently higher impact in visual attention. In this work the temporal component was given a weight slightly higher than 50\%, i.e. $\rho=0.6$.
The parameter $\rho$ is used to give higher relevance to the temporal component, since motion is correlated with error propagation, producing longer artefacts and consequently higher impact on visual attention. In this work the temporal component was given a weight slightly higher than 50\%, i.e. $\rho=0.6$.
On the one hand, higher values of saliency (i.e. regions with higher visual relevance) should result in lower values of $\lambda$, which tend to increase the overhead rate value on those regions, thus reducing the \ac{ec} distortion at the decoder, in case of loss. On the other hand, higher values of $\lambda$ should reduce the overhead, allowing higher \ac{ec} distortion at the decoder. In order to achieve the aforementioned goal, the following definition for $\lambda$ is used:
\begin{equation}\label{eq:lambdasaliency}
@@ -423,13 +424,13 @@ In this section the performance of the proposed method is evaluated only for int
Since intra \acs{cu} result in higher amount of bits, they also require a higher number of packets, making them more prone to transmission errors and losses.
Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets correspond to intra coded slices, which is a non-negligible probability of one of them being hit by transmission errors.
Moreover, by limiting the proposed method to intra frames, the amount of required overhead is reduced.
In this case a constant $\lambda_0=10$ was used, without taking into consideration saliency estimation.
In this case a constant $\lambda_0=10$ was used, without taking into consideration saliency estimation. This value results from an exhaustive search targetting acceptable quality gains with a low overhead (i.e., average overhead under 5\%).
Since a relevant performance indicator is the error propagation over temporally predicted frames, this was evaluated using a single intra-frame loss.
Figures~\ref{FIG5:singlelosslowdelay} and~\ref{FIG5:singlelossramdon} show the \ac{psnr} results when frame Intra-frame $\#16$ is missing, for \ac{ld} and \ac{ra} encoding configurations, respectively.
These figures show the results for the proposed method (\PROP), the motion-copy method (\MC) and also an alternative proposed method using fixed \ac{ecu} size of $16\times16$ pixels.
These results indicate that the proposed method (\PROP) outperforms the reference one and it is able to achieve quality gains up to {2 dB} for \ac{ld} configuration (see Figure~\ref{FIG5:singlelosslowdelay}), decrease the error propagation and increase the quality of the frames affected from the loss of the intra frame, by up to {3 dB}.
The use of \ac{ra} configuration leads to higher quality degradation than LD, due the higher temporal distance of the reference frame used for \ac{ec} (Frame $\#8$ is used to recover the lost Frame~$\#16$). Note that when Intra-frame \#16 is lost, all frames within the range \#9 to \#31 (i.e., next I-frame) are affected by error propagation.
The use of \ac{ra} configuration leads to higher quality degradation than LD, due to the higher temporal distance of the reference frame used for \ac{ec} (Frame $\#8$ is used to recover the lost Frame~$\#16$). Note that when Intra-frame \#16 is lost, all frames within the range \#9 to \#31 (i.e., next I-frame) are affected by error propagation.
Nevertheless, the proposed method still achieves similar quality gains (approximately 2 dB).
\begin{figure}[tb]
@@ -456,7 +457,7 @@ It is also worthwhile to notice that the \ac{psnr} variance obtained with the
\begin{table}[!t]
\newcommand{\somespaces}{}
\centering
\caption{Average quality (\ac{psnr}) of the frames affected by errors and its variance under random packet loss.}
\caption{Average \ac{psnr} (dB) of the frames affected by errors and its variance for two different methods under random loss of packets carrying intra-coded frames.}
%\caption{Average \ac{psnr} and variance of affected frames}
\label{tab:randomloss}
\TABLESVSPACE
@@ -507,7 +508,7 @@ In summary, the proposed technique is able to enhance the performance of the sta
\begin{table}[tbp]
\newcommand{\somespaces}{}
\centering
\caption{Percentage of overhead (\%)}
\caption{Percentage of overhead (\%) when using the proposed method solely for intra-coded frames.}
\label{TAB6:RedundancyIntra}
\TABLESVSPACE
\footnotesize
@@ -618,7 +619,7 @@ This confirms the two-fold superiority of adaptive $\lambda$, i.e. not only the
\subsubsection{Error propagation evaluation}
In this section the error propagation is evaluated using single loss events and compared against the reference methods.
In this simulation a whole frame loss is enforced in order to assess the impact of error propagation in the worst case scenario.
In this simulation a whole frame loss is enforced in order to assess the impact of error propagation on the worst case scenario.
Experimental evaluations were performed for both intra and inter-coded frame loss.
Figure~\ref{FIG5:singlelossInter} shows the \ac{psnr} over a \ac{gop} with an \ac{idr} period of 16 frames, where the whole frame \#6 is missing in sequences Basketball Drill and People On Street.
%The figure shows the results obtained by decoding the stream using the proposed method with adaptive $\lambda$ but without any information sent in the enhancement layer.
@@ -627,10 +628,10 @@ Figure~\ref{FIG5:singlelossIntra} shows the \ac{psnr} for the same conditions bu
\input{042_ConcealmentAwareEncoding/Results/ECMQualitySingleloss.tex}
These results reveal that the proposed method is able to improve the reconstruction of inter-coded frames and also outperforms the existing techniques, whenever a intra-coded frame is missing.
These results also indicate that the proposed method outperforms both reference methods since the time instant when the loss occurs. The lost frame itself is recovered with much higher quality than using the reference \ac{ec} methods.
These results also indicate that the proposed method outperforms both reference methods right after the time instant when the loss occurs. The lost frame itself is recovered with much higher quality than using the reference \ac{ec} methods.
This is due to the use of the optimal \ac{ec} mode to recover each \ac{ecu}, benefiting from the \ac{sei} information received from the encoder.
In this case, the proposed method is able to achieve quality gains up to 5~dB, decreasing the error propagation and increasing the \ac{psnr} of affected frames by up to 6~dB.
Comparing the results of Figures~\ref{FIG5:singlelossInter}~and~~\ref{FIG5:singlelossIntra} one may conclude that the loss of intra-frame leads to higher error propagation. For those cases where an intra-coded frame is missing, not only the reconstructions quality is lower for all methods, but also a longer (in terms of number of frames) error propagation occurs.
The visual impact of using the proposed \ac{ec}-aware encoding can be evaluated by observing the recovered frame (i.e., frame \#6) of Basketball Drill, as an example. Figure~\ref{FIG5:RecoveredExample} shows that higher visual quality and less reconstruction artefacts are obtained in comparison with the \MC method. It is worthwhile to note that the proposed method is able to reconstruct the object edges with higher accuracy than the reference \MC.
@@ -674,7 +675,7 @@ The visible artefacts demonstrate how the decoder is able to recover from whole
\FIGSVSPACE
\end{figure}
A closer look at the Race Horses sequence reveals that the horse rider is not correctly reconstructed by the \MC method (highlighted by the circles and ellipses). As shown in the figure, this not only results in a more accurate reconstruction of the lost frame, but also increases the visual quality of the subsequent frames.
A closer look at the Race Horses sequence reveals that the horse rider is not correctly reconstructed by the \MC method (highlighted by the circles and ellipses). As shown in the figure, the proposed method not only results in a more accurate reconstruction of the lost frame, but also increases the visual quality of the subsequent frames.
In the Kimono sequence, some artefacts are also noticeable in the \MC case, but not visible when the proposed method is used (highlighted with circles).
Moreover, the edges of the women's kimono (highlighted with ellipses) are also improved by the proposed method, resulting in a smooth reconstruction without error propagation.
From the results discussed above, it is clearly observed that the proposed method is able to reconstruct the contours of the moving parts more accurately than traditional \ac{ec} algorithms.
@@ -695,7 +696,7 @@ Since the proposed \ac{ec}-aware scheme is able to achieve higher quality withou
An advantage of using a feedback channel is the low extra complexity required to implement robust coding because the encoder receives specific information about transmission errors. This allows straightforward spatio-temporal localization of the lost data without computational effort.
Table~\ref{tab:overall_gain_ratio} shows the average quality differences between the reference method (\REF) and the proposed one plus three other previous works. Two different test conditions are shown in the table: error-free and PLR=10\%. Moreover, an efficiency measure $R$ is defined, as the ratio between the quality gains obtained under packet loss and the quality decrease in the error-free case, i.e.,
Table~\ref{tab:overall_gain_ratio} shows the average quality differences (\textbf{$\mathbf{\Delta}$ PSNR})between the reference method (\REF) and the proposed one plus three other previous works. Two different test conditions are shown in the table: error-free and PLR=10\%. Moreover, an efficiency measure $R$ is defined, as the ratio between the quality gains obtained under packet loss and the quality decrease in the error-free case, i.e.,
\begin{equation}
R = \frac{\Delta PSNR(\ac{plr}=10\%)}{-\Delta PSNR(Error\text{-}free)}.
\end{equation}
@@ -703,7 +704,7 @@ R = \frac{\Delta PSNR(\ac{plr}=10\%)}{-\Delta PSNR(Error\text{-}free)}.
The results shown in Table~\ref{tab:overall_gain_ratio}, demonstrate that the effectiveness of the \ac{ec}-aware coding scheme is higher than all other methods under comparison. In fact, not only the absolute \ac{psnr} difference (i.e., \ac{psnr} gain) is higher for the same \ac{plr}, but also the efficiency of the overhead is also higher.
This means that the overhead bits spent to encode the signalling information and the enhancement residue carry more useful information than the other methods.
For instance, in comparison with the method in \cite{DualFrameME2015} the proposed method leads to higher quality degradation in the error-free case. However, for PLR=10\% it clearly outperforms the method in \cite{DualFrameME2015} with a much higher value of $R$, indicating a far superior efficiency.
For instance, in comparison with the method in \cite{DualFrameME2015} the proposed method leads to higher quality degradation in the error-free case. However, for PLR=10\% it clearly outperforms the method in \cite{DualFrameME2015}, achieving an higher $R$, indicating a superior efficiency.
Results in Table~\ref{tab:overall_gain_ratio} also show that the small loss of coding efficiency in error-free transmission is clearly surpassed by the quality gains under error-prone transmission conditions.
The methods in \cite{ROPERefGenJan2010,DualFrameME2015} are based on modified predictions, while the method in \cite{VPxResiliencePCS2016} uses redundant data without any further optimisation.
Both of these approaches require low additional computational complexity.
@@ -717,13 +718,13 @@ The proposed method uses \ac{ec} techniques with different levels of computati
\begin{table}[t]
\newcommand{\somespaces}{}
\centering
\caption{Comparison against existing methods.}\label{tab:overall_gain_ratio}
\caption{Comparison of the proposed EC-aware robust coding architecture against existing in error-free and packet loss environments.}\label{tab:overall_gain_ratio}
\footnotesize
% \resizebox{\columnwidth}{!}{
\begin{tabular}{p{2.4cm}x{2.4cm}x{2.4cm}x{2cm}}
% \begin{tabular}{l|c|c|c}
\toprule
\multirow{2}{*}{\vbox{\textbf{Method}}} & \multicolumn{2}{c}{\textbf{$\mathbf{\Delta}$ PSNR}} & \multirow{2}{*}{\centering \vbox{$\mathbf{R}$ }} \\
\multirow{2}{*}{\vbox{\textbf{Method}}} & \multicolumn{2}{c}{\textbf{$\mathbf{\Delta}$ PSNR (dB)}} & \multirow{2}{*}{\centering \vbox{$\mathbf{R}$ }} \\
& {Error-free} & {PLR=10\%} & \\
% \hline
\midrule
042_ConcealmentAwareEncoding/Results/ECMQualitySingleloss.tex
+2 -2
View File
@@ -547,7 +547,7 @@ e
# e
\end{gnuplot}
\vspace{-4mm}
\caption{Error propagation after losing the inter-coded frame $\#6$.} %for Basketball Drill and People On Street sequences.}
\caption{Error propagation after losing the inter-coded frame $\#6$ for the proposed EC-aware approach and three different EC reference methods.} %for Basketball Drill and People On Street sequences.}
\label{FIG5:singlelossInter}
\FIGSVSPACE
\vspace{5mm}
@@ -1026,7 +1026,7 @@ e
# e
\end{gnuplot}
\vspace{-4mm}
\caption{Error propagation after losing intra-coded frame $\#16$.} % for Kimono and Tennis sequences.}
\caption{Error propagation after losing intra-coded frame $\#16$ for the proposed EC-aware approach and three different EC reference methods.}
\label{FIG5:singlelossIntra}
\FIGSVSPACE
\end{figure}
042_ConcealmentAwareEncoding/Results/TableOveralPerformance.tex
+2 -2
View File
@@ -57,7 +57,7 @@
\begin{table}[t]
\centering
\caption{Average quality under 3\% of PLR for different methods and overhead ratios.}
\caption{Average PSNR (dB) under 3\% of PLR for different methods and overhead ratios.}
\label{tab:concealmentResidueRandomLoss_PLR3}
\small
\footnotesize
@@ -100,7 +100,7 @@ Tennis & 33.86 & +0.16 & +0.18 & +0.98 & +1.48 \\ \hline
\begin{table}[t]
\centering
\caption{Average quality under 10\% of PLR.}
\caption{Average PSNR (dB) under 10\% of PLR for different methods and overhead ratios.}
\label{tab:concealmentResidueRandomLoss_PLR10}
\small
\footnotesize