jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Review all chapters and start conclusions

Browse Source
This commit is contained in:
jfmcarreira
2018-09-12 02:44:20 +01:00
parent 461b560b17
commit 8d3a4c72ff
20 changed files with 304 additions and 427 deletions
Download Patch File Download Diff File Expand all files Collapse all files
042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex
+37 -36
View File
@@ -120,7 +120,7 @@ Besides a full search algorithm, where a block-based approach is used to search
\begin{enumerate}
\item a method based on the total variation using the minimization of the \mbox{L-1} norm~\cite{DualFlow2016};
\item a method which approximates the neighbourhood of each pixel by quadratic polynomials on both frames~\cite{PolyFlow2003};
\item a method based on a combination of feature matching and total variation~\cite{DeepFlow2013}. \todo{check here!}
\item a method based on a combination of feature matching and total variation~\cite{DeepFlow2013}. %\todo{check here!}
\end{enumerate}
\subsubsection{Motion estimation based on total variation and regularisation (\textit{DualFlow})}
@@ -321,7 +321,7 @@ Furthermore, image saliency driven by visual attention modelling is used in the
% \HIGHLIGHT{where $R(l)$ represents the number of overhead bits required to encode \ac{ec} mode, and $D(l)$ is mean square error of the \ac{ctu} formed by $4^l$ ECUs.
% }
% \HIGHLIGHT{%
% In equation \ref{eq:optdepth}, besides the parameter $\lambda$ used to trade-off the cost of distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(l)$ according to the duration of the error propagation decreases. This is given by the ratio between the temporal distance from the current frame ($t_0$) to the next refresh frame ($t_I$) and the period of refresh frames ($T_{I}$).
% In equation~\ref{eq:optdepth}, besides the parameter $\lambda$ used to trade-off the cost of distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(l)$ according to the duration of the error propagation decreases. This is given by the ratio between the temporal distance from the current frame ($t_0$) to the next refresh frame ($t_I$) and the period of refresh frames ($T_{I}$).
% }
% \begin{align}
% (ecm^*, l^*) &= arg \min_{\phi \in M} J(ecm, l), \\
@@ -340,7 +340,7 @@ m_{l,i}^* = \argmin\limits_{\footnotesize \substack{l\in\left\{0...3\right\} \\
where $R(m_{l,i})$ represents the number of overhead bits required to encode the \ac{ec} mode, and $D(m_{l,i})$ is the sum of absolute error of the \ac{ecu}.
In equation \ref{EQ6:OptimalDepth}, besides the parameter $\lambda$ used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation. As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques}, this method uses the ratio between the temporal distance from the current frame ($t_C$) to the next refresh frame ($t_I$) and the period of refreshing frames ($T_{I}$).
In equation~\ref{EQ6:OptimalDepth}, besides the parameter $\lambda$ used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation. As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques}, this method uses the ratio between the temporal distance from the current frame ($t_C$) to the next refresh frame ($t_I$) and the period of refreshing frames ($T_{I}$).
%This weight varies between $\left[0;1 \right]$, achieving the lowest value for intra-coded frames and the maximum for the frame located immediately before an intra-coded one.
This behaviour follows the fact that when a lost frame is closer to the next intra-coded frame, error propagation is lower. Thus, it is not worthwhile to spend too many overhead bits to reduce it. This is accomplished by increasing the cost of such bits according to the temporal distance between the missing frame and the next I frame.
Moreover, a dynamic value for $\lambda$ is also used to adjust the amount of overhead within the frame, according to the visual relevance of each image region, determined using a saliency map.
@@ -374,15 +374,6 @@ One should note that this method does not depend on the existence of a saliency
\subsection{Signalling the optimal EC modes}
\begin{figure}[tb]
\centering
\includegraphics[width=0.8\columnwidth]{\ConcealmentAwareDir/Figures/ConcealmentSignalling}
% \input{\ConcealmentAwareDir/Figures/ConcealmentSignalling.tikz}
\caption{Payload structure of the \ac{sei} \acs{nal}} % to signal the optimal error concealment method.}
% Symbols transmitted through \ac{sei} messages to signal the optimal error concealment method.}
\label{FIG5:signalling}
\end{figure}
% \begin{figure}[tb]
% \begin{algorithmic}
% \Procedure{encodectuecm}{$level$}
@@ -408,6 +399,15 @@ Since no split is necessary at the deepest level, then $split=1$ followed by the
This structure allows dynamic selection of different values for the level of the quad-tree partitioning ($l$) within each frame.
Adaptive arithmetic coding~\cite{ArithmeticCodingJun1994} is used to efficiently encode both the $split$ and $ecm$ symbols, further reducing the overhead. Note that the \ac{sei} information uses much less bitrate than the video signal, thus it is very unlikely to be hit during transmission.
\begin{figure}[tb]
\centering
\includegraphics[width=0.8\columnwidth]{\ConcealmentAwareDir/Figures/ConcealmentSignalling}
% \input{\ConcealmentAwareDir/Figures/ConcealmentSignalling.tikz}
\caption{Payload structure of the \ac{sei} \acs{nal}} % to signal the optimal error concealment method.}
% Symbols transmitted through \ac{sei} messages to signal the optimal error concealment method.}
\label{FIG5:signalling}
\end{figure}
\subsection{Enhancement-layer: \ac{ec} residue coding}
@@ -449,10 +449,11 @@ For comparison, it is used the \ac{psnr} obtained from the reference \ac{hevc} d
In this section the performance of the proposed method is evaluated only for intra frames, in order to demonstrate its efficiency in reducing error propagation.
Since intra \acs{cu} result in higher amount of bits, they also require a higher number of packets, making them more prone to transmission errors and losses.
Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets correspond to intra coded slices, which is a non-negligible probability of one of them being hit by transmission errors.
Moreover, by limiting the proposed method to intra frames, the amount of required overhead is reduced.
Moreover, by limiting the proposed method to intra frames, the amount of required overhead is reduced.
In this case a constant $\lambda_0=10$ was considered without taking into consideration saliency estimation.
Since a relevant performance indicator is the error propagation over temporally predicted frames, this was evaluated using a single intra-frame loss.
Figures \ref{FIG5:singlelosslowdelay} and \ref{FIG5:singlelossramdon} show the \ac{psnr} results when frame Intra-frame $\#16$ is missing, for \ac{ld} and \ac{ra} encoding configurations, respectively.
Figures~\ref{FIG5:singlelosslowdelay} and~\ref{FIG5:singlelossramdon} show the \ac{psnr} results when frame Intra-frame $\#16$ is missing, for \ac{ld} and \ac{ra} encoding configurations, respectively.
These figures show the results for the proposed method (\PROP), the motion-copy method (\MC) and also an alternative proposed method using fixed \ac{ecu} size of $16\times16$ pixels.
These results indicate that the proposed method (\PROP) outperforms the reference one and it is able to achieve quality gains up to {2 dB} for \ac{ld} configuration (see Figure~\ref{FIG5:singlelosslowdelay}), decrease the error propagation and increase the quality of the frames affected from the loss of the intra frame, by up to {3 dB}.
The use of \ac{ra} configuration leads to higher quality degradation than LD, due the higher temporal distance of the reference frame used for \ac{ec} (Frame $\#8$ is used to recover the lost Frame~$\#16$). Note that when Intra-frame \#16 is lost, all frames within the range \#9 to \#31 (i.e., next I-frame) are affected by error propagation.
@@ -474,7 +475,7 @@ Further tests were performed to evaluate the effectiveness of the proposed metho
Table~\ref{tab:randomloss} presents the average quality of the frames affected by errors (the lost frame and dependent ones).
This table also shows the \ac{psnr} variance value in parentheses and the difference to the \ac{psnr} obtained with a reference \ac{hevc} decoder.
% The absolute \ac{psnr} value is shown for the reference case (\MC), while the \ac{psnr} difference is presented for the proposed method.
The results confirm the superiority of the proposed method, outperforming the reference \ac{hevc} for both low and high \ac{plr}. Using the \ac{ld} coding configuration, the proposed method is able to achieve quality gains of 0.88 dB for 3\% of \ac{plr} and 1.50 dB for 10\% of \ac{plr}.
The results confirm the superiority of the proposed method, outperforming the reference \ac{hevc} for both low and high \ac{plr}. Using the \ac{ld} coding configuration, the proposed method is able to achieve quality gains of 0.88 dB for 3\% of \ac{plr} and 1.50 dB for 10\% of \ac{plr}.
In case of RA, higher quality gains are achieved, leading to an average improvement of 2.14 dB for 3\% of \ac{plr} and 3.44 dB for 10\% of \ac{plr}.
It is also worthwhile to notice that the \ac{psnr} variance obtained with the \PROP method is lower, which indicates lower quality variations in the video segments affected by errors.
%The quality gains achieved by the proposed method are similar to those achieved by previous works~\cite{ConcealmentIntraDataHidng2009,ConcealmentSignalling2008} for the same \acs{plr}. However, these require a higher amount of overhead (approximately $3\%$), thus the proposed method achieves a better trade-off between quality and overhead.
@@ -523,10 +524,10 @@ It is also worthwhile to notice that the \ac{psnr} variance obtained with the
%\subsubsection{Overhead analysis}
Subsequently, the percentage of overhead bitrate introduced by the proposed method is compared against fixed \ac{ecu} sizes of $32\times32$ and $16\times16$, i.e., $Level=1$ and $Level=2$.
These results are presented in Table \ref{TAB6:RedundancyIntra} and were obtained for a total bitrate, as shown in Table~\ref{TableTestSequences}.
These results are presented in Table~\ref{TAB6:RedundancyIntra} and were obtained for a total bitrate, as shown in Table~\ref{TableTestSequences}.
It is noticeable from the results that reducing the \ac{ecu} size (see results for $32\times32$ and $16\times16$) increases the overhead ratios as more symbols are transmitted (one for each \ac{ecu}).
While the maximum overhead of these fixed partitions is $1.81\%$ (Traffic), the proposed dynamic partitioning method is able to reduce the overhead to a maximum of $0.19\%$ (Kendo).
Moreover, comparing the results of Table \ref{TAB6:RedundancyIntra} with the video quality results obtained in case of packet losses reveals in the proposed method only a small reduction in the reconstruction quality, even with a significantly lower overhead.
Moreover, comparing the results of Table~\ref{TAB6:RedundancyIntra} with the video quality results obtained in case of packet losses reveals in the proposed method only a small reduction in the reconstruction quality, even with a significantly lower overhead.
This indicates that using dynamic partitioning allows to achieve a better trade-off between the amount of overhead and the \ac{ec} performance.
In summary, the proposed technique is able to enhance the performance of the standard decoding of intra coded frames by finding the best error concealment method at the encoder and sending such information with a small amount of signalling overhead.
@@ -582,11 +583,11 @@ This section is organised in the following subsections:
\subsubsection{\ac{ec} mode usage distribution}
The \ac{ec} mode usage distribution is an indicator of the usefulness of each \ac{ec} mode.
Table \ref{tab:useratio} shows the average $Level$ selected for the \ac{ecu} quadtree and the usage percentage of each \ac{ec} mode.
Table~\ref{tab:useratio} shows the average $Level$ selected for the \ac{ecu} quadtree and the usage percentage of each \ac{ec} mode.
It is noticeable from these results that the proposed method leads to low average $Level$ values (lower than 1), resulting in larger \acs{ecu}, which requires a lower amount of \ac{ecu} to be transmitted than if higher values of $Level$ were chosen. This implicitly results in low signalling information.
This also justifies the low overhead shown in Table~\ref{tab:redundancy_signalling}.
The results in Table \ref{tab:useratio} also show that all \ac{ec} modes are effectively used, which indicates that all of them are found relevant to recover the missing data.
The results in Table~\ref{tab:useratio} also show that all \ac{ec} modes are effectively used, which indicates that all of them are found relevant to recover the missing data.
The \ac{ec} candidate methods based on motion field estimation (i.e., M1, M2) are selected more often because they provide more accurate representation of the scene's motion than the others (i.e., M3, M4).
\begin{table}[tb]
@@ -617,10 +618,10 @@ The \ac{ec} candidate methods based on motion field estimation (i.e., M1, M2) ar
\subsubsection{Evaluation of optimal \ac{ec} mode decision}
\label{sec:ECModeSelectionPerformance}
The evaluation of the proposed saliency-weighted \ac{ec} mode decision (\DYNLAMBDA) was carried out under random loss events and it was compared with the reference methods.
The evaluation of the proposed saliency-weighted \ac{ec} mode decision (\DYNLAMBDA) was carried out under random loss events and it was compared with the reference methods.
The video quality was evaluated by using two quality metrics: \ac{psnr} and a \ac{fsim}~\cite{FSIM_TIP2011}, in order to validate the performance of the proposed method in a consolidated manner.
Realistic conditions were used for transmission, by allowing independent losses for any type of \acs{nal}, hence there is no guarantee that the decoder conditions are always equal to those at the encoder, which means that sub-optimal \ac{ec} may be used at the decoder.
The results obtained without saliency weighting are also shown for comparison (\FIXEDLAMBDA). In this case a constant $\lambda_0=10$ is used.
The results obtained without saliency weighting are also shown for comparison (\FIXEDLAMBDA). As in the previous subsection, a $\lambda_0=10$ was used.
Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM} show the average quality under different \acs{plr} for the \ac{ld} configuration.
Table~\ref{TAB4:SignallingPSNR} shows the \ac{psnr} and weighted \ac{psnr} (\ac{wpsnr}) between parentheses.
@@ -635,26 +636,26 @@ Moreover, the results of Table~\ref{TAB4:SignallingFSIM} confirm the quality imp
Table~\ref{tab:redundancy_signalling} shows the percentage of overhead in the total rate with and without saliency-weighted optimisation, i.e., \FIXEDLAMBDA and \DYNLAMBDA, respectively.
As shown in this table, the saliency-weighted optimisation reduces the amount of overhead by dynamically assigning unequal number of bits to image regions of different visual importance.
These results show that using an adaptive value of $\lambda$, based on the saliency values, leads to an average overhead reduction of 28\% in comparison with the case where no saliency is used, i.e., constant $\lambda$.
It is worth to note that using the saliency-weighted optimisation to reduce the overhead does not compromise the video quality, as can be seen in the results shown in Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM}.
It is worth to note that using the saliency-weighted optimisation to reduce the overhead does not compromise the video quality, as can be seen in the results shown in Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM}.
The \ac{psnr} values for \FIXEDLAMBDA and \DYNLAMBDA are very close, but in terms of overhead savings the results are significant for most sequences.
The achieved \ac{wpsnr} and \ac{fsim} results further demonstrate the effectiveness of the saliency-based optimisation, because the lower overhead does not affect the most relevant image regions, i.e., those with higher saliency values. Thus, the proposed method using a adaptive $\lambda$ (i.e., using saliency) is able to not only achieve higher \ac{ec} quality than the reference methods, but also higher quality gains than using a constant $\lambda$ (i.e., without saliency).
This confirms the two-fold superiority of adaptive $\lambda$, i.e. not only the total overhead is reduced but also higher quality gains are achieved.
This confirms the two-fold superiority of adaptive $\lambda$, i.e. not only the total overhead is reduced but also higher quality gains are achieved.
%\todo{\HERE}
\subsubsection{Error propagation evaluation}
In this section the error propagation is evaluated using single loss events and compared against the reference methods.
In this simulation a whole frame loss is enforced in order to assess the impact of error propagation in the worst case scenario.
In this section the error propagation is evaluated using single loss events and compared against the reference methods.
In this simulation a whole frame loss is enforced in order to assess the impact of error propagation in the worst case scenario.
Experimental evaluations were performed for both intra and inter-coded frame loss.
Figure \ref{FIG5:singlelossInter} shows the \ac{psnr} over a \ac{gop} with an \ac{idr} period of 16 frames, where the whole frame \#6 is missing in sequences Basketball Drill and People On Street.
%The figure shows the results obtained by decoding the stream using the proposed method with adaptive $\lambda$ but without any information sent in the enhancement layer.
Figure \ref{FIG5:singlelossIntra} shows the \ac{psnr} for the same conditions but with a loss event on the intra-coded Frame \#16, in sequences Kimono and Tennis.
Figure~\ref{FIG5:singlelossInter} shows the \ac{psnr} over a \ac{gop} with an \ac{idr} period of 16 frames, where the whole frame \#6 is missing in sequences Basketball Drill and People On Street.
%The figure shows the results obtained by decoding the stream using the proposed method with adaptive $\lambda$ but without any information sent in the enhancement layer.
Figure~\ref{FIG5:singlelossIntra} shows the \ac{psnr} for the same conditions but with a loss event on the intra-coded Frame \#16, in sequences Kimono and Tennis.
\input{042_ConcealmentAwareEncoding/Results/ECMQualitySingleloss.tex}
These results reveal that the proposed method is able to improve the reconstruction of inter-coded frames and also outperforms the existing techniques, whenever a intra-coded frame is missing.
These results also indicate that the proposed method outperforms both reference methods since the time instant when the loss occurs. The lost frame itself is recovered with much higher quality than using the reference \ac{ec} methods.
These results also indicate that the proposed method outperforms both reference methods since the time instant when the loss occurs. The lost frame itself is recovered with much higher quality than using the reference \ac{ec} methods.
This is due to the use of the optimal \ac{ec} mode to recover each \ac{ecu}, benefiting from the \ac{sei} information received from the encoder.
In this case, the proposed method is able to achieve quality gains up to 5~dB, decreasing the error propagation and increasing the \ac{psnr} of affected frames by up to 6~dB.
@@ -702,9 +703,9 @@ The visible artefacts demonstrate how the decoder is able to recover from whole
\FIGSVSPACE
\end{figure}
A closer look at the Race Horses sequence reveals that the horse rider is not correctly reconstructed by the \MC method (highlighted by the circles and ellipses). As shown in the figure, this not only results in a more accurate reconstruction of the lost frame, but also increases the visual quality of the subsequent frames.
In the Kimono sequence, some artefacts are also noticeable in the \MC case, but not visible when the proposed method is used (highlighted with circles).
Moreover, the edges of the women's kimono (highlighted with ellipses) are also improved by the proposed method, resulting in a smooth reconstruction without error propagation.
A closer look at the Race Horses sequence reveals that the horse rider is not correctly reconstructed by the \MC method (highlighted by the circles and ellipses). As shown in the figure, this not only results in a more accurate reconstruction of the lost frame, but also increases the visual quality of the subsequent frames.
In the Kimono sequence, some artefacts are also noticeable in the \MC case, but not visible when the proposed method is used (highlighted with circles).
Moreover, the edges of the women's kimono (highlighted with ellipses) are also improved by the proposed method, resulting in a smooth reconstruction without error propagation.
From the results discussed above, it is clearly observed that the proposed method is able to reconstruct the contours of the moving parts more accurately than traditional \ac{ec} algorithms.
@@ -712,7 +713,7 @@ From the results discussed above, it is clearly observed that the proposed metho
The achieved results were compared with previous works, where fair comparisons could be made.
In \cite{ConcealmentSignalling2008} a method was proposed to improve the recovery quality by signalling the suitable \ac{mv} to be used for \ac{ec}.
The proposed \ac{ec}-aware scheme of Figure \ref{FIG5:BlockDiagram} is able to achieve similar quality gains with less amounts of redundancy, allowing a better trade-off between quality and overhead.
The proposed \ac{ec}-aware scheme of Figure~\ref{FIG5:BlockDiagram} is able to achieve similar quality gains with less amounts of redundancy, allowing a better trade-off between quality and overhead.
In comparison with the work described in \cite{SignallingRobustScalable2015}, the proposed method is able to select among more complex \ac{ec} methods, which allows for higher quality gains.
Both of the above mentioned methods search for the optimal \ac{ec} method at the encoder-side, which results in similar complexity as the proposed method.
@@ -729,12 +730,12 @@ R = \frac{\Delta PSNR(\ac{plr}=10\%)}{-\Delta PSNR(Error\text{-}free)}.
\end{equation}
The results shown in Table~\ref{tab:overall_gain_ratio}, demonstrate that the effectiveness of the \ac{ec}-aware coding scheme is higher than all other methods under comparison. In fact, not only the absolute \ac{psnr} difference (i.e., \ac{psnr} gain) is higher for the same \ac{plr}, but also the efficiency of the overhead is also higher.
The results shown in Table~\ref{tab:overall_gain_ratio}, demonstrate that the effectiveness of the \ac{ec}-aware coding scheme is higher than all other methods under comparison. In fact, not only the absolute \ac{psnr} difference (i.e., \ac{psnr} gain) is higher for the same \ac{plr}, but also the efficiency of the overhead is also higher.
This means that the overhead bits spent to encode the signalling information and the enhancement residue carry more useful information than the other methods.
For instance, in comparison with the method in \cite{DualFrameME2015} the proposed method leads to higher quality degradation in the error-free case. However, for PLR=10\% it clearly outperforms the method in \cite{DualFrameME2015} with a much higher value of $R$, indicating a far superior efficiency.
Results in the table~\ref{tab:overall_gain_ratio} also show that the small loss of coding efficiency in error-free transmission is clearly surpassed by the quality gains under error-prone transmission conditions.
The methods in \cite{ROPERefGenJan2010,DualFrameME2015} are based on modified predictions, while the method in \cite{VPxResiliencePCS2016} uses redundant data without any further optimisation.
Both of these approaches require low additional computational complexity.
Both of these approaches require low additional computational complexity.
However, as observed in the results shown in Table~\ref{tab:overall_gain_ratio}, the lower additional complexity also corresponds to lower quality gains in case of transmission errors.
@@ -773,6 +774,6 @@ The proposed method uses \ac{ec} techniques with different levels of computati
In this chapter an \ac{ec}-aware encoding scheme is proposed by incorporating loss simulation at the encoder-side to enhance the EC efficiency at the decoder.
% The proposed method relies on loss simulation at the encoder-side to optimise the error concealment proposed. Then, the optimal error concealment is signalled to the decoder.
The novelty of this method is to include optimal \ac{ec} mode decision at the encoder, to be used by the decoder in case of packet loss, and also to allow encoding of the \ac{ec} residue using the enhancement layer of the \ac{hevc} standard.
The novelty of this method is to include optimal \ac{ec} mode decision at the encoder, to be used by the decoder in case of packet loss, and also to allow encoding of the \ac{ec} residue using the enhancement layer of the \ac{hevc} standard. The use of \ac{ec} signalling was firstly proposed for the use on intra-coded frames in \ref{Conf:ConcAware}, and then improved and proposed for a generic robust video transmission, resulting in a more detailed analysis published in \ref{Journal:ConcAware}.
The overall results demonstrate that the proposed method is able to efficiently reconstruct lost frames and reduce the error propagation compared with other reference methods.
A consistent quality improvement achieved at the cost of low overhead allows the use of this method in broadcasting video services and applications where packet loss probability is not negligible.