jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Review chapter 7

Browse Source
This commit is contained in:
jfmcarreira
2018-12-15 01:41:56 +00:00
parent dd6023d285
commit 15f95a314b
1 changed files with 20 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
043_DynamicErrorRobustness/DynamicErrorRobustness.tex
+20 -20
View File
@@ -34,7 +34,7 @@ In the scope of this thesis different robust coding techniques were investigated
Different algorithms were devised in the proposed methods, such as, reference frame selection, motion vector redundancies and error concealment signalling.
In the research work described in Chapters~\ref{Chapter:ErrorResilienceReview}~to~\ref{Chapter:ConcealmentAwareTechniques}, these methods were evaluated against existing techniques, revealing superior performance and better efficiency to deal with error robustness problems in high efficient video coding.
Overall, the performance results of these methods have demonstrated that they can efficiently increase the robustness of video streams against transmission errors and losses, leading to higher decoded video quality.
Overall, the performance results of these methods have demonstrated that they can efficiently increase the robustness of video streams against errors and data loss, leading to higher decoded video quality.
However, each method on its own does not adapt to different conditions.
For instance, while the reference frame selection algorithm presented in Chapter~\ref{Chapter:ErrorResilienceTechniques} introduces low overhead in comparison with existing techniques, the method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} achieves higher quality gains at cost of slightly higher bitrate increase.
Therefore, one may conclude that each method may be further optimised for different applications, video characteristics and network conditions.
@@ -46,7 +46,7 @@ This scheme can be used in transmission systems where different video bitstreams
In order to address the above problem of adaptive robust coding, a simulation study was carried out to compare the error robustness performance of the previously proposed methods for different conditions.
This study extends the previous evaluations presented in Chapters~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, as they target a wider range of test material and network conditions.
Moreover, it aims at comparing the performance of different techniques across a variety of video content, bitrates and loss conditions, in order to find relevant insights about the most affect the decoded video quality. Consequently, these relevant insights may be used to develop an adaptive robust coding method.
Moreover, it aims at comparing the performance of different techniques across a variety of video content, bitrates and loss conditions, in order to find relevant insights on how the decoded video quality varies with each parameter. Consequently, these relevant insights may be used to develop an adaptive robust coding method.
%In order to perform a wider evaluation different bitrate and \acs{plr} were used. Table~\ref{Table6:TestVariables} illustrates the selected configurations. As shown in the table a total of 15 test sequences were used encoder using 5 different target bitrates. One should note that the main bitrate varies based on the test sequence and it is shown in third column of Table~\ref{TableTestSequences}. Different network conditions were evaluated using a total of 13 different \acs{plr}, with higher special focus on lower loss ratios, which are more relevant to modern video streaming applications.
The following methods were evaluated in the simulation study, namely:
@@ -137,8 +137,8 @@ The proposed approach results in four different bitstreams with different charac
(iii) \SMEXPELTEN: robust video coding using selective reference frames and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue;
(iv) \SMEXPELTWEN: same as the previous bitstream but with 20\% bitrate allocated for the scalable~\ac{el}.
The video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and feedback from the transmission channel.
From the input video signal, relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, further information is also extracted from the coded bitstream using a stream parser.
The video stream with the optimal error robustness is selected using an \ac{nn}, which receives different parameters extracted from the video signal, the compressed bitstream and information from the network conditions.
From the input video, relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, further information is also extracted from the coded bitstream using a stream parser.
Finally, the packet loss ratio of the transmission channel is also used.
These parameters are fed into the optimisation block to select the best bitstream for transmission.
The optimisation process is described in detail in the following section.
@@ -165,7 +165,7 @@ This section describes both the input parameters and the neural network used in
The proposed method includes an optimisation stage that uses a neural network classifier, in order to select the optimal stream to be used for transmission.
This stream is able to obtain the highest quality for a particular transmission scenario.
The stream that should be selected is the one which obtains the highest quality for a particular transmission scenario.
Therefore, several test cases were simulated in order to obtain results for a wide range of conditions. These tests are mainly characterised by three elements:
\begin{itemize}
\item Video signal;
@@ -190,10 +190,10 @@ From the input video the two parameters are measured in a frame by frame basis,
The characterisation of these three aspects of the video transmission results in a total of six parameters:
\begin{enumerate}
\item \textit{\ac{si}}: parameter expressing the amount of spatial detail, with impact on the error recovery performance~\cite{ITUTP9102008}.
\item \textit{\glsfirst{si}}: parameter expressing the amount of spatial detail, with impact on the error recovery performance~\cite{ITUTP9102008}.
\item \textit{\ac{ti}}: parameter expressing the amount of difference between temporally adjacent frames~\cite{ITUTP9102008}.
This has a direct relation in the impact of errors and performance of the error resilience techniques, as shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, which were discussed in Section~\ref{Section6:Problem}.
\item \textit{\glsfirst{ti}}: parameter expressing the amount of difference between temporally adjacent frames~\cite{ITUTP9102008}.
This has a direct relation to the impact of errors and performance of the error resilience techniques, as shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, which were discussed in Section~\ref{Section6:Problem}.
\item \textit{Inter (percentage)}: the percentage of the image that is compressed using inter-frame prediction techniques.
This is an indication of the distribution of spatial and temporal dependencies.
@@ -201,11 +201,11 @@ This is calculated as the ratio between inter-coded blocks and intra-coded ones.
Normally, coded streams with higher number of temporal dependencies are more prone to error propagation, thus higher number of bits should be allocated for error protection \cite{UEPScalable_TCSVT2010}.
\item \textit{AVG\_MV}: for the inter-coded regions, the motion information is analysed and the average \ac{mv} magnitude is calculated to provide information about the motion intensity.
As shown in previous studies, motion-based \ac{ec} algorithms are less efficient for video signals with higher motion intensity, as \acs{mv} are less predictable using algorithms, such as \ac{mc} or \ac{mve} \cite{HybridConcealmentTB2008}.
As shown in previous studies~\cite{PixelBasedMVE2004,PelBasedMVE_TCE2007,HybridConcealmentTB2008}, motion-based \ac{ec} algorithms are less efficient for video signals with higher motion intensity, as \acs{mv} are less predictable using algorithms, such as \ac{mc} or \ac{mve}.
\item \textit{Bitrate}: information regarding the average bit allocation per frames is calculated, since it will affect the video quality and the amount of bits that can be available for auxiliary information. This value is expressed in Mbps, as illustrated in Table~\ref{Table6:TestMaterialCharacteristics}.
\item \textit{\ac{plr}:} feedback information from the network to characterise the transmission losses, expressed by the percentage of packets loss. This is used since for higher loss ratio there is a clear increase in the quality degradation, which affects the performance of the robust coding techniques.
\item \textit{\glsfirst{plr}:} feedback information from the network to characterise the transmission losses, expressed by the percentage of packets loss. This is used since for higher loss ratio there is a clear increase in the quality degradation, which affects the performance of the robust coding techniques.
Transmission networks with higher loss ratios require a higher bit allocation for robust techniques, therefore the error resilience technique should be able to adapt based on the value of the \ac{plr}.
@@ -236,15 +236,14 @@ This aims at finding the correlation between the input parameters and the output
In this study the streams are characterised by a number between 1 and 4.
Table~\ref{Table6:Correlation} shows the absolute Person Correlation coefficient obtained across all sequences, bitrates and \acs{plr}.
The table shows the results for the parameters previously described, as well as, a new parameter obtained from the average magnitude of the optical flow measured using the method in~\cite{DeepFlow2013} (\textit{OPT\_FLOW}).
Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable.
Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks having the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable.
Since both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method, due to its higher Person Correlation coefficient.
\subsection{Neural network}
A \ac{nn} is a computational model based on the structure and functions of biological neural networks.
%In terms of computer science, \acs{nn} work similarly to a human nervous system for receiving, processing, and transmitting information between neurons.
Mainly, a \ac{nn} consists of three different layers: the input layer which receives data from an external environment, the hidden layer which process the inputs and an output layer~\cite{NeuralNetworks1994}.
Mainly, a \ac{nn} consists of three different layers: the input layer which receives data from an external environment, the hidden layer which processes the inputs and an output layer~\cite{NeuralNetworks1994}.
In the past \acs{nn} have been widely used in video processing applications such as, quality evaluation~\cite{VideoQualityNN_TNN2006}, video analysis~\cite{VideoAnalysisNN_2018} and video coding~\cite{RateDistortionNN_VCIP2017,HEVCComplexityNN_TIP2018}.
In the proposed method a feed-forward network is used, which connects every neuron in a given layer to the neurons of the subsequent layer.
@@ -263,17 +262,18 @@ Subsequently, the network transforms the input data through a series of neurons
The output layer comprises four neurons corresponding to each available video stream from \SMEC to \SMEXPELTWEN (different error resilience techniques).
Finally, the output with the highest value is selected and the corresponding stream is used for transmission, since it is the one expected to achieve the highest video quality for a particular scenario.
In the proposed approach a shallow \ac{nn} is used for simplicity and to avoid over-fitting, due to the limited test cases.
In the proposed approach a shallow \ac{nn} using three layer, with 32 nodes in the hidden layer is used for simplicity and to avoid over-fitting, due to the limited test cases~\cite{NeuralNetworkDesign_2014}.
By avoiding over-fitting, it is guaranteed that the proposed method would continue to work in future test conditions, not used for the network training.
The number of nodes in the hidden layer was determined by comparing the performance obtained for different number of neurons in this layer using the cross-entropy performance metric~\ref{CrossEntropy_2005}.
However, different network topologies with different number of neurons in each layer could also be used to predict between different error resilience techniques, as in the proposed method.
\subsubsection{Training and validation}
%As mentioned before a total of \NumCases test cases are used for the evaluation of the proposed method.
To effectively train and validate the \ac{nn}, firstly the cases corresponding to three video sequence were left out, to be used in the performance evaluation of the proposed method.
The remaining data set was randomly partitioned into three different groups: (i) 65\% of the test cases are used for training, (ii) 15\% for validation and (iii) 20\% for testing the \ac{nn} performance.
The \ac{nn} supervised training uses 557 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993}.
Then, 129 test conditions were used for \ac{nn} validation.
The remaining data set was randomly partitioned into three different groups: (i) 65\% of the test cases are used for training, (ii) 15\% for validation and (iii) 20\% for testing the \ac{nn} performance.
\HIGHLIGHT{These ratios were chosen to use a relevant number of samples for testing \ac{nn} performance (i.e., 20\%), and the remaining were used mainly for training, leaving a 15\% samples for validation and stop criteria of the supervised training.
The \ac{nn} supervised training uses 557 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993} and stop when the cross-entropy obtained with the validation samples increases, revealing that the training starts to over-fit the training samples and is not being general enough to predict new data.
Therefore, 129 test conditions were used for \ac{nn} validation.}
Finally, 172 conditions are used to evaluate the accuracy of the proposed method after training with different conditions.
Tables~\ref{Table6:ConfusionTraining}, \ref{Table6:ConfusionVal} and \ref{Table6:ConfusionTest} show the confusion matrices obtained for training, validation and test sets, respectively.
@@ -385,7 +385,7 @@ In case of data loss, the results show that the proposed method is able to match
\begin{table}[tb]
\centering
\caption{Average quality gains ($\Delta$PSNR) in comparison with the reference \ac{hevc} across different bitrates and packet loss ratios.}
\caption{Average quality gains ($\Delta$PSNR in dB) in comparison with the reference \ac{hevc} across different bitrates and packet loss ratios.}
\label{Table6:AverageGains}
\TABLESVSPACE
\newcommand{\colwidth}{2cm}
@@ -445,7 +445,7 @@ This confirms the advantage of the proposed method to adaptively select among di
\begin{table}[tb]
\centering
\caption{Average quality gain ($\Delta$PSNR) in comparison with the reference \ac{hevc} for video sequences not used in the NN training.}
\caption{Average quality gain ($\Delta$PSNR in dB) in comparison with the reference \ac{hevc} for video sequences not used in the NN training.}
\label{Table6:AverageGainBasketBallDrive}
\TABLESVSPACE
\newcommand{\colwidth}{1.7cm}