jfmcarreira/doc_phd_thesis
1
0 Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Review all chapters and start conclusions

Browse Source
This commit is contained in:
jfmcarreira committed 2018-09-12 02:44:20 +01:00
1 parent 461b560b17
commit 8d3a4c72ff
20 files changed
+304 -427
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

043_EndToEndRobustSystem/EndToEndRobustSystem.tex
+82 -76
View File
@@ -1,7 +1,7 @@
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CHAPTER
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Adaptive Error Robustness for Highly Efficient Video Coding \VERSION{1}}
\chapter{Adaptive Error Robustness for Highly Efficient Video Coding \VERSION{2}}
\label{Chapter:EndToEndRobustSystem}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -15,13 +15,13 @@
The effectiveness of different error resilience techniques is rather variable depending on the video content, the transmitted bitrate, network conditions, among others.
Therefore, it is not straightforward to select a single technique which achieves the highest efficiency for every possible scenario and also for the whole range of networking conditions.
This chapter addresses this problem by proposing an adaptive network which optimises the efficiency of the techniques previously described in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}
At the encoder-side different video streams are generated with different levels of compression efficiency and error robustness.
Then, a pre-trained neural network is used to predict which video stream will achieve the highest decoded video quality for a particular loss case.
The selection is based on different input parameters which include information from the video signal, the coded stream and the transmission network. Using this approach the overall performance of the proposed adaptive method significantly increases yielding a better trade-off between compression efficiency and error robustness.
This chapter addresses this problem by proposing an adaptive network that optimises the efficiency of the techniques previously described in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}.
At the encoder-side, a pre-trained neural network predicts the video stream that is able to achieve the highest decoded quality for a particular loss cases, among the various video streams encoded with different levels of compression efficiency and error robustness.
The selection is based on different input parameters that include information from the video signal, the coded stream and the transmission network.
Using this approach the overall performance of the proposed adaptive method significantly increases, yielding a better trade-off between compression efficiency and error robustness.
This chapter is organised as follows.
Section~\ref{Section6:Problem} presents the problem addressed in this chapter and experimental evidence as motivation for the method investigated.
Section~\ref{Section6:Problem} presents the problem addressed in this chapter and experimental evidence, as motivation for the method investigated.
Section~\ref{Section6:ProposedMethod} provides an overview of architecture that supports the implementation of the proposed method and Section~\ref{Section6:OptimisationMethod} describes the technical details.
Section~\ref{Section6:ExperimentalResults} presents the performance evaluation and a discussion of results.
Finally, Section~\ref{Section6:Summary} concludes the chapter.
@@ -42,7 +42,7 @@ However, each method on its own does not adapt to different conditions.
While the reference frame selection algorithm presented in Chapter~\ref{Chapter:ErrorResilienceTechniques} introduces low overhead in comparison with existing techniques, the method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} achieves higher quality gains at cost of slightly higher bitrate increase.
Therefore, one may conclude that each method may be further optimised for different applications, video characteristics and network conditions.
This chapter advances one step forward , by investigating a joint robust coding method using different techniques to optimise the robust coder across different conditions, such as, the input content and the network information.
This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and can selected based on the current conditions.
This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and can be selected based on the current conditions.
\subsection{Comparison of the proposed methods}
@@ -52,24 +52,24 @@ This study extends the previous evaluations presented in Chapter~\ref{Chapter:Er
Moreover, it aims at comparing the performance of different techniques across a variety of video contents, bitrates and loss conditions, in order to find relevant insights which affect the decoded video quality. Consequently, these relevant insights may be used to develop an adaptive robust coding method.
%In order to perform a wider evaluation different bitrate and \acs{plr} were used. Table~\ref{Table6:TestVariables} illustrates the selected configurations. As shown in the table a total of 15 test sequences were used encoder using 5 different target bitrates. One should note that the main bitrate varies based on the test sequence and it is shown in third column of Table~\ref{TableTestSequences}. Different network conditions were evaluated using a total of 13 different \acs{plr}, with higher special focus on lower loss ratios, which are more relevant to modern video streaming applications.
The following methods are evaluated in the simulation study, namely:
The following methods were evaluated in the simulation study, namely:
\begin{itemize}
\item the default \ac{hevc} decoder with frame-copy \ac{ec} (\REF), as reference method;
\item the default \ac{hevc} encoder without robust coding-techniques and motion vector extrapolation \ac{ec} (\MVE);
\item the proposed reference frame selection scheme (\EXP);
\item the \EXP method in combination with an \ac{ec} optimisation with 10\% of bitrate allocate for the scalable enhancement layer (\EXPELTEN);
\item the same as above, with 20\% of bitrate allocate for the scalable enhancement layer (\EXPELTWEN).
\item the \EXP method in combination with an \ac{ec} optimisation with 10\% of bitrate allocated for the scalable enhancement layer (\EXPELTEN);
\item the same as above, with 20\% of bitrate allocated for the scalable enhancement layer (\EXPELTWEN).
\end{itemize}
These methods were selected as they are the main techniques proposed within this research work and cover two aspects of a robust video transmission: (i) robust video encoding by dynamically select the reference frames and (ii) robust video decoding by improving the error concealment performance.
\input{043_EndToEndRobustSystem/Results/AdaptiveSequenceList.tex}
Table~\ref{Table6:TestMaterialCharacteristics} presents the list of test sequences used. These sequences were selected to cover a wide range of video content, with different levels of spatial and temporal complexity, as demonstrated by the values of \ac{si} and \ac{ti}, respectively.
Table~\ref{Table6:TestMaterialCharacteristics} presents the list of test sequences used. These sequences were selected to cover a wide range of video content, with different levels of spatial and temporal complexity, as demonstrated by the values of \ac{si} and \ac{ti}~\cite{ITUTP9102008}, respectively.
Moreover, from the bitstream encoded with the reference implementation of the \ac{hevc} encoder, the percentage of the inter-coded blocks and the average \ac{mv} magnitude was extracted and presented in the table (columns 4 and 5, respectively). These values were extracted to compare them against the performance obtained with the proposed robust coding methods.
The remaining configuration were kept as described in Section~\ref{Section3:ErrorResilienceEvaluation}.
The remaining configuration was kept as described in Section~\ref{Section3:ErrorResilienceEvaluation}.
%The experimental methodology carried out in this section follows the same conditions described in Section~\ref{Section3:ErrorResilienceEvaluation}.
\subsubsection{Simulation results and discussion}
\subsubsection{Preliminary simulation results and discussion}
\begin{figure}[!t]
\centering
@@ -92,16 +92,16 @@ Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained by reducin
\label{Figure6:AverageQualityReduction}
\end{figure}
Firstly, comparing the results for different bitrates, it is noticeable that lower quality is obtained for the lower bitrates, which results from the higher compression ratios used. It is also noticed that lower quality gains, when comparing with the reference \ac{hevc} encoder (\REF), are obtained for the video streams encoded with a 40\% reduction of bitrate. This indicates that for lower bitrates the performance of the proposed methods is decreased, especially for the \EXPELTWEN case where 20\% of the bitrate is allocated for the enhancement layer. Therefore, for lower bitrates, using methods that have a lower impact on the coding efficiency (e.g., \EXP) or even disable the robust coding techniques (\MVE) may lead to higher performance.
Firstly, comparing the results for different bitrates, it is noticeable that lower quality is obtained for the lower bitrates, which results from the higher compression ratios used. It is also noticed that lower quality gains, when comparing with the reference \ac{hevc} encoder (\REF), are obtained for the video streams encoded with a 40\% reduction of bitrate. This indicates that for lower bitrates the performance of the proposed methods decreases, especially for the \EXPELTWEN case, where 20\% of the bitrate is allocated for the enhancement layer. Therefore, for lower bitrates, using methods that have a lower impact on the coding efficiency (e.g., \EXP) or even disable the robust coding techniques (\MVE) may lead to higher performance.
Secondly, comparing the results across different video content, i.e., different test sequences, one can notice that the performance of the different methods is not consistent.
For instance, in the result of Figure~\ref{Figure6:AverageQualityMain}, for Tennis sequence the \EXPELTWEN clearly outperforms the remaining methods, however, for the BQSquare sequence higher performance is achieved by using less bitrate allocated for the \ac{el} (i.e., 10\%). This is a consequence of the higher quality degradation due to higher motion in Tennis sequence (see results for TI and \ac{mv} size in Table~\ref{Table6:TestMaterialCharacteristics}) which required higher redundant information to increase the overall quality.
%\HIGHLIGHT{Mention SI!!}
Finally, comparing the results for all test sequences and both bitrates it is clearly noticed that the \acs{plr} has a significant impact on the performance of the error robustness techniques. In case of the proposed reference frame selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions. For the \EXPELTWEN case higher \acs{plr} are required to be able to outperform both the reference \ac{hevc} and the \ac{mve} method.
Finally, comparing the results for all test sequences and both bitrates it can be clearly noticed that the \acs{plr} have a significant impact on the performance of the error robustness techniques. In case of the proposed reference frame selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions. For the \EXPELTWEN case, higher \acs{plr} are required to be able to outperform both the reference \ac{hevc} and the \ac{mve} method.
Summarising, the performance of the proposed methods is not similar across different video content and bitrates, opening research to adaptively choose the best method that should be applied to achieve the highest decoded video quality.
In order to achieve increased efficiency and higher video quality, the best trade-off between auxiliary information and video bitrate should be found by selecting the optimal robust coding algorithm.
In order to achieve increased efficiency and higher video quality, the best trade-off between auxiliary information and video bitrate can be found by selecting the optimal robust coding algorithm.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
@@ -134,7 +134,7 @@ The proposed approach results in four different bitstreams with different charac
(i) S1: reference stream without error robust coding;
(ii) S2: robust video coding using reference frame selection;
(iii) S3: robust video coding using selective reference frames and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue;
(iv) S4: same as the previous bitstream but with 20\% bitrate allocated for the scalable \ac{el}.
(iv) S4: same as the previous bitstream but with 20\% bitrate allocated for the scalable~\ac{el}.
Finally, the video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and feedback from the transmission channel.
From the input video signal, relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, further information is also extracted from the coded bitstream using a stream parser.
@@ -143,7 +143,7 @@ These parameters are fed into the optimisation block to select the best bitstrea
The optimisation process is described in detail in the following section.
The main purpose of this architecture is to allow dynamic stream switching during real time transmission.
To accomplish that without causing drift at the decoder, the switching procedure should take place at refresh points, i.e., \ac{idr} and \ac{cra} frames, which correspond to clean random access points. In the proposed scheme, \ac{idr} frames are introduced every 16 frames, resulting in between 1 to 3 switching points every second, depending on the frame rate of the video signal.
To accomplish this without causing drift at the decoder, the switching procedure should take place at refresh points, i.e., \ac{idr} and \ac{cra} frames, which correspond to clean random access points. In the proposed scheme, \ac{idr} frames are introduced every 16 frames, resulting in between 1 to 3 switching points every second, depending on the frame rate of the video signal.
@@ -155,7 +155,7 @@ To accomplish that without causing drift at the decoder, the switching procedure
In this section the optimisation method used to select a robust video stream is presented.
The selection method is based on a shallow neural network which predicts the optimal stream based on different parameters.
This section describes both the input parameters and the neural network used in the optimal stream selection process. Moreover, a description data set used to train and evaluate the neural network used is also provided
This section describes both the input parameters and the neural network used in the optimal stream selection process. Moreover, a description data set used to train and evaluate the neural network used is also provided.
\subsection{Data set}
@@ -163,16 +163,16 @@ This section describes both the input parameters and the neural network used in
\def\NumCases{910\xspace}
The proposed method includes an optimisation stage, which used a neural network classifier in order to select the optimal stream to be used for transmission.
This stream is able to obtain the highest quality for particular transmission scenario.
The proposed method includes an optimisation stage that uses a neural network classifier, in order to select the optimal stream to be used for transmission.
This stream is able to obtain the highest quality for a particular transmission scenario.
Therefore, several test cases were simulated in order to obtain results for a wide range of conditions. These tests are mainly characterised by three elements:
\begin{enumerate}
\begin{itemize}
\item Video signal;
\item Target bitrate;
\item Packet loss ratio.
\end{enumerate}
\end{itemize}
To obtain the data set 14 different test sequences with different scene content, listed in Table~\ref{Table6:TestMaterialCharacteristics}. Those sequences were encoded with the main bitrate (see Table~\ref{Table6:TestMaterialCharacteristics}), as well as four different others obtained by reducing the main bitrate by 10\%, 20\%, 30\% and 40\%.
To obtain the data set, 14 different test sequences with different scene content were used, as listed in Table~\ref{Table6:TestMaterialCharacteristics}. Those sequences were encoded with the main bitrate (see Table~\ref{Table6:TestMaterialCharacteristics}), as well as four different others, obtained by reducing the main bitrate by 10\%, 20\%, 30\% and 40\%.
Then, the following \acs{plr} were used to cover a wide range of loss ratios giving more detail to low \acs{plr}: 0\%, 0.5\%, 1.0\%, 1.2\%, 1.6\%, 2.0\%, 2.2\%, 2.4\%, 2.6\%, 2.8\%, 3.0\%, 5.0\% and 10.0\%.
This results in a total of \NumCases test conditions.
@@ -189,12 +189,12 @@ From the input video the two parameters are measured in a frame by frame basis,
%From the coded video streams complementary information to characterise the streaming content is also extracted. A stream parser is applied to the reference bitstream without error robustness techniques. Therefore, the prediction information extracted is related to a stream coded based on optimal \ac{rd} optimisation which was not affected by robust coding techniques. The following is extracted:
The characterisation of these three aspects of the video transmission result in a total of six parameters:
\begin{itemize}
The characterisation of these three aspects of the video transmission results in a total of six parameters:
\begin{enumerate}
\item \textit{\ac{si}}: parameter expressing the amount of spatial detail, which consequently affects the error recovery operations, leading to higher quality degradation~\cite{ITUTP9102008}.
\item \textit{\ac{ti}}: parameter expressing the amount of difference between temporally adjacent frames~\cite{ITUTP9102008}. This has a direct impact on the impact of errors, and thus the performance of the error resilience techniques as shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, which were discussed in Section~\ref{Section6:Problem}.
\item \textit{\ac{ti}}: parameter expressing the amount of difference between temporally adjacent frames~\cite{ITUTP9102008}. This has a direct relation to the impact of errors, and thus the performance of the error resilience techniques, as shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, which were discussed in Section~\ref{Section6:Problem}.
\item \textit{Inter (percentage)}: the percentage of the image that is compressed using inter-frame prediction techniques.
This is an indication of the distribution of spatial and temporal dependencies.
@@ -202,33 +202,44 @@ This is calculated as the ratio of inter-coded blocks over intra-coded ones.
Normally, coded streams with higher number of temporal dependencies are more prone to error propagation, thus higher number of bits should be allocated for error protection \cite{UEPScalable_TCSVT2010}.
\item \textit{AVG\_MV}: for the inter-coded regions, the motion information is analysed and the average \ac{mv} magnitude is calculated to provide information about the motion intensity.
As shown in previous studies, motion-based \ac{ec} algorithms are less efficient for video signals with higher motion intensity, as \acs{mv} are less predictable using algorithms such as \ac{mc} or \ac{mve} \cite{HybridConcealmentTB2008}.
As shown in previous studies, motion-based \ac{ec} algorithms are less efficient for video signals with higher motion intensity, as \acs{mv} are less predictable using algorithms, such as \ac{mc} or \ac{mve} \cite{HybridConcealmentTB2008}.
\item \textit{Bitrate}: information regarding the average bit allocation per frames is extracted, since it will affect the video quality and the amount of bits that can be available for auxiliary information. This value is expressed in Mbps, as illustrated in Table~\ref{Table6:TestMaterialCharacteristics}.
\item \textit{Bitrate}: information regarding the average bit allocation per frames is calculated, since it will affect the video quality and the amount of bits that can be available for auxiliary information. This value is expressed in Mbps, as illustrated in Table~\ref{Table6:TestMaterialCharacteristics}.
\item \textit{\ac{plr}:} feedback information from the network to characterise the transmission losses, expressed by the percentage of packets loss. This is used since for higher the loss rations there is a clear increase in the quality degradation which affect the performance of the robust coding techniques.
\item \textit{\ac{plr}:} feedback information from the network to characterise the transmission losses, expressed by the percentage of packets loss. This is used since for higher loss ratio there is a clear increase in the quality degradation, which affects the performance of the robust coding techniques.
Transmission networks with higher loss ratios require a higher bit allocation for robust techniques, therefore the error resilience technique should adapt based on the value of the \ac{plr}.
\end{itemize}
\end{enumerate}
\begin{table}[tb]
%To characterise the input video signal the following parameters are used: \ac{si} and \ac{ti}~\cite{ITUTP9102008}.
%These parameters are obtained as described in Chapter~\ref{Chapter:HEVCErrorResilience}, and aim at evaluating the spatial complexity and motion complexity of the video scenes, which have relevant impact on the error robustness of the coded streams.
%As shown, test material with higher \ac{si} or \ac{ti} suffer from higher quality degradation as the \ac{plr} increases, e.g., Book Arrival and Tennis sequences.
These parameters were compared against the optimal stream, selected from S1 to S4.
This aims at finding the correlation between the input parameters and the output variable, i.e., target stream.
In this study the streams are characterised by a number between 1 and 4.
Table~\ref{Table6:Correlation} shows the absolute Person Correlation coefficient obtained across all sequences, bitrates and \acs{plr}.
The table shows the results for the parameters previously described, as well as, a new parameter obtained from the average magnitude of the optical flow measured using the method in~\cite{DeepFlow2013} (\textit{OPT\_FLOW}).
Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable. As both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method, as it achieves a higher Person Correlation coefficient.
\begin{table}[b]
\centering
\caption{Pearson correlation coefficient between the different parameters and the optimal robust coding techniques (optimal stream).}
\caption{Pearson Correlation coefficient between the different parameters and the optimal robust coding techniques (optimal stream).}
\label{Table6:Correlation}
\TABLESVSPACE
\renewcommand{\arraystretch}{1.2}
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{p{4cm}x{4cm}}
\toprule
Parameter & Absolute pearson coefficient \\ \hline
% \textit{SI} & -0.1553 \\ \hline
% \textit{TI} & 0.6079 \\ \hline
% \textit{Inter (percentage)} & -0.6487 \\ \hline
% \textit{OPT\_FLOW} & 0.6953 \\ \hline
% \textit{AVG\_MV} & 0.4647 \\ \hline
% \textit{Bitrate} & 0.3580 \\ \hline
% \textit{PLR} & 0.5755 \\ \hline
Parameter & Absolute Pearson Correlation \\ \hline
% \textit{SI} & -0.1553 \\ \hline
% \textit{TI} & 0.6079 \\ \hline
% \textit{Inter (percentage)} & -0.6487 \\ \hline
% \textit{OPT\_FLOW} & 0.6953 \\ \hline
% \textit{AVG\_MV} & 0.4647 \\ \hline
% \textit{Bitrate} & 0.3580 \\ \hline
% \textit{PLR} & 0.5755 \\ \hline
\textit{SI} & 0.3953 \\ \hline
\textit{TI} & 0.6079 \\ \hline
@@ -241,20 +252,9 @@ Transmission networks with higher loss ratios require a higher bit allocation fo
\end{table}
%To characterise the input video signal the following parameters are used: \ac{si} and \ac{ti}~\cite{ITUTP9102008}.
%These parameters are obtained as described in Chapter~\ref{Chapter:HEVCErrorResilience}, and aim at evaluating the spatial complexity and motion complexity of the video scenes, which have relevant impact on the error robustness of the coded streams.
%As shown, test material with higher \ac{si} or \ac{ti} suffer from higher quality degradation as the \ac{plr} increases, e.g., Book Arrival and Tennis sequences.
These parameters were compared against the optimal stream which should be selected (S1...S4).
This aims at finding the correlation between the input parameters and the output variable, i.e., target stream.
In this study the streams are characterised by a number between 1 and 4, respectively.
Table~\ref{Table6:Correlation} shows the absolute person correlation coefficient obtained across all sequences, bitrates and \acs{plr}.
The table shows the results for the parameters previously described, as well as, a new parameter obtained from the average magnitude of the optical flow measured using the method in~\cite{DeepFlow2013} (\textit{OPT\_FLOW}).
Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable. As both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method as it achieves a higher person correlation coefficient.
% Podia se citar este paper
% \cite{IntraRefreshICIP2009}
%\HERE
\subsection{Neural network}
\glsreset{nn}
@@ -263,7 +263,7 @@ A \ac{nn} is a computational model based on the structure and functions of biolo
%In terms of computer science, \acs{nn} work similarly to a human nervous system for receiving, processing, and transmitting information between neurons.
Mainly, a \ac{nn} consists of three different layers: the input layer which receives data from an external environment, the hidden layer which process the inputs and an output layer~\cite{NeuralNetworks1994}.
In the past \acs{nn} have been widely used in video processing applications such as, quality evaluation~\cite{VideoQualityNN_TNN2006}, video analysis~\cite{VideoAnalysisNN_2018} and video coding~\cite{RateDistortionNN_VCIP2017,HEVCComplexityNN_TIP2018}.
In the proposed method a feed-forward network is used which connects every neuron in a given layer to the neurons of the subsequent layer.
In the proposed method a feed-forward network is used, which connects every neuron in a given layer to the neurons of the subsequent layer.
Figure~\ref{FIG6:NeuralNetwork} shows the diagram of the \ac{nn} used in the proposed method.
The network contains and input and output layer, as well as, a single hidden layer.
@@ -281,22 +281,23 @@ Finally, the output with the highest value is selected and the corresponding str
\FIGSVSPACE
\end{figure}
In the proposed approach a shallow \ac{nn} is used for simplicity and to avoid over-fitting due to the limited test cases.
By avoiding over-fitting, it is guaranteed that the proposed method would continue to work in future test conditions not used for the network training.
In the proposed approach a shallow \ac{nn} is used for simplicity and to avoid over-fitting, due to the limited test cases.
By avoiding over-fitting, it is guaranteed that the proposed method would continue to work in future test conditions, not used for the network training.
However, different network topologies with different number of neurons in each layer could also be used to predict between different error resilience techniques, as in the proposed method.
\subsubsection{Training and validation}
%As mentioned before a total of \NumCases test cases are used for the evaluation of the proposed method.
To effectively train and validate the \ac{nn} the data set is randomly in three different groups: (i) 55\% of the test cases are used for training, (ii) 15\% for validation and (iii) 30\% for testing the \ac{nn} performance.
To effectively train and validate the \ac{nn}, the data set is randomly partitioned into three different groups: (i) 55\% of the test cases are used for training, (ii) 15\% for validation and (iii) 30\% for testing the \ac{nn} performance.
The \ac{nn} supervised training uses 591 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993}.
The remaining 254 conditions are used to evaluate the accuracy of the proposed method after training using different conditions. From this test one video sequences was left out, to be used in the performance evaluation of the proposed method.
The remaining 254 conditions are used to evaluate the accuracy of the proposed method after training with different conditions. From this test one video sequence was left out, to be used in the performance evaluation of the proposed method.
Table~\ref{Table6:ConfusionTraining} shows the confusion matrix of the \ac{nn} training phase.
In this matrix each column corresponds to the target result, i.e., the cases where the coded streams S1 to S4 would lead to the highest decoded video quality, respectively.
In this matrix each column corresponds to the target result, i.e., the cases where the coded streams, S1 to S4, would lead to the highest decoded video quality.
Similarly, each row corresponds to the \ac{nn} selected by the trained networks.
Thus, those cases where the horizontal and vertical labels match, indicate that the network produced the correct output. The remaining positions correspond to the erroneous outputs.
Thus, those cases where the horizontal and vertical labels match, indicate that the network produced the correct output.
The remaining positions correspond to the erroneous outputs.
The last line shows the accuracy for each target output.
These results correspond to the dataset used to train the \ac{nn}.
@@ -319,8 +320,8 @@ These results correspond to the dataset used to train the \ac{nn}.
\end{tabular}
\end{table}
Results in Table~\ref{Table6:ConfusionTraining} shows that streams S1 (i.e., without robust coding techniques) and S4 (i.e., using optimal \ac{ec} with 20\% of bitrate in the \ac{el}) lead to the highest quality in the majority of the test cases (are selected more often).
The \ac{nn} is able to achieve a high accuracy for S1 and S4 streams, correctly selecting the output stream in more than 90\% of cases. Regarding the remaining robust streams (i.e., S2 and S3) an acceptable accuracy of 70.8\% and 74.4\% is achieved, respectively. This reveals that the \ac{nn} used in the proposed method is able to predict the coded stream which leads to the highest video quality across different conditions with acceptable accuracy.
Results in Table~\ref{Table6:ConfusionTraining} shows that streams S1 (i.e., without robust coding techniques) and S4 (i.e., using optimal \ac{ec} with 20\% of bitrate in the \ac{el}) lead to the highest quality in the majority of the test cases (are more often selected).
The \ac{nn} is able to achieve a high accuracy for S1 and S4 streams, correctly selecting the output stream in more than 90\% of cases. Regarding the remaining robust streams (i.e., S2 and S3), an acceptable accuracy of 70.8\% and 74.4\% is achieved, respectively. This reveals that the \ac{nn} used in the proposed method is able to predict the coded stream which leads to the highest video quality across different conditions with acceptable accuracy.
\begin{table}[tb]
\centering
@@ -342,7 +343,8 @@ The \ac{nn} is able to achieve a high accuracy for S1 and S4 streams, correctly
\end{tabular}
\end{table}
To further evaluate the performance of the \ac{nn}, the confusion matrix of the data set usef for testing is shown in the Table~\ref{Table6:ConfusionTest}. This table shows the same results as Table~\ref{Table6:ConfusionTraining} but for different test cases, which were not used in the training phase.
To further evaluate the performance of the \ac{nn}, the confusion matrix of the data set used for testing is shown in the Table~\ref{Table6:ConfusionTest}.
This table shows the same results as Table~\ref{Table6:ConfusionTraining}, but for different test cases, which were not used in the training phase.
% Therefore, the trained network is not aware of the test cases used for validation.
Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 62.2\% with a maximum value of 89.6\% when the S4 is the target stream. Although a lower accuracy is obtained using the validation test cases, the overall accuracy is still acceptable, since it allows to select the optimal robust coding technique (i.e., S1 to S4) for the majority of the conditions.
In summary, the results presented in this sub-section reveal that the input parameters used in the proposed \ac{nn} are correlated with the decoded video quality, as they can effectively be used to perform data categorisation used in the selection of the optimal error robustness coding technique for the \ac{hevc} standard.
@@ -354,7 +356,7 @@ In summary, the results presented in this sub-section reveal that the input para
In this section the performance of the proposed method is evaluated and discussed.
The overall goal of the proposed method is to achieve improved quality using a combination of different robust coding techniques adaptively selected using a feed-forward \ac{nn}.
The error resilience techniques used in the proposed method are used in the context of the \ac{hevc} standard and compared with the reference implementation (\REF).
The error resilience techniques comprising the proposed method are used in the context of the \ac{hevc} standard and compared with the reference implementation (\REF).
The quality obtained with the proposed optimal error robustness scheme (\PROP) is compared against the reference \ac{hevc} (\REF) and the methods used to generate the streams S1 to S4.
\begin{figure}[!t]
@@ -366,12 +368,12 @@ The quality obtained with the proposed optimal error robustness scheme (\PROP) i
\end{figure}
Figure~\ref{Figure6:ProposedQuality} shows the quality obtained for the proposed method and the \REF case for the main target bitrate shown in Table~\ref{TableTestSequences}. Moreover, the quality obtained with the target stream (\BEST) is also shown.
One should note that the \BEST case corresponds to the optimal selection which can only be performed after quality evaluation at the decoder, therefore not suitable for real time applications.
It is only used to establish the optimal case for comparison.
One should note that the \BEST case corresponds to the optimal selection, which can only be performed after quality evaluation at the decoder, therefore it is not suitable for real time applications.
It is only used to establish the optimal case for comparison purposes.
The results in the figure allows to evaluate the impact in the decoded quality of the selection used in the proposed method and compare it with the optimal case.
As shown in Figure~\ref{Figure6:ProposedQuality} the proposed method clearly outperforms the reference \ac{hevc} for all \acs{plr}.
Moreover, in the error-free case (\ac{plr}=0) it leads to imperceptible quality reduction since the proposed approach is able to select the optimal stream, thus disabling any robust coding technique when the transmission network does not have packet losses.
As shown in Figure~\ref{Figure6:ProposedQuality}, the proposed method clearly outperforms the reference \ac{hevc} for all \acs{plr}.
Moreover, in the error-free case (\ac{plr}=0), it leads to imperceptible quality reduction, since the proposed approach is able to select the optimal stream, thus disabling any robust coding technique when the transmission network does not have packet losses.
In case of data loss, the results show that the proposed method is able to match the quality obtained by the \BEST case in most cases, revealing that the proposed method is able to efficiently select the optimal stream for transmission.
\begin{table}[tb]
@@ -413,13 +415,16 @@ Subsequently, a comparison between the different methods is performed, in order
%for every test case.
Table~\ref{Table6:AverageGains} shows the average quality gains in comparison with the reference \ac{hevc} obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams S1 to S4. The average quality gains are measured across all test cases, i.e., different bitrates and different loss ratios.
Results in Table~\ref{Table6:AverageGains} shows that the robust coded streams S1 to S4 are able to achieve acceptable average quality gains up to 1.86 dB (see results for S2).
Results in Table~\ref{Table6:AverageGains} shows that the robust coded streams S1 to S4 are able to achieve acceptable average quality gains up to 1.51 dB (see results for S2).
However, since the implementation of robust coding techniques decreases the coding efficiency, it also leads to lower quality than the reference \ac{hevc}. This is noticeable in the results obtained for the Four People sequence in the streams S2 to S4 and for the Kristen and Sara sequence in the streams S3 and S4 (see red ellipses).
The overall performance of such methods reveals that they are not suitable for transmission conditions with low \acs{plr}, e.g., lower than 3\% (higher number of conditions have \acs{plr} lower than 3\%).
On the contrary, the proposed method is always able to outperform the reference \ac{hevc} and achieve quality gains of at least 0.25 dB for the Four People sequence. Moreover, for the Kimono and Race Horses sequences it is able to achieve the highest quality gains of up 3.41 dB and 4.39 dB, respectively (see blue ellipses). Summarising, by adaptively selecting the video stream, and consequently the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness achieving an average quality gain across different bitrates and \acs{plr} of 1.88 dB.
On the contrary, the proposed method is always able to outperform the reference \ac{hevc} and achieve quality gains of at least 0.25 dB for the Four People sequence.
Moreover, for the Kimono and Race Horses sequences, it is able to achieve the highest quality gains, of up 3.41 dB and 4.39 dB, respectively (see blue ellipses). Summarising, by adaptively selecting the video stream, and consequently the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness, achieving an average quality gain across different bitrates and \acs{plr} of 1.88 dB.
Finally, in order the performance evaluation of the proposed method is evaluated for a video sequence, which was not fed into the \ac{nn} training phase. This study aims at finding the accuracy of the optimal stream selection in case of new video content. For this purpose the Basketball Drive sequence was not used in the previous simulations. Table~\ref{Table6:AverageGainBasketBallDrive} shows the average quality results obtained with this sequence, similarly as Table~\ref{Table6:AverageGains}. These results confirm that the proposed method is able to select the optimal robust stream to be used for transmission, even when new video content is transmitted. The proposed method achieves a quality gain of 3.55 dB in comparison with the reference \ac{hevc}, outperforming the quality achieved with the stream S4 (i.e., 3.45 dB).
This confirms the advantage of the proposed method to adaptively select among different stream with different levels of error protections.
Finally, the performance evaluation of the proposed method is evaluated for a video sequence, which was not fed into the \ac{nn} training phase. This study aims at finding the accuracy of the optimal stream selection in case of a new video content. For this purpose the Basketball Drive sequence was randomly selected, and not used in previous simulations.
Table~\ref{Table6:AverageGainBasketBallDrive} shows the average quality results obtained with this sequence, similarly to Table~\ref{Table6:AverageGains}.
These results confirm that the proposed method is able to select the optimal robust stream to be used for transmission, even when new video content is transmitted. The proposed method achieves a quality gain of 3.55 dB in comparison with the reference \ac{hevc}, outperforming the quality achieved with the stream S4 (i.e., 3.45 dB).
This confirms the advantage of the proposed method to adaptively select among different streams with different levels of error protections.
\begin{table}[tb]
\centering
@@ -442,8 +447,9 @@ This confirms the advantage of the proposed method to adaptively select among di
\section{Summary}
\label{Section6:Summary}
In this chapter an adaptive error robustness method is proposed based on reference frame selection and \ac{ec}-aware optimisation. By adaptively selecting a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the reference techniques.
In this chapter an adaptive error robustness method is proposed, based on reference frame selection and \ac{ec}-aware optimisation. By adaptively selecting a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the reference techniques.
%Moreover, the proposed method is able to outperform those cases where a single error resilience technique is used.
The overall results demonstrate that the proposed method is able to increase the reconstruction video quality in case of errors, and by adapting the transmitted stream to the network conditions a better trade-off between coding efficiency and error robustness is achieved.
The consistent quality improvement achieved, allows the use of this method in dynamic broadcasting video services where network feedback is available and where packet loss probability is not negligible.
The overall results demonstrate that the proposed method is able to increase the reconstruction video quality in case of errors.
Also, by adapting the transmitted stream to the network conditions, a better trade-off between coding efficiency and error robustness can be achieved.
The consistent quality improvement, allows the use of this method in dynamic broadcasting video services where network feedback is available.
This architecture is suitable for dynamic streaming technologies, e.g., HTTP streaming based on MPEG-DASH~\cite{HEVCHTTPStreamAug2016} or content delivery over heterogeneous networks using the \ac{mmt}~\cite{ATSCMMT2016}.