367 lines
32 KiB
TeX
367 lines
32 KiB
TeX
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
% CHAPTER
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\chapter{Adaptive Error Robustness for Highly Efficient Video Coding}
|
|
\label{Chapter:EndToEndRobustSystem}
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Introduction}
|
|
|
|
The effectiveness of the error resilience techniques varies depending on the video content, as well as, the transmitted bitrate and network conditions.
|
|
Therefore, it is not straightforward to select a single technique which achieves high efficiency for every scenario.
|
|
Thus, in this chapter a novel adaptive error robustness method is proposed in order to optimise the techniques previously described in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}.
|
|
At the encoder-side different video streams are generated with different levels compression efficiency and error robustness. Then, a pre-training \ac{nn} is used to predict which video stream will achieve the highest decoded video quality for a particular scenario. The selection is based on different input parameters which include information from the video signal, the coded stream and the transmission network. Using this approach the overall performance of the proposed method significantly increases by achieving a better trade-off between compression efficient and error robustness.
|
|
|
|
This chapter is organised as follows.
|
|
Section~\ref{Section6:Problem} presents the problem formulation and preliminary results which serve as motivation for the method proposed in this chapter.
|
|
Section~\ref{Section6:ProposedMethod} provides an overview of the proposed method architecture and Section~\ref{Section6:OptimisationMethod} describes the method in detail.
|
|
Section~\ref{Section6:ExperimentalResults} shows the experimental evaluation of the proposed approach. Finally, Section~\ref{Section6:Summary} concludes this chapter.
|
|
|
|
\clearpage
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Problem formulation}
|
|
\label{Section6:Problem}
|
|
|
|
In this research work different robust coding techniques were proposed to increase the overall performance of video transmission under error-prone conditions.
|
|
Difference algorithms are used in the proposed methods, such as, reference frame selection, motion vector redundancies and error concealment signalling.
|
|
In previous chapters, these methods were evaluated against existing techniques revealing their superior performance and efficiency to deal with error robustness problems in high efficient video coding.
|
|
The evaluation of these methods revealed that they can efficiently increase the robustness against transmission errors and losses, leading to higher decoded video quality. Results in Chapter~\ref{Chapter:ErrorResilienceTechniques} show that the proposed reference frame selection algorithm introduces low overhead in comparison with existing techniques. However, the method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} achieves higher quality gains at cost of slightly higher bitrate increase.
|
|
Therefore, one may conclude that each method can be optimised for different applications, video characteristics and network conditions.
|
|
|
|
To advance one step forward, a joint robust coding method using different techniques can be used to increase the decoder video quality across difference conditions, by taking into consideration the input contents, the network information to optimise the robust coder to each scenario.
|
|
%In order to overcome the above problem, a joint evaluation study is necessary to identify the most efficient techniques across different conditions.
|
|
|
|
|
|
\subsection{Comparison of the proposed methods}
|
|
|
|
In order to develop a joint robust coder to cope with transmission errors a comparison study was carried out, to compare the error robustness performance of the proposed methods for different conditions.
|
|
%This study extends the previous evaluations presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, as they target a wider range of test material and network conditions.
|
|
The experimental evaluation carried out in this section follows the same condition described in Section~\ref{Section3:ErrorResilienceEvaluation}.
|
|
%In order to perform a wider evaluation different bitrate and \acs{plr} were used. Table~\ref{Table6:TestVariables} illustrates the selected configurations. As shown in the table a total of 15 test sequences were used encoder using 5 different target bitrates. One should note that the main bitrate varies based on the test sequence and it is shown in third column of Table~\ref{TableTestSequences}. Different network conditions were evaluated using a total of 13 different \acs{plr}, with higher special focus on lower loss ratios, which are more relevant to modern video streaming applications.
|
|
The following methods are evaluated, namely:
|
|
\begin{itemize}
|
|
\item the default \ac{hevc} decoder with frame-copy \ac{ec} (\REF), as reference method;
|
|
\item the proposed reference frame selection scheme (\EXP);
|
|
\item the \EXP method in combination with an \ac{ec} optimisation with 10\% of bitrate on the scalable enhancement layer (\EXPELTEN);
|
|
\item the same as above, with 20\% of bitrate on the scalable enhancement layer (\EXPELTWEN).
|
|
\end{itemize}
|
|
|
|
|
|
\begin{figure}[!t]
|
|
\centering
|
|
\input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex}
|
|
\vspace{-4mm}
|
|
\caption{Average decoded video quality using the main target bitrate.}
|
|
\label{Figure6:AverageQualityMain}
|
|
\end{figure}
|
|
|
|
\subsubsection{Experimental results and discussion}
|
|
|
|
Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} shows the average decoded video quality obtained from 50 trials, measured with the \ac{psnr} quality metric, for different loss conditions.
|
|
Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate listed in the third column of Table~\ref{TableTestSequences}. Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained using a lower amount of coded bits, corresponding to a 40\% reduction of the main target bitrate. Comparing both results aims at finding the impact of the video signal and network bitrate on the performance of the proposed methods.
|
|
|
|
\begin{figure}[!t]
|
|
\centering
|
|
\input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex}
|
|
\vspace{-4mm}
|
|
\caption{Average decoded video quality using a 40\% reduction of the main target bitrate}
|
|
\label{Figure6:AverageQualityReduction}
|
|
\end{figure}
|
|
|
|
Results of both figures show that the proposed methods (\EXP, \EXPELTEN and \EXPELTWEN) are all able to outperform the reference \ac{hevc} for high \acs{plr}. In case of the proposed reference frame selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions.
|
|
Comparing the results for both target bitrates also shows that the proposed methods are able to achieve higher quality gains for higher bitrates. Therefore, the for a lower bitrate (see results of Figure~\ref{Figure6:AverageQualityReduction}) the proposed methods only outperform the reference case for slightly higher \acs{plr}. This is more noticeable in the \EXPELTWEN case.
|
|
|
|
Summarising, the performance of the proposed methods is not similar across different video content and target bitrate, which impose a challenge on which method should be applied to achieve the highest decoded video quality in case of transmission errors. In order to achieve increased efficiency and higher video quality, the optimal trade-off between auxiliary information and video bitrate should be obtained by selecting the optimal robust coding algorithm.
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Proposed architecture}
|
|
\label{Section6:ProposedMethod}
|
|
|
|
This section provides an overview of the proposed method, which aims at obtaining the optimal error robustness method for different streaming scenarios, i.e., video content and network conditions.
|
|
The main advantage of using the proposed algorithm is that it leads to an optimal trade-off between error robustness tools and coding efficiency.
|
|
This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and can selected based on the current conditions.
|
|
This architecture is suitable for dynamic streaming technologies, e.g., HTTP streaming based on MPEG-DASH~\cite{HEVCHTTPStreamAug2016} or content delivery over heterogeneous networks using the \ac{mmt}~\cite{ATSCMMT2016}.
|
|
In the proposed method, firstly the input signals are coded using different levels of error protections. Subsequently, an optimisation algorithm is used to decide which bitstream should be transmitted at any given instant, based on different parameters. The selected stream is expected to obtain the highest quality at the decoder-side for the given conditions.
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
%\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram}
|
|
\input{043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz}
|
|
\caption{Architecture of the proposed scheme.}
|
|
\label{FIG6:BlockDiagram}
|
|
\FIGSVSPACE
|
|
\end{figure}
|
|
|
|
Figure~\ref{FIG6:BlockDiagram} illustrates the architecture of the proposed encoding scheme.
|
|
Firstly, a reference video coder is used to compress the bitstream using highly efficient coding techniques in order to obtain the best \ac{rd} performance for error-free environments.
|
|
This is used as the reference bitstream in the proposed approach.
|
|
Subsequently, a transcoding operation is applied to the reference bitstream aided by the input video signal to efficiently select the reference frames used for inter prediction. The reference frame selection is applied as proposed in Chapter~\ref{Chapter:ErrorResilienceTechniques}.
|
|
Moreover, loss simulation and a saliency-based \ac{ec} optimisation is applied on the robust bitstream. This is used to select the optimal \ac{ec} algorithm which should be applied at the decoder.
|
|
Finally, since the optimal \ac{ec} mode might not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el} using two different bitrates, i.e., 10\% and 20\% of the total bitrate, providing two different levels of error robustness.
|
|
The \ac{ec} optimisation and \ac{el} coding is performed as proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques}.
|
|
The proposed approach results in four different bitstreams with different characteristics: (i) reference stream without error robust coding techniques; (ii) robust video coding using a reference frame selection algorithm which leads to lower error propagation in case of data-loss; (iii) robust video coding using selective reference frames and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue; (iv) same as the previous bitstream but with 20\% bitrate allocated for the scalable \ac{el}. These bitstreams are referred to as "S1" to "S4", respectively, as shown in Figure~\ref{FIG6:BlockDiagram}.
|
|
|
|
Finally, the video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and from the transmission channel.
|
|
From the input video signal relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, further information is also extracted from the coded bitstream using a stream parser.
|
|
%the predictions used by the video encoder as well as the target bitrate are extracted using a stream parser.
|
|
Finally, a parameter regarding the transmission channel is also used to predict the data loss.
|
|
These parameters are fed into the error robustness optimisation block to select the suitable bitstream which should be transmitted to obtain the highest video quality. The optimisation process is described in detail in the following section.
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Error robustness selection method}
|
|
\label{Section6:OptimisationMethod}
|
|
|
|
In this section the optimisation method used to select a robust highly efficiently coded stream is presented.
|
|
The proposed scheme aims at selecting the best method which should be used for transmission, that achieves the highest quality in error-prone scenarios.
|
|
The selection method is based on a shallow neural network which predicts the optimal stream based on different parameters.
|
|
This section describes of both the input parameters and the neural network used in the proposed scheme.
|
|
|
|
\subsection{Input parameters}
|
|
|
|
\glsreset{si}
|
|
\glsreset{ti}
|
|
|
|
A set of different parameters are used to predict the optimal approach that achieves the highest quality in specific network conditions. Based on these inputs the proposed scheme is able to select the appropriate robust coding scheme which should be used. These parameters aim at covering three different aspects of the video transmission: two parameters from the input video, three parameters from the video stream and one parameter from the transmission network.
|
|
|
|
To characterise the input video signal the following parameters are used: \ac{si} and \ac{ti}~\cite{ITUTP9112008}.
|
|
These parameters are obtained as described in Chapter~\ref{Chapter:HEVCErrorResilience}, and aim at evaluating the spatial complexity and motion complexity of the video scenes, which have relevant impact on the error robustness of the coded streams.
|
|
As shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, test material with higher \ac{si} or \ac{ti} suffer from higher quality degradation as the \ac{plr} increases, e.g., Book Arrival and Tennis sequences.
|
|
|
|
From the coded video streams complementary information to characterise the streaming content is also extracted. A stream parser is applied to the reference bitstream without error robustness techniques. Therefore, the prediction information extracted is related to a stream coded based on optimal \ac{rd} optimisation which was not affected by robust coding techniques. The following is extracted:
|
|
\begin{itemize}
|
|
\item \textit{Inter(\%)}: the relation between the ratio of the image which is coded based on intra and inter frame predictions, indication the distribution of spatial and temporal dependencies.
|
|
This is expressed by the ratio of blocks coded using inter frame predictions.
|
|
Normally, coded streams with higher number of temporal dependencies are more prone to error propagation, thus higher number of bits should be allocated for error protection \cite{UEPScalable_TCSVT2010}.
|
|
|
|
\item \textit{AVG\_MV}: from those regions coded using inter frame prediction, the motion information is analysed and the average \ac{mv} size is used to provide information regarding the motion intensity. As shown in previous studies, motion based \ac{ec} algorithms are less efficient for video signals with higher motion intensity as \acs{mv} are less predictable using algorithms such as \ac{mc} or \ac{mve} \cite{HybridConcealmentTB2008}.
|
|
|
|
\item \textit{Bitrate}: information regarding the bit allocation is extracted, since it will affect the video quality and the amount of bits that can be available for auxiliary information.
|
|
\end{itemize}
|
|
|
|
Finally, since the amount of data loss affects the decoded video quality the transmission network \ac{plr} is also used as input parameter. Transmission networks with higher loss ratios require a higher bit allocation for robust techniques, therefore the error resilience technique should adapt based on the value of the \ac{plr}.
|
|
|
|
% Podia se citar este paper
|
|
% \cite{IntraRefreshICIP2009}
|
|
|
|
\subsection{Neural network}
|
|
|
|
\glsreset{nn}
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
%\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram}
|
|
\input{043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz}
|
|
\caption{Representation of the neural network used in the proposed method.}
|
|
\label{FIG6:NeuralNetwork}
|
|
\FIGSVSPACE
|
|
\end{figure}
|
|
|
|
A \ac{nn} is a computational model based on the structure and functions of biological neural networks.
|
|
%In terms of computer science, \acs{nn} work similarly to a human nervous system for receiving, processing, and transmitting information between neurons.
|
|
Mainly, a \ac{nn} consists of three different layers: the input layer which receives data from an external environment, the hidden layer which process the inputs and an output layer~\cite{NeuralNetworks1994}.
|
|
In the past \acs{nn} have been widely used in video processing applications such as, quality evaluation~\cite{VideoQualityNN_TNN2006}, video analysis~\cite{VideoAnalysisNN_2018} and video coding~\cite{RateDistortionNN_VCIP2017,HEVCComplexityNN_TIP2018}.
|
|
In the proposed method a feed-forward network is used which connects every neuron in a given layer to the neurons of the subsequent layer.
|
|
|
|
Figure~\ref{FIG6:NeuralNetwork} shows the diagram of the \ac{nn} used in the proposed method.
|
|
The network contains and input and output layer, as well as, a single hidden layer.
|
|
The input layer receives the six inputs, as described in the previous sub-section. These inputs are correlated with the optimisation variable being maximised, i.e., the output video quality.
|
|
Subsequently, the network transforms the input data through a series of neurons in the hidden layer.
|
|
The output layer comprises four neurons corresponding to each available video stream from S1 to S4 (different error resilience techniques).
|
|
Finally, the output with the highest value is selected and the corresponding stream is used for transmission, since it is the one expected to achieve the highest video quality for a particular scenario.
|
|
|
|
In the proposed approach a shallow \ac{nn} is used for simplicity and to avoid over-fitting due to the limited test cases.
|
|
By avoiding over-fitting, it is guaranteed that the proposed method would continue to work in future test conditions not used for the network training.
|
|
However, different network topologies with different number of neurons in each layer could also be used to predict between different error resilience techniques, as in the proposed method.
|
|
|
|
|
|
\section{Experimental evaluation}
|
|
\label{Section6:ExperimentalResults}
|
|
|
|
\def\NumCases{845\xspace}
|
|
|
|
In this section the experimental results obtained with the proposed method are presented and discussed.
|
|
The error resilience techniques used in the proposed method are used in the context of the \ac{hevc} standard and compared with the reference implementation (\REF).
|
|
%The experimental results were obtained with the methodology described in Section~\ref{ExperimentalTestSetup}.
|
|
To demonstrate the effectiveness of the proposed optimal error robustness scheme several test cases were used.
|
|
In this experimental evaluation 13 test sequences were used, encoded using at 5 different bitrates.
|
|
Moreover, the following \acs{plr} were used to cover a wide range of loss rations giving more detail to low \acs{plr}: 0\%, 0.5\%, 1.0\%, 1.2\%, 1.6\%, 2.0\%, 2.2\%, 2.4\%, 2.6\%, 2.8\%, 3.0\%, 5.0\% and 10.0\%.
|
|
This results in a total of \NumCases test conditions.
|
|
|
|
The performance evaluation of the proposed method is carried out using two different approaches.
|
|
In the former, a description of the \ac{nn} training is presented along with the validation results.
|
|
In the latter, the quality obtained with the proposed optimal error robustness scheme is evaluated and compared with the reference methods.
|
|
|
|
%\begin{table}[t]
|
|
% \centering
|
|
% \caption{List of the test conditions tested.}
|
|
% \label{Table6:TestVariables}
|
|
% \TABLESVSPACE
|
|
% \begin{tabular}{m{4cm}|y{4cm}|y{6cm}}
|
|
% \toprule
|
|
% \textbf{Sequence} & \textbf{Bitrate} & \textbf{\ac{plr}} \\ \midrule
|
|
% Basketball Drill & Main (see Table~\ref{TableTestSequences}) & Error-free \\
|
|
% Basketball Drive & 10\% reduction & 0.5\% \\
|
|
% Book Arrival & 20\% reduction & 1.0\% \\
|
|
% BQSquare & 30\% reduction & 1.2\% \\
|
|
% Cactus & 40\% reduction & 1.6\% \\
|
|
% Four People & 50\% reduction & 2.0\% \\
|
|
% Kendo & & 2.2\% \\
|
|
% Kimono & & 2.4\% \\
|
|
% Kristen and Sara & & 2.6\% \\
|
|
% Park Scene & & 2.8\% \\
|
|
% People on Street & & 3.0\% \\
|
|
% Race Horses & & 5.0\% \\
|
|
% Tennis & & 10.0\% \\
|
|
% Traffic & & \\
|
|
% \bottomrule
|
|
% \end{tabular}
|
|
%\end{table}
|
|
|
|
|
|
\subsection{Network training and validation}
|
|
|
|
%As mentioned before a total of \NumCases test cases are used for the evaluation of the proposed method.
|
|
To effectively train and validate the \ac{nn} used in the proposed method the data set is randomly divided based on the following: 70\% of the test cases are used for training and 30\% for validation.
|
|
The \ac{nn} supervised training uses 591 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993}.
|
|
The remaining 273 conditions are used to evaluate the accuracy of the proposed method after training using different conditions.
|
|
|
|
|
|
\begin{table}[tb]
|
|
\centering
|
|
\caption{Confusion matrix of the training test cases.}
|
|
\label{Table6:ConfusionTraining}
|
|
\TABLESVSPACE
|
|
\renewcommand{\arraystretch}{1.2}
|
|
\begin{tabular}{x{.5cm}x{1cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}}
|
|
& & \multicolumn{4}{c}{Target} \\
|
|
\multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}}
|
|
& & S1 & S2 & S3 & S4 \\ \hline
|
|
& S1 & 122 & 17 & 3 & 0 \\
|
|
\cline{2-6} & S2 & 13 & 80 & 19 & 2 \\
|
|
\cline{2-6} & S3 & 0 & 12 & 96 & 9 \\
|
|
\cline{2-6} & S4 & 0 & 4 & 11 & 202 \\
|
|
\hline
|
|
\multicolumn{2}{x{1.7cm}|}{Accuracy} & 90.3\% & 70.8\% & 74.4\% & 94.8\% \\
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
Table~\ref{Table6:ConfusionTraining} shows the confusion matrix of the \ac{nn} training phase.
|
|
In this matrix each column corresponds to the target result, i.e., the cases where the coded streams S1 to S4 would lead to the highest decoded video quality, respectively.
|
|
Similarly, each row corresponds to the \ac{nn} selected by the trained networks.
|
|
Thus, those cases where the horizontal and vertical labels match, indicate that the network produced the correct output. The remaining positions correspond to the erroneous outputs.
|
|
The last line shows the accuracy for each target output.
|
|
These results correspond to the dataset used to train the \ac{nn}.
|
|
|
|
Results in Table~\ref{Table6:ConfusionTraining} shows that streams S1 (i.e., without robust coding techniques) and S4 (i.e., using optimal \ac{ec} with 20\% of bitrate in the \ac{el}) lead to the highest quality in the majority of the test cases (are selected more often).
|
|
The \ac{nn} is able to achieve a high accuracy for S1 and S4 streams, correctly selecting the output stream in more than 90\% of cases. Regarding the remaining robust streams (i.e., S2 and S3) an acceptable accuracy of 70.8\% and 74.4\% is achieved, respectively. This reveals that the \ac{nn} used in the proposed method is able to correctly predict the coded stream which leads to the highest video quality across different conditions.
|
|
|
|
\begin{table}[tb]
|
|
\centering
|
|
\caption{Confusion matrix of the validation test cases.}
|
|
\label{Table6:ConfusionTest}
|
|
\TABLESVSPACE
|
|
\renewcommand{\arraystretch}{1.2}
|
|
\begin{tabular}{x{.5cm}x{1cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}}
|
|
& & \multicolumn{4}{c}{Target} \\
|
|
\multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}}
|
|
& & S1 & S2 & S3 & S4 \\ \hline
|
|
& S1 & 56 & 7 & 2 & 0 \\
|
|
\cline{2-6} & S2 & 10 & 28 & 6 & 2 \\
|
|
\cline{2-6} & S3 & 1 & 5 & 31 & 8 \\
|
|
\cline{2-6} & S4 & 0 & 5 & 7 & 86 \\
|
|
\hline
|
|
\multicolumn{2}{x{1.7cm}|}{Accuracy} & 83.6\% & 62.2\% & 67.4\% & 89.6\% \\
|
|
%\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
To further evaluate the performance of the \ac{nn}, the confusion matrix of the validation set is shown in the Table~\ref{Table6:ConfusionTest}. This table shows the same results as Table~\ref{Table6:ConfusionTraining} but for different test cases, which were not used in the training phase.
|
|
% Therefore, the trained network is not aware of the test cases used for validation.
|
|
Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 62.2\% with a maximum value of 89.6\% when the S4 is the target stream. Although a lower accuracy is obtained using the validation test cases, the overall accuracy is still acceptable, since it allows to select the optimal robust coding technique (i.e., S1 to S4) for the majority of the conditions.
|
|
In summary, the results presented in this sub-section reveal that the input parameters used in the proposed \ac{nn} are correlated with the decoded video quality, as they can effectively be used to perform data categorisation used in the selection of the optimal error robustness coding technique for the \ac{hevc} standard.
|
|
|
|
|
|
|
|
|
|
|
|
\subsection{Quality evaluation}
|
|
|
|
The overall goal of the proposed method is to achieve improved quality using a combination of different robust coding techniques adaptively selected using a feed-forward \ac{nn}. In this sub-section the quality obtained with the proposed method (\PROP) is compared against the reference \ac{hevc} (\REF) and the methods used to generate the streams S1 to S4.
|
|
|
|
\begin{figure}[!t]
|
|
\centering
|
|
\input{043_EndToEndRobustSystem/Results/PlotQualty_vs_PLR_Predicted.tex}
|
|
\vspace{-2mm}
|
|
\caption{Average video quality using different error resilience techniques.}
|
|
\label{Figure6:ProposedQuality}
|
|
\end{figure}
|
|
|
|
Figure~\ref{Figure6:ProposedQuality} shows the quality obtained for the proposed method and the \REF case for the main target bitrate shown in Table~\ref{TableTestSequences}. Moreover, the quality obtained with the target stream (\BEST) is also shown. One should note that the \BEST case corresponds to the optimal selection which can only be performed after quality evaluation at the decoder, therefore not suitable for real time applications and it is only used to illustrate the optimal scenario.
|
|
The results in the figure allows to evaluate the impact in the decoded quality of the selection used in the proposed method and compare it with the optimal case.
|
|
|
|
As shown in the Figure~\ref{Figure6:ProposedQuality} the proposed method clearly outperforms the reference \ac{hevc} for all \acs{plr}. Moreover, in the error-free case (\ac{plr}=0) it does not lead to quality reduction since the proposed approach is able to select the optimal stream, thus disabling any robust coding technique when the transmission network does not have packet losses. In case of data loss results show that the proposed method is able to match the quality obtained by the \BEST case in most of the cases, revealing that the proposed method is able to efficiently select the optimal stream for transmission.
|
|
|
|
\begin{table}[tbp]
|
|
\centering
|
|
\caption{Average quality gains ($\Delta$PSNR) in comparison with the reference \ac{hevc} across different bitrates and loss ratios.}
|
|
\label{Table6:AverageGains}
|
|
\TABLESVSPACE
|
|
\newcommand{\colwidth}{2cm}
|
|
\begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}}
|
|
\toprule
|
|
\multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\
|
|
& S1 & S2 & S3 & S4 & \PROP \\ \hline
|
|
Basketball Drill & +1.61 & +2.66 & +2.48 & +2.44 & +2.75 \\ \hline
|
|
Basketball Drive & +2.17 & +3.46 & +3.22 & +3.63 & +3.69 \\ \hline
|
|
Book Arrival & +0.96 & +1.66 & +1.43 & +1.59 & +1.74 \\ \hline
|
|
BQSquare & +0.92 & +0.96 & +0.80 & +0.62 & +1.09 \\ \hline
|
|
Cactus & +1.28 & +2.04 & +1.89 & +1.96 & +2.09 \\ \hline
|
|
Four People & +0.28 & -0.23 & -0.88 & -0.99 & +0.25 \\ \hline
|
|
Kendo & +0.81 & +0.78 & +0.20 & +0.61 & +1.02 \\ \hline
|
|
Kimono & +1.65 & +3.04 & +3.26 & +3.27 & +3.41 \\ \hline
|
|
Kristen Sara & +0.36 & +0.15 & -0.33 & -0.50 & +0.38 \\ \hline
|
|
Park Scene & +1.64 & +2.18 & +1.88 & +2.02 & +2.25 \\ \hline
|
|
People on Street & +1.00 & +1.27 & +0.91 & +1.49 & +1.63 \\ \hline
|
|
Race Horses & +1.93 & +3.89 & +4.06 & +4.30 & +4.39 \\ \hline
|
|
Tennis & +1.15 & +2.24 & +2.17 & +3.12 & +3.18 \\ \hline
|
|
\textbf{Average} & \textbf{+1.21} & \textbf{+1.86} & \textbf{+1.62} & \textbf{+1.81} & \textbf{+2.15} \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\begin{tikzpicture}[overlay]
|
|
\draw[red, line width=1.2pt] (1.8,4.65) ellipse (3.5cm and 0.3cm);
|
|
\draw[red, line width=1.2pt] (3,3.05) ellipse (2.3cm and 0.3cm);
|
|
\draw[blue, line width=1.2pt] (6.7,1.45) ellipse (.85cm and 0.25cm);
|
|
\draw[blue, line width=1.2pt] (6.7,6.75) ellipse (.85cm and 0.25cm);
|
|
\end{tikzpicture}
|
|
\end{table}
|
|
|
|
Subsequently, a comparison between the different methods is performed, in order to evaluate the quality gains obtained with the proposed method in comparison with the different techniques.
|
|
%for every test case.
|
|
Table~\ref{Table6:AverageGains} shows the average quality gains in comparison with the reference \ac{hevc} obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams S1 to S4. The average quality gains are measured across all test cases, i.e., different bitrates and different loss ratios.
|
|
|
|
Results in Table~\ref{Table6:AverageGains} shows that the robust coded streams S1 to S4 are able to achieve acceptable average quality gains up to 1.86 dB (see results for S2).
|
|
However, since the introduction of robust coding techniques decreases the coding efficiency it may lead to lower quality than the reference \ac{hevc}. This is noticeable in the results obtained for the Four People sequence in the streams S2 to S4 and for the Kristen and Sara sequence and in the streams S3 and S4 (see red ellipses). This reveals that the overall performance of such methods is not suitable for transmission networks subject to low \acs{plr}, e.g., lower than 3\% (higher number of conditions have \acs{plr} lower than 3\%).
|
|
On contrary, the proposed method is always able to outperform the reference \ac{hevc} and achieve quality gains of at least 0.25 dB for the Four People sequence. Moreover, for the Basketball Drive and Race Horses sequences it is able to achieve the highest quality gains of up 3.69 dB and 4.39 dB, respectively (see blue ellipses). Summarising, by adaptively selecting the video stream, and consequently the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness achieving an average quality gains across different bitrates and \acs{plr} of 2.15 dB.
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Summary}
|
|
\label{Section6:Summary}
|
|
|
|
In this chapter an adaptive error robustness method is proposed based on reference frame selection and \ac{ec}-aware optimisation. By adaptively select a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the reference techniques.
|
|
%Moreover, the proposed method is able to outperform those cases where a single error resilience technique is used.
|
|
The overall results demonstrate that the proposed method is able to increase the reconstruction video quality in case of errors, and by adapting the transmitted stream to the network conditions a better trade-off between coding efficiency and error robusntess is achieved.
|
|
The consistent quality improvement achieved, allows the use of this method in dynamic broadcasting video services where network feedback is available and where packet loss probability is not negligible.
|
|
|