%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % CHAPTER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \chapter{Adaptive error robustness for highly efficient video coding \VERSION{3}} \label{Chapter:EndToEndRobustSystem} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \vspace*{\fill} \section{Introduction} \glsunset{hevc} \glsreset{ec} \glsreset{rd} \glsreset{el} \glsreset{nn} The effectiveness of different error resilience techniques is rather variable depending on the video content, the transmitted bitrate, network conditions, among others. Therefore, it is not straightforward to select a single technique capable of guaranteeing the highest efficiency for every possible scenario and also for the whole range of networking conditions. This chapter addresses this problem by proposing an adaptive error robustness scheme that optimises the efficiency of the techniques previously described in Chapters~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}. At the encoder-side, a pre-trained neural network predicts the video stream that is able to achieve the highest decoded quality for particular loss cases, among various video streams encoded with different levels of compression efficiency and error robustness. The stream selection is based on different input parameters that include information from the video signal, the coded stream and the transmission network. Using this approach the overall performance of the proposed adaptive method significantly increases, achieving a better trade-off between compression efficiency and error robustness. This chapter is organised as follows. Section~\ref{Section6:Problem} presents the problem addressed in this chapter and experimental evidence, as motivation for the method investigated. Section~\ref{Section6:ProposedMethod} provides an overview of architecture that supports the implementation of the proposed method and Section~\ref{Section6:OptimisationMethod} describes the technical details. Section~\ref{Section6:ExperimentalResults} presents the performance evaluation and a discussion of results. Finally, Section~\ref{Section6:Summary} concludes the chapter. \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The problem of adaptive error robustness} \label{Section6:Problem} In the scope of this thesis different robust coding techniques were investigated and described in previous chapters to increase the overall performance of video transmission under error-prone conditions. Different algorithms were devised in the proposed methods, such as, reference frame selection, motion vector redundancies and error concealment signalling. In the research work described in Chapters~\ref{Chapter:ErrorResilienceReview}~to~\ref{Chapter:ConcealmentAwareTechniques}, these methods were evaluated against existing techniques, revealing superior performance and better efficiency to deal with error robustness problems in high efficient video coding. Overall, the performance results of these methods have demonstrated that they can efficiently increase the robustness of video streams against transmission errors and losses, leading to higher decoded video quality. However, each method on its own does not adapt to different conditions. For instance, while the reference frame selection algorithm presented in Chapter~\ref{Chapter:ErrorResilienceTechniques} introduces low overhead in comparison with existing techniques, the method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} achieves higher quality gains at cost of slightly higher bitrate increase. Therefore, one may conclude that each method may be further optimised for different applications, video characteristics and network conditions. This is the motivation for the research presented in this chapter, which advances one step forward by investigating a joint robust coding method using different techniques to optimise the video coding process across different conditions, such as, the input content and the network information. This scheme can be used in transmission systems where different video bitstreams can be made available at the encoder-side to be selected based on the dynamic conditions. \subsection{Comparison of the proposed methods} In order to address the above problem of adaptive robust coding, a simulation study was carried out to compare the error robustness performance of the previously proposed methods for different conditions. This study extends the previous evaluations presented in Chapters~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, as they target a wider range of test material and network conditions. Moreover, it aims at comparing the performance of different techniques across a variety of video content, bitrates and loss conditions, in order to find relevant insights about the most affect the decoded video quality. Consequently, these relevant insights may be used to develop an adaptive robust coding method. %In order to perform a wider evaluation different bitrate and \acs{plr} were used. Table~\ref{Table6:TestVariables} illustrates the selected configurations. As shown in the table a total of 15 test sequences were used encoder using 5 different target bitrates. One should note that the main bitrate varies based on the test sequence and it is shown in third column of Table~\ref{TableTestSequences}. Different network conditions were evaluated using a total of 13 different \acs{plr}, with higher special focus on lower loss ratios, which are more relevant to modern video streaming applications. The following methods were evaluated in the simulation study, namely: \begin{itemize} \item the default \ac{hevc} decoder with frame-copy \ac{ec} (\REF), as reference method; \item the default \ac{hevc} encoder without robust coding-techniques and motion vector extrapolation \ac{ec} (\MEC); \item the proposed reference frame selection scheme (\MEXP); \item the method \MEXP in combination with an \ac{ec} optimisation with 10\% of bitrate allocated for the scalable enhancement layer (\MEXPELTEN); \item the same as above, with 20\% of bitrate allocated for the scalable enhancement layer (\MEXPELTWEN). \end{itemize} These methods were selected because they are the main techniques proposed within this research work and cover two elements of a robust video communication chain: (i) robust video encoding by dynamically select the reference frames and (ii) robust video decoding by improving the \ac{ec} performance. \input{043_EndToEndRobustSystem/Results/AdaptiveSequenceList.tex} Table~\ref{Table6:TestMaterialCharacteristics} presents the list of test sequences used. These sequences were selected to cover a wide range of video content, with different levels of spatial and temporal complexity, as demonstrated by the values of \ac{si} and \ac{ti}~\cite{ITUTP9102008}, respectively. Moreover, from the bitstream encoded with the reference implementation of the \ac{hevc} encoder, the percentage of the inter-coded blocks and the average \ac{mv} magnitude was extracted and presented in the table (columns 4 and 5, respectively). These parameters were considered due to their possible relation with the performance obtained by the proposed methods. The remaining configuration was kept as described in Section~\ref{Section3:ErrorResilienceEvaluation}. %The experimental methodology carried out in this section follows the same conditions described in Section~\ref{Section3:ErrorResilienceEvaluation}. \subsubsection{Preliminary simulation results and discussion} Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} show the average decoded video quality (\ac{psnr}) obtained for different \acs{plr}. Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate listed in Table~\ref{Table6:TestMaterialCharacteristics}, referred to as the main bitrate. Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained by reducing the main bitrate by 40\%. This is used to evaluate the impact of the network bandwidth on the decoded video quality. The results shown in the figures correspond to the same test sequences, in order to allow an accurate and fair comparison. \begin{figure}[!t] \centering \input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex} \vspace{-4mm} \caption{Average decoded video quality for the bitstreams encoded with the main bitrate (bps).} \label{Figure6:AverageQualityMain} \end{figure} \begin{figure}[!t] \centering \input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex} \vspace{-4mm} \caption{Average decoded video quality using a 40\% reduction of the main bitrate (bps).} \label{Figure6:AverageQualityReduction} \end{figure} Firstly, comparing the results for different bitrates, it is noticeable that lower quality is obtained for the lower bitrates, which obviously results from the higher compression ratios used. It is also noticed that lower quality gains, when comparing with the reference \ac{hevc} encoder (\REF), are obtained for the video streams encoded with a 40\% reduction of bitrate. This indicates that for lower bitrates the performance of the proposed methods decreases, especially for the \MEXPELTWEN case, where 20\% of the bitrate is allocated for the enhancement layer. Therefore, for lower bitrates, using methods that have a lower impact on the coding efficiency (e.g., \MEXP) or even disable the robust coding techniques (\MEC) may lead to higher performance. Secondly, comparing the results across different video content, i.e., different test sequences, one can notice that the performance of the different methods is not consistent. For instance, in the results of Figure~\ref{Figure6:AverageQualityMain}, for Tennis sequence the method \MEXPELTWEN clearly outperforms the remaining ones. However for the BQSquare sequence, higher performance is achieved by using less bitrate (i.e., 10\%) allocated for the \ac{el} (\MEXPELTEN). This is a consequence of the higher quality degradation due to higher motion in Tennis sequence (see results for TI and \ac{mv} size in Table~\ref{Table6:TestMaterialCharacteristics}) which requires higher redundant information to increase the overall quality. %\HIGHLIGHT{Mention SI!!} Finally, by comparing the results for all test sequences and both bitrates it can be clearly noticed that the \ac{plr} has a significant impact on the performance of the error robustness techniques. In case of the proposed reference frame selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions. For method \MEXPELTWEN, higher \acs{plr} are required to be able to outperform both the reference \ac{hevc} and the \ac{mve} method. Summarising, the performance of the proposed methods does not exhibit a consistent behaviour across different video content and bitrates, opening a new research goal to adaptively choose the best method that should be applied to achieve the highest decoded video quality. In order to achieve increased efficiency and higher video quality, the best trade-off between auxiliary information and video bitrate must be found by selecting the optimal robust coding algorithm. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Architecture for adaptive error robustness} \label{Section6:ProposedMethod} This section describes the method proposed to attain the research goal defined above, which aims at obtaining optimal error robustness for different streaming scenarios, characterised by different video content and network conditions. Figure~\ref{Figure6:BlockDiagram} illustrates the proposed architecture, whose main objective is to achieve an optimal trade-off between error robustness tools and coding efficiency. \begin{figure}[tb] \centering %\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram} \input{043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz} \caption{Adaptive error robustness architecture for 4 streams (\SMEC...\SMEXPELTWEN).} \label{Figure6:BlockDiagram} \FIGSVSPACE \end{figure} Firstly, the input signals are coded using different levels of error protection. Then, an optimisation algorithm is used to decide which bitstream should be transmitted at any given instant, based on different parameters. The selected stream is expected to obtain the highest quality at the decoder-side for the given conditions. A reference video coder is used to compress the video signal using highly efficient coding techniques, obtaining the best \ac{rd} performance for error-free environments. This is used as the reference bitstream. Subsequently, a transcoding operation is applied to this reference bitstream aided by the input video signal to efficiently select the reference frames used for inter prediction. The reference frame selection mechanism proposed in Chapter~\ref{Chapter:ErrorResilienceTechniques} is used for this purpose. Moreover, loss simulation and a saliency-based \ac{ec} optimisation are also used in the robust coding process, which enables the selection of the optimal \ac{ec} algorithm to be applied at the decoder. Finally, since the optimal \ac{ec} mode might not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el} using two different bitrates. In this case 10\% and 20\% of the total bitrate were chosen, providing two different levels of error robustness. The \ac{ec} optimisation and \ac{el} coding is performed as described in Chapter~\ref{Chapter:ConcealmentAwareTechniques}. The proposed approach results in four different bitstreams with different characteristics (referred to as \SMEC to \SMEXPELTWEN, as shown in Figure~\ref{Figure6:BlockDiagram}): (i) \SMEC: reference stream without error robust coding; (ii) \SMEXP: robust video coding using reference frame selection; (iii) \SMEXPELTEN: robust video coding using selective reference frames and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue; (iv) \SMEXPELTWEN: same as the previous bitstream but with 20\% bitrate allocated for the scalable~\ac{el}. The video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and feedback from the transmission channel. From the input video signal, relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, further information is also extracted from the coded bitstream using a stream parser. Finally, the packet loss ratio of the transmission channel is also used. These parameters are fed into the optimisation block to select the best bitstream for transmission. The optimisation process is described in detail in the following section. % TODO: dizer que é assumido PLR com variações lentas para o atraso do feedback da rede nao ter influencia. The main purpose of this architecture is to allow dynamic stream switching during real time transmission. To accomplish this without causing drift at the decoder, the switching procedure should take place at refresh points, i.e., \ac{idr} and \ac{cra} frames, which correspond to clean random access points. In the proposed scheme, \ac{idr} frames are introduced every 16 frames, resulting in between 1 to 3 switching points every second, depending on the frame rate of the video signal. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Robust stream selection method} \label{Section6:OptimisationMethod} In this section the optimisation method used to select a robust video stream is presented. The selection method is based on a shallow neural network which predicts the optimal stream based on different parameters. This section describes both the input parameters and the neural network used in the optimal stream selection process. Moreover, a description data set used to train and evaluate the neural network is also provided. \subsection{Data set} %\def\NumCases{845\xspace} \def\NumCases{1092\xspace} The proposed method includes an optimisation stage that uses a neural network classifier, in order to select the optimal stream to be used for transmission. This stream is able to obtain the highest quality for a particular transmission scenario. Therefore, several test cases were simulated in order to obtain results for a wide range of conditions. These tests are mainly characterised by three elements: \begin{itemize} \item Video signal; \item Target bitrate; \item Packet loss ratio. \end{itemize} To obtain the data set, 14 different test sequences with different scene content were used, as listed in Table~\ref{Table6:TestMaterialCharacteristics}. Those sequences were encoded with the main bitrate (see Table~\ref{Table6:TestMaterialCharacteristics}), four different others, obtained by reducing the main bitrate by 10\%, 20\%, 30\% and 40\%, and one with 10\% of bitrate increase. Then, the following \acs{plr} were used to cover a wide range of loss ratios giving more detail to low \acs{plr}: 0\%, 0.5\%, 1.0\%, 1.2\%, 1.6\%, 2.0\%, 2.2\%, 2.4\%, 2.6\%, 2.8\%, 3.0\%, 5.0\% and 10.0\%. This results in a total of \NumCases test conditions. \subsection{Input parameters} \glsreset{si} \glsreset{ti} A set of different parameters is used to predict the optimal stream that achieves the highest quality in specific network conditions. Based on these inputs the proposed scheme should be able to select the appropriate robust coding stream that should be transmitted. The parameter set characterises three different aspects of the video transmission system; (i) input video (two parameters), (ii) video stream (three parameters) and (iii) transmission network (one parameter). %These parameters aim at covering three different aspects of the video transmission: two parameters from the input video, three parameters from the video stream and one parameter from the transmission network. From the input video the two parameters are measured in a frame by frame basis, while for the video stream information a stream parser is applied to the reference bitstream that was compressed only based optimal \ac{rd} optimisation without considering any robust coding techniques, as shown in Figure~\ref{Figure6:BlockDiagram}. Finally, the parameter related with the transmission network can be obtained using feedback information from the decoder or the network itself. %From the coded video streams complementary information to characterise the streaming content is also extracted. A stream parser is applied to the reference bitstream without error robustness techniques. Therefore, the prediction information extracted is related to a stream coded based on optimal \ac{rd} optimisation which was not affected by robust coding techniques. The following is extracted: The characterisation of these three aspects of the video transmission results in a total of six parameters: \begin{enumerate} \item \textit{\ac{si}}: parameter expressing the amount of spatial detail, with impact on the error recovery performance~\cite{ITUTP9102008}. \item \textit{\ac{ti}}: parameter expressing the amount of difference between temporally adjacent frames~\cite{ITUTP9102008}. This has a direct relation in the impact of errors and performance of the error resilience techniques, as shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, which were discussed in Section~\ref{Section6:Problem}. \item \textit{Inter (percentage)}: the percentage of the image that is compressed using inter-frame prediction techniques. This is an indication of the distribution of spatial and temporal dependencies. This is calculated as the ratio between inter-coded blocks and intra-coded ones. Normally, coded streams with higher number of temporal dependencies are more prone to error propagation, thus higher number of bits should be allocated for error protection \cite{UEPScalable_TCSVT2010}. \item \textit{AVG\_MV}: for the inter-coded regions, the motion information is analysed and the average \ac{mv} magnitude is calculated to provide information about the motion intensity. As shown in previous studies, motion-based \ac{ec} algorithms are less efficient for video signals with higher motion intensity, as \acs{mv} are less predictable using algorithms, such as \ac{mc} or \ac{mve} \cite{HybridConcealmentTB2008}. \item \textit{Bitrate}: information regarding the average bit allocation per frames is calculated, since it will affect the video quality and the amount of bits that can be available for auxiliary information. This value is expressed in Mbps, as illustrated in Table~\ref{Table6:TestMaterialCharacteristics}. \item \textit{\ac{plr}:} feedback information from the network to characterise the transmission losses, expressed by the percentage of packets loss. This is used since for higher loss ratio there is a clear increase in the quality degradation, which affects the performance of the robust coding techniques. Transmission networks with higher loss ratios require a higher bit allocation for robust techniques, therefore the error resilience technique should be able to adapt based on the value of the \ac{plr}. \end{enumerate} \begin{table}[tb] \centering \caption{Pearson Correlation coefficient between the different parameters and the optimal robust coding techniques (optimal stream).} \label{Table6:Correlation} \TABLESVSPACE \renewcommand{\arraystretch}{1.2} \begin{tabular}{p{4cm}x{4cm}} \toprule Parameter & Absolute Pearson Correlation \\ \hline \textit{SI} & 0.3953 \\ \hline \textit{TI} & 0.6079 \\ \hline \textit{Inter (percentage)} & 0.6487 \\ \hline \textit{OPT\_FLOW} & 0.1553 \\ \hline \textit{AVG\_MV} & 0.4647 \\ \hline \textit{Bitrate} & 0.3580 \\ \hline \textit{PLR} & 0.5755 \\ \hline \end{tabular} \end{table} These parameters were compared against the optimal stream, selected from \SMEC to \SMEXPELTWEN. This aims at finding the correlation between the input parameters and the output variable, i.e., target stream. In this study the streams are characterised by a number between 1 and 4. Table~\ref{Table6:Correlation} shows the absolute Person Correlation coefficient obtained across all sequences, bitrates and \acs{plr}. The table shows the results for the parameters previously described, as well as, a new parameter obtained from the average magnitude of the optical flow measured using the method in~\cite{DeepFlow2013} (\textit{OPT\_FLOW}). Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable. Since both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method, due to its higher Person Correlation coefficient. \subsection{Neural network} \glsreset{nn} A \ac{nn} is a computational model based on the structure and functions of biological neural networks. %In terms of computer science, \acs{nn} work similarly to a human nervous system for receiving, processing, and transmitting information between neurons. Mainly, a \ac{nn} consists of three different layers: the input layer which receives data from an external environment, the hidden layer which process the inputs and an output layer~\cite{NeuralNetworks1994}. In the past \acs{nn} have been widely used in video processing applications such as, quality evaluation~\cite{VideoQualityNN_TNN2006}, video analysis~\cite{VideoAnalysisNN_2018} and video coding~\cite{RateDistortionNN_VCIP2017,HEVCComplexityNN_TIP2018}. In the proposed method a feed-forward network is used, which connects every neuron in a given layer to the neurons of the subsequent layer. \begin{figure}[tb] \centering %\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram} \input{043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz} \caption{Representation of the neural network used in the proposed method.} \label{FIG6:NeuralNetwork} \FIGSVSPACE \end{figure} Figure~\ref{FIG6:NeuralNetwork} shows the diagram of the \ac{nn} used in the proposed method. The network contains an input and output layer, as well as, a single hidden layer. The input layer receives the six inputs, as described in the previous sub-section. These inputs are correlated with the optimisation variable being maximised, i.e., the output video quality. Subsequently, the network transforms the input data through a series of neurons in the hidden layer. The output layer comprises four neurons corresponding to each available video stream from \SMEC to \SMEXPELTWEN (different error resilience techniques). Finally, the output with the highest value is selected and the corresponding stream is used for transmission, since it is the one expected to achieve the highest video quality for a particular scenario. In the proposed approach a shallow \ac{nn} is used for simplicity and to avoid over-fitting, due to the limited test cases. By avoiding over-fitting, it is guaranteed that the proposed method would continue to work in future test conditions, not used for the network training. However, different network topologies with different number of neurons in each layer could also be used to predict between different error resilience techniques, as in the proposed method. \subsubsection{Training and validation} %As mentioned before a total of \NumCases test cases are used for the evaluation of the proposed method. To effectively train and validate the \ac{nn}, firstly the cases corresponding to three video sequence were left out, to be used in the performance evaluation of the proposed method. The remaining data set was randomly partitioned into three different groups: (i) 65\% of the test cases are used for training, (ii) 15\% for validation and (iii) 20\% for testing the \ac{nn} performance. The \ac{nn} supervised training uses 557 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993}. Then, 129 test conditions were used for \ac{nn} validation. Finally, 172 conditions are used to evaluate the accuracy of the proposed method after training with different conditions. Tables~\ref{Table6:ConfusionTraining}, \ref{Table6:ConfusionVal} and \ref{Table6:ConfusionTest} show the confusion matrices obtained for training, validation and test sets, respectively. In these matrix each column corresponds to the target result, i.e., the cases where the coded streams, \SMEC to \SMEXPELTWEN, would lead to the highest decoded video quality. Similarly, each row corresponds to the \ac{nn} selected by the trained network. Thus, those cases where the horizontal and vertical labels match, indicate that the network produced the correct output. The remaining positions correspond to the erroneous outputs. The last line shows the accuracy for each target output. These results correspond to the dataset used to train the \ac{nn}. \begin{table}[tb] \centering \caption{Confusion matrix of the training set.} \label{Table6:ConfusionTraining} \TABLESVSPACE \renewcommand{\arraystretch}{1.2} \begin{tabular}{x{.5cm}x{1cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}} & & \multicolumn{4}{c}{Target} \\ \multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} & & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN \\ \hline & \SMEC & 132 & 32 & 6 & 6 \\ \cline{2-6} & \SMEXP & 16 & 57 & 13 & 0 \\ \cline{2-6} & \SMEXPELTEN & 4 & 10 & 69 & 18 \\ \cline{2-6} & \SMEXPELTWEN & 9 & 10 & 17 & 158 \\ \hline \multicolumn{2}{x{1.7cm}|}{Accuracy} & 82.0\% & 52.3\% & 65.7\% & 86.8\% \\ \end{tabular} \end{table} Results in Table~\ref{Table6:ConfusionTraining} shows that streams \SMEC (i.e., without robust coding techniques) and \SMEXPELTWEN (i.e., using optimal \ac{ec} with 20\% of bitrate in the \ac{el}) lead to the highest quality in the majority of the test cases (are more often selected). The \ac{nn} is able to achieve a high accuracy for \SMEC and \SMEXPELTWEN streams, correctly selecting the output stream in more than 80\% of the cases. Regarding the remaining robust streams (i.e., \SMEXP and \SMEXPELTEN), an acceptable accuracy of 52.2\% and 60.6\% is achieved, respectively. This reveals that the \ac{nn} used in the proposed method is able to predict the coded stream which leads to the highest video quality across different conditions with acceptable accuracy. The results from the validation set, shown in Table~\ref{Table6:ConfusionVal}, reveal that the \ac{nn} training did not over-fit, since it shows similar accuracy as the results shown in Table~\ref{Table6:ConfusionTraining}. \begin{table}[tb] \centering \caption{Confusion matrix of the validation set.} \label{Table6:ConfusionVal} \TABLESVSPACE \renewcommand{\arraystretch}{1.2} \begin{tabular}{x{.5cm}x{1cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}} & & \multicolumn{4}{c}{Target} \\ \multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} & & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN \\ \hline & \SMEC & 28 & 4 & 1 & 2 \\ \cline{2-6} & \SMEXP & 4 & 12 & 6 & 0 \\ \cline{2-6} & \SMEXPELTEN & 1 & 7 & 20 & 5 \\ \cline{2-6} & \SMEXPELTWEN & 1 & 0 & 6 & 32 \\ \hline \multicolumn{2}{x{1.7cm}|}{Accuracy} & 82.4\% & 52.2\% & 60.6\% & 82.1\% \\ %\bottomrule \end{tabular} \end{table} \begin{table}[tb] \centering \caption{Confusion matrix of the test set.} \label{Table6:ConfusionTest} \TABLESVSPACE \renewcommand{\arraystretch}{1.2} \begin{tabular}{x{.5cm}x{1cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}} & & \multicolumn{4}{c}{Target} \\ \multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} & & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN \\ \hline & \SMEC & 41 & 10 & 5 & 1 \\ \cline{2-6} & \SMEXP & 5 & 14 & 2 & 0 \\ \cline{2-6} & \SMEXPELTEN & 2 & 0 & 23 & 7 \\ \cline{2-6} & \SMEXPELTWEN & 3 & 4 & 10 & 45 \\ \hline \multicolumn{2}{x{1.7cm}|}{Accuracy} & 80.4\% & 50.0\% & 57.5\% & 84.9\% \\ %\bottomrule \end{tabular} \end{table} To further evaluate the performance of the \ac{nn}, the confusion matrix of the data set used for testing (test set) is shown in the Table~\ref{Table6:ConfusionTest}. This table shows the same results as Table~\ref{Table6:ConfusionTraining}, but for different test cases, which were not used in the training phase. % Therefore, the trained network is not aware of the test cases used for validation. Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 50\% with a maximum value of 84.9\% when the \SMEXPELTWEN is the target stream. Although a lower accuracy is obtained using the test set, the overall accuracy is still acceptable, since it allows to select the optimal robust coding technique (i.e., \SMEC to \SMEXPELTWEN) for the majority of the conditions. In summary, the results presented in this sub-section reveal that the input parameters used in the proposed \ac{nn} are correlated with the decoded video quality, as they can effectively be used to perform data categorisation used in the selection of the optimal error robustness coding technique for the \ac{hevc} standard. \section{Performance evaluation} \label{Section6:ExperimentalResults} In this section the performance of the proposed method is evaluated and discussed. The overall goal of the proposed method is to achieve improved quality using a combination of different robust coding techniques adaptively selected using a feed-forward \ac{nn}. The error resilience techniques comprising the proposed method are used in the context of the \ac{hevc} standard and compared with the reference implementation (\REF). The quality obtained with the proposed optimal error robustness scheme (\PROP) is compared against the reference \ac{hevc} (\REF) and the individual methods used to generate the streams \SMEC to \SMEXPELTWEN. \begin{figure}[!t] \centering \input{043_EndToEndRobustSystem/Results/PlotQualty_vs_PLR_Predicted.tex} \vspace{-2mm} \caption{Average video quality using different error resilience techniques.} \label{Figure6:ProposedQuality} \end{figure} Figure~\ref{Figure6:ProposedQuality} shows the quality obtained for the proposed method and the \REF case for the main target bitrate shown in Table~\ref{TableTestSequences}. Moreover, the quality obtained with the target stream (\BEST) is also shown. One should note that the \BEST case corresponds to the optimal selection, which can only be performed after quality evaluation at the decoder, therefore it is not suitable for real time applications. It is only used to establish the optimal case for comparison purposes. The results in the figure allow the comparison between three relevant cases: proposed stream selection method, the optimal case and the reference method. As shown in Figure~\ref{Figure6:ProposedQuality}, the proposed method clearly outperforms the reference \ac{hevc} for all \acs{plr}. Moreover, in the error-free case (\ac{plr}=0), it leads to imperceptible quality reduction, since the proposed approach is able to select the optimal stream, thus disabling any robust coding technique when the transmission network does not have packet losses. In case of data loss, the results show that the proposed method is able to match the quality obtained by the \BEST case in most cases, revealing that the \ac{nn} is able to efficiently select the optimal stream for transmission. \begin{table}[tb] \centering \caption{Average quality gains ($\Delta$PSNR) in comparison with the reference \ac{hevc} across different bitrates and packet loss ratios.} \label{Table6:AverageGains} \TABLESVSPACE \newcommand{\colwidth}{2cm} \begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} \toprule \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\ & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN & \PROP \\ \hline Basketball Drill & +1.68 & +2.13 & +2.81 & +2.62 & +2.81 \\ \hline Book Arrival & +1.00 & +1.51 & +1.79 & +1.74 & +1.87 \\ \hline BQSquare & +0.98 & +1.14 & +1.06 & +0.73 & +1.20 \\ \hline Cactus & +1.22 & +1.36 & +1.95 & +1.85 & +1.88 \\ \hline Four People & +0.30 & -1.19 & -0.12 & -0.61 & +0.28 \\ \hline Kimono & +1.55 & +2.68 & +2.86 & +3.06 & +3.07 \\ \hline Kristen Sara & +0.38 & -0.52 & +0.20 & -0.38 & +0.30 \\ \hline Park Scene & +1.65 & +1.73 & +2.24 & +2.10 & +1.83 \\ \hline Party Scene & +0.21 & +0.11 & +0.17 & +0.44 & +0.30 \\ \hline People on Street & +1.01 & +1.13 & +1.38 & +1.62 & +1.76 \\ \hline Tennis & +1.18 & +2.11 & +2.43 & +3.28 & +3.28 \\ \hline \textbf{Average} & \textbf{+1.01} & \textbf{+1.11} & \textbf{+1.52} & \textbf{+1.50} & \textbf{+1.69} \\ \bottomrule \end{tabular} \begin{tikzpicture}[overlay] \draw[red, line width=1.2pt] (1.8,4.1) ellipse (3.5cm and 0.3cm); \draw[red, line width=1.2pt] (1.8,3.05) ellipse (3.5cm and 0.3cm); \draw[blue, line width=1.2pt] (6.7,.95) ellipse (.85cm and 0.25cm); \draw[blue, line width=1.2pt] (6.7,3.6) ellipse (.85cm and 0.25cm); \end{tikzpicture} \end{table} Subsequently, a comparison between the different methods is performed, in order to evaluate the quality gains obtained with the proposed method in comparison with each of the different techniques. %for every test case. Table~\ref{Table6:AverageGains} shows the average quality gains in comparison with the reference \ac{hevc} for the test set. These were obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams \SMEC to \SMEXPELTWEN. The average quality gains are measured across all test cases, i.e., test set with different bitrates and different packet loss ratios. Results in Table~\ref{Table6:AverageGains} show that the robust coded streams \SMEC to \SMEXPELTWEN are able to achieve acceptable average quality gains up to 1.52 dB (see results for \SMEXPELTEN). However, since the implementation of robust coding techniques decreases the coding efficiency, it also leads to lower quality than the reference \ac{hevc}. This is noticeable in the results obtained for the Four People sequence in the streams \SMEXP to \SMEXPELTWEN and for the Kristen and Sara sequence in the streams \SMEXPELTEN and \SMEXPELTWEN (see red ellipses). The overall performance of such methods reveals that they are not the best options for transmission conditions with low \acs{plr} (higher number of conditions have \acs{plr} lower than 3\%). On the contrary, the proposed method is always capable of outperforming the reference \ac{hevc} and achieve quality gains from 0.28 dB to 3.28 dB. For the Kimono and Tennis sequences, it is able to achieve the highest quality gains, of up 3.07 dB and 3.28 dB, respectively (see blue ellipses). Summarising, by adaptively selecting the video stream, and implicitly the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness, achieving an average quality gain across different bitrates and \acs{plr} of 1.69 dB. Finally, the performance of the proposed method is evaluated for video sequences, which was not fed into the \ac{nn} training phase. This study aims at finding the accuracy of the optimal stream selection in case of a new video content. For this purpose the Basketball Drive sequence was randomly selected, and not used in previous simulations. Table~\ref{Table6:AverageGainBasketBallDrive} shows the average quality results obtained with these sequences, similarly to Table~\ref{Table6:AverageGains}. These results also confirm that the proposed method is able to select the optimal robust stream to be used for transmission, even when new video content is transmitted. The proposed method achieves a quality gain up to 4.46 dB in comparison with the reference \ac{hevc}, outperforming the quality achieved with the stream \SMEXPELTWEN (i.e., 4.19 dB). This confirms the advantage of the proposed method to adaptively select among different streams with different levels of error protections. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Summary} \label{Section6:Summary} \begin{table}[tb] \centering \caption{Average quality gain ($\Delta$PSNR) in comparison with the reference \ac{hevc} for video sequences not used in the NN training.} \label{Table6:AverageGainBasketBallDrive} \TABLESVSPACE \newcommand{\colwidth}{1.7cm} \begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} \toprule \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\ & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN & \PROP \\ \hline Basketball Drive & +2.10 & +2.68 & +3.36 & +3.45 & +3.55 \\ \hline Kendo & +1.92 & +3.24 & +3.94 & +4.19 & +4.46 \\ \hline Race Horses & +0.85 & +1.02 & +0.88 & +0.75 & +1.06 \\ \bottomrule \end{tabular} \end{table} In this chapter an adaptive error robustness method is proposed, based on reference frame selection and \ac{ec}-aware optimisation. By adaptively selecting a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the individual reference techniques. %Moreover, the proposed method is able to outperform those cases where a single error resilience technique is used. The overall results demonstrate that the proposed method is able to increase the reconstruction video quality in case of errors. Also, by adapting the transmitted stream to the network conditions, a better trade-off between coding efficiency and error robustness can be achieved. The consistent quality improvement, allows the use of this method in dynamic broadcasting video services where network feedback is available. This architecture is suitable for dynamic streaming technologies, e.g., HTTP streaming based on MPEG-DASH~\cite{HEVCHTTPStreamAug2016} or content delivery over heterogeneous networks using the \ac{mmt}~\cite{ATSCMMT2016}.