%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % CHAPTER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \chapter{Adaptive Error Robustness for Highly Efficient Video Coding \VERSION{2}} \label{Chapter:EndToEndRobustSystem} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Introduction} \glsunset{hevc} \glsunset{ec} \glsreset{nn} The effectiveness of different error resilience techniques is rather variable depending on the video content, the transmitted bitrate, network conditions, among others. Therefore, it is not straightforward to select a single technique which achieves the highest efficiency for every possible scenario and also for the whole range of networking conditions. This chapter addresses this problem by proposing an adaptive network that optimises the efficiency of the techniques previously described in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}. At the encoder-side, a pre-trained neural network predicts the video stream that is able to achieve the highest decoded quality for a particular loss cases, among the various video streams encoded with different levels of compression efficiency and error robustness. The selection is based on different input parameters that include information from the video signal, the coded stream and the transmission network. Using this approach the overall performance of the proposed adaptive method significantly increases, yielding a better trade-off between compression efficiency and error robustness. This chapter is organised as follows. Section~\ref{Section6:Problem} presents the problem addressed in this chapter and experimental evidence, as motivation for the method investigated. Section~\ref{Section6:ProposedMethod} provides an overview of architecture that supports the implementation of the proposed method and Section~\ref{Section6:OptimisationMethod} describes the technical details. Section~\ref{Section6:ExperimentalResults} presents the performance evaluation and a discussion of results. Finally, Section~\ref{Section6:Summary} concludes the chapter. \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The problem of adaptive error robustness} \label{Section6:Problem} In the scope of this thesis different robust coding techniques were investigated to increase the overall performance of video transmission under error-prone conditions. Different algorithms were devised in the proposed methods, such as, reference frame selection, motion vector redundancies and error concealment signalling. In previous chapters, these methods were evaluated against existing techniques, revealing superior performance and better efficiency to deal with error robustness problems in high efficient video coding. Overall, the performance results of these methods have demonstrated that they can efficiently increase the robustness of video streams against transmission errors and losses, leading to higher decoded video quality. However, each method on its own does not adapt to different conditions. While the reference frame selection algorithm presented in Chapter~\ref{Chapter:ErrorResilienceTechniques} introduces low overhead in comparison with existing techniques, the method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} achieves higher quality gains at cost of slightly higher bitrate increase. Therefore, one may conclude that each method may be further optimised for different applications, video characteristics and network conditions. This chapter advances one step forward , by investigating a joint robust coding method using different techniques to optimise the robust coder across different conditions, such as, the input content and the network information. This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and can be selected based on the current conditions. \subsection{Comparison of the proposed methods} In order to address the above problem of adaptive robust coding, a simulation study was carried out to compare the error robustness performance of the previously proposed methods for different conditions. This study extends the previous evaluations presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, as they target a wider range of test material and network conditions. Moreover, it aims at comparing the performance of different techniques across a variety of video contents, bitrates and loss conditions, in order to find relevant insights which affect the decoded video quality. Consequently, these relevant insights may be used to develop an adaptive robust coding method. %In order to perform a wider evaluation different bitrate and \acs{plr} were used. Table~\ref{Table6:TestVariables} illustrates the selected configurations. As shown in the table a total of 15 test sequences were used encoder using 5 different target bitrates. One should note that the main bitrate varies based on the test sequence and it is shown in third column of Table~\ref{TableTestSequences}. Different network conditions were evaluated using a total of 13 different \acs{plr}, with higher special focus on lower loss ratios, which are more relevant to modern video streaming applications. The following methods were evaluated in the simulation study, namely: \begin{itemize} \item the default \ac{hevc} decoder with frame-copy \ac{ec} (\REF), as reference method; \item the default \ac{hevc} encoder without robust coding-techniques and motion vector extrapolation \ac{ec} (\MVE); \item the proposed reference frame selection scheme (\EXP); \item the \EXP method in combination with an \ac{ec} optimisation with 10\% of bitrate allocated for the scalable enhancement layer (\EXPELTEN); \item the same as above, with 20\% of bitrate allocated for the scalable enhancement layer (\EXPELTWEN). \end{itemize} These methods were selected as they are the main techniques proposed within this research work and cover two aspects of a robust video transmission: (i) robust video encoding by dynamically select the reference frames and (ii) robust video decoding by improving the error concealment performance. \input{043_EndToEndRobustSystem/Results/AdaptiveSequenceList.tex} Table~\ref{Table6:TestMaterialCharacteristics} presents the list of test sequences used. These sequences were selected to cover a wide range of video content, with different levels of spatial and temporal complexity, as demonstrated by the values of \ac{si} and \ac{ti}~\cite{ITUTP9102008}, respectively. Moreover, from the bitstream encoded with the reference implementation of the \ac{hevc} encoder, the percentage of the inter-coded blocks and the average \ac{mv} magnitude was extracted and presented in the table (columns 4 and 5, respectively). These values were extracted to compare them against the performance obtained with the proposed robust coding methods. The remaining configuration was kept as described in Section~\ref{Section3:ErrorResilienceEvaluation}. %The experimental methodology carried out in this section follows the same conditions described in Section~\ref{Section3:ErrorResilienceEvaluation}. \subsubsection{Preliminary simulation results and discussion} \begin{figure}[!t] \centering \input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex} \vspace{-4mm} \caption{Average decoded video quality for the bitstreams encoded with the main bitrate (bps).} \label{Figure6:AverageQualityMain} \end{figure} Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} shows the average decoded video quality \ac{psnr} obtained from 50 trials, for different loss conditions. Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate listed in Table~\ref{Table6:TestMaterialCharacteristics}, referred to as main bitrate. Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained by reducing the main bitrate by 40\%. This is used to evaluate the impact of the network bitrate on the decoded video quality. The results shown in the figures corresponds to the same test sequences, in order to carry out an accurate comparison. \begin{figure}[!t] \centering \input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex} \vspace{-4mm} \caption{Average decoded video quality using a 40\% reduction of the main bitrate (bps).} \label{Figure6:AverageQualityReduction} \end{figure} Firstly, comparing the results for different bitrates, it is noticeable that lower quality is obtained for the lower bitrates, which results from the higher compression ratios used. It is also noticed that lower quality gains, when comparing with the reference \ac{hevc} encoder (\REF), are obtained for the video streams encoded with a 40\% reduction of bitrate. This indicates that for lower bitrates the performance of the proposed methods decreases, especially for the \EXPELTWEN case, where 20\% of the bitrate is allocated for the enhancement layer. Therefore, for lower bitrates, using methods that have a lower impact on the coding efficiency (e.g., \EXP) or even disable the robust coding techniques (\MVE) may lead to higher performance. Secondly, comparing the results across different video content, i.e., different test sequences, one can notice that the performance of the different methods is not consistent. For instance, in the result of Figure~\ref{Figure6:AverageQualityMain}, for Tennis sequence the \EXPELTWEN clearly outperforms the remaining methods, however, for the BQSquare sequence higher performance is achieved by using less bitrate allocated for the \ac{el} (i.e., 10\%). This is a consequence of the higher quality degradation due to higher motion in Tennis sequence (see results for TI and \ac{mv} size in Table~\ref{Table6:TestMaterialCharacteristics}) which required higher redundant information to increase the overall quality. %\HIGHLIGHT{Mention SI!!} Finally, comparing the results for all test sequences and both bitrates it can be clearly noticed that the \acs{plr} have a significant impact on the performance of the error robustness techniques. In case of the proposed reference frame selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions. For the \EXPELTWEN case, higher \acs{plr} are required to be able to outperform both the reference \ac{hevc} and the \ac{mve} method. Summarising, the performance of the proposed methods is not similar across different video content and bitrates, opening research to adaptively choose the best method that should be applied to achieve the highest decoded video quality. In order to achieve increased efficiency and higher video quality, the best trade-off between auxiliary information and video bitrate can be found by selecting the optimal robust coding algorithm. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Architecture for adaptive error robustness} \label{Section6:ProposedMethod} This section describes the proposed method, which aims at obtaining optimal error robustness for different streaming scenarios, characterised by different video content and network conditions. The main objective of this robust coding scheme it to achieve an optimal trade-off between error robustness tools and coding efficiency. \begin{figure}[tb] \centering %\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram} \input{043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz} \caption{Adaptive error robustness architecture for 4 streams (S1...S4).} \label{Figure6:BlockDiagram} \FIGSVSPACE \end{figure} Figure~\ref{Figure6:BlockDiagram} illustrates the proposed architecture. Firstly, the input signals are coded using different levels of error protections. Then, an optimisation algorithm is used to decide which bitstream should be transmitted at any given instant, based on different parameters. The selected stream is expected to obtain the highest quality at the decoder-side for the given conditions. A reference video coder is used to compress the video signal using highly efficient coding techniques, obtaining the best \ac{rd} performance for error-free environments. This is used as the reference bitstream. Subsequently, a transcoding operation is applied to this reference bitstream aided by the input video signal to efficiently select the reference frames used for inter prediction. The reference frame selection mechanism proposed in Chapter~\ref{Chapter:ErrorResilienceTechniques} is used for this purpose. Moreover, loss simulation and a saliency-based \ac{ec} optimisation are also used in the robust coding process, which enables the selection of the optimal \ac{ec} algorithm to be applied at the decoder. Finally, since the optimal \ac{ec} mode might not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el} using two different bitrates. In this case 10\% and 20\% of the total bitrate were chosen, providing two different levels of error robustness. The \ac{ec} optimisation and \ac{el} coding is performed as described in Chapter~\ref{Chapter:ConcealmentAwareTechniques}. The proposed approach results in four different bitstreams with different characteristics (referred to as "S1" to "S4", respectively, as shown in Figure~\ref{Figure6:BlockDiagram}): (i) S1: reference stream without error robust coding; (ii) S2: robust video coding using reference frame selection; (iii) S3: robust video coding using selective reference frames and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue; (iv) S4: same as the previous bitstream but with 20\% bitrate allocated for the scalable~\ac{el}. Finally, the video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and feedback from the transmission channel. From the input video signal, relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, further information is also extracted from the coded bitstream using a stream parser. Finally, the loss ratio of the transmission channel is also used. These parameters are fed into the optimisation block to select the best bitstream for transmission. The optimisation process is described in detail in the following section. The main purpose of this architecture is to allow dynamic stream switching during real time transmission. To accomplish this without causing drift at the decoder, the switching procedure should take place at refresh points, i.e., \ac{idr} and \ac{cra} frames, which correspond to clean random access points. In the proposed scheme, \ac{idr} frames are introduced every 16 frames, resulting in between 1 to 3 switching points every second, depending on the frame rate of the video signal. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Error robustness selection method} \label{Section6:OptimisationMethod} In this section the optimisation method used to select a robust video stream is presented. The selection method is based on a shallow neural network which predicts the optimal stream based on different parameters. This section describes both the input parameters and the neural network used in the optimal stream selection process. Moreover, a description data set used to train and evaluate the neural network used is also provided. \subsection{Data set} %\def\NumCases{845\xspace} \def\NumCases{910\xspace} The proposed method includes an optimisation stage that uses a neural network classifier, in order to select the optimal stream to be used for transmission. This stream is able to obtain the highest quality for a particular transmission scenario. Therefore, several test cases were simulated in order to obtain results for a wide range of conditions. These tests are mainly characterised by three elements: \begin{itemize} \item Video signal; \item Target bitrate; \item Packet loss ratio. \end{itemize} To obtain the data set, 14 different test sequences with different scene content were used, as listed in Table~\ref{Table6:TestMaterialCharacteristics}. Those sequences were encoded with the main bitrate (see Table~\ref{Table6:TestMaterialCharacteristics}), as well as four different others, obtained by reducing the main bitrate by 10\%, 20\%, 30\% and 40\%. Then, the following \acs{plr} were used to cover a wide range of loss ratios giving more detail to low \acs{plr}: 0\%, 0.5\%, 1.0\%, 1.2\%, 1.6\%, 2.0\%, 2.2\%, 2.4\%, 2.6\%, 2.8\%, 3.0\%, 5.0\% and 10.0\%. This results in a total of \NumCases test conditions. \subsection{Input parameters} \glsreset{si} \glsreset{ti} A set of different parameters is used to predict the optimal stream that achieves the highest quality in specific network conditions. Based on these inputs the proposed scheme is able to select the appropriate robust coding scheme which should be used. These parameters aim at characterise three different aspects of the video transmission system; (i) input video (two parameters), (ii) video stream (three parameters) and (iii) transmission network (one parameter). %These parameters aim at covering three different aspects of the video transmission: two parameters from the input video, three parameters from the video stream and one parameter from the transmission network. From the input video the two parameters are measured in a frame by frame basis, while for the video stream information a stream parser is applied to the reference bitstream that was compressed only based optimal \ac{rd} optimisation without considering any robust coding techniques, as shown in Figure~\ref{Figure6:BlockDiagram}. Finally, the parameter related with the transmission network can be obtained using feedback information from the decoder. %From the coded video streams complementary information to characterise the streaming content is also extracted. A stream parser is applied to the reference bitstream without error robustness techniques. Therefore, the prediction information extracted is related to a stream coded based on optimal \ac{rd} optimisation which was not affected by robust coding techniques. The following is extracted: The characterisation of these three aspects of the video transmission results in a total of six parameters: \begin{enumerate} \item \textit{\ac{si}}: parameter expressing the amount of spatial detail, which consequently affects the error recovery operations, leading to higher quality degradation~\cite{ITUTP9102008}. \item \textit{\ac{ti}}: parameter expressing the amount of difference between temporally adjacent frames~\cite{ITUTP9102008}. This has a direct relation to the impact of errors, and thus the performance of the error resilience techniques, as shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, which were discussed in Section~\ref{Section6:Problem}. \item \textit{Inter (percentage)}: the percentage of the image that is compressed using inter-frame prediction techniques. This is an indication of the distribution of spatial and temporal dependencies. This is calculated as the ratio of inter-coded blocks over intra-coded ones. Normally, coded streams with higher number of temporal dependencies are more prone to error propagation, thus higher number of bits should be allocated for error protection \cite{UEPScalable_TCSVT2010}. \item \textit{AVG\_MV}: for the inter-coded regions, the motion information is analysed and the average \ac{mv} magnitude is calculated to provide information about the motion intensity. As shown in previous studies, motion-based \ac{ec} algorithms are less efficient for video signals with higher motion intensity, as \acs{mv} are less predictable using algorithms, such as \ac{mc} or \ac{mve} \cite{HybridConcealmentTB2008}. \item \textit{Bitrate}: information regarding the average bit allocation per frames is calculated, since it will affect the video quality and the amount of bits that can be available for auxiliary information. This value is expressed in Mbps, as illustrated in Table~\ref{Table6:TestMaterialCharacteristics}. \item \textit{\ac{plr}:} feedback information from the network to characterise the transmission losses, expressed by the percentage of packets loss. This is used since for higher loss ratio there is a clear increase in the quality degradation, which affects the performance of the robust coding techniques. Transmission networks with higher loss ratios require a higher bit allocation for robust techniques, therefore the error resilience technique should adapt based on the value of the \ac{plr}. \end{enumerate} %To characterise the input video signal the following parameters are used: \ac{si} and \ac{ti}~\cite{ITUTP9102008}. %These parameters are obtained as described in Chapter~\ref{Chapter:HEVCErrorResilience}, and aim at evaluating the spatial complexity and motion complexity of the video scenes, which have relevant impact on the error robustness of the coded streams. %As shown, test material with higher \ac{si} or \ac{ti} suffer from higher quality degradation as the \ac{plr} increases, e.g., Book Arrival and Tennis sequences. These parameters were compared against the optimal stream, selected from S1 to S4. This aims at finding the correlation between the input parameters and the output variable, i.e., target stream. In this study the streams are characterised by a number between 1 and 4. Table~\ref{Table6:Correlation} shows the absolute Person Correlation coefficient obtained across all sequences, bitrates and \acs{plr}. The table shows the results for the parameters previously described, as well as, a new parameter obtained from the average magnitude of the optical flow measured using the method in~\cite{DeepFlow2013} (\textit{OPT\_FLOW}). Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable. As both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method, as it achieves a higher Person Correlation coefficient. \begin{table}[b] \centering \caption{Pearson Correlation coefficient between the different parameters and the optimal robust coding techniques (optimal stream).} \label{Table6:Correlation} \TABLESVSPACE \renewcommand{\arraystretch}{1.2} \begin{tabular}{p{4cm}x{4cm}} \toprule Parameter & Absolute Pearson Correlation \\ \hline % \textit{SI} & -0.1553 \\ \hline % \textit{TI} & 0.6079 \\ \hline % \textit{Inter (percentage)} & -0.6487 \\ \hline % \textit{OPT\_FLOW} & 0.6953 \\ \hline % \textit{AVG\_MV} & 0.4647 \\ \hline % \textit{Bitrate} & 0.3580 \\ \hline % \textit{PLR} & 0.5755 \\ \hline \textit{SI} & 0.3953 \\ \hline \textit{TI} & 0.6079 \\ \hline \textit{Inter (percentage)} & 0.6487 \\ \hline \textit{OPT\_FLOW} & 0.1553 \\ \hline \textit{AVG\_MV} & 0.4647 \\ \hline \textit{Bitrate} & 0.3580 \\ \hline \textit{PLR} & 0.5755 \\ \hline \end{tabular} \end{table} % Podia se citar este paper % \cite{IntraRefreshICIP2009} \subsection{Neural network} \glsreset{nn} A \ac{nn} is a computational model based on the structure and functions of biological neural networks. %In terms of computer science, \acs{nn} work similarly to a human nervous system for receiving, processing, and transmitting information between neurons. Mainly, a \ac{nn} consists of three different layers: the input layer which receives data from an external environment, the hidden layer which process the inputs and an output layer~\cite{NeuralNetworks1994}. In the past \acs{nn} have been widely used in video processing applications such as, quality evaluation~\cite{VideoQualityNN_TNN2006}, video analysis~\cite{VideoAnalysisNN_2018} and video coding~\cite{RateDistortionNN_VCIP2017,HEVCComplexityNN_TIP2018}. In the proposed method a feed-forward network is used, which connects every neuron in a given layer to the neurons of the subsequent layer. Figure~\ref{FIG6:NeuralNetwork} shows the diagram of the \ac{nn} used in the proposed method. The network contains and input and output layer, as well as, a single hidden layer. The input layer receives the six inputs, as described in the previous sub-section. These inputs are correlated with the optimisation variable being maximised, i.e., the output video quality. Subsequently, the network transforms the input data through a series of neurons in the hidden layer. The output layer comprises four neurons corresponding to each available video stream from S1 to S4 (different error resilience techniques). Finally, the output with the highest value is selected and the corresponding stream is used for transmission, since it is the one expected to achieve the highest video quality for a particular scenario. \begin{figure}[tb] \centering %\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram} \input{043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz} \caption{Representation of the neural network used in the proposed method.} \label{FIG6:NeuralNetwork} \FIGSVSPACE \end{figure} In the proposed approach a shallow \ac{nn} is used for simplicity and to avoid over-fitting, due to the limited test cases. By avoiding over-fitting, it is guaranteed that the proposed method would continue to work in future test conditions, not used for the network training. However, different network topologies with different number of neurons in each layer could also be used to predict between different error resilience techniques, as in the proposed method. \subsubsection{Training and validation} %As mentioned before a total of \NumCases test cases are used for the evaluation of the proposed method. To effectively train and validate the \ac{nn}, the data set is randomly partitioned into three different groups: (i) 55\% of the test cases are used for training, (ii) 15\% for validation and (iii) 30\% for testing the \ac{nn} performance. The \ac{nn} supervised training uses 591 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993}. The remaining 254 conditions are used to evaluate the accuracy of the proposed method after training with different conditions. From this test one video sequence was left out, to be used in the performance evaluation of the proposed method. Table~\ref{Table6:ConfusionTraining} shows the confusion matrix of the \ac{nn} training phase. In this matrix each column corresponds to the target result, i.e., the cases where the coded streams, S1 to S4, would lead to the highest decoded video quality. Similarly, each row corresponds to the \ac{nn} selected by the trained networks. Thus, those cases where the horizontal and vertical labels match, indicate that the network produced the correct output. The remaining positions correspond to the erroneous outputs. The last line shows the accuracy for each target output. These results correspond to the dataset used to train the \ac{nn}. \begin{table}[tb] \centering \caption{Confusion matrix of the training and validation test cases.} \label{Table6:ConfusionTraining} \TABLESVSPACE \renewcommand{\arraystretch}{1.2} \begin{tabular}{x{.5cm}x{1cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}} & & \multicolumn{4}{c}{Target} \\ \multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} & & S1 & S2 & S3 & S4 \\ \hline & S1 & 122 & 17 & 3 & 0 \\ \cline{2-6} & S2 & 13 & 80 & 19 & 2 \\ \cline{2-6} & S3 & 0 & 12 & 96 & 9 \\ \cline{2-6} & S4 & 0 & 4 & 11 & 202 \\ \hline \multicolumn{2}{x{1.7cm}|}{Accuracy} & 90.3\% & 70.8\% & 74.4\% & 94.8\% \\ \end{tabular} \end{table} Results in Table~\ref{Table6:ConfusionTraining} shows that streams S1 (i.e., without robust coding techniques) and S4 (i.e., using optimal \ac{ec} with 20\% of bitrate in the \ac{el}) lead to the highest quality in the majority of the test cases (are more often selected). The \ac{nn} is able to achieve a high accuracy for S1 and S4 streams, correctly selecting the output stream in more than 90\% of cases. Regarding the remaining robust streams (i.e., S2 and S3), an acceptable accuracy of 70.8\% and 74.4\% is achieved, respectively. This reveals that the \ac{nn} used in the proposed method is able to predict the coded stream which leads to the highest video quality across different conditions with acceptable accuracy. \begin{table}[tb] \centering \caption{Confusion matrix of the test cases not used for testing.} \label{Table6:ConfusionTest} \TABLESVSPACE \renewcommand{\arraystretch}{1.2} \begin{tabular}{x{.5cm}x{1cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}} & & \multicolumn{4}{c}{Target} \\ \multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} & & S1 & S2 & S3 & S4 \\ \hline & S1 & 56 & 7 & 2 & 0 \\ \cline{2-6} & S2 & 10 & 28 & 6 & 2 \\ \cline{2-6} & S3 & 1 & 5 & 31 & 8 \\ \cline{2-6} & S4 & 0 & 5 & 7 & 86 \\ \hline \multicolumn{2}{x{1.7cm}|}{Accuracy} & 83.6\% & 62.2\% & 67.4\% & 89.6\% \\ %\bottomrule \end{tabular} \end{table} To further evaluate the performance of the \ac{nn}, the confusion matrix of the data set used for testing is shown in the Table~\ref{Table6:ConfusionTest}. This table shows the same results as Table~\ref{Table6:ConfusionTraining}, but for different test cases, which were not used in the training phase. % Therefore, the trained network is not aware of the test cases used for validation. Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 62.2\% with a maximum value of 89.6\% when the S4 is the target stream. Although a lower accuracy is obtained using the validation test cases, the overall accuracy is still acceptable, since it allows to select the optimal robust coding technique (i.e., S1 to S4) for the majority of the conditions. In summary, the results presented in this sub-section reveal that the input parameters used in the proposed \ac{nn} are correlated with the decoded video quality, as they can effectively be used to perform data categorisation used in the selection of the optimal error robustness coding technique for the \ac{hevc} standard. \section{Performance evaluation} \label{Section6:ExperimentalResults} In this section the performance of the proposed method is evaluated and discussed. The overall goal of the proposed method is to achieve improved quality using a combination of different robust coding techniques adaptively selected using a feed-forward \ac{nn}. The error resilience techniques comprising the proposed method are used in the context of the \ac{hevc} standard and compared with the reference implementation (\REF). The quality obtained with the proposed optimal error robustness scheme (\PROP) is compared against the reference \ac{hevc} (\REF) and the methods used to generate the streams S1 to S4. \begin{figure}[!t] \centering \input{043_EndToEndRobustSystem/Results/PlotQualty_vs_PLR_Predicted.tex} \vspace{-2mm} \caption{Average video quality using different error resilience techniques.} \label{Figure6:ProposedQuality} \end{figure} Figure~\ref{Figure6:ProposedQuality} shows the quality obtained for the proposed method and the \REF case for the main target bitrate shown in Table~\ref{TableTestSequences}. Moreover, the quality obtained with the target stream (\BEST) is also shown. One should note that the \BEST case corresponds to the optimal selection, which can only be performed after quality evaluation at the decoder, therefore it is not suitable for real time applications. It is only used to establish the optimal case for comparison purposes. The results in the figure allows to evaluate the impact in the decoded quality of the selection used in the proposed method and compare it with the optimal case. As shown in Figure~\ref{Figure6:ProposedQuality}, the proposed method clearly outperforms the reference \ac{hevc} for all \acs{plr}. Moreover, in the error-free case (\ac{plr}=0), it leads to imperceptible quality reduction, since the proposed approach is able to select the optimal stream, thus disabling any robust coding technique when the transmission network does not have packet losses. In case of data loss, the results show that the proposed method is able to match the quality obtained by the \BEST case in most cases, revealing that the proposed method is able to efficiently select the optimal stream for transmission. \begin{table}[tb] \centering \caption{Average quality gains ($\Delta$PSNR) in comparison with the reference \ac{hevc} across different bitrates and loss ratios.} \label{Table6:AverageGains} \TABLESVSPACE \newcommand{\colwidth}{2cm} \begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} \toprule \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\ & S1 & S2 & S3 & S4 & \PROP \\ \hline Basketball Drill & +1.61 & +2.66 & +2.48 & +2.44 & +2.75 \\ \hline %Basketball Drive & +2.17 & +3.46 & +3.22 & +3.63 & +3.69 \\ \hline Book Arrival & +0.96 & +1.66 & +1.43 & +1.59 & +1.74 \\ \hline BQSquare & +0.92 & +0.96 & +0.80 & +0.62 & +1.09 \\ \hline Cactus & +1.28 & +2.04 & +1.89 & +1.96 & +2.09 \\ \hline Four People & +0.28 & -0.23 & -0.88 & -0.99 & +0.25 \\ \hline Kendo & +0.81 & +0.78 & +0.20 & +0.61 & +1.02 \\ \hline Kimono & +1.65 & +3.04 & +3.26 & +3.27 & +3.41 \\ \hline Kristen Sara & +0.36 & +0.15 & -0.33 & -0.50 & +0.38 \\ \hline Park Scene & +1.64 & +2.18 & +1.88 & +2.02 & +2.25 \\ \hline Party Scene & +0.21 & +0.20 & -0.07 & -0.24 & +0.29 \\ \hline People on Street & +1.00 & +1.27 & +0.91 & +1.49 & +1.63 \\ \hline Race Horses & +1.93 & +3.89 & +4.06 & +4.30 & +4.39 \\ \hline Tennis & +1.15 & +2.24 & +2.17 & +3.12 & +3.18 \\ \hline \textbf{Average} & \textbf{+1.06} & \textbf{+1.60} & \textbf{+1.37} & \textbf{+1.51} & \textbf{+1.88} \\ \bottomrule \end{tabular} \begin{tikzpicture}[overlay] \draw[red, line width=1.2pt] (1.8,5.2) ellipse (3.5cm and 0.3cm); \draw[red, line width=1.2pt] (3,3.6) ellipse (2.3cm and 0.3cm); \draw[blue, line width=1.2pt] (6.7,1.45) ellipse (.85cm and 0.25cm); \draw[blue, line width=1.2pt] (6.7,4.10) ellipse (.85cm and 0.25cm); \end{tikzpicture} \end{table} Subsequently, a comparison between the different methods is performed, in order to evaluate the quality gains obtained with the proposed method in comparison with the different techniques. %for every test case. Table~\ref{Table6:AverageGains} shows the average quality gains in comparison with the reference \ac{hevc} obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams S1 to S4. The average quality gains are measured across all test cases, i.e., different bitrates and different loss ratios. Results in Table~\ref{Table6:AverageGains} shows that the robust coded streams S1 to S4 are able to achieve acceptable average quality gains up to 1.51 dB (see results for S2). However, since the implementation of robust coding techniques decreases the coding efficiency, it also leads to lower quality than the reference \ac{hevc}. This is noticeable in the results obtained for the Four People sequence in the streams S2 to S4 and for the Kristen and Sara sequence in the streams S3 and S4 (see red ellipses). The overall performance of such methods reveals that they are not suitable for transmission conditions with low \acs{plr}, e.g., lower than 3\% (higher number of conditions have \acs{plr} lower than 3\%). On the contrary, the proposed method is always able to outperform the reference \ac{hevc} and achieve quality gains of at least 0.25 dB for the Four People sequence. Moreover, for the Kimono and Race Horses sequences, it is able to achieve the highest quality gains, of up 3.41 dB and 4.39 dB, respectively (see blue ellipses). Summarising, by adaptively selecting the video stream, and consequently the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness, achieving an average quality gain across different bitrates and \acs{plr} of 1.88 dB. Finally, the performance evaluation of the proposed method is evaluated for a video sequence, which was not fed into the \ac{nn} training phase. This study aims at finding the accuracy of the optimal stream selection in case of a new video content. For this purpose the Basketball Drive sequence was randomly selected, and not used in previous simulations. Table~\ref{Table6:AverageGainBasketBallDrive} shows the average quality results obtained with this sequence, similarly to Table~\ref{Table6:AverageGains}. These results confirm that the proposed method is able to select the optimal robust stream to be used for transmission, even when new video content is transmitted. The proposed method achieves a quality gain of 3.55 dB in comparison with the reference \ac{hevc}, outperforming the quality achieved with the stream S4 (i.e., 3.45 dB). This confirms the advantage of the proposed method to adaptively select among different streams with different levels of error protections. \begin{table}[tb] \centering \caption{Average quality gain ($\Delta$PSNR) in comparison with the reference \ac{hevc} for the Basketball Drive sequence.} \label{Table6:AverageGainBasketBallDrive} \TABLESVSPACE \newcommand{\colwidth}{1.7cm} \begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} \toprule \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\ & S1 & S2 & S3 & S4 & \PROP \\ \hline Basketball Drive & +2.10 & +2.68 & +3.36 & +3.45 & +3.55 \\ \bottomrule \end{tabular} \end{table} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Summary} \label{Section6:Summary} In this chapter an adaptive error robustness method is proposed, based on reference frame selection and \ac{ec}-aware optimisation. By adaptively selecting a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the reference techniques. %Moreover, the proposed method is able to outperform those cases where a single error resilience technique is used. The overall results demonstrate that the proposed method is able to increase the reconstruction video quality in case of errors. Also, by adapting the transmitted stream to the network conditions, a better trade-off between coding efficiency and error robustness can be achieved. The consistent quality improvement, allows the use of this method in dynamic broadcasting video services where network feedback is available. This architecture is suitable for dynamic streaming technologies, e.g., HTTP streaming based on MPEG-DASH~\cite{HEVCHTTPStreamAug2016} or content delivery over heterogeneous networks using the \ac{mmt}~\cite{ATSCMMT2016}.