diff --git a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex index e4f5fa5..6d5cd8c 100644 --- a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex +++ b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex @@ -69,7 +69,7 @@ These methods were selected because they are the main techniques proposed within Table~\ref{Table6:TestMaterialCharacteristics} presents the list of test sequences used. These sequences were selected to cover a wide range of video content, with different levels of spatial and temporal complexity, as demonstrated by the values of \ac{si} and \ac{ti}~\cite{ITUTP9102008}, respectively. -Moreover, from the bitstream encoded with the reference implementation of the \ac{hevc} encoder, the percentage of the inter-coded blocks and the average \ac{mv} magnitude was extracted and presented in the table (columns 4 and 5, respectively). +Moreover, from the bitstream encoded with the reference implementation of the \ac{hevc} encoder, the percentage of the inter-coded blocks and the average \ac{mv} magnitude was extracted and presented in the table (columns 4 and 5, respectively). These parameters were considered due to their possible relation with the performance obtained by the proposed methods. The remaining configuration was kept as described in Section~\ref{Section3:ErrorResilienceEvaluation}. %The experimental methodology carried out in this section follows the same conditions described in Section~\ref{Section3:ErrorResilienceEvaluation}. @@ -101,7 +101,7 @@ Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained by reducin Firstly, comparing the results for different bitrates, it is noticeable that lower quality is obtained for the lower bitrates, which obviously results from the higher compression ratios used. It is also noticed that lower quality gains, when comparing with the reference \ac{hevc} encoder (\REF), are obtained for the video streams encoded with a 40\% reduction of bitrate. This indicates that for lower bitrates the performance of the proposed methods decreases, especially for the \MEXPELTWEN case, where 20\% of the bitrate is allocated for the enhancement layer. Therefore, for lower bitrates, using methods that have a lower impact on the coding efficiency (e.g., \MEXP) or even disable the robust coding techniques (\MEC) may lead to higher performance. Secondly, comparing the results across different video content, i.e., different test sequences, one can notice that the performance of the different methods is not consistent. -For instance, in the results of Figure~\ref{Figure6:AverageQualityMain}, for Tennis sequence the method \MEXPELTWEN clearly outperforms the remaining ones. +For instance, in the results of Figure~\ref{Figure6:AverageQualityMain}, for Tennis sequence the method \MEXPELTWEN clearly outperforms the remaining ones. However for the BQSquare sequence, higher performance is achieved by using less bitrate (i.e., 10\%) allocated for the \ac{el} (\MEXPELTEN). This is a consequence of the higher quality degradation due to higher motion in Tennis sequence (see results for TI and \ac{mv} size in Table~\ref{Table6:TestMaterialCharacteristics}) which requires higher redundant information to increase the overall quality. %\HIGHLIGHT{Mention SI!!} @@ -202,7 +202,7 @@ The characterisation of these three aspects of the video transmission results in \item \textit{\ac{si}}: parameter expressing the amount of spatial detail, with impact on the error recovery performance~\cite{ITUTP9102008}. -\item \textit{\ac{ti}}: parameter expressing the amount of difference between temporally adjacent frames~\cite{ITUTP9102008}. +\item \textit{\ac{ti}}: parameter expressing the amount of difference between temporally adjacent frames~\cite{ITUTP9102008}. This has a direct relation in the impact of errors and performance of the error resilience techniques, as shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, which were discussed in Section~\ref{Section6:Problem}. \item \textit{Inter (percentage)}: the percentage of the image that is compressed using inter-frame prediction techniques. @@ -246,7 +246,7 @@ This aims at finding the correlation between the input parameters and the output In this study the streams are characterised by a number between 1 and 4. Table~\ref{Table6:Correlation} shows the absolute Person Correlation coefficient obtained across all sequences, bitrates and \acs{plr}. The table shows the results for the parameters previously described, as well as, a new parameter obtained from the average magnitude of the optical flow measured using the method in~\cite{DeepFlow2013} (\textit{OPT\_FLOW}). -Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable. +Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable. Since both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method, due to its higher Person Correlation coefficient. @@ -285,9 +285,9 @@ However, different network topologies with different number of neurons in each l %As mentioned before a total of \NumCases test cases are used for the evaluation of the proposed method. To effectively train and validate the \ac{nn}, firstly the cases corresponding to three video sequence were left out, to be used in the performance evaluation of the proposed method. The remaining data set was randomly partitioned into three different groups: (i) 65\% of the test cases are used for training, (ii) 15\% for validation and (iii) 20\% for testing the \ac{nn} performance. -The \ac{nn} supervised training uses 557 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993}. +The \ac{nn} supervised training uses 557 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993}. Then, 129 test conditions were used for \ac{nn} validation. -Finally, 172 conditions are used to evaluate the accuracy of the proposed method after training with different conditions. +Finally, 172 conditions are used to evaluate the accuracy of the proposed method after training with different conditions. Tables~\ref{Table6:ConfusionTraining}, \ref{Table6:ConfusionVal} and \ref{Table6:ConfusionTest} show the confusion matrices obtained for training, validation and test sets, respectively. In these matrix each column corresponds to the target result, i.e., the cases where the coded streams, \SMEC to \SMEXPELTWEN, would lead to the highest decoded video quality. @@ -311,14 +311,14 @@ These results correspond to the dataset used to train the \ac{nn}. \cline{2-6} & \SMEXP & 16 & 57 & 13 & 0 \\ \cline{2-6} & \SMEXPELTEN & 4 & 10 & 69 & 18 \\ \cline{2-6} & \SMEXPELTWEN & 9 & 10 & 17 & 158 \\ - \hline + \hline \multicolumn{2}{x{1.7cm}|}{Accuracy} & 82.0\% & 52.3\% & 65.7\% & 86.8\% \\ \end{tabular} \end{table} Results in Table~\ref{Table6:ConfusionTraining} shows that streams \SMEC (i.e., without robust coding techniques) and \SMEXPELTWEN (i.e., using optimal \ac{ec} with 20\% of bitrate in the \ac{el}) lead to the highest quality in the majority of the test cases (are more often selected). -The \ac{nn} is able to achieve a high accuracy for \SMEC and \SMEXPELTWEN streams, correctly selecting the output stream in more than 80\% of the cases. Regarding the remaining robust streams (i.e., \SMEXP and \SMEXPELTEN), an acceptable accuracy of 52.2\% and 60.6\% is achieved, respectively. -This reveals that the \ac{nn} used in the proposed method is able to predict the coded stream which leads to the highest video quality across different conditions with acceptable accuracy. +The \ac{nn} is able to achieve a high accuracy for \SMEC and \SMEXPELTWEN streams, correctly selecting the output stream in more than 80\% of the cases. Regarding the remaining robust streams (i.e., \SMEXP and \SMEXPELTEN), an acceptable accuracy of 52.2\% and 60.6\% is achieved, respectively. +This reveals that the \ac{nn} used in the proposed method is able to predict the coded stream which leads to the highest video quality across different conditions with acceptable accuracy. The results from the validation set, shown in Table~\ref{Table6:ConfusionVal}, reveal that the \ac{nn} training did not over-fit, since it shows similar accuracy as the results shown in Table~\ref{Table6:ConfusionTraining}. @@ -366,7 +366,7 @@ The results from the validation set, shown in Table~\ref{Table6:ConfusionVal}, r To further evaluate the performance of the \ac{nn}, the confusion matrix of the data set used for testing (test set) is shown in the Table~\ref{Table6:ConfusionTest}. This table shows the same results as Table~\ref{Table6:ConfusionTraining}, but for different test cases, which were not used in the training phase. % Therefore, the trained network is not aware of the test cases used for validation. -Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 50\% with a maximum value of 84.9\% when the \SMEXPELTWEN is the target stream. +Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 50\% with a maximum value of 84.9\% when the \SMEXPELTWEN is the target stream. Although a lower accuracy is obtained using the test set, the overall accuracy is still acceptable, since it allows to select the optimal robust coding technique (i.e., \SMEC to \SMEXPELTWEN) for the majority of the conditions. In summary, the results presented in this sub-section reveal that the input parameters used in the proposed \ac{nn} are correlated with the decoded video quality, as they can effectively be used to perform data categorisation used in the selection of the optimal error robustness coding technique for the \ac{hevc} standard. @@ -433,22 +433,22 @@ Tennis & +1.18 & +2.11 & +2.43 & +3.28 & +3.28 \\ \hline Subsequently, a comparison between the different methods is performed, in order to evaluate the quality gains obtained with the proposed method in comparison with each of the different techniques. %for every test case. Table~\ref{Table6:AverageGains} shows the average quality gains in comparison with the reference \ac{hevc} for the test set. -These were obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams \SMEC to \SMEXPELTWEN. +These were obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams \SMEC to \SMEXPELTWEN. The average quality gains are measured across all test cases, i.e., test set with different bitrates and different packet loss ratios. -Results in Table~\ref{Table6:AverageGains} show that the robust coded streams \SMEC to \SMEXPELTWEN are able to achieve acceptable average quality gains up to 1.50 dB (see results for \SMEXPELTEN). -However, since the implementation of robust coding techniques decreases the coding efficiency, it also leads to lower quality than the reference \ac{hevc}. +Results in Table~\ref{Table6:AverageGains} show that the robust coded streams \SMEC to \SMEXPELTWEN are able to achieve acceptable average quality gains up to 1.52 dB (see results for \SMEXPELTEN). +However, since the implementation of robust coding techniques decreases the coding efficiency, it also leads to lower quality than the reference \ac{hevc}. This is noticeable in the results obtained for the Four People sequence in the streams \SMEXP to \SMEXPELTWEN and for the Kristen and Sara sequence in the streams \SMEXPELTEN and \SMEXPELTWEN (see red ellipses). The overall performance of such methods reveals that they are not the best options for transmission conditions with low \acs{plr} (higher number of conditions have \acs{plr} lower than 3\%). On the contrary, the proposed method is always capable of outperforming the reference \ac{hevc} and achieve quality gains from 0.28 dB to 3.28 dB. -For the Kimono and Tennis sequences, it is able to achieve the highest quality gains, of up 3.07 dB and 3.28 dB, respectively (see blue ellipses). -Summarising, by adaptively selecting the video stream, and implicitly the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness, achieving an average quality gain across different bitrates and \acs{plr} of 1.88 dB. +For the Kimono and Tennis sequences, it is able to achieve the highest quality gains, of up 3.07 dB and 3.28 dB, respectively (see blue ellipses). +Summarising, by adaptively selecting the video stream, and implicitly the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness, achieving an average quality gain across different bitrates and \acs{plr} of 1.69 dB. Finally, the performance of the proposed method is evaluated for video sequences, which was not fed into the \ac{nn} training phase. -This study aims at finding the accuracy of the optimal stream selection in case of a new video content. +This study aims at finding the accuracy of the optimal stream selection in case of a new video content. For this purpose the Basketball Drive sequence was randomly selected, and not used in previous simulations. Table~\ref{Table6:AverageGainBasketBallDrive} shows the average quality results obtained with these sequences, similarly to Table~\ref{Table6:AverageGains}. -These results also confirm that the proposed method is able to select the optimal robust stream to be used for transmission, even when new video content is transmitted. +These results also confirm that the proposed method is able to select the optimal robust stream to be used for transmission, even when new video content is transmitted. The proposed method achieves a quality gain up to 4.46 dB in comparison with the reference \ac{hevc}, outperforming the quality achieved with the stream \SMEXPELTWEN (i.e., 4.19 dB). This confirms the advantage of the proposed method to adaptively select among different streams with different levels of error protections.