From ba6b505983f727ec9a260218595d3d8bcec8fe06 Mon Sep 17 00:00:00 2001 From: Joao Carreira Date: Fri, 24 Aug 2018 16:05:49 +0100 Subject: [PATCH] Commit chapter 7 --- .../EndToEndRobustSystem.tex | 342 +++++++---- .../Figures/NeuralNetwork.tikz | 4 +- .../Results/PlotQualty_vs_PLR_Predicted.tex | 276 +++++++++ .../Results/Quality_PLR_Methods.tex | 2 +- .../Results/pyPlotQuality.bash | 569 ------------------ 5 files changed, 495 insertions(+), 698 deletions(-) create mode 100644 043_EndToEndRobustSystem/Results/PlotQualty_vs_PLR_Predicted.tex delete mode 100644 043_EndToEndRobustSystem/Results/pyPlotQuality.bash diff --git a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex index dd7f9b0..b8e5cc2 100644 --- a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex +++ b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex @@ -1,7 +1,7 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % CHAPTER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\chapter{\HIGHLIGHT{Error Robustness Optimisation for Highly Efficient Video Coding}} +\chapter{Adaptive Error Robustness for Highly Efficient Video Coding} \label{Chapter:EndToEndRobustSystem} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -9,12 +9,15 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Introduction} +The effectiveness of the error resilience techniques varies depending on the video content, as well as, the transmitted bitrate and network conditions. +Therefore, it is not straightforward to select a single technique which achieves high efficiency for every scenario. +Thus, in this chapter a novel adaptive error robustness method is proposed in order to optimise the techniques previously described in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}. +At the encoder-side different video streams are generated with different levels compression efficiency and error robustness. Then, a pre-training \ac{nn} is used to predict which video stream will achieve the highest decoded video quality for a particular scenario. The selection is based on different input parameters which include information from the video signal, the coded stream and the transmission network. Using this approach the overall performance of the proposed method significantly increases by achieving a better trade-off between compression efficient and error robustness. -In this chapter the error resilience and error concealment approaches proposed in the previous chapters, namely Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, are combined to achieved a higher decoder video quality across different network conditions and applications. -This chapter presents a combined evaluation of different approaches and proposes a method to efficiently select the optimal one in terms of reconstructed quality. - -This chapter is organised as follows. Section~\ref{Section6:Problem} presents the problem formulation and motivation for the method proposed in this chapter. \HIGHLIGHT{...} -Section~\ref{Section6:Summary} concludes this chapter. +This chapter is organised as follows. +Section~\ref{Section6:Problem} presents the problem formulation and preliminary results which serve as motivation for the method proposed in this chapter. +Section~\ref{Section6:ProposedMethod} provides an overview of the proposed method architecture and Section~\ref{Section6:OptimisationMethod} describes the method in detail. +Section~\ref{Section6:ExperimentalResults} shows the experimental evaluation of the proposed approach. Finally, Section~\ref{Section6:Summary} concludes this chapter. \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -23,27 +26,28 @@ Section~\ref{Section6:Summary} concludes this chapter. \section{Problem formulation} \label{Section6:Problem} -In this research work different robust coding techniques were proposed to increase the overall performance of video transmission under error-prone conditions. The methods proposed used different algorithms, as reference picture selection, motion vector redundancies and error concealment signalling. In previous chapters theses methods were evaluated against existing techniques revealing their superior performance and efficiency to deal with error robustness problems in high efficient video coding, e.g., using the \ac{hevc} standard. +In this research work different robust coding techniques were proposed to increase the overall performance of video transmission under error-prone conditions. +Difference algorithms are used in the proposed methods, such as, reference frame selection, motion vector redundancies and error concealment signalling. +In previous chapters, these methods were evaluated against existing techniques revealing their superior performance and efficiency to deal with error robustness problems in high efficient video coding. +The evaluation of these methods revealed that they can efficiently increase the robustness against transmission errors and losses, leading to higher decoded video quality. Results in Chapter~\ref{Chapter:ErrorResilienceTechniques} show that the proposed reference frame selection algorithm introduces low overhead in comparison with existing techniques. However, the method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} achieves higher quality gains at cost of slightly higher bitrate increase. +Therefore, one may conclude that each method can be optimised for different applications, video characteristics and network conditions. -The evaluation of these methods revealed that they can efficiently increase the robustness to transmission errors and losses, leading to higher decoded video quality. Results in Chapter~\ref{Chapter:ErrorResilienceTechniques} show that the proposed reference picture selection algorithm introduces low overhead in comparison with existing techniques. However, the method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} achieves higher quality gains at cost of slightly higher bitrate increase. -Therefore, it is noticeable that each method might be optimised for different applications, video characteristics and network conditions. - -To advance one step forward, a jointly robust coding method using different techniques can be used to increase the decoder video quality across difference conditions. -Such approach can be based on both the input contents and the network information, such as bitrate and packet loss conditions to optimise the robust coders to each application scenario. In order to overcome the above problem, a joint evaluation study is necessary to identify the most efficient techniques across different conditions. +To advance one step forward, a joint robust coding method using different techniques can be used to increase the decoder video quality across difference conditions, by taking into consideration the input contents, the network information to optimise the robust coder to each scenario. +%In order to overcome the above problem, a joint evaluation study is necessary to identify the most efficient techniques across different conditions. \subsection{Comparison of the proposed methods} -In order to develop a joint robust coder to cope with transmission errors an evaluation study was carried out, to compare the error robustness performance of the proposed methods for different conditions with each others. -%This study extends the previous evaluations presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, as they target a wider range of test material and network conditions. -The experimental evaluation carried out in this section follows the same condition described in Section~\ref{Section3:ErrorResilienceEvaluation}. +In order to develop a joint robust coder to cope with transmission errors a comparison study was carried out, to compare the error robustness performance of the proposed methods for different conditions. +%This study extends the previous evaluations presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, as they target a wider range of test material and network conditions. +The experimental evaluation carried out in this section follows the same condition described in Section~\ref{Section3:ErrorResilienceEvaluation}. %In order to perform a wider evaluation different bitrate and \acs{plr} were used. Table~\ref{Table6:TestVariables} illustrates the selected configurations. As shown in the table a total of 15 test sequences were used encoder using 5 different target bitrates. One should note that the main bitrate varies based on the test sequence and it is shown in third column of Table~\ref{TableTestSequences}. Different network conditions were evaluated using a total of 13 different \acs{plr}, with higher special focus on lower loss ratios, which are more relevant to modern video streaming applications. -In this experimental evaluation the following methods are evaluated, namely: +The following methods are evaluated, namely: \begin{itemize} \item the default \ac{hevc} decoder with frame-copy \ac{ec} (\REF), as reference method; - \item the proposed reference picture selection scheme (\EXP); - \item the \EXP method in combination with an \ac{ec} optimisation with 10\% of bitrate of the scalable enhancement layer (\EXPELTEN); - \item the same as above with 20\% of bitrate for the scalable enhancement layer (\EXPELTWEN). + \item the proposed reference frame selection scheme (\EXP); + \item the \EXP method in combination with an \ac{ec} optimisation with 10\% of bitrate on the scalable enhancement layer (\EXPELTEN); + \item the same as above, with 20\% of bitrate on the scalable enhancement layer (\EXPELTWEN). \end{itemize} @@ -57,7 +61,8 @@ In this experimental evaluation the following methods are evaluated, namely: \subsubsection{Experimental results and discussion} -Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} shows the average decoded video quality obtained from 50 different trials, measured with the \ac{psnr} quality metric. The figures shows the quality obtained for different loss conditions. Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate shown in the third column of Table~\ref{TableTestSequences}. Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained using a lower amount of coded bits, corresponding to a 40\% reduction of the main target bitrate. This aims at finding the impact of the video and network bitrate on the performance of the proposed methods. +Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} shows the average decoded video quality obtained from 50 trials, measured with the \ac{psnr} quality metric, for different loss conditions. +Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate listed in the third column of Table~\ref{TableTestSequences}. Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained using a lower amount of coded bits, corresponding to a 40\% reduction of the main target bitrate. Comparing both results aims at finding the impact of the video signal and network bitrate on the performance of the proposed methods. \begin{figure}[!t] \centering @@ -67,10 +72,10 @@ Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReductio \label{Figure6:AverageQualityReduction} \end{figure} -Results on both figures show that the proposed methods (\EXP, \EXPELTEN and \EXPELTWEN) are all able to outperform the reference \ac{hevc} for high \acs{plr}. In case of the proposed reference picture selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions shown. -Comparing the results for both target bitrates also shows that the proposed methods are able to achieve higher quality gains for higher bitrates. Therefore, the for a lower bitrate (see results of Figure~\ref{Figure6:AverageQualityReduction}) the proposed methods only outperform the reference method for slightly higher \acs{plr}. This is more noticeable in the \EXPELTWEN case. +Results of both figures show that the proposed methods (\EXP, \EXPELTEN and \EXPELTWEN) are all able to outperform the reference \ac{hevc} for high \acs{plr}. In case of the proposed reference frame selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions. +Comparing the results for both target bitrates also shows that the proposed methods are able to achieve higher quality gains for higher bitrates. Therefore, the for a lower bitrate (see results of Figure~\ref{Figure6:AverageQualityReduction}) the proposed methods only outperform the reference case for slightly higher \acs{plr}. This is more noticeable in the \EXPELTWEN case. -Summarising, the performance of the proposed methods is not similar across different video content and target bitrate, which impose a challenge on which method should be applied to achieve the highest decoded video quality in case of transmission errors. In order to achieve the highest quality the optimal trade-off between auxiliary information and video bitrate should be obtained by selecting the optimal error robust coding algorithm. +Summarising, the performance of the proposed methods is not similar across different video content and target bitrate, which impose a challenge on which method should be applied to achieve the highest decoded video quality in case of transmission errors. In order to achieve increased efficiency and higher video quality, the optimal trade-off between auxiliary information and video bitrate should be obtained by selecting the optimal robust coding algorithm. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% @@ -78,10 +83,11 @@ Summarising, the performance of the proposed methods is not similar across diffe \section{Proposed architecture} \label{Section6:ProposedMethod} -This section provides an overview of the proposed method, which aims at obtaining the optimal error robustness method for different streaming conditions, i.e., video content and network conditions. -The main advantage of using the proposed algorithm is that it leads to an optimal trade-off between error robustness tools and coding efficiency. -This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and are selected based the current conditions, i.e., video signal and network conditions. This architecture is suitable for dynamic streaming, e.g., HTTP streaming based on MPEG-DASH~\cite{HEVCHTTPStreamAug2016} or content delivery over heterogeneous networks using the \ac{mmt}~\cite{ATSCMMT2016}. -In order to accomplish that, at the encoder-side, the input signals are coded using different levels of error protections. Subsequently, an optimisation algorithm is used to decide which bitstream should be transmitted at any given instant, based of different parameters. The selected stream is expected to obtain the highest quality at the decoder-side for the given conditions. +This section provides an overview of the proposed method, which aims at obtaining the optimal error robustness method for different streaming scenarios, i.e., video content and network conditions. +The main advantage of using the proposed algorithm is that it leads to an optimal trade-off between error robustness tools and coding efficiency. +This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and can selected based on the current conditions. +This architecture is suitable for dynamic streaming technologies, e.g., HTTP streaming based on MPEG-DASH~\cite{HEVCHTTPStreamAug2016} or content delivery over heterogeneous networks using the \ac{mmt}~\cite{ATSCMMT2016}. +In the proposed method, firstly the input signals are coded using different levels of error protections. Subsequently, an optimisation algorithm is used to decide which bitstream should be transmitted at any given instant, based on different parameters. The selected stream is expected to obtain the highest quality at the decoder-side for the given conditions. \begin{figure}[tb] \centering @@ -92,21 +98,20 @@ In order to accomplish that, at the encoder-side, the input signals are coded us \FIGSVSPACE \end{figure} -Figure~\ref{FIG6:BlockDiagram} illustrates the architecture of the proposed encoding scheme. -Firstly, a reference video coder is used to compress the bitstream using highly efficient coding techniques in order to obtain the best \ac{rd} performance for error-free environments. +Figure~\ref{FIG6:BlockDiagram} illustrates the architecture of the proposed encoding scheme. +Firstly, a reference video coder is used to compress the bitstream using highly efficient coding techniques in order to obtain the best \ac{rd} performance for error-free environments. This is used as the reference bitstream in the proposed approach. -Subsequently, a transcoding operation based on the reference bitstream and the input video is used to efficiently select the reference pictures used for inter-frame prediction in terms of error propagation. -The reference picture selection is applied as proposed in Chapter~\ref{Chapter:ErrorResilienceTechniques}. -Moreover, a loss simulation and a saliency-based \ac{ec} optimisation is also applied on the robust bitstream. This is used to select the optimal \ac{ec} algorithm which should be selected at the decoder-side. -Since the optimal \ac{ec} mode might not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el} using two different bitrates, i.e., with 10\% and 20\% of the total bitrate. This provides two different levels of error robustness. The \ac{ec} optimisation and \ac{el} coding is performed as proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques}. +Subsequently, a transcoding operation is applied to the reference bitstream aided by the input video signal to efficiently select the reference frames used for inter prediction. The reference frame selection is applied as proposed in Chapter~\ref{Chapter:ErrorResilienceTechniques}. +Moreover, loss simulation and a saliency-based \ac{ec} optimisation is applied on the robust bitstream. This is used to select the optimal \ac{ec} algorithm which should be applied at the decoder. +Finally, since the optimal \ac{ec} mode might not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el} using two different bitrates, i.e., 10\% and 20\% of the total bitrate, providing two different levels of error robustness. +The \ac{ec} optimisation and \ac{el} coding is performed as proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques}. +The proposed approach results in four different bitstreams with different characteristics: (i) reference stream without error robust coding techniques; (ii) robust video coding using a reference frame selection algorithm which leads to lower error propagation in case of data-loss; (iii) robust video coding using selective reference frames and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue; (iv) same as the previous bitstream but with 20\% bitrate allocated for the scalable \ac{el}. These bitstreams are referred to as "S1" to "S4", respectively, as shown in Figure~\ref{FIG6:BlockDiagram}. -The proposed approach results in four different bitstreams: (i) reference stream without error robustness; (ii) robust video coding using a reference picture selection algorithm which leads to lower error propagation in case of data-loss; (iii) robust video coding and decoding using selective reference pictures and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue; (iv) same as the previous bitstream but with 20\% of the bitrate allocated for the scalable \ac{el}. These bitstreams are referred to as "S1" to "S4", respectively, as shown in Figure~\ref{FIG6:BlockDiagram}. - -Finally, the video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and from the transmission channel. -From the input video signal relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, informations is also extracted from the coded bitstream using a stream parser. -%the predictions used by the video encoder as well as the target bitrate are extracted using a stream parser. +Finally, the video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and from the transmission channel. +From the input video signal relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, further information is also extracted from the coded bitstream using a stream parser. +%the predictions used by the video encoder as well as the target bitrate are extracted using a stream parser. Finally, a parameter regarding the transmission channel is also used to predict the data loss. -These parameters are fed into the error robustness optimisation block to select the suitable bitstream which should be transmitted to obtain the highest video quality. The optimisation process is described in detail in Section~\ref{Section6:OptimisationMethod}. +These parameters are fed into the error robustness optimisation block to select the suitable bitstream which should be transmitted to obtain the highest video quality. The optimisation process is described in detail in the following section. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -115,56 +120,85 @@ These parameters are fed into the error robustness optimisation block to select \section{Error robustness selection method} \label{Section6:OptimisationMethod} -In this section the optimisation method used to select a robust highly efficiently coded stream is presented. -As mentioned before, the proposed scheme aims at predict the best method which should be used at the encoder-side, that achieves the highest quality. The selection method is based on a shallow neural network which predicts the optimal stream based on different parameters. Moreover, in this section a description of the input parameters obtained from the stream parser and the video analyser is also described. +In this section the optimisation method used to select a robust highly efficiently coded stream is presented. +The proposed scheme aims at selecting the best method which should be used for transmission, that achieves the highest quality in error-prone scenarios. +The selection method is based on a shallow neural network which predicts the optimal stream based on different parameters. +This section describes of both the input parameters and the neural network used in the proposed scheme. \subsection{Input parameters} \glsreset{si} \glsreset{ti} -A set of different parameters are used to predict the optimal approach that achieves the higher quality in specific network conditions. Based on these inputs the proposed scheme is able to select the appropriate robust coding scheme to be used. These parameters aim at covering three different aspects of the video transmission: two parameters from the input video, three parameters from the video stream and two parameters from the transmission network. +A set of different parameters are used to predict the optimal approach that achieves the highest quality in specific network conditions. Based on these inputs the proposed scheme is able to select the appropriate robust coding scheme which should be used. These parameters aim at covering three different aspects of the video transmission: two parameters from the input video, three parameters from the video stream and one parameter from the transmission network. -To characterise the input video signal the following parameters are used: \ac{si} and \ac{ti}~\cite{ITUTP9112008}. -These parameters are obtained as described in Chapter~\ref{Chapter:HEVCErrorResilience}, and aim at evaluating the spatial complexity of the video scenes and the motion complexity, which have impact on the error robustness of the coded streams. +To characterise the input video signal the following parameters are used: \ac{si} and \ac{ti}~\cite{ITUTP9112008}. +These parameters are obtained as described in Chapter~\ref{Chapter:HEVCErrorResilience}, and aim at evaluating the spatial complexity and motion complexity of the video scenes, which have relevant impact on the error robustness of the coded streams. As shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, test material with higher \ac{si} or \ac{ti} suffer from higher quality degradation as the \ac{plr} increases, e.g., Book Arrival and Tennis sequences. -From the coded video streams complementary information to characterise the streaming content is also extracted. A stream parser is applied to the reference bitstream without error robustness techniques. Therefore, the information extracted regarding the prediction information is related to a stream coded based on an optimal \ac{rd} optimisation which was not affected by robust coding techniques. The following information is extracted: +From the coded video streams complementary information to characterise the streaming content is also extracted. A stream parser is applied to the reference bitstream without error robustness techniques. Therefore, the prediction information extracted is related to a stream coded based on optimal \ac{rd} optimisation which was not affected by robust coding techniques. The following is extracted: \begin{itemize} -\item \textit{Inter(\%)}: the relation between ration of the image which is coded based on intra and inter frame predictions. This is expressed by the ration of blocks coded using inter frame predictions. -The number of inter-coded blocks indicates the distribution of spatial and temporal dependencies. -Normally, coded streams with higher number of temporal dependencies are more prone to error propagation, thus higher number of bits should be allocated for error protection \HIGHLIGHT{[XX]}. +\item \textit{Inter(\%)}: the relation between the ratio of the image which is coded based on intra and inter frame predictions, indication the distribution of spatial and temporal dependencies. +This is expressed by the ratio of blocks coded using inter frame predictions. +Normally, coded streams with higher number of temporal dependencies are more prone to error propagation, thus higher number of bits should be allocated for error protection \cite{UEPScalable_TCSVT2010}. -\item \textit{AVG\_MV}: from those regions coded using inter frame predictions the motion information is analysed and the average \ac{mv} size is used to provide information regarding the motion intensity. Motion based \ac{ec} algorithms are less efficient \HIGHLIGHT{[XX]} for video signals with higher motion intensity as \acs{mv} are less predictable using algorithms such as \ac{mc} and \ac{mve}. +\item \textit{AVG\_MV}: from those regions coded using inter frame prediction, the motion information is analysed and the average \ac{mv} size is used to provide information regarding the motion intensity. As shown in previous studies, motion based \ac{ec} algorithms are less efficient for video signals with higher motion intensity as \acs{mv} are less predictable using algorithms such as \ac{mc} or \ac{mve} \cite{HybridConcealmentTB2008}. -\item \textit{Bitrate}: information regarding the bit allocation is extracted, since it will affect the video quality and the amount of bits that can be allocated for robust coding algorithms and auxiliary information. +\item \textit{Bitrate}: information regarding the bit allocation is extracted, since it will affect the video quality and the amount of bits that can be available for auxiliary information. \end{itemize} Finally, since the amount of data loss affects the decoded video quality the transmission network \ac{plr} is also used as input parameter. Transmission networks with higher loss ratios require a higher bit allocation for robust techniques, therefore the error resilience technique should adapt based on the value of the \ac{plr}. +% Podia se citar este paper +% \cite{IntraRefreshICIP2009} \subsection{Neural network} -Neural Networks are a computational model based on the structure and functions of biological neural networks. -In terms of computer science, a neural network works artificial human nervous system for receiving, processing, and transmitting information between neurons. Mainly, a neural network consists of three different layers: the input layer which receives data from an external environment, the hidden layer which process the inputs and an output layer~\cite{NeuralNetworks1994}. -In the proposed method a feed-forward network is used which connects every neuron a given layer to the neurons of the subsequent layer. +\glsreset{nn} \begin{figure}[tb] -\centering -%\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram} -\input{043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz} -\caption{Architecture of the proposed scheme.} -\label{FIG6:BlockDiagram} -\FIGSVSPACE + \centering + %\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram} + \input{043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz} + \caption{Representation of the neural network used in the proposed method.} + \label{FIG6:NeuralNetwork} + \FIGSVSPACE \end{figure} -Figure~\ref{FIG6:BlockDiagram} shows the diagram of the neural network used in the proposed method. \HERE -In the proposed method the input layer receives the six input described above which are related with the output video quality which is the optimisation variable. Subsequently, the neural network transforms the input data through a series of neurons in two hidden layers. +A \ac{nn} is a computational model based on the structure and functions of biological neural networks. +%In terms of computer science, \acs{nn} work similarly to a human nervous system for receiving, processing, and transmitting information between neurons. +Mainly, a \ac{nn} consists of three different layers: the input layer which receives data from an external environment, the hidden layer which process the inputs and an output layer~\cite{NeuralNetworks1994}. +In the past \acs{nn} have been widely used in video processing applications such as, quality evaluation~\cite{VideoQualityNN_TNN2006}, video analysis~\cite{VideoAnalysisNN_2018} and video coding~\cite{RateDistortionNN_VCIP2017,HEVCComplexityNN_TIP2018}. +In the proposed method a feed-forward network is used which connects every neuron in a given layer to the neurons of the subsequent layer. + +Figure~\ref{FIG6:NeuralNetwork} shows the diagram of the \ac{nn} used in the proposed method. +The network contains and input and output layer, as well as, a single hidden layer. +The input layer receives the six inputs, as described in the previous sub-section. These inputs are correlated with the optimisation variable being maximised, i.e., the output video quality. +Subsequently, the network transforms the input data through a series of neurons in the hidden layer. +The output layer comprises four neurons corresponding to each available video stream from S1 to S4 (different error resilience techniques). +Finally, the output with the highest value is selected and the corresponding stream is used for transmission, since it is the one expected to achieve the highest video quality for a particular scenario. + +In the proposed approach a shallow \ac{nn} is used for simplicity and to avoid over-fitting due to the limited test cases. +By avoiding over-fitting, it is guaranteed that the proposed method would continue to work in future test conditions not used for the network training. +However, different network topologies with different number of neurons in each layer could also be used to predict between different error resilience techniques, as in the proposed method. -\section{Performance evaluation} +\section{Experimental evaluation} +\label{Section6:ExperimentalResults} +\def\NumCases{910\xspace} +In this section the experimental results obtained with the proposed method are presented and discussed. +The error resilience techniques used in the proposed method are used in the context of the \ac{hevc} standard and compared with the reference implementation (\REF). +%The experimental results were obtained with the methodology described in Section~\ref{ExperimentalTestSetup}. +To demonstrate the effectiveness of the proposed optimal error robustness scheme several test cases were used. +In this experimental evaluation 14 test sequences were used, encoded using at 5 different bitrates. +Moreover, the following \acs{plr} were used to cover a wide range of loss rations giving more detail to low \acs{plr}: 0\%, 0.5\%, 1.0\%, 1.2\%, 1.6\%, 2.0\%, 2.2\%, 2.4\%, 2.6\%, 2.8\%, 3.0\%, 5.0\% and 10.0\%. +This results in a total of \NumCases test conditions. + +The performance evaluation of the proposed method is carried out using two different approaches. +In the former, a description of the \ac{nn} training is presented along with the validation results. +In the latter, the quality obtained with the proposed optimal error robustness scheme is evaluated and compared with the reference methods. %\begin{table}[t] % \centering @@ -175,102 +209,158 @@ In the proposed method the input layer receives the six input described above wh % \toprule % \textbf{Sequence} & \textbf{Bitrate} & \textbf{\ac{plr}} \\ \midrule % Basketball Drill & Main (see Table~\ref{TableTestSequences}) & Error-free \\ -% Book Arrival & 10\% reduction & 0.5\% \\ -% Bosphorus & 20\% reduction & 1.0\% \\ +% Basketball Drive & 10\% reduction & 0.5\% \\ +% Book Arrival & 20\% reduction & 1.0\% \\ % BQSquare & 30\% reduction & 1.2\% \\ % Cactus & 40\% reduction & 1.6\% \\ % Four People & 50\% reduction & 2.0\% \\ -% Jockey & & 2.2\% \\ -% Kendo & & 2.4\% \\ -% Kimono & & 2.6\% \\ -% Kristen and Sara & & 2.8\% \\ -% Park Scene & & 3.0\% \\ -% People on Street & & 5.0\% \\ -% Race Horses & & 10.0\% \\ -% Tennis & & \\ +% Kendo & & 2.2\% \\ +% Kimono & & 2.4\% \\ +% Kristen and Sara & & 2.6\% \\ +% Park Scene & & 2.8\% \\ +% People on Street & & 3.0\% \\ +% Race Horses & & 5.0\% \\ +% Tennis & & 10.0\% \\ % Traffic & & \\ % \bottomrule % \end{tabular} %\end{table} -\subsection{Selection Accuracy} +\subsection{Network training and validation} + +%As mentioned before a total of \NumCases test cases are used for the evaluation of the proposed method. +To effectively train and validate the \ac{nn} used in the proposed method the data set is randomly divided based on the following: 70\% of the test cases are used for training and 30\% for validation. +The \ac{nn} supervised training uses 637 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993}. +The remaining 273 conditions are used to evaluate the accuracy of the proposed method after training using different conditions. -\begin{table}[t] -\centering -\caption{Confusion matrix of the training and validation set.} -\label{Table6:ConfusionTraining} -\TABLESVSPACE -\renewcommand{\arraystretch}{1.2} -\begin{tabular}{x{.5cm}x{1cm}|x{1.1cm}|x{1.1cm}|x{1.1cm}|x{1.1cm}|} -& & \multicolumn{4}{c}{Optimal} \\ -\multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} -& & S1 & S2 & S3 & S4 \\ \hline - & S1 & 122 & 17 & 3 & 0 \\ -\cline{2-6} & S2 & 13 & 80 & 19 & 2 \\ -\cline{2-6} & S3 & 0 & 12 & 96 & 9 \\ -\cline{2-6} & S4 & 0 & 4 & 11 & 202 \\ -\hline -\multicolumn{2}{x{1.7cm}|}{Accuracy} & 90.3\% & 70.8\% & 74.4\% & 94.8\% \\ -\end{tabular} +\begin{table}[tb] + \centering + \caption{Confusion matrix of the training test cases.} + \label{Table6:ConfusionTraining} + \TABLESVSPACE + \renewcommand{\arraystretch}{1.2} + \begin{tabular}{x{.5cm}x{1cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}} + & & \multicolumn{4}{c}{Target} \\ + \multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} + & & S1 & S2 & S3 & S4 \\ \hline + & S1 & 122 & 17 & 3 & 0 \\ + \cline{2-6} & S2 & 13 & 80 & 19 & 2 \\ + \cline{2-6} & S3 & 0 & 12 & 96 & 9 \\ + \cline{2-6} & S4 & 0 & 4 & 11 & 202 \\ + \hline + \multicolumn{2}{x{1.7cm}|}{Accuracy} & 90.3\% & 70.8\% & 74.4\% & 94.8\% \\ + \end{tabular} \end{table} +Table~\ref{Table6:ConfusionTraining} shows the confusion matrix of the \ac{nn} training phase. +In this matrix each column corresponds to the target result, i.e., the cases where the coded streams S1 to S4 would lead to the highest decoded video quality, respectively. +Similarly, each row corresponds to the \ac{nn} selected by the trained networks. +Thus, those cases where the horizontal and vertical labels match, indicate that the network produced the correct output. The remaining positions correspond to the erroneous outputs. +The last line shows the accuracy for each target output. +These results correspond to the dataset used to train the \ac{nn}. -\begin{table}[t] -\centering -\caption{Confusion matrix of the validation test set.} -\label{Table6:ConfusionTest} -\TABLESVSPACE -\renewcommand{\arraystretch}{1.2} -\begin{tabular}{x{.5cm}x{1cm}|x{1.1cm}|x{1.1cm}|x{1.1cm}|x{1.1cm}|} - & & \multicolumn{4}{c}{Optimal} \\ -\multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} -& & S1 & S2 & S3 & S4 \\ \hline - & S1 & 56 & 7 & 2 & 0 \\ -\cline{2-6} & S2 & 10 & 28 & 6 & 2 \\ -\cline{2-6} & S3 & 1 & 5 & 31 & 8 \\ -\cline{2-6} & S4 & 0 & 5 & 7 & 86 \\ -\hline - \multicolumn{2}{x{1.7cm}|}{Accuracy} & 83.6\% & 62.2\% & 67.4\% & 89.6\% \\ -%\bottomrule -\end{tabular} +Results in Table~\ref{Table6:ConfusionTraining} shows that streams S1 (i.e., without robust coding techniques) and S4 (i.e., using optimal \ac{ec} with 20\% of bitrate in the \ac{el}) lead to the highest quality in the majority of the test cases (are selected more often). +The \ac{nn} is able to achieve a high accuracy for S1 and S4 streams, correctly selecting the output stream in more than 90\% of cases. Regarding the remaining robust streams (i.e., S2 and S3) an acceptable accuracy of 70.8\% and 74.4\% is achieved, respectively. This reveals that the \ac{nn} used in the proposed method is able to correctly predict the coded stream which leads to the highest video quality across different conditions. + +\begin{table}[tb] + \centering + \caption{Confusion matrix of the validation test cases.} + \label{Table6:ConfusionTest} + \TABLESVSPACE + \renewcommand{\arraystretch}{1.2} + \begin{tabular}{x{.5cm}x{1cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}} + & & \multicolumn{4}{c}{Target} \\ + \multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} + & & S1 & S2 & S3 & S4 \\ \hline + & S1 & 56 & 7 & 2 & 0 \\ + \cline{2-6} & S2 & 10 & 28 & 6 & 2 \\ + \cline{2-6} & S3 & 1 & 5 & 31 & 8 \\ + \cline{2-6} & S4 & 0 & 5 & 7 & 86 \\ + \hline + \multicolumn{2}{x{1.7cm}|}{Accuracy} & 83.6\% & 62.2\% & 67.4\% & 89.6\% \\ + %\bottomrule + \end{tabular} \end{table} +To further evaluate the performance of the \ac{nn}, the confusion matrix of the validation set is shown in the Table~\ref{Table6:ConfusionTest}. This table shows the same results as Table~\ref{Table6:ConfusionTraining} but for different test cases, which were not used in the training phase. +% Therefore, the trained network is not aware of the test cases used for validation. +Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 62.2\% with a maximum value of 89.6\% when the S4 is the target stream. Although a lower accuracy is obtained using the validation test cases, the overall accuracy is still acceptable, since it allows to select the optimal robust coding technique (i.e., S1 to S4) for the majority of the conditions. +In summary, the results presented in this sub-section reveal that the input parameters used in the proposed \ac{nn} are correlated with the decoded video quality, as they can effectively be used to perform data categorisation used in the selection of the optimal error robustness coding technique for the \ac{hevc} standard. + + + + \subsection{Quality evaluation} +The overall goal of the proposed method is to achieve improved quality using a combination of different robust coding techniques adaptively selected using a feed-forward \ac{nn}. In this sub-section the quality obtained with the proposed method (\PROP) is compared against the reference \ac{hevc} (\REF) and the methods used to generate the streams S1 to S4. -\begin{table}[t] +\begin{figure}[!t] + \centering + \input{043_EndToEndRobustSystem/Results/PlotQualty_vs_PLR_Predicted.tex} + \vspace{-2mm} + \caption{Average video quality using different error resilience techniques.} + \label{Figure6:ProposedQuality} +\end{figure} + +Figure~\ref{Figure6:ProposedQuality} shows the quality obtained for the proposed method and the \REF case for the main target bitrate shown in Table~\ref{TableTestSequences}. Moreover, the quality obtained with the target stream (\BEST) is also shown. One should note that the \BEST case corresponds to the optimal selection which can only be performed after quality evaluation at the decoder, therefore not suitable for real time applications and it is only used to illustrate the optimal scenario. +The results in the figure allows to evaluate the impact in the decoded quality of the selection used in the proposed method and compare it with the optimal case. + +As shown in the Figure~\ref{Figure6:ProposedQuality} the proposed method clearly outperforms the reference \ac{hevc} for all \acs{plr}. Moreover, in the error-free case (\ac{plr}=0) it does not lead to quality reduction since the proposed approach is able to select the optimal stream, thus disabling any robust coding technique when the transmission network does not have packet losses. In case of data loss results show that the proposed method is able to match the quality obtained by the \BEST case in most of the cases, revealing that the proposed method is able to efficiently select the optimal stream for transmission. + +\begin{table}[tbp] \centering \caption{Average quality gains ($\Delta$PSNR) in comparison with the reference \ac{hevc} across different bitrates and loss ratios.} \label{Table6:AverageGains} \TABLESVSPACE \newcommand{\colwidth}{2cm} \begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} -\toprule -\multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\ - & S1 & S2 & S3 & S4 & Optimal (\textit{Proposed}) \\ \hline -Basketball Drill & 1.61 & 2.66 & 2.48 & 2.44 & 2.75 \\ \hline -Basketball Drive & 2.17 & 3.46 & 3.22 & 3.63 & 3.69 \\ \hline -Book Arrival & 0.96 & 1.66 & 1.43 & 1.59 & 1.74 \\ \hline -BQSquare & 0.92 & 0.96 & 0.80 & 0.62 & 1.09 \\ \hline -Cactus & 1.28 & 2.04 & 1.89 & 1.96 & 2.09 \\ \hline -Four People & 0.28 & -0.23 & -0.88 & -0.99 & 0.25 \\ \hline -Kendo & 0.81 & 0.78 & 0.20 & 0.61 & 1.02 \\ \hline -Kimono & 1.65 & 3.04 & 3.26 & 3.27 & 3.41 \\ \hline -Kristen Sara & 0.36 & 0.15 & -0.33 & -0.50 & 0.38 \\ \hline -Park Scene & 1.64 & 2.18 & 1.88 & 2.02 & 2.25 \\ \hline -People on Street & 1.00 & 1.27 & 0.91 & 1.49 & 1.63 \\ \hline -Race Horses & 1.93 & 3.89 & 4.06 & 4.30 & 4.39 \\ \hline -Tennis & 1.15 & 2.24 & 2.17 & 3.12 & 3.18 \\ \hline -\textbf{Average} & \textbf{1.21} & \textbf{1.86} & \textbf{1.62} & \textbf{1.81} & \textbf{2.15} \\ +\toprule +\multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\ + & S1 & S2 & S3 & S4 & \PROP \\ \hline +Basketball Drill & +1.61 & +2.66 & +2.48 & +2.44 & +2.75 \\ \hline +Basketball Drive & +2.17 & +3.46 & +3.22 & +3.63 & +3.69 \\ \hline +Book Arrival & +0.96 & +1.66 & +1.43 & +1.59 & +1.74 \\ \hline +BQSquare & +0.92 & +0.96 & +0.80 & +0.62 & +1.09 \\ \hline +Cactus & +1.28 & +2.04 & +1.89 & +1.96 & +2.09 \\ \hline +Four People & +0.28 & -0.23 & -0.88 & -0.99 & +0.25 \\ \hline +Kendo & +0.81 & +0.78 & +0.20 & +0.61 & +1.02 \\ \hline +Kimono & +1.65 & +3.04 & +3.26 & +3.27 & +3.41 \\ \hline +Kristen Sara & +0.36 & +0.15 & -0.33 & -0.50 & +0.38 \\ \hline +Park Scene & +1.64 & +2.18 & +1.88 & +2.02 & +2.25 \\ \hline +People on Street & +1.00 & +1.27 & +0.91 & +1.49 & +1.63 \\ \hline +Race Horses & +1.93 & +3.89 & +4.06 & +4.30 & +4.39 \\ \hline +Tennis & +1.15 & +2.24 & +2.17 & +3.12 & +3.18 \\ \hline +\textbf{Average} & \textbf{+1.21} & \textbf{+1.86} & \textbf{+1.62} & \textbf{+1.81} & \textbf{+2.15} \\ \bottomrule \end{tabular} +\begin{tikzpicture}[overlay] +\draw[red, line width=1.2pt] (1.8,4.65) ellipse (3.5cm and 0.3cm); +\draw[red, line width=1.2pt] (3,3.05) ellipse (2.3cm and 0.3cm); +\draw[blue, line width=1.2pt] (6.7,1.45) ellipse (.85cm and 0.25cm); +\draw[blue, line width=1.2pt] (6.7,6.75) ellipse (.85cm and 0.25cm); +\end{tikzpicture} \end{table} +Subsequently, a comparison between the different methods is performed, in order to evaluate the quality gains obtained with the proposed method in comparison with the different techniques. +%for every test case. +Table~\ref{Table6:AverageGains} shows the average quality gains in comparison with the reference \ac{hevc} obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams S1 to S4. The average quality gains are measured across all test cases, i.e., different bitrates and different loss ratios. + +Results in Table~\ref{Table6:AverageGains} shows that the robust coded streams S1 to S4 are able to achieve acceptable average quality gains up to 1.86 dB (see results for S2). +However, since the introduction of robust coding techniques decreases the coding efficiency it may lead to lower quality than the reference \ac{hevc}. This is noticeable in the results obtained for the Four People sequence in the streams S2 to S4 and for the Kristen and Sara sequence and in the streams S3 and S4 (see red ellipses). This reveals that the overall performance of such methods is not suitable for transmission networks subject to low \acs{plr}, e.g., lower than 3\% (higher number of conditions have \acs{plr} lower than 3\%). +On contrary, the proposed method is always able to outperform the reference \ac{hevc} and achieve quality gains of at least 0.25 dB for the Four People sequence. Moreover, for the Basketball Drive and Race Horses sequences it is able to achieve the highest quality gains of up 3.69 dB and 4.39 dB, respectively (see blue ellipses). Summarising, by adaptively selecting the video stream, and consequently the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness achieving an average quality gains across different bitrates and \acs{plr} of 2.15 dB. + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Summary} \label{Section6:Summary} + +In this chapter an adaptive error robustness method is proposed based on reference frame selection and \ac{ec}-aware optimisation. By adaptively select a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the reference techniques. +%Moreover, the proposed method is able to outperform those cases where a single error resilience technique is used. +The overall results demonstrate that the proposed method is able to increase the reconstruction video quality in case of errors, and by adapting the transmitted stream to the network conditions a better trade-off between coding efficiency and error robusntess is achieved. +The consistent quality improvement achieved, allows the use of this method in dynamic broadcasting video services where network feedback is available and where packet loss probability is not negligible. + diff --git a/043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz b/043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz index 83d2524..413d799 100644 --- a/043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz +++ b/043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz @@ -1,5 +1,5 @@ -\def\layersep{3cm} -\begin{tikzpicture}[shorten >=1pt,->, node distance=\layersep,font=\small] +\def\layersep{3cm} +\begin{tikzpicture}[shorten >=1pt,->, node distance=\layersep,font=\small, yscale=1.2] \tikzstyle{neuron}=[circle,fill=black!25,minimum size=17pt,inner sep=0pt] \tikzstyle{input neuron}=[neuron, fill=TikZColorFillDark]; \tikzstyle{hidden neuron}=[neuron, fill=TikZColorFill]; diff --git a/043_EndToEndRobustSystem/Results/PlotQualty_vs_PLR_Predicted.tex b/043_EndToEndRobustSystem/Results/PlotQualty_vs_PLR_Predicted.tex new file mode 100644 index 0000000..653214d --- /dev/null +++ b/043_EndToEndRobustSystem/Results/PlotQualty_vs_PLR_Predicted.tex @@ -0,0 +1,276 @@ +\newcommand{\pltw}{\columnwidth} +\newcommand{\plth}{0.75\textheight} + +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' + +set multiplot layout 3,2 +set key bottom left Left reverse width 5 + +set title '\TitleFont Basketball Drill' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \BEST', '-' using 2:3 w lp ls 3 title '\KeyFont \PROP' +"EF" 0.0 39.90 +"1.0%" 1.0 36.95 +"2.0%" 2.0 35.16 +"3.0%" 3.0 33.31 +"5.0%" 5.0 31.39 +"10.0%" 10.0 28.22 +e +"EF" 0.0 39.90 +"1.0%" 1.0 38.97 +"2.0%" 2.0 38.38 +"3.0%" 3.0 37.93 +"5.0%" 5.0 37.19 +"10.0%" 10.0 35.12 +e +"EF" 0.0 39.90 +"1.0%" 1.0 38.97 +"2.0%" 2.0 38.38 +"3.0%" 3.0 37.86 +"5.0%" 5.0 37.19 +"10.0%" 10.0 35.12 +e + +set title '\TitleFont Book Arrival' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \BEST', '-' using 2:3 w lp ls 3 title '\KeyFont \PROP' +"EF" 0.0 40.28 +"1.0%" 1.0 38.58 +"2.0%" 2.0 37.58 +"3.0%" 3.0 36.21 +"5.0%" 5.0 34.40 +"10.0%" 10.0 31.01 +e +"EF" 0.0 40.28 +"1.0%" 1.0 39.72 +"2.0%" 2.0 39.42 +"3.0%" 3.0 39.19 +"5.0%" 5.0 38.78 +"10.0%" 10.0 37.32 +e +"EF" 0.0 40.28 +"1.0%" 1.0 39.72 +"2.0%" 2.0 39.34 +"3.0%" 3.0 39.19 +"5.0%" 5.0 38.78 +"10.0%" 10.0 37.32 +e + +# set title '\TitleFont BQSquare' +# plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \BEST', '-' using 2:3 w lp ls 3 title '\KeyFont \PROP' +# "EF" 0.0 39.30 +# "1.0%" 1.0 37.91 +# "2.0%" 2.0 36.72 +# "3.0%" 3.0 35.64 +# "5.0%" 5.0 34.29 +# "10.0%" 10.0 31.36 +# e +# "EF" 0.0 39.30 +# "1.0%" 1.0 38.56 +# "2.0%" 2.0 38.06 +# "3.0%" 3.0 37.65 +# "5.0%" 5.0 36.76 +# "10.0%" 10.0 34.91 +# e +# "EF" 0.0 39.30 +# "1.0%" 1.0 38.53 +# "2.0%" 2.0 38.06 +# "3.0%" 3.0 37.60 +# "5.0%" 5.0 36.76 +# "10.0%" 10.0 34.91 +# e + +# set title '\TitleFont Cactus' +# plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \BEST', '-' using 2:3 w lp ls 3 title '\KeyFont \PROP' +# "EF" 0.0 38.00 +# "1.0%" 1.0 36.51 +# "2.0%" 2.0 35.51 +# "3.0%" 3.0 34.78 +# "5.0%" 5.0 33.68 +# "10.0%" 10.0 31.93 +# e +# "EF" 0.0 38.00 +# "1.0%" 1.0 37.70 +# "2.0%" 2.0 37.38 +# "3.0%" 3.0 37.09 +# "5.0%" 5.0 36.27 +# "10.0%" 10.0 33.87 +# e +# "EF" 0.0 37.92 +# "1.0%" 1.0 37.70 +# "2.0%" 2.0 37.38 +# "3.0%" 3.0 37.09 +# "5.0%" 5.0 36.09 +# "10.0%" 10.0 33.85 +# e + +set title '\TitleFont Four People' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \BEST', '-' using 2:3 w lp ls 3 title '\KeyFont \PROP' +"EF" 0.0 40.13 +"1.0%" 1.0 39.78 +"2.0%" 2.0 39.51 +"3.0%" 3.0 39.16 +"5.0%" 5.0 38.54 +"10.0%" 10.0 37.12 +e +"EF" 0.0 40.13 +"1.0%" 1.0 39.96 +"2.0%" 2.0 39.81 +"3.0%" 3.0 39.70 +"5.0%" 5.0 39.51 +"10.0%" 10.0 38.93 +e +"EF" 0.0 40.13 +"1.0%" 1.0 39.96 +"2.0%" 2.0 39.81 +"3.0%" 3.0 39.70 +"5.0%" 5.0 39.51 +"10.0%" 10.0 38.93 +e + +set title '\TitleFont Kendo' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \BEST', '-' using 2:3 w lp ls 3 title '\KeyFont \PROP' +"EF" 0.0 41.15 +"1.0%" 1.0 39.54 +"2.0%" 2.0 38.27 +"3.0%" 3.0 37.35 +"5.0%" 5.0 34.86 +"10.0%" 10.0 31.57 +e +"EF" 0.0 41.15 +"1.0%" 1.0 40.24 +"2.0%" 2.0 39.49 +"3.0%" 3.0 38.95 +"5.0%" 5.0 38.36 +"10.0%" 10.0 36.62 +e +"EF" 0.0 41.15 +"1.0%" 1.0 40.11 +"2.0%" 2.0 39.18 +"3.0%" 3.0 38.69 +"5.0%" 5.0 38.36 +"10.0%" 10.0 36.62 +e + +# set title '\TitleFont Kimono' +# plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \BEST', '-' using 2:3 w lp ls 3 title '\KeyFont \PROP' +# "EF" 0.0 42.88 +# "1.0%" 1.0 39.02 +# "2.0%" 2.0 37.48 +# "3.0%" 3.0 36.19 +# "5.0%" 5.0 34.60 +# "10.0%" 10.0 32.24 +# e +# "EF" 0.0 42.88 +# "1.0%" 1.0 42.00 +# "2.0%" 2.0 40.75 +# "3.0%" 3.0 39.67 +# "5.0%" 5.0 36.61 +# "10.0%" 10.0 32.67 +# e +# "EF" 0.0 42.88 +# "1.0%" 1.0 41.94 +# "2.0%" 2.0 40.75 +# "3.0%" 3.0 39.67 +# "5.0%" 5.0 36.51 +# "10.0%" 10.0 32.03 +# e + +# set title '\TitleFont Park Scene' +# plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \BEST', '-' using 2:3 w lp ls 3 title '\KeyFont \PROP' +# "EF" 0.0 39.47 +# "1.0%" 1.0 36.97 +# "2.0%" 2.0 35.56 +# "3.0%" 3.0 34.54 +# "5.0%" 5.0 32.92 +# "10.0%" 10.0 30.42 +# e +# "EF" 0.0 39.47 +# "1.0%" 1.0 38.84 +# "2.0%" 2.0 38.31 +# "3.0%" 3.0 37.77 +# "5.0%" 5.0 36.85 +# "10.0%" 10.0 34.06 +# e +# "EF" 0.0 39.47 +# "1.0%" 1.0 38.84 +# "2.0%" 2.0 38.31 +# "3.0%" 3.0 37.77 +# "5.0%" 5.0 36.85 +# "10.0%" 10.0 34.06 +# e + +set title '\TitleFont People On Street' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \BEST', '-' using 2:3 w lp ls 3 title '\KeyFont \PROP' +"EF" 0.0 35.90 +"1.0%" 1.0 32.55 +"2.0%" 2.0 30.71 +"3.0%" 3.0 29.48 +"5.0%" 5.0 27.77 +"10.0%" 10.0 25.05 +e +"EF" 0.0 35.90 +"1.0%" 1.0 34.11 +"2.0%" 2.0 33.29 +"3.0%" 3.0 32.56 +"5.0%" 5.0 30.85 +"10.0%" 10.0 27.14 +e +"EF" 0.0 35.90 +"1.0%" 1.0 34.11 +"2.0%" 2.0 33.29 +"3.0%" 3.0 32.56 +"5.0%" 5.0 30.85 +"10.0%" 10.0 27.14 +e + +set title '\TitleFont Race Horses' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \BEST', '-' using 2:3 w lp ls 3 title '\KeyFont \PROP' +"EF" 0.0 39.87 +"1.0%" 1.0 33.45 +"2.0%" 2.0 30.74 +"3.0%" 3.0 29.08 +"5.0%" 5.0 26.98 +"10.0%" 10.0 24.19 +e +"EF" 0.0 39.87 +"1.0%" 1.0 37.73 +"2.0%" 2.0 36.27 +"3.0%" 3.0 35.11 +"5.0%" 5.0 32.76 +"10.0%" 10.0 27.93 +e +"EF" 0.0 39.36 +"1.0%" 1.0 37.73 +"2.0%" 2.0 36.27 +"3.0%" 3.0 35.11 +"5.0%" 5.0 32.76 +"10.0%" 10.0 27.93 +e + +# set title '\TitleFont Tennis' +# plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \BEST', '-' using 2:3 w lp ls 3 title '\KeyFont \PROP' +# "EF" 0.0 40.68 +# "1.0%" 1.0 36.81 +# "2.0%" 2.0 34.46 +# "3.0%" 3.0 32.71 +# "5.0%" 5.0 30.36 +# "10.0%" 10.0 27.08 +# e +# "EF" 0.0 40.68 +# "1.0%" 1.0 39.37 +# "2.0%" 2.0 38.55 +# "3.0%" 3.0 37.68 +# "5.0%" 5.0 36.61 +# "10.0%" 10.0 33.69 +# e +# "EF" 0.0 40.68 +# "1.0%" 1.0 39.37 +# "2.0%" 2.0 38.55 +# "3.0%" 3.0 37.68 +# "5.0%" 5.0 36.61 +# "10.0%" 10.0 33.69 +# e + + + +\end{gnuplot} diff --git a/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex b/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex index 4f10f1f..abee388 100644 --- a/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex +++ b/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex @@ -1,5 +1,5 @@ \newcommand{\pltw}{\columnwidth} -\newcommand{\plth}{0.7\textheight} +\newcommand{\plth}{0.75\textheight} \begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] load 'carreira_phd_thesis_gnuplot_template.cfg' diff --git a/043_EndToEndRobustSystem/Results/pyPlotQuality.bash b/043_EndToEndRobustSystem/Results/pyPlotQuality.bash deleted file mode 100644 index bbfab0a..0000000 --- a/043_EndToEndRobustSystem/Results/pyPlotQuality.bash +++ /dev/null @@ -1,569 +0,0 @@ -#!/bin/bash -gnuplot << _EOF -set terminal pdfcairo mono \ -font 'TimesNewRoman,14' - -set datafile missing '-' - -set grid -#set size square {1,.5} - -set title center offset character 0, -.9 - -set xlabel center -set ylabel center offset character 1, 0 - - -set key box spacing 1 width 0 - -set style line 4 lc 1 lt 1 lw 2 pt 2 ps 1 -set style line 2 lc 2 lt 2 lw 2 pt 3 ps 1 -set style line 3 lc 3 lt 4 lw 2 pt 4 ps 1 -set style line 1 lc 4 lt 4 lw 2 pt 10 ps 1 -set style line 5 lc 7 lt 3 lw 2 pt 6 ps 1 -set style line 6 lc 8 lt 6 lw 2 pt 8 ps 1 -set style line 7 lc 2 lt 7 lw 2 pt 8 ps 1 -set style line 8 lc 3 lt 8 lw 2 pt 9 ps 1 -set style line 9 lc 4 lt 9 lw 2 pt 10 ps 1 - -set style line 100 lc 1 lw 3 -set style line 101 lc 4 lw 3 -set style line 102 lc 2 lw 3 -set style line 103 lc 3 lw 3 - -#set rmargin 1 - -set xlabel 'Packet Loss Ratio [%]' -set key bottom left Left reverse -set output 'lowdelay_P_4R_slice_1200B_seq_basketball_drill_rate_single.pdf' -set title 'Low-Delay - 1200B - Basketball Drill - Main rate' -plot '-' using 2:1 w lp ls 1 title 'FC', '-' using 2:1 w lp ls 2 title 'EXP+EMVE', '-' using 2:1 w lp ls 3 title 'EL10', '-' using 2:1 w lp ls 4 title 'EL20' -39.900 0.0 "EF" -38.057 0.5 "0.5%" -36.949 1.0 "1.0%" -35.859 1.6 "1.2%" -35.157 2.0 "1.6%" -33.890 2.6 "2.0%" -33.305 3.0 "3.0%" -31.385 5.0 "5.0%" -28.217 10.0 "10.0%" -e -39.600 0.0 "EF" -38.847 0.5 "0.5%" -38.305 1.0 "1.0%" -37.485 1.6 "1.2%" -37.178 2.0 "1.6%" -36.436 2.6 "2.0%" -36.137 3.0 "3.0%" -34.691 5.0 "5.0%" -31.288 10.0 "10.0%" -e -39.480 0.0 "EF" -39.191 0.5 "0.5%" -38.973 1.0 "1.0%" -38.586 1.6 "1.2%" -38.385 2.0 "1.6%" -38.093 2.6 "2.0%" -37.929 3.0 "3.0%" -37.015 5.0 "5.0%" -34.474 10.0 "10.0%" -e -38.880 0.0 "EF" -38.729 0.5 "0.5%" -38.550 1.0 "1.0%" -38.353 1.6 "1.2%" -38.208 2.0 "1.6%" -37.956 2.6 "2.0%" -37.863 3.0 "3.0%" -37.186 5.0 "5.0%" -35.117 10.0 "10.0%" -e - -set output 'lowdelay_P_4R_slice_1200B_seq_basketball_drill_rate_single_60pc.pdf' -set title 'Low-Delay - 1200B - Basketball Drill - 40% Reduction ' -plot '-' using 2:1 w lp ls 1 title 'FC', '-' using 2:1 w lp ls 2 title 'EXP+EMVE', '-' using 2:1 w lp ls 3 title 'EL10', '-' using 2:1 w lp ls 4 title 'EL20' -37.570 0.0 "EF" -36.658 0.5 "0.5%" -35.927 1.0 "1.0%" -35.287 1.6 "1.2%" -34.683 2.0 "1.6%" -33.776 2.6 "2.0%" -33.583 3.0 "3.0%" -31.726 5.0 "5.0%" -28.381 10.0 "10.0%" -e -37.120 0.0 "EF" -36.831 0.5 "0.5%" -36.405 1.0 "1.0%" -36.152 1.6 "1.2%" -35.895 2.0 "1.6%" -35.391 2.6 "2.0%" -35.099 3.0 "3.0%" -34.150 5.0 "5.0%" -31.861 10.0 "10.0%" -e -36.940 0.0 "EF" -36.737 0.5 "0.5%" -36.569 1.0 "1.0%" -36.487 1.6 "1.2%" -36.357 2.0 "1.6%" -36.072 2.6 "2.0%" -35.991 3.0 "3.0%" -35.431 5.0 "5.0%" -33.753 10.0 "10.0%" -e -36.130 0.0 "EF" -36.038 0.5 "0.5%" -35.975 1.0 "1.0%" -35.885 1.6 "1.2%" -35.749 2.0 "1.6%" -35.627 2.6 "2.0%" -35.617 3.0 "3.0%" -35.114 5.0 "5.0%" -33.962 10.0 "10.0%" -e - -set output 'lowdelay_P_4R_slice_1200B_seq_book_arrival_c6_rate_single.pdf' -set title 'Low-Delay - 1200B - Book Arrival - Main rate' -plot '-' using 2:1 w lp ls 1 title 'FC', '-' using 2:1 w lp ls 2 title 'EXP+EMVE', '-' using 2:1 w lp ls 3 title 'EL10', '-' using 2:1 w lp ls 4 title 'EL20' -40.280 0.0 "EF" -39.122 0.5 "0.5%" -38.578 1.0 "1.0%" -37.776 1.6 "1.2%" -37.584 2.0 "1.6%" -36.699 2.6 "2.0%" -36.206 3.0 "3.0%" -34.396 5.0 "5.0%" -31.010 10.0 "10.0%" -e -39.970 0.0 "EF" -39.391 0.5 "0.5%" -39.053 1.0 "1.0%" -38.553 1.6 "1.2%" -38.419 2.0 "1.6%" -37.928 2.6 "2.0%" -37.719 3.0 "3.0%" -36.705 5.0 "5.0%" -33.853 10.0 "10.0%" -e -40.030 0.0 "EF" -39.788 0.5 "0.5%" -39.716 1.0 "1.0%" -39.460 1.6 "1.2%" -39.339 2.0 "1.6%" -39.088 2.6 "2.0%" -38.918 3.0 "3.0%" -38.348 5.0 "5.0%" -36.607 10.0 "10.0%" -e -39.790 0.0 "EF" -39.679 0.5 "0.5%" -39.517 1.0 "1.0%" -39.463 1.6 "1.2%" -39.416 2.0 "1.6%" -39.125 2.6 "2.0%" -39.193 3.0 "3.0%" -38.781 5.0 "5.0%" -37.323 10.0 "10.0%" -e - -set output 'lowdelay_P_4R_slice_1200B_seq_book_arrival_c6_rate_single_60pc.pdf' -set title 'Low-Delay - 1200B - Book Arrival - 40% Reduction ' -plot '-' using 2:1 w lp ls 1 title 'FC', '-' using 2:1 w lp ls 2 title 'EXP+EMVE', '-' using 2:1 w lp ls 3 title 'EL10', '-' using 2:1 w lp ls 4 title 'EL20' -38.920 0.0 "EF" -38.641 0.5 "0.5%" -37.769 1.0 "1.0%" -37.575 1.6 "1.2%" -36.545 2.0 "1.6%" -36.302 2.6 "2.0%" -36.189 3.0 "3.0%" -35.001 5.0 "5.0%" -31.592 10.0 "10.0%" -e -38.400 0.0 "EF" -38.253 0.5 "0.5%" -37.711 1.0 "1.0%" -37.676 1.6 "1.2%" -37.137 2.0 "1.6%" -37.037 2.6 "2.0%" -37.005 3.0 "3.0%" -36.027 5.0 "5.0%" -34.522 10.0 "10.0%" -e -38.460 0.0 "EF" -38.305 0.5 "0.5%" -38.169 1.0 "1.0%" -38.029 1.6 "1.2%" -37.780 2.0 "1.6%" -37.651 2.6 "2.0%" -37.582 3.0 "3.0%" -36.977 5.0 "5.0%" -35.674 10.0 "10.0%" -e -37.830 0.0 "EF" -37.715 0.5 "0.5%" -37.487 1.0 "1.0%" -37.537 1.6 "1.2%" -37.469 2.0 "1.6%" -37.419 2.6 "2.0%" -37.321 3.0 "3.0%" -36.980 5.0 "5.0%" -35.818 10.0 "10.0%" -e - -set output 'lowdelay_P_4R_slice_1200B_seq_bqsquare_rate_single.pdf' -set title 'Low-Delay - 1200B - BQSquare - Main rate' -plot '-' using 2:1 w lp ls 1 title 'FC', '-' using 2:1 w lp ls 2 title 'EXP+EMVE', '-' using 2:1 w lp ls 3 title 'EL10', '-' using 2:1 w lp ls 4 title 'EL20' -39.300 0.0 "EF" -38.381 0.5 "0.5%" -37.908 1.0 "1.0%" -37.233 1.6 "1.2%" -36.716 2.0 "1.6%" -35.946 2.6 "2.0%" -35.639 3.0 "3.0%" -34.289 5.0 "5.0%" -31.365 10.0 "10.0%" -e -39.150 0.0 "EF" -38.735 0.5 "0.5%" -38.522 1.0 "1.0%" -38.077 1.6 "1.2%" -37.747 2.0 "1.6%" -37.252 2.6 "2.0%" -36.985 3.0 "3.0%" -35.971 5.0 "5.0%" -33.495 10.0 "10.0%" -e -38.750 0.0 "EF" -38.558 0.5 "0.5%" -38.371 1.0 "1.0%" -38.163 1.6 "1.2%" -37.981 2.0 "1.6%" -37.673 2.6 "2.0%" -37.652 3.0 "3.0%" -36.734 5.0 "5.0%" -34.690 10.0 "10.0%" -e -38.160 0.0 "EF" -38.015 0.5 "0.5%" -37.897 1.0 "1.0%" -37.767 1.6 "1.2%" -37.652 2.0 "1.6%" -37.447 2.6 "2.0%" -37.411 3.0 "3.0%" -36.765 5.0 "5.0%" -34.911 10.0 "10.0%" -e - -set output 'lowdelay_P_4R_slice_1200B_seq_bqsquare_rate_single_60pc.pdf' -set title 'Low-Delay - 1200B - BQSquare - 40% Reduction ' -plot '-' using 2:1 w lp ls 1 title 'FC', '-' using 2:1 w lp ls 2 title 'EXP+EMVE', '-' using 2:1 w lp ls 3 title 'EL10', '-' using 2:1 w lp ls 4 title 'EL20' -36.780 0.0 "EF" -36.370 0.5 "0.5%" -36.129 1.0 "1.0%" -35.735 1.6 "1.2%" -35.480 2.0 "1.6%" -34.912 2.6 "2.0%" -34.947 3.0 "3.0%" -33.946 5.0 "5.0%" -31.479 10.0 "10.0%" -e -36.610 0.0 "EF" -36.340 0.5 "0.5%" -36.261 1.0 "1.0%" -36.051 1.6 "1.2%" -35.839 2.0 "1.6%" -35.513 2.6 "2.0%" -35.547 3.0 "3.0%" -34.913 5.0 "5.0%" -33.003 10.0 "10.0%" -e -36.230 0.0 "EF" -36.135 0.5 "0.5%" -36.021 1.0 "1.0%" -35.888 1.6 "1.2%" -35.787 2.0 "1.6%" -35.586 2.6 "2.0%" -35.569 3.0 "3.0%" -35.200 5.0 "5.0%" -33.764 10.0 "10.0%" -e -35.700 0.0 "EF" -35.627 0.5 "0.5%" -35.560 1.0 "1.0%" -35.459 1.6 "1.2%" -35.399 2.0 "1.6%" -35.297 2.6 "2.0%" -35.258 3.0 "3.0%" -34.891 5.0 "5.0%" -33.942 10.0 "10.0%" -e - -set output 'lowdelay_P_4R_slice_1200B_seq_kendo_c3_rate_single.pdf' -set title 'Low-Delay - 1200B - Kendo - Main rate' -plot '-' using 2:1 w lp ls 1 title 'FC', '-' using 2:1 w lp ls 2 title 'EXP+EMVE', '-' using 2:1 w lp ls 3 title 'EL10', '-' using 2:1 w lp ls 4 title 'EL20' -41.150 0.0 "EF" -40.679 0.5 "0.5%" -39.544 1.0 "1.0%" -38.839 1.6 "1.2%" -38.271 2.0 "1.6%" -37.512 2.6 "2.0%" -37.349 3.0 "3.0%" -34.856 5.0 "5.0%" -31.573 10.0 "10.0%" -e -40.270 0.0 "EF" -40.009 0.5 "0.5%" -39.304 1.0 "1.0%" -39.111 1.6 "1.2%" -38.638 2.0 "1.6%" -38.516 2.6 "2.0%" -38.401 3.0 "3.0%" -37.110 5.0 "5.0%" -34.829 10.0 "10.0%" -e -40.730 0.0 "EF" -40.427 0.5 "0.5%" -40.156 1.0 "1.0%" -39.496 1.6 "1.2%" -39.319 2.0 "1.6%" -39.305 2.6 "2.0%" -38.692 3.0 "3.0%" -37.258 5.0 "5.0%" -35.144 10.0 "10.0%" -e -40.260 0.0 "EF" -40.065 0.5 "0.5%" -39.801 1.0 "1.0%" -39.599 1.6 "1.2%" -39.492 2.0 "1.6%" -39.238 2.6 "2.0%" -38.950 3.0 "3.0%" -38.358 5.0 "5.0%" -36.619 10.0 "10.0%" -e - -set output 'lowdelay_P_4R_slice_1200B_seq_kendo_c3_rate_single_60pc.pdf' -set title 'Low-Delay - 1200B - Kendo - 40% Reduction ' -plot '-' using 2:1 w lp ls 1 title 'FC', '-' using 2:1 w lp ls 2 title 'EXP+EMVE', '-' using 2:1 w lp ls 3 title 'EL10', '-' using 2:1 w lp ls 4 title 'EL20' -38.730 0.0 "EF" -38.365 0.5 "0.5%" -37.714 1.0 "1.0%" -37.599 1.6 "1.2%" -36.762 2.0 "1.6%" -36.695 2.6 "2.0%" -35.859 3.0 "3.0%" -34.589 5.0 "5.0%" -31.635 10.0 "10.0%" -e -37.550 0.0 "EF" -37.280 0.5 "0.5%" -37.189 1.0 "1.0%" -36.991 1.6 "1.2%" -36.823 2.0 "1.6%" -36.388 2.6 "2.0%" -36.283 3.0 "3.0%" -35.449 5.0 "5.0%" -33.936 10.0 "10.0%" -e -38.080 0.0 "EF" -37.819 0.5 "0.5%" -37.382 1.0 "1.0%" -37.364 1.6 "1.2%" -36.767 2.0 "1.6%" -36.901 2.6 "2.0%" -36.804 3.0 "3.0%" -35.920 5.0 "5.0%" -34.085 10.0 "10.0%" -e -37.260 0.0 "EF" -37.147 0.5 "0.5%" -36.921 1.0 "1.0%" -36.823 1.6 "1.2%" -36.588 2.0 "1.6%" -36.552 2.6 "2.0%" -36.383 3.0 "3.0%" -35.746 5.0 "5.0%" -34.202 10.0 "10.0%" -e - -set output 'lowdelay_P_4R_slice_1200B_seq_park_rate_single.pdf' -set title 'Low-Delay - 1200B - Park Scene - Main rate' -plot '-' using 2:1 w lp ls 1 title 'FC', '-' using 2:1 w lp ls 2 title 'EXP+EMVE', '-' using 2:1 w lp ls 3 title 'EL10', '-' using 2:1 w lp ls 4 title 'EL20' -39.470 0.0 "EF" -38.067 0.5 "0.5%" -36.969 1.0 "1.0%" -36.205 1.6 "1.2%" -35.555 2.0 "1.6%" -34.863 2.6 "2.0%" -34.545 3.0 "3.0%" -32.923 5.0 "5.0%" -30.423 10.0 "10.0%" -e -38.900 0.0 "EF" -38.528 0.5 "0.5%" -38.113 1.0 "1.0%" -37.721 1.6 "1.2%" -37.380 2.0 "1.6%" -37.064 2.6 "2.0%" -36.846 3.0 "3.0%" -35.771 5.0 "5.0%" -32.539 10.0 "10.0%" -e -39.160 0.0 "EF" -38.957 0.5 "0.5%" -38.779 1.0 "1.0%" -38.509 1.6 "1.2%" -38.212 2.0 "1.6%" -37.920 2.6 "2.0%" -37.767 3.0 "3.0%" -36.847 5.0 "5.0%" -33.341 10.0 "10.0%" -e -38.820 0.0 "EF" -38.681 0.5 "0.5%" -38.589 1.0 "1.0%" -38.287 1.6 "1.2%" -38.127 2.0 "1.6%" -37.869 2.6 "2.0%" -37.693 3.0 "3.0%" -36.777 5.0 "5.0%" -34.059 10.0 "10.0%" -e - -set output 'lowdelay_P_4R_slice_1200B_seq_park_rate_single_60pc.pdf' -set title 'Low-Delay - 1200B - Park Scene - 40% Reduction ' -plot '-' using 2:1 w lp ls 1 title 'FC', '-' using 2:1 w lp ls 2 title 'EXP+EMVE', '-' using 2:1 w lp ls 3 title 'EL10', '-' using 2:1 w lp ls 4 title 'EL20' -37.810 0.0 "EF" -36.969 0.5 "0.5%" -36.189 1.0 "1.0%" -35.422 1.6 "1.2%" -35.190 2.0 "1.6%" -34.477 2.6 "2.0%" -34.091 3.0 "3.0%" -32.873 5.0 "5.0%" -30.508 10.0 "10.0%" -e -37.230 0.0 "EF" -37.020 0.5 "0.5%" -36.750 1.0 "1.0%" -36.504 1.6 "1.2%" -36.361 2.0 "1.6%" -36.002 2.6 "2.0%" -35.951 3.0 "3.0%" -35.174 5.0 "5.0%" -33.007 10.0 "10.0%" -e -37.460 0.0 "EF" -37.289 0.5 "0.5%" -37.189 1.0 "1.0%" -37.070 1.6 "1.2%" -36.905 2.0 "1.6%" -36.740 2.6 "2.0%" -36.517 3.0 "3.0%" -36.009 5.0 "5.0%" -34.323 10.0 "10.0%" -e -37.010 0.0 "EF" -36.903 0.5 "0.5%" -36.868 1.0 "1.0%" -36.735 1.6 "1.2%" -36.625 2.0 "1.6%" -36.501 2.6 "2.0%" -36.354 3.0 "3.0%" -35.955 5.0 "5.0%" -34.623 10.0 "10.0%" -e - -set output 'lowdelay_P_4R_slice_1200B_seq_tennis_rate_single.pdf' -set title 'Low-Delay - 1200B - Tennis - Main rate' -plot '-' using 2:1 w lp ls 1 title 'FC', '-' using 2:1 w lp ls 2 title 'EXP+EMVE', '-' using 2:1 w lp ls 3 title 'EL10', '-' using 2:1 w lp ls 4 title 'EL20' -40.680 0.0 "EF" -38.243 0.5 "0.5%" -36.815 1.0 "1.0%" -35.278 1.6 "1.2%" -34.460 2.0 "1.6%" -33.418 2.6 "2.0%" -32.709 3.0 "3.0%" -30.360 5.0 "5.0%" -27.083 10.0 "10.0%" -e -40.030 0.0 "EF" -38.966 0.5 "0.5%" -38.059 1.0 "1.0%" -36.983 1.6 "1.2%" -36.429 2.0 "1.6%" -35.351 2.6 "2.0%" -35.068 3.0 "3.0%" -33.155 5.0 "5.0%" -29.764 10.0 "10.0%" -e -40.390 0.0 "EF" -39.420 0.5 "0.5%" -38.912 1.0 "1.0%" -38.123 1.6 "1.2%" -37.448 2.0 "1.6%" -36.907 2.6 "2.0%" -36.522 3.0 "3.0%" -34.991 5.0 "5.0%" -31.664 10.0 "10.0%" -e -40.030 0.0 "EF" -39.612 0.5 "0.5%" -39.371 1.0 "1.0%" -38.763 1.6 "1.2%" -38.549 2.0 "1.6%" -38.128 2.6 "2.0%" -37.679 3.0 "3.0%" -36.606 5.0 "5.0%" -33.692 10.0 "10.0%" -e - -set output 'lowdelay_P_4R_slice_1200B_seq_tennis_rate_single_60pc.pdf' -set title 'Low-Delay - 1200B - Tennis - 40% Reduction ' -plot '-' using 2:1 w lp ls 1 title 'FC', '-' using 2:1 w lp ls 2 title 'EXP+EMVE', '-' using 2:1 w lp ls 3 title 'EL10', '-' using 2:1 w lp ls 4 title 'EL20' -39.020 0.0 "EF" -37.569 0.5 "0.5%" -36.453 1.0 "1.0%" -35.481 1.6 "1.2%" -34.685 2.0 "1.6%" -33.662 2.6 "2.0%" -33.442 3.0 "3.0%" -31.232 5.0 "5.0%" -27.690 10.0 "10.0%" -e -38.170 0.0 "EF" -37.482 0.5 "0.5%" -36.874 1.0 "1.0%" -36.215 1.6 "1.2%" -35.711 2.0 "1.6%" -35.236 2.6 "2.0%" -34.863 3.0 "3.0%" -33.289 5.0 "5.0%" -30.118 10.0 "10.0%" -e -38.580 0.0 "EF" -37.739 0.5 "0.5%" -37.295 1.0 "1.0%" -36.679 1.6 "1.2%" -36.384 2.0 "1.6%" -35.732 2.6 "2.0%" -35.538 3.0 "3.0%" -34.074 5.0 "5.0%" -31.364 10.0 "10.0%" -e -37.960 0.0 "EF" -37.569 0.5 "0.5%" -37.403 1.0 "1.0%" -36.947 1.6 "1.2%" -36.903 2.0 "1.6%" -36.413 2.6 "2.0%" -36.227 3.0 "3.0%" -35.540 5.0 "5.0%" -32.985 10.0 "10.0%" -e - -_EOF -CONV_FILENAMES="lowdelay_P_4R_slice_1200B_seq_basketball_drill_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_basketball_drill_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_book_arrival_c6_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_book_arrival_c6_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_bqsquare_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_bqsquare_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_kendo_c3_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_kendo_c3_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_park_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_park_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_tennis_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_tennis_rate_single_60pc.pdf " -gs -q -sPAPERSIZE=letter -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=pyPlotQuality.pdf $CONV_FILENAMES -rm ${CONV_FILENAMES}