diff --git a/021_VideoCoding/VideoCodingReview.tex b/021_VideoCoding/VideoCodingReview.tex index 3eb6258..49fc1ee 100755 --- a/021_VideoCoding/VideoCodingReview.tex +++ b/021_VideoCoding/VideoCodingReview.tex @@ -140,6 +140,9 @@ Moreover, new pictures types were introduced to increase the random-access flexi Similar to \ac{h264}~\cite{H264AVCStandard2003}, the \ac{hevc} streams are based on \glsfirst{nal} for stream partitioning and synchronization, which is useful to deal with lossy network environments~\cite{H264OverIPJul2003}. Using this concept the stream is composed of different \ac{nal} units. The \ac{nal} concept provides the ability to map the \ac{vcl} data that represents coded video slices (or frames) on various transport layers, such as \ac{rtp}, MPEG-2 Systems and MPEG DASH. The~\ac{nal} units are classified into two categories: (i) \ac{vcl} and (ii) non-\ac{vcl}, as they carry coded video or associated data, respectively. +The \ac{hevc} standard defines two bytes to describe the content of each \ac{nal} unit. +These bytes are used to identify the \ac{nal} type are transmitted using a fixed coding format length, i.e., without variable length codes. This allow extra error robustness but increases the bit overhead. Figure~\ref{Figure2:NalHeader} shows the structure of the \ac{hevc} \ac{nal} unit header. The first bit \textit{F}, referred to as \textit{forbiden\_bit}, is always zero in order to ensure that a start code can be detected across different protocol standards. This is followed by the \ac{nal} type, which identifies its content, enabling the decoder to either use it if required for a particular decoding purpose or discard it. Another syntax element introduced by the \ac{hevc} standard is the \textit{temporal\_id\_plus\_one} (TIDP), which defines the temporal layer where the \ac{nal} unit belongs to. This allows implicit temporal scalability (layers ranging from 0 to 6) by immediately discarding the \ac{nal} unit if it belongs to a layer higher than the desired one. In the header, six bits are reserved for future extensions, preventing the creation of extra \ac{nal} units for scalable and multiview extensions, as done in \ac{h264}. + \begin{figure}[tb] \centering \vspace{5mm} @@ -155,11 +158,6 @@ Similar to \ac{h264}~\cite{H264AVCStandard2003}, the \ac{hevc} streams are based \label{Figure2:NalHeader} \end{figure} -The \ac{hevc} standard defines two bytes to describe the content of each \ac{nal} unit. -These bytes are used to identify the \ac{nal} type are transmitted using a fixed coding format length, i.e., without variable length codes. This allow extra error robustness but increases the bit overhead. Figure~\ref{Figure2:NalHeader} shows the structure of the \ac{hevc} \ac{nal} unit header. The first bit \textit{F}, referred to as \textit{forbiden\_bit}, is always zero in order to ensure that a start code can be detected across different protocol standards. This is followed by the \ac{nal} type, which identifies its content, enabling the decoder to either use it if required for a particular decoding purpose or discard it. Another syntax element introduced by the \ac{hevc} standard is the \textit{temporal\_id\_plus\_one} (TIDP), which defines the temporal layer where the \ac{nal} unit belongs to. This allows implicit temporal scalability (layers ranging from 0 to 6) by immediately discarding the \ac{nal} unit if it belongs to a layer higher than the desired one. In the header, six bits are reserved for future extensions, preventing the creation of extra \ac{nal} units for scalable and multiview extensions, as done in \ac{h264}. - - - \subsection{Parameters set} \glsreset{vps} @@ -249,13 +247,6 @@ Therefore, in order to make splicing operation straightforward, the \ac{nal} uni \subsubsection{Leading and trailing pictures} -\begin{figure}[tb] -\centering -\input{\FigsPath/PictureTypes_Leading_Trailing.tikz} -\caption{Leading and trailing pictures associated with a random-access point.} -\label{Figure2:LeadingTralingPics} -\end{figure} - Leading pictures correspond to frames that follow a particular \ac{rap} frame in the decoding order, but precede it in the display order. A trailing picture is a frame that follows a particular \ac{rap} frame in both decoding and display order. %A trailing picture is a frame that follows a particular \ac{rap} frame in both decoding and display order and must not depend on any leading frames nor any trailing frame associated to a previous \ac{rap}. @@ -264,6 +255,13 @@ Leading and trailing pictures are considered to be associated with the closest p Also, all leading frames of a \ac{rap} frame must precede in decoding order all trailing frames that are associated with the same \ac{rap}. This means that the following order is imposed by the \ac{hevc} standard: 1) \ac{rap} picture, 2) associated leading pictures and 3) associated trailing pictures. Due to the introduction of more flexible random-access points using \ac{cra} pictures, it is important to mark the leading and trailing pictures, so the decoder can be aware of which frames can be correctly decodable whenever it starts decoding the bitstream. +\begin{figure}[tb] + \centering + \input{\FigsPath/PictureTypes_Leading_Trailing.tikz} + \caption{Leading and trailing pictures associated with a random-access point.} + \label{Figure2:LeadingTralingPics} +\end{figure} + There are two types of leading pictures, as shown in Table~\ref{Table2:PictureTypes}. The \ac{radl} pictures, which can be correctly decoded since they only depend on the associated \ac{rap} picture and do not have any prior dependencies to trailing pictures associated with the previous \ac{rap} frame. On contrary, the \ac{rasl} pictures may have dependencies to prior trailing pictures. @@ -341,6 +339,10 @@ The use of tiles provide several advantages. Specifically, they improve frame pa As mentioned before, the use of either slices or tiles to process each frame using parallel threads require the use of different entropy coding contexts for each slice/tile, in order to make them independently decodable. As consequence, the compression efficiency decreases. In order to overcome this issue, the concept of \ac{wpp} was introduced, which enables coding using efficient parallel processing~\cite{HEVCParallelCSVT2016}. +In \ac{wpp}, each slice (or tile) is divided into single rows of \acs{ctu} which can be processed in parallel, but entropy coding and prediction are enabled across \acs{ctu} of different rows, in order to avoid loss in coding efficiency. +To re-use the same context in the whole slice (or tile) each row requires a delay of two \acs{ctu} in order to ensure that all necessary dependencies have already been encoded/decoded. Figure~\ref{Figure2:WPP} illustrates an example of \ac{wpp} using at least five different threads. +Each square corresponds to a \ac{ctu} and the numbers correspond to the time instance when a \ac{ctu} starts being processed. Using this staggered start (which appears like a wavefront, as represented in Figure~\ref{Figure2:WPP}), the parallelisation may use as many threads as the number of \acs{ctu} rows available in the video frame. However, although the number of threads does not affect the coding efficiency, the required inter-process communication substantially increases. + \begin{figure}[tb] \centering \input{\FigsPath/WPP.tikz} @@ -348,10 +350,6 @@ As mentioned before, the use of either slices or tiles to process each frame usi \label{Figure2:WPP} \end{figure} -In \ac{wpp}, each slice (or tile) is divided into single rows of \acs{ctu} which can be processed in parallel, but entropy coding and prediction are enabled across \acs{ctu} of different rows, in order to avoid loss in coding efficiency. -To re-use the same context in the whole slice (or tile) each row requires a delay of two \acs{ctu} in order to ensure that all necessary dependencies have already been encoded/decoded. Figure~\ref{Figure2:WPP} illustrates an example of \ac{wpp} using at least five different threads. -Each square corresponds to a \ac{ctu} and the numbers correspond to the time instance when a \ac{ctu} starts being processed. Using this staggered start (which appears like a wavefront, as represented in Figure~\ref{Figure2:WPP}), the parallelisation may use as many threads as the number of \acs{ctu} rows available in the video frame. However, although the number of threads does not affect the coding efficiency, the required inter-process communication substantially increases. - %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -450,19 +448,19 @@ To improve the efficiency of angular prediction, when the reference samples need \subsubsection{Planar and DC} -\begin{figure}[tb] -\centering -\input{021_VideoCoding/Figures/IntraPredictionPlanar.tikz} -\caption{Example of Planar prediction for a $4\times4$ block.} -\label{Figure2:IntraModePlanar} -\end{figure} - The Planar and DC modes are particularly efficient in the prediction of smooth regions. The DC mode uses the average or the neighbouring reference samples to generate a constant prediction for the current \ac{pu} as a whole, i.e., all pixels are predicted from the same value. The Planar mode predicts the \ac{pu} through a linear interpolation from the four closest neighbouring reference pixels. Figure~\ref{Figure2:IntraModePlanar} shows the reference pixels (i.e., $p_1$, $p'_2$, $p_3$ and $p'_4$) and arrows pointing to the pixel being predicted. Since the right and bottom neighbourhood is not available (not coded yet), the same pixel $p_2$ is repeated to form a column of reference pixels, marked with an ellipse in Figure~\ref{Figure2:IntraModePlanar}. A similar approach is also used to create the bottom row of reference pixels from the pixel $p_4$ . %As the Planar mode used linear surface for prediction it prevents the discontinuities at \ac{pu} boundaries. +\begin{figure}[tb] + \centering + \input{021_VideoCoding/Figures/IntraPredictionPlanar.tikz} + \caption{Example of Planar prediction for a $4\times4$ block.} + \label{Figure2:IntraModePlanar} +\end{figure} + \subsubsection{Sample smoothing} @@ -525,6 +523,15 @@ Once a frame is marked as \emph{unused for reference} it is no longer used for m In \ac{h264} a sliding window approach is used to implicitly manage the reference picture list. This process discards a given frame whenever the \ac{dpb} has the number of defined reference frames (which is controlled by the encoder). Moreover, \ac{h264} uses an explicitly \ac{mmco} to send commands to the decoder and change the normal reference picture management process. +In the \ac{hevc} standard the status of the \ac{dpb} is encoded for every slice, instead of the implicit managing used in the past. +The most fundamental difference in \ac{rpset} concept compared to \ac{mmco}/sliding window of \ac{h264} is that each particular frame has a complete set of the reference frames that are used or will be used. +Thus, a complete set of all frames that must be kept in the \ac{dpb} is explicitly provided. +This is different from \ac{h264} where only relative changes to the \ac{dpb} are encoded. +With the \ac{rpset} concept, no information from earlier frames in decoding order is needed to maintain the correct status of the \ac{dpb}. +Table~\ref{Table2:ReferencePictureListExample} shows the \ac{rpset} for some frames illustrated in Figure~\ref{Figure2:LeadingTralingPics}. +As expected, for the \ac{cra} frame ($I_1$) no information is sent to the decoder because there is not inter-prediction and the \ac{dpb} should be empty. +For the remaining frames, two types of information are encoded: frame number and a binary flag to indicate whether it is currently used. This is required because any frame not listed in a given \ac{rpset} will be discarded by the decoder and then subsequent frames cannot use them as reference. + \begin{table}[tb] \centering \caption{Reference Picture Set example.} @@ -544,16 +551,6 @@ This process discards a given frame whenever the \ac{dpb} has the number of defi \end{tabular} \end{table} -In the \ac{hevc} standard the status of the \ac{dpb} is encoded for every slice, instead of the implicit managing used in the past. -The most fundamental difference in \ac{rpset} concept compared to \ac{mmco}/sliding window of \ac{h264} is that each particular frame has a complete set of the reference frames that are used or will be used. -Thus, a complete set of all frames that must be kept in the \ac{dpb} is explicitly provided. -This is different from \ac{h264} where only relative changes to the \ac{dpb} are encoded. -With the \ac{rpset} concept, no information from earlier frames in decoding order is needed to maintain the correct status of the \ac{dpb}. -Table~\ref{Table2:ReferencePictureListExample} shows the \ac{rpset} for some frames illustrated in Figure~\ref{Figure2:LeadingTralingPics}. -As expected, for the \ac{cra} frame ($I_1$) no information is sent to the decoder because there is not inter-prediction and the \ac{dpb} should be empty. -For the remaining frames, two types of information are encoded: frame number and a binary flag to indicate whether it is currently used. This is required because any frame not listed in a given \ac{rpset} will be discarded by the decoder and then subsequent frames cannot use them as reference. - - The \ac{rpset} concept provides a basic level of error robustness to the reference picture management. In \ac{h264}, whenever a reference frame is lost the decoder can look at the value of the \emph{FrameNum} (incremental value associated with each reference frame) to detect the loss event. However, as gaps in \emph{FrameNum} are allowed, the decoder might not be able to decide whether the missing frame was intentional or resulted from a transmission loss event. @@ -562,16 +559,6 @@ Using the \ac{rpset} concept, the decoder is always able to detect loss events a \subsubsection{Asymmetric block partitions} -\begin{figure}[tb] - \centering - \begin{tabular}{x{.8\columnwidth}x{.1\columnwidth}} - \input{\FigsPath/PUSymetric.tikz} & (a)\\ - \input{\FigsPath/PUAsymetric.tikz} & (b)\\ - \end{tabular} - \caption{Partition types of an inter coded CU into PUs~\cite{HEVCBlock2012}.} - \label{Figure2:InterPuPartitions} -\end{figure} - When inter prediction mode is selected a higher range of partitioning schemes is allowed, thus dividing the \ac{cu} into one, two or four \acs{pu}. The splitting into four \acs{pu} is only allowed when the \ac{cu} size is equal to the minimum. @@ -585,15 +572,18 @@ Finally, the $\frac{N}{2} \times \frac{N}{2}$ case results in four \acs{pu}. The partitions shown in Figure~\ref{Figure2:InterPuPartitions}~(b) correspond to the asymmetric partitioning, which is one of the main improvements introduced in \ac{hevc}, as it allows higher flexibility in motion estimation, leading to higher compression efficiency~\cite{HEVCBlock2012}. These partitions types are only allowed for \acs{cu} with $16\times16$ pixels or larger. +\begin{figure}[tb] + \centering + \begin{tabular}{x{.8\columnwidth}x{.1\columnwidth}} + \input{\FigsPath/PUSymetric.tikz} & (a)\\ + \input{\FigsPath/PUAsymetric.tikz} & (b)\\ + \end{tabular} + \caption{Partition types of an inter coded CU into PUs~\cite{HEVCBlock2012}.} + \label{Figure2:InterPuPartitions} +\end{figure} + \subsubsection{Merge mode} -\begin{figure}[tb] -\centering -\input{\FigsPath/MergeModeHEVC.tikz} -%\input{\FigsPath/HEVC/MergeModeHEVCTwoFrames.tikz} -\caption{Illustration of the MV prediction candidates in the AMVP mode.} -\label{Figure2:MVCandidates} -\end{figure} The Merge mode was introduced in \ac{hevc} as a new technique to derive motion information from spatial and temporal neighbouring blocks~\cite{MotionCodingHEVC2013}. This technique extends the concepts of the Direct and Skip modes used in \ac{h264}~\cite{H264OverviewJul2003}, using a more sophisticated approach with two key differences. Firstly, the Merge Mode allows for more \ac{mv} candidates than the Skip in \ac{h264}. @@ -610,6 +600,13 @@ The Merge Mode defines how to encode the \ac{mv} candidate and the residual info In \ac{hevc}, the Skip mode, i.e., transmitting only the \ac{mv} without residual information, is treated as a special case of the Merge Mode. In this case, only a skip flag is enabled and the corresponding merge index is transmitted. +\begin{figure}[tb] + \centering + \input{\FigsPath/MergeModeHEVC.tikz} + %\input{\FigsPath/HEVC/MergeModeHEVCTwoFrames.tikz} + \caption{Illustration of the MV prediction candidates in the AMVP mode.} + \label{Figure2:MVCandidates} +\end{figure} \subsubsection{Motion vector prediction} diff --git a/022_ErrorResilience/RobustTransmissionReview.tex b/022_ErrorResilience/RobustTransmissionReview.tex index d382fe4..3ecce61 100755 --- a/022_ErrorResilience/RobustTransmissionReview.tex +++ b/022_ErrorResilience/RobustTransmissionReview.tex @@ -95,7 +95,7 @@ Normally, such optimisation is equivalent to the minimisation of the Lagrangian \begin{equation} J = D + \lambda R, \end{equation} -where $D$ is the distortion of all pixels, $R$ the bitrate and $\lambda$ is the Lagrangian multiplier. +where $D$ is the distortion of all pixels, $R$ the bitrate and $\lambda$ is the Lagrangian multiplier~\cite{LagragianMultiplier1982}. Such optimisation only takes into account the coding distortion ($D$), which is only suitable for error-free environments. However, some alternative approaches with error resilient \ac{rd} optimisation take into consideration possible network losses~\cite{DistortionTrackingH264_ICIP2004}. Other advanced approaches can take advantage of a \ac{rope} of the end-to-end distortion at the encoder-side~\cite{ROPEJun2000}. @@ -245,8 +245,6 @@ In \cite{ResilienceDualFrameBuffer2004}, an optimisation algorithm was devised t The same method can also be used in combination with feedback information from the network to improve the references frame selection. In the decoder-side, an \ac{ec} method was later used to take advantage of such prediction structure \cite{ConcealmentDualFrameBuffer2005}. - - A feedback information from the network was also used in \cite{ErrorResilienceRPSFeb2009}, to reduce error propagation at end-user decoders, by dynamically selecting the reference frame, only from those correctly received. Figure~\ref{fig2:ReferencePictureSelection} shows an example of an error resilience transcoding scheme, where the original temporal predictions (represented by the solid arrows) are transcoded to use only correctly received frames as reference, i.e., prediction represented by the dashed arrow. This is done based on network feedback. A more advanced scheme may be considered, where only the regions affected by mismatch predictions are processed in the compressed domain, and subsequently transcoded to limit the dependencies from lost frames. diff --git a/031_ErrorResilienceEvaluation/DynamicTmvp/DynamicTMVP.tikz b/031_ErrorResilienceEvaluation/DynamicTmvp/DynamicTMVP.tikz index aabecdd..8bba7e9 100644 --- a/031_ErrorResilienceEvaluation/DynamicTmvp/DynamicTMVP.tikz +++ b/031_ErrorResilienceEvaluation/DynamicTmvp/DynamicTMVP.tikz @@ -8,7 +8,7 @@ matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} % \draw[draw=black] (0,0) rectangle ++(21.58cm,27.94cm); -\newcommand{\scale}{0.7} +\newcommand{\scale}{0.68} % \newcommand{\TikZyslant}{-0.2} \newcommand{\xslant}{0} \newcommand{\xframeshift}{6.5cm} diff --git a/031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNR.tex b/031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNR.tex index 331e5ee..463271e 100644 --- a/031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNR.tex +++ b/031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNR.tex @@ -11,37 +11,37 @@ & & & \textbf{1\%} & \textbf{5\%} \\ \midrule \multirow{\nlines}{*}{\vbox{Basketball Drill}} - & \TIDR~\cite{DynamicTMVP2011} & -0.011 & 0.092 & 0.365\\ - & \PROP & -0.054 & 0.218 & 0.823\\ + & \TIDR~\cite{DynamicTMVP2011} & -0.011 & 0.09 & 0.37\\ + & \PROP & -0.054 & 0.22 & 0.82\\ % & Without TMVP & -0.094 & 0.574 & 2.165\\ \hline \multirow{\nlines}{*}{\vbox{Book Arrival}} - & \TIDR~\cite{DynamicTMVP2011} & -0.003 & 0.099 & 0.428\\ - & \PROP & -0.012 & 0.359 & 1.436\\ + & \TIDR~\cite{DynamicTMVP2011} & -0.003 & 0.10 & 0.43\\ + & \PROP & -0.012 & 0.36 & 1.44\\ % & Without TMVP & -0.021 & 0.641 & 2.494\\\ \hline \multirow{\nlines}{*}{\vbox{BQSquare}} - & \TIDR~\cite{DynamicTMVP2011} & -0.005 & 0.418 & 1.572\\ - & \PROP & -0.032 & 0.539 & 1.967\\ + & \TIDR~\cite{DynamicTMVP2011} & -0.005 & 0.42 & 1.57\\ + & \PROP & -0.032 & 0.54 & 1.97\\ % & Without TMVP & -0.033 & 1.328 & 4.897\\ \hline \multirow{\nlines}{*}{\vbox{Kendo}} - & \TIDR~\cite{DynamicTMVP2011} & -0.008 & 0.150 & 0.618\\ - & \PROP & -0.031 & 0.380 & 1.539\\ + & \TIDR~\cite{DynamicTMVP2011} & -0.008 & 0.15 & 0.62\\ + & \PROP & -0.031 & 0.38 & 1.54\\ % & Without TMVP & -0.035 & 0.569 & 2.277\\ \hline \multirow{\nlines}{*}{\vbox{Race Horses}} - & \TIDR~\cite{DynamicTMVP2011} & -0.010 & 0.179 & 0.689\\ - & \PROP & -0.040 & 0.236 & 0.909\\ + & \TIDR~\cite{DynamicTMVP2011} & -0.010 & 0.18 & 0.69\\ + & \PROP & -0.040 & 0.24 & 0.91\\ % & Without TMVP & -0.050 & 0.776 & 2.796\\ \hline \multirow{\nlines}{*}{\vbox{Tennis}} - & \TIDR~\cite{DynamicTMVP2011} & -0.003 & 0.099 & 0.494\\ - & \PROP & -0.041 & 0.214 & 0.932\\ + & \TIDR~\cite{DynamicTMVP2011} & -0.003 & 0.10 & 0.49\\ + & \PROP & -0.041 & 0.21 & 0.93\\ % & Without TMVP & -0.035 & 0.599 & 2.488\\ \midrule \multirow{\nlines}{*}{\vbox{Average}} - & \TIDR~\cite{DynamicTMVP2011} & -0.006 & 0.171 & 0.695 \\ - & \PROP & -0.034 & 0.342 & 1.339 \\ + & \TIDR~\cite{DynamicTMVP2011} & -0.006 & 0.17 & 0.70 \\ + & \PROP & -0.034 & 0.34 & 1.34 \\ \bottomrule \end{tabular} diff --git a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex index 0584c5b..e7f7d56 100644 --- a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex +++ b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex @@ -14,8 +14,8 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Introduction} -The use of \ac{hevc} in different type of application and video services over networks that are prone to errors and packet losses, requires mechanisms to minimise the effect o such probabilistic events. This chapter starts by presenting an evaluation study about the error resilience of \ac{hevc} in Section~\ref{Section3:ErrorResilienceEvaluation} , with the aim of defining the weak elements of coded streams and level of error robustness under different transmission loss conditions. The evaluation study covers different coding configuration to evaluate the individual impact of some new coding tools introduced, e.g., increased partition size, intra-refreshing, slice partitioning and advanced motion vector prediction. -In Section~\ref{Section3:DynamicMVRefreshHEVC} a novel method to increase the error robustness of the \ac{hevc} \ac{mv} coding is presented and evaluated against existing techniques. The method proposed introduces refresh point in the motion prediction used in the \ac{hevc}, which significant reduces the impact of mismatch \ac{mv} predictions. +The use of \ac{hevc} in different type of application and video services over networks that are prone to errors and packet losses, requires mechanisms to minimise the effect of such probabilistic events. This chapter starts by presenting an evaluation study about the error resilience of \ac{hevc} in Section~\ref{Section3:ErrorResilienceEvaluation} , with the aim of defining the weak elements of coded streams and the level of error robustness under different transmission loss conditions. The evaluation study covers different coding configuration to evaluate the individual impact of some new coding tools introduced, e.g., increased partition size, intra-refreshing, slice partitioning and advanced motion vector prediction. +In Section~\ref{Section3:DynamicMVRefreshHEVC} a novel method to increase the error robustness of the \ac{hevc} \ac{mv} coding is presented and compared to existing techniques. The proposed method introduces refresh point in the motion prediction used in the \ac{hevc}, which significantly reduces the impact of mismatch \ac{mv} predictions. Finally, Section~\ref{Section3:Summary} concludes the chapter. \clearpage @@ -34,23 +34,23 @@ Although the increased complexity may be compensated with more efficient algorit A critical aspect related to the performance of robust video streaming methods is the error detection accuracy. Since \ac{hevc} streaming is expected to use known transport technologies, such as RTP~\cite{HEVCStreamMay2012}, MPEG-2 TS~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016} and MPEG Media Transport\cite{ATSCMMT2016}, the error detection capabilities of the underlying transport protocols have significant impact on the overall system performance. In general, the error detection operation at the transport layers is decoupled from the video coding layer. -To cope with transmission errors the encoding side implements robust coding methods, while the efficient recovery of lost video data is implemented in the decoder through error concealment algorithms. -Between them, signalling between different protocol layers ensures that the decoder is given the necessary information to identify the lost slices/frames in the video stream. +To cope with transmission errors the encoder implements robust coding methods, while the efficient recovery of lost video data is implemented in the decoder, through error concealment algorithms. +Between them, signalling that cope with different protocol layers ensures that the decoder is given the necessary information to identify the lost slices/frames in the video stream. Therefore, the error robustness of video streams can be investigated on its own, assuming any possible form of streaming technologies and error detection. This is the main focus of the study presented in this section. -There are some studies presented in the past regarding the error robustness of the \ac{hevc} standard which are worth to be mention as they provide relevant insights for the research of carried out in this research work. In~\cite{StudyHEVCErrorsDez2012,HEVCWireless_2016} both the coding efficiency and error robustness of the \ac{hevc} are evaluated and compared against the \ac{h264}. Experimental results confirm that \ac{hevc} has reduced error robustness despite its increased coding efficiency. +There are some studies presented in the past regarding the error robustness of the \ac{hevc} standard, which are worth to be mention as they provide relevant insights for the research of carried out in this research work. In~\cite{StudyHEVCErrorsDez2012,HEVCWireless_2016} both, the coding efficiency and error robustness of the \ac{hevc}, are evaluated and compared against the \ac{h264}. Experimental results confirm that \ac{hevc} has reduced error robustness despite its increased coding efficiency. A subjective evaluation study is presented in~\cite{SubjectiveEvalSep2013,QoEHEVCMay2014} revealing that packet loss ratios higher than $3\%$ have significant effect on the perceived video quality. %In both works it is concluded that the robustness of \ac{hevc} decreases for sequences complex motion. The vulnerability of \ac{mv} prediction was investigated and its impact on the error robustness was evaluated \cite{DynamicTMVP2011}. Moreover, a method to introduce \ac{mv} refresh points was proposed, achieving average \ac{psnr} gains of approximately 3~dB. -Summarising, the decrease in error robustness is mostly due to the strong data dependencies imposed by the highly complex prediction modes of \ac{hevc}. Since these are selected according to a rate distortion optimisation criterion, assuming error-free transmission, the decoder faces a different problem whenever errors or data loss occur in the transmission network \cite{ROPE2007}. +Summarising, the decrease in error robustness is mostly due to the strong data dependencies imposed by the highly complex prediction modes of \ac{hevc}. Since these are selected according to a rate distortion optimisation criterion that assumes error-free transmission, the decoder faces different problems whenever errors or data loss occur in the transmission network \cite{ROPE2007}. -The following sub-sections present an error robustness evaluation of \ac{hevc}, focused on the impact of the novel coding techniques introduced in the standard and extending the generic studies presented in previous works. This study provides relevant insights for research directions and motivation to investigate solutions for the error robustness problem in \ac{hevc}. +The following sub-sections present an error robustness evaluation of \ac{hevc} focused on the impact of the novel coding techniques introduced in the standard, extending the generic studies presented in previous works. This study provides relevant insights for research directions and motivation to investigate solutions for the error robustness problem in \ac{hevc}. The following case-studies are covered in this experimental evaluation: \begin{enumerate} \item different maximum \ac{cu} sizes to evaluate the impact of larger frame partitioning; % \item experiments with and without filter operations, i.e., deblocking filter and \ac{sao}, in order to evaluate their impact on decoding video quality; - \item different distances between intra frames to evaluate the impact of introducing different number of refresh points; + \item different distances between intra frames, to evaluate the impact of introducing different number of refresh points; \item different slice partitioning schemes; \item different \ac{mv} coding schemes using different types of \ac{mv} candidates, with analysis at the spatial and temporal level. \end{enumerate} @@ -60,16 +60,16 @@ The following case-studies are covered in this experimental evaluation: \input{031_ErrorResilienceEvaluation/TablesSequences.tex} -The experimental setup used in this study to evaluate the error robustness of the \ac{hevc} used thirteen test sequences, which were chosen to cover different types of motion and texture complexity. +The experimental setup used in this study to evaluate the error robustness of the \ac{hevc} is based on seventeen test sequences, which were chosen to cover different types of motion and texture complexity. % Each sequence comprises 240 frames and the whole set covers. -Table~\ref{TableTestSequences} presents a summary of the main characteristics of the test sequences. Appendix~\ref{Appen:Sequences} shows the first frame of each test sequence. -The spatial and temporal complexity of these video sequences was measured by using the following metrics \cite{ITUTP9112008}: +Table~\ref{TableTestSequences} presents a summary of the main characteristics of the test sequences. Appendix~\ref{Appen:Sequences} shows a single frame of each test sequence. +The spatial and temporal complexity of these video sequences was measured by using the following metrics \cite{ITUTP9102008}: \begin{itemize} - \item {\ac{si}} based on Sobel filter. Each video frame ($f_t$) is firstly filtered, then the standard deviation is computed. Finally, the maximum value over time is chosen to represent the \ac{si} metric. This process can be represented in the following equation: + \item {\ac{si}} based on Sobel filter. Each video frame ($f_t$) is firstly filtered, then the standard deviation is computed. Finally, the maximum value over time is chosen to represent the \ac{si} metric. This process can be represented by the following equation: \begin{equation} SI = max_{time}\Big\{ std_{space} \big( Sobel\left(f_t\right) \big) \Big\}. \end{equation} - \item {\ac{ti}} based upon the motion difference, which is given by the difference between the pixel values at the same spatial location but in consecutive time instances. \ac{ti} is computed as the maximum over time of the standard deviation over space. More motion in adjacent frames will result in higher values of \ac{ti}. This process can be represented by the following: + \item {\ac{ti}} based upon the motion difference, which is given by the difference between the pixel values at the same spatial location but in consecutive time instances. \ac{ti} is computed as the maximum displacement over time of the standard deviation over space. Larger displacements in adjacent frames will result in higher values of \ac{ti}. This process can be represented by the following: \begin{equation} TI = max_{time}\Big\{ std_{space} \big( f_{t}(x,y) - f_{t-1}(x,y) \big) \Big\}. \end{equation} @@ -79,7 +79,7 @@ The values of \ac{si} and \ac{ti} calculated for the test sequences are shown in For the simulation carried out in this study the HM reference software, version $16.2$, was used \cite{HEVCReferenceSoftware162}. The sequences were encoded with an \ac{idr} period of $16$ frames and all the coding modes enabled, following two recommended configurations: \ac{ld}, i.e., P-frames with 4 references frames, and \ac{ra}, i.e., B-frames with 2 reference frames in each list~\cite{HEVCCommonTests2012}. These prediction structures cover the use of P-frames and hierarchical B-frames~\cite{H264ScalableOverviewSep2007}, which are commonly used in HEVC streaming~\cite{RealTimeHEVC2016}. -In order to achieve high quality images for all coded sequences (i.e., \ac{psnr} around 40 dB) different bitrates were used, as listed in Table~\ref{TableTestSequences}. The filtering operations were disabled at slices boundaries in order to keep the slices self-contained. +In order to achieve high quality images for all coded sequences (i.e., \ac{psnr} around 40 dB) different bitrates were used, as listed in Table~\ref{TableTestSequences}. The filtering operations were disabled at slices boundaries, in order to keep the slices self-contained. Moreover, the ratio of inter-coded blocks of each coded sequence using the default coding parameters is also shown in the third column of the table to illustrate the predictions distribution. @@ -87,11 +87,11 @@ Moreover, the ratio of inter-coded blocks of each coded sequence using the defau % At the decoder-side, error concealment is based on motion compensation from previously decoded frames. % The motion field of a missing frame is obtained applying \ac{mv} extrapolation from the two closest neighbours. % Afterwards, the final \acs{mv} are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}. -To simulate the lossy transmission environment, each frame is encoded into several slices, each one packetized into one \ac{nal} unit, which are then transmitted as the payload of independent packets. +To simulate the lossy transmission environment, each frame is divided into several slices, each one encoded and packetized into one \ac{nal} unit, which are then transmitted as the payload of independent packets. In these simulations a whole packet is lost whenever an error occurs, originating a lost slice. Random packet loss was simulated using a two-state Markov model for \ac{plr} of 1\%, 3\%, 5\%, 10\%. For each test condition, 50 runs were performed and the average \ac{psnr} was measured. -The filtering operations were disabled at slices boundaries in order to keep the slices self-contained. The remaining configurations have been kept at their default values as given in the common test conditions~\cite{HEVCCommonTests2012}. +The remaining configurations have been kept at their default values, as given in the common test conditions~\cite{HEVCCommonTests2012}. %\subsection{Experimental results} @@ -100,9 +100,11 @@ The filtering operations were disabled at slices boundaries in order to keep the %The \ac{hevc} standard introduced new partitioning schemes to allow coding improvements in high resolution videos. These partition structures, named \glsfirst{cu}, are allowed to be as large as $64\times64$ pixels. In the first set of experiments the impact of the maximum \ac{cu} size in the decoded video quality is investigated. In this experiment the maximum \ac{cu} size is set to three different sizes, i.e., 16$\times$16, 32$\times$32 and 64$\times$64 pixels. The lower size is targeting the \ac{mb} size used in the \ac{h264}. -Table~\ref{Table3:Quality_BlockSize} shows the average \ac{psnr} obtained for the three test conditions and for four different packet loss ratios. Results for the error-free scenario shows a consistent quality increase as the maximum \ac{cu} size increases, up to a maximum of 1.41~dB for the Tennis sequence when comparing the maximum \ac{cu} size of $64\times64$ with $16\times16$. -This confirms the principle used by the \ac{hevc} standard, that allowing larger coding units lead to higher coding efficiency, achieving a maximum of 1.41~dB for the Tennis sequence which has the higher \ac{ti} parameter (see Table~\ref{TableTestSequences}). Moreover, it is also noticeable that sequences with higher amount of inter-coded blocks take less advantage of the larger \acs{cu}, by the results of the BQSquare sequence (i.e., highest ratio of inter-coded blocks and lowest quality gain in the error-free case). -Overall, comparing the results for the error-free and data loss cases it is noticeable that similar quality gains are obtained, which are achieved due to the superior coding efficiency of larger \acs{cu}. In most of the cases the quality gains slightly decrease for increasing \acs{plr}. +Table~\ref{Table3:Quality_BlockSize} shows the average \ac{psnr} obtained for the three test conditions and for four different packet loss ratios. Results for the error-free scenario shows a consistent quality increase as the maximum \ac{cu} size increases, +% up to a maximum of 1.41~dB for the Tennis sequence +when comparing the maximum \ac{cu} size of $64\times64$ with $16\times16$. +This confirms the principle used by the \ac{hevc} standard, that allowing larger coding units lead to higher coding efficiency, achieving a maximum of 1.41~dB for the Tennis sequence which has the higher \ac{ti} parameter (see Table~\ref{TableTestSequences}). Moreover, it is also noticeable that sequences with higher amount of inter-coded blocks take less advantage of the larger \acs{cu}, in the results of the BQSquare sequence (i.e., highest ratio of inter-coded blocks and lowest quality gain in the error-free case). +Overall, comparing the results for the error-free and data loss cases, it is noticeable that similar quality gains are obtained, which are achieved due to the superior coding efficiency of larger \acs{cu}. In most of the cases the quality gains slightly decrease for increasing \acs{plr}. Therefore, one may conclude that the error robustness of \ac{hevc} streams is not affected by limiting the maximum \ac{cu} size. %\todo{Complete discussion of partition unit size} @@ -120,7 +122,7 @@ Therefore, one may conclude that the error robustness of \ac{hevc} streams is no \end{table} Figure~\ref{Figure3:BlockPartitionBask} illustrates an example of the frame partitions for a region of the Basketball Drill sequence for two cases: (i) maximum \ac{cu} size of $64\times64$ and (i) maximum \ac{cu} size of $16\times16$. -As shown, in the moving regions, e.g., moving player, the partitions obtained for both cases have a similar pattern, mainly composed of \acs{cu} with small size. Larger \acs{cu} are only used in the floor which has a lower spatial complexity and suffers small changes between frames, therefore, easier to compress with $64\times64$ \acs{cu}. Moreover, those regions can be recovered more accurately using neighbouring information (spatial and temporal) correctly decoded, thus not incurring in severe error propagation. +As shown, in the moving regions, e.g., moving player, the partitions obtained for both cases have a similar pattern, mainly composed of \acs{cu} with small size. Larger \acs{cu} are only used in the floor, which has a lower spatial complexity and suffers small changes between frames, therefore, easier to compress with $64\times64$ \acs{cu}. Moreover, those regions can be recovered more accurately using neighbouring information correctly decoded (spatial and temporal), thus not incurring in severe error propagation. Therefore, the overall error robustness is not affected by reducing the maximum \ac{cu} size. One may conclude that the use of larger coding units is able to improve the overall \ac{hevc} efficiency in error-free and error prone applications. \begin{figure}[!t] @@ -171,19 +173,21 @@ This allows the decoder to recover from frame/slice loss and continue error-free Figure~\ref{Figure3:QualityvsIntra} shows the average decoded video quality (\ac{psnr}) for different distance of intra frames, covering \acs{plr} between 3\% and 5\%. Results in the figure indicate that as the distance between intra frames increases, the decoded video quality significantly decreases. The quality decreasing is especially noticeable for intra periods higher than 32 frames, indicating that temporal distances greater than approximately one second are not suitable for video streaming under error-prone conditions. -When a high temporal distance between intra frames is used, the quality decreasing is more noticeable in the People On Street and BQSquare sequences, which have higher spatial complexity (see values for \ac{si}), leading to a reduction of 13 dB and 14 dB, respectively. As spatial complexity increases the encoder rely more on temporal predictions making the bitstream more prone to error propagation. +When a high temporal distance between intra frames is used, the quality decreasing is more noticeable in the People On Street and BQSquare sequences, which have higher spatial complexity (see values for \ac{si}), leading to a reduction of 13 dB and 14 dB, respectively. As spatial complexity increases, the encoder relies more on temporal predictions, which increase the error propagation across temporal adjacent frames, reducing the video quality under errors. + %\todo{Complete discussion of intra refreshing} One should also note from the results of Figure~\ref{Figure3:QualityvsIntra} that using a small distance between intra frames is not always more efficient. Since intra frames require higher amount of bits, the overall coding efficiency is reduced, resulting in an overall quality that decreases even in data loss conditions. This is clear noticeable for the Four People sequence, where increasing the intra period from 8 to 16 leads to a quality increasing of 1 dB for 5\% of \ac{plr}. -For the remaining sequences it can be concluded that using sixteen frames (not higher than half a second) between refresh points (i.e., intra frames) achieves an acceptable efficiency under errors. This provides an efficient trade-off between compression efficiency and error robustness in case of data loss. +For the remaining sequences it can be concluded that using sixteen frames (not higher than half a second) between refresh points (i.e., intra frames) achieves an acceptable efficiency under errors. This provides a good trade-off between compression efficiency and error robustness in case of data loss. \subsection{Evaluation for using different slice partitioning schemes} -The partitioning of the \acs{ctu} into different slices increase the flexibility of video streams to be partitioned into different packets, thus adapting them to the video transmission system. -By partitioning the coded video frame into different slices spatial refresh points are introduced, limiting the error propagation across different \acs{ctu}. -In this sub-section the amount of bytes per slice is evaluated, in order to evaluate the influence of different schemes on the error robustness of \ac{hevc} streams. The quality obtained different mode is compared in order to find the most efficient that suits for video transmission under data loss conditions. +The partitioning of the frame into different slices increase the flexibility of video streams to be divided into different packets, thus adapting them to the video transmission system. +By partitioning the video frame into different slices spatial refresh points are introduced, limiting the error propagation across different \acs{ctu}. +In this sub-section the amount of bytes per slice is evaluated, in order to study the influence of different schemes on the error robustness of \ac{hevc} streams. +A comparison of the quality obtained for different modes is carried out, in order to find the most efficient mode for video transmission under data loss conditions. \begin{figure}[!t] \centering @@ -195,22 +199,22 @@ In this sub-section the amount of bytes per slice is evaluated, in order to eval \end{figure} Figure~\ref{Figure3:SlicePartitioning} shows the average \ac{psnr} for four difference slice partitioning modes and different \acs{plr}. -The first three modes are characterised by a different amount of bytes per slice (B/S), namely 1200, 2400 and 4800 bytes, respectively. In these cases each frame has a variable number of slices. +The first three modes are characterised by a different amount of bytes per slice (B/S), namely 1200, 2400 and 4800 bytes. In these cases each frame has a variable number of slices. The last mode uses a single slice per frame (referred to as \textit{SS}), corresponding to the case where slice partitioning is disabled. -Results in the figure reveals a clear impact of the slice partitioning mode on the error robustness of the \ac{hevc}, as the quality degradation for increasing \ac{plr} significantly changes for each mode. +Results in Figure~\ref{Figure3:SlicePartitioning} reveals a clear impact of the slice partitioning mode on the error robustness of the \ac{hevc}, as the quality degradation for increasing \ac{plr} significantly changes for each mode. Comparing the cases with fixed amount of bytes per slice, it is clearly noticeable that when higher amount of bytes are used the quality degradation increases. This is true for the results of most of the test sequences shown in Figure~\ref{Figure3:SlicePartitioning}. -Although the use of smaller slices increases the overhead used for slice headers, the superior error robustness of achieved with 1200 B/S is able to overcome the inherent degradation on coding efficiency. -This is due to the lack of refresh points within a frame, resulting in errors that affects larger regions of the frame which are propagated to the subsequent inter predicted frames. +Although the use of smaller slices increases the overhead used for slice headers, the superior error robustness achieved with 1200 B/S is able to overcome the inherent degradation on coding efficiency. +This is due to the lack of refresh points within a frame, resulting in errors that affects larger regions of the frame, which are propagated to the subsequent inter predicted frames. -The results for of \textit{SS} (slice partitioning disabled) also reveals that this approach also leads to lower error robustness when comparing with those cases where the frames are divided into multiple slices. Although higher coding efficiency is achieved in this mode the lower error robustness leads to an overall poor performance under frame or slice loss conditions. -Results for Kendo sequence does not follow the main trend. This can be explained by the low bitrate required to compress the video signal, which is the lowest comparing with the other test sequences (see \ref{TableTestSequences}). Therefore, most of the inter-coded frames are able to transmitted using a single slice even with 1200 bytes per slice, resulting in a similar partitioning for all tested modes, and thus similar error robustness. In this case, the higher coding efficiency achieved by the \textit{SS} case consequently leads to higher quality in case of errors. +The results for of \textit{SS} (slice partitioning disabled) also reveals that this approach also leads to lower error robustness when comparing with those cases where the frames are divided into multiple slices. Although higher coding efficiency is achieved in this mode, the lower error robustness leads to an overall poor performance under frame or slice loss conditions. +Results for Kendo sequence does not follow the main trend. This can be explained by the low bitrate required to compress the video signal, which is the lowest when compared with the other test sequences (see Table~\ref{TableTestSequences}). Therefore, most of the inter-coded frames can be transmitted using a single slice, even with 1200 bytes per slice, resulting in a similar partitioning for all tested modes, and thus similar error robustness. In this case, the higher coding efficiency achieved by the \textit{SS} case consequently leads to higher quality in case of errors. %\todo{Complete discussion of slice partitioning} \subsection{Motion vector coding impact on error propagation} -In \ac{hevc}, the new \ac{mv} prediction modes known as \ac{amvp} and Merge Mode~\cite{MotionCodingHEVC2011} may be significant contributors to poor performance under transmission errors. These modes are characterised by coding the motion information using previously encoded \acs{mv}, which obviously leads to prediction mismatches at the decoder-side when such neighbouring information is missing due to network losses. +In \ac{hevc}, the new \ac{mv} prediction modes known as \ac{amvp} and Merge Mode~\cite{MotionCodingHEVC2011}, may be significant contributors to the poor performance under transmission errors. These modes are characterised by coding the motion information using previously encoded \acs{mv}, which obviously leads to prediction mismatches at the decoder-side when such neighbouring information is missing due to network losses. Therefore, in this sub-section the impact of using spatial and temporal \ac{mv} candidates on the decoded video quality under errors is evaluated. \subsubsection{Spatial \ac{mv} candidates} @@ -229,7 +233,7 @@ Figure~\ref{Figure3:AvgPSNRMergeCandidates} shows the average quality results fo The results show that, besides the number of \ac{mv} candidates, similar quality decreasing is achieved as the \ac{plr} increases. Therefore, one may conclude that varying the number of available \ac{mv} candidate does not affect the error resilience of the \ac{hevc}. The occurrence rate for each spatial \ac{mv} candidate was analysed in order to check whether they are uniformly used. -Table~\ref{Table3:MergeIdxRatio} shows the usage percentage of each \ac{mv} candidate when three and five candidates are available, for the slice partitioning mode of 1200 bytes/slice. The results indicate that most predictions use the first candidate and minor differences occur by adding 2 or more \acs{mv}. +Table~\ref{Table3:MergeIdxRatio} shows the usage percentage of each \ac{mv} candidate when three and five candidates are available for the slice partitioning mode of 1200 bytes/slice. The results indicate that most predictions use the first candidate and minor differences occur by adding 2 or more \acs{mv}. Since the mostly used \ac{mv} candidates are the initial ones, changing the number of available candidates does not significantly affects the error resilience. Moreover, the slice partitioning mode used only 1200 bytes per slice, which introduces refresh points in each frame, reducing the spatial error propagation due to mismatched \ac{mv} predictions. @@ -278,11 +282,12 @@ In order to further evaluate the impact of the temporal \ac{mv} dependencies on Table~\ref{Table3:AvgPSNRWithoutTMVP} presents the average quality results, measured using the \ac{psnr}. As expected, the results indicate that in error-free conditions higher quality is achieved by enabling the \ac{tmvp} candidate, since higher compression efficiency is obtained because more \ac{mv} predictors are used. This is true for all both test conditions, \emph{i.e.}, both \ac{ld} and \ac{ra}. -However, in the presence of errors, the video quality significantly decreases due to low error robustness when temporal \ac{mv} candidates are used. Alternatively, average quality up to 5.17 dB of \ac{psnr} when the temporal \ac{mv} predictors are disabled is achieved in the \ac{ld} configuration. +However, in the presence of errors, the video quality significantly decreases due to low error robustness when temporal \ac{mv} candidates are used. +Alternatively, average quality up to 5.17 dB of \ac{psnr} is achieved when the temporal \ac{mv} predictors are disabled in the \ac{ld} configuration. The results also show that the negative impact of using temporal \ac{mv} dependencies is smaller when B-frames are used. For the Random-Access, the average gain of disabling the \ac{tmvp} is 0.78 dB. %This also confirms the previous results of Figure~\ref{Figure3:PSNRvsFrameErrorFreeConc}. A higher negative impact is obtained when only P-frames are used (\ac{ld} configuration), since subsequent frames are temporally closer, leading to more accurate temporal \ac{mv} candidates. -Moreover, the results also indicate that for higher loss ratios the negative impact of using the \ac{tmvp} does not increase as both configuration are subject to high quality degradation. +Moreover, the results also indicate that for higher loss ratios the negative impact of using the \ac{tmvp} does not increase, as both configuration are subject to high quality degradation. \begin{table}[!t] \centering @@ -294,15 +299,15 @@ Moreover, the results also indicate that for higher loss ratios the negative imp \subsection{Remarks} -Overall, the results presented in the previous sub-sections show that \ac{hevc} has poor robustness against transmission errors, resulting in significant quality degradation, though dependent still on the video content and coding options. +Overall, the results presented in the previous sub-sections show that \ac{hevc} has poor robustness against transmission errors, resulting in significant quality degradation, though depending on the video content and coding options. As shown, some coding options, such as slice partition and intra refreshing period, have high correlation with the decoded video quality when transmission errors affect the compressed streams. Therefore, these are important aspects to take into consideration in the design an implementation of broadcasting video services and applications. -One of the main improvements of the \ac{hevc} standard is the support for larger coding units up to $64\times64$, which does not have a negative impact the error robustness of coded video streams, indicating that they should be allowed to achieve higher coding performance. +One of the main improvements of the \ac{hevc} standard is the support for larger coding units up to $64\times64$, which does not have a negative impact on the error robustness of coded video streams, indicating that they should be allowed to achieve higher coding performance. Regarding the use of neighbouring information for \ac{mv} coding, on the one hand the spatial \ac{mv} candidates do not significantly decrease the error robustness. On the other hand, the use of temporal \ac{mv} candidates significantly contribute to increase the temporal dependencies, leading to more severe quality degradation in the presence of packet loss. Based on these evidences, the next section presents an efficient method to improve the error robustness of \ac{hevc}. -The proposed method is based on the underlying principle of reducing the mismatch of \ac{mv} predictions, by selectively disable some \ac{mv} predictions at the encoder-side, reducing the overall temporal dependencies. +The proposed method is based on the underlying principle of reducing the mismatch of \ac{mv} predictions, by selectively disabling some \ac{mv} predictions at the encoder-side, reducing the overall temporal dependencies. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -321,12 +326,12 @@ Based on this evidence, this section proposes a method to make intelligent use o \glsreset{rd} The new approach to deal with temporal motion vector dependencies is accomplished by modifying the \ac{rd} optimisation that is used to select the best motion vector for inter-prediction. -The objective is to increase the number of \acs{mv} that can be independently decoded to improve the error recovery capability after a frame loss event. +The objective is to increase the number of \acs{mv} that can be independently decoded, in order to improve the error recovery capability after a frame loss event. The proposed method also overcomes the limitation of the method presented in \cite{DynamicTMVP2011}, which break the motion information dependencies between frames. Instead of disabling the temporal motion candidate at the frame level, the proposed method aims to remove the dependencies at the \ac{cu} level. To achieve this goal, the temporal dependency of all \ac{mv} candidates is firstly analysed for each block, in order to select on the suitable candidates to be used as predictors for the current \ac{mv}. -Based on the analysis of the temporal dependency, some temporal candidates are marked as unsuitable predictors of the current \ac{mv}, and for this reason are not selected in the Merge mode. In the proposed approach, a temporal \ac{mv} candidate is marked as unsuitable based on the following criteria: +Based on the analysis of the temporal dependency, some temporal candidates are marked as unsuitable predictors of the current \ac{mv}, and for this reason they are not selected in for \ac{mv} prediction. In the proposed approach, a temporal \ac{mv} candidate is marked as unsuitable, based on the following criteria: \begin{enumerate} \item if it is encoded based on another temporal \ac{mv} candidate from a previously encoded frame; \item if it is predicted using a spatial neighbour that was previously encoded using a temporal \ac{mv} candidate. @@ -357,15 +362,16 @@ Moreover, the block corresponding to the arrow (3) cannot use the temporal \ac{m \end{figure} -The performance of the proposed scheme (\PROP) was evaluated against two other approaches: reference \ac{hevc} with TMVP enabled and the \TIDR method as proposed in \cite{DynamicTMVP2011}. To evaluate the proposed method the same experimental setup as presented in Section \ref{Section3:ErrorResilienceEvaluation} was used. +The performance of the proposed scheme (\PROP) was evaluated against two other approaches: reference \ac{hevc} with TMVP enabled and the \TIDR method proposed in \cite{DynamicTMVP2011}. +% To evaluate the proposed method, the same experimental setup as presented in Section \ref{Section3:ErrorResilienceEvaluation} was used. \subsubsection{Error propagation under single loss events} Firstly, the performance of the proposed method was measured under a single loss event (i.e., only one frame is affected by errors). Figure \ref{Figure3:ProposedMethodEvaluationSingle} shows the error propagation for the different methods. -The results show a significant quality difference between the proposed method, the \TIDR method, and the reference \ac{hevc}. -When the temporal \ac{mv} candidate is disabled (proposed and \TIDR method) more error robustness can be achieved. -The results also show the relevance of using the proposed method, as the \TIDR method is only able to recover from the frame loss after the refreshing frame, \emph{i.e.}, frame \#8, especially for the BQSquare sequence +The results show a significant quality difference between the proposed method, the \TIDR method and the reference \ac{hevc}. +When the temporal \ac{mv} candidate is disabled (proposed and \TIDR method) a higher error robustness is achieved. +The results also show the relevance of using the proposed method, as the \TIDR method is only able to recover from the frame loss after the refreshing frame, \emph{i.e.}, frame \#8, especially for the BQSquare sequence. \subsubsection{Quality evaluation under random errors} @@ -380,8 +386,8 @@ The results also show the relevance of using the proposed method, as the \TIDR m Secondly, further simulations were performed to evaluate the effectiveness of the proposed error resiliency method under realistic packet loss patterns. %As in previous experiments, each coded frame is transmitted in one packet. Table~\ref{Table3:AverageStdPSNR} illustrates the average standard deviation of the PSNR, for three of the test sequences. -The results show that increasing the \ac{plr} leads to higher quality deviations, and the impact of errors in the subjective quality experience is higher. -However, reducing the temporal dependencies of the \acs{mv} using the proposed method decreases the quality differences variation, achieving the lowest value for all scenarios tested. The proposed method is able to reduce the quality difference variations up to 0.94 dB in Kendo sequence (3$\%$ of packet loss) compared to the~\TIDR~method. +The results show that increasing the \ac{plr} leads to higher quality deviations and that the impact of errors in the subjective quality experience is higher. +However, reducing the temporal dependencies of the \acs{mv} using the proposed method decreases the quality differences variation, achieving the lowest value for all tested scenarios. The proposed method is able to reduce the quality difference variations up to 0.94 dB for Kendo sequence (3$\%$ of packet loss) compared to the~\TIDR~method. Since the proposed method achieves the lowest \ac{psnr} standard deviation, one may conclude that it is able to reduce the quality variations leading to a better visual quality. %\HIGHLIGHT{vantagem do std} @@ -395,9 +401,11 @@ Since the proposed method achieves the lowest \ac{psnr} standard deviation, one Table~\ref{Table3:AverageResultsProposed} shows the Bjontegaard's average \ac{psnr} increase \cite{BJResults2001}. The results are obtained taking as reference the standard \ac{hevc} encoder. % The results are obtained taking as reference the standard \ac{hevc} encoder, in order to evaluate the quality difference when temporal \ac{mv} candidates are disabled in both error free and under error-prone conditions. The results presented in Table~\ref{Table3:AverageResultsProposed} show that the proposed method presents practically the same rate-distortion performance when compared to \TIDR method. -In general, the average quality, in an error-free scenario, is only decreased by 0.028 dB in comparison to the \TIDR method and by 0.034 dB in comparison to the reference encoder. +In general, the average quality, in an error-free scenario, only decreases 0.028 dB in comparison to the \TIDR method and 0.034 dB in comparison to the reference encoder. However, the proposed method widely outperforms the reference \TIDR technique~\cite{DynamicTMVP2011} in terms of error resiliency performance, achieving in average higher overall quality under error-prone conditions. -The effectiveness of the proposed method increases at higher packet loss rates (\emph{e.g.},~$5\%$), achieving an average gain of 0.573 dB, and a maximum gain of 1 dB (PSNR) for the Book Arrival sequence, when compared to the \TIDR technique. Higher gains are achieved when compared with the reference \ac{hevc} encoder, as shown in Table~\ref{Table3:AverageResultsProposed}. +The effectiveness of the proposed method increases at higher packet loss rates (\emph{e.g.},~$5\%$), achieving an average gain of 0.64 dB ($1.34 - 0.70$), and a maximum gain of 1 dB ($1.44-0.43$) for the Book Arrival sequence , when compared to the \TIDR technique. Higher gains are achieved when compared with the reference \ac{hevc} encoder, as shown in Table~\ref{Table3:AverageResultsProposed}. + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % SECTION @@ -406,7 +414,7 @@ The effectiveness of the proposed method increases at higher packet loss rates ( \label{Section3:Summary} In this chapter, the error resilience performance of \ac{hevc} was studied in detail under error prone conditions. -The simulation results revealed that new coding tools introduced in \ac{hevc} to improve coding efficiency are also responsible for higher error propagation. +The simulation results revealed that new coding tools introduced in \ac{hevc} to improve coding efficiency are partially responsible for higher error propagation. This motivates further research on new techniques to be developed for highly efficient encoders and decoders, in order to increase error resiliency and to improve the overall performance of video broadcast. Then, a new approach to deal with the weak robustness of \ac{mv} coding was proposed, addressing \ac{mv} dependencies at \ac{cu} level. The proposed method selectively disables the temporal \ac{mv} candidates, in order to break the dependencies across more than one frame. The experimental evaluation was able to demonstrate that the proposed method increases the error robustness and outperforms existing state-of-the-art methods and the reference codec. diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/TableMVPs.tex b/031_ErrorResilienceEvaluation/QualityEvaluation/TableMVPs.tex index 31fd01a..dba2c2a 100644 --- a/031_ErrorResilienceEvaluation/QualityEvaluation/TableMVPs.tex +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/TableMVPs.tex @@ -130,7 +130,7 @@ \multirow{2}{*}{\centering\textbf{Sequence}} & \multirow{2}{*}{\vbox{\centering\textbf{TMVP}}} & \multirow{2}{*}{\vbox{\centering\textbf{Error Free}}} & \multicolumn{3}{c}{\textbf{\TABLEPLRLABEL}} \\ & & & \textbf{1\%} & \textbf{5\%} & \textbf{10\%} \\ \hline - \multicolumn{6}{c}{Low-Delay configuration} \\ + \multicolumn{6}{c}{Low-delay configuration (LD)} \\ \hline \multirow{\nlines}{*}{\vbox{Basket Drill}} & Enabled & 38.69 & 34.73 & 27.92 & 25.41 \\ @@ -170,7 +170,7 @@ \multicolumn{2}{c|}{\textbf{Average difference}} & \textbf{-0.04} & \textbf{+3.04} & \textbf{+5.32} & \textbf{+5.17} \\ \midrule \midrule - \multicolumn{6}{c}{Random-Access configuration} \\ + \multicolumn{6}{c}{Random-access configuration (RA)} \\ \hline \multirow{\nlines}{*}{\vbox{Basket Drill}} & Enabled & 39.48 &36.84 & 30.50 & 27.15 \\ diff --git a/031_ErrorResilienceEvaluation/TablesSequences.tex b/031_ErrorResilienceEvaluation/TablesSequences.tex index c7d7d45..69bcd42 100644 --- a/031_ErrorResilienceEvaluation/TablesSequences.tex +++ b/031_ErrorResilienceEvaluation/TablesSequences.tex @@ -41,6 +41,7 @@ {Kimono \newline $1920\times1080@24Hz$} & 23.4 / 32.5 & 83.0 & 8.00 & Capture of a women moving in a forest with moderate motion \\ \hline {Kristen and Sara \newline $1280\times720@60Hz$} & 25.6 / 6.30 & 93.1 & 15.00 & Two persons talking to each other with moderate motion \\ \hline {Park Scene \newline $1920\times1080@24Hz$} & 31.3 / 11.6 & 92.0 & 10.00 & Moderate motion with cyclists passing across the scene \\ \hline +{Party Scene \newline $832\times480@50Hz$} & 52.8 / 11.4 & 89.8 & 4.50 & \\ \hline {People on Street \newline $2560\times1600@24Hz$} & 40.0 / 25.4 & 87.4 & 12.00 & With point capture of people moving; high motion and texture complexity \\ \hline {Race Horses \newline $832\times480@30Hz$} & 43.7 / 24.4 & 79.7 & 8.00 & High motion with several horse riders \\ \hline {Tennis \newline $1920\times1080@30Hz$} & 20.3 / 45.3 & 76.2 & 5.00 & High motion with one moving person in the scene \\ \hline diff --git a/041_ImprovedErrorResilience/ImprovedErrorResilience.tex b/041_ImprovedErrorResilience/ImprovedErrorResilience.tex index eb6c3ca..71d9e0d 100644 --- a/041_ImprovedErrorResilience/ImprovedErrorResilience.tex +++ b/041_ImprovedErrorResilience/ImprovedErrorResilience.tex @@ -3,7 +3,7 @@ \graphicspath{{\ImprovedErrorResilience/Results/}} -\chapter{A Two-stage Approach for Robust Coding and Streaming \VERSION{1}} +\chapter{A Two-stage Approach for Robust Coding and Streaming \VERSION{2}} % \chapter{Error Resilience Techniques for \ac{hevc} Standard} %\chapter{Two-stage Robust Video Coding} \label{Chapter:ErrorResilienceTechniques} @@ -13,6 +13,8 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Introduction} +\glsreset{pu} +\glsreset{ctu} \glsreset{mv} \glsreset{rd} @@ -20,10 +22,10 @@ \glsreset{ld} %The increased compression ratios achieved by the \ac{hevc} standard lead to reduced robustness of coded streams, with increased susceptibility to network errors and consequent video quality degradation, as shown in Chapter~\ref{Chapter:HEVCErrorResilience}. -Therefore, this chapter proposes a coding method, based on a two-stage approach, to improve the error robustness of \ac{hevc} streaming, by reducing temporal error propagation in case of frame loss. -This is accomplished by reducing the prediction mismatch that occurs at the decoder after frame loss through the following two stages: (i) at the encoding stage, the reference pictures are dynamically selected based on constraining conditions and Lagrangian optimisation, which distributes the use of reference pictures throughout each frame, by reducing the number of \acs{pu} that depend on a single reference; (ii) at the streaming stage, a \ac{mv} prioritisation algorithm, based on spatial dependencies, selects an optimal sub-set of \acs{mv} to be transmitted, redundantly, as side information to reduce mismatched \ac{mv} predictions at the decoder. +This chapter proposes a coding method, based on a two-stage approach, to improve the error robustness of \ac{hevc} streaming, by reducing temporal error propagation in case of frame loss. +This is accomplished by reducing the prediction mismatch that occurs at the decoder after frame loss, through the following two stages: (i) at the encoding stage, the reference pictures are dynamically selected based on constraining conditions and Lagrangian optimisation, which distributes the use of reference pictures throughout each frame, by reducing the number of \acs{pu} that depend on a single reference; (ii) at the streaming stage, a \ac{mv} prioritisation algorithm, based on spatial dependencies, selects an optimal sub-set of \acs{mv} to be transmitted, redundantly, as side information to reduce mismatched \ac{mv} predictions at the decoder. -This chapter is organized as follows. Section~\ref{Section4:Problem} presents the problem definition and motivation for the method proposed in this chapter. Section~\ref{Section4:Architecture} provides an overview of the proposed method, describing the functional blocks that compose the two-stage approach for robust coding and streaming. Section~\ref{Section4:RefPicSel} describes the encoding stage of the proposed method, that is used to select the optimal reference picture for inter-prediction. Section~\ref{Section4:MVSelection} describes the streaming stage used to select a sub-set of \acs{mv} to reduce the mismatch motion predictions. Section~\ref{Section4:Performance} provides a performance evaluation of the proposed method, illustrating the advantages of each stage and how they are combined to achieve superior error robustness in \ac{hevc} standard. Finally, Section~\ref{Section4:Summary} concludes the chapter. +This chapter is organized as follows. Section~\ref{Section4:Problem} presents the problem definition and motivation for the method proposed in this chapter. Section~\ref{Section4:Architecture} provides an overview of the proposed method, describing the functional blocks that compose the two-stage approach for robust coding and streaming. Section~\ref{Section4:RefPicSel} describes the encoding stage of the proposed method, that is used to select the optimal reference picture for inter-prediction. Section~\ref{Section4:MVSelection} describes the streaming stage where a sub-set of \acs{mv} are selected to reduce the mismatch motion predictions. Section~\ref{Section4:Performance} provides a performance evaluation of the proposed method, illustrating the advantages of each stage and how they are combined to achieve superior error robustness in \ac{hevc} standard. Finally, Section~\ref{Section4:Summary} concludes the chapter. \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -33,8 +35,8 @@ This chapter is organized as follows. Section~\ref{Section4:Problem} presents th \label{Section4:Problem} -As mentioned before \ac{hevc} bitstreams subject to transmission losses result in significant degradation of both objective and subjective quality. -This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes, which are selected according to a \ac{rd} optimisation criteria, assuming error-free transmission~\cite{ROPE2007}. +As previously mentioned,\ac{hevc} bitstreams subject to transmission losses result in significant degradation of both objective and subjective quality. +This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes, which are selected according to a \ac{rd} optimisation criteria that assumes error-free transmission~\cite{ROPE2007}. Therefore, high compression efficiency is achieved, but resilience to transmission errors is penalised. \begin{figure}[tb] @@ -44,20 +46,21 @@ Therefore, high compression efficiency is achieved, but resilience to transmissi \label{FIG4:ReferenceUsageRatio} \end{figure} -Highly efficient video encoders support the use of multiple reference frames for inter-frame prediction, which are selected based on a \ac{rd} cost which may lead to an unbalance distribution. -In order to show evaluate the unbalanced is the reference frame selection attained by the \ac{rd} implemented in the reference \ac{hevc} encoder, different test sequences are encoded using two reference frames, referred to as $r_1$ and $r_2$, and its selection ratios are analysed. In this study, reference $r_1$ corresponds to the closest neighbouring frame and reference $r_2$ the second closest. +Highly efficient video encoders support the use of multiple reference frames for inter-frame prediction, which are selected based on a \ac{rd} cost that may lead to an unbalance distribution. +In order to evaluate the reference frame selection method attained by the \ac{rd} implemented in the reference \ac{hevc} encoder, different test sequences are encoded using two reference frames, referred to as $r_1$ and $r_2$, and its selection ratios are analysed. In this study, reference $r_1$ corresponds to the closest neighbouring frame and reference $r_2$ the second closest. Figure~\ref{FIG4:ReferenceUsageRatio} illustrates the average usage percentage of reference $r_1$ for different sequences. -The remaining corresponds to those cases where reference $r_2$ were selected. +The remaining corresponds to those cases where reference $r_2$ is selected. These results show that, in general, reference $r_1$ is selected more often than $r_2$. -This is mainly true for sequences with higher motion. For those sequences with lower motion but high spatial complexity, e.g., BQSquare (see \ac{si} in Table~\ref{TableTestSequences}), both reference frames equally (approximately) selected for temporal predictions, with $r_1$ being used for 58\% of cases and $r_2$ for 42\%. -This reveals that even with a configuration with two references the encoder will mainly use the temporally closer one as the reference for inter-frame predictions. +This is mainly true for sequences with higher motion. +For those sequences with lower motion, but high spatial complexity, e.g., BQSquare (see \ac{si} in Table~\ref{TableTestSequences}), both reference frames are equally (approximately) selected for temporal predictions, with $r_1$ being used by 58\% of cases and $r_2$ by 42\%. +This reveals that even with a configuration using only two references the encoder mainly uses the temporally closer one as the reference for inter-frame predictions. Therefore, if such reference frame is lost, a higher number of mismatched predictions will occur, inevitabely leading to higher error propagation. - Another technique that contributes to increased error propagation is the use of predictions in \ac{mv} coding~\cite{MotionCodingHEVC2011}, which contributes to increase the spatial and temporal dependencies. Although the spatial \ac{mv} candidates are used more often, temporally predicted \acs{mv} are responsible for a greater impact on error propagation, as they propagate errors in the \ac{mv} decoding across different time instances. -To overcome these issues, in the next sections a two-fold approach is described to increase the error robustness in video streaming by jointly considering two different stages: (i) while encoding live video, by selecting robust coding modes and (ii) after encoding, by adding robust features to the compressed streams. While in the former this is accomplished by optimising coding parameters and decisions to increase robustness, in the latter some type of compressed domain processing is used to increase error robustness of pre-encoded streams, e.g., error resilient transcoding. Inevitably, in both cases this is achieved at the expense of some loss in coding efficiency, therefore the challenge is to find an optimum trade-off between robustness and bitrate overhead. In this chapter a two-fold approach is used to overcome the two problems mentioned above. +To overcome these issues, in the next sections a two-fold approach is described to increase the error robustness in video streaming by jointly considering two different stages: (i) while encoding live video, by selecting robust coding modes and (ii) after encoding, by adding robust features to the compressed streams. While in the former this is accomplished by optimising coding parameters and decisions to increase robustness, in the latter some type of compressed domain processing is used to increase error robustness of pre-encoded streams, e.g., error resilient transcoding. Inevitably, in both cases this is achieved at the expense of some loss in coding efficiency, therefore the challenge is to find an optimum trade-off between robustness and bitrate overhead. +%In this chapter a two-fold approach is used to overcome the two problems mentioned above. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% @@ -74,9 +77,10 @@ D_t = \tilde f_t - \hat f_t, \text{when } \tilde f_{t-1} \neq \hat f_{t-1}, where $\hat f_t$ and $\tilde f_t$ are the encoded and decoded frames, respectively, at instant $t$. Figure \ref{fig:proposedMethod} illustrates the block diagram of the two-stage architecture proposed in this work. -At the encoding stage, a reference frame selection mechanism is used for each \ac{pu} within a coding tree unit (\ac{ctu}) forcing the use of different reference frames in adjacent \acs{ctu}. The decision criterion contributes to reduce the amount of predictions from a single reference frame, and forces to interleave different reference frames along all encoded \acs{pu}, based on \ac{rd} optimisation. +At the encoding stage, a reference frame selection mechanism is used for each \ac{pu} within a \ac{ctu}, by forcing the use of different reference frames for adjacent \acs{ctu}. +The decision criterion contributes to reduce the amount of predictions from a single reference frame and forces to interleave different reference frames along all encoded \acs{pu}, based on \ac{rd} optimisation. At the streaming stage, the coded stream is parsed to extract the \ac{mv} information, which allows ranking \acs{mv} according to a relevance criterion. Then, a sub-set of these \acs{mv} is selected to be transmitted as side information. The amount of selected \acs{mv} is not fixed and can be optimised for different content and network characteristics. -Note that the streaming stage is an independent and complementary subsystem to the encoder. Thus, it is able to operate at network nodes, including streaming servers delivering pre-encoded streams. +Note that the streaming stage is an independent and complementary subsystem of the encoder. Thus, it is able to operate at network nodes, including streaming servers delivering pre-encoded streams. \begin{figure}[tb] \centering @@ -86,8 +90,8 @@ Note that the streaming stage is an independent and complementary subsystem to t \label{fig:proposedMethod} \end{figure} -In case of frame loss these two stages have the common objective of helping the decoder to recover from mismatched predictions at the expense of a small increase in bitrate. -This is equivalent to reducing the error propagation by using two techniques: (i) reducing the mismatch of inter-frame predictions, by encoding each \ac{pu} from a different reference frame and (ii) reducing the mismatch of \ac{mv} predictions by sending the most important \acs{mv} as side information. These are the two stages of the proposed method described in the following sections. +In case of frame loss, these two stages have the common objective of helping the decoder to recover from mismatched predictions at the expense of a small increase in bitrate. +This is equivalent to reduce the error propagation by using two techniques: (i) reducing the mismatch of inter-frame predictions, by encoding each \ac{pu} from a different reference frame and (ii) reducing the mismatch of \ac{mv} predictions by sending the most important \acs{mv} as side information. These are the two stages of the proposed method described in the following sections. @@ -96,14 +100,14 @@ This is equivalent to reducing the error propagation by using two techniques: (i The reference picture selection mechanism attempts to uniformly distribute the use of different reference pictures across the \ac{ctu} of a coded video frame. -In case of frame loss, the error propagation in the decoder is reduced because the number of \acs{ctu} predicted from the lost frame decreases and thus error concealment benefits as more data is correctly decoded from other frames. +In case of frame loss, the error propagation in the decoder is reduced because the number of \acs{ctu} predicted from the lost frame decreases and thus error concealment benefits as more data is correctly decoded from other reference frames. The underlying principle of this method is to minimise the probability of using the same reference picture for neighbouring \acs{ctu}. %For instance, In a coding structure with two reference frames, as shown in Figure~\ref{fig:checkerboardRefSelection}, the reference frames $r_{1}$ and $r_{2}$ are used to predict the same number of \acs{ctu} in current frame~($C$), resulting in a checkerboard structure, where a different reference frame is used for predicting every other \ac{ctu} within the current frame. If each reference frame is used to predict half of the pixels in the current frame, when any of these references is lost, the subsequent predicted frame is only partially distorted by mismatched predictions. This is because only half of its predictions depend on the missing reference frame, which limits the overall error propagation. -Moreover, since the motion information of the predicted frame is encoded with not only reference to the missing frame but also to the past ones, those \acs{mv} that cross over a missing reference frame, pointing to the previous ones, are still useful for reconstructing correct \acs{mv} and also for error concealment. +Moreover, since the motion information of the predicted frame is encoded using as reference not only the missing frame but also the past ones, those \acs{mv} that cross over a missing reference frame, pointing to the previous ones, are still useful for reconstructing correct \acs{mv} and also for error concealment. In the limit, a straightforward implementation of the previous example leads to a simple interleaved uniform distribution of the reference frames over all \acs{ctu} in the current frame, forcing the checkerboard structure shown in Figure \ref{fig:checkerboardRefSelection}. In the case of two reference frames, 50\% of the \acs{ctu} would use one reference frame, while the other 50\% would use the other reference. However, such solution would not take into consideration coding efficiency, since the selection process of the reference frame would not be \ac{rd} optimised for each \ac{pu} within a \ac{ctu}. @@ -144,7 +148,7 @@ The global and local deviations, $\Delta_G (r_\phi)$ and $\Delta_L (r_\phi)$, r \begin{equation}\label{eq:KFactorXValue} \Delta_\zeta(r_{\phi}) = \mathbf{N}_\zeta(r_\phi) - \frac{1}{S} \sum_{i=1}^{S}\mathbf{N}_\zeta(r_i), \hspace{2mm} \zeta=G,L. \end{equation} -For $\zeta=G$, function $\Delta_G(r_\phi)$ measures the difference between the number of times that reference frame $r_\phi$ was used and the global average use of all $r_i \in F$ in the \acs{pu} of $C$ that were encoded before the current \ac{pu}. $\mathbf{N}_G(r_i)$ is the number of times that each reference frame $r_i$ was used across all previously encoded \acs{pu}. For $\zeta=L$, function $\Delta_L(r_\phi)$ only considers the three top-left neighbouring \acs{ctu} for counting the number of times each reference frame was used, which is expressed by $\mathbf{N}_L(r_i)$. Overall, Equation (\ref{eq:KFactorXValue}) is used to adjust the penalty weight defined by Equation (\ref{eq:KFactor}) in order to achieve an approximately uniform use of all possible reference frames. This corresponds to the general objective of minimising the values of $\Delta_\zeta(r_{\phi})$. +For $\zeta=G$, function $\Delta_G(r_\phi)$ measures the difference between the number of times that reference frame $r_\phi$ was used and the global average use of all $r_i \in F$ in the \acs{pu} of $C$ that were encoded before the current \ac{pu}. $\mathbf{N}_G(r_i)$ is the number of times that each reference frame $r_i$ was used across all previously encoded \acs{pu}. For $\zeta=L$, function $\Delta_L(r_\phi)$ only considers the three top-left neighbouring \acs{ctu} for counting the number of times each reference frame was used, which is expressed by $\mathbf{N}_L(r_i)$. Overall, Equation (\ref{eq:KFactorXValue}) is used to adjust the penalty weight defined by Equation (\ref{eq:KFactor}), in order to achieve an approximately uniform use of all possible reference frames. This corresponds to the general objective of minimising the values of $\Delta_\zeta(r_{\phi})$. The optimum is reached when all reference frames are used to encode the same amount of \acs{pu} (i.e., $\Delta_G(r_{\phi})=\Delta_L(r_{\phi})=0$), and the original Lagrangian cost is not affected, i.e., $e^{W(r_{\phi})}=1$ in Equation (\ref{eq:Lagrangian}). Note that an exponential cost increase is used in Equation (\ref{eq:Lagrangian}) in order to increase the penalty associated to any reference frame that tends to be used much more often than the average. @@ -183,7 +187,7 @@ For instance, for $\gamma=5$, when a reference frame $r_\phi$ is used to predict \FIGSVSPACE \end{figure*} -Figure~\ref{fig:RefPicSelectionExample} illustrates the distribution of reference frames achieved by the proposed method across different \acs{pu}. This result is compared against a fixed checkerboard approach (\CHKB) and the reference \ac{hevc} (\REF). The dark-gray and light-gray squares indicate the reference frame in use, either $r_1$ or $r_2$, respectively. The white squares indicate intra-coded \acs{pu}. Below each method, the percentages of the image areas predicted by each reference frame are shown. As shown in this figure, the proposed method is able to distribute the use of the reference frames closely to the \CHKB method, i.e., $r_2$ is used as reference for approximately the same amount of \acs{pu} as $r_1$ (see use percentages of the predicted image). However, the predicted regions in the proposed method do not exhibit the same regular pattern as in \CHKB method. +Figure~\ref{fig:RefPicSelectionExample} illustrates the distribution of reference frames achieved by the proposed method across different \acs{pu}. This result is compared against a fixed checkerboard approach (\CHKB) and the reference \ac{hevc} (\REF). The dark-gray and light-gray squares indicate the reference frame in use, either $r_1$ or $r_2$, respectively. The unfilled squares indicate intra-coded \acs{pu}. Below each method, the percentages of the image areas predicted by each reference frame are shown. As shown in this figure, the proposed method is able to distribute the use of the reference frames closely to the \CHKB method, i.e., $r_2$ is used as reference for approximately the same amount of \acs{pu} as $r_1$ (see use percentages of the predicted image). However, the predicted regions in the proposed method do not exhibit the same regular pattern as in \CHKB method. It is noticeable in Figure~\ref{fig:RefPicSelectionExample} that for several \acs{ctu} the \PROP method is able to keep the same partitions and reference frames when compared to the optimal encoding (\REF). This can be observed, for example in the small partitions on the left, where the \PROP method uses the same reference frame as \REF. Thus, the proposed method is expected to achieve better coding efficiency than \CHKB, since more optimal coding modes may be chosen. % TID aqui? @@ -212,7 +216,7 @@ In this figure, each block corresponds to a \ac{pu}, which has an associated \ac While the \ac{mv} of \ac{pu} \MVDEPA is used to predict a total of four other \acs{mv}, the \ac{mv} of \ac{pu} \MVDEPB is only used to predict one single \ac{mv}. Thus, in case of data loss (e.g., frame $f_{-1}$), the \ac{mv} of \ac{pu} \MVDEPA has more impact on error propagation to the frame $f_{0}$ and consequently quality degradation than the \ac{mv} of \ac{pu} \MVDEPB, because of its higher number of dependent \acs{mv}. For this reason, in the \ac{mv} selection process, the proposed method assigns higher importance to the \ac{mv} of \ac{pu} \MVDEPA than \ac{pu} \MVDEPB. Moreover, in Figure~\ref{fig:mvdependencies}, it can be observed that although the number of \acs{mv} predicted from the \acs{mv} of \ac{pu} \MVDEPC is higher than \ac{pu} \MVDEPA (6 and 4 \acs{mv}, respectively), -the image area that is dependent from the former is smaller that the latter. This is due to the variable size of \acs{pu}. +the image area that depends on the former is smaller that the latter. This is due to the variable size of \acs{pu}. Therefore, the image area covered by all dependent \acs{pu} is separately considered in the \ac{mv} selection process. \begin{figure}[tb] @@ -242,9 +246,9 @@ where $R_U$ is the amount of bits required to encode the $n$ \acs{mv} of $U$ and \begin{equation} A_{D}(i) = \sum_{j=1}^{DV_i} ( w_j\times h_j ),\hspace{2mm} i = 1, 2 \cdots s, \end{equation} -where $DV_i$ is the number of \acs{mv} dependent from $MV_i$ and $w_j$ and $h_j$ are the width and height of the corresponding \ac{pu}. +where $DV_i$ is the number of \acs{mv} dependent from $MV_i$, and $w_j$ and $h_j$ are the width and height of the corresponding \ac{pu}. A weighting factor is used to control the amount of bitrate overhead ($R_U$), by increasing the cost of the number of encoded bits based on the duration of the error propagation. -It is given by the ratio between the distance from the current frame ($t_c$) to the next refresh frame ($t_I$) and the \ac{idr} period ($T_{I}$), combined with the control parameter $\alpha$. +This duration is given by the ratio between the distance from the current frame ($t_c$) to the next refresh frame ($t_I$) and the \ac{idr} period ($T_{I}$), combined with the control parameter $\alpha$. This weight varies in the range $\left[0;1 \right]$, achieving the lowest value for intra-coded frames and the maximum for the frame located immediately before an intra-coded one. In the proposed approach, for higher values of $\alpha$, less redundant \acs{mv} are selected. As pointed out before, this streaming stage operates on coded data and may be implemented at any point of the network, to increase robustness without fully decoding the stream. Note that the parsing operation represented in Figure~\ref{fig:proposedMethod} does not include decoding of the video frames. Furthermore, only one frame delay is introduced because the \ac{mv} selection process is performed on a frame-by-frame basis. @@ -281,7 +285,7 @@ The evaluation of the encoding-stage is carried out separately from the streamin \subsection{Encoding stage} -The performance of the encoding stage is evaluated on its own in order to only determine the improvements achieved by the dynamic reference frame selection method. +The performance of the encoding stage is evaluated on its own, in order to only determine the improvements achieved by the dynamic reference frame selection method. In this case, the reference frames were reconstructed with distortion, as a result of non-perfect error concealment. Since the encoding stage does not deal with errors in temporal \ac{mv} predictions (this is handled by the streaming-stage), in this evaluation the temporal \ac{mv} predictor is disabled. This is necessary to avoid masking the results with the errors caused by wrong \ac{mv} predictions. @@ -311,9 +315,9 @@ This value was chosen to allow increasing the Lagrangian cost by about $50\%$, \label{fig:RecoveredExample} \end{figure} -The error robustness achieved by the encoding-stage was firstly evaluated in regards to the error propagation resulting from a single loss event in the \ac{ld} configuration. +The error robustness achieved by the encoding-stage was firstly evaluated in regards to the error propagation, resulting from a single loss event in the \ac{ld} configuration. Figure~\ref{fig:RecoveredExample} shows the visual impact of error propagation on a region of the decoded Frame \#10 when Frame \#9 is affected by errors, for Basketball Drill and Race Horses sequences (the \ac{psnr} corresponds to the entire frame). -The proposed method is compared with the method~in~\cite{ROPERefGenJan2010} as both aim at reducing the error propagation. +The proposed method is compared with the method~in~\cite{ROPERefGenJan2010}, as both aim at reducing the error propagation. The method~in~\cite{ROPERefGenJan2010} uses filtered frames as reference for predictions, therefore, in case of errors, the effects of mismatch predictions are significantly reduced by the filtering operation (see Sub-Section~\ref{SECTION2:LeakyPred}). As shown in this figure, the proposed method is able to reduce the impact of mismatch decoding in comparison with the method~in~\cite{ROPERefGenJan2010}. Although both methods still reveal some predictions mismatch leading to inaccurate reconstructed of the objects contours, the proposed method is able to achieve a smoother reconstruction, which leads to higher quality in terms of \ac{psnr}. This is because, in frame \#10, the predictions depending on the missing frame \#9 are not used throughout the entire frame, thus, approximately half of the predictions are not affected by errors. In contrast, the method~in~\cite{ROPERefGenJan2010} requires several filtering operations to be applied (i.e, different reference frames need to be filtered) in order to reduce the effect of mismatch predictions. Summarising, the reconstructed frames illustrated in Figure~\ref{fig:RecoveredExample} reveal the higher efficiency achieved by distributing the use of the references frames. @@ -348,7 +352,7 @@ In \cite{ROPERefGenJan2010}, the use of interpolated references in the encoding The proposed method is also able to outperform the \ac{hevc} reference software when an hierarchical coding structure is used (see results for \ac{ra} configuration in Table~\ref{tab:AverageQualityCHKBEXPRandomAccess}), which implicitly limits the error propagation. In comparison to the reference \ac{hevc}, quality improvements up to $1.42$~dB at PLR=$10\%$ are obtained for Kendo sequence.%, while the average gain across all test sequences is 1~dB. -When comparing results for both test cases, it is noticeable that the proposed method is able to outperform the reference methods both when \ac{idr} frame loss is allowed or not (\LOSTALL and \KEEPINTRA, respectively). As the \ac{plr} increases, the gains of the proposed method decrease in comparison with the case where the \ac{idr} frames are delivered error-free (i.e., case \KEEPINTRA), due to the higher error propagation of the missing \ac{idr} frame. Moreover, the quality gains are lower in the \LOSTALL case when comparing with \KEEPINTRA for the same \ac{plr}, for both the \ac{ld} and \ac{ra}. +When comparing results for both test cases, it is noticeable that the proposed method is able to outperform the reference methods for both cases, when \ac{idr} frame loss is allowed or not (\LOSTALL and \KEEPINTRA, respectively). As the \ac{plr} increases, the gains of the proposed method decrease in comparison with the case where the \ac{idr} frames are delivered error-free (i.e., case \KEEPINTRA), due to the higher error propagation of the missing \ac{idr} frame. Moreover, the quality gains are lower in the \LOSTALL case when comparing with \KEEPINTRA for the same \ac{plr}, for both the \ac{ld} and \ac{ra}. Taking into account the above results and the spatio-temporal information shown in Table~\ref{TableTestSequences}, one can observe that the performance of the proposed method is lower for sequences with high spatial details and low temporal activity (e.g., BQSquare and Park Scene), which indicates higher effectiveness for video content with higher motion activity. %\HIGHLIGHT{complete with analysis of both cases} @@ -359,7 +363,7 @@ Taking into account the above results and the spatio-temporal information shown %\includegraphics[width=\FIGSIZE,page=1]{\ImprovedErrorResilience/Results/PSNRversusGama} %\includegraphics[width=\FIGSIZE,page=2]{\ImprovedErrorResilience/Results/PSNRversusGama} \input{./041_ImprovedErrorResilience/Results/PSNRversusGama.tex} - \caption{Influence of the $\gamma$ parameter in the average quality gains, compared with \REF, for different \acs{plr}.} + \caption{Influence of the $\gamma$ parameter in the average quality gains, comparing the proposed method with \REF, for different \acs{plr}.} \label{fig:QualityVsGamma} \end{figure} @@ -371,7 +375,7 @@ The influence of the $\gamma$ parameter used in Equation (\ref{eq:KFactor}) on t Figure \ref{fig:QualityVsGamma} shows the average quality gains of the proposed method in comparison to the reference \ac{hevc} (\REF) across all test sequences, for different values of $\gamma$. For the error-free case (PLR=0\%), the results in this figure show that increasing $\gamma$ leads to a decrease in the coding efficiency, especially in the \ac{ld} configuration. The reason is that higher values of $\gamma$ lead to higher values of $W(r_{\phi})$, therefore the reference picture selection cost is severely constrained by the exponential weights. -Since $T_{ID}$ is constant for all frames in the \ac{ld}, higher weights are used for all frames ($T_{ID}$ is used to reduce the weights in case of hierarchical coding) and the encoder is not able to select the best reference frame in \c{rd} sense. +Since $T_{ID}$ is constant for all frames in the \ac{ld}, higher weights are used for all frames ($T_{ID}$ is used to reduce the weights in case of hierarchical coding) and the encoder is not able to select the best reference frame in the \ac{rd} sense. In the case where packet loss occurs, higher values of $\gamma$ lead to improved error resilience. Since higher weights are used in the \ac{rd} cost, the encoder is constrained to distribute the use of the reference frames of different \acs{pu}, reducing the error propagation. For very high values (e.g., $\gamma=1000$) the encoder is forced to uniformly select the reference frames, resulting in a fixed checkerboard pattern. @@ -422,7 +426,7 @@ The relative complexity increase of the robust encoder using \PROP, \CHKB and th As shown in this table, the method presented in \cite{ROPERefGenJan2010} is the most time consuming because it requires post-processing of the encoded frames in order to interpolate new references. -The proposed scheme, using a reference picture selection, presents a lower complexity increase than \cite{ROPERefGenJan2010}, while the fixed checkerboard scheme (\CHKB) presents a slightly lower complexity \PROP method. +The proposed scheme, using a reference picture selection, presents a lower complexity increase than \cite{ROPERefGenJan2010}, while the fixed checkerboard scheme (\CHKB) presents a slightly lower complexity than \PROP method. This is due to the fact that in the \CHKB method there is no \ac{rd} optimisation process for the reference frame selection (fixed pattern is used), thus reducing the overall computational complexity. @@ -453,7 +457,14 @@ Different values of $\alpha$ are used to achieve different overhead ratios. All \subsubsection{Dynamic MV selection performance} -\begin{figure}[!b] +Firstly the quality obtained by the proposed method, which dynamically selects the amount of redundant \acs{mv}, is compared to cases where a fixed amount of \acs{mv} per frame are transmitted (i.e., the most important 10\%, 30\% and 50\% $\ac{mv} \in V$). This cased are referred to as "\DYN" and "\FIXED", respectively. + +%In the case where variable redundancy is allowed, \acs{mv} are selected using the proposed dynamic method (Dyn) and a fixed approach (Fixed). +Figure~\ref{fig:QualityVsRedundancyMVSelection} illustrates the relation between the quality obtained (\ac{psnr}) and the amount of redundancy used. +The results indicate that for approximately the same amount of redundant bits are used in both cases, however by using the proposed method to dynamically select the \acs{mv} to be used as side-information (\DYN) is able to outperform the fixed case for all test sequences. +Results also show that for both approaches the decoded video quality increases as the redundancy of the side-information increases, since a higher number of mismatch \ac{mv} predictions are able to be recovered. + +\begin{figure}[tb] \centering \newcommand{\CurrFigSize}{0.49\columnwidth} \includegraphics[width=\CurrFigSize,page=1]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf} @@ -462,19 +473,13 @@ Different values of $\alpha$ are used to achieve different overhead ratios. All \includegraphics[width=\CurrFigSize,page=4]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf} \includegraphics[width=\CurrFigSize,page=5]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf} \includegraphics[width=\CurrFigSize,page=6]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf} - \caption{Decoded video quality for different amounts of redundancy, selected by using a fixed approach and the proposed method.} + \caption{Decoded video quality for different amounts of redundancy, selected by using a fixed approach and the proposed method (\DYN).} \label{fig:QualityVsRedundancyMVSelection} \end{figure} -Firstly the quality obtained by the proposed method, which dynamically selecting the amount of redundant \acs{mv}, is compared to cases where a fixed amount of \acs{mv} per frame are transmitted (i.e., the most important 10\%, 30\% and 50\% $\ac{mv} \in V$). This cased are referred to as "\DYN" and "\FIXED", respectively. - -%In the case where variable redundancy is allowed, \acs{mv} are selected using the proposed dynamic method (Dyn) and a fixed approach (Fixed). -Figure~\ref{fig:QualityVsRedundancyMVSelection} illustrates the relation between the quality obtained (\ac{psnr}) and the amount of redundancy used. -The results indicate that for approximately the same amount of redundant bits are used in both cases, however by using the proposed method to dynamically select the \acs{mv} to be used as side-information (\DYN) is able to outperform the fixed case for all test sequences. -Results also show that for both approaches the decoded video quality increases as the redundancy of the side-information increases, since a higher number of mismatch \ac{mv} predictions are able to be recovered. - -It is noticeable in Figure~\ref{fig:QualityVsRedundancyMVSelection} that for higher \acs{plr}, the proposed method is able to achieve higher gains. This reveals that for increasing data loss it is more important optimise the amount of \acs{mv} selected per frame in order to use more \acs{mv} in those frames which have a higher impact of the error propagation. -For instance, the proposed method is able to gain up to $1.5$~dB for the BQSquare sequence for a PLR=10\$. +It is noticeable in Figure~\ref{fig:QualityVsRedundancyMVSelection} that for higher \acs{plr}, the proposed method is able to achieve higher gains. +This reveals that when data loss increases it is more important optimise the amount of \acs{mv} selected per frame, in order to use more \acs{mv} in those frames that have a higher impact on the error propagation. +For instance, the proposed method is able to gain up to $1.5$~dB for the BQSquare sequence subject to a PLR=10\%. For lower \acs{plr}, since fewer frames are affected by data losses, the use of a fixed approach achieves similar performance as the proposed method (\DYN). @@ -482,13 +487,13 @@ For lower \acs{plr}, since fewer frames are affected by data losses, the use of \subsubsection{Quality evaluation} -Table~\ref{tab:AverageQualityRedundancies} presents the average \ac{psnr} of decoded video for different \ac{plr} (columns~4~to~8) and for various percentages of \ac{mv} redundancy (third column). The absolute quality values are shown for the reference case (\NOREDUND) and \ac{psnr} gains are shown for thee different levels of redundancy introduced by the proposed method (\DYN). -This study also conducted for the two different cases, with and without \ac{idr} loss. +Table~\ref{tab:AverageQualityRedundancies} presents the average \ac{psnr} of decoded video for different \ac{plr} (columns~4~to~8) and for various percentages of \ac{mv} redundancy (third column). The absolute quality values are shown for the reference case (\NOREDUND) and \ac{psnr} gains are shown for three different levels of redundancy introduced by the proposed method (\DYN). +This study was also conducted for the two different cases, with and without \ac{idr} loss. -It can be confirmed from the results of Table~\ref{tab:AverageQualityRedundancies} that, for each sequence, as $\alpha$ decreases more redundant \acs{mv} are used leading to higher overhead rations. This results in higher objective video quality for all tested sequences. +It can be confirmed from the results of Table~\ref{tab:AverageQualityRedundancies} that, for each sequence, as $\alpha$ decreases more redundant \acs{mv} are used, leading to higher overhead rations. This results in higher objective video quality for all tested sequences. This confirms that redundant \acs{mv} are able to minimise the error propagation, which is accomplished at a cost of an acceptable increase in the bitrate, since only a sub-set of \acs{mv} is transmitted. -The use of redundant \acs{mv} yields better video quality for \ac{plr} as low as $1\%$, achieving higher quality gains when \ac{idr} frames are not lost (\KEEPINTRA). This is because of the higher quality degradation when \ac{idr} frames are lost, which is not due to mismatched \ac{mv} predictions and thus cannot be reduced by transmitting extra \acs{mv} as side-information. -The use the proposed method achieves higher quality for sequences with higher motion activity and which makes use of inter-prediction more often, i.e., BQSquare and Traffic sequences (see inter ratio in Table~\ref{TableTestSequences}), leading to a maximum gain of $3.93$~dB for \LOSTALL ($7.82$~dB for \KEEPINTRA). +The use of redundant \acs{mv} yields better video quality for \ac{plr} as low as $1\%$, achieving higher quality gains when \ac{idr} frames are not lost (\KEEPINTRA). This is because of the higher quality degradation when \ac{idr} frames are lost, which is not due to mismatched \ac{mv} predictions, and thus cannot be reduced by transmitting extra \acs{mv} as side-information. +The use the proposed method also achieves higher quality for sequences with higher motion activity that makes use of inter-prediction more often, i.e., BQSquare and Traffic sequences (see Inter ratio in Table~\ref{TableTestSequences}), leading to a maximum gain of $3.93$~dB for \LOSTALL ($7.82$~dB for \KEEPINTRA). \begin{table}[!t] \centering @@ -508,12 +513,12 @@ Thus, using the proposed two-stage approach to recover temporal predictions cons In this chapter a two-stage approach was proposed to increase the error robustness of \ac{hevc} streaming and reduce the error propagation in case of packet loss. A constrained coding approach was devised to select reference frames and dynamically distribute temporal dependencies. -This is jointly used with a controlled amount of side information in coded streams, comprising a small set of the most relevant \acs{mv}, to minimise \ac{mv} mismatch at the decoder in the presence of frame loss. +This is jointly used with a controlled amount of side information in coded streams, comprising a small set of the most relevant \acs{mv}, in order to minimise \ac{mv} mismatch at the decoder in the presence of frame loss. As can be concluded from the results, this method contributes to reduce the temporal dependencies with consistent quality gains for different \ac{plr} and coding configurations, only incurring in a small drop of coding efficiency. -Overall, the proposed approach is an effective method for coping with video transmission over error-prone networks. - - - +Overall, the proposed approach is an effective method to cope with video transmission over error-prone networks. + + +\todo{Papers here??} diff --git a/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex b/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex index 25029ee..5429d06 100644 --- a/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex +++ b/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex @@ -4,7 +4,7 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % CHAPTER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\chapter{Error Concealment-Aware Video Encoding \VERSION{1}} +\chapter{Error Concealment-Aware Video Encoding \VERSION{2}} \label{Chapter:ConcealmentAwareTechniques} %\vspace{-6mm} %\clearpage @@ -19,12 +19,13 @@ \glsreset{ld} %\glsreset{rd} -In this chapter an error concealment-aware encoding scheme is proposed to improve the quality of decoded video in broadcast environments prone to transmission errors and data loss. +In this chapter an error concealment-aware encoding scheme is proposed to improve the quality of decoded video in broadcast environments prone to transmission errors and data loss. The method aims to enhance the reconstruction of recovered lost frames by optimising the efficiency of the \ac{ec}. The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled through \ac{sei} messages. Such optimal \ac{ec} modes are found by simulating transmission losses followed by a Lagrangian optimisation of the \textit{signalling rate - \ac{ec} distortion} cost. A generalised saliency-weighted distortion is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. When data loss occurs the decoder uses the signalling information to improve the reconstruction quality. -This chapter is organized as follows. Section~\ref{Section5:Problem} presents the problem formulation and motivation for the method proposed in this chapter. In Section~\ref{Section5:ECCandidates} does an initial evaluation of different error concealment schemes in order to find the relevant ones to be used in the proposed error concealment-aware coding scheme. -Section~\ref{Section5:Architecture} provides an overview of the architecture of the proposed method followed by a detailed description in Section~\ref{Section5:ProposedMethod}. +This chapter is organized as follows. Section~\ref{Section5:Problem} introduces the problem definition and motivation for the method proposed in this chapter. +Section~\ref{Section5:ECCandidates} presents an initial evaluation of different error concealment schemes in order to find the relevant ones to be used in the proposed error concealment-aware coding scheme. +Section~\ref{Section5:Architecture} provides an overview of the proposed method architecture followed by a detailed description in Section~\ref{Section5:ProposedMethod}. Sections~\ref{Section5:ECSignallingPerformance}~and~\ref{Section5:OverallPerformance} present the performance results and discussion of the proposed \ac{ec} optimisation and signalling. Finally, Section~\ref{Section5:Summary} concludes the chapter. @@ -32,29 +33,29 @@ Finally, Section~\ref{Section5:Summary} concludes the chapter. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\section{Problem formulation} +\section{Problem definition} \label{Section5:Problem} \glsreset{mve} \glsreset{mc} %\glsunset{mb} -In Chapter~\ref{Chapter:ErrorResilienceReview} different \ac{ec} algorithms that can effectively reconstruct lost video were described, though not all of them exhibit the same accuracy for every lost frame or slice. +In Chapter~\ref{Chapter:ErrorResilienceReview} different \ac{ec} algorithms that can effectively reconstruct lost video were described, though not all of them exhibit the same accuracy for every lost frame or slice. Moreover, it was also shown that combining different \ac{ec} approaches at the \ac{mb}/\ac{pu} level to improve the reconstruction accuracy is not straightforward. In the past, hybrid \ac{ec} schemes were proposed to improve the recovery quality of lost frames by either combining spatial and temporal \ac{ec} algorithms~\cite{HybridConcTB2008} or by optimising the \ac{mve} at the pixel and block levels~\cite{HybridConcealmentTB2008,2DConcHybridMotion2010}. Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of motion-based recovery approaches by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}. Although such methods are able to reduce the reconstruction artefacts and improve the overall video quality, decoder-based decisions are still not fully accurate, as they only have limited information to be used as reference in the decision process. -Therefore, higher reconstructed quality can be achieved by estimating the best \ac{ec} techniques at the encoding-side based on the original video signal. +Therefore, higher reconstructed quality can be achieved by estimating the best \ac{ec} techniques at the encoding-side, based on the original video signal. %Such approach not only improves quality but also reduces the complexity of the video decoders. %In this chapter a robust coding driven by \ac{ec} methods is developed with the objective of enhancing the quality of reconstructed video at decoders receiving erroneous streams from the transmission network. The proposed approach combines error resilience tools at the encoder-side with \ac{ec} at the decoder-side. -Different approaches were proposed in the past to address the problem of \ac{ec}-aware video coding~\cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ErrorResilienceModeHEVC2016}. -Extra information was also used to aid the \ac{ec} operation at the decoder-side, either by sending redundant information \cite{ResilienceRECAP_ICASSP2009,SaliencyErrorConcealment_TM2013} or by embedding additional coded data into the bitstream \cite{ConcealmentIntraDataHidng2009}. +Different approaches were proposed in the past to address the problem of \ac{ec}-aware video coding~\cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ErrorResilienceModeHEVC2016}. +Extra information was also used to aid the \ac{ec} operation at the decoder-side, either by sending redundant information \cite{ResilienceRECAP_ICASSP2009,SaliencyErrorConcealment_TM2013} or by embedding additional coded data into the bitstream \cite{ConcealmentIntraDataHidng2009}. Although these methods facilitate the \ac{ec} performance either through slice reordering or based on \acs{mv}, they always rely on the assumption that some fixed \ac{ec} method is used at the decoder. % The work presented in \cite{ConcealmentSignalling2008,SignallingRobustScalable2015} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder. Although such approaches achieves reasonably good performance, it relies on the spatial neighbouring information or multi-layer video, which may not be available in all cases. -To advance one step further, this chapter presents n investigation on concealment-aware video coding and proposes a new scheme to achieve high decoding flexibility in terms of the \ac{ec} approaches, leading to better video quality. +To advance one step further, this chapter presents an investigation on concealment-aware video coding and proposes a new scheme to achieve high decoding flexibility in terms of the \ac{ec} approaches, leading to better video quality. This is accomplished by selecting, at the encoder, among several \ac{ec} algorithms with different characteristics, the one that yields the lowest reconstruction distortion at the decoder and also minimises the \ac{rd} cost considering the necessary signalling rate. Thus, the proposed approach differs from similar previous methods \cite{ConcealmentAwareME2006,ErrorResilienceModeHEVC2016} by selecting the best error concealment method without affecting the encoder optimisation. To further improve the quality of recovered frames, an enhancement layer is encoded with the mismatch residue between the transmitted and recovered frames. This is a dynamic optimisation process, which is more efficient than conventional approaches based on redundant pictures \cite{RedudantPicsJan2009,VPxResiliencePCS2016}, which transmit the whole frame as redundancy. -In contrast with the previous work presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}, which addressed the error resilience problem by optimising the coding dependencies, the approach followed in this chapter considers both \ac{ec} performance and the coded rate required for signalling the best \ac{ec} mode to the decoder. +In contrast with the previous work presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}, which addressed the error resilience problem by optimising the coding dependencies, the approach followed in this chapter considers both, \ac{ec} performance and the coded rate required for signalling the best \ac{ec} mode to the decoder. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -64,7 +65,7 @@ In contrast with the previous work presented in Chapter~\ref{Chapter:ErrorResili \label{Section5:ECCandidates} -In order to proposed a novel error concealment-aware technique different algorithms were taken into consideration. +In order to propose a novel error concealment-aware technique different algorithms were taken into consideration. This section presents a description and evaluation of the most relevant \ac{ec} algorithms used in the proposed method. Firstly, a technical description of each method is provided followed by a performance comparison between them. @@ -93,8 +94,8 @@ x^{'} &= x - v_{0x}, \\ y^{'} &= y - v_{0y}. \end{align} \end{enumerate} -This method is illustrated in Figure~\ref{FIG5:MotionVectorExtrapolation} by the \ac{mv} $v^{2}_{0}$ which results from the extrapolation of the \ac{mv} $v^{2}_{-1}$, and is referred to as \glsfirst{mve}. -One should note that these methods may not result in a complete motion field for the entire frame, resulting in an incomplete reconstruction. For those regions the average of the three top-left neighbouring \acs{mv} are used to complete the motion field. +This method is illustrated in Figure~\ref{FIG5:MotionVectorExtrapolation} by the \ac{mv} $v^{2}_{0}$, which results from the extrapolation of the \ac{mv} $v^{2}_{-1}$, and is referred to as \glsfirst{mve}. +One should note that these methods may not result in a complete motion field for the entire frame, resulting in an incomplete reconstruction. For those regions the average of the three top-left neighbouring \acs{mv} are used to complete the motion field. %\todo{Grid-aligment! Incomplete!} \begin{figure}[tb] @@ -104,7 +105,7 @@ One should note that these methods may not result in a complete motion field for \label{FIG5:MotionVectorExtrapolation} \end{figure} -Finally, as a base for comparison the default \ac{ec} in the reference \ac{hevc} implementation the \ac{fc} algorithm is also evaluated. This method consists in replacing the lost frame with a copy of the previous decoded one, which can also be described as a motion compensated reconstruction using only zero \acs{mv}. +Finally, as a base for comparison, the default \ac{ec} in the reference \ac{hevc} implementation the \ac{fc} algorithm is also evaluated. This method consists in replacing the lost frame with a copy of the previous decoded one, which can also be described as a motion compensated reconstruction using only zero \acs{mv}. \subsection{Algorithms based on motion estimation} @@ -115,11 +116,11 @@ Typically, optical flow estimation algorithms are used to estimate a 2-D motion \begin{equation} E(x,y) = \sum_{x} \sum_{y} \bigg| f_{0}\big(x,y\big) - f_{1}\big( x + v_x(x,y), y+v_y(x,y) \big) \bigg|. \end{equation} -Besides a full search algorithm where a block-based approach is used to search for the best \acs{mv} that minimises the sum of absolute errors, three other approaches were tested to evaluate their effectiveness in \ac{ec} algorithms. These three methods are the following: +Besides a full search algorithm, where a block-based approach is used to search for the best \acs{mv} that minimises the sum of absolute errors, three other approaches were tested to evaluate their effectiveness in \ac{ec} algorithms. These three methods are the following: \begin{enumerate} \item a method based on the total variation using the minimization of the \mbox{L-1} norm~\cite{DualFlow2016}; \item a method which approximates the neighbourhood of each pixel by quadratic polynomials on both frames~\cite{PolyFlow2003}; - \item a method based on a combination of features matching and total variations~\cite{DeepFlow2013}. \HERE + \item a method based on a combination of feature matching and total variation~\cite{DeepFlow2013}. \todo{check here!} \end{enumerate} \subsubsection{Motion estimation based on total variation and regularisation (\textit{DualFlow})} @@ -142,7 +143,7 @@ f(\mathbf{p}) \sim \mathbf{p}\mathbf{A}_1\mathbf{p}^T+\mathbf{b}_1\mathbf{p}^T+C \end{equation} where $\mathbf{p} = [x\ y]$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ is a row vector and $C_1$ is a scalar. The coefficients are estimated using weighted least squares fit to the signal values in the neighbourhood. -Since the result of polynomial expansion is that each neighbourhood is approximated by a polynomial, we start by analysing what happens if a polynomial undergoes an ideal translation. Let's consider the signal $f_1(\mathbf{p})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$ given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following: +Since the result of a polynomial expansion is such that each neighbourhood is approximated by a polynomial, it is analysed what happens if a polynomial undergoes an ideal translation. Let's consider the signal $f_1(\mathbf{p})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$, given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following: \begin{align} f_2(\mathbf{p}) &= f_1(\mathbf{p}-\mathbf{d}) \nonumber \\ @@ -163,15 +164,14 @@ This allows to find the displacement $\mathbf{d}$ for any signal dimensions by s %\todo{Completar o Deep flow} -In this method the motion field is estimated based on extension of the total variational methods using descriptor (feature) matching. +In this method the motion field is estimated based on extension of the total variation methods using descriptor (feature) matching. This allows the optical flow estimation to accurately find large displacements as it is not bound by constrains, such as the Horn \& Such presented in Equation~(\ref{EQ5:DualFlow})\cite{LargeDisplacementOptFlow_TPAMI2011}. Moreover, the correspondences between descriptions are used to apply a coarse-to-fine wrapping, thus avoiding local minimum values. The combination of both approaches increases the matching accuracy. -This was further exploited by introducing deep convolutional networks~\cite{DeepFlow2013}. -This was also exploited in \cite{DeepFlow2013} with the following extensions: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layer optimisation. +This was also exploited in \cite{DeepFlow2013} with the following extensions: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layer optimisation. Figure~\ref{FIG5:DeepFlowDiagram} shows the blocks that are included in the description matching process. The matching algorithm is built upon a multi-stage architecture with six layers with different patches sizes, interleaving convolutions and max-pooling, which is similar to deep convolutional networks \cite{GradientBasedConvNN_ProcIEEE1998}. -As shown in the figure a multi-size response levels are generated from the reference and target image using convolutions with different patches sizes. This method uses bottom-up approach, i.e., starting at a fine level and moving towards the coarser levels (larger patches) which are built as an aggregation of responses of smaller patches. This results in a multi-size response pyramid. +As shown in Figure~\ref{FIG5:DeepFlowDiagram}, a multi-size response levels are generated from the reference and target image using convolutions with different patches sizes. This method uses a bottom-up approach, i.e., starting at a fine level and moving towards the coarser levels (larger patches), which are built as an aggregation of responses of smaller patches. This results in a multi-size response pyramid. Subsequently, a local maximum is extracted from each level to achieve a more accurate motion field (quasi-dense correspondence). Finally, an initial motion field is extracted, referred to as $v'$ . @@ -183,26 +183,26 @@ Finally, an initial motion field is extracted, referred to as $v'$ . \FIGSVSPACE \end{figure} -Subsequently, the final optical flow is obtained by minimising an energy function, which can be described by the following: +Subsequently, the final optical flow is obtained by minimising an energy function, which can be described by the following equation: \begin{equation} E(v) = \sum_{\mathbf{p} \in \Omega} \Big( E_D(\mathbf{p}) + \alpha E_S(\mathbf{p}) + \beta E_M(\mathbf{p}) \Big), \end{equation} where $E_D$ is the data term, $E_S$ is a smoothing term and $E_M$ is the matching term. The data term is obtained from the difference between matching pixels in both video frames. -The smoothness term is a robust penalisation of the gradient of the flow as used in the Equation (\ref{EQ5:DualFlow}), i.e., $E_S(\mathbf{p})=|\nabla v(\mathbf{p})|$. Finally, the matching term ($E_M$) is used to encourage the flow estimation to be similar to the precomputed motion field~($v'$). +The smoothness term is a robust penalisation of the gradient of the flow, similarly to Equation (\ref{EQ5:DualFlow}), i.e., $E_S(\mathbf{p})=|\nabla v(\mathbf{p})|$. Finally, the matching term ($E_M$) is used to encourage the flow estimation to be similar to the precomputed motion field~($v'$). \subsection{Experimental evaluation} -In this section a performance evaluation of the different \ac{ec} schemes previously described is carried out. This evaluation aims at finding the suitable methods to be used for efficiently reconstruct the missing frames in case of data loss. -The two \ac{ec} methods based on the correctly received \acs{mv} are tested, i.e., \ac{mc} and \ac{mve}, using the \ac{fc} method is used as reference for comparison. +In this section a performance evaluation of the different \ac{ec} schemes previously described is carried out. This evaluation aims at finding the suitable methods to be used for efficiently reconstruct the missing frames in case of data loss. +The two \ac{ec} methods based on the correctly received \acs{mv} are tested, i.e., \ac{mc} and \ac{mve}, using the \ac{fc} method is used as reference for comparison. Moreover, the four different approaches to estimate the motion field for the missing frame are compared: (i) full search based on the minimisation of the sum of absolute errors (\FULLSEARCH), (ii) optical flow based on the duality between total variation and smoothness term (\DUALFLOW), (iii) optical flow based on polynomial approximations (\POLYFLOW) and (iv) optical flow based on deep matching (\DEEPFLOW). -Table~\ref{TAB5:ECPerformaceComparison} shows the average quality (\ac{psnr}) obtained for the different methods under investigation. +Table~\ref{TAB5:ECPerformaceComparison} shows the average quality (\ac{psnr}) obtained for the different methods under investigation. The absolute value of \ac{psnr} is shown for the \FC method, while the differential values are shown for the remaining methods. -The results shown in the table reveal that methods \MC and \MVE are both able to overcome \FC, achieving average quality gains of up to 1.41 dB and 1.46 dB, respectively. Moreover, by extrapolating the \acs{mv} from the correctly receiving frame to the lost one (\MVE), it is also possible to outperform motion-copy (\MC). -This can be explained by the objects' trajectories of objects which are preserved in the \MVE method (assuming constant translational motion), in contrast with the \MC method, which assumes that neighbouring objects have parallel motion which may result in inaccurate estimations. +The results shown in the table reveal that methods \MC and \MVE are both able to overcome \FC, achieving average quality gains of up to 1.41 dB and 1.46 dB, respectively. Moreover, by extrapolating the \acs{mv} from the correctly receiving frame to the lost one (\MVE), it is also possible to outperform motion-copy (\MC). +This can be explained by the objects' trajectories of objects which are preserved in the \MVE method (assuming constant translational motion), in contrast with the \MC method, which assumes that neighbouring objects have parallel motion which may result in inaccurate estimations. %\HERE @@ -215,17 +215,17 @@ This can be explained by the objects' trajectories of objects which are preserve \end{table} -According to the results of Table~\ref{TAB5:ECPerformaceComparison} (columns 5 to 8) the \ac{ec} methods based on motion estimation achieve an overall superior performance when compared with methods which rely of correctly received \acs{mv} (i.e., \MC and \MVE). -Results for \FULLSEARCH method reveal that only minimising the sum of absolute errors between two consecutive frames might not always be directly related with objects' motions, which results in inaccurate motion field to be used in the reconstruction of lost frames. -Finally, results from Table~\ref{TAB5:ECPerformaceComparison} reveal that the \DEEPFLOW algorithm leads to the most accurate motion field, in comparison with the other methods. -This is able to achieve an average quality gain of 1.79 dB, and up to 3.39 dB for Book Arrival sequence, where \POLYFLOW only achieves 3.05 dB. +According to the results of Table~\ref{TAB5:ECPerformaceComparison} (columns 5 to 8) the \ac{ec} methods based on motion estimation achieve an overall superior performance when compared with methods which rely on correctly received \acs{mv} (i.e., \MC and \MVE). +Results for \FULLSEARCH method reveal that only minimising the sum of absolute errors between two consecutive frames might not always be directly related with objects' motions, which results in inaccurate motion field to be used in the reconstruction of lost frames. +Finally, results from Table~\ref{TAB5:ECPerformaceComparison} reveal that the \DEEPFLOW algorithm leads to the most accurate motion field, in comparison with the other methods. +It is able to achieve an average quality gain of 1.79 dB, and up to 3.39 dB for Book Arrival sequence, where \POLYFLOW only achieves 3.05 dB. Since \POLYFLOW combines different techniques also used by other algorithms in study, e.g., total variation also used by \DUALFLOW, it is expected to achieve accurate estimations and consequently higher \ac{ec} reconstruction quality. Moreover, using descriptor matching in combination with smoothing conditions, the \DEEPFLOW algorithm is able to correctly estimate large displacements, which results in an improved performance for video signals with higher motion, e.g., Tennis sequence. \subsection*{Remarks} -Summarising, the analysis of the previous results indicates that all methods are able to cope with lost frames and manage to recover the missing data with acceptable quality. -Nevertheless, it is noticeable that lost frames recovered with an estimated motion field result in higher average quality, especially when using the \DEEPFLOW method. However, as the other methods tested are also able to achieve acceptable video quality, they shall not be neglected, as they provide alternative approaches for lost frame recovery with lower computational complexity (i.e., they not require motion estimation). -Thus, in order to achieve higher reconstruction quality different approaches shall be considered, aiming at balancing the overall complexity with the recovery quality. +Summarising, the analysis of the previous results indicates that all methods are able to cope with lost frames and manage to recover the missing data with acceptable quality. +Nevertheless, it is noticeable that lost frames recovered with an estimated motion field result in higher average quality, especially when using the \DEEPFLOW method. However, as the other methods tested are also able to achieve acceptable video quality, they shall not be neglected, as they provide alternative approaches for lost frame recovery with a lower computational complexity (i.e., they not require motion estimation). +Thus, in order to achieve higher reconstruction quality, different approaches shall be considered, aiming at balancing the overall complexity with the recovery quality. %\HERE @@ -235,9 +235,9 @@ Thus, in order to achieve higher reconstruction quality different approaches sha \section{Architecture for the EC-aware video encoding} \label{Section5:Architecture} -%\glsreset{ec} +\glsreset{el} -This section presents the general architecture proposed for \ac{ec}-aware video encoding and describes its main the functional blocks, also providing an introduction for the subsequent detailed descriptions. +This section presents the general architecture proposed for \ac{ec}-aware video encoding, describing its main functional blocks, and provides an introduction for the subsequent detailed descriptions. Figure~\ref{FIG5:BlockDiagram} illustrates the architecture of the proposed \ac{ec}-aware encoding scheme. After the video encoder, it includes data loss simulation at the \ac{nal} for every slice. Subsequently, \ac{ec} is applied to the simulated losses in order to find the best methods that should be used to reconstruct the lost slices in case of transmission errors. Saliency-weighted \ac{ec} mode decision is used to select the optimal \ac{ec} mode from the best trade-off between \ac{ec} distortion, using the original input video~($f$) as reference, and the number of bits required for signalling. @@ -247,8 +247,8 @@ For the \ac{ec} mode decision optimization the missing slices are reconstructed, In this process, each \ac{ctu} is partitioned to form a quadtree structure of \ac{ecu} that are used to test the different \ac{ec} candidate modes. Then, the best trade-off between \ac{ecu} partition and signalling is found through \ac{rd} optimisation. In order to achieve an efficient use of the signalling overhead, a saliency-weighted procedure is embedded in the \ac{rd} optimisation based on a spatio-temporal saliency, where more signalling bits are allocated to regions with higher saliency values. -Finally, the signalling information is transmitted to the decoder by using the \ac{sei} \acs{nalu} type~\cite{HEVCHighLevelReferenceDez2012}. -Since in general, the optimal \ac{ec} mode is still not able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el}. +Finally, the signalling information is transmitted to the decoder by using the \ac{sei} \acs{nal} units~\cite{HEVCHighLevelReferenceDez2012}. +Since, in general, the optimal \ac{ec} mode is still not able to perfectly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el}. This residual information contains the difference between the compressed ($\hat f_t$) and the \ac{ec} frames ($\tilde f$). \begin{figure}[tb] @@ -272,15 +272,16 @@ The remaining of this section describes in detail the building blocks of the pro %\section{Error concealment mode decision} \label{Section5:ProposedMethod} -In this section the proposed saliency-weighted \ac{ec} mode decision algorithm is described. Firstly, the list of \ac{ec} candidates is provided based on the previously described methods. -Then, the \ac{rd} optimisation is proposed using a saliency region in each frame to decrease the overall bitrate without compromising the reconstruction quality. -The signalling coding strategy is also described. -Finally, the introducing of \ac{ec} residue as auxiliary information to improve the reconstruction quality is described. +In this section the proposed saliency-weighted \ac{ec} mode decision algorithm is described. +Firstly, the list of \ac{ec} candidates is provided based on the previously described methods. +Then, the \ac{rd} optimisation is performed using a saliency region in each frame, in order to decrease the overall bitrate without compromising the reconstruction quality. +The signalling coding strategy is also described. +Finally, the \ac{ec} residue is introduced as auxiliary information to improve the reconstruction quality is described. \subsection{\ac{ec} candidates} -A set of four \ac{ec} candidate modes (M1...M4) is used in the optimisation process that selects one of them as the best for each particular \ac{ecu}. Each \ac{ec} mode corresponds to a different \ac{ec} method, as shown in Figure~\ref{FIG5:concealmentcandidates}, namely: +A set of four \ac{ec} candidate modes (M1...M4) is used in the optimisation process that selects one of them as the best match for each particular \ac{ecu}. Each \ac{ec} mode corresponds to a different \ac{ec} method, as shown in Figure~\ref{FIG5:concealmentcandidates}, namely: \begin{itemize} \item M1, M2: These two methods are based on extrapolation of an estimated motion field; \item M3: This method is based on \ac{mv} extrapolation of previously received \acs{mv}; @@ -296,17 +297,18 @@ A set of four \ac{ec} candidate modes (M1...M4) is used in the optimisation pro \FIGSVSPACE \end{figure} -The first two methods M1, M2 rely on an estimated motion field, using the method \DEEPFLOW \cite{DeepFlow2013}, to recover the lost frame. Then, such motion field is extrapolated at the pixel level, i.e., one \ac{mv} is assigned to each pixel, and \ac{pu} level, i.e., one \ac{mv} is assigned to the entire unit. +The first two methods, M1 and M2 rely on an estimated motion field, using the method \DEEPFLOW \cite{DeepFlow2013}, to recover the lost frame. +Such motion field is extrapolated at the pixel level, i.e., one \ac{mv} is assigned to each pixel, and \ac{pu} level, i.e., one \ac{mv} is assigned to the entire unit. This results in two different reconstructions of the missing frame, where the former may increase the accuracy of the pixel extrapolation, but the latter has the advantage of keeping the block structure and generate a smoother reconstruction. -The method M3 is based on a convention \ac{mve}, and finally M4 reconstructs the missing frames using the \ac{mc} algorithm. +The method M3 is based on a conventional \ac{mve}, and finally M4 reconstructs the missing frames using the \ac{mc} algorithm. \subsection{Optimal \ac{ec} mode selection} As previously described in the sub-section~\ref{Section5:ECCandidates}, the proposed method allows to choose between four \ac{ec} candidates, which result in different recovered frames for each lost frame (simulated). The lost frame is partitioned into quadtree structures, each one forming a set of variable size \acs{ecu}. -Then, the process of finding the optimal \ac{ec} method for each \ac{ecu} is formulated as a rate-distortion cost minimisation. To solve such problem, a Lagrangian optimisation is used~\cite{LagragianMultiplier1982} to achieve the best trade-off between reconstruction quality and signalling overhead. +Then, the process of finding the optimal \ac{ec} method for each \ac{ecu} is formulated as a rate-distortion cost minimisation. To solve such problem, a Lagrangian optimisation is used to achieve the best trade-off between reconstruction quality and signalling overhead. Furthermore, image saliency driven by visual attention modelling is used in the optimisation process to weight the cost of the overhead information of each \ac{ecu}, according to the visual relevance of its content. % \HIGHLIGHT{% @@ -327,7 +329,7 @@ Furthermore, image saliency driven by visual attention modelling is used in the % \label{eq:Lagrangian} % \end{align} -The optimal \ac{ec} mode $m_{l,i}^*$, for the $i_{th}$ \ac{ecu} of size defined by the quadtree partitioning level $l$, is obtained by minimising the following lagrangian cost: +The optimal \ac{ec} mode $m_{l,i}^*$, for the $i_{th}$ \ac{ecu} of size defined by the quadtree partitioning level $l$, is obtained by minimising the following Lagrangian cost: \begin{equation}\label{EQ6:OptimalDepth} \small \newcommand{\argmin}{\operatornamewithlimits{arg\ min}} @@ -335,28 +337,28 @@ The optimal \ac{ec} mode $m_{l,i}^*$, for the $i_{th}$ \ac{ecu} of size defined % l^* = \argmin_{l=1..3} \left\{ D(l) + \lambda \left(1-\frac{t_I-t_c}{T_{\ac{ird}}}\right) R(l) \right\}, m_{l,i}^* = \argmin\limits_{\footnotesize \substack{l\in\left\{0...3\right\} \\ i\in\left\{1...4^l\right\}}}\left\{ D(m_{l,i}) + \left(1+\frac{t_C-t_I}{T_{I}}\right) \lambda R(m_{l,i}) \right\}, \end{equation} -where $R(m_{l,i})$ represents the number of overhead bits required to encode the \ac{ec} mode, and $D(m_{l,i})$ is sum of absolute error of the \ac{ecu}. +where $R(m_{l,i})$ represents the number of overhead bits required to encode the \ac{ec} mode, and $D(m_{l,i})$ is the sum of absolute error of the \ac{ecu}. -In equation \ref{EQ6:OptimalDepth}, besides the parameter $\lambda$ used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$ according to the duration of the error propagation. As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques} this method uses the ratio between the temporal distance from the current frame ($t_C$) to the next refresh frame ($t_I$) and the period of refresh frames ($T_{I}$). +In equation \ref{EQ6:OptimalDepth}, besides the parameter $\lambda$ used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation. As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques}, this method uses the ratio between the temporal distance from the current frame ($t_C$) to the next refresh frame ($t_I$) and the period of refreshing frames ($T_{I}$). %This weight varies between $\left[0;1 \right]$, achieving the lowest value for intra-coded frames and the maximum for the frame located immediately before an intra-coded one. This behaviour follows the fact that when a lost frame is closer to the next intra-coded frame, error propagation is lower. Thus, it is not worthwhile to spend too many overhead bits to reduce it. This is accomplished by increasing the cost of such bits according to the temporal distance between the missing frame and the next I frame. -Moreover, a dynamic value for $\lambda$ is also used to adjust the amount of overhead within the frame, according to the visual relevance of each image region given by a saliency map. +Moreover, a dynamic value for $\lambda$ is also used to adjust the amount of overhead within the frame, according to the visual relevance of each image region, determined using a saliency map. -The algorithm described above results in \acs{ecu} with variable size, which is defined by the depth of the quad-tree structure, i.e., level $l$. The partition size is jointly optimised along with the best \ac{ec} candidate method using Equation~\ref{EQ6:OptimalDepth}. Alternatively, a fixed \ac{ecu} size can also be used by defining a fixed depth for all quad-tree structures, i.e., constant $l$ for the entire frame. This can be achieved by using Equation~\ref{EQ6:OptimalDepth} to solely optimise the \ac{ec} methods for each \ac{ecu}. +The algorithm previously described results in \acs{ecu} with variable size, which is defined by the depth of the quad-tree structure, i.e., level $l$. The partition size is jointly optimised along with the best \ac{ec} candidate method using Equation~\ref{EQ6:OptimalDepth}. Alternatively, a fixed \ac{ecu} size can also be used by defining a fixed depth for all quad-tree structures, i.e., constant $l$ for the entire frame. This can be achieved by using Equation~\ref{EQ6:OptimalDepth} to solely optimise the \ac{ec} methods for each \ac{ecu}. Although this reduces the overall complexity of the proposed method, it leads to a non-optimal solution which may required additional overhead. In Section~\ref{Section5:ECSignallingPerformance} a comparison between a dynamic and fixed approach is provided. \subsection{Saliency detection} -The saliency maps were computed based on the well-known Itti's attention model \cite{SaliencyItti1998}, which applies a hierarchical decomposition based on a set of spatial features and then combines them with center-surround weighting to obtain the saliency map. -Since the Itti's model only produces saliency maps for still images, in this work it is further combined with a temporal feature. This is obtained by estimating the optical flow between two consecutive frames, based on the algorithm presented in \cite{HornSchunckOpticalFlow1981}. The final saliency map is obtained by combining the spatial and temporal components, $S_{s}$ and $S_{t}$ respectively, as follows, +The saliency maps are computed based on the well-known Itti's attention model \cite{SaliencyItti1998} that applies a hierarchical decomposition, based on a set of spatial features, which are combined with a center-surround weighting to obtain the saliency map. +Since the Itti's model only produces saliency maps for still images, in this work it is further combined with a temporal feature. This is obtained by estimating the optical flow between two consecutive frames, based on the algorithm presented in \cite{HornSchunckOpticalFlow1981}. The final saliency map is achieved by combining the spatial and temporal components, $S_{s}$ and $S_{t}$ respectively, as follows, \begin{equation}\label{eq:saliency} S = \rho S_{t} + (1-\rho) S_{s} \end{equation} -The parameter $\rho$ is used to give higher relevance to the temporal component, since motion is correlated with error propagation, producing longer artefacts and consequently higher impact in visual attention. In this work the temporal component was given weight slightly higher than 50\%, i.e. $\rho=0.6$. +The parameter $\rho$ is used to give higher relevance to the temporal component, since motion is correlated with error propagation, producing longer artefacts and consequently higher impact in visual attention. In this work the temporal component was given a weight slightly higher than 50\%, i.e. $\rho=0.6$. -On the one hand, higher values of saliency (i.e. regions with higher visual relevance) should result in lower values of $\lambda$, which tend to increase the overhead rate value on those regions, thus reducing \ac{ec} distortion at the decoder in case of loss. On the other hand, higher values of $\lambda$ should reduce the overhead, allowing higher \ac{ec} distortion at the decoder in case of loss. In order to achieve the aforementioned goal the following definition for $\lambda$ is used: +On the one hand, higher values of saliency (i.e. regions with higher visual relevance) should result in lower values of $\lambda$, which tend to increase the overhead rate value on those regions, thus reducing the \ac{ec} distortion at the decoder, in case of loss. On the other hand, higher values of $\lambda$ should reduce the overhead, allowing higher \ac{ec} distortion at the decoder. In order to achieve the aforementioned goal, the following definition for $\lambda$ is used: \begin{equation}\label{eq:lambdasaliency} % \lambda = \lambda_{0} - \frac{S_{\ac{ctu}} - \mean S}{\Delta S}, \Delta S > 0, % \lambda = \lambda_{0} + \frac{\Delta S}{S_{\ac{ctu}} - \mean S}, @@ -367,7 +369,7 @@ On the one hand, higher values of saliency (i.e. regions with higher visual rele \end{cases} \end{equation} where $s$ is the saliency value of the current \ac{ctu}, $\mean S$ is the average saliency value and $\Delta S$ is the difference between the maximum and minimum. -One should note that the method is not dependent on the existence of a saliency map. If the saliency map is constant or non-existent a fixed value of $\lambda$ is used, which means that no distinction is made across image regions. +One should note that this method does not depend on the existence of a saliency map. If the saliency map is constant or non-existent, a fixed value of $\lambda$ is used (i.e., $\lambda_0$) defined at the encoder, which means that no distinction is made across image regions. \subsection{Signalling the optimal EC modes} @@ -402,17 +404,17 @@ One should note that the method is not dependent on the existence of a saliency Figure~\ref{FIG5:signalling} illustrates the structure of the signalling information for each \ac{ctu} comprising a quadtree of \acs{ecu}. In this structure, one bit is used for the $split$ flag and two bits for the \ac{ec} mode associated to each \ac{ecu} ($ecm \in \{M1, M2, M3, M4\}$). The $split$ flag indicates whether the \ac{ecu} is further split into the next~level~($split=1$) or \ac{ecu} partitioning stops at the current level ($split=0$) and the corresponding $ecm$ modes are inserted. -Since no split is necessary at the highest level, then $split=1$ followed by the four $ecm$ modes for each \ac{ecu}. +Since no split is necessary at the deepest level, then $split=1$ followed by the four $ecm$ modes for each \ac{ecu}. This structure allows dynamic selection of different values for the level of the quad-tree partitioning ($l$) within each frame. -Adaptive arithmetic coding~\cite{ArithmeticCodingJun1994} is used to efficiently encode both the $split$ and $ecm$ symbols and further reduce the overhead. Note that the \ac{sei} information uses much less bitrate than the video signal, thus it is very unlikely to be hit during transmission. +Adaptive arithmetic coding~\cite{ArithmeticCodingJun1994} is used to efficiently encode both the $split$ and $ecm$ symbols, further reducing the overhead. Note that the \ac{sei} information uses much less bitrate than the video signal, thus it is very unlikely to be hit during transmission. \subsection{Enhancement-layer: \ac{ec} residue coding} -As described earlier, in order to allow further improvement of recovered slices, besides the best \ac{ec} mode, the proposed encoding scheme also allows encoding a controlled amount of \ac{ec} residue using the scalable extension of \ac{hevc} (see Figure~\ref{FIG5:BlockDiagram}). -The enhancement layer encodes the difference between the coded frame ($\hat f$) and its counterpart recovered by using the optimal \ac{ec} mode ($\tilde f$). +As described earlier in this section, in order to allow further improvement of recovered slices, besides the best \ac{ec} mode, the proposed encoding scheme also allows encoding a controlled amount of \ac{ec} residue using the scalable extension of \ac{hevc} (see Figure~\ref{FIG5:BlockDiagram}). +The enhancement layer module encodes the difference between the coded frame ($\hat f$) and its counterpart recovered by using the optimal \ac{ec} mode ($\tilde f$). Note that, the higher the accuracy of the best \ac{ec} mode, the lower the -amount of bits spent in the enhancement layer. +amount of bits spent by the enhancement layer. At the decoder-side, the reconstruction of a missing frame ($\tilde f$) follows three steps: \begin{enumerate} \item the missing frame is reconstructed using the optimal \ac{ec} methods for each \ac{ecu} $\tilde f'$ received from the encoder; @@ -426,7 +428,7 @@ At the decoder-side, the reconstruction of a missing frame ($\tilde f$) follows The additional residue decoded from the enhancement layer is particularly useful to improve the quality of those \acs{ctu} where the \ac{ec} strategies are not able to produce very accurate reconstruction. Moreover, since the enhancement layer only carries residual information the overhead is affordable. -Note that the generated bitstream is compliant with the standard, as the \ac{ec} signalling is encapsulated in \ac{sei} \ac{nalu}, while the error concealment residue is carried in \acs{nalu} related with the scalable extension. Then, an adapted decoder, capable of decoding both the \ac{sei} \acs{nalu} and the scalable \acs{nalu} is also able to take full benefit from the proposed method. Nevertheless, any standard \ac{hevc} decoder is also able to reconstruct the video, because it simply ignores the extra \acs{nalu}. +Note that the generated bitstream is compliant with the standard, as the \ac{ec} signalling is encapsulated in \ac{sei} \acs{nal}, while the error concealment residue is carried in \acs{nal} related with the scalable extension. Then, an adapted decoder, capable of decoding both the \ac{sei} and the scalable \ac{nal} units is also able to take full benefit from the proposed method. Nevertheless, any standard \ac{hevc} decoder is also able to reconstruct the video, because it simply ignores the extra \ac{nal}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -436,48 +438,27 @@ Note that the generated bitstream is compliant with the standard, as the \ac{ec} \label{Section5:ECSignallingPerformance} This section presents the performance of the proposed \ac{ec}-aware optimisation scheme using the \ac{hevc} reference encoder as the basis of the encoding framework. -Two different types of performance evaluation are presented. -In the first one, only robust \ac{ec} of intra-frames is evaluated, while in the second one the proposed \ac{ec}-aware optimisation is applied to all frames corresponding to a generic robust video transmission using optimal \ac{ec}. +Two different types of performance evaluation are presented. +In the first one, only robust \ac{ec} of intra-frames is evaluated, while in the second one the proposed \ac{ec}-aware optimisation is applied to all frames, corresponding to a generic robust video transmission using optimal \ac{ec}. +For comparison, it is used the \ac{psnr} obtained from the reference \ac{hevc} decoder using motion-copy (\MC) and frame-copy (\REF) for \ac{ec} of the lost frame. \subsection{Intra-frame EC optimisation} \label{SEC4:IntraPerformance} -In this section the performance of the proposed method is evaluated only for intra frames to demonstrate its efficiency in reducing error propagation. +In this section the performance of the proposed method is evaluated only for intra frames, in order to demonstrate its efficiency in reducing error propagation. Since intra \acs{cu} result in higher amount of bits, they also require a higher number of packets, making them more prone to transmission errors and losses. -Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets correspond to intra coded slices, which is a non-negligible probability of one of then being hit by transmission errors. +Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets correspond to intra coded slices, which is a non-negligible probability of one of them being hit by transmission errors. Moreover, by limiting the proposed method to intra frames, the amount of required overhead is reduced. +Since a relevant performance indicator is the error propagation over temporally predicted frames, this was evaluated using a single intra-frame loss. +Figures \ref{FIG5:singlelosslowdelay} and \ref{FIG5:singlelossramdon} show the \ac{psnr} results when frame Intra-frame $\#16$ is missing, for \ac{ld} and \ac{ra} encoding configurations, respectively. +These figures show the results for the proposed method (\PROP), the motion-copy method (\MC) and also an alternative proposed method using fixed \ac{ecu} size of $16\times16$ pixels. +These results indicate that the proposed method (\PROP) outperforms the reference one and it is able to achieve quality gains up to {2 dB} for \ac{ld} configuration (see Figure~\ref{FIG5:singlelosslowdelay}), decrease the error propagation and increase the quality of the frames affected from the loss of the intra frame, by up to {3 dB}. +The use of \ac{ra} configuration leads to higher quality degradation than LD, due the higher temporal distance of the reference frame used for \ac{ec} (Frame $\#8$ is used to recover the lost Frame~$\#16$). Note that when Intra-frame \#16 is lost, all frames within the range \#9 to \#31 (i.e., next I-frame) are affected by error propagation. +Nevertheless, the proposed method still achieves similar quality gains (approximately 2 dB). -% \begin{table}[tb] -% \newcommand{\somespaces}{} -% \centering -% \caption{Usage ratio of each error concealment candidate.} \label{tab:useratio} -% \TABLESVSPACE -% \begin{tabular}{x{2cm}|x{.8cm}|x{.7cm}|x{.7cm}|x{.7cm}|x{.7cm}} -% \toprule -% \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & -% \multirow{2}{*}{\vbox{\centering \textbf{Avg. Depth}}} & -% \multicolumn{4}{c}{\textbf{Avg. usage ratio (\%)}} \\ -% & & M1 & M2 & M3 & M4 \\ -% \hline -% Basketball Drill & 0.60 & 42.00 & 27.27 & 17.93 & 12.80 \\ \hline -% Kendo & 0.31 & 38.69 & 36.63 & 17.20 & 7.48 \\ \hline -% Park Scene & 0.25 & 40.66 & 31.99 & 20.46 & 6.89 \\ \hline -% Traffic & 0.13 & 49.93 & 27.32 & 15.99 & 6.76 \\ \hline -% Jockey & 0.19 & 41.38 & 20.02 & 22.51 & 16.09 \\ -% \bottomrule -% \end{tabular} -% \end{table} - -% Table \ref{tab:useratio} shows the average depth selected for \acs{ctu} and the usage ratio of each error concealment candidate. -% It is noticeable from the results that the proposed method leads to low average depth values (i.e., less than 1), which implicitly results in less signalling information. This also justifies the lower overhead obtained for the \PROP method shown in Table~\ref{tab:redundancy}. -% These results show that all error concealment candidates are used, which indicates that all of them are found relevant by the proposed method to recover the missing frames. -% The candidate methods based on the estimated motion field (i.e., M1, M2) are selected more often because they provide more accurate clues about the scene's motion than the others (i.e., M3, M4). - -%\subsubsection{Quality evaluation} - -\begin{figure}[!t] +\begin{figure}[tb] \centering \input{./042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameLowDelay.tex} \caption{Error propagation using \ac{ld} for the Basketball Drill sequence.} @@ -489,21 +470,16 @@ Moreover, by limiting the proposed method to intra frames, the amount of require \FIGSVSPACE \end{figure} - -Since a relevant performance indicator is the error propagation over temporally predicted frames, this was evaluated using a single intra-frame loss. -Figures \ref{FIG5:singlelosslowdelay} and \ref{FIG5:singlelossramdon} show the \ac{psnr} results when frame Intra-frame $\#16$ is missing, for \ac{ld} and \ac{ra} encoding configurations, respectively. These Figures show the results for the proposed method (\PROP), the motion-copy method (\MC), and also an alternative proposed method using fixed \ac{ecu} size of $16\times16$ pixels. -These results indicate that the proposed method (\PROP) outperforms the reference one and it is able to achieve quality gains up to \HIGHLIGHT{2 dB} for \ac{ld} configuration (see Figure~\ref{FIG5:singlelosslowdelay}), decreasing the error propagation and increasing the quality of the frames affected from the loss of the intra frame, by up to \HIGHLIGHT{3 dB}. -The use of \ac{ra} configuration leads to higher quality degradation than LD, due the higher temporal distance of the reference frame used for \ac{ec} (Frame $\#8$ is used to recover the lost Frame~$\#16$). Note that when Intra-frame \#16 is lost, all frames within the range \#9 to \#31 (i.e., next I-frame) are affected by error propagation. -Nevertheless, the proposed method still achieves similar quality gains (approximately 2 dB). - Further tests were performed to evaluate the effectiveness of the proposed method under different loss rates. In these experiments the packets corresponding to intra frames were randomly discarded to simulate loss events, at different \acs{plr} rates with an average burst length of 5 packets. -For each test condition 25 trials were performed and the average quality of the frames affected by errors (the lost frame and dependent ones) is presented in Table~\ref{tab:randomloss}. The table also shows the \ac{psnr} variance value in parentheses and the difference to the \ac{psnr} obtained with a reference \ac{hevc} decoder. +Table~\ref{tab:randomloss} presents the average quality of the frames affected by errors (the lost frame and dependent ones). +This table also shows the \ac{psnr} variance value in parentheses and the difference to the \ac{psnr} obtained with a reference \ac{hevc} decoder. % The absolute \ac{psnr} value is shown for the reference case (\MC), while the \ac{psnr} difference is presented for the proposed method. -The results confirm the superiority of the proposed method, outperforming the reference \ac{hevc} for both low and high \ac{plr}. Using the \ac{ld} coding configuration, the proposed method is able to achieve quality gains of \HIGHLIGHT{0.88 dB} for 3\% of \ac{plr} and \HIGHLIGHT{1.50 dB} for 10\% of \ac{plr}. In case of RA, higher quality gains are achieved, leading to an average improvement of \HIGHLIGHT{3.44 dB} for 10\% of \ac{plr}. +The results confirm the superiority of the proposed method, outperforming the reference \ac{hevc} for both low and high \ac{plr}. Using the \ac{ld} coding configuration, the proposed method is able to achieve quality gains of 0.88 dB for 3\% of \ac{plr} and 1.50 dB for 10\% of \ac{plr}. +In case of RA, higher quality gains are achieved, leading to an average improvement of 2.14 dB for 3\% of \ac{plr} and 3.44 dB for 10\% of \ac{plr}. It is also worthwhile to notice that the \ac{psnr} variance obtained with the \PROP method is lower, which indicates lower quality variations in the video segments affected by errors. %The quality gains achieved by the proposed method are similar to those achieved by previous works~\cite{ConcealmentIntraDataHidng2009,ConcealmentSignalling2008} for the same \acs{plr}. However, these require a higher amount of overhead (approximately $3\%$), thus the proposed method achieves a better trade-off between quality and overhead. -\begin{table}[tbp] +\begin{table}[!t] \newcommand{\somespaces}{} \centering \caption{Average quality (\ac{psnr}) of the frames affected by errors and its variance under random packet loss.} @@ -518,7 +494,7 @@ It is also worthwhile to notice that the \ac{psnr} variance obtained with the \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{3}{c|}{\textbf{PLR=3\%}} & \multicolumn{2}{c}{\textbf{PLR=10\%}}\\ & \MC & \PROP & $\Delta$PSNR& \MC & \PROP & $\Delta$PSNR \\ \midrule - \multicolumn{7}{c}{Low-delay configuration} \\ [2pt] + \multicolumn{7}{c}{Low-delay configuration (LD)} \\ [2pt] \hline Basketball Drill & 33.15 (0.42) & 34.26 (0.24) & \optbold{+1.11} & 32.97 (0.47) & 35.90 (0.27) & \optbold{+2.93}\\ \hline Race Horses & 30.78 (1.32) & 31.55 (0.94) & \optbold{+0.77} & 30.26 (1.55) & 31.12 (1.20) & \optbold{+0.86}\\ \hline @@ -529,7 +505,7 @@ It is also worthwhile to notice that the \ac{psnr} variance obtained with the Jockey & 36.00 (1.14) & 37.20 (1.05) & \optbold{+1.20} & 35.44 (1.21) & 36.74 (1.18) & \optbold{+1.30}\\ \hline \textbf{Average} & \textbf{34.37 (0.73)} & \textbf{35.25 (0.60)} & \textbf{+0.88} & \textbf{33.64 (0.79)} & \textbf{35.14 (0.65)} & \textbf{+1.50} \\ \midrule - \multicolumn{7}{c}{Random-access configuration} \\ [2pt] + \multicolumn{7}{c}{Random-access configuration (RA)} \\ [2pt] \hline Basketball Drill & 30.22 (8.46) & 30.81 (7.66) & \optbold{+0.59} & 29.19 (7.60) & 29.84 (6.88) & \optbold{+0.65} \\ \hline Race Horses & 27.35 (8.02) & 29.92 (5.89) & \optbold{+2.57} & 25.75 (7.76) & 28.61 (6.14) & \optbold{+2.86} \\ \hline @@ -546,12 +522,12 @@ It is also worthwhile to notice that the \ac{psnr} variance obtained with the %\subsubsection{Overhead analysis} -Subsequently, the percentage of overhead bitrate introduced by the proposed method is compared against fixed \ac{ecu} sizes of $32\times32$ and $16\times16$, i.e., $Level=1$ and $Level=2$. -These results are presented in Table \ref{TAB6:RedundancyIntra} and were obtained for a total bitrate as shown in Table~\ref{TableTestSequences}. +Subsequently, the percentage of overhead bitrate introduced by the proposed method is compared against fixed \ac{ecu} sizes of $32\times32$ and $16\times16$, i.e., $Level=1$ and $Level=2$. +These results are presented in Table \ref{TAB6:RedundancyIntra} and were obtained for a total bitrate, as shown in Table~\ref{TableTestSequences}. It is noticeable from the results that reducing the \ac{ecu} size (see results for $32\times32$ and $16\times16$) increases the overhead ratios as more symbols are transmitted (one for each \ac{ecu}). -While the maximum overhead of these fixed partitions is \HIGHLIGHT{$1.81\%$} (Traffic) the proposed dynamic partitioning method is able to reduce the overhead to a maximum of \HIGHLIGHT{$0.19\%$} (Kendo). -Moreover, comparing the results of Table \ref{TAB6:RedundancyIntra} with the video quality results obtained in case of packet losses reveals that, in the proposed method, only a small reduction in the reconstruction quality even with a significantly lower overhead. -This indicates that using dynamic partitioning allows to achieve a better trade-off between the amount of overhead and the \c{ec} performance. +While the maximum overhead of these fixed partitions is $1.81\%$ (Traffic), the proposed dynamic partitioning method is able to reduce the overhead to a maximum of $0.19\%$ (Kendo). +Moreover, comparing the results of Table \ref{TAB6:RedundancyIntra} with the video quality results obtained in case of packet losses reveals in the proposed method only a small reduction in the reconstruction quality, even with a significantly lower overhead. +This indicates that using dynamic partitioning allows to achieve a better trade-off between the amount of overhead and the \ac{ec} performance. In summary, the proposed technique is able to enhance the performance of the standard decoding of intra coded frames by finding the best error concealment method at the encoder and sending such information with a small amount of signalling overhead. \begin{table}[tbp] @@ -590,7 +566,7 @@ In summary, the proposed technique is able to enhance the performance of the sta % \subsection{Overall performance evaluation} \label{SEC4:OverallPerformance} -In this section the proposed \ac{ec}-aware optimisation scheme is evaluated for a generalised robust video transmission of both intra and inter coded frames. +In this section the proposed \ac{ec}-aware optimisation scheme is evaluated for a generalised robust video transmission of both intra and inter coded frames. The experimental results presented in this section were obtained by applying the proposed technique to all video frames. This section is organised in the following subsections: \begin{itemize} @@ -607,43 +583,43 @@ This section is organised in the following subsections: The \ac{ec} mode usage distribution is an indicator of the usefulness of each \ac{ec} mode. Table \ref{tab:useratio} shows the average $Level$ selected for the \ac{ecu} quadtree and the usage percentage of each \ac{ec} mode. -It is noticeable from these results that the proposed method leads to low average $Level$ values (lower than 1), resulting in larger \acs{ecu}, which requires a lower amount of \ac{ecu} to be transmitted than if higher values of $Level$ were chosen. This implicitly results in low signalling information. +It is noticeable from these results that the proposed method leads to low average $Level$ values (lower than 1), resulting in larger \acs{ecu}, which requires a lower amount of \ac{ecu} to be transmitted than if higher values of $Level$ were chosen. This implicitly results in low signalling information. This also justifies the low overhead shown in Table~\ref{tab:redundancy_signalling}. -\begin{table}[b] -\newcommand{\somespaces}{} -\centering\small -\caption{Average Level and usage ratio of each \ac{ec} mode.} \label{tab:useratio} -\TABLESVSPACE -\footnotesize -\begin{tabular}{p{3.2cm}|x{1.5cm}|x{1.5cm}|x{1.5cm}|x{1.5cm}|x{1.5cm}} -\toprule -\multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & -\multirow{2}{*}{\vbox{\centering \textbf{Avg. Level}}} & -\multicolumn{4}{c}{\textbf{Avg. usage ratio (\%)}} \\ - & & M1 & M2 & M3 & M4 \\ \hline -Basketball Drill & 0.81 & 59.61 & 16.62 & 14.40 & 9.37 \\ \hline -Book Arrival & 0.34 & 66.77 & 18.79 & 7.15 & 7.29 \\ \hline -BQSquare & 0.41 & 15.23 & 0.64 & 72.97 & 11.16 \\ \hline -Four People & 0.17 & 85.84 & 3.22 & 8.09 & 2.86 \\ \hline -Kimono & 0.88 & 33.76 & 23.87 & 26.03 & 16.35 \\ \hline -Park Scene & 0.69 & 39.34 & 13.58 & 34.19 & 12.89 \\ \hline -People on Street & 1.51 & 31.17 & 34.39 & 16.15 & 18.29 \\ \hline -Race Horses & 1.62 & 34.05 & 29.89 & 17.06 & 19.01 \\ \hline -Tennis & 1.17 & 25.27 & 30.03 & 23.22 & 21.48 \\ %\hline -\bottomrule -\end{tabular} -\end{table} - The results in Table \ref{tab:useratio} also show that all \ac{ec} modes are effectively used, which indicates that all of them are found relevant to recover the missing data. The \ac{ec} candidate methods based on motion field estimation (i.e., M1, M2) are selected more often because they provide more accurate representation of the scene's motion than the others (i.e., M3, M4). +\begin{table}[tb] + \newcommand{\somespaces}{} + \centering\small + \caption{Average Level and usage ratio of each \ac{ec} mode.} \label{tab:useratio} + \TABLESVSPACE + \footnotesize + \begin{tabular}{p{3.2cm}|x{1.5cm}|x{1.5cm}|x{1.5cm}|x{1.5cm}|x{1.5cm}} + \toprule + \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & + \multirow{2}{*}{\vbox{\centering \textbf{Avg. Level}}} & + \multicolumn{4}{c}{\textbf{Avg. usage ratio (\%)}} \\ + & & M1 & M2 & M3 & M4 \\ \hline + Basketball Drill & 0.81 & 59.61 & 16.62 & 14.40 & 9.37 \\ \hline + Book Arrival & 0.34 & 66.77 & 18.79 & 7.15 & 7.29 \\ \hline + BQSquare & 0.41 & 15.23 & 0.64 & 72.97 & 11.16 \\ \hline + Four People & 0.17 & 85.84 & 3.22 & 8.09 & 2.86 \\ \hline + Kimono & 0.88 & 33.76 & 23.87 & 26.03 & 16.35 \\ \hline + Park Scene & 0.69 & 39.34 & 13.58 & 34.19 & 12.89 \\ \hline + People on Street & 1.51 & 31.17 & 34.39 & 16.15 & 18.29 \\ \hline + Race Horses & 1.62 & 34.05 & 29.89 & 17.06 & 19.01 \\ \hline + Tennis & 1.17 & 25.27 & 30.03 & 23.22 & 21.48 \\ %\hline + \bottomrule + \end{tabular} +\end{table} \subsubsection{Evaluation of optimal \ac{ec} mode decision} \label{sec:ECModeSelectionPerformance} -The evaluation of the proposed saliency-weighted \ac{ec} mode decision (\DYNLAMBDA) was carried out under random loss events and compared with the reference methods. The video quality was evaluated by using two quality metrics: \ac{psnr} and a \ac{fsim}~\cite{FSIM_TIP2011} in order to validate the performance of the proposed method in a consolidated manner. -Realistic conditions were used for transmission, by allowing independent losses for any of type of \acs{nal}, hence there is no guarantee that the decoder conditions are always equal to those at the encoder, which means that sub-optimal \ac{ec} may be used at the decoder. +The evaluation of the proposed saliency-weighted \ac{ec} mode decision (\DYNLAMBDA) was carried out under random loss events and it was compared with the reference methods. +The video quality was evaluated by using two quality metrics: \ac{psnr} and a \ac{fsim}~\cite{FSIM_TIP2011}, in order to validate the performance of the proposed method in a consolidated manner. +Realistic conditions were used for transmission, by allowing independent losses for any type of \acs{nal}, hence there is no guarantee that the decoder conditions are always equal to those at the encoder, which means that sub-optimal \ac{ec} may be used at the decoder. The results obtained without saliency weighting are also shown for comparison (\FIXEDLAMBDA). In this case a constant $\lambda_0=10$ is used. Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM} show the average quality under different \acs{plr} for the \ac{ld} configuration. @@ -651,48 +627,41 @@ Table~\ref{TAB4:SignallingPSNR} shows the \ac{psnr} and weighted \ac{psnr} (\ac{ The \ac{mse} weights used in the \ac{wpsnr} are defined by the saliency values~\cite{SaliencyQualityMetric2011}. Table~\ref{TAB4:SignallingFSIM} shows the results obtained for the \ac{fsim} quality metric. Note that, the absolute value is shown for the reference case (\REF), while the differential values are shown for the remaining methods. - -\input{\ConcealmentAwareDir/Results/TableSignallingPerformance.tex} - -From the \ac{psnr} results shown in Table~\ref{TAB4:SignallingPSNR}, one may conclude that the proposed method (\DYNLAMBDA) consistently outperforms the reference ones for both low and high \acs{plr}. Average \ac{psnr} gains up to 2.95~dB for PLR=3\% and 2.58~dB for PLR=10\% are achieved. It is also worthwhile to notice that higher performance is obtained for sequences with high motion, such as Basketball Drill, which leads to \ac{psnr} gains up 4.82~dB. +From the \ac{psnr} results shown in Table~\ref{TAB4:SignallingPSNR}, one may conclude that the proposed method (\DYNLAMBDA) consistently outperforms the reference ones for both low and high \acs{plr}. Average \ac{psnr} gains up to 2.95~dB for PLR=3\% and 2.58~dB for PLR=10\% are achieved. It is also worthwhile to notice that higher performance is obtained for sequences with high motion, such as Basketball Drill, which results in \ac{psnr} gains up 4.82~dB. Moreover, the results of Table~\ref{TAB4:SignallingFSIM} confirm the quality improvement achieved by the proposed method, leading to an increase of \ac{fsim} up to 4.32, which indicates that the proposed method is able to recover important image features with higher accuracy than the others. -Table~\ref{tab:redundancy_signalling} shows the percentage of overhead in the total rate with and without saliency-weighted optimisation, i.e., \FIXEDLAMBDA and \DYNLAMBDA, respectively. -As shown in the Table, the saliency-weighted optimisation reduces the amount of overhead by dynamically assigning unequal number of bits to image regions of different visual importance. -These results show that using an adaptive value of $\lambda$, based on the saliency values, leads to an average overhead reduction of 28\% in comparison with the case where no saliency is used, i.e., constant $\lambda$. +\input{042_ConcealmentAwareEncoding/Results/TableSignallingPerformance.tex} -It is worth to note that using the saliency-weighted optimisation to reduce the overhead does not compromise the video quality, as can be seen in the results shown in Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM}. The \ac{psnr} values for \FIXEDLAMBDA and \DYNLAMBDA are very close, but in overhead savings are significant for most sequences. +Table~\ref{tab:redundancy_signalling} shows the percentage of overhead in the total rate with and without saliency-weighted optimisation, i.e., \FIXEDLAMBDA and \DYNLAMBDA, respectively. +As shown in this table, the saliency-weighted optimisation reduces the amount of overhead by dynamically assigning unequal number of bits to image regions of different visual importance. +These results show that using an adaptive value of $\lambda$, based on the saliency values, leads to an average overhead reduction of 28\% in comparison with the case where no saliency is used, i.e., constant $\lambda$. +It is worth to note that using the saliency-weighted optimisation to reduce the overhead does not compromise the video quality, as can be seen in the results shown in Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM}. +The \ac{psnr} values for \FIXEDLAMBDA and \DYNLAMBDA are very close, but in terms of overhead savings the results are significant for most sequences. The achieved \ac{wpsnr} and \ac{fsim} results further demonstrate the effectiveness of the saliency-based optimisation, because the lower overhead does not affect the most relevant image regions, i.e., those with higher saliency values. Thus, the proposed method using a adaptive $\lambda$ (i.e., using saliency) is able to not only achieve higher \ac{ec} quality than the reference methods, but also higher quality gains than using a constant $\lambda$ (i.e., without saliency). -This confirms the two-fold superiority of adaptive $\lambda$, i.e. not only the total overhead is reduced but also higher quality gains are achieved. +This confirms the two-fold superiority of adaptive $\lambda$, i.e. not only the total overhead is reduced but also higher quality gains are achieved. +%\todo{\HERE} \subsubsection{Error propagation evaluation} -In this section the error propagation is evaluated using single loss events and compared against the reference methods. In this simulation a whole frame loss is enforced in order to assess the impact of error propagation in the worst case scenarios. Experimental evaluations were performed for both intra and inter-coded frame loss. -Figure \ref{FIG5:singlelossInter} shows the \ac{psnr} over a \ac{gop} with an \ac{idr} period of 16 frames, whole frame \#6 is missing in sequences Basketball Drill and People On Street. -The figure shows the results obtained by decoding the stream using the proposed method with adaptive $\lambda$ but without any information sent in the enhancement layer. For comparison, the \ac{psnr} obtained from the reference \ac{hevc} decoder using \MC and \REF for \ac{ec} of the lost frame. +In this section the error propagation is evaluated using single loss events and compared against the reference methods. +In this simulation a whole frame loss is enforced in order to assess the impact of error propagation in the worst case scenario. +Experimental evaluations were performed for both intra and inter-coded frame loss. +Figure \ref{FIG5:singlelossInter} shows the \ac{psnr} over a \ac{gop} with an \ac{idr} period of 16 frames, where the whole frame \#6 is missing in sequences Basketball Drill and People On Street. +%The figure shows the results obtained by decoding the stream using the proposed method with adaptive $\lambda$ but without any information sent in the enhancement layer. +Figure \ref{FIG5:singlelossIntra} shows the \ac{psnr} for the same conditions but with a loss event on the intra-coded Frame \#16, in sequences Kimono and Tennis. -\input{\ConcealmentAwareDir/Results/ECMQualitySingleloss.tex} +\input{042_ConcealmentAwareEncoding/Results/ECMQualitySingleloss.tex} -Figure \ref{FIG5:singlelossIntra} shows the \ac{psnr} for the same conditions but with a loss event on the intra-coded Frame \#16, in sequences Kimono and Tennis. These results reveal that the proposed method is able to improve the reconstruction of inter-coded frames and also outperforms the existing techniques whenever a intra-coded frame is missing. -These results also indicate that the proposed method outperforms both reference methods since the time instant when the loss occurs. The lost frame itself is recovered with much higher quality than using the reference \ac{ec} methods. This is due to the use of the optimal \ac{ec} mode to recover each \ac{ecu}, following the \ac{sei} information received from the encoder. -In this case, the proposed method is able to achieve quality gains up to \HIGHLIGHT{5~dB}, decreasing the error propagation and increasing the \ac{psnr} of affected frames by up to \HIGHLIGHT{6~dB}. - -The visual impact of using the proposed \ac{ec}-aware encoding can be evaluated by observing the recovered frame (i.e., frame \#6) of Basketball Drill as an example. Figure~\ref{FIG5:RecoveredExample} shows that higher visual quality and less reconstruction artefacts are obtained in comparison with the \MC method. It is worthwhile to note that the proposed method is able to reconstruct the object edges with higher accuracy than the reference \MC. +These results reveal that the proposed method is able to improve the reconstruction of inter-coded frames and also outperforms the existing techniques, whenever a intra-coded frame is missing. +These results also indicate that the proposed method outperforms both reference methods since the time instant when the loss occurs. The lost frame itself is recovered with much higher quality than using the reference \ac{ec} methods. +This is due to the use of the optimal \ac{ec} mode to recover each \ac{ecu}, benefiting from the \ac{sei} information received from the encoder. +In this case, the proposed method is able to achieve quality gains up to 5~dB, decreasing the error propagation and increasing the \ac{psnr} of affected frames by up to 6~dB. +The visual impact of using the proposed \ac{ec}-aware encoding can be evaluated by observing the recovered frame (i.e., frame \#6) of Basketball Drill, as an example. Figure~\ref{FIG5:RecoveredExample} shows that higher visual quality and less reconstruction artefacts are obtained in comparison with the \MC method. It is worthwhile to note that the proposed method is able to reconstruct the object edges with higher accuracy than the reference \MC. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -% SECTION -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%\section{\HIGHLIGHT{Extensions to the \ac{ec} optimization}} -% -% -%\subsection{\HIGHLIGHT{Increased number of \ac{ec} candidates}} -% -%\subsection{\HIGHLIGHT{\ac{ec} signalling using motion vector differences}} - %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % SECTION %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -707,12 +676,12 @@ In this experiment both layers, i.e., video stream and enhancement layer with th \subsection{Quality evaluation} -\input{\ConcealmentAwareDir/Results/TableOveralPerformance.tex} - Tables~\ref{tab:concealmentResidueRandomLoss_PLR3}~and~\ref{tab:concealmentResidueRandomLoss_PLR10} present the average \ac{psnr} of decoded video for different \acs{plr} and different percentages of total overhead (i.e., signalling rate plus enhancement layer rate). The absolute \ac{psnr} value is shown for the reference case, while the \ac{psnr} difference is presented for the remaining methods. These results show that transmitting the \ac{ec} residue in the enhancement layer further improves the reconstruction quality obtained when only the optimal \ac{ec} method is used (see previous section). By only using 10\% of the total bit rate, the proposed \ac{ec}-aware coding scheme is able to outperform the reference \ac{hevc}, the \MC and the method proposed in \cite{ROPERefGenJan2010} for both \ac{ld} and \ac{ra} configurations. -Using 20\% of the total bitrate the quality gains increase up to \HIGHLIGHT{6.90~dB} for the Basketball Drill sequence when comparing with the reference \ac{hevc}, achieving an average gain of \HIGHLIGHT{3.79~dB} for the \ac{ld} configuration (\HIGHLIGHT{1.33}~dB for the \ac{ra}). +Using 20\% of the total bitrate the quality gains increase up to 6.90~dB for the Basketball Drill sequence when comparing with the reference \ac{hevc}, achieving an average gain of 3.79~dB for the \ac{ld} configuration (1.33~dB for the \ac{ra}). + +\input{042_ConcealmentAwareEncoding/Results/TableOveralPerformance.tex} When comparing the results of Tables~\ref{tab:concealmentResidueRandomLoss_PLR3}~and~\ref{tab:concealmentResidueRandomLoss_PLR10} with the spatial and temporal information of each sequence (SI/TI in Table~\ref{TableTestSequences}) it is noticeable that higher quality gains are achieved for sequences with higher motion information, i.e., Basketball~Drill and Tennis sequences. Moreover, for sequences with higher spatial information (e.g., BQSquare and People on Street) considerable quality gains are also achieved. @@ -725,23 +694,26 @@ To provide an informal subjective quality comparison between the proposed method The visible artefacts demonstrate how the decoder is able to recover from whole frame loss in both methods, and also the impact of error propagation. Figure~\ref{FIG5:ReconstructedFrames} shows that the proposed method clearly improves the reconstructed quality in comparison with \MC. -\begin{figure}[!t] +\begin{figure}[t] \centering - \includegraphics[width=\textwidth]{\ConcealmentAwareDir/Figures/ReconstructedFrames} + \includegraphics[width=\textwidth]{042_ConcealmentAwareEncoding/Figures/ReconstructedFrames} \caption{Crop of the decoded video frame for the Race Horses and Kimono sequences (Loss event at Frame \#6)} \label{FIG5:ReconstructedFrames} \FIGSVSPACE \end{figure} -A closer look at the Race Horses sequence reveals that the horse rider is not correctly reconstructed by the \MC method (highlighted by the circles and ellipses). As shown in the figure, this not only results in a more accurate reconstruction of the lost frame, but also increases the visual quality of the leading frames. In the Kimono sequence, some artefacts are also noticeable in the \MC case, but not visible in the proposed method (highlighted with circles). Moreover, the edges of the women's kimono (highlighted with ellipses) are also improved by the proposed method, resulting in a smooth reconstruction without error propagation. From the results discussed above it is clearly observed that the proposed method is able to reconstruct the contours of the moving parts more accurately than traditional \ac{ec} algorithms. +A closer look at the Race Horses sequence reveals that the horse rider is not correctly reconstructed by the \MC method (highlighted by the circles and ellipses). As shown in the figure, this not only results in a more accurate reconstruction of the lost frame, but also increases the visual quality of the subsequent frames. +In the Kimono sequence, some artefacts are also noticeable in the \MC case, but not visible when the proposed method is used (highlighted with circles). +Moreover, the edges of the women's kimono (highlighted with ellipses) are also improved by the proposed method, resulting in a smooth reconstruction without error propagation. +From the results discussed above, it is clearly observed that the proposed method is able to reconstruct the contours of the moving parts more accurately than traditional \ac{ec} algorithms. \subsection{Comparison with previous works} -The results were compared with previous works where fair comparisons could be made. +The achieved results were compared with previous works, where fair comparisons could be made. In \cite{ConcealmentSignalling2008} a method was proposed to improve the recovery quality by signalling the suitable \ac{mv} to be used for \ac{ec}. -The proposed \ac{ec}-aware scheme of Figure \ref{FIG5:BlockDiagram} is able to achieve similar quality gains with less amounts of redundancy achieving a better trade-off between quality and overhead. -In comparison with the work described in \cite{SignallingRobustScalable2015}, the proposed method is able to select among more complex \ac{ec} methods which allows for higher quality gains. +The proposed \ac{ec}-aware scheme of Figure \ref{FIG5:BlockDiagram} is able to achieve similar quality gains with less amounts of redundancy, allowing a better trade-off between quality and overhead. +In comparison with the work described in \cite{SignallingRobustScalable2015}, the proposed method is able to select among more complex \ac{ec} methods, which allows for higher quality gains. Both of the above mentioned methods search for the optimal \ac{ec} method at the encoder-side, which results in similar complexity as the proposed method. @@ -751,17 +723,19 @@ Since the proposed \ac{ec}-aware scheme is able to achieve higher quality withou An advantage of using a feedback channel is the low extra complexity required to implement robust coding because the encoder receives specific information about transmission errors. This allows straightforward spatio-temporal localization of the lost data without computational effort. -Table~\ref{tab:overall_gain_ratio} shows the average quality differences between the reference method (\REF) and the proposed one plus three other previous works. Two different test conditions are shown in the Table: error-free and PLR=10\%. Moreover, an efficiency measure $R$ is defined as the ratio between the quality gains obtained under packet loss and the quality decrease in the error-free case, i.e., +Table~\ref{tab:overall_gain_ratio} shows the average quality differences between the reference method (\REF) and the proposed one plus three other previous works. Two different test conditions are shown in the table: error-free and PLR=10\%. Moreover, an efficiency measure $R$ is defined, as the ratio between the quality gains obtained under packet loss and the quality decrease in the error-free case, i.e., \begin{equation} -R = \abs{\frac{\Delta PSNR(\ac{plr}=10\%)}{\Delta PSNR(Error\text{-}free)}}. +R = \frac{\Delta PSNR(\ac{plr}=10\%)}{-\Delta PSNR(Error\text{-}free)}. \end{equation} -The results shown in Table~\ref{tab:overall_gain_ratio}, demonstrate that the effectiveness of the \ac{ec}-aware coding scheme is higher than all other methods under comparison. In fact, not only the absolute \ac{psnr} difference (i.e., \ac{psnr} gain) is higher for the same \ac{plr}, but also the efficiency of the overhead is also higher. This means that the rate spent to encode the signalling information and the enhancement residue carry more useful information than the other methods. +The results shown in Table~\ref{tab:overall_gain_ratio}, demonstrate that the effectiveness of the \ac{ec}-aware coding scheme is higher than all other methods under comparison. In fact, not only the absolute \ac{psnr} difference (i.e., \ac{psnr} gain) is higher for the same \ac{plr}, but also the efficiency of the overhead is also higher. +This means that the overhead bits spent to encode the signalling information and the enhancement residue carry more useful information than the other methods. For instance, in comparison with the method in \cite{DualFrameME2015} the proposed method leads to higher quality degradation in the error-free case. However, for PLR=10\% it clearly outperforms the method in \cite{DualFrameME2015} with a much higher value of $R$, indicating a far superior efficiency. Results in the table~\ref{tab:overall_gain_ratio} also show that the small loss of coding efficiency in error-free transmission is clearly surpassed by the quality gains under error-prone transmission conditions. The methods in \cite{ROPERefGenJan2010,DualFrameME2015} are based on modified predictions, while the method in \cite{VPxResiliencePCS2016} uses redundant data without any further optimisation. -Both of these approaches require low additional computation complexity. However, as observed in the results shown in Table ~\ref{tab:overall_gain_ratio} the lower additional complexity also corresponds to lower quality gains in case of transmission errors. +Both of these approaches require low additional computational complexity. +However, as observed in the results shown in Table~\ref{tab:overall_gain_ratio}, the lower additional complexity also corresponds to lower quality gains in case of transmission errors. @@ -771,7 +745,7 @@ The proposed method uses \ac{ec} techniques with different levels of computati \begin{table}[t] \newcommand{\somespaces}{} \centering -\caption{Comparison against existing methods. \HIGHLIGHT{Sort methods}}\label{tab:overall_gain_ratio} +\caption{Comparison against existing methods.}\label{tab:overall_gain_ratio} \footnotesize % \resizebox{\columnwidth}{!}{ \begin{tabular}{p{2.4cm}x{2.4cm}x{2.4cm}x{2cm}} @@ -782,8 +756,8 @@ The proposed method uses \ac{ec} techniques with different levels of computati % \hline \midrule \AVG & -0.88 & 1.64 & 1.9 \\ \midrule %\hline -\textit{Ref~\cite{DualFrameME2015}} & -0.24 & 1.42 & 5.9 \\ \midrule %\hline \textit{Ref~\cite{VPxResiliencePCS2016}} & -0.37 & 2.03 & 5.5 \\ \midrule %\hline +\textit{Ref~\cite{DualFrameME2015}} & -0.24 & 1.42 & 5.9 \\ \midrule %\hline \PROP & -0.35 & 2.10 & 6.0 \\ %\hline \bottomrule \end{tabular} @@ -801,4 +775,4 @@ In this chapter an \ac{ec}-aware encoding scheme is proposed by incorporating lo % The proposed method relies on loss simulation at the encoder-side to optimise the error concealment proposed. Then, the optimal error concealment is signalled to the decoder. The novelty of this method is to include optimal \ac{ec} mode decision at the encoder, to be used by the decoder in case of packet loss, and also to allow encoding of the \ac{ec} residue using the enhancement layer of the \ac{hevc} standard. The overall results demonstrate that the proposed method is able to efficiently reconstruct lost frames and reduce the error propagation compared with other reference methods. -A consistent quality improvement achieved at the cost of low overhead, allows the use of this method in broadcasting video services and applications where packet loss probability is not negligible. +A consistent quality improvement achieved at the cost of low overhead allows the use of this method in broadcasting video services and applications where packet loss probability is not negligible. diff --git a/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameLowDelay.tex b/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameLowDelay.tex index 45db658..2f0572a 100644 --- a/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameLowDelay.tex +++ b/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameLowDelay.tex @@ -1,5 +1,5 @@ -\newcommand{\pltw}{0.75\columnwidth} -\newcommand{\plth}{0.3\textheight} +\newcommand{\pltw}{0.7\columnwidth} +\newcommand{\plth}{0.25\textheight} \begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] load 'carreira_phd_thesis_gnuplot_template.cfg' set xlabel '\TitleFont Frame Number' diff --git a/042_ConcealmentAwareEncoding/Results/TableOveralPerformance.tex b/042_ConcealmentAwareEncoding/Results/TableOveralPerformance.tex index 43a6233..e42c991 100644 --- a/042_ConcealmentAwareEncoding/Results/TableOveralPerformance.tex +++ b/042_ConcealmentAwareEncoding/Results/TableOveralPerformance.tex @@ -19,7 +19,7 @@ % & \multirow{2}{*}{\vbox{\centering \AVG}} & \multicolumn{2}{c}{\PROP} \\ % & & & & $10\%$ & $20\%$ & & & & $10\%$ & $20\%$ \\ % \midrule -% & \multicolumn{10}{c}{Low-delay configuration} \\ [2pt] +% & \multicolumn{10}{c}{Low-delay configuration (LD)} \\ [2pt] % \hline % Basketball Drill & 33.31 & +2.10 & +3.86 & +4.62 & +4.56 & 28.22 & +2.34 & +5.32 & +6.26 & +6.90 \\ \hline % Book Arrival & 36.21 & +1.33 & +2.07 & +2.71 & +2.99 & 31.01 & +2.48 & +4.49 & +5.60 & +6.31 \\ \hline @@ -33,7 +33,7 @@ % \textbf{Average} & \textbf{--} & \textbf{+1.55} & \textbf{+1.88} & \textbf{+3.10} & \textbf{+3.33} % & \textbf{--} & \textbf{+1.36} & \textbf{+1.96} & \textbf{+3.13} & \textbf{+3.79} \\ % \midrule -% & \multicolumn{10}{c}{Random-access configuration} \\ [2pt] +% & \multicolumn{10}{c}{Random-access configuration (RA)} \\ [2pt] % \hline % Basketball Drill & 35.40 & +0.17 & +0.20 & +1.39 & +1.53 & 29.92 & +0.14 & +1.45 & +1.84 & +2.42 \\ \hline % Book Arrival & 37.26 & +0.05 & +0.72 & +1.31 & +1.28 & 32.27 & +0.13 & +2.58 & +2.74 & +2.91 \\ \hline @@ -55,7 +55,7 @@ %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% Split -\begin{table}[!t] +\begin{table}[t] \centering \caption{Average quality under 3\% of PLR for different methods and overhead ratios.} \label{tab:concealmentResidueRandomLoss_PLR3} @@ -68,7 +68,7 @@ & \multirow{2}{*}{\vbox{\centering \REF}} & \multirow{2}{*}{\vbox{\centering \MC}} & \multirow{2}{*}{\vbox{\centering \AVG}} & \multicolumn{2}{c}{\textit{Proposed}} \\ & & & & $10\%$ & $20\%$ \\ \midrule -& \multicolumn{5}{c}{Low-delay configuration} \\ [2pt] +\multicolumn{6}{c}{Low-delay configuration (LD)} \\ [2pt] \hline Basketball Drill & 33.31 & +2.10 & +3.86 & +4.62 & +4.56 \\ \hline Book Arrival & 36.21 & +1.33 & +2.07 & +2.71 & +2.99 \\ \hline @@ -81,7 +81,7 @@ Race Horses & 29.08 & +2.34 & +4.52 & +5.35 & +6.02 \\ \hline Tennis & 32.71 & +1.32 & +3.13 & +3.81 & +4.97 \\ \hline \textbf{Average} & \textbf{--} & \textbf{+1.55} & \textbf{+1.88} & \textbf{+3.10} & \textbf{+3.33} \\ \midrule -& \multicolumn{5}{c}{Random-access configuration} \\ [2pt] +\multicolumn{6}{c}{Random-access configuration (RA)} \\ [2pt] \hline Basketball Drill & 35.40 & +0.17 & +0.20 & +1.39 & +1.53 \\ \hline Book Arrival & 37.26 & +0.05 & +0.72 & +1.31 & +1.28 \\ \hline @@ -98,7 +98,7 @@ Tennis & 33.86 & +0.16 & +0.18 & +0.98 & +1.48 \\ \hline \end{table} -\begin{table}[!t] +\begin{table}[t] \centering \caption{Average quality under 10\% of PLR.} \label{tab:concealmentResidueRandomLoss_PLR10} @@ -111,7 +111,7 @@ Tennis & 33.86 & +0.16 & +0.18 & +0.98 & +1.48 \\ \hline & \multirow{2}{*}{\vbox{\centering \REF}} & \multirow{2}{*}{\vbox{\centering \MC}} & \multirow{2}{*}{\vbox{\centering \AVG}} & \multicolumn{2}{c}{\textit{Proposed}} \\ & & & & $10\%$ & $20\%$\\ \midrule -& \multicolumn{5}{c}{Low-delay configuration} \\ [2pt] +\multicolumn{6}{c}{Low-delay configuration (LD)} \\ [2pt] \hline Basketball Drill & 28.22 & +2.34 & +5.32 & +6.26 & +6.90 \\ \hline Book Arrival & 31.01 & +2.48 & +4.49 & +5.60 & +6.31 \\ \hline @@ -124,7 +124,7 @@ Race Horses & 24.19 & +0.63 & +2.53 & +2.75 & +3.74 \\ \hline Tennis & 27.08 & +1.30 & +3.51 & +4.58 & +6.61 \\ \hline \textbf{Average} & \textbf{--} & \textbf{+1.36} & \textbf{+1.96} & \textbf{+3.13} & \textbf{+3.79} \\ \midrule -& \multicolumn{5}{c}{Random-access configuration} \\ [2pt] +\multicolumn{6}{c}{Random-access configuration (RA)} \\ [2pt] \hline Basketball Drill & 29.92 & +0.14 & +1.45 & +1.84 & +2.42 \\ \hline Book Arrival & 32.27 & +0.13 & +2.58 & +2.74 & +2.91 \\ \hline diff --git a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex index 74d734a..376f611 100644 --- a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex +++ b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex @@ -1,7 +1,7 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % CHAPTER %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\chapter{Adaptive Error Robustness for Highly Efficient Video Coding} +\chapter{Adaptive Error Robustness for Highly Efficient Video Coding \VERSION{1}} \label{Chapter:EndToEndRobustSystem} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -9,109 +9,142 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Introduction} -The effectiveness of the error resilience techniques varies depending on the video content, as well as, the transmitted bitrate and network conditions. -Therefore, it is not straightforward to select a single technique which achieves high efficiency for every scenario. -Thus, in this chapter a novel adaptive error robustness method is proposed in order to optimise the techniques previously described in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}. -At the encoder-side different video streams are generated with different levels compression efficiency and error robustness. Then, a pre-training \ac{nn} is used to predict which video stream will achieve the highest decoded video quality for a particular scenario. The selection is based on different input parameters which include information from the video signal, the coded stream and the transmission network. Using this approach the overall performance of the proposed method significantly increases by achieving a better trade-off between compression efficient and error robustness. +\glsunset{hevc} +\glsunset{ec} +\glsreset{nn} + +The effectiveness of different error resilience techniques is rather variable depending on the video content, the transmitted bitrate, network conditions, among others. +Therefore, it is not straightforward to select a single technique which achieves the highest efficiency for every possible scenario and also for the whole range of networking conditions. +This chapter addresses this problem by proposing an adaptive network which optimises the efficiency of the techniques previously described in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques} +At the encoder-side different video streams are generated with different levels of compression efficiency and error robustness. +Then, a pre-trained neural network is used to predict which video stream will achieve the highest decoded video quality for a particular loss case. +The selection is based on different input parameters which include information from the video signal, the coded stream and the transmission network. Using this approach the overall performance of the proposed adaptive method significantly increases yielding a better trade-off between compression efficiency and error robustness. This chapter is organised as follows. -Section~\ref{Section6:Problem} presents the problem formulation and preliminary results which serve as motivation for the method proposed in this chapter. -Section~\ref{Section6:ProposedMethod} provides an overview of the proposed method architecture and Section~\ref{Section6:OptimisationMethod} describes the method in detail. -Section~\ref{Section6:ExperimentalResults} shows the experimental evaluation of the proposed approach. Finally, Section~\ref{Section6:Summary} concludes this chapter. +Section~\ref{Section6:Problem} presents the problem addressed in this chapter and experimental evidence as motivation for the method investigated. +Section~\ref{Section6:ProposedMethod} provides an overview of architecture that supports the implementation of the proposed method and Section~\ref{Section6:OptimisationMethod} describes the technical details. +Section~\ref{Section6:ExperimentalResults} presents the performance evaluation and a discussion of results. +Finally, Section~\ref{Section6:Summary} concludes the chapter. \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\section{Problem formulation} +\section{The problem of adaptive error robustness} \label{Section6:Problem} -In this research work different robust coding techniques were proposed to increase the overall performance of video transmission under error-prone conditions. -Difference algorithms are used in the proposed methods, such as, reference frame selection, motion vector redundancies and error concealment signalling. -In previous chapters, these methods were evaluated against existing techniques revealing their superior performance and efficiency to deal with error robustness problems in high efficient video coding. -The evaluation of these methods revealed that they can efficiently increase the robustness against transmission errors and losses, leading to higher decoded video quality. Results in Chapter~\ref{Chapter:ErrorResilienceTechniques} show that the proposed reference frame selection algorithm introduces low overhead in comparison with existing techniques. However, the method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} achieves higher quality gains at cost of slightly higher bitrate increase. -Therefore, one may conclude that each method can be optimised for different applications, video characteristics and network conditions. +In the scope of this thesis different robust coding techniques were investigated to increase the overall performance of video transmission under error-prone conditions. +Different algorithms were devised in the proposed methods, such as, reference frame selection, motion vector redundancies and error concealment signalling. +In previous chapters, these methods were evaluated against existing techniques, revealing superior performance and better efficiency to deal with error robustness problems in high efficient video coding. -To advance one step forward, a joint robust coding method using different techniques can be used to increase the decoder video quality across difference conditions, by taking into consideration the input contents, the network information to optimise the robust coder to each scenario. -%In order to overcome the above problem, a joint evaluation study is necessary to identify the most efficient techniques across different conditions. +Overall, the performance results of these methods have demonstrated that they can efficiently increase the robustness of video streams against transmission errors and losses, leading to higher decoded video quality. +However, each method on its own does not adapt to different conditions. +While the reference frame selection algorithm presented in Chapter~\ref{Chapter:ErrorResilienceTechniques} introduces low overhead in comparison with existing techniques, the method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} achieves higher quality gains at cost of slightly higher bitrate increase. +Therefore, one may conclude that each method may be further optimised for different applications, video characteristics and network conditions. +This chapter advances one step forward , by investigating a joint robust coding method using different techniques to optimise the robust coder across different conditions, such as, the input content and the network information. +This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and can selected based on the current conditions. \subsection{Comparison of the proposed methods} -In order to develop a joint robust coder to cope with transmission errors a comparison study was carried out, to compare the error robustness performance of the proposed methods for different conditions. -%This study extends the previous evaluations presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, as they target a wider range of test material and network conditions. -The experimental evaluation carried out in this section follows the same condition described in Section~\ref{Section3:ErrorResilienceEvaluation}. +In order to address the above problem of adaptive robust coding, a simulation study was carried out to compare the error robustness performance of the previously proposed methods for different conditions. +This study extends the previous evaluations presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, as they target a wider range of test material and network conditions. +Moreover, it aims at comparing the performance of different techniques across a variety of video contents, bitrates and loss conditions, in order to find relevant insights which affect the decoded video quality. Consequently, these relevant insights may be used to develop an adaptive robust coding method. + %In order to perform a wider evaluation different bitrate and \acs{plr} were used. Table~\ref{Table6:TestVariables} illustrates the selected configurations. As shown in the table a total of 15 test sequences were used encoder using 5 different target bitrates. One should note that the main bitrate varies based on the test sequence and it is shown in third column of Table~\ref{TableTestSequences}. Different network conditions were evaluated using a total of 13 different \acs{plr}, with higher special focus on lower loss ratios, which are more relevant to modern video streaming applications. -The following methods are evaluated, namely: +The following methods are evaluated in the simulation study, namely: \begin{itemize} \item the default \ac{hevc} decoder with frame-copy \ac{ec} (\REF), as reference method; + \item the default \ac{hevc} encoder without robust coding-techniques and motion vector extrapolation \ac{ec} (\MVE); \item the proposed reference frame selection scheme (\EXP); - \item the \EXP method in combination with an \ac{ec} optimisation with 10\% of bitrate on the scalable enhancement layer (\EXPELTEN); - \item the same as above, with 20\% of bitrate on the scalable enhancement layer (\EXPELTWEN). + \item the \EXP method in combination with an \ac{ec} optimisation with 10\% of bitrate allocate for the scalable enhancement layer (\EXPELTEN); + \item the same as above, with 20\% of bitrate allocate for the scalable enhancement layer (\EXPELTWEN). \end{itemize} +These methods were selected as they are the main techniques proposed within this research work and cover two aspects of a robust video transmission: (i) robust video encoding by dynamically select the reference frames and (ii) robust video decoding by improving the error concealment performance. +\input{043_EndToEndRobustSystem/Results/AdaptiveSequenceList.tex} + +Table~\ref{Table6:TestMaterialCharacteristics} presents the list of test sequences used. These sequences were selected to cover a wide range of video content, with different levels of spatial and temporal complexity, as demonstrated by the values of \ac{si} and \ac{ti}, respectively. +Moreover, from the bitstream encoded with the reference implementation of the \ac{hevc} encoder, the percentage of the inter-coded blocks and the average \ac{mv} magnitude was extracted and presented in the table (columns 4 and 5, respectively). These values were extracted to compare them against the performance obtained with the proposed robust coding methods. +The remaining configuration were kept as described in Section~\ref{Section3:ErrorResilienceEvaluation}. +%The experimental methodology carried out in this section follows the same conditions described in Section~\ref{Section3:ErrorResilienceEvaluation}. + +\subsubsection{Simulation results and discussion} \begin{figure}[!t] -\centering -\input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex} -\vspace{-4mm} -\caption{Average decoded video quality using the main target bitrate.} -\label{Figure6:AverageQualityMain} + \centering + \input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex} + \vspace{-4mm} + \caption{Average decoded video quality for the bitstreams encoded with the main bitrate (bps).} + \label{Figure6:AverageQualityMain} \end{figure} -\subsubsection{Experimental results and discussion} -Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} shows the average decoded video quality obtained from 50 trials, measured with the \ac{psnr} quality metric, for different loss conditions. -Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate listed in the third column of Table~\ref{TableTestSequences}. Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained using a lower amount of coded bits, corresponding to a 40\% reduction of the main target bitrate. Comparing both results aims at finding the impact of the video signal and network bitrate on the performance of the proposed methods. +Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} shows the average decoded video quality \ac{psnr} obtained from 50 trials, for different loss conditions. +Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate listed in Table~\ref{Table6:TestMaterialCharacteristics}, referred to as main bitrate. +Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained by reducing the main bitrate by 40\%. This is used to evaluate the impact of the network bitrate on the decoded video quality. The results shown in the figures corresponds to the same test sequences, in order to carry out an accurate comparison. \begin{figure}[!t] \centering \input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex} \vspace{-4mm} -\caption{Average decoded video quality using a 40\% reduction of the main target bitrate} +\caption{Average decoded video quality using a 40\% reduction of the main bitrate (bps).} \label{Figure6:AverageQualityReduction} \end{figure} -Results of both figures show that the proposed methods (\EXP, \EXPELTEN and \EXPELTWEN) are all able to outperform the reference \ac{hevc} for high \acs{plr}. In case of the proposed reference frame selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions. -Comparing the results for both target bitrates also shows that the proposed methods are able to achieve higher quality gains for higher bitrates. Therefore, the for a lower bitrate (see results of Figure~\ref{Figure6:AverageQualityReduction}) the proposed methods only outperform the reference case for slightly higher \acs{plr}. This is more noticeable in the \EXPELTWEN case. +Firstly, comparing the results for different bitrates, it is noticeable that lower quality is obtained for the lower bitrates, which results from the higher compression ratios used. It is also noticed that lower quality gains, when comparing with the reference \ac{hevc} encoder (\REF), are obtained for the video streams encoded with a 40\% reduction of bitrate. This indicates that for lower bitrates the performance of the proposed methods is decreased, especially for the \EXPELTWEN case where 20\% of the bitrate is allocated for the enhancement layer. Therefore, for lower bitrates, using methods that have a lower impact on the coding efficiency (e.g., \EXP) or even disable the robust coding techniques (\MVE) may lead to higher performance. -Summarising, the performance of the proposed methods is not similar across different video content and target bitrate, which impose a challenge on which method should be applied to achieve the highest decoded video quality in case of transmission errors. In order to achieve increased efficiency and higher video quality, the optimal trade-off between auxiliary information and video bitrate should be obtained by selecting the optimal robust coding algorithm. +Secondly, comparing the results across different video content, i.e., different test sequences, one can notice that the performance of the different methods is not consistent. +For instance, in the result of Figure~\ref{Figure6:AverageQualityMain}, for Tennis sequence the \EXPELTWEN clearly outperforms the remaining methods, however, for the BQSquare sequence higher performance is achieved by using less bitrate allocated for the \ac{el} (i.e., 10\%). This is a consequence of the higher quality degradation due to higher motion in Tennis sequence (see results for TI and \ac{mv} size in Table~\ref{Table6:TestMaterialCharacteristics}) which required higher redundant information to increase the overall quality. +%\HIGHLIGHT{Mention SI!!} + +Finally, comparing the results for all test sequences and both bitrates it is clearly noticed that the \acs{plr} has a significant impact on the performance of the error robustness techniques. In case of the proposed reference frame selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions. For the \EXPELTWEN case higher \acs{plr} are required to be able to outperform both the reference \ac{hevc} and the \ac{mve} method. + +Summarising, the performance of the proposed methods is not similar across different video content and bitrates, opening research to adaptively choose the best method that should be applied to achieve the highest decoded video quality. +In order to achieve increased efficiency and higher video quality, the best trade-off between auxiliary information and video bitrate should be found by selecting the optimal robust coding algorithm. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\section{Proposed architecture} +\section{Architecture for adaptive error robustness} \label{Section6:ProposedMethod} -This section provides an overview of the proposed method, which aims at obtaining the optimal error robustness method for different streaming scenarios, i.e., video content and network conditions. -The main advantage of using the proposed algorithm is that it leads to an optimal trade-off between error robustness tools and coding efficiency. -This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and can selected based on the current conditions. -This architecture is suitable for dynamic streaming technologies, e.g., HTTP streaming based on MPEG-DASH~\cite{HEVCHTTPStreamAug2016} or content delivery over heterogeneous networks using the \ac{mmt}~\cite{ATSCMMT2016}. -In the proposed method, firstly the input signals are coded using different levels of error protections. Subsequently, an optimisation algorithm is used to decide which bitstream should be transmitted at any given instant, based on different parameters. The selected stream is expected to obtain the highest quality at the decoder-side for the given conditions. +This section describes the proposed method, which aims at obtaining optimal error robustness for different streaming scenarios, characterised by different video content and network conditions. +The main objective of this robust coding scheme it to achieve an optimal trade-off between error robustness tools and coding efficiency. \begin{figure}[tb] -\centering -%\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram} -\input{043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz} -\caption{Architecture of the proposed scheme.} -\label{FIG6:BlockDiagram} -\FIGSVSPACE + \centering + %\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram} + \input{043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz} + \caption{Adaptive error robustness architecture for 4 streams (S1...S4).} + \label{Figure6:BlockDiagram} + \FIGSVSPACE \end{figure} -Figure~\ref{FIG6:BlockDiagram} illustrates the architecture of the proposed encoding scheme. -Firstly, a reference video coder is used to compress the bitstream using highly efficient coding techniques in order to obtain the best \ac{rd} performance for error-free environments. -This is used as the reference bitstream in the proposed approach. -Subsequently, a transcoding operation is applied to the reference bitstream aided by the input video signal to efficiently select the reference frames used for inter prediction. The reference frame selection is applied as proposed in Chapter~\ref{Chapter:ErrorResilienceTechniques}. -Moreover, loss simulation and a saliency-based \ac{ec} optimisation is applied on the robust bitstream. This is used to select the optimal \ac{ec} algorithm which should be applied at the decoder. -Finally, since the optimal \ac{ec} mode might not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el} using two different bitrates, i.e., 10\% and 20\% of the total bitrate, providing two different levels of error robustness. -The \ac{ec} optimisation and \ac{el} coding is performed as proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques}. -The proposed approach results in four different bitstreams with different characteristics: (i) reference stream without error robust coding techniques; (ii) robust video coding using a reference frame selection algorithm which leads to lower error propagation in case of data-loss; (iii) robust video coding using selective reference frames and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue; (iv) same as the previous bitstream but with 20\% bitrate allocated for the scalable \ac{el}. These bitstreams are referred to as "S1" to "S4", respectively, as shown in Figure~\ref{FIG6:BlockDiagram}. +Figure~\ref{Figure6:BlockDiagram} illustrates the proposed architecture. +Firstly, the input signals are coded using different levels of error protections. Then, an optimisation algorithm is used to decide which bitstream should be transmitted at any given instant, based on different parameters. The selected stream is expected to obtain the highest quality at the decoder-side for the given conditions. +A reference video coder is used to compress the video signal using highly efficient coding techniques, obtaining the best \ac{rd} performance for error-free environments. +This is used as the reference bitstream. +Subsequently, a transcoding operation is applied to this reference bitstream aided by the input video signal to efficiently select the reference frames used for inter prediction. +The reference frame selection mechanism proposed in Chapter~\ref{Chapter:ErrorResilienceTechniques} is used for this purpose. +Moreover, loss simulation and a saliency-based \ac{ec} optimisation are also used in the robust coding process, which enables the selection of the optimal \ac{ec} algorithm to be applied at the decoder. +Finally, since the optimal \ac{ec} mode might not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el} using two different bitrates. In this case 10\% and 20\% of the total bitrate were chosen, providing two different levels of error robustness. +The \ac{ec} optimisation and \ac{el} coding is performed as described in Chapter~\ref{Chapter:ConcealmentAwareTechniques}. +The proposed approach results in four different bitstreams with different characteristics (referred to as "S1" to "S4", respectively, as shown in Figure~\ref{Figure6:BlockDiagram}): +(i) S1: reference stream without error robust coding; +(ii) S2: robust video coding using reference frame selection; +(iii) S3: robust video coding using selective reference frames and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue; +(iv) S4: same as the previous bitstream but with 20\% bitrate allocated for the scalable \ac{el}. + +Finally, the video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and feedback from the transmission channel. +From the input video signal, relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, further information is also extracted from the coded bitstream using a stream parser. +Finally, the loss ratio of the transmission channel is also used. +These parameters are fed into the optimisation block to select the best bitstream for transmission. +The optimisation process is described in detail in the following section. + +The main purpose of this architecture is to allow dynamic stream switching during real time transmission. +To accomplish that without causing drift at the decoder, the switching procedure should take place at refresh points, i.e., \ac{idr} and \ac{cra} frames, which correspond to clean random access points. In the proposed scheme, \ac{idr} frames are introduced every 16 frames, resulting in between 1 to 3 switching points every second, depending on the frame rate of the video signal. -Finally, the video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and from the transmission channel. -From the input video signal relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, further information is also extracted from the coded bitstream using a stream parser. -%the predictions used by the video encoder as well as the target bitrate are extracted using a stream parser. -Finally, a parameter regarding the transmission channel is also used to predict the data loss. -These parameters are fed into the error robustness optimisation block to select the suitable bitstream which should be transmitted to obtain the highest video quality. The optimisation process is described in detail in the following section. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -120,51 +153,112 @@ These parameters are fed into the error robustness optimisation block to select \section{Error robustness selection method} \label{Section6:OptimisationMethod} -In this section the optimisation method used to select a robust highly efficiently coded stream is presented. -The proposed scheme aims at selecting the best method which should be used for transmission, that achieves the highest quality in error-prone scenarios. +In this section the optimisation method used to select a robust video stream is presented. The selection method is based on a shallow neural network which predicts the optimal stream based on different parameters. -This section describes of both the input parameters and the neural network used in the proposed scheme. +This section describes both the input parameters and the neural network used in the optimal stream selection process. Moreover, a description data set used to train and evaluate the neural network used is also provided + +\subsection{Data set} + +%\def\NumCases{845\xspace} +\def\NumCases{910\xspace} + + +The proposed method includes an optimisation stage, which used a neural network classifier in order to select the optimal stream to be used for transmission. +This stream is able to obtain the highest quality for particular transmission scenario. +Therefore, several test cases were simulated in order to obtain results for a wide range of conditions. These tests are mainly characterised by three elements: +\begin{enumerate} + \item Video signal; + \item Target bitrate; + \item Packet loss ratio. +\end{enumerate} + +To obtain the data set 14 different test sequences with different scene content, listed in Table~\ref{Table6:TestMaterialCharacteristics}. Those sequences were encoded with the main bitrate (see Table~\ref{Table6:TestMaterialCharacteristics}), as well as four different others obtained by reducing the main bitrate by 10\%, 20\%, 30\% and 40\%. +Then, the following \acs{plr} were used to cover a wide range of loss ratios giving more detail to low \acs{plr}: 0\%, 0.5\%, 1.0\%, 1.2\%, 1.6\%, 2.0\%, 2.2\%, 2.4\%, 2.6\%, 2.8\%, 3.0\%, 5.0\% and 10.0\%. +This results in a total of \NumCases test conditions. \subsection{Input parameters} \glsreset{si} \glsreset{ti} -A set of different parameters are used to predict the optimal approach that achieves the highest quality in specific network conditions. Based on these inputs the proposed scheme is able to select the appropriate robust coding scheme which should be used. These parameters aim at covering three different aspects of the video transmission: two parameters from the input video, three parameters from the video stream and one parameter from the transmission network. +A set of different parameters is used to predict the optimal stream that achieves the highest quality in specific network conditions. +Based on these inputs the proposed scheme is able to select the appropriate robust coding scheme which should be used. +These parameters aim at characterise three different aspects of the video transmission system; (i) input video (two parameters), (ii) video stream (three parameters) and (iii) transmission network (one parameter). +%These parameters aim at covering three different aspects of the video transmission: two parameters from the input video, three parameters from the video stream and one parameter from the transmission network. +From the input video the two parameters are measured in a frame by frame basis, while for the video stream information a stream parser is applied to the reference bitstream that was compressed only based optimal \ac{rd} optimisation without considering any robust coding techniques, as shown in Figure~\ref{Figure6:BlockDiagram}. Finally, the parameter related with the transmission network can be obtained using feedback information from the decoder. -To characterise the input video signal the following parameters are used: \ac{si} and \ac{ti}~\cite{ITUTP9112008}. -These parameters are obtained as described in Chapter~\ref{Chapter:HEVCErrorResilience}, and aim at evaluating the spatial complexity and motion complexity of the video scenes, which have relevant impact on the error robustness of the coded streams. -As shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, test material with higher \ac{si} or \ac{ti} suffer from higher quality degradation as the \ac{plr} increases, e.g., Book Arrival and Tennis sequences. +%From the coded video streams complementary information to characterise the streaming content is also extracted. A stream parser is applied to the reference bitstream without error robustness techniques. Therefore, the prediction information extracted is related to a stream coded based on optimal \ac{rd} optimisation which was not affected by robust coding techniques. The following is extracted: -From the coded video streams complementary information to characterise the streaming content is also extracted. A stream parser is applied to the reference bitstream without error robustness techniques. Therefore, the prediction information extracted is related to a stream coded based on optimal \ac{rd} optimisation which was not affected by robust coding techniques. The following is extracted: +The characterisation of these three aspects of the video transmission result in a total of six parameters: \begin{itemize} -\item \textit{Inter(\%)}: the relation between the ratio of the image which is coded based on intra and inter frame predictions, indication the distribution of spatial and temporal dependencies. -This is expressed by the ratio of blocks coded using inter frame predictions. + +\item \textit{\ac{si}}: parameter expressing the amount of spatial detail, which consequently affects the error recovery operations, leading to higher quality degradation~\cite{ITUTP9102008}. + +\item \textit{\ac{ti}}: parameter expressing the amount of difference between temporally adjacent frames~\cite{ITUTP9102008}. This has a direct impact on the impact of errors, and thus the performance of the error resilience techniques as shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, which were discussed in Section~\ref{Section6:Problem}. + +\item \textit{Inter (percentage)}: the percentage of the image that is compressed using inter-frame prediction techniques. +This is an indication of the distribution of spatial and temporal dependencies. +This is calculated as the ratio of inter-coded blocks over intra-coded ones. Normally, coded streams with higher number of temporal dependencies are more prone to error propagation, thus higher number of bits should be allocated for error protection \cite{UEPScalable_TCSVT2010}. -\item \textit{AVG\_MV}: from those regions coded using inter frame prediction, the motion information is analysed and the average \ac{mv} size is used to provide information regarding the motion intensity. As shown in previous studies, motion based \ac{ec} algorithms are less efficient for video signals with higher motion intensity as \acs{mv} are less predictable using algorithms such as \ac{mc} or \ac{mve} \cite{HybridConcealmentTB2008}. +\item \textit{AVG\_MV}: for the inter-coded regions, the motion information is analysed and the average \ac{mv} magnitude is calculated to provide information about the motion intensity. +As shown in previous studies, motion-based \ac{ec} algorithms are less efficient for video signals with higher motion intensity, as \acs{mv} are less predictable using algorithms such as \ac{mc} or \ac{mve} \cite{HybridConcealmentTB2008}. + +\item \textit{Bitrate}: information regarding the average bit allocation per frames is extracted, since it will affect the video quality and the amount of bits that can be available for auxiliary information. This value is expressed in Mbps, as illustrated in Table~\ref{Table6:TestMaterialCharacteristics}. + +\item \textit{\ac{plr}:} feedback information from the network to characterise the transmission losses, expressed by the percentage of packets loss. This is used since for higher the loss rations there is a clear increase in the quality degradation which affect the performance of the robust coding techniques. +Transmission networks with higher loss ratios require a higher bit allocation for robust techniques, therefore the error resilience technique should adapt based on the value of the \ac{plr}. + -\item \textit{Bitrate}: information regarding the bit allocation is extracted, since it will affect the video quality and the amount of bits that can be available for auxiliary information. \end{itemize} -Finally, since the amount of data loss affects the decoded video quality the transmission network \ac{plr} is also used as input parameter. Transmission networks with higher loss ratios require a higher bit allocation for robust techniques, therefore the error resilience technique should adapt based on the value of the \ac{plr}. + +\begin{table}[tb] + \centering + \caption{Pearson correlation coefficient between the different parameters and the optimal robust coding techniques (optimal stream).} + \label{Table6:Correlation} + \TABLESVSPACE + \renewcommand{\arraystretch}{1.2} + \begin{tabular}{p{4cm}x{4cm}} + \toprule + Parameter & Absolute pearson coefficient \\ \hline +% \textit{SI} & -0.1553 \\ \hline +% \textit{TI} & 0.6079 \\ \hline +% \textit{Inter (percentage)} & -0.6487 \\ \hline +% \textit{OPT\_FLOW} & 0.6953 \\ \hline +% \textit{AVG\_MV} & 0.4647 \\ \hline +% \textit{Bitrate} & 0.3580 \\ \hline +% \textit{PLR} & 0.5755 \\ \hline + + \textit{SI} & 0.3953 \\ \hline + \textit{TI} & 0.6079 \\ \hline + \textit{Inter (percentage)} & 0.6487 \\ \hline + \textit{OPT\_FLOW} & 0.1553 \\ \hline + \textit{AVG\_MV} & 0.4647 \\ \hline + \textit{Bitrate} & 0.3580 \\ \hline + \textit{PLR} & 0.5755 \\ \hline + \end{tabular} +\end{table} + + +%To characterise the input video signal the following parameters are used: \ac{si} and \ac{ti}~\cite{ITUTP9102008}. +%These parameters are obtained as described in Chapter~\ref{Chapter:HEVCErrorResilience}, and aim at evaluating the spatial complexity and motion complexity of the video scenes, which have relevant impact on the error robustness of the coded streams. +%As shown, test material with higher \ac{si} or \ac{ti} suffer from higher quality degradation as the \ac{plr} increases, e.g., Book Arrival and Tennis sequences. + +These parameters were compared against the optimal stream which should be selected (S1...S4). +This aims at finding the correlation between the input parameters and the output variable, i.e., target stream. +In this study the streams are characterised by a number between 1 and 4, respectively. +Table~\ref{Table6:Correlation} shows the absolute person correlation coefficient obtained across all sequences, bitrates and \acs{plr}. +The table shows the results for the parameters previously described, as well as, a new parameter obtained from the average magnitude of the optical flow measured using the method in~\cite{DeepFlow2013} (\textit{OPT\_FLOW}). +Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable. As both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method as it achieves a higher person correlation coefficient. % Podia se citar este paper % \cite{IntraRefreshICIP2009} - +%\HERE \subsection{Neural network} \glsreset{nn} -\begin{figure}[tb] - \centering - %\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram} - \input{043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz} - \caption{Representation of the neural network used in the proposed method.} - \label{FIG6:NeuralNetwork} - \FIGSVSPACE -\end{figure} - A \ac{nn} is a computational model based on the structure and functions of biological neural networks. %In terms of computer science, \acs{nn} work similarly to a human nervous system for receiving, processing, and transmitting information between neurons. Mainly, a \ac{nn} consists of three different layers: the input layer which receives data from an external environment, the hidden layer which process the inputs and an output layer~\cite{NeuralNetworks1994}. @@ -178,82 +272,27 @@ Subsequently, the network transforms the input data through a series of neurons The output layer comprises four neurons corresponding to each available video stream from S1 to S4 (different error resilience techniques). Finally, the output with the highest value is selected and the corresponding stream is used for transmission, since it is the one expected to achieve the highest video quality for a particular scenario. +\begin{figure}[tb] + \centering + %\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram} + \input{043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz} + \caption{Representation of the neural network used in the proposed method.} + \label{FIG6:NeuralNetwork} + \FIGSVSPACE +\end{figure} + In the proposed approach a shallow \ac{nn} is used for simplicity and to avoid over-fitting due to the limited test cases. By avoiding over-fitting, it is guaranteed that the proposed method would continue to work in future test conditions not used for the network training. However, different network topologies with different number of neurons in each layer could also be used to predict between different error resilience techniques, as in the proposed method. - -\section{Experimental evaluation} -\label{Section6:ExperimentalResults} - -\def\NumCases{845\xspace} - -In this section the experimental results obtained with the proposed method are presented and discussed. -The error resilience techniques used in the proposed method are used in the context of the \ac{hevc} standard and compared with the reference implementation (\REF). -%The experimental results were obtained with the methodology described in Section~\ref{ExperimentalTestSetup}. -To demonstrate the effectiveness of the proposed optimal error robustness scheme several test cases were used. -In this experimental evaluation 13 test sequences were used, encoded using at 5 different bitrates. -Moreover, the following \acs{plr} were used to cover a wide range of loss rations giving more detail to low \acs{plr}: 0\%, 0.5\%, 1.0\%, 1.2\%, 1.6\%, 2.0\%, 2.2\%, 2.4\%, 2.6\%, 2.8\%, 3.0\%, 5.0\% and 10.0\%. -This results in a total of \NumCases test conditions. - -The performance evaluation of the proposed method is carried out using two different approaches. -In the former, a description of the \ac{nn} training is presented along with the validation results. -In the latter, the quality obtained with the proposed optimal error robustness scheme is evaluated and compared with the reference methods. - -%\begin{table}[t] -% \centering -% \caption{List of the test conditions tested.} -% \label{Table6:TestVariables} -% \TABLESVSPACE -% \begin{tabular}{m{4cm}|y{4cm}|y{6cm}} -% \toprule -% \textbf{Sequence} & \textbf{Bitrate} & \textbf{\ac{plr}} \\ \midrule -% Basketball Drill & Main (see Table~\ref{TableTestSequences}) & Error-free \\ -% Basketball Drive & 10\% reduction & 0.5\% \\ -% Book Arrival & 20\% reduction & 1.0\% \\ -% BQSquare & 30\% reduction & 1.2\% \\ -% Cactus & 40\% reduction & 1.6\% \\ -% Four People & 50\% reduction & 2.0\% \\ -% Kendo & & 2.2\% \\ -% Kimono & & 2.4\% \\ -% Kristen and Sara & & 2.6\% \\ -% Park Scene & & 2.8\% \\ -% People on Street & & 3.0\% \\ -% Race Horses & & 5.0\% \\ -% Tennis & & 10.0\% \\ -% Traffic & & \\ -% \bottomrule -% \end{tabular} -%\end{table} - - -\subsection{Network training and validation} +\subsubsection{Training and validation} %As mentioned before a total of \NumCases test cases are used for the evaluation of the proposed method. -To effectively train and validate the \ac{nn} used in the proposed method the data set is randomly divided based on the following: 70\% of the test cases are used for training and 30\% for validation. +To effectively train and validate the \ac{nn} the data set is randomly in three different groups: (i) 55\% of the test cases are used for training, (ii) 15\% for validation and (iii) 30\% for testing the \ac{nn} performance. The \ac{nn} supervised training uses 591 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993}. -The remaining 273 conditions are used to evaluate the accuracy of the proposed method after training using different conditions. +The remaining 254 conditions are used to evaluate the accuracy of the proposed method after training using different conditions. From this test one video sequences was left out, to be used in the performance evaluation of the proposed method. -\begin{table}[tb] - \centering - \caption{Confusion matrix of the training test cases.} - \label{Table6:ConfusionTraining} - \TABLESVSPACE - \renewcommand{\arraystretch}{1.2} - \begin{tabular}{x{.5cm}x{1cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}} - & & \multicolumn{4}{c}{Target} \\ - \multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} - & & S1 & S2 & S3 & S4 \\ \hline - & S1 & 122 & 17 & 3 & 0 \\ - \cline{2-6} & S2 & 13 & 80 & 19 & 2 \\ - \cline{2-6} & S3 & 0 & 12 & 96 & 9 \\ - \cline{2-6} & S4 & 0 & 4 & 11 & 202 \\ - \hline - \multicolumn{2}{x{1.7cm}|}{Accuracy} & 90.3\% & 70.8\% & 74.4\% & 94.8\% \\ - \end{tabular} -\end{table} - Table~\ref{Table6:ConfusionTraining} shows the confusion matrix of the \ac{nn} training phase. In this matrix each column corresponds to the target result, i.e., the cases where the coded streams S1 to S4 would lead to the highest decoded video quality, respectively. Similarly, each row corresponds to the \ac{nn} selected by the trained networks. @@ -261,12 +300,31 @@ Thus, those cases where the horizontal and vertical labels match, indicate that The last line shows the accuracy for each target output. These results correspond to the dataset used to train the \ac{nn}. +\begin{table}[tb] + \centering + \caption{Confusion matrix of the training and validation test cases.} + \label{Table6:ConfusionTraining} + \TABLESVSPACE + \renewcommand{\arraystretch}{1.2} + \begin{tabular}{x{.5cm}x{1cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}} + & & \multicolumn{4}{c}{Target} \\ + \multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} + & & S1 & S2 & S3 & S4 \\ \hline + & S1 & 122 & 17 & 3 & 0 \\ + \cline{2-6} & S2 & 13 & 80 & 19 & 2 \\ + \cline{2-6} & S3 & 0 & 12 & 96 & 9 \\ + \cline{2-6} & S4 & 0 & 4 & 11 & 202 \\ + \hline + \multicolumn{2}{x{1.7cm}|}{Accuracy} & 90.3\% & 70.8\% & 74.4\% & 94.8\% \\ + \end{tabular} +\end{table} + Results in Table~\ref{Table6:ConfusionTraining} shows that streams S1 (i.e., without robust coding techniques) and S4 (i.e., using optimal \ac{ec} with 20\% of bitrate in the \ac{el}) lead to the highest quality in the majority of the test cases (are selected more often). -The \ac{nn} is able to achieve a high accuracy for S1 and S4 streams, correctly selecting the output stream in more than 90\% of cases. Regarding the remaining robust streams (i.e., S2 and S3) an acceptable accuracy of 70.8\% and 74.4\% is achieved, respectively. This reveals that the \ac{nn} used in the proposed method is able to correctly predict the coded stream which leads to the highest video quality across different conditions. +The \ac{nn} is able to achieve a high accuracy for S1 and S4 streams, correctly selecting the output stream in more than 90\% of cases. Regarding the remaining robust streams (i.e., S2 and S3) an acceptable accuracy of 70.8\% and 74.4\% is achieved, respectively. This reveals that the \ac{nn} used in the proposed method is able to predict the coded stream which leads to the highest video quality across different conditions with acceptable accuracy. \begin{table}[tb] \centering - \caption{Confusion matrix of the validation test cases.} + \caption{Confusion matrix of the test cases not used for testing.} \label{Table6:ConfusionTest} \TABLESVSPACE \renewcommand{\arraystretch}{1.2} @@ -284,20 +342,22 @@ The \ac{nn} is able to achieve a high accuracy for S1 and S4 streams, correctly \end{tabular} \end{table} -To further evaluate the performance of the \ac{nn}, the confusion matrix of the validation set is shown in the Table~\ref{Table6:ConfusionTest}. This table shows the same results as Table~\ref{Table6:ConfusionTraining} but for different test cases, which were not used in the training phase. +To further evaluate the performance of the \ac{nn}, the confusion matrix of the data set usef for testing is shown in the Table~\ref{Table6:ConfusionTest}. This table shows the same results as Table~\ref{Table6:ConfusionTraining} but for different test cases, which were not used in the training phase. % Therefore, the trained network is not aware of the test cases used for validation. Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 62.2\% with a maximum value of 89.6\% when the S4 is the target stream. Although a lower accuracy is obtained using the validation test cases, the overall accuracy is still acceptable, since it allows to select the optimal robust coding technique (i.e., S1 to S4) for the majority of the conditions. In summary, the results presented in this sub-section reveal that the input parameters used in the proposed \ac{nn} are correlated with the decoded video quality, as they can effectively be used to perform data categorisation used in the selection of the optimal error robustness coding technique for the \ac{hevc} standard. +\section{Performance evaluation} +\label{Section6:ExperimentalResults} +In this section the performance of the proposed method is evaluated and discussed. +The overall goal of the proposed method is to achieve improved quality using a combination of different robust coding techniques adaptively selected using a feed-forward \ac{nn}. +The error resilience techniques used in the proposed method are used in the context of the \ac{hevc} standard and compared with the reference implementation (\REF). +The quality obtained with the proposed optimal error robustness scheme (\PROP) is compared against the reference \ac{hevc} (\REF) and the methods used to generate the streams S1 to S4. -\subsection{Quality evaluation} - -The overall goal of the proposed method is to achieve improved quality using a combination of different robust coding techniques adaptively selected using a feed-forward \ac{nn}. In this sub-section the quality obtained with the proposed method (\PROP) is compared against the reference \ac{hevc} (\REF) and the methods used to generate the streams S1 to S4. - -\begin{figure}[!t] +\begin{figure}[!t] \centering \input{043_EndToEndRobustSystem/Results/PlotQualty_vs_PLR_Predicted.tex} \vspace{-2mm} @@ -305,12 +365,16 @@ The overall goal of the proposed method is to achieve improved quality using a c \label{Figure6:ProposedQuality} \end{figure} -Figure~\ref{Figure6:ProposedQuality} shows the quality obtained for the proposed method and the \REF case for the main target bitrate shown in Table~\ref{TableTestSequences}. Moreover, the quality obtained with the target stream (\BEST) is also shown. One should note that the \BEST case corresponds to the optimal selection which can only be performed after quality evaluation at the decoder, therefore not suitable for real time applications and it is only used to illustrate the optimal scenario. +Figure~\ref{Figure6:ProposedQuality} shows the quality obtained for the proposed method and the \REF case for the main target bitrate shown in Table~\ref{TableTestSequences}. Moreover, the quality obtained with the target stream (\BEST) is also shown. +One should note that the \BEST case corresponds to the optimal selection which can only be performed after quality evaluation at the decoder, therefore not suitable for real time applications. +It is only used to establish the optimal case for comparison. The results in the figure allows to evaluate the impact in the decoded quality of the selection used in the proposed method and compare it with the optimal case. -As shown in the Figure~\ref{Figure6:ProposedQuality} the proposed method clearly outperforms the reference \ac{hevc} for all \acs{plr}. Moreover, in the error-free case (\ac{plr}=0) it does not lead to quality reduction since the proposed approach is able to select the optimal stream, thus disabling any robust coding technique when the transmission network does not have packet losses. In case of data loss results show that the proposed method is able to match the quality obtained by the \BEST case in most of the cases, revealing that the proposed method is able to efficiently select the optimal stream for transmission. +As shown in Figure~\ref{Figure6:ProposedQuality} the proposed method clearly outperforms the reference \ac{hevc} for all \acs{plr}. +Moreover, in the error-free case (\ac{plr}=0) it leads to imperceptible quality reduction since the proposed approach is able to select the optimal stream, thus disabling any robust coding technique when the transmission network does not have packet losses. +In case of data loss, the results show that the proposed method is able to match the quality obtained by the \BEST case in most cases, revealing that the proposed method is able to efficiently select the optimal stream for transmission. -\begin{table}[tbp] +\begin{table}[tb] \centering \caption{Average quality gains ($\Delta$PSNR) in comparison with the reference \ac{hevc} across different bitrates and loss ratios.} \label{Table6:AverageGains} @@ -321,7 +385,7 @@ As shown in the Figure~\ref{Figure6:ProposedQuality} the proposed method clearly \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\ & S1 & S2 & S3 & S4 & \PROP \\ \hline Basketball Drill & +1.61 & +2.66 & +2.48 & +2.44 & +2.75 \\ \hline -Basketball Drive & +2.17 & +3.46 & +3.22 & +3.63 & +3.69 \\ \hline +%Basketball Drive & +2.17 & +3.46 & +3.22 & +3.63 & +3.69 \\ \hline Book Arrival & +0.96 & +1.66 & +1.43 & +1.59 & +1.74 \\ \hline BQSquare & +0.92 & +0.96 & +0.80 & +0.62 & +1.09 \\ \hline Cactus & +1.28 & +2.04 & +1.89 & +1.96 & +2.09 \\ \hline @@ -330,17 +394,18 @@ Kendo & +0.81 & +0.78 & +0.20 & +0.61 & +1.02 \\ \hline Kimono & +1.65 & +3.04 & +3.26 & +3.27 & +3.41 \\ \hline Kristen Sara & +0.36 & +0.15 & -0.33 & -0.50 & +0.38 \\ \hline Park Scene & +1.64 & +2.18 & +1.88 & +2.02 & +2.25 \\ \hline +Party Scene & +0.21 & +0.20 & -0.07 & -0.24 & +0.29 \\ \hline People on Street & +1.00 & +1.27 & +0.91 & +1.49 & +1.63 \\ \hline Race Horses & +1.93 & +3.89 & +4.06 & +4.30 & +4.39 \\ \hline Tennis & +1.15 & +2.24 & +2.17 & +3.12 & +3.18 \\ \hline -\textbf{Average} & \textbf{+1.21} & \textbf{+1.86} & \textbf{+1.62} & \textbf{+1.81} & \textbf{+2.15} \\ +\textbf{Average} & \textbf{+1.06} & \textbf{+1.60} & \textbf{+1.37} & \textbf{+1.51} & \textbf{+1.88} \\ \bottomrule \end{tabular} \begin{tikzpicture}[overlay] -\draw[red, line width=1.2pt] (1.8,4.65) ellipse (3.5cm and 0.3cm); -\draw[red, line width=1.2pt] (3,3.05) ellipse (2.3cm and 0.3cm); +\draw[red, line width=1.2pt] (1.8,5.2) ellipse (3.5cm and 0.3cm); +\draw[red, line width=1.2pt] (3,3.6) ellipse (2.3cm and 0.3cm); \draw[blue, line width=1.2pt] (6.7,1.45) ellipse (.85cm and 0.25cm); -\draw[blue, line width=1.2pt] (6.7,6.75) ellipse (.85cm and 0.25cm); +\draw[blue, line width=1.2pt] (6.7,4.10) ellipse (.85cm and 0.25cm); \end{tikzpicture} \end{table} @@ -349,9 +414,27 @@ Subsequently, a comparison between the different methods is performed, in order Table~\ref{Table6:AverageGains} shows the average quality gains in comparison with the reference \ac{hevc} obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams S1 to S4. The average quality gains are measured across all test cases, i.e., different bitrates and different loss ratios. Results in Table~\ref{Table6:AverageGains} shows that the robust coded streams S1 to S4 are able to achieve acceptable average quality gains up to 1.86 dB (see results for S2). -However, since the introduction of robust coding techniques decreases the coding efficiency it may lead to lower quality than the reference \ac{hevc}. This is noticeable in the results obtained for the Four People sequence in the streams S2 to S4 and for the Kristen and Sara sequence and in the streams S3 and S4 (see red ellipses). This reveals that the overall performance of such methods is not suitable for transmission networks subject to low \acs{plr}, e.g., lower than 3\% (higher number of conditions have \acs{plr} lower than 3\%). -On contrary, the proposed method is always able to outperform the reference \ac{hevc} and achieve quality gains of at least 0.25 dB for the Four People sequence. Moreover, for the Basketball Drive and Race Horses sequences it is able to achieve the highest quality gains of up 3.69 dB and 4.39 dB, respectively (see blue ellipses). Summarising, by adaptively selecting the video stream, and consequently the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness achieving an average quality gains across different bitrates and \acs{plr} of 2.15 dB. +However, since the implementation of robust coding techniques decreases the coding efficiency, it also leads to lower quality than the reference \ac{hevc}. This is noticeable in the results obtained for the Four People sequence in the streams S2 to S4 and for the Kristen and Sara sequence in the streams S3 and S4 (see red ellipses). +The overall performance of such methods reveals that they are not suitable for transmission conditions with low \acs{plr}, e.g., lower than 3\% (higher number of conditions have \acs{plr} lower than 3\%). +On the contrary, the proposed method is always able to outperform the reference \ac{hevc} and achieve quality gains of at least 0.25 dB for the Four People sequence. Moreover, for the Kimono and Race Horses sequences it is able to achieve the highest quality gains of up 3.41 dB and 4.39 dB, respectively (see blue ellipses). Summarising, by adaptively selecting the video stream, and consequently the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness achieving an average quality gain across different bitrates and \acs{plr} of 1.88 dB. +Finally, in order the performance evaluation of the proposed method is evaluated for a video sequence, which was not fed into the \ac{nn} training phase. This study aims at finding the accuracy of the optimal stream selection in case of new video content. For this purpose the Basketball Drive sequence was not used in the previous simulations. Table~\ref{Table6:AverageGainBasketBallDrive} shows the average quality results obtained with this sequence, similarly as Table~\ref{Table6:AverageGains}. These results confirm that the proposed method is able to select the optimal robust stream to be used for transmission, even when new video content is transmitted. The proposed method achieves a quality gain of 3.55 dB in comparison with the reference \ac{hevc}, outperforming the quality achieved with the stream S4 (i.e., 3.45 dB). +This confirms the advantage of the proposed method to adaptively select among different stream with different levels of error protections. + +\begin{table}[tb] + \centering + \caption{Average quality gain ($\Delta$PSNR) in comparison with the reference \ac{hevc} for the Basketball Drive sequence.} + \label{Table6:AverageGainBasketBallDrive} + \TABLESVSPACE + \newcommand{\colwidth}{1.7cm} + \begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} + \toprule + \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\ + & S1 & S2 & S3 & S4 & \PROP \\ \hline + Basketball Drive & +2.10 & +2.68 & +3.36 & +3.45 & +3.55 \\ + \bottomrule + \end{tabular} +\end{table} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% @@ -359,8 +442,8 @@ On contrary, the proposed method is always able to outperform the reference \ac{ \section{Summary} \label{Section6:Summary} -In this chapter an adaptive error robustness method is proposed based on reference frame selection and \ac{ec}-aware optimisation. By adaptively select a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the reference techniques. +In this chapter an adaptive error robustness method is proposed based on reference frame selection and \ac{ec}-aware optimisation. By adaptively selecting a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the reference techniques. %Moreover, the proposed method is able to outperform those cases where a single error resilience technique is used. -The overall results demonstrate that the proposed method is able to increase the reconstruction video quality in case of errors, and by adapting the transmitted stream to the network conditions a better trade-off between coding efficiency and error robusntess is achieved. +The overall results demonstrate that the proposed method is able to increase the reconstruction video quality in case of errors, and by adapting the transmitted stream to the network conditions a better trade-off between coding efficiency and error robustness is achieved. The consistent quality improvement achieved, allows the use of this method in dynamic broadcasting video services where network feedback is available and where packet loss probability is not negligible. - +This architecture is suitable for dynamic streaming technologies, e.g., HTTP streaming based on MPEG-DASH~\cite{HEVCHTTPStreamAug2016} or content delivery over heterogeneous networks using the \ac{mmt}~\cite{ATSCMMT2016}. diff --git a/043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz b/043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz index d570540..a4ce646 100644 --- a/043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz +++ b/043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz @@ -1,4 +1,4 @@ - + \usetikzlibrary{arrows,matrix} \usetikzlibrary{fit} \usetikzlibrary{intersections} @@ -115,7 +115,7 @@ % % %% TS Mux and output \path (Encoder.south) -- (Encoder |- ConcOpt.north) node[midway] (middleRef) {}; % Find middle ref -\node [propBlock, right=9 of middleRef, text width=5.5cm, rotate=-90, anchor=center, minimum height=1.1cm] (Mux) {Error robustness optimisation}; +\node [propBlock, right=9 of middleRef, text width=5.5cm, rotate=-90, anchor=center, minimum height=1.1cm] (Mux) {Optimal stream selection}; \node [right of=Mux, node distance=2.8cm, coordinate] (Out) {}; \tikzstyle{OutputLabels} = [pos=1, anchor=south east, text width=2cm, align=right] diff --git a/043_EndToEndRobustSystem/Results/AdaptiveSequenceList.tex b/043_EndToEndRobustSystem/Results/AdaptiveSequenceList.tex new file mode 100644 index 0000000..07c2623 --- /dev/null +++ b/043_EndToEndRobustSystem/Results/AdaptiveSequenceList.tex @@ -0,0 +1,31 @@ +\begin{table}[t] + \centering + \caption{Test material used in the experimental simulation and its characteristics.} + \label{Table6:TestMaterialCharacteristics} + \TABLESVSPACE + \newcommand{\colsize}{1.3cm} + \begin{tabular}{m{4cm}|x{\colsize}|x{\colsize}|x{\colsize}|x{2.2cm}|x{2.8cm}} + \toprule + \centering\let\newline\\\arraybackslash\hspace{0pt}\textbf{Sequence} + & \textbf{SI} + & \textbf{TI} + & \textbf{Inter (\%)} + & \textbf{MV Magnitude} + & \textbf{Main bitrate (Mbps)} \\ \hline + Basketball Drill & 33.4 & 14.4 & 88.1 & 6.40 & 4.50 \\ \hline + Basketball Drive & 33.0 & 15.2 & 85.0 & 27.4 & 15.00 \\ \hline + Book Arrival & 28.4 & 21.7 & 90.7 & 4.67 & 1.50 \\ \hline + BQSquare & 63.2 & 11.5 & 93.7 & 1.51 & 4.00 \\ \hline + Cactus & 30.5 & 11.4 & 89.7 & 5.90 & 20.00 \\ \hline + Four People & 31.3 & 6.90 & 93.3 & 0.76 & 2.00 \\ \hline + Kendo & 19.6 & 16.1 & 88.0 & 9.85 & 0.60 \\ \hline + Kimono & 23.4 & 32.5 & 83.0 & 14.8 & 8.00 \\ \hline + Kristen and Sara & 25.6 & 6.30 & 93.1 & 1.75 & 15.00 \\ \hline + Park Scene & 31.3 & 11.6 & 92.0 & 7.26 & 10.00 \\ \hline + Party Scene & 52.6 & 11.4 & 89.8 & 3.42 & 4.50 \\ \hline + People on Street & 40.0 & 25.4 & 87.4 & 7.80 & 12.00 \\ \hline + Race Horses & 43.7 & 24.4 & 79.7 & 22.5 & 8.00 \\ \hline + Tennis & 20.3 & 45.3 & 76.2 & 91.0 & 5.00 \\ + \bottomrule + \end{tabular} +\end{table} diff --git a/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.bash b/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.bash new file mode 100644 index 0000000..9f3de98 --- /dev/null +++ b/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.bash @@ -0,0 +1,1317 @@ +#!/bin/bash +gnuplot << _EOF +set terminal pdfcairo mono \ +font 'TimesNewRoman,14' + +set datafile missing '-' + +set grid +#set size square {1,.5} + +set title center offset character 0, -.9 + +set xlabel center +set ylabel center offset character 1, 0 + +set key spacing 1 width 0 + +set style line 4 lc 1 lt 1 lw 2 pt 2 ps 1 +set style line 2 lc 2 lt 2 lw 2 pt 3 ps 1 +set style line 3 lc 3 lt 4 lw 2 pt 4 ps 1 +set style line 1 lc 4 lt 4 lw 2 pt 10 ps 1 +set style line 5 lc 7 lt 3 lw 2 pt 6 ps 1 +set style line 6 lc 8 lt 6 lw 2 pt 8 ps 1 +set style line 7 lc 2 lt 7 lw 2 pt 8 ps 1 +set style line 8 lc 3 lt 8 lw 2 pt 9 ps 1 +set style line 9 lc 4 lt 9 lw 2 pt 10 ps 1 + +set style line 100 lc 1 lw 3 +set style line 101 lc 4 lw 3 +set style line 102 lc 2 lw 3 +set style line 103 lc 3 lw 3 + +#set rmargin 1 + + +# Extra template +set xlabel 'Packet Loss Ratio [%]' +set ylabel 'Packet Loss Ratio [%]' +set key bottom left Left reverse +set output 'lowdelay_P_4R_slice_1200B_seq_basketball_drill_rate_single.pdf' +set title 'Basketball Drill - Main rate' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 39.900 +"0.5%" 0.5 38.057 +"1.0%" 1.0 36.949 +"2.0%" 2.0 35.157 +"2.6%" 2.6 33.890 +"2.8%" 2.8 33.513 +"3.0%" 3.0 33.305 +"5.0%" 5.0 31.385 +e +"EF" 0.0 39.900 +"0.5%" 0.5 38.854 +"1.0%" 1.0 38.079 +"2.0%" 2.0 36.974 +"2.6%" 2.6 36.173 +"2.8%" 2.8 35.783 +"3.0%" 3.0 35.546 +"5.0%" 5.0 33.776 +e +"EF" 0.0 39.600 +"0.5%" 0.5 38.847 +"1.0%" 1.0 38.305 +"2.0%" 2.0 37.178 +"2.6%" 2.6 36.436 +"2.8%" 2.8 36.267 +"3.0%" 3.0 36.137 +"5.0%" 5.0 34.691 +e +"EF" 0.0 39.480 +"0.5%" 0.5 39.191 +"1.0%" 1.0 38.973 +"2.0%" 2.0 38.385 +"2.6%" 2.6 38.093 +"2.8%" 2.8 37.790 +"3.0%" 3.0 37.929 +"5.0%" 5.0 37.015 +e +"EF" 0.0 38.880 +"0.5%" 0.5 38.729 +"1.0%" 1.0 38.550 +"2.0%" 2.0 38.208 +"2.6%" 2.6 37.956 +"2.8%" 2.8 37.846 +"3.0%" 3.0 37.863 +"5.0%" 5.0 37.186 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_basketball_drill_rate_single_60pc.pdf' +set title 'Basketball Drill - 40% Reduction ' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 37.570 +"0.5%" 0.5 36.658 +"1.0%" 1.0 35.927 +"2.0%" 2.0 34.683 +"2.6%" 2.6 33.776 +"2.8%" 2.8 33.730 +"3.0%" 3.0 33.583 +"5.0%" 5.0 31.726 +e +"EF" 0.0 37.570 +"0.5%" 0.5 37.067 +"1.0%" 1.0 36.571 +"2.0%" 2.0 35.863 +"2.6%" 2.6 35.256 +"2.8%" 2.8 35.199 +"3.0%" 3.0 35.111 +"5.0%" 5.0 33.670 +e +"EF" 0.0 37.120 +"0.5%" 0.5 36.831 +"1.0%" 1.0 36.405 +"2.0%" 2.0 35.895 +"2.6%" 2.6 35.391 +"2.8%" 2.8 35.375 +"3.0%" 3.0 35.099 +"5.0%" 5.0 34.150 +e +"EF" 0.0 36.940 +"0.5%" 0.5 36.737 +"1.0%" 1.0 36.569 +"2.0%" 2.0 36.357 +"2.6%" 2.6 36.072 +"2.8%" 2.8 36.029 +"3.0%" 3.0 35.991 +"5.0%" 5.0 35.431 +e +"EF" 0.0 36.130 +"0.5%" 0.5 36.038 +"1.0%" 1.0 35.975 +"2.0%" 2.0 35.749 +"2.6%" 2.6 35.627 +"2.8%" 2.8 35.561 +"3.0%" 3.0 35.617 +"5.0%" 5.0 35.114 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_basketball_drive_rate_single.pdf' +set title 'Basketball Drive - Main rate' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 38.470 +"0.5%" 0.5 36.114 +"1.0%" 1.0 34.743 +"2.0%" 2.0 32.612 +"2.6%" 2.6 31.746 +"2.8%" 2.8 31.541 +"3.0%" 3.0 31.189 +"5.0%" 5.0 29.464 +e +"EF" 0.0 38.470 +"0.5%" 0.5 37.389 +"1.0%" 1.0 36.546 +"2.0%" 2.0 35.163 +"2.6%" 2.6 34.483 +"2.8%" 2.8 34.152 +"3.0%" 3.0 33.831 +"5.0%" 5.0 31.865 +e +"EF" 0.0 38.100 +"0.5%" 0.5 37.353 +"1.0%" 1.0 36.695 +"2.0%" 2.0 35.356 +"2.6%" 2.6 34.719 +"2.8%" 2.8 34.648 +"3.0%" 3.0 34.420 +"5.0%" 5.0 32.685 +e +"EF" 0.0 38.280 +"0.5%" 0.5 37.928 +"1.0%" 1.0 37.626 +"2.0%" 2.0 36.795 +"2.6%" 2.6 36.337 +"2.8%" 2.8 36.368 +"3.0%" 3.0 36.040 +"5.0%" 5.0 34.351 +e +"EF" 0.0 38.040 +"0.5%" 0.5 37.835 +"1.0%" 1.0 37.601 +"2.0%" 2.0 37.020 +"2.6%" 2.6 36.564 +"2.8%" 2.8 36.508 +"3.0%" 3.0 36.405 +"5.0%" 5.0 34.887 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_basketball_drive_rate_single_60pc.pdf' +set title 'Basketball Drive - 40% Reduction ' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 37.340 +"0.5%" 0.5 35.557 +"1.0%" 1.0 34.518 +"2.0%" 2.0 32.703 +"2.6%" 2.6 31.689 +"2.8%" 2.8 31.447 +"3.0%" 3.0 31.231 +"5.0%" 5.0 29.515 +e +"EF" 0.0 37.340 +"0.5%" 0.5 36.549 +"1.0%" 1.0 36.021 +"2.0%" 2.0 34.997 +"2.6%" 2.6 34.267 +"2.8%" 2.8 33.982 +"3.0%" 3.0 33.784 +"5.0%" 5.0 32.369 +e +"EF" 0.0 36.730 +"0.5%" 0.5 36.177 +"1.0%" 1.0 35.732 +"2.0%" 2.0 34.927 +"2.6%" 2.6 34.475 +"2.8%" 2.8 34.359 +"3.0%" 3.0 34.131 +"5.0%" 5.0 32.929 +e +"EF" 0.0 36.990 +"0.5%" 0.5 36.660 +"1.0%" 1.0 36.431 +"2.0%" 2.0 35.862 +"2.6%" 2.6 35.465 +"2.8%" 2.8 35.428 +"3.0%" 3.0 35.317 +"5.0%" 5.0 34.106 +e +"EF" 0.0 36.570 +"0.5%" 0.5 36.371 +"1.0%" 1.0 36.236 +"2.0%" 2.0 35.815 +"2.6%" 2.6 35.657 +"2.8%" 2.8 35.523 +"3.0%" 3.0 35.399 +"5.0%" 5.0 34.647 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_book_arrival_c6_rate_single.pdf' +set title 'Book Arrival - Main rate' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 40.280 +"0.5%" 0.5 39.122 +"1.0%" 1.0 38.578 +"2.0%" 2.0 37.584 +"2.6%" 2.6 36.699 +"2.8%" 2.8 35.703 +"3.0%" 3.0 36.206 +"5.0%" 5.0 34.396 +e +"EF" 0.0 40.280 +"0.5%" 0.5 39.471 +"1.0%" 1.0 39.127 +"2.0%" 2.0 38.514 +"2.6%" 2.6 37.941 +"2.8%" 2.8 37.139 +"3.0%" 3.0 37.505 +"5.0%" 5.0 36.119 +e +"EF" 0.0 39.970 +"0.5%" 0.5 39.391 +"1.0%" 1.0 39.053 +"2.0%" 2.0 38.419 +"2.6%" 2.6 37.928 +"2.8%" 2.8 37.366 +"3.0%" 3.0 37.719 +"5.0%" 5.0 36.705 +e +"EF" 0.0 40.030 +"0.5%" 0.5 39.788 +"1.0%" 1.0 39.716 +"2.0%" 2.0 39.339 +"2.6%" 2.6 39.088 +"2.8%" 2.8 38.949 +"3.0%" 3.0 38.918 +"5.0%" 5.0 38.348 +e +"EF" 0.0 39.790 +"0.5%" 0.5 39.679 +"1.0%" 1.0 39.517 +"2.0%" 2.0 39.416 +"2.6%" 2.6 39.125 +"2.8%" 2.8 39.151 +"3.0%" 3.0 39.193 +"5.0%" 5.0 38.781 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_book_arrival_c6_rate_single_60pc.pdf' +set title 'Book Arrival - 40% Reduction ' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 38.920 +"0.5%" 0.5 38.641 +"1.0%" 1.0 37.769 +"2.0%" 2.0 36.545 +"2.6%" 2.6 36.302 +"2.8%" 2.8 36.515 +"3.0%" 3.0 36.189 +"5.0%" 5.0 35.001 +e +"EF" 0.0 38.920 +"0.5%" 0.5 38.736 +"1.0%" 1.0 38.174 +"2.0%" 2.0 37.237 +"2.6%" 2.6 37.134 +"2.8%" 2.8 37.221 +"3.0%" 3.0 36.922 +"5.0%" 5.0 36.287 +e +"EF" 0.0 38.400 +"0.5%" 0.5 38.253 +"1.0%" 1.0 37.711 +"2.0%" 2.0 37.137 +"2.6%" 2.6 37.037 +"2.8%" 2.8 36.572 +"3.0%" 3.0 37.005 +"5.0%" 5.0 36.027 +e +"EF" 0.0 38.460 +"0.5%" 0.5 38.305 +"1.0%" 1.0 38.169 +"2.0%" 2.0 37.780 +"2.6%" 2.6 37.651 +"2.8%" 2.8 37.596 +"3.0%" 3.0 37.582 +"5.0%" 5.0 36.977 +e +"EF" 0.0 37.830 +"0.5%" 0.5 37.715 +"1.0%" 1.0 37.487 +"2.0%" 2.0 37.469 +"2.6%" 2.6 37.419 +"2.8%" 2.8 37.328 +"3.0%" 3.0 37.321 +"5.0%" 5.0 36.980 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_bqsquare_rate_single.pdf' +set title 'BQSquare - Main rate' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 39.300 +"0.5%" 0.5 38.381 +"1.0%" 1.0 37.908 +"2.0%" 2.0 36.716 +"2.6%" 2.6 35.946 +"2.8%" 2.8 35.576 +"3.0%" 3.0 35.639 +"5.0%" 5.0 34.289 +e +"EF" 0.0 39.300 +"0.5%" 0.5 38.873 +"1.0%" 1.0 38.552 +"2.0%" 2.0 37.882 +"2.6%" 2.6 37.351 +"2.8%" 2.8 37.163 +"3.0%" 3.0 37.101 +"5.0%" 5.0 36.155 +e +"EF" 0.0 39.150 +"0.5%" 0.5 38.735 +"1.0%" 1.0 38.522 +"2.0%" 2.0 37.747 +"2.6%" 2.6 37.252 +"2.8%" 2.8 36.924 +"3.0%" 3.0 36.985 +"5.0%" 5.0 35.971 +e +"EF" 0.0 38.750 +"0.5%" 0.5 38.558 +"1.0%" 1.0 38.371 +"2.0%" 2.0 37.981 +"2.6%" 2.6 37.673 +"2.8%" 2.8 37.640 +"3.0%" 3.0 37.652 +"5.0%" 5.0 36.734 +e +"EF" 0.0 38.160 +"0.5%" 0.5 38.015 +"1.0%" 1.0 37.897 +"2.0%" 2.0 37.652 +"2.6%" 2.6 37.447 +"2.8%" 2.8 37.372 +"3.0%" 3.0 37.411 +"5.0%" 5.0 36.765 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_bqsquare_rate_single_60pc.pdf' +set title 'BQSquare - 40% Reduction ' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 36.780 +"0.5%" 0.5 36.370 +"1.0%" 1.0 36.129 +"2.0%" 2.0 35.480 +"2.6%" 2.6 34.912 +"2.8%" 2.8 34.873 +"3.0%" 3.0 34.947 +"5.0%" 5.0 33.946 +e +"EF" 0.0 36.780 +"0.5%" 0.5 36.567 +"1.0%" 1.0 36.448 +"2.0%" 2.0 36.048 +"2.6%" 2.6 35.715 +"2.8%" 2.8 35.698 +"3.0%" 3.0 35.807 +"5.0%" 5.0 35.108 +e +"EF" 0.0 36.610 +"0.5%" 0.5 36.340 +"1.0%" 1.0 36.261 +"2.0%" 2.0 35.839 +"2.6%" 2.6 35.513 +"2.8%" 2.8 35.581 +"3.0%" 3.0 35.547 +"5.0%" 5.0 34.913 +e +"EF" 0.0 36.230 +"0.5%" 0.5 36.135 +"1.0%" 1.0 36.021 +"2.0%" 2.0 35.787 +"2.6%" 2.6 35.586 +"2.8%" 2.8 35.602 +"3.0%" 3.0 35.569 +"5.0%" 5.0 35.200 +e +"EF" 0.0 35.700 +"0.5%" 0.5 35.627 +"1.0%" 1.0 35.560 +"2.0%" 2.0 35.399 +"2.6%" 2.6 35.297 +"2.8%" 2.8 35.236 +"3.0%" 3.0 35.258 +"5.0%" 5.0 34.891 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_cactus_rate_single.pdf' +set title 'Cactus - Main rate' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 38.000 +"0.5%" 0.5 37.133 +"1.0%" 1.0 36.510 +"2.0%" 2.0 35.505 +"2.6%" 2.6 34.986 +"2.8%" 2.8 34.829 +"3.0%" 3.0 34.785 +"5.0%" 5.0 33.677 +e +"EF" 0.0 38.000 +"0.5%" 0.5 37.653 +"1.0%" 1.0 37.316 +"2.0%" 2.0 36.773 +"2.6%" 2.6 36.351 +"2.8%" 2.8 36.327 +"3.0%" 3.0 36.216 +"5.0%" 5.0 35.151 +e +"EF" 0.0 37.740 +"0.5%" 0.5 37.507 +"1.0%" 1.0 37.213 +"2.0%" 2.0 36.809 +"2.6%" 2.6 36.429 +"2.8%" 2.8 36.367 +"3.0%" 3.0 36.278 +"5.0%" 5.0 35.387 +e +"EF" 0.0 37.840 +"0.5%" 0.5 37.753 +"1.0%" 1.0 37.624 +"2.0%" 2.0 37.375 +"2.6%" 2.6 37.149 +"2.8%" 2.8 37.134 +"3.0%" 3.0 37.093 +"5.0%" 5.0 36.273 +e +"EF" 0.0 37.650 +"0.5%" 0.5 37.571 +"1.0%" 1.0 37.499 +"2.0%" 2.0 37.259 +"2.6%" 2.6 37.105 +"2.8%" 2.8 37.003 +"3.0%" 3.0 36.937 +"5.0%" 5.0 36.089 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_cactus_rate_single_60pc.pdf' +set title 'Cactus - 40% Reduction ' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 37.130 +"0.5%" 0.5 36.267 +"1.0%" 1.0 35.582 +"2.0%" 2.0 34.475 +"2.6%" 2.6 33.934 +"2.8%" 2.8 33.655 +"3.0%" 3.0 33.502 +"5.0%" 5.0 32.178 +e +"EF" 0.0 37.130 +"0.5%" 0.5 36.753 +"1.0%" 1.0 36.471 +"2.0%" 2.0 35.935 +"2.6%" 2.6 35.510 +"2.8%" 2.8 35.401 +"3.0%" 3.0 35.202 +"5.0%" 5.0 34.377 +e +"EF" 0.0 36.780 +"0.5%" 0.5 36.524 +"1.0%" 1.0 36.349 +"2.0%" 2.0 35.789 +"2.6%" 2.6 35.516 +"2.8%" 2.8 35.389 +"3.0%" 3.0 35.291 +"5.0%" 5.0 34.573 +e +"EF" 0.0 36.880 +"0.5%" 0.5 36.791 +"1.0%" 1.0 36.701 +"2.0%" 2.0 36.469 +"2.6%" 2.6 36.319 +"2.8%" 2.8 36.282 +"3.0%" 3.0 36.235 +"5.0%" 5.0 35.723 +e +"EF" 0.0 36.560 +"0.5%" 0.5 36.494 +"1.0%" 1.0 36.431 +"2.0%" 2.0 36.265 +"2.6%" 2.6 36.145 +"2.8%" 2.8 36.100 +"3.0%" 3.0 36.097 +"5.0%" 5.0 35.670 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_four_people_rate_single.pdf' +set title 'Four People - Main rate' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 40.130 +"0.5%" 0.5 39.933 +"1.0%" 1.0 39.775 +"2.0%" 2.0 39.507 +"2.6%" 2.6 39.205 +"2.8%" 2.8 39.176 +"3.0%" 3.0 39.161 +"5.0%" 5.0 38.544 +e +"EF" 0.0 40.130 +"0.5%" 0.5 40.034 +"1.0%" 1.0 39.953 +"2.0%" 2.0 39.806 +"2.6%" 2.6 39.625 +"2.8%" 2.8 39.617 +"3.0%" 3.0 39.634 +"5.0%" 5.0 39.292 +e +"EF" 0.0 39.770 +"0.5%" 0.5 39.686 +"1.0%" 1.0 39.622 +"2.0%" 2.0 39.529 +"2.6%" 2.6 39.411 +"2.8%" 2.8 39.380 +"3.0%" 3.0 39.391 +"5.0%" 5.0 39.147 +e +"EF" 0.0 39.800 +"0.5%" 0.5 39.760 +"1.0%" 1.0 39.717 +"2.0%" 2.0 39.680 +"2.6%" 2.6 39.632 +"2.8%" 2.8 39.592 +"3.0%" 3.0 39.565 +"5.0%" 5.0 39.470 +e +"EF" 0.0 39.250 +"0.5%" 0.5 39.223 +"1.0%" 1.0 39.207 +"2.0%" 2.0 39.165 +"2.6%" 2.6 39.127 +"2.8%" 2.8 39.116 +"3.0%" 3.0 39.099 +"5.0%" 5.0 38.981 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_four_people_rate_single_60pc.pdf' +set title 'Four People - 40% Reduction ' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 36.080 +"0.5%" 0.5 36.020 +"1.0%" 1.0 35.942 +"2.0%" 2.0 35.809 +"2.6%" 2.6 35.726 +"2.8%" 2.8 35.744 +"3.0%" 3.0 35.690 +"5.0%" 5.0 35.469 +e +"EF" 0.0 36.080 +"0.5%" 0.5 36.045 +"1.0%" 1.0 35.983 +"2.0%" 2.0 35.914 +"2.6%" 2.6 35.864 +"2.8%" 2.8 35.874 +"3.0%" 3.0 35.849 +"5.0%" 5.0 35.718 +e +"EF" 0.0 35.350 +"0.5%" 0.5 35.340 +"1.0%" 1.0 35.315 +"2.0%" 2.0 35.269 +"2.6%" 2.6 35.251 +"2.8%" 2.8 35.246 +"3.0%" 3.0 35.244 +"5.0%" 5.0 35.142 +e +"EF" 0.0 35.790 +"0.5%" 0.5 35.778 +"1.0%" 1.0 35.763 +"2.0%" 2.0 35.731 +"2.6%" 2.6 35.705 +"2.8%" 2.8 35.692 +"3.0%" 3.0 35.688 +"5.0%" 5.0 35.624 +e +"EF" 0.0 35.610 +"0.5%" 0.5 35.603 +"1.0%" 1.0 35.590 +"2.0%" 2.0 35.567 +"2.6%" 2.6 35.561 +"2.8%" 2.8 35.533 +"3.0%" 3.0 35.544 +"5.0%" 5.0 35.479 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_kendo_c3_rate_single.pdf' +set title 'Kendo - Main rate' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 41.150 +"0.5%" 0.5 40.679 +"1.0%" 1.0 39.544 +"2.0%" 2.0 38.271 +"2.6%" 2.6 37.512 +"2.8%" 2.8 36.995 +"3.0%" 3.0 37.349 +"5.0%" 5.0 34.856 +e +"EF" 0.0 41.150 +"0.5%" 0.5 40.852 +"1.0%" 1.0 40.082 +"2.0%" 2.0 39.122 +"2.6%" 2.6 38.690 +"2.8%" 2.8 38.217 +"3.0%" 3.0 38.537 +"5.0%" 5.0 36.565 +e +"EF" 0.0 40.270 +"0.5%" 0.5 40.009 +"1.0%" 1.0 39.304 +"2.0%" 2.0 38.638 +"2.6%" 2.6 38.516 +"2.8%" 2.8 38.231 +"3.0%" 3.0 38.401 +"5.0%" 5.0 37.110 +e +"EF" 0.0 40.730 +"0.5%" 0.5 40.427 +"1.0%" 1.0 40.156 +"2.0%" 2.0 39.319 +"2.6%" 2.6 39.305 +"2.8%" 2.8 38.770 +"3.0%" 3.0 38.692 +"5.0%" 5.0 37.258 +e +"EF" 0.0 40.260 +"0.5%" 0.5 40.065 +"1.0%" 1.0 39.801 +"2.0%" 2.0 39.492 +"2.6%" 2.6 39.238 +"2.8%" 2.8 39.172 +"3.0%" 3.0 38.950 +"5.0%" 5.0 38.358 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_kendo_c3_rate_single_60pc.pdf' +set title 'Kendo - 40% Reduction ' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 38.730 +"0.5%" 0.5 38.365 +"1.0%" 1.0 37.714 +"2.0%" 2.0 36.762 +"2.6%" 2.6 36.695 +"2.8%" 2.8 35.899 +"3.0%" 3.0 35.859 +"5.0%" 5.0 34.589 +e +"EF" 0.0 38.730 +"0.5%" 0.5 38.466 +"1.0%" 1.0 38.034 +"2.0%" 2.0 37.402 +"2.6%" 2.6 37.279 +"2.8%" 2.8 36.751 +"3.0%" 3.0 36.538 +"5.0%" 5.0 35.744 +e +"EF" 0.0 37.550 +"0.5%" 0.5 37.280 +"1.0%" 1.0 37.189 +"2.0%" 2.0 36.823 +"2.6%" 2.6 36.388 +"2.8%" 2.8 36.455 +"3.0%" 3.0 36.283 +"5.0%" 5.0 35.449 +e +"EF" 0.0 38.080 +"0.5%" 0.5 37.819 +"1.0%" 1.0 37.382 +"2.0%" 2.0 36.767 +"2.6%" 2.6 36.901 +"2.8%" 2.8 36.967 +"3.0%" 3.0 36.804 +"5.0%" 5.0 35.920 +e +"EF" 0.0 37.260 +"0.5%" 0.5 37.147 +"1.0%" 1.0 36.921 +"2.0%" 2.0 36.588 +"2.6%" 2.6 36.552 +"2.8%" 2.8 36.316 +"3.0%" 3.0 36.383 +"5.0%" 5.0 35.746 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_kimono_rate_single.pdf' +set title 'Kimono - Main rate' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 42.880 +"0.5%" 0.5 40.547 +"1.0%" 1.0 39.019 +"2.0%" 2.0 37.479 +"2.6%" 2.6 36.643 +"2.8%" 2.8 36.385 +"3.0%" 3.0 36.193 +"5.0%" 5.0 34.604 +e +"EF" 0.0 42.880 +"0.5%" 0.5 41.665 +"1.0%" 1.0 40.699 +"2.0%" 2.0 39.153 +"2.6%" 2.6 37.937 +"2.8%" 2.8 37.643 +"3.0%" 3.0 37.622 +"5.0%" 5.0 35.445 +e +"EF" 0.0 42.600 +"0.5%" 0.5 41.650 +"1.0%" 1.0 41.121 +"2.0%" 2.0 39.675 +"2.6%" 2.6 38.961 +"2.8%" 2.8 38.645 +"3.0%" 3.0 38.491 +"5.0%" 5.0 36.605 +e +"EF" 0.0 42.720 +"0.5%" 0.5 42.313 +"1.0%" 1.0 41.996 +"2.0%" 2.0 40.649 +"2.6%" 2.6 39.681 +"2.8%" 2.8 39.435 +"3.0%" 3.0 39.280 +"5.0%" 5.0 36.611 +e +"EF" 0.0 42.540 +"0.5%" 0.5 42.187 +"1.0%" 1.0 41.936 +"2.0%" 2.0 40.747 +"2.6%" 2.6 40.183 +"2.8%" 2.8 39.813 +"3.0%" 3.0 39.665 +"5.0%" 5.0 36.513 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_kimono_rate_single_60pc.pdf' +set title 'Kimono - 40% Reduction ' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 42.070 +"0.5%" 0.5 39.871 +"1.0%" 1.0 38.394 +"2.0%" 2.0 36.475 +"2.6%" 2.6 35.665 +"2.8%" 2.8 35.391 +"3.0%" 3.0 35.102 +"5.0%" 5.0 33.411 +e +"EF" 0.0 42.070 +"0.5%" 0.5 41.097 +"1.0%" 1.0 40.226 +"2.0%" 2.0 38.742 +"2.6%" 2.6 38.046 +"2.8%" 2.8 37.839 +"3.0%" 3.0 37.529 +"5.0%" 5.0 35.479 +e +"EF" 0.0 41.640 +"0.5%" 0.5 40.969 +"1.0%" 1.0 40.533 +"2.0%" 2.0 39.207 +"2.6%" 2.6 38.533 +"2.8%" 2.8 38.588 +"3.0%" 3.0 38.227 +"5.0%" 5.0 36.521 +e +"EF" 0.0 41.890 +"0.5%" 0.5 41.503 +"1.0%" 1.0 41.238 +"2.0%" 2.0 40.550 +"2.6%" 2.6 40.104 +"2.8%" 2.8 39.691 +"3.0%" 3.0 39.645 +"5.0%" 5.0 37.736 +e +"EF" 0.0 41.680 +"0.5%" 0.5 41.458 +"1.0%" 1.0 41.292 +"2.0%" 2.0 40.671 +"2.6%" 2.6 40.028 +"2.8%" 2.8 39.892 +"3.0%" 3.0 39.903 +"5.0%" 5.0 38.453 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_kristen_sara_rate_single.pdf' +set title 'Kristen Sara - Main rate' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 40.500 +"0.5%" 0.5 40.319 +"1.0%" 1.0 40.165 +"2.0%" 2.0 39.834 +"2.6%" 2.6 39.683 +"2.8%" 2.8 39.572 +"3.0%" 3.0 39.544 +"5.0%" 5.0 38.968 +e +"EF" 0.0 40.500 +"0.5%" 0.5 40.414 +"1.0%" 1.0 40.342 +"2.0%" 2.0 40.171 +"2.6%" 2.6 40.149 +"2.8%" 2.8 40.071 +"3.0%" 3.0 40.060 +"5.0%" 5.0 39.712 +e +"EF" 0.0 40.120 +"0.5%" 0.5 40.049 +"1.0%" 1.0 39.986 +"2.0%" 2.0 39.913 +"2.6%" 2.6 39.814 +"2.8%" 2.8 39.760 +"3.0%" 3.0 39.763 +"5.0%" 5.0 39.541 +e +"EF" 0.0 40.240 +"0.5%" 0.5 40.199 +"1.0%" 1.0 40.178 +"2.0%" 2.0 40.110 +"2.6%" 2.6 40.050 +"2.8%" 2.8 40.033 +"3.0%" 3.0 40.003 +"5.0%" 5.0 39.889 +e +"EF" 0.0 39.970 +"0.5%" 0.5 39.935 +"1.0%" 1.0 39.888 +"2.0%" 2.0 39.867 +"2.6%" 2.6 39.831 +"2.8%" 2.8 39.815 +"3.0%" 3.0 39.791 +"5.0%" 5.0 39.696 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_kristen_sara_rate_single_60pc.pdf' +set title 'Kristen Sara - 40% Reduction ' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 39.040 +"0.5%" 0.5 38.938 +"1.0%" 1.0 38.819 +"2.0%" 2.0 38.659 +"2.6%" 2.6 38.445 +"2.8%" 2.8 38.467 +"3.0%" 3.0 38.383 +"5.0%" 5.0 38.029 +e +"EF" 0.0 39.040 +"0.5%" 0.5 39.000 +"1.0%" 1.0 38.928 +"2.0%" 2.0 38.845 +"2.6%" 2.6 38.765 +"2.8%" 2.8 38.751 +"3.0%" 3.0 38.725 +"5.0%" 5.0 38.573 +e +"EF" 0.0 38.440 +"0.5%" 0.5 38.415 +"1.0%" 1.0 38.369 +"2.0%" 2.0 38.323 +"2.6%" 2.6 38.258 +"2.8%" 2.8 38.257 +"3.0%" 3.0 38.243 +"5.0%" 5.0 38.087 +e +"EF" 0.0 38.290 +"0.5%" 0.5 38.256 +"1.0%" 1.0 38.223 +"2.0%" 2.0 38.171 +"2.6%" 2.6 38.141 +"2.8%" 2.8 38.115 +"3.0%" 3.0 38.114 +"5.0%" 5.0 37.980 +e +"EF" 0.0 36.380 +"0.5%" 0.5 36.360 +"1.0%" 1.0 36.337 +"2.0%" 2.0 36.303 +"2.6%" 2.6 36.285 +"2.8%" 2.8 36.279 +"3.0%" 3.0 36.284 +"5.0%" 5.0 36.189 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_park_rate_single.pdf' +set title 'Park Scene - Main rate' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 39.470 +"0.5%" 0.5 38.067 +"1.0%" 1.0 36.969 +"2.0%" 2.0 35.555 +"2.6%" 2.6 34.863 +"2.8%" 2.8 34.589 +"3.0%" 3.0 34.545 +"5.0%" 5.0 32.923 +e +"EF" 0.0 39.470 +"0.5%" 0.5 38.965 +"1.0%" 1.0 38.381 +"2.0%" 2.0 37.537 +"2.6%" 2.6 36.963 +"2.8%" 2.8 36.761 +"3.0%" 3.0 36.738 +"5.0%" 5.0 35.165 +e +"EF" 0.0 38.900 +"0.5%" 0.5 38.528 +"1.0%" 1.0 38.113 +"2.0%" 2.0 37.380 +"2.6%" 2.6 37.064 +"2.8%" 2.8 36.909 +"3.0%" 3.0 36.846 +"5.0%" 5.0 35.771 +e +"EF" 0.0 39.160 +"0.5%" 0.5 38.957 +"1.0%" 1.0 38.779 +"2.0%" 2.0 38.212 +"2.6%" 2.6 37.920 +"2.8%" 2.8 37.964 +"3.0%" 3.0 37.767 +"5.0%" 5.0 36.847 +e +"EF" 0.0 38.820 +"0.5%" 0.5 38.681 +"1.0%" 1.0 38.589 +"2.0%" 2.0 38.127 +"2.6%" 2.6 37.869 +"2.8%" 2.8 37.901 +"3.0%" 3.0 37.693 +"5.0%" 5.0 36.777 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_park_rate_single_60pc.pdf' +set title 'Park Scene - 40% Reduction ' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 37.810 +"0.5%" 0.5 36.969 +"1.0%" 1.0 36.189 +"2.0%" 2.0 35.190 +"2.6%" 2.6 34.477 +"2.8%" 2.8 34.028 +"3.0%" 3.0 34.091 +"5.0%" 5.0 32.873 +e +"EF" 0.0 37.810 +"0.5%" 0.5 37.511 +"1.0%" 1.0 37.109 +"2.0%" 2.0 36.703 +"2.6%" 2.6 36.252 +"2.8%" 2.8 35.948 +"3.0%" 3.0 36.012 +"5.0%" 5.0 35.173 +e +"EF" 0.0 37.230 +"0.5%" 0.5 37.020 +"1.0%" 1.0 36.750 +"2.0%" 2.0 36.361 +"2.6%" 2.6 36.002 +"2.8%" 2.8 35.964 +"3.0%" 3.0 35.951 +"5.0%" 5.0 35.174 +e +"EF" 0.0 37.460 +"0.5%" 0.5 37.289 +"1.0%" 1.0 37.189 +"2.0%" 2.0 36.905 +"2.6%" 2.6 36.740 +"2.8%" 2.8 36.747 +"3.0%" 3.0 36.517 +"5.0%" 5.0 36.009 +e +"EF" 0.0 37.010 +"0.5%" 0.5 36.903 +"1.0%" 1.0 36.868 +"2.0%" 2.0 36.625 +"2.6%" 2.6 36.501 +"2.8%" 2.8 36.395 +"3.0%" 3.0 36.354 +"5.0%" 5.0 35.955 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_people_rate_single.pdf' +set title 'People - Main rate' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 35.900 +"0.5%" 0.5 33.738 +"1.0%" 1.0 32.546 +"2.0%" 2.0 30.711 +"2.6%" 2.6 29.942 +"2.8%" 2.8 29.719 +"3.0%" 3.0 29.479 +"5.0%" 5.0 27.770 +e +"EF" 0.0 35.900 +"0.5%" 0.5 34.590 +"1.0%" 1.0 33.780 +"2.0%" 2.0 32.097 +"2.6%" 2.6 31.417 +"2.8%" 2.8 31.237 +"3.0%" 3.0 31.104 +"5.0%" 5.0 28.863 +e +"EF" 0.0 34.970 +"0.5%" 0.5 34.074 +"1.0%" 1.0 33.432 +"2.0%" 2.0 32.157 +"2.6%" 2.6 31.646 +"2.8%" 2.8 31.461 +"3.0%" 3.0 31.369 +"5.0%" 5.0 29.743 +e +"EF" 0.0 35.380 +"0.5%" 0.5 34.610 +"1.0%" 1.0 34.107 +"2.0%" 2.0 33.082 +"2.6%" 2.6 32.416 +"2.8%" 2.8 31.950 +"3.0%" 3.0 32.117 +"5.0%" 5.0 29.939 +e +"EF" 0.0 34.810 +"0.5%" 0.5 34.418 +"1.0%" 1.0 34.077 +"2.0%" 2.0 33.289 +"2.6%" 2.6 32.871 +"2.8%" 2.8 32.593 +"3.0%" 3.0 32.557 +"5.0%" 5.0 30.851 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_people_rate_single_60pc.pdf' +set title 'People - 40% Reduction ' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 33.230 +"0.5%" 0.5 32.010 +"1.0%" 1.0 31.419 +"2.0%" 2.0 29.977 +"2.6%" 2.6 29.311 +"2.8%" 2.8 29.090 +"3.0%" 3.0 28.889 +"5.0%" 5.0 27.509 +e +"EF" 0.0 33.230 +"0.5%" 0.5 32.475 +"1.0%" 1.0 32.143 +"2.0%" 2.0 31.069 +"2.6%" 2.6 30.581 +"2.8%" 2.8 30.357 +"3.0%" 3.0 30.220 +"5.0%" 5.0 28.946 +e +"EF" 0.0 32.040 +"0.5%" 0.5 31.566 +"1.0%" 1.0 31.307 +"2.0%" 2.0 30.577 +"2.6%" 2.6 30.199 +"2.8%" 2.8 29.993 +"3.0%" 3.0 29.898 +"5.0%" 5.0 28.964 +e +"EF" 0.0 32.560 +"0.5%" 0.5 31.977 +"1.0%" 1.0 31.727 +"2.0%" 2.0 30.743 +"2.6%" 2.6 30.357 +"2.8%" 2.8 30.141 +"3.0%" 3.0 30.224 +"5.0%" 5.0 29.111 +e +"EF" 0.0 31.730 +"0.5%" 0.5 31.474 +"1.0%" 1.0 31.292 +"2.0%" 2.0 30.753 +"2.6%" 2.6 30.416 +"2.8%" 2.8 30.208 +"3.0%" 3.0 30.284 +"5.0%" 5.0 29.347 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_race_rate_single.pdf' +set title 'Race Horses - Main rate' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 39.870 +"0.5%" 0.5 35.527 +"1.0%" 1.0 33.448 +"2.0%" 2.0 30.742 +"2.6%" 2.6 29.755 +"2.8%" 2.8 29.231 +"3.0%" 3.0 29.082 +"5.0%" 5.0 26.981 +e +"EF" 0.0 39.870 +"0.5%" 0.5 36.995 +"1.0%" 1.0 35.462 +"2.0%" 2.0 32.993 +"2.6%" 2.6 32.041 +"2.8%" 2.8 31.617 +"3.0%" 3.0 31.471 +"5.0%" 5.0 28.718 +e +"EF" 0.0 39.250 +"0.5%" 0.5 37.371 +"1.0%" 1.0 35.972 +"2.0%" 2.0 33.958 +"2.6%" 2.6 32.988 +"2.8%" 2.8 32.808 +"3.0%" 3.0 32.431 +"5.0%" 5.0 30.051 +e +"EF" 0.0 39.360 +"0.5%" 0.5 38.191 +"1.0%" 1.0 37.495 +"2.0%" 2.0 35.867 +"2.6%" 2.6 35.199 +"2.8%" 2.8 34.740 +"3.0%" 3.0 34.431 +"5.0%" 5.0 32.030 +e +"EF" 0.0 38.820 +"0.5%" 0.5 38.161 +"1.0%" 1.0 37.732 +"2.0%" 2.0 36.272 +"2.6%" 2.6 35.635 +"2.8%" 2.8 35.377 +"3.0%" 3.0 35.106 +"5.0%" 5.0 32.759 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_race_rate_single_60pc.pdf' +set title 'Race Horses - 40% Reduction ' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 37.590 +"0.5%" 0.5 34.817 +"1.0%" 1.0 33.087 +"2.0%" 2.0 30.779 +"2.6%" 2.6 29.742 +"2.8%" 2.8 29.288 +"3.0%" 3.0 29.214 +"5.0%" 5.0 27.179 +e +"EF" 0.0 37.590 +"0.5%" 0.5 35.844 +"1.0%" 1.0 34.645 +"2.0%" 2.0 32.860 +"2.6%" 2.6 32.052 +"2.8%" 2.8 31.613 +"3.0%" 3.0 31.569 +"5.0%" 5.0 29.399 +e +"EF" 0.0 36.910 +"0.5%" 0.5 35.765 +"1.0%" 1.0 34.811 +"2.0%" 2.0 33.397 +"2.6%" 2.6 32.733 +"2.8%" 2.8 32.256 +"3.0%" 3.0 32.528 +"5.0%" 5.0 30.556 +e +"EF" 0.0 37.140 +"0.5%" 0.5 36.365 +"1.0%" 1.0 35.903 +"2.0%" 2.0 34.706 +"2.6%" 2.6 34.058 +"2.8%" 2.8 33.845 +"3.0%" 3.0 33.565 +"5.0%" 5.0 31.996 +e +"EF" 0.0 36.640 +"0.5%" 0.5 36.172 +"1.0%" 1.0 35.803 +"2.0%" 2.0 34.910 +"2.6%" 2.6 34.685 +"2.8%" 2.8 34.420 +"3.0%" 3.0 33.957 +"5.0%" 5.0 33.076 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_tennis_rate_single.pdf' +set title 'Tennis - Main rate' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 40.680 +"0.5%" 0.5 38.243 +"1.0%" 1.0 36.815 +"2.0%" 2.0 34.460 +"2.6%" 2.6 33.418 +"2.8%" 2.8 32.690 +"3.0%" 3.0 32.709 +"5.0%" 5.0 30.360 +e +"EF" 0.0 40.680 +"0.5%" 0.5 38.889 +"1.0%" 1.0 37.856 +"2.0%" 2.0 35.901 +"2.6%" 2.6 34.967 +"2.8%" 2.8 34.325 +"3.0%" 3.0 34.247 +"5.0%" 5.0 31.923 +e +"EF" 0.0 40.030 +"0.5%" 0.5 38.966 +"1.0%" 1.0 38.059 +"2.0%" 2.0 36.429 +"2.6%" 2.6 35.351 +"2.8%" 2.8 34.823 +"3.0%" 3.0 35.068 +"5.0%" 5.0 33.155 +e +"EF" 0.0 40.390 +"0.5%" 0.5 39.420 +"1.0%" 1.0 38.912 +"2.0%" 2.0 37.448 +"2.6%" 2.6 36.907 +"2.8%" 2.8 36.713 +"3.0%" 3.0 36.522 +"5.0%" 5.0 34.991 +e +"EF" 0.0 40.030 +"0.5%" 0.5 39.612 +"1.0%" 1.0 39.371 +"2.0%" 2.0 38.549 +"2.6%" 2.6 38.128 +"2.8%" 2.8 37.883 +"3.0%" 3.0 37.679 +"5.0%" 5.0 36.606 +e + +set output 'lowdelay_P_4R_slice_1200B_seq_tennis_rate_single_60pc.pdf' +set title 'Tennis - 40% Reduction ' +plot '-' using 2:3 w lp ls 1 title 'FC', '-' using 2:3 w lp ls 2 title 'MC', '-' using 2:3 w lp ls 3 title 'EXP+EMVE', '-' using 2:3 w lp ls 4 title 'EL10', '-' using 2:3 w lp ls 5 title 'EL20' +"EF" 0.0 39.020 +"0.5%" 0.5 37.569 +"1.0%" 1.0 36.453 +"2.0%" 2.0 34.685 +"2.6%" 2.6 33.662 +"2.8%" 2.8 33.320 +"3.0%" 3.0 33.442 +"5.0%" 5.0 31.232 +e +"EF" 0.0 39.020 +"0.5%" 0.5 37.958 +"1.0%" 1.0 37.021 +"2.0%" 2.0 35.698 +"2.6%" 2.6 34.791 +"2.8%" 2.8 34.513 +"3.0%" 3.0 34.600 +"5.0%" 5.0 32.544 +e +"EF" 0.0 38.170 +"0.5%" 0.5 37.482 +"1.0%" 1.0 36.874 +"2.0%" 2.0 35.711 +"2.6%" 2.6 35.236 +"2.8%" 2.8 35.002 +"3.0%" 3.0 34.863 +"5.0%" 5.0 33.289 +e +"EF" 0.0 38.580 +"0.5%" 0.5 37.739 +"1.0%" 1.0 37.295 +"2.0%" 2.0 36.384 +"2.6%" 2.6 35.732 +"2.8%" 2.8 35.556 +"3.0%" 3.0 35.538 +"5.0%" 5.0 34.074 +e +"EF" 0.0 37.960 +"0.5%" 0.5 37.569 +"1.0%" 1.0 37.403 +"2.0%" 2.0 36.903 +"2.6%" 2.6 36.413 +"2.8%" 2.8 36.403 +"3.0%" 3.0 36.227 +"5.0%" 5.0 35.540 +e + +_EOF +CONV_FILENAMES="lowdelay_P_4R_slice_1200B_seq_basketball_drill_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_basketball_drill_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_basketball_drive_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_basketball_drive_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_book_arrival_c6_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_book_arrival_c6_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_bqsquare_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_bqsquare_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_cactus_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_cactus_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_four_people_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_four_people_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_kendo_c3_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_kendo_c3_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_kimono_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_kimono_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_kristen_sara_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_kristen_sara_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_park_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_park_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_people_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_people_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_race_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_race_rate_single_60pc.pdf lowdelay_P_4R_slice_1200B_seq_tennis_rate_single.pdf lowdelay_P_4R_slice_1200B_seq_tennis_rate_single_60pc.pdf " +gs -q -sPAPERSIZE=letter -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=Quality_PLR_Methods.pdf $CONV_FILENAMES +rm ${CONV_FILENAMES} diff --git a/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.pdf b/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.pdf new file mode 100644 index 0000000..e797ec7 Binary files /dev/null and b/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.pdf differ diff --git a/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex b/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex index abee388..5fb2973 100644 --- a/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex +++ b/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex @@ -10,184 +10,205 @@ set key bottom left Left reverse width 8.5 set ytics 1, 2,100 set yrange[29:41] set title '\TitleFont Basketball Drill' -plot '-' using 2:1 title '\KeyFont \REF' w lp ls 1, '-' using 2:1 title '\KeyFont \EXP' w lp ls 2, '-' using 2:1 title '\KeyFont \EXPELTEN' w lp ls 3, '-' using 2:1 title '\KeyFont \EXPELTWEN' w lp ls 4 -39.900 0.0 "EF" -38.057 0.5 "0.5%" -36.949 1.0 "1.0%" -35.859 1.6 "1.2%" -35.157 2.0 "1.6%" -33.890 2.6 "2.0%" -33.305 3.0 "3.0%" -31.385 5.0 "5.0%" -#28.217 10.0 "10.0%" +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +"EF" 0.0 39.900 +"0.5%" 0.5 38.057 +"1.0%" 1.0 36.949 +"1.6%" 1.6 35.859 +"2.0%" 2.0 35.157 +"2.6%" 2.6 33.890 +"3.0%" 3.0 33.305 +"5.0%" 5.0 31.385 e -39.600 0.0 "EF" -38.847 0.5 "0.5%" -38.305 1.0 "1.0%" -37.485 1.6 "1.2%" -37.178 2.0 "1.6%" -36.436 2.6 "2.0%" -36.137 3.0 "3.0%" -34.691 5.0 "5.0%" -#31.288 10.0 "10.0%" +"EF" 0.0 39.900 +"0.5%" 0.5 38.919 +"1.0%" 1.0 38.201 +"1.6%" 1.6 37.490 +"2.0%" 2.0 37.093 +"2.6%" 2.6 36.382 +"3.0%" 3.0 35.747 +"5.0%" 5.0 34.027 e -39.480 0.0 "EF" -39.191 0.5 "0.5%" -38.973 1.0 "1.0%" -38.586 1.6 "1.2%" -38.385 2.0 "1.6%" -38.093 2.6 "2.0%" -37.929 3.0 "3.0%" -37.015 5.0 "5.0%" -#34.474 10.0 "10.0%" +"EF" 0.0 39.600 +"0.5%" 0.5 38.847 +"1.0%" 1.0 38.305 +"1.6%" 1.6 37.485 +"2.0%" 2.0 37.178 +"2.6%" 2.6 36.436 +"3.0%" 3.0 36.137 +"5.0%" 5.0 34.691 e -38.880 0.0 "EF" -38.729 0.5 "0.5%" -38.550 1.0 "1.0%" -38.353 1.6 "1.2%" -38.208 2.0 "1.6%" -37.956 2.6 "2.0%" -37.863 3.0 "3.0%" -37.186 5.0 "5.0%" -#35.117 10.0 "10.0%" +"EF" 0.0 39.480 +"0.5%" 0.5 39.191 +"1.0%" 1.0 38.973 +"1.6%" 1.6 38.586 +"2.0%" 2.0 38.385 +"2.6%" 2.6 38.093 +"3.0%" 3.0 37.929 +"5.0%" 5.0 37.015 e +"EF" 0.0 38.880 +"0.5%" 0.5 38.729 +"1.0%" 1.0 38.550 +"1.6%" 1.6 38.353 +"2.0%" 2.0 38.208 +"2.6%" 2.6 37.956 +"3.0%" 3.0 37.863 +"5.0%" 5.0 37.186 +e + set autoscale y set ytics 1, 2,100 set yrange[33:41] set title '\TitleFont Book Arrival' -plot '-' using 2:1 title '\KeyFont \REF' w lp ls 1, '-' using 2:1 title '\KeyFont \EXP' w lp ls 2, '-' using 2:1 title '\KeyFont \EXPELTEN' w lp ls 3, '-' using 2:1 title '\KeyFont \EXPELTWEN' w lp ls 4 -40.280 0.0 "EF" -39.122 0.5 "0.5%" -38.578 1.0 "1.0%" -37.776 1.6 "1.2%" -37.584 2.0 "1.6%" -36.699 2.6 "2.0%" -36.206 3.0 "3.0%" -34.396 5.0 "5.0%" -#31.010 10.0 "10.0%" +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +"EF" 0.0 40.280 +"0.5%" 0.5 39.122 +"1.0%" 1.0 38.578 +"1.6%" 1.6 37.776 +"2.0%" 2.0 37.584 +"2.6%" 2.6 36.699 +"3.0%" 3.0 36.206 +"5.0%" 5.0 34.396 e -39.970 0.0 "EF" -39.391 0.5 "0.5%" -39.053 1.0 "1.0%" -38.553 1.6 "1.2%" -38.419 2.0 "1.6%" -37.928 2.6 "2.0%" -37.719 3.0 "3.0%" -36.705 5.0 "5.0%" -#33.853 10.0 "10.0%" +"EF" 0.0 40.280 +"0.5%" 0.5 39.509 +"1.0%" 1.0 39.199 +"1.6%" 1.6 38.741 +"2.0%" 2.0 38.589 +"2.6%" 2.6 38.058 +"3.0%" 3.0 37.627 +"5.0%" 5.0 36.326 e -40.030 0.0 "EF" -39.788 0.5 "0.5%" -39.716 1.0 "1.0%" -39.460 1.6 "1.2%" -39.339 2.0 "1.6%" -39.088 2.6 "2.0%" -38.918 3.0 "3.0%" -38.348 5.0 "5.0%" -# 36.607 10.0 "10.0%" +"EF" 0.0 39.970 +"0.5%" 0.5 39.391 +"1.0%" 1.0 39.053 +"1.6%" 1.6 38.553 +"2.0%" 2.0 38.419 +"2.6%" 2.6 37.928 +"3.0%" 3.0 37.719 +"5.0%" 5.0 36.705 e -39.790 0.0 "EF" -39.679 0.5 "0.5%" -39.517 1.0 "1.0%" -39.463 1.6 "1.2%" -39.416 2.0 "1.6%" -39.125 2.6 "2.0%" -39.193 3.0 "3.0%" -38.781 5.0 "5.0%" -# 37.323 10.0 "10.0%" +"EF" 0.0 40.030 +"0.5%" 0.5 39.788 +"1.0%" 1.0 39.716 +"1.6%" 1.6 39.460 +"2.0%" 2.0 39.339 +"2.6%" 2.6 39.088 +"3.0%" 3.0 38.918 +"5.0%" 5.0 38.348 +e +"EF" 0.0 39.790 +"0.5%" 0.5 39.679 +"1.0%" 1.0 39.517 +"1.6%" 1.6 39.463 +"2.0%" 2.0 39.416 +"2.6%" 2.6 39.125 +"3.0%" 3.0 39.193 +"5.0%" 5.0 38.781 e set autoscale y set ytics 2, 2, 100 set yrange[32:40] set title '\TitleFont BQSquare' -plot '-' using 2:1 title '\KeyFont \REF' w lp ls 1, '-' using 2:1 title '\KeyFont \EXP' w lp ls 2, '-' using 2:1 title '\KeyFont \EXPELTEN' w lp ls 3, '-' using 2:1 title '\KeyFont \EXPELTWEN' w lp ls 4 -39.300 0.0 "EF" -38.381 0.5 "0.5%" -37.908 1.0 "1.0%" -37.233 1.6 "1.2%" -36.716 2.0 "1.6%" -35.946 2.6 "2.0%" -35.639 3.0 "3.0%" -34.289 5.0 "5.0%" -# 31.365 10.0 "10.0%" +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +"EF" 0.0 39.300 +"0.5%" 0.5 38.381 +"1.0%" 1.0 37.908 +"1.6%" 1.6 37.233 +"2.0%" 2.0 36.716 +"2.6%" 2.6 35.946 +"3.0%" 3.0 35.639 +"5.0%" 5.0 34.289 e -39.150 0.0 "EF" -38.735 0.5 "0.5%" -38.522 1.0 "1.0%" -38.077 1.6 "1.2%" -37.747 2.0 "1.6%" -37.252 2.6 "2.0%" -36.985 3.0 "3.0%" -35.971 5.0 "5.0%" -# 33.495 10.0 "10.0%" +"EF" 0.0 39.300 +"0.5%" 0.5 38.861 +"1.0%" 1.0 38.531 +"1.6%" 1.6 38.110 +"2.0%" 2.0 37.861 +"2.6%" 2.6 37.318 +"3.0%" 3.0 37.069 +"5.0%" 5.0 36.117 e -38.750 0.0 "EF" -38.558 0.5 "0.5%" -38.371 1.0 "1.0%" -38.163 1.6 "1.2%" -37.981 2.0 "1.6%" -37.673 2.6 "2.0%" -37.652 3.0 "3.0%" -36.734 5.0 "5.0%" -# 34.690 10.0 "10.0%" +"EF" 0.0 39.150 +"0.5%" 0.5 38.735 +"1.0%" 1.0 38.522 +"1.6%" 1.6 38.077 +"2.0%" 2.0 37.747 +"2.6%" 2.6 37.252 +"3.0%" 3.0 36.985 +"5.0%" 5.0 35.971 e -38.160 0.0 "EF" -38.015 0.5 "0.5%" -37.897 1.0 "1.0%" -37.767 1.6 "1.2%" -37.652 2.0 "1.6%" -37.447 2.6 "2.0%" -37.411 3.0 "3.0%" -36.765 5.0 "5.0%" -# 34.911 10.0 "10.0%" +"EF" 0.0 38.750 +"0.5%" 0.5 38.558 +"1.0%" 1.0 38.371 +"1.6%" 1.6 38.163 +"2.0%" 2.0 37.981 +"2.6%" 2.6 37.673 +"3.0%" 3.0 37.652 +"5.0%" 5.0 36.734 +e +"EF" 0.0 38.160 +"0.5%" 0.5 38.015 +"1.0%" 1.0 37.897 +"1.6%" 1.6 37.767 +"2.0%" 2.0 37.652 +"2.6%" 2.6 37.447 +"3.0%" 3.0 37.411 +"5.0%" 5.0 36.765 e set autoscale y set ytics 1, 2,100 set yrange[33:42] set title '\TitleFont Kendo' -plot '-' using 2:1 title '\KeyFont \REF' w lp ls 1, '-' using 2:1 title '\KeyFont \EXP' w lp ls 2, '-' using 2:1 title '\KeyFont \EXPELTEN' w lp ls 3, '-' using 2:1 title '\KeyFont \EXPELTWEN' w lp ls 4 -41.150 0.0 "EF" -40.679 0.5 "0.5%" -39.544 1.0 "1.0%" -38.839 1.6 "1.2%" -38.271 2.0 "1.6%" -37.512 2.6 "2.0%" -37.349 3.0 "3.0%" -34.856 5.0 "5.0%" -# 31.573 10.0 "10.0%" +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +"EF" 0.0 41.150 +"0.5%" 0.5 40.679 +"1.0%" 1.0 39.544 +"1.6%" 1.6 38.839 +"2.0%" 2.0 38.271 +"2.6%" 2.6 37.512 +"3.0%" 3.0 37.349 +"5.0%" 5.0 34.856 e -40.270 0.0 "EF" -40.009 0.5 "0.5%" -39.304 1.0 "1.0%" -39.111 1.6 "1.2%" -38.638 2.0 "1.6%" -38.516 2.6 "2.0%" -38.401 3.0 "3.0%" -37.110 5.0 "5.0%" -# 34.829 10.0 "10.0%" +"EF" 0.0 41.150 +"0.5%" 0.5 40.863 +"1.0%" 1.0 40.112 +"1.6%" 1.6 39.675 +"2.0%" 2.0 39.178 +"2.6%" 2.6 38.746 +"3.0%" 3.0 38.606 +"5.0%" 5.0 36.655 e -40.730 0.0 "EF" -40.427 0.5 "0.5%" -40.156 1.0 "1.0%" -39.496 1.6 "1.2%" -39.319 2.0 "1.6%" -39.305 2.6 "2.0%" -38.692 3.0 "3.0%" -37.258 5.0 "5.0%" -# 35.144 10.0 "10.0%" +"EF" 0.0 40.270 +"0.5%" 0.5 40.009 +"1.0%" 1.0 39.304 +"1.6%" 1.6 39.111 +"2.0%" 2.0 38.638 +"2.6%" 2.6 38.516 +"3.0%" 3.0 38.401 +"5.0%" 5.0 37.110 e -40.260 0.0 "EF" -40.065 0.5 "0.5%" -39.801 1.0 "1.0%" -39.599 1.6 "1.2%" -39.492 2.0 "1.6%" -39.238 2.6 "2.0%" -38.950 3.0 "3.0%" -38.358 5.0 "5.0%" -# 36.619 10.0 "10.0%" +"EF" 0.0 40.730 +"0.5%" 0.5 40.427 +"1.0%" 1.0 40.156 +"1.6%" 1.6 39.496 +"2.0%" 2.0 39.319 +"2.6%" 2.6 39.305 +"3.0%" 3.0 38.692 +"5.0%" 5.0 37.258 +e +"EF" 0.0 40.260 +"0.5%" 0.5 40.065 +"1.0%" 1.0 39.801 +"1.6%" 1.6 39.599 +"2.0%" 2.0 39.492 +"2.6%" 2.6 39.238 +"3.0%" 3.0 38.950 +"5.0%" 5.0 38.358 e @@ -195,93 +216,105 @@ set autoscale y set ytics 1, 2,100 set yrange[31:40] set title '\TitleFont Park Scene' -plot '-' using 2:1 title '\KeyFont \REF' w lp ls 1, '-' using 2:1 title '\KeyFont \EXP' w lp ls 2, '-' using 2:1 title '\KeyFont \EXPELTEN' w lp ls 3, '-' using 2:1 title '\KeyFont \EXPELTWEN' w lp ls 4 -39.470 0.0 "EF" -38.067 0.5 "0.5%" -36.969 1.0 "1.0%" -36.205 1.6 "1.2%" -35.555 2.0 "1.6%" -34.863 2.6 "2.0%" -34.545 3.0 "3.0%" -32.923 5.0 "5.0%" -# 30.423 10.0 "10.0%" +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +"EF" 0.0 39.470 +"0.5%" 0.5 38.067 +"1.0%" 1.0 36.969 +"1.6%" 1.6 36.205 +"2.0%" 2.0 35.555 +"2.6%" 2.6 34.863 +"3.0%" 3.0 34.545 +"5.0%" 5.0 32.923 e -38.900 0.0 "EF" -38.528 0.5 "0.5%" -38.113 1.0 "1.0%" -37.721 1.6 "1.2%" -37.380 2.0 "1.6%" -37.064 2.6 "2.0%" -36.846 3.0 "3.0%" -35.771 5.0 "5.0%" -# 32.539 10.0 "10.0%" +"EF" 0.0 39.470 +"0.5%" 0.5 38.988 +"1.0%" 1.0 38.423 +"1.6%" 1.6 37.992 +"2.0%" 2.0 37.596 +"2.6%" 2.6 37.029 +"3.0%" 3.0 36.816 +"5.0%" 5.0 35.236 e -39.160 0.0 "EF" -38.957 0.5 "0.5%" -38.779 1.0 "1.0%" -38.509 1.6 "1.2%" -38.212 2.0 "1.6%" -37.920 2.6 "2.0%" -37.767 3.0 "3.0%" -36.847 5.0 "5.0%" -# 33.341 10.0 "10.0%" +"EF" 0.0 38.900 +"0.5%" 0.5 38.528 +"1.0%" 1.0 38.113 +"1.6%" 1.6 37.721 +"2.0%" 2.0 37.380 +"2.6%" 2.6 37.064 +"3.0%" 3.0 36.846 +"5.0%" 5.0 35.771 e -38.820 0.0 "EF" -38.681 0.5 "0.5%" -38.589 1.0 "1.0%" -38.287 1.6 "1.2%" -38.127 2.0 "1.6%" -37.869 2.6 "2.0%" -37.693 3.0 "3.0%" -36.777 5.0 "5.0%" -# 34.059 10.0 "10.0%" +"EF" 0.0 39.160 +"0.5%" 0.5 38.957 +"1.0%" 1.0 38.779 +"1.6%" 1.6 38.509 +"2.0%" 2.0 38.212 +"2.6%" 2.6 37.920 +"3.0%" 3.0 37.767 +"5.0%" 5.0 36.847 e +"EF" 0.0 38.820 +"0.5%" 0.5 38.681 +"1.0%" 1.0 38.589 +"1.6%" 1.6 38.287 +"2.0%" 2.0 38.127 +"2.6%" 2.6 37.869 +"3.0%" 3.0 37.693 +"5.0%" 5.0 36.777 +e + set autoscale y set ytics 2, 2,100 set yrange[28:41] set title '\TitleFont Tennis' -plot '-' using 2:1 title '\KeyFont \REF' w lp ls 1, '-' using 2:1 title '\KeyFont \EXP' w lp ls 2, '-' using 2:1 title '\KeyFont \EXPELTEN' w lp ls 3, '-' using 2:1 title '\KeyFont \EXPELTWEN' w lp ls 4 -40.680 0.0 "EF" -38.243 0.5 "0.5%" -36.815 1.0 "1.0%" -35.278 1.6 "1.2%" -34.460 2.0 "1.6%" -33.418 2.6 "2.0%" -32.709 3.0 "3.0%" -30.360 5.0 "5.0%" -# 27.083 10.0 "10.0%" +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +"EF" 0.0 40.680 +"0.5%" 0.5 38.243 +"1.0%" 1.0 36.815 +"1.6%" 1.6 35.278 +"2.0%" 2.0 34.460 +"2.6%" 2.6 33.418 +"3.0%" 3.0 32.709 +"5.0%" 5.0 30.360 e -40.030 0.0 "EF" -38.966 0.5 "0.5%" -38.059 1.0 "1.0%" -36.983 1.6 "1.2%" -36.429 2.0 "1.6%" -35.351 2.6 "2.0%" -35.068 3.0 "3.0%" -33.155 5.0 "5.0%" -# 29.764 10.0 "10.0%" +"EF" 0.0 40.680 +"0.5%" 0.5 38.978 +"1.0%" 1.0 37.925 +"1.6%" 1.6 36.541 +"2.0%" 2.0 36.013 +"2.6%" 2.6 35.063 +"3.0%" 3.0 34.362 +"5.0%" 5.0 32.073 e -40.390 0.0 "EF" -39.420 0.5 "0.5%" -38.912 1.0 "1.0%" -38.123 1.6 "1.2%" -37.448 2.0 "1.6%" -36.907 2.6 "2.0%" -36.522 3.0 "3.0%" -34.991 5.0 "5.0%" -# 31.664 10.0 "10.0%" +"EF" 0.0 40.030 +"0.5%" 0.5 38.966 +"1.0%" 1.0 38.059 +"1.6%" 1.6 36.983 +"2.0%" 2.0 36.429 +"2.6%" 2.6 35.351 +"3.0%" 3.0 35.068 +"5.0%" 5.0 33.155 e -40.030 0.0 "EF" -39.612 0.5 "0.5%" -39.371 1.0 "1.0%" -38.763 1.6 "1.2%" -38.549 2.0 "1.6%" -38.128 2.6 "2.0%" -37.679 3.0 "3.0%" -36.606 5.0 "5.0%" -# 33.692 10.0 "10.0%" +"EF" 0.0 40.390 +"0.5%" 0.5 39.420 +"1.0%" 1.0 38.912 +"1.6%" 1.6 38.123 +"2.0%" 2.0 37.448 +"2.6%" 2.6 36.907 +"3.0%" 3.0 36.522 +"5.0%" 5.0 34.991 e +"EF" 0.0 40.030 +"0.5%" 0.5 39.612 +"1.0%" 1.0 39.371 +"1.6%" 1.6 38.763 +"2.0%" 2.0 38.549 +"2.6%" 2.6 38.128 +"3.0%" 3.0 37.679 +"5.0%" 5.0 36.606 +e + \end{gnuplot} diff --git a/043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex b/043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex index 020ba7b..64ea211 100644 --- a/043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex +++ b/043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex @@ -10,138 +10,153 @@ set key bottom left Left reverse width 9 set ytics 1, 2, 100 set yrange[29:41] set title '\TitleFont Basketball Drill' -plot '-' using 2:1 title '\KeyFont \REF' w lp ls 1, '-' using 2:1 title '\KeyFont \EXP' w lp ls 2, '-' using 2:1 title '\KeyFont \EXPELTEN' w lp ls 3, '-' using 2:1 title '\KeyFont \EXPELTWEN' w lp ls 4 -37.570 0.0 "EF" -36.658 0.5 "0.5%" -35.927 1.0 "1.0%" -35.287 1.6 "1.2%" -34.683 2.0 "1.6%" -33.776 2.6 "2.0%" -33.583 3.0 "3.0%" -31.726 5.0 "5.0%" -# 28.381 10.0 "10.0%" +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +"EF" 0.0 37.570 +"0.5%" 0.5 36.658 +"1.0%" 1.0 35.927 +"1.6%" 1.6 35.287 +"2.0%" 2.0 34.683 +"2.6%" 2.6 33.776 +"3.0%" 3.0 33.583 +"5.0%" 5.0 31.726 e -37.120 0.0 "EF" -36.831 0.5 "0.5%" -36.405 1.0 "1.0%" -36.152 1.6 "1.2%" -35.895 2.0 "1.6%" -35.391 2.6 "2.0%" -35.099 3.0 "3.0%" -34.150 5.0 "5.0%" -# 31.861 10.0 "10.0%" +"EF" 0.0 37.570 +"0.5%" 0.5 37.118 +"1.0%" 1.0 36.640 +"1.6%" 1.6 36.239 +"2.0%" 2.0 35.973 +"2.6%" 2.6 35.386 +"3.0%" 3.0 35.252 +"5.0%" 5.0 33.857 e -36.940 0.0 "EF" -36.737 0.5 "0.5%" -36.569 1.0 "1.0%" -36.487 1.6 "1.2%" -36.357 2.0 "1.6%" -36.072 2.6 "2.0%" -35.991 3.0 "3.0%" -35.431 5.0 "5.0%" -# 33.753 10.0 "10.0%" +"EF" 0.0 37.120 +"0.5%" 0.5 36.831 +"1.0%" 1.0 36.405 +"1.6%" 1.6 36.152 +"2.0%" 2.0 35.895 +"2.6%" 2.6 35.391 +"3.0%" 3.0 35.099 +"5.0%" 5.0 34.150 e -36.130 0.0 "EF" -36.038 0.5 "0.5%" -35.975 1.0 "1.0%" -35.885 1.6 "1.2%" -35.749 2.0 "1.6%" -35.627 2.6 "2.0%" -35.617 3.0 "3.0%" -35.114 5.0 "5.0%" -# 33.962 10.0 "10.0%" +"EF" 0.0 36.940 +"0.5%" 0.5 36.737 +"1.0%" 1.0 36.569 +"1.6%" 1.6 36.487 +"2.0%" 2.0 36.357 +"2.6%" 2.6 36.072 +"3.0%" 3.0 35.991 +"5.0%" 5.0 35.431 +e +"EF" 0.0 36.130 +"0.5%" 0.5 36.038 +"1.0%" 1.0 35.975 +"1.6%" 1.6 35.885 +"2.0%" 2.0 35.749 +"2.6%" 2.6 35.627 +"3.0%" 3.0 35.617 +"5.0%" 5.0 35.114 e set autoscale y set ytics 1, 2, 100 set yrange[33:41] set title '\TitleFont Book Arrival' -plot '-' using 2:1 title '\KeyFont \REF' w lp ls 1, '-' using 2:1 title '\KeyFont \EXP' w lp ls 2, '-' using 2:1 title '\KeyFont \EXPELTEN' w lp ls 3, '-' using 2:1 title '\KeyFont \EXPELTWEN' w lp ls 4 -38.920 0.0 "EF" -38.641 0.5 "0.5%" -37.769 1.0 "1.0%" -37.575 1.6 "1.2%" -36.545 2.0 "1.6%" -36.302 2.6 "2.0%" -36.189 3.0 "3.0%" -35.001 5.0 "5.0%" -# 31.592 10.0 "10.0%" +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +"EF" 0.0 38.920 +"0.5%" 0.5 38.641 +"1.0%" 1.0 37.769 +"1.6%" 1.6 37.575 +"2.0%" 2.0 36.545 +"2.6%" 2.6 36.302 +"3.0%" 3.0 36.189 +"5.0%" 5.0 35.001 e -38.400 0.0 "EF" -38.253 0.5 "0.5%" -37.711 1.0 "1.0%" -37.676 1.6 "1.2%" -37.137 2.0 "1.6%" -37.037 2.6 "2.0%" -37.005 3.0 "3.0%" -36.027 5.0 "5.0%" -# 34.522 10.0 "10.0%" +"EF" 0.0 38.920 +"0.5%" 0.5 38.751 +"1.0%" 1.0 38.215 +"1.6%" 1.6 38.061 +"2.0%" 2.0 37.341 +"2.6%" 2.6 37.219 +"3.0%" 3.0 36.998 +"5.0%" 5.0 36.417 e -38.460 0.0 "EF" -38.305 0.5 "0.5%" -38.169 1.0 "1.0%" -38.029 1.6 "1.2%" -37.780 2.0 "1.6%" -37.651 2.6 "2.0%" -37.582 3.0 "3.0%" -36.977 5.0 "5.0%" -# 35.674 10.0 "10.0%" +"EF" 0.0 38.400 +"0.5%" 0.5 38.253 +"1.0%" 1.0 37.711 +"1.6%" 1.6 37.676 +"2.0%" 2.0 37.137 +"2.6%" 2.6 37.037 +"3.0%" 3.0 37.005 +"5.0%" 5.0 36.027 e -37.830 0.0 "EF" -37.715 0.5 "0.5%" -37.487 1.0 "1.0%" -37.537 1.6 "1.2%" -37.469 2.0 "1.6%" -37.419 2.6 "2.0%" -37.321 3.0 "3.0%" -36.980 5.0 "5.0%" -# 35.818 10.0 "10.0%" +"EF" 0.0 38.460 +"0.5%" 0.5 38.305 +"1.0%" 1.0 38.169 +"1.6%" 1.6 38.029 +"2.0%" 2.0 37.780 +"2.6%" 2.6 37.651 +"3.0%" 3.0 37.582 +"5.0%" 5.0 36.977 +e +"EF" 0.0 37.830 +"0.5%" 0.5 37.715 +"1.0%" 1.0 37.487 +"1.6%" 1.6 37.537 +"2.0%" 2.0 37.469 +"2.6%" 2.6 37.419 +"3.0%" 3.0 37.321 +"5.0%" 5.0 36.980 e set autoscale y set ytics 2, 2,100 set yrange[32:40] set title '\TitleFont BQSquare' -plot '-' using 2:1 title '\KeyFont \REF' w lp ls 1, '-' using 2:1 title '\KeyFont \EXP' w lp ls 2, '-' using 2:1 title '\KeyFont \EXPELTEN' w lp ls 3, '-' using 2:1 title '\KeyFont \EXPELTWEN' w lp ls 4 -36.780 0.0 "EF" -36.370 0.5 "0.5%" -36.129 1.0 "1.0%" -35.735 1.6 "1.2%" -35.480 2.0 "1.6%" -34.912 2.6 "2.0%" -34.947 3.0 "3.0%" -33.946 5.0 "5.0%" -# 31.479 10.0 "10.0%" +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +"EF" 0.0 36.780 +"0.5%" 0.5 36.370 +"1.0%" 1.0 36.129 +"1.6%" 1.6 35.735 +"2.0%" 2.0 35.480 +"2.6%" 2.6 34.912 +"3.0%" 3.0 34.947 +"5.0%" 5.0 33.946 e -36.610 0.0 "EF" -36.340 0.5 "0.5%" -36.261 1.0 "1.0%" -36.051 1.6 "1.2%" -35.839 2.0 "1.6%" -35.513 2.6 "2.0%" -35.547 3.0 "3.0%" -34.913 5.0 "5.0%" -# 33.003 10.0 "10.0%" +"EF" 0.0 36.780 +"0.5%" 0.5 36.565 +"1.0%" 1.0 36.441 +"1.6%" 1.6 36.181 +"2.0%" 2.0 36.037 +"2.6%" 2.6 35.697 +"3.0%" 3.0 35.773 +"5.0%" 5.0 35.013 e -36.230 0.0 "EF" -36.135 0.5 "0.5%" -36.021 1.0 "1.0%" -35.888 1.6 "1.2%" -35.787 2.0 "1.6%" -35.586 2.6 "2.0%" -35.569 3.0 "3.0%" -35.200 5.0 "5.0%" -# 33.764 10.0 "10.0%" +"EF" 0.0 36.610 +"0.5%" 0.5 36.340 +"1.0%" 1.0 36.261 +"1.6%" 1.6 36.051 +"2.0%" 2.0 35.839 +"2.6%" 2.6 35.513 +"3.0%" 3.0 35.547 +"5.0%" 5.0 34.913 e -35.700 0.0 "EF" -35.627 0.5 "0.5%" -35.560 1.0 "1.0%" -35.459 1.6 "1.2%" -35.399 2.0 "1.6%" -35.297 2.6 "2.0%" -35.258 3.0 "3.0%" -34.891 5.0 "5.0%" -# 33.942 10.0 "10.0%" +"EF" 0.0 36.230 +"0.5%" 0.5 36.135 +"1.0%" 1.0 36.021 +"1.6%" 1.6 35.888 +"2.0%" 2.0 35.787 +"2.6%" 2.6 35.586 +"3.0%" 3.0 35.569 +"5.0%" 5.0 35.200 +e +"EF" 0.0 35.700 +"0.5%" 0.5 35.627 +"1.0%" 1.0 35.560 +"1.6%" 1.6 35.459 +"2.0%" 2.0 35.399 +"2.6%" 2.6 35.297 +"3.0%" 3.0 35.258 +"5.0%" 5.0 34.891 e @@ -149,138 +164,152 @@ set autoscale y set ytics 1, 2, 100 set yrange[33:42] set title '\TitleFont Kendo' -plot '-' using 2:1 title '\KeyFont \REF' w lp ls 1, '-' using 2:1 title '\KeyFont \EXP' w lp ls 2, '-' using 2:1 title '\KeyFont \EXPELTEN' w lp ls 3, '-' using 2:1 title '\KeyFont \EXPELTWEN' w lp ls 4 -38.730 0.0 "EF" -38.365 0.5 "0.5%" -37.714 1.0 "1.0%" -37.599 1.6 "1.2%" -36.762 2.0 "1.6%" -36.695 2.6 "2.0%" -35.859 3.0 "3.0%" -34.589 5.0 "5.0%" -# 31.635 10.0 "10.0%" +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +"EF" 0.0 38.730 +"0.5%" 0.5 38.365 +"1.0%" 1.0 37.714 +"1.6%" 1.6 37.599 +"2.0%" 2.0 36.762 +"2.6%" 2.6 36.695 +"3.0%" 3.0 35.859 +"5.0%" 5.0 34.589 e -37.550 0.0 "EF" -37.280 0.5 "0.5%" -37.189 1.0 "1.0%" -36.991 1.6 "1.2%" -36.823 2.0 "1.6%" -36.388 2.6 "2.0%" -36.283 3.0 "3.0%" -35.449 5.0 "5.0%" -# 33.936 10.0 "10.0%" +"EF" 0.0 38.730 +"0.5%" 0.5 38.476 +"1.0%" 1.0 38.049 +"1.6%" 1.6 37.966 +"2.0%" 2.0 37.427 +"2.6%" 2.6 37.311 +"3.0%" 3.0 36.554 +"5.0%" 5.0 35.775 e -38.080 0.0 "EF" -37.819 0.5 "0.5%" -37.382 1.0 "1.0%" -37.364 1.6 "1.2%" -36.767 2.0 "1.6%" -36.901 2.6 "2.0%" -36.804 3.0 "3.0%" -35.920 5.0 "5.0%" -# 34.085 10.0 "10.0%" +"EF" 0.0 37.550 +"0.5%" 0.5 37.280 +"1.0%" 1.0 37.189 +"1.6%" 1.6 36.991 +"2.0%" 2.0 36.823 +"2.6%" 2.6 36.388 +"3.0%" 3.0 36.283 +"5.0%" 5.0 35.449 e -37.260 0.0 "EF" -37.147 0.5 "0.5%" -36.921 1.0 "1.0%" -36.823 1.6 "1.2%" -36.588 2.0 "1.6%" -36.552 2.6 "2.0%" -36.383 3.0 "3.0%" -35.746 5.0 "5.0%" -# 34.202 10.0 "10.0%" +"EF" 0.0 38.080 +"0.5%" 0.5 37.819 +"1.0%" 1.0 37.382 +"1.6%" 1.6 37.364 +"2.0%" 2.0 36.767 +"2.6%" 2.6 36.901 +"3.0%" 3.0 36.804 +"5.0%" 5.0 35.920 +e +"EF" 0.0 37.260 +"0.5%" 0.5 37.147 +"1.0%" 1.0 36.921 +"1.6%" 1.6 36.823 +"2.0%" 2.0 36.588 +"2.6%" 2.6 36.552 +"3.0%" 3.0 36.383 +"5.0%" 5.0 35.746 e - set autoscale y set ytics 1, 2,100 set yrange[31:40] set title '\TitleFont Park Scene' -plot '-' using 2:1 title '\KeyFont \REF' w lp ls 1, '-' using 2:1 title '\KeyFont \EXP' w lp ls 2, '-' using 2:1 title '\KeyFont \EXPELTEN' w lp ls 3, '-' using 2:1 title '\KeyFont \EXPELTWEN' w lp ls 4 -37.810 0.0 "EF" -36.969 0.5 "0.5%" -36.189 1.0 "1.0%" -35.422 1.6 "1.2%" -35.190 2.0 "1.6%" -34.477 2.6 "2.0%" -34.091 3.0 "3.0%" -32.873 5.0 "5.0%" -# 30.508 10.0 "10.0%" +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +"EF" 0.0 37.810 +"0.5%" 0.5 36.969 +"1.0%" 1.0 36.189 +"1.6%" 1.6 35.422 +"2.0%" 2.0 35.190 +"2.6%" 2.6 34.477 +"3.0%" 3.0 34.091 +"5.0%" 5.0 32.873 e -37.230 0.0 "EF" -37.020 0.5 "0.5%" -36.750 1.0 "1.0%" -36.504 1.6 "1.2%" -36.361 2.0 "1.6%" -36.002 2.6 "2.0%" -35.951 3.0 "3.0%" -35.174 5.0 "5.0%" -# 33.007 10.0 "10.0%" +"EF" 0.0 37.810 +"0.5%" 0.5 37.526 +"1.0%" 1.0 37.148 +"1.6%" 1.6 36.818 +"2.0%" 2.0 36.749 +"2.6%" 2.6 36.307 +"3.0%" 3.0 36.087 +"5.0%" 5.0 35.288 e -37.460 0.0 "EF" -37.289 0.5 "0.5%" -37.189 1.0 "1.0%" -37.070 1.6 "1.2%" -36.905 2.0 "1.6%" -36.740 2.6 "2.0%" -36.517 3.0 "3.0%" -36.009 5.0 "5.0%" -# 34.323 10.0 "10.0%" +"EF" 0.0 37.230 +"0.5%" 0.5 37.020 +"1.0%" 1.0 36.750 +"1.6%" 1.6 36.504 +"2.0%" 2.0 36.361 +"2.6%" 2.6 36.002 +"3.0%" 3.0 35.951 +"5.0%" 5.0 35.174 e -37.010 0.0 "EF" -36.903 0.5 "0.5%" -36.868 1.0 "1.0%" -36.735 1.6 "1.2%" -36.625 2.0 "1.6%" -36.501 2.6 "2.0%" -36.354 3.0 "3.0%" -35.955 5.0 "5.0%" -# 34.623 10.0 "10.0%" +"EF" 0.0 37.460 +"0.5%" 0.5 37.289 +"1.0%" 1.0 37.189 +"1.6%" 1.6 37.070 +"2.0%" 2.0 36.905 +"2.6%" 2.6 36.740 +"3.0%" 3.0 36.517 +"5.0%" 5.0 36.009 +e +"EF" 0.0 37.010 +"0.5%" 0.5 36.903 +"1.0%" 1.0 36.868 +"1.6%" 1.6 36.735 +"2.0%" 2.0 36.625 +"2.6%" 2.6 36.501 +"3.0%" 3.0 36.354 +"5.0%" 5.0 35.955 e set autoscale y set ytics 2, 2,100 set yrange[28:41] set title '\TitleFont Tennis' -plot '-' using 2:1 title '\KeyFont \REF' w lp ls 1, '-' using 2:1 title '\KeyFont \EXP' w lp ls 2, '-' using 2:1 title '\KeyFont \EXPELTEN' w lp ls 3, '-' using 2:1 title '\KeyFont \EXPELTWEN' w lp ls 4 -39.020 0.0 "EF" -37.569 0.5 "0.5%" -36.453 1.0 "1.0%" -35.481 1.6 "1.2%" -34.685 2.0 "1.6%" -33.662 2.6 "2.0%" -33.442 3.0 "3.0%" -31.232 5.0 "5.0%" -# 27.690 10.0 "10.0%" +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +"EF" 0.0 39.020 +"0.5%" 0.5 37.569 +"1.0%" 1.0 36.453 +"1.6%" 1.6 35.481 +"2.0%" 2.0 34.685 +"2.6%" 2.6 33.662 +"3.0%" 3.0 33.442 +"5.0%" 5.0 31.232 e -38.170 0.0 "EF" -37.482 0.5 "0.5%" -36.874 1.0 "1.0%" -36.215 1.6 "1.2%" -35.711 2.0 "1.6%" -35.236 2.6 "2.0%" -34.863 3.0 "3.0%" -33.289 5.0 "5.0%" -# 30.118 10.0 "10.0%" +"EF" 0.0 39.020 +"0.5%" 0.5 37.965 +"1.0%" 1.0 37.052 +"1.6%" 1.6 36.376 +"2.0%" 2.0 35.737 +"2.6%" 2.6 34.841 +"3.0%" 3.0 34.667 +"5.0%" 5.0 32.678 e -38.580 0.0 "EF" -37.739 0.5 "0.5%" -37.295 1.0 "1.0%" -36.679 1.6 "1.2%" -36.384 2.0 "1.6%" -35.732 2.6 "2.0%" -35.538 3.0 "3.0%" -34.074 5.0 "5.0%" -# 31.364 10.0 "10.0%" +"EF" 0.0 38.170 +"0.5%" 0.5 37.482 +"1.0%" 1.0 36.874 +"1.6%" 1.6 36.215 +"2.0%" 2.0 35.711 +"2.6%" 2.6 35.236 +"3.0%" 3.0 34.863 +"5.0%" 5.0 33.289 e -37.960 0.0 "EF" -37.569 0.5 "0.5%" -37.403 1.0 "1.0%" -36.947 1.6 "1.2%" -36.903 2.0 "1.6%" -36.413 2.6 "2.0%" -36.227 3.0 "3.0%" -35.540 5.0 "5.0%" -# 32.985 10.0 "10.0%" +"EF" 0.0 38.580 +"0.5%" 0.5 37.739 +"1.0%" 1.0 37.295 +"1.6%" 1.6 36.679 +"2.0%" 2.0 36.384 +"2.6%" 2.6 35.732 +"3.0%" 3.0 35.538 +"5.0%" 5.0 34.074 +e +"EF" 0.0 37.960 +"0.5%" 0.5 37.569 +"1.0%" 1.0 37.403 +"1.6%" 1.6 36.947 +"2.0%" 2.0 36.903 +"2.6%" 2.6 36.413 +"3.0%" 3.0 36.227 +"5.0%" 5.0 35.540 e \end{gnuplot} diff --git a/060_Appendix/Papers.tex b/060_Appendix/Papers.tex index 5135f8d..e527b51 100755 --- a/060_Appendix/Papers.tex +++ b/060_Appendix/Papers.tex @@ -9,9 +9,9 @@ Submitted papers are also presented. \begin{description} \item[\namedlabel{Journal:TwoStage}{J1}] -J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, ``A two-stage approach for robust HEVC coding and streaming,'' \emph{IEEE Transactions on Circuits and Systems for Video Technology}, vol. 28, no. 8, pp. 1960-1973, Aug. 2018. +J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, ``A two-stage approach for robust HEVC coding and streaming,'' \emph{IEEE Transactions on Circuits and Systems for Video Technology}, vol. 28, no. 8, pp. 1960-1973, Aug. 2018 (Apr. 2017). \item[\namedlabel{Journal:ConcAware}{J2}] -J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, ``Error Concealment-Aware Encoding for Robust Video Transmission,'' \emph{IEEE Transactions on Broadcasting}. +J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, ``Error Concealment-Aware Encoding for Robust Video Transmission,'' \emph{IEEE Transactions on Broadcasting} (Aug. 2018). \end{description} \section{Conference publications} diff --git a/060_Appendix/Sequences.tex b/060_Appendix/Sequences.tex index a615847..a0f301e 100755 --- a/060_Appendix/Sequences.tex +++ b/060_Appendix/Sequences.tex @@ -79,11 +79,11 @@ This appendix illustrates the original test signals used in simulations througho \caption{Park Scene, $1920\times1080$, 24 fps, 240 frames.} \end{figure} -%\begin{figure}[tbp] -% \centering -% \includegraphics[height=\SeuqenceSize]{\AppenAFigs/PartyScene_832x480_50_FRAME_100.jpg} -% \caption{Party Scene, $832\times480$, 50 fps, 240 frames.} -%\end{figure} +\begin{figure}[tbp] + \centering + \includegraphics[height=\SeuqenceSize]{\AppenAFigs/PartyScene_832x480_50_FRAME_100.jpg} + \caption{Party Scene, $832\times480$, 50 fps, 240 frames.} +\end{figure} \begin{figure}[tbp] \centering diff --git a/carreira_phd_thesis.sty b/carreira_phd_thesis.sty index 11fd1ae..dafe38e 100644 --- a/carreira_phd_thesis.sty +++ b/carreira_phd_thesis.sty @@ -77,7 +77,8 @@ \newcommand{\yfshift}{0.05cm} \definecolor{TikZColorFill}{RGB}{220,220,220} \definecolor{TikZColorFillDark}{RGB}{120,120,120} -\definecolor{TikZColorFillProp}{RGB}{159, 161, 249} +%\definecolor{TikZColorFillProp}{RGB}{159, 161, 249} +\definecolor{TikZColorFillProp}{RGB}{86, 151, 255} %\definecolor{colorBackground}{RGB}{194,215,249} %\definecolor{colorBackgroundSecondary}{RGB}{158,195,255} \definecolor{TikZColorBlock0}{RGB}{0,95,120} @@ -100,7 +101,8 @@ % *** FLOAT PACKAGES *** % Coloca sempre as figuras dentro das secções a que pertencem \usepackage{float} -%\usepackage[section]{placeins} +\usepackage[section]{placeins} +% \usepackage{flafter} % *** MATH PACKAGES *** diff --git a/carreira_phd_thesis.tex b/carreira_phd_thesis.tex index f847010..14b632f 100644 --- a/carreira_phd_thesis.tex +++ b/carreira_phd_thesis.tex @@ -44,7 +44,7 @@ \titlefigs{\includegraphics[width=10cm]{lborolodonlogo.png}} % make the title area -%\listoftodos +\listoftodos \maketitle @@ -85,13 +85,13 @@ \include{050_Conclusions/Conclusion} -\backmatter \appendix \include{060_Appendix/Sequences} \include{060_Appendix/Papers} %\def\extraresults{0} %\include{060_Appendix/Results} +\backmatter \input{References.tex} % that's all folks diff --git a/carreira_phd_thesis_abr.sty b/carreira_phd_thesis_abr.sty index 543fd8e..cd5ac73 100644 --- a/carreira_phd_thesis_abr.sty +++ b/carreira_phd_thesis_abr.sty @@ -187,7 +187,7 @@ \def\CHKB{\mbox{\textit{Chkb}}\xspace} \def\EXP{\mbox{\textit{Exp}}\xspace} -\def\FIXED{\mbox{\textit{Dyn}}\xspace} +\def\FIXED{\mbox{\textit{Fixed}}\xspace} %\def\DYN{\mbox{\textit{Prop-MV}}\xspace} \def\DYN{\mbox{\textit{Dyn}}\xspace} \def\NOREDUND{\mbox{Without MVs}\xspace}