From 63208b1c3db4fb064d575d562db5635bdac1d323 Mon Sep 17 00:00:00 2001 From: Joao Carreira Date: Thu, 13 Dec 2018 15:34:25 +0000 Subject: [PATCH] Review chapter 4 --- .latexmkrc | 2 +- .../ErrorResilienceEvaluation.tex | 170 +++++++++--------- .../QualityEvaluation/PSNRvsIntra.tex | 2 +- carreira_phd_thesis.sty | 3 +- 4 files changed, 89 insertions(+), 88 deletions(-) diff --git a/.latexmkrc b/.latexmkrc index 5ddd588..8c199ec 100644 --- a/.latexmkrc +++ b/.latexmkrc @@ -35,5 +35,5 @@ sub makeglossaries { return 0; } -$clean_ext = "bbl nav out snm acn aux bbl blg fls ist log lof lot maf out toc"; +$clean_ext = "out snm acn acr aux alg bbl blg fls ist log lof lot maf out toc nav xdy"; diff --git a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex index 0888e2f..d139a11 100644 --- a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex +++ b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex @@ -15,7 +15,7 @@ \vspace*{\fill} \section{Introduction} -The use of \ac{hevc} in different type of application and video services over networks that are prone to errors and packet losses, requires mechanisms to minimise the effect of such probabilistic events. This chapter starts by presenting an evaluation study about the error resilience of \ac{hevc} in Section~\ref{Section3:ErrorResilienceEvaluation} , with the aim of defining the weak elements of coded streams and the level of error robustness under different transmission loss conditions. The evaluation study covers different coding configuration to evaluate the individual impact of some new coding tools introduced in this standard, e.g., increased partition size, intra-refreshing, slice partitioning and advanced motion vector prediction. +The use of \ac{hevc} in different type of application and video services over networks that are prone to errors and packet losses, requires mechanisms to minimise the effect of such probabilistic events. This chapter starts by presenting an evaluation study about the error resilience of \ac{hevc} in Section~\ref{Section3:ErrorResilienceEvaluation}, with the aim of defining the weak elements of coded streams and the level of error robustness under different transmission loss conditions. The evaluation study covers different coding configuration to evaluate the individual impact of some new coding tools introduced in this standard, e.g., increased partition size, intra-refreshing, slice partitioning and advanced motion vector prediction. Then, in Section~\ref{Section3:DynamicMVRefreshHEVC}, based on the results of the simulation study, a novel method to increase the error robustness of the \ac{mv} coding is presented and compared to existing techniques. The proposed method introduces refresh points in the motion prediction used in the \ac{hevc}, which significantly reduces the impact of mismatch \ac{mv} predictions. @@ -43,40 +43,39 @@ Interlayer communication must be used to cope with different protocol layers, en Therefore, the error robustness of video streams can be investigated on its own, assuming any possible form of streaming/transport technologies and error detection. This is the main focus of the study presented in this section. -There are some studies presented in the past regarding the error robustness of the \ac{hevc} standard, which are worth to be mentioned as they provide relevant insights for the research carried out in this work. In~\cite{StudyHEVCErrorsDez2012,HEVCWireless_2016} both, the coding efficiency and error robustness of the \ac{hevc} are evaluated and compared against the \ac{h264}. Experimental results confirm that \ac{hevc} has reduced error robustness despite its increased coding efficiency. +There are some studies presented in the past regarding the error robustness of the \ac{hevc} standard, which are worth to be mentioned as they provide relevant insights for the research carried out in this work. In~\cite{StudyHEVCErrorsDez2012,HEVCWireless_2016} the coding efficiency and error robustness of the \ac{hevc} are evaluated and compared against the \ac{h264}. Experimental results confirm that \ac{hevc} has reduced error robustness despite its increased coding efficiency. A subjective evaluation study is presented in~\cite{SubjectiveEvalSep2013,QoEHEVCMay2014} revealing that packet loss ratios higher than $3\%$ have significant effect on the perceived video quality. -The vulnerability of \ac{mv} prediction was investigated and its impact on the error robustness was evaluated in \cite{DynamicTMVP2011}. -Moreover, a method to introduce \ac{mv} refresh points was proposed, achieving average \ac{psnr} gains of approximately 3~dB. +The vulnerability of \ac{mv} prediction is investigated and its impact on the error robustness is evaluated in \cite{DynamicTMVP2011}. +Moreover, in the same study, a method to introduce \ac{mv} refresh points is proposed, achieving average \ac{psnr} gains of approximately 3~dB. Summarising, the decrease in error robustness is mostly due to the strong data dependencies imposed by the highly complex prediction modes of \ac{hevc}. Since these are selected according to a rate distortion optimisation criterion that assumes error-free transmission, the decoder faces different problems whenever errors or data loss occur in the transmission network \cite{ROPE2007}. The following sub-sections present an error robustness evaluation of \ac{hevc} focused on the impact of the novel coding techniques introduced in the standard, extending the generic studies presented in previous works. This study provides relevant insights for research directions and motivation to investigate solutions for the error robustness problem in \ac{hevc}. The following case-studies are covered in this experimental evaluation: \begin{enumerate} - \item different maximum \ac{cu} sizes to evaluate the impact of larger frame partitioning; + \item different distances between intra frames, to evaluate the impact of introducing different number of refresh points; + \item different maximum \ac{cu} sizes to evaluate the impact of larger frame partitioning; % \item experiments with and without filter operations, i.e., deblocking filter and \ac{sao}, in order to evaluate their impact on decoding video quality; - \item different distances between intra frames, to evaluate the impact of introducing different number of refresh points; - \item different slice partitioning schemes; - \item different \ac{mv} coding schemes using different types of \ac{mv} candidates, with analysis at the spatial and temporal levels. + \item different slice partitioning schemes; + \item different \ac{mv} coding schemes using different types of \ac{mv} candidates, with analysis at the spatial and temporal levels. \end{enumerate} \subsection{Characterisation of the experiments} \label{ExperimentalTestSetup} - - -The experimental setup used in this study to evaluate the error robustness of the \ac{hevc} was based on seventeen test sequences, which were chosen to cover different types of motion and texture complexity. -% Each sequence comprises 240 frames and the whole set covers. +The experimental setup used in this study to evaluate the error robustness of the \ac{hevc} is based on seventeen test sequences, which were chosen to cover different types of motion and texture complexity. Table~\ref{TableTestSequences} presents a summary of the main characteristics of the test sequences. Appendix~\ref{Appen:Sequences} shows a single frame of each test sequence. -The spatial and temporal complexity of these video sequences was measured by using the following metrics \cite{ITUTP9102008}: +Considering a test sequence with $N$ frames $f(t)$ the pixel values (Luma component) of the frame at instant $t$, the spatial and temporal complexity of the video sequences was measured by using the following metrics \cite{ITUTP9102008}: \begin{itemize} - \item {\ac{si}} based on Sobel filter. Each video frame ($f_t$) is firstly filtered, then the standard deviation is computed. Finally, the maximum value over time is chosen to represent the \ac{si} metric. This process can be represented by the following equation: - \begin{equation} - SI = max_{time}\Big\{ std_{space} \big( Sobel\left(f_t\right) \big) \Big\}. - \end{equation} - \item {\ac{ti}} based upon the motion difference, which is given by the difference between the pixel values at the same spatial location but in consecutive time instances. \ac{ti} is computed as the maximum displacement over time of the standard deviation over space. Larger displacements in adjacent frames will result in higher values of \ac{ti}. This process can be represented by the following: - \begin{equation} - TI = max_{time}\Big\{ std_{space} \big( f_{t}(x,y) - f_{t-1}(x,y) \big) \Big\}. - \end{equation} + \item {\ac{si}} based on Sobel filter. Each video frame is firstly filtered, then the standard deviation ($std$) of the pixel values is computed. Finally, the maximum value over time ($\maxop_{t\in [1..N]}$), is chosen to represent the \ac{si} metric. This process can be represented by the following equation: + \begin{equation} +% SI = max_{time}\Big\{ std \big( Sobel\left(f_t\right) \big) \Big\}. + SI = \maxop_{t\in [1..N]} \Big\{ std \big( sobel\left(f\left(t\right)\right) \big) \Big\}. + \end{equation} + \item {\ac{ti}} based upon the motion difference, which is given by the difference between the pixel values at the same spatial location but in consecutive time instances. \ac{ti} is computed as the maximum displacement over time of the standard deviation over space. Larger displacements in adjacent frames will result in higher values of \ac{ti}. This process can be represented by the following: + \begin{equation} +% TI = max_{time}\Big\{ std \big( f_t - f_{t-1} \big) \Big\}. + TI = \maxop_{t\in [2..N]} \Big\{ std \big( f\left(t\right) - f\left(t-1\right) \big) \Big\}. + \end{equation} \end{itemize} The values of \ac{si} and \ac{ti} calculated for the test sequences are shown in the Table~\ref{TableTestSequences} (second column). The percentage of inter-coded blocks and the bitrates are also shown in the table and explained below. @@ -84,47 +83,61 @@ The percentage of inter-coded blocks and the bitrates are also shown in the tabl \input{031_ErrorResilienceEvaluation/TablesSequences.tex} For the simulation carried out in this study the \ac{hm} reference software, version $16.2$, was used \cite{HEVCReferenceSoftware162}. -The sequences were encoded with an \ac{idr} period of $16$ frames and all the coding modes enabled, following two recommended configurations: \ac{ld}, i.e., P-frames with 4 references frames, and \ac{ra}, i.e., B-frames with 2 reference frames in each list~\cite{HEVCCommonTests2012}. -These prediction structures cover the use of P-frames and hierarchical B-frames~\cite{H264ScalableOverviewSep2007}, which are commonly used in HEVC streaming~\cite{RealTimeHEVC2016}. +The sequences were encoded with all the coding modes enabled, following two recommended configurations: \ac{ld}, i.e., P-frames with 4 references frames, and \ac{ra}, i.e., B-frames with 2 reference frames in each list~\cite{HEVCCommonTests2012}. +These prediction structures cover the use of P-frames and hierarchical B-frames~\cite{H264ScalableOverviewSep2007}, which are commonly used in HEVC streaming~\cite{RealTimeHEVC2016}. An \ac{idr} period of $16$ frames is used, derived from the recommended configurations. Further \ac{idr} periods are evaluated in Section~\ref{SEC3:IntraRefreshEvaluation}. In order to achieve high quality images for all coded sequences (i.e., \ac{psnr} around 40 dB) different bitrates were used, as listed in Table~\ref{TableTestSequences}. The filtering operations were disabled at slices boundaries, in order to keep the slices self-contained. Moreover, the ratio of inter-coded \acs{cu} of each coded sequence using the default coding parameters is also shown in the third column of the table since this is an important parameter to analyse error propagation. -%The following bitrates were selected to achieve high quality (i.e., around 40 dB): 1.0 Mbits/s (Kendo), 1.5 Mbits/s (Book Arrival), 4.0 Mbits/s (BQSquare, Park Scene and Race Horses) and 12.0 Mbits/s (People on Street). -% At the decoder-side, error concealment is based on motion compensation from previously decoded frames. -% The motion field of a missing frame is obtained applying \ac{mv} extrapolation from the two closest neighbours. -% Afterwards, the final \acs{mv} are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}. -To simulate the lossy transmission environment, each frame is divided into several slices, each one encoded and packetized into one \ac{nal} unit, which are then transmitted as the payload of independent packets. +\begin{table}[t] + \centering + \caption{Characteristics of the loss patterns used in the experimental simulations.} + \label{TAB3:LossPatterns} + \footnotesize + \begin{tabular}{ccc} + \toprule + \textbf{Packet Loss Ratio (PLR)} & \textbf{Average burst length} & \textbf{Maximum burst size} \\ + \midrule + 1\% & 1.24 & 2 \\ \hline + 3\% & 1.47 & 3 \\ \hline + 5\% & 1.83 & 5 \\ \hline + 10\% & 2.05 & 7 \\ + \bottomrule + \end{tabular} +\end{table} + +To simulate the lossy transmission environment, each frame is divided into several slices and each slice encoded and packetized into one \ac{nal} unit, which are then transmitted as the payload of independent packets. In these simulations a whole packet is lost whenever an error occurs, originating a lost slice. -Random packet loss was simulated using a two-state Markov model for \ac{plr} of 1\%, 3\%, 5\%, 10\%. -For each test condition, 50 runs were performed and the average \ac{psnr} was measured. +\HIGHLIGHT{Random packet loss was simulated using a two-state Markov model, derived from the classical Gilbert-Elliot \cite{GilbertModel_1960, ElliotModel_1963}, which can be used to describe error patterns in real time video applications~\cite{GilbertElliotModelPlr_2008}. In this study a simplified version of the Gilbert-Elliot model was used by only defining the loss probability without control over burst loss events. +The application of this model results in loss patterns with approximately the same average burst loss as the ones provided in \cite{JCTVCLossSim_2012}. +For each test condition, 50 runs were performed and the average \ac{psnr} was measured. Four different \acs{plr} are used resulting in the characteristics presented in Table~\ref{TAB3:LossPatterns}.} The remaining configurations have been kept at their default values, as given in the common test conditions~\cite{HEVCCommonTests2012}. -%\subsection{Experimental results} \subsection{Evaluation of the impact of the partition units size} In the first set of experiments the impact of the maximum \ac{cu} size in the decoded video quality is investigated. In this experiment the maximum \ac{cu} size is set to three different sizes, i.e., 16$\times$16, 32$\times$32 and 64$\times$64 pixels. The lower size is targeting the \ac{mb} size used in the \ac{h264}. -Table~\ref{Table3:Quality_BlockSize} shows the average \ac{psnr} obtained for the three test conditions and for four different packet loss ratios. Results for the error-free scenario shows a consistent quality increase as the maximum \ac{cu} size increases, +\begin{table}[!t] + \centering + \caption{Average \ac{psnr} (dB) for different maximum CU size.} + \label{Table3:Quality_BlockSize} + \TABLESVSPACE + \input{./031_ErrorResilienceEvaluation/QualityEvaluation/TableCUSizes.tex} + % \vspace{6mm} +\end{table} + + +Table~\ref{Table3:Quality_BlockSize} shows the average \ac{psnr} (dB) obtained for the three test conditions and for three different packet loss ratios. Results for the error-free scenario shows a consistent quality increase as the maximum \ac{cu} size increases, % up to a maximum of 1.41~dB for the Tennis sequence when comparing the maximum \ac{cu} size of $64\times64$ with $16\times16$. This confirms the principle used by the \ac{hevc} standard, whereby larger coding units lead to higher coding efficiency. -For the Tennis sequence, which has the higher \ac{ti} parameter (see Table~\ref{TableTestSequences}), a maximum of 1.41~dB is achieved. +For the Tennis sequence, which has the highest \ac{ti} parameter (see Table~\ref{TableTestSequences}), a maximum of 1.41~dB is achieved. Moreover, it is also noticeable that sequences with higher amount of inter-coded blocks take less advantage of the larger \acs{cu}. This is noticeable in the results of the BQSquare sequence (i.e., highest ratio of inter-coded blocks and lowest quality gain in the error-free case). -Overall, comparing the results for the error-free and data loss cases, it is noticeable that similar quality gains are obtained,mostly due to the superior coding efficiency of larger \acs{cu}. +Overall, comparing the results for the error-free and data loss cases, it is noticeable that similar quality gains are obtained, mostly due to the superior coding efficiency of larger \acs{cu}. In most cases the quality gains slightly decrease for increasing \acs{plr}. Therefore, one may conclude that the error robustness of \ac{hevc} streams is not affected by limiting the maximum \ac{cu} size. -\begin{table}[!t] - \centering - \caption{Average \ac{psnr} for different maximum CU size.} - \label{Table3:Quality_BlockSize} - \TABLESVSPACE - \input{./031_ErrorResilienceEvaluation/QualityEvaluation/TableCUSizes.tex} -% \vspace{6mm} -\end{table} - \begin{figure}[!t] \centering \renewcommand{\FIGSIZE}{0.38\textwidth} @@ -140,41 +153,26 @@ Therefore, one may conclude that the error robustness of \ac{hevc} streams is no \label{Figure3:BlockPartitionBask} \end{figure} - -Figure~\ref{Figure3:BlockPartitionBask} illustrates an example of the frame partitions for a region of the Basketball Drill sequence for two cases: (i) maximum \ac{cu} size of $64\times64$ and (i) maximum \ac{cu} size of $16\times16$. +Figure~\ref{Figure3:BlockPartitionBask} illustrates an example of the frame partitions for a region of the Basketball Drill sequence for two cases: (i) maximum \ac{cu} size of $64\times64$ and (ii) maximum \ac{cu} size of $16\times16$. As shown, in the moving regions, e.g., moving player, the partitions obtained for both cases have a similar pattern, mainly composed of \acs{cu} with small size. Larger \acs{cu} are only used in the floor, which has a lower spatial complexity and suffers small changes between frames, therefore, easier to compress with $64\times64$ \acs{cu}. Moreover, those regions can be more accurately recovered, by using neighbouring information correctly decoded (spatial and temporal), thus not incurring in severe error propagation. Therefore, the overall error robustness is not affected by reducing the maximum \ac{cu} size. One may conclude that the use of larger coding units is able to improve the overall \ac{hevc} efficiency in error-free and error prone communications. -% \subsection{Impact of filtering operation} -% -% The \ac{hevc} introduced new in-loop filter operations, extending the deblocking filter of the H.264/AVC. One of the filtering operations present in the standard is the \glsfirst{sao}. As mentioned before, the filter is able to reduce the reconstruction distortion by conditionally adding an offset value to each sample after the application of the deblocking filter. In this sub-section the impact of the \ac{sao} is evaluated, by comparing the quality obtained under packet loss with the filter enabled and disabled. -% -% -% \begin{table}[!t] -% \centering -% \caption{Average \ac{psnr} for different filtering configurations.} -% \label{TAB3:QualitySao} -% \input{\HEVCRobustnessEvaluation/TableFilters.tex} -% \end{table} -% -% Table~\ref{TAB3:QualitySao} shows the average decoded quality for the two tested scenario. The case where the filter is enabled is used as reference for comparison and the absolute value of the \ac{psnr} is shown. For the case where the \ac{sao} is disabled the quality differences are illustrated. -% Results confirm the superiority of enabling the the \ac{sao} both in error-free but also in data loss environments. Although only a small quality impact occurs in the case where the filter is disabled is, - \subsection{Evaluation of the impact of intra refreshing} +\label{SEC3:IntraRefreshEvaluation} In this sub-section the relation between the temporal distance of consecutive intra-coded frames, i.e., period of the intra frames, and the video quality under transmission errors is evaluated. As the intra-coded frames have no dependencies to prior coded pictures, they introduce a refreshing point into the bitstream, which stops temporal error propagation along several frames or slices. This allows the decoder to recover from frame/slice loss and continue error-free decoding in a seamless manner. \begin{figure}[!t] - \centering - \input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex} - \caption{Decoded video quality for different intra periods.} - % \caption{Decoded video quality for different intra periods, i.e., distance between intra-frames (Complementary results in Figure~\ref{Appendix:Figure3:QualityvsIntra}).} - \label{Figure3:QualityvsIntra} + \centering + \input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex} + \caption{Decoded video quality for different intra periods.} + % \caption{Decoded video quality for different intra periods, i.e., distance between intra-frames (Complementary results in Figure~\ref{Appendix:Figure3:QualityvsIntra}).} + \label{Figure3:QualityvsIntra} \end{figure} Figure~\ref{Figure3:QualityvsIntra} shows the average decoded video quality (\ac{psnr}) for different distances of intra frames, covering \acs{plr} between 3\% and 5\%. @@ -186,15 +184,17 @@ As spatial complexity increases, the prediction residue energy also increases, w One should also note from the results of Figure~\ref{Figure3:QualityvsIntra} that, using a small distance between intra frames is not always more efficient. Since intra frames require higher amount of bits, the overall coding efficiency is reduced, resulting in lower overall quality in case of data loss, when comparing with other cases. This is clearly noticeable for the Four People sequence, where replacing the intra period from 8 to 16 frames leads to a quality increase of 1 dB for 5\% of \ac{plr}. -For the remaining sequences it can be concluded that using sixteen frames (not higher than half a second) between refresh points (i.e., intra frames) yelds an acceptable efficiency under errors. This provides a good trade-off between compression efficiency and error robustness in case of data loss. +For the remaining sequences it can be concluded that using sixteen frames (not higher than half a second) between refresh points (i.e., intra frames) yields an acceptable efficiency under errors. +This provides a good trade-off between compression efficiency and error robustness in case of data loss. +For this reason for the remaining experimental evaluation an \ac{idr} of 16 frame will be used. + \subsection{Evaluation for different slice partitioning schemes} -The partitioning of the frame into different slices increase the flexibility of video stream packetisation, along packets of different sizes, thus providing adequate adaptation of coded data units to different transport layers and networks. - +The partitioning of a frame into different slices increases the flexibility of video stream packetisation, along packets of different sizes, thus providing adequate adaptation of coded data units to different transport layers and networks. By partitioning the video frame into different slices, spatial refresh points are introduced, limiting the error propagation across different \acs{ctu}. -In this sub-section the amount of bytes per slice is evaluated, in order to study the influence of different slice partitioning schemes on the error robustness of \ac{hevc} streams. -A comparison of the quality obtained for different modes is also carried out, in order to find the most efficient mode for video transmission under data loss conditions. +In this sub-section the effect of the slice size is evaluated, to study the influence of different slice partitioning schemes on the error robustness of \ac{hevc} streams. +A comparison of the quality obtained for different partition modes is also carried out, in order to find the most efficient mode for video transmission under data loss conditions. \begin{figure}[!t] \centering @@ -203,10 +203,10 @@ A comparison of the quality obtained for different modes is also carried out, in \label{Figure3:SlicePartitioning} \end{figure} -Figure~\ref{Figure3:SlicePartitioning} shows the average \ac{psnr} for four difference slice partitioning modes and different \acs{plr}. +Figure~\ref{Figure3:SlicePartitioning} shows the average \ac{psnr} for four different slice partitioning modes and different \acs{plr}. The first three modes are characterised by a different amount of bytes per slice (B/S), namely 1200, 2400 and 4800 bytes. In these cases each frame has a variable number of slices. The last mode uses a single slice per frame (referred to as \textit{SS}), corresponding to the case where slice partitioning is disabled. -The results shown in Figure~\ref{Figure3:SlicePartitioning} reveal a clear impact of the slice partitioning mode on the error robustness of \ac{hevc}, as the quality degradation for increasing \ac{plr} significantly changes for each mode. +The results shown in Figure~\ref{Figure3:SlicePartitioning} reveal a clear impact of the slice partitioning mode on the error robustness of \ac{hevc}, as the quality degradation for increasing \ac{plr} changes significantly for each mode. Comparing the cases with fixed amount of bytes per slice, it is clearly noticeable that a higher amount of bytes also corresponds to a higher quality degradation. This is true for the results of most test sequences shown in Figure~\ref{Figure3:SlicePartitioning}. Although the use of smaller slices increases the overhead required for slice headers, the superior error robustness achieved with 1200 B/S is able to overcome the inherent degradation on coding efficiency. @@ -221,7 +221,7 @@ In this case, the higher coding efficiency achieved by the \textit{SS} case also \subsection{Motion vector coding impact on error propagation} - +\label{Section3:MotionVectorCodingTests} In \ac{hevc}, the new \ac{mv} prediction modes known as \ac{amvp} and Merge Mode~\cite{MotionCodingHEVC2011}, may be significant contributors to the poor performance under transmission errors. These modes are characterised by coding the motion information using previously encoded \acs{mv}, which obviously leads to prediction mismatches at the decoder-side when such neighbouring information is missing due to network losses. Therefore, in this sub-section the impact of using spatial and temporal \ac{mv} candidates on the decoded video quality under errors is evaluated. @@ -232,14 +232,14 @@ Therefore, in this sub-section the impact of using spatial and temporal \ac{mv} \centering \renewcommand{\FIGSIZE}{0.75\textwidth} \input{031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsMergeCand.tex} - \caption{Average quality using different number of \ac{mv} candidates.} + \caption{Average quality using different number of spatial MV candidates.} \label{Figure3:AvgPSNRMergeCandidates} \end{figure} -The first set of experiments aimed to investigate the impact of the amount of spatial \ac{mv} candidates in the video quality. +The first set of experiments aimed to investigate the impact of the amount of spatial \ac{mv} candidates on the video quality. As mentioned before, \ac{hevc} allows up to five candidates to be used, which can be provided solely by spatial neighbours. In order to check the impact of the \ac{mv} candidates, different coding configurations were used, each one characterised by different amounts of spatial \ac{mv} candidates, i.e., 1, 3 and 5 \acs{mv}. Figure~\ref{Figure3:AvgPSNRMergeCandidates} shows the average quality results for the Basketball Drill and People On Street sequences, under different packet loss conditions. -The results show that, besides the number of \ac{mv} candidates, similar quality decreasing is achieved as the \ac{plr} increases. Therefore, one may conclude that varying the number of available \ac{mv} candidate does not affect the error resilience of the \ac{hevc}. +The results show that, for different number of \ac{mv} candidates, similar quality decreasing is achieved as the \ac{plr} increases. Therefore, one may conclude that varying the number of available \ac{mv} candidate does not affect the error resilience of the \ac{hevc}. The occurrence rate for each spatial \ac{mv} candidate was also analysed in order to check whether they are uniformly used. Table~\ref{Table3:MergeIdxRatio} shows the usage percentage of each \ac{mv} candidate when three and five candidates are available for the slice partitioning mode of 1200 B/S. The results indicate that most predictions use the first candidate and minor differences occur by adding 2 or more \acs{mv}. @@ -274,14 +274,14 @@ Moreover, the slice partitioning mode used only 1200 bytes per slice, which intr \renewcommand{\FIGSIZE}{0.76\textwidth} % \includegraphics[width=\FIGSIZE,page=2]{\HEVCRobustnessEvaluation/PSNRvsFrameSingleFrameTMVP} \input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP} - \caption{Error propagation for Kendo sequence when Frame~\#6 is lost for the \ac{ld} configuration.} + \caption{Error propagation for Kendo sequence when Frame~\#6 is lost for the \ac{ld} configuration using different TMVP configurations.} \label{Figure3:PSNRvsFrameErrorFreeConc} \end{figure} The second set of experiments aimed to find the influence of losing temporal motion predictors on the error resilience of \ac{hevc}. In order to perform this test, a single frame loss in enforced, and the error propagation is evaluated for two configurations: (i) with the temporal \ac{mv} candidate enabled and (ii) disabled. -Figure~\ref{Figure3:PSNRvsFrameErrorFreeConc} illustrates the error propagation, when a single frame is lost. For this particular experiment an intra-refreshing period of 32 frames was used to evaluate the impact on long error propagation periods. +Figure~\ref{Figure3:PSNRvsFrameErrorFreeConc} illustrates the error propagation, when a single frame is lost. For this particular experiment an intra-refreshing period of 32 frames was used to evaluate the impact on long error propagation periods. This is commonly used in quality evaluation of erroneous video decoding and follows the reference conditions~\cite{HEVCCommonTests2012}. The results show that when the \ac{tmvp} mode is enabled, the loss of Frame \#6 leads to significant reduction in the reconstruction quality of the subsequent frames. On the contrary, when the temporal \ac{mv} candidate is disabled, the motion information on subsequent frames is not affected by the loss of a single frame and significantly higher quality is obtained. This is the effect of breaking the temporal dependencies between \acs{mv}, which prevents erroneous \acs{mv} to propagate throughout the set of dependent frames. @@ -289,7 +289,7 @@ This is the effect of breaking the temporal dependencies between \acs{mv}, which \begin{table}[!t] \centering %\caption{Average \ac{psnr} using different motion vector candidates.} - \caption{\ac{psnr} under random packet loss.} \label{Table3:AvgPSNRWithoutTMVP} + \caption{Average PSNR (dB) under random losses for different TMVP configurations.} \label{Table3:AvgPSNRWithoutTMVP} \TABLESVSPACE \input{./031_ErrorResilienceEvaluation/QualityEvaluation/TableMVPs.tex} \end{table} @@ -332,7 +332,7 @@ Based on this evidence, this section proposes a method to make intelligent use o \subsection{Proposed method} -The new approach to deal with temporal \ac{mv} dependencies is accomplished by modifying the \ac{rd} optimisation that is used to select the best \ac{mv} for inter-prediction. +The new approach to dealing with temporal \ac{mv} dependencies is accomplished by modifying the \ac{rd} optimisation that is used to select the best \ac{mv} for inter-prediction. The objective is to increase the number of \acs{mv} that can be independently decoded, in order to improve the error recovery capability after an arbitrary frame loss event. The proposed method also overcomes the limitation of the method presented in \cite{DynamicTMVP2011}, which breaks the motion information dependencies between frames. @@ -349,7 +349,7 @@ In the proposed approach, a temporal \ac{mv} candidate is marked as unsuitable, \centering % \includegraphics[width=0.6\columnwidth]{\DynamicTMVP/DynamicTMVP} \input{./031_ErrorResilienceEvaluation/DynamicTmvp/DynamicTMVP.tikz} - \caption{Proposed approach to constrain the temporal \ac{mv} dependencies.} + \caption{Proposed approach to constrain the temporal \ac{mv} dependencies (arrow point to the MV predictor).} \label{ProposedDynamicMethod} \FIGSVSPACE \end{figure} @@ -367,7 +367,7 @@ Therefore, coding efficiency is not significantly penalised. \begin{figure}[tb] \centering \input{031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingle.tex} - \caption{Quality obtained with the different approaches tested in the \ac{hevc} when the frame \#5 is missing for Kendo and BQSquare sequences.} + \caption{Quality obtained with the proposed scheme, \TIDR and reference HEVC when the frame \#5 is missing for Kendo and BQSquare sequences.} \label{Figure3:ProposedMethodEvaluationSingle} \end{figure} @@ -389,7 +389,7 @@ While the \TIDR method is only able to recover from the frame loss after the ref \begin{table}[tbp] \centering - \caption{Average standard deviation of the decoded video quality (PSNR).} \label{Table3:AverageStdPSNR} + \caption{Average standard deviation of the PSNR (dB) for the proposed scheme, \TIDR and reference HEVC.} \label{Table3:AverageStdPSNR} \TABLESVSPACE \input{./031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNRSTD.tex} \end{table} @@ -398,7 +398,7 @@ Secondly, further simulations were performed to evaluate the effectiveness of th %As in previous experiments, each coded frame is transmitted in one packet. Table~\ref{Table3:AverageStdPSNR} illustrates the average standard deviation of the PSNR, for three of the test sequences. The results show that increasing the \ac{plr} leads to higher quality variations, thus the impact of errors in the subjective quality experience is higher. -However, reducing the temporal dependencies of the \acs{mv} using the proposed method decreases such quality variations, achieving the lowest value for all tested scenarios. +However, reducing the temporal dependencies of the \acs{mv} using the proposed method decreases such quality variations, achieving the lowest variation value for all tested scenarios. The proposed method is able to reduce the quality variations up to 0.94 dB for Kendo sequence (3$\%$ of packet loss) compared to the~\TIDR~method. Therefore, since the proposed method achieves the lowest \ac{psnr} standard deviation, one may conclude that it is able to reduce the video quality variations leading to a better visual quality. @@ -406,7 +406,7 @@ Therefore, since the proposed method achieves the lowest \ac{psnr} standard dev \begin{table}[tbp] \centering -\caption{BD-PSNR (dB) differences for all sequences tested at different frame loss conditions.} \label{Table3:AverageResultsProposed} +\caption{BD-PSNR (dB) differences for the proposed method and \TIDR at different frame loss conditions.} \label{Table3:AverageResultsProposed} \input{031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNR.tex} \end{table} diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex index ec38a27..5af9396 100644 --- a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex @@ -6,7 +6,7 @@ load 'carreira_phd_thesis_gnuplot_template.cfg' set multiplot layout 3,2 #set multiplot layout 3,1 -set xlabel '\TitleFont Distance between intra-frames' +set xlabel '\TitleFont Distance between intra-frames (frames)' set xrange[8:128] set xtics 8,24,128 set key width 0 diff --git a/carreira_phd_thesis.sty b/carreira_phd_thesis.sty index ddab783..bf4a49c 100644 --- a/carreira_phd_thesis.sty +++ b/carreira_phd_thesis.sty @@ -107,7 +107,8 @@ \usepackage{mdwmath,amsmath} \usepackage{array,mdwmath,amsmath,commath} \newcommand*\mean[1]{\bar{#1}} - +\newcommand{\argmax}{\operatornamewithlimits{arg\ max}} +\newcommand{\maxop}{\operatornamewithlimits{max}} % *** PAGE ROTATION PACKAGES *** \usepackage{pdflscape}