Files
doc_phd_thesis/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex
T
2018-07-06 17:56:46 +01:00

375 lines
36 KiB
TeX

% Figure folders
\newcommand{\HEVCRobustnessEvaluation}{./031_ErrorResilienceEvaluation/QualityEvaluation}
\newcommand{\DynamicTMVP}{031_ErrorResilienceEvaluation/DynamicTmvp}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CHAPTER
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{A Method to Increase the Error Robustness of HEVC}
\label{Chapter:HEVCErrorResilience}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
The \ac{hevc} standard is expected be used by the same range of applications as its predecessors (\emph{e.g.}, H.264/AVC), such as, broadcasting, internet streaming, two-way communications, and storage~\cite{HEVCLayerIntegrationDez2012}.
More advanced prediction structures, as presented in Section~\ref{Chapter:HEVCReview}, allow bitrate savings, but also to some disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and reduced error robustness. Therefore, it is important to evaluate the error robustness of the current standard in order to check for its weaknesses. This can provide relevant insights on future directions on how to improve the standard robustness to transmission losses.
The results provided by this experimental study allowed to improve the \ac{hevc} encoder in order to avoid ...
In this chapter an error resilience evaluation of the \ac{hevc} is presented in Section~\ref{Section3:ErrorResilienceEvaluation}. The evaluation covers different coding configuration to evaluate the individual impact of some of the new coding features introduced, e.g., increased partition size, intra-refreshing, slice partitioning and advanced motion vector prediction. In Section~\ref{Section3:DynamicMVRefreshHEVC} a novel method to increase the error robustness of the \ac{hevc} \ac{mv} coding is presented and evaluated against existing techniques. Finally, Section~\ref{Section3:Summary} concludes this chapter.
\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% SECTION
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Evaluation study of the error robustness of HEVC}
\label{Section3:ErrorResilienceEvaluation}
\subsection{Motivation and objectives}
As mentioned before the \ac{hevc} standard provide improvements in terms of coding efficiency, however at the cost of more advanced predictions structures, which increases complexity and bitstream dependencies.
Although the complexity may not pose significant problems due to the rapid evolution of hardware technologies, reduced error robustness strongly affects quality under packet loss conditions.
A critical aspect related to the performance of robust video streaming methods is the error detection accuracy.
Since \ac{hevc} streaming is expected to use known transport technologies, such as RTP~\cite{HEVCStreamMay2012}, MPEG-2 TS~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016} and MPEG Media Transport\cite{ATSCMMT2016}, their error detection capabilities have significant impact on the overall system performance.
In general, the error detection operation at the transport layers is decoupled from the video coding layer.
The encoding side implements the robust coding methods, while the efficient recovery of lost video data is implemented in the decoder.
Between them, an inter-layer signalling ensures that the decoder is given the necessary information to identify the lost slices/frames in the video data. Therefore, stream robustness can be investigated on its own, assuming any possible form of error detection and streaming technologies. This is the main focus of the study presented in this chapter.
The are some studies presented in the past regarding the error robustness of the \ac{hevc} standard which are worth to be mention as they provide relevant insights for the research of carried out in this research work. In~\cite{StudyHEVCErrorsDez2012} the both the coding efficiency and error robustness of the \ac{hevc} are evaluated, and compared against the \ac{h264}. Experimental results show that \ac{hevc} has reduced error robustness despite its increased coding efficiency. A subjective analysis is presented in~\cite{SubjectiveEvalSep2013,QoEHEVCMay2014} revealing that packet loss ratios higher than $3\%$ have a significant effect on the perceived video quality. In both works it is concluded that the robustness of \ac{hevc} decreases for high motion sequences. Moreover, in~\cite{DynamicTMVP2011} a the vulnerability of \ac{mv} prediction is presented and its impact of the error robustness is evaluated, showing its weak error resilience in contrast with the coding improvements. Summarising, the decreasing in error robustness is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes of \ac{hevc}, which are selected according to a rate distortion optimisation criteria, assuming error-free transmission\cite{ROPE2007}.
This section presents an error robustness evaluation of the \ac{hevc} standard. This study focus on the novel techniques introduced extending the generic studies presented in previous works mentioned above. This study may provide relevant insights for future directions where the error robustness problem should be tackled in the \ac{hevc} standard.
The following cases studies are covered in this experimental evaluation:
\begin{enumerate}
\item different maximum \ac{cu} sizes to evaluate the impact of higher larger frame partitioning;
% \item experiments with and without filter operations, i.e., deblocking filter and \ac{sao}, in order to evaluate their impact on decoding video quality;
\item results for difference distances between intra frames, i.e., evaluate the impact of introducing different amount of refresh points to the overall quality;
\item different slice partitioning schemes by varying the number of bytes per slice;
\item different \ac{mv} coding schemes using different types of \ac{mv} candidates, with an analysis at the spatial and temporal level.
\end{enumerate}
\subsection{Characterisation of the experiments}
\input{031_ErrorResilienceEvaluation/TablesSequences.tex}
The experimental setup used in this work to evaluate the error robustness of the \ac{hevc} standard used thirteen test sequences, which were chosen to cover different types of motion and texture complexity.
% Each sequence comprises 240 frames and the whole set covers.
Table~\ref{TableTestSequences} presents a summary of the main characteristics of the video sequences. Appendix~\ref{Appen:Sequences} shows a list of the first frame of each test sequence.
In order to evaluation the spatial and temporal complexity of the video sequences the following metrics \cite{ITUTP9112008} were used:
\begin{itemize}
\item {\ac{si})} based on Sobel filter. Each video frame ($f_t$) is firstly filtered, then the standard deviation is computed. Finally, the maximum value over time is chosen to represent the \ac{si} metric. This process can be represented in the following equation:
\begin{equation}
SI = max_{time}\Big\{ std_{space} \big( Sobel\left(f_t\right) \big) \Big\}.
\end{equation}
\item {\ac{ti}} based upon the motion difference feature, which is the difference between the pixel values at the same location in space but in consecutive time instances. \ac{ti} is computed as the maximum over time of the standard deviation over space. More motion in adjacent frames will result in higher values of \ac{ti}. This process can be represented by the following:
\begin{equation}
TI = max_{time}\Big\{ std_{space} \big( f_{t}(x,y) - f_{t-1}(x,y) \big) \Big\}.
\end{equation}
\end{itemize}
The values of the metrics described above are represented in the Table~\ref{TableTestSequences} (see second column).
The experimental results for this study were obtained by using the HM reference software, version $16.2$.
The sequences were encoded with an IDR period of $16$ frames and all the coding modes enabled, following two recommended configurations: Low-Delay (P-frames with 4 references frames) and Random-Access (B-frames with 2 reference frames in each list)~\cite{HEVCCommonTests2012}.
In order to achieve high quality images for all coded sequences (i.e., \ac{psnr} around 40 dB) different bitrates were used.
%The following bitrates were selected to achieve high quality (i.e., around 40 dB): 1.0 Mbits/s (Kendo), 1.5 Mbits/s (Book Arrival), 4.0 Mbits/s (BQSquare, Park Scene and Race Horses) and 12.0 Mbits/s (People on Street).
% At the decoder side, error concealment is based on motion compensation from previously decoded frames.
% The motion field of a missing frame is obtained applying \ac{mv} extrapolation from the two closest neighbours.
% Afterwards, the final \acs{mv} are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}.
To simulate the lossy transmission environment, each frame is encoded into several slices, each one packetized into one \ac{nal} unit, which are then transmitted as the payload of independent packets.
In our experiments a whole packet is lost whenever an error occurs, originating a lost slice.
Random packet loss was simulated using a two-state Markov model for \ac{plr} of 1\%, 3\%, 5\%, 10\%.
For each test condition, 50 runs were performed and the average \ac{psnr} was measured.
The filtering operations were disabled at slices boundaries in order to keep the slices self-contained. The remaining configurations have been kept at their default values as given in the common test conditions~\cite{HEVCCommonTests2012}.
%\subsection{Experimental results}
\subsection{Evaluation of the impact of the partition units size}
The \ac{hevc} standard introduced new partitioning schemes to allow coding improvements in high resolution videos. These partition structures, named \glsfirst{cu}, are allowed to be as large as $64\times64$ pixels. In the first set of experiments the impact of the maximum \ac{cu} size in the decoded video quality is investigated. In this experiment the maximum \ac{cu} size is set to three different sizes, i.e., 16$\times$16, 32$\times$32 and 64$\times$64. The lower size is targeting the \ac{mb} size used in the H.264/AVC.
\begin{table}[!t]
\centering
\caption{Average \ac{psnr} for different maximum CU size.}
\label{Table3:Quality_BlockSize}
\TABLESVSPACE
\input{./031_ErrorResilienceEvaluation/QualityEvaluation/TableCUSizes.tex}
\vspace{2mm}
\end{table}
Table~\ref{Table3:Quality_BlockSize} shows the average \ac{psnr} obtained for the three condition tested, for error-free and 4 different packet loss ratios. Results for the error-free scenario shows a consistent quality increasing as the maximum \ac{cu} size increases, to a maximum of 1.41 dB for the Tennis sequence when comparing the maximum \ac{cu} size of $64\times64$ with $16\times16$. This confirms the principle used by the \ac{hevc} standard, that larger coding units lead to higher coding efficiency. In case of data loss, results indicate that larger coding units should be maintained. Even though larger frame partitions lead to larger coding dependencies, it will not affect the robustness to packet loss and error propagation. This is mainly due to the fact that error propagation occurs mostly in complex and moving regions, which normally are coded using smaller partition units.
\begin{figure}[t]
\centering
% \small
\renewcommand{\FIGSIZE}{0.35\textwidth}
% \resizebox{\columnwidth}{!}{
\begin{tabular}{y{.45\columnwidth}y{.45\columnwidth}}
\includegraphics[width=\FIGSIZE]{\HEVCRobustnessEvaluation/Bask_Partitions_16.png} &
\includegraphics[width=\FIGSIZE]{\HEVCRobustnessEvaluation/Bask_Partitions_64.png} \\
(a) $16\times16$ & (b) $64\times64$ \\
\end{tabular}
% }
\caption{Illustration of the block partition and prediction mode for a region of the Basketball Drill sequence.}
\label{Figure3:BlockPartitionBask}
\end{figure}
Figure~\ref{Figure3:BlockPartitionBask} illustrates an example of the frame partitions for a region of the Basketball Drill sequence for two cases: (i) maximum \ac{cu} size of $64\times64$ and (i) maximum \ac{cu} size of $16\times16$. As shown, in the moving regions, e.g., moving player, the partitions obtained for both cases have a similar pattern, mainly composed of \acs{cu} with small size. The differences in the size of the \acs{cu} are only noticeable in the background, which is less prone to errors. Therefore, the overall error robustness is not affected by reducing the maximum \ac{cu} size. One may conclude that the use of larger coding units is able to improve the overall \ac{hevc} efficiency in error-free and error prone applications.
% \subsection{Impact of filtering operation}
%
% The \ac{hevc} introduced new in-loop filter operations, extending the deblocking filter of the H.264/AVC. One of the filtering operations present in the standard is the \glsfirst{sao}. As mentioned before, the filter is able to reduce the reconstruction distortion by conditionally adding an offset value to each sample after the application of the deblocking filter. In this sub-section the impact of the \ac{sao} is evaluated, by comparing the quality obtained under packet loss with the filter enabled and disabled.
%
%
% \begin{table}[!t]
% \centering
% \caption{Average \ac{psnr} for different filtering configurations.}
% \label{TAB3:QualitySao}
% \input{\HEVCRobustnessEvaluation/TableFilters.tex}
% \end{table}
%
% Table~\ref{TAB3:QualitySao} shows the average decoded quality for the two tested scenario. The case where the filter is enabled is used as reference for comparison and the absolute value of the \ac{psnr} is shown. For the case where the \ac{sao} is disabled the quality differences are illustrated.
% Results confirm the superiority of enabling the the \ac{sao} both in error-free but also in data loss environments. Although only a small quality impact occurs in the case where the filter is disabled is,
\subsection{Evaluation of the impact of intra refreshing}
\begin{figure}[!t]
\centering
% \renewcommand{\FIGSIZE}{0.49\textwidth}
% \includegraphics[width=\FIGSIZE,page=1]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot}
% \includegraphics[width=\FIGSIZE,page=2]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot}
% \includegraphics[width=\FIGSIZE,page=3]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot}
% \includegraphics[width=\FIGSIZE,page=4]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot}
% \includegraphics[width=\FIGSIZE,page=5]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot}
% \includegraphics[width=\FIGSIZE,page=6]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot}
% \includegraphics[width=\FIGSIZE,page=7]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot}
% \includegraphics[width=\FIGSIZE,page=8]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot}
% \includegraphics[width=\FIGSIZE,page=9]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot}
% \includegraphics[width=\FIGSIZE,page=10]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot}
% \includegraphics[width=\FIGSIZE,page=11]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot}
% \includegraphics[width=\FIGSIZE,page=12]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot}
\input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex.bak}
\caption{Decoded video quality for different intra periods, i.e., distance between intra-frames.}
% \caption{Decoded video quality for different intra periods, i.e., distance between intra-frames (Complementary results in Figure~\ref{Appendix:Figure3:QualityvsIntra}).}
\label{Figure3:QualityvsIntra}
\end{figure}
In this section the relation between the distance of subsequent intra-coded frames, i.e., period of the intra frames, and the quality under errors is evaluated. As the intra-coded frames have no dependencies to prior coded pictures, they introduce a refresh point into the bitstream. This allows the decoded to recover from slice loss and continue a error-free decoding.
Results on Figure~\ref{Figure3:QualityvsIntra} shows the average decoded video quality for different distance of intra frames. The figure covers packet loss ratios between 3\% and 5\%. Results indicate that has the distance between intra frames increases the decoded video quality significantly decreases. This is especially noticeable for an intra period higher than 32 frames, indicating that it is not suitable for video streaming under error-prone conditions.
One should also be noted from the results of Figure~\ref{Figure3:QualityvsIntra} that using a small distance between intra frames is not always more efficient. Since intra frames require higher amount of bytes, the overall coding efficiency is reduced, resulting in an overall quality decreasing even with a 5\% f \ac{plr}. This is clear noticeable for the Four People sequence. From the results for Park Scene sequence (and Cactus sequence in Figure~\ref{Appendix:Figure3:QualityvsIntra}) it can be concluded that using sixteen frames between refresh points (i.e., intra frames) achieves an acceptable efficiency under errors, without compromising the coding efficiency.
\subsection{Evaluation for using different slice partitioning schemes}
The partitioning of the \acs{ctu} into different slices introduce flexibility into video bitstream, by introducing spatial refresh points and avoiding the error propagation across different \acs{ctu}. In this section the amount of bytes per slices is evaluated, since it the most efficient partitioning scheme in the \ac{hevc} standard that suits the network requirements of fixed packaging. As a reference for comparison a single slice per frame is also evaluated.
\begin{figure}[!t]
\centering
\input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsSlices.tex.bak}
\caption{Decoded video quality for different slice partitioning schemes.}
%\input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsSlices.tex}
%\caption{Decoded video quality for different slice partitioning schemes (Complementary results in Figure~\ref{Appendix:Figure3:SlicePartitioning}).}
\label{Figure3:SlicePartitioning}
\end{figure}
Figure~\ref{Figure3:SlicePartitioning} shows the average \ac{psnr} for three difference slice partitioning schemes. Results reveals a clear impact of the slice partitioning on the error robustness of the \ac{hevc}, as the quality degradation as the \ac{plr} increases varies for each scheme.
Firstly it is clearly notice the disadvantage of using a single slice per frame as it leads to the lower quality when comparing the test cases studied. This is due to the lack of refresh points within a frame, resulting in loss events that affects the whole frame which will propagate to the subsequent inter predicted frames.
Secondly, comparing slices of different sizes, i.e., comparing the use of 1200 and 2400 bytes per slice, reveals that using small slices leads to higher quality which is able to overcome the inherent degradation on coding efficiency due to higher amount of slice headers.
\subsection{Motion vector coding impact on error propagation}
\begin{figure}[!t]
\centering
\renewcommand{\FIGSIZE}{0.75\textwidth}
% \includegraphics[width=\FIGSIZE,page=1]{\HEVCRobustnessEvaluation/PSNRvsLossNoTMVPMergeSpatial_Plot}
% \includegraphics[width=\FIGSIZE,page=9]{\HEVCRobustnessEvaluation/PSNRvsLossNoTMVPMergeSpatial_Plot}
\input{031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsMergeCand.tex}
\caption{Average quality using different number of \ac{mv} candidates.}
\label{Figure3:AvgPSNRMergeCandidates}
\end{figure}
In \ac{hevc}, new motion vector prediction modes are introduced, such as the \ac{amvp} and Merge Mode~\cite{MotionCodingHEVC2011}. In this section the the error robustness of the Merge Mode is evaluated, which predicts \acs{mv} for the current block from its neighbours, which obviously leads to prediction mismatches at the decoder side in case of packet loss.
\subsubsection{Spatial \ac{mv} candidates}
The first set of experiments aimed to evaluate the impact of using several \ac{mv} candidates in the Merge Mode. As mentioned before, \ac{hevc} allows up to five candidates to be used, which may increase the amount of spatial dependencies.
In order to check the impact of the \ac{mv} candidates, different coding configurations were used, by reducing the number of available candidates. Figure~\ref{Figure3:AvgPSNRMergeCandidates} shows the average quality results for the Basketball Drill sequence under different packet loss conditions. The results show that, besides the amount of \ac{mv} candidates, the error resilience of the \ac{hevc} is not compromised.
\begin{table}[t]
\newcommand{\somespaces}{}
\newcommand{\colwidth}{1.8cm}
\centering
\caption{Relation between the number of available \ac{mv} candidates and its usage} \label{Table3:MergeIdxRatio}
\footnotesize
\TABLESVSPACE
\begin{tabular}{x{3.3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}}
\toprule
\multirow{2}{*}{\vbox{\centering \textbf{Max. number of candidates}}} & \multicolumn{5}{c}{\textbf{Candidate selected (ratio)}} \\
\cline{2-6}
& 1 & 2 & 3 & 4 & 5 \\
\hline
3 \acs{mv} & $66.17\%$ & 22.79$\%$ & 11.04$\%$ & - & - \\
\hline
5 \acs{mv} & 64.54$\%$ & 22.24$\%$ & 7.38$\%$ & 3.37$\%$ & 2.47$\%$ \\
\bottomrule
\end{tabular}
\end{table}
The occurrence rate for each spatial \ac{mv} candidate was analysed in order to check if they are uniformly used. Table~\ref{Table3:MergeIdxRatio} shows the use of each \ac{mv} candidate when three and five candidates are available. The results indicate that most predictions use the first candidate and minor differences occur by adding 2 or more \acs{mv}.
Since the mostly used \acs{mv} are the initial ones, changing the number of available candidates does not significantly affects the error resilience.
Moreover, the slice partitioning scheme used (\emph{i.e.}, 1200 bytes per slice) adds more refresh points in each frame reducing the spatial error propagation due to mismatch \ac{mv} predictions.
% \HERE
\subsubsection{Temporal \ac{mv} candidates}
\begin{figure}[!t]
\centering
\renewcommand{\FIGSIZE}{0.76\textwidth}
% \includegraphics[width=\FIGSIZE,page=2]{\HEVCRobustnessEvaluation/PSNRvsFrameSingleFrameTMVP}
\input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP}
\caption{Error propagation for Kendo sequence when Frame~\#6 is lost for the Low-Delay configuration.}
\label{Figure3:PSNRvsFrameErrorFreeConc}
\end{figure}
The second set of experiments aimed to find the influence of losing temporal motion predictors on the error resilience of the \ac{hevc} standard.
In order to perform this test, a single frame is lost, and the error propagation is evaluated for two configurations: with the temporal \ac{mv} candidate enabled and disabled.
Figure~\ref{Figure3:PSNRvsFrameErrorFreeConc} illustrates the error propagation, when a single frame is lost. For this particular experiment a intra-refreshing period of 32 frames was used to evaluate a long error propagation period.
The results show that when the \ac{tmvp} mode is enabled, the loss of Frame \#6 leads to significant reduction in the reconstruction quality of the subsequent frames.
% This is true even for sequences with low motion activity as per Table~\ref{tab:sequences}.
On the contrary, when the temporal \ac{mv} candidate is disabled, motion information on subsequent frames is not affected by the loss of a single frame, since it is not temporal dependent, achieving significant quality increasing. This is more noticeable in the Low-Delay configuration since more error propagation occurs in P-frames and it is propagated for several frames until an intra frame is decoded.
\begin{table}[!t]
\centering
%\caption{Average \ac{psnr} using different motion vector candidates.}
\caption{\ac{psnr} under random packet loss.} \label{Table3:AvgPSNRWithoutTMVP}
\TABLESVSPACE
\input{./031_ErrorResilienceEvaluation/QualityEvaluation/TableMVPs.tex}
\end{table}
In order to further evaluate the impact of the temporal \ac{mv} dependencies, the \ac{hevc} streams were subject to random packet losses.
Table~\ref{Table3:AvgPSNRWithoutTMVP} presents the average quality results, measured using the \ac{psnr}.
As expected, the results indicate that in error free environment, \ac{hevc} achieves higher quality by enabling the \ac{tmvp} candidate, since it presents higher compression efficiency due to more \ac{mv} predictors in the Merge Mode. This is true for all common test conditions, \emph{i.e.}, both Low Delay and Random Access.
However, in the presence of errors, the quality significantly decreases due to low error robustness when temporal \ac{mv} candidates are used, resulting in an average of 2.65 dB of \ac{psnr} increasing when temporal \ac{mv} predictor is disabled across all configuration tested.
The results also show that the negative impact of using temporal \ac{mv} dependencies decreases as the number of B-frames increases. The average gain of disabling the temporal \ac{mv} predictor is 4.51 dB (Low Delay) and 0.78 dB (Random Access).
This also confirms the previous results of Figure~\ref{Figure3:PSNRvsFrameErrorFreeConc}.
A higher negative impact is obtained when only P-frames are used, since subsequent frames are temporally closer, leading to more accurate temporal \ac{mv} candidates.
Moreover, the results also indicate that for higher loss ratios the negative impact of using the \ac{tmvp} does not increase as both configuration are subject to high quality degradation.
\subsection{Discussion and conclusions}
The results presented in the previous sub-sections show that \ac{hevc} has poor robustness against transmission errors, resulting in significant quality degradation, though dependent on the video content and coding options. As shown coding options as slice partition and intra refreshing period have high correlation with the decoded video quality under errors, which shall be taken into consideration when designing a broadcast application.
One of the main improvements of the \ac{hevc} standard is related with the high coding units which does not have an impact of its error robustness, so shall be configured to achieve the highest coding performance. Another improvement in the standard was the Merge Mode, which significantly improves the flexibility in coding and signalling the motion information. One the one hand the spatial \ac{mv} candidates do not significantly decrease the error robustness. On the other hand, the introduction of temporal \ac{mv} candidates significantly contribute to increase the temporal dependencies, leading to more severe quality degradation in the presence of slice loss.
Based on these evidences, the next section presents an efficient method to improve the error robustness of \ac{hevc} based on the underlying principle of reducing the mismatch in \ac{mv} predictions caused by temporal dependencies.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% SECTION
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Dynamic motion vector refreshing}
\label{Section3:DynamicMVRefreshHEVC}
%\subsection{Problem formulation}
The aforementioned results revealed a lack of error resilience in the \ac{hevc} standard motion prediction. Based on this evidence, this section proposes a method to make intelligent use of temporal \ac{mv} candidates during the motion estimation process, in order to decrease the temporal dependency, and improve the error resiliency without penalising the \ac{rd} performance.
\subsection{Proposed method}
In this section a new approach to deal with temporal motion vector dependencies is proposed, aiming at reducing the amount of temporally dependent \acs{mv}. This is accomplished by modifying the rate distortion optimisation process in \ac{hevc}. Since the number of \acs{mv} that can be independently decoded increases, the error recovery percentage is improved after a frame loss event.
In order to overcome in the limitation of the method presented in \cite{DynamicTMVP2011}, a new method that breaks the motion information dependencies between frames is proposed, which is used in the Merge Mode. Instead of disabling the temporal motion candidate at the frame level, the proposed method aims to remove the dependencies at the block (CU) level. To achieve this, the temporal dependency of all \ac{mv} candidates is analysed for each block, in order to decide on the suitable candidates to be used to predict the current \ac{mv}. Based on the analysis of the temporal dependency, some temporal candidates are marked as unsuitable for the prediction of the current \ac{mv}, and are not selected in the merge modes. In the proposed approach, a temporal \ac{mv} candidate is marked as unsuitable based on the following criteria:
\begin{enumerate}
\item if it is encoded based on another temporal \ac{mv} candidate from a previous encoded frame;
\item if it is predicted using a spatial neighbour that was previously encoded using a temporal \ac{mv} candidate.
\end{enumerate}
\begin{figure}[t]
\centering
% \includegraphics[width=0.6\columnwidth]{\DynamicTMVP/DynamicTMVP}
\input{./031_ErrorResilienceEvaluation/DynamicTmvp/DynamicTMVP.tikz}
\caption{Proposed approach to use the temporal motion vector candidate in the \ac{hevc} encoder.}
\label{ProposedDynamicMethod}
\FIGSVSPACE
\end{figure}
Figure \ref{ProposedDynamicMethod} can be used to explain the proposed method, illustrating different blocks and different \ac{mv} dependencies represented with arrows. Note that the squares illustrate a generic block size; the representation of the temporal dependencies is simplified, and in some cases may not correspond to the exact co-located block in the previous frame. In frame $f_{t-1}$ there are several \acs{mv} encoded using spatial predictions and one \ac{mv} encoded using a temporal prediction (arrow pointing to $f_{t-2}$). In frame $f_{t}$, the \ac{mv} corresponding to the block (1) may use the temporal \ac{mv} candidate, since the co-located block in $f_{t-1}$ is not temporally dependent. However, the \ac{mv} prediction marked with the arrow (2) is not allowed in the proposed scheme to avoid propagation from temporal dependencies. Moreover, the block corresponding to the arrow (3) cannot use the temporal \ac{mv} candidate since it already depends on a \ac{mv} that was previously encoded using the temporal candidate. The proposed method reduces the temporal dependencies of the \glspl{mv} at the block level, providing improved error robustness to the encoded streams. Using this approach, the dependencies are selectively removed, improving the error recovery, without fully disabling the temporal \gls{mv} candidates for a given frame.
\subsection{Performance evaluation}
\label{sec:ProposedMethodEvaluation}
\begin{figure}[tb]
\centering
\input{031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingle.tex}
\caption{Quality obtained with the different approaches tested in the \ac{hevc} when the frame \#5 is missing for Kendo and BQSquare sequences.}
\label{Figure3:ProposedMethodEvaluationSingle}
\end{figure}
In this section the performance of the proposed scheme (\PROP) is evaluated against two other approaches: reference \ac{hevc} with TMVP enabled and the \TIDR method as proposed in \cite{DynamicTMVP2011}. To evaluate the proposed method the same experimental setup as presented in Section \ref{Section3:ErrorResilienceEvaluation} was used.
\subsubsection{Error propagation under single loss events}
Firstly, the performance of the proposed method was measured under a single loss event. Figure \ref{Figure3:ProposedMethodEvaluationSingle} shows the error propagation for various tested methods. The results show a significant quality difference between the proposed method, the \TIDR method, and the reference \ac{hevc}.
When the temporal \ac{mv} candidate is disabled (proposed and \TIDR method) more error robustness can be achieved.
The results also show the relevance of using the proposed method, as the \TIDR is only able to recover from the frame loss after the frame 8 (\emph{i.e.}, the refresh frame).
\subsubsection{Quality evaluation under random errors}
\begin{table}[tbp]
\centering
\caption{Average standard deviation of the decoded video quality (PSNR).} \label{Table3:AverageStdPSNR}
\TABLESVSPACE
\input{./031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNRSTD.tex}
\end{table}
Secondly, further tests were performed to evaluate the effectiveness of the proposed error resiliency method under realistic packet loss patterns. As in previous experiments, each coded frame is transmitted in one packet.
Table~\ref{Table3:AverageStdPSNR} illustrates the average standard deviation of the PSNR, for three of the sequences tested. The results show that increasing the packet loss rate leads to higher quality deviations, and the impact of errors in the subjective quality experience is higher. However, reducing the temporal dependencies of the \acs{mv} using the proposed method decreases the quality differences variation, achieving the lowest value for all scenarios tested. The proposed method is able to reduce the quality difference variations up to 0.94 dB in Kendo sequence (3$\%$ of packet loss) compared to the~\TIDR~method.
\begin{table}[tbp]
\centering
\caption{Bjontegaard's average \ac{psnr} (dB) differences for all sequences tested at different frame loss conditions.} \label{Table3:AverageResultsProposed}
\input{031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNR.tex}
\end{table}
Table~\ref{Table3:AverageResultsProposed} shows the Bjontegaard's average \ac{psnr} increase \cite{BJResults2001}. The results are obtained taking as reference the standard \ac{hevc} encoder.
% The results are obtained taking as reference the standard \ac{hevc} encoder, in order to evaluate the quality difference when temporal \ac{mv} candidates are disabled in both error free and under error-prone conditions.
The results presented in Table~\ref{Table3:AverageResultsProposed} show that the proposed method presents practically the same rate-distortion performance when compared to \TIDR method.
In general, the average quality, in an error-free scenario, is only decreased by 0.028 dB in comparison to the \TIDR method and by 0.034 dB in comparison to the reference encoder.
However, the proposed method widely outperforms the reference \TIDR technique~\cite{DynamicTMVP2011} in terms of error resiliency performance, achieving in average higher overall quality under error-prone conditions.
The effectiveness of the proposed method increases at higher packet loss rates (\emph{e.g.},~$5\%$), achieving an average gain of 0.573 dB, and a maximum gain of 1 dB (PSNR) for the Book Arrival sequence, when compared to the \TIDR technique. Higher gains are achieved when compared with the reference \ac{hevc} encoder, as shown in Table~\ref{Table3:AverageResultsProposed}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% SECTION
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Summary}
\label{Section3:Summary}
In this chapter, the error resilience performance of the \ac{hevc} standard was analysed under error prone conditions. Results revealed that new coding tool introduced in \ac{hevc} lead to higher error propagation. This reveals that new techniques should be employed within \ac{hevc} encoder and decoder to increase its resilience to errors, achieving higher overall performance under broadcast environments.
Therefore, a new approach to deal with a drawback in \ac{mv} coding was proposed, addressing \ac{mv} dependencies at the block level. The proposed method selectively disables the temporal \ac{mv} candidates, in order to break the dependencies across more than one frame. The experimental evaluation shows that the proposed method increases error robustness and outperforms the existing state-of-the-art method and the standard. The results also show that the proposed method is able to provide a good trade-off between the coding performance and error resilience. The results demonstrate that selective usage of temporal \ac{mv} candidates is an efficient method to enhance error resilience in \ac{hevc}.