jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Files
35de982fc20ae535e6c8ca1db0871188878ea5ea
doc_phd_thesis/041_ImprovedErrorResilience/ImprovedErrorResilience.tex
T
jfmcarreira deffaca3fc Version 0.1
2018-07-06 17:56:46 +01:00

455 lines
45 KiB
TeX
Raw Blame History

% Figure folders
\newcommand{\ImprovedErrorResilience}{./041_ImprovedErrorResilience}
\graphicspath{{\ImprovedErrorResilience/Results/}}
\chapter{A Two-stage Approach for Robust Coding and Streaming}
% \chapter{Error Resilience Techniques for \ac{hevc} Standard}
%\chapter{Two-stage Robust Video Coding}
\label{Chapter:ErrorResilienceTechniques}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
\glsreset{mv}
\glsreset{rd}
The increased compression ratios achieved by the \ac{hevc} standard lead to reduced robustness of coded streams, with increased susceptibility to network errors and consequent video quality degradation, as shown in Chapter~\ref{Chapter:HEVCErrorResilience}.
Therefore, this chapter proposes a two-stage approach method to improve the error robustness of \ac{hevc} streaming, by reducing temporal error propagation in case of frame loss.
The prediction mismatch that occurs at the decoder after frame loss is reduced through the following two stages: (i) at the encoding stage, the reference pictures are dynamically selected based on constraining conditions and Lagrangian optimisation, which distributes the use of reference pictures, by reducing the number of \acs{pu} that depend on a single reference; (ii) at the streaming stage, a \ac{mv} prioritisation algorithm, based on spatial dependencies, selects an optimal sub-set of \acs{mv} to be transmitted, redundantly, as side information to reduce mismatched \ac{mv} predictions at the decoder.
This chapter is organized as follows. Section~\ref{Section4:Problem} presents the problem formulation and motivation for the method proposed in this chapter. Section~\ref{Section4:Introduction} provides an overview of the proposed method, describing the functional blocks that compose the two-stage approach for robust coding and streaming. Section~\ref{Section4:RefPicSel} describes the encoding stage of the proposed method, that is used to selected the optimal reference picture for inter-prediction. Section~\ref{Section4:MVSelection} describes the streaming stage used to select a sub-set of \acs{mv} to reduce the mismatch motion predictions. Section~\ref{Section4:Performance} provides a performance evaluation of the proposed method, illustrating the advantages of each stage and how they are combined together to achieve superior error robustness in \ac{hevc} standard. Finally, Section~\ref{Section4:Summary} concludes this chapter.
\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Problem formulation}
\label{Section4:Problem}
The error robustness characteristics of \ac{hevc} analysed in Chapter~\ref{Chapter:HEVCErrorResilience} confirmed that, in general, \ac{hevc} presents poor error resilience performance. Thus, \ac{hevc} bitstreams subject to transmission losses result in significant degradation of both objective and subjective quality. This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes of \ac{hevc}, which are selected according to a \ac{rd} optimisation criteria, assuming error-free transmission~\cite{ROPE2007}. Therefore, high compression efficiency is achieved, but resilience to transmission errors is penalised. This compromises the error robustness of video decoders which leads to poor performance under error conditions. Moreover, the use of advanced \ac{mv} prediction in \ac{hevc} \cite{MotionCodingHEVC2011} also contributes to increase the spatial and temporal dependencies to a great extent. Although the spatial MV candidates are used more often, temporally predicted \acs{mv} are responsible for a greater impact on error propagation.
\input{./041_ImprovedErrorResilience/Results/AverageRefPicUsage.tex}
As mentioned before the video decoder may rely of a distributed use of references frames to achieve higher video quality, by experienced lower error propagation and by optimising the error concealment algorithms to extract information from correctly decoded reference frames. In order to evaluate how the reference \ac{rd} optimisation presented in the reference \ac{hevc} encoder different test sequences were encoded using a configuration with two reference frames, referred to as $r_1$ and $r_2$. Figure~\ref{FIG4:ReferenceUsageRatio} illustrates the average usage ratio of reference $r_1$ for different sequences. The remaining corresponds to those cases where reference $r_2$ were selected.
Results in Figure~\ref{FIG4:ReferenceUsageRatio} shows that the reference $r_1$ is selected in most of the cases when two references are available. This is mainly true for sequences with higher motion. For those cases with lower motion but high spatial complexity, e.g., BQSquare sequence (see \ac{ti} in Table~\ref{TableTestSequences}), both frames are selected for temporal prediction, with $r_1$ being used for 58\% of cases and $r_2$ for 42\%. This reveals that even a configuration with two references the encoder will mainly use the temporally closer one as the reference for inter-frame predictions. This will lead to a higher number of mismatch predictions in case of frame loss and consequently to higher error propagation.
As mentioned in Chapter~\ref{Chapter:HEVCErrorResilience}, another technique that contributes to increased error propagation is the use of advanced \ac{mv} prediction in \ac{hevc} \cite{MotionCodingHEVC2011}. This contributes to increase the spatial and temporal dependencies to a great extent. Although the spatial \ac{mv} candidates are used more often, temporally predicted \acs{mv} are responsible for a greater impact on error propagation, as they propagate errors in the \ac{mv} decoding to across different time instances.
In general, increased error robustness in video streaming can be jointly achieved at two different stages: (i) while encoding live video, by selecting robust coding modes and (ii) after encoding, by adding robust features to the compressed streams. While in the former this is accomplished by optimising coding parameters and decisions to increase robustness, in the latter some type of compressed domain processing must be used to increase error robustness of pre-encoded streams, e.g., error resilient transcoding. Inevitably, in both cases this is achieved at the expense of some loss in coding efficiency, therefore the challenge is to find an optimum trade-off between robustness and bitrate overhead. In this chapter a two-fold approach is used to overcome the two problems mentioned above.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Proposed architecture}
\label{Section4:Introduction}
The goal of the proposed method is to reduce the error propagation in decoded video in case of frame loss. This is accomplished by constraining inter-frame prediction dependencies, in order to minimise the distortion $D_t$ given by
\begin{equation}
D_t = \tilde f_t - \hat f_t, \text{when } \tilde f_{t-1} \neq \hat f_{t-1},
\label{eq1}
\end{equation}
where $\hat f_t$ and $\tilde f_t$ are the encoded and decoded frames, respectively, at instant $t$.
Figure \ref{fig:proposedMethod} illustrates the block diagram of the two-stage method proposed in this work.
At the encoding stage, a reference frame selection mechanism is used for each \ac{pu} within a coding tree unit (\ac{ctu}) forcing the use of different reference frames in adjacent \acs{ctu}. The decision criterion contributes to reduce the amount of predictions from a single reference frame, and forces to interleave different reference frames along all encoded \acs{pu}, based on a \ac{rd} optimised criterion.
At the streaming stage, the coded stream is parsed to extract the \ac{mv} information, which allows ranking \acs{mv} according to a relevance criterion. Then, a sub-set of these \acs{mv} is selected to be transmitted as side information. The amount of selected \acs{mv} is not fixed and can be optimised for different content and end-network characteristics.
Note that the streaming stage is an independent and complementary subsystem to the encoder. Thus, it is able to operate at network nodes, including the streaming server on pre-encoded streams.
\begin{figure}[tb]
\centering
% \includegraphics[width=0.85\columnwidth]{041_ImprovedErrorResilience/Figures/BlockDiagram}
\input{041_ImprovedErrorResilience/Figures/BlockDiagram.tikz}
\caption{Functional blocks of the proposed method (\textsl{gray} blocks) in the context of the \ac{hevc} coding and streaming.}
\label{fig:proposedMethod}
\end{figure}
In case of frame loss these two stages have the common objective of helping the decoder to recover from mismatched predictions at the expense of a small increase in bitrate.
This is equivalent to reducing the error propagation by using two techniques: (i) reducing the mismatch of inter-frame predictions, by encoding each \ac{pu} from a different reference frame and (ii) reducing the mismatch of \ac{mv} predictions by sending the most important \acs{mv} as side information. These are the two stages of the proposed method described in the following sections.
\section{Encoding stage: Reference frame selection}
\label{Section4:RefPicSel}
The reference picture selection mechanism attempts to uniformly distribute the use of different reference pictures across the \ac{ctu} of a coded video frame. In case of frame loss, the error propagation in the decoder is reduced because the number of \acs{ctu} not predicted from the lost frame decreases and thus error concealment benefits from more data correctly decoded. Note that standard compliance is maintained.
The underlying principle of this method is to minimise the probability of using the same reference picture for neighbouring \acs{ctu}, which tends to result in a checkerboard structure, where a different reference frame is used for predicting every other \ac{ctu} within the current frame.
For instance, in a coding structure with two reference frames, as shown in Figure~\ref{fig:checkerboardRefSelection}, the reference frames $r_{1}$ and $r_{2}$ are used to predict the same number of \acs{ctu} in current frame~($C$).
If each reference frame is used to predict half of the pixels in the current frame, when any of these references is lost, the subsequent predicted frame is only partially distorted by mismatched predictions.
This is because only half of its predictions depend on the missing reference frame, which limits the overall error propagation.
Moreover, since the motion information of the predicted frame is encoded with reference not only to the missing frame but also to the past ones, those \acs{mv} that cross over a missing reference frame, pointing to the previous ones, are still useful for reconstructing correct \acs{mv} and also for error concealment.
In the limit, a straightforward implementation of the previous example leads to a simple interleaved uniform distribution of the reference frames over all \acs{ctu} in the current frame, forcing the checkerboard structure shown in Figure \ref{fig:checkerboardRefSelection}. In the case of two reference frames, 50\% of the \acs{ctu} would use one reference frame, while the other 50\% would use the other reference.
However, such solution would not take into consideration coding efficiency, since the selection process of the reference frame would not be \ac{rd} optimised for each \ac{pu} within a \ac{ctu}.
Therefore, a more efficient approach to improve error robustness was devised to dynamically choose the coding mode for each \ac{pu}, that ensures a balanced use of all possible reference frames $F=\{r_1,r_2,...r_S\}$ across the \acs{ctu} of the current frame $C$, also optimising the \ac{rd} cost.
This leads to the overall goal of finding the optimum R-D coding mode for each \ac{pu}, as the one which tends to minimise the standard deviation of the reference frames use count $\mathbf{N}_C(r_i)$, considering all encoded \acs{pu} of the current frame $C$, as follows:
\begin{equation}
\sigma_C=std\Big(\mathbf{N}_C(r_i) \Big), i=1, 2,\cdots S.
\label{eq:StdX}
\end{equation}
In this case $\sigma_C$ measures the deviation from the checkerboard solution, where $\sigma_C = 0$, since all possible reference frames $r_i\in F$ are uniformly used throughout the \acs{pu} of $C$.
\begin{figure}[t]
\centering
\input{\ImprovedErrorResilience/Figures/RefPicSelection.tikz}
\caption{Example of the reference picture selection applied to the frame $C$ based on the checkerboard at \ac{ctu} level (\acs{pu} represented by small partitions).}
\label{fig:checkerboardRefSelection}
\end{figure}
To attain the goal defined above, the best coding mode for each \ac{pu} is chosen through a Lagrangian \ac{rd} optimisation that tends to minimise $\sigma_C$.
This is done by penalising (or benefiting) the \ac{rd} cost of those coding modes which intend to use reference frames that have already been used more (or less) frequently than their average use in previous \acs{pu} of the current frame.
For each \ac{pu}, this mechanism has the effect of reducing (or increasing) the probability of using each possible reference frame $r_i\in F$, according to the number of previous \acs{pu} that have used each one.
Such approach allows to select the reference frames, by independently processing each \ac{pu} in order to minimise the global standard deviation given by~Equation~(\ref{eq:StdX}). Thus, the optimum coding mode $\phi^*$, selected from a set of possible coding modes $M$, is derived based on the following Lagrangian optimisation:
\begin{align}
\phi^* &= arg \min_{\phi \in M} J(\phi) \\
J(\phi) &= \Big( D(\phi) + \lambda \times R(\phi) \Big) \times e^{W(r_{\phi})}
\label{eq:Lagrangian}
\end{align}
where $D(\phi)$ and $R(\phi)$ are the distortion and rate, respectively, associated with the coding mode $\phi$.
The exponential weighting factor $e^{W(r_{\phi})}$ is used to penalise (or benefit) the Lagrangian cost of a given coding mode $\phi$ associated to reference frame $r_\phi$.
$W(r_{\phi})$ is obtained according to the global and local deviation between the number of times that $r_\phi$ was used and the average use of all $r_i \in F$ across the \acs{pu} of current frame $C$, as follows,
\begin{equation}\label{eq:KFactor}
W(r_{\phi}) = \Big( \Delta_G(r_{\phi}) + \Delta_L(r_{\phi}) \Big)\times 2^{-T_{ID}} \times \gamma
\end{equation}
where $T_{ID}$ is given by the temporal hierarchy in \ac{hevc}, and $\gamma$ is a parameter to control the slope of ${W(r_\phi)}$.
The global and local deviations, $\Delta_G (r_\phi)$ and $\Delta_L (r_\phi)$, respectively, are computed as follows,
\begin{equation}\label{eq:KFactorXValue}
\Delta_\zeta(r_{\phi}) = \mathbf{N}_\zeta(r_\phi) - \frac{1}{S} \sum_{i=1}^{S}\mathbf{N}_\zeta(r_i), \hspace{2mm} \zeta=G,L.
\end{equation}
For $\zeta=G$, function $\Delta_G(r_\phi)$ measures the difference between the number of times that reference frame $r_\phi$ was used and the global average use of all $r_i \in F$ in the \acs{pu} of $C$ that were encoded before the current \ac{pu}. $\mathbf{N}_G(r_i)$ is the number of times that each reference frame $r_i$ was used across all previously encoded \acs{pu}. For $\zeta=L$, function $\Delta_L(r_\phi)$ only considers the three top-left neighbouring \acs{ctu} for counting the number of times each reference frame was used, which is expressed by $\mathbf{N}_L(r_i)$. Overall, Equation (\ref{eq:KFactorXValue}) is used to adjust the penalty weight defined by Equation (\ref{eq:KFactor}) in order to achieve an approximately uniform use of all possible reference frames. This corresponds to the general objective of minimising the values of $\Delta_\zeta(r_{\phi})$.
The optimum is reached when all reference frames are used to encode the same amount of \acs{pu} (i.e., $\Delta_G(r_{\phi})=\Delta_L(r_{\phi})=0$), and the original Lagrangian cost is not affected, i.e., $e^{W(r_{\phi})}=1$ in Equation (\ref{eq:Lagrangian}).
Note that an exponential cost increase is used in Equation (\ref{eq:Lagrangian}) in order to increase the penalty associated to any reference frame that tends to be used much more often than the average.
\begin{figure}[t]
\centering
\includegraphics[width=0.8\columnwidth]{\ImprovedErrorResilience/Figures/ExponentialWeight}
\caption{Exponential weight as a function of the global and local deviations}
\label{fig:weightfct}
\end{figure}
In Equation (\ref{eq:KFactor}), the $\gamma$ constant controls the slope of $W(r_\phi)$, leading to higher weights as the value of $\gamma$ increases.
On the one hand a very high value for $\gamma$ leads to the checkerboard pattern in the reference picture usage, as previously described.
On the other hand, a low value of $\gamma$ reduces the importance of the choice of the reference frames in the Lagrangian cost.
Therefore, the impact of the proposed method on the coding efficiency can be controlled through $\gamma$. Another way of controlling the impact of the proposed method is the use of the $T_{ID}$. This allows using lower weights in higher temporal layers, since such frames cannot be used as references for others with lower temporal ID, thus having less impact on error propagation.
Therefore, there is a reduction of the impact of the proposed method when hierarchical coding configuration is adopted without compromising the error robustness.
As an example, Figure~\ref{fig:weightfct} illustrates the exponential weight used in the proposed method for different values of $\gamma$ and $T_{ID}=0$. The horizontal axis represents the sum of $\Delta_G(r_\phi)$~plus~$\Delta_L(r_\phi)$.
For instance, for $\gamma=5$, when a reference frame $r_\phi$ is used to predict 3 more \acs{ctu} ($\Delta_G(r_\phi) + \Delta_L(r_\phi)=3$) than the other reference frames (all \acs{pu} within those \acs{ctu} use the same reference), then the Lagrangian cost is increased by $50\%$ ($e^{W(r_{\phi})}\approx1.5$).
\begin{figure*}[tb]
\centering
\newcommand{\figsize}{0.22\textwidth}
\resizebox{\textwidth}{!}{
\begin{tabular}{y{.24\textwidth}y{.24\textwidth}y{.24\textwidth}y{.24\textwidth}}
\small Original frame & \small \REF & \small \CHKB & \small \EXP \\
\includegraphics[width=\figsize]{SelectionExample/ORG} &
\includegraphics[width=\figsize]{SelectionExample/REF} &
\includegraphics[width=\figsize]{SelectionExample/CHKB} &
\includegraphics[width=\figsize]{SelectionExample/EXP} \\
& \footnotesize $r_{1}$: $61.8\%$ | $r_{2}$: $25.7\%$ & \footnotesize $r_{1}$: $42.2\%$ | $r_{2}$: $39.3\%$ & \footnotesize $r_{1}$: $45.22\%$ | $r_{2}$: $43.2\%$ \\
\end{tabular}
}
\caption{Example of the partitions and references pictures selected using the different encoding methods (the dark gray squares indicate that the \ac{mv} is using the reference $r_{1}$, light~gray squares corresponds to the reference $r_{2}$ and unfilled regions are intra-coded blocks).}
\label{fig:RefPicSelectionExample}
\FIGSVSPACE
\end{figure*}
Figure \ref{fig:RefPicSelectionExample} illustrates the distribution of reference frames achieved by the proposed method (\EXP) across different \acs{pu}. This result is compared against a fixed checkerboard approach (\CHKB) and the reference \ac{hevc} (\REF). The dark-gray and light-gray squares indicate the reference frame in use, either $r_1$ or $r_2$, respectively. The white squares indicate intra-coded \acs{pu}. Below each method, the percentages of the image areas predicted by each reference frame are shown. As shown in this figure, the proposed method (\EXP) is able to distribute the use of the reference frames closely to the \CHKB method, i.e., $r_2$ is used as reference for approximately the same amount of \acs{pu} as $r_1$ (see use percentages of the predicted image). However, the predicted regions in the proposed method do not exhibit the same regular pattern as in \CHKB method.
It is noticeable in the figure that for several \acs{ctu} the \EXP method is able to keep the same partitions and reference frames when compared to the optimal encoding (\REF). This can be observed, for example in the small partitions on the left, where the \EXP method uses the same reference frame as \REF.Thus, \EXP is expected to achieve better coding efficiency than \CHKB, since more optimal coding modes may be chosen.
Summarising, the underlying principle of Equations~(\ref{eq:Lagrangian})~and~(\ref{eq:KFactor}) is that the cost of using a given reference frame increases with the number of \acs{pu} using it for prediction. However, the reference frames are not hard-selected without taking into account the R-D cost. In general, it is more difficult to reach a uniform use of reference frames in complex sequences than simpler ones.
Note that standard compatibility is maintained, since the proposed optimisation method does not require any syntax change. Moreover, as the proposed approach is applied at the Lagrangian computation already present in the \ac{hevc} optimisation, the complexity of the encoder is not significantly affected.
\section{Streaming stage: \ac{mv} selection}
\label{Section4:MVSelection}
Since \ac{hevc} uses differential and predictive \ac{mv} coding, whenever a \ac{mv} is lost the subsequent ones will also be affected, until a refresh point is reached to break the dependency chain. This increases the error propagation whenever the temporal \ac{mv} candidate is used in the Merge Mode.
To increase the number of possible refresh points and improve error robustness, a reduced number of \acs{mv} are selected to be transmitted as side information. The selection of these redundant \acs{mv} is based on a trade-off that attempts to maximise the image area covered by such \acs{mv} using the lowest number of bits to encode them.
\def\MVDEPA{\mbox{(a)}\xspace}
\def\MVDEPB{\mbox{(b)}\xspace}
\def\MVDEPC{\mbox{(c)}\xspace}
Figure~\ref{fig:mvdependencies} illustrates how \acs{mv} have different importance in error propagation.
In this figure, each block corresponds to a \ac{pu}, which has an associated \ac{mv}. The arrows point to the \ac{pu} containing the \ac{mv} used as predictor. In the current frame $f_0$, two \acs{pu} have temporally dependent \acs{mv}.
While the \ac{mv} of \ac{pu} \MVDEPA is used to predict a total of four other \acs{mv}, the \ac{mv} of \ac{pu} \MVDEPB is only used to predict one single \ac{mv}. Thus, in case of data loss, the \ac{mv} of \ac{pu} \MVDEPA has more impact on error propagation and quality degradation than the \ac{mv} of \ac{pu} \MVDEPB, because of its higher number of dependent \acs{mv}.
For the this reason, in the \ac{mv} selection process, the proposed method assigns higher importance to the \ac{mv} of \ac{pu} \MVDEPA than \ac{pu} \MVDEPB. Moreover in Figure~\ref{fig:mvdependencies}, it can be observed that although the number of \acs{mv} predicted from the \acs{mv} of \ac{pu} \MVDEPC is higher than \ac{pu} \MVDEPA (6 and 4 \acs{mv}, respectively),
the image area dependent from the former is smaller that the latter. This is due to the variable size of \acs{pu}.
Therefore, the image area covered by all dependent \acs{pu} is separately considered in the \ac{mv} selection process.
\begin{figure}[tb]
\centering
\input{\ImprovedErrorResilience/Figures/MVDependencies.tikz}
\caption{Example of temporal and spatial \ac{mv} dependencies in \ac{hevc}.}
\label{fig:mvdependencies}
\end{figure}
Based on the evidence mentioned above, the best \acs{mv} to be encoded redundantly as side information for each frame are selected through the following procedure:
\begin{enumerate}
\item
the \acs{mv} with temporal dependencies are firstly selected as the most important set: $V$ = \{\emph{\ac{mv}: \ac{mv} is temporally dependent from others}\}.
\item
the $MVs \in V$ are ranked according to the number of spatial dependencies associated to each one.
\item
the optimal sub-set $U \subset V$ is found as the best trade-off between the image area covered by the dependent \acs{pu} and bitrate overhead, i.e., the one that maximises the cost function in Equation (\ref{eq:MVSelectOpt}).
\end{enumerate}
Considering $V$ the ordered set of \acs{mv} encoded for the current frame, i.e., $V=\{MV_1, MV_2, \cdots MV_s\}$ and $U \subset V$ always starting in the first vector, i.e., \mbox{$U=\{MV_1, MV_2, \cdots MV_n\}$} with $n\leq s$, the best set of redundant \acs{mv} ($U^*)$ is selected based on the optimisation procedure given by Equation (\ref{eq:MVSelectOpt}). Note that any value of $n$ corresponds to a unique subset $U$.
\begin{equation}\label{eq:MVSelectOpt}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
n^* = \argmax_{n\leq s} \left\{ \sum_{i=1}^n A_{D}(i) - \alpha \frac{t_C-t_I}{T_{IDR}} R_U \right\},
\end{equation}
where $R_U$ is the amount of bits required to encode the $n$ \acs{mv} of $U$ and $A_{D}(i)$ is the total area of the \acs{pu} whose \acs{mv} are dependently encoded from $\ac{mv}_i$:
\begin{equation}
A_{D}(i) = \sum_{j=1}^{DV_i} ( w_j\times h_j ),\hspace{2mm} i = 1, 2 \cdots s,
\end{equation}
where $DV_i$ is the number of \acs{mv} dependent from $MV_i$ and $w_j$ and $h_j$ are the width and height of the corresponding \ac{pu}.
The weighting factor used to control the amount of redundancy $R_U$ is used to increase the cost of the overhead rate as the duration of error propagation decreases.
It is given by the ratio between the distance from the current frame ($t_c$) to the next \ac{idr} ($t_I$) and the \ac{idr} period ($T_{IDR}$), combined with the $\alpha$ parameter. Therefore, for higher values of $\alpha$, less redundant \acs{mv} are selected.
As pointed out before, this streaming stage may be implemented at any point of the network, to increase robustness without fully decoding the stream. Note that the parsing operation represented in Figure~\ref{fig:proposedMethod} does not decode the video frames, introducing only one frame delay because the \ac{mv} selection is performed on a frame-by-frame basis.
To maintain compatibility with the \ac{hevc} standard, the redundant motion information is transmitted using the \ac{sei} \ac{nal} units \cite{HEVCHighLevelReferenceDez2012}. Thus, this information can be multiplexed into the coded bitstream without affecting the compressed video.
The subset of selected \acs{mv} is independently encoded, ensuring that no reference information is needed in the decoder to properly decode the redundant \acs{mv}. In this work arithmetic coding~\cite{ArithmeticCodingJun1994} was used for the redundant \acs{mv}, but any other entropy coding method can be used.
\section{Performance evaluation}
\label{Section4:Performance}
In this section the robustness of the proposed method is evaluated and discussed for two cases: (i) only using the encoding stage and (ii) using the streaming stage in addition to the encoding stage. In the former case, identified as ``\EXP'', only the dynamic reference picture selection method described in Section~\ref{Section4:RefPicSel} is evaluated. In the latter case, identified as ``\DYN'', besides the previous method of reference picture selection, redundant \acs{mv} are encoded as described in Section~\ref{Section4:MVSelection}, resulting in a combination of both.
As references for comparison, three other methods are used: (i) a fixed checkerboard approach (\CHKB), (ii) the reference picture generation method proposed in \cite{ROPERefGenJan2010}, identified as ``\AVG'', and (iii) long-term reference frames (\LONG). The \LONG method uses only the key frames as reference, i.e., in the Random-Access configuration only one out of eight frames is used as reference frame, allowing a more robust transmission, as the reference frames are less likely to be hit by errors.
The experimental setup used in this section is the same as the one previously described in Section~\ref{Section3:ErrorResilienceEvaluation}. The evaluation of the encoding-stage is carried out separately from the streaming-stage.
\subsection{Encoding stage}
The performance of the encoding stage is evaluated on its own in order to consider only the improvements achieved by the dynamic reference frame selection method, when reference frames were reconstructed with distortion (e.g. as a result of non-perfect error concealment).
Since the encoding stage does not deal with errors in temporal \ac{mv} predictions (this is handled by the streaming-stage of the proposed method), in this evaluation the temporal \ac{mv} predictions were not used. This is necessary to avoid masking the results with the errors caused by wrong \ac{mv} predictions.
The performance of the proposed encoding stage (\EXP) was evaluated using $\gamma = 5$. This allows to increase the Lagrangian cost by about $50\%$, when a reference frame is used for three or more \acs{ctu} in comparison to the usage of the other ones (see Figure~\ref{fig:weightfct}). Further results for a wide range of $\gamma$ values are also presented.
\subsubsection{Performance under error-free conditions}
Table~\ref{tab:bd_error_free} shows the Bjontegaard's Delta PSNR (BD-PSNR) values compared to the reference \ac{hevc} encoder for different coding configurations in error-free decoding. Four QPs were used from the common test conditions: 22, 27, 32, 37.
The aim is to evaluate the \ac{rd} penalty incurred by the~\EXP method in comparison with \CHKB, as well as the other reference methods.
The use of long-term reference pictures (\LONG) is only tested for the case of Random-Access, where key frames are available, while method \cite{ROPERefGenJan2010} is tested for the Low-Delay configuration.
The results in Table~\ref{tab:bd_error_free} indicate a small loss of coding efficiency when the proposed reference picture selection approach is used under error-free conditions. As expected, this is due to the sub-optimal \ac{rd} optimisation of the proposed method, because the cost function also includes the cost associated to the choice of the reference frame, in addition to the rate and distortion.
\begin{table}[t]
\centering
\newcommand{\resultswidth}{1.5cm}
\caption{BD-PSNR in error-free conditions.}
\TABLESVSPACE
\footnotesize
\label{tab:bd_error_free}
\begin{tabular}{p{3cm}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}}
\toprule
\multirow{2}{*}{\centering\textbf{Sequence}} & \multicolumn{3}{c|}{\textbf{Low-Delay}} & \multicolumn{3}{c}{\textbf{Random-Access}} \\
& \AVG & \CHKB & \EXP & \LONG & \CHKB & \EXP \\ \hline
Basketball Drill & -1.14 & -1.15 & -0.51 & -0.85 & -0.29 & -0.22 \\ \hline
Kendo & -2.10 & -1.39 & -0.99 & -0.91 & -0.39 & -0.28 \\ \hline
Park Scene & -1.79 & -1.38 & -0.81 & -0.68 & -0.38 & -0.28 \\ \hline
Traffic & -1.91 & -1.21 & -0.71 & -0.55 & -0.36 & -0.21 \\ \hline
Bosphorus & -0.97 & -0.95 & -1.10 & -0.64 & -0.20 & -0.20 \\ \hline
\textbf{Average} & \textbf{-1.58} & \textbf{-1.22} & \textbf{-0.82} & \textbf{-0.73} & \textbf{-0.32} & \textbf{-0.24} \\
\bottomrule
\end{tabular}
\end{table}
While the fixed checkerboard approach leads to an average quality reduction of $1.22$~dB for the Low-Delay ($0.32$~dB for Random-Access), the proposed method only loses $0.82$~dB and $0.24$~dB in the Low-Delay and Random-Access configurations, respectively. In comparison to \LONG and the method in~\cite{ROPERefGenJan2010}, the proposed method is still more efficient, consistently achieving better results.
Overall, the loss in the quality compared to the reference \ac{hevc} is considered acceptable, given the increase in robustness obtained in lossy transmission, as discussed in the following sections.
%\subsubsection{Reference picture selection results}
\subsubsection{Quality evaluation under error-prone conditions}
The robustness achieved by the proposed method was evaluated by comparison of the error propagation resulting from a single loss event in the Low-Delay configuration.
Figure~\ref{fig:RecoveredExample} shows the visual impact of error propagation on a region of the decoded Frame \#10 when Frame \#9 is affected by errors, for Basketball Drill and Race Horses sequences (the PSNR corresponds to the entire frame). As shown in this figure, the proposed method (\EXP) is able to reduce the impact of mismatch decoding in comparison with the method~in~\cite{ROPERefGenJan2010}, revealing the higher efficiency achieving by distributing the use of the references frames.
\begin{figure}[tb]
\centering
\footnotesize
\newcommand{\CurrFigSize}{0.23\columnwidth}
\begin{tabular}{y{.23\columnwidth}y{.23\columnwidth}|y{.23\columnwidth}y{.23\columnwidth}}
\multicolumn{2}{c}{Basketball Drill} & \multicolumn{2}{|c}{Race Horses} \\
\AVG & \EXP & \AVG & \EXP \\
\includegraphics[width=\CurrFigSize]{Figure_6_1} &
\includegraphics[width=\CurrFigSize]{Figure_6_2} &
\includegraphics[width=\CurrFigSize]{Figure_6_3} &
\includegraphics[width=\CurrFigSize]{Figure_6_4} \\
PSNR = 31.93 dB & PSNR = 33.83 dB & PSNR = 24.12 dB & PSNR = 31.93 dB \\
\end{tabular}
\caption{Zoom of recovered Frame \#10 after a loss event at Frame \#9 for Basketball Drill and Race Horses sequence.}
\label{fig:RecoveredExample}
\end{figure}
\begin{figure}[t]
\renewcommand{\FIGSIZE}{0.8\columnwidth}
\centering
% \includegraphics[width=\FIGSIZE]{\ImprovedErrorResilience/Results/PSNRversusFrameKendo}
\input{041_ImprovedErrorResilience/Results/PSNRversusFrameKendo.tex}
\caption{Error propagation when errors affect Frame \#7 in Kendo sequence with an \ac{idr} period of 32 frames.}
\label{fig:PSNRvsFrameForRefPicSel}
\end{figure}
\input{\ImprovedErrorResilience/Results/TableResultsRandomLossRefPicSelection.tex}
Figure~\ref{fig:PSNRvsFrameForRefPicSel}~shows the PSNR over a GOP with an \ac{idr} period of 32 frames, encoded at the same constant bitrate.
This figure shows that all methods are able to outperform the reference \ac{hevc} (\REF), gaining approximately $3$~dB in reconstruction quality.
Since the proposed method is more efficient than the others for error-free transmission (see~Table~\ref{tab:bd_error_free}), and it also provides better error robustness, it can be considered as an efficient solution to better deal with transmission errors.
The similar quality levels achieved by \CHKB and \EXP reveal that dynamic selection of the reference frames is more robust because a better trade-off between coding efficiency and error resilience is attained.
Overall, the method proposed for the encoding stage is able to increase the error robustness of \ac{hevc} without severely compromising the coding efficiency.
Further tests were run to evaluate the effectiveness of the proposed method under various packet loss rates.
For each test condition, 50 trials were performed and the average quality obtained across all trials for both coding configurations are shown in Table~\ref{tab:AverageQualityCHKBEXPLowDelay}~and~\ref{tab:AverageQualityCHKBEXPRandomAccess}, for the Low-Delay and Random-Access, respectively.
In both tables the absolute PSNR value is shown for the reference \ac{hevc} case (\REF), while the PSNR difference is presented for the other cases.
Results show that the proposed method is able to outperform other methods, by increasing the robustness of \ac{hevc}. An average gain up to $1.47$~dB is obtained for the Low-Delay configuration (see Table~\ref{tab:AverageQualityCHKBEXPLowDelay}), in comparison to the \REF, while the fixed checkerboard pattern approach (\CHKB) only achieves up to $1.33$~dB on average.
In \cite{ROPERefGenJan2010}, the use of interpolated references in the encoding loop reveals to reduce error propagation, at a cost of high degradation of the coding efficiency in the error-free environment, thus the overall performance is lower (see results for ``\AVG'').
The proposed method is also able to outperform the \ac{hevc} standard when an hierarchical coding structure is used (see results for Random-Access configuration in Table~\ref{tab:AverageQualityCHKBEXPRandomAccess}), which inherently limits the error propagation.
In comparison to the reference \ac{hevc}, quality improvements up to $1.42$~dB at PLR=$10\%$ are obtained (Kendo), while the average gain across all test sequences is 1~dB.
When comparing results for both cases, it is noticeable that the proposed method is able to outperform the reference methods both when \ac{idr} frame loss is allowed or not (case 1 and 2, respectively).
As the \ac{plr} increases the gains of the proposed method decreases when comparing with the case where the \ac{idr} frames delivered error-free (i.e., case 1), due to the higher error propagation of the missing \ac{idr} frame.
Taking into account the above results and the spatio-temporal information shown in Table~\ref{TableTestSequences}, one can observe that the performance of the proposed method is lower for sequences with high spatial details and low temporal activity (see results for BQSquare and Park Scene), which indicates higher effectiveness for video sequences with higher motion activity.
\begin{figure}[!b]
\renewcommand{\FIGSIZE}{0.8\columnwidth}
\centering
%\includegraphics[width=\FIGSIZE,page=1]{\ImprovedErrorResilience/Results/PSNRversusGama}
%\includegraphics[width=\FIGSIZE,page=2]{\ImprovedErrorResilience/Results/PSNRversusGama}
\input{./041_ImprovedErrorResilience/Results/PSNRversusGama.tex}
\caption{Influence of the $\gamma$ parameter in the average quality gains, comparing with \REF, for different packet loss ratios.}
\label{fig:QualityVsGamma}
\end{figure}
\subsubsection{Influence of parameter $\gamma$}
The influence of the $\gamma$ parameter used in Equation (\ref{eq:KFactor}) was also studied.
Figure \ref{fig:QualityVsGamma} shows the average quality gains in comparison to the reference \ac{hevc} (\REF) across all test sequences, when using different values of $\gamma$ in the proposed method (\EXP).
For the error-free case, the results in this figure show that increasing $\gamma$ leads to a decrease in the coding efficiency, especially for the Low-Delay configuration. The reason is that higher values of $\gamma$ lead to higher values of $W(r_{\phi})$, therefore the reference picture selection is more constrained by the exponential weights. Since $T_{ID}$ is constant for all frames in the Low-Delay, higher weights are used for all frames ($T_{ID}$ is used to reduce the weights in case of hierarchical coding) and the encoder is not able to select the best reference frame in terms of R-D optimisation.
In the case where packet loss occurs, higher values of $\gamma$ lead to higher error resilience. Since higher weights used in the \ac{rd} cost, the encoder is constrained to distribute the use of the reference frames, reducing the error propagation.
For very high values (e.g., $\gamma=1000$) the encoder is forced to uniformly select the reference frames, resulting in a fixed checkerboard pattern.
Overall, using the $\gamma$ parameter in the proposed method, the impact of the reference frame, selected by the \ac{rd} optimisation, can be controlled, and thus a better trade-off between higher error resilience and loss of coding efficiency can be achieved.
\subsubsection{Complexity overhead}
The implementation of the proposed method has slightly increased the complexity of the standard non-robust encoder. Therefore, the computational complexity has also been considered as a performance metric in this research, which is measured as the encoding time in a controlled hardware platform. The reference used for comparison with other methods is the \ac{hevc} reference software encoding time, running on the same platform with the same coding configuration.
The relative complexity increase of the robust encoder using \EXP, \CHKB and the method presented in \cite{ROPERefGenJan2010} is shown in Table~\ref{tab:cpu_time}.
\begin{table}[b]
\newcommand{\resultswidth}{2.2cm}
\centering
\caption{Relative CPU times of tested methods.}\TABLESVSPACE
\label{tab:cpu_time}
\footnotesize
\begin{tabular}{p{4cm}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}}
\toprule
\multirow{2}{*}{\centering\textbf{Sequence}} & \multicolumn{3}{c}{\textbf{Method}} \\
& \AVG & \CHKB & \EXP \\ \hline
Basketball Drill & +4.19\% & +2.13\% & +2.93\% \\ \hline
Kendo & +4.96\% & +1.83\% & +2.05\% \\ \hline
Park Scene & +4.98\% & +3.90\% & +4.04\% \\ \hline
Traffic & +8.65\% & +2.28\% & +3.05\% \\
\bottomrule
\end{tabular}
\end{table}
As shown in this table, the method presented in \cite{ROPERefGenJan2010} is the most time consuming because it requires post-processing of the encoded frames in order to interpolate new references.
The proposed scheme, using a reference picture selection, \EXP presents a lower complexity increase than \cite{ROPERefGenJan2010}, while the fixed checkerboard scheme (\CHKB) presents a slightly lower complexity than \EXP.
\subsection{Streaming stage}
In this section the coding performance that results from using redundant \ac{mv} (streaming stage) in combination with the reference picture selection scheme (encoding stage) is presented. In the experimental tests the temporal \ac{mv} candidates are enabled and redundant \acs{mv} are added into the stream, as described in Section \ref{Section4:MVSelection}. This is referred to as ``\DYN'', and compared to cases where a fixed amount of redundant \acs{mv} per frame are transmitted (i.e., the most important 10\%, 30\% and 50\% $\ac{mv} \in V$). Different $\alpha$ values are used for different redundancy ratios. All streams were encoded with the same total bitrate, including the redundant bits, in order to make a fair quality comparison.
% \begin{figure}[tbp]
% \centering
% \newcommand{\CurrFigSize}{0.45\columnwidth}
%
% \caption{Decoded video quality for different amounts of redundancy, selected using a fixed approach and the proposed method (Continuation).}
% \label{fig:QualityVsRedundancy\ac{mv}SelectionContinuation}
% \end{figure}
\subsubsection{Dynamic MV selection performance}
\begin{figure}[!b]
\centering
\newcommand{\CurrFigSize}{0.49\columnwidth}
% \includegraphics[width=\CurrFigSize,page=1]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
% \includegraphics[width=\CurrFigSize,page=2]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
\includegraphics[width=\CurrFigSize,page=3]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
\includegraphics[width=\CurrFigSize,page=4]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
\includegraphics[width=\CurrFigSize,page=5]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
\includegraphics[width=\CurrFigSize,page=6]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
\caption{Decoded video quality for different amounts of redundancy, selected using a fixed approach and the proposed method.}
\label{fig:QualityVsRedundancyMVSelection}
\end{figure}
In the case where variable redundancy is allowed, \acs{mv} are selected using the proposed dynamic method (Dyn) and a fixed approach (Fixed).
Figure~\ref{fig:QualityVsRedundancyMVSelection} illustrates the relation between the quality obtained and the amount of redundancy.
The results indicate that for approximately the same amount of redundant bits, dynamic selection of \acs{mv} lead to higher performance in all test sequences. For instance, the proposed method is able to gain up to $1.5$~dB for the BQSquare sequence. Moreover, the results are consistent for different \ac{plr}.
\subsubsection{Quality evaluation}
\begin{table}[!t]
\centering
\caption{Average quality using redundant \ac{mv} selected by the proposed approach.}
\TABLESVSPACE
\label{tab:AverageQualityRedundancies}
\TABLESVSPACE
\input{\ImprovedErrorResilience/Results/TableResultsRandomLossMvRedundancies.tex}
\end{table}
Table~\ref{tab:AverageQualityRedundancies} presents the average PSNR of decoded video for different \ac{plr} (columns~4~to~8) and for various percentages of \ac{mv} redundancy (third column).
It can be confirmed from the results that, for each sequence, as $\alpha$ decreases more redundant \acs{mv} are used, resulting in higher objective image quality for all tested sequences.
This indicates that redundant \acs{mv} are able to minimise the error propagation. This is accomplished with an acceptable increase in the bitrate, since only a sub-set of \acs{mv} is transmitted.
The use of redundant \acs{mv} reveals better results for \ac{plr} as low as $1\%$, achieving a maximum gain of $3.93$~dB for \LOSTALL ($7.82$~dB for \KEEPINTRA) for the BQSquare sequence at PLR=$5\%$.
In summary, the proposed method is able to select the most relevant motion \ac{mv} information and when combined with a reference picture selection scheme the problem of mismatch \ac{mv} predictions is mitigated.
Moreover, the dynamic approach devised to select the best \acs{mv} also contribute to increase the overall performance.
Thus, using the proposed two-stage approach to recover temporal predictions consistently improves the error robustness of \ac{hevc}.
\section{Summary}
\label{Section4:Summary}
In this chapter a two-stage approach is proposed to increase the error robustness of \ac{hevc} streaming and reduce the error propagation in case of packet loss.
A constrained coding approach was devised to select reference frames and dynamically distribute temporal dependencies.
This is jointly used with a controlled amount of side information in coded streams, comprising a small set of the most relevant \acs{mv}, to minimise \ac{mv} mismatch at the decoder in the presence of frame loss.
As can be concluded from the results, this method contributes to reduce the temporal dependencies with consistent gains for different \ac{plr} and coding configurations, only incurring in a small drop of coding efficiency.
Overall, the proposed approach is an effective method for coping with video transmission over error-prone networks.
Reference in New Issue View Git Blame Copy Permalink