Fix abreviations and equations referencing

This commit is contained in:
jfmcarreira committed 2018-12-10 19:19:32 +00:00
1 parent 935eb96ed4
commit 52d96c729a
11 files changed
+40 -144

No files matched your search

@@ -15,10 +15,6 @@
\vspace*{\fill}
\section{Introduction}
\glsreset{rd}
\glsreset{pu}
\glsreset{mv}
This chapter proposes a new two-stage approach to improve the error robustness of \ac{hevc} streaming, by reducing temporal error propagation in case of frame loss.
This is accomplished by reducing the prediction mismatch that occurs at the decoder after frame loss.
The proposed approach comprises the following two stages: (i) at the encoding stage, the reference pictures are dynamically selected based on constraining conditions and Lagrangian optimisation, which distributes the use of reference pictures throughout each frame, reducing the number of \ac{pu} that depend on a single reference; (ii) at the streaming stage, a \ac{mv} prioritisation algorithm, based on spatial dependencies, selects an optimal sub-set of \acs{mv} to be transmitted as side redundant information to reduce mismatched \ac{mv} predictions at the decoder.
@@ -32,16 +28,6 @@ This chapter is organized as follows. Section~\ref{Section4:Problem} presents th
\section{Problem definition}
\label{Section4:Problem}
\glsreset{rd}
\glsreset{ec}
\glsreset{pu}
\glsreset{ctu}
\glsreset{cu}
\glsreset{mv}
\glsreset{ra}
\glsreset{ld}
As previously mentioned, when\ac{hevc} bitstreams are subjected to transmission losses, the decoded video signal incurs in significant degradation of both objective and subjective quality.
This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes, which are selected according to \ac{rd} optimisation criteria, which always assume error-free transmission~\cite{ROPE2007}.
Therefore, high compression efficiency is achieved, but resilience to transmission errors is penalised.
@@ -62,7 +48,7 @@ These results show that, in general, reference $r_1$ is selected more often than
This is mainly true for sequences with higher motion.
For those sequences with lower motion, but high spatial complexity, e.g., BQSquare (see \ac{si} in Table~\ref{TableTestSequences}), both reference frames are equally (approximately) selected for temporal predictions, with $r_1$ being used by 58\% of cases and $r_2$ by 42\%.
This reveals that, even though two references are used, the encoder mainly uses the temporally closer one as the reference for inter-frame predictions.
Therefore, if such reference frame is lost, a higher number of mismatched predictions will occur, inevitabely leading to higher error propagation.
Therefore, if such reference frame is lost, a higher number of mismatched predictions will occur, inevitably leading to higher error propagation.
Another coding technique that contributes to increase error propagation is the use of predictions in \ac{mv} coding~\cite{MotionCodingHEVC2011}, which also contributes to increase the total amount of spatial and temporal dependencies.
As shown in Chapter~\ref{Chapter:HEVCErrorResilience}, temporally predicted \acs{mv} are responsible for a greater impact on error propagation, as they propagate errors in the \ac{mv} decoding across different time instances.
@@ -134,12 +120,10 @@ This leads to the overall goal of finding the optimum R-D coding mode for each \
\end{equation}
In this case $\sigma_C$ measures the deviation from the checkerboard solution, where $\sigma_C = 0$, since all possible reference frames $r_i\in F$ are uniformly used throughout the \acs{pu} of $C$.
To attain the goal defined above, the best coding mode for each \ac{pu} is chosen through a Lagrangian \ac{rd} optimisation that tends to minimise $\sigma_C$.
This is done by penalising (or benefiting) the \ac{rd} cost of those coding modes which intend to use reference frames that have already been used more (or less) frequently than their average use in previous \acs{pu} of the current frame.
For each \ac{pu}, this mechanism has the effect of reducing (or increasing) the probability of using each possible reference frame $r_i\in F$, according to the number of previous \acs{pu} that have used each one.
Such approach allows to select the reference frames, by independently processing each \ac{pu} in order to minimise the global standard deviation given by~Equation~(\ref{eq:StdX}). Thus, the optimum coding mode $\phi^*$, selected from a set of possible coding modes $M$, is derived based on the following Lagrangian optimisation:
Such approach allows to select the reference frames, by independently processing each \ac{pu} in order to minimise the global standard deviation given by~(\ref{eq:StdX}). Thus, the optimum coding mode $\phi^*$, selected from a set of possible coding modes $M$, is derived based on the following Lagrangian optimisation:
\begin{align}
\phi^* &= arg \min_{\phi \in M} J(\phi) \\
J(\phi) &= \Big( D(\phi) + \lambda \times R(\phi) \Big) \times e^{W(r_{\phi})}
@@ -156,9 +140,9 @@ The global and local deviations, $\Delta_G (r_\phi)$ and $\Delta_L (r_\phi)$, r
\begin{equation}\label{eq:KFactorXValue}
\Delta_\zeta(r_{\phi}) = \mathbf{N}_\zeta(r_\phi) - \frac{1}{S} \sum_{i=1}^{S}\mathbf{N}_\zeta(r_i), \hspace{2mm} \zeta=G,L.
\end{equation}
For $\zeta=G$, function $\Delta_G(r_\phi)$ measures the difference between the number of times that reference frame $r_\phi$ was used and the global average use of all $r_i \in F$ in the \acs{pu} of $C$ that were encoded before the current \ac{pu}. $\mathbf{N}_G(r_i)$ is the number of times that each reference frame $r_i$ was used across all previously encoded \acs{pu}. For $\zeta=L$, function $\Delta_L(r_\phi)$ only considers the three top-left neighbouring \acs{ctu} for counting the number of times each reference frame was used, which is expressed by $\mathbf{N}_L(r_i)$. Overall, Equation (\ref{eq:KFactorXValue}) is used to adjust the penalty weight defined by Equation (\ref{eq:KFactor}), in order to achieve an approximately uniform use of all possible reference frames. This corresponds to the general objective of minimising the values of $\Delta_\zeta(r_{\phi})$.
The optimum is reached when all reference frames are used to encode the same amount of \acs{pu} (i.e., $\Delta_G(r_{\phi})=\Delta_L(r_{\phi})=0$), and the original Lagrangian cost is not affected, i.e., $e^{W(r_{\phi})}=1$ in Equation (\ref{eq:Lagrangian}).
Note that an exponential cost increase is used in Equation (\ref{eq:Lagrangian}) in order to increase the penalty associated to any reference frame that tends to be used much more often than the average.
For $\zeta=G$, function $\Delta_G(r_\phi)$ measures the difference between the number of times that reference frame $r_\phi$ was used and the global average use of all $r_i \in F$ in the \acs{pu} of $C$ that were encoded before the current \ac{pu}. $\mathbf{N}_G(r_i)$ is the number of times that each reference frame $r_i$ was used across all previously encoded \acs{pu}. For $\zeta=L$, function $\Delta_L(r_\phi)$ only considers the three top-left neighbouring \acs{ctu} for counting the number of times each reference frame was used, which is expressed by $\mathbf{N}_L(r_i)$. Overall, (\ref{eq:KFactorXValue}) is used to adjust the penalty weight defined by (\ref{eq:KFactor}), in order to achieve an approximately uniform use of all possible reference frames. This corresponds to the general objective of minimising the values of $\Delta_\zeta(r_{\phi})$.
The optimum is reached when all reference frames are used to encode the same amount of \acs{pu} (i.e., $\Delta_G(r_{\phi})=\Delta_L(r_{\phi})=0$), and the original Lagrangian cost is not affected, i.e., $e^{W(r_{\phi})}=1$ in (\ref{eq:Lagrangian}).
Note that an exponential cost increase is used in (\ref{eq:Lagrangian}) in order to increase the penalty associated to any reference frame that tends to be used much more often than the average.
\begin{figure}[t]
\centering
@@ -167,7 +151,7 @@ Note that an exponential cost increase is used in Equation (\ref{eq:Lagrangian})
\label{fig:weightfct}
\end{figure}
In Equation (\ref{eq:KFactor}), the $\gamma$ constant controls the slope of $W(r_\phi)$, leading to higher weights as the value of $\gamma$ increases.
In (\ref{eq:KFactor}), the $\gamma$ constant controls the slope of $W(r_\phi)$, leading to higher weights as the value of $\gamma$ increases.
On the one hand a very high value for $\gamma$ leads to the checkerboard pattern in the reference picture usage, as previously described.
On the other hand, a low value of $\gamma$ reduces the importance of the choice of the reference frames in the Lagrangian cost.
Therefore, the impact of the proposed method on the coding efficiency can be controlled through $\gamma$. Another way of controlling the impact of the proposed method is the use of the $T_{ID}$. This allows using lower weights in higher temporal layers, since such frames cannot be used as references for others with lower temporal ID, thus having less impact on error propagation.
@@ -198,8 +182,7 @@ For instance, for $\gamma=5$, when a reference frame $r_\phi$ is used to predict
Figure~\ref{fig:RefPicSelectionExample} illustrates the distribution of reference frames achieved by the proposed method across different \acs{pu}. This result is compared against a fixed checkerboard approach (\CHKB) and the reference \ac{hevc} (\REF). The dark-gray and light-gray squares indicate the reference frame in use, either $r_1$ or $r_2$, respectively. The unfilled squares indicate intra-coded \acs{pu}. Below each method, the percentages of the image areas predicted by each reference frame are shown. As shown in this figure, the proposed method is able to distribute the use of the reference frames closely to the \CHKB method, i.e., $r_2$ is used as reference for approximately the same amount of \acs{pu} as $r_1$ (see use percentages of the predicted image). However, the predicted regions in the proposed method do not exhibit the same regular pattern as in \CHKB method.
It is noticeable in Figure~\ref{fig:RefPicSelectionExample} that for several \acs{ctu} the \PROP method is able to keep the same partitions and reference frames when compared to the optimal encoding (\REF). This can be observed, for example in the small partitions on the left, where the \PROP method uses the same reference frame as \REF. Thus, the proposed method is expected to achieve better coding efficiency than \CHKB, since more optimal coding modes may be chosen.
% TID aqui?
Summarising, by using Equations~(\ref{eq:Lagrangian})~and~(\ref{eq:KFactor}), the cost of using a given reference frame increases as the number of \acs{pu} using it for prediction increases.
Summarising, by using~(\ref{eq:Lagrangian})~and~(\ref{eq:KFactor}), the cost of using a given reference frame increases as the number of \acs{pu} using it for prediction increases.
However, the reference frames are not hard-selected without taking into account the R-D cost.
In general, it is more difficult to reach a uniform use of reference frames in complex sequences than in simpler ones.
Note that standard compatibility is maintained, since the proposed optimisation method does not require any syntax change. Moreover, since the proposed approach is employed at the Lagrangian optimisation already implemented in the \ac{hevc} reference software, the complexity of the encoder is not significantly affected.
@@ -208,8 +191,6 @@ Note that standard compatibility is maintained, since the proposed optimisation
\section{Streaming stage: \ac{mv} selection}
\label{Section4:MVSelection}
\glsreset{tmvp}
Since \ac{hevc} uses differential and predictive \ac{mv} coding, whenever a \ac{mv} is lost the subsequent ones will also be affected, until a refresh point (i.e., \ac{mv} coded on its own, without using \ac{mv} prediction) is reached to break the dependency chain.
Thus, the amount of error propagation strongly depends on whether the \ac{tmvp} is used.
In this streaming stage, to increase the number of possible refresh points and improve error robustness, a small number of \acs{mv} are selected to be transmitted as side information. The selection of these redundant \acs{mv} is based on a trade-off between the image area covered by such \acs{mv} and the number of bits to encode them.
@@ -242,10 +223,10 @@ the \acs{mv} with temporal dependencies are firstly selected as the most importa
\item
the sub-set of $MVs \in V$ are ranked according to the number of spatial dependencies associated to each one.
\item
the optimal sub-set $U \subset V$ is found as the best trade-off between the image area covered by the dependent \acs{pu} and bitrate overhead, i.e., the one that maximises the cost function in Equation (\ref{eq:MVSelectOpt}).
the optimal sub-set $U \subset V$ is found as the best trade-off between the image area covered by the dependent \acs{pu} and bitrate overhead, i.e., the one that maximises the cost function in (\ref{eq:MVSelectOpt}).
\end{enumerate}
This can be formulated as follows. Considering $V$ the ordered set of \acs{mv} encoded for the current frame, i.e., ${V=\{MV_1, MV_2, \cdots MV_s\}}$ and ${U \subset V}$ always starting in the first vector, i.e., ${U_n=\{MV_1, MV_2, \cdots MV_n\}}$ with $n\leq s$, the best set of \acs{mv} ($U_n^*)$ is selected based on the optimisation procedure given by Equation (\ref{eq:MVSelectOpt}). Note that any value of $n$ corresponds to a unique subset $U$.
This can be formulated as follows. Considering $V$ the ordered set of \acs{mv} encoded for the current frame, i.e., ${V=\{MV_1, MV_2, \cdots MV_s\}}$ and ${U \subset V}$ always starting in the first vector, i.e., ${U_n=\{MV_1, MV_2, \cdots MV_n\}}$ with $n\leq s$, the best set of \acs{mv} ($U_n^*)$ is selected based on the optimisation procedure given by (\ref{eq:MVSelectOpt}). Note that any value of $n$ corresponds to a unique subset $U$.
\begin{equation}\label{eq:MVSelectOpt}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
@@ -280,17 +261,6 @@ At the decoder-side, \ac{ec} is based on motion compensation from previously dec
The evaluation of the encoding-stage is carried out separately from the streaming-stage.
%This section presents an evaluation study about the error robustness of HEVC using different sequences and coding configurations. Table~\ref{tab:sequences} presents a summary of the main characteristics of the video sequences used in this work. Each sequence comprises 240 frames and the whole set covers different types of motion and texture complexity. This table also shows the spatial information (SI) and the temporal information (TI), as defined in \cite{ITUTP9111998}. The experimental results for this study were obtained by using the HM reference software, version $16.2$.
%The sequences were encoded with all coding modes enabled and following two recommended prediction structures: Low-Delay (P-frames with 4 references frames) and \ac{ra} (B-frames with 2 reference frames in each list)~\cite{HEVCCommonTests2012}. These prediction structures cover the use of P-frames and hierarchical B-frames~\cite{H264ScalableOverviewSep2007}, which are commonly used in HEVC streaming~\cite{RealTimeHEVC2016}.
%To simulate a broadcast transmission an \ac{idr} period of $32$ frames was used and the filtering operations were disabled at slices boundaries in order to keep the slices self-contained. The remaining configurations have been kept at their default values as given in the common test conditions~\cite{HEVCCommonTests2012}.
%In order to achieve high quality images for all coded sequences (i.e., PSNR around 40 dB) different bitrates were used.
%At the decoder-side, error concealment is based on motion compensation from previously decoded frames. The motion field of a missing frame is obtained applying MV extrapolation from the two closest neighbours. Afterwards, the final MVs are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}.
%To simulate the lossy transmission environment, each frame is encoded into several slices, each one packetized into one NAL unit, which are then transmitted as the payload of independent packets.
%
%In our experiments a whole packet is lost whenever an error occurs, originating a lost slice. Random packet loss was simulated using a two-state Markov model with the following average burst length 1.24, 1.47, 1.83 and 2.05, and a maximum burst length of 2, 3, 5 and 7, for packet loss ratios (PLR) of 1\%, 3\%, 5\%, 10\%, respectively.
\subsection{Encoding stage}
@@ -382,7 +352,7 @@ Taking into account the above results and the spatio-temporal information shown
\label{fig:QualityVsGamma}
\end{figure}
The influence of the $\gamma$ parameter used in Equation (\ref{eq:KFactor}) on the video quality was also evaluated, in order to check how the cost constrains affect the performance of the proposed method.
The influence of the $\gamma$ parameter used in (\ref{eq:KFactor}) on the video quality was also evaluated, in order to check how the cost constrains affect the performance of the proposed method.
Figure~\ref{fig:QualityVsGamma} shows the average quality gains of the proposed method in comparison to the reference \ac{hevc} (\REF) across all test sequences, for different values of $\gamma$.
For the error-free case (PLR=0\%), the results in this figure show that increasing $\gamma$ leads to a decrease in the coding efficiency, especially in the \ac{ld} configuration.
The reason is that higher values of $\gamma$ lead to higher values of $W(r_{\phi})$, therefore the reference picture selection cost is severely constrained by the exponential weights.