jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Complete review of PA comments in chapter 7 and SF coments on chapters 1 to 6

Browse Source
This commit is contained in:
jfmcarreira
2018-09-11 03:08:00 +01:00
parent 1c13a4e48e
commit 461b560b17
23 changed files with 2475 additions and 997 deletions
Download Patch File Download Diff File Expand all files Collapse all files
041_ImprovedErrorResilience/ImprovedErrorResilience.tex
+57 -52
View File
@@ -3,7 +3,7 @@
\graphicspath{{\ImprovedErrorResilience/Results/}}
\chapter{A Two-stage Approach for Robust Coding and Streaming \VERSION{1}}
\chapter{A Two-stage Approach for Robust Coding and Streaming \VERSION{2}}
% \chapter{Error Resilience Techniques for \ac{hevc} Standard}
%\chapter{Two-stage Robust Video Coding}
\label{Chapter:ErrorResilienceTechniques}
@@ -13,6 +13,8 @@
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
\glsreset{pu}
\glsreset{ctu}
\glsreset{mv}
\glsreset{rd}
@@ -20,10 +22,10 @@
\glsreset{ld}
%The increased compression ratios achieved by the \ac{hevc} standard lead to reduced robustness of coded streams, with increased susceptibility to network errors and consequent video quality degradation, as shown in Chapter~\ref{Chapter:HEVCErrorResilience}.
Therefore, this chapter proposes a coding method, based on a two-stage approach, to improve the error robustness of \ac{hevc} streaming, by reducing temporal error propagation in case of frame loss.
This is accomplished by reducing the prediction mismatch that occurs at the decoder after frame loss through the following two stages: (i) at the encoding stage, the reference pictures are dynamically selected based on constraining conditions and Lagrangian optimisation, which distributes the use of reference pictures throughout each frame, by reducing the number of \acs{pu} that depend on a single reference; (ii) at the streaming stage, a \ac{mv} prioritisation algorithm, based on spatial dependencies, selects an optimal sub-set of \acs{mv} to be transmitted, redundantly, as side information to reduce mismatched \ac{mv} predictions at the decoder.
This chapter proposes a coding method, based on a two-stage approach, to improve the error robustness of \ac{hevc} streaming, by reducing temporal error propagation in case of frame loss.
This is accomplished by reducing the prediction mismatch that occurs at the decoder after frame loss, through the following two stages: (i) at the encoding stage, the reference pictures are dynamically selected based on constraining conditions and Lagrangian optimisation, which distributes the use of reference pictures throughout each frame, by reducing the number of \acs{pu} that depend on a single reference; (ii) at the streaming stage, a \ac{mv} prioritisation algorithm, based on spatial dependencies, selects an optimal sub-set of \acs{mv} to be transmitted, redundantly, as side information to reduce mismatched \ac{mv} predictions at the decoder.
This chapter is organized as follows. Section~\ref{Section4:Problem} presents the problem definition and motivation for the method proposed in this chapter. Section~\ref{Section4:Architecture} provides an overview of the proposed method, describing the functional blocks that compose the two-stage approach for robust coding and streaming. Section~\ref{Section4:RefPicSel} describes the encoding stage of the proposed method, that is used to select the optimal reference picture for inter-prediction. Section~\ref{Section4:MVSelection} describes the streaming stage used to select a sub-set of \acs{mv} to reduce the mismatch motion predictions. Section~\ref{Section4:Performance} provides a performance evaluation of the proposed method, illustrating the advantages of each stage and how they are combined to achieve superior error robustness in \ac{hevc} standard. Finally, Section~\ref{Section4:Summary} concludes the chapter.
This chapter is organized as follows. Section~\ref{Section4:Problem} presents the problem definition and motivation for the method proposed in this chapter. Section~\ref{Section4:Architecture} provides an overview of the proposed method, describing the functional blocks that compose the two-stage approach for robust coding and streaming. Section~\ref{Section4:RefPicSel} describes the encoding stage of the proposed method, that is used to select the optimal reference picture for inter-prediction. Section~\ref{Section4:MVSelection} describes the streaming stage where a sub-set of \acs{mv} are selected to reduce the mismatch motion predictions. Section~\ref{Section4:Performance} provides a performance evaluation of the proposed method, illustrating the advantages of each stage and how they are combined to achieve superior error robustness in \ac{hevc} standard. Finally, Section~\ref{Section4:Summary} concludes the chapter.
\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -33,8 +35,8 @@ This chapter is organized as follows. Section~\ref{Section4:Problem} presents th
\label{Section4:Problem}
As mentioned before \ac{hevc} bitstreams subject to transmission losses result in significant degradation of both objective and subjective quality.
This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes, which are selected according to a \ac{rd} optimisation criteria, assuming error-free transmission~\cite{ROPE2007}.
As previously mentioned,\ac{hevc} bitstreams subject to transmission losses result in significant degradation of both objective and subjective quality.
This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes, which are selected according to a \ac{rd} optimisation criteria that assumes error-free transmission~\cite{ROPE2007}.
Therefore, high compression efficiency is achieved, but resilience to transmission errors is penalised.
\begin{figure}[tb]
@@ -44,20 +46,21 @@ Therefore, high compression efficiency is achieved, but resilience to transmissi
\label{FIG4:ReferenceUsageRatio}
\end{figure}
Highly efficient video encoders support the use of multiple reference frames for inter-frame prediction, which are selected based on a \ac{rd} cost which may lead to an unbalance distribution.
In order to show evaluate the unbalanced is the reference frame selection attained by the \ac{rd} implemented in the reference \ac{hevc} encoder, different test sequences are encoded using two reference frames, referred to as $r_1$ and $r_2$, and its selection ratios are analysed. In this study, reference $r_1$ corresponds to the closest neighbouring frame and reference $r_2$ the second closest.
Highly efficient video encoders support the use of multiple reference frames for inter-frame prediction, which are selected based on a \ac{rd} cost that may lead to an unbalance distribution.
In order to evaluate the reference frame selection method attained by the \ac{rd} implemented in the reference \ac{hevc} encoder, different test sequences are encoded using two reference frames, referred to as $r_1$ and $r_2$, and its selection ratios are analysed. In this study, reference $r_1$ corresponds to the closest neighbouring frame and reference $r_2$ the second closest.
Figure~\ref{FIG4:ReferenceUsageRatio} illustrates the average usage percentage of reference $r_1$ for different sequences.
The remaining corresponds to those cases where reference $r_2$ were selected.
The remaining corresponds to those cases where reference $r_2$ is selected.
These results show that, in general, reference $r_1$ is selected more often than $r_2$.
This is mainly true for sequences with higher motion. For those sequences with lower motion but high spatial complexity, e.g., BQSquare (see \ac{si} in Table~\ref{TableTestSequences}), both reference frames equally (approximately) selected for temporal predictions, with $r_1$ being used for 58\% of cases and $r_2$ for 42\%.
This reveals that even with a configuration with two references the encoder will mainly use the temporally closer one as the reference for inter-frame predictions.
This is mainly true for sequences with higher motion.
For those sequences with lower motion, but high spatial complexity, e.g., BQSquare (see \ac{si} in Table~\ref{TableTestSequences}), both reference frames are equally (approximately) selected for temporal predictions, with $r_1$ being used by 58\% of cases and $r_2$ by 42\%.
This reveals that even with a configuration using only two references the encoder mainly uses the temporally closer one as the reference for inter-frame predictions.
Therefore, if such reference frame is lost, a higher number of mismatched predictions will occur, inevitabely leading to higher error propagation.
Another technique that contributes to increased error propagation is the use of predictions in \ac{mv} coding~\cite{MotionCodingHEVC2011}, which contributes to increase the spatial and temporal dependencies.
Although the spatial \ac{mv} candidates are used more often, temporally predicted \acs{mv} are responsible for a greater impact on error propagation, as they propagate errors in the \ac{mv} decoding across different time instances.
To overcome these issues, in the next sections a two-fold approach is described to increase the error robustness in video streaming by jointly considering two different stages: (i) while encoding live video, by selecting robust coding modes and (ii) after encoding, by adding robust features to the compressed streams. While in the former this is accomplished by optimising coding parameters and decisions to increase robustness, in the latter some type of compressed domain processing is used to increase error robustness of pre-encoded streams, e.g., error resilient transcoding. Inevitably, in both cases this is achieved at the expense of some loss in coding efficiency, therefore the challenge is to find an optimum trade-off between robustness and bitrate overhead. In this chapter a two-fold approach is used to overcome the two problems mentioned above.
To overcome these issues, in the next sections a two-fold approach is described to increase the error robustness in video streaming by jointly considering two different stages: (i) while encoding live video, by selecting robust coding modes and (ii) after encoding, by adding robust features to the compressed streams. While in the former this is accomplished by optimising coding parameters and decisions to increase robustness, in the latter some type of compressed domain processing is used to increase error robustness of pre-encoded streams, e.g., error resilient transcoding. Inevitably, in both cases this is achieved at the expense of some loss in coding efficiency, therefore the challenge is to find an optimum trade-off between robustness and bitrate overhead.
%In this chapter a two-fold approach is used to overcome the two problems mentioned above.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
@@ -74,9 +77,10 @@ D_t = \tilde f_t - \hat f_t, \text{when } \tilde f_{t-1} \neq \hat f_{t-1},
where $\hat f_t$ and $\tilde f_t$ are the encoded and decoded frames, respectively, at instant $t$.
Figure \ref{fig:proposedMethod} illustrates the block diagram of the two-stage architecture proposed in this work.
At the encoding stage, a reference frame selection mechanism is used for each \ac{pu} within a coding tree unit (\ac{ctu}) forcing the use of different reference frames in adjacent \acs{ctu}. The decision criterion contributes to reduce the amount of predictions from a single reference frame, and forces to interleave different reference frames along all encoded \acs{pu}, based on \ac{rd} optimisation.
At the encoding stage, a reference frame selection mechanism is used for each \ac{pu} within a \ac{ctu}, by forcing the use of different reference frames for adjacent \acs{ctu}.
The decision criterion contributes to reduce the amount of predictions from a single reference frame and forces to interleave different reference frames along all encoded \acs{pu}, based on \ac{rd} optimisation.
At the streaming stage, the coded stream is parsed to extract the \ac{mv} information, which allows ranking \acs{mv} according to a relevance criterion. Then, a sub-set of these \acs{mv} is selected to be transmitted as side information. The amount of selected \acs{mv} is not fixed and can be optimised for different content and network characteristics.
Note that the streaming stage is an independent and complementary subsystem to the encoder. Thus, it is able to operate at network nodes, including streaming servers delivering pre-encoded streams.
Note that the streaming stage is an independent and complementary subsystem of the encoder. Thus, it is able to operate at network nodes, including streaming servers delivering pre-encoded streams.
\begin{figure}[tb]
\centering
@@ -86,8 +90,8 @@ Note that the streaming stage is an independent and complementary subsystem to t
\label{fig:proposedMethod}
\end{figure}
In case of frame loss these two stages have the common objective of helping the decoder to recover from mismatched predictions at the expense of a small increase in bitrate.
This is equivalent to reducing the error propagation by using two techniques: (i) reducing the mismatch of inter-frame predictions, by encoding each \ac{pu} from a different reference frame and (ii) reducing the mismatch of \ac{mv} predictions by sending the most important \acs{mv} as side information. These are the two stages of the proposed method described in the following sections.
In case of frame loss, these two stages have the common objective of helping the decoder to recover from mismatched predictions at the expense of a small increase in bitrate.
This is equivalent to reduce the error propagation by using two techniques: (i) reducing the mismatch of inter-frame predictions, by encoding each \ac{pu} from a different reference frame and (ii) reducing the mismatch of \ac{mv} predictions by sending the most important \acs{mv} as side information. These are the two stages of the proposed method described in the following sections.
@@ -96,14 +100,14 @@ This is equivalent to reducing the error propagation by using two techniques: (i
The reference picture selection mechanism attempts to uniformly distribute the use of different reference pictures across the \ac{ctu} of a coded video frame.
In case of frame loss, the error propagation in the decoder is reduced because the number of \acs{ctu} predicted from the lost frame decreases and thus error concealment benefits as more data is correctly decoded from other frames.
In case of frame loss, the error propagation in the decoder is reduced because the number of \acs{ctu} predicted from the lost frame decreases and thus error concealment benefits as more data is correctly decoded from other reference frames.
The underlying principle of this method is to minimise the probability of using the same reference picture for neighbouring \acs{ctu}.
%For instance,
In a coding structure with two reference frames, as shown in Figure~\ref{fig:checkerboardRefSelection}, the reference frames $r_{1}$ and $r_{2}$ are used to predict the same number of \acs{ctu} in current frame~($C$), resulting in a checkerboard structure, where a different reference frame is used for predicting every other \ac{ctu} within the current frame.
If each reference frame is used to predict half of the pixels in the current frame, when any of these references is lost, the subsequent predicted frame is only partially distorted by mismatched predictions.
This is because only half of its predictions depend on the missing reference frame, which limits the overall error propagation.
Moreover, since the motion information of the predicted frame is encoded with not only reference to the missing frame but also to the past ones, those \acs{mv} that cross over a missing reference frame, pointing to the previous ones, are still useful for reconstructing correct \acs{mv} and also for error concealment.
Moreover, since the motion information of the predicted frame is encoded using as reference not only the missing frame but also the past ones, those \acs{mv} that cross over a missing reference frame, pointing to the previous ones, are still useful for reconstructing correct \acs{mv} and also for error concealment.
In the limit, a straightforward implementation of the previous example leads to a simple interleaved uniform distribution of the reference frames over all \acs{ctu} in the current frame, forcing the checkerboard structure shown in Figure \ref{fig:checkerboardRefSelection}. In the case of two reference frames, 50\% of the \acs{ctu} would use one reference frame, while the other 50\% would use the other reference.
However, such solution would not take into consideration coding efficiency, since the selection process of the reference frame would not be \ac{rd} optimised for each \ac{pu} within a \ac{ctu}.
@@ -144,7 +148,7 @@ The global and local deviations, $\Delta_G (r_\phi)$ and $\Delta_L (r_\phi)$, r
\begin{equation}\label{eq:KFactorXValue}
\Delta_\zeta(r_{\phi}) = \mathbf{N}_\zeta(r_\phi) - \frac{1}{S} \sum_{i=1}^{S}\mathbf{N}_\zeta(r_i), \hspace{2mm} \zeta=G,L.
\end{equation}
For $\zeta=G$, function $\Delta_G(r_\phi)$ measures the difference between the number of times that reference frame $r_\phi$ was used and the global average use of all $r_i \in F$ in the \acs{pu} of $C$ that were encoded before the current \ac{pu}. $\mathbf{N}_G(r_i)$ is the number of times that each reference frame $r_i$ was used across all previously encoded \acs{pu}. For $\zeta=L$, function $\Delta_L(r_\phi)$ only considers the three top-left neighbouring \acs{ctu} for counting the number of times each reference frame was used, which is expressed by $\mathbf{N}_L(r_i)$. Overall, Equation (\ref{eq:KFactorXValue}) is used to adjust the penalty weight defined by Equation (\ref{eq:KFactor}) in order to achieve an approximately uniform use of all possible reference frames. This corresponds to the general objective of minimising the values of $\Delta_\zeta(r_{\phi})$.
For $\zeta=G$, function $\Delta_G(r_\phi)$ measures the difference between the number of times that reference frame $r_\phi$ was used and the global average use of all $r_i \in F$ in the \acs{pu} of $C$ that were encoded before the current \ac{pu}. $\mathbf{N}_G(r_i)$ is the number of times that each reference frame $r_i$ was used across all previously encoded \acs{pu}. For $\zeta=L$, function $\Delta_L(r_\phi)$ only considers the three top-left neighbouring \acs{ctu} for counting the number of times each reference frame was used, which is expressed by $\mathbf{N}_L(r_i)$. Overall, Equation (\ref{eq:KFactorXValue}) is used to adjust the penalty weight defined by Equation (\ref{eq:KFactor}), in order to achieve an approximately uniform use of all possible reference frames. This corresponds to the general objective of minimising the values of $\Delta_\zeta(r_{\phi})$.
The optimum is reached when all reference frames are used to encode the same amount of \acs{pu} (i.e., $\Delta_G(r_{\phi})=\Delta_L(r_{\phi})=0$), and the original Lagrangian cost is not affected, i.e., $e^{W(r_{\phi})}=1$ in Equation (\ref{eq:Lagrangian}).
Note that an exponential cost increase is used in Equation (\ref{eq:Lagrangian}) in order to increase the penalty associated to any reference frame that tends to be used much more often than the average.
@@ -183,7 +187,7 @@ For instance, for $\gamma=5$, when a reference frame $r_\phi$ is used to predict
\FIGSVSPACE
\end{figure*}
Figure~\ref{fig:RefPicSelectionExample} illustrates the distribution of reference frames achieved by the proposed method across different \acs{pu}. This result is compared against a fixed checkerboard approach (\CHKB) and the reference \ac{hevc} (\REF). The dark-gray and light-gray squares indicate the reference frame in use, either $r_1$ or $r_2$, respectively. The white squares indicate intra-coded \acs{pu}. Below each method, the percentages of the image areas predicted by each reference frame are shown. As shown in this figure, the proposed method is able to distribute the use of the reference frames closely to the \CHKB method, i.e., $r_2$ is used as reference for approximately the same amount of \acs{pu} as $r_1$ (see use percentages of the predicted image). However, the predicted regions in the proposed method do not exhibit the same regular pattern as in \CHKB method.
Figure~\ref{fig:RefPicSelectionExample} illustrates the distribution of reference frames achieved by the proposed method across different \acs{pu}. This result is compared against a fixed checkerboard approach (\CHKB) and the reference \ac{hevc} (\REF). The dark-gray and light-gray squares indicate the reference frame in use, either $r_1$ or $r_2$, respectively. The unfilled squares indicate intra-coded \acs{pu}. Below each method, the percentages of the image areas predicted by each reference frame are shown. As shown in this figure, the proposed method is able to distribute the use of the reference frames closely to the \CHKB method, i.e., $r_2$ is used as reference for approximately the same amount of \acs{pu} as $r_1$ (see use percentages of the predicted image). However, the predicted regions in the proposed method do not exhibit the same regular pattern as in \CHKB method.
It is noticeable in Figure~\ref{fig:RefPicSelectionExample} that for several \acs{ctu} the \PROP method is able to keep the same partitions and reference frames when compared to the optimal encoding (\REF). This can be observed, for example in the small partitions on the left, where the \PROP method uses the same reference frame as \REF. Thus, the proposed method is expected to achieve better coding efficiency than \CHKB, since more optimal coding modes may be chosen.
% TID aqui?
@@ -212,7 +216,7 @@ In this figure, each block corresponds to a \ac{pu}, which has an associated \ac
While the \ac{mv} of \ac{pu} \MVDEPA is used to predict a total of four other \acs{mv}, the \ac{mv} of \ac{pu} \MVDEPB is only used to predict one single \ac{mv}.
Thus, in case of data loss (e.g., frame $f_{-1}$), the \ac{mv} of \ac{pu} \MVDEPA has more impact on error propagation to the frame $f_{0}$ and consequently quality degradation than the \ac{mv} of \ac{pu} \MVDEPB, because of its higher number of dependent \acs{mv}.
For this reason, in the \ac{mv} selection process, the proposed method assigns higher importance to the \ac{mv} of \ac{pu} \MVDEPA than \ac{pu} \MVDEPB. Moreover, in Figure~\ref{fig:mvdependencies}, it can be observed that although the number of \acs{mv} predicted from the \acs{mv} of \ac{pu} \MVDEPC is higher than \ac{pu} \MVDEPA (6 and 4 \acs{mv}, respectively),
the image area that is dependent from the former is smaller that the latter. This is due to the variable size of \acs{pu}.
the image area that depends on the former is smaller that the latter. This is due to the variable size of \acs{pu}.
Therefore, the image area covered by all dependent \acs{pu} is separately considered in the \ac{mv} selection process.
\begin{figure}[tb]
@@ -242,9 +246,9 @@ where $R_U$ is the amount of bits required to encode the $n$ \acs{mv} of $U$ and
\begin{equation}
A_{D}(i) = \sum_{j=1}^{DV_i} ( w_j\times h_j ),\hspace{2mm} i = 1, 2 \cdots s,
\end{equation}
where $DV_i$ is the number of \acs{mv} dependent from $MV_i$ and $w_j$ and $h_j$ are the width and height of the corresponding \ac{pu}.
where $DV_i$ is the number of \acs{mv} dependent from $MV_i$, and $w_j$ and $h_j$ are the width and height of the corresponding \ac{pu}.
A weighting factor is used to control the amount of bitrate overhead ($R_U$), by increasing the cost of the number of encoded bits based on the duration of the error propagation.
It is given by the ratio between the distance from the current frame ($t_c$) to the next refresh frame ($t_I$) and the \ac{idr} period ($T_{I}$), combined with the control parameter $\alpha$.
This duration is given by the ratio between the distance from the current frame ($t_c$) to the next refresh frame ($t_I$) and the \ac{idr} period ($T_{I}$), combined with the control parameter $\alpha$.
This weight varies in the range $\left[0;1 \right]$, achieving the lowest value for intra-coded frames and the maximum for the frame located immediately before an intra-coded one.
In the proposed approach, for higher values of $\alpha$, less redundant \acs{mv} are selected.
As pointed out before, this streaming stage operates on coded data and may be implemented at any point of the network, to increase robustness without fully decoding the stream. Note that the parsing operation represented in Figure~\ref{fig:proposedMethod} does not include decoding of the video frames. Furthermore, only one frame delay is introduced because the \ac{mv} selection process is performed on a frame-by-frame basis.
@@ -281,7 +285,7 @@ The evaluation of the encoding-stage is carried out separately from the streamin
\subsection{Encoding stage}
The performance of the encoding stage is evaluated on its own in order to only determine the improvements achieved by the dynamic reference frame selection method.
The performance of the encoding stage is evaluated on its own, in order to only determine the improvements achieved by the dynamic reference frame selection method.
In this case, the reference frames were reconstructed with distortion, as a result of non-perfect error concealment.
Since the encoding stage does not deal with errors in temporal \ac{mv} predictions (this is handled by the streaming-stage), in this evaluation the temporal \ac{mv} predictor is disabled.
This is necessary to avoid masking the results with the errors caused by wrong \ac{mv} predictions.
@@ -311,9 +315,9 @@ This value was chosen to allow increasing the Lagrangian cost by about $50\%$,
\label{fig:RecoveredExample}
\end{figure}
The error robustness achieved by the encoding-stage was firstly evaluated in regards to the error propagation resulting from a single loss event in the \ac{ld} configuration.
The error robustness achieved by the encoding-stage was firstly evaluated in regards to the error propagation, resulting from a single loss event in the \ac{ld} configuration.
Figure~\ref{fig:RecoveredExample} shows the visual impact of error propagation on a region of the decoded Frame \#10 when Frame \#9 is affected by errors, for Basketball Drill and Race Horses sequences (the \ac{psnr} corresponds to the entire frame).
The proposed method is compared with the method~in~\cite{ROPERefGenJan2010} as both aim at reducing the error propagation.
The proposed method is compared with the method~in~\cite{ROPERefGenJan2010}, as both aim at reducing the error propagation.
The method~in~\cite{ROPERefGenJan2010} uses filtered frames as reference for predictions, therefore, in case of errors, the effects of mismatch predictions are significantly reduced by the filtering operation (see Sub-Section~\ref{SECTION2:LeakyPred}).
As shown in this figure, the proposed method is able to reduce the impact of mismatch decoding in comparison with the method~in~\cite{ROPERefGenJan2010}. Although both methods still reveal some predictions mismatch leading to inaccurate reconstructed of the objects contours, the proposed method is able to achieve a smoother reconstruction, which leads to higher quality in terms of \ac{psnr}. This is because, in frame \#10, the predictions depending on the missing frame \#9 are not used throughout the entire frame, thus, approximately half of the predictions are not affected by errors. In contrast, the method~in~\cite{ROPERefGenJan2010} requires several filtering operations to be applied (i.e, different reference frames need to be filtered) in order to reduce the effect of mismatch predictions.
Summarising, the reconstructed frames illustrated in Figure~\ref{fig:RecoveredExample} reveal the higher efficiency achieved by distributing the use of the references frames.
@@ -348,7 +352,7 @@ In \cite{ROPERefGenJan2010}, the use of interpolated references in the encoding
The proposed method is also able to outperform the \ac{hevc} reference software when an hierarchical coding structure is used (see results for \ac{ra} configuration in Table~\ref{tab:AverageQualityCHKBEXPRandomAccess}), which implicitly limits the error propagation.
In comparison to the reference \ac{hevc}, quality improvements up to $1.42$~dB at PLR=$10\%$ are obtained for Kendo sequence.%, while the average gain across all test sequences is 1~dB.
When comparing results for both test cases, it is noticeable that the proposed method is able to outperform the reference methods both when \ac{idr} frame loss is allowed or not (\LOSTALL and \KEEPINTRA, respectively). As the \ac{plr} increases, the gains of the proposed method decrease in comparison with the case where the \ac{idr} frames are delivered error-free (i.e., case \KEEPINTRA), due to the higher error propagation of the missing \ac{idr} frame. Moreover, the quality gains are lower in the \LOSTALL case when comparing with \KEEPINTRA for the same \ac{plr}, for both the \ac{ld} and \ac{ra}.
When comparing results for both test cases, it is noticeable that the proposed method is able to outperform the reference methods for both cases, when \ac{idr} frame loss is allowed or not (\LOSTALL and \KEEPINTRA, respectively). As the \ac{plr} increases, the gains of the proposed method decrease in comparison with the case where the \ac{idr} frames are delivered error-free (i.e., case \KEEPINTRA), due to the higher error propagation of the missing \ac{idr} frame. Moreover, the quality gains are lower in the \LOSTALL case when comparing with \KEEPINTRA for the same \ac{plr}, for both the \ac{ld} and \ac{ra}.
Taking into account the above results and the spatio-temporal information shown in Table~\ref{TableTestSequences}, one can observe that the performance of the proposed method is lower for sequences with high spatial details and low temporal activity (e.g., BQSquare and Park Scene), which indicates higher effectiveness for video content with higher motion activity.
%\HIGHLIGHT{complete with analysis of both cases}
@@ -359,7 +363,7 @@ Taking into account the above results and the spatio-temporal information shown
%\includegraphics[width=\FIGSIZE,page=1]{\ImprovedErrorResilience/Results/PSNRversusGama}
%\includegraphics[width=\FIGSIZE,page=2]{\ImprovedErrorResilience/Results/PSNRversusGama}
\input{./041_ImprovedErrorResilience/Results/PSNRversusGama.tex}
\caption{Influence of the $\gamma$ parameter in the average quality gains, compared with \REF, for different \acs{plr}.}
\caption{Influence of the $\gamma$ parameter in the average quality gains, comparing the proposed method with \REF, for different \acs{plr}.}
\label{fig:QualityVsGamma}
\end{figure}
@@ -371,7 +375,7 @@ The influence of the $\gamma$ parameter used in Equation (\ref{eq:KFactor}) on t
Figure \ref{fig:QualityVsGamma} shows the average quality gains of the proposed method in comparison to the reference \ac{hevc} (\REF) across all test sequences, for different values of $\gamma$.
For the error-free case (PLR=0\%), the results in this figure show that increasing $\gamma$ leads to a decrease in the coding efficiency, especially in the \ac{ld} configuration.
The reason is that higher values of $\gamma$ lead to higher values of $W(r_{\phi})$, therefore the reference picture selection cost is severely constrained by the exponential weights.
Since $T_{ID}$ is constant for all frames in the \ac{ld}, higher weights are used for all frames ($T_{ID}$ is used to reduce the weights in case of hierarchical coding) and the encoder is not able to select the best reference frame in \c{rd} sense.
Since $T_{ID}$ is constant for all frames in the \ac{ld}, higher weights are used for all frames ($T_{ID}$ is used to reduce the weights in case of hierarchical coding) and the encoder is not able to select the best reference frame in the \ac{rd} sense.
In the case where packet loss occurs, higher values of $\gamma$ lead to improved error resilience.
Since higher weights are used in the \ac{rd} cost, the encoder is constrained to distribute the use of the reference frames of different \acs{pu}, reducing the error propagation.
For very high values (e.g., $\gamma=1000$) the encoder is forced to uniformly select the reference frames, resulting in a fixed checkerboard pattern.
@@ -422,7 +426,7 @@ The relative complexity increase of the robust encoder using \PROP, \CHKB and th
As shown in this table, the method presented in \cite{ROPERefGenJan2010} is the most time consuming because it requires post-processing of the encoded frames in order to interpolate new references.
The proposed scheme, using a reference picture selection, presents a lower complexity increase than \cite{ROPERefGenJan2010}, while the fixed checkerboard scheme (\CHKB) presents a slightly lower complexity \PROP method.
The proposed scheme, using a reference picture selection, presents a lower complexity increase than \cite{ROPERefGenJan2010}, while the fixed checkerboard scheme (\CHKB) presents a slightly lower complexity than \PROP method.
This is due to the fact that in the \CHKB method there is no \ac{rd} optimisation process for the reference frame selection (fixed pattern is used), thus reducing the overall computational complexity.
@@ -453,7 +457,14 @@ Different values of $\alpha$ are used to achieve different overhead ratios. All
\subsubsection{Dynamic MV selection performance}
\begin{figure}[!b]
Firstly the quality obtained by the proposed method, which dynamically selects the amount of redundant \acs{mv}, is compared to cases where a fixed amount of \acs{mv} per frame are transmitted (i.e., the most important 10\%, 30\% and 50\% $\ac{mv} \in V$). This cased are referred to as "\DYN" and "\FIXED", respectively.
%In the case where variable redundancy is allowed, \acs{mv} are selected using the proposed dynamic method (Dyn) and a fixed approach (Fixed).
Figure~\ref{fig:QualityVsRedundancyMVSelection} illustrates the relation between the quality obtained (\ac{psnr}) and the amount of redundancy used.
The results indicate that for approximately the same amount of redundant bits are used in both cases, however by using the proposed method to dynamically select the \acs{mv} to be used as side-information (\DYN) is able to outperform the fixed case for all test sequences.
Results also show that for both approaches the decoded video quality increases as the redundancy of the side-information increases, since a higher number of mismatch \ac{mv} predictions are able to be recovered.
\begin{figure}[tb]
\centering
\newcommand{\CurrFigSize}{0.49\columnwidth}
\includegraphics[width=\CurrFigSize,page=1]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
@@ -462,19 +473,13 @@ Different values of $\alpha$ are used to achieve different overhead ratios. All
\includegraphics[width=\CurrFigSize,page=4]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
\includegraphics[width=\CurrFigSize,page=5]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
\includegraphics[width=\CurrFigSize,page=6]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
\caption{Decoded video quality for different amounts of redundancy, selected by using a fixed approach and the proposed method.}
\caption{Decoded video quality for different amounts of redundancy, selected by using a fixed approach and the proposed method (\DYN).}
\label{fig:QualityVsRedundancyMVSelection}
\end{figure}
Firstly the quality obtained by the proposed method, which dynamically selecting the amount of redundant \acs{mv}, is compared to cases where a fixed amount of \acs{mv} per frame are transmitted (i.e., the most important 10\%, 30\% and 50\% $\ac{mv} \in V$). This cased are referred to as "\DYN" and "\FIXED", respectively.
%In the case where variable redundancy is allowed, \acs{mv} are selected using the proposed dynamic method (Dyn) and a fixed approach (Fixed).
Figure~\ref{fig:QualityVsRedundancyMVSelection} illustrates the relation between the quality obtained (\ac{psnr}) and the amount of redundancy used.
The results indicate that for approximately the same amount of redundant bits are used in both cases, however by using the proposed method to dynamically select the \acs{mv} to be used as side-information (\DYN) is able to outperform the fixed case for all test sequences.
Results also show that for both approaches the decoded video quality increases as the redundancy of the side-information increases, since a higher number of mismatch \ac{mv} predictions are able to be recovered.
It is noticeable in Figure~\ref{fig:QualityVsRedundancyMVSelection} that for higher \acs{plr}, the proposed method is able to achieve higher gains. This reveals that for increasing data loss it is more important optimise the amount of \acs{mv} selected per frame in order to use more \acs{mv} in those frames which have a higher impact of the error propagation.
For instance, the proposed method is able to gain up to $1.5$~dB for the BQSquare sequence for a PLR=10\$.
It is noticeable in Figure~\ref{fig:QualityVsRedundancyMVSelection} that for higher \acs{plr}, the proposed method is able to achieve higher gains.
This reveals that when data loss increases it is more important optimise the amount of \acs{mv} selected per frame, in order to use more \acs{mv} in those frames that have a higher impact on the error propagation.
For instance, the proposed method is able to gain up to $1.5$~dB for the BQSquare sequence subject to a PLR=10\%.
For lower \acs{plr}, since fewer frames are affected by data losses, the use of a fixed approach achieves similar performance as the proposed method (\DYN).
@@ -482,13 +487,13 @@ For lower \acs{plr}, since fewer frames are affected by data losses, the use of
\subsubsection{Quality evaluation}
Table~\ref{tab:AverageQualityRedundancies} presents the average \ac{psnr} of decoded video for different \ac{plr} (columns~4~to~8) and for various percentages of \ac{mv} redundancy (third column). The absolute quality values are shown for the reference case (\NOREDUND) and \ac{psnr} gains are shown for thee different levels of redundancy introduced by the proposed method (\DYN).
This study also conducted for the two different cases, with and without \ac{idr} loss.
Table~\ref{tab:AverageQualityRedundancies} presents the average \ac{psnr} of decoded video for different \ac{plr} (columns~4~to~8) and for various percentages of \ac{mv} redundancy (third column). The absolute quality values are shown for the reference case (\NOREDUND) and \ac{psnr} gains are shown for three different levels of redundancy introduced by the proposed method (\DYN).
This study was also conducted for the two different cases, with and without \ac{idr} loss.
It can be confirmed from the results of Table~\ref{tab:AverageQualityRedundancies} that, for each sequence, as $\alpha$ decreases more redundant \acs{mv} are used leading to higher overhead rations. This results in higher objective video quality for all tested sequences.
It can be confirmed from the results of Table~\ref{tab:AverageQualityRedundancies} that, for each sequence, as $\alpha$ decreases more redundant \acs{mv} are used, leading to higher overhead rations. This results in higher objective video quality for all tested sequences.
This confirms that redundant \acs{mv} are able to minimise the error propagation, which is accomplished at a cost of an acceptable increase in the bitrate, since only a sub-set of \acs{mv} is transmitted.
The use of redundant \acs{mv} yields better video quality for \ac{plr} as low as $1\%$, achieving higher quality gains when \ac{idr} frames are not lost (\KEEPINTRA). This is because of the higher quality degradation when \ac{idr} frames are lost, which is not due to mismatched \ac{mv} predictions and thus cannot be reduced by transmitting extra \acs{mv} as side-information.
The use the proposed method achieves higher quality for sequences with higher motion activity and which makes use of inter-prediction more often, i.e., BQSquare and Traffic sequences (see inter ratio in Table~\ref{TableTestSequences}), leading to a maximum gain of $3.93$~dB for \LOSTALL ($7.82$~dB for \KEEPINTRA).
The use of redundant \acs{mv} yields better video quality for \ac{plr} as low as $1\%$, achieving higher quality gains when \ac{idr} frames are not lost (\KEEPINTRA). This is because of the higher quality degradation when \ac{idr} frames are lost, which is not due to mismatched \ac{mv} predictions, and thus cannot be reduced by transmitting extra \acs{mv} as side-information.
The use the proposed method also achieves higher quality for sequences with higher motion activity that makes use of inter-prediction more often, i.e., BQSquare and Traffic sequences (see Inter ratio in Table~\ref{TableTestSequences}), leading to a maximum gain of $3.93$~dB for \LOSTALL ($7.82$~dB for \KEEPINTRA).
\begin{table}[!t]
\centering
@@ -508,12 +513,12 @@ Thus, using the proposed two-stage approach to recover temporal predictions cons
In this chapter a two-stage approach was proposed to increase the error robustness of \ac{hevc} streaming and reduce the error propagation in case of packet loss.
A constrained coding approach was devised to select reference frames and dynamically distribute temporal dependencies.
This is jointly used with a controlled amount of side information in coded streams, comprising a small set of the most relevant \acs{mv}, to minimise \ac{mv} mismatch at the decoder in the presence of frame loss.
This is jointly used with a controlled amount of side information in coded streams, comprising a small set of the most relevant \acs{mv}, in order to minimise \ac{mv} mismatch at the decoder in the presence of frame loss.
As can be concluded from the results, this method contributes to reduce the temporal dependencies with consistent quality gains for different \ac{plr} and coding configurations, only incurring in a small drop of coding efficiency.
Overall, the proposed approach is an effective method for coping with video transmission over error-prone networks.
Overall, the proposed approach is an effective method to cope with video transmission over error-prone networks.
\todo{Papers here??}