Review thesis after PA comments

This commit is contained in:
jfmcarreira committed 2018-09-17 23:11:37 +01:00
1 parent 2a97ab6c09
commit 9bdc2ee860
24 files changed
+798 -749

No files matched your search

@@ -3,7 +3,7 @@
\graphicspath{{\ImprovedErrorResilience/Results/}}
\chapter{A Two-stage Approach for Robust Coding and Streaming \VERSION{2}}
\chapter{A two-stage approach for robust coding and streaming \VERSION{2}}
% \chapter{Error Resilience Techniques for \ac{hevc} Standard}
%\chapter{Two-stage Robust Video Coding}
\label{Chapter:ErrorResilienceTechniques}
@@ -13,19 +13,15 @@
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
\glsreset{pu}
\glsreset{ctu}
\glsreset{mv}
\glsreset{rd}
\glsreset{pu}
\glsreset{mv}
\glsreset{ra}
\glsreset{ld}
This chapter proposes a new two-stage approach to improve the error robustness of \ac{hevc} streaming, by reducing temporal error propagation in case of frame loss.
This is accomplished by reducing the prediction mismatch that occurs at the decoder after frame loss.
The proposed approach comprises the following two stages: (i) at the encoding stage, the reference pictures are dynamically selected based on constraining conditions and Lagrangian optimisation, which distributes the use of reference pictures throughout each frame, reducing the number of \ac{pu} that depend on a single reference; (ii) at the streaming stage, a \ac{mv} prioritisation algorithm, based on spatial dependencies, selects an optimal sub-set of \acs{mv} to be transmitted as side redundant information to reduce mismatched \ac{mv} predictions at the decoder.
%The increased compression ratios achieved by the \ac{hevc} standard lead to reduced robustness of coded streams, with increased susceptibility to network errors and consequent video quality degradation, as shown in Chapter~\ref{Chapter:HEVCErrorResilience}.
This chapter proposes a coding method, based on a two-stage approach, to improve the error robustness of \ac{hevc} streaming, by reducing temporal error propagation in case of frame loss.
This is accomplished by reducing the prediction mismatch that occurs at the decoder after frame loss, through the following two stages: (i) at the encoding stage, the reference pictures are dynamically selected based on constraining conditions and Lagrangian optimisation, which distributes the use of reference pictures throughout each frame, by reducing the number of \acs{pu} that depend on a single reference; (ii) at the streaming stage, a \ac{mv} prioritisation algorithm, based on spatial dependencies, selects an optimal sub-set of \acs{mv} to be transmitted, redundantly, as side information to reduce mismatched \ac{mv} predictions at the decoder.
This chapter is organized as follows. Section~\ref{Section4:Problem} presents the problem definition and motivation for the method proposed in this chapter. Section~\ref{Section4:Architecture} provides an overview of the proposed method, describing the functional blocks that compose the two-stage approach for robust coding and streaming. Section~\ref{Section4:RefPicSel} describes the encoding stage of the proposed method, that is used to select the optimal reference picture for inter-prediction. Section~\ref{Section4:MVSelection} describes the streaming stage where a sub-set of \acs{mv} are selected to reduce the mismatch motion predictions. Section~\ref{Section4:Performance} provides a performance evaluation of the proposed method, illustrating the advantages of each stage and how they are combined to achieve superior error robustness in \ac{hevc} standard. Finally, Section~\ref{Section4:Summary} concludes the chapter.
This chapter is organized as follows. Section~\ref{Section4:Problem} presents the problem definition and motivation for the method proposed in the chapter. Section~\ref{Section4:Architecture} provides an overview of the proposed method, describing the functional blocks that compose the two-stage approach for robust coding and streaming. Section~\ref{Section4:RefPicSel} describes the encoding stage of the proposed method, that is used to select the optimal reference picture for inter-prediction. Section~\ref{Section4:MVSelection} describes the streaming stage where a sub-set of \acs{mv} are selected to reduce the mismatch motion predictions. Section~\ref{Section4:Performance} provides a performance evaluation of the proposed method, illustrating the advantages of each stage and how they are combined to achieve superior error robustness in \ac{hevc} standard. Finally, Section~\ref{Section4:Summary} concludes the chapter.
\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -34,9 +30,18 @@ This chapter is organized as follows. Section~\ref{Section4:Problem} presents th
\section{Problem definition}
\label{Section4:Problem}
\glsreset{rd}
\glsreset{ec}
\glsreset{pu}
\glsreset{ctu}
\glsreset{cu}
\glsreset{mv}
As previously mentioned,\ac{hevc} bitstreams subject to transmission losses result in significant degradation of both objective and subjective quality.
This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes, which are selected according to a \ac{rd} optimisation criteria that assumes error-free transmission~\cite{ROPE2007}.
\glsreset{ra}
\glsreset{ld}
As previously mentioned, when\ac{hevc} bitstreams are subjected to transmission losses, the decoded video signal incurs in significant degradation of both objective and subjective quality.
This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes, which are selected according to \ac{rd} optimisation criteria, which always assume error-free transmission~\cite{ROPE2007}.
Therefore, high compression efficiency is achieved, but resilience to transmission errors is penalised.
\begin{figure}[tb]
@@ -46,21 +51,22 @@ Therefore, high compression efficiency is achieved, but resilience to transmissi
\label{FIG4:ReferenceUsageRatio}
\end{figure}
Highly efficient video encoders support the use of multiple reference frames for inter-frame prediction, which are selected based on a \ac{rd} cost that may lead to an unbalance distribution.
In order to evaluate the reference frame selection method attained by the \ac{rd} implemented in the reference \ac{hevc} encoder, different test sequences are encoded using two reference frames, referred to as $r_1$ and $r_2$, and its selection ratios are analysed. In this study, reference $r_1$ corresponds to the closest neighbouring frame and reference $r_2$ the second closest.
Highly efficient video encoders support the use of multiple reference frames for inter-frame prediction, which are selected based on minimisation of \ac{rd} cost that may lead to an unbalanced use of various reference frames.
In order to analyse the distribution of the reference frames selected by the \ac{rd} decision method used in the reference \ac{hevc} encoder, different test sequences were encoded using two reference frames, referred to as $r_1$ and $r_2$, and their selection ratios were analysed.
In this study, reference $r_1$ corresponds to the closest neighbouring frame and reference $r_2$ the second closest.
Figure~\ref{FIG4:ReferenceUsageRatio} illustrates the average usage percentage of reference $r_1$ for different sequences.
The remaining corresponds to those cases where reference $r_2$ is selected.
These results show that, in general, reference $r_1$ is selected more often than $r_2$.
This is mainly true for sequences with higher motion.
For those sequences with lower motion, but high spatial complexity, e.g., BQSquare (see \ac{si} in Table~\ref{TableTestSequences}), both reference frames are equally (approximately) selected for temporal predictions, with $r_1$ being used by 58\% of cases and $r_2$ by 42\%.
This reveals that even with a configuration using only two references the encoder mainly uses the temporally closer one as the reference for inter-frame predictions.
This reveals that, even though two references are used, the encoder mainly uses the temporally closer one as the reference for inter-frame predictions.
Therefore, if such reference frame is lost, a higher number of mismatched predictions will occur, inevitabely leading to higher error propagation.
Another technique that contributes to increased error propagation is the use of predictions in \ac{mv} coding~\cite{MotionCodingHEVC2011}, which contributes to increase the spatial and temporal dependencies.
Although the spatial \ac{mv} candidates are used more often, temporally predicted \acs{mv} are responsible for a greater impact on error propagation, as they propagate errors in the \ac{mv} decoding across different time instances.
Another coding technique that contributes to increase error propagation is the use of predictions in \ac{mv} coding~\cite{MotionCodingHEVC2011}, which also contributes to increase the total amount of spatial and temporal dependencies.
As shown in Chapter~\ref{Chapter:HEVCErrorResilience}, temporally predicted \acs{mv} are responsible for a greater impact on error propagation, as they propagate errors in the \ac{mv} decoding across different time instances.
To overcome these issues, in the next sections a two-fold approach is described to increase the error robustness in video streaming by jointly considering two different stages: (i) while encoding live video, by selecting robust coding modes and (ii) after encoding, by adding robust features to the compressed streams. While in the former this is accomplished by optimising coding parameters and decisions to increase robustness, in the latter some type of compressed domain processing is used to increase error robustness of pre-encoded streams, e.g., error resilient transcoding. Inevitably, in both cases this is achieved at the expense of some loss in coding efficiency, therefore the challenge is to find an optimum trade-off between robustness and bitrate overhead.
%In this chapter a two-fold approach is used to overcome the two problems mentioned above.
To overcome these issues a two-fold approach was devised to increase the error robustness in video streaming by jointly considering two different stages: (i) while encoding live video, by selecting robust coding modes and (ii) after encoding, by adding robust features to the compressed streams. While in the former this is accomplished by optimising coding parameters and decisions to increase robustness, in the latter some type of compressed domain processing is used to increase error robustness of pre-encoded streams, e.g., error resilient transcoding. Inevitably, in both cases this is achieved at the expense of some loss in coding efficiency, therefore the challenge is to find an optimum trade-off between robustness and bitrate overhead.
The proposed method is described in the remaining of this chapter.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
@@ -68,8 +74,8 @@ To overcome these issues, in the next sections a two-fold approach is described
\section{Proposed two-stage architecture}
\label{Section4:Architecture}
The goal of the proposed method is to reduce the error propagation in decoded video in case of frame loss.
The underlying idea is to constrain inter-frame prediction dependencies, in order to minimise the distortion $D_t$ given by
The goal of the proposed method is to reduce the error propagation in decoded video when the received stream was incurred in frame loss.
The underlying idea is to constrain inter-frame prediction dependencies, in order to minimise the frame distortion $D_t$ given by
\begin{equation}
D_t = \tilde f_t - \hat f_t, \text{when } \tilde f_{t-1} \neq \hat f_{t-1},
\label{eq1}
@@ -91,23 +97,23 @@ Note that the streaming stage is an independent and complementary subsystem of t
\end{figure}
In case of frame loss, these two stages have the common objective of helping the decoder to recover from mismatched predictions at the expense of a small increase in bitrate.
This is equivalent to reduce the error propagation by using two techniques: (i) reducing the mismatch of inter-frame predictions, by encoding each \ac{pu} from a different reference frame and (ii) reducing the mismatch of \ac{mv} predictions by sending the most important \acs{mv} as side information. These are the two stages of the proposed method described in the following sections.
This is equivalent to reduce the error propagation by means of two different mechanisms: (i) reducing the mismatch of inter-frame predictions, by encoding each \ac{pu} from a different reference frame and (ii) reducing the mismatch of \ac{mv} predictions by sending the most important \acs{mv} as side information. These are the two stages of the proposed method described in the following sections.
\section{Encoding stage: Reference frame selection}
\section{Encoding stage: reference frame selection}
\label{Section4:RefPicSel}
The reference picture selection mechanism attempts to uniformly distribute the use of different reference pictures across the \ac{ctu} of a coded video frame.
In case of frame loss, the error propagation in the decoder is reduced because the number of \acs{ctu} predicted from the lost frame decreases and thus error concealment benefits as more data is correctly decoded from other reference frames.
In case of frame loss, the error propagation in the decoder is reduced because the number of \acs{ctu} predicted from the lost frame decreases and thus \ac{ec} benefits as more data is correctly decoded from other reference frames.
The underlying principle of this method is to minimise the probability of using the same reference picture for neighbouring \acs{ctu}.
%For instance,
In a coding structure with two reference frames, as shown in Figure~\ref{fig:checkerboardRefSelection}, the reference frames $r_{1}$ and $r_{2}$ are used to predict the same number of \acs{ctu} in current frame~($C$), resulting in a checkerboard structure, where a different reference frame is used for predicting every other \ac{ctu} within the current frame.
If each reference frame is used to predict half of the pixels in the current frame, when any of these references is lost, the subsequent predicted frame is only partially distorted by mismatched predictions.
This is because only half of its predictions depend on the missing reference frame, which limits the overall error propagation.
Moreover, since the motion information of the predicted frame is encoded using as reference not only the missing frame but also the past ones, those \acs{mv} that cross over a missing reference frame, pointing to the previous ones, are still useful for reconstructing correct \acs{mv} and also for error concealment.
Moreover, since the motion information of the predicted frame is encoded using, not only the missing frame as reference but also the past ones, those \acs{mv} that cross over a missing reference frame, pointing to the previous ones, are still useful for reconstructing accurate \acs{mv} and also for \ac{ec}.
In the limit, a straightforward implementation of the previous example leads to a simple interleaved uniform distribution of the reference frames over all \acs{ctu} in the current frame, forcing the checkerboard structure shown in Figure~\ref{fig:checkerboardRefSelection}. In the case of two reference frames, 50\% of the \acs{ctu} would use one reference frame, while the other 50\% would use the other reference.
However, such solution would not take into consideration coding efficiency, since the selection process of the reference frame would not be \ac{rd} optimised for each \ac{pu} within a \ac{ctu}.
@@ -215,7 +221,8 @@ Figure~\ref{fig:mvdependencies} illustrates how \acs{mv} have different importan
In this figure, each block corresponds to a \ac{pu}, which has an associated \ac{mv}. The arrows point to the \ac{pu} containing the \ac{mv} used as predictor. In the current frame ($f_0$), two \acs{pu} have temporally dependent \acs{mv}.
While the \ac{mv} of \ac{pu} \MVDEPA is used to predict a total of four other \acs{mv}, the \ac{mv} of \ac{pu} \MVDEPB is only used to predict one single \ac{mv}.
Thus, in case of data loss (e.g., frame $f_{-1}$), the \ac{mv} of \ac{pu} \MVDEPA has more impact on error propagation to the frame $f_{0}$ and consequently quality degradation than the \ac{mv} of \ac{pu} \MVDEPB, because of its higher number of dependent \acs{mv}.
For this reason, in the \ac{mv} selection process, the proposed method assigns higher importance to the \ac{mv} of \ac{pu} \MVDEPA than \ac{pu} \MVDEPB. Moreover, in Figure~\ref{fig:mvdependencies}, it can be observed that although the number of \acs{mv} predicted from the \acs{mv} of \ac{pu} \MVDEPC is higher than \ac{pu} \MVDEPA (6 and 4 \acs{mv}, respectively),
For this reason, in the \ac{mv} selection process, the proposed method assigns higher importance to the \ac{mv} of \ac{pu} \MVDEPA than \ac{pu} \MVDEPB.
Moreover, in Figure~\ref{fig:mvdependencies} it can be observed that, although the number of \acs{mv} predicted from the \acs{mv} of \ac{pu} \MVDEPC is higher than \ac{pu} \MVDEPA (6 and 4 \acs{mv}, respectively),
the image area that depends on the former is smaller that the latter. This is due to the variable size of \acs{pu}.
Therefore, the image area covered by all dependent \acs{pu} is separately considered in the \ac{mv} selection process.
@@ -231,12 +238,12 @@ Based on the evidence mentioned above, the best \acs{mv} to be encoded as side i
\item
the \acs{mv} with temporal dependencies are firstly selected as the most important set: $V$ = \{\emph{\ac{mv}: \ac{mv} is temporally dependent from others}\}.
\item
the $MVs \in V$ are ranked according to the number of spatial dependencies associated to each one.
the sub-set of $MVs \in V$ are ranked according to the number of spatial dependencies associated to each one.
\item
the optimal sub-set $U \subset V$ is found as the best trade-off between the image area covered by the dependent \acs{pu} and bitrate overhead, i.e., the one that maximises the cost function in Equation (\ref{eq:MVSelectOpt}).
\end{enumerate}
Considering $V$ the ordered set of \acs{mv} encoded for the current frame, i.e., $V=\{MV_1, MV_2, \cdots MV_s\}$ and $U \subset V$ always starting in the first vector, i.e., \mbox{$U_n=\{MV_1, MV_2, \cdots MV_n\}$} with $n\leq s$, the best set of \acs{mv} ($U_n^*)$ is selected based on the optimisation procedure given by Equation (\ref{eq:MVSelectOpt}). Note that any value of $n$ corresponds to a unique subset $U$.
This can be formulated as follows. Considering $V$ the ordered set of \acs{mv} encoded for the current frame, i.e., ${V=\{MV_1, MV_2, \cdots MV_s\}}$ and ${U \subset V}$ always starting in the first vector, i.e., ${U_n=\{MV_1, MV_2, \cdots MV_n\}}$ with $n\leq s$, the best set of \acs{mv} ($U_n^*)$ is selected based on the optimisation procedure given by Equation (\ref{eq:MVSelectOpt}). Note that any value of $n$ corresponds to a unique subset $U$.
\begin{equation}\label{eq:MVSelectOpt}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
@@ -261,13 +268,13 @@ The subset of selected \acs{mv} is independently encoded, ensuring that no refer
\label{Section4:Performance}
In this section the robustness of the proposed two-stage architecture is evaluated and discussed for two cases: (i) only using the encoding stage and (ii) using the streaming stage in addition to the encoding stage. In the former case, identified as ``\PROP'', only the dynamic reference picture selection method described in Section~\ref{Section4:RefPicSel} is evaluated. In the latter case, identified as ``\DYN'', besides the previous method of reference picture selection, redundant \acs{mv} are encoded as described in Section~\ref{Section4:MVSelection}, resulting in a combination of both stages.
In this section the performance of the proposed two-stage architecture is evaluated and discussed for two cases: (i) only using the encoding stage and (ii) using the streaming stage in addition to the encoding stage. In the former case, identified as ``\PROP'', only the dynamic reference picture selection method described in Section~\ref{Section4:RefPicSel} is evaluated. In the latter case, identified as ``\DYN'', besides the previous method of reference picture selection, redundant \acs{mv} are encoded as described in Section~\ref{Section4:MVSelection}, resulting in a combination of both stages.
As references for comparison, three other methods are used: (i) a fixed checkerboard approach (\CHKB), (ii) the reference picture generation method based on leaky predictions proposed in \cite{ROPERefGenJan2010}, identified as ``\AVG'', and (iii) long-term reference frames (\LONG). The \LONG method only uses the key frames as reference, i.e., in the \ac{ra} configuration only one out of eight frames is used as reference for prediction, allowing a more robust transmission, as the reference frames are less likely to be hit by errors.
The experimental setup used in this section is the same as the one previously described in Section~\ref{Section3:ErrorResilienceEvaluation}.
Two test cases were considered for simulation: (i) in \emph{\LOSTALL} all frames may be equally affected by packet loss, including \ac{idr} frames; (ii) in \emph{\KEEPINTRA} \ac{idr} frames are assumed to be prioritised in the network and delivered error-free, or using some combined approach as recently proposed in \cite{ErrorResilienceHeterogenousNetworks2016}.
At the decoder-side, error concealment is based on motion compensation from previously decoded frames. The motion field of a missing frame is obtained applying \ac{mv} extrapolation from the two closest neighbours. Afterwards, the final \acs{mv} are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}.
At the decoder-side, \ac{ec} is based on motion compensation from previously decoded frames. The motion field of a missing frame is obtained applying \ac{mv} extrapolation from the two closest neighbours. Afterwards, the final \acs{mv} are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}.
The evaluation of the encoding-stage is carried out separately from the streaming-stage.
@@ -286,7 +293,7 @@ The evaluation of the encoding-stage is carried out separately from the streamin
The performance of the encoding stage is evaluated on its own, in order to only determine the improvements achieved by the dynamic reference frame selection method.
In this case, the reference frames were reconstructed with distortion, as a result of non-perfect error concealment.
In this case, the reference frames were reconstructed with distortion, as a result of non-perfect recovery.
Since the encoding stage does not deal with errors in temporal \ac{mv} predictions (this is handled by the streaming-stage), in this evaluation the temporal \ac{mv} predictor is disabled.
This is necessary to avoid masking the results with the errors caused by wrong \ac{mv} predictions.
@@ -318,8 +325,11 @@ This value was chosen to allow increasing the Lagrangian cost by about $50\%$,
The error robustness achieved by the encoding-stage was firstly evaluated in regards to the error propagation, resulting from a single loss event in the \ac{ld} configuration.
Figure~\ref{fig:RecoveredExample} shows the visual impact of error propagation on a region of the decoded Frame \#10 when Frame \#9 is affected by errors, for Basketball Drill and Race Horses sequences (the \ac{psnr} corresponds to the entire frame).
The proposed method is compared with the method~in~\cite{ROPERefGenJan2010}, as both aim at reducing the error propagation.
The method~in~\cite{ROPERefGenJan2010} uses filtered frames as reference for predictions, therefore, in case of errors, the effects of mismatch predictions are significantly reduced by the filtering operation (see Sub-Section~\ref{SECTION2:LeakyPred}).
As shown in this figure, the proposed method is able to reduce the impact of mismatch decoding in comparison with the method~in~\cite{ROPERefGenJan2010}. Although both methods still reveal some predictions mismatch leading to inaccurate reconstructed of the objects contours, the proposed method is able to achieve a smoother reconstruction, which leads to higher quality in terms of \ac{psnr}. This is because, in frame \#10, the predictions depending on the missing frame \#9 are not used throughout the entire frame, thus, approximately half of the predictions are not affected by errors. In contrast, the method~in~\cite{ROPERefGenJan2010} requires several filtering operations to be applied (i.e, different reference frames need to be filtered) in order to reduce the effect of mismatch predictions.
The method~in~\cite{ROPERefGenJan2010} uses filtered frames as reference for predictions, therefore, in case of errors, the effects of mismatched predictions are significantly reduced by the filtering operation (see Sub-Section~\ref{SECTION2:LeakyPred}).
As shown in Figure~\ref{fig:RecoveredExample}, the proposed method is able to reduce the impact of mismatch decoding in comparison with the method~in~\cite{ROPERefGenJan2010}.
Although both methods still reveal some predictions mismatch leading to inaccurate reconstructed of the objects contours, the proposed method is able to achieve a smoother reconstruction, which leads to higher quality in terms of \ac{psnr}.
This is because, in frame \#10, these predictions that depend on the missing frame \#9 are not used throughout the entire frame, thus, approximately half of the predictions are not affected by errors.
In contrast, the method~in~\cite{ROPERefGenJan2010} requires several filtering operations to be applied (i.e, different reference frames need to be filtered) in order to reduce the effect of mismatched predictions.
Summarising, the reconstructed frames illustrated in Figure~\ref{fig:RecoveredExample} reveal the higher efficiency achieved by distributing the use of the references frames.
@@ -335,7 +345,7 @@ Summarising, the reconstructed frames illustrated in Figure~\ref{fig:RecoveredEx
Figure~\ref{fig:PSNRvsFrameForRefPicSel}~shows the \ac{psnr} over a \ac{gop} with an \ac{idr} period of 32 frames, using the same constant bitrate for all streams.
This figure shows that all methods are able to outperform the reference \ac{hevc} (\REF), gaining approximately $3$~dB in reconstruction quality.
The similar quality levels achieved by \CHKB and \PROP reveal that the dynamic selection of the reference frames used in \PROP is more robust because a better trade-off between coding efficiency and error resilience is attained.
The similar quality levels achieved by \CHKB and \PROP reveal that the dynamic selection of the reference frames used in \PROP is more robust, because a better trade-off between coding efficiency and error resilience is attained.
Moreover, results in Figure~\ref{fig:PSNRvsFrameForRefPicSel} are in line with the reconstruction quality shown in Figure~\ref{fig:RecoveredExample}, confirming that the proposed method is able to outperform the reference frame generation technique in~\cite{ROPERefGenJan2010}.
%Since the proposed method is more efficient than the others for error-free transmission (see~Table~\ref{tab:bd_error_free}), and it also provides better error robustness, it can be considered as an efficient solution to better deal with transmission errors.
Overall, the method proposed for the encoding stage is able to increase the error robustness of \ac{hevc} without severely compromising the coding efficiency.
@@ -352,7 +362,8 @@ In \cite{ROPERefGenJan2010}, the use of interpolated references in the encoding
The proposed method is also able to outperform the \ac{hevc} reference software when an hierarchical coding structure is used (see results for \ac{ra} configuration in Table~\ref{tab:AverageQualityCHKBEXPRandomAccess}), which implicitly limits the error propagation.
In comparison to the reference \ac{hevc}, quality improvements up to $1.42$~dB at PLR=$10\%$ are obtained for Kendo sequence.%, while the average gain across all test sequences is 1~dB.
When comparing results for both test cases, it is noticeable that the proposed method is able to outperform the reference methods for both cases, when \ac{idr} frame loss is allowed or not (\LOSTALL and \KEEPINTRA, respectively). As the \ac{plr} increases, the gains of the proposed method decrease in comparison with the case where the \ac{idr} frames are delivered error-free (i.e., case \KEEPINTRA), due to the higher error propagation of the missing \ac{idr} frame. Moreover, the quality gains are lower in the \LOSTALL case when comparing with \KEEPINTRA for the same \ac{plr}, for both the \ac{ld} and \ac{ra}.
When comparing results for both test cases, it is noticeable that the proposed method is able to outperform the reference methods for both cases, regardless whether \ac{idr} frame loss is allowed or not (\LOSTALL and \KEEPINTRA, respectively).
As the \ac{plr} increases, the gains of the proposed method decrease in comparison with the case where the \ac{idr} frames are delivered error-free (i.e., case \KEEPINTRA), due to the higher error propagation of the missing \ac{idr} frame. Moreover, the quality gains are lower in the \LOSTALL case when comparing with \KEEPINTRA for the same \ac{plr}, for both the \ac{ld} and \ac{ra}.
Taking into account the above results and the spatio-temporal information shown in Table~\ref{TableTestSequences}, one can observe that the performance of the proposed method is lower for sequences with high spatial details and low temporal activity (e.g., BQSquare and Park Scene), which indicates higher effectiveness for video content with higher motion activity.
%\HIGHLIGHT{complete with analysis of both cases}
@@ -411,9 +422,11 @@ Table~\ref{tab:bd_error_free} shows the Bjontegaard's Delta PSNR (BD-PSNR) valu
\end{table}
The results in Table~\ref{tab:bd_error_free} indicate a small loss of coding efficiency of the proposed method, as well as, the reference methods tested.
The fixed checkerboard approach leads to an average quality reduction of $1.22$~dB for the \ac{ld} ($0.32$~dB for \ac{ra}), while the proposed method only loses $0.82$~dB and $0.24$~dB in the \ac{ld} and \ac{ra} configurations, respectively. This results clearly shows that dynamically selecting the reference frames is more efficient that using a fixed pattern, confirming the previous results for error-prone conditions.
Moreover, in comparison to \LONG and the method in~\cite{ROPERefGenJan2010}, the proposed method is still more efficient, consistently achieving better results. Comparing, the results for both error-free and error-prone conditions, one may conclude that the proposed method not only achieves higher quality gains, but also does reduces the impact on coding efficiency.
Overall, the loss in the quality compared to the reference \ac{hevc} is considered acceptable, given the increase in robustness obtained in lossy transmission demonstrated in the previous sub-sections.
The fixed checkerboard approach leads to an average quality reduction of $1.22$~dB for the \ac{ld} ($0.32$~dB for \ac{ra}), while the proposed method only loses $0.82$~dB and $0.24$~dB in the \ac{ld} and \ac{ra} configurations, respectively.
This results clearly show that dynamically selecting the reference frames is more efficient than using a fixed pattern, confirming the previous results for error-prone conditions.
Moreover, in comparison to \LONG and the method in~\cite{ROPERefGenJan2010}, the proposed method is still more efficient, because it consistently achieves better results.
Comparing, the results for both error-free and error-prone conditions, one may conclude that the proposed method not only achieves higher quality gains, but also reduces the impact on coding efficiency.
Overall, the loss in the quality compared to the reference \ac{hevc} is considered acceptable, given the increase in robustness obtained in lossy transmission.
%As expected, this is due to the sub-optimal \ac{rd} optimisation of the proposed method, because the cost function also includes the cost associated to the choice of the reference frame, in addition to the rate and distortion.
@@ -473,12 +486,12 @@ Results also show that for both approaches the decoded video quality increases a
\includegraphics[width=\CurrFigSize,page=4]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
\includegraphics[width=\CurrFigSize,page=5]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
\includegraphics[width=\CurrFigSize,page=6]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
\caption{Decoded video quality for different amounts of redundancy, selected by using a fixed approach and the proposed method (\DYN).}
\caption{Decoded video quality for different amounts of redundancy, selected by using a fixed approach and the proposed dynamic method (\DYN).}
\label{fig:QualityVsRedundancyMVSelection}
\end{figure}
It is noticeable in Figure~\ref{fig:QualityVsRedundancyMVSelection} that for higher \acs{plr}, the proposed method is able to achieve higher gains.
This reveals that when data loss increases it is more important optimise the amount of \acs{mv} selected per frame, in order to use more \acs{mv} in those frames that have a higher impact on the error propagation.
This reveals that when data loss increases it is more important to optimise the amount of \acs{mv} selected per frame, in order to use more \acs{mv} in those frames that have a higher impact on the error propagation.
For instance, the proposed method is able to gain up to $1.5$~dB for the BQSquare sequence subject to a PLR=10\%.
For lower \acs{plr}, since fewer frames are affected by data losses, the use of a fixed approach achieves similar performance as the proposed method (\DYN).
@@ -489,9 +502,9 @@ Table~\ref{tab:AverageQualityRedundancies} presents the average \ac{psnr} of dec
This study was also conducted for the two different cases, with and without \ac{idr} loss.
It can be confirmed from the results of Table~\ref{tab:AverageQualityRedundancies} that, for each sequence, as $\alpha$ decreases more redundant \acs{mv} are used, leading to higher overhead rations. This results in higher objective video quality for all tested sequences.
This confirms that redundant \acs{mv} are able to minimise the error propagation, which is accomplished at a cost of an acceptable increase in the bitrate, since only a sub-set of \acs{mv} is transmitted.
These results also confirm that redundant \acs{mv} are able to minimise the error propagation, which is accomplished at the cost of an acceptable increase in the bitrate, since only a sub-set of \acs{mv} is transmitted.
The use of redundant \acs{mv} yields better video quality for \ac{plr} as low as $1\%$, achieving higher quality gains when \ac{idr} frames are not lost (\KEEPINTRA). This is because of the higher quality degradation when \ac{idr} frames are lost, which is not due to mismatched \ac{mv} predictions, and thus cannot be reduced by transmitting extra \acs{mv} as side-information.
The use the proposed method also achieves higher quality for sequences with higher motion activity that makes use of inter-prediction more often, i.e., BQSquare and Traffic sequences (see Inter ratio in Table~\ref{TableTestSequences}), leading to a maximum gain of $3.93$~dB for \LOSTALL ($7.82$~dB for \KEEPINTRA).
The use the proposed method also achieves higher quality for sequences with higher motion activity that make use of inter-prediction more often, i.e., BQSquare and Traffic sequences (see Inter ratio in Table~\ref{TableTestSequences}), leading to a maximum gain of $3.93$~dB for \LOSTALL ($7.82$~dB for \KEEPINTRA).
In summary, the proposed method is able to select the most relevant motion \ac{mv} information and when combined with a reference picture selection scheme the problem of mismatch \ac{mv} predictions is mitigated.
Moreover, the dynamic approach devised to select the best \acs{mv} also contributes to increase the overall performance.
@@ -511,7 +524,7 @@ Thus, using the proposed two-stage approach to recover temporal predictions cons
In this chapter a two-stage approach was proposed to increase the error robustness of \ac{hevc} streaming and reduce the error propagation in case of packet loss.
A constrained coding approach was devised to select reference frames and dynamically distribute temporal dependencies.
This is jointly used with a controlled amount of side information in coded streams, comprising a small set of the most relevant \acs{mv}, in order to minimise \ac{mv} mismatch at the decoder in the presence of frame loss. The use of \acs{mv} as side information was firstly proposed and evaluated in the publication~\ref{Conf:MotionVectorRedundancies} and the reference frame selection was further discussed in the publication~\ref{Conf:Chkb}. Subsequently, a wider analysis was carried out and published in \ref{Journal:TwoStage}.
This is jointly used with a controlled amount of side information in coded streams, comprising a small set of the most relevant \acs{mv}, in order to minimise \ac{mv} mismatch at the decoder in the presence of frame loss. The use of \acs{mv} as side information was firstly proposed and evaluated in the publication~(\ref{Conf:MotionVectorRedundancies}) and the reference frame selection was further discussed in the publication~(\ref{Conf:Chkb}). Subsequently, a wider analysis was carried out and published in (\ref{Journal:TwoStage}).
As can be concluded from the results, this method contributes to reduce the temporal dependencies with consistent quality gains for different \ac{plr} and coding configurations, only incurring in a small drop of coding efficiency.
Overall, the proposed approach is an effective method to cope with video transmission over error-prone networks.