diff --git a/000_Preamble/abstract.tex b/000_Preamble/abstract.tex index e4b96c4..370fd2b 100644 --- a/000_Preamble/abstract.tex +++ b/000_Preamble/abstract.tex @@ -6,43 +6,20 @@ \vspace{-5mm} -%This thesis investigates the problem of robust coding and Error Concealment (EC) in High Efficiency Video Coding (HEVC). -%After a review of the current state of the art, a simulation study about error robustness, revealed that the HEVC has weak protection against transmission errors with significant impact on video quality degradation. -%Based on this evidence, the first contribution of this work is a new method to reduce the temporal dependencies between Motion Vectors (MVs), by improving the decoded video quality without compromising the compression efficiency. -% -%The second contribution of this thesis is a two-stage approach for reducing the mismatch of temporal predictions in case of video streams received with errors or lost data. -%At the encoder stage, the reference pictures are dynamically distributed based on a constrained Lagrangian rate-distortion optimization to reduce the number of predictions from a single reference. -%At the streaming stage, a prioritization algorithm, based on spatial dependencies, selects a reduced set of MVs to be transmitted, as side information, to reduce mismatched MV predictions at the decoder. -% -%The problem of EC-aware video coding is also investigated to enhance the overall error robustness. -%A new approach based on scalable coding and optimally EC selection is proposed, where the optimal EC modes are found by simulating transmission losses, followed by a saliency-weighted optimisation. -%Moreover, EC residual information is encoded using a rate-controlled enhancement layer. -%Both are transmitted to the decoder to be used in case of data loss. -% -%Finally, an adaptive error resilience scheme is proposed to dynamically predict the video stream that achieves the highest decoded quality for a particular loss case. -%A neural network selects among the various video streams, encoded with different levels of compression efficiency and error protection, based on information from the video signal, the coded stream and the transmission network. -% -%Overall, the new robust video coding methods investigated in this thesis yield consistent quality gains in comparison with other existing methods and also the ones implemented in the HEVC reference software. -%Furthermore, the trade-off between coding efficiency and error robustness is also better in the proposed methods. - -%\vspace{2mm} -%\textbf{Keywords:} 3D video, frame loss error concealment, subjective quality assessment. - - This thesis investigates the problem of robust coding and error concealment in High Efficiency Video Coding (HEVC). -After a review of the current state of the art, a simulation study about error robustness, revealed that the HEVC has weak protection against transmission errors with significant impact on video quality degradation. +After a review of the current state of the art, a simulation study about error robustness, revealed that the HEVC has weak protection against network losses with significant impact on video quality degradation. Based on this evidence, the first contribution of this work is a new method to reduce the temporal dependencies between motion vectors, by improving the decoded video quality without compromising the compression efficiency. The second contribution of this thesis is a two-stage approach for reducing the mismatch of temporal predictions in case of video streams received with errors or lost data. -At the encoding stage, the reference pictures are dynamically distributed based on a constrained Lagrangian rate-distortion optimization to reduce the number of predictions from a single reference. -At the streaming stage, a prioritization algorithm, based on spatial dependencies, selects a reduced set of motion vectors to be transmitted, as side information, to reduce mismatched motion predictions at the decoder. +At the encoding stage, the reference pictures are dynamically distributed based on a constrained Lagrangian rate-distortion optimization to reduce the number of predictions from a single reference. +At the streaming stage, a prioritization algorithm, based on spatial dependencies, selects a reduced set of motion vectors to be transmitted, as side information, to reduce mismatched motion predictions at the decoder. The problem of error concealment-aware video coding is also investigated to enhance the overall error robustness. A new approach based on scalable coding and optimally error concealment selection is proposed, where the optimal error concealment modes are found by simulating transmission losses, followed by a saliency-weighted optimisation. Moreover, recovery residual information is encoded using a rate-controlled enhancement layer. Both are transmitted to the decoder to be used in case of data loss. -Finally, an adaptive error resilience scheme is proposed to dynamically predict the video stream that achieves the highest decoded quality for a particular loss case. +Finally, an adaptive error resilience scheme is proposed to dynamically predict the video stream that achieves the highest decoded quality for a particular loss case. A neural network selects among the various video streams, encoded with different levels of compression efficiency and error protection, based on information from the video signal, the coded stream and the transmission network. Overall, the new robust video coding methods investigated in this thesis yield consistent quality gains in comparison with other existing methods and also the ones implemented in the HEVC reference software. diff --git a/010_Introduction/Introduction.tex b/010_Introduction/Introduction.tex index c095de6..bb4177f 100644 --- a/010_Introduction/Introduction.tex +++ b/010_Introduction/Introduction.tex @@ -19,7 +19,6 @@ To cope with such demanding requirements a new coding standard was developed by The \ac{hevc} is currently the main video coding framework to extend current services towards higher resolution formats delivered with better compression efficiency. Similar to its predecessor, \ac{hevc} is a block-based hybrid video codec using predictive, transform and entropy coding. The coding flexibility and efficiency of \ac{hevc} results from the use of new coding tools and block structures, which enables new block partitions\cite{HEVCBlock2012}, improved prediction modes \cite{IntraCodingHEVC2012,AMPCodingHEVCJun2012} and new high-level features~\cite{HEVCHighLevelReferenceDez2012}, such as explicit reference picture management and new parameter sets. -%However, it is known that higher coding efficiency leads to lower robustness in the presence of transmission errors and data loss. However, new coding tools of \ac{hevc} not only bring high compression efficiency~\cite{HEVCComparisonDec2012}, but also some other disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and inherent reduction of error robustness~\cite{StudyHEVCErrorsDez2012}. Earlier studies about the error robustness characteristics of \ac{hevc} have shown that, in general, the use of high efficiency coding tools results in poor error resilience performance (score \emph{slightly annoying} in the \ac{mos} scale), in comparison with previous standards~\cite{QoEHEVCMay2014,SubjectiveEvalSep2013}. Thus, when \ac{hevc} streams incur in transmission losses, this leads to significant degradation of both objective and subjective quality. @@ -51,7 +50,7 @@ The following objectives were pursued in this work: \begin{enumerate} \item \textbf{Study and evaluation of error robustness in highly compressed video:} -The objective of this research study is to evaluate the robustness against transmission errors of highly efficiently coded video. +The objective of this research study is to evaluate the robustness against network losses of highly efficiently coded video. The study and its conclusions aim at finding relevant research directions to develop new methods for increasing the error robustness of coded streams. The results and conclusions of this study were published in conference paper (\ref{Conf:ErrorResilience}) and in journal paper (\ref{Journal:TwoStage}) (see Appendix~\ref{Appen:Papers}). diff --git a/021_VideoCoding/VideoCodingReview.tex b/021_VideoCoding/VideoCodingReview.tex index 43b436f..920d435 100644 --- a/021_VideoCoding/VideoCodingReview.tex +++ b/021_VideoCoding/VideoCodingReview.tex @@ -672,7 +672,7 @@ At the high-level, \ac{hevc} introduces new \ac{nal} types and also new pictures These allow the implementation of various random-access strategies, more flexibility and higher coding efficiency than its predecessor \ac{h264}. Another major improvement comes from the inter-frame prediction. On the one hand new high-level reference picture management was introduced, which increases the error robustness, as it explicitly defines the active reference frames and the \ac{dpb} status. On the other hand it introduces the \ac{amvp} and the Merge mode, which efficiently reduce the required bits for coding the motion information. The newly developed coding tools and prediction modes described in this chapter have resulted in approximately 50\% of bitrate savings for the same subjective quality in comparison with the previous standard. -However, such high coding efficiency implies higher data dependencies, which are more sensible to data loss and increase the error propagation to subsequent coding structures. Therefore, evaluating the impact of transmission errors and error propagation in \ac{hevc} is of most importance to develop efficient techniques to recover from data loss and reduce the error propagation. +However, such high coding efficiency implies higher data dependencies, which are more sensible to data loss and increase the error propagation to subsequent coding structures. Therefore, evaluating the impact of network errors and error propagation in \ac{hevc} is of most importance to develop efficient techniques to recover from data loss and reduce the error propagation. diff --git a/022_ErrorResilience/RobustTransmissionReview.tex b/022_ErrorResilience/RobustTransmissionReview.tex index 4be3257..c2f3033 100644 --- a/022_ErrorResilience/RobustTransmissionReview.tex +++ b/022_ErrorResilience/RobustTransmissionReview.tex @@ -9,7 +9,7 @@ -In general, digital communications through networks are prone to transmission errors or data loss. +In general, digital communications through networks are prone to errors or data loss. These errors can be handled in different ways, such as retransmission, error correction, and \ac{ec} \cite{ErrorResilience2000,PixelInterlacingIntraConcealment2011}. This chapter reviews the existing techniques for robust video coding that can be applied at the \ac{vcl} to reduce the error propagation and increase the decoded video quality. Several techniques have been proposed in the past to deal with the problem of robust video transmission, especially for video coding standards with high temporal dependencies, such as \ac{h264} and more recently \ac{hevc}. @@ -23,7 +23,7 @@ Section~\ref{Section22:Localisation} describes the localisation techniques, whic Section~\ref{Section22:DataPartitionRedundant} provides an overview on methods based on data partition, classification and bitstream redundancy. %These techniques ease the introduction of extra information in order to recover from lost data. Section~\ref{Section22:Concealment} covers the techniques used at the decoder-side, as post-processing, to reduce the effect of errors and improve visual quality and smoothness. -Section~\ref{Section22:ConcealmentAware} provides a description of methods that use \ac{ec} techniques to cope with transmission errors and improve error robustness. +Section~\ref{Section22:ConcealmentAware} provides a description of methods that use \ac{ec} techniques to cope with packet loss and improve error robustness. Finally, Section~\ref{Section22:Summary} concludes the chapter. % by providing a overall comparison between the different approaches. @@ -133,11 +133,11 @@ The results obtained in \cite{ROPEBurst2009} revealed that higher video quality In \cite{E2EDistortion2011} temporal \ac{ec} is considered in the distortion estimation, which increases the performance of the proposed method, by avoiding frame-copy that provides a lower bound on reconstruction quality. More advanced temporal \ac{ec} methods were also considered in the work presented in~\cite{EndToEndDistortionConcealment2014}. The \acs{mv} used for motion compensation reconstruction are obtained using boundary matching search, which is able to achieve lower distortion when compared with frame-copy. Using more advanced \ac{ec} algorithms leads to a more accurate estimation of the end-to-end distortion, thus increasing the error robustness of the transmitted video. -To take into account the delay introduced in the encoding loop due to hierarchical prediction structures, delay-based \ac{rd} optimisation was proposed in \cite{DelayRD_TCVT2015} by estimating both the distortion and the end-to-end delay that is minimised along with the required bitrate. +To take into account the delay introduced in the encoding loop due to hierarchical prediction structures, delay-based \ac{rd} optimisation was proposed in \cite{DelayRD_TCVT2015} by estimating both the distortion and the end-to-end delay that is minimised along with the required bitrate. Using this approach, the total delay can be controlled without severely compromising the coding efficiency. In the context of \ac{hevc}, end-to-end distortion estimation was also used in the \ac{rd} optimisation showing significant quality improvements when compared to with the reference implementation \cite{ErrorResilienceModeHEVC2016}. -Different quality metrics can also be used for end-to-end distortion estimation. +Different quality metrics can also be used for end-to-end distortion estimation. The method presented in \cite{SSIMModeling2015} proposes to estimate the distortion by using the \ac{ssim} \cite{QualitySSIMRef2002} in order to increase the correlation between the estimated distortion and the perceptual quality noticed by the human visual system~\cite{VisualQualityMetrics2011}. Moreover, the \ac{ssim} quality metric was also used in learning-based \ac{rd} models~\cite{RateDistortionNN_VCIP2017}, using a \ac{cnn} to predict the end-to-end distortion and the required rate, showing good accuracy in the \ac{rd} cost estimation. @@ -182,7 +182,7 @@ A more recent approach was proposed using a perceptual noticeable difference mod This method is able to improve the results of a previous work \cite{IntraRefresh2005}, especially for high motion sequences. This is due to the fact that such previous approach only used an estimation of the error propagation without accounting for perceptual information. A similar approach was also exploited for \ac{hevc} in \cite{HEVCROIErrorResilience2015,HEVCRoiIntraRefresh_DSP2016}, by using a feedback channel to provide information about the network errors, which is then used by the encoder to introduce intra-coded blocks into the \ac{roi}. -The simulation results reveal an increase on the video quality when higher number of intra blocks are used. +The simulation results reveal an increase on the video quality when higher number of intra blocks are used. Moreover, applying the proposed method only to the \ac{roi}, the bitrate overhead introduced is lower, leading to higher \ac{rd} gains. @@ -209,8 +209,8 @@ Alternatively, the two previously encoded frames can be used to generate an inte \check f_t^i = \alpha \hat f_t^i + (1- \alpha) \hat f_{t-1}^i. \end{equation} Therefore, it is possible to decrease the error propagation using $\check f_t^i$ as reference frame for prediction, if only one of the original encoded frames is lost (i.e., $\hat f_{t}^i$ or $\hat f_{t-1}^i$). This is due to the fact that $\check f_t^i$ contains information from both $\hat f_{t}^i$ and $\hat f_{t-1}^i$. -In~(\ref{EQ2:LeakyPredDual}) $\hat f_{t}^i$ and $\hat f_{t-1}^i$ always correspond to the encoded frames and the generated frames ($\check f_t^i $) are not used to interpolate subsequent frames. This corresponds to a finite impulse response filter. -Although the use of two reference frames for leaky prediction is an effective way to reduce the error propagation with a small penalty in coding efficiency, the use of interpolated frame obtained using (\ref{EQ2:LeakyPredDual}) still leads to a severe error propagation in case of burst frame loss. +In~(\ref{EQ2:LeakyPredDual}) $\hat f_{t}^i$ and $\hat f_{t-1}^i$ always correspond to the encoded frames and the generated frames ($\check f_t^i $) are not used to interpolate subsequent frames. This corresponds to a finite impulse response filter. +Although the use of two reference frames for leaky prediction is an effective way to reduce the error propagation with a small penalty in coding efficiency, the use of interpolated frame obtained using (\ref{EQ2:LeakyPredDual}) still leads to a severe error propagation in case of burst frame loss. In this case both frames would be affect by packet loss, which would reduce the effectiveness of this method. @@ -290,7 +290,7 @@ In compressed video streams different symbols have different relevance in terms Therefore, there is an unequal contribution of different syntax elements to the decoding distortion and error propagation, depending on the symbols that are lost in each case. Data partition techniques have been developed to provide a mechanism to cope with the unequal importance of coded information by grouping different parts of a compressed bitstream, according to their importance to the decoded quality. Thus, some of these groups can be protected more efficiently than others using \ac{uep} schemes. -These are normally associated with \ac{fec} algorithms, which are typically applied at the channel level which is not aware of the video content, leading to a sub-optimal error protection~\cite{UEPMotion_TM2006}. +These are normally associated with \ac{fec} algorithms, which are typically applied at the channel level which is not aware of the video content, leading to a sub-optimal error protection~\cite{UEPMotion_TM2006}. Using \ac{uep}, a source-channel optimisation is applied, whereby multi-levels of \ac{fec} are used to achieve different levels of error protection, overcoming this problem. This is presented and discussed with more detail in Sub-section~\ref{SEC22:UEP} @@ -311,7 +311,7 @@ Then, it was further improved in \ac{h264}, allowing the symbols in each slice t \item Type C: inter-coded information and inter coefficients. This type also requires the information of type A, but can be decoded without intra-coded information (type B). \end{itemize} These partitions have different levels of importance, according to the type of information carried in each one. -Data partitions of type A are the most important as they carry the header information that is required to decode the remaining partitions in the video stream. +Data partitions of type A are the most important as they carry the header information that is required to decode the remaining partitions in the video stream. Then, since partitions of type B carry intra-coded information, they are more important than type C, because they can stop temporal error propagation. An error resilience tool named \ac{fmo} technique was introduced in \ac{h264}, allowing each \ac{mb} to be freely assigned to a specific slice group (i.e., a set of slices), organised in a mixed-up fashion~\cite{H264FMO2006}. @@ -325,8 +325,8 @@ Figure~\ref{fig2:FMOTypes} shows \ac{fmo} types 0 to 5 only, since the type 6 ha In the \ac{fmo} type 0 each slice group is defined by a maximum number of \acs{mb}, and a new slice group begins whenever his maximum value is reached. The same slice group can be cyclic repeated within a frame. \ac{fmo} type 1 corresponds to scattered slices, where the \acs{mb} are assigned to each slice group with a fixed formula. Figure~\ref{fig2:FMOTypes} represents the \ac{fmo} type 1 with two slice groups resulting in a checkerboard pattern. In the \ac{fmo} type 2, slice groups are defined by rectangular shapes bounded by the top-left and bottom-right corners. -Although the slice groups are allowed to overlap each other, each \ac{mb} may only belong to one slice group. -Similar to \ac{fmo} type 0, in the types 3, 4 and 5 in Figure~\ref{fig2:FMOTypes}, slice groups are defined by a maximum number of \acs{mb}. +Although the slice groups are allowed to overlap each other, each \ac{mb} may only belong to one slice group. +Similar to \ac{fmo} type 0, in the types 3, 4 and 5 in Figure~\ref{fig2:FMOTypes}, slice groups are defined by a maximum number of \acs{mb}. However, for these types the configuration is allowed to change for every frame, contrary of \ac{fmo} type 0. In order to keep the signalling overhead affordable, only two slice groups are allowed in these types. The \ac{fmo} type 6 is the most generic case, where the slice groups are explicitly defined using high level packets (i.e., signalling \ac{nal} units), allowing more flexibility at the cost of higher signalling overhead. @@ -341,7 +341,7 @@ The \ac{fmo} type 6 is the most generic case, where the slice groups are explici In the past, \ac{fmo} has been widely used to increase the error robustness of video transmissions in different communication scenarios \cite{FMOErrorResilience2007,FMOResilience2010,FMOResilience2011,FMOROI2012,MotionEnergyDataPart2014}. In~\cite{FMOErrorResilience2007} \ac{fmo} is used to increase the \ac{ec} performance and thus reduce the error propagation in prioritised video transmission. -The method divides the \acs{mb} into two slice groups based on an estimation of the \ac{ec} distortion using a two-pass approach. +The method divides the \acs{mb} into two slice groups based on an estimation of the \ac{ec} distortion using a two-pass approach. Then, the slice group which has the most impact on the \ac{ec} distortion is transmitted using a reliable channel, and it is assumed to be received error-free. The experimental results shows that the proposed approach is able to outperform the dispersed mode (\ac{fmo} type 1)~\cite{FMOErrorResilience2007}. Slice classification can also be based on the weighted sum of the motion vector information and the rate-distortion cost~\cite{FMOResilience2010}. @@ -450,7 +450,7 @@ In this example, the encoder-side creates two descriptions, i.e., $R_1$ and $R_2 As shown in this diagram, three different outputs may be produced at the decoder-side: both descriptions are correctly received and the central decoder is used ($D_0$) or, either one of the descriptions is not received and then only one side decoder is used ($D_{1,1}$ or $D_{1,2}$). When both bitstreams are received, the central decoder combines the information carried by each description achieving higher quality than any single side decoder. -In \ac{mdc} encoding, the redundancy among descriptions can be controlled by changing the correlation between them. +In \ac{mdc} encoding, the redundancy among descriptions can be controlled by changing the correlation between them. Higher correlation leads to higher redundancy and also higher robustness, also leading to higher quality in the output of the side decoders. In contrast, with the case of redundant pictures, the redundant \ac{mdc} streams do not carry the same information, which means that if both are received, then the decoded quality may be increased. However, a combination of \ac{mdc} and redundant pictures can also be used to achieve higher quality gains \cite{MDCRedundantPicsJan2010}. @@ -527,8 +527,8 @@ This approach has shown to improve the performance of the \ac{mc} method, with g \subsubsection{Boundary match algorithms} -\Gls{bma} are used to recover a missing region based on the available neighbouring regions. -By using the missing block neighbourhood, a new \ac{mv} can be estimated for the missing block. +\Gls{bma} are used to recover a missing region based on the available neighbouring regions. +By using the missing block neighbourhood, a new \ac{mv} can be estimated for the missing block. Then, the recovery of the missing pixels is accomplished based on motion compensation. A low complexity approach may use the boundary pixels to search for the \ac{mv}, based on a spatial smoothness constraint imposed on the boundaries of the lost block \cite{BMAAlgorithm2008}. Figure~\ref{FIG3:BMAPixelInformation} represents the boundary information used to estimate the \ac{mv} for the lost block from a set of different candidates. The \ac{bma} uses the available \acs{mv} provided by the top (T), bottom (B), left (L) and right (R) blocks to simulate different reconstructions of the lost block. @@ -549,13 +549,13 @@ Although \ac{bma} and outer \ac{bma} have the same design principle of minimisin It was observed in the experimental results of \cite{OBMAMethod2008} that the outer \ac{bma} has better performance in video recovery than conventional \ac{bma}. The performance improvements are explained by the distortion minimisation of the outer boundary, which incorporates the edge information located in the neighbourhood of the lost block in the decision process to select the most accurate \ac{mv}. In \cite{BMAAlgorithm2008} a novel distortion function was proposed, to choose among different candidate MVs. The distortion function exploits both spatial and temporal smooth constraints. The temporal function is based on the outer boundary distortion, as in \cite{OBMAMethod2008}. The spatial distortion aims to reduce the gradient variances in boundaries of the reconstructed block, so the image edges may be guaranteed. -%Several experimental results were obtained for different loss events, covering from an isolated block loss to the loss of a complete row of blocks. -This method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method. +%Several experimental results were obtained for different loss events, covering from an isolated block loss to the loss of a complete row of blocks. +This method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method. However, although subjective analysis revealed that this method is able to recover the edges in the lost image, complex mathematical operations are required, which may not be practical in real time applications. \subsubsection{Motion vector extrapolation} -As mentioned before, the \ac{mc} method is a simple way to reconstruct an estimation of a missing frame using motion information from previously decoded frames. +As mentioned before, the \ac{mc} method is a simple way to reconstruct an estimation of a missing frame using motion information from previously decoded frames. Although this method can provide an acceptable performance comparing with the \ac{fc}, as shown in \cite{H264RefConc2006}, more complex algorithms that use the assumption of motion continuity across consecutive frames are used to recover the missing information. \ac{mve} \cite{BlockBasedMVE2002,BlockBasedMVEOverllaped2011} methods are efficient techniques to reconstruct the \acs{mv} of a lost frame, which in turn are used for motion compensation. diff --git a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex index 6bdd316..d4c7c4c 100644 --- a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex +++ b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex @@ -35,7 +35,7 @@ Although the increased complexity may be compensated with more efficient algorit A critical aspect related to the performance of robust video streaming methods is the error detection accuracy. Since \ac{hevc} streaming is expected to use known transport technologies, such as RTP~\cite{HEVCStreamMay2012}, MPEG-2 TS~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016} and MPEG Media Transport\cite{ATSCMMT2016}, the error detection capabilities of the underlying transport protocols have significant impact on the overall system performance. In general, the error detection operation at the transport layers is decoupled from the video coding layer. -To cope with transmission errors, the encoder implements robust coding methods, while the efficient recovery of lost video data is implemented in the decoder, through \ac{ec} algorithms. +To cope with data loss, the encoder implements robust coding methods, while the efficient recovery of lost video data is implemented in the decoder, through \ac{ec} algorithms. Interlayer communication must be used to cope with different protocol layers, ensuring that the video decoder is given the necessary information to identify the lost slices/frames in the video stream. Therefore, the error robustness of video streams can be investigated on its own, assuming any possible form of streaming/transport technologies and error detection. This is the main focus of the study presented in this section. @@ -160,7 +160,7 @@ Therefore, the overall error robustness is not affected by reducing the maximum \subsection{Evaluation of the impact of intra refreshing} \label{SEC3:IntraRefreshEvaluation} -In this sub-section the relation between the temporal distance of consecutive intra-coded frames, i.e., period of the intra frames, and the video quality under transmission errors is evaluated. +In this sub-section the relation between the temporal distance of consecutive intra-coded frames, i.e., period of the intra frames, and the video quality under packet loss is evaluated. As the intra-coded frames have no dependencies to prior coded pictures, they introduce a refreshing point into the bitstream, which stops temporal error propagation along several frames or slices. This allows the decoder to recover from frame/slice loss and continue error-free decoding in a seamless manner. @@ -220,7 +220,7 @@ In this case, the higher coding efficiency achieved by the \textit{SS} case also \subsection{Motion vector coding impact on error propagation} \label{Section3:MotionVectorCodingTests} -In \ac{hevc}, the new \ac{mv} prediction modes known as \ac{amvp} and Merge Mode~\cite{MotionCodingHEVC2011}, may be significant contributors to the poor performance under transmission errors. These modes are characterised by coding the motion information using previously encoded \acs{mv}, which obviously leads to prediction mismatches at the decoder-side when such neighbouring information is missing due to network losses. +In \ac{hevc}, the new \ac{mv} prediction modes known as \ac{amvp} and Merge Mode~\cite{MotionCodingHEVC2011}, may be significant contributors to the poor performance under network errors. These modes are characterised by coding the motion information using previously encoded \acs{mv}, which obviously leads to prediction mismatches at the decoder-side when such neighbouring information is missing due to network losses. Therefore, in this sub-section the impact of using spatial and temporal \ac{mv} candidates on the decoded video quality under errors is evaluated. \subsubsection{Spatial \ac{mv} candidates} @@ -305,8 +305,8 @@ Moreover, these results also indicate that for higher \ac{plr} the negative impa \subsection*{Remarks on error robustness of HEVC} -Overall, the results presented in the previous sub-sections show that \ac{hevc} has poor robustness against transmission errors, resulting in significant quality degradation, though depending on the video content and coding configurations. -As shown, some coding configurations, such as slice partition and intra refreshing period, have high correlation with the decoded video quality when transmission errors affect the compressed streams. +Overall, the results presented in the previous sub-sections show that \ac{hevc} has poor robustness against network losses, resulting in significant quality degradation, though depending on the video content and coding configurations. +As shown, some coding configurations, such as slice partition and intra refreshing period, have high correlation with the decoded video quality when errors affect the compressed streams. Therefore, these are important aspects to take into consideration in the design and implementation of video delivery services and applications over error-prone networks. One of the main improvements of the \ac{hevc} standard is the support for larger coding units up to $64\times64$, which does not have a negative impact on the error robustness of coded video streams, indicating that they should be allowed to achieve higher coding performance. diff --git a/041_ImprovedErrorResilience/ImprovedErrorResilience.tex b/041_ImprovedErrorResilience/ImprovedErrorResilience.tex index b4c14ca..16acffe 100644 --- a/041_ImprovedErrorResilience/ImprovedErrorResilience.tex +++ b/041_ImprovedErrorResilience/ImprovedErrorResilience.tex @@ -26,7 +26,7 @@ This chapter is organized as follows. Section~\ref{Section4:Problem} presents th As previously mentioned, when \ac{hevc} bitstreams are subject to transmission losses, the decoded video signal incurs in significant degradation of both objective and subjective quality. This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes, which are selected according to \ac{rd} optimisation criteria, which always assume error-free transmission~\cite{ROPE2007}. -Therefore, high compression efficiency is achieved, but resilience to transmission errors is penalised. +Therefore, high compression efficiency is achieved, but resilience to errors and data loss is penalised. \begin{figure}[tb] \centering @@ -314,7 +314,6 @@ Figure~\ref{fig:PSNRvsFrameForRefPicSel}~shows the \ac{psnr} over a \ac{gop} wit This figure shows that all methods are able to outperform the reference \ac{hevc} (\REF), gaining approximately $3$~dB in reconstruction quality. The similar quality levels achieved by \CHKB and \PROP reveal that the dynamic selection of the reference frames used in \PROP is more robust, because a better trade-off between coding efficiency and error resilience is attained. Moreover, results in Figure~\ref{fig:PSNRvsFrameForRefPicSel} are in line with the reconstruction quality shown in Figure~\ref{fig:RecoveredExample}, confirming that the proposed method is able to outperform the reference frame generation technique in~\cite{ROPERefGenJan2010}. -%Since the proposed method is more efficient than the others for error-free transmission (see~Table~\ref{tab:bd_error_free}), and it also provides better error robustness, it can be considered as an efficient solution to better deal with transmission errors. Overall, the method proposed for the encoding stage is able to increase the error robustness of \ac{hevc} without severely compromising the coding efficiency. \input{./041_ImprovedErrorResilience/Results/TableResultsRandomLossRefPicSelection.tex} diff --git a/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex b/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex index 2e79bdf..c2622be 100644 --- a/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex +++ b/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex @@ -226,7 +226,7 @@ Thus, in order to achieve higher reconstruction quality, different approaches sh This section presents the general architecture proposed for \ac{ec}-aware video encoding, describing its main functional blocks, and providing an introduction for the subsequent detailed descriptions. Figure~\ref{FIG5:BlockDiagram} illustrates the architecture of the proposed \ac{ec}-aware encoding scheme. -After the video encoder, it includes data loss simulation at the \ac{nal} for every slice. Subsequently, \ac{ec} is applied to the simulated losses in order to find the best methods that should be used to reconstruct the lost slices in case of transmission errors. +After the video encoder, it includes data loss simulation at the \ac{nal} for every slice. Subsequently, \ac{ec} is applied to the simulated losses in order to find the best methods that should be used to reconstruct the lost slices in case of errors and packet loss. Saliency-weighted \ac{ec} mode decision is used to select the optimal \ac{ec} mode from the best trade-off between \ac{ec} distortion, using the original input video~($f$) as reference, and the number of bits required for signalling. Then, the optimal \ac{ec} mode is sent to the decoder along with the mismatch residue computed as a difference between the transmitted and recovered frames. Finally, the scalable extension of the \ac{hevc} standard is used to encode this residue information. @@ -421,8 +421,8 @@ Realistic conditions were used for transmission, by allowing independent losses \label{SEC4:IntraPerformance} In this section the performance of the proposed method is evaluated only for intra frames, in order to demonstrate its efficiency in reducing error propagation. -Since intra \acs{cu} result in higher amount of bits, they also require a higher number of packets, making them more prone to transmission errors and losses. -Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets correspond to intra coded slices, which is a non-negligible probability of one of them being hit by transmission errors. +Since intra \acs{cu} result in higher amount of bits, they also require a higher number of packets, making them more prone to errors and data loss. +Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets correspond to intra coded slices, which is a non-negligible probability of one of them being hit by transmission errors which would lead to packet loss. Moreover, by limiting the proposed method to intra frames, the amount of required overhead is reduced. In this case a constant $\lambda_0=10$ was used, without taking into consideration saliency estimation. This value results from an exhaustive search targetting acceptable quality gains with a low overhead (i.e., average overhead under 5\%). @@ -708,7 +708,7 @@ For instance, in comparison with the method in \cite{DualFrameME2015} the propos Results in Table~\ref{tab:overall_gain_ratio} also show that the small loss of coding efficiency in error-free transmission is clearly surpassed by the quality gains under error-prone transmission conditions. The methods in \cite{ROPERefGenJan2010,DualFrameME2015} are based on modified predictions, while the method in \cite{VPxResiliencePCS2016} uses redundant data without any further optimisation. Both of these approaches require low additional computational complexity. -However, as observed in the results shown in Table~\ref{tab:overall_gain_ratio}, the lower additional complexity also corresponds to lower quality gains in case of transmission errors. +However, as observed in the results shown in Table~\ref{tab:overall_gain_ratio}, the lower additional complexity also corresponds to lower quality gains in case of packet loss. diff --git a/carreira_phd_thesis.tex b/carreira_phd_thesis.tex index d0f0a1f..92ef690 100644 --- a/carreira_phd_thesis.tex +++ b/carreira_phd_thesis.tex @@ -28,7 +28,7 @@ %\includeonly{031_ErrorResilienceEvaluation/ErrorResilienceEvaluation} %\includeonly{041_ImprovedErrorResilience/ImprovedErrorResilience} %\includeonly{042_ConcealmentAwareEncoding/ConcealmentAwareEncoding} -%\includeonly{043_EndToEndRobustSystem/EndToEndRobustSystem} +%\includeonly{043_DynamicErrorRobustness/DynamicErrorRobustness} %\includeonly{050_Conclusions/Conclusion} \renewcommand{\baselinestretch}{1.3} diff --git a/carreira_phd_thesis_abr.sty b/carreira_phd_thesis_abr.sty index ff329b4..872e96b 100644 --- a/carreira_phd_thesis_abr.sty +++ b/carreira_phd_thesis_abr.sty @@ -146,6 +146,8 @@ \def\LOSTALL{$IDR_{Loss}$\xspace} \def\KEEPINTRA{$IDR_{NoLoss}$\xspace} +\def\NOLOSS{No Loss} + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Reference methods \def\REF{\mbox{\textit{{Ref}}}}