jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Review the use of transmission losses to network losses

Browse Source
This commit is contained in:
jfmcarreira
2018-12-15 01:42:24 +00:00
parent 77cfe2f79d
commit 57fdfd0863
9 changed files with 37 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files
000_Preamble/abstract.tex
+1 -24
View File
@@ -6,31 +6,8 @@
\vspace{-5mm}
%This thesis investigates the problem of robust coding and Error Concealment (EC) in High Efficiency Video Coding (HEVC).
%After a review of the current state of the art, a simulation study about error robustness, revealed that the HEVC has weak protection against transmission errors with significant impact on video quality degradation.
%Based on this evidence, the first contribution of this work is a new method to reduce the temporal dependencies between Motion Vectors (MVs), by improving the decoded video quality without compromising the compression efficiency.
%
%The second contribution of this thesis is a two-stage approach for reducing the mismatch of temporal predictions in case of video streams received with errors or lost data.
%At the encoder stage, the reference pictures are dynamically distributed based on a constrained Lagrangian rate-distortion optimization to reduce the number of predictions from a single reference.
%At the streaming stage, a prioritization algorithm, based on spatial dependencies, selects a reduced set of MVs to be transmitted, as side information, to reduce mismatched MV predictions at the decoder.
%
%The problem of EC-aware video coding is also investigated to enhance the overall error robustness.
%A new approach based on scalable coding and optimally EC selection is proposed, where the optimal EC modes are found by simulating transmission losses, followed by a saliency-weighted optimisation.
%Moreover, EC residual information is encoded using a rate-controlled enhancement layer.
%Both are transmitted to the decoder to be used in case of data loss.
%
%Finally, an adaptive error resilience scheme is proposed to dynamically predict the video stream that achieves the highest decoded quality for a particular loss case.
%A neural network selects among the various video streams, encoded with different levels of compression efficiency and error protection, based on information from the video signal, the coded stream and the transmission network.
%
%Overall, the new robust video coding methods investigated in this thesis yield consistent quality gains in comparison with other existing methods and also the ones implemented in the HEVC reference software.
%Furthermore, the trade-off between coding efficiency and error robustness is also better in the proposed methods.
%\vspace{2mm}
%\textbf{Keywords:} 3D video, frame loss error concealment, subjective quality assessment.
This thesis investigates the problem of robust coding and error concealment in High Efficiency Video Coding (HEVC).
After a review of the current state of the art, a simulation study about error robustness, revealed that the HEVC has weak protection against transmission errors with significant impact on video quality degradation.
After a review of the current state of the art, a simulation study about error robustness, revealed that the HEVC has weak protection against network losses with significant impact on video quality degradation.
Based on this evidence, the first contribution of this work is a new method to reduce the temporal dependencies between motion vectors, by improving the decoded video quality without compromising the compression efficiency.
The second contribution of this thesis is a two-stage approach for reducing the mismatch of temporal predictions in case of video streams received with errors or lost data.
010_Introduction/Introduction.tex
+1 -2
View File
@@ -19,7 +19,6 @@ To cope with such demanding requirements a new coding standard was developed by
The \ac{hevc} is currently the main video coding framework to extend current services towards higher resolution formats delivered with better compression efficiency.
Similar to its predecessor, \ac{hevc} is a block-based hybrid video codec using predictive, transform and entropy coding.
The coding flexibility and efficiency of \ac{hevc} results from the use of new coding tools and block structures, which enables new block partitions\cite{HEVCBlock2012}, improved prediction modes \cite{IntraCodingHEVC2012,AMPCodingHEVCJun2012} and new high-level features~\cite{HEVCHighLevelReferenceDez2012}, such as explicit reference picture management and new parameter sets.
%However, it is known that higher coding efficiency leads to lower robustness in the presence of transmission errors and data loss.
However, new coding tools of \ac{hevc} not only bring high compression efficiency~\cite{HEVCComparisonDec2012}, but also some other disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and inherent reduction of error robustness~\cite{StudyHEVCErrorsDez2012}.
Earlier studies about the error robustness characteristics of \ac{hevc} have shown that, in general, the use of high efficiency coding tools results in poor error resilience performance (score \emph{slightly annoying} in the \ac{mos} scale), in comparison with previous standards~\cite{QoEHEVCMay2014,SubjectiveEvalSep2013}.
Thus, when \ac{hevc} streams incur in transmission losses, this leads to significant degradation of both objective and subjective quality.
@@ -51,7 +50,7 @@ The following objectives were pursued in this work:
\begin{enumerate}
\item
\textbf{Study and evaluation of error robustness in highly compressed video:}
The objective of this research study is to evaluate the robustness against transmission errors of highly efficiently coded video.
The objective of this research study is to evaluate the robustness against network losses of highly efficiently coded video.
The study and its conclusions aim at finding relevant research directions to develop new methods for increasing the error robustness of coded streams.
The results and conclusions of this study were published in conference paper (\ref{Conf:ErrorResilience}) and in journal paper (\ref{Journal:TwoStage}) (see Appendix~\ref{Appen:Papers}).
021_VideoCoding/VideoCodingReview.tex
+1 -1
View File
@@ -672,7 +672,7 @@ At the high-level, \ac{hevc} introduces new \ac{nal} types and also new pictures
These allow the implementation of various random-access strategies, more flexibility and higher coding efficiency than its predecessor \ac{h264}.
Another major improvement comes from the inter-frame prediction. On the one hand new high-level reference picture management was introduced, which increases the error robustness, as it explicitly defines the active reference frames and the \ac{dpb} status. On the other hand it introduces the \ac{amvp} and the Merge mode, which efficiently reduce the required bits for coding the motion information.
The newly developed coding tools and prediction modes described in this chapter have resulted in approximately 50\% of bitrate savings for the same subjective quality in comparison with the previous standard.
However, such high coding efficiency implies higher data dependencies, which are more sensible to data loss and increase the error propagation to subsequent coding structures. Therefore, evaluating the impact of transmission errors and error propagation in \ac{hevc} is of most importance to develop efficient techniques to recover from data loss and reduce the error propagation.
However, such high coding efficiency implies higher data dependencies, which are more sensible to data loss and increase the error propagation to subsequent coding structures. Therefore, evaluating the impact of network errors and error propagation in \ac{hevc} is of most importance to develop efficient techniques to recover from data loss and reduce the error propagation.
022_ErrorResilience/RobustTransmissionReview.tex
+2 -2
View File
@@ -9,7 +9,7 @@
In general, digital communications through networks are prone to transmission errors or data loss.
In general, digital communications through networks are prone to errors or data loss.
These errors can be handled in different ways, such as retransmission, error correction, and \ac{ec} \cite{ErrorResilience2000,PixelInterlacingIntraConcealment2011}.
This chapter reviews the existing techniques for robust video coding that can be applied at the \ac{vcl} to reduce the error propagation and increase the decoded video quality.
Several techniques have been proposed in the past to deal with the problem of robust video transmission, especially for video coding standards with high temporal dependencies, such as \ac{h264} and more recently \ac{hevc}.
@@ -23,7 +23,7 @@ Section~\ref{Section22:Localisation} describes the localisation techniques, whic
Section~\ref{Section22:DataPartitionRedundant} provides an overview on methods based on data partition, classification and bitstream redundancy.
%These techniques ease the introduction of extra information in order to recover from lost data.
Section~\ref{Section22:Concealment} covers the techniques used at the decoder-side, as post-processing, to reduce the effect of errors and improve visual quality and smoothness.
Section~\ref{Section22:ConcealmentAware} provides a description of methods that use \ac{ec} techniques to cope with transmission errors and improve error robustness.
Section~\ref{Section22:ConcealmentAware} provides a description of methods that use \ac{ec} techniques to cope with packet loss and improve error robustness.
Finally, Section~\ref{Section22:Summary} concludes the chapter.
% by providing a overall comparison between the different approaches.
031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex
+5 -5
View File
@@ -35,7 +35,7 @@ Although the increased complexity may be compensated with more efficient algorit
A critical aspect related to the performance of robust video streaming methods is the error detection accuracy.
Since \ac{hevc} streaming is expected to use known transport technologies, such as RTP~\cite{HEVCStreamMay2012}, MPEG-2 TS~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016} and MPEG Media Transport\cite{ATSCMMT2016}, the error detection capabilities of the underlying transport protocols have significant impact on the overall system performance.
In general, the error detection operation at the transport layers is decoupled from the video coding layer.
To cope with transmission errors, the encoder implements robust coding methods, while the efficient recovery of lost video data is implemented in the decoder, through \ac{ec} algorithms.
To cope with data loss, the encoder implements robust coding methods, while the efficient recovery of lost video data is implemented in the decoder, through \ac{ec} algorithms.
Interlayer communication must be used to cope with different protocol layers, ensuring that the video decoder is given the necessary information to identify the lost slices/frames in the video stream.
Therefore, the error robustness of video streams can be investigated on its own, assuming any possible form of streaming/transport technologies and error detection.
This is the main focus of the study presented in this section.
@@ -160,7 +160,7 @@ Therefore, the overall error robustness is not affected by reducing the maximum
\subsection{Evaluation of the impact of intra refreshing}
\label{SEC3:IntraRefreshEvaluation}
In this sub-section the relation between the temporal distance of consecutive intra-coded frames, i.e., period of the intra frames, and the video quality under transmission errors is evaluated.
In this sub-section the relation between the temporal distance of consecutive intra-coded frames, i.e., period of the intra frames, and the video quality under packet loss is evaluated.
As the intra-coded frames have no dependencies to prior coded pictures, they introduce a refreshing point into the bitstream, which stops temporal error propagation along several frames or slices.
This allows the decoder to recover from frame/slice loss and continue error-free decoding in a seamless manner.
@@ -220,7 +220,7 @@ In this case, the higher coding efficiency achieved by the \textit{SS} case also
\subsection{Motion vector coding impact on error propagation}
\label{Section3:MotionVectorCodingTests}
In \ac{hevc}, the new \ac{mv} prediction modes known as \ac{amvp} and Merge Mode~\cite{MotionCodingHEVC2011}, may be significant contributors to the poor performance under transmission errors. These modes are characterised by coding the motion information using previously encoded \acs{mv}, which obviously leads to prediction mismatches at the decoder-side when such neighbouring information is missing due to network losses.
In \ac{hevc}, the new \ac{mv} prediction modes known as \ac{amvp} and Merge Mode~\cite{MotionCodingHEVC2011}, may be significant contributors to the poor performance under network errors. These modes are characterised by coding the motion information using previously encoded \acs{mv}, which obviously leads to prediction mismatches at the decoder-side when such neighbouring information is missing due to network losses.
Therefore, in this sub-section the impact of using spatial and temporal \ac{mv} candidates on the decoded video quality under errors is evaluated.
\subsubsection{Spatial \ac{mv} candidates}
@@ -305,8 +305,8 @@ Moreover, these results also indicate that for higher \ac{plr} the negative impa
\subsection*{Remarks on error robustness of HEVC}
Overall, the results presented in the previous sub-sections show that \ac{hevc} has poor robustness against transmission errors, resulting in significant quality degradation, though depending on the video content and coding configurations.
As shown, some coding configurations, such as slice partition and intra refreshing period, have high correlation with the decoded video quality when transmission errors affect the compressed streams.
Overall, the results presented in the previous sub-sections show that \ac{hevc} has poor robustness against network losses, resulting in significant quality degradation, though depending on the video content and coding configurations.
As shown, some coding configurations, such as slice partition and intra refreshing period, have high correlation with the decoded video quality when errors affect the compressed streams.
Therefore, these are important aspects to take into consideration in the design and implementation of video delivery services and applications over error-prone networks.
One of the main improvements of the \ac{hevc} standard is the support for larger coding units up to $64\times64$, which does not have a negative impact on the error robustness of coded video streams, indicating that they should be allowed to achieve higher coding performance.
041_ImprovedErrorResilience/ImprovedErrorResilience.tex
+1 -2
View File
@@ -26,7 +26,7 @@ This chapter is organized as follows. Section~\ref{Section4:Problem} presents th
As previously mentioned, when \ac{hevc} bitstreams are subject to transmission losses, the decoded video signal incurs in significant degradation of both objective and subjective quality.
This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes, which are selected according to \ac{rd} optimisation criteria, which always assume error-free transmission~\cite{ROPE2007}.
Therefore, high compression efficiency is achieved, but resilience to transmission errors is penalised.
Therefore, high compression efficiency is achieved, but resilience to errors and data loss is penalised.
\begin{figure}[tb]
\centering
@@ -314,7 +314,6 @@ Figure~\ref{fig:PSNRvsFrameForRefPicSel}~shows the \ac{psnr} over a \ac{gop} wit
This figure shows that all methods are able to outperform the reference \ac{hevc} (\REF), gaining approximately $3$~dB in reconstruction quality.
The similar quality levels achieved by \CHKB and \PROP reveal that the dynamic selection of the reference frames used in \PROP is more robust, because a better trade-off between coding efficiency and error resilience is attained.
Moreover, results in Figure~\ref{fig:PSNRvsFrameForRefPicSel} are in line with the reconstruction quality shown in Figure~\ref{fig:RecoveredExample}, confirming that the proposed method is able to outperform the reference frame generation technique in~\cite{ROPERefGenJan2010}.
%Since the proposed method is more efficient than the others for error-free transmission (see~Table~\ref{tab:bd_error_free}), and it also provides better error robustness, it can be considered as an efficient solution to better deal with transmission errors.
Overall, the method proposed for the encoding stage is able to increase the error robustness of \ac{hevc} without severely compromising the coding efficiency.
\input{./041_ImprovedErrorResilience/Results/TableResultsRandomLossRefPicSelection.tex}
042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex
+4 -4
View File
@@ -226,7 +226,7 @@ Thus, in order to achieve higher reconstruction quality, different approaches sh
This section presents the general architecture proposed for \ac{ec}-aware video encoding, describing its main functional blocks, and providing an introduction for the subsequent detailed descriptions.
Figure~\ref{FIG5:BlockDiagram} illustrates the architecture of the proposed \ac{ec}-aware encoding scheme.
After the video encoder, it includes data loss simulation at the \ac{nal} for every slice. Subsequently, \ac{ec} is applied to the simulated losses in order to find the best methods that should be used to reconstruct the lost slices in case of transmission errors.
After the video encoder, it includes data loss simulation at the \ac{nal} for every slice. Subsequently, \ac{ec} is applied to the simulated losses in order to find the best methods that should be used to reconstruct the lost slices in case of errors and packet loss.
Saliency-weighted \ac{ec} mode decision is used to select the optimal \ac{ec} mode from the best trade-off between \ac{ec} distortion, using the original input video~($f$) as reference, and the number of bits required for signalling.
Then, the optimal \ac{ec} mode is sent to the decoder along with the mismatch residue computed as a difference between the transmitted and recovered frames.
Finally, the scalable extension of the \ac{hevc} standard is used to encode this residue information.
@@ -421,8 +421,8 @@ Realistic conditions were used for transmission, by allowing independent losses
\label{SEC4:IntraPerformance}
In this section the performance of the proposed method is evaluated only for intra frames, in order to demonstrate its efficiency in reducing error propagation.
Since intra \acs{cu} result in higher amount of bits, they also require a higher number of packets, making them more prone to transmission errors and losses.
Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets correspond to intra coded slices, which is a non-negligible probability of one of them being hit by transmission errors.
Since intra \acs{cu} result in higher amount of bits, they also require a higher number of packets, making them more prone to errors and data loss.
Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets correspond to intra coded slices, which is a non-negligible probability of one of them being hit by transmission errors which would lead to packet loss.
Moreover, by limiting the proposed method to intra frames, the amount of required overhead is reduced.
In this case a constant $\lambda_0=10$ was used, without taking into consideration saliency estimation. This value results from an exhaustive search targetting acceptable quality gains with a low overhead (i.e., average overhead under 5\%).
@@ -708,7 +708,7 @@ For instance, in comparison with the method in \cite{DualFrameME2015} the propos
Results in Table~\ref{tab:overall_gain_ratio} also show that the small loss of coding efficiency in error-free transmission is clearly surpassed by the quality gains under error-prone transmission conditions.
The methods in \cite{ROPERefGenJan2010,DualFrameME2015} are based on modified predictions, while the method in \cite{VPxResiliencePCS2016} uses redundant data without any further optimisation.
Both of these approaches require low additional computational complexity.
However, as observed in the results shown in Table~\ref{tab:overall_gain_ratio}, the lower additional complexity also corresponds to lower quality gains in case of transmission errors.
However, as observed in the results shown in Table~\ref{tab:overall_gain_ratio}, the lower additional complexity also corresponds to lower quality gains in case of packet loss.
carreira_phd_thesis.tex
+1 -1
View File
@@ -28,7 +28,7 @@
%\includeonly{031_ErrorResilienceEvaluation/ErrorResilienceEvaluation}
%\includeonly{041_ImprovedErrorResilience/ImprovedErrorResilience}
%\includeonly{042_ConcealmentAwareEncoding/ConcealmentAwareEncoding}
%\includeonly{043_EndToEndRobustSystem/EndToEndRobustSystem}
%\includeonly{043_DynamicErrorRobustness/DynamicErrorRobustness}
%\includeonly{050_Conclusions/Conclusion}
\renewcommand{\baselinestretch}{1.3}
carreira_phd_thesis_abr.sty
+2
View File
@@ -146,6 +146,8 @@
\def\LOSTALL{$IDR_{Loss}$\xspace}
\def\KEEPINTRA{$IDR_{NoLoss}$\xspace}
\def\NOLOSS{No Loss}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Reference methods
\def\REF{\mbox{\textit{{Ref}}}}