jfmcarreira
4ebe3013c0
INSTRUCTIONS Px stands for paragraph number GENERAL CORRECTIONS Fix the reference of equation "in Equation (5.2)" -> "in (5.2)" ACKNOWLEDGMENTS: Replace Dr. with Prof. where appropriate. Add reference to Loughborough University London. ABSTRACT: Remove abbreviations. Add MPEG-DASH, MOS, HM to the list. Replaced LTE with LTE-A. LIST OF ABBREVIATIONS: Remove trailing dot and add page where it is defined in the text. CHAPTER 1 Section 1.1 P1: internet -> Internet, added mention to 360-degree video and Facebook. P2: Remove sentence that was lacking reference: "However, it is known that higher coding efficiency leads to lower robustness in the presence of transmission errors and data loss." Subsequent sentence expresses the same idea as it includes reference already. P2: Add an objective score to the "poor error resilience". Section 1.2 P1: Capitalise the first letter of each objective. P1: Fix "error robust" spelling error. Section 1.3 P2: Replace "with" by "to" and efficiently with efficient. P7: Replace "material" with "materials". CHAPTER 2 Section 2.1 P1: Replace "in" with "on". Section 2.2 P1: Fix three spelling errors. P3: Replace "between" with "among" and add missing "a". P4: Add missing "a". Figure 2.1: Add description of T and Q. Section 2.3 P1: Replace "complaint" with "conforming". P2: Add missing either. Figure 2.2: Change figure to avoid confusing in arrow direction. Section 2.4 P1: Fix typo in "pictures types". Section 2.4.2 P1: Remove last part of the second sentence. P2: Replace “with” by “to”. Section 2.4.4 P1: Add missing “that”. P7: Fix two spelling errors. Section 2.5 P2: Add missing pixels. Figure 2.8: Correct the coding order Section 2.6 P1: Fix three typos. Section 2.6.1 P1: Fix third sentence to: “When the intra prediction mode is chosen for a CU, the size of the PU needs to be the same as the CU, except when the smallest CU size is selected and further division into four PUs is allowed.” P7: Fix “three-tap”. P8 (page 24): The last sentence of the paragraph was revised to: “Only the mentioned boundaries are filtered, as they are the ones most likely to introduce discontinuities.” Section 2.6.2 P1: Replace “rely” with “relies”. P2: Replace twice “in” to “of”. P2: Replace “different” with “difference”. P3: Fix two typos. P4: Add HM acronym definition. P6: Replace “not” with “no”. Table 2.2: Remove acronym in the title. Figure 2.11: Review figure and caption to become self-contained. Section 2.7 P1: remove comma after Similarly. P3: Replace “intra coded” with “intra-coded”. P4: Replace last sentence with “This procedure is the main cause for the coding distortion, as it performs a many-to-one mapping, which cannot be reversed.” Section 2.9 P1: Revise the third sentence to: “Such grid-based restriction reduces the computational complexity and facilitates the parallel-processing, which is one of the main goals of HEVC design.” P2: Revise the third sentence: “For each CTU, the encoder decides whether to apply the SAO filter or not, and if used, one of the two filter types is applied, namely the band offset or the edge offset.”
113 lines
14 KiB
TeX
113 lines
14 KiB
TeX
\graphicspath{{010_Introduction}}
|
||
|
||
\chapter{Introduction \VERSION{3}}
|
||
\label{Chapter:Intro}
|
||
|
||
\section{Context and motivation}
|
||
|
||
|
||
|
||
|
||
In recent years new advances in video compression and networking technologies have enabled the development of many multimedia applications over the Internet, such as video conferencing, video streaming, video publishing, among others.
|
||
The increasing diversity of services, the demand for high quality multimedia and the introduction of \ac{uhd} formats (\emph{e.g.}, $4k$ or $8k$ resolution and beyond) are pushing forward existing technologies and opening new research directions.
|
||
Moreover, the emerging new video formats with increased resolutions have created stronger needs for higher coding efficiency, beyond the previous standards, i.e., the \ac{h264}~\cite{H264AVCStandard2003,H264OverviewJul2003}.
|
||
Such need has been driven not only by higher resolutions but also by 360-degree, stereoscopic and multi-view video formats, which increase the number of viewpoints that need to be encoded and delivered simultaneously.
|
||
Moreover, the ever increasing traffic generated by mobile multimedia applications \cite{CiscoMobile1621_Feb2017} and different video services with growing popularity (e.g., Netflix and Facebook) impose new challenges to the existing networks.
|
||
To cope with such demanding requirements a new coding standard was developed by the \ac{itut} \ac{vceg} and the \ac{iso}/\ac{iec} \ac{mpeg} standardisation organisations, referred to as H.265 - \ac{hevc}~\cite{H265StandardFeb2018,HEVCOverviewDez2012}.
|
||
|
||
\glsunset{h264}
|
||
|
||
The \ac{hevc} is currently the main video coding framework to extend current services towards higher resolution formats delivered with better compression efficiency.
|
||
Similar to its predecessor, \ac{hevc} is a block-based hybrid video codec using predictive, transform and entropy coding.
|
||
The coding flexibility and efficiency of \ac{hevc} results from the use of new coding tools and block structures, which enables new block partitions\cite{HEVCBlock2012}, improved prediction modes \cite{IntraCodingHEVC2012,AMPCodingHEVCJun2012} and new high-level features~\cite{HEVCHighLevelReferenceDez2012}, such as explicit reference picture management and new parameter sets.
|
||
%However, it is known that higher coding efficiency leads to lower robustness in the presence of transmission errors and data loss.
|
||
However, new coding tools of \ac{hevc} not only bring high compression efficiency~\cite{HEVCComparisonDec2012}, but also some other disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and inherent reduction of error robustness~\cite{StudyHEVCErrorsDez2012}.
|
||
Earlier studies about the error robustness characteristics of \ac{hevc} have shown that, in general, the use of high efficiency coding tools results in poor error resilience performance (score \emph{slightly annoying} in the \ac{mos} scale), in comparison with previous standards~\cite{QoEHEVCMay2014,SubjectiveEvalSep2013}.
|
||
Thus, when \ac{hevc} streams incur in transmission losses, this leads to significant degradation of both objective and subjective quality.
|
||
This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes of \ac{hevc}, which assume \mbox{error-free} transmission~\cite{ROPE2007}.
|
||
A possible approach to encode robust video streams is to increase the error resilience of \ac{hevc} by including \mbox{error-aware} mechanisms in the coding process. As a result, the effect of error propagation, due to prediction mismatch at the decoding side, will be limited.
|
||
|
||
|
||
|
||
The widespread use of \ac{hevc} in broadcast and multimedia delivery services is based on transport technologies such as \ac{rtp}~\cite{HEVCStreamMay2012}, \ac{ts}~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as \ac{dash}~\cite{HEVCHTTPStreamAug2016}.
|
||
Recent developments also enable combinations of different technologies, e.g, \mbox{HTTP-based} streaming and \ac{ts}, to increase the flexibility of over-the-top video services\cite{Broadcast2017}. For content delivery over heterogeneous networks, the \ac{mmt}~\cite{ATSCMMT2016} is another possible technology to be used with the \ac{hevc} standard. Besides the transport technologies,
|
||
the overall error robustness of video transmission may also benefit from the increasing flexibility of multimedia systems to cope with heterogeneous networks, by using auxiliary streams that allow the video decoder to increase the quality of the recovered lost slices~\cite{ErrorResilienceHeterogenousNetworks2016,HeterogeneousStreaming2016}.
|
||
Moreover, error resilience techniques may also take advantage of the multi-stream based approaches, where each stream allows for different levels of error robustness~\cite{HEVCHTTPStreamAug2016,PerfMediaTransport2017}.
|
||
|
||
|
||
In summary, increasing the error robustness of the highly compressed video streams is a relevant research challenge, whose solutions will be useful to increase the adaptability for transmission over diverse network conditions and also cope with communication errors, providing increased levels of \ac{qoe} to the end users at all times.
|
||
In general, increased error robustness in video transmission can be jointly achieved at three different stages: (i) using robust video encoding, by selecting robust coding modes, (ii) after encoding, by adding robust features to the compressed streams, as side-information, and (iii) using robust video decoding techniques, e.g., efficient \ac{ec}. This is exemplified in Figure~\ref{FIG1:RobustVideoTransmission} where at the encoder different layers are used to increase the error robustness, which results in a video stream composed by a \ac{vcl} and \ac{sei} layers. At the decoder two processing layers are used to achieve robust video decoding.
|
||
The \ac{hevc} reference codec is continuously being improved and extended by multiple proponents from around the world, which also address the problem of error resilience and robust delivered transmission.
|
||
Although different \ac{ec} methods have been developed during the past years for \ac{h264}~\cite{2DRecursiveRefinement2010} and more recently for \ac{hevc}~\cite{HEVCMCBlockAug2013}, a combined and optimised approach involving the encoder and decoder has never been fully exploited. This would improve the users’ \ac{qoe}, by exploiting shared information between the encoder and the decoder-sides.
|
||
|
||
\begin{figure}[tb]
|
||
\centering
|
||
\input{./010_Introduction/RobustVideoSystem.tikz}
|
||
\caption{Generic robust video transmission system.}
|
||
\label{FIG1:RobustVideoTransmission}
|
||
\end{figure}
|
||
|
||
\section{Main objectives and contributions}
|
||
|
||
The research objectives of this thesis are defined within the scope of robust error resilience for efficient error concealment in highly efficiency video decoders.
|
||
Thus, this work investigates novel techniques to increase the video quality of \ac{hevc} bitstreams delivered through error-prone networks.
|
||
The following objectives were pursued in this work:
|
||
\begin{enumerate}
|
||
\item
|
||
\textbf{Study and evaluation of error robustness in highly compressed video:}
|
||
The objective of this research study is to evaluate the robustness against transmission errors of highly efficiently coded video.
|
||
The study and its conclusions aim at finding relevant research directions to develop new methods for increasing the error robustness of coded streams.
|
||
The results and conclusions of this study were published in conference paper (\ref{Conf:ErrorResilience}) and in journal paper (\ref{Journal:TwoStage}) (see Appendix~\ref{Appen:Papers}).
|
||
|
||
\item
|
||
\textbf{Robust video coding:}
|
||
At the encoder-side, novel techniques to extend the existing error robustness of highly predictive video coding techniques are investigated in order to increase the received video quality over error-prone networks.
|
||
This research topic resulted in a new method to reduce the error propagation in motion vector coding algorithms, which was published in conference paper (\ref{Conf:DynamicTMVP}).
|
||
Moreover, a novel two-stage approach was devised to improve the error robustness of highly efficiently coded video at the encoder and streaming levels, resulting in two conference publications (\ref{Conf:MotionVectorRedundancies} and~\ref{Conf:Chkb}) and a journal publication (\ref{Journal:TwoStage}).
|
||
|
||
\item \textbf{Robust video decoding:} error concealment techniques are used to increase the error robustness of video decoding and improve the perceived video quality.
|
||
The paradigm of \ac{ec} is evaluated at the encoder-side, as part of the encoding process, in order to optimise the reconstruction process at the decoder-side. This research resulted in a novel error concealment-aware technique to optimise the error concealment process at the encoder-side, which was proposed for intra-frame only in a conference paper (\ref{Conf:ConcAware}) and for generic robust video transmission in the journal (\ref{Journal:ConcAware}).
|
||
|
||
%\item \textbf{jointly error resilience and concealment:} the ultimate goal of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet higher quality demands for different video contents characteristics and network conditions.
|
||
\end{enumerate}
|
||
%\HIGHLIGHT{Contributions}
|
||
|
||
%Finally, the ultimate goal of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet a set of \ac{qos}/\ac{qoe} requirements.
|
||
%Since the ultimate quality judgement is made by end users, subjective tests will constitute a part of the research work in order to validate the objective gains reported as a result of the deployment of various proposed and implemented methods. Given the importance of the \ac{hevc} standard in the near future, the simulation scenarios will be defined based on practical service requirements, such as, bit rate allocation, transmission delay and required QoS.
|
||
|
||
\section{Outline of this thesis}
|
||
|
||
This thesis is organised as follows. The current chapter introduces the context, motivation and objectives of this thesis, as well as the original contributions.
|
||
|
||
The first two chapters review the \mbox{state-of-the-art} concepts and previous work related to this research. Chapter~\ref{Chapter:HEVCReview} presents an overview of the \ac{hevc} standard, focusing on the concepts explored in this work, e.g., high-level features and temporal predictions. Chapter~\ref{Chapter:ErrorResilienceReview} reviews the most important work dealing with robust video transmission also providing relevant background knowledge to this work.
|
||
|
||
Chapter~\ref{Chapter:HEVCErrorResilience} firstly presents a study about the error robustness of high efficiently coded video, using the \ac{hevc} as the underlying framework for various coding configurations.
|
||
The study presents the relevant tools that mostly affect the error robustness.
|
||
Secondly, a new method is proposed to decrease the vulnerability of temporal motion prediction and decrease the error propagation, leading to higher video quality under error-prone conditions.
|
||
|
||
Chapter~\ref{Chapter:ErrorResilienceTechniques} proposes a novel error resilience approach for high efficient video codecs.
|
||
A new method to distribute the use of reference frames is proposed to reduce the error propagation and improve the \ac{ec} performance.
|
||
Moreover, to reduce the effect of mismatched motion predictions (shown in Chapter~\ref{Chapter:HEVCErrorResilience}), a method based on selective motion vector redundancies is proposed.
|
||
This method is able to improve the decoded video quality using a small overhead.
|
||
|
||
Chapter~\ref{Chapter:ConcealmentAwareTechniques} addresses the problem of robust \ac{ec} of high efficiently coded video streams.
|
||
%This work was motivated by the requirements for more accurate \ac{ec} algorithms which are able to reduce the error propagation.
|
||
Different \ac{ec} algorithms were taken into consideration in order to develop a concealment-aware coding scheme.
|
||
The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled to the decoder.
|
||
%Such optimal EC\ac{ec} modes are found by simulating transmission losses followed by a \ac{rd} optimisation of the signalling rate and \ac{ec} distortion cost.
|
||
A generalised saliency-weighted \ac{rd} optimisation is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer.
|
||
%When data loss occurs the decoder uses the signalling information is used at the decoder, in case of data loss, to improve the reconstruction quality.
|
||
%The simulation results show that the proposed method achieves consistent quality gains in comparison with other reference methods and previous works.
|
||
|
||
|
||
Chapter~\ref{Chapter:EndToEndRobustSystem} proposes a novel adaptive error robustness method, which combines the different techniques proposed in this thesis.
|
||
At the encoder-side various video streams are generated with different levels of compression efficiency and error robustness.
|
||
Then, a pre-training \ac{nn} is used to predict which video stream is expected to achieve the highest decoded video quality for a particular scenario. The selection is based on different input parameters, which include information from the video signal, the coded stream and the transmission network.
|
||
|
||
Finally, Chapter~\ref{Chapter:Conclusions} concludes this dissertation and presents some suggestions for future work.
|
||
Appendix~\ref{Appen:Sequences} illustrates the original test signals used in simulations throughout the thesis. This includes a partial set of the \ac{hevc} test materials, as well as, other external test sequences.
|
||
Appendix~\ref{Appen:Papers} presents a summary of the published papers, describing the contributions of the research work of this thesis.
|
||
|
||
|
||
|