jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Review introduction after PA comments

Browse Source
This commit is contained in:
jfmcarreira
2018-07-09 20:01:17 +01:00
parent 01abdabf71
commit 35de982fc2
6 changed files with 62 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
010_Introduction/Introduction.tex
+56 -30
View File
@@ -8,30 +8,38 @@
\glsreset{h264}
The developments in video compression and networking technologies over the recent years have made the deployment of many applications over the internet possible such as video conferencing, video streaming, video publishing, among others. The increasing diversity of services, the demands for high quality multimedia and the growing popularity of \ac{uhd} formats (\emph{e.g.}, $4k\times2k$ or $8k\times4k$ resolution and beyond) are pushing technologies forward and opening new research directions.
Moreover, the emergent video formats and increased resolutions are creating even stronger needs for coding efficiency superior to the MPEG-4/\ac{h264}~\cite{H264AVCStandard2003,H264OverviewJul2003} capabilities.
The need is even stronger when higher resolutions are needed by stereoscopic and multi-view videos, due to the increased number of camera views that need to be delivered simultaneously. Moreover, the traffic generated by mobile multimedia applications and the video-on-demand services of ever increasing popularity impose new challenges to the existing networks.
Thus, a new coding standard was developed by the ITU-T \ac{vceg} and the \ac{iso}/\ac{iec} \ac{mpeg} standardisation organisations, referred to as H.265 - \ac{hevc}~\cite{H265StandardFeb2018,HEVCOverviewDez2012}.
Over the recent years new advances in video compression and networking technologies have been enabling the development of many multimedia applications over the internet, such as video conferencing, video streaming, video publishing, among others.
The increasing diversity of services, the demands for high quality multimedia and the growing popularity of \ac{uhd} formats (\emph{e.g.}, $4k\times2k$ or $8k\times4k$ resolution and beyond) are pushing technologies forward and opening new research directions.
Moreover, the emerging new video formats with increased resolutions have created stronger needs for higher coding efficiency, beyond the previous standards, i.e., the MPEG-4/\ac{h264}~\cite{H264AVCStandard2003,H264OverviewJul2003}.
Such need has been driven not only by higher resolutions but also by stereoscopic and multi-view video formats, which increase the number of viewpoints that need to be captured, coded and delivered simultaneously.
Moreover, the traffic generated by mobile multimedia applications \HIGHLIGHT{[XX]} and different video services with ever increasing popularity, e.g., Netflix, impose new challenges to the existing networks.
To cope with such demanding requirements a new coding standard was developed by the ITU-T \ac{vceg} and the \ac{iso}/\ac{iec} \ac{mpeg} standardisation organisations, referred to as H.265 - \ac{hevc}~\cite{H265StandardFeb2018,HEVCOverviewDez2012}.
\glsunset{h264}
The \ac{hevc} is foreseen as the main video coding framework to extend current services towards higher resolution formats delivered with better compression efficiency. Similar to \ac{h264}, \ac{hevc} is a hybrid video codec based on predictive transform coding and an entropy coding step. The coding flexibility and efficiency of \ac{hevc}, results from the use of new coding tools and block structures, which enables new block partitions\cite{HEVCBlock2012}, improved prediction modes \cite{IntraCodingHEVC2012,AMPCodingHEVCJun2012} and new high-level features~\cite{HEVCHighLevelReferenceDez2012}, such as explicit reference picture management and new parameter sets. However, it is known that higher coding efficiency leads to lower robustness to transmission errors and data loss. Summarising, the new coding tools in \ac{hevc} bring certain compression efficiency~\cite{HEVCComparisonDec2012}, but also has some other disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and inherent reduction in error robustness.
The \ac{hevc} is currently the main video coding framework to extend current services towards higher resolution formats delivered with better compression efficiency.
Similar to its predecessor, \ac{hevc} is a block-based hybrid video codec using predictive transform and entropy coding.
The coding flexibility and efficiency of \ac{hevc}, results from the use of new coding tools and block structures, which enables new block partitions\cite{HEVCBlock2012}, improved prediction modes \cite{IntraCodingHEVC2012,AMPCodingHEVCJun2012} and new high-level features~\cite{HEVCHighLevelReferenceDez2012}, such as explicit reference picture management and new parameter sets.
However, it is known that higher coding efficiency leads to lower robustness to transmission errors and data loss.
Therefore, the new coding tools in \ac{hevc} bring not only high compression efficiency~\cite{HEVCComparisonDec2012}, but also some other disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and inherent reduction in error robustness.
\begin{figure}[tb]
\centering
%\includegraphics[width=0.5\columnwidth]{./010_Introduction/HEVCTransmissionStack}
\includegraphics[width=0.7\columnwidth]{./010_Introduction/HEVCTransmissionStackOverview}
\caption{System layer stack compatible with \ac{hevc} video transmission~\cite{HEVCLayerIntegrationDez2012}.}
\label{fig:HEVCStack}
\end{figure}
%\begin{figure}[tb]
%\centering
%%\includegraphics[width=0.5\columnwidth]{./010_Introduction/HEVCTransmissionStack}
%\includegraphics[width=0.7\columnwidth]{./010_Introduction/HEVCTransmissionStackOverview}
%\caption{System layer stack compatible with \ac{hevc} video transmission~\cite{HEVCLayerIntegrationDez2012}.}
%\label{fig:HEVCStack}
%\end{figure}
The error robustness characteristics of \ac{hevc} were analysed in \cite{StudyHEVCErrorsDez2012} confirming that, in general, the \ac{hevc} presents poor error resilience performance, despite its superior coding efficiency in comparison with previous standards.
Thus, when \ac{hevc} streams incur in transmission losses, this results in significant degradation of both objective and subjective quality~\cite{QoEHEVCMay2014,SubjectiveEvalSep2013}.
This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes of \ac{hevc}, which are selected according to a \mbox{rate-distortion~(R-D)} optimisation criteria, always assuming \mbox{error-free} transmission~\cite{ROPE2007}. A possible approach to encode robust video streams is to increase the error resilience of \ac{hevc} by including \mbox{error-aware} mechanisms in the coding process. As a result, the effect of error propagation due to prediction mismatch at the decoding side, will be limited.
%\cite{StudyHEVCErrorsDez2012}
Earlier studies about the error robustness characteristics of \ac{hevc} have shown that in general the use of high efficiency coding tools results in poor error resilience performance, in comparison with previous standards.
Thus, when \ac{hevc} streams incur in transmission losses, this leads to significant degradation of both objective and subjective quality~\cite{QoEHEVCMay2014,SubjectiveEvalSep2013}.
This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes of \ac{hevc}, which assume \mbox{error-free} transmission~\cite{ROPE2007}.
A possible approach to encode robust video streams is to increase the error resilience of \ac{hevc} by including \mbox{error-aware} mechanisms in the coding process. As a result, the effect of error propagation due to prediction mismatch at the decoding side, will be limited.
The widespread use of \ac{hevc} in broadcast and multimedia delivery services, will be based on transport technologies such as RTP~\cite{HEVCStreamMay2012}, \mbox{MPEG-2 TS}~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016}.
Recent developments also enable combinations of different technologies, e.g, \mbox{HTTP-based} streaming and \mbox{MPEG-2 TS}, to increase the flexibility of over-the-top (OTT) video services\cite{Broadcast2017}. For content delivery over heterogeneous networks, the MPEG Media Transport (MMT)~\cite{ATSCMMT2016} is another possible technology to be used with the \ac{hevc} standard. Besides the transport technologies,
the overall error robustness of video transmission may also take advantage of the increasing flexibility of multimedia systems to cope with heterogeneous networks, by using auxiliary streams to aid the video decoder to increase the quality of recovered lost slices~\cite{ErrorResilienceHeterogenousNetworks2016,HeterogeneousStreaming2016}. Moreover, error resilience techniques may also take advantage of the multi-stream based approach, where each stream allows different levels of robustness to errors~\mbox{\cite{HEVCHTTPStreamAug2016,PerfMediaTransport2017}}.
The widespread use of \ac{hevc} in broadcast and multimedia delivery services, is be based on transport technologies such as \ac{rtp}~\cite{HEVCStreamMay2012}, \mbox{MPEG-2 TS}~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016}.
Recent developments also enable combinations of different technologies, e.g, \mbox{HTTP-based} streaming and \mbox{MPEG-2 TS}, to increase the flexibility of over-the-top (OTT) video services\cite{Broadcast2017}. For content delivery over heterogeneous networks, the \ac{mmt}~\cite{ATSCMMT2016} is another possible technology to be used with the \ac{hevc} standard. Besides the transport technologies,
the overall error robustness of video transmission may also benefit from the increasing flexibility of multimedia systems to cope with heterogeneous networks, by using auxiliary streams to aid the video decoder to increase the quality of recovered lost slices~\cite{ErrorResilienceHeterogenousNetworks2016,HeterogeneousStreaming2016}. Moreover, error resilience techniques may also take advantage of the multi-stream based approach, where each stream allows different levels of robustness to errors~\mbox{\cite{HEVCHTTPStreamAug2016,PerfMediaTransport2017}}.
%Despite the different characteristics of the existing services and applications, the users' \ac{qoe} consistently depends on how the compressed streams are adapted to the diverse networking technologies, as well as on how the transmission errors and/or packet losses are handled at the application level ~\cite{HEVCStreamMay2012,HEVCLayerIntegrationDez2012,StudyHEVCErrorsDez2012}. As mentioned before, the \ac{hevc} standard is expected to replace the \ac{h264} in future video services.
@@ -54,36 +62,54 @@ the overall error robustness of video transmission may also take advantage of th
In summary, it is an emergent research problem to increase the adaptability of \ac{hevc} compressed streams for diverse network conditions and its robustness against transmission errors, in order to provide acceptable levels of \ac{qoe} to the end users at all times. The \ac{hevc} reference codec is still under development by multiple proponents from around the world, and there are open questions in terms of error resilience and techniques to increase the concealment performance on the decoder side. Although the \ac{ec} methods have been developed during the past years for \ac{h264}~\cite{2DRecursiveRefinement2010} and recently for \ac{hevc}~\cite{HEVCMCBlockAug2013}, a combined and optimised approach between encoder and decoder is yet to be developed, which would improve the users’ \ac{qoe} by exploiting shared information between the encoder and the decoder sides.
In summary, increasing the error robustness of the highly compressed video streams is a relevant research problem whose solutions will be useful to increase the adaptability for transmission diverse network conditions and to achieve robust communications in order to provide acceptable levels of \ac{qoe} to the end users at all times.
The \ac{hevc} reference codec is continuously being improved and extended development by multiple proponents from around the world, also addressing the problem of error resilience and robust delivered transmission
Although the \ac{ec} methods have been developed during the past years for \ac{h264}~\cite{2DRecursiveRefinement2010} and recently for \ac{hevc}~\cite{HEVCMCBlockAug2013}, a combined and optimised approach between encoder and decoder is yet to be developed, which would improve the users’ \ac{qoe} by exploiting shared information between the encoder and the decoder sides.
\section{\HIGHLIGHT{Main objectives and contributions}}
This research plan addresses robust error resilience and efficient error concealment for the forthcoming multimedia services, based on the recent \ac{hevc} standard. Thus, this work focuses on novel techniques to increase delivered quality of \ac{hevc} bitstreams in the following respects:
The research objectives of this thesis are focused in robust error resilience for efficient error concealment in highly efficiency video decoders.
Thus, this work investigates novel techniques to increase the video quality of \ac{hevc} bitstreams delivered through error-prone networks.
The following objectives were pursued in this work:
\begin{enumerate}
\item \textbf{evaluate the error robustness of \ac{hevc}:}
\item \textbf{evaluation study of the error robustness:} the first objective of this research work is to evaluate the robustness against transmission errors of highly efficiently coded video
\item \textbf{source coding level:} At the encoder side, novel techniques to extend the existing error robustness features of the \ac{hevc} standard will be investigated in order to increase the received video quality over error-prone networks. The new coding features introduced in the standard increase the compression ratio, resulting in more complex dependencies that lead to increased error propagation rate.
\item \textbf{robust video coding:} At the encoder side, novel techniques to extend the existing error robustness of highly predictive video coding techniques are investigated in order to increase the received video quality over error-prone networks.
\item \textbf{error concealment of the received video:} At the decoder side, the error concealment techniques improve the perceived video quality by reconstructing the missing information. The paradigm of error concealment should be evaluated at the encoder side, as part of the encoding process, in order to evaluate the concealment performance and share used information with the decoder to improve the overall performance of the transmission system.
\item \textbf{robust video decoding:} error concealment techniques are used to increase the error robustness of video decoder and improve the perceived video quality.
The paradigm of error concealment are evaluated at the encoder side, as part of the encoding process, in order to optimise the reconstruction process at the decoder side.
\item \textbf{jointly error resilience and concealment:} the ultimate goal of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet higher quality demands for different video contents characteristics and network conditions.
%\item \textbf{jointly error resilience and concealment:} the ultimate goal of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet higher quality demands for different video contents characteristics and network conditions.
\end{enumerate}
\HIGHLIGHT{Contributions}
%Finally, the ultimate goal of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet a set of \ac{qos}/\ac{qoe} requirements.
%Since the ultimate quality judgement is made by end users, subjective tests will constitute a part of the research work in order to validate the objective gains reported as a result of the deployment of various proposed and implemented methods. Given the importance of the \ac{hevc} standard in the near future, the simulation scenarios will be defined based on practical service requirements, such as, bit rate allocation, transmission delay and required QoS.
\section{\HIGHLIGHT{Outline of this thesis}}
This thesis is organised as follows. The current chapter introduces the main research topics of this thesis, describing the motivations and objectives to achieve with this work, as well as the original contributions of this thesis.
This thesis is organised as follows. The current chapter introduces the context, motivation and objectives of this thesis, as well as the original contributions of this thesis.
The first two chapters review the \mbox{state-of-the-art} concepts related with this research work. Chapter~\ref{Chapter:HEVCReview} presents an overview of the \ac{hevc} standard, with some details in the concepts explored in this work, e.g., high-level features and temporal predictions. Chapter~\ref{Chapter:ErrorResilienceReview} reviews some of the work regarding robust video transmission proposed over the last years, since they provide relevant knowledge to the main objectives of this work.
The first two chapters review the \mbox{state-of-the-art} concepts and previous work related with this research. Chapter~\ref{Chapter:HEVCReview} presents an overview of the \ac{hevc} standard, focusing on the concepts explored in this work, e.g., high-level features and temporal predictions. Chapter~\ref{Chapter:ErrorResilienceReview} reviews the most important work dealing with robust video transmission also providing relevant background knowledge to this work.
Chapter~\ref{Chapter:HEVCErrorResilience} firstly presents an evaluation study of the performance of the \ac{hevc} standard under errors for various coding configuration. The study aims at finding the relevant tools introduced in the standard that highly affects the error robustness. Secondly, a new method is proposed to decrease the vulnerability of \ac{hevc} temporal prediction and decrease error propagation, leading to higher video quality in error-prone conditions.
Chapter~\ref{Chapter:HEVCErrorResilience} firstly presents an evaluation study about the error robustness of high efficiently coded video, using the \ac{hevc} as the underlying framework for various coding configuration.
The study presents the relevant tools that mostly affect the error robustness.
Secondly, a new method is proposed to decrease the vulnerability of temporal motion prediction and decrease error propagation, leading to higher video quality in error-prone conditions.
Chapter~\ref{Chapter:ErrorResilienceTechniques} proposes a novel error resilience approach in the context of the \ac{hevc} standard. The motivation for this works is related with the poor error robustness performance show in Chapter~\ref{Chapter:HEVCErrorResilience}. Thus, a new method to distribute the use of reference frames was proposed to reduce the error propagation and improve the \ac{ec} performance. Moreover, to reduce the effect of mismatch motion predictions (also shown in Chapter~\ref{Chapter:HEVCErrorResilience}) a method based on selective motion vector redundancies is proposed. This method is able to improve the decoded video quality using a low amount of overhead.
Chapter~\ref{Chapter:ErrorResilienceTechniques} proposes a novel error resilience approach for high efficiently video codecs.
A new method to distribute the use of reference frames is proposed to reduce the error propagation and improve the \ac{ec} performance.
Moreover, to reduce the effect of mismatch motion predictions (shown in Chapter~\ref{Chapter:HEVCErrorResilience}) a method based on selective motion vector redundancies is proposed.
This method is able to improve the decoded video quality using a small amount of overhead.
Chapter~\ref{Chapter:ConcealmentAwareTechniques} addresses the problem of efficient \ac{ec} in the \ac{hevc}. This work was motivated by the requirements for more accurate \ac{ec} algorithms which are able to reduce the error propagation. Different \ac{ec} algorithms were taken into consideration in order to develop a concealment-aware coding scheme. The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled to the decoder. Such optimal EC\ac{ec} modes are found by simulating transmission losses followed by a \ac{rd} optimisation of the signalling rate and \ac{ec} distortion cost.
A generalised saliency-weighted distortion is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. When data loss occurs the decoder uses the signalling information is used at the decoder, in case of data loss, to improve the reconstruction quality. The simulation results show that the proposed method achieves consistent quality gains in comparison with other reference methods and previous works.
Chapter~\ref{Chapter:ConcealmentAwareTechniques} addresses the problem of robust \ac{ec} of highly efficiently video streams.
%This work was motivated by the requirements for more accurate \ac{ec} algorithms which are able to reduce the error propagation.
Different \ac{ec} algorithms were taken into consideration in order to develop a concealment-aware coding scheme.
The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled to the decoder.
%Such optimal EC\ac{ec} modes are found by simulating transmission losses followed by a \ac{rd} optimisation of the signalling rate and \ac{ec} distortion cost.
A generalised saliency-weighted \ac{rd} optimisation is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer.
%When data loss occurs the decoder uses the signalling information is used at the decoder, in case of data loss, to improve the reconstruction quality.
%The simulation results show that the proposed method achieves consistent quality gains in comparison with other reference methods and previous works.
Chapter~\ref{Chapter:EndToEndRobustSystem} ... \HERE