\graphicspath{{010_Introduction}} \glsreset{hevc} \chapter{Introduction} \label{Chapter:Intro} \section{Context and motivation} \glsreset{h264} Over the recent years new advances in video compression and networking technologies have been enabling the development of many multimedia applications over the internet, such as video conferencing, video streaming, video publishing, among others. The increasing diversity of services, the demands for high quality multimedia and the growing popularity of \ac{uhd} formats (\emph{e.g.}, $4k\times2k$ or $8k\times4k$ resolution and beyond) are pushing technologies forward and opening new research directions. Moreover, the emerging new video formats with increased resolutions have created stronger needs for higher coding efficiency, beyond the previous standards, i.e., the MPEG-4/\ac{h264}~\cite{H264AVCStandard2003,H264OverviewJul2003}. Such need has been driven not only by higher resolutions but also by stereoscopic and multi-view video formats, which increase the number of viewpoints that need to be captured, coded and delivered simultaneously. Moreover, the ever increasing traffic generated by mobile multimedia applications \cite{CiscoMobile1621_Feb2017} and different video services with ever increasing popularity, e.g., Netflix, impose new challenges to the existing networks. To cope with such demanding requirements a new coding standard was developed by the ITU-T \ac{vceg} and the \ac{iso}/\ac{iec} \ac{mpeg} standardisation organisations, referred to as H.265 - \ac{hevc}~\cite{H265StandardFeb2018,HEVCOverviewDez2012}. \glsunset{h264} The \ac{hevc} is currently the main video coding framework to extend current services towards higher resolution formats delivered with better compression efficiency. Similar to its predecessor, \ac{hevc} is a block-based hybrid video codec using predictive transform and entropy coding. The coding flexibility and efficiency of \ac{hevc}, results from the use of new coding tools and block structures, which enables new block partitions\cite{HEVCBlock2012}, improved prediction modes \cite{IntraCodingHEVC2012,AMPCodingHEVCJun2012} and new high-level features~\cite{HEVCHighLevelReferenceDez2012}, such as explicit reference picture management and new parameter sets. However, it is known that higher coding efficiency leads to lower robustness to transmission errors and data loss. Therefore, the new coding tools in \ac{hevc} bring not only high compression efficiency~\cite{HEVCComparisonDec2012}, but also some other disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and inherent reduction in error robustness. %\cite{StudyHEVCErrorsDez2012} Earlier studies about the error robustness characteristics of \ac{hevc} have shown that in general the use of high efficiency coding tools results in poor error resilience performance, in comparison with previous standards. Thus, when \ac{hevc} streams incur in transmission losses, this leads to significant degradation of both objective and subjective quality~\cite{QoEHEVCMay2014,SubjectiveEvalSep2013}. This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes of \ac{hevc}, which assume \mbox{error-free} transmission~\cite{ROPE2007}. A possible approach to encode robust video streams is to increase the error resilience of \ac{hevc} by including \mbox{error-aware} mechanisms in the coding process. As a result, the effect of error propagation due to prediction mismatch at the decoding side, will be limited. The widespread use of \ac{hevc} in broadcast and multimedia delivery services, is be based on transport technologies such as \ac{rtp}~\cite{HEVCStreamMay2012}, \mbox{MPEG-2 TS}~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016}. Recent developments also enable combinations of different technologies, e.g, \mbox{HTTP-based} streaming and \mbox{MPEG-2 TS}, to increase the flexibility of over-the-top (OTT) video services\cite{Broadcast2017}. For content delivery over heterogeneous networks, the \ac{mmt}~\cite{ATSCMMT2016} is another possible technology to be used with the \ac{hevc} standard. Besides the transport technologies, the overall error robustness of video transmission may also benefit from the increasing flexibility of multimedia systems to cope with heterogeneous networks, by using auxiliary streams to aid the video decoder to increase the quality of recovered lost slices~\cite{ErrorResilienceHeterogenousNetworks2016,HeterogeneousStreaming2016}. Moreover, error resilience techniques may also take advantage of the multi-stream based approach, where each stream allows different levels of robustness to errors~\mbox{\cite{HEVCHTTPStreamAug2016,PerfMediaTransport2017}}. %Despite the different characteristics of the existing services and applications, the users' \ac{qoe} consistently depends on how the compressed streams are adapted to the diverse networking technologies, as well as on how the transmission errors and/or packet losses are handled at the application level ~\cite{HEVCStreamMay2012,HEVCLayerIntegrationDez2012,StudyHEVCErrorsDez2012}. As mentioned before, the \ac{hevc} standard is expected to replace the \ac{h264} in future video services. %The system interface of \ac{hevc} is an important component %in the media access chain, and a prerequisite for its successful %deployment in various consumer electronics applications. We %consider four broad categories of systems: broadcast delivery %over digital television and Internet Protocol Television (IPTV) %channels including video-on-demand (VoD), streaming over %the Internet using the Hypertext Transport Protocol (HTTP), %real-time communications such as video conferencing, and %store-forward delivery based on a file or a physical memory %delivered to the end user (such as: camcorded videos or Blu- %ray Disks). To this end, different system technologies have %been standardized. These technologies can be summarized as %follows. \begin{figure}[tb] \centering \input{./010_Introduction/RobustVideoSystem.tikz} \caption{Generalised diagram of a robust video transmission system.} \label{FIG1:RobustVideoTransmission} \end{figure} In summary, increasing the error robustness of the highly compressed video streams is a relevant research problem whose solutions will be useful to increase the adaptability for transmission diverse network conditions and to achieve robust communications in order to provide acceptable levels of \ac{qoe} to the end users at all times. In general, increased error robustness in video transmission can be jointly achieved at three different stages: (i) using robust video encoding, by selecting robust coding modes, (ii) after encoding, by adding robust features to the compressed streams as side-information and (iii) using robust video decoding techniques, e.g., efficient \ac{ec}. This is exemplified in Figure~\ref{FIG1:RobustVideoTransmission} where at the encoder different layer are used to increase the error robustness, which results in a video stream composed by a \ac{vcl} and \ac{sei} layers. Finally, at the decoder two processing layers are used to achieve robust video decoding. The \ac{hevc} reference codec is continuously being improved and extended development by multiple proponents from around the world, also addressing the problem of error resilience and robust delivered transmission Although the \ac{ec} methods have been developed during the past years for \ac{h264}~\cite{2DRecursiveRefinement2010} and recently for \ac{hevc}~\cite{HEVCMCBlockAug2013}, a combined and optimised approach between encoder and decoder is yet to be developed, which would improve the users’ \ac{qoe} by exploiting shared information between the encoder and the decoder sides. \section{\HIGHLIGHT{Main objectives and contributions}} The research objectives of this thesis are focused in robust error resilience for efficient error concealment in highly efficiency video decoders. Thus, this work investigates novel techniques to increase the video quality of \ac{hevc} bitstreams delivered through error-prone networks. The following objectives were pursued in this work: \begin{enumerate} \item \textbf{evaluation study of the error robustness:} the first objective of this research work is to evaluate the robustness against transmission errors of highly efficiently coded video. The study conclusions aimed at finding relevant research directions to increase the error robust of coded streams. The results and conclusions of this study were published in the conference paper \ref{Conf:ErrorResilience} and are part of the journal paper \ref{Journal:TwoStage} (see Appendix \ref{Appen:Papers}). \item \textbf{robust video coding:} At the encoder side, novel techniques to extend the existing error robustness of highly predictive video coding techniques are investigated in order to increase the received video quality over error-prone networks. This research topic resulted in a new method to reduce the error propagation in motion vector coding algorithms published in the conference paper \ref{Conf:DynamicTMVP}. Moreover, a novel two-stage approach was devised to improve the error robustness of highly efficiently coded video at the encoder and streaming levels, resulting in two conference publications \ref{Conf:MotionVectorRedundancies} and \ref{Conf:Chkb}, and a journal publication \ref{Journal:TwoStage}. \item \textbf{robust video decoding:} error concealment techniques are used to increase the error robustness of video decoder and improve the perceived video quality. The paradigm of error concealment are evaluated at the encoder side, as part of the encoding process, in order to optimise the reconstruction process at the decoder side. This research resulted in a novel error concealment-aware technique to optimise the error concealment process at the encoder-side, which was proposed for intra-frame only in the conference paper \ref{Conf:ConcAware} and for a generic robust video transmission in the journal \ref{Journal:ConcAware}. %\item \textbf{jointly error resilience and concealment:} the ultimate goal of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet higher quality demands for different video contents characteristics and network conditions. \end{enumerate} %\HIGHLIGHT{Contributions} %Finally, the ultimate goal of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet a set of \ac{qos}/\ac{qoe} requirements. %Since the ultimate quality judgement is made by end users, subjective tests will constitute a part of the research work in order to validate the objective gains reported as a result of the deployment of various proposed and implemented methods. Given the importance of the \ac{hevc} standard in the near future, the simulation scenarios will be defined based on practical service requirements, such as, bit rate allocation, transmission delay and required QoS. \section{\HIGHLIGHT{Outline of this thesis}} This thesis is organised as follows. The current chapter introduces the context, motivation and objectives of this thesis, as well as the original contributions of this thesis. The first two chapters review the \mbox{state-of-the-art} concepts and previous work related with this research. Chapter~\ref{Chapter:HEVCReview} presents an overview of the \ac{hevc} standard, focusing on the concepts explored in this work, e.g., high-level features and temporal predictions. Chapter~\ref{Chapter:ErrorResilienceReview} reviews the most important work dealing with robust video transmission also providing relevant background knowledge to this work. Chapter~\ref{Chapter:HEVCErrorResilience} firstly presents an evaluation study about the error robustness of high efficiently coded video, using the \ac{hevc} as the underlying framework for various coding configuration. The study presents the relevant tools that mostly affect the error robustness. Secondly, a new method is proposed to decrease the vulnerability of temporal motion prediction and decrease error propagation, leading to higher video quality in error-prone conditions. Chapter~\ref{Chapter:ErrorResilienceTechniques} proposes a novel error resilience approach for high efficiently video codecs. A new method to distribute the use of reference frames is proposed to reduce the error propagation and improve the \ac{ec} performance. Moreover, to reduce the effect of mismatch motion predictions (shown in Chapter~\ref{Chapter:HEVCErrorResilience}) a method based on selective motion vector redundancies is proposed. This method is able to improve the decoded video quality using a small amount of overhead. Chapter~\ref{Chapter:ConcealmentAwareTechniques} addresses the problem of robust \ac{ec} of highly efficiently video streams. %This work was motivated by the requirements for more accurate \ac{ec} algorithms which are able to reduce the error propagation. Different \ac{ec} algorithms were taken into consideration in order to develop a concealment-aware coding scheme. The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled to the decoder. %Such optimal EC\ac{ec} modes are found by simulating transmission losses followed by a \ac{rd} optimisation of the signalling rate and \ac{ec} distortion cost. A generalised saliency-weighted \ac{rd} optimisation is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. %When data loss occurs the decoder uses the signalling information is used at the decoder, in case of data loss, to improve the reconstruction quality. %The simulation results show that the proposed method achieves consistent quality gains in comparison with other reference methods and previous works. Chapter~\ref{Chapter:EndToEndRobustSystem} ... \HERE Chapter~\ref{Chapter:Conclusions} concludes this dissertation and presents some suggestions for future work. Appendix~\ref{Appen:Sequences} illustrates the original test signals used in simulations throughout the thesis. This includes a partial set of the \ac{hevc} test material and other external test material. Appendix~\ref{Appen:Papers} presents a summary of the published papers, describing the contributions of the research work of this thesis.