% Figure folders \renewcommand{\FigsPath}{./021_VideoCoding/Figures} % \chapter{\HIGHLIGHT{The \ac{hevc} Standard}} \chapter[The High Efficiency Video Coding Standard]{The High Efficiency Video \\Coding Standard} %\chapter{\HIGHLIGHT{The State-of-Art Video Coding}} \label{Chapter:HEVCReview} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Introduction} \glsunset{h264} This chapter presents an overview of the H.265 - \ac{hevc}~\cite{H265StandardFeb2018} standard, focusing on the most relevant features of the encoding algorithm for the work developed in this thesis. The \ac{hevc} standard aims to address essentially all existing applications of \ac{h264}~\cite{H264AVCStandard2003}, extending it in two keys aspects: increased video resolution and increased use of parallel processing architectures. In order to increase the flexibility and follow the increase in the video resolution, the \ac{hevc} standard adopts a new block partition structure, as well as, new prediction modes. In this chapter these new techniques introduced are described, especially those which may have a significant impact in the error robustness characteristics of the \ac{hevc}. %The standard also improves the intra~\cite{IntraCodingHEVC2012} and inter~\cite{MotionCodingHEVC2013} coding and includes new high-level features~\cite{HEVCHighLevelReferenceDez2012}. This chapter will describe main features and improvements introduced in the standard. This chapter is organized as follows. Section~\ref{Section2:GeneraVideoSystem} introduced the functional blocks of a generic video coding system, and Section~\ref{Section2:Overview} provides the overview of the \ac{hevc} standard features. Subsequently, Section~\ref{Section2:HighLevel} describes the high-level features which are used to carry relevant information regarding the bitstream, Section~\ref{Section2:CodingStructures} describes the coding structures used to divide the video frames into smaller partitions and Section~\ref{Section2:PredictionModes} describes the main predictions techniques, namely intra and inter coding. Finally, a briefly description of the transform and quantisation, entropy coding and in-loop filters is given in Sections~\ref{Section2:TransformQuant}, \ref{Section2:Entropy} and \ref{Section2:Filters}, respectively. Section~\ref{Section2:Summary} concludes this chapter. \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{General hybrid video coding} \label{Section2:GeneraVideoSystem} A digital video is a temporal succession of images, also referred to as pictures or frames, which usually presents a significant amount of temporal and spatial correlations. In general, video coding and distribution systems follow a generic set of tools: pre-processing, encoding, storage or transmission, decoding and post-processing. The pre-processing unit is used adapt the raw video signal, in which if the format of raw videos from video cameras is analogue, it will be firstly digitized into RGB images and then converted into YUV representation~\cite{DataCompression1997}. In the next processing tool, the video data will be compressed by the respective encoding unit by reducing its redundancy. The encoding process is able to compress the raw video in a high compression ratio and maintains a good trade-off between video quality and bitrate through high efficient algorithms. On the contrary, the output video for displaying can be acquired through decoding and post-processing modules. % \begin{figure}[tb] % \centering % \input{\FigsPath/GeneralVideoSystem.tikz} % \caption{Block diagram of a generic video coding system.} % \label{FIG2:GeneralVideoSystem} % \end{figure} Since H.261, the hybrid video coding scheme has been the basic structure for all video coding standards and recommendations of ITU-T \ac{vceg} and \ac{iso}/\ac{iec} \ac{mpeg}. While the structure has not been changed, the algorithms represented by the building blocks have been refined and the applicable configuration for the algorithms has become more and more flexible over years. The coding scheme is called hybrid as it combines temporal prediction between pictures of the video sequence with transform coding techniques for the prediction error \cite{VideoCodingFundamentals2015} as shown in Figure~\ref{Figure2:HybridCodec}~\cite{BookLucas2017}. \begin{figure}[tb] \centering \input{\FigsPath/HybridCodingBlockDiagramPrePost.tikz} \caption{Block diagram of a generic video coding system, including the inner blocks of a generic hybrid encoding and decoding scheme.} \label{Figure2:HybridCodec} \end{figure} As aforementioned, the main purpose of video coding is to compress the video by reducing its redundancy. In general, there are two kinds of redundancies existing in video signal: spatial and temporal. On the one hand, the spatial redundancy is exploited through intra frame prediction, of which the mechanism is to find the best matching block for current one from its neighbours. This is due to a high correlation between neighbouring pixels inside a frame. As result, in intra frame prediction a block is reconstructed by the blocks that has already been encoded and stored in the bitstream for output. On the other hand, inter frame prediction is designed to deal with the temporal redundancy existed between consecutive frames such as the frame in the past or in the future. Temporal redundancy is normally referred as the correlation between two adjacent frames since they usually have similar backgrounds or a same object with a relative motion. Thus, the purpose of inter-frame prediction is to search for the best matching block for the current one in the reference frames, ranging from past to future frames. After the best matching candidate for the current block is found by either intra- or inter-frame prediction, only the difference between the original block and the predicted one needs to be encoded. As seen from Figure~\ref{Figure2:HybridCodec}, these differences (also called residual signal) will be further processed and converted into the coefficients of transform domain (T). In general, redundant information can be eliminated through transforming. Subsequently, the transformed coefficients are quantized (Q) to reduce the range of the signal and save the number of bits required for signalling. Finally, the quantized transform coefficients together with the prediction information are compressed to form an output bitstream for transmission by the entropy encoder, as shown in Figure~\ref{Figure2:HybridCodec}. On contrary, the decoding part performs the inverse operations on the bitstream to reconstruct the video for playing. Subsequently, the aforementioned blocks are described in the context of \ac{hevc} standard. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Overview the standard features} \label{Section2:Overview} As for all the previous coding standards developed by JCT-VC, in \ac{hevc}~\cite{HEVCOverviewJul2012,HEVCOverviewDez2012,BookHEVC2014} only the bitstream structure and the coding syntax are standardised. Included are the bitstream constraints, the definition of the semantic meaning of syntax elements and the decoding process, such that every decoder conforming to the standard generates the same output when given a correct bitstream. Such limitation of scope in the standard allows maximal freedom for developers and enables specific optimisations to cope with various applications’ requirements (\emph{e.g.}, balancing the compression ratio and the coding complexity). Therefore, in order to be considered as an \ac{hevc} encoder, the algorithm needs to ensure that the produced bitstream complies with the standard specifications. Figure \ref{fig:HEVCBlockDiagram} illustrates the block diagram of the hybrid \ac{hevc} encoder. It follows the same overall architecture as its predecessors starting from H.261. The encoder uses both intra- and inter-picture prediction to exploit spatial and temporal redundancies, respectively. Subsequently, the prediction residue transformed by linear spatial transform, which is then quantised, and entropy coded. This allows to reduce the bitstream redundancy to a minimum. The resulting bitstream is combined with control information, which signals the prediction chosen, motion vectors, filtering operations and bitstream headers. \begin{figure}[tb] \centering % \includegraphics[width=0.75\columnwidth]{\FigsPath/HEVC/HEVCBlockDiagram} \input{\FigsPath/HEVC/HEVCBlockDiagram.tikz} \caption{Generic \ac{hevc} standard architecture.} \label{fig:HEVCBlockDiagram} \end{figure} As depicted in Figure \ref{fig:HEVCBlockDiagram}, each picture can be compressed with intra coding algorithms or inter prediction using previously encoded and reconstructed pictures kept in the \ac{dpb} as reference. The encoding process follows a block-based approach as in previous standards with the larger units referred to as \ac{ctu} with a maximum size of $64\times64$. In order to improve the overall quality of the reconstructed pictures and smooth the coding artefacts, \ac{hevc} uses the deblocking and \ac{sao} filters. The encoder has a built-in decoder to generate the reference pictures. In the decoding loop, the residual information is delivered to the inverse quantisation and transform modules. The finally reconstructed samples are obtained by adding the predicted signal and applying the filtering operations. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{High-level syntax} \label{Section2:HighLevel} This section describes the high-level components of the \ac{hevc} standard that provide signalling information and enable bitstream partitioning. \ac{hevc} inherits several high-level features from \ac{h264}, e.g., the \ac{nal} units, parameter sets, the use of \ac{poc}, and \ac{sei} messages for auxiliary data. Some high-level features that exist in \ac{h264} were not included in \ac{hevc}, for example \ac{fmo}, redundant slices, arbitrary slice order, data partitioning, and switching slices. Several new high-level features are introduced, such as the \ac{vps}, the tiles and wavefront tools for parallel processing, the dependent slices for reduced delay, and a new reference picture management concept. Moreover, new pictures types were introduced new picture types to increase the random-access flexibility, and temporal sub-layer access pictures. \subsection{Network Abstraction Layer} Similar to \ac{h264}~\cite{H264AVCStandard2003}, the \ac{hevc} standard used a bitstream based on \glsfirst{nal} units, which are useful to deal with lossy network environments, as it is useful for bitstream partitioning and synchronization \cite{H264OverIPJul2003}. The \ac{nal} provides the ability to map the \ac{vcl} data that represents video frames on various transport layers, such as \ac{rtp}, MPEG-2 Systems and MPEG DASH. \Gls{nal} units are classified into two categories: (i) \ac{vcl} and (ii) non-\ac{vcl}, as they carry coded video frames or associated data, respectively. In the \ac{hevc} standard two bytes are allocated to describe the content of each \ac{nal} unit. These bytes are used to identify which type of \ac{nal} is being decoded and are transmitted in a fixed format form without variable length codes. This allow extra error robustness but increases the bit allocation. Figure~\ref{Figure2:NalHeader} show the structure of the \ac{hevc} \ac{nal} unit header. The first bit \textit{F}, referred to as \textit{forbiden\_bit}, is always zero in order to ensure that a start code can be detected across different protocol standards. This is followed by the \ac{nal} unit type which identifies its content, enabling the decoder to access if it is required for a particular decoding purpose, or if it can be discarded. Another syntax element defined by the \ac{hevc} standard in the \ac{nal} unit header is the TIDP which marks the temporal layer to which the unit belongs. This allows for temporal scalability (layers ranging from 0 to 6) by immediately discarding the \ac{nal} unit if it belongs to a layer higher than the desired one. Finally, six bits are reserved for future extensions, preventing the creating of extra \gls{nal} units for scalable and multiview extensions as it was done in \ac{h264}. \begin{figure}[tb] \centering \newcommand\bithead[1]{\bitbox[]{#1}{\scriptsize #1}} \newcommand{\colorbitbox}[3]{% \rlap{\bitbox{#2}{\color{#1}\rule{\width}{\height}}}% \bitbox{#2}{#3}} \begin{bytefield}[bitwidth=20pt]{16} % \bitheader{0-15} \\ \colorbitbox{\TikZColorFill}{1}{F} & \colorbitbox{gray!60}{6}{NAL Type} & \colorbitbox{\TikZColorFill}{6}{Reserved} & \colorbitbox{gray!60}{3}{TIDP} \end{bytefield} \caption{Two bytes \ac{nal} unit header in \ac{hevc}.} \label{Figure2:NalHeader} \end{figure} \subsection{Parameters set} The encoder uses a set of signalling packets as a robust method to share data with the decoder, referred to as parameter set units. This information was introduced in previous standards to deal with vulnerabilities in the transport of signalling information between the encoder and decoder, which compromised the streaming applications. Within this approach the set of parameters are included in the bitstream and are repeated as many times as required by the application. This structure is inherited from the \ac{h264} standard\cite{H264OverIPJul2003} with modifications to cope with new coding tools. The \ac{hevc} includes three sets of parameters, \ac{sps}, \ac{pps} and the newly introduced \glsfirst{vps}\cite{HEVCHighLevelReferenceDez2012}. % \ac{sps} include sequence related information, that include the decoder operation point, flags to enable or disable certain tools within a given profile. \Gls{pps} include picture related information, that may change for every picture, e.g., \ac{qp} and tilling information. The new set of parameters introduced in the \ac{hevc} standard (\ac{vps}) contains information related to the different layers of the video, avoiding duplications like in the \ac{h264} and its extensions. \Gls{vps} aims to provide signalling for every layer, guaranteeing a clean and extendible standard, capable of supporting multiple layers. Moreover, \ac{vps} carries information for session negotiation (\emph{e.g.} profile, level and tier). The other sets are the \ac{sps} and \ac{pps}. \Glspl{sps} are related to the whole sequence (\emph{i.e.}, should affect all coded slices), and contain information related with the decoder operation point, as well as, flags to control optional tools and scalability control. \Glspl{pps} carry information that may change for every picture, such as, initial \ac{qp}, flags for picture related tools and tilling information. \subsection{Picture types} \begin{table}[tb] \centering \caption{Picture categories in \ac{hevc} standard.}\label{Table2:PictureTypes} \TABLESVSPACE\small \begin{tabular}{m{1.3cm}|m{5.2cm}|m{6.3cm}} \toprule \multicolumn{3}{c}{\textbf{Random-access points pictures}} \\ \midrule IDR & Instantaneous decoding refresh & All leading pictures can be decoded \\ \midrule CRA & Clean random access & Pictures in the display order can be decoded\\ \midrule BLA & Broken link access & Leading pictures cannot be decoded \\ \midrule \midrule \multicolumn{3}{c}{\textbf{Leading pictures}} \\ \midrule RADL & \multicolumn{2}{l}{Random access decodable picture}\\ \midrule RASL & \multicolumn{2}{l}{Random access skipped picture}\\ \midrule\midrule \multicolumn{3}{c}{\textbf{Temporal sub-layer access pictures}} \\ \midrule TSA & Temporal sub-layer access & Allows switching to a given temporal layer or higher \\ \midrule STSA & Step-wise temporal sub-layer access & Allows switching to a given layer only \\ \bottomrule \end{tabular} \end{table} The \ac{hevc} standard introduces several pictures types as show in Table~\ref{Table2:PictureTypes}. These pictures are divided in the following categories: (i) random-access points, (ii) leading pictures and (iii) temporal sub-layer access pictures~\cite{HEVCHighLevelReferenceDez2012,BookHEVC2014}. \subsubsection{Random-Access Point} The \ac{rap} pictures are used to provide points in the bitstream where it is possible to start the decoding process. The standard defines that a conforming bitstream must start with a \ac{rap} picture. These frames must belong to temporal sub-layer 0 and be coded using intra coding techniques, i.e., must not using previous decoded frames as reference. One should not that frames may be compressed with solely intra prediction and not be marked as random-access points. It is always possible to decode from the \ac{rap} frame and onwards to output any subsequent pictures in output order even if all pictures that precede the \ac{rap} in decoding order are discarded from the bitstream. The \ac{idr} frame is an intra coded picture that completely refresh the decoding process and starts a new conforming video stream. This means that neither the \ac{idr} nor any other frame following in the decoding order have dependencies to a prior frame. \ac{idr} frames are allowed to have leading pictures, but they must be decodable, thus creating a true random-access point where all subsequent frames in the decoding order can be decoder and displayed. In order to increase the coding efficiency, the \ac{hevc} standard introduces a new \ac{rap} picture, referred to as \ac{cra} picture. Theses frames do not refresh the decoding process and allows leading pictures to depend on frames that precede the \ac{cra} picture in the decoding order. This enables a more efficient prediction structure, while maintaining an access point to begin the decoding process. The introduction of \ac{cra} frames in \ac{hevc} were able to improve the coding efficiency by up to 6\% \cite{HEVCCRAPics2011}. The \ac{hevc} supports bitstream splicing, which means taking a particular bitstream starting from a \ac{rap} (both \ac{idr} and \ac{cra} frames) and introduce it in another bitstream at a random-access instance. Therefore, whenever this is performed for a bitstream starting from a \ac{cra} frame, the leading frames associated might not be correctly decoded. Therefore, in order to make splicing operation straightforward the \ac{nal} units of the \ac{cra} picture are changed to \ac{nal} units of type \ac{bla}. In case of bitstream splicing starting from an \ac{idr} frame the leading pictures does not have dependencies to frames prior the \ac{rap}, thus converting it to \ac{bla} is not required. \subsubsection{Leading and trailing pictures} \begin{figure}[tb] \centering \input{\FigsPath/HEVC/PictureTypes_Leading_Trailing.tikz} \caption{Leading and trailing pictures associated with a random-access point.} \label{Figure2:LeadingTralingPics} \end{figure} Leading pictures correspond to those frame that follow a particular \ac{rap} frame in the decoding order but precedes it in the output order. A trailing picture is a frame that follows a particular \ac{rap} frame in both decoding and output order and must not depend on any leading frames nor any trailing frame belonging to a previous \ac{rap}. Figure~\ref{Figure2:LeadingTralingPics} shows examples of leading and trailing pictures. In the figure, frames are illustrated in the output order and the number associated with the frame type (i.e., I, P and B) represents the decoding order. In this example frame $I_1$ corresponds to a \ac{cra} picture. Leading and trailing pictures are considered to be associated with the closest previous \ac{rap} picture in decoding order, such as frame $I_1$ in Figure~\ref{Figure2:LeadingTralingPics}. Also, all leading frame of a \ac{rap} frame must precede in decoding order all trailing frames that are associated with the same \ac{rap}. This means that the following order is imposed by the \ac{hevc} standard: 1) \ac{rap} picture, 2) associated leading pictures and 3) associated trailing pictures. There is two types of leading pictures as shown in Table~\ref{Table2:PictureTypes}. On the one hand, the \ac{radl} pictures only depend on the associated \ac{rap} pictures and do not have any prior dependencies to trailing pictures associated with previous \ac{rap} frame. These frames can be correctly decoded. On the other hand, \ac{rasl} picture may have dependencies to a prior trailing picture. Therefore, when a random-access is performed to the associated \ac{rap} frame, these frames cannot be correctly decoded, and should be skipped. \subsubsection{Temporal sub-layer switching pictures} \begin{figure}[tb] \centering % \includegraphics[width=0.75\columnwidth]{\FigsPath/HEVC/Pictures_TSA} \input{\FigsPath/HEVC/PictureTypes_TSA.tikz} \caption{Temporal sub-layer access pictures example.} \label{Figure2:TSAExample} \end{figure} A \ac{tsa} picture is a trailing picture that marks a temporal sub-layer switching point. When a decoder is decoding a sub-set of the temporal sub-layers in the bitstream and encounters a \ac{tsa} picture type of the temporal sub-layer just above the maximum temporal sub-layer it is decoding, it is possible for the decoder to switch up to and decode any number of additional temporal sub-layers. For example, picture $P_6$ in Figure~\ref{Figure2:TSAExample} can use the \ac{tsa} frame type since it only depends on frame belonging to temporal sub-layer 0 for prediction, as well as, any subsequent frame predicted from the \ac{tsa} frame $P_6$. In the example of figure if the decoder was only decoding the temporal sub-layer 0 it can after $P_6$ starts decoding both sub-layer 0 and 1 or decode all the available three sub-layers. Although theses frame allows higher flexibility in terms of sub-layer switching, they severely constrain the prediction of trailing the \ac{tsa} frame. To reduce the constrains in frame prediction due to sub-layer switching, \ac{hevc} introduces the \ac{stsa} picture. These pictures have similar purpose as the \ac{tsa} frames but they only guarantee that frames in the same temporal sub-layer as the \ac{stsa} frame in the decoding order are able to be correctly reconstructed. This is guaranteed by preventing any dependency to frame preceding the \ac{stsa} to be used as reference in its temporal sub-layer. One example of a \ac{stsa} frame is given in also given in Figure~\ref{Figure2:TSAExample} in frame $P_2$. This frame can be used to switch to the sub-layer 1 as it does not have dependencies to any prior frame in the same sub-layer ($P_2$ only depends on $I_0$). However, it cannot be classified as \ac{tsa} picture since $P_3$ has dependencies to prior frame in the decoding order to $P_1$. Summarising, \ac{stsa} frames can be used to switch to a particular sub-layer as the \ac{tsa} can be used to switch to any sub-layer. \subsection{Picture partitioning} The high-level segmentation of a picture in \ac{hevc} is achieved through four techniques: regular slices, dependent slices, tiles and \glsfirst{wpp}. Picture partitioning normally serves one or more of the following three purposes: \begin{description} \item[Error robustness:] partitioning the picture into smaller self-contained units in order to gain error robustness by the ability to re-synchronize both the parsing and decoding processes in case of data losses. \item[Network adaptation:] adapt to the network constraint of \ac{mtu} size, found for example in \ac{ip} networks. Such packetisation scheme restricts the maximum number of payload bits within a slice regardless of the size of the coded frame. To keep each slice within this limit and minimise the packetisation overhead, a variable number of coding units is used for each slice. \item[Parallel processing:] partitioning the coded frame into units which can be processed in parallel. This is achieved by dividing the coding units in such way that they can be encoded and decoded independently of each other’s. \end{description} \subsubsection{Slices} The \ac{hevc} standard preserves the slice structure present in the \ac{h264} standard. A slice may comprise either the entire frame or a section of it, and all the associated data (\emph{i.e.}, entropy symbols, prediction and residue information) can be independently decoded. Each slice is transported in a different \ac{nal} unit. As dependencies across slice boundaries are disabled, each slice can be independently reconstructed, regardless if other slices were lost or incorrectly decoded. Slice partitioning is the only tool that can be used for parallel processing in a virtually identical form as in \ac{h264}. As they are independent from each other, it is straightforward to process multiple slices in parallel, without inter-process communication (except for inter-frame prediction). Although this is a simple approach, it incurs in substantial coding overhead due to the higher number of slice headers, and reduction of causal neighbours for prediction (due to lack of predictions across slice boundaries). Another disadvantage of using slices for parallel processing is that they are also used to meet the \gls{mtu} size constrains. Therefore, it might be impossible to meet both requirements using only one technique. Another partitioning tool in \ac{hevc} standard is the dependent slices. These slices allow the partitioning of the frame at coding unit boundaries without breaking intra-frame predictions. Moreover, as they have a smaller slice header they are more efficient in terms of signalling overhead than regular slices. Dependent slices are normally used to reduce the end-to-end delay by allowing part of the slice to transmitted while the rest of it are still being processed. \begin{figure}[tb] \centering \begin{tabular}{x{.45\columnwidth}x{.45\columnwidth}} \input{\FigsPath/HEVC/Tiles_Slices_Example_1.tikz} & \input{\FigsPath/HEVC/Tiles_Slices_Example_2.tikz} \\ (a) & (b) \\ \end{tabular} \caption{Examples of the interaction between tiles and slices.} \label{Figure2:HEVCTiles} \end{figure} \subsubsection{Tiles} A new data partition has been included in the \ac{hevc} standard, referred to as Tiles\cite{HEVCTilesDec2013}. Tiles enable the partition of a coded frame into a smaller group of \acs{ctu} using horizontal and/or vertical boundaries. Figure~\ref{Figure2:HEVCTiles} shows two examples of slice partitioning using tiles. In both cases the frame is partitioned into three tiles by using two vertical boundaries. By defining tiles using only boundaries it allows for an efficient signalling with a small overhead. These boundaries are defined in the \ac{pps} \ac{nal} units. Although the \ac{hevc} standard enables the use of both slices and tiles at the same time, it impose constrains in order to reduce the implementation complexity. One of two cases must be used: (i) all \acs{ctu} within a tile must belong to the same slice, or (ii) all \acs{ctu} within a slice must belong to the same tile. These cases are illustrated in Figure~\ref{Figure2:HEVCTiles} (a) and (b), respectively. The use of tiles provide several advantages. Specifically, they improve frame partitioning for parallel processing~\cite{WPPTilesParallelHEVC2017} when compared to slices, by reducing the required overhead. Moreover, they also improve the \ac{mtu} size matching and reduce the line buffer memory (by dividing the frame as in Figure~\ref{Figure2:HEVCTiles}). Finally, as tiles allow flexible partitioning,they enable the definition of \acs{roi} for asymmetric video coding. \subsubsection{Wavefront Parallel Processing} As mentioned before, the use of either slices or tiles to process each frame using parallel threads require the use of different entropy coding contexts for each slice/tile, in order to make them independently decodable. In consequence the compression efficiency decreases. In order to overcome this issue, the concept of \ac{wpp} was introduced, which enables coding efficient parallel processing~\cite{HEVCParallelCSVT2016}. \begin{figure}[tb] \centering \input{\FigsPath/HEVC/WPP.tikz} \caption{Illustration of the wavefront parallel processing.} \label{Figure2:WPP} \end{figure} In \ac{wpp}, each slice is divided into single rows of \acs{ctu} which can be processed in parallel. Entropy decoding and prediction are enabled across \acs{ctu} of different rows. Therefore, higher compression efficiency can be achieved. In order to re-use the same context in the whole slice each row requires a two \ac{ctu} delay in order to make sure the required dependencies have already been coded/decoded. Figure~\ref{Figure2:WPP} illustrates an example of \ac{wpp} using at least five different threads. Each square corresponds to a \ac{ctu} and the numbers corresponds to the time instance in which a \ac{ctu} may start being processed. Using this staggered start (which appears like a wavefront as represented in the figure), the parallelisation may use as many threads as the number of \acs{ctu} rows available in the video frame. Although, the number of threads does not affect the coding efficiency, the required inter-process communication substantially increases. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Coding structures} \label{Section2:CodingStructures} \glsreset{ctu} Despite the fact that the \ac{hevc} standard follows a traditional hybrid video coding architecture, it introduces significant changes in the data structures when compared to the previous standards (e.g., \ac{h264}). In \ac{h264} each frame is divided into units so-called \acs{mb}, which are composed by a $16\times16$ block of luma samples and two $8\times8$ blocks of chroma samples, assuming video data in YUV 4:2:0 sub-sampling. The \ac{mb} is considered the basic processing unit in the \ac{h264}, which define if the containing pixels are compressed with intra or inter prediction techniques. Each \ac{mb} could also be partitioned into sub-blocks which should follow the same prediction type. \begin{figure}[tb] \centering \input{\FigsPath/HEVC/BlockPartitionsWithTree.tikz} \caption{Representation of a CTU ($64\times64$) and its partitioning into CUs and PUs (left); corresponding quad-tree with coding order (right)} \label{Figure2:HEVCCodingStructure} \end{figure} In \ac{hevc} each picture is divided into blocks so-called \acs{ctu}, which are composed by the luma and chroma \acs{ctb} and corresponding signalling~\cite{BlockPartHEVC2012}. The maximum \ac{ctu} size is configured by the encoder and signalled to the decoder, and should be one of the following: $64\times64$, $32\times32$ and $16\times16$. The use of coding units with larger size in \ac{hevc} was adopted to achieve higher coding efficiency especially in high resolution video. Each \ac{ctu} can be recursively partitioned into four smaller units, referred to as \acs{cu}, until it reaches the minimum unit size, which cannot be inferior to $8\times8$ pixels. Similar to the maximum \ac{ctu} size the minimum size is also defined at the encoding time. Figure~\ref{Figure2:HEVCCodingStructure} illustrates an example of a \ac{ctu} partitioning into \acs{cu}. The partitioning illustrated in the figure leads to the quad-tree structure represented on the right side with the corresponding coding order. \begin{figure}[tb] \centering \begin{tabular}{x{.8\columnwidth}x{.1\columnwidth}} \input{\FigsPath/HEVC/PUSymetric.tikz} & (a)\\ \input{\FigsPath/HEVC/PUAsymetric.tikz} & (b)\\ \end{tabular} \caption{Partitioning modes of an inter coded \ac{cu} into \acs{pu}~\cite{HEVCBlock2012}.} \label{Figure2:InterPuPartitions} \end{figure} The \ac{cu} is the basic processing unit in \ac{hevc} for prediction. It is further partitioned into two types of units, specifically \ac{pu} and \ac{tu}, which are used for prediction and transform operations, respectively. Each \ac{pu} is composed by \acs{pb}, and each \ac{tu} is composed by \acs{tb}. The decision whether to use intra-frame or inter-frame prediction modes is applied at the \ac{cu} level. However, depending on the partitioning of the \ac{cu} into \acs{pu} different intra or inter coding modes can be used for each \ac{pu}. When the intra prediction mode is signalled for a \ac{cu}, the size of the \ac{pb} need to be the same as the \ac{cb}, except when the smaller \ac{cb} is selected. In this case further splitting into four \ac{pb} of smaller size. For example, when the smaller \ac{cb} size is $8\times8$, the encoder may decide to divide it into four \acs{pb} of $4\times4$ pixels, each one having its own intra prediction mode. This is signalled using an one bit flag. The use of small \acs{pb} is useful for complex regions that required a fine-granularity prediction. When inter prediction mode is signalled, \ac{hevc} defines a higher range of partitioning schemes, allowing the \ac{cu} to be divided into one, two or four \acs{pu}. The splitting into four \acs{pu} is only allowed when the \ac{cu} size is equal to the minimum. Otherwise, the splitting should be carried out at the \ac{cu} level which leads to the same partition tree. Figure~\ref{Figure2:InterPuPartitions} illustrates the possible \ac{cu} partitioning into \acs{pu}. One should note that the partition applied to the \ac{cu} is also applied to divide the \ac{cb} into \acs{pb}. Figure~\ref{Figure2:InterPuPartitions}~(a) shows the symmetric partitions. The $N \times N$ corresponds to the case where the \ac{cu} is not further divided. Then, the standard defines the horizontal and vertical division into two \acs{pu}, corresponding to the $\frac{N}{2}\times N$ and $N \times \frac{N}{2}$ cases, respectively. Finally, the $\frac{N}{2} \times \frac{N}{2}$ case result in four \acs{pu}. The partitions on Figure~\ref{Figure2:InterPuPartitions}~(b) correspond to the asymmetric partitioning, which is one of the main improvements introduced in \ac{hevc}, as it allow for higher flexibility when performing motion estimation and compensation, leading to higher compression efficiency~\cite{HEVCBlock2012}. These modes are only allowed for \acs{cu} with $16\times16$ pixels or larger. %The main key feature of \ac{hevc} is the new block partition structure, which achieves more flexibility and efficiency. Four different block concepts have been introduced in the standard: \ac{ctu}, \ac{cu}, \ac{pu}, and \ac{tu}, all of which have different roles. Figure \ref{fig:HEVCBlockStructure} illustrates an example of a CTU partitioning in HEVC, and it is compared to the \ac{h264} \ac{mb} structure. CTUs are defined as the high-level partition within each slice and can be as large as 64$\times$64, which can be recursively partitioned into four smaller blocks referred to as CUs. The CU is the basic units for the intra/inter prediction and can be as small as 8$\times$8. Subsequently, the CU are divided into up to four PUs which are used for inter prediction. For intra coding PU is not used. Finally, the CU is divided into a transform unit tree, which may include several TUs, in a quad-tree structure. Transform units are the basic units for the transform operations applied to the coding residue. % %Each CU may be split into two or four Pus resulting in either symmetric or asymmetric partitions. Asymmetric partitioning of the CU into two PUs enables more efficient motion estimation since it allows more flexible partitioning of the picture~\cite{BlockPartHEVC2012}. This scheme is able to improve the coding performance by up to $1\%$ when compared with rectangular symmetric partitioning. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Prediction methods} \label{Section2:PredictionModes} Prediction methods are responsible for an important role in image and video coding standards, due to their ability to reduce the signal redundancy based on the previously encoded pixels. This section describes the prediction techniques used in the \ac{hevc} standard. These techniques include the directional intra prediction for efficiently reduce spatial redundancy and inter-frame motion compensation for remove the temporal redundancies. \subsection{Intra-frame} In previous standards, intra-frame prediction was used to efficiently reduce the spatial redundancy in video frames. Intra predicted blocks are obtained using the previously encoded pixels around the current \ac{pu}. \ac{hevc} adopts three types of intra-frame prediction: directional, Planar and DC~\cite{IntraCodingHEVC2012,HEVCIntraperformance2012}. Figure~\ref{Figure2:IntraModesHEVC} shows the available intra prediction mode available in \ac{hevc}. As shown, the predictions are applied to square blocks only. The example in the figure corresponds to \ac{pu} of $8\times8$ pixels, but the standard allows block sizes ranging from $4\times4$ to $32\times32$ The encoder tests all the available intra prediction modes and signals the best mode to the decoder. The decoder must generate an identical prediction block, based on the reconstructed pixels. The reconstructed reference samples used in the prediction process belong to the neighbouring blocks at left-down, left, top-left, top and top-right positions, as can be noticeable in Figure~\ref{Figure2:IntraModesHEVC}. Given a \ac{pu} with size $N \times N$, intra prediction requires a top neighbouring row of $2N$ pixels, a left column of $2N$ pixels plus the top-left neighbouring pixel. Due to unit boundaries, e.g., slice or tile boundaries, neighbouring pixels might not be all available to apply the intra prediction. In addition, when constrained intra prediction is enabled, reference samples belonging to inter-predicted \acs{pu} are not used in order to remove error propagation from potentially erroneously reconstructed reference frames. In order to overcome this issue and being able to apply all possible modes, \ac{hevc} uses reference sample substitution to replace the unavailable reference samples with the closer available ones, so that all intra prediction modes can be used. \begin{figure}[tb] \centering %\begin{tabular}{x{.5\columnwidth}x{.5\columnwidth}} %\input{\FigsPath/HEVC/IntraPredictions.tikz} & %\input{\FigsPath/HEVC/IntraPredictionsExample.tikz} \\ %\end{tabular} %\caption{List of the available \ac{hevc} angular intra prediction modes (left) and example of the angular mode 29 (right).} \includegraphics[width=0.9\columnwidth]{\FigsPath/HEVC/IntraPrediction} \caption{Example of \ac{hevc} intra prediction modes for a $8\times8$ block~\cite{BookHEVC2014}.} \label{Figure2:IntraModesHEVC} \end{figure} \subsubsection{Angular modes} Directional intra prediction was first introduced in the \ac{h264} standard~\cite{H264OverviewJul2003}, to estimate regions with a structured texture or directional edge. In its process, directional prediction projects the reconstructed samples in the block neighbourhood along a specific direction. The \ac{hevc} improves the number of available direction from 8 in \ac{h264} to 33 possible directions~\cite{IntraCodingHEVC2012}. The result of the projections in each possible direction is illustrated in Figure~\ref{Figure2:IntraModesHEVC}. The angular modes are designed to provide a dense coverage near the horizontal and vertical directions. It is not noticeable in the Figure~\ref{Figure2:IntraModesHEVC}, but the angular directions are more coarse as it gets closer to the diagonals. This reflects the observed statistical prevalence of the angles and the prediction efficiency. In order to improve the efficiency of angular prediction the when the reference samples need to be projected a bilinear interpolation is used from the two closest pixels using a $1/32$ accuracy. \subsubsection{Planar and DC} In contrast, with the angular mode which cover those regions with strong direction edges, \ac{hevc} also implements the Planar and DC modes targeting smooth regions. These modes are useful for smooth regions as they result in a linear or constant prediction. The DC mode uses the average or the neighbouring reference samples to generate a constant prediction for the current \ac{pu}. The Planar mode predicts the \ac{pu} through linear predictions from vertical and horizontal slopes derived from the neighbouring reference samples. As the Planar mode used linear surface for prediction it prevents the discontinuities at \ac{pu} boundaries. Both of these modes are also illustrated in Figure~\ref{Figure2:IntraModesHEVC}. \subsubsection{Sample smoothing} In \ac{hevc} sample smoothing is applied at two distinct occasions two improve the overall performance of the prediction accuracy. Firstly, the reference samples are filtered and secondly the \ac{pu} is filtered after prediction. This improves the previous standard \ac{h264} where only reference samples were filtered. In \ac{hevc} the reference sample smoothing is conditionally applied, based on the \ac{pu} and intra prediction mode. Similar to \ac{h264}, a three tap $\left[1\ 2\ 1\right]/4$ smoothing filter is applied, except for \ac{pu} sizes of $4\times4$. The prediction results from Planar mode is also filtered when the \ac{pu} size is greater than $8\times8$, and it is not used for the DC mode as it would not be useful. The second filter in \ac{hevc} is the boundary smoothing. This filter aims at removing the discontinuities along block boundaries due to intra prediction. The application of this filter is more exclusive, and it is only applied to the DC and angular mode 10 and 26 (exactly the horizontal and vertical directions, respectively). A two-tap filter $\left[3\ 1\right]/4$ fed by the original samples and the adjacent reference sample. For the DC mode this filter is applied to both the top row and the left-most column. In case of the angular mode 10 and 26 only the left-most column and the top row are filtered, respectively. Only the mentioned boundaries are filtered, as they are the most likely to introduce discontinuities. The result of the boundary smoothing can be noticed in Figure~\ref{Figure2:IntraModesHEVC} for the aforementioned modes. %\subsubsection{Rough Mode Decision} %\cite{HEVCROughModeDecision} \subsection{Inter-frame} The efficiency of video coding algorithms consistently relies on inter-frame prediction techniques used to reduce temporal redundancy. The underlying idea of inter-frame prediction is to estimate the current frame from one or more previously encoded reference frames using block-based motion compensated. The most common technique used for motion estimation in \ac{hevc}, as well as in \ac{h264}, is the boundary matching algorithm. In the \ac{hevc} standard an \ac{amvp} for efficient motion vector signalling, which defines the Merge Mode and differential coding of the motion information. \subsubsection{Motion estimation and compensation} Motion estimation consists in searching for the current block with the highest similarity with the target block to be predicted (\ac{pu} in \ac{hevc}), using block-matching algorithms over the previously encoded frames, i.e. the reference frames. Typically, reference frames correspond to past or future temporal frames. The frame encoding order determines which reference frames are selected and whether future frames are available. Typically, video frame type P normally are predicted from the past as B frame are predicted from frames both in past and future. Block-matching consist on subtracting the current block from one of equal size in the reference frame in a pre-defined window. The reference block that results in the lowest difference is selected for prediction. In order to signal the selected block to be used for motion compensation the different between the target and the reference block positions are computed to obtain a two-dimensional vector, referred to as \ac{mv}. This vector is signalled to the decoder, in order to apply the same prediction. Similar to \ac{h264} the \ac{hevc} allows for either one or two \ac{mv} to be selected to compensate the target \ac{pu}. This results in uni-predictive and bi-predictive coding, respectively. \ac{hevc} also allows for weighted prediction, which consists in scaling and offset operations the predicted block using determined factors which are explicitly sent to the decoder. The motion vectors are obtained from frames included in the reference frame list. Each frame is identified by an index, therefore to fully describe inter-frame prediction a combination of two parameters is required: \ac{mv} and reference index. In recent standards motion compensation has been performed using a variable block size approach. In \ac{h264} block sizes ranging from $4\times4$ up to $16\times$ pixels were allowed. In \ac{hevc} a significant improvement was introduced by extending the block sizes up to $64\times64$ pixels. As mentioned before it also allows asymmetric rectangular block sizes. By using larger prediction units, \ac{hevc} is able to save bits in the signalling of the selected \ac{mv}, because less \ac{mv} are required. In order to increase the efficiency of motion estimation using fractional \acs{mv} are used. This has been proven to improve accuracy of motion compensation. To accomplish this, frame interpolation is performed to generated non-integer pixels in the reference frame. Both \ac{h264} and \ac{hevc} standards use \acs{mv} up to quarter-pixel accuracy for luma component. For chroma samples, the \ac{mv} accuracy is determined according to the chroma sampling format, which for 4:2:0 results in eighth-pixel accuracy. In contrast, with the two-stage approach used in \ac{h264}, \ac{hevc} uses a single separable approach for the interpolation process without intermediate rounding operations. This improves the precision and simplifies the architecture of the fractional interpolation. The block matching-based motion estimation is one of the most computationally complex procedures in the \ac{mv} encoder. Normally, the optimal solution for motion estimation is provided by the full-search algorithm, which tests all the possible candidate blocks in the search area. Although the full-search algorithm achieves the optimal performance, it requires the highest computational complexity. Therefore, sub-optimal algorithms have been proposed over the years to reduce the motion estimation complexity and decrease encoding time\cite{HEVCME2012,HEVCME2017,HEVCME2014}. For example, the HM reference software implements a fast algorithm based on Test Zone Search ~\cite{HEVCME2012}. Since motion estimation is a non-normative procedure, encoders may implement any alternative search algorithms that suits its needs (i.e., compression ratios or encoding time). \subsubsection{Reference picture management} Another improvement in \ac{hevc} is related with the reference picture management scheme. \ac{hevc} introduces the \ac{rpset} concept~\cite{HEVCHighLevelReferenceDez2012}, which defines how previously decoded pictures are managed in the \ac{dpb}. Decoded frame in the \ac{dpb} are mark grouped in one of the following categories: (i) \emph{short-term reference}, (ii) \emph{long-term reference} and (iii) \emph{unused for reference}. Once a frame is marked as \emph{unused for reference} it is no longer used for motion compensation and it will be discarded from the \ac{dpb} after being displayed/outputted. In \ac{h264} a sliding windows approach was used to implicitly manage the reference picture list. This process discards a given frame whenever the \ac{dpb} has the number of defined reference frames (which is signalled by the encoder). Moreover, \ac{h264} used an explicitly \ac{mmco} to send commands to the decoder and change the normal reference picture management process. \begin{table}[tb] \centering \caption{Reference Picture Set example.}\label{Table2:ReferencePictureListExample} \small \TABLESVSPACE \begin{tabular}{x{1.5cm}|ccc} \toprule Frame & \multicolumn{3}{c}{\ac{rpset} (\{frame, is used\}) } \\ \midrule $I_1$ & -- & & \\ $P_5$ & \{$I_1$, 1\} & & \\ $B_6$ & \{$I_1$, 1\} & \{$P_5$, 1\} & \\ $B_7$ & \{$I_1$, 1\} & \{$B_6$, 1\} & \{$P_5$, 0\}\\ $B_8$ & \{$B_6$, 1\} & \{$P_5$, 1\} &\\ \bottomrule \end{tabular} \end{table} In the \ac{hevc} standard instead of a signalling relative changes, the status of the \ac{dpb} is signalled for every slice. The most fundamental difference with the \ac{rpset} concept compared to \ac{mmco}/sliding window of \ac{h264} is that for each particular slice a complete set of the reference frames that are used by the current or any subsequent frames are be provided. Thus, a complete set of all frames that must be kept in the \ac{dpb} (in the present and future) are signalled. This is different from the \ac{h264} scheme where only relative changes to the \ac{dpb} are signalled. With the \ac{rpset} concept, no information from earlier frames in decoding order is needed to maintain the correct status of the \ac{dpb}. Table~\ref{Table2:ReferencePictureListExample} shows the \ac{rpset} for some frames illustrated in Figure~\ref{Figure2:LeadingTralingPics}. As expected for the \ac{cra} frame ($I_2$) nothing is signalled. For the remaining frames, two types of signalled are used: frame number and a flag to mark if it is currently used. This is required because any frame not listed in a given \ac{rpset} will be discarded and subsequent frames cannot use it as reference. The \ac{rpset} concept in \ac{hevc} provides a basic level of error robustness to the reference picture management. In \ac{h264} whenever a reference frame is lost the decoder can look at the value of the \emph{FrameNum} (incremental value associated with each reference frame) to detect the loss event. However, as gaps in \emph{FrameNum} are allowed the decoded might not be able to decide the missing frame was intentional or if it resulted from a loss event in the transmission. Moreover, even if the decoded if able to detect the loss event it might not be able to correctly compute the value the \ac{poc} associated with the concealed frame. Using the \ac{rps} concept in the \ac{hevc} the decoder is always able to detect loss events and correctly identify the missing frames, preventing it from use wrong frames as reference. \subsubsection{Merge mode} \begin{figure}[tb] \centering \input{\FigsPath/HEVC/MergeModeHEVC.tikz} %\input{\FigsPath/HEVC/MergeModeHEVCTwoFrames.tikz} \caption{Illustration of the \ac{mv} prediction candidates in the AMVP.} \label{Figure2:MVCandidates} \end{figure} The Merge mode was introduced in \ac{hevc} standard, as a new technique to derive motion information from spatial and temporal neighbouring blocks~\cite{MotionCodingHEVC2013}. This technique extends the concepts of the direct and skip modes used in \ac{h264}~\cite{H264OverviewJul2003}, using a more sophisticated approach with two key differences. Firstly, the Merge Mode allows for more \ac{mv} candidates. Thus, an index value is included to select one out of several \ac{mv} candidates. Secondly, it explicitly identifies the reference picture index used for the selected candidate, as well as, the reference picture list. This results in an improved flexibility, compared with the predefined values used in \ac{h264}. The \ac{mv} candidates allowed in the Merge Mode\cite{MotionCodingHEVC2013} are illustrated in Figure \ref{Figure2:MVCandidates}. The figure illustrates the spatial candidates' positions ($A_0$, $A_1$, $B_0$, $B_1$, $B_2$). Moreover, it also uses a temporal motion vector candidate, derived from the co-located position on the temporal adjacent frame at positions $T_0$ and $T_1$. From this \ac{mv} candidate list the unavailable candidates are not accounted and duplicated candidates (i.e. with the same motion information) are removed. In the Merge Mode, a $C-1$ number of candidates are used, selected from the aforementioned order and one temporal \ac{mv} candidate is selected to fulfil the total number of \ac{mv} ($C$). The value of $C$ is defined at the slice header and a flag controlling the use of temporal \ac{mv} candidates is sent at the \ac{pps}-level. For B-slices, additional \ac{mv} candidates are generated by choosing two existing candidates according to a predefined order for reference picture list 0 and list 1. A total number of twelve combination are specified in \ac{hevc} which are listed in ~\cite{HEVCOverviewDez2012}. For P-slices if the number of candidates is less than $C$ zero motion vectors are included to fill the remaining entries. The Merge Mode allows for the signalling of a \ac{mv} candidate and the residual information. In \ac{hevc}, the Skip mode, i.e., transmitting only the \ac{mv} without residual information, is treated as a special case of the Merge Mode. In this case only a skip flag is enabled, and the corresponding merge index is transmitted. \subsubsection{Motion vector prediction} The \ac{hevc} also improves the performance of the motion vector coding for those cased where the Merge Mode is not used. \ac{hevc} always use a difference encoding approach for efficiently represents the motion information. It considers the same \ac{mv} candidate of the Merge mode, illustrated in Figure~\ref{Figure2:MVCandidates}, to derive the motion vector predictors from which the difference vector is obtained. From these \ac{mv} candidates only two are selected based on the following: \begin{enumerate} \item Left candidate: first available from $A_0$ and $A_1$. \item Top candidate: first available from $B_0$, $B_1$ and $B_2$. \item Temporal candidate: first available from $T_0$ and $T_1$. \end{enumerate} Therefore, the use of the temporal \ac{mv} candidate is limited to those cases where the spatial neighbours are not able to provide two valid candidates. \ac{hevc} only allows a much lower number of candidates to be used in the motion vector prediction process, since it can still send a motion vector difference. Moreover, the encoder will perform motion estimation for each candidate, which increase the encoding complexity for every possibility tested. For those cases where the reference index is not the same in the current \ac{pu} and the neighbouring one, a scaling version of the \ac{mv} is used. %In the \ac{h264} standard, along different motion compensated coding techniques~\cite{H264OverviewJul2003}, the so-called P\_SKIP mode was introduced. In this mode, neither the quantised residual information, nor the \ac{mv} is transmitted. This mode is useful for large areas with no change or just constant motion, and it is only enabled for $16\times16$ blocks, \emph{i.e.}, the whole \ac{mb}. The \ac{mv} for the P\_SKIP mode is derived based on the surrounding neighbour MBs\cite{H264MotionVectorPrediction2003}. When no neighbours are available, or the left or top \acs{mv} are equal to zero, a zero \ac{mv} is used. Otherwise, the \ac{mv} is derived from the median value of the following three neighbours: left, above and above right, as shown in Figure \ref{SkipMergeMode}~(a) by $MV_A$, $MV_B$ and $MV_C$, respectively. Whenever the $MV_C$ is not available, the $MV_D$ is used instead. % %In contrast with the \ac{h264} standard, the \ac{hevc} standard allows the usage of a temporal \ac{mv} predictor, as well as, more spatial candidates to predict the current motion vector. Despite the improvements in the flexibility and coding efficiency, the increased levels of dependencies between the \acs{mv} make the motion information more prone to errors. Especially the introduction of the temporal candidate in the set of possible \acs{mv} increases the dependencies between subsequent frames. This leads to higher quality degradation in the presence of transmission errors, when compared to \ac{h264}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Transform and quantisation} \label{Section2:TransformQuant} \glsreset{tb} Similarly, to the previous standards, \ac{hevc} uses a transform followed by a quantisation to compress the residual signal. The residual block is partitioned into multiple square blocks, referred to as \ac{tb}. \subsubsection{Core transform} A 2D transform of the residual information is obtained by applying 1D transforms in both the horizontal and vertical directions. The core transform matrices are obtained from the \ac{dct} basis functions to define a integer matrix of $32\times32$ points. Sub-sampled versions of this matrix are used to derive the remaining transform matrix sizes down to $4\times4$. A special case is applied for the residual coding of a $4\times4$ intra coded block. In this case an alternative integer transform based on the \ac{dst}. The motivation for \ac{dst} is related to the statistics of the residual signal of these blocks, which tend to present higher residue as the distance to the boundary used for prediction (i.e, top or left boundary) increases. Experiments have shown that DST provides up to 1\% of bitrate savings for the compression of $4\times4$ intra predicted blocks, without significantly increase the computational complexity \cite{DCTDSTComp2013}. \subsubsection{Quantisation} \ac{hevc} standard follows the same principle introduced in \ac{h264} for the quantisation method. After obtaining the residual transform a quantisation is applied using a uniform-reconstruction quantiser scheme based on the \ac{qp}. The standard supports \ac{qp} values ranging from 0 to 51, and a increase of 6 in the \ac{qp} corresponds to doubling the quantisation step. This procedure is the main responsible to introduce the coding errors as it performs many-to-one mapping, which cannot be reversed. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Entropy coding} \label{Section2:Entropy} In contrast to \ac{h264} standard, which allows two entropy coding methods, \ac{hevc} only uses the \ac{cabac} \cite{HEVCCabac2003}. \Gls{cabac} is an arithmetic coding method that only uses binary symbols, considering different probability models for each symbol. Entropy coding efficiency is closely linked with the context model selected. Therefore, this was carefully designed in \ac{hevc} standard, extending the functionality presented in \ac{h264} For example, the \ac{hevc} standard exploits the depth of the partition tree or residual transform tree in order to derive the context models for several syntax elements. Regarding transform coefficient coding, \ac{cabac} uses a scanning method to firstly organise the coefficients and then transmits the position of the last non-zero transform coefficient. Moreover, it transmits, a significance map, the sign bits and the levels for the transform coefficients. Three coefficient scanning methods are available: the diagonal up-right, the horizontal and the vertical scan. The coefficient scanning is implicitly selected and always performed in $4\times4$ sub-blocks for all transform block sizes. % Details about the context models selection and the compression of each syntax element of \ac{hevc} can be found in [1]. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{In-loop filters} \label{Section2:Filters} The \ac{hevc} standard uses two filtering procedures over the reconstructed pixels before writing them into the frame buffer, namely the Deblocking Filter and the \ac{sao} filter. The main purpose of deblocking filter is to reduce blocking artefacts caused by the block-based coding tools, operating similarly to the deblocking filter of the \ac{h264} standard. DBF is applied to the samples adjacent to the \ac{pu} or \ac{tu} boundaries, that are aligned on a $8\times8$ grid. Such grid-based restriction reduces the computational complexity and improves parallel-processing implementation which is one of the main goals of \ac{hevc}. Three strength levels can be used by filter depending on the coded block characteristics. The \ac{sao} is a non-linear filter, that is adaptively applied to all samples of the image, after the deblocking filter. In its operation, the \ac{sao} filter modifies the samples by adding an offset value, extracted from lookup tables transmitted by the encoder. For each \ac{ctu}, the encoder decides whether to apply the \ac{sao} filter or not, and if used, one of the two available filter types is applied, namely the band offset or the edge offset. In the band offset mode, the added offset value depends on the sample amplitude, while the edge offset uses the gradient to classify and derive the offset value. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Summary} \label{Section2:Summary} In this chapter we described the main features of the \glsfirst{hevc} standard, giving a special focus on the high-level features, coding structures and newly introduced prediction modes. At the high-level \ac{hevc} introduces new \ac{nal} units types, not only for new signalling features but also for the new pictures types introduced. \ac{hevc} introduces new picture types to handle different types of random-access strategies, which allow more flexibility and higher coding efficiency, when compared to the random-access in previous standard such \ac{h264}. Another major improvement comes from the inter-frame prediction. On the one hand new high-level reference picture management was introduced, which increase the error robustness, as it explicitly defines the active reference frame and the \ac{dpb} status. On the other hand it introduces the \ac{amvp} and the Merge mode which efficiently reduce the required bits or motion signalling. In general, \ac{hevc} achieves higher coding performance than its predecessors. Previous studies reported an average bitrate saving of approximately 50\% for the same subjective quality. Such amount of bitrate saving is normally related with increased prediction structures with higher dependencies. Such dependencies are more sensible to data loss, which will increase the error propagation to subsequent coding structures. Therefore, it is emergent to evaluate the impact of the transmission errors in the \ac{hevc} and develop efficient techniques to cope with data loss and reduce the error propagation.