jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Review all chapters and start conclusions

Browse Source
This commit is contained in:
jfmcarreira
2018-09-12 02:44:20 +01:00
parent 461b560b17
commit 8d3a4c72ff
20 changed files with 304 additions and 427 deletions
Download Patch File Download Diff File Expand all files Collapse all files
021_VideoCoding/VideoCodingReview.tex
+15 -15
View File
@@ -27,7 +27,7 @@ This chapter is organised as follows.
Section~\ref{Section2:GeneraVideoSystem} introduces the functional blocks of a generic hybrid video coding system and Section~\ref{Section2:Overview} provides an overview of the \ac{hevc} standard.
Then, Section~\ref{Section2:HighLevel} describes the high-level syntax elements, which are used to carry relevant information in coded streams.
Section~\ref{Section2:CodingStructures} describes the data structures used to divide the video frames into smaller partitions and Section~\ref{Section2:PredictionModes} describes particular aspects of the main predictions techniques, namely intra and inter coding.
Finally, a brief description of the transform and quantisation, entropy coding and in-loop filters is provided in Sections~\ref{Section2:TransformQuant}, \ref{Section2:Entropy} and \ref{Section2:Filters}, respectively.
Finally, a brief description of the transform and quantisation, entropy coding and in-loop filters is provided in Sections~\ref{Section2:TransformQuant},~\ref{Section2:Entropy} and~\ref{Section2:Filters}, respectively.
Section~\ref{Section2:Summary} concludes the chapter.
\clearpage
@@ -78,7 +78,7 @@ As shown in Figure~\ref{Figure2:HybridCodec}, these differences (also known as r
Such a transformation decorrelates the residue signal and concentrates the signal energy in few coefficients, mostly at low frequencies.
Subsequently, the transformed coefficients are quantised (Q) to reduce the representation accuracy of the signal, and consequently to reduce the number of bits required for coding.
Finally, the quantised transform coefficients together with the prediction information are further compressed by the entropy encoder, forming an output stream suitable for transmission or storage, as shown in Figure~\ref{Figure2:HybridCodec}.
On the other side of the communication chain, the decoding stage performs the inverse operations on the stream to reconstruct the video signal.
On the other side of the communication chain, the decoding stage performs the inverse operations on the stream to reconstruct the video signal.
In the next section, the aforementioned blocks are described in the context of the \ac{hevc} standard.
@@ -100,7 +100,7 @@ Similar to the previous coding standards developed by JCT-VC, in \ac{hevc}~\cite
coding syntax and semantics are standardised.
The constrains and functionalities are defined through profiles and levels, while the semantic meaning of syntax elements define the decoding process, such that every decoder conforming to the standard generates the same output when given a complaint stream.
Such delimitation of scope in the standard allows maximum freedom for developers and manufactures to develop proprietary optimisations to cope with various applications’ requirements (\emph{e.g.}, balancing the compression ratio and the coding complexity).
Figure \ref{fig:HEVCBlockDiagram} illustrates the block diagram of the \ac{hevc} encoder. It follows the same overall hybrid architecture as its predecessors starting from H.261.
Figure~\ref{fig:HEVCBlockDiagram} illustrates the block diagram of the \ac{hevc} encoder. It follows the same overall hybrid architecture as its predecessors starting from H.261.
The encoder uses both intra- and inter-picture prediction to exploit spatial and temporal redundancies, respectively. Subsequently, the prediction residue transformed by linear spatial transform, which is then quantised, and entropy coded, allowing to further reduce the bitstream redundancy.
%The resulting bitstream is combined with control information, which signals the prediction chosen, motion vectors, filtering operations and bitstream headers.
@@ -112,7 +112,7 @@ The encoder uses both intra- and inter-picture prediction to exploit spatial and
\label{fig:HEVCBlockDiagram}
\end{figure}
However, there are some novelties introduced in the \ac{hevc} standard. As depicted in Figure \ref{fig:HEVCBlockDiagram}, the encoding process follows a block-based approach, therefore each picture is partitioned into larger blocks, referred to as \acs{ctu}, with a maximum size of $64\times64$ pixels.
However, there are some novelties introduced in the \ac{hevc} standard. As depicted in Figure~\ref{fig:HEVCBlockDiagram}, the encoding process follows a block-based approach, therefore each picture is partitioned into larger blocks, referred to as \acs{ctu}, with a maximum size of $64\times64$ pixels.
Subsequently, a dynamic quad-tree partitioning is used to divide the picture into \acs{cu}. These units are introduced as the basic processing units for prediction, and for the transform and quantisation operations, \acs{tu} are used.
In order to generate the prediction residue the \ac{hevc} may use one of 34 intra-prediction modes or take advantage of symmetric and asymmetric partitions for motion estimation and inter-prediction.
The motion estimation is performed in the pictures listed in the \ac{rpset}, which was introduced in the \ac{hevc} to increase the robustness to errors and data loss of reference picture management.
@@ -140,7 +140,7 @@ Moreover, new pictures types were introduced to increase the random-access flexi
Similar to \ac{h264}~\cite{H264AVCStandard2003}, the \ac{hevc} streams are based on \glsfirst{nal} for stream partitioning and synchronization, which is useful to deal with lossy network environments~\cite{H264OverIPJul2003}. Using this concept the stream is composed of different \ac{nal} units. The \ac{nal} concept provides the ability to map the \ac{vcl} data that represents coded video slices (or frames) on various transport layers, such as \ac{rtp}, MPEG-2 Systems and MPEG DASH. The~\ac{nal} units are classified into two categories: (i) \ac{vcl} and (ii) non-\ac{vcl}, as they carry coded video or associated data, respectively.
The \ac{hevc} standard defines two bytes to describe the content of each \ac{nal} unit.
The \ac{hevc} standard defines two bytes to describe the content of each \ac{nal} unit.
These bytes are used to identify the \ac{nal} type are transmitted using a fixed coding format length, i.e., without variable length codes. This allow extra error robustness but increases the bit overhead. Figure~\ref{Figure2:NalHeader} shows the structure of the \ac{hevc} \ac{nal} unit header. The first bit \textit{F}, referred to as \textit{forbiden\_bit}, is always zero in order to ensure that a start code can be detected across different protocol standards. This is followed by the \ac{nal} type, which identifies its content, enabling the decoder to either use it if required for a particular decoding purpose or discard it. Another syntax element introduced by the \ac{hevc} standard is the \textit{temporal\_id\_plus\_one} (TIDP), which defines the temporal layer where the \ac{nal} unit belongs to. This allows implicit temporal scalability (layers ranging from 0 to 6) by immediately discarding the \ac{nal} unit if it belongs to a layer higher than the desired one. In the header, six bits are reserved for future extensions, preventing the creation of extra \ac{nal} units for scalable and multiview extensions, as done in \ac{h264}.
\begin{figure}[tb]
@@ -307,8 +307,8 @@ The high-level segmentation of a picture in \ac{hevc} is achieved through four t
\subsubsection{Slices}
The \ac{hevc} standard preserves the slice structure defined in the \ac{h264} standard. A slice may comprise either the entire frame or a section of it, and all the associated data (\emph{i.e.}, entropy symbols, prediction and residue information) can be independently decoded. Each slice is transported in a different \ac{nal} unit.
As some dependencies across slice boundaries are disabled, each slice can be independently reconstructed, regardless whether previous slices were lost or incorrectly decoded. Slice partitioning is the only tool that can be used for parallel processing in a virtually identical form as in \ac{h264}.
The \ac{hevc} standard preserves the slice structure defined in the \ac{h264} standard. A slice may comprise either the entire frame or a section of it, and all the associated data (\emph{i.e.}, entropy symbols, prediction and residue information) can be independently decoded. Each slice is transported in a different \ac{nal} unit.
As some dependencies across slice boundaries are disabled, each slice can be independently reconstructed, regardless whether previous slices were lost or incorrectly decoded. Slice partitioning is the only tool that can be used for parallel processing in a virtually identical form as in \ac{h264}.
As each slice is independent from each other, it is straightforward to process multiple slices in parallel, without inter-process communication (except for inter-frame prediction). Although this is a simple approach, it incurs in substantial coding overhead due to the higher number of slice headers, and reduction of causal neighbours for prediction (due to lack of predictions across slice boundaries). Another disadvantage of using slices for parallel processing is that they are also used to meet the \gls{mtu} size constrains. Therefore, it might be impossible to meet both requirements using only one technique.
Another partitioning tool in \ac{hevc} standard is the dependent slices. These slices allow the partitioning of a frame at the coding unit boundaries without breaking intra-frame predictions. Moreover, as they have a smaller slice header they are more efficient in terms of signalling overhead than regular slices. Dependent slices are normally used to reduce the end-to-end delay by allowing part of the slice to transmitted while the rest of its data is still being processed.
@@ -340,7 +340,7 @@ The use of tiles provide several advantages. Specifically, they improve frame pa
As mentioned before, the use of either slices or tiles to process each frame using parallel threads require the use of different entropy coding contexts for each slice/tile, in order to make them independently decodable. As consequence, the compression efficiency decreases. In order to overcome this issue, the concept of \ac{wpp} was introduced, which enables coding using efficient parallel processing~\cite{HEVCParallelCSVT2016}.
In \ac{wpp}, each slice (or tile) is divided into single rows of \acs{ctu} which can be processed in parallel, but entropy coding and prediction are enabled across \acs{ctu} of different rows, in order to avoid loss in coding efficiency.
To re-use the same context in the whole slice (or tile) each row requires a delay of two \acs{ctu} in order to ensure that all necessary dependencies have already been encoded/decoded. Figure~\ref{Figure2:WPP} illustrates an example of \ac{wpp} using at least five different threads.
To re-use the same context in the whole slice (or tile) each row requires a delay of two \acs{ctu} in order to ensure that all necessary dependencies have already been encoded/decoded. Figure~\ref{Figure2:WPP} illustrates an example of \ac{wpp} using at least five different threads.
Each square corresponds to a \ac{ctu} and the numbers correspond to the time instance when a \ac{ctu} starts being processed. Using this staggered start (which appears like a wavefront, as represented in Figure~\ref{Figure2:WPP}), the parallelisation may use as many threads as the number of \acs{ctu} rows available in the video frame. However, although the number of threads does not affect the coding efficiency, the required inter-process communication substantially increases.
\begin{figure}[tb]
@@ -357,14 +357,14 @@ Each square corresponds to a \ac{ctu} and the numbers correspond to the time ins
\label{Section2:CodingStructures}
Despite the fact that the \ac{hevc} standard follows a traditional hybrid video coding architecture, it introduces significant changes in the data structures when compared to the previous standards (e.g., \ac{h264}).
In \ac{h264} each frame is divided into units so-called \acs{mb}, which are composed by a $16\times16$ block of luma samples and two $8\times8$ blocks of chroma samples, assuming video data in YUV 4:2:0 chroma sub-sampling. The \ac{mb} is considered the basic processing unit in the \ac{h264}, and the respective coding mode define whether the corresponding pixels are coded as intra or intra prediction.
In \ac{h264} each frame is divided into units so-called \acs{mb}, which are composed by a $16\times16$ block of luma samples and two $8\times8$ blocks of chroma samples, assuming video data in YUV 4:2:0 chroma sub-sampling. The \ac{mb} is considered the basic processing unit in the \ac{h264}, and the respective coding mode define whether the corresponding pixels are coded as intra or intra prediction.
Each \ac{mb} could also be partitioned into sub-blocks which should follow the same prediction mode.
\glsreset{ctu}
\glsreset{cu}
The basic data structures in \ac{hevc} are quite different from \ac{h264}. Each picture is divided into blocks named as \ac{ctu}~\cite{BlockPartHEVC2012}. The maximum \ac{ctu} size is an encoder configuration parameter that is signalled to the decoder, and should be one of the following: $64\times64$, $32\times32$ and $16\times16$. The use of data structures with larger size in \ac{hevc} was adopted to achieve higher coding efficiency especially in high resolution video. Each \ac{ctu} can be recursively partitioned into four smaller units, referred to as \acs{cu}, until it reaches the minimum unit size, which cannot be smaller than $8\times8$ pixels.
The basic data structures in \ac{hevc} are quite different from \ac{h264}. Each picture is divided into blocks named as \ac{ctu}~\cite{BlockPartHEVC2012}. The maximum \ac{ctu} size is an encoder configuration parameter that is signalled to the decoder, and should be one of the following: $64\times64$, $32\times32$ and $16\times16$. The use of data structures with larger size in \ac{hevc} was adopted to achieve higher coding efficiency especially in high resolution video. Each \ac{ctu} can be recursively partitioned into four smaller units, referred to as \acs{cu}, until it reaches the minimum unit size, which cannot be smaller than $8\times8$ pixels.
Similar to the maximum \ac{ctu} size, the minimum size is also defined at the encoder.
For instance, in case of homogeneous regions, large \acs{cu} can be used to represent those regions
by using a smaller number of symbols than is the case of using several small units.
@@ -386,7 +386,7 @@ The decision whether to use intra-frame or inter-frame prediction modes is taken
%The main key feature of \ac{hevc} is the new block partition structure, which achieves more flexibility and efficiency. Four different block concepts have been introduced in the standard: \ac{ctu}, \ac{cu}, \ac{pu}, and \ac{tu}, all of which have different roles. Figure \ref{fig:HEVCBlockStructure} illustrates an example of a CTU partitioning in HEVC, and it is compared to the \ac{h264} \ac{mb} structure. CTUs are defined as the high-level partition within each slice and can be as large as 64$\times$64, which can be recursively partitioned into four smaller blocks referred to as CUs. The CU is the basic units for the intra/inter prediction and can be as small as 8$\times$8. Subsequently, the CU are divided into up to four PUs which are used for inter prediction. For intra coding PU is not used. Finally, the CU is divided into a transform unit tree, which may include several TUs, in a quad-tree structure. Transform units are the basic units for the transform operations applied to the coding residue.
%The main key feature of \ac{hevc} is the new block partition structure, which achieves more flexibility and efficiency. Four different block concepts have been introduced in the standard: \ac{ctu}, \ac{cu}, \ac{pu}, and \ac{tu}, all of which have different roles. Figure~\ref{fig:HEVCBlockStructure} illustrates an example of a CTU partitioning in HEVC, and it is compared to the \ac{h264} \ac{mb} structure. CTUs are defined as the high-level partition within each slice and can be as large as 64$\times$64, which can be recursively partitioned into four smaller blocks referred to as CUs. The CU is the basic units for the intra/inter prediction and can be as small as 8$\times$8. Subsequently, the CU are divided into up to four PUs which are used for inter prediction. For intra coding PU is not used. Finally, the CU is divided into a transform unit tree, which may include several TUs, in a quad-tree structure. Transform units are the basic units for the transform operations applied to the coding residue.
%
%Each CU may be split into two or four Pus resulting in either symmetric or asymmetric partitions. Asymmetric partitioning of the CU into two PUs enables more efficient motion estimation since it allows more flexible partitioning of the picture~\cite{BlockPartHEVC2012}. This scheme is able to improve the coding performance by up to $1\%$ when compared with rectangular symmetric partitioning.
@@ -406,7 +406,7 @@ This section describes the prediction techniques used in the \ac{hevc} standard
Intra-frame prediction is used to efficiently reduce the spatial redundancy within video frames.
Intra predicted blocks are obtained using the previously encoded pixels around the current \ac{pu}.
When the intra prediction mode is chosen for a \ac{cu}, the size of the \ac{pu} need to be the same as the \ac{cu}, except when the smaller \ac{cu} is selected and further divided into four \acs{pu} is done.
When the intra prediction mode is chosen for a \ac{cu}, the size of the \ac{pu} need to be the same as the \ac{cu}, except when the smaller \ac{cu} is selected and further divided into four \acs{pu} is done.
For example, when the smaller \ac{cu} size is $8\times8$, the encoder may decide to divide it into four \acs{pu} of $4\times4$ pixels, each one having its own intra prediction mode. This is signalled using a binary flag.
The use of small \acs{pu} is useful for regions with many details that require fine-granularity prediction.
@@ -590,14 +590,14 @@ Firstly, the Merge Mode allows for more \ac{mv} candidates than the Skip in \ac{
Thus, an index is used to identify one out of several \ac{mv} candidates.
Secondly, the reference picture index is explicitly identified.
This results in improved flexibility, compared with the predefined values used in \ac{h264}.
The \ac{mv} candidates allowed in the Merge Mode\cite{MotionCodingHEVC2013} are illustrated in Figure \ref{Figure2:MVCandidates}.
The \ac{mv} candidates allowed in the Merge Mode\cite{MotionCodingHEVC2013} are illustrated in Figure~\ref{Figure2:MVCandidates}.
The figure shows the positions of the spatial \ac{mv} candidates ($A_0$, $A_1$, $B_0$, $B_1$, $B_2$).
Moreover, it also uses a temporal \ac{mv} candidate, derived from the co-located position on the temporally adjacent frame, at positions $T_0$ or $T_1$. From the \ac{mv} candidate list the unavailable candidates are not considered, and duplicated candidates (i.e. with the same motion information) are removed.
The Merge Mode uses a $C-1$ spatial \ac{mv} candidates, selected from the aforementioned order, and one \ac{tmvp} to fulfil the total number of \acs{mv} ($C$). The value of $C$ is defined at the slice header and a flag controlling the use of \ac{tmvp} is encoded at the \ac{pps}-level.
For B-slices, additional \ac{mv} candidates are considered by choosing two existing candidates, according to a predefined order for reference picture list 0 and list 1. A total number of twelve combinations are specified in \ac{hevc}, which are listed in~\cite{HEVCOverviewDez2012}. For P-slices, if the number of candidates is less than $C$, then zero \acs{mv} are included to fill the remaining entries.
The Merge Mode defines how to encode the \ac{mv} candidate and the residual information.
In \ac{hevc}, the Skip mode, i.e., transmitting only the \ac{mv} without residual information, is treated as a special case of the Merge Mode.
In \ac{hevc}, the Skip mode, i.e., transmitting only the \ac{mv} without residual information, is treated as a special case of the Merge Mode.
In this case, only a skip flag is enabled and the corresponding merge index is transmitted.
\begin{figure}[tb]
@@ -624,7 +624,7 @@ Therefore, the use of the temporal \ac{mv} candidate is limited to those cases w
For \ac{mv} prediction only a much lower number of candidates is allowed, since the encoder can still send a \ac{mv} difference. Moreover, the encoder performs motion estimation for each candidate, which increases the encoding complexity for every possibility tested.
%For those cases where the reference index is not the same as the current \ac{pu} and the neighbouring one, a scaling version of the \ac{mv} is used.
%In the \ac{h264} standard, along different motion compensated coding techniques~\cite{H264OverviewJul2003}, the so-called P\_SKIP mode was introduced. In this mode, neither the quantised residual information, nor the \ac{mv} is transmitted. This mode is useful for large areas with no change or just constant motion, and it is only enabled for $16\times16$ blocks, \emph{i.e.}, the whole \ac{mb}. The \ac{mv} for the P\_SKIP mode is derived based on the surrounding neighbour MBs\cite{H264MotionVectorPrediction2003}. When no neighbours are available, or the left or top \acs{mv} are equal to zero, a zero \ac{mv} is used. Otherwise, the \ac{mv} is derived from the median value of the following three neighbours: left, above and above right, as shown in Figure \ref{SkipMergeMode}~(a) by $MV_A$, $MV_B$ and $MV_C$, respectively. Whenever the $MV_C$ is not available, the $MV_D$ is used instead.
%In the \ac{h264} standard, along different motion compensated coding techniques~\cite{H264OverviewJul2003}, the so-called P\_SKIP mode was introduced. In this mode, neither the quantised residual information, nor the \ac{mv} is transmitted. This mode is useful for large areas with no change or just constant motion, and it is only enabled for $16\times16$ blocks, \emph{i.e.}, the whole \ac{mb}. The \ac{mv} for the P\_SKIP mode is derived based on the surrounding neighbour MBs\cite{H264MotionVectorPrediction2003}. When no neighbours are available, or the left or top \acs{mv} are equal to zero, a zero \ac{mv} is used. Otherwise, the \ac{mv} is derived from the median value of the following three neighbours: left, above and above right, as shown in Figure~\ref{SkipMergeMode}~(a) by $MV_A$, $MV_B$ and $MV_C$, respectively. Whenever the $MV_C$ is not available, the $MV_D$ is used instead.
%
%In contrast with the \ac{h264} standard, the \ac{hevc} standard allows the usage of a temporal \ac{mv} predictor, as well as, more spatial candidates to predict the current motion vector. Despite the improvements in the flexibility and coding efficiency, the increased levels of dependencies between the \acs{mv} make the motion information more prone to errors. Especially the introduction of the temporal candidate in the set of possible \acs{mv} increases the dependencies between subsequent frames. This leads to higher quality degradation in the presence of transmission errors, when compared to \ac{h264}.