Complete review of PA comments in chapter 7 and SF coments on chapters 1 to 6
This commit is contained in:
@@ -140,6 +140,9 @@ Moreover, new pictures types were introduced to increase the random-access flexi
|
||||
|
||||
Similar to \ac{h264}~\cite{H264AVCStandard2003}, the \ac{hevc} streams are based on \glsfirst{nal} for stream partitioning and synchronization, which is useful to deal with lossy network environments~\cite{H264OverIPJul2003}. Using this concept the stream is composed of different \ac{nal} units. The \ac{nal} concept provides the ability to map the \ac{vcl} data that represents coded video slices (or frames) on various transport layers, such as \ac{rtp}, MPEG-2 Systems and MPEG DASH. The~\ac{nal} units are classified into two categories: (i) \ac{vcl} and (ii) non-\ac{vcl}, as they carry coded video or associated data, respectively.
|
||||
|
||||
The \ac{hevc} standard defines two bytes to describe the content of each \ac{nal} unit.
|
||||
These bytes are used to identify the \ac{nal} type are transmitted using a fixed coding format length, i.e., without variable length codes. This allow extra error robustness but increases the bit overhead. Figure~\ref{Figure2:NalHeader} shows the structure of the \ac{hevc} \ac{nal} unit header. The first bit \textit{F}, referred to as \textit{forbiden\_bit}, is always zero in order to ensure that a start code can be detected across different protocol standards. This is followed by the \ac{nal} type, which identifies its content, enabling the decoder to either use it if required for a particular decoding purpose or discard it. Another syntax element introduced by the \ac{hevc} standard is the \textit{temporal\_id\_plus\_one} (TIDP), which defines the temporal layer where the \ac{nal} unit belongs to. This allows implicit temporal scalability (layers ranging from 0 to 6) by immediately discarding the \ac{nal} unit if it belongs to a layer higher than the desired one. In the header, six bits are reserved for future extensions, preventing the creation of extra \ac{nal} units for scalable and multiview extensions, as done in \ac{h264}.
|
||||
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
\vspace{5mm}
|
||||
@@ -155,11 +158,6 @@ Similar to \ac{h264}~\cite{H264AVCStandard2003}, the \ac{hevc} streams are based
|
||||
\label{Figure2:NalHeader}
|
||||
\end{figure}
|
||||
|
||||
The \ac{hevc} standard defines two bytes to describe the content of each \ac{nal} unit.
|
||||
These bytes are used to identify the \ac{nal} type are transmitted using a fixed coding format length, i.e., without variable length codes. This allow extra error robustness but increases the bit overhead. Figure~\ref{Figure2:NalHeader} shows the structure of the \ac{hevc} \ac{nal} unit header. The first bit \textit{F}, referred to as \textit{forbiden\_bit}, is always zero in order to ensure that a start code can be detected across different protocol standards. This is followed by the \ac{nal} type, which identifies its content, enabling the decoder to either use it if required for a particular decoding purpose or discard it. Another syntax element introduced by the \ac{hevc} standard is the \textit{temporal\_id\_plus\_one} (TIDP), which defines the temporal layer where the \ac{nal} unit belongs to. This allows implicit temporal scalability (layers ranging from 0 to 6) by immediately discarding the \ac{nal} unit if it belongs to a layer higher than the desired one. In the header, six bits are reserved for future extensions, preventing the creation of extra \ac{nal} units for scalable and multiview extensions, as done in \ac{h264}.
|
||||
|
||||
|
||||
|
||||
\subsection{Parameters set}
|
||||
|
||||
\glsreset{vps}
|
||||
@@ -249,13 +247,6 @@ Therefore, in order to make splicing operation straightforward, the \ac{nal} uni
|
||||
|
||||
\subsubsection{Leading and trailing pictures}
|
||||
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
\input{\FigsPath/PictureTypes_Leading_Trailing.tikz}
|
||||
\caption{Leading and trailing pictures associated with a random-access point.}
|
||||
\label{Figure2:LeadingTralingPics}
|
||||
\end{figure}
|
||||
|
||||
Leading pictures correspond to frames that follow a particular \ac{rap} frame in the decoding order, but precede it in the display order.
|
||||
A trailing picture is a frame that follows a particular \ac{rap} frame in both decoding and display order.
|
||||
%A trailing picture is a frame that follows a particular \ac{rap} frame in both decoding and display order and must not depend on any leading frames nor any trailing frame associated to a previous \ac{rap}.
|
||||
@@ -264,6 +255,13 @@ Leading and trailing pictures are considered to be associated with the closest p
|
||||
Also, all leading frames of a \ac{rap} frame must precede in decoding order all trailing frames that are associated with the same \ac{rap}. This means that the following order is imposed by the \ac{hevc} standard: 1) \ac{rap} picture, 2) associated leading pictures and 3) associated trailing pictures.
|
||||
Due to the introduction of more flexible random-access points using \ac{cra} pictures, it is important to mark the leading and trailing pictures, so the decoder can be aware of which frames can be correctly decodable whenever it starts decoding the bitstream.
|
||||
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
\input{\FigsPath/PictureTypes_Leading_Trailing.tikz}
|
||||
\caption{Leading and trailing pictures associated with a random-access point.}
|
||||
\label{Figure2:LeadingTralingPics}
|
||||
\end{figure}
|
||||
|
||||
There are two types of leading pictures, as shown in Table~\ref{Table2:PictureTypes}.
|
||||
The \ac{radl} pictures, which can be correctly decoded since they only depend on the associated \ac{rap} picture and do not have any prior dependencies to trailing pictures associated with the previous \ac{rap} frame.
|
||||
On contrary, the \ac{rasl} pictures may have dependencies to prior trailing pictures.
|
||||
@@ -341,6 +339,10 @@ The use of tiles provide several advantages. Specifically, they improve frame pa
|
||||
|
||||
As mentioned before, the use of either slices or tiles to process each frame using parallel threads require the use of different entropy coding contexts for each slice/tile, in order to make them independently decodable. As consequence, the compression efficiency decreases. In order to overcome this issue, the concept of \ac{wpp} was introduced, which enables coding using efficient parallel processing~\cite{HEVCParallelCSVT2016}.
|
||||
|
||||
In \ac{wpp}, each slice (or tile) is divided into single rows of \acs{ctu} which can be processed in parallel, but entropy coding and prediction are enabled across \acs{ctu} of different rows, in order to avoid loss in coding efficiency.
|
||||
To re-use the same context in the whole slice (or tile) each row requires a delay of two \acs{ctu} in order to ensure that all necessary dependencies have already been encoded/decoded. Figure~\ref{Figure2:WPP} illustrates an example of \ac{wpp} using at least five different threads.
|
||||
Each square corresponds to a \ac{ctu} and the numbers correspond to the time instance when a \ac{ctu} starts being processed. Using this staggered start (which appears like a wavefront, as represented in Figure~\ref{Figure2:WPP}), the parallelisation may use as many threads as the number of \acs{ctu} rows available in the video frame. However, although the number of threads does not affect the coding efficiency, the required inter-process communication substantially increases.
|
||||
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
\input{\FigsPath/WPP.tikz}
|
||||
@@ -348,10 +350,6 @@ As mentioned before, the use of either slices or tiles to process each frame usi
|
||||
\label{Figure2:WPP}
|
||||
\end{figure}
|
||||
|
||||
In \ac{wpp}, each slice (or tile) is divided into single rows of \acs{ctu} which can be processed in parallel, but entropy coding and prediction are enabled across \acs{ctu} of different rows, in order to avoid loss in coding efficiency.
|
||||
To re-use the same context in the whole slice (or tile) each row requires a delay of two \acs{ctu} in order to ensure that all necessary dependencies have already been encoded/decoded. Figure~\ref{Figure2:WPP} illustrates an example of \ac{wpp} using at least five different threads.
|
||||
Each square corresponds to a \ac{ctu} and the numbers correspond to the time instance when a \ac{ctu} starts being processed. Using this staggered start (which appears like a wavefront, as represented in Figure~\ref{Figure2:WPP}), the parallelisation may use as many threads as the number of \acs{ctu} rows available in the video frame. However, although the number of threads does not affect the coding efficiency, the required inter-process communication substantially increases.
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
%% SECTION %%
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
@@ -450,19 +448,19 @@ To improve the efficiency of angular prediction, when the reference samples need
|
||||
|
||||
\subsubsection{Planar and DC}
|
||||
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
\input{021_VideoCoding/Figures/IntraPredictionPlanar.tikz}
|
||||
\caption{Example of Planar prediction for a $4\times4$ block.}
|
||||
\label{Figure2:IntraModePlanar}
|
||||
\end{figure}
|
||||
|
||||
The Planar and DC modes are particularly efficient in the prediction of smooth regions.
|
||||
The DC mode uses the average or the neighbouring reference samples to generate a constant prediction for the current \ac{pu} as a whole, i.e., all pixels are predicted from the same value.
|
||||
The Planar mode predicts the \ac{pu} through a linear interpolation from the four closest neighbouring reference pixels. Figure~\ref{Figure2:IntraModePlanar} shows the reference pixels (i.e., $p_1$, $p'_2$, $p_3$ and $p'_4$) and arrows pointing to the pixel being predicted.
|
||||
Since the right and bottom neighbourhood is not available (not coded yet), the same pixel $p_2$ is repeated to form a column of reference pixels, marked with an ellipse in Figure~\ref{Figure2:IntraModePlanar}. A similar approach is also used to create the bottom row of reference pixels from the pixel $p_4$ .
|
||||
%As the Planar mode used linear surface for prediction it prevents the discontinuities at \ac{pu} boundaries.
|
||||
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
\input{021_VideoCoding/Figures/IntraPredictionPlanar.tikz}
|
||||
\caption{Example of Planar prediction for a $4\times4$ block.}
|
||||
\label{Figure2:IntraModePlanar}
|
||||
\end{figure}
|
||||
|
||||
|
||||
\subsubsection{Sample smoothing}
|
||||
|
||||
@@ -525,6 +523,15 @@ Once a frame is marked as \emph{unused for reference} it is no longer used for m
|
||||
In \ac{h264} a sliding window approach is used to implicitly manage the reference picture list.
|
||||
This process discards a given frame whenever the \ac{dpb} has the number of defined reference frames (which is controlled by the encoder). Moreover, \ac{h264} uses an explicitly \ac{mmco} to send commands to the decoder and change the normal reference picture management process.
|
||||
|
||||
In the \ac{hevc} standard the status of the \ac{dpb} is encoded for every slice, instead of the implicit managing used in the past.
|
||||
The most fundamental difference in \ac{rpset} concept compared to \ac{mmco}/sliding window of \ac{h264} is that each particular frame has a complete set of the reference frames that are used or will be used.
|
||||
Thus, a complete set of all frames that must be kept in the \ac{dpb} is explicitly provided.
|
||||
This is different from \ac{h264} where only relative changes to the \ac{dpb} are encoded.
|
||||
With the \ac{rpset} concept, no information from earlier frames in decoding order is needed to maintain the correct status of the \ac{dpb}.
|
||||
Table~\ref{Table2:ReferencePictureListExample} shows the \ac{rpset} for some frames illustrated in Figure~\ref{Figure2:LeadingTralingPics}.
|
||||
As expected, for the \ac{cra} frame ($I_1$) no information is sent to the decoder because there is not inter-prediction and the \ac{dpb} should be empty.
|
||||
For the remaining frames, two types of information are encoded: frame number and a binary flag to indicate whether it is currently used. This is required because any frame not listed in a given \ac{rpset} will be discarded by the decoder and then subsequent frames cannot use them as reference.
|
||||
|
||||
\begin{table}[tb]
|
||||
\centering
|
||||
\caption{Reference Picture Set example.}
|
||||
@@ -544,16 +551,6 @@ This process discards a given frame whenever the \ac{dpb} has the number of defi
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
In the \ac{hevc} standard the status of the \ac{dpb} is encoded for every slice, instead of the implicit managing used in the past.
|
||||
The most fundamental difference in \ac{rpset} concept compared to \ac{mmco}/sliding window of \ac{h264} is that each particular frame has a complete set of the reference frames that are used or will be used.
|
||||
Thus, a complete set of all frames that must be kept in the \ac{dpb} is explicitly provided.
|
||||
This is different from \ac{h264} where only relative changes to the \ac{dpb} are encoded.
|
||||
With the \ac{rpset} concept, no information from earlier frames in decoding order is needed to maintain the correct status of the \ac{dpb}.
|
||||
Table~\ref{Table2:ReferencePictureListExample} shows the \ac{rpset} for some frames illustrated in Figure~\ref{Figure2:LeadingTralingPics}.
|
||||
As expected, for the \ac{cra} frame ($I_1$) no information is sent to the decoder because there is not inter-prediction and the \ac{dpb} should be empty.
|
||||
For the remaining frames, two types of information are encoded: frame number and a binary flag to indicate whether it is currently used. This is required because any frame not listed in a given \ac{rpset} will be discarded by the decoder and then subsequent frames cannot use them as reference.
|
||||
|
||||
|
||||
The \ac{rpset} concept provides a basic level of error robustness to the reference picture management.
|
||||
In \ac{h264}, whenever a reference frame is lost the decoder can look at the value of the \emph{FrameNum} (incremental value associated with each reference frame) to detect the loss event.
|
||||
However, as gaps in \emph{FrameNum} are allowed, the decoder might not be able to decide whether the missing frame was intentional or resulted from a transmission loss event.
|
||||
@@ -562,16 +559,6 @@ Using the \ac{rpset} concept, the decoder is always able to detect loss events a
|
||||
|
||||
\subsubsection{Asymmetric block partitions}
|
||||
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
\begin{tabular}{x{.8\columnwidth}x{.1\columnwidth}}
|
||||
\input{\FigsPath/PUSymetric.tikz} & (a)\\
|
||||
\input{\FigsPath/PUAsymetric.tikz} & (b)\\
|
||||
\end{tabular}
|
||||
\caption{Partition types of an inter coded CU into PUs~\cite{HEVCBlock2012}.}
|
||||
\label{Figure2:InterPuPartitions}
|
||||
\end{figure}
|
||||
|
||||
|
||||
When inter prediction mode is selected a higher range of partitioning schemes is allowed, thus dividing the \ac{cu} into one, two or four \acs{pu}.
|
||||
The splitting into four \acs{pu} is only allowed when the \ac{cu} size is equal to the minimum.
|
||||
@@ -585,15 +572,18 @@ Finally, the $\frac{N}{2} \times \frac{N}{2}$ case results in four \acs{pu}.
|
||||
The partitions shown in Figure~\ref{Figure2:InterPuPartitions}~(b) correspond to the asymmetric partitioning, which is one of the main improvements introduced in \ac{hevc}, as it allows higher flexibility in motion estimation, leading to higher compression efficiency~\cite{HEVCBlock2012}.
|
||||
These partitions types are only allowed for \acs{cu} with $16\times16$ pixels or larger.
|
||||
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
\begin{tabular}{x{.8\columnwidth}x{.1\columnwidth}}
|
||||
\input{\FigsPath/PUSymetric.tikz} & (a)\\
|
||||
\input{\FigsPath/PUAsymetric.tikz} & (b)\\
|
||||
\end{tabular}
|
||||
\caption{Partition types of an inter coded CU into PUs~\cite{HEVCBlock2012}.}
|
||||
\label{Figure2:InterPuPartitions}
|
||||
\end{figure}
|
||||
|
||||
\subsubsection{Merge mode}
|
||||
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
\input{\FigsPath/MergeModeHEVC.tikz}
|
||||
%\input{\FigsPath/HEVC/MergeModeHEVCTwoFrames.tikz}
|
||||
\caption{Illustration of the MV prediction candidates in the AMVP mode.}
|
||||
\label{Figure2:MVCandidates}
|
||||
\end{figure}
|
||||
|
||||
The Merge mode was introduced in \ac{hevc} as a new technique to derive motion information from spatial and temporal neighbouring blocks~\cite{MotionCodingHEVC2013}. This technique extends the concepts of the Direct and Skip modes used in \ac{h264}~\cite{H264OverviewJul2003}, using a more sophisticated approach with two key differences.
|
||||
Firstly, the Merge Mode allows for more \ac{mv} candidates than the Skip in \ac{h264}.
|
||||
@@ -610,6 +600,13 @@ The Merge Mode defines how to encode the \ac{mv} candidate and the residual info
|
||||
In \ac{hevc}, the Skip mode, i.e., transmitting only the \ac{mv} without residual information, is treated as a special case of the Merge Mode.
|
||||
In this case, only a skip flag is enabled and the corresponding merge index is transmitted.
|
||||
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
\input{\FigsPath/MergeModeHEVC.tikz}
|
||||
%\input{\FigsPath/HEVC/MergeModeHEVCTwoFrames.tikz}
|
||||
\caption{Illustration of the MV prediction candidates in the AMVP mode.}
|
||||
\label{Figure2:MVCandidates}
|
||||
\end{figure}
|
||||
|
||||
\subsubsection{Motion vector prediction}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user