jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Review of chapter 2

Browse Source
This commit is contained in:
jfmcarreira
2018-07-30 15:38:33 +01:00
parent f3f362359e
commit 74109b2c4c
31 changed files with 751 additions and 352 deletions
Download Patch File Download Diff File Expand all files Collapse all files
043_EndToEndRobustSystem/EndToEndRobustSystem.tex
+164 -43
View File
@@ -1,7 +1,7 @@
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CHAPTER
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{\HIGHLIGHT{Optimal Robust Video Coding for High Efficient Video Coding}}
\chapter{\HIGHLIGHT{Error Robustness Optimisation for Highly Efficient Video Coding}}
\label{Chapter:EndToEndRobustSystem}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -50,19 +50,19 @@ In this experimental evaluation the following methods are evaluated, namely:
\begin{figure}[!t]
\centering
\input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex}
\vspace{-2mm}
\vspace{-4mm}
\caption{Average decoded video quality using the main target bitrate.}
\label{Figure6:AverageQualityMain}
\end{figure}
\subsubsection{Experimental results}
\subsubsection{Experimental results and discussion}
Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} shows the average decoded video quality obtained from 50 different trials, measured with the \ac{psnr} quality metric. The figures shows the quality obtained for different loss conditions. Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate shown in the third column of Table~\ref{TableTestSequences}. Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained using a lower amount of coded bits, corresponding to a 40\% reduction of the main target bitrate. This aims at finding the impact of the video and network bitrate on the performance of the proposed methods.
\begin{figure}[!t]
\centering
\input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex}
\vspace{-2mm}
\vspace{-4mm}
\caption{Average decoded video quality using a 40\% reduction of the main target bitrate}
\label{Figure6:AverageQualityReduction}
\end{figure}
@@ -78,8 +78,10 @@ Summarising, the performance of the proposed methods is not similar across diffe
\section{Proposed architecture}
\label{Section6:ProposedMethod}
In this section the architecture of the proposed method is described. The goal of the proposed method is to obtain an optimal error robustness method for a wide range of applications, i.e., video content and network conditions.
In order to accomplish that, at the encoder-side, the input signals are coded using different levels of error protections. Subsequently, an optimisation algorithm is used to decide which bitstream should be transmitted at any given instant, depending of different parameters.
This section provides an overview of the proposed method, which aims at obtaining the optimal error robustness method for different streaming conditions, i.e., video content and network conditions.
The main advantage of using the proposed algorithm is that it leads to an optimal trade-off between error robustness tools and coding efficiency.
This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and are selected based the current conditions, i.e., video signal and network conditions. This architecture is suitable for dynamic streaming, e.g., HTTP streaming based on MPEG-DASH~\cite{HEVCHTTPStreamAug2016} or content delivery over heterogeneous networks using the \ac{mmt}~\cite{ATSCMMT2016}.
In order to accomplish that, at the encoder-side, the input signals are coded using different levels of error protections. Subsequently, an optimisation algorithm is used to decide which bitstream should be transmitted at any given instant, based of different parameters. The selected stream is expected to obtain the highest quality at the decoder-side for the given conditions.
\begin{figure}[tb]
\centering
@@ -91,62 +93,181 @@ In order to accomplish that, at the encoder-side, the input signals are coded us
\end{figure}
Figure~\ref{FIG6:BlockDiagram} illustrates the architecture of the proposed encoding scheme.
Firstly, a reference video coder is used to compress the bitstream using highly efficiency coding techniques in order to obtain the best \ac{rd} performance for error-free environments. This is used as the reference bitstream in the proposed approach.
Subsequently, a transcoding operation based on the reference bitstream and the input video is used to efficiently select the reference pictures used for inter-frame prediction in terms of error propagation. The reference picture selection is applied as proposed in Chapter~\ref{Chapter:ErrorResilienceTechniques}.
Moreover, a loss simulation and saliency-based \ac{ec} optimisation is also applied on the error robust bitstream. This is used to select the optimal \ac{ec} algorithm which should be used at the decoder side.
Since the optimal \ac{ec} mode might not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el} using two different bitrates, i.e., with 10\% and 20\% of the total bitrate. This provides two different levels of error robustness. This \ac{ec} optimisation and the \ac{el} coding is applied as proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques}
Firstly, a reference video coder is used to compress the bitstream using highly efficient coding techniques in order to obtain the best \ac{rd} performance for error-free environments.
This is used as the reference bitstream in the proposed approach.
Subsequently, a transcoding operation based on the reference bitstream and the input video is used to efficiently select the reference pictures used for inter-frame prediction in terms of error propagation.
The reference picture selection is applied as proposed in Chapter~\ref{Chapter:ErrorResilienceTechniques}.
Moreover, a loss simulation and a saliency-based \ac{ec} optimisation is also applied on the robust bitstream. This is used to select the optimal \ac{ec} algorithm which should be selected at the decoder-side.
Since the optimal \ac{ec} mode might not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el} using two different bitrates, i.e., with 10\% and 20\% of the total bitrate. This provides two different levels of error robustness. The \ac{ec} optimisation and \ac{el} coding is performed as proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques}.
The proposed approach results in four different bitstreams: (i) reference stream without error robustness; (ii) robust video coded using a reference picture selection algorithm which leads to lower error propagation in case of data-loss; (iii) robust video coding and decoding using optimal reference pictures and optimal \ac{ec} method signalling and 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue; (iv) same as the previous bitstream but with 20\$ of the bitrate allocated for the scalable \ac{el}. These bitstreams are referred in Figure~\ref{FIG6:BlockDiagram} as \textit{Stream\_1} to \textit{Stream\_4}, respectively.
The proposed approach results in four different bitstreams: (i) reference stream without error robustness; (ii) robust video coding using a reference picture selection algorithm which leads to lower error propagation in case of data-loss; (iii) robust video coding and decoding using selective reference pictures and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue; (iv) same as the previous bitstream but with 20\% of the bitrate allocated for the scalable \ac{el}. These bitstreams are referred to as "S1" to "S4", respectively, as shown in Figure~\ref{FIG6:BlockDiagram}.
Finally, the optimal robust stream is selected using a selection algorithm based on
Finally, the video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and from the transmission channel.
From the input video signal relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, informations is also extracted from the coded bitstream using a stream parser.
%the predictions used by the video encoder as well as the target bitrate are extracted using a stream parser.
Finally, a parameter regarding the transmission channel is also used to predict the data loss.
These parameters are fed into the error robustness optimisation block to select the suitable bitstream which should be transmitted to obtain the highest video quality. The optimisation process is described in detail in Section~\ref{Section6:OptimisationMethod}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Error robustness optimisation method}
\section{Error robustness selection method}
\label{Section6:OptimisationMethod}
In this section the optimisation method used to select a robust highly efficiently coded stream is presented.
As mentioned before, the proposed scheme aims at predict the best method which should be used at the encoder-side, that achieves the highest quality. The selection method is based on a shallow neural network which predicts the optimal stream based on different parameters. Moreover, in this section a description of the input parameters obtained from the stream parser and the video analyser is also described.
\subsection{Input parameters}
\glsreset{si}
\glsreset{ti}
A set of different parameters are used to predict the optimal approach that achieves the higher quality in specific network conditions. Based on these inputs the proposed scheme is able to select the appropriate robust coding scheme to be used. These parameters aim at covering three different aspects of the video transmission: two parameters from the input video, three parameters from the video stream and two parameters from the transmission network.
To characterise the input video signal the following parameters are used: \ac{si} and \ac{ti}~\cite{ITUTP9112008}.
These parameters are obtained as described in Chapter~\ref{Chapter:HEVCErrorResilience}, and aim at evaluating the spatial complexity of the video scenes and the motion complexity, which have impact on the error robustness of the coded streams.
As shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, test material with higher \ac{si} or \ac{ti} suffer from higher quality degradation as the \ac{plr} increases, e.g., Book Arrival and Tennis sequences.
From the coded video streams complementary information to characterise the streaming content is also extracted. A stream parser is applied to the reference bitstream without error robustness techniques. Therefore, the information extracted regarding the prediction information is related to a stream coded based on an optimal \ac{rd} optimisation which was not affected by robust coding techniques. The following information is extracted:
\begin{itemize}
\item \textit{Inter(\%)}: the relation between ration of the image which is coded based on intra and inter frame predictions. This is expressed by the ration of blocks coded using inter frame predictions.
The number of inter-coded blocks indicates the distribution of spatial and temporal dependencies.
Normally, coded streams with higher number of temporal dependencies are more prone to error propagation, thus higher number of bits should be allocated for error protection \HIGHLIGHT{[XX]}.
\item \textit{AVG\_MV}: from those regions coded using inter frame predictions the motion information is analysed and the average \ac{mv} size is used to provide information regarding the motion intensity. Motion based \ac{ec} algorithms are less efficient \HIGHLIGHT{[XX]} for video signals with higher motion intensity as \acs{mv} are less predictable using algorithms such as \ac{mc} and \ac{mve}.
\item \textit{Bitrate}: information regarding the bit allocation is extracted, since it will affect the video quality and the amount of bits that can be allocated for robust coding algorithms and auxiliary information.
\end{itemize}
Finally, since the amount of data loss affects the decoded video quality the transmission network \ac{plr} is also used as input parameter. Transmission networks with higher loss ratios require a higher bit allocation for robust techniques, therefore the error resilience technique should adapt based on the value of the \ac{plr}.
\subsection{Neural network}
Neural Networks are a computational model based on the structure and functions of biological neural networks.
In terms of computer science, a neural network works artificial human nervous system for receiving, processing, and transmitting information between neurons. Mainly, a neural network consists of three different layers: the input layer which receives data from an external environment, the hidden layer which process the inputs and an output layer~\cite{NeuralNetworks1994}.
In the proposed method a feed-forward network is used which connects every neuron a given layer to the neurons of the subsequent layer.
\begin{figure}[tb]
\centering
%\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram}
\input{043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz}
\caption{Architecture of the proposed scheme.}
\label{FIG6:BlockDiagram}
\FIGSVSPACE
\end{figure}
Figure~\ref{FIG6:BlockDiagram} shows the diagram of the neural network used in the proposed method. \HERE
In the proposed method the input layer receives the six input described above which are related with the output video quality which is the optimisation variable. Subsequently, the neural network transforms the input data through a series of neurons in two hidden layers.
\section{Performance evaluation}
%\begin{table}[t]
% \centering
% \caption{List of the test conditions tested.}
% \label{Table6:TestVariables}
% \TABLESVSPACE
% \begin{tabular}{m{4cm}|y{4cm}|y{6cm}}
% \toprule
% \textbf{Sequence} & \textbf{Bitrate} & \textbf{\ac{plr}} \\ \midrule
% Basketball Drill & Main (see Table~\ref{TableTestSequences}) & Error-free \\
% Book Arrival & 10\% reduction & 0.5\% \\
% Bosphorus & 20\% reduction & 1.0\% \\
% BQSquare & 30\% reduction & 1.2\% \\
% Cactus & 40\% reduction & 1.6\% \\
% Four People & 50\% reduction & 2.0\% \\
% Jockey & & 2.2\% \\
% Kendo & & 2.4\% \\
% Kimono & & 2.6\% \\
% Kristen and Sara & & 2.8\% \\
% Park Scene & & 3.0\% \\
% People on Street & & 5.0\% \\
% Race Horses & & 10.0\% \\
% Tennis & & \\
% Traffic & & \\
% \bottomrule
% \end{tabular}
%\end{table}
\subsection{Selection Accuracy}
\begin{table}[t]
\centering
\caption{List of the test conditions tested.}
\label{Table6:TestVariables}
\TABLESVSPACE
\begin{tabular}{m{4cm}|y{4cm}|y{6cm}}
\toprule
\textbf{Sequence} & \textbf{Bitrate} & \textbf{\ac{plr}} \\ \midrule
Basketball Drill & Main (see Table~\ref{TableTestSequences}) & Error-free \\
Book Arrival & 10\% reduction & 0.5\% \\
Bosphorus & 20\% reduction & 1.0\% \\
BQSquare & 30\% reduction & 1.2\% \\
Cactus & 40\% reduction & 1.6\% \\
Four People & 50\% reduction & 2.0\% \\
Jockey & & 2.2\% \\
Kendo & & 2.4\% \\
Kimono & & 2.6\% \\
Kristen and Sara & & 2.8\% \\
Park Scene & & 3.0\% \\
People on Street & & 5.0\% \\
Race Horses & & 10.0\% \\
Tennis & & \\
Traffic & & \\
\bottomrule
\end{tabular}
\centering
\caption{Confusion matrix of the training and validation set.}
\label{Table6:ConfusionTraining}
\TABLESVSPACE
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{x{.5cm}x{1cm}|x{1.1cm}|x{1.1cm}|x{1.1cm}|x{1.1cm}|}
& & \multicolumn{4}{c}{Optimal} \\
\multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}}
& & S1 & S2 & S3 & S4 \\ \hline
& S1 & 122 & 17 & 3 & 0 \\
\cline{2-6} & S2 & 13 & 80 & 19 & 2 \\
\cline{2-6} & S3 & 0 & 12 & 96 & 9 \\
\cline{2-6} & S4 & 0 & 4 & 11 & 202 \\
\hline
\multicolumn{2}{x{1.7cm}|}{Accuracy} & 90.3\% & 70.8\% & 74.4\% & 94.8\% \\
\end{tabular}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% SECTION %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\section{Error concealment optimisation based on MV differences}
\begin{table}[t]
\centering
\caption{Confusion matrix of the validation test set.}
\label{Table6:ConfusionTest}
\TABLESVSPACE
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{x{.5cm}x{1cm}|x{1.1cm}|x{1.1cm}|x{1.1cm}|x{1.1cm}|}
& & \multicolumn{4}{c}{Optimal} \\
\multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}}
& & S1 & S2 & S3 & S4 \\ \hline
& S1 & 56 & 7 & 2 & 0 \\
\cline{2-6} & S2 & 10 & 28 & 6 & 2 \\
\cline{2-6} & S3 & 1 & 5 & 31 & 8 \\
\cline{2-6} & S4 & 0 & 5 & 7 & 86 \\
\hline
\multicolumn{2}{x{1.7cm}|}{Accuracy} & 83.6\% & 62.2\% & 67.4\% & 89.6\% \\
%\bottomrule
\end{tabular}
\end{table}
\subsection{Experimental results}
\subsection{Quality evaluation}
\begin{table}[t]
\centering
\caption{Average quality gains ($\Delta$PSNR) in comparison with the reference \ac{hevc} across different bitrates and loss ratios.}
\label{Table6:AverageGains}
\TABLESVSPACE
\newcommand{\colwidth}{2cm}
\begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}}
\toprule
\multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\
& S1 & S2 & S3 & S4 & Optimal (\textit{Proposed}) \\ \hline
Basketball Drill & 1.61 & 2.66 & 2.48 & 2.44 & 2.75 \\ \hline
Basketball Drive & 2.17 & 3.46 & 3.22 & 3.63 & 3.69 \\ \hline
Book Arrival & 0.96 & 1.66 & 1.43 & 1.59 & 1.74 \\ \hline
BQSquare & 0.92 & 0.96 & 0.80 & 0.62 & 1.09 \\ \hline
Cactus & 1.28 & 2.04 & 1.89 & 1.96 & 2.09 \\ \hline
Four People & 0.28 & -0.23 & -0.88 & -0.99 & 0.25 \\ \hline
Kendo & 0.81 & 0.78 & 0.20 & 0.61 & 1.02 \\ \hline
Kimono & 1.65 & 3.04 & 3.26 & 3.27 & 3.41 \\ \hline
Kristen Sara & 0.36 & 0.15 & -0.33 & -0.50 & 0.38 \\ \hline
Park Scene & 1.64 & 2.18 & 1.88 & 2.02 & 2.25 \\ \hline
People on Street & 1.00 & 1.27 & 0.91 & 1.49 & 1.63 \\ \hline
Race Horses & 1.93 & 3.89 & 4.06 & 4.30 & 4.39 \\ \hline
Tennis & 1.15 & 2.24 & 2.17 & 3.12 & 3.18 \\ \hline
\textbf{Average} & \textbf{1.21} & \textbf{1.86} & \textbf{1.62} & \textbf{1.81} & \textbf{2.15} \\
\bottomrule
\end{tabular}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%