277 lines
21 KiB
TeX
277 lines
21 KiB
TeX
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
% CHAPTER
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\chapter{\HIGHLIGHT{Error Robustness Optimisation for Highly Efficient Video Coding}}
|
|
\label{Chapter:EndToEndRobustSystem}
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Introduction}
|
|
|
|
|
|
In this chapter the error resilience and error concealment approaches proposed in the previous chapters, namely Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, are combined to achieved a higher decoder video quality across different network conditions and applications.
|
|
This chapter presents a combined evaluation of different approaches and proposes a method to efficiently select the optimal one in terms of reconstructed quality.
|
|
|
|
This chapter is organised as follows. Section~\ref{Section6:Problem} presents the problem formulation and motivation for the method proposed in this chapter. \HIGHLIGHT{...}
|
|
Section~\ref{Section6:Summary} concludes this chapter.
|
|
|
|
\clearpage
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Problem formulation}
|
|
\label{Section6:Problem}
|
|
|
|
In this research work different robust coding techniques were proposed to increase the overall performance of video transmission under error-prone conditions. The methods proposed used different algorithms, as reference picture selection, motion vector redundancies and error concealment signalling. In previous chapters theses methods were evaluated against existing techniques revealing their superior performance and efficiency to deal with error robustness problems in high efficient video coding, e.g., using the \ac{hevc} standard.
|
|
|
|
The evaluation of these methods revealed that they can efficiently increase the robustness to transmission errors and losses, leading to higher decoded video quality. Results in Chapter~\ref{Chapter:ErrorResilienceTechniques} show that the proposed reference picture selection algorithm introduces low overhead in comparison with existing techniques. However, the method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} achieves higher quality gains at cost of slightly higher bitrate increase.
|
|
Therefore, it is noticeable that each method might be optimised for different applications, video characteristics and network conditions.
|
|
|
|
To advance one step forward, a jointly robust coding method using different techniques can be used to increase the decoder video quality across difference conditions.
|
|
Such approach can be based on both the input contents and the network information, such as bitrate and packet loss conditions to optimise the robust coders to each application scenario. In order to overcome the above problem a jointly evaluation study is necessary to identify the most efficient techniques and to evaluate their performance for different conditions.
|
|
|
|
|
|
\subsection{Comparison of the proposed methods}
|
|
|
|
In order to develop a jointly robust coder to cope with transmission errors an evaluation study was carried out, to compare the error robustness performance of the proposed methods for different conditions with each others.
|
|
%This study extends the previous evaluations presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, as they target a wider range of test material and network conditions.
|
|
The experimental evaluation carried out in this section follows the same condition described in Section~\ref{Section3:ErrorResilienceEvaluation}.
|
|
%In order to perform a wider evaluation different bitrate and \acs{plr} were used. Table~\ref{Table6:TestVariables} illustrates the selected configurations. As shown in the table a total of 15 test sequences were used encoder using 5 different target bitrates. One should note that the main bitrate varies based on the test sequence and it is shown in third column of Table~\ref{TableTestSequences}. Different network conditions were evaluated using a total of 13 different \acs{plr}, with higher special focus on lower loss ratios, which are more relevant to modern video streaming applications.
|
|
In this experimental evaluation the following methods are evaluated, namely:
|
|
\begin{itemize}
|
|
\item the default \ac{hevc} decoder with frame-copy \ac{ec} (\REF), as reference method;
|
|
\item the proposed reference picture selection scheme (\EXP);
|
|
\item the \EXP method in combination with an \ac{ec} optimisation with 10\% of bitrate of the scalable enhancement layer (\EXPELTEN);
|
|
\item the same as above with 20\% of bitrate for the scalable enhancement layer (\EXPELTWEN).
|
|
\end{itemize}
|
|
|
|
|
|
\begin{figure}[!t]
|
|
\centering
|
|
\input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex}
|
|
\vspace{-4mm}
|
|
\caption{Average decoded video quality using the main target bitrate.}
|
|
\label{Figure6:AverageQualityMain}
|
|
\end{figure}
|
|
|
|
\subsubsection{Experimental results and discussion}
|
|
|
|
Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} shows the average decoded video quality obtained from 50 different trials, measured with the \ac{psnr} quality metric. The figures shows the quality obtained for different loss conditions. Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate shown in the third column of Table~\ref{TableTestSequences}. Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained using a lower amount of coded bits, corresponding to a 40\% reduction of the main target bitrate. This aims at finding the impact of the video and network bitrate on the performance of the proposed methods.
|
|
|
|
\begin{figure}[!t]
|
|
\centering
|
|
\input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex}
|
|
\vspace{-4mm}
|
|
\caption{Average decoded video quality using a 40\% reduction of the main target bitrate}
|
|
\label{Figure6:AverageQualityReduction}
|
|
\end{figure}
|
|
|
|
Results on both figures show that the proposed methods (i.e., \EXP, \EXPELTEN and \EXPELTWEN) are all able to outperform the reference \ac{hevc} for high \acs{plr}. In case of the proposed reference picture selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions shown.
|
|
Comparing the results for both target bitrates also shows that the proposed methods are able to achieve higher quality gains for higher bitrates. Therefore, the for a lower bitrate (see results of Figure~\ref{Figure6:AverageQualityReduction}) the proposed methods only outperform the reference method for slightly higher \acs{plr}. This is more noticeable in the \EXPELTWEN case.
|
|
|
|
Summarising, the performance of the proposed methods is not similar across different video content and target bitrate, which impose a challenge on which method should be applied to achieve the highest decoded video quality in case of transmission errors. In order to achieve the highest quality the optimal trade-off between auxiliary information and video bitrate should be obtained by selecting the optimal error robust coding algorithm.
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Proposed architecture}
|
|
\label{Section6:ProposedMethod}
|
|
|
|
This section provides an overview of the proposed method, which aims at obtaining the optimal error robustness method for different streaming conditions, i.e., video content and network conditions.
|
|
The main advantage of using the proposed algorithm is that it leads to an optimal trade-off between error robustness tools and coding efficiency.
|
|
This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and are selected based the current conditions, i.e., video signal and network conditions. This architecture is suitable for dynamic streaming, e.g., HTTP streaming based on MPEG-DASH~\cite{HEVCHTTPStreamAug2016} or content delivery over heterogeneous networks using the \ac{mmt}~\cite{ATSCMMT2016}.
|
|
In order to accomplish that, at the encoder-side, the input signals are coded using different levels of error protections. Subsequently, an optimisation algorithm is used to decide which bitstream should be transmitted at any given instant, based of different parameters. The selected stream is expected to obtain the highest quality at the decoder-side for the given conditions.
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
%\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram}
|
|
\input{043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz}
|
|
\caption{Architecture of the proposed scheme.}
|
|
\label{FIG6:BlockDiagram}
|
|
\FIGSVSPACE
|
|
\end{figure}
|
|
|
|
Figure~\ref{FIG6:BlockDiagram} illustrates the architecture of the proposed encoding scheme.
|
|
Firstly, a reference video coder is used to compress the bitstream using highly efficient coding techniques in order to obtain the best \ac{rd} performance for error-free environments.
|
|
This is used as the reference bitstream in the proposed approach.
|
|
Subsequently, a transcoding operation based on the reference bitstream and the input video is used to efficiently select the reference pictures used for inter-frame prediction in terms of error propagation.
|
|
The reference picture selection is applied as proposed in Chapter~\ref{Chapter:ErrorResilienceTechniques}.
|
|
Moreover, a loss simulation and a saliency-based \ac{ec} optimisation is also applied on the robust bitstream. This is used to select the optimal \ac{ec} algorithm which should be selected at the decoder-side.
|
|
Since the optimal \ac{ec} mode might not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el} using two different bitrates, i.e., with 10\% and 20\% of the total bitrate. This provides two different levels of error robustness. The \ac{ec} optimisation and \ac{el} coding is performed as proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques}.
|
|
|
|
The proposed approach results in four different bitstreams: (i) reference stream without error robustness; (ii) robust video coding using a reference picture selection algorithm which leads to lower error propagation in case of data-loss; (iii) robust video coding and decoding using selective reference pictures and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue; (iv) same as the previous bitstream but with 20\% of the bitrate allocated for the scalable \ac{el}. These bitstreams are referred to as "S1" to "S4", respectively, as shown in Figure~\ref{FIG6:BlockDiagram}.
|
|
|
|
Finally, the video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and from the transmission channel.
|
|
From the input video signal relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, informations is also extracted from the coded bitstream using a stream parser.
|
|
%the predictions used by the video encoder as well as the target bitrate are extracted using a stream parser.
|
|
Finally, a parameter regarding the transmission channel is also used to predict the data loss.
|
|
These parameters are fed into the error robustness optimisation block to select the suitable bitstream which should be transmitted to obtain the highest video quality. The optimisation process is described in detail in Section~\ref{Section6:OptimisationMethod}.
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Error robustness selection method}
|
|
\label{Section6:OptimisationMethod}
|
|
|
|
In this section the optimisation method used to select a robust highly efficiently coded stream is presented.
|
|
As mentioned before, the proposed scheme aims at predict the best method which should be used at the encoder-side, that achieves the highest quality. The selection method is based on a shallow neural network which predicts the optimal stream based on different parameters. Moreover, in this section a description of the input parameters obtained from the stream parser and the video analyser is also described.
|
|
|
|
\subsection{Input parameters}
|
|
|
|
\glsreset{si}
|
|
\glsreset{ti}
|
|
|
|
A set of different parameters are used to predict the optimal approach that achieves the higher quality in specific network conditions. Based on these inputs the proposed scheme is able to select the appropriate robust coding scheme to be used. These parameters aim at covering three different aspects of the video transmission: two parameters from the input video, three parameters from the video stream and two parameters from the transmission network.
|
|
|
|
To characterise the input video signal the following parameters are used: \ac{si} and \ac{ti}~\cite{ITUTP9112008}.
|
|
These parameters are obtained as described in Chapter~\ref{Chapter:HEVCErrorResilience}, and aim at evaluating the spatial complexity of the video scenes and the motion complexity, which have impact on the error robustness of the coded streams.
|
|
As shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, test material with higher \ac{si} or \ac{ti} suffer from higher quality degradation as the \ac{plr} increases, e.g., Book Arrival and Tennis sequences.
|
|
|
|
From the coded video streams complementary information to characterise the streaming content is also extracted. A stream parser is applied to the reference bitstream without error robustness techniques. Therefore, the information extracted regarding the prediction information is related to a stream coded based on an optimal \ac{rd} optimisation which was not affected by robust coding techniques. The following information is extracted:
|
|
\begin{itemize}
|
|
\item \textit{Inter(\%)}: the relation between ration of the image which is coded based on intra and inter frame predictions. This is expressed by the ration of blocks coded using inter frame predictions.
|
|
The number of inter-coded blocks indicates the distribution of spatial and temporal dependencies.
|
|
Normally, coded streams with higher number of temporal dependencies are more prone to error propagation, thus higher number of bits should be allocated for error protection \HIGHLIGHT{[XX]}.
|
|
|
|
\item \textit{AVG\_MV}: from those regions coded using inter frame predictions the motion information is analysed and the average \ac{mv} size is used to provide information regarding the motion intensity. Motion based \ac{ec} algorithms are less efficient \HIGHLIGHT{[XX]} for video signals with higher motion intensity as \acs{mv} are less predictable using algorithms such as \ac{mc} and \ac{mve}.
|
|
|
|
\item \textit{Bitrate}: information regarding the bit allocation is extracted, since it will affect the video quality and the amount of bits that can be allocated for robust coding algorithms and auxiliary information.
|
|
\end{itemize}
|
|
|
|
Finally, since the amount of data loss affects the decoded video quality the transmission network \ac{plr} is also used as input parameter. Transmission networks with higher loss ratios require a higher bit allocation for robust techniques, therefore the error resilience technique should adapt based on the value of the \ac{plr}.
|
|
|
|
|
|
\subsection{Neural network}
|
|
|
|
Neural Networks are a computational model based on the structure and functions of biological neural networks.
|
|
In terms of computer science, a neural network works artificial human nervous system for receiving, processing, and transmitting information between neurons. Mainly, a neural network consists of three different layers: the input layer which receives data from an external environment, the hidden layer which process the inputs and an output layer~\cite{NeuralNetworks1994}.
|
|
In the proposed method a feed-forward network is used which connects every neuron a given layer to the neurons of the subsequent layer.
|
|
|
|
\begin{figure}[tb]
|
|
\centering
|
|
%\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram}
|
|
\input{043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz}
|
|
\caption{Architecture of the proposed scheme.}
|
|
\label{FIG6:BlockDiagram}
|
|
\FIGSVSPACE
|
|
\end{figure}
|
|
|
|
Figure~\ref{FIG6:BlockDiagram} shows the diagram of the neural network used in the proposed method. \HERE
|
|
In the proposed method the input layer receives the six input described above which are related with the output video quality which is the optimisation variable. Subsequently, the neural network transforms the input data through a series of neurons in two hidden layers.
|
|
|
|
|
|
\section{Performance evaluation}
|
|
|
|
|
|
|
|
%\begin{table}[t]
|
|
% \centering
|
|
% \caption{List of the test conditions tested.}
|
|
% \label{Table6:TestVariables}
|
|
% \TABLESVSPACE
|
|
% \begin{tabular}{m{4cm}|y{4cm}|y{6cm}}
|
|
% \toprule
|
|
% \textbf{Sequence} & \textbf{Bitrate} & \textbf{\ac{plr}} \\ \midrule
|
|
% Basketball Drill & Main (see Table~\ref{TableTestSequences}) & Error-free \\
|
|
% Book Arrival & 10\% reduction & 0.5\% \\
|
|
% Bosphorus & 20\% reduction & 1.0\% \\
|
|
% BQSquare & 30\% reduction & 1.2\% \\
|
|
% Cactus & 40\% reduction & 1.6\% \\
|
|
% Four People & 50\% reduction & 2.0\% \\
|
|
% Jockey & & 2.2\% \\
|
|
% Kendo & & 2.4\% \\
|
|
% Kimono & & 2.6\% \\
|
|
% Kristen and Sara & & 2.8\% \\
|
|
% Park Scene & & 3.0\% \\
|
|
% People on Street & & 5.0\% \\
|
|
% Race Horses & & 10.0\% \\
|
|
% Tennis & & \\
|
|
% Traffic & & \\
|
|
% \bottomrule
|
|
% \end{tabular}
|
|
%\end{table}
|
|
|
|
|
|
\subsection{Selection Accuracy}
|
|
|
|
|
|
\begin{table}[t]
|
|
\centering
|
|
\caption{Confusion matrix of the training and validation set.}
|
|
\label{Table6:ConfusionTraining}
|
|
\TABLESVSPACE
|
|
\renewcommand{\arraystretch}{1.2}
|
|
\begin{tabular}{x{.5cm}x{1cm}|x{1.1cm}|x{1.1cm}|x{1.1cm}|x{1.1cm}|}
|
|
& & \multicolumn{4}{c}{Optimal} \\
|
|
\multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}}
|
|
& & S1 & S2 & S3 & S4 \\ \hline
|
|
& S1 & 122 & 17 & 3 & 0 \\
|
|
\cline{2-6} & S2 & 13 & 80 & 19 & 2 \\
|
|
\cline{2-6} & S3 & 0 & 12 & 96 & 9 \\
|
|
\cline{2-6} & S4 & 0 & 4 & 11 & 202 \\
|
|
\hline
|
|
\multicolumn{2}{x{1.7cm}|}{Accuracy} & 90.3\% & 70.8\% & 74.4\% & 94.8\% \\
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
|
|
\begin{table}[t]
|
|
\centering
|
|
\caption{Confusion matrix of the validation test set.}
|
|
\label{Table6:ConfusionTest}
|
|
\TABLESVSPACE
|
|
\renewcommand{\arraystretch}{1.2}
|
|
\begin{tabular}{x{.5cm}x{1cm}|x{1.1cm}|x{1.1cm}|x{1.1cm}|x{1.1cm}|}
|
|
& & \multicolumn{4}{c}{Optimal} \\
|
|
\multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}}
|
|
& & S1 & S2 & S3 & S4 \\ \hline
|
|
& S1 & 56 & 7 & 2 & 0 \\
|
|
\cline{2-6} & S2 & 10 & 28 & 6 & 2 \\
|
|
\cline{2-6} & S3 & 1 & 5 & 31 & 8 \\
|
|
\cline{2-6} & S4 & 0 & 5 & 7 & 86 \\
|
|
\hline
|
|
\multicolumn{2}{x{1.7cm}|}{Accuracy} & 83.6\% & 62.2\% & 67.4\% & 89.6\% \\
|
|
%\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
|
|
\subsection{Quality evaluation}
|
|
|
|
|
|
\begin{table}[t]
|
|
\centering
|
|
\caption{Average quality gains ($\Delta$PSNR) in comparison with the reference \ac{hevc} across different bitrates and loss ratios.}
|
|
\label{Table6:AverageGains}
|
|
\TABLESVSPACE
|
|
\newcommand{\colwidth}{2cm}
|
|
\begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}}
|
|
\toprule
|
|
\multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\
|
|
& S1 & S2 & S3 & S4 & Optimal (\textit{Proposed}) \\ \hline
|
|
Basketball Drill & 1.61 & 2.66 & 2.48 & 2.44 & 2.75 \\ \hline
|
|
Basketball Drive & 2.17 & 3.46 & 3.22 & 3.63 & 3.69 \\ \hline
|
|
Book Arrival & 0.96 & 1.66 & 1.43 & 1.59 & 1.74 \\ \hline
|
|
BQSquare & 0.92 & 0.96 & 0.80 & 0.62 & 1.09 \\ \hline
|
|
Cactus & 1.28 & 2.04 & 1.89 & 1.96 & 2.09 \\ \hline
|
|
Four People & 0.28 & -0.23 & -0.88 & -0.99 & 0.25 \\ \hline
|
|
Kendo & 0.81 & 0.78 & 0.20 & 0.61 & 1.02 \\ \hline
|
|
Kimono & 1.65 & 3.04 & 3.26 & 3.27 & 3.41 \\ \hline
|
|
Kristen Sara & 0.36 & 0.15 & -0.33 & -0.50 & 0.38 \\ \hline
|
|
Park Scene & 1.64 & 2.18 & 1.88 & 2.02 & 2.25 \\ \hline
|
|
People on Street & 1.00 & 1.27 & 0.91 & 1.49 & 1.63 \\ \hline
|
|
Race Horses & 1.93 & 3.89 & 4.06 & 4.30 & 4.39 \\ \hline
|
|
Tennis & 1.15 & 2.24 & 2.17 & 3.12 & 3.18 \\ \hline
|
|
\textbf{Average} & \textbf{1.21} & \textbf{1.86} & \textbf{1.62} & \textbf{1.81} & \textbf{2.15} \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%% SECTION %%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\section{Summary}
|
|
\label{Section6:Summary}
|