Finish thesis review
This commit is contained in:
@@ -14,7 +14,7 @@ The effectiveness of different error resilience techniques is rather variable de
|
||||
Therefore, it is not straightforward to select a single technique capable of guaranteeing the highest efficiency for every possible scenario and also for the whole range of networking conditions.
|
||||
This chapter addresses this problem by proposing an adaptive error robustness scheme that optimises the efficiency of the techniques previously described in Chapters~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}.
|
||||
At the encoder-side, a pre-trained neural network predicts the video stream that is able to achieve the highest decoded quality for particular loss cases, among various video streams encoded with different levels of compression efficiency and error robustness.
|
||||
The stream selection is based on different input parameters that in\textsl{}clude information from the video signal, the coded stream and the transmission network.
|
||||
The stream selection is based on different input parameters that include information from the video signal, the coded stream and the transmission network.
|
||||
Using this approach the overall performance of the proposed adaptive method significantly increases, achieving a better trade-off between compression efficiency and error robustness.
|
||||
|
||||
This chapter is organised as follows.
|
||||
@@ -70,19 +70,18 @@ The remaining configuration was kept as described in Section~\ref{Section3:Error
|
||||
|
||||
\subsubsection{Preliminary simulation results and discussion}
|
||||
|
||||
\begin{figure}[!t]
|
||||
\centering
|
||||
\input{043_DynamicErrorRobustness/Results/Quality_PLR_Methods.tex}
|
||||
\vspace{-4mm}
|
||||
\caption{Average decoded video quality for the bitstreams encoded with the main bitrate (bps).}
|
||||
\label{Figure6:AverageQualityMain}
|
||||
\end{figure}
|
||||
|
||||
Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} show the average decoded video quality (\ac{psnr}) obtained for different \acs{plr}.
|
||||
Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate listed in Table~\ref{Table6:TestMaterialCharacteristics}, referred to as the main bitrate.
|
||||
Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained by reducing the main bitrate by 40\%. This is used to evaluate the impact of the network bandwidth on the decoded video quality. The results shown in the figures correspond to the same test sequences, in order to allow an accurate and fair comparison.
|
||||
|
||||
\begin{figure}[!t]
|
||||
\centering
|
||||
\input{043_DynamicErrorRobustness/Results/Quality_PLR_Methods.tex}
|
||||
\vspace{-4mm}
|
||||
\caption{Average decoded video quality for the bitstreams encoded with the main bitrate (bps).}
|
||||
\label{Figure6:AverageQualityMain}
|
||||
\end{figure}
|
||||
|
||||
|
||||
\begin{figure}[!t]
|
||||
\centering
|
||||
@@ -123,7 +122,7 @@ Figure~\ref{Figure6:BlockDiagram} illustrates the proposed architecture, whose m
|
||||
|
||||
|
||||
Firstly, the input signals are coded using different levels of error protection. Then, an optimisation algorithm is used to decide which bitstream should be transmitted at any given instant, based on different parameters. The selected stream is expected to obtain the highest quality at the decoder-side for the given conditions.
|
||||
A reference video coder is used to compress the video signal using highly efficient coding techniques, obtaining the best \ac{rd} performance for error-free environments.
|
||||
A reference video coder is used to compress the video signal using highly efficient coding techniques, obtaining the best \ac{rd} performance for packet loss free environments.
|
||||
This is used as the reference bitstream.
|
||||
Subsequently, a transcoding operation is applied to this reference bitstream aided by the input video signal to efficiently select the reference frames used for inter prediction.
|
||||
The reference frame selection mechanism proposed in Chapter~\ref{Chapter:ErrorResilienceTechniques} is used for this purpose.
|
||||
@@ -136,8 +135,10 @@ The proposed approach results in four different bitstreams with different charac
|
||||
(iii) \SMEXPELTEN: robust video coding using selective reference frames and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue;
|
||||
(iv) \SMEXPELTWEN: same as the previous bitstream but with 20\% bitrate allocated for the scalable~\ac{el}.
|
||||
|
||||
The video stream with the optimal error robustness is selected using an \ac{nn}, which receives different parameters extracted from the video signal, the compressed bitstream and information from the network conditions.
|
||||
From the input video, relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, further information is also extracted from the coded bitstream using a stream parser.
|
||||
The video stream with the optimal error robustness is selected using an \ac{nn}, which receives different parameters extracted from the input video, the compressed bitstream and information from the network conditions.
|
||||
The relevant video characteristics that are useful for this algorithm are known measures of spatial and temporal complexity~\cite{ITUTP9102008}, defined in Section~\ref{Section6:InputParameters}.
|
||||
These are computed by the functional block defined as "Video analyser" in Figure~\ref{Figure6:BlockDiagram}.
|
||||
Moreover, further information is also extracted from the coded bitstream using a stream parser.
|
||||
Finally, the packet loss ratio of the transmission channel is also used.
|
||||
These parameters are fed into the optimisation block to select the best bitstream for transmission.
|
||||
The optimisation process is described in detail in the following section.
|
||||
@@ -177,6 +178,7 @@ Then, the following \acs{plr} were used to cover a wide range of loss ratios gi
|
||||
This results in a total of \NumCases test conditions.
|
||||
|
||||
\subsection{Input parameters}
|
||||
\label{Section6:InputParameters}
|
||||
|
||||
A set of different parameters is used to predict the optimal stream that achieves the highest quality in specific network conditions.
|
||||
Based on these inputs the proposed scheme should be able to select the appropriate robust coding stream that should be transmitted.
|
||||
@@ -233,7 +235,7 @@ Transmission networks with higher loss ratios require a higher bit allocation fo
|
||||
These parameters were compared against the optimal stream, selected from \SMEC to \SMEXPELTWEN.
|
||||
This aims at finding the correlation between the input parameters and the output variable, i.e., target stream.
|
||||
In this study the streams are characterised by a number between 1 and 4.
|
||||
Table~\ref{Table6:Correlation} shows the absolute Person Correlation coefficient obtained across all sequences, bitrates and \acs{plr}.
|
||||
Table~\ref{Table6:Correlation} shows the absolute Person Correlation coefficient \cite{PersonCorrelation_2015} obtained across all sequences, bitrates and \acs{plr}.
|
||||
The table shows the results for the parameters previously described, as well as, a new parameter obtained from the average magnitude of the optical flow measured using the method in~\cite{DeepFlow2013} (\textit{OPT\_FLOW}).
|
||||
Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks having the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable.
|
||||
Since both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method, due to its higher Person Correlation coefficient.
|
||||
@@ -244,7 +246,8 @@ Since both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar informatio
|
||||
A \ac{nn} is a computational model based on the structure and functions of biological neural networks.
|
||||
Mainly, a \ac{nn} consists of three different layers: the input layer which receives data from an external environment, the hidden layer which processes the inputs and an output layer~\cite{NeuralNetworks1994}.
|
||||
In the past \acs{nn} have been widely used in video processing applications such as, quality evaluation~\cite{VideoQualityNN_TNN2006}, video analysis~\cite{VideoAnalysisNN_2018} and video coding~\cite{RateDistortionNN_VCIP2017,HEVCComplexityNN_TIP2018}.
|
||||
In the proposed method a feed-forward network is used, which connects every neuron in a given layer to the neurons of the subsequent layer.
|
||||
Neural networks are used in the proposed method since they are universal approximators, which can fit arbitrary complex data that could not be approximated using analytical models~\cite{NeuralNetworks_2016}.
|
||||
In this work a feed-forward network is used, which connects every neuron in a given layer to the neurons of the subsequent layer.
|
||||
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
@@ -261,10 +264,10 @@ Subsequently, the network transforms the input data through a series of neurons
|
||||
The output layer comprises four neurons corresponding to each available video stream from \SMEC to \SMEXPELTWEN (different error resilience techniques).
|
||||
Finally, the output with the highest value is selected and the corresponding stream is used for transmission, since it is the one expected to achieve the highest video quality for a particular scenario.
|
||||
|
||||
In the proposed approach a shallow \ac{nn} using three layer, with 32 nodes in the hidden layer is used for simplicity and to avoid over-fitting, due to the limited test cases~\cite{NeuralNetworkDesign_2014}.
|
||||
In the proposed approach a shallow \ac{nn} is used for simplicity and to avoid over-fitting, due to the limited test cases~\cite{NeuralNetworkDesign_2014}. As shown in Figure~\ref{FIG6:NeuralNetwork}, the \ac{nn} using in this method has three layer: input layer with 6 nodes, an hidden layer with 32 nodes and an output layer with 4 nodes.
|
||||
By avoiding over-fitting, it is guaranteed that the proposed method would continue to work in future test conditions, not used for the network training.
|
||||
The number of nodes in the hidden layer was determined by comparing the performance obtained for different number of neurons in this layer using the cross-entropy performance metric~\cite{CrossEntropy_2005}.
|
||||
However, different network topologies with different number of neurons in each layer could also be used to predict between different error resilience techniques, as in the proposed method.
|
||||
The number of nodes in the hidden layer was determined by comparing the performance obtained for different cases using the cross-entropy performance metric~\cite{CrossEntropy_2005}.
|
||||
%However, different network topologies with different number of neurons in each layer could also be used to predict between different error resilience techniques, as in the proposed method.
|
||||
|
||||
\subsubsection{Training and validation}
|
||||
|
||||
@@ -379,7 +382,7 @@ It is only used to establish the optimal case for comparison purposes.
|
||||
The results in the figure allow the comparison between three relevant cases: proposed stream selection method, the optimal case and the reference method.
|
||||
|
||||
As shown in Figure~\ref{Figure6:ProposedQuality}, the proposed method clearly outperforms the reference \ac{hevc} for all \acs{plr}.
|
||||
Moreover, in the error-free case (\ac{plr}=0), it leads to imperceptible quality reduction, since the proposed approach is able to select the optimal stream, thus disabling any robust coding technique when the transmission network does not have packet losses.
|
||||
Moreover, in the packet loss free case (\ac{plr}=0), it leads to imperceptible quality reduction, since the proposed approach is able to select the optimal stream, thus disabling any robust coding technique when the transmission network does not have packet losses.
|
||||
In case of data loss, the results show that the proposed method is able to match the quality obtained by the \BEST case in most cases, revealing that the \ac{nn} is able to efficiently select the optimal stream for transmission.
|
||||
|
||||
\begin{table}[tb]
|
||||
@@ -436,29 +439,30 @@ These results also confirm that the proposed method is able to select the optima
|
||||
The proposed method achieves a quality gain up to 4.46 dB in comparison with the reference \ac{hevc}, outperforming the quality achieved with the stream \SMEXPELTWEN (i.e., 4.19 dB).
|
||||
This confirms the advantage of the proposed method to adaptively select among different streams with different levels of error protections.
|
||||
|
||||
\begin{table}[tb]
|
||||
\centering
|
||||
\caption{Average quality gain ($\Delta$PSNR in dB) in comparison with the reference \ac{hevc} for video sequences not used in the NN training.}
|
||||
\label{Table6:AverageGainBasketBallDrive}
|
||||
\TABLESVSPACE
|
||||
\newcommand{\colwidth}{1.7cm}
|
||||
\begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}}
|
||||
\toprule
|
||||
\multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\
|
||||
& \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN & \PROP \\ \hline
|
||||
Basketball Drive & +2.10 & +2.68 & +3.36 & +3.45 & +3.55 \\ \hline
|
||||
Kendo & +1.92 & +3.24 & +3.94 & +4.19 & +4.46 \\ \hline
|
||||
Race Horses & +0.85 & +1.02 & +0.88 & +0.75 & +1.06 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
%% SECTION %%
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
\section{Summary}
|
||||
\label{Section6:Summary}
|
||||
|
||||
\begin{table}[tb]
|
||||
\centering
|
||||
\caption{Average quality gain ($\Delta$PSNR in dB) in comparison with the reference \ac{hevc} for video sequences not used in the NN training.}
|
||||
\label{Table6:AverageGainBasketBallDrive}
|
||||
\TABLESVSPACE
|
||||
\newcommand{\colwidth}{1.7cm}
|
||||
\begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}}
|
||||
\toprule
|
||||
\multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\
|
||||
& \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN & \PROP \\ \hline
|
||||
Basketball Drive & +2.10 & +2.68 & +3.36 & +3.45 & +3.55 \\ \hline
|
||||
Kendo & +1.92 & +3.24 & +3.94 & +4.19 & +4.46 \\ \hline
|
||||
Race Horses & +0.85 & +1.02 & +0.88 & +0.75 & +1.06 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
In this chapter an adaptive error robustness method is proposed, based on reference frame selection and \ac{ec}-aware optimisation. By adaptively selecting a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the individual reference techniques.
|
||||
%Moreover, the proposed method is able to outperform those cases where a single error resilience technique is used.
|
||||
The overall results demonstrate that the proposed method is able to increase the reconstruction video quality in case of errors.
|
||||
|
||||
@@ -85,7 +85,7 @@
|
||||
\path[MainArrow] (Scalable0) -- (Mux.south |- Scalable0) node[OutputLabels] {\SMEXPELTWEN};
|
||||
\path[MainArrow] (Mux) -- (Out) node[above, midway, text width=2cm, text centered] {Optimal \\ Robust Steam};
|
||||
|
||||
\node [text centered, text width=1.5cm, minimum height=1cm, below=.3 of Mux.east, xshift=1.9cm] (Channel) {Channel Conditions};
|
||||
\node [text centered, text width=1.5cm, minimum height=1cm, below=.3 of Mux.east, xshift=1.9cm] (Channel) {Network Conditions};
|
||||
|
||||
|
||||
\path[SignallArrow] (StreamParser) -| (Mux.-6) node (b1) [pos=.3, anchor=center] {\footnotesize /} node[right=-.2 of b1, yshift=-6, inner sep=0] {\footnotesize 3};
|
||||
|
||||
Reference in New Issue
Block a user