Review all chapters and start conclusions
This commit is contained in:
+12
-1
@@ -1,7 +1,18 @@
|
||||
\dedication{
|
||||
Anyone who has never\\ made a mistake has never\\ tried anything new.
|
||||
|
||||
\vspace{5cm}
|
||||
Dedicated to my future wife, \textbf{Bianca}
|
||||
and my parents, \textbf{Antonio} and \textbf{Maria}
|
||||
|
||||
\vspace{15cm}
|
||||
|
||||
Anyone who has never\\ made a mistake has never\\ tried anything new.\\
|
||||
\vspace{0.5cm}
|
||||
\textbf{(Albert Einstein)}
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
}
|
||||
|
||||
|
||||
@@ -1,19 +1,19 @@
|
||||
\chapter*{\HIGHLIGHT{\glsknowledgmentsname}}
|
||||
\chapter*{\glsknowledgmentsname}
|
||||
\addcontentsline{toc}{chapter}{\glsknowledgmentsname}
|
||||
|
||||
I would like to thank to everyone that help during this research work, and that make it possible to accomplish.
|
||||
|
||||
I would like to express my gratitude to my advisers Prof. Pedro António Amado Assunção, Prof. Sérgio Manuel Maciel de Faria, Doctor Erhan Ekmekcioglu and Prof. Ahmet Kondoz for their support, guidance and availability that were essential for the proper conduct of this research work.
|
||||
Firstly, I would like to express my gratitude to my advisors, Dr. Pedro António Amado Assunção, Dr. Sérgio Manuel Maciel de Faria, Dr Erhan Ekmekcioglu and Dr. Ahmet Kondoz for the tireless support of my PhD studies and related research, for their motivation, patience, and immense knowledge. I am thankful for their guidance and availability that were essential for the proper
|
||||
conduct of this research work. I would also like to acknowledge Dr. Varuna de Silva for the initial support on this research~work.
|
||||
|
||||
I would like to thank Funda\c{c}\~{a}o para a Ci\^{e}ncia e a Tecnologia (FCT), which provided
|
||||
the financial support for this PhD, under the PhD grant SFRH/BD/86368/2012.
|
||||
I am also grateful to the Instituto de Telecomunicações and to the Institute for Digital Technologies, which provided the computer resources and laboratory facilities with excellent conditions for my research work.
|
||||
|
||||
%I would like to acknowledge my research colleagues ... \HERE
|
||||
%of Instituto de Telecomunicações~-~Leiria, Sylvain Marcelino, Lino Ferreira, Nelson Francisco, Luís Pinto, Luís Lucas, Anderson and Auridélia Moura, for the friendship and the wonderful work environment. I also want to express my gratitude for my graduation and non-graduation friends, for the moments outside the working environment, friendship and support, which were important along this research work.
|
||||
I would like to acknowledge my research colleagues of Instituto de Telecomunicações~-~Leiria, in special Lucas, João, Ricardo, Nuno and Rodolfo for the friendship and the wonderful work environment, for sharing new ideas, and also for the moments outside the laboratory.
|
||||
I also thank all my friends, that accompanied me during this journey, for the distracting and social moments.
|
||||
|
||||
I would like to thank to all communities around the world that contribute for the development of many free and open-source tools used in this research work, namely the GNU/Linux, KDE software, Ubuntu and Gentoo communities. I also acknowledge the partnership of Luis Lucas in the Calyp software, and his contribution to the Remoterun scripts and the general computer management, which improved our developer capabilities and ease the experimental evaluations.
|
||||
I must thank all communities around the world that contribute for the development of many free and open-source tools used in this research work, namely the GNU/Linux, KDE software, Ubuntu and Gentoo communities. I also acknowledge the partnership of Luis Lucas in the Calyp software, and his contribution to the Remoterun scripts and the general computer management, which improved our developer capabilities and ease the experimental evaluations.
|
||||
|
||||
Finally, a special thanks to my parents, António and Maria, and my sister Marta to whom I owe all that I have become. I also would like to express a special gratitude to Bianca Pires for her support and love during this journey.
|
||||
A special thanks to my parents, António and Maria, and my sister Marta to whom I owe all that I have become. Also, I must a acknowledge my uncles and cousins living in the United Kingdom for the kind welcome into their home, which significantly ease my research experience abroad.
|
||||
Last but not least, I also would like to express a special gratitude to Bianca Pires for her unconditional support and love during this journey.
|
||||
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
|
||||
Over the recent years new advances in video compression and networking technologies have enabled the development of many multimedia applications over the internet, such as video conferencing, video streaming, video publishing, among others.
|
||||
The increasing diversity of services, the demands for high quality multimedia and the growing popularity of \ac{uhd} formats (\emph{e.g.}, $4k$ or $8k$ resolution and beyond) are pushing technologies forward and opening new research directions.
|
||||
Moreover, the emerging new video formats with increased resolutions have created stronger needs for higher coding efficiency, beyond the previous standards, i.e., the MPEG-4/\ac{h264}~\cite{H264AVCStandard2003,H264OverviewJul2003}.
|
||||
Moreover, the emerging new video formats with increased resolutions, such as, 360-degree video, have created stronger needs for higher coding efficiency, beyond the previous standards, i.e., the MPEG-4/\ac{h264}~\cite{H264AVCStandard2003,H264OverviewJul2003}.
|
||||
Such need has been driven not only by higher resolutions but also by stereoscopic and multi-view video formats, which increase the number of viewpoints that need to be encoded and delivered simultaneously.
|
||||
Moreover, the ever increasing traffic generated by mobile multimedia applications \cite{CiscoMobile1621_Feb2017} and different video services with ever increasing popularity, e.g., Netflix, impose new challenges to the existing networks.
|
||||
To cope with such demanding requirements a new coding standard was developed by the ITU-T \ac{vceg} and the \ac{iso}/\ac{iec} \ac{mpeg} standardisation organisations, referred to as H.265 - \ac{hevc}~\cite{H265StandardFeb2018,HEVCOverviewDez2012}.
|
||||
@@ -67,19 +67,15 @@ Although different \ac{ec} methods have been developed during the past years for
|
||||
|
||||
\section{Main objectives and contributions}
|
||||
|
||||
|
||||
|
||||
|
||||
The research objectives of this thesis are focused in robust error resilience for efficient error concealment in highly efficiency video decoders.
|
||||
|
||||
Thus, this work investigates novel techniques to increase the video quality of \ac{hevc} bitstreams delivered through error-prone networks.
|
||||
The following objectives were pursued in this work:
|
||||
\begin{enumerate}
|
||||
\item
|
||||
\textbf{evaluation and study of the error robustness:} The first objective of this research work is to evaluate the robustness against transmission errors of highly efficiently coded video. The study conclusions aimed at finding relevant research directions to increase the error robust of coded streams. The results and conclusions of this study were published in conference paper (\ref{Conf:ErrorResilience}) and in journal paper (\ref{Journal:TwoStage}) (see Appendix \ref{Appen:Papers}).
|
||||
\textbf{evaluation and study of the error robustness:} The first objective of this research work is to evaluate the robustness against transmission errors of highly efficiently coded video. The study conclusions aimed at finding relevant research directions to increase the error robust of coded streams. The results and conclusions of this study were published in conference paper (\ref{Conf:ErrorResilience}) and in journal paper (\ref{Journal:TwoStage}) (see Appendix~\ref{Appen:Papers}).
|
||||
|
||||
\item
|
||||
\textbf{robust video coding:} At the encoder-side, novel techniques to extend the existing error robustness of highly predictive video coding techniques are investigated in order to increase the received video quality over error-prone networks. This research topic resulted in a new method to reduce the error propagation in motion vector coding algorithms, which was published in conference paper (\ref{Conf:DynamicTMVP}). Moreover, a novel two-stage approach was devised to improve the error robustness of highly efficiently coded video at the encoder and streaming levels, resulting in two conference publications (\ref{Conf:MotionVectorRedundancies} and \ref{Conf:Chkb}) and a journal publication (\ref{Journal:TwoStage}).
|
||||
\textbf{robust video coding:} At the encoder-side, novel techniques to extend the existing error robustness of highly predictive video coding techniques are investigated in order to increase the received video quality over error-prone networks. This research topic resulted in a new method to reduce the error propagation in motion vector coding algorithms, which was published in conference paper (\ref{Conf:DynamicTMVP}). Moreover, a novel two-stage approach was devised to improve the error robustness of highly efficiently coded video at the encoder and streaming levels, resulting in two conference publications (\ref{Conf:MotionVectorRedundancies} and~\ref{Conf:Chkb}) and a journal publication (\ref{Journal:TwoStage}).
|
||||
|
||||
\item \textbf{robust video decoding:} error concealment techniques are used to increase the error robustness of video decoder and improve the perceived video quality.
|
||||
The paradigm of error concealment is evaluated at the encoder-side, as part of the encoding process, in order to optimise the reconstruction process at the decoder-side. This research resulted in a novel error concealment-aware technique to optimise the error concealment process at the encoder-side, which was proposed for intra-frame only in a conference paper (\ref{Conf:ConcAware}) and for a generic robust video transmission in the journal (\ref{Journal:ConcAware}).
|
||||
|
||||
@@ -53,8 +53,8 @@
|
||||
row sep=-\pgflinewidth,
|
||||
column sep=-\pgflinewidth,
|
||||
nodes={rectangle,text width=1.5cm,align=center},
|
||||
text depth=1.25ex,
|
||||
text height=.7ex,
|
||||
text depth=1ex,
|
||||
minimum height=2mm,
|
||||
nodes in empty cells
|
||||
}
|
||||
}
|
||||
@@ -66,23 +66,22 @@
|
||||
};
|
||||
|
||||
% The labels
|
||||
\node[text centered, xshift=.7cm] at (arrow-1-1) {\footnotesize SEI};
|
||||
\node[text centered, xshift=.7cm, text width=2.2cm] at (arrow-2-1.south) {\footnotesize VCL};
|
||||
\node[text centered, xshift=1cm] at (arrow-1-1) {\footnotesize SEI};
|
||||
\node[text centered, xshift=1cm, text width=2.2cm] at (arrow-2-1.south) {\footnotesize VCL};
|
||||
|
||||
|
||||
\fill[white] (last1.north) -- (last2.east) -- (last1.north east) -- cycle;
|
||||
\fill[white] (last3.south) -- (last3.south east) -- (last2.east) -- cycle;
|
||||
\fill[white] (arrow-1-1.north west) -- ($(arrow-2-1.center)+(-.4,0)$) -- (arrow-3-1.south west) -- cycle;
|
||||
|
||||
\coordinate (div0) at (-.8,-1.3);
|
||||
\coordinate (div1) at ($(div0)+(-20:.2cm)+(-20:.2cm)+(60:3.1cm)$);
|
||||
\coordinate (div0) at (-.8,-1.1);
|
||||
\coordinate (div1) at ($(div0)+(-20:.2cm)+(-20:.2cm)+(60:2.5cm)$);
|
||||
|
||||
\fill[fill=white] ($(div0)+(-20:-.2cm)$) -- ++(60:1.8cm) -- ++(-30:.4cm) -- ($(div1)+(-20:-.2cm)$) -- ++(-30:.4cm) -- ++(60:-1.85cm) -- ++(-30:-.4cm) -- ($(div0)+(-20:.2cm)$);
|
||||
|
||||
%\draw ( $(div0)$ ) -- ( $(div0)+(60:1.7cm)$ );
|
||||
\path[decoration={markings, mark=at position .5 with \node[transform shape] {\scriptsize Error-prone};}, postaction={decorate}]
|
||||
( $(div0)$ ) -- ( $(div0)+(60:1.7cm)$ );
|
||||
|
||||
%\draw ($(div1)+(60:-1.8cm)$) -- ($(div1)$);
|
||||
\path[decoration={markings, mark=at position .5 with \node[transform shape] {\scriptsize Transmission};}, postaction={decorate}]
|
||||
($(div1)+(60:-1.8cm)$) -- ($(div1)$);
|
||||
|
||||
@@ -100,7 +99,7 @@
|
||||
|
||||
|
||||
%% Encoding
|
||||
\node[fill=TikZColorFillProp, coders, minimum height=3.1cm, text width=2.6cm, left=-.5 of arrow] (SideInformation) {};
|
||||
\node[fill=TikZColorFillProp, coders, minimum height=3.1cm, text width=2.6cm, left=0 of arrow] (SideInformation) {};
|
||||
\node[fill=TikZColorFillProp, coders, minimum height=2.1cm, text width=2.2cm, yshift=-7] (RobustEncoding) at (SideInformation) {};
|
||||
\node[fill=TikZColorFill, coders, minimum height=1.1cm, text width=1.5cm, yshift=-13] (Encoding) at (SideInformation) {Encoding};
|
||||
|
||||
@@ -111,7 +110,7 @@
|
||||
($(SideInformation.west)+(.3,.3)$) to [bend left=50] ($(SideInformation.east)+(-.3,.3)$);
|
||||
|
||||
%% Decoding
|
||||
\node[fill=TikZColorFillProp, coders, minimum height=2.1cm, text width=2.2cm, right=-.2 of arrow] (RobustDecoding) {};
|
||||
\node[fill=TikZColorFillProp, coders, minimum height=2.1cm, text width=2.2cm, right=.1 of arrow] (RobustDecoding) {};
|
||||
\node[fill=TikZColorFill, coders, minimum height=1.1cm, text width=1.5cm, yshift=-7] (Decoding) at (RobustDecoding) {Decoding};
|
||||
|
||||
\path[postaction={decorate,decoration={text along path,text align=center,text={|\small|Robust Decoding}}}]
|
||||
|
||||
@@ -27,7 +27,7 @@ This chapter is organised as follows.
|
||||
Section~\ref{Section2:GeneraVideoSystem} introduces the functional blocks of a generic hybrid video coding system and Section~\ref{Section2:Overview} provides an overview of the \ac{hevc} standard.
|
||||
Then, Section~\ref{Section2:HighLevel} describes the high-level syntax elements, which are used to carry relevant information in coded streams.
|
||||
Section~\ref{Section2:CodingStructures} describes the data structures used to divide the video frames into smaller partitions and Section~\ref{Section2:PredictionModes} describes particular aspects of the main predictions techniques, namely intra and inter coding.
|
||||
Finally, a brief description of the transform and quantisation, entropy coding and in-loop filters is provided in Sections~\ref{Section2:TransformQuant}, \ref{Section2:Entropy} and \ref{Section2:Filters}, respectively.
|
||||
Finally, a brief description of the transform and quantisation, entropy coding and in-loop filters is provided in Sections~\ref{Section2:TransformQuant},~\ref{Section2:Entropy} and~\ref{Section2:Filters}, respectively.
|
||||
Section~\ref{Section2:Summary} concludes the chapter.
|
||||
|
||||
\clearpage
|
||||
@@ -100,7 +100,7 @@ Similar to the previous coding standards developed by JCT-VC, in \ac{hevc}~\cite
|
||||
coding syntax and semantics are standardised.
|
||||
The constrains and functionalities are defined through profiles and levels, while the semantic meaning of syntax elements define the decoding process, such that every decoder conforming to the standard generates the same output when given a complaint stream.
|
||||
Such delimitation of scope in the standard allows maximum freedom for developers and manufactures to develop proprietary optimisations to cope with various applications’ requirements (\emph{e.g.}, balancing the compression ratio and the coding complexity).
|
||||
Figure \ref{fig:HEVCBlockDiagram} illustrates the block diagram of the \ac{hevc} encoder. It follows the same overall hybrid architecture as its predecessors starting from H.261.
|
||||
Figure~\ref{fig:HEVCBlockDiagram} illustrates the block diagram of the \ac{hevc} encoder. It follows the same overall hybrid architecture as its predecessors starting from H.261.
|
||||
The encoder uses both intra- and inter-picture prediction to exploit spatial and temporal redundancies, respectively. Subsequently, the prediction residue transformed by linear spatial transform, which is then quantised, and entropy coded, allowing to further reduce the bitstream redundancy.
|
||||
%The resulting bitstream is combined with control information, which signals the prediction chosen, motion vectors, filtering operations and bitstream headers.
|
||||
|
||||
@@ -112,7 +112,7 @@ The encoder uses both intra- and inter-picture prediction to exploit spatial and
|
||||
\label{fig:HEVCBlockDiagram}
|
||||
\end{figure}
|
||||
|
||||
However, there are some novelties introduced in the \ac{hevc} standard. As depicted in Figure \ref{fig:HEVCBlockDiagram}, the encoding process follows a block-based approach, therefore each picture is partitioned into larger blocks, referred to as \acs{ctu}, with a maximum size of $64\times64$ pixels.
|
||||
However, there are some novelties introduced in the \ac{hevc} standard. As depicted in Figure~\ref{fig:HEVCBlockDiagram}, the encoding process follows a block-based approach, therefore each picture is partitioned into larger blocks, referred to as \acs{ctu}, with a maximum size of $64\times64$ pixels.
|
||||
Subsequently, a dynamic quad-tree partitioning is used to divide the picture into \acs{cu}. These units are introduced as the basic processing units for prediction, and for the transform and quantisation operations, \acs{tu} are used.
|
||||
In order to generate the prediction residue the \ac{hevc} may use one of 34 intra-prediction modes or take advantage of symmetric and asymmetric partitions for motion estimation and inter-prediction.
|
||||
The motion estimation is performed in the pictures listed in the \ac{rpset}, which was introduced in the \ac{hevc} to increase the robustness to errors and data loss of reference picture management.
|
||||
@@ -386,7 +386,7 @@ The decision whether to use intra-frame or inter-frame prediction modes is taken
|
||||
|
||||
|
||||
|
||||
%The main key feature of \ac{hevc} is the new block partition structure, which achieves more flexibility and efficiency. Four different block concepts have been introduced in the standard: \ac{ctu}, \ac{cu}, \ac{pu}, and \ac{tu}, all of which have different roles. Figure \ref{fig:HEVCBlockStructure} illustrates an example of a CTU partitioning in HEVC, and it is compared to the \ac{h264} \ac{mb} structure. CTUs are defined as the high-level partition within each slice and can be as large as 64$\times$64, which can be recursively partitioned into four smaller blocks referred to as CUs. The CU is the basic units for the intra/inter prediction and can be as small as 8$\times$8. Subsequently, the CU are divided into up to four PUs which are used for inter prediction. For intra coding PU is not used. Finally, the CU is divided into a transform unit tree, which may include several TUs, in a quad-tree structure. Transform units are the basic units for the transform operations applied to the coding residue.
|
||||
%The main key feature of \ac{hevc} is the new block partition structure, which achieves more flexibility and efficiency. Four different block concepts have been introduced in the standard: \ac{ctu}, \ac{cu}, \ac{pu}, and \ac{tu}, all of which have different roles. Figure~\ref{fig:HEVCBlockStructure} illustrates an example of a CTU partitioning in HEVC, and it is compared to the \ac{h264} \ac{mb} structure. CTUs are defined as the high-level partition within each slice and can be as large as 64$\times$64, which can be recursively partitioned into four smaller blocks referred to as CUs. The CU is the basic units for the intra/inter prediction and can be as small as 8$\times$8. Subsequently, the CU are divided into up to four PUs which are used for inter prediction. For intra coding PU is not used. Finally, the CU is divided into a transform unit tree, which may include several TUs, in a quad-tree structure. Transform units are the basic units for the transform operations applied to the coding residue.
|
||||
%
|
||||
%Each CU may be split into two or four Pus resulting in either symmetric or asymmetric partitions. Asymmetric partitioning of the CU into two PUs enables more efficient motion estimation since it allows more flexible partitioning of the picture~\cite{BlockPartHEVC2012}. This scheme is able to improve the coding performance by up to $1\%$ when compared with rectangular symmetric partitioning.
|
||||
|
||||
@@ -590,7 +590,7 @@ Firstly, the Merge Mode allows for more \ac{mv} candidates than the Skip in \ac{
|
||||
Thus, an index is used to identify one out of several \ac{mv} candidates.
|
||||
Secondly, the reference picture index is explicitly identified.
|
||||
This results in improved flexibility, compared with the predefined values used in \ac{h264}.
|
||||
The \ac{mv} candidates allowed in the Merge Mode\cite{MotionCodingHEVC2013} are illustrated in Figure \ref{Figure2:MVCandidates}.
|
||||
The \ac{mv} candidates allowed in the Merge Mode\cite{MotionCodingHEVC2013} are illustrated in Figure~\ref{Figure2:MVCandidates}.
|
||||
The figure shows the positions of the spatial \ac{mv} candidates ($A_0$, $A_1$, $B_0$, $B_1$, $B_2$).
|
||||
Moreover, it also uses a temporal \ac{mv} candidate, derived from the co-located position on the temporally adjacent frame, at positions $T_0$ or $T_1$. From the \ac{mv} candidate list the unavailable candidates are not considered, and duplicated candidates (i.e. with the same motion information) are removed.
|
||||
The Merge Mode uses a $C-1$ spatial \ac{mv} candidates, selected from the aforementioned order, and one \ac{tmvp} to fulfil the total number of \acs{mv} ($C$). The value of $C$ is defined at the slice header and a flag controlling the use of \ac{tmvp} is encoded at the \ac{pps}-level.
|
||||
@@ -624,7 +624,7 @@ Therefore, the use of the temporal \ac{mv} candidate is limited to those cases w
|
||||
For \ac{mv} prediction only a much lower number of candidates is allowed, since the encoder can still send a \ac{mv} difference. Moreover, the encoder performs motion estimation for each candidate, which increases the encoding complexity for every possibility tested.
|
||||
%For those cases where the reference index is not the same as the current \ac{pu} and the neighbouring one, a scaling version of the \ac{mv} is used.
|
||||
|
||||
%In the \ac{h264} standard, along different motion compensated coding techniques~\cite{H264OverviewJul2003}, the so-called P\_SKIP mode was introduced. In this mode, neither the quantised residual information, nor the \ac{mv} is transmitted. This mode is useful for large areas with no change or just constant motion, and it is only enabled for $16\times16$ blocks, \emph{i.e.}, the whole \ac{mb}. The \ac{mv} for the P\_SKIP mode is derived based on the surrounding neighbour MBs\cite{H264MotionVectorPrediction2003}. When no neighbours are available, or the left or top \acs{mv} are equal to zero, a zero \ac{mv} is used. Otherwise, the \ac{mv} is derived from the median value of the following three neighbours: left, above and above right, as shown in Figure \ref{SkipMergeMode}~(a) by $MV_A$, $MV_B$ and $MV_C$, respectively. Whenever the $MV_C$ is not available, the $MV_D$ is used instead.
|
||||
%In the \ac{h264} standard, along different motion compensated coding techniques~\cite{H264OverviewJul2003}, the so-called P\_SKIP mode was introduced. In this mode, neither the quantised residual information, nor the \ac{mv} is transmitted. This mode is useful for large areas with no change or just constant motion, and it is only enabled for $16\times16$ blocks, \emph{i.e.}, the whole \ac{mb}. The \ac{mv} for the P\_SKIP mode is derived based on the surrounding neighbour MBs\cite{H264MotionVectorPrediction2003}. When no neighbours are available, or the left or top \acs{mv} are equal to zero, a zero \ac{mv} is used. Otherwise, the \ac{mv} is derived from the median value of the following three neighbours: left, above and above right, as shown in Figure~\ref{SkipMergeMode}~(a) by $MV_A$, $MV_B$ and $MV_C$, respectively. Whenever the $MV_C$ is not available, the $MV_D$ is used instead.
|
||||
%
|
||||
|
||||
%In contrast with the \ac{h264} standard, the \ac{hevc} standard allows the usage of a temporal \ac{mv} predictor, as well as, more spatial candidates to predict the current motion vector. Despite the improvements in the flexibility and coding efficiency, the increased levels of dependencies between the \acs{mv} make the motion information more prone to errors. Especially the introduction of the temporal candidate in the set of possible \acs{mv} increases the dependencies between subsequent frames. This leads to higher quality degradation in the presence of transmission errors, when compared to \ac{h264}.
|
||||
|
||||
@@ -122,7 +122,7 @@ E\left\{\left( \tilde f_t^i \right)^2 \right\} = \left(1-p\right) \left( \left(
|
||||
\end{equation}
|
||||
assuming that the pixel $i$ is predicted from pixel $j$ in the previous frame.
|
||||
The prediction error $e_t^i$ is quantised to the value $\hat e_t^i$, which is transmitted to the decoder along with the motion information.
|
||||
Equations \ref{EQ2:E2EDIntra} to \ref{EQ2:E2EDInter} provide a measure of the distortion, taking into consideration not only the coding distortion but also the possibility of frame loss and consequent \ac{ec} distortion.
|
||||
Equations~\ref{EQ2:E2EDIntra} to~\ref{EQ2:E2EDInter} provide a measure of the distortion, taking into consideration not only the coding distortion but also the possibility of frame loss and consequent \ac{ec} distortion.
|
||||
|
||||
This technique was used in \cite{ROPEJun2000} to switch between intra and inter coding modes with the aim of reducing the temporal dependencies, thus achieving enhanced error robustness in video transmission.
|
||||
In \cite{AdvancesROPEHE2642007} this technique was optimised to quarter pixel estimation, and in \cite{ROPEBurst2009} the method was extended for burst losses, by introducing the burst loss probability into the original end-to-end estimation.
|
||||
@@ -222,7 +222,7 @@ In Equation (\ref{EQ2:LeakyPredDual}) $\hat f_{n}^i$ and $\hat f_{n-1}^i$ always
|
||||
|
||||
|
||||
In \cite{ROILeakyPred2009} a method is presented, where the leaky prediction is confined to a pre-defined \ac{roi}, in order to increase the error robustness with lower impact on the coding efficiency.
|
||||
Figure \ref{fig2:ErrorPropagationROI} shows an example of error propagation within two different regions: \ac{roi} and background.
|
||||
Figure~\ref{fig2:ErrorPropagationROI} shows an example of error propagation within two different regions: \ac{roi} and background.
|
||||
As show in this figure, since the \ac{roi} uses background regions for predictions, error propagation occurs across regions.
|
||||
To overcome this problem, the method in \cite{ROILeakyPred2009} uses a weighted prediction for the \ac{roi} region obtained from the following predictions: (i) a globally optimal prediction obtained by unrestricted motion estimation, (ii) a prediction obtained by restricting motion estimation within the \ac{roi} region.
|
||||
This limits the impact of errors in the background region on the \ac{roi}.
|
||||
@@ -432,7 +432,7 @@ These approaches increase the overall system performance by presenting an effici
|
||||
\end{figure}
|
||||
|
||||
In \ac{mdc} schemes two or more bitstreams, named descriptions, are encoded and transmitted through different channels, such that an acceptable level of quality is achieved when only some of them reach the decoder \cite{MDC1993,MDC2005}.
|
||||
Figure \ref{fig2:MDCSystem} illustrates a basic architecture of a \ac{mdc} encoder and decoder using two descriptions.
|
||||
Figure~\ref{fig2:MDCSystem} illustrates a basic architecture of a \ac{mdc} encoder and decoder using two descriptions.
|
||||
In this example, the encoder-side creates two descriptions, i.e., $R_1$ and $R_2$, which shall be transmitted using two different channels, reducing the probability of the same video data being lost at the same time in both descriptions.
|
||||
As shown in this diagram, three different outputs may be produced at the decoder-side: both descriptions are correctly received and the central decoder is used($D_0$), or either one of the descriptions is not received and then one of the side decoders is used ($D_{1,1}$ and $D_{1,2}$).
|
||||
When both bitstreams are received, the central decoder combines the information carried by each description achieving higher quality than any single side decoder.
|
||||
|
||||
@@ -346,7 +346,7 @@ Based on the analysis of the temporal dependency, some temporal candidates are m
|
||||
\FIGSVSPACE
|
||||
\end{figure}
|
||||
|
||||
Figure \ref{ProposedDynamicMethod} can be used to explain the proposed method, illustrating different blocks and different \ac{mv} dependencies represented with arrows. Note that the squares illustrate a generic block size; the representation of the temporal dependencies is simplified, and in some cases may not correspond to the exact co-located block in the previous frame. In frame $f_{t-1}$ there are several \acs{mv} encoded using spatial predictions and one \ac{mv} encoded using a temporal prediction (arrow pointing to $f_{t-2}$). In frame $f_{t}$, the \ac{mv} corresponding to the block (1) may use the temporal \ac{mv} candidate, since the co-located block in $f_{t-1}$ is not temporally dependent.
|
||||
Figure~\ref{ProposedDynamicMethod} can be used to explain the proposed method, illustrating different blocks and different \ac{mv} dependencies represented with arrows. Note that the squares illustrate a generic block size; the representation of the temporal dependencies is simplified, and in some cases may not correspond to the exact co-located block in the previous frame. In frame $f_{t-1}$ there are several \acs{mv} encoded using spatial predictions and one \ac{mv} encoded using a temporal prediction (arrow pointing to $f_{t-2}$). In frame $f_{t}$, the \ac{mv} corresponding to the block (1) may use the temporal \ac{mv} candidate, since the co-located block in $f_{t-1}$ is not temporally dependent.
|
||||
However, in the proposed scheme the \ac{mv} prediction marked with the arrow (2) is not allowed, thus preventing the propagation of temporal dependencies.
|
||||
Moreover, the block corresponding to the arrow (3) cannot use the temporal \ac{mv} candidate since it already depends on a \ac{mv} that was previously encoded using the temporal candidate. The proposed method reduces the temporal dependencies of the \glspl{mv} at the block level, providing improved error robustness to the encoded streams. Using this approach, the dependencies are selectively removed, improving the error recovery, without fully disabling the temporal \gls{mv} candidates for a given frame. Therefore, coding efficiency is not significantly penalised.
|
||||
|
||||
@@ -363,12 +363,12 @@ Moreover, the block corresponding to the arrow (3) cannot use the temporal \ac{m
|
||||
|
||||
|
||||
The performance of the proposed scheme (\PROP) was evaluated against two other approaches: reference \ac{hevc} with TMVP enabled and the \TIDR method proposed in \cite{DynamicTMVP2011}.
|
||||
% To evaluate the proposed method, the same experimental setup as presented in Section \ref{Section3:ErrorResilienceEvaluation} was used.
|
||||
% To evaluate the proposed method, the same experimental setup as presented in Section~\ref{Section3:ErrorResilienceEvaluation} was used.
|
||||
|
||||
\subsubsection{Error propagation under single loss events}
|
||||
|
||||
Firstly, the performance of the proposed method was measured under a single loss event (i.e., only one frame is affected by errors).
|
||||
Figure \ref{Figure3:ProposedMethodEvaluationSingle} shows the error propagation for the different methods.
|
||||
Figure~\ref{Figure3:ProposedMethodEvaluationSingle} shows the error propagation for the different methods.
|
||||
The results show a significant quality difference between the proposed method, the \TIDR method and the reference \ac{hevc}.
|
||||
When the temporal \ac{mv} candidate is disabled (proposed and \TIDR method) a higher error robustness is achieved.
|
||||
The results also show the relevance of using the proposed method, as the \TIDR method is only able to recover from the frame loss after the refreshing frame, \emph{i.e.}, frame \#8, especially for the BQSquare sequence.
|
||||
@@ -413,10 +413,11 @@ The effectiveness of the proposed method increases at higher packet loss rates (
|
||||
\section{Summary}
|
||||
\label{Section3:Summary}
|
||||
|
||||
In this chapter, the error resilience performance of \ac{hevc} was studied in detail under error prone conditions.
|
||||
In this chapter, the error resilience performance of \ac{hevc} was studied in detail under error prone conditions. This study was partially discussed and published in the paper~(\ref{Conf:ErrorResilience}).
|
||||
The simulation results revealed that new coding tools introduced in \ac{hevc} to improve coding efficiency are partially responsible for higher error propagation.
|
||||
This motivates further research on new techniques to be developed for highly efficient encoders and decoders, in order to increase error resiliency and to improve the overall performance of video broadcast.
|
||||
Then, a new approach to deal with the weak robustness of \ac{mv} coding was proposed, addressing \ac{mv} dependencies at \ac{cu} level.
|
||||
The proposed method selectively disables the temporal \ac{mv} candidates, in order to break the dependencies across more than one frame. The experimental evaluation was able to demonstrate that the proposed method increases the error robustness and outperforms existing state-of-the-art methods and the reference codec.
|
||||
The proposed method selectively disables the temporal \ac{mv} candidates, in order to break the dependencies across more than one frame. This method was proposed in the publication (\ref{Conf:DynamicTMVP}).
|
||||
The experimental evaluation was able to demonstrate that the proposed method increases the error robustness and outperforms existing state-of-the-art methods and the reference codec.
|
||||
The results also show that the proposed method is able to provide a good trade-off between coding efficiency and error resilience.
|
||||
Overall. these new findings demonstrate that selective usage of temporal \ac{mv} candidates is an efficient approach to enhance error resilience in high efficiency video encoders.
|
||||
Overall, these new findings demonstrate that selective usage of temporal \ac{mv} candidates is an efficient approach to enhance error resilience in high efficiency video encoders.
|
||||
|
||||
@@ -130,7 +130,7 @@
|
||||
\multirow{2}{*}{\centering\textbf{Sequence}} & \multirow{2}{*}{\vbox{\centering\textbf{TMVP}}} & \multirow{2}{*}{\vbox{\centering\textbf{Error Free}}} & \multicolumn{3}{c}{\textbf{\TABLEPLRLABEL}} \\
|
||||
& & & \textbf{1\%} & \textbf{5\%} & \textbf{10\%} \\
|
||||
\hline
|
||||
\multicolumn{6}{c}{Low-delay configuration (LD)} \\
|
||||
\multicolumn{6}{c}{Low-delay configuration (LD)} \\ [2pt]
|
||||
\hline
|
||||
\multirow{\nlines}{*}{\vbox{Basket Drill}}
|
||||
& Enabled & 38.69 & 34.73 & 27.92 & 25.41 \\
|
||||
@@ -170,7 +170,7 @@
|
||||
\multicolumn{2}{c|}{\textbf{Average difference}} & \textbf{-0.04} & \textbf{+3.04} & \textbf{+5.32} & \textbf{+5.17} \\
|
||||
\midrule
|
||||
\midrule
|
||||
\multicolumn{6}{c}{Random-access configuration (RA)} \\
|
||||
\multicolumn{6}{c}{Random-access configuration (RA)} \\ [2pt]
|
||||
\hline
|
||||
\multirow{\nlines}{*}{\vbox{Basket Drill}}
|
||||
& Enabled & 39.48 &36.84 & 30.50 & 27.15 \\
|
||||
|
||||
@@ -41,7 +41,7 @@
|
||||
{Kimono \newline $1920\times1080@24Hz$} & 23.4 / 32.5 & 83.0 & 8.00 & Capture of a women moving in a forest with moderate motion \\ \hline
|
||||
{Kristen and Sara \newline $1280\times720@60Hz$} & 25.6 / 6.30 & 93.1 & 15.00 & Two persons talking to each other with moderate motion \\ \hline
|
||||
{Park Scene \newline $1920\times1080@24Hz$} & 31.3 / 11.6 & 92.0 & 10.00 & Moderate motion with cyclists passing across the scene \\ \hline
|
||||
{Party Scene \newline $832\times480@50Hz$} & 52.8 / 11.4 & 89.8 & 4.50 & \\ \hline
|
||||
{Party Scene \newline $832\times480@50Hz$} & 52.8 / 11.4 & 89.8 & 4.50 & Moderate motion with three children playing \\ \hline
|
||||
{People on Street \newline $2560\times1600@24Hz$} & 40.0 / 25.4 & 87.4 & 12.00 & With point capture of people moving; high motion and texture complexity \\ \hline
|
||||
{Race Horses \newline $832\times480@30Hz$} & 43.7 / 24.4 & 79.7 & 8.00 & High motion with several horse riders \\ \hline
|
||||
{Tennis \newline $1920\times1080@30Hz$} & 20.3 / 45.3 & 76.2 & 5.00 & High motion with one moving person in the scene \\ \hline
|
||||
|
||||
@@ -76,7 +76,7 @@ D_t = \tilde f_t - \hat f_t, \text{when } \tilde f_{t-1} \neq \hat f_{t-1},
|
||||
\end{equation}
|
||||
where $\hat f_t$ and $\tilde f_t$ are the encoded and decoded frames, respectively, at instant $t$.
|
||||
|
||||
Figure \ref{fig:proposedMethod} illustrates the block diagram of the two-stage architecture proposed in this work.
|
||||
Figure~\ref{fig:proposedMethod} illustrates the block diagram of the two-stage architecture proposed in this work.
|
||||
At the encoding stage, a reference frame selection mechanism is used for each \ac{pu} within a \ac{ctu}, by forcing the use of different reference frames for adjacent \acs{ctu}.
|
||||
The decision criterion contributes to reduce the amount of predictions from a single reference frame and forces to interleave different reference frames along all encoded \acs{pu}, based on \ac{rd} optimisation.
|
||||
At the streaming stage, the coded stream is parsed to extract the \ac{mv} information, which allows ranking \acs{mv} according to a relevance criterion. Then, a sub-set of these \acs{mv} is selected to be transmitted as side information. The amount of selected \acs{mv} is not fixed and can be optimised for different content and network characteristics.
|
||||
@@ -108,7 +108,7 @@ In a coding structure with two reference frames, as shown in Figure~\ref{fig:che
|
||||
If each reference frame is used to predict half of the pixels in the current frame, when any of these references is lost, the subsequent predicted frame is only partially distorted by mismatched predictions.
|
||||
This is because only half of its predictions depend on the missing reference frame, which limits the overall error propagation.
|
||||
Moreover, since the motion information of the predicted frame is encoded using as reference not only the missing frame but also the past ones, those \acs{mv} that cross over a missing reference frame, pointing to the previous ones, are still useful for reconstructing correct \acs{mv} and also for error concealment.
|
||||
In the limit, a straightforward implementation of the previous example leads to a simple interleaved uniform distribution of the reference frames over all \acs{ctu} in the current frame, forcing the checkerboard structure shown in Figure \ref{fig:checkerboardRefSelection}. In the case of two reference frames, 50\% of the \acs{ctu} would use one reference frame, while the other 50\% would use the other reference.
|
||||
In the limit, a straightforward implementation of the previous example leads to a simple interleaved uniform distribution of the reference frames over all \acs{ctu} in the current frame, forcing the checkerboard structure shown in Figure~\ref{fig:checkerboardRefSelection}. In the case of two reference frames, 50\% of the \acs{ctu} would use one reference frame, while the other 50\% would use the other reference.
|
||||
However, such solution would not take into consideration coding efficiency, since the selection process of the reference frame would not be \ac{rd} optimised for each \ac{pu} within a \ac{ctu}.
|
||||
|
||||
\begin{figure}[tb]
|
||||
@@ -372,7 +372,7 @@ Taking into account the above results and the spatio-temporal information shown
|
||||
|
||||
|
||||
The influence of the $\gamma$ parameter used in Equation (\ref{eq:KFactor}) on the video quality was also evaluated, in order to check how the cost constrains affect the performance of the proposed method.
|
||||
Figure \ref{fig:QualityVsGamma} shows the average quality gains of the proposed method in comparison to the reference \ac{hevc} (\REF) across all test sequences, for different values of $\gamma$.
|
||||
Figure~\ref{fig:QualityVsGamma} shows the average quality gains of the proposed method in comparison to the reference \ac{hevc} (\REF) across all test sequences, for different values of $\gamma$.
|
||||
For the error-free case (PLR=0\%), the results in this figure show that increasing $\gamma$ leads to a decrease in the coding efficiency, especially in the \ac{ld} configuration.
|
||||
The reason is that higher values of $\gamma$ lead to higher values of $W(r_{\phi})$, therefore the reference picture selection cost is severely constrained by the exponential weights.
|
||||
Since $T_{ID}$ is constant for all frames in the \ac{ld}, higher weights are used for all frames ($T_{ID}$ is used to reduce the weights in case of hierarchical coding) and the encoder is not able to select the best reference frame in the \ac{rd} sense.
|
||||
@@ -451,7 +451,7 @@ This is due to the fact that in the \CHKB method there is no \ac{rd} optimisatio
|
||||
\subsection{Streaming stage}
|
||||
|
||||
In this section the coding performance that results from transmitting redundant \ac{mv} (streaming stage) in combination with the reference picture selection scheme (encoding stage) is presented.
|
||||
In these experimental tests the temporal \ac{mv} candidates are enabled and \acs{mv} are dynamically added as side-information into the stream, as described in Section \ref{Section4:MVSelection}.
|
||||
In these experimental tests the temporal \ac{mv} candidates are enabled and \acs{mv} are dynamically added as side-information into the stream, as described in Section~\ref{Section4:MVSelection}.
|
||||
Different values of $\alpha$ are used to achieve different overhead ratios. All streams were encoded with the same total bitrate, including the redundant bits, in order to make a fair quality comparison.
|
||||
|
||||
|
||||
@@ -464,7 +464,7 @@ Figure~\ref{fig:QualityVsRedundancyMVSelection} illustrates the relation between
|
||||
The results indicate that for approximately the same amount of redundant bits are used in both cases, however by using the proposed method to dynamically select the \acs{mv} to be used as side-information (\DYN) is able to outperform the fixed case for all test sequences.
|
||||
Results also show that for both approaches the decoded video quality increases as the redundancy of the side-information increases, since a higher number of mismatch \ac{mv} predictions are able to be recovered.
|
||||
|
||||
\begin{figure}[tb]
|
||||
\begin{figure}[!t]
|
||||
\centering
|
||||
\newcommand{\CurrFigSize}{0.49\columnwidth}
|
||||
\includegraphics[width=\CurrFigSize,page=1]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
|
||||
@@ -483,8 +483,6 @@ For instance, the proposed method is able to gain up to $1.5$~dB for the BQSquar
|
||||
For lower \acs{plr}, since fewer frames are affected by data losses, the use of a fixed approach achieves similar performance as the proposed method (\DYN).
|
||||
|
||||
|
||||
|
||||
|
||||
\subsubsection{Quality evaluation}
|
||||
|
||||
Table~\ref{tab:AverageQualityRedundancies} presents the average \ac{psnr} of decoded video for different \ac{plr} (columns~4~to~8) and for various percentages of \ac{mv} redundancy (third column). The absolute quality values are shown for the reference case (\NOREDUND) and \ac{psnr} gains are shown for three different levels of redundancy introduced by the proposed method (\DYN).
|
||||
@@ -495,6 +493,10 @@ This confirms that redundant \acs{mv} are able to minimise the error propagation
|
||||
The use of redundant \acs{mv} yields better video quality for \ac{plr} as low as $1\%$, achieving higher quality gains when \ac{idr} frames are not lost (\KEEPINTRA). This is because of the higher quality degradation when \ac{idr} frames are lost, which is not due to mismatched \ac{mv} predictions, and thus cannot be reduced by transmitting extra \acs{mv} as side-information.
|
||||
The use the proposed method also achieves higher quality for sequences with higher motion activity that makes use of inter-prediction more often, i.e., BQSquare and Traffic sequences (see Inter ratio in Table~\ref{TableTestSequences}), leading to a maximum gain of $3.93$~dB for \LOSTALL ($7.82$~dB for \KEEPINTRA).
|
||||
|
||||
In summary, the proposed method is able to select the most relevant motion \ac{mv} information and when combined with a reference picture selection scheme the problem of mismatch \ac{mv} predictions is mitigated.
|
||||
Moreover, the dynamic approach devised to select the best \acs{mv} also contributes to increase the overall performance.
|
||||
Thus, using the proposed two-stage approach to recover temporal predictions consistently improves the error robustness of \ac{hevc}.
|
||||
|
||||
\begin{table}[!t]
|
||||
\centering
|
||||
\caption{Average quality using redundant \ac{mv} selected by the proposed approach.}
|
||||
@@ -504,21 +506,15 @@ The use the proposed method also achieves higher quality for sequences with high
|
||||
\input{\ImprovedErrorResilience/Results/TableResultsRandomLossMvRedundancies.tex}
|
||||
\end{table}
|
||||
|
||||
In summary, the proposed method is able to select the most relevant motion \ac{mv} information and when combined with a reference picture selection scheme the problem of mismatch \ac{mv} predictions is mitigated.
|
||||
Moreover, the dynamic approach devised to select the best \acs{mv} also contributes to increase the overall performance.
|
||||
Thus, using the proposed two-stage approach to recover temporal predictions consistently improves the error robustness of \ac{hevc}.
|
||||
|
||||
\section{Summary}
|
||||
\label{Section4:Summary}
|
||||
|
||||
In this chapter a two-stage approach was proposed to increase the error robustness of \ac{hevc} streaming and reduce the error propagation in case of packet loss.
|
||||
A constrained coding approach was devised to select reference frames and dynamically distribute temporal dependencies.
|
||||
This is jointly used with a controlled amount of side information in coded streams, comprising a small set of the most relevant \acs{mv}, in order to minimise \ac{mv} mismatch at the decoder in the presence of frame loss.
|
||||
This is jointly used with a controlled amount of side information in coded streams, comprising a small set of the most relevant \acs{mv}, in order to minimise \ac{mv} mismatch at the decoder in the presence of frame loss. The use of \acs{mv} as side information was firstly proposed and evaluated in the publication~\ref{Conf:MotionVectorRedundancies} and the reference frame selection was further discussed in the publication~\ref{Conf:Chkb}. Subsequently, a wider analysis was carried out and published in \ref{Journal:TwoStage}.
|
||||
As can be concluded from the results, this method contributes to reduce the temporal dependencies with consistent quality gains for different \ac{plr} and coding configurations, only incurring in a small drop of coding efficiency.
|
||||
Overall, the proposed approach is an effective method to cope with video transmission over error-prone networks.
|
||||
|
||||
|
||||
\todo{Papers here??}
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
\newcommand\nlines{4}
|
||||
\newcommand{\resultswidth}{1.3cm}
|
||||
\centering
|
||||
\caption{Average quality (PSNR): Low-Delay.}
|
||||
\caption{Average quality (PSNR): Low-delay configuration (LD).}
|
||||
\TABLESVSPACE
|
||||
\small
|
||||
\footnotesize
|
||||
@@ -94,7 +94,7 @@
|
||||
\newcommand{\resultswidth}{1.3cm}
|
||||
\newcommand\nlines{4}
|
||||
\centering
|
||||
\caption{Average quality (PSNR): Random-Access.}
|
||||
\caption{Average quality (PSNR): Random-access configuration (RA).}
|
||||
\TABLESVSPACE
|
||||
\small
|
||||
\footnotesize
|
||||
|
||||
@@ -120,7 +120,7 @@ Besides a full search algorithm, where a block-based approach is used to search
|
||||
\begin{enumerate}
|
||||
\item a method based on the total variation using the minimization of the \mbox{L-1} norm~\cite{DualFlow2016};
|
||||
\item a method which approximates the neighbourhood of each pixel by quadratic polynomials on both frames~\cite{PolyFlow2003};
|
||||
\item a method based on a combination of feature matching and total variation~\cite{DeepFlow2013}. \todo{check here!}
|
||||
\item a method based on a combination of feature matching and total variation~\cite{DeepFlow2013}. %\todo{check here!}
|
||||
\end{enumerate}
|
||||
|
||||
\subsubsection{Motion estimation based on total variation and regularisation (\textit{DualFlow})}
|
||||
@@ -321,7 +321,7 @@ Furthermore, image saliency driven by visual attention modelling is used in the
|
||||
% \HIGHLIGHT{where $R(l)$ represents the number of overhead bits required to encode \ac{ec} mode, and $D(l)$ is mean square error of the \ac{ctu} formed by $4^l$ ECUs.
|
||||
% }
|
||||
% \HIGHLIGHT{%
|
||||
% In equation \ref{eq:optdepth}, besides the parameter $\lambda$ used to trade-off the cost of distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(l)$ according to the duration of the error propagation decreases. This is given by the ratio between the temporal distance from the current frame ($t_0$) to the next refresh frame ($t_I$) and the period of refresh frames ($T_{I}$).
|
||||
% In equation~\ref{eq:optdepth}, besides the parameter $\lambda$ used to trade-off the cost of distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(l)$ according to the duration of the error propagation decreases. This is given by the ratio between the temporal distance from the current frame ($t_0$) to the next refresh frame ($t_I$) and the period of refresh frames ($T_{I}$).
|
||||
% }
|
||||
% \begin{align}
|
||||
% (ecm^*, l^*) &= arg \min_{\phi \in M} J(ecm, l), \\
|
||||
@@ -340,7 +340,7 @@ m_{l,i}^* = \argmin\limits_{\footnotesize \substack{l\in\left\{0...3\right\} \\
|
||||
where $R(m_{l,i})$ represents the number of overhead bits required to encode the \ac{ec} mode, and $D(m_{l,i})$ is the sum of absolute error of the \ac{ecu}.
|
||||
|
||||
|
||||
In equation \ref{EQ6:OptimalDepth}, besides the parameter $\lambda$ used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation. As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques}, this method uses the ratio between the temporal distance from the current frame ($t_C$) to the next refresh frame ($t_I$) and the period of refreshing frames ($T_{I}$).
|
||||
In equation~\ref{EQ6:OptimalDepth}, besides the parameter $\lambda$ used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation. As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques}, this method uses the ratio between the temporal distance from the current frame ($t_C$) to the next refresh frame ($t_I$) and the period of refreshing frames ($T_{I}$).
|
||||
%This weight varies between $\left[0;1 \right]$, achieving the lowest value for intra-coded frames and the maximum for the frame located immediately before an intra-coded one.
|
||||
This behaviour follows the fact that when a lost frame is closer to the next intra-coded frame, error propagation is lower. Thus, it is not worthwhile to spend too many overhead bits to reduce it. This is accomplished by increasing the cost of such bits according to the temporal distance between the missing frame and the next I frame.
|
||||
Moreover, a dynamic value for $\lambda$ is also used to adjust the amount of overhead within the frame, according to the visual relevance of each image region, determined using a saliency map.
|
||||
@@ -374,15 +374,6 @@ One should note that this method does not depend on the existence of a saliency
|
||||
|
||||
\subsection{Signalling the optimal EC modes}
|
||||
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
\includegraphics[width=0.8\columnwidth]{\ConcealmentAwareDir/Figures/ConcealmentSignalling}
|
||||
% \input{\ConcealmentAwareDir/Figures/ConcealmentSignalling.tikz}
|
||||
\caption{Payload structure of the \ac{sei} \acs{nal}} % to signal the optimal error concealment method.}
|
||||
% Symbols transmitted through \ac{sei} messages to signal the optimal error concealment method.}
|
||||
\label{FIG5:signalling}
|
||||
\end{figure}
|
||||
|
||||
% \begin{figure}[tb]
|
||||
% \begin{algorithmic}
|
||||
% \Procedure{encodectuecm}{$level$}
|
||||
@@ -408,6 +399,15 @@ Since no split is necessary at the deepest level, then $split=1$ followed by the
|
||||
This structure allows dynamic selection of different values for the level of the quad-tree partitioning ($l$) within each frame.
|
||||
Adaptive arithmetic coding~\cite{ArithmeticCodingJun1994} is used to efficiently encode both the $split$ and $ecm$ symbols, further reducing the overhead. Note that the \ac{sei} information uses much less bitrate than the video signal, thus it is very unlikely to be hit during transmission.
|
||||
|
||||
\begin{figure}[tb]
|
||||
\centering
|
||||
\includegraphics[width=0.8\columnwidth]{\ConcealmentAwareDir/Figures/ConcealmentSignalling}
|
||||
% \input{\ConcealmentAwareDir/Figures/ConcealmentSignalling.tikz}
|
||||
\caption{Payload structure of the \ac{sei} \acs{nal}} % to signal the optimal error concealment method.}
|
||||
% Symbols transmitted through \ac{sei} messages to signal the optimal error concealment method.}
|
||||
\label{FIG5:signalling}
|
||||
\end{figure}
|
||||
|
||||
|
||||
\subsection{Enhancement-layer: \ac{ec} residue coding}
|
||||
|
||||
@@ -450,9 +450,10 @@ In this section the performance of the proposed method is evaluated only for int
|
||||
Since intra \acs{cu} result in higher amount of bits, they also require a higher number of packets, making them more prone to transmission errors and losses.
|
||||
Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets correspond to intra coded slices, which is a non-negligible probability of one of them being hit by transmission errors.
|
||||
Moreover, by limiting the proposed method to intra frames, the amount of required overhead is reduced.
|
||||
In this case a constant $\lambda_0=10$ was considered without taking into consideration saliency estimation.
|
||||
|
||||
Since a relevant performance indicator is the error propagation over temporally predicted frames, this was evaluated using a single intra-frame loss.
|
||||
Figures \ref{FIG5:singlelosslowdelay} and \ref{FIG5:singlelossramdon} show the \ac{psnr} results when frame Intra-frame $\#16$ is missing, for \ac{ld} and \ac{ra} encoding configurations, respectively.
|
||||
Figures~\ref{FIG5:singlelosslowdelay} and~\ref{FIG5:singlelossramdon} show the \ac{psnr} results when frame Intra-frame $\#16$ is missing, for \ac{ld} and \ac{ra} encoding configurations, respectively.
|
||||
These figures show the results for the proposed method (\PROP), the motion-copy method (\MC) and also an alternative proposed method using fixed \ac{ecu} size of $16\times16$ pixels.
|
||||
These results indicate that the proposed method (\PROP) outperforms the reference one and it is able to achieve quality gains up to {2 dB} for \ac{ld} configuration (see Figure~\ref{FIG5:singlelosslowdelay}), decrease the error propagation and increase the quality of the frames affected from the loss of the intra frame, by up to {3 dB}.
|
||||
The use of \ac{ra} configuration leads to higher quality degradation than LD, due the higher temporal distance of the reference frame used for \ac{ec} (Frame $\#8$ is used to recover the lost Frame~$\#16$). Note that when Intra-frame \#16 is lost, all frames within the range \#9 to \#31 (i.e., next I-frame) are affected by error propagation.
|
||||
@@ -523,10 +524,10 @@ It is also worthwhile to notice that the \ac{psnr} variance obtained with the
|
||||
%\subsubsection{Overhead analysis}
|
||||
|
||||
Subsequently, the percentage of overhead bitrate introduced by the proposed method is compared against fixed \ac{ecu} sizes of $32\times32$ and $16\times16$, i.e., $Level=1$ and $Level=2$.
|
||||
These results are presented in Table \ref{TAB6:RedundancyIntra} and were obtained for a total bitrate, as shown in Table~\ref{TableTestSequences}.
|
||||
These results are presented in Table~\ref{TAB6:RedundancyIntra} and were obtained for a total bitrate, as shown in Table~\ref{TableTestSequences}.
|
||||
It is noticeable from the results that reducing the \ac{ecu} size (see results for $32\times32$ and $16\times16$) increases the overhead ratios as more symbols are transmitted (one for each \ac{ecu}).
|
||||
While the maximum overhead of these fixed partitions is $1.81\%$ (Traffic), the proposed dynamic partitioning method is able to reduce the overhead to a maximum of $0.19\%$ (Kendo).
|
||||
Moreover, comparing the results of Table \ref{TAB6:RedundancyIntra} with the video quality results obtained in case of packet losses reveals in the proposed method only a small reduction in the reconstruction quality, even with a significantly lower overhead.
|
||||
Moreover, comparing the results of Table~\ref{TAB6:RedundancyIntra} with the video quality results obtained in case of packet losses reveals in the proposed method only a small reduction in the reconstruction quality, even with a significantly lower overhead.
|
||||
This indicates that using dynamic partitioning allows to achieve a better trade-off between the amount of overhead and the \ac{ec} performance.
|
||||
In summary, the proposed technique is able to enhance the performance of the standard decoding of intra coded frames by finding the best error concealment method at the encoder and sending such information with a small amount of signalling overhead.
|
||||
|
||||
@@ -582,11 +583,11 @@ This section is organised in the following subsections:
|
||||
\subsubsection{\ac{ec} mode usage distribution}
|
||||
|
||||
The \ac{ec} mode usage distribution is an indicator of the usefulness of each \ac{ec} mode.
|
||||
Table \ref{tab:useratio} shows the average $Level$ selected for the \ac{ecu} quadtree and the usage percentage of each \ac{ec} mode.
|
||||
Table~\ref{tab:useratio} shows the average $Level$ selected for the \ac{ecu} quadtree and the usage percentage of each \ac{ec} mode.
|
||||
It is noticeable from these results that the proposed method leads to low average $Level$ values (lower than 1), resulting in larger \acs{ecu}, which requires a lower amount of \ac{ecu} to be transmitted than if higher values of $Level$ were chosen. This implicitly results in low signalling information.
|
||||
This also justifies the low overhead shown in Table~\ref{tab:redundancy_signalling}.
|
||||
|
||||
The results in Table \ref{tab:useratio} also show that all \ac{ec} modes are effectively used, which indicates that all of them are found relevant to recover the missing data.
|
||||
The results in Table~\ref{tab:useratio} also show that all \ac{ec} modes are effectively used, which indicates that all of them are found relevant to recover the missing data.
|
||||
The \ac{ec} candidate methods based on motion field estimation (i.e., M1, M2) are selected more often because they provide more accurate representation of the scene's motion than the others (i.e., M3, M4).
|
||||
|
||||
\begin{table}[tb]
|
||||
@@ -620,7 +621,7 @@ The \ac{ec} candidate methods based on motion field estimation (i.e., M1, M2) ar
|
||||
The evaluation of the proposed saliency-weighted \ac{ec} mode decision (\DYNLAMBDA) was carried out under random loss events and it was compared with the reference methods.
|
||||
The video quality was evaluated by using two quality metrics: \ac{psnr} and a \ac{fsim}~\cite{FSIM_TIP2011}, in order to validate the performance of the proposed method in a consolidated manner.
|
||||
Realistic conditions were used for transmission, by allowing independent losses for any type of \acs{nal}, hence there is no guarantee that the decoder conditions are always equal to those at the encoder, which means that sub-optimal \ac{ec} may be used at the decoder.
|
||||
The results obtained without saliency weighting are also shown for comparison (\FIXEDLAMBDA). In this case a constant $\lambda_0=10$ is used.
|
||||
The results obtained without saliency weighting are also shown for comparison (\FIXEDLAMBDA). As in the previous subsection, a $\lambda_0=10$ was used.
|
||||
|
||||
Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM} show the average quality under different \acs{plr} for the \ac{ld} configuration.
|
||||
Table~\ref{TAB4:SignallingPSNR} shows the \ac{psnr} and weighted \ac{psnr} (\ac{wpsnr}) between parentheses.
|
||||
@@ -647,9 +648,9 @@ This confirms the two-fold superiority of adaptive $\lambda$, i.e. not only the
|
||||
In this section the error propagation is evaluated using single loss events and compared against the reference methods.
|
||||
In this simulation a whole frame loss is enforced in order to assess the impact of error propagation in the worst case scenario.
|
||||
Experimental evaluations were performed for both intra and inter-coded frame loss.
|
||||
Figure \ref{FIG5:singlelossInter} shows the \ac{psnr} over a \ac{gop} with an \ac{idr} period of 16 frames, where the whole frame \#6 is missing in sequences Basketball Drill and People On Street.
|
||||
Figure~\ref{FIG5:singlelossInter} shows the \ac{psnr} over a \ac{gop} with an \ac{idr} period of 16 frames, where the whole frame \#6 is missing in sequences Basketball Drill and People On Street.
|
||||
%The figure shows the results obtained by decoding the stream using the proposed method with adaptive $\lambda$ but without any information sent in the enhancement layer.
|
||||
Figure \ref{FIG5:singlelossIntra} shows the \ac{psnr} for the same conditions but with a loss event on the intra-coded Frame \#16, in sequences Kimono and Tennis.
|
||||
Figure~\ref{FIG5:singlelossIntra} shows the \ac{psnr} for the same conditions but with a loss event on the intra-coded Frame \#16, in sequences Kimono and Tennis.
|
||||
|
||||
\input{042_ConcealmentAwareEncoding/Results/ECMQualitySingleloss.tex}
|
||||
|
||||
@@ -712,7 +713,7 @@ From the results discussed above, it is clearly observed that the proposed metho
|
||||
|
||||
The achieved results were compared with previous works, where fair comparisons could be made.
|
||||
In \cite{ConcealmentSignalling2008} a method was proposed to improve the recovery quality by signalling the suitable \ac{mv} to be used for \ac{ec}.
|
||||
The proposed \ac{ec}-aware scheme of Figure \ref{FIG5:BlockDiagram} is able to achieve similar quality gains with less amounts of redundancy, allowing a better trade-off between quality and overhead.
|
||||
The proposed \ac{ec}-aware scheme of Figure~\ref{FIG5:BlockDiagram} is able to achieve similar quality gains with less amounts of redundancy, allowing a better trade-off between quality and overhead.
|
||||
In comparison with the work described in \cite{SignallingRobustScalable2015}, the proposed method is able to select among more complex \ac{ec} methods, which allows for higher quality gains.
|
||||
Both of the above mentioned methods search for the optimal \ac{ec} method at the encoder-side, which results in similar complexity as the proposed method.
|
||||
|
||||
@@ -773,6 +774,6 @@ The proposed method uses \ac{ec} techniques with different levels of computati
|
||||
|
||||
In this chapter an \ac{ec}-aware encoding scheme is proposed by incorporating loss simulation at the encoder-side to enhance the EC efficiency at the decoder.
|
||||
% The proposed method relies on loss simulation at the encoder-side to optimise the error concealment proposed. Then, the optimal error concealment is signalled to the decoder.
|
||||
The novelty of this method is to include optimal \ac{ec} mode decision at the encoder, to be used by the decoder in case of packet loss, and also to allow encoding of the \ac{ec} residue using the enhancement layer of the \ac{hevc} standard.
|
||||
The novelty of this method is to include optimal \ac{ec} mode decision at the encoder, to be used by the decoder in case of packet loss, and also to allow encoding of the \ac{ec} residue using the enhancement layer of the \ac{hevc} standard. The use of \ac{ec} signalling was firstly proposed for the use on intra-coded frames in \ref{Conf:ConcAware}, and then improved and proposed for a generic robust video transmission, resulting in a more detailed analysis published in \ref{Journal:ConcAware}.
|
||||
The overall results demonstrate that the proposed method is able to efficiently reconstruct lost frames and reduce the error propagation compared with other reference methods.
|
||||
A consistent quality improvement achieved at the cost of low overhead allows the use of this method in broadcasting video services and applications where packet loss probability is not negligible.
|
||||
|
||||
@@ -546,7 +546,7 @@ e
|
||||
# 19 36.70 41.39 42.01
|
||||
# e
|
||||
\end{gnuplot}
|
||||
\vspace{-8mm}
|
||||
\vspace{-4mm}
|
||||
\caption{Error propagation after losing the inter-coded frame $\#6$.} %for Basketball Drill and People On Street sequences.}
|
||||
\label{FIG5:singlelossInter}
|
||||
\FIGSVSPACE
|
||||
@@ -1025,7 +1025,7 @@ e
|
||||
# 30 30.56 40.46 40.42
|
||||
# e
|
||||
\end{gnuplot}
|
||||
\vspace{-8mm}
|
||||
\vspace{-4mm}
|
||||
\caption{Error propagation after losing intra-coded frame $\#16$.} % for Kimono and Tennis sequences.}
|
||||
\label{FIG5:singlelossIntra}
|
||||
\FIGSVSPACE
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
% CHAPTER
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
\chapter{Adaptive Error Robustness for Highly Efficient Video Coding \VERSION{1}}
|
||||
\chapter{Adaptive Error Robustness for Highly Efficient Video Coding \VERSION{2}}
|
||||
\label{Chapter:EndToEndRobustSystem}
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
@@ -15,13 +15,13 @@
|
||||
|
||||
The effectiveness of different error resilience techniques is rather variable depending on the video content, the transmitted bitrate, network conditions, among others.
|
||||
Therefore, it is not straightforward to select a single technique which achieves the highest efficiency for every possible scenario and also for the whole range of networking conditions.
|
||||
This chapter addresses this problem by proposing an adaptive network which optimises the efficiency of the techniques previously described in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}
|
||||
At the encoder-side different video streams are generated with different levels of compression efficiency and error robustness.
|
||||
Then, a pre-trained neural network is used to predict which video stream will achieve the highest decoded video quality for a particular loss case.
|
||||
The selection is based on different input parameters which include information from the video signal, the coded stream and the transmission network. Using this approach the overall performance of the proposed adaptive method significantly increases yielding a better trade-off between compression efficiency and error robustness.
|
||||
This chapter addresses this problem by proposing an adaptive network that optimises the efficiency of the techniques previously described in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}.
|
||||
At the encoder-side, a pre-trained neural network predicts the video stream that is able to achieve the highest decoded quality for a particular loss cases, among the various video streams encoded with different levels of compression efficiency and error robustness.
|
||||
The selection is based on different input parameters that include information from the video signal, the coded stream and the transmission network.
|
||||
Using this approach the overall performance of the proposed adaptive method significantly increases, yielding a better trade-off between compression efficiency and error robustness.
|
||||
|
||||
This chapter is organised as follows.
|
||||
Section~\ref{Section6:Problem} presents the problem addressed in this chapter and experimental evidence as motivation for the method investigated.
|
||||
Section~\ref{Section6:Problem} presents the problem addressed in this chapter and experimental evidence, as motivation for the method investigated.
|
||||
Section~\ref{Section6:ProposedMethod} provides an overview of architecture that supports the implementation of the proposed method and Section~\ref{Section6:OptimisationMethod} describes the technical details.
|
||||
Section~\ref{Section6:ExperimentalResults} presents the performance evaluation and a discussion of results.
|
||||
Finally, Section~\ref{Section6:Summary} concludes the chapter.
|
||||
@@ -42,7 +42,7 @@ However, each method on its own does not adapt to different conditions.
|
||||
While the reference frame selection algorithm presented in Chapter~\ref{Chapter:ErrorResilienceTechniques} introduces low overhead in comparison with existing techniques, the method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} achieves higher quality gains at cost of slightly higher bitrate increase.
|
||||
Therefore, one may conclude that each method may be further optimised for different applications, video characteristics and network conditions.
|
||||
This chapter advances one step forward , by investigating a joint robust coding method using different techniques to optimise the robust coder across different conditions, such as, the input content and the network information.
|
||||
This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and can selected based on the current conditions.
|
||||
This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and can be selected based on the current conditions.
|
||||
|
||||
|
||||
\subsection{Comparison of the proposed methods}
|
||||
@@ -52,24 +52,24 @@ This study extends the previous evaluations presented in Chapter~\ref{Chapter:Er
|
||||
Moreover, it aims at comparing the performance of different techniques across a variety of video contents, bitrates and loss conditions, in order to find relevant insights which affect the decoded video quality. Consequently, these relevant insights may be used to develop an adaptive robust coding method.
|
||||
|
||||
%In order to perform a wider evaluation different bitrate and \acs{plr} were used. Table~\ref{Table6:TestVariables} illustrates the selected configurations. As shown in the table a total of 15 test sequences were used encoder using 5 different target bitrates. One should note that the main bitrate varies based on the test sequence and it is shown in third column of Table~\ref{TableTestSequences}. Different network conditions were evaluated using a total of 13 different \acs{plr}, with higher special focus on lower loss ratios, which are more relevant to modern video streaming applications.
|
||||
The following methods are evaluated in the simulation study, namely:
|
||||
The following methods were evaluated in the simulation study, namely:
|
||||
\begin{itemize}
|
||||
\item the default \ac{hevc} decoder with frame-copy \ac{ec} (\REF), as reference method;
|
||||
\item the default \ac{hevc} encoder without robust coding-techniques and motion vector extrapolation \ac{ec} (\MVE);
|
||||
\item the proposed reference frame selection scheme (\EXP);
|
||||
\item the \EXP method in combination with an \ac{ec} optimisation with 10\% of bitrate allocate for the scalable enhancement layer (\EXPELTEN);
|
||||
\item the same as above, with 20\% of bitrate allocate for the scalable enhancement layer (\EXPELTWEN).
|
||||
\item the \EXP method in combination with an \ac{ec} optimisation with 10\% of bitrate allocated for the scalable enhancement layer (\EXPELTEN);
|
||||
\item the same as above, with 20\% of bitrate allocated for the scalable enhancement layer (\EXPELTWEN).
|
||||
\end{itemize}
|
||||
These methods were selected as they are the main techniques proposed within this research work and cover two aspects of a robust video transmission: (i) robust video encoding by dynamically select the reference frames and (ii) robust video decoding by improving the error concealment performance.
|
||||
|
||||
\input{043_EndToEndRobustSystem/Results/AdaptiveSequenceList.tex}
|
||||
|
||||
Table~\ref{Table6:TestMaterialCharacteristics} presents the list of test sequences used. These sequences were selected to cover a wide range of video content, with different levels of spatial and temporal complexity, as demonstrated by the values of \ac{si} and \ac{ti}, respectively.
|
||||
Table~\ref{Table6:TestMaterialCharacteristics} presents the list of test sequences used. These sequences were selected to cover a wide range of video content, with different levels of spatial and temporal complexity, as demonstrated by the values of \ac{si} and \ac{ti}~\cite{ITUTP9102008}, respectively.
|
||||
Moreover, from the bitstream encoded with the reference implementation of the \ac{hevc} encoder, the percentage of the inter-coded blocks and the average \ac{mv} magnitude was extracted and presented in the table (columns 4 and 5, respectively). These values were extracted to compare them against the performance obtained with the proposed robust coding methods.
|
||||
The remaining configuration were kept as described in Section~\ref{Section3:ErrorResilienceEvaluation}.
|
||||
The remaining configuration was kept as described in Section~\ref{Section3:ErrorResilienceEvaluation}.
|
||||
%The experimental methodology carried out in this section follows the same conditions described in Section~\ref{Section3:ErrorResilienceEvaluation}.
|
||||
|
||||
\subsubsection{Simulation results and discussion}
|
||||
\subsubsection{Preliminary simulation results and discussion}
|
||||
|
||||
\begin{figure}[!t]
|
||||
\centering
|
||||
@@ -92,16 +92,16 @@ Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained by reducin
|
||||
\label{Figure6:AverageQualityReduction}
|
||||
\end{figure}
|
||||
|
||||
Firstly, comparing the results for different bitrates, it is noticeable that lower quality is obtained for the lower bitrates, which results from the higher compression ratios used. It is also noticed that lower quality gains, when comparing with the reference \ac{hevc} encoder (\REF), are obtained for the video streams encoded with a 40\% reduction of bitrate. This indicates that for lower bitrates the performance of the proposed methods is decreased, especially for the \EXPELTWEN case where 20\% of the bitrate is allocated for the enhancement layer. Therefore, for lower bitrates, using methods that have a lower impact on the coding efficiency (e.g., \EXP) or even disable the robust coding techniques (\MVE) may lead to higher performance.
|
||||
Firstly, comparing the results for different bitrates, it is noticeable that lower quality is obtained for the lower bitrates, which results from the higher compression ratios used. It is also noticed that lower quality gains, when comparing with the reference \ac{hevc} encoder (\REF), are obtained for the video streams encoded with a 40\% reduction of bitrate. This indicates that for lower bitrates the performance of the proposed methods decreases, especially for the \EXPELTWEN case, where 20\% of the bitrate is allocated for the enhancement layer. Therefore, for lower bitrates, using methods that have a lower impact on the coding efficiency (e.g., \EXP) or even disable the robust coding techniques (\MVE) may lead to higher performance.
|
||||
|
||||
Secondly, comparing the results across different video content, i.e., different test sequences, one can notice that the performance of the different methods is not consistent.
|
||||
For instance, in the result of Figure~\ref{Figure6:AverageQualityMain}, for Tennis sequence the \EXPELTWEN clearly outperforms the remaining methods, however, for the BQSquare sequence higher performance is achieved by using less bitrate allocated for the \ac{el} (i.e., 10\%). This is a consequence of the higher quality degradation due to higher motion in Tennis sequence (see results for TI and \ac{mv} size in Table~\ref{Table6:TestMaterialCharacteristics}) which required higher redundant information to increase the overall quality.
|
||||
%\HIGHLIGHT{Mention SI!!}
|
||||
|
||||
Finally, comparing the results for all test sequences and both bitrates it is clearly noticed that the \acs{plr} has a significant impact on the performance of the error robustness techniques. In case of the proposed reference frame selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions. For the \EXPELTWEN case higher \acs{plr} are required to be able to outperform both the reference \ac{hevc} and the \ac{mve} method.
|
||||
Finally, comparing the results for all test sequences and both bitrates it can be clearly noticed that the \acs{plr} have a significant impact on the performance of the error robustness techniques. In case of the proposed reference frame selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions. For the \EXPELTWEN case, higher \acs{plr} are required to be able to outperform both the reference \ac{hevc} and the \ac{mve} method.
|
||||
|
||||
Summarising, the performance of the proposed methods is not similar across different video content and bitrates, opening research to adaptively choose the best method that should be applied to achieve the highest decoded video quality.
|
||||
In order to achieve increased efficiency and higher video quality, the best trade-off between auxiliary information and video bitrate should be found by selecting the optimal robust coding algorithm.
|
||||
In order to achieve increased efficiency and higher video quality, the best trade-off between auxiliary information and video bitrate can be found by selecting the optimal robust coding algorithm.
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
%% SECTION %%
|
||||
@@ -134,7 +134,7 @@ The proposed approach results in four different bitstreams with different charac
|
||||
(i) S1: reference stream without error robust coding;
|
||||
(ii) S2: robust video coding using reference frame selection;
|
||||
(iii) S3: robust video coding using selective reference frames and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue;
|
||||
(iv) S4: same as the previous bitstream but with 20\% bitrate allocated for the scalable \ac{el}.
|
||||
(iv) S4: same as the previous bitstream but with 20\% bitrate allocated for the scalable~\ac{el}.
|
||||
|
||||
Finally, the video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and feedback from the transmission channel.
|
||||
From the input video signal, relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, further information is also extracted from the coded bitstream using a stream parser.
|
||||
@@ -143,7 +143,7 @@ These parameters are fed into the optimisation block to select the best bitstrea
|
||||
The optimisation process is described in detail in the following section.
|
||||
|
||||
The main purpose of this architecture is to allow dynamic stream switching during real time transmission.
|
||||
To accomplish that without causing drift at the decoder, the switching procedure should take place at refresh points, i.e., \ac{idr} and \ac{cra} frames, which correspond to clean random access points. In the proposed scheme, \ac{idr} frames are introduced every 16 frames, resulting in between 1 to 3 switching points every second, depending on the frame rate of the video signal.
|
||||
To accomplish this without causing drift at the decoder, the switching procedure should take place at refresh points, i.e., \ac{idr} and \ac{cra} frames, which correspond to clean random access points. In the proposed scheme, \ac{idr} frames are introduced every 16 frames, resulting in between 1 to 3 switching points every second, depending on the frame rate of the video signal.
|
||||
|
||||
|
||||
|
||||
@@ -155,7 +155,7 @@ To accomplish that without causing drift at the decoder, the switching procedure
|
||||
|
||||
In this section the optimisation method used to select a robust video stream is presented.
|
||||
The selection method is based on a shallow neural network which predicts the optimal stream based on different parameters.
|
||||
This section describes both the input parameters and the neural network used in the optimal stream selection process. Moreover, a description data set used to train and evaluate the neural network used is also provided
|
||||
This section describes both the input parameters and the neural network used in the optimal stream selection process. Moreover, a description data set used to train and evaluate the neural network used is also provided.
|
||||
|
||||
\subsection{Data set}
|
||||
|
||||
@@ -163,16 +163,16 @@ This section describes both the input parameters and the neural network used in
|
||||
\def\NumCases{910\xspace}
|
||||
|
||||
|
||||
The proposed method includes an optimisation stage, which used a neural network classifier in order to select the optimal stream to be used for transmission.
|
||||
This stream is able to obtain the highest quality for particular transmission scenario.
|
||||
The proposed method includes an optimisation stage that uses a neural network classifier, in order to select the optimal stream to be used for transmission.
|
||||
This stream is able to obtain the highest quality for a particular transmission scenario.
|
||||
Therefore, several test cases were simulated in order to obtain results for a wide range of conditions. These tests are mainly characterised by three elements:
|
||||
\begin{enumerate}
|
||||
\begin{itemize}
|
||||
\item Video signal;
|
||||
\item Target bitrate;
|
||||
\item Packet loss ratio.
|
||||
\end{enumerate}
|
||||
\end{itemize}
|
||||
|
||||
To obtain the data set 14 different test sequences with different scene content, listed in Table~\ref{Table6:TestMaterialCharacteristics}. Those sequences were encoded with the main bitrate (see Table~\ref{Table6:TestMaterialCharacteristics}), as well as four different others obtained by reducing the main bitrate by 10\%, 20\%, 30\% and 40\%.
|
||||
To obtain the data set, 14 different test sequences with different scene content were used, as listed in Table~\ref{Table6:TestMaterialCharacteristics}. Those sequences were encoded with the main bitrate (see Table~\ref{Table6:TestMaterialCharacteristics}), as well as four different others, obtained by reducing the main bitrate by 10\%, 20\%, 30\% and 40\%.
|
||||
Then, the following \acs{plr} were used to cover a wide range of loss ratios giving more detail to low \acs{plr}: 0\%, 0.5\%, 1.0\%, 1.2\%, 1.6\%, 2.0\%, 2.2\%, 2.4\%, 2.6\%, 2.8\%, 3.0\%, 5.0\% and 10.0\%.
|
||||
This results in a total of \NumCases test conditions.
|
||||
|
||||
@@ -189,12 +189,12 @@ From the input video the two parameters are measured in a frame by frame basis,
|
||||
|
||||
%From the coded video streams complementary information to characterise the streaming content is also extracted. A stream parser is applied to the reference bitstream without error robustness techniques. Therefore, the prediction information extracted is related to a stream coded based on optimal \ac{rd} optimisation which was not affected by robust coding techniques. The following is extracted:
|
||||
|
||||
The characterisation of these three aspects of the video transmission result in a total of six parameters:
|
||||
\begin{itemize}
|
||||
The characterisation of these three aspects of the video transmission results in a total of six parameters:
|
||||
\begin{enumerate}
|
||||
|
||||
\item \textit{\ac{si}}: parameter expressing the amount of spatial detail, which consequently affects the error recovery operations, leading to higher quality degradation~\cite{ITUTP9102008}.
|
||||
|
||||
\item \textit{\ac{ti}}: parameter expressing the amount of difference between temporally adjacent frames~\cite{ITUTP9102008}. This has a direct impact on the impact of errors, and thus the performance of the error resilience techniques as shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, which were discussed in Section~\ref{Section6:Problem}.
|
||||
\item \textit{\ac{ti}}: parameter expressing the amount of difference between temporally adjacent frames~\cite{ITUTP9102008}. This has a direct relation to the impact of errors, and thus the performance of the error resilience techniques, as shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, which were discussed in Section~\ref{Section6:Problem}.
|
||||
|
||||
\item \textit{Inter (percentage)}: the percentage of the image that is compressed using inter-frame prediction techniques.
|
||||
This is an indication of the distribution of spatial and temporal dependencies.
|
||||
@@ -202,26 +202,37 @@ This is calculated as the ratio of inter-coded blocks over intra-coded ones.
|
||||
Normally, coded streams with higher number of temporal dependencies are more prone to error propagation, thus higher number of bits should be allocated for error protection \cite{UEPScalable_TCSVT2010}.
|
||||
|
||||
\item \textit{AVG\_MV}: for the inter-coded regions, the motion information is analysed and the average \ac{mv} magnitude is calculated to provide information about the motion intensity.
|
||||
As shown in previous studies, motion-based \ac{ec} algorithms are less efficient for video signals with higher motion intensity, as \acs{mv} are less predictable using algorithms such as \ac{mc} or \ac{mve} \cite{HybridConcealmentTB2008}.
|
||||
As shown in previous studies, motion-based \ac{ec} algorithms are less efficient for video signals with higher motion intensity, as \acs{mv} are less predictable using algorithms, such as \ac{mc} or \ac{mve} \cite{HybridConcealmentTB2008}.
|
||||
|
||||
\item \textit{Bitrate}: information regarding the average bit allocation per frames is extracted, since it will affect the video quality and the amount of bits that can be available for auxiliary information. This value is expressed in Mbps, as illustrated in Table~\ref{Table6:TestMaterialCharacteristics}.
|
||||
\item \textit{Bitrate}: information regarding the average bit allocation per frames is calculated, since it will affect the video quality and the amount of bits that can be available for auxiliary information. This value is expressed in Mbps, as illustrated in Table~\ref{Table6:TestMaterialCharacteristics}.
|
||||
|
||||
\item \textit{\ac{plr}:} feedback information from the network to characterise the transmission losses, expressed by the percentage of packets loss. This is used since for higher the loss rations there is a clear increase in the quality degradation which affect the performance of the robust coding techniques.
|
||||
\item \textit{\ac{plr}:} feedback information from the network to characterise the transmission losses, expressed by the percentage of packets loss. This is used since for higher loss ratio there is a clear increase in the quality degradation, which affects the performance of the robust coding techniques.
|
||||
Transmission networks with higher loss ratios require a higher bit allocation for robust techniques, therefore the error resilience technique should adapt based on the value of the \ac{plr}.
|
||||
|
||||
|
||||
\end{itemize}
|
||||
\end{enumerate}
|
||||
|
||||
|
||||
\begin{table}[tb]
|
||||
%To characterise the input video signal the following parameters are used: \ac{si} and \ac{ti}~\cite{ITUTP9102008}.
|
||||
%These parameters are obtained as described in Chapter~\ref{Chapter:HEVCErrorResilience}, and aim at evaluating the spatial complexity and motion complexity of the video scenes, which have relevant impact on the error robustness of the coded streams.
|
||||
%As shown, test material with higher \ac{si} or \ac{ti} suffer from higher quality degradation as the \ac{plr} increases, e.g., Book Arrival and Tennis sequences.
|
||||
|
||||
These parameters were compared against the optimal stream, selected from S1 to S4.
|
||||
This aims at finding the correlation between the input parameters and the output variable, i.e., target stream.
|
||||
In this study the streams are characterised by a number between 1 and 4.
|
||||
Table~\ref{Table6:Correlation} shows the absolute Person Correlation coefficient obtained across all sequences, bitrates and \acs{plr}.
|
||||
The table shows the results for the parameters previously described, as well as, a new parameter obtained from the average magnitude of the optical flow measured using the method in~\cite{DeepFlow2013} (\textit{OPT\_FLOW}).
|
||||
Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable. As both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method, as it achieves a higher Person Correlation coefficient.
|
||||
|
||||
\begin{table}[b]
|
||||
\centering
|
||||
\caption{Pearson correlation coefficient between the different parameters and the optimal robust coding techniques (optimal stream).}
|
||||
\caption{Pearson Correlation coefficient between the different parameters and the optimal robust coding techniques (optimal stream).}
|
||||
\label{Table6:Correlation}
|
||||
\TABLESVSPACE
|
||||
\renewcommand{\arraystretch}{1.2}
|
||||
\begin{tabular}{p{4cm}x{4cm}}
|
||||
\toprule
|
||||
Parameter & Absolute pearson coefficient \\ \hline
|
||||
Parameter & Absolute Pearson Correlation \\ \hline
|
||||
% \textit{SI} & -0.1553 \\ \hline
|
||||
% \textit{TI} & 0.6079 \\ \hline
|
||||
% \textit{Inter (percentage)} & -0.6487 \\ \hline
|
||||
@@ -241,20 +252,9 @@ Transmission networks with higher loss ratios require a higher bit allocation fo
|
||||
\end{table}
|
||||
|
||||
|
||||
%To characterise the input video signal the following parameters are used: \ac{si} and \ac{ti}~\cite{ITUTP9102008}.
|
||||
%These parameters are obtained as described in Chapter~\ref{Chapter:HEVCErrorResilience}, and aim at evaluating the spatial complexity and motion complexity of the video scenes, which have relevant impact on the error robustness of the coded streams.
|
||||
%As shown, test material with higher \ac{si} or \ac{ti} suffer from higher quality degradation as the \ac{plr} increases, e.g., Book Arrival and Tennis sequences.
|
||||
|
||||
These parameters were compared against the optimal stream which should be selected (S1...S4).
|
||||
This aims at finding the correlation between the input parameters and the output variable, i.e., target stream.
|
||||
In this study the streams are characterised by a number between 1 and 4, respectively.
|
||||
Table~\ref{Table6:Correlation} shows the absolute person correlation coefficient obtained across all sequences, bitrates and \acs{plr}.
|
||||
The table shows the results for the parameters previously described, as well as, a new parameter obtained from the average magnitude of the optical flow measured using the method in~\cite{DeepFlow2013} (\textit{OPT\_FLOW}).
|
||||
Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable. As both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method as it achieves a higher person correlation coefficient.
|
||||
|
||||
% Podia se citar este paper
|
||||
% \cite{IntraRefreshICIP2009}
|
||||
%\HERE
|
||||
|
||||
\subsection{Neural network}
|
||||
|
||||
\glsreset{nn}
|
||||
@@ -263,7 +263,7 @@ A \ac{nn} is a computational model based on the structure and functions of biolo
|
||||
%In terms of computer science, \acs{nn} work similarly to a human nervous system for receiving, processing, and transmitting information between neurons.
|
||||
Mainly, a \ac{nn} consists of three different layers: the input layer which receives data from an external environment, the hidden layer which process the inputs and an output layer~\cite{NeuralNetworks1994}.
|
||||
In the past \acs{nn} have been widely used in video processing applications such as, quality evaluation~\cite{VideoQualityNN_TNN2006}, video analysis~\cite{VideoAnalysisNN_2018} and video coding~\cite{RateDistortionNN_VCIP2017,HEVCComplexityNN_TIP2018}.
|
||||
In the proposed method a feed-forward network is used which connects every neuron in a given layer to the neurons of the subsequent layer.
|
||||
In the proposed method a feed-forward network is used, which connects every neuron in a given layer to the neurons of the subsequent layer.
|
||||
|
||||
Figure~\ref{FIG6:NeuralNetwork} shows the diagram of the \ac{nn} used in the proposed method.
|
||||
The network contains and input and output layer, as well as, a single hidden layer.
|
||||
@@ -281,22 +281,23 @@ Finally, the output with the highest value is selected and the corresponding str
|
||||
\FIGSVSPACE
|
||||
\end{figure}
|
||||
|
||||
In the proposed approach a shallow \ac{nn} is used for simplicity and to avoid over-fitting due to the limited test cases.
|
||||
By avoiding over-fitting, it is guaranteed that the proposed method would continue to work in future test conditions not used for the network training.
|
||||
In the proposed approach a shallow \ac{nn} is used for simplicity and to avoid over-fitting, due to the limited test cases.
|
||||
By avoiding over-fitting, it is guaranteed that the proposed method would continue to work in future test conditions, not used for the network training.
|
||||
However, different network topologies with different number of neurons in each layer could also be used to predict between different error resilience techniques, as in the proposed method.
|
||||
|
||||
\subsubsection{Training and validation}
|
||||
|
||||
%As mentioned before a total of \NumCases test cases are used for the evaluation of the proposed method.
|
||||
To effectively train and validate the \ac{nn} the data set is randomly in three different groups: (i) 55\% of the test cases are used for training, (ii) 15\% for validation and (iii) 30\% for testing the \ac{nn} performance.
|
||||
To effectively train and validate the \ac{nn}, the data set is randomly partitioned into three different groups: (i) 55\% of the test cases are used for training, (ii) 15\% for validation and (iii) 30\% for testing the \ac{nn} performance.
|
||||
The \ac{nn} supervised training uses 591 test conditions and a scaled conjugate gradient algorithm~\cite{NNScaledConjugateTraining_1993}.
|
||||
The remaining 254 conditions are used to evaluate the accuracy of the proposed method after training using different conditions. From this test one video sequences was left out, to be used in the performance evaluation of the proposed method.
|
||||
The remaining 254 conditions are used to evaluate the accuracy of the proposed method after training with different conditions. From this test one video sequence was left out, to be used in the performance evaluation of the proposed method.
|
||||
|
||||
|
||||
Table~\ref{Table6:ConfusionTraining} shows the confusion matrix of the \ac{nn} training phase.
|
||||
In this matrix each column corresponds to the target result, i.e., the cases where the coded streams S1 to S4 would lead to the highest decoded video quality, respectively.
|
||||
In this matrix each column corresponds to the target result, i.e., the cases where the coded streams, S1 to S4, would lead to the highest decoded video quality.
|
||||
Similarly, each row corresponds to the \ac{nn} selected by the trained networks.
|
||||
Thus, those cases where the horizontal and vertical labels match, indicate that the network produced the correct output. The remaining positions correspond to the erroneous outputs.
|
||||
Thus, those cases where the horizontal and vertical labels match, indicate that the network produced the correct output.
|
||||
The remaining positions correspond to the erroneous outputs.
|
||||
The last line shows the accuracy for each target output.
|
||||
These results correspond to the dataset used to train the \ac{nn}.
|
||||
|
||||
@@ -319,8 +320,8 @@ These results correspond to the dataset used to train the \ac{nn}.
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
Results in Table~\ref{Table6:ConfusionTraining} shows that streams S1 (i.e., without robust coding techniques) and S4 (i.e., using optimal \ac{ec} with 20\% of bitrate in the \ac{el}) lead to the highest quality in the majority of the test cases (are selected more often).
|
||||
The \ac{nn} is able to achieve a high accuracy for S1 and S4 streams, correctly selecting the output stream in more than 90\% of cases. Regarding the remaining robust streams (i.e., S2 and S3) an acceptable accuracy of 70.8\% and 74.4\% is achieved, respectively. This reveals that the \ac{nn} used in the proposed method is able to predict the coded stream which leads to the highest video quality across different conditions with acceptable accuracy.
|
||||
Results in Table~\ref{Table6:ConfusionTraining} shows that streams S1 (i.e., without robust coding techniques) and S4 (i.e., using optimal \ac{ec} with 20\% of bitrate in the \ac{el}) lead to the highest quality in the majority of the test cases (are more often selected).
|
||||
The \ac{nn} is able to achieve a high accuracy for S1 and S4 streams, correctly selecting the output stream in more than 90\% of cases. Regarding the remaining robust streams (i.e., S2 and S3), an acceptable accuracy of 70.8\% and 74.4\% is achieved, respectively. This reveals that the \ac{nn} used in the proposed method is able to predict the coded stream which leads to the highest video quality across different conditions with acceptable accuracy.
|
||||
|
||||
\begin{table}[tb]
|
||||
\centering
|
||||
@@ -342,7 +343,8 @@ The \ac{nn} is able to achieve a high accuracy for S1 and S4 streams, correctly
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
To further evaluate the performance of the \ac{nn}, the confusion matrix of the data set usef for testing is shown in the Table~\ref{Table6:ConfusionTest}. This table shows the same results as Table~\ref{Table6:ConfusionTraining} but for different test cases, which were not used in the training phase.
|
||||
To further evaluate the performance of the \ac{nn}, the confusion matrix of the data set used for testing is shown in the Table~\ref{Table6:ConfusionTest}.
|
||||
This table shows the same results as Table~\ref{Table6:ConfusionTraining}, but for different test cases, which were not used in the training phase.
|
||||
% Therefore, the trained network is not aware of the test cases used for validation.
|
||||
Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 62.2\% with a maximum value of 89.6\% when the S4 is the target stream. Although a lower accuracy is obtained using the validation test cases, the overall accuracy is still acceptable, since it allows to select the optimal robust coding technique (i.e., S1 to S4) for the majority of the conditions.
|
||||
In summary, the results presented in this sub-section reveal that the input parameters used in the proposed \ac{nn} are correlated with the decoded video quality, as they can effectively be used to perform data categorisation used in the selection of the optimal error robustness coding technique for the \ac{hevc} standard.
|
||||
@@ -354,7 +356,7 @@ In summary, the results presented in this sub-section reveal that the input para
|
||||
|
||||
In this section the performance of the proposed method is evaluated and discussed.
|
||||
The overall goal of the proposed method is to achieve improved quality using a combination of different robust coding techniques adaptively selected using a feed-forward \ac{nn}.
|
||||
The error resilience techniques used in the proposed method are used in the context of the \ac{hevc} standard and compared with the reference implementation (\REF).
|
||||
The error resilience techniques comprising the proposed method are used in the context of the \ac{hevc} standard and compared with the reference implementation (\REF).
|
||||
The quality obtained with the proposed optimal error robustness scheme (\PROP) is compared against the reference \ac{hevc} (\REF) and the methods used to generate the streams S1 to S4.
|
||||
|
||||
\begin{figure}[!t]
|
||||
@@ -366,12 +368,12 @@ The quality obtained with the proposed optimal error robustness scheme (\PROP) i
|
||||
\end{figure}
|
||||
|
||||
Figure~\ref{Figure6:ProposedQuality} shows the quality obtained for the proposed method and the \REF case for the main target bitrate shown in Table~\ref{TableTestSequences}. Moreover, the quality obtained with the target stream (\BEST) is also shown.
|
||||
One should note that the \BEST case corresponds to the optimal selection which can only be performed after quality evaluation at the decoder, therefore not suitable for real time applications.
|
||||
It is only used to establish the optimal case for comparison.
|
||||
One should note that the \BEST case corresponds to the optimal selection, which can only be performed after quality evaluation at the decoder, therefore it is not suitable for real time applications.
|
||||
It is only used to establish the optimal case for comparison purposes.
|
||||
The results in the figure allows to evaluate the impact in the decoded quality of the selection used in the proposed method and compare it with the optimal case.
|
||||
|
||||
As shown in Figure~\ref{Figure6:ProposedQuality} the proposed method clearly outperforms the reference \ac{hevc} for all \acs{plr}.
|
||||
Moreover, in the error-free case (\ac{plr}=0) it leads to imperceptible quality reduction since the proposed approach is able to select the optimal stream, thus disabling any robust coding technique when the transmission network does not have packet losses.
|
||||
As shown in Figure~\ref{Figure6:ProposedQuality}, the proposed method clearly outperforms the reference \ac{hevc} for all \acs{plr}.
|
||||
Moreover, in the error-free case (\ac{plr}=0), it leads to imperceptible quality reduction, since the proposed approach is able to select the optimal stream, thus disabling any robust coding technique when the transmission network does not have packet losses.
|
||||
In case of data loss, the results show that the proposed method is able to match the quality obtained by the \BEST case in most cases, revealing that the proposed method is able to efficiently select the optimal stream for transmission.
|
||||
|
||||
\begin{table}[tb]
|
||||
@@ -413,13 +415,16 @@ Subsequently, a comparison between the different methods is performed, in order
|
||||
%for every test case.
|
||||
Table~\ref{Table6:AverageGains} shows the average quality gains in comparison with the reference \ac{hevc} obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams S1 to S4. The average quality gains are measured across all test cases, i.e., different bitrates and different loss ratios.
|
||||
|
||||
Results in Table~\ref{Table6:AverageGains} shows that the robust coded streams S1 to S4 are able to achieve acceptable average quality gains up to 1.86 dB (see results for S2).
|
||||
Results in Table~\ref{Table6:AverageGains} shows that the robust coded streams S1 to S4 are able to achieve acceptable average quality gains up to 1.51 dB (see results for S2).
|
||||
However, since the implementation of robust coding techniques decreases the coding efficiency, it also leads to lower quality than the reference \ac{hevc}. This is noticeable in the results obtained for the Four People sequence in the streams S2 to S4 and for the Kristen and Sara sequence in the streams S3 and S4 (see red ellipses).
|
||||
The overall performance of such methods reveals that they are not suitable for transmission conditions with low \acs{plr}, e.g., lower than 3\% (higher number of conditions have \acs{plr} lower than 3\%).
|
||||
On the contrary, the proposed method is always able to outperform the reference \ac{hevc} and achieve quality gains of at least 0.25 dB for the Four People sequence. Moreover, for the Kimono and Race Horses sequences it is able to achieve the highest quality gains of up 3.41 dB and 4.39 dB, respectively (see blue ellipses). Summarising, by adaptively selecting the video stream, and consequently the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness achieving an average quality gain across different bitrates and \acs{plr} of 1.88 dB.
|
||||
On the contrary, the proposed method is always able to outperform the reference \ac{hevc} and achieve quality gains of at least 0.25 dB for the Four People sequence.
|
||||
Moreover, for the Kimono and Race Horses sequences, it is able to achieve the highest quality gains, of up 3.41 dB and 4.39 dB, respectively (see blue ellipses). Summarising, by adaptively selecting the video stream, and consequently the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness, achieving an average quality gain across different bitrates and \acs{plr} of 1.88 dB.
|
||||
|
||||
Finally, in order the performance evaluation of the proposed method is evaluated for a video sequence, which was not fed into the \ac{nn} training phase. This study aims at finding the accuracy of the optimal stream selection in case of new video content. For this purpose the Basketball Drive sequence was not used in the previous simulations. Table~\ref{Table6:AverageGainBasketBallDrive} shows the average quality results obtained with this sequence, similarly as Table~\ref{Table6:AverageGains}. These results confirm that the proposed method is able to select the optimal robust stream to be used for transmission, even when new video content is transmitted. The proposed method achieves a quality gain of 3.55 dB in comparison with the reference \ac{hevc}, outperforming the quality achieved with the stream S4 (i.e., 3.45 dB).
|
||||
This confirms the advantage of the proposed method to adaptively select among different stream with different levels of error protections.
|
||||
Finally, the performance evaluation of the proposed method is evaluated for a video sequence, which was not fed into the \ac{nn} training phase. This study aims at finding the accuracy of the optimal stream selection in case of a new video content. For this purpose the Basketball Drive sequence was randomly selected, and not used in previous simulations.
|
||||
Table~\ref{Table6:AverageGainBasketBallDrive} shows the average quality results obtained with this sequence, similarly to Table~\ref{Table6:AverageGains}.
|
||||
These results confirm that the proposed method is able to select the optimal robust stream to be used for transmission, even when new video content is transmitted. The proposed method achieves a quality gain of 3.55 dB in comparison with the reference \ac{hevc}, outperforming the quality achieved with the stream S4 (i.e., 3.45 dB).
|
||||
This confirms the advantage of the proposed method to adaptively select among different streams with different levels of error protections.
|
||||
|
||||
\begin{table}[tb]
|
||||
\centering
|
||||
@@ -442,8 +447,9 @@ This confirms the advantage of the proposed method to adaptively select among di
|
||||
\section{Summary}
|
||||
\label{Section6:Summary}
|
||||
|
||||
In this chapter an adaptive error robustness method is proposed based on reference frame selection and \ac{ec}-aware optimisation. By adaptively selecting a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the reference techniques.
|
||||
In this chapter an adaptive error robustness method is proposed, based on reference frame selection and \ac{ec}-aware optimisation. By adaptively selecting a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the reference techniques.
|
||||
%Moreover, the proposed method is able to outperform those cases where a single error resilience technique is used.
|
||||
The overall results demonstrate that the proposed method is able to increase the reconstruction video quality in case of errors, and by adapting the transmitted stream to the network conditions a better trade-off between coding efficiency and error robustness is achieved.
|
||||
The consistent quality improvement achieved, allows the use of this method in dynamic broadcasting video services where network feedback is available and where packet loss probability is not negligible.
|
||||
The overall results demonstrate that the proposed method is able to increase the reconstruction video quality in case of errors.
|
||||
Also, by adapting the transmitted stream to the network conditions, a better trade-off between coding efficiency and error robustness can be achieved.
|
||||
The consistent quality improvement, allows the use of this method in dynamic broadcasting video services where network feedback is available.
|
||||
This architecture is suitable for dynamic streaming technologies, e.g., HTTP streaming based on MPEG-DASH~\cite{HEVCHTTPStreamAug2016} or content delivery over heterogeneous networks using the \ac{mmt}~\cite{ATSCMMT2016}.
|
||||
|
||||
@@ -1,95 +0,0 @@
|
||||
|
||||
\usetikzlibrary{arrows,matrix}
|
||||
\usetikzlibrary{fit}
|
||||
\usetikzlibrary{intersections}
|
||||
\usetikzlibrary{intersections}
|
||||
% \usetkzobj{all}
|
||||
|
||||
\pgfdeclarelayer{background}
|
||||
\pgfdeclarelayer{foreground}
|
||||
\pgfsetlayers{background,main,foreground}
|
||||
|
||||
|
||||
\def\scale{1}
|
||||
|
||||
|
||||
\begin{tikzpicture}[
|
||||
font=\scriptsize,
|
||||
auto,
|
||||
node distance=1cm,
|
||||
scale=\scale,
|
||||
every node/.style={scale=\scale},
|
||||
template/.style={thick, text centered},
|
||||
block/.style={template, rectangle, text width=1.3cm, ,inner sep=3, minimum height=.6cm},
|
||||
arrow/.style={template, draw, -latex},
|
||||
point/.style={template, draw, fill=black, circle,inner sep=0, minimum size=3}
|
||||
]
|
||||
|
||||
% Define extra block styles
|
||||
\tikzstyle{sumBlock} = [template, draw, circle, inner sep=0cm, minimum size=0.3cm]
|
||||
\tikzstyle{propBlock} = [block, draw, fill=blue!15]
|
||||
\tikzstyle{refBlock} = [block, draw, fill=gray!10]
|
||||
\tikzstyle{propBlockNoDraw} = [block, fill=TikZColorFillProp]
|
||||
\tikzstyle{refBlockNoDraw} = [block, fill=TikZColorFill]
|
||||
|
||||
|
||||
\newcommand{\DrawEdgesTwoNodes}[2]{\path[arrow,-] (#1.south west) |- (#2.north east) |- (#1.south west);}
|
||||
|
||||
%% Encoder/Decoder
|
||||
\node [refBlockNoDraw, text width=1.1cm, minimum height=.6cm] (Encoder) at (0,0) {Encoder};
|
||||
\node [refBlockNoDraw, text width=0.6cm, minimum height=.6cm, right=-0.03 of Encoder.south east, anchor=south west, text centered] (EncoderNAL) {NAL};
|
||||
\node [refBlockNoDraw, text width=1.1cm, minimum height=.6cm, above=-0.03 of Encoder.north west, anchor=south west] (Decoder) {Decoder};
|
||||
\path[arrow,-] (Encoder.south west) |- (Decoder.north east) |- (EncoderNAL.north east) |- (Encoder.south west);
|
||||
\path[arrow,-,densely dotted] (EncoderNAL.south west -| Decoder.south east) -- (Decoder.south east); % Draw inside lines
|
||||
\path[arrow,-,densely dotted] (Decoder.south west |- EncoderNAL.north west) -- (EncoderNAL.north west); % Draw inside lines
|
||||
|
||||
|
||||
%% NALU Loss and EC optimisation
|
||||
\node [propBlock, above=2.2 of Encoder.west, anchor=south west, text width=3.8cm, minimum height=2.4cm] (ConcOpt) {Saliency-weighted EC Mode Decision};
|
||||
\node [propBlock, below=.8 of ConcOpt.south east, anchor=north east, text width=1.7cm, minimum height=.9cm] (LossSim) {NALU Loss Simulation};
|
||||
\path[arrow] (EncoderNAL -| LossSim.south) -- (LossSim) node[point] at (EncoderNAL -| LossSim) {};;
|
||||
\path[arrow] (Decoder) -- (ConcOpt.south -| Decoder) node[pos=.6,coordinate] (p2) {} node[point] at (p2) {};
|
||||
\path[arrow] (LossSim) -- (ConcOpt.south -| LossSim);
|
||||
|
||||
|
||||
%% EL encoder
|
||||
\node [propBlockNoDraw, right=4cm of ConcOpt.south, anchor=south west, text width=1.4cm, minimum height=1cm] (Scalable) {EL\\Encoder};
|
||||
\node [refBlockNoDraw, text width=.2cm, right=-.025 of Scalable.south east, anchor=south west, text centered, minimum height=1cm] (ScalableNAL) {N\\[-2pt]A\\[-2pt]L};
|
||||
\DrawEdgesTwoNodes{Scalable}{ScalableNAL}; % Draw border
|
||||
\path[arrow,-,densely dotted] (ScalableNAL.south west) -- (ScalableNAL.north west); % Draw inside lines
|
||||
|
||||
|
||||
%% Diff
|
||||
\node [sumBlock, left=.75 of Scalable] (ConcDiff) {$-$};
|
||||
\path[arrow] (ConcDiff) -- (Scalable) node[above, pos=0, xshift=.2cm] {$e^c$};
|
||||
\path[arrow] (ConcOpt.east |- ConcDiff) -- (ConcDiff) node[above, pos=1, xshift=-.3cm] {$\tilde f$};
|
||||
\path[arrow,-, shorten >=3] (p2) -- (p2 -| LossSim.north);
|
||||
\path[arrow, shorten <=3] (p2 -| LossSim.north) -| (ConcDiff) node[right, pos=1, yshift=-.3cm] {$\hat f$};
|
||||
|
||||
|
||||
%% EC mode coding
|
||||
\node [propBlockNoDraw, anchor=north west, text width=1.4cm, minimum height=1cm] at (ConcOpt.north -| Scalable.west) (Signalling) {EC Mode Coding};
|
||||
\node [refBlockNoDraw, text width=.2cm, right=-.025 of Signalling.south east, anchor=south west, text centered, minimum height=1cm] (SignallingNAL) {N\\[-2pt]A\\[-2pt]L};
|
||||
\DrawEdgesTwoNodes{Signalling}{SignallingNAL}; % Draw border
|
||||
\path[arrow,-,densely dotted] (SignallingNAL.south west) -- (SignallingNAL.north west); % Draw inside lines
|
||||
\path[arrow] (ConcOpt.east |- Signalling) -- (Signalling) node[above, pos=.45, yshift=-1] {\scriptsize M1...M4};
|
||||
|
||||
%% Input
|
||||
\node[text width=.5cm, text centered] at (-1.7,4.5) (Input) {Input video};
|
||||
\path[arrow] (Input) |- (Encoder);
|
||||
\path[arrow] (Input |- ConcOpt) -- (ConcOpt) node[point] at (Input |- ConcOpt) {};
|
||||
|
||||
%% TS Mux and output
|
||||
\path (Encoder.south) -- (Encoder |- Signalling.north) node[midway] (middleRef) {}; % Find middle ref
|
||||
\node [refBlock, right=8.5 of middleRef, text width=.7cm, minimum height=5.5cm] (NAL) {TS Mux};
|
||||
\node [right of=NAL, node distance=1.5cm, coordinate] (Out) {};
|
||||
|
||||
\path[arrow] (EncoderNAL) -- (NAL.west |- EncoderNAL) node[above,pos=1, xshift=-.4cm] {VCL$_0$};
|
||||
\path[arrow] (ScalableNAL) -- (NAL.west |- Scalable) node[above,pos=1, xshift=-.4cm] {VCL$_1$};
|
||||
\path[arrow] (SignallingNAL) -- (NAL.west |- Signalling) node[above,pos=1, xshift=-.4cm] {SEI};
|
||||
\path[arrow] (NAL) -- (Out) node[above,pos=1, xshift=-.2cm, text width=1.3cm] {Transport Stream};
|
||||
|
||||
|
||||
\end{tikzpicture}
|
||||
|
||||
|
||||
@@ -2,8 +2,6 @@
|
||||
\usetikzlibrary{arrows,matrix}
|
||||
\usetikzlibrary{fit}
|
||||
\usetikzlibrary{intersections}
|
||||
\usetikzlibrary{intersections}
|
||||
% \usetkzobj{all}
|
||||
|
||||
\pgfdeclarelayer{background}
|
||||
\pgfdeclarelayer{foreground}
|
||||
@@ -39,9 +37,6 @@
|
||||
|
||||
%% Encoder/Decoder
|
||||
\node [refBlock, text width=1.6cm, minimum height=1cm] (Encoder) at (0,0) {Reference \\ Encoder};
|
||||
% \node [refBlockNoDraw, text width=2.6cm, minimum height=.6cm, above=-0.03 of Encoder.north west, anchor=south west] (Decoder) {Decoder};
|
||||
% \path[arrow,-] (Encoder.south west) |- (Decoder.north east) |- (Encoder.south west);
|
||||
% \path[arrow,-,densely dotted] (Decoder.south west) -- (Decoder.south east); % Draw inside lines
|
||||
|
||||
%% RPS
|
||||
\node [propBlockNoDraw, text width=2.6cm, minimum height=1cm, right=.3 of Encoder, yshift=1.3cm] (EXP) {Reference Selection \\ Transcoding};
|
||||
@@ -62,58 +57,22 @@
|
||||
\path[MainArrow] (ConcOpt.east |- Scalable0) -- (Scalable0);
|
||||
\path[MainArrow] (ConcOpt.east |- Scalable1) -- (Scalable1);
|
||||
|
||||
%
|
||||
% %% EL encoder
|
||||
% \node [propBlockNoDraw, right=4cm of ConcOpt.south, anchor=south west, text width=1.4cm, minimum height=1cm] (Scalable) {EL\\Encoder};
|
||||
% \node [refBlockNoDraw, text width=.2cm, right=-.025 of Scalable.south east, anchor=south west, text centered, minimum height=1cm] (ScalableNAL) {N\\[-2pt]A\\[-2pt]L};
|
||||
% \DrawEdgesTwoNodes{Scalable}{ScalableNAL}; % Draw border
|
||||
% \path[arrow,-,densely dotted] (ScalableNAL.south west) -- (ScalableNAL.north west); % Draw inside lines
|
||||
%
|
||||
%
|
||||
% %% Diff
|
||||
% \node [sumBlock, left=.75 of Scalable] (ConcDiff) {$-$};
|
||||
% \path[arrow] (ConcDiff) -- (Scalable) node[above, pos=0, xshift=.2cm] {$e^c$};
|
||||
% \path[arrow] (ConcOpt.east |- ConcDiff) -- (ConcDiff) node[above, pos=1, xshift=-.3cm] {$\tilde f$};
|
||||
% \path[arrow,-, shorten >=3] (p2) -- (p2 -| LossSim.north);
|
||||
% \path[arrow, shorten <=3] (p2 -| LossSim.north) -| (ConcDiff) node[right, pos=1, yshift=-.3cm] {$\hat f$};
|
||||
%
|
||||
%
|
||||
% %% EC mode coding
|
||||
% \node [propBlockNoDraw, anchor=north west, text width=1.4cm, minimum height=1cm] at (ConcOpt.north -| Scalable.west) (Signalling) {EC Mode Coding};
|
||||
% \node [refBlockNoDraw, text width=.2cm, right=-.025 of Signalling.south east, anchor=south west, text centered, minimum height=1cm] (SignallingNAL) {N\\[-2pt]A\\[-2pt]L};
|
||||
% \DrawEdgesTwoNodes{Signalling}{SignallingNAL}; % Draw border
|
||||
% \path[arrow,-,densely dotted] (SignallingNAL.south west) -- (SignallingNAL.north west); % Draw inside lines
|
||||
% \path[arrow] (ConcOpt.east |- Signalling) -- (Signalling) node[above, pos=.45, yshift=-1] {\scriptsize M1...M4};
|
||||
%
|
||||
|
||||
\node [propBlock, minimum height=.8cm, text width=2cm, right=2 of Encoder, yshift=-1cm] (StreamParser) {Stream Parser};
|
||||
\path[MainArrow] (Encoder -| StreamParser) node[point] {} -- (StreamParser);
|
||||
|
||||
\node [propBlock, minimum height=.8cm, text width=2cm, below=.3 of StreamParser] (VideoParser) {Video analyser};
|
||||
|
||||
\node[text width=.7cm, text centered] (Input) at (-2,5) {Input video};
|
||||
|
||||
%% Input
|
||||
% \begin{scope}[xshift=-2.5cm, yshift=4cm]
|
||||
% \def\spacing{.2}
|
||||
% \coordinate(Input) at (0.5,0.3);
|
||||
% \foreach \i in {0,...,3}
|
||||
% \draw[thick,fill=TikZColorFill] (\spacing*\i,-.15*\i) rectangle ++(1,1);
|
||||
% % \node[text width=.7cm, text centered] at (1.6,1.1) {Video Stream};
|
||||
% \node[text width=.7cm, text centered] at (1.1,.1) {Input video};
|
||||
% \end{scope}
|
||||
% \path[MainArrow, shorten <=.9cm] (Input) |- (Encoder);
|
||||
% \path[MainArrow] (Input |- EXP) node[point] {} |- (EXP);
|
||||
% \path[MainArrow, shorten <=1.2cm] (Input.east |- ConcOpt) -- (ConcOpt);
|
||||
|
||||
\node[text width=.7cm, text centered] (Input) at (-2,4.3) {Input video};
|
||||
\path[MainArrow] (Input |- Encoder) node[point] {} |- (VideoParser);
|
||||
\path[MainArrow] (Input) |- (Encoder);
|
||||
\path[MainArrow] (Input |- EXP) node[point] {} |- (EXP);
|
||||
\path[MainArrow] (Input.east |- ConcOpt) -- (ConcOpt);
|
||||
\path[MainArrow] (Input |- ConcOpt) node[point] {} -- (ConcOpt);
|
||||
|
||||
|
||||
%
|
||||
% %% TS Mux and output
|
||||
% Mux and output
|
||||
\path (Encoder.south) -- (Encoder |- ConcOpt.north) node[midway] (middleRef) {}; % Find middle ref
|
||||
\node [propBlock, right=9 of middleRef, text width=5.5cm, rotate=-90, anchor=center, minimum height=1.1cm] (Mux) {Optimal stream selection};
|
||||
\node [right of=Mux, node distance=2.8cm, coordinate] (Out) {};
|
||||
@@ -128,22 +87,11 @@
|
||||
|
||||
\node [text centered, text width=1.5cm, minimum height=1cm, below=.3 of Mux.east, xshift=1.9cm] (Channel) {Channel Conditions};
|
||||
|
||||
% \path[SignallArrow] (StreamParser.9) -| (Mux.-12);
|
||||
% \path[SignallArrow] (StreamParser.0) -| (Mux.-8);
|
||||
% \path[SignallArrow] (StreamParser.-9) -| (Mux.-4);
|
||||
% \path[SignallArrow] (VideoParser.7) -| (Mux.2);
|
||||
% \path[SignallArrow] (VideoParser.-7) -| (Mux.6);
|
||||
% \path[SignallArrow] (Channel) -| (Mux.12);
|
||||
|
||||
|
||||
\path[SignallArrow] (StreamParser) -| (Mux.-6) node (b1) [pos=.3, anchor=center] {\footnotesize /} node[right=-.2 of b1, yshift=-6, inner sep=0] {\footnotesize 3};
|
||||
\path[SignallArrow] (VideoParser) -| (Mux.0) node (b2) at( b1 |- VideoParser) [anchor=center] {\footnotesize /} node[right=-.2 of b2, yshift=-6, inner sep=0] {\footnotesize 2};
|
||||
\path[SignallArrow] (Channel) -| (Mux.6);
|
||||
|
||||
% \node at ($ (ah) - (0.55, 0) $) {\scriptsize /};
|
||||
|
||||
|
||||
|
||||
\end{tikzpicture}
|
||||
|
||||
|
||||
|
||||
@@ -1,37 +1,61 @@
|
||||
|
||||
\chapter{\HIGHLIGHT{Conclusion and future work}}
|
||||
\chapter{Conclusion and future work \VERSION{0}}
|
||||
\label{Chapter:Conclusions}
|
||||
|
||||
This chapter concludes this thesis, presenting some conclusions about this research work and some future research perspectives in the area of video transmission, error resilience and error concealment.
|
||||
\glsresetall
|
||||
\glsunset{h264}
|
||||
|
||||
%\vspace{10mm}
|
||||
\section{Conclusions}
|
||||
|
||||
% This section presents a summary of the of the chapters presented before and a discussion of the contributions presented.
|
||||
%
|
||||
% Chapter~\ref{CHAP:3DVideo} presented a review on the main technologies for 3D video systems related with the formats for representation, coding methods and proposed storage and transmission systems. This chapter also describes the main features of the MVC extension of the H.264/MVC standard, which is the coding standard for the most 3D applications \mbox{(\emph{e.g.}, Blu-ray)}. Moreover, as this research focuses in transmission errors and error concealment, a review of some concealment methods proposed over the last years was presented in the Chapter~\ref{CHAP:Concealment}.
|
||||
%
|
||||
% The research work presented in this thesis leaded
|
||||
% %different contributions
|
||||
% to the development of an advanced 3D video transmission system. More specifically, our research focused on the study of the subjective impacts of transmission errors in 3D video, when the auxiliary view is subject to frame loss. These results were presented in Chapter~\ref{CHAP:SubjectiveEval}. The results demonstrated that the disparity is intrinsically related with the quality of the concealed 3D video, and that errors have higher effects in those sequences with higher disparity. Moreover, for longer error bursts, the results showed that it is better to switch into 2D visualisation rather than trying to restore the 3D effect by using frame concealment methods.
|
||||
%
|
||||
% The results presented in the Chapter~\ref{CHAP:SubjectiveEval} leaded to further experiments in order to evaluate the objective quality of the transmitted 3D video using different objective metrics. A transmission framework was presented in Chapter~\ref{CHAP:HierarchicalDVBT3D} and the quality evaluation was performed. The quality models, devised from the analysis of the quality results, are a novel contribution to the research community and also for the network planners. Through those models, estimated values of the quality using different metrics can be obtained from the packet loss ratio of the transmission channel. This can be easily measured at the receiver side. The quality models are able to cover different video quality features, as the signal-to-noise relation, the structural similarity and a more specific feature of the 3D video, the stereo perception.
|
||||
% The tools developed are part of the first contribution of this research work, which resulted in an efficient 3D receiver to cope with transmission errors in 3D video streams, delivered using the MPEG-2 TS encapsulation. Moreover, as part of the receiver side, an efficient 3D video decoder was achieved, that is able to detect and conceal the missing frames using different concealment techniques. This set of tools provides relevant knowledge, for further investigations in quality evaluation and error concealment.
|
||||
%
|
||||
% The second contribution of this research work was the development of efficient concealment methods, able to recover the motion field for missing frames in 3D video. Assuming the hierarchical transmission system, as presented in Chapter~\ref{CHAP:HierarchicalDVBT3D}, the proposed methods are an efficient alternative to the existing ones. They are mainly based on the information presented in the error-free view (\emph{i.e.}, the base view), since these information can provide more accurate clues to estimate the loss data in the auxiliary view, as described in Chapter~\ref{CHAP:PropFrameLossConc}. The presented results showed the effectiveness of the proposed techniques, when compared with conventional motion-based algorithms, achieving an average gain of $1.89\ dB$ in random frame loss and up to $6\ dB$ of gain in the objective quality of the reconstructed frame. Moreover, an improvement to the proposed method was devised, and a motion based decision approach was incorporated, combining both methods, thus allowing to improve the quality of the reconstructed frames up $2\ dB$.
|
||||
%
|
||||
% One may conclude that the research on alternative frame loss concealment techniques may bring new insights to achieve a more efficiently video decoding in the presence of error prone channels. Moreover, the subjective study presented in this dissertation provided relevant subjective factors that may be used in future developments of error concealment techniques. New representation formats for 3D video and novel coding approaches are being proposed, leading to new error concealment problems, and different ways to recovery the missing information, thus, the research work in this area should continue.
|
||||
This thesis investigated new algorithms and techniques for efficient error robustness in highly efficiency video coders.
|
||||
The importance of these techniques is justified by the spread of digital video contents, including high resolution formats, multi-view and 360-degree video formats, which significantly the network bandwidth requirements. Consequently, errors and data loss may occur along the transmission system.
|
||||
Moreover, in order to meet the network bandwidth constrains video coding standard have been improved, by including advanced predictions techniques, which results in increased bitstream dependencies.
|
||||
This increase the negative impact of errors, leading to higher quality degradation.
|
||||
To overcome this problem, in this thesis different methods were proposed, addressing the problem of robust coding and efficient \ac{ec}.
|
||||
%Subsequently, for each chapter of this thesis, the main conclusions are presented.
|
||||
|
||||
|
||||
\section{Future work}
|
||||
Firstly, an introduction of the main motivation for this research work is presented, enumerating its main objectives and contributions.
|
||||
Then, the main functional blocks of the state-of-the-art \ac{hevc} are described.
|
||||
This description covered the high-level features, such as the new picture types and the slice partitioning schemes, the new block partitioning scheme, which allow multi-level hierarchical structures with size-independent representation, the main predictions techniques, including the newly introduced advanced motion vector prediction technique.
|
||||
This aimed at introducing the novel techniques in \ac{hevc}, which may have a relevant impact on the error robustness of the coded streams.
|
||||
The main algorithms proposed in the past to increase the error robustness of video streams were presented, describing techniques based on error resilience \ac{rd} optimisation, redundant based approaches and \ac{ec}-based methods.
|
||||
It was concluded that, although the previous works provide efficient techniques to reduce the bitstream dependencies and ease the error recovery, the \ac{hevc} standard introduces new problems that could not be fully addressed with existing techniques.
|
||||
Such problems range from new \ac{mv} coding techniques to the lack of robust coding techniques, such as \ac{fmo} or data partitioning.
|
||||
Moreover, there was also open issues regarding robust \ac{ec}-based encoding, as existing method do not fully exploit the use of different methods with different levels of complexity, especially the use of advanced \ac{ec} techniques.
|
||||
|
||||
% In this research work, the technologies and the subjective quality impact of errors in 3D video were studied. Nevertheless, there are still a number of topics that require further investigation, and may lead to more efficient 3D video decoders, namely:
|
||||
% \begin{itemize}
|
||||
% \item \textbf{3D video quality models extension to the channel level.} The 3D video quality models presented in Chapter~\ref{CHAP:HierarchicalDVBT3D} revealed to match with the experimental data. Thus, the combination with the hierarchical DVB-T channel models could extend the proposed models to a new level, where the quality of the 3D video could be estimated from the channel conditions, instead of the packet loss. This would extend the relevance of the proposed models to the network planners and developers.
|
||||
%
|
||||
% \item \textbf{Extension of the proposed method to a multi-view scenario.} The proposed concealment methods, described in Chapter~\ref{CHAP:PropFrameLossConc}, revealed good performance to conceal missing frames in 3D video (\emph{i.e.}, stereoscopic video). Therefore, the same techniques are expected to have similar behaviour under a multi-view scenario, where, besides the base view frames, other auxiliary view frames that may provide motion information clues to estimate the lost frames' motion field.
|
||||
%
|
||||
% \item \textbf{Extend the concealment algorithm to MVD coding scenario.} Although the proposed concealment methods did not focus in the recovery of the depth information, they could be used to recover texture in multi-view plus depth video signals. In this case, besides disparity information, depth maps are available to extract relevant information, for example, to detect occlusions and errors in the disparity map. The performance of the proposed method could also be investigated to reconstruct the lost depth map.
|
||||
%
|
||||
% \item \textbf{Performance evaluation of the concealment methods in a MDC.} The multi-description coding approach is being investigated over the last years, as a novel alternative for the emergent multi-path networks. The algorithms presented in the proposed method have great flexibility and could be applied under different coding scenarios. Moreover, since MDC introduces new possibilities to error concealment by using the redundant information in each description, such characteristic will be used to investigate how different types of redundancy can be jointly used with the proposed concealment techniques to enhance the quality of received views.
|
||||
% \end{itemize}
|
||||
To investigate new techniques for robust video coding an evaluation study was firstly carried out, which aimed at finding the influence of the new coding tools on the error resilience of the \ac{hevc} streams.
|
||||
Existing studies only performed generic comparisons between \ac{hevc} and previous standards (e.g., \ac{h264}).
|
||||
This study extended existing ones by considering different coding configurations, which allowed to evaluate the impact of new coding tools. The study revealed that some features, such as increased block size, does not have significant impact of the error robustness performance.
|
||||
However, slice partitioning and intra-refreshing have a noticeable impact on both, coding efficiency and error robustness.
|
||||
Therefore, a small compromise in coding efficiency should be reasoned, in order to achieve an acceptable quality in case of errors.
|
||||
Finally, one can also conclude that the use of temporal predictions in \ac{mv} coding significantly increases temporal dependencies, leading to increased quality degradation compromising the performance of an \ac{hevc}-based video transmission system.
|
||||
|
||||
Based on the above evidences, a new \ac{mv} coding scheme was used in \ac{hevc}, aiming at reducing the temporal dependencies.
|
||||
In this scheme the use of the temporal \ac{mv} candidate is selectively used in some \acs{pu}, preventing temporal dependencies across several frames.
|
||||
As this method significantly reduces the temporal dependencies, the number of mismatched \acs{mv} is reduced, consequently reducing the error propagation.
|
||||
Moreover, as the temporal candidate is not fully disabled the impact on coding efficiency is reduced.
|
||||
Experimental results showed a clear improvement on decoded video quality, across different video content, with a small bitrate increase.
|
||||
|
||||
|
||||
The reference frame distribution in highly efficient video coders was investigated, as it may increase the number of mismatch predictions in case of erroneous reference frames.
|
||||
Since the reference frame usage is consistently unbalanced, a new method based on constrained \ac{rd} optimisation was proposed to equally distribute the use of multiple reference frames across neighbouring \acs{pu}.
|
||||
An extension using selective \ac{mv} redundancies was also investigated, in order to recover from erroneous motion predictions.
|
||||
This resulted in a two-fold approach to reduce error propagation.
|
||||
As part of the proposed method can be applied at the streaming-stage, using a low complexity parser, this method can be used to adapt existing streams to error-prone transmission channels.
|
||||
The proposed method was compared against existing techniques, and the simulation results revealed a consistently superior performance of using the two-fold approach, making it suitable for video services where packet loss cannot be negligible.
|
||||
|
||||
To address the problem of robust video decoding and increase the quality of the reconstructed frames, \ac{ec}-aware resilience techniques were investigated.
|
||||
To extend existing techniques, a saliency-based \ac{ec} optimisation was proposed to optimally select the \ac{ec} which should be applied at the decoder in case of frame loss. This information is transmitted to the decoder as side-information.
|
||||
Besides a saliency estimation, a coefficient based on the temporal duration of the error propagation was also considered, in order to keep the signalling overhead affordable. Finally, a small bit allocation was used to transmit an \ac{el}, which was used to improve the reconstruction quality, in those regions where the optimal \ac{ec} was not accurate.
|
||||
Experimental results revealed that using different \ac{ec} algorithms increase the performance of the \ac{ec}-aware coding scheme and achieve superior performance when compared with existing methods.
|
||||
|
||||
Finally, the problem of adaptive robust coding was investigated, in order to achieve superior quality with the error robustness techniques proposed in this research work.
|
||||
Preliminary results showed that the robust coding technique which obtains the higher quality depends on different factors, such as video content, bitrate and network conditions.
|
||||
Therefore, an adaptive scheme based on six input parameters and a pre-trained \ac{nn} was proposed, to predicts the video stream that is able to achieve the highest decoded quality.
|
||||
Results revealed that using the proposed scheme, an efficient trade-off between coding efficiency and error resilience is achieved for the majority of test conditions.
|
||||
Using this architecture, the problem of robust video coding and decoding is addressed using a combination of different methods, which leads to a superior performance when compared with individual use of a single technique.
|
||||
|
||||
As a final remark, one may conclude that the use of novel methods, proposed in the context of this research work, are able to increase the robustness to transmission errors of the state-of-the-art video coding standard.
|
||||
This was jointly achieved by achieving implementing robust video coding techniques with efficient error concealment algorithms optimised at the encoder.
|
||||
|
||||
|
||||
\subsection*{Future work}
|
||||
@@ -77,10 +77,7 @@
|
||||
\newcommand{\yfshift}{0.05cm}
|
||||
\definecolor{TikZColorFill}{RGB}{220,220,220}
|
||||
\definecolor{TikZColorFillDark}{RGB}{120,120,120}
|
||||
%\definecolor{TikZColorFillProp}{RGB}{159, 161, 249}
|
||||
\definecolor{TikZColorFillProp}{RGB}{86, 151, 255}
|
||||
%\definecolor{colorBackground}{RGB}{194,215,249}
|
||||
%\definecolor{colorBackgroundSecondary}{RGB}{158,195,255}
|
||||
\definecolor{TikZColorFillProp}{RGB}{136, 180, 255}
|
||||
\definecolor{TikZColorBlock0}{RGB}{0,95,120}
|
||||
\definecolor{TikZColorBlock1}{RGB}{0,160,200}
|
||||
\def\TikZyslant{-0.2}
|
||||
|
||||
+10
-18
@@ -18,7 +18,7 @@
|
||||
% \newcommand{\HIGHLIGHT}[1]{\textcolor{red}{#1}}
|
||||
\newcommand{\HIGHLIGHT}[1]{\textcolor{red}{#1}}
|
||||
\newcommand{\HERE}{\textcolor{red}{CHECK HERE!}}
|
||||
\newcommand{\VERSION}[1]{\textcolor{red}{r#1}}
|
||||
\newcommand{\VERSION}[1]{\textcolor{red}{(r#1)}}
|
||||
|
||||
%\includeonly{010_Introduction/Introduction}
|
||||
%\includeonly{021_VideoCoding/VideoCodingReview}
|
||||
@@ -27,6 +27,7 @@
|
||||
%\includeonly{041_ImprovedErrorResilience/ImprovedErrorResilience}
|
||||
%\includeonly{042_ConcealmentAwareEncoding/ConcealmentAwareEncoding}
|
||||
%\includeonly{043_EndToEndRobustSystem/EndToEndRobustSystem}
|
||||
%\includeonly{050_Conclusions/Conclusion}
|
||||
|
||||
\begin{document}
|
||||
|
||||
@@ -44,23 +45,20 @@
|
||||
\titlefigs{\includegraphics[width=10cm]{lborolodonlogo.png}}
|
||||
|
||||
% make the title area
|
||||
\listoftodos
|
||||
%\listoftodos
|
||||
|
||||
\maketitle
|
||||
|
||||
\frontmatter
|
||||
% PDF exports of Word Documents available (Exported August 2012)
|
||||
% Thesis Access Form
|
||||
\clearpage
|
||||
\includepdf[pages=1, pagecommand=]{000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf}
|
||||
\cleardoublepage
|
||||
\includepdf[pages=2, pagecommand=]{000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf}
|
||||
% Certificate of Originality
|
||||
%\includepdf[pages=-, pagecommand=, templatesize={5in}{10in}]{000_Preamble/LU/origin.pdf}
|
||||
%\clearpage
|
||||
\includepdf[pagecommand=]{000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf}
|
||||
%\cleardoublepage
|
||||
%\includepdf[pages=2, pagecommand=]{000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf}
|
||||
|
||||
%\input{000_Preamble/dedic}
|
||||
%\input{000_Preamble/thanks}
|
||||
%\input{000_Preamble/abstract}
|
||||
\input{000_Preamble/dedic}
|
||||
\input{000_Preamble/thanks}
|
||||
\input{000_Preamble/abstract}
|
||||
|
||||
%\dominitoc
|
||||
\glsunsetall
|
||||
@@ -71,20 +69,14 @@
|
||||
\glsresetall
|
||||
|
||||
\mainmatter
|
||||
% \pagenumbering{gobble}
|
||||
|
||||
%\setcounter{mtc}{6}
|
||||
\include{010_Introduction/Introduction}
|
||||
|
||||
\include{021_VideoCoding/VideoCodingReview}
|
||||
\include{022_ErrorResilience/RobustTransmissionReview}
|
||||
\include{031_ErrorResilienceEvaluation/ErrorResilienceEvaluation}
|
||||
\include{041_ImprovedErrorResilience/ImprovedErrorResilience}
|
||||
\include{042_ConcealmentAwareEncoding/ConcealmentAwareEncoding}
|
||||
\include{043_EndToEndRobustSystem/EndToEndRobustSystem}
|
||||
|
||||
\include{050_Conclusions/Conclusion}
|
||||
|
||||
\appendix
|
||||
\include{060_Appendix/Sequences}
|
||||
\include{060_Appendix/Papers}
|
||||
|
||||
+15
-14
@@ -250,31 +250,32 @@ T\kern-.1667em\lower.5ex\hbox{E}\kern-.125emX\spacefactor1000}
|
||||
\pagestyle{plain}
|
||||
\titleformat{\chapter}[display]
|
||||
{\normalfont\Large\bfseries}
|
||||
{\sc\chaptertitlename\ \thechapter}{15pt}
|
||||
{\sc\chaptertitlename\ \thechapter}{20pt}
|
||||
{\Huge \thispagestyle{empty}}
|
||||
[\vspace{5mm}\titlerule]
|
||||
[]
|
||||
}
|
||||
\renewcommand\mainmatter{%
|
||||
\estg@mainBegin
|
||||
\cleardoublepage
|
||||
\@mainmattertrue
|
||||
% \titleformat{\chapter}[display]
|
||||
% {\normalfont\Large\bfseries}
|
||||
% {\sc\chaptertitlename\ \thechapter}{15pt}
|
||||
% {\Huge \thispagestyle{empty}}
|
||||
% [\vspace{1cm}\titlerule\minitoc\glsresetall]
|
||||
\titleformat{\chapter}[display]{\normalfont\Huge\bfseries}{\sc \chaptertitlename\ \thechapter}{20pt}{\Huge \thispagestyle{empty}}[\vspace{5mm}]
|
||||
%\titlerule\minitoc\glsresetall
|
||||
\titleformat{\chapter}[display]
|
||||
{\normalfont\Huge\bfseries} % format
|
||||
{\sc \chaptertitlename\ \thechapter} % label
|
||||
{20pt} % sep
|
||||
{\Huge \thispagestyle{empty}} % before
|
||||
[\vspace{5mm}]
|
||||
\properpagestyle
|
||||
\pagenumbering{arabic}}
|
||||
|
||||
\renewcommand\backmatter{%
|
||||
\cleardoublepage
|
||||
\addcontentsline{toc}{chapter}{\name@bib}
|
||||
\titleformat{\chapter}[display]
|
||||
{\normalfont\Large\bfseries\raggedleft}
|
||||
{\sc\chaptertitlename\ \thechapter}{15pt}
|
||||
{\Huge \thispagestyle{empty}}
|
||||
[\vspace{1cm}\titlerule]
|
||||
% \titleformat{\chapter}[display]
|
||||
% {\normalfont\Large\bfseries\raggedleft}
|
||||
% {\sc\chaptertitlename\ \thechapter}{15pt}
|
||||
% {\Huge \thispagestyle{empty}}
|
||||
% [\vspace{1cm}\titlerule]
|
||||
}
|
||||
|
||||
\newcommand\titlefigs[1]{%
|
||||
@@ -398,7 +399,7 @@ T\kern-.1667em\lower.5ex\hbox{E}\kern-.125emX\spacefactor1000}
|
||||
\thispagestyle{empty}
|
||||
\vspace*{\fill}
|
||||
\begin{flushright}
|
||||
\begin{minipage}{60mm}
|
||||
\begin{minipage}{70mm}
|
||||
\raggedleft \it \normalsize \@dedic
|
||||
\end{minipage}
|
||||
\end{flushright}
|
||||
|
||||
Reference in New Issue
Block a user