Fix abreviations and equations referencing
This commit is contained in:
@@ -6,23 +6,23 @@
|
||||
|
||||
\vspace{-5mm}
|
||||
|
||||
This thesis investigates the problem of robust coding and \ac{ec} in highly efficient video coders.
|
||||
After a review of the current state-of-art, a simulation study about error robustness, revealed that the \ac{hevc} has weak protection against transmission errors with significant impact on video quality degradation.
|
||||
This thesis investigates the problem of robust coding and Error Concealment (EC) in \HIGHLIGHT{highly efficient video coders}.
|
||||
After a review of the current state-of-art, a simulation study about error robustness, revealed that the High Efficiency Video Coding (HEVC) has weak protection against transmission errors with significant impact on video quality degradation.
|
||||
Based on this evidence, the first contribution of this work is a new method to reduce the temporal dependencies between \acs{mv}, by improving the decoded video quality without compromising the compression efficiency.
|
||||
|
||||
The second contribution of this thesis is a two-stage approach for reducing the mismatch of temporal predictions in case of video streams received with errors or lost data.
|
||||
At the encoder stage, the reference pictures are dynamically distributed based on a constrained Lagrangian \ac{rd} optimization to reduce the number of predictions from a single reference.
|
||||
At the encoder stage, the reference pictures are dynamically distributed based on a constrained Lagrangian rate-distortion optimization to reduce the number of predictions from a single reference.
|
||||
At the streaming stage, a prioritization algorithm, based on spatial dependencies, selects a reduced set of MVs to be transmitted, as side information, to reduce mismatched MV predictions at the decoder.
|
||||
|
||||
The problem of EC-aware video coding is also investigated to enhance the overall error robustness.
|
||||
A new approach based on scalable coding and optimally EC selection is proposed, where the optimal EC modes are found by simulating transmission losses, followed by a saliency-weighted \ac{rd} optimisation.
|
||||
Moreover, EC residual information is encoded using a rate-controlled \ac{el}.
|
||||
A new approach based on scalable coding and optimally EC selection is proposed, where the optimal EC modes are found by simulating transmission losses, followed by a saliency-weighted optimisation.
|
||||
Moreover, EC residual information is encoded using a rate-controlled enhancement layer.
|
||||
Both are transmitted to the decoder to be used in case of data loss.
|
||||
|
||||
Finally, an adaptive error resilience scheme is proposed to dynamically predict the video stream that achieves the highest decoded quality for a particular loss case.
|
||||
A pre-trained neural network selects among the various video streams, encoded with different levels of compression efficiency and error protection, based on information from the video signal, the coded stream and the transmission network.
|
||||
|
||||
Overall, the new robust video coding methods investigated in this thesis yield consistent quality gains in comparison with other existing methods and also the ones implemented in the \ac{hevc} reference software.
|
||||
Overall, the new robust video coding methods investigated in this thesis yield consistent quality gains in comparison with other existing methods and also the ones implemented in the HEVC reference software.
|
||||
Furthermore, the trade-off between coding efficiency and error robustness is also better in the proposed methods.
|
||||
|
||||
%\vspace{2mm}
|
||||
|
||||
@@ -1,22 +1,18 @@
|
||||
\graphicspath{{010_Introduction}}
|
||||
|
||||
\glsreset{hevc}
|
||||
\chapter{Introduction \VERSION{3}}
|
||||
\label{Chapter:Intro}
|
||||
|
||||
\section{Context and motivation}
|
||||
|
||||
\glsreset{h264}
|
||||
\glsunset{itut}
|
||||
\glsunset{iso}
|
||||
\glsunset{iec}
|
||||
|
||||
|
||||
In recent years new advances in video compression and networking technologies have enabled the development of many multimedia applications over the internet, such as video conferencing, video streaming, video publishing, among others.
|
||||
|
||||
In recent years new advances in video compression and networking technologies have enabled the development of many multimedia applications over the Internet, such as video conferencing, video streaming, video publishing, among others.
|
||||
The increasing diversity of services, the demand for high quality multimedia and the introduction of \ac{uhd} formats (\emph{e.g.}, $4k$ or $8k$ resolution and beyond) are pushing forward existing technologies and opening new research directions.
|
||||
Moreover, the emerging new video formats with increased resolutions have created stronger needs for higher coding efficiency, beyond the previous standards, i.e., the \ac{h264}~\cite{H264AVCStandard2003,H264OverviewJul2003}.
|
||||
Such need has been driven not only by higher resolutions but also by stereoscopic and multi-view video formats, which increase the number of viewpoints that need to be encoded and delivered simultaneously.
|
||||
Moreover, the ever increasing traffic generated by mobile multimedia applications \cite{CiscoMobile1621_Feb2017} and different video services with growing popularity, e.g., Netflix, impose new challenges to the existing networks.
|
||||
Such need has been driven not only by higher resolutions but also by 360-degree, stereoscopic and multi-view video formats, which increase the number of viewpoints that need to be encoded and delivered simultaneously.
|
||||
Moreover, the ever increasing traffic generated by mobile multimedia applications \cite{CiscoMobile1621_Feb2017} and different video services with growing popularity (e.g., Netflix and Facebook) impose new challenges to the existing networks.
|
||||
To cope with such demanding requirements a new coding standard was developed by the \ac{itut} \ac{vceg} and the \ac{iso}/\ac{iec} \ac{mpeg} standardisation organisations, referred to as H.265 - \ac{hevc}~\cite{H265StandardFeb2018,HEVCOverviewDez2012}.
|
||||
|
||||
\glsunset{h264}
|
||||
|
||||
Binary file not shown.
@@ -13,17 +13,12 @@
|
||||
\vspace*{\fill}
|
||||
\section{Introduction}
|
||||
|
||||
\glsunset{h264}
|
||||
\glsreset{hevc}
|
||||
|
||||
This chapter presents an overview of the \ac{hevc}~\cite{H265StandardFeb2018} standard, focusing on the most relevant features of the encoding algorithm, considering the scope of the work developed in this thesis.
|
||||
In short, the \ac{hevc} standard essentially aims to support all existing applications of \ac{h264}~\cite{H264AVCStandard2003}, while extending two keys aspects: increased video resolution with higher compression ratio and also more efficient use of parallel processing architectures.
|
||||
In order to increase the coding flexibility and higher video resolutions, the \ac{hevc} standard adopts a new block partition structure, as well as, new prediction modes.
|
||||
In this chapter the new data structures and coding techniques, specifically introduced to increase the coding efficiency, are described.
|
||||
Particular emphasis is given to those which may have a significant impact in the error robustness characteristics of \ac{hevc} streams.
|
||||
|
||||
\glsunset{hevc}
|
||||
|
||||
The chapter is organised as follows.
|
||||
Section~\ref{Section2:GeneraVideoSystem} introduces the functional blocks of a generic hybrid video coding system and Section~\ref{Section2:Overview} provides an overview of the \ac{hevc} standard.
|
||||
Then, Section~\ref{Section2:HighLevel} describes the high-level syntax elements, which are used to carry relevant information in coded streams.
|
||||
@@ -69,9 +64,9 @@ This is the reason why the coding scheme is called hybrid, as it combines quiet
|
||||
|
||||
In general, there are two kinds of redundancy in video signal: spatial and temporal.
|
||||
On the one hand, the spatial redundancy is exploited through intra frame prediction, where the underlying mechanism is to find the best matching block for the current one, by searching the neighbouring region.
|
||||
This is possible due to the existence of high correlation between neighbouring pixels within a frame.
|
||||
This is possible due to the existence of high correlation among neighbouring pixels within a frame.
|
||||
As result, when intra frame prediction is used, a block of pixels is reconstructed from the neighbouring blocks previously processed.
|
||||
On the other hand, inter frame prediction is designed to exploit the temporal redundancy that exists between consecutive frames, such as those temporally adjacent, e.g., the past frame or the future frame in regard to the current one.
|
||||
On the other hand, inter frame prediction is designed to exploit the temporal redundancy that exists among consecutive frames, such as those temporally adjacent, e.g., the past frame or the future frame in regard to the current one.
|
||||
Temporal redundancy is normally referred to as the correlation between two adjacent frames since they usually have quiet similar visual content, such as foreground objects and background.
|
||||
Thus, the purpose of inter-frame prediction is to search for the best matching block for the current one in a set of reference frames.
|
||||
|
||||
@@ -101,7 +96,7 @@ In the next section, the aforementioned blocks are described in the context of t
|
||||
|
||||
Similar to the previous coding standards developed by \ac{jctvc}, in \ac{hevc}~\cite{HEVCOverviewJul2012,HEVCOverviewDez2012,BookHEVC2014} only the
|
||||
coding syntax and semantics are standardised.
|
||||
The constrains and functionalities are defined through profiles and levels, while the semantic meaning of syntax elements define the decoding process, such that every decoder conforming to the standard generates the same output when given a complaint stream.
|
||||
The constrains and functionalities are defined through profiles and levels, while the semantic meaning of syntax elements define the decoding process, such that every decoder conforming to the standard generates the same output when given a conforming stream.
|
||||
Such delimitation of scope in the standard concede maximum freedom for developers and manufactures to develop proprietary optimisations, allowing to cope with various applications’ requirements (\emph{e.g.}, balancing the compression ratio and the coding complexity).
|
||||
Figure~\ref{fig:HEVCBlockDiagram} illustrates the block diagram of the \ac{hevc} encoder. It follows the same overall hybrid architecture as its predecessors starting from H.261.
|
||||
The encoder uses both intra- and inter-picture prediction to exploit spatial and temporal redundancies, respectively.
|
||||
@@ -119,7 +114,7 @@ Subsequently, the prediction residue is transformed by linear spatial transform
|
||||
However, there are some novelties introduced in the \ac{hevc} standard. As depicted in Figure~\ref{fig:HEVCBlockDiagram}, the encoding process follows a block-based approach, therefore each picture is partitioned into larger blocks, referred to as \acs{ctu}, with a maximum size of $64\times64$ pixels.
|
||||
Subsequently, a dynamic quad-tree partitioning is used to divide the picture into \acs{cu}.
|
||||
These units are introduced as the basic processing units for prediction, while for the transform and quantisation operations, \acs{tu} are used.
|
||||
In order to generate the prediction residue, the \ac{hevc} may use one of 34 intra-prediction modes or take advantage of symmetric and asymmetric partitions for motion estimation and inter-prediction.
|
||||
In order to generate the prediction residue, the \ac{hevc} may either use one of 34 intra-prediction modes or take advantage of symmetric and asymmetric partitions for motion estimation and inter-prediction.
|
||||
The motion estimation is performed in the pictures listed in the \ac{rpset}, which was introduced in the \ac{hevc} to increase the robustness to errors and data loss of reference picture management.
|
||||
To improve the overall quality of the reconstructed pictures and smooth the coding artefacts, \ac{hevc} not only uses a deblocking filter but also a \ac{sao} filter. Finally, the control data, quantised coefficients, prediction modes, motion vectors and filter information are compressed using the \ac{cabac}. The compressed bitstream is then allocated to various \ac{nal} units.
|
||||
|
||||
@@ -170,7 +165,7 @@ Figure~\ref{Figure2:NalHeader} shows the structure of the \ac{hevc} \ac{nal} uni
|
||||
|
||||
\glsreset{vps}
|
||||
|
||||
The encoder uses a set of signalling packets as a robust method to share data with the decoder, referred to as parameter set units. This information was introduced in previous standards to deal with vulnerabilities in the transport of signalling information between the encoder and decoder, which may compromise the streaming applications.
|
||||
The encoder uses a set of signalling packets as a robust method to share data with the decoder, referred to as parameter set units. This information was introduced in previous standards to deal with vulnerabilities in the transport of signalling information between the encoder and decoder.
|
||||
In this approach, such set of parameters is multiplexed in the bitstream and repeated as many times as required by the application, e.g., streaming, storage among others. In \ac{hevc}, this structure is inherited from \ac{h264}\cite{H264OverIPJul2003} with modifications to cope with new coding tools.
|
||||
The \ac{hevc} includes three sets of parameters, \ac{sps}, \ac{pps} and the newly introduced \ac{vps}\cite{HEVCHighLevelReferenceDez2012}.
|
||||
% \ac{sps} include sequence related information, that include the decoder operation point, flags to enable or disable certain tools within a given profile. \Gls{pps} include picture related information, that may change for every picture, e.g., \ac{qp} and tilling information.
|
||||
|
||||
@@ -125,7 +125,7 @@ E\left\{\left( \tilde f_t^i \right)^2 \right\} = \left(1-p\right) \left( \left(
|
||||
\end{equation}
|
||||
assuming that the pixel $i$ is predicted from pixel $j$ in the previous frame.
|
||||
The prediction error $e_t^i$ is quantised to the value $\hat e_t^i$, which is transmitted to the decoder along with the motion information.
|
||||
Equations~\ref{EQ2:E2EDIntra} to~\ref{EQ2:E2EDInter} provide a measure of the distortion, taking into consideration not only the coding distortion but also the possibility of frame loss and consequent \ac{ec} distortion.
|
||||
Equations~(\ref{EQ2:E2EDIntra}) to~(\ref{EQ2:E2EDInter}) provide a measure of the distortion, taking into consideration not only the coding distortion but also the possibility of frame loss and consequent \ac{ec} distortion.
|
||||
|
||||
This method was used in \cite{ROPEJun2000} to switch between intra and inter coding modes with the aim of reducing the temporal dependencies, thus achieving enhanced error robustness in video transmission.
|
||||
In \cite{AdvancesROPEHE2642007} this was optimised to quarter pixel estimation, and in \cite{ROPEBurst2009} the same method was extended for burst losses, by introducing the burst loss probability into the original end-to-end estimation.
|
||||
@@ -199,7 +199,7 @@ In such approach, the reconstructed reference frames are scaled down by applying
|
||||
\check f_t^i = \alpha \hat f_t^i + (1- \alpha)C,
|
||||
\end{equation}
|
||||
where $\hat f_t^i$ is a reference frame, $\alpha$ is the leaky factor used to control the amount of information from the reconstructed frame present in the new frame $\check f_t^i$, and $C$ is a constant value.
|
||||
Leaky prediction has been used in the past for error control, especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in Equation~(\ref{EQ2:LeakyPredConstant}) is replaced by a lower layer error-free reconstructed frame. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate, as shown in \cite{LeakyPredConstant1973}. The $\alpha$ parameter allows to control the effect of the constant value, and thus the coding efficiency.
|
||||
Leaky prediction has been used in the past for error control, especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in~(\ref{EQ2:LeakyPredConstant}) is replaced by a lower layer error-free reconstructed frame. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate, as shown in \cite{LeakyPredConstant1973}. The $\alpha$ parameter allows to control the effect of the constant value, and thus the coding efficiency.
|
||||
By decreasing $\alpha$, the coding efficiency increases, but the exponential decay in error propagation becomes less significant.
|
||||
|
||||
% MHMCP_TCSVT2006
|
||||
@@ -211,7 +211,7 @@ Alternatively, the two previously encoded frames can be used to generate an inte
|
||||
\check f_t^i = \alpha \hat f_t^i + (1- \alpha) \hat f_{t-1}^i.
|
||||
\end{equation}
|
||||
Therefore, it is possible to decrease the error propagation using $\check f_t^i$ as reference frame for prediction, if only one of the original encoded frames is lost (i.e., $\hat f_{t}^i$ or $\hat f_{t-1}^i$). This is due to the fact that $\check f_t^i$ contains information from both $\hat f_{t}^i$ and $\hat f_{t-1}^i$.
|
||||
In Equation~(\ref{EQ2:LeakyPredDual}) $\hat f_{n}^i$ and $\hat f_{n-1}^i$ always correspond to the encoded frames and the generated frames ($\check f_t^i $) are not used to interpolate subsequent frames. This corresponds to a finite impulse response filter.
|
||||
In~(\ref{EQ2:LeakyPredDual}) $\hat f_{n}^i$ and $\hat f_{n-1}^i$ always correspond to the encoded frames and the generated frames ($\check f_t^i $) are not used to interpolate subsequent frames. This corresponds to a finite impulse response filter.
|
||||
|
||||
|
||||
\begin{figure}[tb]
|
||||
@@ -234,12 +234,12 @@ This limits the impact of errors in the background region to affect the \ac{roi}
|
||||
Moreover, since the leaky predictions are only confined to some specific regions, higher coding efficiency is achieved when compared with other methods.
|
||||
|
||||
|
||||
The method proposed in \cite{ROPERefGenJan2010} extends the traditional leaky prediction expressed in Equation~(\ref{EQ2:LeakyPredDual}) by replacing the previous encoded frame ($\hat f_{t-1}^i.$) by the previous interpolated frame ($\check f_t^i$). This results in the following:
|
||||
The method proposed in \cite{ROPERefGenJan2010} extends the traditional leaky prediction expressed in~(\ref{EQ2:LeakyPredDual}) by replacing the previous encoded frame ($\hat f_{t-1}^i.$) by the previous interpolated frame ($\check f_t^i$). This results in the following:
|
||||
\begin{equation}\label{EQ2:LeakyPredIIR}
|
||||
\check f_t^i = \alpha \hat f_t^i + (1- \alpha) \check f_{t-1}^i.
|
||||
\end{equation}
|
||||
The above equation corresponds to an infinite impulse response filter.
|
||||
Comparing both Equations~\ref{EQ2:LeakyPredDual}~and~\ref{EQ2:LeakyPredIIR} for the same $\alpha$, the latter implies stronger filtering than the former.
|
||||
Comparing both~(\ref{EQ2:LeakyPredDual})~and~(\ref{EQ2:LeakyPredIIR}) for the same $\alpha$, the latter implies stronger filtering than the former.
|
||||
Therefore, reference frames generated with heavier filtering have higher resilience against error propagation, but less correlation with the original frame, which negatively impacts inter predictions and hence coding efficiency.
|
||||
The idea of combining multiple predictions was also applied in the \ac{hevc} standard by combining it with end-to-end distortion to select the optimal predictions~\cite{DualFrameME2015}.
|
||||
|
||||
|
||||
@@ -15,12 +15,6 @@
|
||||
\vspace*{\fill}
|
||||
\section{Introduction}
|
||||
|
||||
\glsreset{mv}
|
||||
\glsreset{ec}
|
||||
\glsreset{cu}
|
||||
\glsreset{ld}
|
||||
\glsreset{ra}
|
||||
|
||||
The use of \ac{hevc} in different type of application and video services over networks that are prone to errors and packet losses, requires mechanisms to minimise the effect of such probabilistic events. This chapter starts by presenting an evaluation study about the error resilience of \ac{hevc} in Section~\ref{Section3:ErrorResilienceEvaluation} , with the aim of defining the weak elements of coded streams and the level of error robustness under different transmission loss conditions. The evaluation study covers different coding configuration to evaluate the individual impact of some new coding tools introduced in this standard, e.g., increased partition size, intra-refreshing, slice partitioning and advanced motion vector prediction.
|
||||
|
||||
Then, in Section~\ref{Section3:DynamicMVRefreshHEVC}, based on the results of the simulation study, a novel method to increase the error robustness of the \ac{mv} coding is presented and compared to existing techniques.
|
||||
@@ -338,8 +332,6 @@ Based on this evidence, this section proposes a method to make intelligent use o
|
||||
|
||||
\subsection{Proposed method}
|
||||
|
||||
\glsreset{rd}
|
||||
|
||||
The new approach to deal with temporal \ac{mv} dependencies is accomplished by modifying the \ac{rd} optimisation that is used to select the best \ac{mv} for inter-prediction.
|
||||
The objective is to increase the number of \acs{mv} that can be independently decoded, in order to improve the error recovery capability after an arbitrary frame loss event.
|
||||
|
||||
|
||||
@@ -15,10 +15,6 @@
|
||||
\vspace*{\fill}
|
||||
\section{Introduction}
|
||||
|
||||
\glsreset{rd}
|
||||
\glsreset{pu}
|
||||
\glsreset{mv}
|
||||
|
||||
This chapter proposes a new two-stage approach to improve the error robustness of \ac{hevc} streaming, by reducing temporal error propagation in case of frame loss.
|
||||
This is accomplished by reducing the prediction mismatch that occurs at the decoder after frame loss.
|
||||
The proposed approach comprises the following two stages: (i) at the encoding stage, the reference pictures are dynamically selected based on constraining conditions and Lagrangian optimisation, which distributes the use of reference pictures throughout each frame, reducing the number of \ac{pu} that depend on a single reference; (ii) at the streaming stage, a \ac{mv} prioritisation algorithm, based on spatial dependencies, selects an optimal sub-set of \acs{mv} to be transmitted as side redundant information to reduce mismatched \ac{mv} predictions at the decoder.
|
||||
@@ -32,16 +28,6 @@ This chapter is organized as follows. Section~\ref{Section4:Problem} presents th
|
||||
\section{Problem definition}
|
||||
\label{Section4:Problem}
|
||||
|
||||
\glsreset{rd}
|
||||
\glsreset{ec}
|
||||
\glsreset{pu}
|
||||
\glsreset{ctu}
|
||||
\glsreset{cu}
|
||||
\glsreset{mv}
|
||||
|
||||
\glsreset{ra}
|
||||
\glsreset{ld}
|
||||
|
||||
As previously mentioned, when\ac{hevc} bitstreams are subjected to transmission losses, the decoded video signal incurs in significant degradation of both objective and subjective quality.
|
||||
This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes, which are selected according to \ac{rd} optimisation criteria, which always assume error-free transmission~\cite{ROPE2007}.
|
||||
Therefore, high compression efficiency is achieved, but resilience to transmission errors is penalised.
|
||||
@@ -62,7 +48,7 @@ These results show that, in general, reference $r_1$ is selected more often than
|
||||
This is mainly true for sequences with higher motion.
|
||||
For those sequences with lower motion, but high spatial complexity, e.g., BQSquare (see \ac{si} in Table~\ref{TableTestSequences}), both reference frames are equally (approximately) selected for temporal predictions, with $r_1$ being used by 58\% of cases and $r_2$ by 42\%.
|
||||
This reveals that, even though two references are used, the encoder mainly uses the temporally closer one as the reference for inter-frame predictions.
|
||||
Therefore, if such reference frame is lost, a higher number of mismatched predictions will occur, inevitabely leading to higher error propagation.
|
||||
Therefore, if such reference frame is lost, a higher number of mismatched predictions will occur, inevitably leading to higher error propagation.
|
||||
|
||||
Another coding technique that contributes to increase error propagation is the use of predictions in \ac{mv} coding~\cite{MotionCodingHEVC2011}, which also contributes to increase the total amount of spatial and temporal dependencies.
|
||||
As shown in Chapter~\ref{Chapter:HEVCErrorResilience}, temporally predicted \acs{mv} are responsible for a greater impact on error propagation, as they propagate errors in the \ac{mv} decoding across different time instances.
|
||||
@@ -134,12 +120,10 @@ This leads to the overall goal of finding the optimum R-D coding mode for each \
|
||||
\end{equation}
|
||||
In this case $\sigma_C$ measures the deviation from the checkerboard solution, where $\sigma_C = 0$, since all possible reference frames $r_i\in F$ are uniformly used throughout the \acs{pu} of $C$.
|
||||
|
||||
|
||||
|
||||
To attain the goal defined above, the best coding mode for each \ac{pu} is chosen through a Lagrangian \ac{rd} optimisation that tends to minimise $\sigma_C$.
|
||||
This is done by penalising (or benefiting) the \ac{rd} cost of those coding modes which intend to use reference frames that have already been used more (or less) frequently than their average use in previous \acs{pu} of the current frame.
|
||||
For each \ac{pu}, this mechanism has the effect of reducing (or increasing) the probability of using each possible reference frame $r_i\in F$, according to the number of previous \acs{pu} that have used each one.
|
||||
Such approach allows to select the reference frames, by independently processing each \ac{pu} in order to minimise the global standard deviation given by~Equation~(\ref{eq:StdX}). Thus, the optimum coding mode $\phi^*$, selected from a set of possible coding modes $M$, is derived based on the following Lagrangian optimisation:
|
||||
Such approach allows to select the reference frames, by independently processing each \ac{pu} in order to minimise the global standard deviation given by~(\ref{eq:StdX}). Thus, the optimum coding mode $\phi^*$, selected from a set of possible coding modes $M$, is derived based on the following Lagrangian optimisation:
|
||||
\begin{align}
|
||||
\phi^* &= arg \min_{\phi \in M} J(\phi) \\
|
||||
J(\phi) &= \Big( D(\phi) + \lambda \times R(\phi) \Big) \times e^{W(r_{\phi})}
|
||||
@@ -156,9 +140,9 @@ The global and local deviations, $\Delta_G (r_\phi)$ and $\Delta_L (r_\phi)$, r
|
||||
\begin{equation}\label{eq:KFactorXValue}
|
||||
\Delta_\zeta(r_{\phi}) = \mathbf{N}_\zeta(r_\phi) - \frac{1}{S} \sum_{i=1}^{S}\mathbf{N}_\zeta(r_i), \hspace{2mm} \zeta=G,L.
|
||||
\end{equation}
|
||||
For $\zeta=G$, function $\Delta_G(r_\phi)$ measures the difference between the number of times that reference frame $r_\phi$ was used and the global average use of all $r_i \in F$ in the \acs{pu} of $C$ that were encoded before the current \ac{pu}. $\mathbf{N}_G(r_i)$ is the number of times that each reference frame $r_i$ was used across all previously encoded \acs{pu}. For $\zeta=L$, function $\Delta_L(r_\phi)$ only considers the three top-left neighbouring \acs{ctu} for counting the number of times each reference frame was used, which is expressed by $\mathbf{N}_L(r_i)$. Overall, Equation (\ref{eq:KFactorXValue}) is used to adjust the penalty weight defined by Equation (\ref{eq:KFactor}), in order to achieve an approximately uniform use of all possible reference frames. This corresponds to the general objective of minimising the values of $\Delta_\zeta(r_{\phi})$.
|
||||
The optimum is reached when all reference frames are used to encode the same amount of \acs{pu} (i.e., $\Delta_G(r_{\phi})=\Delta_L(r_{\phi})=0$), and the original Lagrangian cost is not affected, i.e., $e^{W(r_{\phi})}=1$ in Equation (\ref{eq:Lagrangian}).
|
||||
Note that an exponential cost increase is used in Equation (\ref{eq:Lagrangian}) in order to increase the penalty associated to any reference frame that tends to be used much more often than the average.
|
||||
For $\zeta=G$, function $\Delta_G(r_\phi)$ measures the difference between the number of times that reference frame $r_\phi$ was used and the global average use of all $r_i \in F$ in the \acs{pu} of $C$ that were encoded before the current \ac{pu}. $\mathbf{N}_G(r_i)$ is the number of times that each reference frame $r_i$ was used across all previously encoded \acs{pu}. For $\zeta=L$, function $\Delta_L(r_\phi)$ only considers the three top-left neighbouring \acs{ctu} for counting the number of times each reference frame was used, which is expressed by $\mathbf{N}_L(r_i)$. Overall, (\ref{eq:KFactorXValue}) is used to adjust the penalty weight defined by (\ref{eq:KFactor}), in order to achieve an approximately uniform use of all possible reference frames. This corresponds to the general objective of minimising the values of $\Delta_\zeta(r_{\phi})$.
|
||||
The optimum is reached when all reference frames are used to encode the same amount of \acs{pu} (i.e., $\Delta_G(r_{\phi})=\Delta_L(r_{\phi})=0$), and the original Lagrangian cost is not affected, i.e., $e^{W(r_{\phi})}=1$ in (\ref{eq:Lagrangian}).
|
||||
Note that an exponential cost increase is used in (\ref{eq:Lagrangian}) in order to increase the penalty associated to any reference frame that tends to be used much more often than the average.
|
||||
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
@@ -167,7 +151,7 @@ Note that an exponential cost increase is used in Equation (\ref{eq:Lagrangian})
|
||||
\label{fig:weightfct}
|
||||
\end{figure}
|
||||
|
||||
In Equation (\ref{eq:KFactor}), the $\gamma$ constant controls the slope of $W(r_\phi)$, leading to higher weights as the value of $\gamma$ increases.
|
||||
In (\ref{eq:KFactor}), the $\gamma$ constant controls the slope of $W(r_\phi)$, leading to higher weights as the value of $\gamma$ increases.
|
||||
On the one hand a very high value for $\gamma$ leads to the checkerboard pattern in the reference picture usage, as previously described.
|
||||
On the other hand, a low value of $\gamma$ reduces the importance of the choice of the reference frames in the Lagrangian cost.
|
||||
Therefore, the impact of the proposed method on the coding efficiency can be controlled through $\gamma$. Another way of controlling the impact of the proposed method is the use of the $T_{ID}$. This allows using lower weights in higher temporal layers, since such frames cannot be used as references for others with lower temporal ID, thus having less impact on error propagation.
|
||||
@@ -198,8 +182,7 @@ For instance, for $\gamma=5$, when a reference frame $r_\phi$ is used to predict
|
||||
Figure~\ref{fig:RefPicSelectionExample} illustrates the distribution of reference frames achieved by the proposed method across different \acs{pu}. This result is compared against a fixed checkerboard approach (\CHKB) and the reference \ac{hevc} (\REF). The dark-gray and light-gray squares indicate the reference frame in use, either $r_1$ or $r_2$, respectively. The unfilled squares indicate intra-coded \acs{pu}. Below each method, the percentages of the image areas predicted by each reference frame are shown. As shown in this figure, the proposed method is able to distribute the use of the reference frames closely to the \CHKB method, i.e., $r_2$ is used as reference for approximately the same amount of \acs{pu} as $r_1$ (see use percentages of the predicted image). However, the predicted regions in the proposed method do not exhibit the same regular pattern as in \CHKB method.
|
||||
It is noticeable in Figure~\ref{fig:RefPicSelectionExample} that for several \acs{ctu} the \PROP method is able to keep the same partitions and reference frames when compared to the optimal encoding (\REF). This can be observed, for example in the small partitions on the left, where the \PROP method uses the same reference frame as \REF. Thus, the proposed method is expected to achieve better coding efficiency than \CHKB, since more optimal coding modes may be chosen.
|
||||
|
||||
% TID aqui?
|
||||
Summarising, by using Equations~(\ref{eq:Lagrangian})~and~(\ref{eq:KFactor}), the cost of using a given reference frame increases as the number of \acs{pu} using it for prediction increases.
|
||||
Summarising, by using~(\ref{eq:Lagrangian})~and~(\ref{eq:KFactor}), the cost of using a given reference frame increases as the number of \acs{pu} using it for prediction increases.
|
||||
However, the reference frames are not hard-selected without taking into account the R-D cost.
|
||||
In general, it is more difficult to reach a uniform use of reference frames in complex sequences than in simpler ones.
|
||||
Note that standard compatibility is maintained, since the proposed optimisation method does not require any syntax change. Moreover, since the proposed approach is employed at the Lagrangian optimisation already implemented in the \ac{hevc} reference software, the complexity of the encoder is not significantly affected.
|
||||
@@ -208,8 +191,6 @@ Note that standard compatibility is maintained, since the proposed optimisation
|
||||
\section{Streaming stage: \ac{mv} selection}
|
||||
\label{Section4:MVSelection}
|
||||
|
||||
\glsreset{tmvp}
|
||||
|
||||
Since \ac{hevc} uses differential and predictive \ac{mv} coding, whenever a \ac{mv} is lost the subsequent ones will also be affected, until a refresh point (i.e., \ac{mv} coded on its own, without using \ac{mv} prediction) is reached to break the dependency chain.
|
||||
Thus, the amount of error propagation strongly depends on whether the \ac{tmvp} is used.
|
||||
In this streaming stage, to increase the number of possible refresh points and improve error robustness, a small number of \acs{mv} are selected to be transmitted as side information. The selection of these redundant \acs{mv} is based on a trade-off between the image area covered by such \acs{mv} and the number of bits to encode them.
|
||||
@@ -242,10 +223,10 @@ the \acs{mv} with temporal dependencies are firstly selected as the most importa
|
||||
\item
|
||||
the sub-set of $MVs \in V$ are ranked according to the number of spatial dependencies associated to each one.
|
||||
\item
|
||||
the optimal sub-set $U \subset V$ is found as the best trade-off between the image area covered by the dependent \acs{pu} and bitrate overhead, i.e., the one that maximises the cost function in Equation (\ref{eq:MVSelectOpt}).
|
||||
the optimal sub-set $U \subset V$ is found as the best trade-off between the image area covered by the dependent \acs{pu} and bitrate overhead, i.e., the one that maximises the cost function in (\ref{eq:MVSelectOpt}).
|
||||
\end{enumerate}
|
||||
|
||||
This can be formulated as follows. Considering $V$ the ordered set of \acs{mv} encoded for the current frame, i.e., ${V=\{MV_1, MV_2, \cdots MV_s\}}$ and ${U \subset V}$ always starting in the first vector, i.e., ${U_n=\{MV_1, MV_2, \cdots MV_n\}}$ with $n\leq s$, the best set of \acs{mv} ($U_n^*)$ is selected based on the optimisation procedure given by Equation (\ref{eq:MVSelectOpt}). Note that any value of $n$ corresponds to a unique subset $U$.
|
||||
This can be formulated as follows. Considering $V$ the ordered set of \acs{mv} encoded for the current frame, i.e., ${V=\{MV_1, MV_2, \cdots MV_s\}}$ and ${U \subset V}$ always starting in the first vector, i.e., ${U_n=\{MV_1, MV_2, \cdots MV_n\}}$ with $n\leq s$, the best set of \acs{mv} ($U_n^*)$ is selected based on the optimisation procedure given by (\ref{eq:MVSelectOpt}). Note that any value of $n$ corresponds to a unique subset $U$.
|
||||
|
||||
\begin{equation}\label{eq:MVSelectOpt}
|
||||
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
|
||||
@@ -280,17 +261,6 @@ At the decoder-side, \ac{ec} is based on motion compensation from previously dec
|
||||
The evaluation of the encoding-stage is carried out separately from the streaming-stage.
|
||||
|
||||
|
||||
|
||||
%This section presents an evaluation study about the error robustness of HEVC using different sequences and coding configurations. Table~\ref{tab:sequences} presents a summary of the main characteristics of the video sequences used in this work. Each sequence comprises 240 frames and the whole set covers different types of motion and texture complexity. This table also shows the spatial information (SI) and the temporal information (TI), as defined in \cite{ITUTP9111998}. The experimental results for this study were obtained by using the HM reference software, version $16.2$.
|
||||
%The sequences were encoded with all coding modes enabled and following two recommended prediction structures: Low-Delay (P-frames with 4 references frames) and \ac{ra} (B-frames with 2 reference frames in each list)~\cite{HEVCCommonTests2012}. These prediction structures cover the use of P-frames and hierarchical B-frames~\cite{H264ScalableOverviewSep2007}, which are commonly used in HEVC streaming~\cite{RealTimeHEVC2016}.
|
||||
%To simulate a broadcast transmission an \ac{idr} period of $32$ frames was used and the filtering operations were disabled at slices boundaries in order to keep the slices self-contained. The remaining configurations have been kept at their default values as given in the common test conditions~\cite{HEVCCommonTests2012}.
|
||||
%In order to achieve high quality images for all coded sequences (i.e., PSNR around 40 dB) different bitrates were used.
|
||||
%At the decoder-side, error concealment is based on motion compensation from previously decoded frames. The motion field of a missing frame is obtained applying MV extrapolation from the two closest neighbours. Afterwards, the final MVs are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}.
|
||||
%To simulate the lossy transmission environment, each frame is encoded into several slices, each one packetized into one NAL unit, which are then transmitted as the payload of independent packets.
|
||||
%
|
||||
%In our experiments a whole packet is lost whenever an error occurs, originating a lost slice. Random packet loss was simulated using a two-state Markov model with the following average burst length 1.24, 1.47, 1.83 and 2.05, and a maximum burst length of 2, 3, 5 and 7, for packet loss ratios (PLR) of 1\%, 3\%, 5\%, 10\%, respectively.
|
||||
|
||||
|
||||
\subsection{Encoding stage}
|
||||
|
||||
|
||||
@@ -382,7 +352,7 @@ Taking into account the above results and the spatio-temporal information shown
|
||||
\label{fig:QualityVsGamma}
|
||||
\end{figure}
|
||||
|
||||
The influence of the $\gamma$ parameter used in Equation (\ref{eq:KFactor}) on the video quality was also evaluated, in order to check how the cost constrains affect the performance of the proposed method.
|
||||
The influence of the $\gamma$ parameter used in (\ref{eq:KFactor}) on the video quality was also evaluated, in order to check how the cost constrains affect the performance of the proposed method.
|
||||
Figure~\ref{fig:QualityVsGamma} shows the average quality gains of the proposed method in comparison to the reference \ac{hevc} (\REF) across all test sequences, for different values of $\gamma$.
|
||||
For the error-free case (PLR=0\%), the results in this figure show that increasing $\gamma$ leads to a decrease in the coding efficiency, especially in the \ac{ld} configuration.
|
||||
The reason is that higher values of $\gamma$ lead to higher values of $W(r_{\phi})$, therefore the reference picture selection cost is severely constrained by the exponential weights.
|
||||
|
||||
@@ -13,16 +13,6 @@
|
||||
\vspace*{\fill}
|
||||
\section{Introduction}
|
||||
|
||||
\glsreset{rd}
|
||||
\glsreset{mv}
|
||||
\glsreset{ec}
|
||||
\glsreset{cu}
|
||||
\glsreset{ld}
|
||||
\glsreset{ra}
|
||||
\glsreset{ra}
|
||||
\glsreset{ld}
|
||||
|
||||
|
||||
In this chapter an error concealment-aware encoding scheme is proposed to improve the quality of decoded video in broadcast environments prone to transmission errors and data loss.
|
||||
The method aims to enhance the reconstruction of recovered lost frames by optimising the efficiency of the \ac{ec}.
|
||||
The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled through \ac{sei} messages. Such optimal \ac{ec} modes are found by simulating transmission losses followed by a Lagrangian optimisation of the \textit{signalling rate - \ac{ec} distortion} cost. A generalised saliency-weighted distortion is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. When data loss occurs the decoder uses the signalling information to improve the reconstruction quality.
|
||||
@@ -40,10 +30,6 @@ Finally, Section~\ref{Section5:Summary} concludes the chapter.
|
||||
\section{Problem definition}
|
||||
\label{Section5:Problem}
|
||||
|
||||
\glsreset{mve}
|
||||
\glsreset{mc}
|
||||
%\glsunset{mb}
|
||||
|
||||
In Chapter~\ref{Chapter:ErrorResilienceReview} different \ac{ec} algorithms that can effectively reconstruct lost video were described, though not all of them exhibit the same accuracy for every lost frame or slice.
|
||||
Moreover, it was also shown that combining different \ac{ec} approaches at the block level to improve the reconstruction accuracy is not straightforward.
|
||||
In the past, hybrid \ac{ec} schemes were proposed to improve the quality of recovered frames by either combining spatial and temporal \ac{ec} algorithms~\cite{HybridConcTB2008} or by optimising the \ac{mve} at the pixel and block levels~\cite{HybridConcealmentTB2008,2DConcHybridMotion2010}.
|
||||
@@ -142,7 +128,7 @@ E(v) =
|
||||
\sum_{\mathbf{p} \in \Omega} \Big( \big| f_{-1}(\mathbf{p}) - f_{-2}( \mathbf{p} + v(\mathbf{p}))\big| + \big|\nabla v(\mathbf{p})\big| \Big)%d\mathbf{p},\ \mathbf{p} \in \Omega
|
||||
\end{equation}
|
||||
where $f_{-1}$ and $f_{-2}$ are the two previous frames at time instants $t_{-1}$ and $t_{-2}$, as represented in Figure~\ref{FIG5:MotionVectorExtrapolation}.
|
||||
In Equation~\ref{EQ5:DualFlow}, $\Omega$ defines the sets of pixels of those frames, and $|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of the optical flow which effectively disallow discontinuities.
|
||||
In~(\ref{EQ5:DualFlow}), $\Omega$ defines the sets of pixels of those frames, and $|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of the optical flow which effectively disallow discontinuities.
|
||||
This is based on the assumption that neighbouring regions belong to the same object, thus such regions should have similar motion.
|
||||
|
||||
|
||||
@@ -156,7 +142,7 @@ f(\mathbf{p}) \sim \mathbf{p}\mathbf{A}_1\mathbf{p}^T+\mathbf{b}_1\mathbf{p}^T+C
|
||||
where $\mathbf{p} = [x\ y]$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ is a row vector and $C_1$ is a scalar. The coefficients are estimated using weighted least squares fit to the signal values in the neighbourhood.
|
||||
|
||||
Since the result of a polynomial expansion is such that each neighbourhood is approximated by a polynomial, it is necessary to analyse what happens if a polynomial undergoes an ideal translation.
|
||||
Let's consider the signal $f_1(\mathbf{p})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$, given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following:
|
||||
Let's consider the signal $f_1(\mathbf{p})$ represented by~(\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$, given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following:
|
||||
\begin{align}
|
||||
f_2(\mathbf{p})
|
||||
&= f_1(\mathbf{p}-\mathbf{d}) \nonumber \\
|
||||
@@ -175,10 +161,8 @@ This allows to find the displacement $\mathbf{d}$ for any signal dimensions by s
|
||||
|
||||
\subsubsection{Large displacement optical flow with deep matching (\textit{DeepFlow})}
|
||||
|
||||
%\todo{Completar o Deep flow}
|
||||
|
||||
In this method the motion field is estimated based on extension of the total variation methods using descriptor (feature) matching.
|
||||
This allows the optical flow estimation to accurately find large displacements, because it is not fully bound by any constrains, such as the Horn \& Such presented in Equation~(\ref{EQ5:DualFlow})\cite{LargeDisplacementOptFlow_TPAMI2011}. Moreover, the correspondences between descriptions are used to apply a coarse-to-fine wrapping, which avoids local minimum values.
|
||||
This allows the optical flow estimation to accurately find large displacements, because it is not fully bound by any constrains, such as the Horn \& Such presented in~(\ref{EQ5:DualFlow})\cite{LargeDisplacementOptFlow_TPAMI2011}. Moreover, the correspondences between descriptions are used to apply a coarse-to-fine wrapping, which avoids local minimum values.
|
||||
The combination of both approaches increases the matching accuracy.
|
||||
|
||||
This approach was also exploited in \cite{DeepFlow2013} with the following extensions: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layer optimisation.
|
||||
@@ -194,7 +178,7 @@ E(v) =
|
||||
\end{equation}
|
||||
where $E_D$ is the data term, $E_S$ is a smoothing term and $E_M$ is the matching term.
|
||||
The data term is obtained from the difference between matching pixels in both video frames.
|
||||
The smoothness term is a robust penalisation of the gradient of the flow, similarly to Equation (\ref{EQ5:DualFlow}), i.e., $E_S(\mathbf{p})=|\nabla v(\mathbf{p})|$. Finally, the matching term ($E_M$) is used to encourage the flow estimation to be similar to the precomputed motion field~($v'$).
|
||||
The smoothness term is a robust penalisation of the gradient of the flow, similarly to~(\ref{EQ5:DualFlow}), i.e., $E_S(\mathbf{p})=|\nabla v(\mathbf{p})|$. Finally, the matching term ($E_M$) is used to encourage the flow estimation to be similar to the precomputed motion field~($v'$).
|
||||
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
@@ -215,7 +199,6 @@ Table~\ref{TAB5:ECPerformaceComparison} shows the average quality (\ac{psnr}) ob
|
||||
The absolute value of \ac{psnr} is shown for the \FC method, while the differential values are shown for the remaining methods.
|
||||
The results shown in the table reveal that methods \MC and \MVE are both able to overcome \FC, achieving average quality gains of up to 1.41 dB and 1.46 dB, respectively. Moreover, by extrapolating the \acs{mv} from the correctly receiving frame to the lost one (\MVE), it is also possible to outperform motion-copy (\MC).
|
||||
This can be explained by the objects' trajectories of objects which are preserved in the \MVE method (assuming constant translational motion), in contrast with the \MC method, which assumes that neighbouring objects have parallel motion which may result in inaccurate estimations.
|
||||
%\HERE
|
||||
|
||||
|
||||
\begin{table}[tb]
|
||||
@@ -247,8 +230,6 @@ Thus, in order to achieve higher reconstruction quality, different approaches sh
|
||||
\section{Architecture for the EC-aware video encoding}
|
||||
\label{Section5:Architecture}
|
||||
|
||||
\glsreset{el}
|
||||
|
||||
This section presents the general architecture proposed for \ac{ec}-aware video encoding, describing its main functional blocks, and providing an introduction for the subsequent detailed descriptions.
|
||||
Figure~\ref{FIG5:BlockDiagram} illustrates the architecture of the proposed \ac{ec}-aware encoding scheme.
|
||||
After the video encoder, it includes data loss simulation at the \ac{nal} for every slice. Subsequently, \ac{ec} is applied to the simulated losses in order to find the best methods that should be used to reconstruct the lost slices in case of transmission errors.
|
||||
@@ -324,24 +305,6 @@ The lost frame is partitioned into quadtree structures, each one forming a set o
|
||||
Then, the process of finding the optimal \ac{ec} method for each \ac{ecu} is formulated as a rate-distortion cost minimisation. To solve such problem, a Lagrangian optimisation is used to achieve the best trade-off between reconstruction quality and signalling overhead.
|
||||
Furthermore, image saliency driven by visual attention modelling is used in the optimisation process to weight the cost of the overhead information of each \ac{ecu}, according to the visual relevance of its content.
|
||||
|
||||
% \HIGHLIGHT{%
|
||||
% The optimal depth $Level$ ($l^*$) that determines the best \ac{ecu} size is obtained by minimising the following lagragian cost.}
|
||||
% \begin{equation}\label{eq:optdepth}
|
||||
% \newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
|
||||
% l^* = \argmin_{l=0..3} \left\{ D(l) + \left(1+\frac{t_0-t_I}{T_{I}}\right) \lambda R(l) \right\},
|
||||
% % l^* = \argmin_{l=1..3} \left\{ D(l) + \lambda \left(1-\frac{t_I-t_c}{T_{\ac{ird}}}\right) R(l) \right\},
|
||||
% \end{equation}
|
||||
% \HIGHLIGHT{where $R(l)$ represents the number of overhead bits required to encode \ac{ec} mode, and $D(l)$ is mean square error of the \ac{ctu} formed by $4^l$ ECUs.
|
||||
% }
|
||||
% \HIGHLIGHT{%
|
||||
% In equation~\ref{eq:optdepth}, besides the parameter $\lambda$ used to trade-off the cost of distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(l)$ according to the duration of the error propagation decreases. This is given by the ratio between the temporal distance from the current frame ($t_0$) to the next refresh frame ($t_I$) and the period of refresh frames ($T_{I}$).
|
||||
% }
|
||||
% \begin{align}
|
||||
% (ecm^*, l^*) &= arg \min_{\phi \in M} J(ecm, l), \\
|
||||
% J(ecm,l) &= D(ecm,l) + \left(1+\frac{t_0-t_I}{T_{I}}\right) \lambda R(ecm,l),
|
||||
% \label{eq:Lagrangian}
|
||||
% \end{align}
|
||||
|
||||
The optimal \ac{ec} mode $m_{l,i}^*$, for the $i_{th}$ \ac{ecu} of size defined by the quadtree partitioning level $l$, is obtained by minimising the following Lagrangian cost:
|
||||
\begin{equation}\label{EQ6:OptimalDepth}
|
||||
\small
|
||||
@@ -353,14 +316,14 @@ m_{l,i}^* = \argmin\limits_{\footnotesize \substack{l\in\left\{0...3\right\} \\
|
||||
where $R(m_{l,i})$ represents the number of overhead bits required to encode the \ac{ec} mode, and $D(m_{l,i})$ is the sum of absolute error of the \ac{ecu}.
|
||||
|
||||
|
||||
In equation~\ref{EQ6:OptimalDepth}, besides the parameter $\lambda$, used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation.
|
||||
In~(\ref{EQ6:OptimalDepth}), besides the parameter $\lambda$, used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation.
|
||||
As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques}, this method uses the ratio between the temporal distance from the current frame ($t_C$) to the \HIGHLIGHT{previous refresh frame} ($t_I$) and the period of refreshing frames ($T_{I}$).
|
||||
%This weight varies between $\left[0;1 \right]$, achieving the lowest value for intra-coded frames and the maximum for the frame located immediately before an intra-coded one.
|
||||
This behaviour follows the fact that when a lost frame is closer to the next intra-coded frame, the error propagation is lower.
|
||||
Thus, it is not worthwhile to spend too many overhead bits to reduce it. This is accomplished by increasing the cost of such bits according to the temporal distance between the missing frame and the next I frame.
|
||||
Moreover, a dynamic value for $\lambda$ is also used to adjust the amount of overhead within the frame, according to the visual relevance of each image region, defined by a saliency map.
|
||||
|
||||
The algorithm previously described results in \acs{ecu} with variable size, which is defined by the depth of the quad-tree structure, i.e., level $l$. The partition size is jointly optimised along with the best \ac{ec} candidate method using Equation~\ref{EQ6:OptimalDepth}. Alternatively, a fixed \ac{ecu} size can also be used by defining a fixed depth for all quad-tree structures, i.e., constant $l$ for the entire frame. This can be achieved by using Equation~\ref{EQ6:OptimalDepth} to solely optimise the \ac{ec} methods for each \ac{ecu}.
|
||||
The algorithm previously described results in \acs{ecu} with variable size, which is defined by the depth of the quad-tree structure, i.e., level $l$. The partition size is jointly optimised along with the best \ac{ec} candidate method using~(\ref{EQ6:OptimalDepth}). Alternatively, a fixed \ac{ecu} size can also be used by defining a fixed depth for all quad-tree structures, i.e., constant $l$ for the entire frame. This can be achieved by using~(\ref{EQ6:OptimalDepth}) to solely optimise the \ac{ec} methods for each \ac{ecu}.
|
||||
Although this reduces the overall complexity of the proposed method, it leads to a non-optimal solution which may require additional overhead. In Section~\ref{Section5:ECSignallingPerformance} a comparison between a dynamic and fixed approach is provided.
|
||||
|
||||
|
||||
@@ -575,7 +538,6 @@ In summary, the proposed technique is able to enhance the performance of the sta
|
||||
|
||||
|
||||
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
% SECTION
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
@@ -685,7 +647,6 @@ The visual impact of using the proposed \ac{ec}-aware encoding can be evaluated
|
||||
\section{Overall performance evaluation}
|
||||
\label{Section5:OverallPerformance}
|
||||
|
||||
%\subsection{Quality evaluation}
|
||||
|
||||
As in the previous simulations, the packets were randomly discarded to simulate loss events, at different \acs{plr}.
|
||||
The quality obtained with the proposed method is compared with the default method used by reference \ac{hevc} , i.e., frame-copy (\REF), motion-copy (\MC)~\cite{HEVCMCResidueDec2013}, and the method proposed in \cite{ROPERefGenJan2010}, which implements a robust scheme to minimise error propagation.
|
||||
|
||||
@@ -10,17 +10,11 @@
|
||||
\vspace*{\fill}
|
||||
\section{Introduction}
|
||||
|
||||
\glsunset{hevc}
|
||||
\glsreset{ec}
|
||||
\glsreset{rd}
|
||||
\glsreset{el}
|
||||
\glsreset{nn}
|
||||
|
||||
The effectiveness of different error resilience techniques is rather variable depending on the video content, the transmitted bitrate, network conditions, among others.
|
||||
Therefore, it is not straightforward to select a single technique capable of guaranteeing the highest efficiency for every possible scenario and also for the whole range of networking conditions.
|
||||
This chapter addresses this problem by proposing an adaptive error robustness scheme that optimises the efficiency of the techniques previously described in Chapters~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}.
|
||||
At the encoder-side, a pre-trained neural network predicts the video stream that is able to achieve the highest decoded quality for particular loss cases, among various video streams encoded with different levels of compression efficiency and error robustness.
|
||||
The stream selection is based on different input parameters that include information from the video signal, the coded stream and the transmission network.
|
||||
The stream selection is based on different input parameters that in\textsl{}clude information from the video signal, the coded stream and the transmission network.
|
||||
Using this approach the overall performance of the proposed adaptive method significantly increases, achieving a better trade-off between compression efficiency and error robustness.
|
||||
|
||||
This chapter is organised as follows.
|
||||
@@ -186,9 +180,6 @@ This results in a total of \NumCases test conditions.
|
||||
|
||||
\subsection{Input parameters}
|
||||
|
||||
\glsreset{si}
|
||||
\glsreset{ti}
|
||||
|
||||
A set of different parameters is used to predict the optimal stream that achieves the highest quality in specific network conditions.
|
||||
Based on these inputs the proposed scheme should be able to select the appropriate robust coding stream that should be transmitted.
|
||||
The parameter set characterises three different aspects of the video transmission system; (i) input video (two parameters), (ii) video stream (three parameters) and (iii) transmission network (one parameter).
|
||||
@@ -252,8 +243,6 @@ Since both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar informatio
|
||||
|
||||
\subsection{Neural network}
|
||||
|
||||
\glsreset{nn}
|
||||
|
||||
A \ac{nn} is a computational model based on the structure and functions of biological neural networks.
|
||||
%In terms of computer science, \acs{nn} work similarly to a human nervous system for receiving, processing, and transmitting information between neurons.
|
||||
Mainly, a \ac{nn} consists of three different layers: the input layer which receives data from an external environment, the hidden layer which process the inputs and an output layer~\cite{NeuralNetworks1994}.
|
||||
@@ -321,8 +310,6 @@ The \ac{nn} is able to achieve a high accuracy for \SMEC and \SMEXPELTWEN stream
|
||||
This reveals that the \ac{nn} used in the proposed method is able to predict the coded stream which leads to the highest video quality across different conditions with acceptable accuracy.
|
||||
The results from the validation set, shown in Table~\ref{Table6:ConfusionVal}, reveal that the \ac{nn} training did not over-fit, since it shows similar accuracy as the results shown in Table~\ref{Table6:ConfusionTraining}.
|
||||
|
||||
|
||||
|
||||
\begin{table}[tb]
|
||||
\centering
|
||||
\caption{Confusion matrix of the validation set.}
|
||||
@@ -452,8 +439,6 @@ These results also confirm that the proposed method is able to select the optima
|
||||
The proposed method achieves a quality gain up to 4.46 dB in comparison with the reference \ac{hevc}, outperforming the quality achieved with the stream \SMEXPELTWEN (i.e., 4.19 dB).
|
||||
This confirms the advantage of the proposed method to adaptively select among different streams with different levels of error protections.
|
||||
|
||||
|
||||
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
%% SECTION %%
|
||||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||||
|
||||
@@ -2,10 +2,6 @@
|
||||
\chapter{Conclusion and future work \VERSION{3}}
|
||||
\label{Chapter:Conclusions}
|
||||
|
||||
\glsresetall
|
||||
\glsunset{hevc}
|
||||
\glsunset{h264}
|
||||
|
||||
This thesis investigated new algorithms and techniques for efficient error robustness in highly efficiency video coders.
|
||||
The importance of this research work is justified by the widespread of digital video content, including high resolution formats, multi-view and 360-degree video formats, which significantly increase the network bandwidth requirements.
|
||||
Consequently, errors and data loss may occur along the transmission system, despite the fact that video coding standards have been improved, in order to meet the network bandwidth constrains.
|
||||
|
||||
@@ -68,8 +68,9 @@
|
||||
\listoftables
|
||||
\printglossary[title=List of Abbreviations]
|
||||
\glsresetall
|
||||
|
||||
\glsunset{ip}
|
||||
\glsunset{itut}
|
||||
\glsunset{iso}
|
||||
\glsunset{iec}
|
||||
|
||||
\mainmatter
|
||||
\include{010_Introduction/Introduction}
|
||||
|
||||
Reference in New Issue
Block a user