diff --git a/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex b/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex index 96693f5..2d4b37e 100644 --- a/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex +++ b/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex @@ -35,11 +35,11 @@ Finally, Section~\ref{Section5:Summary} concludes this chapter. \glsreset{mve} \glsunset{mb} -In Chapter~\ref{Chapter:ErrorResilienceReview} different \ac{ec} algorithms that can effectively reconstruct lost data were described. However, different algorithms might not present the same accuracy for every lost frame or slice. Moreover, selection at the \ac{mb}/\ac{pu} level is not straightforward. -In the past, hybrid \ac{ec} schemes were proposed to improve the reconstruction quality by either combining spatial and temporal \ac{ec} algorithms~\cite{HybridConcTB2008} or by optimising the \ac{mve} at the pixel and block levels~\cite{HybridConcealmentTB2008,2DConcHybridMotion2010}. -Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of the motion-based recovery by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}. Although such methods are able to reduce the reconstruction artefacts and improve the overall quality, decoder-based decisions are still not fully accurate, as they do not have enough reference information to ease the decision process. Therefore, higher performance can be achieved by performing encoding-side estimation and decision based on the error-free information. +In Chapter~\ref{Chapter:ErrorResilienceReview} different \ac{ec} algorithms that can effectively reconstruct lost data were described. However, different algorithms might not present the same accuracy for every lost frame or slice. Moreover, selection at the \ac{mb}/\ac{pu} level is not straightforward. +In the past, hybrid \ac{ec} schemes were proposed to improve the reconstruction quality by either combining spatial and temporal \ac{ec} algorithms~\cite{HybridConcTB2008} or by optimising the \ac{mve} at the pixel and block levels~\cite{HybridConcealmentTB2008,2DConcHybridMotion2010}. +Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of the motion-based recovery by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}. Although such methods are able to reduce the reconstruction artefacts and improve the overall quality, decoder-based decisions are still not fully accurate, as they do not have enough reference information to ease the decision process. Therefore, higher performance can be achieved by performing encoding-side estimation and decision based on the error-free information. %Such approach not only improves quality but also reduces the complexity of the video decoders. -%In this chapter a robust coding driven by \ac{ec} methods is developed with the objective of enhancing the quality of reconstructed video at decoders receiving erroneous streams from the transmission network. The proposed approach combines error resilience tools at the encoder-side with \ac{ec} at the decoder-side. +%In this chapter a robust coding driven by \ac{ec} methods is developed with the objective of enhancing the quality of reconstructed video at decoders receiving erroneous streams from the transmission network. The proposed approach combines error resilience tools at the encoder-side with \ac{ec} at the decoder-side. Different approaches were proposed in the past to address the problem of \ac{ec} at the encoder-side \cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ErrorResilienceModeHEVC2016}. Moreover, extra information was used to aid the \ac{ec} operation at the decoder-side, either by sending redundant data \cite{ResilienceRECAP_ICASSP2009,SaliencyErrorConcealment_TM2013} or by embedding data into the bitstream \cite{ConcealmentIntraDataHidng2009}. Although the above methods facilitate the \ac{ec} performance either through slice reordering or based on \acs{mv}, they always rely on the assumption that a fixed \ac{ec} method is used at the decoder. % The work presented in \cite{ConcealmentSignalling2008,SignallingRobustScalable2015} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder. Although such approaches achieves reasonably good performance, it relies on the spatial neighbouring information or multi-layer video, which may not be available in all cases. @@ -55,13 +55,13 @@ In contrast with previous work presented in Chapter~\ref{Chapter:ErrorResilience \section{Traditional error concealment algorithms} \label{Section5:ECCandidates} -In order to proposed a novel error concealment-aware technique different algorithms were taken into consideration. This section provides a description and an evaluation of the error concealment candidates used in the proposed method. +In order to proposed a novel error concealment-aware technique different algorithms were taken into consideration. This section provides a description and an evaluation of the error concealment candidates used in the proposed method. Firstly, a detailed description of each approach is provided followed by a performance comparison between each method. \subsection{Motion-compensation algorithms} %\subsection{Motion-compensated reconstruction} -The most common algorithms to recover missing frames takes advantage of the correctly received \acs{mv} from the neighbouring frames. +The most common algorithms to recover missing frames takes advantage of the correctly received \acs{mv} from the neighbouring frames. This has been widely used in the past for \ac{h264} standard \cite{H264RefConc2006,BlockBasedMVE2002,BlockBasedMVEOverllaped2011} and also in the \ac{hevc} standard~\cite{HEVCMCBlockAug2013,HEVCMCResidueDec2013,DownhillHEVCCon_ICCE2017}. Let's consider the example of Figure~\ref{FIG5:MotionVectorExtrapolation}, where a missing frame ($f_0$) at time instance $t_0$ is reconstructed based the information present in the neighbouring frames, $f_{-1}$ and $f_{-2}$. @@ -69,13 +69,13 @@ One of the common approaches, i.e., motion-copy based algorithms reconstructs th An alternative approach reconstructs the missing frame by using motion compensation through the correctly received \acs{mv}, of frame $f_{-1}$ at time instant $t_{-1}$, according to the following procedure: \begin{enumerate} -\item +\item the \acs{mv} ($v_{-1}$) of the neighbouring frame ($f_{-1}$) are extrapolated to compute the set of \acs{mv} ($v_{0}$) for the missing frame as follows: \begin{equation} % v^\zeta_0 = \frac{ t_0 - t_{C} }{ t_{-1} - t_{R} } \times v^\zeta_{-1},\ \zeta=s,r v_0 = \frac{ t_0 - t_{-1} }{ t_{-1} - t_{R} } \times v_{-1}, \end{equation} -where $t_{R}$ is the time instant of the reference frame pointed by the original \acs{mv} $v_{-1}$. +where $t_{R}$ is the time instant of the reference frame pointed by the original \acs{mv} $v_{-1}$. \item the \ac{mv} associated to the block $b_{-1}(x,y)$ at spatial position $(x,y)$ in $f_{-1}$ may be used to recover the block $b^{'}_{{0}}(x^{'},y^{'})$ in $f_0$, at position $(x^{'},y^{'})$ obtained from $v_0$ components as follows: \begin{align} @@ -96,7 +96,7 @@ Finally, as a base for comparison the default \ac{ec} in the reference \ac{hevc} \subsection{Algorithms based on motion estimation} -In this sub-section different method that can be used to estimate a motion-field are described, which can be used to reconstruct the missing motion information. +In this sub-section different method that can be used to estimate a motion-field are described, which can be used to reconstruct the missing motion information. The obtained motion-field is used in combination with the \ac{mve} algorithm to recover the lost frame. Typically, optical flow estimation algorithms are used to estimate a 2-D \ac{mv} field, $v=(v_x, v_y)$, for each pixel, such as the following energy function is minimised: @@ -127,29 +127,30 @@ $|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of In this method the motion field is estimated based on a polynomial expansion a neighbourhood of a pixel~\cite{PolyFlow2003}. In this method only quadratic polynomials are considered, expressed as: \begin{equation}\label{EQ5:PolyFlow} -f(\mathbf{p}) \sim \mathbf{p}^TA_1\mathbf{p}+\mathbf{b}_1^T\mathbf{p}+C_1 +f(\mathbf{p}) \sim \mathbf{p}A_1\mathbf{p}^T+\mathbf{b}_1\mathbf{p}^T+C_1 \end{equation} -where $\mathbf{p} = (x,y)^T$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ a vector and $C_1$ a scalar. The coefficients are -estimated from a weighted least squares fit to the signal values in the neighbourhood. +where $\mathbf{p} = [x\ y]$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ is a row vector and $C_1$ is a scalar. The coefficients are estimated from a weighted least squares fit to the signal values in the neighbourhood. -Since the result of polynomial expansion is that each neighbourhood is approximated by a polynomial, we start by analysing what happens if a polynomial undergoes an ideal translation. Let's consider the signal $f_1(\mathbf{p})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$ given by $\mathbf{d}=(v_x,v_y)^T$. This can be expressed by the following: +Since the result of polynomial expansion is that each neighbourhood is approximated by a polynomial, we start by analysing what happens if a polynomial undergoes an ideal translation. Let's consider the signal $f_1(\mathbf{p})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$ given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following: \begin{align} f_2(\mathbf{p}) -&= f_1(\mathbf{p}+\mathbf{d}) \nonumber \\ -&= (\mathbf{p}+\mathbf{d})^TA_1(\mathbf{p}+\mathbf{d})+\mathbf{b}_1^T(\mathbf{p}+\mathbf{d})+C_1 \\ -&= \mathbf{p}^TA_2\mathbf{p}+\mathbf{b}_2^T\mathbf{p}+C_2, \nonumber +&= f_1(\mathbf{p}-\mathbf{d}) \nonumber \\ +&= (\mathbf{p}-\mathbf{d})A_1(\mathbf{p}-\mathbf{d})^T+\mathbf{b}_1(\mathbf{p}+\mathbf{d})^T+C_1 \\ +&= \mathbf{p}\mathbf{A}_1\mathbf{p}^T - 2\mathbf{d}\mathbf{A}_1\mathbf{p}^T+ \mathbf{d}\mathbf{A}_1\mathbf{d}^T+ \mathbf{b}_1\mathbf{p}^T-\mathbf{b}_1\mathbf{d}^T + C_1 \nonumber \\ +&= \mathbf{p}\mathbf{A}_1\mathbf{p}^T + (\mathbf{b}_1-2\mathbf{d}\mathbf{A}_1)\mathbf{p}^T + \mathbf{d}\mathbf{A}_1\mathbf{d}^T - \mathbf{b}_1\mathbf{d}^T + C_1\\ +&= \mathbf{p}A_2\mathbf{p}^T+\mathbf{b}_2\mathbf{p}^T+C_2, \nonumber \end{align} -which results in the following relations, +which by equating the coefficients of the quadratic polynomials results in the following relations: \begin{align} \mathbf{A}_2 &= \mathbf{A}_1 \nonumber \\ -\mathbf{b}_2 &= \mathbf{b}_1 + 2 \mathbf{A}_1\mathbf{d}, \\ -C_2 &= \mathbf{d}^T\mathbf{A}_1\mathbf{d}+\mathbf{b}_1\mathbf{d}+C_1. \nonumber +\mathbf{b}_2 &= \mathbf{b}_1 - 2 \mathbf{d} \mathbf{A}_1, \\ +C_2 &= \mathbf{d}\mathbf{A}_1\mathbf{d}^T + \mathbf{b}_1\mathbf{d}^T + C_1. \nonumber \end{align} -This allows to find the displacement $\mathbf{d}$ for any signal dimensions by solving the above equations. Detailed implementations of the polynomial expansion and solving can be found in~\cite{NormalizedDiffConvolution1993,farneback2002polynomial}. +This allows to find the displacement $\mathbf{d}$ for any signal dimensions by solving the above equations. Detailed implementations of the polynomial expansion and respecting solving can be found in~\cite{NormalizedDiffConvolution1993,farneback2002polynomial}. \subsubsection{Large displacement optical flow with deep matching (\textit{DeepFlow})} -In this method the motion field is estimated based on the large displacement optical flow presented in \cite{LargeDisplacementOptFlow_TPAMI2011}, which extends the total variation methods, e.g. the one expressed by Equation (\ref{EQ5:DualFlow}), by including descriptor matching. +In this method the motion field is estimated based on the large displacement optical flow presented in \cite{LargeDisplacementOptFlow_TPAMI2011}, which extends the total variation methods, e.g. the one expressed by Equation (\ref{EQ5:DualFlow}), by including descriptor matching. The correspondences between descriptions are used to apply a coarse-to-fine wrapping, thus avoiding local minimum values. The matching algorithm is built upon a multi-stage architecture with six layers with different patches sizes, interleaving convolutions and max-pooling \cite{DeepFlow2013}. The combination of both approaches increases the matching accuracy. In \cite{DeepFlow2013} to improve the estimated motion field accuracy, the following extension to the method proposed in~\cite{LargeDisplacementOptFlow_TPAMI2011} were introduced: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layers of optimisation. Figure~\ref{FIG5:DeepFlowDiagram} shows the blocks that are included in the estimation process. As shown in the figure a multi-size optimisation is applied by using different patches sizes. This is used to estimate an initial motion field, referred to as $v'$ . @@ -397,7 +398,7 @@ Note that the generated bitstream is compliant with the standard, as the error c \section{Error concealment signalling evaluation} \label{Section5:ECSignallingPerformance} -This section presents the performance of the proposed \ac{ec}-aware optimisation scheme using the \ac{hevc} standard as the basis of the encoding framework. +This section presents the performance of the proposed \ac{ec}-aware optimisation scheme using the \ac{hevc} standard as the basis of the encoding framework. Two different evaluations are presented. In the former, only robust \ac{ec} of intra-frames is evaluated, while in the latter the proposed \ac{ec}-aware optimisation is applied to all frames corresponding to a generic robust video transmission using optimal \ac{ec}. @@ -405,7 +406,7 @@ Two different evaluations are presented. In the former, only robust \ac{ec} of i \label{SEC4:IntraPerformance} As mention before, in this section the evaluation of the proposed method is performed only for intra frames to study their impact on error propagation. As intra coding units result in higher amount of bits, they also need a higher number of packets, making them more prone to transmission errors and losses. -Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets corresponds to intra coded slices. +Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets corresponds to intra coded slices. %Also, by limiting the proposed method to intra frames, the amount of required overhead is reduced. % \begin{table}[tb]