From 935eb96ed4fb0d2036a04195fd75f6eb2a075a21 Mon Sep 17 00:00:00 2001 From: Joao Carreira Date: Fri, 7 Dec 2018 14:08:48 +0000 Subject: [PATCH] Commit before examiners corrections --- 021_VideoCoding/Figures/HEVCBlockDiagram.tikz | 1 + .../HybridCodingBlockDiagramPrePost.tikz | 2 +- .../ConcealmentAwareEncoding.tex | 36 +++++++++---------- 3 files changed, 20 insertions(+), 19 deletions(-) diff --git a/021_VideoCoding/Figures/HEVCBlockDiagram.tikz b/021_VideoCoding/Figures/HEVCBlockDiagram.tikz index 75fb4b5..65df3b8 100644 --- a/021_VideoCoding/Figures/HEVCBlockDiagram.tikz +++ b/021_VideoCoding/Figures/HEVCBlockDiagram.tikz @@ -99,6 +99,7 @@ % \path[blockArrow] (ResidueDiff) -- (Transf); \path[blockArrow] (ResidueDiff) -- (FTu); \path[blockArrow] (FTu) -- (Transf); +\path[blockArrow] (Transf) -- (Quant); \path[blockArrow] (pointIn) |- (GCC); \path[blockArrow] (GCC) -- (ControlData); diff --git a/021_VideoCoding/Figures/HybridCodingBlockDiagramPrePost.tikz b/021_VideoCoding/Figures/HybridCodingBlockDiagramPrePost.tikz index 3953471..c5731ef 100644 --- a/021_VideoCoding/Figures/HybridCodingBlockDiagramPrePost.tikz +++ b/021_VideoCoding/Figures/HybridCodingBlockDiagramPrePost.tikz @@ -121,7 +121,7 @@ \path (w) -- (w -| ReconSumDec) node [point] (z) {}; \draw[-latex,densely dashed] (w1) to [bend right=60] (w2); \draw [thick,blockArrow] (z1)+(-0.1,-0.1) -- (z); -\path[blockArrow] (z) -- node[pos=0.96,left] {-} (ReconSumDec); +\path[blockArrow] (z) -- (ReconSumDec); \path[blockArrow] (outputPosition) |- (RefBufferDec); diff --git a/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex b/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex index ed706f4..7dab0fc 100644 --- a/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex +++ b/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex @@ -47,7 +47,7 @@ Finally, Section~\ref{Section5:Summary} concludes the chapter. In Chapter~\ref{Chapter:ErrorResilienceReview} different \ac{ec} algorithms that can effectively reconstruct lost video were described, though not all of them exhibit the same accuracy for every lost frame or slice. Moreover, it was also shown that combining different \ac{ec} approaches at the block level to improve the reconstruction accuracy is not straightforward. In the past, hybrid \ac{ec} schemes were proposed to improve the quality of recovered frames by either combining spatial and temporal \ac{ec} algorithms~\cite{HybridConcTB2008} or by optimising the \ac{mve} at the pixel and block levels~\cite{HybridConcealmentTB2008,2DConcHybridMotion2010}. -Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of motion-based recovery approaches by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}. +Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of motion-based recovery approaches by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}. Although such methods are able to reduce the reconstruction artefacts and improve the overall video quality, decoder-based decisions are still not fully accurate, as they are based on limited information to be used as reference in the decision process. Therefore, higher reconstructed quality can be achieved by estimating the best \ac{ec} techniques at the encoding-side, based on the original video signal. %Such approach not only improves quality but also reduces the complexity of the video decoders. @@ -59,7 +59,7 @@ Although these methods facilitate the \ac{ec} performance either through slice r To advance one step further, this chapter presents an investigation on \ac{ec}-aware video coding and proposes a new scheme to achieve high decoding flexibility in terms of the \ac{ec} approaches, leading to better video quality. This is accomplished by selecting, at the encoder, among several \ac{ec} algorithms with different characteristics, the one that yields the lowest reconstruction distortion at the decoder and also minimises the \ac{rd} cost considering the necessary signalling rate. Thus, the proposed approach differs from similar previous methods \cite{ConcealmentAwareME2006,ErrorResilienceModeHEVC2016} by selecting the best \ac{ec} method without affecting the encoder optimisation. -To further improve the quality of recovered frames, an enhancement layer is encoded with the mismatch residue computed as a difference between the transmitted and recovered frames. +To further improve the quality of recovered frames, an enhancement layer is encoded with the mismatch residue computed as a difference between the transmitted and recovered frames. This is a dynamic optimisation process, which is more efficient than conventional approaches based on redundant pictures \cite{RedudantPicsJan2009,VPxResiliencePCS2016}, which transmit the whole frame as redundancy. In contrast with the previous work presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}, which addressed the error resilience problem by optimising the coding dependencies, the approach followed in this chapter considers both, \ac{ec} performance and the coded rate required for signalling the best \ac{ec} mode to the decoder. @@ -101,7 +101,7 @@ y^{'} &= y - v_{0y}. \end{align} \end{enumerate} This method, referred to as \glsfirst{mve}, is illustrated in Figure~\ref{FIG5:MotionVectorExtrapolation}, where the \ac{mv} $v^{2}_{0}$ results from the extrapolation of $v^{2}_{-1}$. -One should note that these methods may not result in a complete motion field for the entire frame, resulting in an incomplete reconstruction. +One should note that these methods may not result in a complete motion field for the entire frame, resulting in an incomplete reconstruction. For the regions without \acs{mv} the average of the three top-left neighbouring \acs{mv} are used to complete the motion field. %\todo{Grid-aligment! Incomplete!} @@ -112,7 +112,7 @@ For the regions without \acs{mv} the average of the three top-left neighbouring \label{FIG5:MotionVectorExtrapolation} \end{figure} -Finally, as a base for comparison, the default \ac{ec} in the reference \ac{hevc} implementation the \ac{fc} algorithm is also evaluated. +Finally, as a base for comparison, the default \ac{ec} in the reference \ac{hevc} implementation the \ac{fc} algorithm is also evaluated. This method consists in replacing the lost frame with a copy of the previously decoded one, which can also be described as a motion compensated reconstruction using only zero \acs{mv}. \subsection{Algorithms based on motion estimation} @@ -124,7 +124,7 @@ Typically, optical flow estimation algorithms are used to estimate a 2-D motion \begin{equation} E(x,y) = \sum_{x} \sum_{y} \bigg| f_{0}\big(x,y\big) - f_{1}\big( x + v_x(x,y), y+v_y(x,y) \big) \bigg|. \end{equation} -Besides a full search algorithm, where a block-based approach is used to search for the best \acs{mv} that minimise the sum of absolute errors, three other approaches were tested to evaluate their effectiveness in \ac{ec} algorithms. +Besides a full search algorithm, where a block-based approach is used to search for the best \acs{mv} that minimise the sum of absolute errors, three other approaches were tested to evaluate their effectiveness in \ac{ec} algorithms. These three methods are the following: \begin{enumerate} \item a method based on the total variation using the minimization of the \mbox{L-1} norm~\cite{DualFlow2016}; @@ -134,15 +134,15 @@ These three methods are the following: \subsubsection{Motion estimation based on total variation and regularisation (\textit{DualFlow})} -In this method the motion field is estimated based on the minimization of the total variation obtained using the L-1 norm \cite{DualFlow2007,DualFlow2016}, taking into account the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}. +In this method the motion field is estimated based on the minimization of the total variation obtained using the L-1 norm \cite{DualFlow2007,DualFlow2016}, taking into account the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}. This condition forces the estimated field to be regular across neighbouring pixels by penalising deviations in a quadratic sense. Using this method, for each pixel $\mathbf{p}$, the motion field $v(\mathbf{p})$ is obtained by minimising the following energy function, \begin{equation}\label{EQ5:DualFlow} E(v) = \sum_{\mathbf{p} \in \Omega} \Big( \big| f_{-1}(\mathbf{p}) - f_{-2}( \mathbf{p} + v(\mathbf{p}))\big| + \big|\nabla v(\mathbf{p})\big| \Big)%d\mathbf{p},\ \mathbf{p} \in \Omega \end{equation} -where $f_{-1}$ and $f_{-2}$ are the two previous frames at time instants $t_{-1}$ and $t_{-2}$, as represented in Figure~\ref{FIG5:MotionVectorExtrapolation}. -In Equation~\ref{EQ5:DualFlow}, $\Omega$ defines the sets of pixels of those frames, and $|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of the optical flow which effectively disallow discontinuities. +where $f_{-1}$ and $f_{-2}$ are the two previous frames at time instants $t_{-1}$ and $t_{-2}$, as represented in Figure~\ref{FIG5:MotionVectorExtrapolation}. +In Equation~\ref{EQ5:DualFlow}, $\Omega$ defines the sets of pixels of those frames, and $|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variations of the optical flow which effectively disallow discontinuities. This is based on the assumption that neighbouring regions belong to the same object, thus such regions should have similar motion. @@ -155,7 +155,7 @@ f(\mathbf{p}) \sim \mathbf{p}\mathbf{A}_1\mathbf{p}^T+\mathbf{b}_1\mathbf{p}^T+C \end{equation} where $\mathbf{p} = [x\ y]$, $\mathbf{A_1}$ is a symmetric matrix, $\mathbf{b_1}$ is a row vector and $C_1$ is a scalar. The coefficients are estimated using weighted least squares fit to the signal values in the neighbourhood. -Since the result of a polynomial expansion is such that each neighbourhood is approximated by a polynomial, it is necessary to analyse what happens if a polynomial undergoes an ideal translation. +Since the result of a polynomial expansion is such that each neighbourhood is approximated by a polynomial, it is necessary to analyse what happens if a polynomial undergoes an ideal translation. Let's consider the signal $f_1(\mathbf{p})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$, given by $\mathbf{d}=[v_x \ v_y]$. This can be expressed by the following: \begin{align} f_2(\mathbf{p}) @@ -206,7 +206,7 @@ The smoothness term is a robust penalisation of the gradient of the flow, simila \subsection{Experimental evaluation} -This section presents a simulation study a performance evaluation of the different \ac{ec} schemes previously described. +This section presents a simulation study a performance evaluation of the different \ac{ec} schemes previously described. This evaluation aims at finding the suitable methods to be used for efficient reconstruction of the missing frames in case of data loss. The two \ac{ec} methods based on the correctly received \acs{mv} are tested, i.e., \ac{mc} and \ac{mve}, while the \ac{fc} method is used as reference for comparison. Moreover, the four different approaches to estimate the motion field for the missing frame are compared: (i) full search based on the minimisation of the sum of absolute errors (\FULLSEARCH), (ii) optical flow based on the duality between total variation and smoothness term (\DUALFLOW), (iii) optical flow based on polynomial approximations (\POLYFLOW) and (iv) optical flow based on deep matching (\DEEPFLOW). @@ -231,13 +231,13 @@ According to the results of Table~\ref{TAB5:ECPerformaceComparison} (columns 5 t Results for \FULLSEARCH method reveal that only minimising the sum of absolute errors between two consecutive frames might not always be directly related with objects' motions, which results in inaccurate motion field to be used in the reconstruction of lost frames. Finally, results from Table~\ref{TAB5:ECPerformaceComparison} reveal that the \DEEPFLOW algorithm leads to the most accurate motion field, in comparison with the other methods. It is able to achieve an average quality gain of 1.79 dB, and up to 3.39 dB for Book Arrival sequence, where \POLYFLOW only achieves 3.05 dB. -Since \POLYFLOW combines different techniques also used by other algorithms in study, e.g., total variation also used by \DUALFLOW, it is expected to achieve accurate estimations and consequently higher \ac{ec} reconstruction quality. +Since \POLYFLOW combines different techniques also used by other algorithms in study, e.g., total variation also used by \DUALFLOW, it is expected to achieve accurate estimations and consequently higher \ac{ec} reconstruction quality. Moreover, using feature matching in combination with smoothing conditions, the \DEEPFLOW algorithm is able to correctly estimate large displacements, which results in improved performance for video signals with higher motion, e.g., Tennis sequence. \subsection*{Remarks} Summarising, the analysis of the previous results indicates that all methods are able to cope with lost frames and manage to recover the missing data with reasonable quality. -Nevertheless, it is noticeable that lost frames recovered with an estimated motion field result in higher average quality, especially when using the \DEEPFLOW method. +Nevertheless, it is noticeable that lost frames recovered with an estimated motion field result in higher average quality, especially when using the \DEEPFLOW method. However, since the other methods are also able to achieve acceptable video quality, they shall not be neglected, as they provide alternative approaches for lost frame recovery with a lower computational complexity (i.e., they not require motion estimation). Thus, in order to achieve higher reconstruction quality, different approaches shall be considered, aiming at balancing the overall complexity with the quality of the reconstructed frames. @@ -356,7 +356,7 @@ where $R(m_{l,i})$ represents the number of overhead bits required to encode the In equation~\ref{EQ6:OptimalDepth}, besides the parameter $\lambda$, used to trade-off distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$, according to the duration of the error propagation. As in the method proposed in the Chapter~\ref{Chapter:ErrorResilienceTechniques}, this method uses the ratio between the temporal distance from the current frame ($t_C$) to the \HIGHLIGHT{previous refresh frame} ($t_I$) and the period of refreshing frames ($T_{I}$). %This weight varies between $\left[0;1 \right]$, achieving the lowest value for intra-coded frames and the maximum for the frame located immediately before an intra-coded one. -This behaviour follows the fact that when a lost frame is closer to the next intra-coded frame, the error propagation is lower. +This behaviour follows the fact that when a lost frame is closer to the next intra-coded frame, the error propagation is lower. Thus, it is not worthwhile to spend too many overhead bits to reduce it. This is accomplished by increasing the cost of such bits according to the temporal distance between the missing frame and the next I frame. Moreover, a dynamic value for $\lambda$ is also used to adjust the amount of overhead within the frame, according to the visual relevance of each image region, defined by a saliency map. @@ -465,7 +465,7 @@ Realistic conditions were used for transmission, by allowing independent losses In this section the performance of the proposed method is evaluated only for intra frames, in order to demonstrate its efficiency in reducing error propagation. Since intra \acs{cu} result in higher amount of bits, they also require a higher number of packets, making them more prone to transmission errors and losses. Results indicate that for an intra-period of 16 frames, approximately 42\% of the packets correspond to intra coded slices, which is a non-negligible probability of one of them being hit by transmission errors. -Moreover, by limiting the proposed method to intra frames, the amount of required overhead is reduced. +Moreover, by limiting the proposed method to intra frames, the amount of required overhead is reduced. In this case a constant $\lambda_0=10$ was used, without taking into consideration saliency estimation. Since a relevant performance indicator is the error propagation over temporally predicted frames, this was evaluated using a single intra-frame loss. @@ -584,7 +584,7 @@ In summary, the proposed technique is able to enhance the performance of the sta \label{SEC4:OverallPerformance} In this section the proposed \ac{ec}-aware optimisation scheme is evaluated for a generalised robust video transmission of both intra and inter coded frames. -The experimental results presented in this section were obtained by applying the proposed technique to all video frames. +The experimental results presented in this section were obtained by applying the proposed technique to all video frames. As in the previous subsection, a $\lambda_0=10$ was used. This section is organised in the following subsections: \begin{itemize} @@ -637,10 +637,10 @@ The \ac{ec} candidate methods based on motion field estimation (i.e., M1, M2) ar The evaluation of the proposed saliency-weighted \ac{ec} mode decision (\DYNLAMBDA) was carried out under random loss events and it was compared with the reference methods. The video quality was evaluated by using two quality metrics: \ac{psnr} and a \ac{fsim}~\cite{FSIM_TIP2011}, in order to validate the performance of the proposed method in a consolidated manner. -The results obtained without saliency weighting are also shown for comparison (\FIXEDLAMBDA). +The results obtained without saliency weighting are also shown for comparison (\FIXEDLAMBDA). Tables~\ref{TAB4:SignallingPSNR}~and~\ref{TAB4:SignallingFSIM} show the average quality under different \acs{plr} for the \ac{ld} configuration. -Table~\ref{TAB4:SignallingPSNR} shows the \ac{psnr} and weighted \ac{psnr} (\ac{wpsnr}) between parentheses. +Table~\ref{TAB4:SignallingPSNR} shows the \ac{psnr} and \ac{wpsnr} between parentheses. The \ac{mse} weights used in the \ac{wpsnr} are defined by the saliency values~\cite{SaliencyQualityMetric2011}. Table~\ref{TAB4:SignallingFSIM} shows the results obtained for the \ac{fsim} quality metric. Note that, the absolute value is shown for the reference case (\REF), while the differential values are shown for the remaining methods. @@ -790,6 +790,6 @@ The proposed method uses \ac{ec} techniques with different levels of computati In this chapter a novel \ac{ec}-aware encoding scheme was proposed by incorporating loss simulation at the encoder-side to enhance the EC efficiency at the decoder. % The proposed method relies on loss simulation at the encoder-side to optimise the error concealment proposed. Then, the optimal error concealment is signalled to the decoder. -The novelty of this method is to include optimal \ac{ec} mode decision at the encoder, to be used by the decoder in case of packet loss, and also to allow encoding of the \ac{ec} residue using the enhancement layer of the \ac{hevc} standard. The use of \ac{ec} signalling was firstly proposed for the use on intra-coded frames in (\ref{Conf:ConcAware}), and then improved and proposed for a generic robust video transmission, resulting in a more detailed analysis published in (\ref{Journal:ConcAware}). +The novelty of this method is to include optimal \ac{ec} mode decision at the encoder, to be used by the decoder in case of packet loss, and also to allow encoding of the \ac{ec} residue using the enhancement layer of the \ac{hevc} standard. The use of \ac{ec} signalling was firstly proposed for the use on intra-coded frames in (\ref{Conf:ConcAware}), and then improved and proposed for a generic robust video transmission, resulting in a more detailed analysis published in (\ref{Journal:ConcAware}). The overall results demonstrate that the proposed method is able to efficiently reconstruct lost frames and reduce the error propagation compared with other reference methods. A consistent quality improvement achieved at the cost of low overhead allows the use of this method in broadcasting video services and network applications where packet loss probability is not negligible.