|
|
|
@@ -1,10 +1,4 @@
|
|
|
|
|
% Figure folders
|
|
|
|
|
\newcommand{\ErrorResilienceReviewFigs}{./022_ErrorResilience/Figures}
|
|
|
|
|
\newcommand{\TransmissionResiliencefigs}{./022_ErrorResilience/Figures}
|
|
|
|
|
\newcommand{\ErrorConcealmentReviewFigs}{./022_ErrorResilience/Figures}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\chapter{Robust Video Coding: A Review}
|
|
|
|
|
\chapter{Robust Video Coding: A Review \VERSION{1}}
|
|
|
|
|
\label{Chapter:ErrorResilienceReview}
|
|
|
|
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
@@ -77,150 +71,200 @@ Since these techniques break the predictive coding, they have a negative impact
|
|
|
|
|
\begin{figure}[tb]
|
|
|
|
|
\centering
|
|
|
|
|
\input{./022_ErrorResilience/Figures/LocalizationTechniques.tikz}
|
|
|
|
|
\caption{Examples of spatial and temporal error propagation and how it can be reduced using localisation techniques \cite{ErrorResilienceAug2005}.}
|
|
|
|
|
\caption{Examples of spatial and temporal error propagation and how it can be reduced with spatial and temporal error resilience techniques.}
|
|
|
|
|
\label{fig2:ErrorPropagation}
|
|
|
|
|
%\cite{ErrorResilienceAug2005}
|
|
|
|
|
\end{figure}
|
|
|
|
|
|
|
|
|
|
%\begin{figure}[tb]
|
|
|
|
|
% \centering
|
|
|
|
|
% \includegraphics[width=0.7\columnwidth]{\TransmissionResiliencefigs/ErrorPropagation_1.png}
|
|
|
|
|
% \includegraphics[width=0.7\columnwidth]{./022_ErrorResilience/Figures/ErrorPropagation_1.png}
|
|
|
|
|
% \caption{Examples of spatial and temporal error propagation and how it can be reduce using localisation techniques \cite{ErrorResilienceAug2005}.}
|
|
|
|
|
% \label{fig2:ErrorPropagation}
|
|
|
|
|
%\end{figure}
|
|
|
|
|
|
|
|
|
|
This group of error resilience techniques are able to reduce the error propagation at two levels, i.e., temporal and spatial levels.
|
|
|
|
|
This is illustrated in in Figure~\ref{fig2:ErrorPropagation}, where one can observe that spatial techniques break the dependencies within a single frame, while temporal error propagation can be reduce by selectively breaking the dependencies across different frames.
|
|
|
|
|
This group of error resilience techniques are able to reduce the error propagation at two levels, i.e., temporal and spatial levels. Figure~\ref{fig2:ErrorPropagation} is used to demonstrate the principle of error propagation and how it can be reduced using localisation techniques. The top of the figure illustrates a lost block and the correspondent spatial and temporal error propagation on the left and right, respectively.
|
|
|
|
|
Spatial localisation techniques
|
|
|
|
|
To reduce the error propagation within a frame a row of refresh blocks can be introduced to stop the error propagation.
|
|
|
|
|
To address the temporal error propagation refresh blocks (e.g., intra coded blocks) can be randomly introduce to stop error propagation.
|
|
|
|
|
As shown in the figure the refresh blocks are not affected error propagation.
|
|
|
|
|
In both examples of Figure~\ref{fig2:ErrorPropagation} the regions affected by errors are clearly reduced.
|
|
|
|
|
|
|
|
|
|
\HERE
|
|
|
|
|
|
|
|
|
|
\subsection{Error resilient rate-distortion optimisation}
|
|
|
|
|
|
|
|
|
|
A common localization technique addresses the error resilience problem of the rate-distortion optimisation on video encoders. Normally, such optimisation is equivalent to the minimisation of the Lagrangian cost given by:
|
|
|
|
|
\glsreset{rd}
|
|
|
|
|
|
|
|
|
|
A common localization technique deals with the error resilience problem within the scope of the \ac{rd} optimisation process used in video coding.
|
|
|
|
|
Normally, such optimisation is equivalent to the minimisation of the Lagrangian cost given by:
|
|
|
|
|
\begin{equation}
|
|
|
|
|
J = D + \lambda R,
|
|
|
|
|
\end{equation}
|
|
|
|
|
where $D$ is the distortion of all pixels, $R$ the required bitrate, and $\lambda$ is the Lagrangian multiplier.
|
|
|
|
|
In \cite{ROPEJun2000,ROPE2007,ROPERefGenJan2010} a rate distortion optimisation for error resilience was presented.
|
|
|
|
|
In order to take into consideration possible network losses a \ac{rope} of the end-to-end distortion is applied at the encoder-side.
|
|
|
|
|
Let $f_t^i$ denote the original value of pixel $i$ in frame at instant $t$, $\hat f_t^i$ denote its encoder reconstruction and $\tilde f_t^i$ the decoder reconstruction after error concealment if needed.
|
|
|
|
|
Since data loss may occur in the transmission channel, $\tilde f_t^i$ is considered a random variable at the encoder.
|
|
|
|
|
The expected mean square error of each pixel is given by:
|
|
|
|
|
\begin{align}
|
|
|
|
|
E\left\{d_t^i\right\} &= E\left\{ \left( f_t^i - \tilde f_t^i\right)^2 \right\} \nonumber \\
|
|
|
|
|
&= (f_t^i)^2 - 2 f_t^i E\left\{ \tilde f_t^i \right\} + E\left\{ \left(\tilde f_t^i\right)^2 \right\}
|
|
|
|
|
\end{align}
|
|
|
|
|
Then to take into consideration the channel losses packet loss probability ($p$) must be used in the distortion measurement.
|
|
|
|
|
This leads to the following for intra-coded pixels:
|
|
|
|
|
where $D$ is the distortion of all pixels, $R$ the bitrate and $\lambda$ is the Lagrangian multiplier.
|
|
|
|
|
Such optimisation only takes into account the coding distortion ($D$), which is only suitable for error-free environments.
|
|
|
|
|
Therefore, alternative approaches error resilient \ac{rd} optimisation, which take into consideration possible network losses~\cite{DistortionTrackingH264_ICIP2004}.
|
|
|
|
|
More advanced approaches can take advantage of a \ac{rope} of the end-to-end distortion at the encoder-side~\cite{ROPEJun2000}.
|
|
|
|
|
Let $f_t^i$ denote the original value of pixel $i$ in frame at instant $t$, $\hat f_t^i$ denote its reconstruction in the encoder and $\tilde f_t^i$ the reconstruction in the decoder after \ac{ec}.
|
|
|
|
|
Since data loss may occur in the transmission channel, $\tilde f_t^i$ is modelled as a random variable at the encoder.
|
|
|
|
|
The expected \ac{mse} of each pixel is given by:
|
|
|
|
|
\begin{equation}
|
|
|
|
|
E\left\{d_t^i\right\} = E\left\{ \left( f_t^i - \tilde f_t^i\right)^2 \right\} = (f_t^i)^2 - 2 f_t^i E\left\{ \tilde f_t^i \right\} + E\left\{ \left(\tilde f_t^i\right)^2 \right\}
|
|
|
|
|
\end{equation}
|
|
|
|
|
Then the packet loss probability ($p$) is taken into consideration and used in the distortion measurement.
|
|
|
|
|
In case of intra-coded pixels the following equation used:
|
|
|
|
|
\begin{align}\label{EQ2:E2EDIntra}
|
|
|
|
|
E\left\{ \tilde f_t^i \right\} &= \left(1-p\right) \hat f_t^i + p E\left\{ \tilde f_{n-1}^i \right\} \nonumber \\
|
|
|
|
|
E\left\{ \tilde f_t^i \right\} &= \left(1-p\right) \hat f_t^i + p E\left\{ \tilde f_{n-1}^i \right\} \\
|
|
|
|
|
E\left\{\left( \tilde f_t^i \right)^2 \right\} &= \left(1-p\right) \left( \hat f_t^i \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\},
|
|
|
|
|
\end{align}
|
|
|
|
|
and inter-coded pixels:
|
|
|
|
|
while in the case inter-coded pixels:
|
|
|
|
|
\begin{align}\label{EQ2:E2EDInter}
|
|
|
|
|
E\left\{ \tilde f_t^i \right\} &= \left(1-p\right) \left( \hat e_t^i + E\left\{ \tilde f_{n-1}^{j} \right\} \right) + p E\left\{ \tilde f_{n-1}^i \right\} \nonumber \\
|
|
|
|
|
E\left\{ \tilde f_t^i \right\} &= \left(1-p\right) \left( \hat e_t^i + E\left\{ \tilde f_{n-1}^{j} \right\} \right) + p E\left\{ \tilde f_{n-1}^i \right\} \\
|
|
|
|
|
E\left\{\left( \tilde f_t^i \right)^2 \right\} &= \left(1-p\right) \left( \left( \hat e_t^i \right)^2 + 2 \hat e_t^i E\left\{ \tilde f_{n-1}^{j} \right\} + E\left\{ \left( \tilde f_{n-1}^{j} \right)^2 \right\} \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\}
|
|
|
|
|
\end{align}
|
|
|
|
|
where the inter-coded pixel $i$ is predicted from pixel $j$ in the previous frame.
|
|
|
|
|
assuming that the pixel $i$ is predicted from pixel $j$ in the previous frame.
|
|
|
|
|
The prediction error $e_t^i$, is quantized to the value $\hat e_t^i$, which is transmitted to the decoder along with the motion information.
|
|
|
|
|
Equations \ref{EQ2:E2EDIntra} and \ref{EQ2:E2EDInter} allow to measure the distortion taking into consideration the compression error but also the possible of frame loss and consequent \ac{ec} distortion.
|
|
|
|
|
This technique was used in \cite{ROPEJun2000} to switch between intra and inter coding modes reducing the temporal dependencies, and, thus achieving superior error robustness in video transmission. In \cite{AdvancesROPEHE2642007} this technique was optimized to quarter pixel estimation in \ac{h264} standard. Moreover, in \cite{ROPEBurst2009} this method is extended for burst losses by introducing the probability of burst losses into the original end-to-end estimation. This shown to improve the quality under frame loss.
|
|
|
|
|
In \cite{EndToEndDistortionConcealment2014} the distortion estimation assumes a temporal \ac{ec} at the decoder-side.
|
|
|
|
|
This increases the performance of the proposed as it does not rely of frame-copy which leads to lower reconstruction quality. More advanced \ac{ec} method were also consider in the study presented in~\cite{EndToEndDistortionConcealment2014}.
|
|
|
|
|
Equations \ref{EQ2:E2EDIntra} and \ref{EQ2:E2EDInter} provide a measure of the distortion taking into consideration not only the coding distortion but also the possibility of frame loss and consequent \ac{ec} distortion.
|
|
|
|
|
|
|
|
|
|
This technique was used in \cite{ROPEJun2000} to switch between intra and inter coding modes with the aim of reducing the temporal dependencies, and, thus achieving enhanced error robustness in video transmission.
|
|
|
|
|
In \cite{AdvancesROPEHE2642007} this technique was optimized to quarter pixel estimation, and then in \cite{ROPEBurst2009} the method was extended for burst losses, by introducing the burst loss probability into the original end-to-end estimation.
|
|
|
|
|
Results revealed that higher video quality was achieved by considering both probabilities.
|
|
|
|
|
In \cite{E2EDistortion2011} a temporal \ac{ec} is considered in the distortion estimation, which increases the performance of the proposed method, by avoiding frame-copy that provides a lower bound on reconstruction quality.
|
|
|
|
|
More advanced temporal \ac{ec} methods were also considered in the work presented in~\cite{EndToEndDistortionConcealment2014}.
|
|
|
|
|
The \acs{mv} used for motion compensation reconstruction is obtained using boundary matching search, which is able to achieve lower distortion when compared with frame-copy. Using more advanced \ac{ec} algorithms leads to a more accurate estimation of the end-to-end distortion, thus increasing the error robustness of the transmitted video.
|
|
|
|
|
To take into account the delay introduced in the encoding loop due to hierarchical prediction structures, delay based \ac{rd} optimisation was proposed in \cite{DelayRD_TCVT2015} by estimating both the distortion and the end-to-end delay which is minimised along the required bitrate. Using this approach the total delay can be controlled without severely compromising the coding efficiency.
|
|
|
|
|
In the context of \ac{hevc} end-to-end distortion estimation was also used in the \ac{rd} optimisation \cite{ErrorResilienceModeHEVC2016} showing significant quality improvements comparing with the reference implementation.
|
|
|
|
|
|
|
|
|
|
Different quality metrics can also be used for end-to-end distortion estimation. The method presented in \cite{SSIMModeling2015} proposes the distortion estimation using the \ac{ssim} \cite{QualitySSIMRef2002} in order to increase the correlation between the estimated distortion and the perceptual quality noticed by the human visual system~\cite{VisualQualityMetrics2011}. Moreover, the \ac{ssim} quality metric in learning based \ac{rd} modelling~\cite{RateDistortionNN_VCIP2017}, using a \acs{cnn} to predict both the distortion and the required rate, showing good accuracy in the \ac{rd} cost estimation.
|
|
|
|
|
|
|
|
|
|
In order to increase the error robustness of the \ac{hevc} standard distortion estimation was also used in the \ac{rd} optimisation \cite{ErrorResilienceModeHEVC2016} showing significant quality improvements comparing with the reference \ac{hevc} implementation.
|
|
|
|
|
The method in \cite{SSIMModeling2015} proposes the estimation of the end-to-end distortion using the \ac{ssim} \cite{QualitySSIMRef2002} due to its trade-off between complexity and efficiency to evaluate the perceptual quality \cite{VisualQualityMetrics2011}.
|
|
|
|
|
|
|
|
|
|
\subsection{Intra-refreshing}
|
|
|
|
|
|
|
|
|
|
In order to reduce the spatial dependencies one can introduce refresh points into the bitstream through intra-coded blocks \cite{IntraRefresh1999,IntraRefreshAttentionCM2007,IntraRefreshRefSelectionJan2009,IntraRefreshICIP2009,SandroIntraRefresh2011,JointIntraRefresh2014,IntraRefreshFeedback2015,IntraRefreshRPS2017}.
|
|
|
|
|
These techniques are fairly efficient to promote the error robustness of bitstreams, which avoid long error propagation at the temporal level.
|
|
|
|
|
The main feature of these techniques is to increase the number of intra-coded blocks as they have no dependencies to previous frames, however at the expense of higher amount of bits~\cite{IntraRefreshAttentionCM2007}.
|
|
|
|
|
Therefore, different alternatives have been proposed optimising the number of intra-coded blocks introduced by reducing it the refresh regions.
|
|
|
|
|
Such refresh regions can be introduced randomly, periodically, based on the motion information or on the end-to-end distortion estimation.
|
|
|
|
|
Moreover, whenever is possible the encoder can use a feedback channel to decide where to introduce refresh regions~\cite{IntraRefresh1999,IntraRefreshFeedback2015}.
|
|
|
|
|
|
|
|
|
|
In the \ac{h264} reference software it was introduced a non-normative technique for random intra refreshing which introduces cyclic refresh not in raster scan order. Instead a random pattern is applied. This technique guarantees that all \acs{mb} are refreshed at some point, therefore errors do not propagate indefinitely. However, this technique refresh all \ac{mb} exactly the same number of times without consideration for rate distortion optimisation neither the region containing the \acs{mb}.
|
|
|
|
|
In \cite{IntraRefreshICIP2008} a rate control scheme is used to decide if a given \ac{mb} should be refreshed using a intra-coded block. The decision is performed based on the ratio of the Lagrangian cost between the best intra and inter mode. Whenever this ration is lower than a threshold a intra mode is selected. Using this guarantees that intra refreshing only forces an intra prediction mode if its Lagrangian cost is closer to optimal coding method. In \cite{IntraRefreshICIP2009} this is further extended to adapt the intra refresh algorithm to the network conditions, i.e., the threshold value is obtained from the bitrate and the \ac{plr} value. This technique automatically increases the number of intra-coded block as the bitrate and the packet loss increases. In case of \ac{hevc} intra refresh is also being used to reduce error propagation \cite{HEVCIntraRefreshICIP2017}. It was proposed to use joint intra-inter prediction to introduce soft-refresh points in the bitstream. Although such regions are not correctly decoded in case of errors (as happens with intra-refresh), it significantly decreases the bitrate increasing due the amount of intra-coded blocks. When compared with traditional intra-refresh bitrate saving of up to 12\% are achieved for a 5\% \gls{plr}.
|
|
|
|
|
|
|
|
|
|
In order to reduce the effects of error propagation within a single frame spatial and temporal dependencies can be limited by introducing refresh points into the bitstream by forcing intra-coded blocks\cite{IntraRefresh1999,IntraRefreshAttentionCM2007,IntraRefreshRefSelectionJan2009,IntraRefreshICIP2009,SandroIntraRefresh2011,JointIntraRefresh2014,IntraRefreshFeedback2015,IntraRefreshRPS2017}.
|
|
|
|
|
This technique is fairly efficient to improve the error robustness of video streams, by avoiding long error propagation both within a single frame but also across several subsequent frames.
|
|
|
|
|
The main characteristic of this group of techniques is to increase the number of intra-coded blocks as they have no dependencies to previous frames.
|
|
|
|
|
However, this is done at the expense of higher amount of bits~\cite{IntraRefreshAttentionCM2007}.
|
|
|
|
|
Therefore, different alternatives have been proposed, which optimise the number of intra-coded blocks, which can be introduced randomly, periodically~\cite{SandroIntraRefresh2011}, based on the motion information or rate-distortion optimisation~\cite{IntraRefreshICIP2008,HEVCIntraRefreshICIP2017}.
|
|
|
|
|
Moreover, whenever is possible the encoder can use a feedback channel to decide where refresh regions should be introduced~\cite{IntraRefresh1999,IntraRefreshFeedback2015}.
|
|
|
|
|
|
|
|
|
|
\begin{figure}[tb]
|
|
|
|
|
\centering
|
|
|
|
|
\includegraphics[width=0.7\columnwidth]{\TransmissionResiliencefigs/ROI_IntraRefresh_1.png}
|
|
|
|
|
\caption{Examples of spatial and temporal error propagation and how it can be reduce using localisation techniques \cite{IntraRefreshAttentionCM2007}.}
|
|
|
|
|
\label{fig2:IntraRefreshAttention}
|
|
|
|
|
\centering
|
|
|
|
|
\input{./022_ErrorResilience/Figures/IntraRefresh.tikz}
|
|
|
|
|
\caption{Examples of spatial and temporal error propagation and how it can be reduce using localisation techniques.}
|
|
|
|
|
\label{FIG2:IntraRefresh}
|
|
|
|
|
\end{figure}
|
|
|
|
|
|
|
|
|
|
A different approach introduces refresh points based on the video contents. In~\cite{IntraRefreshAttentionCM2007} the error resilience of video streams is increased combining rate control and attention based models to define \acs{roi}. A weighting factor is used in the Lagrangian optimization to increase the number of intra blocks areas determined by the attention models. Using the aforementioned method lower refresh points are required to achieve high error resilience in the \ac{roi}.
|
|
|
|
|
A more recent approach was proposed in \cite{IntraRefreshPerceptual2010} using a perceptual noticeable difference model to determine an estimation of noticeable distortion. Then, using a visual attention model presented in [XX] intra refresh blocks are introduce in the bitstream. This method is able to improve the results of a previous work \cite{IntraRefresh2005}, especially for high motion sequences. This is due to the fact the previous approach only used an estimation of the error propagation without accounting for perceptual information.
|
|
|
|
|
A non-normative method for random intra refreshing is implemented in the \ac{h264} reference software, which introduces cyclic refresh points using a random pattern.
|
|
|
|
|
Figure~\ref{FIG2:IntraRefresh} (a) shows an example of intra-coded blocks (dark gray squares) being randomly places into the coded frames to prevent error propagation.
|
|
|
|
|
This approach guarantees that all \acs{mb} are refreshed at some point, therefore errors do not propagate indefinitely.
|
|
|
|
|
However, all \acs{mb} are refreshed exactly the same number of times without consideration for rate distortion optimisation neither the image region containing the \acs{mb}.
|
|
|
|
|
In \cite{IntraRefreshICIP2008} a rate control scheme is used to decide if a given block should be refreshed using a intra-coded prediction.
|
|
|
|
|
The decision is performed based on the ratio between the Lagrangian cost of the best intra and inter modes.
|
|
|
|
|
Whenever this ratio is lower than a certain predefined threshold an intra mode is selected, which guarantees that intra refreshing is only forced when the Lagrangian cost of the intra mode is close to the optimal coding method.
|
|
|
|
|
This is further extended to adapt the intra refresh algorithm to the network conditions, i.e., the threshold value is obtained from the bitrate and the loss ratio~\cite{IntraRefreshICIP2009}.
|
|
|
|
|
Therefore, the number of intra-coded block automatically increases as the bitrate and the loss ratio increases.
|
|
|
|
|
In the context of \ac{hevc} intra refresh is also being used to reduce error propagation.
|
|
|
|
|
Soft-refresh points using joint intra-inter prediction were proposed for this purpose~\cite{HEVCIntraRefreshICIP2017}.
|
|
|
|
|
Although the image regions cannot be correctly decoded in case of errors (as happens with intra-refresh), this method significantly decreases the bitrate increase due to extra intra-coded blocks.
|
|
|
|
|
When compared with traditional intra-refresh, bitrate savings of up to 12\% are achieved for a 5\% \gls{plr}.
|
|
|
|
|
|
|
|
|
|
This was be also exploited for the \ac{hevc} standard \cite{HEVCROIErrorResilience2015} in combination with a feedback channel. Based on the received information the encoder can introduce intra \acs{cu} in the \ac{roi} areas. Experimental results reveal an increasing on the achieved video quality when higher number of intra \acs{cu} are used. Moreover, applying the proposed method to the \ac{roi} exclusively lower bitrate overhead is introduced leading to higher rate distortion gains.
|
|
|
|
|
\glsreset{roi}
|
|
|
|
|
|
|
|
|
|
A different approach is to introduce refresh points based on the video content.
|
|
|
|
|
In~\cite{IntraRefreshAttentionCM2007} the error resilience of video streams is increased by combining rate control and attention based models to define \acs{roi}.
|
|
|
|
|
A weighting factor is used in the Lagrangian optimization to increase the number of intra-coded blocks in the image regions determined by the attention models.
|
|
|
|
|
Figure~\ref{FIG2:IntraRefresh} (b) shows an example with a \ac{roi} marked with a black line which varies for every frame. In this case the intra-coded blocks are only introduced inside the marked region, in contrast to what is shown in Figure~\ref{FIG2:IntraRefresh} (a).
|
|
|
|
|
Results show that using the aforementioned method, less refresh points are required to improve the error robustness of the \ac{roi}. When comparing with \cite{DistortionTrackingH264_ICIP2004}, approximately 4 dB of \ac{psnr} gains are achieved in the \ac{roi} without compromising the overall video quality.
|
|
|
|
|
A more recent approach was proposed using a perceptual noticeable difference model to estimate the noticeable distortion, and then using a visual attention model \cite{SaliencyItti1998} to decide which blocks should be used as intra refresh in the bitstream~\cite{IntraRefreshPerceptual2010}.
|
|
|
|
|
This method is able to improve the results of a previous work \cite{IntraRefresh2005}, especially for high motion sequences. This is due to the fact that such previous approach only used an estimation of the error propagation without accounting for perceptual information.
|
|
|
|
|
|
|
|
|
|
A similar approach was also exploited for \ac{hevc} in \cite{HEVCROIErrorResilience2015,HEVCRoiIntraRefresh_DSP2016}, by using a feedback channel to provide information about the network errors, which is then used by the encoder to introduce intra-coded blocks into the \ac{roi}.
|
|
|
|
|
The simulation results reveal an increasing on the achieved video quality when higher number of intra blocks are used. Moreover, applying the proposed method only to the \ac{roi}, the bitrate overhead introduced is lower, leading to higher \ac{rd} gains.
|
|
|
|
|
|
|
|
|
|
% \HIGHLIGHT{INTRA REFRESHING}
|
|
|
|
|
|
|
|
|
|
\subsection{Leaky-predictions}
|
|
|
|
|
|
|
|
|
|
Directly using reconstructed past frames as reference frames for predictive coding, may result in substantial error propagation whenever the reference frames are lost.
|
|
|
|
|
In order to reduce error propagation one can employ leaky predictions \cite{LeakyPred2009,ROILeakyPred2009}, which scales down reconstructed frames.
|
|
|
|
|
This leads to an exponential decay of error propagation, which leads to higher error robustness.
|
|
|
|
|
In order to generate such reference frames the following can be applied
|
|
|
|
|
Directly using reconstructed past frames as reference frames for predictive coding, may result in substantial error propagation whenever the reference frames are either lost or reconstructed from other distorted frames.
|
|
|
|
|
To reduce error propagation one can employ leaky predictions, which scales down reconstructed frames before being used as reference~\cite{LeakyPred2009,ROILeakyPred2009}.
|
|
|
|
|
This leads to an exponential decay of error propagation, with direct impact on error robustness.
|
|
|
|
|
In this approach, the reconstructed reference frames are scaled down by applying the following:
|
|
|
|
|
\begin{equation}\label{EQ2:LeakyPredConstant}
|
|
|
|
|
\check f_t^i = \alpha \hat f_t^i + (1- \alpha)C,
|
|
|
|
|
\end{equation}
|
|
|
|
|
where $\check f_t^i$ is a reference frame pixel, $\alpha$ is the leak factor, and $C$ is a constant value.
|
|
|
|
|
The use of leaky predictions has been used in the past for error control especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in Equation~(\ref{EQ2:LeakyPredConstant}) is replace by a lower layer error-free reconstruction. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate as shown in \cite{LeakyPredConstant1973}.
|
|
|
|
|
where $\check f_t^i$ is a reference frame, $\alpha$ is the leaky factor used to control the amount of information from the reconstructed frame present in the new frame $\check f_t^i$, and $C$ is a constant value.
|
|
|
|
|
Leaky prediction has been used in the past for error control, especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in Equation~(\ref{EQ2:LeakyPredConstant}) is replaced by a lower layer error-free reconstructed frame. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate as shown in \cite{LeakyPredConstant1973}. The $\alpha$ parameter allows to control the effect of the constant value, and thus the coding efficiency. By decreasing $\alpha$ the coding efficiency increases, however, the exponential decay in error propagation becomes less significant.
|
|
|
|
|
|
|
|
|
|
In order to overcome the coding efficiency problem weighted prediction can be used with similar advantages. Since \ac{h264} motion compensation can be applied from two individual previously encoded frames and combined together to generate a single prediction for the current frame \cite{WeightedBiPred2005}. In the case of \ac{hevc}, a method was proposed to force the use of dual inter-frame prediction~\cite{DualFrameME2015} leading to lower error propagations.
|
|
|
|
|
In case of leaky prediction instead of applying motion compensation is applied once, but with an interpolated frame obtained as follow:
|
|
|
|
|
% MHMCP_TCSVT2006
|
|
|
|
|
|
|
|
|
|
In order to overcome the loss in coding efficiency, weighted prediction can achieve similar advantages, by using motion compensation from two previously encoded frames to generate a single prediction for the current frame \cite{WeightedBiPred2005}.
|
|
|
|
|
Alternatively, the two previously encoded frames can be used to generate an interpolated frame ($ \check f_t^i $) to be used as reference, using:
|
|
|
|
|
%In case of leaky prediction instead of applying motion compensation is applied once, but with an interpolated frame obtained as follow:
|
|
|
|
|
\begin{equation}\label{EQ2:LeakyPredDual}
|
|
|
|
|
\check f_t^i = \alpha \hat f_t^i + (1- \alpha) \hat f_{n-1}^i.
|
|
|
|
|
\tilde f_t^i = \alpha \hat f_t^i + (1- \alpha) \hat f_{n-1}^i.
|
|
|
|
|
\end{equation}
|
|
|
|
|
Using $\check f_t^i$ as reference frame for prediction significantly decrease error propagation if only one of the original coded frames is lost (i.e., $\hat f_{n}^i$ or $\hat f_{n-1}^i$).
|
|
|
|
|
Therefore, it is possible to decrease the error propagation using $\tilde f_t^i$ as reference frame for prediction, if only one of the original encoded frames is lost (i.e., $\hat f_{n}^i$ or $\hat f_{n-1}^i$). This is due to the fact that $\tilde f_t^i$ contains information from both $\hat f_{n}^i$ and $\hat f_{n-1}^i$.
|
|
|
|
|
In Equation (\ref{EQ2:LeakyPredDual}) $\hat f_{n}^i$ and $\hat f_{n-1}^i$ always correspond to the encoded frames and the generated frames ($\tilde f_t^i $) are not used to interpolate subsequent frames. This corresponds to a finite impulse response filter.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\begin{figure}[tb]
|
|
|
|
|
\centering
|
|
|
|
|
\begin{tabular}{x{4cm}x{4cm}x{4cm}}
|
|
|
|
|
\multicolumn{3}{c}{\includegraphics[width=12cm]{\TransmissionResiliencefigs/ROILeakyPrediction.png}} \\
|
|
|
|
|
$f_{n-1}$ & $f_{n}$ & $f_{n+1}$ \\
|
|
|
|
|
\end{tabular}
|
|
|
|
|
\caption{Example of error propagation with a ROI (shaded area indicates error, and the arrows indicate temporal predictions to previous frame) \cite{ROILeakyPred2009}.}
|
|
|
|
|
\label{fig2:ErrorPropagationROI}
|
|
|
|
|
\centering
|
|
|
|
|
% \begin{tabular}{x{4cm}x{4cm}x{4cm}}
|
|
|
|
|
% \multicolumn{3}{c}{\includegraphics[width=12cm]{./022_ErrorResilience/Figures/ROILeakyPrediction.png}} \\
|
|
|
|
|
% $f_{n-1}$ & $f_{n}$ & $f_{n+1}$ \\
|
|
|
|
|
% \end{tabular}
|
|
|
|
|
% \caption{Example of error propagation with a ROI (shaded area indicates error, and the arrows indicate temporal predictions to previous frame) \cite{ROILeakyPred2009}.}
|
|
|
|
|
\input{./022_ErrorResilience/Figures/ROILeakyPrediction.tikz}
|
|
|
|
|
\caption{Example of error propagation with two different regions; as shown errors can propagate from the background to the ROI.}
|
|
|
|
|
\label{fig2:ErrorPropagationROI}
|
|
|
|
|
\end{figure}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In \cite{ROILeakyPred2009} in order to increase the error robustness with lower impact on the coding efficiency leaky prediction is confined to a pre-defined \ac{roi}.
|
|
|
|
|
Figure \ref{fig2:ErrorPropagationROI} shows an example of error propagation within two different regions: \ac{roi} region and background.
|
|
|
|
|
As show in the figure, since the \ac{roi} regions have predictions from regions in the background error propagation across regions occurs.
|
|
|
|
|
Therefore, the method in \cite{ROILeakyPred2009} uses a weighted prediction for the \ac{roi} region obtained as follows: 1) a globally optimal prediction obtained by unrestricted motion estimation, 2) a prediction obtained by restricting motion estimation within the \ac{roi} region. This limits the impact on \ac{roi} from errors that affect background region. Moreover, since the leaky predictions are confined to a specific regions higher coding efficiency is achieved when compared with other methods.
|
|
|
|
|
Equation (\ref{EQ2:LeakyPredDual}) results in a finite impulse response based weighting of reference frames, as previously weighted frames are not considered. In \cite{ROPERefGenJan2010} an infinitive impulse response based scheme is proposed to increase the decay in error propagation. To accomplish that, in Equation (\ref{EQ2:LeakyPredDual}) the use of the previous encoded frame ($\hat f_{n-1}^i.$) is replaced by the previous interpolated frame ($\check f_t^i$).
|
|
|
|
|
The idea of combining multiple predictions was also been used in the \ac{hevc} standard by combining it with end-to-end distortion to select the optimal predictions.
|
|
|
|
|
In \cite{ROILeakyPred2009} a method is presented where the leaky prediction is confined to a pre-defined \ac{roi}, in order to increase the error robustness with lower impact on the coding efficiency.
|
|
|
|
|
Figure \ref{fig2:ErrorPropagationROI} shows an example of error propagation within two different regions: \ac{roi} and background. As show in the figure, since the \ac{roi} uses background regions for predictions error propagation occurs across regions.
|
|
|
|
|
To overcome this problem, the method in \cite{ROILeakyPred2009} uses a weighted prediction for the \ac{roi} region obtained from the following predictions: (i) a globally optimal prediction obtained by unrestricted motion estimation, (ii) a prediction obtained by restricting motion estimation within the \ac{roi} region.
|
|
|
|
|
This limits the impact of errors in the background region on the \ac{roi}.
|
|
|
|
|
Moreover, since the leaky predictions are only confined to some specific regions, higher coding efficiency is achieved when compared with other methods.
|
|
|
|
|
|
|
|
|
|
%\HERE
|
|
|
|
|
|
|
|
|
|
The method proposed in \cite{ROPERefGenJan2010} extends the traditional leaky prediction expressed in Equation (\ref{EQ2:LeakyPredDual}) by replacing the previous encoded frame ($\hat f_{n-1}^i.$) with the previous interpolated frame ($\tilde f_t^i$). This results in the following:
|
|
|
|
|
\begin{equation}\label{EQ2:LeakyPredIIR}
|
|
|
|
|
\tilde f_t^i = \alpha \hat f_t^i + (1- \alpha) \tilde f_{n-1}^i.
|
|
|
|
|
\end{equation}
|
|
|
|
|
This corresponds to infinite impulse response filter.
|
|
|
|
|
Comparing both equations for the same $\alpha$, the Equation (\ref{EQ2:LeakyPredIIR}) implies stronger filtering than Equation (\ref{EQ2:LeakyPredDual}). Therefore, reference frames generated with heavier filtering have higher resilience against error propagation, but less correlation with the original frame, which negatively impacts inter predictions and hence coding efficiency.
|
|
|
|
|
The idea of combining multiple predictions was also been used in the \ac{hevc} standard by combining it with end-to-end distortion to select the optimal predictions~\cite{DualFrameME2015}.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\subsection{Reference picture selection}
|
|
|
|
|
|
|
|
|
|
In order to reduce the temporal dependencies and reduce the error propagation reference picture selection has been widely used in the past \cite{ResilienceDualFrameBuffer2004,ErrorResilienceRPSFeb2009,RPSErrorsFeb2009,MVConcatenationAug2011}.
|
|
|
|
|
In \cite{ResilienceDualFrameBuffer2004}, an optimisation algorithm was used to select between short and long term references based on the end-to-end distortion obtained using the \ac{rope} method, in order to increase the prediction distance and reduce error propagation.
|
|
|
|
|
This method can also be used in combination with feedback channels to improve the references frame selection.
|
|
|
|
|
In the decoder side, an \ac{ec} method was later used to take advantage of such prediction structure \cite{ConcealmentDualFrameBuffer2005}.
|
|
|
|
|
Reference picture selection has been widely used in the past to reduce the temporal dependencies and reduce the error propagation \cite{ResilienceDualFrameBuffer2004,ErrorResilienceRPSFeb2009,RPSErrorsFeb2009,MVConcatenationAug2011}.
|
|
|
|
|
In \cite{ResilienceDualFrameBuffer2004}, an optimisation algorithm was devised to select between short and long term references based on the \ac{rope} end-to-end distortion estimation method. This was used to increase the prediction distance and reduce error propagation.
|
|
|
|
|
The same method can also be used in combination with feedback information from the network to improve the references frame selection.
|
|
|
|
|
In the decoder-side, an \ac{ec} method was later used to take advantage of such prediction structure \cite{ConcealmentDualFrameBuffer2005}.
|
|
|
|
|
|
|
|
|
|
\begin{figure}[b]
|
|
|
|
|
\begin{figure}[tb]
|
|
|
|
|
\centering
|
|
|
|
|
\includegraphics[width=0.95\columnwidth]{\TransmissionResiliencefigs/RefPicSelection.png}
|
|
|
|
|
\caption{Examples of reference picture selection in error resilience video transcoding \cite{ErrorResilienceRPSFeb2009}.}
|
|
|
|
|
\includegraphics[width=0.95\columnwidth]{./022_ErrorResilience/Figures/RefPicSelection.png}
|
|
|
|
|
\caption{Examples of dynamic reference picture selection used in error resilient video transcoding \cite{ErrorResilienceRPSFeb2009}.}
|
|
|
|
|
\label{fig2:ReferencePictureSelection}
|
|
|
|
|
\end{figure}
|
|
|
|
|
|
|
|
|
|
A feedback channel was also used in \cite{ErrorResilienceRPSFeb2009}, to reduce error propagation at end-user decoders, by dynamically selecting the correctly received frames as references. Figure~\ref{fig2:ReferencePictureSelection} shows an example of error resilience transcoding, where based on network feedback the original temporal predictions (represented by the solid arrows) are transcoded to use the correctly received frame, i.e., prediction represented by the dashed arrow.
|
|
|
|
|
A more advanced scheme may be considered, where only the regions affected by mismatch predictions are process in the compressed domain, and subsequently transcoded, thus limit the dependencies to lost frames.
|
|
|
|
|
This was further exploited in \cite{MVConcatenationAug2011}. To limit complexity motion vector concatenation is employed, in order to obtain new inter-predictions to error-free frames.
|
|
|
|
|
In \cite{ChannelAdaptiveVideoStreaming2002} feedback-based is combined with path diversity to select the reference frames that are more likely to be received at the decoder.
|
|
|
|
|
In the proposed approach, the feedback is not directly used to select the reference frames.
|
|
|
|
|
Instead, the reference frames are selected through an end-to-end distortion estimation, taking into account the source distortion and the channel distortion (concealment distortion is also taken into account in channel distortion).
|
|
|
|
|
In \cite{ErrorResilienceModeHEVC2017} a \ac{roi} is defined based on the motion objects is used to limit the reference picture selection based on a feedback channel.
|
|
|
|
|
Moreover, intra refresh is also used based on the temporal distance to the previous intra-refresh frame.
|
|
|
|
|
A feedback information from the network was also used in \cite{ErrorResilienceRPSFeb2009}, to reduce error propagation at end-user decoders, by dynamically selecting the reference frame, only from those correctly received.
|
|
|
|
|
Figure~\ref{fig2:ReferencePictureSelection} shows an example of an error resilience transcoding scheme, where the original temporal predictions (represented by the solid arrows) are transcoded to use only correctly received frames as reference, i.e., prediction represented by the dashed arrow. This is done based on network feedback.
|
|
|
|
|
A more advanced scheme may be considered, where only the regions affected by mismatch predictions are processed in the compressed domain, and subsequently transcoded to limit the dependencies from lost frames. To achieve low complexity \ac{mv} concatenation can be used, in order to obtain new inter-predictions for error-free frames \cite{MVConcatenationAug2011}.
|
|
|
|
|
In \cite{ChannelAdaptiveVideoStreaming2002} network feedback is combined with path diversity to select the reference frames that are more likely to be received at the decoder.
|
|
|
|
|
In the proposed approach, the feedback information is not directly used to select the reference frames.
|
|
|
|
|
Instead, the reference frames are selected through an end-to-end distortion estimation process, taking into account the coding distortion and distortion resulting from channel errors (\ac{ec} distortion is also taken into account).
|
|
|
|
|
In \cite{ErrorResilienceModeHEVC2017} a \ac{roi} is defined based on the motion objects to limit the regions where the mechanism of reference picture selection is applied. Intra refresh is also used based on the temporal distance to the previous intra-refresh frame. This method not only achieves significant quality gains when compared with the reference \ac{hevc} standard but also when comparing with tradition reference selection schemes.
|
|
|
|
|
|
|
|
|
|
In~\cite{IntraRefreshRefSelectionJan2009} an algorithm to introduce intra refresh points at the macroblock level was proposed for the \ac{h264} standard. This method takes into consideration the distortion when no concealment is applied, the distortion when frame-copy concealment is applied and the packet loss conditions, in order to select the refresh blocks. Those refresh blocks are encoded using either intra prediction or inter prediction through a secondary algorithm to select the most reliable reference frame (i.e., frame with less total expected distortion).
|
|
|
|
|
In~\cite{IntraRefreshRefSelectionJan2009} an algorithm to introduce intra refresh points at the \ac{mb} level was proposed.
|
|
|
|
|
This method takes into consideration the decoding distortion assuming that frame-copy concealment is applied for \ac{ec} when packet loss occurs.
|
|
|
|
|
The refresh blocks are encoded using either intra prediction or inter prediction through a secondary algorithm to select the most reliable reference frame (i.e., frame with less total expected distortion).
|
|
|
|
|
|
|
|
|
|
In the methods mentioned above the dependencies between the frames are analysed and reduced in order to decrease the error propagation in cases of data loss. Since the existing video coding standards use intense predictive coding, these methods present an efficient approach in highly erroneous conditions.
|
|
|
|
|
In the methods mentioned above the dependencies between the frames are analysed and reduced in order to decrease the error propagation in cases of data loss.
|
|
|
|
|
Since the existing video coding standards are strongly based on predictive coding, these methods present an efficient approach in highly erroneous conditions.
|
|
|
|
|
% However, the method in~\cite{RDReferenceSelectionJul2012} uses more complex distortion estimations, and in ideal conditions multiple loop encoders are needed to select the reference frames optimally.
|
|
|
|
|
Although these closed-loop techniques may offer a reliable control of the decoder distortion, the feedback required from the receiver side and multipath networking are not always available neither feasible in real-time and unidirectional channels (e.g. broadcast).
|
|
|
|
|
However, although these closed-loop techniques may offer a reliable control of the decoder distortion, the feedback required from the receiver side and multipath networking are not always available neither feasible in real-time and unidirectional channels (e.g. broadcast).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
@@ -229,62 +273,115 @@ Although these closed-loop techniques may offer a reliable control of the decode
|
|
|
|
|
\section{Data partition and redundant coding}
|
|
|
|
|
\label{Section22:DataPartitionRedundant}
|
|
|
|
|
|
|
|
|
|
In a compressed bitstream different symbols have different relevance in terms of error propagation, therefore the impact on the decoded quality is dependent on lost symbols. Therefore, data partition techniques have been developed to group different parts of the compressed bitstream according to their importance to the decoded quality. Then, some of those groups can be protected more efficiently than others using \ac{uep}. These techniques are normally associated with \ac{fec} algorithms. By applying multi-levels of \ac{fec}, different levels of error protection are achieved.
|
|
|
|
|
In a compressed video streams different symbols have different relevance in terms of their impact on error propagation.
|
|
|
|
|
Therefore, there is an unequal contribution of different syntax elements to the decoding distortion by error propagation, depending on the lost symbols.
|
|
|
|
|
Data partition techniques have been developed to provide a mechanism to cope with the unequal importance of coded information by grouping different parts of a compressed bitstream according to their importance to the decoded quality.
|
|
|
|
|
Then, some of these groups can be protected more efficiently than others using \ac{uep} schemes. These are normally associated with \ac{fec} algorithms, whereby multi-levels of \ac{fec} are applied to achieve different levels of error protection. This is presented and discussed with more detail in Sub-section~\ref{SEC22:UEP}
|
|
|
|
|
|
|
|
|
|
In a single channel transmission, the important groups can be protected with stronger channel codes than the remaining groups. In case of multi-channel the important groups can be routed over the reliable channels. Moreover, to achieve higher quality in multi-channel transmission one can use extra information to increase the error resilience of video streams, and be able to decode then in case of errors. Redundancy can be added in an explicit form, e.g., redundant picture, or it can be implicit, e.g., multiple description coding \cite{MDC2005}.
|
|
|
|
|
In a single channel transmission, the most important data or syntax elements (e.g., headers) can be protected with stronger channel codes than the remaining ones.
|
|
|
|
|
In case of multi-channel communications, the important data can be routed over the most reliable channels.
|
|
|
|
|
Moreover, to achieve higher quality in multi-channel transmission one can use extra information to increase the error resilience of video streams, in order to increase the probability of being able to decode then in case of errors.
|
|
|
|
|
For instance, redundancy can be added either in an explicit form, e.g., redundant picture, or implicit, e.g., multiple description coding \cite{MDC1993}. Section~\ref{SEC22:RedundantPics}~and~\ref{SEC22:MDC} present a detailed review of these two approaches, respectively.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\subsection{Flexible Macroblock Ordering}
|
|
|
|
|
\subsection{Data partitioning and flexible macroblock ordering}
|
|
|
|
|
|
|
|
|
|
The \ac{h264}~\cite{H264OverviewJul2003} introduced an error resilience tool named \ac{fmo} technique~\cite{H264FMO2006}. This technique allows each \ac{mb} to be assigned freely to a specific slice group (i.e., a set of slices), organized in a mixed-up fashion. The standard supports up to eight slices groups in each picture, and \acs{mb} within a group can be assigned to a several slices in the default raster scan order.
|
|
|
|
|
The case where only one slice groups is used corresponds to the case where the \ac{fmo} tools is disabled.
|
|
|
|
|
\ac{fmo} increases the error resilience by partitioning the bitstream into several groups. Results show that by using \ac{fmo} it is possible to enhance the subjective and objective quality especially for high packet losses with at cost of an affordable overhead~\cite{H264FMO2006}.
|
|
|
|
|
\glsreset{fmo}
|
|
|
|
|
The concept of data partitioning was firstly introduced in the H.263 coding standard to split the coded data into different categories\cite{H264OverIPJul2003}. It was further improved in the next standard and it allows the symbols in each slice to be divided into three partitions:
|
|
|
|
|
\begin{itemize}
|
|
|
|
|
\item Type A: headers, \ac{mb} types, quantization parameters and motion information.
|
|
|
|
|
\item Type B: intra-coded information, such as, intra blocks signalling and intra coefficients. This type cannot be used without the partitions of type A being present.
|
|
|
|
|
\item Type C: inter-coded information and inter coefficients. This type also requires the information of type A but can be decoder without intra-coded information (type B).
|
|
|
|
|
\end{itemize}
|
|
|
|
|
These partitions have different levels of importance, being data partitions of type A the most important as they carry the header information which are required by the remaining partitions. Then, as partitions of type B carry intra-coded information they are more important than type C information as it can stop temporal error propagation.
|
|
|
|
|
|
|
|
|
|
\ac{h264} \ac{fmo} support six types of slice groups. Figure~\ref{fig2:FMOTypes} shows \ac{fmo} types 0 to 5. In the \ac{fmo} Type 0 each slice groups has a maximum number of \acs{mb} before the next slice group begins. However, the same slice groups can appear more than once in a frame. \ac{fmo} Type 1 corresponds to scattered slices, where the \acs{mb} are assigned to each slice groups with a fixed formula. In \ref{fig2:FMOTypes} it is represented the \ac{fmo} type 1 case with two slice groups resulting in a checkerboard pattern. In the \ac{fmo} Type 2 slice groups are defined by rectangular shapes defined by the top-left and bottom-right corners. Although the standard allows slice groups to overlap each other, each \ac{mb} may only belong to one slice group. \ac{fmo} Type 3, 4 and 5 are special types each allow the configuration to change on every frame. It only allows two slice groups, in order to keep the signalling overhead affordable. The last \ac{fmo} type is the most general case, where the slice groups are explicitly defined using high level packets (i.e., signalling \ac{nal} units), allow more flexibility at the cost of higher signalling overhead.
|
|
|
|
|
The \ac{h264} introduced an error resilience tool named \ac{fmo} technique, which allows each \ac{mb} to be freely assigned to a specific slice group (i.e., a set of slices), organized in a mixed-up fashion~\cite{H264FMO2006}.
|
|
|
|
|
The standard supports up to eight slice groups in each picture, and the \acs{mb} within a group can be assigned to several slices in the default raster scan order.
|
|
|
|
|
The case where only one slice group is used corresponds to the case where the \ac{fmo} tool is disabled.
|
|
|
|
|
\ac{fmo} increases the error resilience by partitioning the bitstream into several groups, which is able to reduce the error propagation.
|
|
|
|
|
Results show that the use of \ac{fmo} contributes to enhance the subjective and objective quality, especially for high packet loss rates at cost of an affordable overhead~\cite{H264FMO2006}.
|
|
|
|
|
|
|
|
|
|
Six types of slice groups are supported by \ac{fmo}. Figure~\ref{fig2:FMOTypes} shows \ac{fmo} types 0 to 5.
|
|
|
|
|
In the \ac{fmo} type 0 each slice group is defined by a maximum number of \acs{mb}, and a new slice group beings whenever this maximum value is reached. The same slice group can be cyclic repeated within a frame.
|
|
|
|
|
\ac{fmo} type 1 corresponds to scattered slices, where the \acs{mb} are assigned to each slice group with a fixed formula. Figure~\ref{fig2:FMOTypes} represents the \ac{fmo} type 1 with two slice groups resulting in a checkerboard pattern.
|
|
|
|
|
In the \ac{fmo} type 2 slice groups are defined by rectangular shapes bounded by the top-left and bottom-right corners.
|
|
|
|
|
Although the slice groups are allowed to overlap each other, each \ac{mb} may only belong to one slice group. \ac{fmo} type 3, 4 and 5 are special types, each one allowing the configuration to change on every frame.
|
|
|
|
|
Nevertheless, only two slice groups are allowed in \ac{fmo} type 3 to 5, in order to keep the signalling overhead affordable.
|
|
|
|
|
The last \ac{fmo} type is the most generic case, where the slice groups are explicitly defined using high level packets (i.e., signalling \ac{nal} units), allowing more flexibility at the cost of higher signalling overhead.
|
|
|
|
|
|
|
|
|
|
\begin{figure}[tb]
|
|
|
|
|
\centering
|
|
|
|
|
\includegraphics[width=0.8\columnwidth]{\TransmissionResiliencefigs/FMO_Slices_Type.png}
|
|
|
|
|
\includegraphics[width=0.8\columnwidth]{./022_ErrorResilience/Figures/FMO_Slices_Type.png}
|
|
|
|
|
\caption{Different types of flexible macroblock reordering \cite{H264FMO2006}.}
|
|
|
|
|
\label{fig2:FMOTypes}
|
|
|
|
|
\end{figure}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In the past \ac{fmo} has been used widely used to increase the error robustness of video transmissions~ \cite{FMOErrorResilience2007,FMOResilience2010,FMOResilience2011,FMOROI2012,MotionEnergyDataPart2014} in \ac{h264} streams. In ~\cite{FMOErrorResilience2007} \ac{fmo} is used to increase the \ac{ec} performance and thus reduce the error propagation in H.264. The method divides the \acs{mb} into the two slice groups based on the \ac{ec} distortion using a two-pass approach. Then, the slice groups which has more impact of the \ac{ec} distortion is transmitted using a reliable channel, and it is assumed to be received error-free. Results show that the proposed approach is able to outperform the dispersed mode (\ac{fmo} type 1).
|
|
|
|
|
In \cite{FMOResilience2010} slice classification is also used based on the weighted sum of the motion vector information and the rate-distortion cost. As in the previous approach, the \acs{mb} are ranked based on the importance and distributed in the two groups until the same amount of \acs{mb} are presented in the two slice groups. This was further improved in \cite{FMOResilience2011} by applying a prediction based on the previous frame to estimate the mapping of the \acs{mb} into the slice groups. This reduces the computational complexity of the \ac{mb} grouping while achieving similar error resilience performance.
|
|
|
|
|
The method in \cite{FMOROI2012} divides the video into two regions which are then grouped in different slice groups. In order to classify the different \ac{mb} spatial information is devised from the \ac{mb} bit allocation, and temporal information is extracted based on the \ac{ec} distortion. This technique is able to reduce the number of erroneous \ac{mb} up to 82\% achieving quality of up to 3.97\%, measured with \ac{psnr}, when comparing to the case where \ac{fmo} is not used.
|
|
|
|
|
In \cite{MotionEnergyDataPart2014} a method based on motion energy is also used to classify \acs{mb} into different groups by .
|
|
|
|
|
In order to classify a given \ac{mb} into a groups its motion energy is compared with the motion energy of its neighbours and the average motion energy of the entire frame. This allows to obtain a dynamic threshold which is then used to determine the \ac{mb} importance. This method is compared with other \ac{uep} error protection schemes showing quality improvements under different \acs{plr}.
|
|
|
|
|
In the past, \ac{fmo} has been widely used to increase the error robustness of video transmissions in different communication scenarios \cite{FMOErrorResilience2007,FMOResilience2010,FMOResilience2011,FMOROI2012,MotionEnergyDataPart2014}.
|
|
|
|
|
In ~\cite{FMOErrorResilience2007} \ac{fmo} is used to increase the \ac{ec} performance and thus reduce the error propagation in prioritised video transmission.
|
|
|
|
|
The method divides the \acs{mb} into the two slice groups based on an estimation of the \ac{ec} distortion using a two-pass approach. Then, the slice group which has the most impact on the \ac{ec} distortion is transmitted using a reliable channel and it is assumed to be received error-free.
|
|
|
|
|
Results show that the proposed approach is able to outperform the dispersed mode (\ac{fmo} type 1).
|
|
|
|
|
Slice classification can also be based on the weighted sum of the motion vector information and the rate-distortion cost~\cite{FMOResilience2010}.
|
|
|
|
|
As in \cite{FMOErrorResilience2007}, the \acs{mb} are ranked based on the importance and distributed in the two groups with the same amount of blocks. In order to reduce the complexity of mapping the \acs{mb} into groups the same map can be repeated for several subsequent P-frames \cite{FMOResilience2011}.
|
|
|
|
|
This reduces the computational complexity of the \ac{mb} grouping while achieving similar error resilience performance.
|
|
|
|
|
The method in \cite{FMOROI2012} splits the video frames into two regions, which are then grouped in different slice groups.To classify the different \acs{mb}, spatial information is extracted from the \ac{mb} bit allocation, and temporal information is also extracted based on the estimation of the \ac{ec} distortion. This technique is able to reduce the number of erroneous \acs{mb} up to 82\% achieving a quality gain of up to 3.97\%, in terms of \ac{psnr}, in comparison with the case where \ac{fmo} is not used.
|
|
|
|
|
Motion energy was also used to classify \acs{mb} into different groups \cite{MotionEnergyDataPart2014}, by comparing the motion energy of a given \ac{mb} with the motion energy of its neighbours and the average motion energy of the entire frame.
|
|
|
|
|
This allowed the authors to obtain a dynamic threshold which is then used to determine the \ac{mb} importance. In comparison with other \ac{uep} error protection schemes, this method achieve good quality improvements under different \acs{plr}.
|
|
|
|
|
|
|
|
|
|
In \cite{FMORedudantSlices_TB2007} a new \ac{fmo} type is proposed combining the type 1 and 3 in order to obtain a checkerboard pattern in a box-out scanning order, thus increasing the scattering of lost \acs{mb}. Results show that the proposed \ac{fmo} type is able to outperform the existing types under single and burst loss events.
|
|
|
|
|
The method described in \cite{FMOTypeROI2014} introduces a new type in order to efficiently divide the video contents in two regions with different levels of importance, based on the \ac{fmo} Type 6. Then, a spiral protection is applied to scan the \acs{mb} grouping then in two groups. Moreover, by proposing a new \ac{fmo} type they are able to signal several \ac{roi} areas achieving lower overhead than using the original \ac{fmo} Type 6.
|
|
|
|
|
In \cite{FMORedudantSlices_TB2007} a new \ac{fmo} type is proposed combining the slice groups type 1 and 3 in order to obtain a checkerboard pattern in a box-out scanning order, with the objective of increasing the scattering of lost \acs{mb}. Results show that the proposed \ac{fmo} type is able to outperform the existing types under single and burst loss events.
|
|
|
|
|
The method described in \cite{FMOTypeROI2014} introduces a new type in order to efficiently divide the video content of each frame in two regions with different levels of importance, based on the \ac{fmo} type 6.
|
|
|
|
|
%Then, a spiral protection is applied to scan the \acs{mb} grouping then in two groups.
|
|
|
|
|
By proposing a new \ac{fmo} type they are able to signal several \ac{roi} areas achieving lower overhead than using the original \ac{fmo} Type 6.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\subsection{Unequal error protection}
|
|
|
|
|
\label{SEC22:UEP}
|
|
|
|
|
|
|
|
|
|
\glsreset{uep}
|
|
|
|
|
|
|
|
|
|
Video signals normally have different types of content, both at the temporal and spatial level. Therefore, there is the possibility of categorising it into different groups. Moreover, it is well-know that different regions of a video bitstream are affected differently by transmission errors.
|
|
|
|
|
This motivated the use of \Gls{uep} techniques in video streaming, which allows different parts of bitstream to be better protected against errors. This not only ensures that those regions will be delivered with higher quality, but also leads to a reduction of the total overhead required to increase the error robustness of video streams.
|
|
|
|
|
\Gls{uep} techniques are used to protect different parts of a video stream against errors according to their importance for the decoding quality. \ac{uep} approaches take advantage of the video stream characteristics that result from different types of content, both at the temporal and spatial level. This allows categorisation of coded data into different groups, by exploiting the fact that different regions are affected differently by transmission errors.
|
|
|
|
|
This not only ensures that those regions will be delivered with higher quality, but also leads to efficient use of the total overhead required to achieve increased the error robustness in video transmission.
|
|
|
|
|
|
|
|
|
|
\ac{uep} has been used for a long time to increase the error robustness of video transmission. A wide variety of unequal loss protection schemes have been developed in the literature to optimize the performance of video communication over networks with packet loss. It has been observed that, within a \ac{gop}, different frame types (e.g. I, P, or B frames) have different levels of contribution to the reconstructed video quality. This has been exploited for unequal error protection in video transmission as shown in~\cite{UEP_TM2005,UEPMotion_TM2006}.
|
|
|
|
|
A wide variety of \ac{uep} schemes have been presented in the literature to optimize the performance of video communication over networks where the probability of packet loss is not negligible.
|
|
|
|
|
It has been observed that, within a \ac{gop}, different frame types (e.g. I, P, or B frames) have different levels of contribution to the reconstructed video quality when transmission losses occur. This has been exploited for \ac{uep} in video transmission as shown in~\cite{UEP_TM2005,UEPMotion_TM2006}.
|
|
|
|
|
|
|
|
|
|
In \cite{UEPLayered_TB1999} an evaluation of layered video was described. In order to increase the error robustness, redundant packets are used which ease the recovering of lost ones. In order to reduce the overhead for error protection each video layer was given a different importance and protected using different levels of redundancy, thus guaranteeing that lower level packets have a higher probability of being decoded without errors. This is a simple approach not dependent on the network capabilities, neither requires a deep knowledge of the video signals. A lightweight model is used to assign packets priorities to be used by error protection algorithms~\cite{UEP_TCE2011}. This way an \ac{uep} can be achieved. The model is based on a weighted sum of four components: (i) a slice type parameter giving higher importance to refresh and reference frames; (ii) a binary flag indicating where a network packet transports video related headers, i.e., \ac{nal} or slice header; (iii) a parameter indicating the distance to the packet containing the next slice; (iv) a parameter indicating the distance to the next refresh frame. The encoder-side is able to extract these parameters based on the \gls{nal} header, which makes it suitable to low-end encoders or even to prioritise each packet at a streaming level.
|
|
|
|
|
In \cite{UEPLayered_TB1999} an evaluation of layered video was proposed to increase the error robustness, by using redundant packets for helping the recovery of lost ones.
|
|
|
|
|
In order to achieve reduced overhead for error protection, each video layer has a different importance and is protected using different levels of redundancy, thus guaranteeing that packets carrying data of lower layers have higher probability of being decoded without errors.
|
|
|
|
|
This is a simple approach, not dependent on the network capabilities, neither requiring a deep knowledge of the video contents.
|
|
|
|
|
A lightweight model is used to assign packet priorities to be used by \ac{uep} algorithms~\cite{UEP_TCE2011}. Such model is based on a weighted sum of four components: (i) a slice type parameter giving higher importance to refresh and reference frames; (ii) a binary flag indicating whether a network packet transports video stream headers, i.e., \ac{nal} or slice header; (iii) a parameter indicating the distance in packets to the next slice; (iv) a parameter indicating the temporal distance to the next refresh frame.
|
|
|
|
|
These parameters can be obtained by parsing the \gls{nal} header, which makes it suitable to low-end encoders or even to prioritise each packet at a streaming level.
|
|
|
|
|
|
|
|
|
|
However, \ac{uep} techniques usually require a deep video analysis which is difficult its integration in every transmission scenario.
|
|
|
|
|
For instance, one can use the motion information to classify different frames or even different regions within a frame, e.g., \ac{mb} in \ac{h264}\cite{UEPMotion_TM2006}. The motion information is used to determine the motion energy which is given by the L2-norm of the \acs{mv} within a determined region.
|
|
|
|
|
In~\cite{UEPMotionEnergy_WCNC2013} the motion energy of each frame is computed and compared with neighbouring frames. To achieve this the energy of co-located \acs{mb} in subsequent frames are compared, and higher importance is given to frames with higher motion energy differences. This method was compared with equal error protection revealing higher objective and subjective quality revealing the relation between motion information and impact of errors.
|
|
|
|
|
This work was extended in order to use a two levels unequal protection~\cite{UEPMotion_ICSPCC2012,UEPMotion_ICSPCS2013}. Not only \ac{mb} and frame importance is taken into consideration but also the combined importance of different frames are used to determine the importance of each \ac{gop}.
|
|
|
|
|
However, \ac{uep} techniques usually require a deep video analysis which in general contributes to increase the encoding complexity.
|
|
|
|
|
Nevertheless, one can use information already needed for encoding, such as motion, to classify different frames or even different regions within a frame~\cite{UEPMotion_TM2006}.
|
|
|
|
|
The motion information is used to determine the motion energy which is given by the L2-norm of the \acs{mv} within a predictive regions.
|
|
|
|
|
For instance, in~\cite{UEPMotionEnergy_WCNC2013} the motion energy of each frame is computed and compared with neighbouring frames. The energy of co-located block in adjacent frame is compared, and higher importance is given to those frames that have higher motion energy differences.
|
|
|
|
|
This method was compared with equal error protection, resulting in higher objective and subjective quality and revealing the relation between motion information and impact on errors.
|
|
|
|
|
This work was extended in ~\cite{UEPMotion_ICSPCC2012,UEPMotion_ICSPCS2013} to use a two levels \ac{uep} approach, where besides taking into consideration the \ac{mb} and frame importance, the combined importance of different frames is also used to determine the level of importance of each \ac{gop}.
|
|
|
|
|
|
|
|
|
|
In \cite{Zhang20131390} the transmission distortion is estimated at the packet level, to assess the impact of each packet in the error propagation. Then, different priorities are applied to each packet allow different levels of error protection. This was also be used to schedule the packet transmission through multi path networks\cite{PacketScheduling2010}.
|
|
|
|
|
In order to improve the classification of different regions of a video stream a perceptual algorithm can be used. This increases the correlation between the classification and subjective impact of errors.
|
|
|
|
|
In \cite{UEPSaliency_TCSVT2010} an method based on a combination of two parameters: a spatial parameter based on a perceptual attention model and a temporal parameters based on the frame position within \ac{gop}. The parameter is then used for \ac{fec} assignment, and experimental results show a significant quality improvement both used objective metrics and subjective evaluation.
|
|
|
|
|
Another work dealing with \ac{uep} is described in~\cite{Zhang20131390}, where the transmission distortion is estimated at the packet level, to assess the impact of each packet in the error propagation.
|
|
|
|
|
Then, different priorities are applied to each packet, allowing different levels of error protection. This was also used to schedule the packet transmission through multi-path networks\cite{PacketScheduling2010}.
|
|
|
|
|
Furthermore, in order to improve the classification of different regions of a video stream a perceptual algorithm can be used.
|
|
|
|
|
This strategy increases the correlation between the classification and subjective impact of errors.
|
|
|
|
|
In \cite{UEPSaliency_TCSVT2010} a method based on a combination of two parameters is described: a spatial parameter based on a perceptual attention model and a temporal parameter based on the frame position within the \ac{gop}.
|
|
|
|
|
These parameters are then used to improve the efficiency of \ac{fec} assignment, and experimental results show a significant quality improvement both using objective metrics and subjective evaluation.
|
|
|
|
|
|
|
|
|
|
In scalable video coding scheme, the layered structure of the bit stream provides a natural framework for \ac{uep} \cite{UEPScalable_TCE2008,UEPScalable_TCSVT2010,UEPScalable_ICSPICES2015,UEP_TM2018}.
|
|
|
|
|
In~\cite{UEPScalable_TCSVT2010} an algorithm to estimate the overall distortion of reconstructed frame in the decoder is proposed for scalable video. At the encoder-side different enhancement layer truncation, error propagation and \ac{ec} at taken into account to perform the distortion estimation. The method performs a recursively computation of the total expected distortion in the decoder at the picture-level for each layer in the prediction hierarchy. This is used to perform a cross-layer optimisation and apply different levels of error protection.
|
|
|
|
|
The work in~\cite{UEPLayerVideo_TM2014} proposes a technique for finding the optimal coding rates for coded bit-streams at the encoder-side, in order to optimise the necessary \ac{fec} codes at the enhancement layer which aid the decoding of erroneous base-layer video. This not only improves the overall decoded video quality but also prevents that the correctly received enhancement layers are discarded due to missing dependencies.
|
|
|
|
|
\glsreset{el}
|
|
|
|
|
|
|
|
|
|
The scalable video coding framework provides implicit support for \ac{uep} through the layered structures defined with different levels of quality~\cite{UEPScalable_TCE2008,UEPScalable_TCSVT2010,UEPLayerVideo_TM2014, UEPScalable_ICSPICES2015}.
|
|
|
|
|
In~\cite{UEPScalable_TCSVT2010} an algorithm to recursively estimate the total expected distortion in the decoder at the picture-level for each layer is proposed.
|
|
|
|
|
This method takes into consideration \ac{el} truncation, i.e., a specific layer being discarded, and \ac{ec} to measure an accurate estimation of the end-to-end distortion. The estimation is carried out for every layer in the video stream, in order to apply an optimisation of different levels of error protection across several layers.
|
|
|
|
|
Also based on scalable coding, the work in~\cite{UEPLayerVideo_TM2014} proposes a technique for finding the optimal \ac{fec} coding rates, which allow for a correct \ac{el} decoding when errors affect the base layer. This prevents discarding correctly received \acs{el} frames due to missing dependencies, increasing the overall video quality.
|
|
|
|
|
|
|
|
|
|
In the context of \ac{hevc} there are also methods to achieve \ac{uep}.
|
|
|
|
|
Since techniques like \ac{fmo} are not available, different approaches were investigated \cite{SliceImportance_ICHNRSR2015,HEVCUepLte_IWCMCC2015,UEPVideoDegradation_ISCAS2017,UEP_TM2018}
|
|
|
|
|
The work in~\cite{SliceImportance_ICHNRSR2015} proposes a resource allocation scheme based on prioritisation at the slice level (i.e., \ac{nal} unit level) for \ac{lte} Advanced networks~\cite{HEVCUepLte_IWCMCC2015}.
|
|
|
|
|
The prioritisation is performed by analysing the inter-frame predictions dependencies, giving higher rankings to slices having higher number of dependent pixels.
|
|
|
|
|
In~\cite{UEPVideoDegradation_ISCAS2017} a quality degradation model is proposed for \ac{hevc} video, by
|
|
|
|
|
simulating error events and measuring the quality using the \ac{psnr} metric.
|
|
|
|
|
From the empirical results a model was devised to predict the quality under different events.
|
|
|
|
|
Then, the model is used to control the amount of redundant codes to be applied at the network level.
|
|
|
|
|
The scalable extension of the \ac{hevc} standard was also used to support \gls{uep} for video storage in cloud environments~\cite{UEP_TM2018}, by combining the layer information (i.e., base or enhancement layer) and the picture type (i.e., I, P or B-frame) to maximise the performance of the \ac{fec} codes both at the temporal and quality/resolution levels.
|
|
|
|
|
|
|
|
|
|
In the context of \ac{hevc} there are also methods to achieve unequal error protection. Since techniques like \ac{fmo} are not available, different approaches were taken into account \cite{HEVCUepLte_IWCMCC2015,UEPVideoDegradation_ISCAS2017,UEP_TM2018}.The work in~\cite{SliceImportance_ICHNRSR2015} proposes a resource allocation scheme based on prioritisation for \ac{lte} Advanced networks~\cite{HEVCUepLte_IWCMCC2015}. The prioritisation is performed at the slice level (i.e., \ac{nal} unit level) based on the inter-frame predictions. Such predictions are analysed and a ranking is given based on the amount of pixels predicted based on the current slice. In~\cite{UEPVideoDegradation_ISCAS2017} a quality degradation model is proposed for video encoded using the \ac{hevc} standard. The quality degradation is obtained by simulation error events and measuring the quality using the \ac{psnr} metric. From the empirical results a model can be devised to predict the quality under different events. This model is used to control the amount of redundant codes are applied at the network level. The work in~\cite{UEP_TM2018} proposes the use of \gls{uep} for video storage in cloud environments in the context of the scalable extension of the \ac{hevc} standard. The proposed method not only takes into consideration the layer information, but also considers the picture type (i.e., I, P or B-frame) to achieve a cross-optimisation.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -293,49 +390,62 @@ In the context of \ac{hevc} there are also methods to achieve unequal error prot
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
|
%\section{Redundant Coding}
|
|
|
|
|
\subsection{Redundant pictures}
|
|
|
|
|
\label{SEC22:RedundantPics}
|
|
|
|
|
|
|
|
|
|
Redundant pictures is a well-known technique adopted in the \ac{h264} standard \cite{RedudantPicsJan2009}, which allows for different representations of the same picture to be transmitted using different encoding parameters.
|
|
|
|
|
Redundant pictures is a well-known technique adopted in the \ac{h264} standard \cite{RedudantPicsJan2009}, which allows for different representations of the same coded picture to be transmitted using different encoding parameters.
|
|
|
|
|
For example, the primary picture may be coded with fine quantization, while the redundant one with coarse quantisation at lower bitrate.
|
|
|
|
|
One the one hand if the primary picture is received, the redundant information is discarded.
|
|
|
|
|
One the other hand if errors occur and the primary picture is lost, the redundant one is used to provide a lower level of reconstructed quality and contain the error propagation.
|
|
|
|
|
On the one hand if the primary picture is received, the redundant information is discarded.
|
|
|
|
|
On the other hand if errors occur and the primary picture is lost, the redundant one is used to provide a lower level of reconstructed quality and limiting the error propagation.
|
|
|
|
|
|
|
|
|
|
Zhu \emph{et al.} investigated the paradigm of redundant picture coding in \ac{h264} standard~\cite{RedudantPicsJan2009}.
|
|
|
|
|
Different approaches were investigated in order to achieve a mixed method based on both reference picture selection to obtain a clean decoding without error propagation and a hierarchical allocation of the redundant pictures to increase the error robustness. In~\cite{VPxResiliencePCS2016} the key frames are encoded and transmitted at lower bitrate to be used in case of error to replace the missing frame.
|
|
|
|
|
Redundant pictures achieve good performance under error prone conditions, since they introduce secondary pictures that can be decoded when the primary picture is not available.
|
|
|
|
|
Zhu \emph{et al.} investigated the paradigm of redundant picture coding~\cite{RedudantPicsJan2009}, considering different approaches in order to develop a mixed method based on both reference picture selection to obtain a clean decoding without error propagation and hierarchical allocation of the redundant pictures to increase error robustness.
|
|
|
|
|
In~\cite{VPxResiliencePCS2016} the key frames are encoded and transmitted at lower bitrate to be used in case of error to replace the missing frame.
|
|
|
|
|
Redundant pictures are a good mechanism to improve performance under error prone networking conditions, since they introduce secondary pictures that can be decoded when the primary picture is not available.
|
|
|
|
|
|
|
|
|
|
Moreover, redundant pictures might also work as localisation techniques when different reference frames are selected. However, such approaches generate large amount of redundant data and overheads. In~\cite{RedundantMVsJune2008} a newer approach was proposed to reduce the amount of information in the redundant pictures of the \ac{h264} standard, by introducing only redundant \acs{mv}. This was also used in~\cite{VPxResilience2016}, where the motion information is used to recover the missing frame.
|
|
|
|
|
These approaches increases the overall system performance by presenting an efficient trade-off between the redundant bitrate and the error robustness.
|
|
|
|
|
Moreover, redundant pictures might also work as localisation techniques when different reference frames are selected. As the number of redundant pictures increases, the amount of mismatch temporal predictions in case of data loss is reduced, since the redundant picture can replace the erroneous ones.
|
|
|
|
|
This achieves a similar goal as reducing the video stream dependencies, i.e., localisation techniques.
|
|
|
|
|
The main drawback of such approaches is the large amount of redundant data and overheads that are required for their efficient implementation.
|
|
|
|
|
In~\cite{RedundantMVsJune2008} a newer approach was proposed to reduce the amount of information needed to encode redundant pictures, by introducing only redundant \acs{mv}.
|
|
|
|
|
This was also used in~\cite{VPxResilience2016}, where the motion information is used to recover the missing frame.
|
|
|
|
|
These approaches increase the overall system performance by presenting an efficient trade-off between the redundant bitrate and the error robustness.
|
|
|
|
|
|
|
|
|
|
\subsection{Multiple description coding}
|
|
|
|
|
\label{SEC22:MDC}
|
|
|
|
|
|
|
|
|
|
\begin{figure}[tb]
|
|
|
|
|
\centering
|
|
|
|
|
\includegraphics[width=0.7\columnwidth]{\TransmissionResiliencefigs/MDC.png}
|
|
|
|
|
\includegraphics[width=0.7\columnwidth]{./022_ErrorResilience/Figures/MDC.png}
|
|
|
|
|
\caption{Architecture of multiple description system \cite{MDC2005}.}
|
|
|
|
|
\label{fig2:MDCSystem}
|
|
|
|
|
\end{figure}
|
|
|
|
|
|
|
|
|
|
In a \ac{mdc} scheme different bitstreams, named descriptions, are encoded such that a acceptable level of quality is achieve with one of the bitstreams isolated~\cite{MDC2005}.
|
|
|
|
|
Figure \ref{fig2:MDCSystem} illustrates the a basic architecture of encoder and decoder used by the \ac{mdc} technique.
|
|
|
|
|
The encoder-side creates two descriptions, i.e., $R_1$ and $R_2$, which shall be transmitted using two different channels, reducing the probability of both being lost at the same time.
|
|
|
|
|
As shown in the figure three situation may happen at the decoder-side: both descriptions are correctly received and decoder using the central decoder, or either one of the descriptions are not received and then one of the side decoders is used.
|
|
|
|
|
When both bitstreams are received, the central decoder combines the information carried by each description achieving higher quality.
|
|
|
|
|
In the \ac{mdc} encoder the redundancy can be controlled by changing the correlation between descriptions. Higher correlation leads to higher redundancy but also higher quality in the output of the side decoders.
|
|
|
|
|
In contrast with redundant pictures the redundant streams do not carry the same information which means that if both are received the decoded quality is increased.
|
|
|
|
|
In \ac{mdc} schemes two or more bitstreams, named descriptions, are encoded and transmitted through different channels, such that an acceptable level of quality is achieved when only some of them reach the decoder \cite{MDC1993}.
|
|
|
|
|
Figure \ref{fig2:MDCSystem} illustrates a basic architecture of a \ac{mdc} encoder and decoder using two descriptions.
|
|
|
|
|
The encoder-side creates two descriptions, i.e., $R_1$ and $R_2$, which shall be transmitted using two different channels, reducing the probability of the same video data being lost at the same time in both descriptions.
|
|
|
|
|
As shown in the figure three different outputs may be produced at the decoder-side: both descriptions are correctly received and the central decoder is used, or either one of the descriptions is not received and then one of the side decoders is used.
|
|
|
|
|
When both bitstreams are received, the central decoder combines the information carried by each description achieving higher quality than any single side decoder.
|
|
|
|
|
|
|
|
|
|
Different types of multiple description coding \cite{MDCHierarBFeb2012} can also be used to improve the quality under transmission errors in multiple path networks \cite{MDCCorreiaJun2012}. Although these techniques can be very effective in multi-path networks, a significant amount of overhead is required to create the different descriptions.
|
|
|
|
|
In the \ac{mdc} encoder the redundancy among descriptions can be controlled by changing the correlation between them. Higher correlation leads to higher redundancy but also higher quality in the output of the side decoders.
|
|
|
|
|
In contrast with the case of redundant pictures, the redundant \ac{mdc} streams do not carry the same information which means that if both are received, then the decoded quality is increased. However, a combination of \ac{mdc} and redundant pictures can also be used to achieve increase quality gains \cite{MDCRedundantPicsJan2010}.
|
|
|
|
|
Different types of \ac{mdc} approaches \cite{MDC2005,MDCHierarBFeb2012,MDCCorreiaJun2012} can be used to improve the quality under transmission errors, either by taking advantage of hierarchical B-pictures~\cite{MDCHierarBFeb2012} or by introducing extra auxiliary information to limit the error propagation in case of single description decoding~\cite{MDCCorreiaJun2012}.
|
|
|
|
|
Although these techniques can be very effective in multi-path networks, a significant amount of overhead is required to create the different descriptions\cite{MDCCorreiaJun2012}.
|
|
|
|
|
|
|
|
|
|
%\HERE
|
|
|
|
|
|
|
|
|
|
\cite{MDCRedundantPicsJan2010}
|
|
|
|
|
\cite{MDC_TCSVT2011}
|
|
|
|
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
|
%% SECTION %%
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
|
\section{Error concealment by post-processing}
|
|
|
|
|
\label{Section22:Concealment}
|
|
|
|
|
|
|
|
|
|
\glsreset{ec}
|
|
|
|
|
%\glsreset{ec}
|
|
|
|
|
|
|
|
|
|
In general, the \ac{ec} operation is a post processing step in the decoder side and it is the last step to handle errors in the communication chain\cite{ConcealmentReview1998}. Thus, concealment of missing data should take place within the video decoder to minimise perceived effects of lost data. In summary, to enhance the 3D video quality and depth perception, the design of the transmission system must involve the development of adequate \ac{ec} tools. This section reviews several \ac{ec} methods based on spatial and temporal redundancies.
|
|
|
|
|
In general, \ac{ec} functions are implemented as post-processing step in the decoder and thus is the last module in the communication chain capable of handling errors~\cite{ConcealmentReview1998}.
|
|
|
|
|
Thus, reconstruction of missing data should take place within the video decoder to minimise the perceived effects of lost data.
|
|
|
|
|
%Overall, to enhance the video quality, the design of the transmission system must incorporate adequate \ac{ec} tools.
|
|
|
|
|
This section reviews several \ac{ec} methods based on spatial and temporal redundancies, as these play an important role in the overall performance of robust communication systems.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\subsection{Spatial error concealment techniques}
|
|
|
|
@@ -347,7 +457,8 @@ In general, the \ac{ec} operation is a post processing step in the decoder side
|
|
|
|
|
%\subsection{Pixel level concealment}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
A typical approach to conceal missing regions in an image is based on linear interpolation of neighbouring pixels \cite{ErrorConcSpatial1995,ConcTCE2001}. Figure~\ref{FIG3:pixel_interpolation} shows the estimation process of a lost pixel ($lp$), within the lost region ($R_L$), using the four closest pixels ($p_i$) as reference, and the distance between the lost pixel and the reference ones ($d_i$). The pixel value ($lp$) is recovered using a weighted average, as follows:
|
|
|
|
|
A typical approach to recover missing regions in an image is to estimate the lost regions based on linear interpolation of neighbouring pixels \cite{ErrorConcSpatial1995,ConcTCE2001}.
|
|
|
|
|
Figure~\ref{FIG3:pixel_interpolation} shows the estimation process of a lost pixel ($lp$), within the lost region ($R_L$), using the four closest pixels ($p_i$) as reference, and the distance between the lost pixel and the reference ones ($d_i$). The pixel value ($lp$) is recovered using a weighted average, as follows:
|
|
|
|
|
\begin{equation}
|
|
|
|
|
\label{eq:pixel_interpolation}
|
|
|
|
|
lp = \left. \sum_{i}^n{ \frac{p_i}{d_i} } \middle/ \sum_{i}{ \frac{1}{d_i} } \right. .
|
|
|
|
@@ -357,40 +468,40 @@ lp = \left. \sum_{i}^n{ \frac{p_i}{d_i} } \middle/ \sum_{i}{ \frac{1}{d_i} } \ri
|
|
|
|
|
%\normalsize lp = \frac{\sum_i{ \frac{p_i}{d_i} } }{ \sum_{i}{ \frac{1}{d_i} } }.
|
|
|
|
|
%\end{equation}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Liner interpolation is a simple yet effective method to recover lost regions in digital images. However, as each lost pixel is recovered by giving the same importance to each reference, the image discontinuities are not taken into account. Thus, some pixels used as reference may belong to distinct regions of the image, without significant correlation with the lost region itself.
|
|
|
|
|
This leads to a poor estimation of lost pixels, resulting in noticeable artefacts in the reconstructed image. For example, in Figure~\ref{FIG3:pixel_interpolation} the lost pixel ($lp$) and the pixel $p_4$ are separated by an edge. Therefore, $p_4$ may not be suitable to recover the lost pixel value. So, to ensure a correct concealment of the lost region, the edges and image transitions must be considered, by selecting correct pixels to be used in the interpolation function. To overcome this problem, a directional interpolation can be performed, by recovering the lost pixels based on those within the same region, according to the edge directions.
|
|
|
|
|
Liner interpolation is a simple yet effective method to recover lost regions in digital images. However, as each lost pixel is recovered by giving the same importance to each reference direction, the image discontinuities are not taken into account. Thus, some pixels used as reference may belong to distinct regions of the image, without significant correlation with the lost region itself.
|
|
|
|
|
This leads to a poor estimation of lost pixels, resulting in noticeable artefacts in the reconstructed image. For example, in Figure~\ref{FIG3:pixel_interpolation} the lost pixel ($lp$) and the pixel $p_4$ belong to different regions separated by an edge. Therefore, $p_4$ may not be suitable to recover the lost pixel value. So, to
|
|
|
|
|
improve the estimation accuracy of the lost region, the edges and image transitions must be considered, by selecting correct pixels to be used in the interpolation function. To overcome this problem, directional interpolation can be used, by recovering the lost pixels based on those within the same region, according to the edge directions.
|
|
|
|
|
|
|
|
|
|
\begin{figure}[tb]
|
|
|
|
|
\centering
|
|
|
|
|
\input{\ErrorConcealmentReviewFigs/LinearInterpolation}
|
|
|
|
|
\input{./022_ErrorResilience/Figures/LinearInterpolation}
|
|
|
|
|
\caption{Lost pixel interpolation from neighbouring error-free pixels. }
|
|
|
|
|
\label{FIG3:pixel_interpolation}
|
|
|
|
|
\end{figure}
|
|
|
|
|
|
|
|
|
|
The directional interpolation improves the performance of the pixel recovery, by estimating the lost edges using either correctly received or previously recovered pixels. In order to obtain the best direction to interpolate, edges are calculated using the image gradient, for example applying the Sobel operator \cite{SignalImageVideoProcessing2012}. This operator gives the first-order approximation of the gradient of the image. This is widely used in image processing to locate the image edges, and uses an horizontal and vertical mask given by:
|
|
|
|
|
\begin{equation}
|
|
|
|
|
G_x =
|
|
|
|
|
\begin{bmatrix}
|
|
|
|
|
-1 & 0 & 1 \\
|
|
|
|
|
-2 & 0 & 2 \\
|
|
|
|
|
-1 & 0 & 1
|
|
|
|
|
\end{bmatrix},\ \
|
|
|
|
|
G_y =
|
|
|
|
|
\begin{bmatrix}
|
|
|
|
|
-1 & -2 & -1 \\
|
|
|
|
|
0 & 0 & 0 \\
|
|
|
|
|
1 & 2 & 1
|
|
|
|
|
\end{bmatrix}.
|
|
|
|
|
\end{equation}
|
|
|
|
|
The convolution between the image pixels and the Sobel matrixes results in both horizontal ($g_x$) and vertical ($g_y$) gradients, from which the magnitude and the angular direction are computed using,
|
|
|
|
|
\begin{equation}
|
|
|
|
|
\begin{aligned}
|
|
|
|
|
G &= \sqrt{g^2_x + g^2_y}, \\
|
|
|
|
|
\theta &= tan^{-1} \left( \frac{g_y}{g_x} \right).\\
|
|
|
|
|
\end{aligned}
|
|
|
|
|
\end{equation}
|
|
|
|
|
|
|
|
|
|
Directional interpolation improves the performance of the pixel recovery, by firstly estimating the lost edges using either correctly received or previously recovered pixels. In order to obtain the best direction to interpolate, edges are calculated using the image gradient, for example applying the Sobel operator \cite{SignalImageVideoProcessing2012}.
|
|
|
|
|
%This operator gives the first-order approximation of the gradient of the image. This is widely used in image processing to locate the image edges, and uses an horizontal and vertical mask given by:
|
|
|
|
|
%\begin{equation}
|
|
|
|
|
%G_x =
|
|
|
|
|
%\begin{bmatrix}
|
|
|
|
|
%-1 & 0 & 1 \\
|
|
|
|
|
%-2 & 0 & 2 \\
|
|
|
|
|
%-1 & 0 & 1
|
|
|
|
|
%\end{bmatrix},\ \
|
|
|
|
|
%G_y =
|
|
|
|
|
%\begin{bmatrix}
|
|
|
|
|
%-1 & -2 & -1 \\
|
|
|
|
|
% 0 & 0 & 0 \\
|
|
|
|
|
% 1 & 2 & 1
|
|
|
|
|
%\end{bmatrix}.
|
|
|
|
|
%\end{equation}
|
|
|
|
|
%The convolution between the image pixels and the Sobel matrixes results in both horizontal ($g_x$) and vertical ($g_y$) gradients, from which the magnitude and the angular direction are computed using,
|
|
|
|
|
%\begin{equation}
|
|
|
|
|
%\begin{aligned}
|
|
|
|
|
%G &= \sqrt{g^2_x + g^2_y}, \\
|
|
|
|
|
%\theta &= tan^{-1} \left( \frac{g_y}{g_x} \right).\\
|
|
|
|
|
%\end{aligned}
|
|
|
|
|
%\end{equation}
|
|
|
|
|
The Sobel operator is an efficient algorithm to detect edges, as it can reach extreme values at the image edges, so it can be used in pixel interpolation. The usage of this operator for \ac{ec} was evaluated in \cite{IntraConc2010,FineDirectTCE2006,SpatialEdgeConsecutiveBlocks2004,SpatialEdgeModConsecutiveBlocks2005,SpatialInterpolation2008}, and it is characterised as a fast and efficient method due to its small kernel \cite{GradEstimation2009}.
|
|
|
|
|
|
|
|
|
|
% \abbrev{FC}{Frame-Copy}
|
|
|
|
@@ -398,54 +509,47 @@ The Sobel operator is an efficient algorithm to detect edges, as it can reach ex
|
|
|
|
|
|
|
|
|
|
\subsection{Temporal error concealment techniques}
|
|
|
|
|
|
|
|
|
|
\subsubsection{Motion-copy}
|
|
|
|
|
%\subsubsection{Frame and motion copy}
|
|
|
|
|
|
|
|
|
|
The simplest way to recover missing frames in the temporal domain is using the \ac{fc} or the \ac{mc} method. These two basic concealment methods were proposed for the H.264 reference software in \cite{H264RefConc2006}. The \ac{fc} method consists in repeating (copying) the temporally closest frame in order to fill the gap of the missing frame. The \ac{mc} method uses the motion information of the closest frame to recover the missing one, thus the missing pixels are recovered through motion compensation. The experimental results showed that the \mbox{motion-copy} method achieves higher quality in the reconstructed frames in most of the cases. However, the complexity increases with the implementation of the \ac{mc} method, when compared with the \ac{fc} method.
|
|
|
|
|
The conventional way to recover missing frames in the temporal domain are the so-called \ac{fc} and \ac{mc}.
|
|
|
|
|
These \ac{ec} methods were implemented for the \ac{h264} reference software in \cite{H264RefConc2006}.
|
|
|
|
|
The \ac{fc} method consists in repeating (copying) the frame that has the shortest temporal distance in order to to the missing one, into the time instant where the loss occurred.
|
|
|
|
|
The \ac{mc} method uses the motion information of the closest frame to recover the missing one, thus the missing pixels are recovered through motion compensation.
|
|
|
|
|
Previous works and simulation results showed that the \ac{mc} method achieves higher quality in the reconstructed frames in most of the cases~\cite{2DRecursiveRefinement2010}. However, the complexity increases with the implementation of the \ac{mc} method, when compared with \ac{fc}.
|
|
|
|
|
|
|
|
|
|
In \cite{2DRecursiveRefinement2010} a whole frame loss error concealment algorithm was proposed to improve the performance of the \ac{mc} method, since no refinement is performed in the \acs{mv} obtained from the closest decoded frame. Moreover, in this work the \acs{mv} between consecutive frames are assumed to have a strong correlation, so the motion information of several preceding frames is taken into account to estimate the lost one. Then, the motion vector differences between the vectors of successive frame pairs are recursively calculated, as well as the next refinement area is determined, until it becomes unchanged. This is shown to improve the performance of the \ac{mc} algorithm, achieving gains up $0.47\ dB$ (\ac{psnr}) for a \ac{plr} of $20\%$.
|
|
|
|
|
As an example, in \cite{2DRecursiveRefinement2010} a whole frame loss \ac{ec} algorithm was proposed to improve the performance of the \ac{mc} method, by using a refinement algorithm to improve the accuracy of the \acs{mv} obtained from the closest decoded frame.
|
|
|
|
|
Moreover, in this work the \acs{mv} between consecutive frames are assumed to be strongly correlated, so the motion information of several preceding frames is taken into account to estimate the lost one. Then, the \ac{mv} differences between the vectors of successive frame pairs are recursively calculated along with the next refinement, until the changes become negligibly small (e.g., close to zero).
|
|
|
|
|
This is shown to improve the performance of the \ac{mc} method, achieving gains up $0.47\ dB$ (\ac{psnr}) for a \ac{plr} of $20\%$.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\subsubsection{Boundary match algorithms}
|
|
|
|
|
% \abbrev{ICMV}{Interpolated Candidate Motion Vectors}
|
|
|
|
|
|
|
|
|
|
The recovery of missing blocks in video frames may rely on the available neighbouring regions, to find the suitable \ac{mv} to reconstruct the missing pixels.
|
|
|
|
|
A low complexity approach may use the boundary information of surrounding blocks to search for the motion vector based on a spatial smoothness constraint on boundaries of a lost \ac{mb} \cite{BMAAlgorithm2008}. Figure~\ref{FIG3:BMAPixelInformation} represents the boundary information used to estimate the \ac{mv} for the lost block from a set of different \acs{mv} candidates.
|
|
|
|
|
The \ac{bma} uses the available \acs{mv} provided by the top (T), bottom (B), left (L) and right (R) blocks (see Figure~\ref{FIG3:BMAPixelInformation}) to fetch the pixels from the previous decoded frame.
|
|
|
|
|
Then, the optimal \ac{mv} is selected according to the sum of differences, between pixels on the internal boundary in the recovered block and the corresponding ones on the external boundary.
|
|
|
|
|
These pixels are marked in the figure.
|
|
|
|
|
\Gls{bma} are used to recover a missing region based on the available neighbouring regions. By using the missing block neighbourhood, a new \ac{mv} can be estimated for the missing block. Then, recovery based on motion compensation can be achieved.
|
|
|
|
|
A low complexity approach may use the boundary pixels to search for the \ac{mv} based on a spatial smoothness constraint imposed on the boundaries of the lost block \cite{BMAAlgorithm2008}. Figure~\ref{FIG3:BMAPixelInformation} represents the boundary information used to estimate the \ac{mv} for the lost block from a set of different candidates.
|
|
|
|
|
The \ac{bma} uses the available \acs{mv} provided by the top (T), bottom (B), left (L) and right (R) blocks to simulate different reconstructions of the lost block.
|
|
|
|
|
Then, the optimal reconstruction is selected according to the sum of differences, between the pixels on the internal boundary in the recovered block and the corresponding ones on the external boundary (see Figure~\ref{FIG3:BMAPixelInformation}).
|
|
|
|
|
|
|
|
|
|
\begin{figure}[tb]
|
|
|
|
|
\centering
|
|
|
|
|
%\includegraphics[width=0.45\columnwidth]{\ErrorConcealmentReviewFigs/BMAAlgorithm}
|
|
|
|
|
\input{\ErrorConcealmentReviewFigs/BMAAlgorithm}
|
|
|
|
|
%\includegraphics[width=0.45\columnwidth]{./022_ErrorResilience/Figures/BMAAlgorithm}
|
|
|
|
|
\input{./022_ErrorResilience/Figures/BMAAlgorithm}
|
|
|
|
|
\caption{Illustration of the boundary match algorithm.}
|
|
|
|
|
\label{FIG3:BMAPixelInformation}
|
|
|
|
|
\end{figure}
|
|
|
|
|
|
|
|
|
|
In \cite{BMAAlgorithm2006}, error concealment based on the boundary matching algorithm is improved, by introducing the concept of the interpolated candidate motion vectors.
|
|
|
|
|
The candidate \acs{mv} are calculated with an interpolation of the motion vectors of the neighbouring block in the missing frame and the region around the corresponding block in the previous frame.
|
|
|
|
|
The interpolated candidate \acs{mv} are created based on the change of magnitude and angle of the top, bottom, left and right neighbouring \acs{mv} of the lost block from the previous frame to the current one.
|
|
|
|
|
Then, these modifications are applied to the \ac{mv} of the co-located block with the lost one in the previous frame.
|
|
|
|
|
A more generalised proposal is to apply such transformation not only to the closest neighbours but also to a wider area of blocks.
|
|
|
|
|
The concept of interpolated candidate \acs{mv} can be applied to any method based on \ac{bma}, improving its performance, by introducing more accurate candidates.
|
|
|
|
|
|
|
|
|
|
Another improvement to \ac{bma} was obtained with the outer boundary matching algorithm\cite{OBMAMethod2008}, also known as decoder \ac{mv} estimation algorithm in \cite{DMVE2000}.
|
|
|
|
|
An implementation of the \ac{bma} was proposed in \cite{BMAAlgorithm2006}, by analysing the change of magnitude and angle between the \acs{mv} of the neighbouring blocks and the co-located ones in the previous frames. The average change is applied to the \ac{mv} of block co-located lost one to compute the \ac{mv} candidate for recovery.
|
|
|
|
|
An improvement to \ac{bma} was obtained with the outer boundary matching algorithm\cite{OBMAMethod2008}, also known as decoder \ac{mv} estimation algorithm in \cite{DMVE2000}.
|
|
|
|
|
In this method the distortion is calculated by the difference of two outer boundaries of reconstructed block, instead of internal and external boundary in \ac{bma}.
|
|
|
|
|
Although \ac{bma} and outer \ac{bma} have the same design principle of minimising the mismatch distortion, in later the distortion function is measured in the outer boundaries, \emph{i.e.}, external boundaries of the reconstructed block. It was observed in the experimental results that outer \ac{bma} has better performance in video recovery \cite{OBMAMethod2008} than conventional \ac{bma}. The performance improvements are explained by the usage of the outer boundary in the distortion functions, which incorporates the edge information within the neighbourhood of the lost block in the decision of the most accurate \ac{mv} to recover the missing pixels \cite{OBMAMethod2008}. Moreover, in the method presented in \cite{OBMAMethod2008} further \ac{mv} candidates were introduced, a refined local search is performed and multiple boundary layers, \emph{i.e.}, increase the overlapped region to measure the distortion and to select the final MV, were tested. The experimental results revealed that the usage of a single layer match gives better performance than multiple layers, while selective search gives excellent trade-off between the image recovery and computation complexity
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
% \abbrev{OBMA}{Outer Boundary Matching Algorithm}
|
|
|
|
|
% \abbrev{DMVE}{Decoder Motion Vector Estimation}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Although \ac{bma} and outer \ac{bma} have the same design principle of minimising the mismatch distortion, in latter the distortion is measured in the outer boundaries, \emph{i.e.}, external boundaries of the reconstructed block. It was observed in the experimental results \cite{OBMAMethod2008} that outer \ac{bma} has better performance in video recovery than conventional \ac{bma}. The performance improvements are explained by the distortion minimisation of the outer boundary, which incorporates the edge information located in the neighbourhood of the lost block in the decision process to select the most accurate \ac{mv}.
|
|
|
|
|
%Moreover, in the method presented in \cite{OBMAMethod2008} further \ac{mv} candidates were introduced, a refined local search is performed and multiple boundary layers, \emph{i.e.}, increase the overlapped region to measure the distortion and to select the final MV, were tested. The experimental results revealed that the usage of a single layer match gives better performance than multiple layers, while selective search gives excellent trade-off between the image recovery and computation complexity
|
|
|
|
|
% In \cite{DynamicBMA2010} a novel approach to measure the spatial distortion on boundaries of the reconstructed block is presented. Instead of using only horizontal and vertical directions, diagonal and anti-diagonal are also tested, and the best direction is chosen from the reference frame. Since block boundaries may change abruptly, instead of minimising the distortion function the proposed method aims to approximate this distortion to an optimal difference. This optimal difference are determined in the neighbourhood of the co-located block in the reference frame.
|
|
|
|
|
% In \cite{BMAAlgorithm2008} a novel distortion function was proposed, to choose among different candidate MVs. The distortion function exploits both spatial and temporal smooth constraints. The temporal function is based on the outer boundary distortion, as in \cite{OBMAMethod2008}.
|
|
|
|
|
% The spatial distortion aims to reduce the gradient variances in boundaries of the reconstructed block, so the image edges may be guaranteed. Several experimental results were obtained for different loss events, covering from isolated block lost to the loss of a complete row of blocks. The proposed method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method. Moreover, subjective analysis reveal that this method recover the edges in the lost image. However, the method uses complex mathematical operations, which may not be feasible in real time applications.
|
|
|
|
|
In \cite{BMAAlgorithm2008} a novel distortion function was proposed, to choose among different candidate MVs. The distortion function exploits both spatial and temporal smooth constraints. The temporal function is based on the outer boundary distortion, as in \cite{OBMAMethod2008}.
|
|
|
|
|
The spatial distortion aims to reduce the gradient variances in boundaries of the reconstructed block, so the image edges may be guaranteed. Several experimental results were obtained for different loss events, covering from isolated block lost to the loss of a complete row of blocks. The proposed method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method. Moreover, subjective analysis reveal that this method recover the edges in the lost image. However, the method uses complex mathematical operations, which may not be feasible in real time applications.
|
|
|
|
|
|
|
|
|
|
\subsubsection{Motion vector extrapolation}
|
|
|
|
|
|
|
|
|
|
As mentioned before, the \ac{mc} method is a simple way of concealment a missing frame using motion information from previously decoded frames. Although this method can provide an acceptable performance comparing with the \ac{fc}, as shown in \cite{H264RefConc2006}, more complex algorithms that use the assumption of motion continuity across consecutive frames are proposed to conceal, not only missing blocks, but also whole frames in compressed video.
|
|
|
|
|
As mentioned before, the \ac{mc} method is a simple way to reconstruct an estimation of a missing frame using motion information from previously decoded frames. Although this method can provide an acceptable performance comparing with the \ac{fc}, as shown in \cite{H264RefConc2006}, more complex algorithms that use the assumption of motion continuity across consecutive frames are proposed to recover, not only missing blocks, but also whole frames in compressed video.
|
|
|
|
|
|
|
|
|
|
% \abbrev{MMA}{Multi-frame Motion vector Averaging}
|
|
|
|
|
|
|
|
|
@@ -459,57 +563,68 @@ As mentioned before, the \ac{mc} method is a simple way of concealment a missing
|
|
|
|
|
% \item Finally, the fully reconstructed frame at a half-pixel level is filtered using a mean filter with a $2\times2$ kernel, and down sampled.
|
|
|
|
|
% \end{enumerate}
|
|
|
|
|
|
|
|
|
|
% This method usually produces relatively high quality in the reconstructed frame, but in low motion cases it behaves worse than the \ac{fc} method under a few conditions, as indicated by the experimental results in \cite{MMATempConc2003}. This phenomenon happens because the oldest reference frames usually do not have \ac{mv} similar to the missing one. Moreover, if $L$ increases, the complexity of the proposed algorithm may not be useful to be used in error concealment, due to complexity and memory requirements to determine the \ac{mv} history.
|
|
|
|
|
% This method usually produces relatively high quality in the reconstructed frame, but in low motion cases it behaves worse than the \ac{fc} method under a few conditions, as indicated by the experimental results in \cite{MMATempConc2003}. This phenomenon happens because the oldest reference frames usually do not have \ac{mv} similar to the missing one. Moreover, if $L$ increases, the complexity of the proposed algorithm may not be useful to be used in \ac{ec}, due to complexity and memory requirements to determine the \ac{mv} history.
|
|
|
|
|
|
|
|
|
|
\begin{figure}[t]
|
|
|
|
|
\centering
|
|
|
|
|
\includegraphics[width=0.9\columnwidth]{\ErrorConcealmentReviewFigs/MVE}
|
|
|
|
|
\caption{Motion vector extrapolation with the corresponding extrapolated macroblock, based on the method presented in \cite{BlockBasedMVE2002}.}
|
|
|
|
|
\label{FIG3:MVEExtrapolation}
|
|
|
|
|
\end{figure}
|
|
|
|
|
\glsreset{mve}
|
|
|
|
|
|
|
|
|
|
\ac{mve} \cite{BlockBasedMVE2002,BlockBasedMVEOverllaped2011} methods are a simple but efficient way to reconstruct the \acs{mv} of a lost frame, and they are able to achieve good performance. In the method presented in \cite{BlockBasedMVE2002}, \acs{mv} are extrapolated from the last decoded frame \mbox{(reference frame)} onto the missing one. Firstly, each \ac{mb} in the reference frame is projected to the missing one with its own direction given by the corresponding extrapolated motion vector ($mv'$). An example of motion vector extrapolation is shown in Figure~\ref{FIG3:MVEExtrapolation}, where the vectors of the frame $t-1$ are projected onto the missing frame at instant $t$. Note that, in the reference frame ($f_{t-1}$) the motion vectors may point to different frames ($f_{t-2}$~and~$f_{t-3}$), previously decoded, so \ac{mv} extrapolation must take into account the difference between the original temporal distance and the projected one.
|
|
|
|
|
Secondly, to improve the accuracy of \ac{mve}, the damaged region is divided into $4$ blocks ($8\times8$ pixels). Then, according to the matching area between the extrapolated block and the damaged $8\times8$ region, the best \ac{mv} is chosen.
|
|
|
|
|
In order words, the \ac{mv} for the each block is obtained from the extrapolated block that achieves a higher number of overlapped pixels with the missing $8\times8$ region. Finally, the reconstructed \acs{mv} are used to fetch the missing pixels from the previously decoded frame ($f_{t-1}$). Although this method is able to overcome the disadvantage of incorrect \ac{mb} displacement, the choice of the best \ac{mv} is not very effective \cite{PixelBasedMVE2004}. In \cite{BlockBasedMVE2002} the \ac{mve} is used to extrapolate the \ac{mb} of the reference frame.
|
|
|
|
|
This results in overlapped regions, which are recovered using a weighted average of the multiple pixels candidates.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\ac{mve} \cite{BlockBasedMVE2002,BlockBasedMVEOverllaped2011} methods are implemented quiet simple yet efficient techniques that reconstruct the \acs{mv} of a lost frame which in turn are used for motion compensation.
|
|
|
|
|
These \acs{mv} are extrapolated from the last decoded frame \mbox{(reference frame)} onto the missing one~\cite{BlockBasedMVE2002}.
|
|
|
|
|
Each block in the reference frame is projected to the missing one with its own direction given by the corresponding extrapolated \ac{mv} ($mv'$).
|
|
|
|
|
An example of \ac{mve} is shown in Figure~\ref{FIG3:MVEExtrapolation}, where the vectors of frame $t-1$ are projected onto the missing frame at instant $t$. Note that, in the reference frame ($f_{t-1}$) the \acs{mv} may point to different frames ($f_{t-2}$~and~$f_{t-3}$), previously decoded, so \ac{mve} must take into account the temporal distance between different frames.
|
|
|
|
|
To improve the accuracy of \ac{mve}, the damaged region is divided smaller blocks, and according to the matching area between the extrapolated block and the damaged one, the best \ac{mv} is chosen.
|
|
|
|
|
In order words, the \ac{mv} for the each block is obtained from the extrapolated block with the highest number of pixels matching the missing locations.
|
|
|
|
|
Note that the final motion information need to be aligned to a fixed size grid, e.g., $4\times4$ grid, which leads to a misalignment of the extrapolated \acs{mv}.
|
|
|
|
|
Alternatively, in \cite{PixelBasedMVE2004} several blocks are extrapolated to the missing frame, resulting in overlapped regions, which are recovered using a weighted average of the multiple pixels candidates.
|
|
|
|
|
The weight coefficients assume a maximum value in the center of the extrapolated region, and a minimum value at the boundaries.
|
|
|
|
|
|
|
|
|
|
Since the choice of the best candidate \ac{mv} relies on the overlapped area of the extrapolated block with the lost region, the method may give erroneously estimated \ac{mv} in large motion scenes. The example of Figure~\ref{FIG3:MVEPixelBased} shows a lost $8\times8$ block (in the centre of the figure) and four possible extrapolated \acs{pu} that can be used to estimate lost the motion information. As illustrated in the figure, although the largest overlapped area belongs to the PU$_2$, the overlapped pixels correspond to a small portion of the whole missing block. Moreover, the majority of the lost block's area was not covered by any extrapolated \ac{pu}. Thus, motion recovery using only the information available in PU$_2$ will not properly characterise the motion information for all lost pixels.
|
|
|
|
|
|
|
|
|
|
To overcome the previous limitation and to achieve smoother and finer results, a \ac{pmve} algorithm was proposed in \cite{PixelBasedMVE2004}. This motion vector extrapolation method is divided into two cases:
|
|
|
|
|
\begin{enumerate}
|
|
|
|
|
\item The pixels in the lost region that are covered by at least one extrapolated block, such as circle points in Figure~\ref{FIG3:MVEPixelBased}, are recovered using the average of the \acs{mv} of the overlapped block.
|
|
|
|
|
\item Those pixels which are not covered by any extrapolation, such as the triangle point in Figure~\ref{FIG3:MVEPixelBased}, the \ac{mv} of the co-located pixel in the previous frame is used.
|
|
|
|
|
\end{enumerate}
|
|
|
|
|
%
|
|
|
|
|
%Since the choice of the best candidate \ac{mv} relies on the overlapped area of the extrapolated block with the lost region, the method may give erroneously estimated \ac{mv} in large motion scenes. The example of Figure~\ref{FIG3:MVEPixelBased} shows a lost $8\times8$ block (in the centre of the figure) and four possible extrapolated \acs{pu} that can be used to estimate lost the motion information. As illustrated in the figure, although the largest overlapped area belongs to the PU$_2$, the overlapped pixels correspond to a small portion of the whole missing block. Moreover, the majority of the lost block's area was not covered by any extrapolated \ac{pu}. Thus, motion recovery using only the information available in PU$_2$ will not properly characterise the motion information for all lost pixels.
|
|
|
|
|
%
|
|
|
|
|
%\begin{figure}[t]
|
|
|
|
|
%\centering
|
|
|
|
|
%\input{./022_ErrorResilience/Figures/PixelBasedMVE}
|
|
|
|
|
%\caption{Pixel based motion vector extrapolation \cite{PixelBasedMVE2004}.}
|
|
|
|
|
%\label{FIG3:MVEPixelBased}
|
|
|
|
|
%\vspace{3mm}
|
|
|
|
|
%\end{figure}
|
|
|
|
|
%
|
|
|
|
|
%To overcome the previous limitation and to achieve smoother and finer results, a \ac{pmve} algorithm was proposed in \cite{PixelBasedMVE2004}. This motion vector extrapolation method is divided into two cases:
|
|
|
|
|
%\begin{enumerate}
|
|
|
|
|
% \item The pixels in the lost region that are covered by at least one extrapolated block, such as circle points in Figure~\ref{FIG3:MVEPixelBased}, are recovered using the average of the \acs{mv} of the overlapped block.
|
|
|
|
|
% \item Those pixels which are not covered by any extrapolation, such as the triangle point in Figure~\ref{FIG3:MVEPixelBased}, the \ac{mv} of the co-located pixel in the previous frame is used.
|
|
|
|
|
%\end{enumerate}
|
|
|
|
|
%
|
|
|
|
|
%Then, if the estimated \ac{mv} is $(MV_x,MV_y)$, then the estimation of the missing pixel, $p_m(x,y)$, is obtained by:
|
|
|
|
|
%\begin{equation}\label{EQ:MotionComp}
|
|
|
|
|
% p_m(x,y) = p_r(x+MV_x,y+MV_y),
|
|
|
|
|
%\end{equation}
|
|
|
|
|
%where $p_r$ refers to the pixels in the previous frame. This pixel-based \ac{mve} provides similar results than the block-based \ac{mve} algorithm in low motion scenarios. Nevertheless, it is able to overcome the previous method in large motion scenes.
|
|
|
|
|
An extension to the previous approach was considered in \HIGHLIGHT{\cite{PelBasedMVE_TCE2007}}, by using the \acs{mv} of subsequent available frames. Thus, to estimate the lost motion information, forward and backward extrapolations are performed, in order to obtain two candidates to each lost pixel. Then, the lost pixels are obtained through motion compensation using the \acs{mv} determined by the average of two \ac{mv} candidates. The experimental results revealed that the bi-directional \ac{pmve} outperforms the isolated use of forward or backward \ac{pmve}. Moreover, the bi-directional \ac{pmve} achieves gains up to $1.85$ dB and $1.32$ dB (\ac{psnr}) comparing with the work in \cite{MMATempConc2003} and \ac{mve} \cite{BlockBasedMVE2002}.
|
|
|
|
|
|
|
|
|
|
\begin{figure}[t]
|
|
|
|
|
\centering
|
|
|
|
|
\input{\ErrorConcealmentReviewFigs/PixelBasedMVE}
|
|
|
|
|
\caption{Pixel based motion vector extrapolation \cite{PixelBasedMVE2004}.}
|
|
|
|
|
\label{FIG3:MVEPixelBased}
|
|
|
|
|
\vspace{3mm}
|
|
|
|
|
\centering
|
|
|
|
|
\includegraphics[width=0.9\columnwidth]{./022_ErrorResilience/Figures/MVE}
|
|
|
|
|
\caption{Motion vector extrapolation with the corresponding extrapolated macroblock, based on the method presented in \cite{BlockBasedMVE2002}.}
|
|
|
|
|
\label{FIG3:MVEExtrapolation}
|
|
|
|
|
\end{figure}
|
|
|
|
|
|
|
|
|
|
Then, if the estimated \ac{mv} is $(MV_x,MV_y)$, then the estimation of the missing pixel, $p_m(x,y)$, is obtained by:
|
|
|
|
|
\begin{equation}\label{EQ:MotionComp}
|
|
|
|
|
p_m(x,y) = p_r(x+MV_x,y+MV_y),
|
|
|
|
|
\end{equation}
|
|
|
|
|
where $p_r$ refers to the pixels in the previous frame. This pixel-based \ac{mve} provides similar results than the block-based \ac{mve} algorithm in low motion scenarios. Nevertheless, it is able to overcome the previous method in large motion scenes.
|
|
|
|
|
An extension to the previous approach was considered in \cite{PixelBasedMVE2004}, by using the \acs{mv} of subsequent available frames. Thus, to estimate the lost motion information, forward and backward extrapolations are performed, in order to obtain two candidates to each lost pixel. Then, the lost pixel is estimated using the Equation~\ref{EQ:MotionComp}, using the \ac{mv} determined by the average of two \ac{mv} candidates. The experimental results revealed that the bi-directional \ac{pmve} outperforms the isolated use of forward or backward \ac{pmve}. Moreover, the bi-directional \ac{pmve} achieves gains up to $1.85$ dB and $1.32$ dB (\ac{psnr}) comparing with the work in \cite{MMATempConc2003} and \ac{mve} \cite{BlockBasedMVE2002}.
|
|
|
|
|
In \cite{2DConcHybridMotion2010} a \ac{hmve} algorithm was proposed, to improve the accuracy of the \acs{mv} of the \ac{pmve} method, by using the extrapolated vectors at the pixel and block level. Moreover, the \acs{mv} that were wrongly extrapolated are discarded in order to obtain a more accurate MV.
|
|
|
|
|
In this \ac{ec} method, firstly a pixel based motion extrapolation is performed and the pixels are organised into three groups.
|
|
|
|
|
%Group A contains the pixels that are overlapped by at least one extrapolated block; group B includes pixels that are not covered by any extrapolated block, but belong to a block that is covered, \emph{e.g.}, in Figure~\ref{FIG3:MVEPixelBased} the triangle point was not covered by any extrapolated \ac{mb} but it belongs to $8\times8$ block that is covered by 4 blocks.
|
|
|
|
|
%The remaining pixels are included in group C (star pixels in the figure).
|
|
|
|
|
Then, through block-based \ac{mve} two new \acs{mv} are determined: the first one corresponds to the \ac{mv} of the most overlapped block with the current one, and second one is obtained by the weighted average value of the \acs{mv} corresponding to different overlapped blocks.
|
|
|
|
|
To find the more accurate \ac{mv} from the \ac{mv} set composed described above. To achieve this, the \ac{mv} set is further refined using the distance between each pair of \acs{mv}, in order to discard the wrongly extrapolated \acs{mv}.
|
|
|
|
|
% Finally, the pixels of the lost regions (belonging to the groups A and B) are obtained through motion compensation, using the average of the remaining vectors in the \ac{mv} set. The pixels of the group C, although the \ac{mv} set is empty, the same process is applied using the \ac{mv} of the co-located pixel in the previous frame.
|
|
|
|
|
This algorithm is able to outperform the \ac{pmve} for different packet loss probabilities, as shown by the results in \cite{2DConcHybridMotion2010}.
|
|
|
|
|
The method was used to recover both intra and inter frames in video sequences compressed with the \ac{h264}, achieving \ac{psnr} gains above $1.09$ dB in the erroneous frames.
|
|
|
|
|
|
|
|
|
|
In \cite{2DConcHybridMotion2010} a \ac{hmve} algorithm was proposed, to provide more accuracy to the estimated motion information. The method aims to improve the accuracy of the \acs{mv} of the \ac{pmve} method, by using the extrapolated vectors at the pixel and block level. Moreover, the proposed method discards the \ac{mv} that were wrongly extrapolated in order to obtain an accurate MV. In this concealment method, firstly a pixel based motion extrapolation is performed and the pixels is organised into three groups. Group A contains the pixels that are overlapped by at least one extrapolated block (both circle and diamond points in Figure~\ref{FIG3:MVEPixelBased}).
|
|
|
|
|
Group B includes the pixels that are not covered by any extrapolated block, but belong to a block that is covered, \emph{e.g.}, in Figure~\ref{FIG3:MVEPixelBased} the triangle point was not covered by any extrapolated \ac{mb} but it belongs to $8\times8$ block that is covered by 4 blocks.
|
|
|
|
|
The remaining pixels are included in group C (star pixels in the figure). Then, through the block-based \ac{mve} method described in \cite{BlockBasedMVE2002} two new \acs{mv} are determined: the first one corresponds to the \ac{mv} of the most overlapped block with the current one, and second one is obtained by the weighted average value of the \acs{mv} corresponding to different overlapped blocks.
|
|
|
|
|
Previous works presented for the \ac{hevc} standard are mainly based on temporal \ac{ec} approaches.
|
|
|
|
|
For instance, in~\cite{HEVCMCBlockAug2013} the block partitions of the neighbouring frames were used to improve the performance of \ac{mve}.
|
|
|
|
|
The reconstructed frame keeps smooth boundaries, leading to higher objective and visual quality.
|
|
|
|
|
Moreover, in order to select only the reliable motion information, the residue information can also be used to decide the optimal \ac{mv}, as shown in~\cite{HEVCMCResidueDec2013}.
|
|
|
|
|
|
|
|
|
|
The \ac{hmve} method aims to find the more accurate \ac{mv} from the \ac{mv} set composed described above. To achieve this, the \ac{mv} set is further refined using the distance between each pair of \acs{mv}, in order to discard the wrongly extrapolated \acs{mv}. Finally, the pixels of the lost regions (belonging to the groups A and B) are obtained through motion compensation, using the average of the remaining vectors in the \ac{mv} set. The pixels of the group C, although the \ac{mv} set is empty, the same process is applied using the \ac{mv} of the co-located pixel in the previous frame.
|
|
|
|
|
This algorithm is able to outperform the \ac{pmve} for different packet loss probabilities, as shown by the results in \cite{2DConcHybridMotion2010}. The method is used to conceal both I (INTRA) and P frames in video sequence compressed with the \ac{h264}, achieving more $1.09$ dB (\ac{psnr}) in the erroneous frames.
|
|
|
|
|
|
|
|
|
|
%\HIGHLIGHT{HEVC}
|
|
|
|
|
In the case of \ac{hevc}, there are also a few works worth to be mentioned. The works presented for the \ac{hevc} standard are mainly based on temporal \ac{ec} approaches.
|
|
|
|
|
In~\cite{HEVCMCBlockAug2013} the block partitions of the neighbouring frames were used to improve the performance motion vector extrapolation. Using this the reconstructed frame keeps the boundaries smooth, leading to higher objective and visual quality.
|
|
|
|
|
Moreover, in order to select only the reliable motion information, the residue information can be exploited, as shown in~\cite{HEVCMCResidueDec2013}.
|
|
|
|
|
|
|
|
|
|
Overall, the aforementioned methods rely on post-processing without any assistance from the encoder-side, which may lead to inaccurate recovery of the lost information. Thus, in order to achieve higher reconstruction quality further information need to be shared with the decoder to minimise the residual distortion of \ac{ec}.
|
|
|
|
|
Overall, the aforementioned methods rely on post-processing without any assistance from the encoder-side, which usually leads to inaccurate recovery of the lost information. Thus, in order to achieve higher reconstruction quality, further information need to be shared with the decoder to minimise the residual distortion of \ac{ec}.
|
|
|
|
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
|
%% SECTION %%
|
|
|
|
@@ -517,22 +632,22 @@ Overall, the aforementioned methods rely on post-processing without any assistan
|
|
|
|
|
\section{Concealment-aware error resilience}
|
|
|
|
|
\label{Section22:ConcealmentAware}
|
|
|
|
|
|
|
|
|
|
The last category of techniques for robust video transmission address those algorithms that help the \ac{ec} process at the decoder-side. Although such techniques might overlap with concepts described above, such as, data partition or redundant information, they have the main goal of improving the \ac{ec} performance. This group of algorithms combine the error resilience capabilities with \ac{ec} methods to achieve an improved overall performance.
|
|
|
|
|
The last category of techniques for robust video transmission address those algorithms that help the \ac{ec} process at the decoder-side. Although such techniques might overlap with some of the concepts described above, such as, data partition or redundant information, they have the main goal of improving the \ac{ec} performance. This class of algorithms combine the error resilience capabilities with \ac{ec} methods to achieve an improved overall performance.
|
|
|
|
|
|
|
|
|
|
One of the approaches that can be used to improve the \ac{ec} performance is to estimate, at the encoding time, the \ac{ec} distortion introduced by the decoder in case of data loss.
|
|
|
|
|
In \cite{ConcealmentDrivenSliceOrdering2008} the \ac{ec} distortion estimation is used for slice classification and \ac{uep}. Such distortion is also used to optimise motion estimation~\cite{ConcealmentAwareME2006}, by including the distortion of the subsequent frame (assuming it is lost and recovered) in the \ac{rd} cost optimisation of the current frame. This was also used in the context of the \ac{hevc} to improve its error robustness as shown in \cite{ErrorResilienceModeHEVC2016}. This kinda of approaches take advantage of location techniques and data partitioning to improve the reconstruction quality achieved by the \ac{ec} algorithms. Normally, they use a single \ac{ec} method that is implemented both at the encoder and decoder, based on temporal algorithms, e.g., \ac{mc} and \ac{mve}.
|
|
|
|
|
In \cite{ConcealmentDrivenSliceOrdering2008} the \ac{ec} distortion estimation is used for slice classification and \ac{uep}. Such distortion is also used to optimise motion estimation~\cite{ConcealmentAwareME2006}, by including the distortion of the subsequent frame (assuming it is lost and recovered) in the \ac{rd} cost optimisation of the current frame. This was also used in the context of \ac{hevc} to improve its error robustness, as shown in \cite{ErrorResilienceModeHEVC2016}. This kind of approaches take advantage of location techniques and data partitioning to improve the reconstruction quality achieved by \ac{ec} algorithms. Normally, they use a single \ac{ec} method that is implemented both at the encoder and decoder, based on temporal algorithms, e.g., \ac{mc} and \ac{mve}.
|
|
|
|
|
|
|
|
|
|
Another approach is the use \ac{ec} motion vectors which are \acs{mv} that might be carried by intra-coded units for the purpose of concealing errors. When an error occurs in a given unit, the concealment motion vector from the neighbouring units can be used to form a prediction from previously received frames. This concept was exploited in \cite{ConcealmentIntraDataHidng2009} by using data-hiding to carry additional motion information in the \ac{dct} coefficients. This technique not only improves the reconstruction quality but also shift the algorithm complexity from the decoder to the encoder-side.
|
|
|
|
|
Another approach is the use \ac{ec} \ac{mv} which are vectors that might be carried by intra-coded units for the purpose of error recovery. When an error occurs in a given coding unit, the \ac{ec} \ac{mv} from the neighbouring units can be used to form a prediction from previously received frames. This concept was exploited in \cite{ConcealmentIntraDataHidng2009} by using data-hiding to carry additional motion information in the \ac{dct} coefficients. This improves the reconstruction quality and also adds complexity from the decoder to the encoder.
|
|
|
|
|
|
|
|
|
|
A different approach to improve the \ac{ec} performance was presented in \cite{PixelInterlacingIntraConcealment2011}. Pixel interlacing is used to divide the frame into different packets in order to enhance the performance of intra-frame \ac{ec} using linear interpolation. This technique reduces the probabilities of burst losses, therefore, the neighbouring pixels surrounding the lost regions are available to be used as reference for the concealment operation.
|
|
|
|
|
The above methods facilitate the \ac{ec} performance, either through slice reordering or based on \acs{mv}, they always rely on the assumption that a fixed \ac{ec} method is used at the decoder.
|
|
|
|
|
In \cite{ResilienceRECAP_ICASSP2009} a low-resolution video is transmitted alongside the main bitstream in order to be used as reference for concealment. The redundant stream is used by the decoder to improve the estimation accuracy of motion information. This method was extended in \cite{SaliencyErrorConcealment_TM2013} by using asymmetric coding of the redundant information based on Itti's attention model~\cite{SaliencyItti1998}.
|
|
|
|
|
A different approach to improve the \ac{ec} performance was presented in \cite{PixelInterlacingIntraConcealment2011}. Pixel interlacing is used to divide the frame into different packets in order to enhance the performance of intra-frame \ac{ec} by using linear interpolation. This technique reduces the probability of burst losses, therefore, the neighbouring pixels surrounding the lost regions are available to be used as reference for the concealment operation.
|
|
|
|
|
The above methods facilitate the \ac{ec} performance, either through slice reordering or based on \acs{mv}, but they always rely on the assumption that a fixed \ac{ec} method is used at the decoder.
|
|
|
|
|
In \cite{ResilienceRECAP_ICASSP2009} a low-resolution video is transmitted alongside the main bitstream in order to be used as reference for \ac{ec}. Such redundant stream is used by the decoder to improve the estimation accuracy of motion information. This method was extended in \cite{SaliencyErrorConcealment_TM2013} by using asymmetric coding of the redundant information based on Itti's attention model~\cite{SaliencyItti1998}.
|
|
|
|
|
|
|
|
|
|
The work presented in \cite{ConcealmentSignalling2008,BurstConcealmentSignalling2008} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder, using \ac{roi} to identify the most sensitive image area.
|
|
|
|
|
Although such an approach achieves reasonably good performance, it relies on the spatial neighbouring information, which may not be available in case of full frame loss. Moreover, it requires transmission of the \ac{roi} map to the decoder, which increases the overall bitrate.
|
|
|
|
|
More recently a EC signalling methods was proposed for scalable video \cite{SignallingRobustScalable2015}, in order to select the best frame to be used to replace the missing one.
|
|
|
|
|
More recently a \ac{ec} signalling methods was proposed for scalable video \cite{SignallingRobustScalable2015}, in order to select the best frame to replace the missing one.
|
|
|
|
|
|
|
|
|
|
Summarising, \ac{ec}-aware error techniques allow the combination of efficient algorithms, e.g., redundant pictures or error concealment-aware motion estimation or embedded information, with algorithms at the decoder which take advantages of such extra information. Such techniques are able to achieve quality improvements in comparison with the application of error recovery approaches a the decoder on its own. Although effective techniques have been proposed in the past, the exploration of dynamic techniques which explore the potential of different error concealment techniques with adaptable decoder have not been fully exploited.
|
|
|
|
|
Summarising, \ac{ec}-aware error techniques allow the combination of efficient algorithms, e.g., redundant pictures or \ac{ec}-aware motion estimation or embedded information, with algorithms at the decoder which take advantages of such extra information. Such techniques are able to achieve quality improvements in comparison with error recovery approaches at the decoder on its own. Although effective techniques have been proposed in the past, the exploration of dynamic techniques which explore the potential of different \ac{ec} techniques with adaptable decoder have not been fully exploited.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -543,12 +658,8 @@ Summarising, \ac{ec}-aware error techniques allow the combination of efficient a
|
|
|
|
|
\section{Summary}
|
|
|
|
|
\label{Section22:Summary}
|
|
|
|
|
|
|
|
|
|
This chapter presented an overview of the different error resilience techniques proposed over the last years, based on they key features. Different approaches were described covering techniques for robust video transmission from dependency reduction at the encoder-side to error concealment by post-processing at the decoder.
|
|
|
|
|
|
|
|
|
|
The techniques described in this chapter provide relevant insights for the methods proposed in this work. In order to achieve increase the error robustness of the \ac{hevc} bitstrems and improved reconstruction quality in case of errors the different algorithms presented before are considered and combined. In Chapter~\ref{Chapter:HEVCErrorResilience} localisation techniques are used to decrease motion vector dependencies in \ac{hevc} standard achieving higher quality at cost of small coding efficiency loss. Chapter~\ref{Chapter:ErrorResilienceTechniques} takes advantage of reference picture selection scheme in combination with motion vector redundancies to improve \ac{hevc} streaming.
|
|
|
|
|
Chapter~\ref{Chapter:ConcealmentAwareTechniques} proposes an error-concealment aware for increase reconstruction quality.
|
|
|
|
|
Finally, Chapter~\ref{Chapter:EndToEndRobustSystem} proposes an end-to-end robust video transmission scheme by combining the error resilience and error concealment based on content, network conditions and application.
|
|
|
|
|
%
|
|
|
|
|
This chapter presented an overview of the different error resilience techniques proposed over the last years, based on their main characteristics. Different approaches were described covering techniques for robust video transmission, like dependency reduction at the encoder-side and \ac{ec} by post-processing at the decoder. The techniques described in this chapter provide relevant insights for the methods investigated in this work.
|
|
|
|
|
Different algorithms were considered and combined to increase the error robustness of highly efficient coded video and improve the reconstruction quality in case of errors.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
