jfmcarreira/doc_phd_thesis
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Commit before SF changes

Browse Source
This commit is contained in:
jfmcarreira
2018-09-03 02:03:59 +01:00
parent 594357af56
commit 8fb90680d2
10 changed files with 270 additions and 280 deletions
Download Patch File Download Diff File Expand all files Collapse all files
010_Introduction/Introduction.tex
+24 -22
View File
@@ -1,36 +1,36 @@
\graphicspath{{010_Introduction}}
\glsreset{hevc}
\chapter{Introduction \VERSION{1}}
\chapter{Introduction \VERSION{2}}
\label{Chapter:Intro}
\section{Context and motivation}
\glsreset{h264}
Over the recent years new advances in video compression and networking technologies have been enabling the development of many multimedia applications over the internet, such as video conferencing, video streaming, video publishing, among others.
The increasing diversity of services, the demands for high quality multimedia and the growing popularity of \ac{uhd} formats (\emph{e.g.}, $4k\times2k$ or $8k\times4k$ resolution and beyond) are pushing technologies forward and opening new research directions.
Over the recent years new advances in video compression and networking technologies have enabled the development of many multimedia applications over the internet, such as video conferencing, video streaming, video publishing, among others.
The increasing diversity of services, the demands for high quality multimedia and the growing popularity of \ac{uhd} formats (\emph{e.g.}, $4k$ or $8k$ resolution and beyond) are pushing technologies forward and opening new research directions.
Moreover, the emerging new video formats with increased resolutions have created stronger needs for higher coding efficiency, beyond the previous standards, i.e., the MPEG-4/\ac{h264}~\cite{H264AVCStandard2003,H264OverviewJul2003}.
Such need has been driven not only by higher resolutions but also by stereoscopic and multi-view video formats, which increase the number of viewpoints that need to be captured, coded and delivered simultaneously.
Such need has been driven not only by higher resolutions but also by stereoscopic and multi-view video formats, which increase the number of viewpoints that need to be encoded and delivered simultaneously.
Moreover, the ever increasing traffic generated by mobile multimedia applications \cite{CiscoMobile1621_Feb2017} and different video services with ever increasing popularity, e.g., Netflix, impose new challenges to the existing networks.
To cope with such demanding requirements a new coding standard was developed by the ITU-T \ac{vceg} and the \ac{iso}/\ac{iec} \ac{mpeg} standardisation organisations, referred to as H.265 - \ac{hevc}~\cite{H265StandardFeb2018,HEVCOverviewDez2012}.
\glsunset{h264}
The \ac{hevc} is currently the main video coding framework to extend current services towards higher resolution formats delivered with better compression efficiency.
Similar to its predecessor, \ac{hevc} is a block-based hybrid video codec using predictive transform and entropy coding.
The coding flexibility and efficiency of \ac{hevc}, results from the use of new coding tools and block structures, which enables new block partitions\cite{HEVCBlock2012}, improved prediction modes \cite{IntraCodingHEVC2012,AMPCodingHEVCJun2012} and new high-level features~\cite{HEVCHighLevelReferenceDez2012}, such as explicit reference picture management and new parameter sets.
However, it is known that higher coding efficiency leads to lower robustness to transmission errors and data loss.
Therefore, the new coding tools in \ac{hevc} bring not only high compression efficiency~\cite{HEVCComparisonDec2012}, but also some other disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and inherent reduction in error robustness.
Similar to its predecessor, \ac{hevc} is a block-based hybrid video codec using predictive, transform and entropy coding.
The coding flexibility and efficiency of \ac{hevc} results from the use of new coding tools and block structures, which enables new block partitions\cite{HEVCBlock2012}, improved prediction modes \cite{IntraCodingHEVC2012,AMPCodingHEVCJun2012} and new high-level features~\cite{HEVCHighLevelReferenceDez2012}, such as explicit reference picture management and new parameter sets.
However, it is known that higher coding efficiency leads to lower robustness in the presence of transmission errors and data loss.
Therefore, the \ac{hevc} new coding tools bring not only high compression efficiency~\cite{HEVCComparisonDec2012}, but also some other disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and inherent reduction of error robustness.
%\cite{StudyHEVCErrorsDez2012}
Earlier studies about the error robustness characteristics of \ac{hevc} have shown that in general the use of high efficiency coding tools results in poor error resilience performance, in comparison with previous standards.
Earlier studies about the error robustness characteristics of \ac{hevc} have shown that, in general, the use of high efficiency coding tools results in poor error resilience performance, in comparison with previous standards.
Thus, when \ac{hevc} streams incur in transmission losses, this leads to significant degradation of both objective and subjective quality~\cite{QoEHEVCMay2014,SubjectiveEvalSep2013}.
This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes of \ac{hevc}, which assume \mbox{error-free} transmission~\cite{ROPE2007}.
A possible approach to encode robust video streams is to increase the error resilience of \ac{hevc} by including \mbox{error-aware} mechanisms in the coding process. As a result, the effect of error propagation due to prediction mismatch at the decoding side, will be limited.
A possible approach to encode robust video streams is to increase the error resilience of \ac{hevc} by including \mbox{error-aware} mechanisms in the coding process. As a result, the effect of error propagation, due to prediction mismatch at the decoding side, will be limited.
The widespread use of \ac{hevc} in broadcast and multimedia delivery services, is be based on transport technologies such as \ac{rtp}~\cite{HEVCStreamMay2012}, \mbox{MPEG-2 TS}~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016}.
The widespread use of \ac{hevc} in broadcast and multimedia delivery services is based on transport technologies such as \ac{rtp}~\cite{HEVCStreamMay2012}, \mbox{MPEG-2 TS}~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016}.
Recent developments also enable combinations of different technologies, e.g, \mbox{HTTP-based} streaming and \mbox{MPEG-2 TS}, to increase the flexibility of over-the-top (OTT) video services\cite{Broadcast2017}. For content delivery over heterogeneous networks, the \ac{mmt}~\cite{ATSCMMT2016} is another possible technology to be used with the \ac{hevc} standard. Besides the transport technologies,
the overall error robustness of video transmission may also benefit from the increasing flexibility of multimedia systems to cope with heterogeneous networks, by using auxiliary streams to aid the video decoder to increase the quality of recovered lost slices~\cite{ErrorResilienceHeterogenousNetworks2016,HeterogeneousStreaming2016}. Moreover, error resilience techniques may also take advantage of the multi-stream based approach, where each stream allows different levels of robustness to errors~\mbox{\cite{HEVCHTTPStreamAug2016,PerfMediaTransport2017}}.
the overall error robustness of video transmission may also benefit from the increasing flexibility of multimedia systems to cope with heterogeneous networks, by using auxiliary streams that allow the video decoder to increase the quality of the recovered lost slices~\cite{ErrorResilienceHeterogenousNetworks2016,HeterogeneousStreaming2016}. Moreover, error resilience techniques may also take advantage of the multi-stream based approach, where each stream allows different levels of robustness to errors~\mbox{\cite{HEVCHTTPStreamAug2016,PerfMediaTransport2017}}.
%Despite the different characteristics of the existing services and applications, the users' \ac{qoe} consistently depends on how the compressed streams are adapted to the diverse networking technologies, as well as on how the transmission errors and/or packet losses are handled at the application level ~\cite{HEVCStreamMay2012,HEVCLayerIntegrationDez2012,StudyHEVCErrorsDez2012}. As mentioned before, the \ac{hevc} standard is expected to replace the \ac{h264} in future video services.
@@ -51,17 +51,19 @@ the overall error robustness of video transmission may also benefit from the inc
%follows.
\begin{figure}[tb]
\centering
\input{./010_Introduction/RobustVideoSystem.tikz}
\caption{Generalised diagram of a robust video transmission system.}
\label{FIG1:RobustVideoTransmission}
\end{figure}
In summary, increasing the error robustness of the highly compressed video streams is a relevant research problem whose solutions will be useful to increase the adaptability for transmission diverse network conditions and to achieve robust communications in order to provide acceptable levels of \ac{qoe} to the end users at all times.
In general, increased error robustness in video transmission can be jointly achieved at three different stages: (i) using robust video encoding, by selecting robust coding modes, (ii) after encoding, by adding robust features to the compressed streams as side-information and (iii) using robust video decoding techniques, e.g., efficient \ac{ec}. This is exemplified in Figure~\ref{FIG1:RobustVideoTransmission} where at the encoder different layer are used to increase the error robustness, which results in a video stream composed by a \ac{vcl} and \ac{sei} layers. Finally, at the decoder two processing layers are used to achieve robust video decoding.
The \ac{hevc} reference codec is continuously being improved and extended development by multiple proponents from around the world, also addressing the problem of error resilience and robust delivered transmission
Although the \ac{ec} methods have been developed during the past years for \ac{h264}~\cite{2DRecursiveRefinement2010} and recently for \ac{hevc}~\cite{HEVCMCBlockAug2013}, a combined and optimised approach between encoder and decoder is yet to be developed, which would improve the users’ \ac{qoe} by exploiting shared information between the encoder and the decoder sides.
In summary, increasing the error robustness of the highly compressed video streams is a relevant research problem, whose solutions will be useful to increase the adaptability for transmission over diverse network conditions and to achieve robust communications, thus providing acceptable levels of \ac{qoe} to the end users at all times.
In general, increased error robustness in video transmission can be jointly achieved at three different stages: (i) using robust video encoding, by selecting robust coding modes, (ii) after encoding, by adding robust features to the compressed streams, as side-information, and (iii) using robust video decoding techniques, e.g., efficient \ac{ec}. This is exemplified in Figure~\ref{FIG1:RobustVideoTransmission} where at the encoder different layers are used to increase the error robustness, which results in a video stream composed by a \ac{vcl} and \ac{sei} layers. Finally, at the decoder two processing layers are used to achieve robust video decoding.
The \ac{hevc} reference codec is continuously being improved and extended by multiple proponents from around the world, which also address the problem of error resilience and robust delivered transmission.
Although different \ac{ec} methods have been developed during the past years for \ac{h264}~\cite{2DRecursiveRefinement2010} and recently for \ac{hevc}~\cite{HEVCMCBlockAug2013}, a combined and optimised approach involving the encoder and decoder is yet to be developed, which would improve the users’ \ac{qoe}, by exploiting shared information between the encoder and the decoder sides.
\begin{figure}[tb]
\centering
\input{./010_Introduction/RobustVideoSystem.tikz}
\caption{Generalised diagram of a robust video transmission system.}
\label{FIG1:RobustVideoTransmission}
\end{figure}
\section{Main objectives and contributions}
021_VideoCoding/VideoCodingReview.tex
+2 -2
View File
@@ -34,7 +34,7 @@ Section~\ref{Section2:Summary} concludes this chapter.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% SECTION %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{General hybrid video coding}
\section{Generic hybrid video coding}
\label{Section2:GeneraVideoSystem}
A digital video signal is a temporal succession of images, also referred to as pictures or frames, which usually presents a significant amount of temporal and spatial correlations.
@@ -586,7 +586,7 @@ These partitions types are only allowed for \acs{cu} with $16\times16$ pixels or
\centering
\input{\FigsPath/MergeModeHEVC.tikz}
%\input{\FigsPath/HEVC/MergeModeHEVCTwoFrames.tikz}
\caption{Illustration of the MV prediction candidates in the AMVP.}
\caption{Illustration of the MV prediction candidates in the AMVP mode.}
\label{Figure2:MVCandidates}
\end{figure}
031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex
+43 -43
View File
@@ -6,7 +6,7 @@
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CHAPTER
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Error robustness of HEVC - Evaluation and improvements}
\chapter{Error robustness of HEVC - Evaluation and improvements \VERSION{1}}
\label{Chapter:HEVCErrorResilience}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -103,7 +103,8 @@ In the first set of experiments the impact of the maximum \ac{cu} size in the de
Table~\ref{Table3:Quality_BlockSize} shows the average \ac{psnr} obtained for the three test conditions and for four different packet loss ratios. Results for the error-free scenario shows a consistent quality increase as the maximum \ac{cu} size increases, up to a maximum of 1.41 dB for the Tennis sequence when comparing the maximum \ac{cu} size of $64\times64$ with $16\times16$.
This confirms the principle used by the \ac{hevc} standard, that allowing larger coding units lead to higher coding efficiency, achieving a maximum of 1.41~dB for the Tennis sequence which has the higher \ac{ti} parameter (see Table~\ref{TableTestSequences}). Moreover, it is also noticeable that sequences with higher amount of inter-coded blocks take less advantage of the larger \acs{cu}, by the results of the BQSquare sequence (i.e., highest ratio of inter-coded blocks and lowest quality gain in the error-free case).
Overall, comparing the results for the error-free and data loss cases it is noticeable that similar quality gains are obtained, which are achieved due to the superior coding efficiency of larger \acs{cu}. In most of the cases the quality gains slightly decrease for increasing \acs{plr}.
Therefore, one may conclude that the error robustness of \ac{hevc} streams is not affected by limiting the maximum \ac{cu} size. \todo{Complete discussion of partition unit size}
Therefore, one may conclude that the error robustness of \ac{hevc} streams is not affected by limiting the maximum \ac{cu} size.
%\todo{Complete discussion of partition unit size}
%Results for the case of data loss indicate that using larger coding units leads to higher video quality.
@@ -171,7 +172,7 @@ Figure~\ref{Figure3:QualityvsIntra} shows the average decoded video quality (\ac
Results in the figure indicate that as the distance between intra frames increases, the decoded video quality significantly decreases.
The quality decreasing is especially noticeable for intra periods higher than 32 frames, indicating that temporal distances greater than approximately one second are not suitable for video streaming under error-prone conditions.
When a high temporal distance between intra frames is used, the quality decreasing is more noticeable in the People On Street and BQSquare sequences, which have higher spatial complexity (see values for \ac{si}), leading to a reduction of 13 dB and 14 dB, respectively. As spatial complexity increases the encoder rely more on temporal predictions making the bitstream more prone to error propagation.
\todo{Complete discussion of intra refreshing}
%\todo{Complete discussion of intra refreshing}
One should also note from the results of Figure~\ref{Figure3:QualityvsIntra} that using a small distance between intra frames is not always more efficient.
Since intra frames require higher amount of bits, the overall coding efficiency is reduced, resulting in an overall quality that decreases even in data loss conditions.
@@ -182,8 +183,7 @@ For the remaining sequences it can be concluded that using sixteen frames (not h
The partitioning of the \acs{ctu} into different slices increase the flexibility of video streams to be partitioned into different packets, thus adapting them to the video transmission system.
By partitioning the coded video frame into different slices spatial refresh points are introduced, limiting the error propagation across different \acs{ctu}.
In this sub-section the amount of bits per slice is evaluated, in order to evaluate the influence of different schemes on the error robustness of \ac{hevc} streams. The quality obtained different mode is compared in order to find the most efficient that suits for video transmission under data loss conditions.
%As a reference for comparison a single slice per frame is also evaluated.
In this sub-section the amount of bytes per slice is evaluated, in order to evaluate the influence of different schemes on the error robustness of \ac{hevc} streams. The quality obtained different mode is compared in order to find the most efficient that suits for video transmission under data loss conditions.
\begin{figure}[!t]
\centering
@@ -194,15 +194,18 @@ In this sub-section the amount of bits per slice is evaluated, in order to evalu
\label{Figure3:SlicePartitioning}
\end{figure}
Figure~\ref{Figure3:SlicePartitioning} shows the average \ac{psnr} for three difference slice partitioning modes and different \acs{plr}.
The first mode uses a single slice per frame, corresponding to the case where slice partitioning is disabled.
The two remaining are characterised by a different amount of bits per slice, namely 1200 and 2400 bytes per slice, respectively. In these two cases each frame has a variable number of slices.
\todo{Complete discussion of slice partitioning}
Figure~\ref{Figure3:SlicePartitioning} shows the average \ac{psnr} for four difference slice partitioning modes and different \acs{plr}.
The first three modes are characterised by a different amount of bytes per slice (B/S), namely 1200, 2400 and 4800 bytes, respectively. In these cases each frame has a variable number of slices.
The last mode uses a single slice per frame (referred to as \textit{SS}), corresponding to the case where slice partitioning is disabled.
Results in the figure reveals a clear impact of the slice partitioning mode on the error robustness of the \ac{hevc}, as the quality degradation for increasing \ac{plr} significantly changes for each mode.
Comparing the cases with fixed amount of bytes per slice, it is clearly noticeable that when higher amount of bytes are used the quality degradation increases. This is true for the results of most of the test sequences shown in Figure~\ref{Figure3:SlicePartitioning}.
Although the use of smaller slices increases the overhead used for slice headers, the superior error robustness of achieved with 1200 B/S is able to overcome the inherent degradation on coding efficiency.
This is due to the lack of refresh points within a frame, resulting in errors that affects larger regions of the frame which are propagated to the subsequent inter predicted frames.
Results reveals a clear impact of the slice partitioning on the error robustness of the \ac{hevc}, as the quality degradation as the \ac{plr} increases varies for each scheme.
Firstly it is clearly notice the disadvantage of using a single slice per frame as it leads to the lower quality when comparing the test cases studied. This is due to the lack of refresh points within a frame, resulting in loss events that affects the whole frame which will propagate to the subsequent inter predicted frames.
Secondly, comparing slices of different sizes, i.e., comparing the use of 1200 and 2400 bytes per slice, reveals that using small slices leads to higher quality which is able to overcome the inherent degradation on coding efficiency due to higher amount of slice headers.
The results for of \textit{SS} (slice partitioning disabled) also reveals that this approach also leads to lower error robustness when comparing with those cases where the frames are divided into multiple slices. Although higher coding efficiency is achieved in this mode the lower error robustness leads to an overall poor performance under frame or slice loss conditions.
Results for Kendo sequence does not follow the main trend. This can be explained by the low bitrate required to compress the video signal, which is the lowest comparing with the other test sequences (see \ref{TableTestSequences}). Therefore, most of the inter-coded frames are able to transmitted using a single slice even with 1200 bytes per slice, resulting in a similar partitioning for all tested modes, and thus similar error robustness. In this case, the higher coding efficiency achieved by the \textit{SS} case consequently leads to higher quality in case of errors.
%\todo{Complete discussion of slice partitioning}
\subsection{Motion vector coding impact on error propagation}
@@ -212,47 +215,44 @@ Therefore, in this sub-section the impact of using spatial and temporal \ac{mv}
\subsubsection{Spatial \ac{mv} candidates}
\begin{figure}[!t]
\centering
\renewcommand{\FIGSIZE}{0.75\textwidth}
\input{031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsMergeCand.tex}
\caption{Average quality using different number of \ac{mv} candidates.}
\label{Figure3:AvgPSNRMergeCandidates}
\end{figure}
The first set of experiments aimed to investigate the impact of the amount of spatial \ac{mv} candidates in the video quality.
As mentioned before, \ac{hevc} allows up to five candidates to be used, which can be provided solely by spatial neighbours. In order to check the impact of the \ac{mv} candidates, different coding configurations were used, each one characterised by different amounts of spatial \ac{mv} candidates, i.e., 1, 3 and 5 \acs{mv}.
\begin{figure}[!t]
\centering
\renewcommand{\FIGSIZE}{0.75\textwidth}
\input{031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsMergeCand.tex}
\caption{Average quality using different number of \ac{mv} candidates.}
\label{Figure3:AvgPSNRMergeCandidates}
\end{figure}
Figure~\ref{Figure3:AvgPSNRMergeCandidates} shows the average quality results for the Basketball Drill and People On Street sequences under different packet loss conditions.
The results show that, besides the number of \ac{mv} candidates, similar quality decreasing is achieved as the \ac{plr} increases. Therefore, one may conclude that varying the number of available \ac{mv} candidate does not affect the error resilience of the \ac{hevc}.
\begin{table}[!t]
\newcommand{\somespaces}{}
\newcommand{\colwidth}{1.8cm}
\centering
\caption{Relation between the number of available \ac{mv} candidates and its usage} \label{Table3:MergeIdxRatio}
\footnotesize
\TABLESVSPACE
\begin{tabular}{x{3.3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}}
\toprule
\multirow{2}{*}{\vbox{\centering \textbf{Max. number of candidates}}} & \multicolumn{5}{c}{\textbf{Candidate selected (ratio)}} \\
\cline{2-6}
& 1 & 2 & 3 & 4 & 5 \\
\hline
3 \acs{mv} & $66.17\%$ & 22.79$\%$ & 11.04$\%$ & - & - \\
\hline
5 \acs{mv} & 64.54$\%$ & 22.24$\%$ & 7.38$\%$ & 3.37$\%$ & 2.47$\%$ \\
\bottomrule
\end{tabular}
\end{table}
The occurrence rate for each spatial \ac{mv} candidate was analysed in order to check whether they are uniformly used.
Table~\ref{Table3:MergeIdxRatio} shows the usage percentage of each \ac{mv} candidate when three and five candidates are available, for the slice partitioning mode of 1200 bytes/slice. The results indicate that most predictions use the first candidate and minor differences occur by adding 2 or more \acs{mv}.
Since the mostly used \ac{mv} candidates are the initial ones, changing the number of available candidates does not significantly affects the error resilience.
Moreover, the slice partitioning mode used only 1200 bytes per slice, which introduces refresh points in each frame, reducing the spatial error propagation due to mismatched \ac{mv} predictions.
\begin{table}[tb]
\newcommand{\somespaces}{}
\newcommand{\colwidth}{1.8cm}
\centering
\caption{Relation between the number of available \ac{mv} candidates and its usage} \label{Table3:MergeIdxRatio}
\footnotesize
\TABLESVSPACE
\begin{tabular}{x{3.3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}}
\toprule
\multirow{2}{*}{\vbox{\centering \textbf{Max. number of candidates}}} & \multicolumn{5}{c}{\textbf{Candidate selected (ratio)}} \\
\cline{2-6}
& 1 & 2 & 3 & 4 & 5 \\
\hline
3 \acs{mv} & $66.17\%$ & 22.79$\%$ & 11.04$\%$ & - & - \\
\hline
5 \acs{mv} & 64.54$\%$ & 22.24$\%$ & 7.38$\%$ & 3.37$\%$ & 2.47$\%$ \\
\bottomrule
\end{tabular}
\end{table}
\subsubsection{Temporal \ac{mv} candidates}
@@ -292,7 +292,7 @@ Moreover, the results also indicate that for higher loss ratios the negative imp
\input{./031_ErrorResilienceEvaluation/QualityEvaluation/TableMVPs.tex}
\end{table}
\subsection{Discussion}
\subsection{Remarks}
Overall, the results presented in the previous sub-sections show that \ac{hevc} has poor robustness against transmission errors, resulting in significant quality degradation, though dependent still on the video content and coding options.
As shown, some coding options, such as slice partition and intra refreshing period, have high correlation with the decoded video quality when transmission errors affect the compressed streams.
031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsMergeCand.tex
+1 -1
View File
@@ -1,5 +1,5 @@
\newcommand{\pltw}{0.75\columnwidth}
\newcommand{\plth}{0.5\textheight}
\newcommand{\plth}{0.4\textheight}
\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}]
load 'carreira_phd_thesis_gnuplot_template.cfg'
031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsSlices.tex
+160 -184
View File
@@ -1,156 +1,97 @@
\newcommand{\ft}[0]{\footnotesize}
\newcommand{\pltw}{\columnwidth}
\newcommand{\plth}{0.6\textheight}
\newcommand{\plth}{0.7\textheight}
\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}]
load 'carreira_phd_thesis_gnuplot_template.cfg'
set multiplot layout 3,2
set key width -3
set key width -1
set yrange[28:42]
set ytics 0,2,60
set title '\TitleFont Basketball Drill'
plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3
0.0 40.080
1.0 39.198
3.0 37.351
5.0 35.953
10.0 33.024
e
0.0 39.900
1.0 38.462
3.0 36.705
5.0 35.526
10.0 33.680
e
0.0 39.970
1.0 37.988
3.0 34.884
5.0 32.921
10.0 29.059
e
# set title '\TitleFont Book Arrival'
# set yrange[28:42]
# set ytics 0,2,60
# set title '\TitleFont Basketball Drill'
# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3
# 0.0 40.390
# 1.0 39.635
# 3.0 37.954
# 5.0 36.821
# 10.0 34.185
# 0.0 40.080
# 1.0 39.198
# 3.0 37.351
# 5.0 35.953
# 10.0 33.024
# e
# 0.0 40.280
# 1.0 38.818
# 3.0 36.334
# 5.0 34.694
# 10.0 31.357
# 0.0 39.900
# 1.0 38.462
# 3.0 36.705
# 5.0 35.526
# 10.0 33.680
# e
# 0.0 40.320
# 1.0 39.093
# 3.0 36.939
# 5.0 34.280
# 10.0 30.948
# 0.0 39.970
# 1.0 37.988
# 3.0 34.884
# 5.0 32.921
# 10.0 29.059
# e
set yrange[31:41]
set ytics 31,2,60
set title '\TitleFont BQSquare'
plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3
0.0 39.430
1.0 38.946
3.0 37.851
5.0 36.991
10.0 35.078
# set yrange[31:41]
# set ytics 31,2,60
# set title '\TitleFont BQSquare'
# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3
# 0.0 39.430
# 1.0 38.946
# 3.0 37.851
# 5.0 36.991
# 10.0 35.078
# e
# 0.0 39.300
# 1.0 38.486
# 3.0 36.973
# 5.0 35.779
# 10.0 33.624
# e
# 0.0 39.350
# 1.0 38.468
# 3.0 36.429
# 5.0 34.475
# 10.0 31.059
# e
set yrange[30:42]
set ytics 30,2,60
set title '\TitleFont Kendo'
plot '-' using 1:2 title '\KeyFont 1200 B/S' w lp ls 1, '-' using 1:2 title '\KeyFont 2400 B/S' w lp ls 2, '-' using 1:2 title '\KeyFont 4800 B/S' w lp ls 3, '-' using 1:2 title '\KeyFont SS' w lp ls 4
0.0 41.150
1.0 39.809
3.0 37.264
5.0 34.900
10.0 30.713
e
0.0 39.300
1.0 38.486
3.0 36.973
5.0 35.779
10.0 33.624
0.0 41.240
1.0 40.124
3.0 37.752
5.0 35.788
10.0 32.144
e
0.0 39.350
1.0 38.468
3.0 36.429
5.0 34.475
10.0 31.059
0.0 41.290
1.0 40.297
3.0 38.161
5.0 36.641
10.0 33.139
e
0.0 41.320
1.0 40.405
3.0 38.475
5.0 37.084
10.0 34.076
e
# set title '\TitleFont Cactus'
# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3
# 0.0 38.060
# 1.0 37.545
# 3.0 36.519
# 5.0 35.651
# 10.0 33.521
# e
# 0.0 38.000
# 1.0 37.707
# 3.0 37.400
# 5.0 37.270
# 10.0 36.930
# e
# 0.0 38.030
# 1.0 37.719
# 3.0 37.329
# 5.0 37.122
# 10.0 36.666
# e
# set title '\TitleFont Four People'
# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3
# 0.0 40.270
# 1.0 40.081
# 3.0 39.678
# 5.0 39.272
# 10.0 38.230
# e
# 0.0 40.130
# 1.0 39.862
# 3.0 39.406
# 5.0 39.005
# 10.0 38.115
# e
# 0.0 40.180
# 1.0 39.947
# 3.0 39.328
# 5.0 38.753
# 10.0 37.494
# e
# set title '\TitleFont Kendo'
# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3
# 0.0 41.320
# 1.0 40.405
# 3.0 38.475
# 5.0 37.084
# 10.0 34.076
# e
# 0.0 41.150
# 1.0 39.809
# 3.0 37.264
# 5.0 34.900
# 10.0 30.713
# e
# 0.0 41.240
# 1.0 40.124
# 3.0 37.752
# 5.0 35.788
# 10.0 32.144
# e
set yrange[36:44]
set ytics 36,2,60
# unset yrange
# unset ytics
set yrange[36.5:43.5]
set ytics 37,1,60
set title '\TitleFont Kimono'
plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3
0.0 42.940
1.0 42.180
3.0 40.589
5.0 39.315
10.0 36.789
e
plot '-' using 1:2 title '\KeyFont 1200 B/S' w lp ls 1, '-' using 1:2 title '\KeyFont 2400 B/S' w lp ls 2, '-' using 1:2 title '\KeyFont 4800 B/S' w lp ls 3, '-' using 1:2 title '\KeyFont SS' w lp ls 4
0.0 42.880
1.0 42.247
3.0 41.587
@@ -163,6 +104,18 @@ e
5.0 40.650
10.0 39.841
e
0.0 42.910
1.0 42.001
3.0 40.519
5.0 39.585
10.0 38.254
e
0.0 42.940
1.0 42.180
3.0 40.589
5.0 39.315
10.0 36.789
e
# set title '\TitleFont Kristen Sara'
@@ -186,16 +139,10 @@ e
# 10.0 38.111
# e
set yrange[34:42]
set ytics 34,2,60
set yrange[34:41]
set ytics 34,1,60
set title '\TitleFont Park Scene'
plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3
0.0 39.570
1.0 39.033
3.0 37.859
5.0 36.923
10.0 34.945
e
plot '-' using 1:2 title '\KeyFont 1200 B/S' w lp ls 1, '-' using 1:2 title '\KeyFont 2400 B/S' w lp ls 2, '-' using 1:2 title '\KeyFont 4800 B/S' w lp ls 3, '-' using 1:2 title '\KeyFont SS' w lp ls 4
0.0 39.470
1.0 38.905
3.0 38.385
@@ -208,17 +155,23 @@ e
5.0 37.856
10.0 37.133
e
set yrange[29:37]
set ytics 29,2,60
set title '\TitleFont People On Street'
plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3
0.0 36.140
1.0 35.257
3.0 33.619
5.0 32.391
10.0 29.845
0.0 39.530
1.0 38.806
3.0 37.747
5.0 37.014
10.0 35.329
e
0.0 39.570
1.0 39.033
3.0 37.859
5.0 36.923
10.0 34.945
e
set yrange[28:36.5]
set ytics 28,2,60
set title '\TitleFont People On Street'
plot '-' using 1:2 title '\KeyFont 1200 B/S' w lp ls 1, '-' using 1:2 title '\KeyFont 2400 B/S' w lp ls 2, '-' using 1:2 title '\KeyFont 4800 B/S' w lp ls 3, '-' using 1:2 title '\KeyFont SS' w lp ls 4
0.0 35.900
1.0 34.736
3.0 33.795
@@ -231,17 +184,23 @@ e
5.0 32.490
10.0 31.323
e
0.0 36.030
1.0 34.236
3.0 31.765
5.0 30.494
10.0 28.042
e
0.0 36.140
1.0 35.257
3.0 33.619
5.0 32.391
10.0 29.845
e
set yrange[30:42]
set yrange[30:40.5]
set ytics 30,2,60
set title '\TitleFont Race Horses'
plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3
0.0 40.030
1.0 38.846
3.0 36.403
5.0 34.608
10.0 30.944
e
plot '-' using 1:2 title '\KeyFont 1200 B/S' w lp ls 1, '-' using 1:2 title '\KeyFont 2400 B/S' w lp ls 2, '-' using 1:2 title '\KeyFont 4800 B/S' w lp ls 3, '-' using 1:2 title '\KeyFont SS' w lp ls 4
0.0 39.870
1.0 38.724
3.0 37.661
@@ -254,30 +213,47 @@ e
5.0 35.580
10.0 34.204
e
0.0 39.960
1.0 37.729
3.0 34.768
5.0 32.972
10.0 30.390
e
0.0 40.030
1.0 38.846
3.0 36.403
5.0 34.608
10.0 30.944
e
#
# set title '\TitleFont Tennis'
# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3
# 0.0 40.900
# 1.0 39.773
# 3.0 37.491
# 5.0 35.896
# 10.0 32.303
# e
# 0.0 40.680
# 1.0 39.242
# 3.0 37.515
# 5.0 36.545
# 10.0 35.064
# e
# 0.0 40.770
# 1.0 38.369
# 3.0 35.105
# 5.0 33.079
# 10.0 29.667
# e
set yrange[27:42]
set ytics 27,2,60
set title '\TitleFont Tennis'
plot '-' using 1:2 title '\KeyFont 1200 B/S' w lp ls 1, '-' using 1:2 title '\KeyFont 2400 B/S' w lp ls 2, '-' using 1:2 title '\KeyFont 4800 B/S' w lp ls 3, '-' using 1:2 title '\KeyFont SS' w lp ls 4
0.0 40.680
1.0 39.242
3.0 37.515
5.0 36.545
10.0 35.064
e
0.0 40.770
1.0 38.369
3.0 35.105
5.0 33.079
10.0 29.667
e
0.0 40.830
1.0 38.307
3.0 34.743
5.0 32.276
10.0 27.524
e
0.0 40.900
1.0 39.773
3.0 37.491
5.0 35.896
10.0 32.303
e
\end{gnuplot}
041_ImprovedErrorResilience/ImprovedErrorResilience.tex
+5 -5
View File
@@ -3,7 +3,7 @@
\graphicspath{{\ImprovedErrorResilience/Results/}}
\chapter{A Two-stage Approach for Robust Coding and Streaming}
\chapter{A Two-stage Approach for Robust Coding and Streaming \VERSION{1}}
% \chapter{Error Resilience Techniques for \ac{hevc} Standard}
%\chapter{Two-stage Robust Video Coding}
\label{Chapter:ErrorResilienceTechniques}
@@ -110,7 +110,7 @@ However, such solution would not take into consideration coding efficiency, sinc
\begin{figure}[tb]
\centering
\input{\ImprovedErrorResilience/Figures/RefPicSelection.tikz}
\caption{Example of the reference picture selection applied to the frame $C$ based on the checkerboard at \ac{ctu} level (\acs{pu} represented by small partitions).}
\caption{Example of the reference picture selection applied to the frame $C$ based on the checkerboard at \ac{ctu} level (\acs{pu} are represented by small partitions).}
\label{fig:checkerboardRefSelection}
\end{figure}
@@ -359,7 +359,7 @@ Taking into account the above results and the spatio-temporal information shown
%\includegraphics[width=\FIGSIZE,page=1]{\ImprovedErrorResilience/Results/PSNRversusGama}
%\includegraphics[width=\FIGSIZE,page=2]{\ImprovedErrorResilience/Results/PSNRversusGama}
\input{./041_ImprovedErrorResilience/Results/PSNRversusGama.tex}
\caption{Influence of the $\gamma$ parameter in the average quality gains, comparing with \REF, for different \acs{plr}.}
\caption{Influence of the $\gamma$ parameter in the average quality gains, compared with \REF, for different \acs{plr}.}
\label{fig:QualityVsGamma}
\end{figure}
@@ -429,7 +429,7 @@ This is due to the fact that in the \CHKB method there is no \ac{rd} optimisatio
\begin{table}[tb]
\newcommand{\resultswidth}{2.2cm}
\centering
\caption{Relative CPU times of tested methods.}\TABLESVSPACE
\caption{Relative CPU times of the tested methods.}\TABLESVSPACE
\label{tab:cpu_time}
\footnotesize
\begin{tabular}{p{4cm}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}}
@@ -462,7 +462,7 @@ Different values of $\alpha$ are used to achieve different overhead ratios. All
\includegraphics[width=\CurrFigSize,page=4]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
\includegraphics[width=\CurrFigSize,page=5]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
\includegraphics[width=\CurrFigSize,page=6]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf}
\caption{Decoded video quality for different amounts of redundancy, selected using a fixed approach and the proposed method.}
\caption{Decoded video quality for different amounts of redundancy, selected by using a fixed approach and the proposed method.}
\label{fig:QualityVsRedundancyMVSelection}
\end{figure}
042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex
+32 -21
View File
@@ -4,7 +4,7 @@
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CHAPTER
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Error Concealment-Aware Video Encoding}
\chapter{Error Concealment-Aware Video Encoding \VERSION{1}}
\label{Chapter:ConcealmentAwareTechniques}
%\vspace{-6mm}
%\clearpage
@@ -119,7 +119,7 @@ Besides a full search algorithm where a block-based approach is used to search f
\begin{enumerate}
\item a method based on the total variation using the minimization of the \mbox{L-1} norm~\cite{DualFlow2016};
\item a method which approximates the neighbourhood of each pixel by quadratic polynomials on both frames~\cite{PolyFlow2003};
\item a method based on a combination of descriptor matching and large displacement estimation~\cite{DeepFlow2013}. \HERE
\item a method based on a combination of features matching and total variations~\cite{DeepFlow2013}. \HERE
\end{enumerate}
\subsubsection{Motion estimation based on total variation and regularisation (\textit{DualFlow})}
@@ -161,14 +161,21 @@ This allows to find the displacement $\mathbf{d}$ for any signal dimensions by s
\subsubsection{Large displacement optical flow with deep matching (\textit{DeepFlow})}
\todo{Completar o Deep flow}
%\todo{Completar o Deep flow}
In this method the motion field is estimated based on the large displacement optical flow presented in \cite{LargeDisplacementOptFlow_TPAMI2011}, which extends the total variation methods, e.g. the one expressed by Equation (\ref{EQ5:DualFlow}), by including descriptor matching.
The correspondences between descriptions are used to apply a coarse-to-fine wrapping, thus avoiding local minimum values. The matching algorithm is built upon a multi-stage architecture with six layers with different patches sizes, interleaving convolutions and max-pooling \cite{DeepFlow2013}. The combination of both approaches increases the matching accuracy.
In this method the motion field is estimated based on extension of the total variational methods using descriptor (feature) matching.
This allows the optical flow estimation to accurately find large displacements as it is not bound by constrains, such as the Horn \& Such presented in Equation~(\ref{EQ5:DualFlow})\cite{LargeDisplacementOptFlow_TPAMI2011}. Moreover, the correspondences between descriptions are used to apply a coarse-to-fine wrapping, thus avoiding local minimum values.
The combination of both approaches increases the matching accuracy.
In \cite{DeepFlow2013} to improve the estimated motion field accuracy, the following extension to the method proposed in~\cite{LargeDisplacementOptFlow_TPAMI2011} were introduced: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layers of optimisation. Figure~\ref{FIG5:DeepFlowDiagram} shows the blocks that are included in the estimation process. As shown in the figure a multi-size optimisation is applied by using different patches sizes. This is used to estimate an initial motion field, referred to as $v'$ .
This was further exploited by introducing deep convolutional networks~\cite{DeepFlow2013}.
This was also exploited in \cite{DeepFlow2013} with the following extensions: (i) use of description matching in the minimisation problem; (ii) introduction of a normalisation parameter to decrease the impact of regions with high spatial changes; (iii) use of different weights for each layer optimisation.
Figure~\ref{FIG5:DeepFlowDiagram} shows the blocks that are included in the description matching process.
The matching algorithm is built upon a multi-stage architecture with six layers with different patches sizes, interleaving convolutions and max-pooling, which is similar to deep convolutional networks \cite{GradientBasedConvNN_ProcIEEE1998}.
As shown in the figure a multi-size response levels are generated from the reference and target image using convolutions with different patches sizes. This method uses bottom-up approach, i.e., starting at a fine level and moving towards the coarser levels (larger patches) which are built as an aggregation of responses of smaller patches. This results in a multi-size response pyramid.
Subsequently, a local maximum is extracted from each level to achieve a more accurate motion field (quasi-dense correspondence).
Finally, an initial motion field is extracted, referred to as $v'$ .
\begin{figure}[b]
\begin{figure}[tb]
\centering
\includegraphics[width=.95\columnwidth]{\ConcealmentAwareDir/Figures/DeepFlowBlockDiagram}
\caption{Block diagram of the deep matching algorithm \cite{DeepFlow2013}.}
@@ -194,7 +201,10 @@ Moreover, the four different approaches to estimate the motion field for the mis
Table~\ref{TAB5:ECPerformaceComparison} shows the average quality (\ac{psnr}) obtained for the different methods under investigation.
The absolute value of \ac{psnr} is shown for the \FC method, while the differential values are shown for the remaining methods.
The results shown in the table reveal that methods \MC and \MVE are both able to overcome \FC, achieving average quality gains of up to 1.41 dB and 1.46 dB, respectively. Moreover, by extrapolating the \acs{mv} from the correctly receiving frame to the lost one (\MVE), it is also possible to outperform motion-copy (\MC).
The results shown in the table reveal that methods \MC and \MVE are both able to overcome \FC, achieving average quality gains of up to 1.41 dB and 1.46 dB, respectively. Moreover, by extrapolating the \acs{mv} from the correctly receiving frame to the lost one (\MVE), it is also possible to outperform motion-copy (\MC).
This can be explained by the objects' trajectories of objects which are preserved in the \MVE method (assuming constant translational motion), in contrast with the \MC method, which assumes that neighbouring objects have parallel motion which may result in inaccurate estimations.
%\HERE
\begin{table}[tb]
\centering
@@ -204,10 +214,12 @@ The results shown in the table reveal that methods \MC and \MVE are both able to
\input{\ConcealmentAwareDir/Results/PerformanceConcealmentMethods.tex}
\end{table}
\HERE
Results for \ac{ec} methods based on motion estimation (see columns 5 to 8 of Table~\ref{TAB5:ECPerformaceComparison}) reveal an overall superior performance when comparing with methods which rely of correctly received \acs{mv}.
Results for \FULLSEARCH method reveals that only minimising the sum of absolute errors between two consecutive frames might not always be directly related with objects' motions. Thus, this results in an inaccurate motion estimation to be used for the reconstruction of lost frames. Finally, results from Table~\ref{TAB5:ECPerformaceComparison} reveal that the \DEEPFLOW algorithm leads to the most accurate motion field to be used for \ac{ec}, in comparison with the other methods. It is able to achieve an average quality gain of 1.79 dB, and up to 3.39 dB for Book Arrival sequence, where \POLYFLOW only achieves 3.05 dB.
According to the results of Table~\ref{TAB5:ECPerformaceComparison} (columns 5 to 8) the \ac{ec} methods based on motion estimation achieve an overall superior performance when compared with methods which rely of correctly received \acs{mv} (i.e., \MC and \MVE).
Results for \FULLSEARCH method reveal that only minimising the sum of absolute errors between two consecutive frames might not always be directly related with objects' motions, which results in inaccurate motion field to be used in the reconstruction of lost frames.
Finally, results from Table~\ref{TAB5:ECPerformaceComparison} reveal that the \DEEPFLOW algorithm leads to the most accurate motion field, in comparison with the other methods.
This is able to achieve an average quality gain of 1.79 dB, and up to 3.39 dB for Book Arrival sequence, where \POLYFLOW only achieves 3.05 dB.
Since \POLYFLOW combines different techniques also used by other algorithms in study, e.g., total variation also used by \DUALFLOW, it is expected to achieve accurate estimations and consequently higher \ac{ec} reconstruction quality. Moreover, using descriptor matching in combination with smoothing conditions, the \DEEPFLOW algorithm is able to correctly estimate large displacements, which results in an improved performance for video signals with higher motion, e.g., Tennis sequence.
\subsection*{Remarks}
@@ -231,16 +243,6 @@ After the video encoder, it includes data loss simulation at the \ac{nal} for ev
Saliency-weighted \ac{ec} mode decision is used to select the optimal \ac{ec} mode from the best trade-off between \ac{ec} distortion, using the original input video~($f$) as reference, and the number of bits required for signalling.
Then, the optimal \ac{ec} mode is sent to the decoder along with the residue formed between the error-free decoded frame and its error concealed counterpart. The scalable extension of the \ac{hevc} standard is used to encode this residue information.
\begin{figure}[tb]
\centering
%\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram}
\input{042_ConcealmentAwareEncoding/Figures/BlockDiagram.tikz}
\caption{Architecture of the error concealment-aware encoding scheme.}
\label{FIG5:BlockDiagram}
\FIGSVSPACE
\end{figure}
For the \ac{ec} mode decision optimization the missing slices are reconstructed, as they would be in the decoder, i.e., recovered at the encoder by simulating the decoder \ac{ec}, but using different \ac{ec} methods based on motion estimation and extrapolation techniques, to selected the best one at the encoder-side.
In this process, each \ac{ctu} is partitioned to form a quadtree structure of \ac{ecu} that are used to test the different \ac{ec} candidate modes.
Then, the best trade-off between \ac{ecu} partition and signalling is found through \ac{rd} optimisation.
@@ -249,6 +251,15 @@ Finally, the signalling information is transmitted to the decoder by using the \
Since in general, the optimal \ac{ec} mode is still not able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el}.
This residual information contains the difference between the compressed ($\hat f_t$) and the \ac{ec} frames ($\tilde f$).
\begin{figure}[tb]
\centering
%\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram}
\input{042_ConcealmentAwareEncoding/Figures/BlockDiagram.tikz}
\caption{Architecture of the error concealment-aware encoding scheme.}
\label{FIG5:BlockDiagram}
\FIGSVSPACE
\end{figure}
The transmitted information is used to assist the \ac{ec} operation at the decoder to recover erroneous frames with high accuracy, by using the best \ac{ec} mode and also residual information to reduce error propagation.
The remaining of this section describes in detail the building blocks of the proposed encoding architecture, i.e., the \ac{ec} candidate modes, the saliency-weighted optimal \ac{ec} mode decision, signalling and enhancement layer coding.
042_ConcealmentAwareEncoding/Results/TableSignallingPerformance.tex
+1 -1
View File
@@ -120,7 +120,7 @@ Tennis & 28.47 (30.79) & +1.66 (+1.27) & +2.
\begin{sidewaystable}
\newcommand{\somespaces}{}
\centering
\caption{Average \ac{fsim} of the entire video sequence in frame loss conditions (scaled between $[0-100]$).}
\caption{Average \ac{fsim} under frame loss conditions (scaled between $[0-100]$).}
\label{TAB4:SignallingFSIM}
\TABLESVSPACE
\small
carreira_phd_thesis.tex
+1 -1
View File
@@ -44,7 +44,7 @@
\titlefigs{\includegraphics[width=10cm]{lborolodonlogo.png}}
% make the title area
\listoftodos
%\listoftodos
\maketitle
clean_files.bash
+1
View File
@@ -18,6 +18,7 @@ PATTERN_LIST="
*.maf
*.mtc*
*.out
*.tdo
*.toc
*.synctex.gz
*_latexmk