commit deffaca3fc3c8b879f4bf35918ef42799173b432 Author: Joao Carreira Date: Fri Jul 6 17:56:46 2018 +0100 Version 0.1 diff --git a/000_Preamble/LU/access.pdf b/000_Preamble/LU/access.pdf new file mode 100644 index 0000000..2cb1ea0 Binary files /dev/null and b/000_Preamble/LU/access.pdf differ diff --git a/000_Preamble/LU/origin.pdf b/000_Preamble/LU/origin.pdf new file mode 100644 index 0000000..5b0ace5 Binary files /dev/null and b/000_Preamble/LU/origin.pdf differ diff --git a/000_Preamble/abstract.tex b/000_Preamble/abstract.tex new file mode 100755 index 0000000..7a541f7 --- /dev/null +++ b/000_Preamble/abstract.tex @@ -0,0 +1,27 @@ + +\selectlanguage{english} + +\chapter*{\HIGHLIGHT{Abstract}} +\addcontentsline{toc}{chapter}{Abstract} + +% The increasing diversity of services, the demands for high quality multimedia and the growing popularity of Ultra-HD formats are pushing technologies forward and opening new research directions. +% The recently developments in video coding techniques resulted in the novel standard, referred to as High Efficiency Video Coding (HEVC). +% The high level of compression efficiency achieved by \ac{hevc} coding techniques decreases the error resilience performance under error prone conditions. +% +% In the scope of this research work the error resilience of the \ac{hevc} standard was firstly analysed. The results reveal the lack of robustness of the motion vector (MV) coding, leading to quality degradation due to error propagation. It is also shown that the temporal dependency of motion information is comparatively higher than that in the H.264/AVC standard. +% Based on this evidence, this paper proposes a method to make intelligent use of temporal motion vector (MV) candidates during the motion estimation process, in order to decrease the temporal dependency, and improve the error resiliency without penalising the rate-distortion performance. +% The simulation results show that the proposed method improves the error resilience under tested conditions, compared to the reference method that always enables temporal \ac{mv} candidates. +% +% Moreover, a method based on the prediction dependency of \acs{mv}is proposed, to select the most relevant ones for redundant coding with reduced overhead. +% The \ac{mv} dependencies are analysed in the encoder to prioritise the motion information , based on the number of subsequent dependent coding units. +% Then a subset of prioritised \acs{mv}selected with the previous is transmitted as redundancy, to reduce the use and propagation of mismatched \ac{mv} predictions in case of transmission errors or data loss. +% The simulation results show that the proposed \acs{mv} selection method can effectively identify the most relevant motion field, achieving improved error robustness with a reduced redundancy overhead. +% +% Further research will be pursed to improve the error robustness of the \ac{hevc} standard and develop novel concealment methods using combined info between the encoder and decoder side. + +The increasing diversity of services and the demand for ever higher quality multimedia at wider resolutions, which leads to the growing popularity of Ultra-HD (UHD) format, are pushing technologies forward and opening new research directions. The recent developments in video coding techniques have resulted in a novel standard called \ac{hevc} that is the next breakthrough in multimedia compression after the globally adopted MPEG-4/\ac{h264} standard. However, the significantly improved levels of compression efficiency achieved by the \ac{hevc} standard lead to an inherent reduction in the error resilience performance under error prone conditions. It should be aimed at ensuring an adequate level of protection against network and transmission induced impairments (e.g., packet losses), without sacrificing the compression efficiency thanks to the novel block level operations done by \ac{hevc}. + +My research work takes the \ac{hevc} standard to its core and builds new techniques on top of it to enhance its error resilience performance, and develop new processing techniques to improve decoder side error concealment performance. The first scope of this research work is to make an extensive analysis of various error resilience techniques developed for the \ac{hevc} standard, but also for the previous video coding standards (most notably \ac{h264}); identify their shortcomings and address those shortcomings. Up to date work has revealed that the \ac{mv} coding technique lacks robustness, which leads to quality degradation due to error propagation in lossy transmission environments. It has also been seen that the temporal dependency of motion information is comparatively higher than that in the \ac{h264} standard. Based on this evidence, a method was developed to make intelligent use of temporal \ac{mv} candidates during the motion estimation process, in order to decrease the temporal dependency, and improve the error resiliency without penalising the rate-distortion performance. The simulation results have shown that the proposed method improves the error resilience under tested conditions, compared to the reference method that always enables temporal \acs{mv} candidates. Another proposed method has exploited the prediction dependency of \glspl{mv}in order to select the most relevant ones for redundant coding with reduced overhead. Simulation results have shown that the proposed \gls{mv} selection method can effectively identify the most relevant motion fields, achieving improved error robustness with a significantly reduced redundancy overhead. + +The second scope of this research work is to develop new processing techniques for the concealment of the coding elements lost during the transmission and delivery system, using combined information between the encoder and decoder sides. Error concealment techniques mitigate the detrimental effects of network impairments on the perceptual multimedia quality and the performance these techniques determine the quality of a system directly. The third and the ultimate aim of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet a set of QoS/QoE requirements associated with current and future multimedia-based services (e.g., real-time multi-party video conferencing, Ultra-HD video on demand, etc.), such as bandwidth and delay. It is envisaged to produce a set of configuration profiles for chosen services based around the developed tools for \ac{hevc}. + diff --git a/000_Preamble/dedic.tex b/000_Preamble/dedic.tex new file mode 100755 index 0000000..08e4dc2 --- /dev/null +++ b/000_Preamble/dedic.tex @@ -0,0 +1,7 @@ +\dedication{ +Anyone who has never\\ made a mistake has never\\ tried anything new. + +\vspace{0.5cm} +\textbf{(Albert Einstein)} +} + diff --git a/000_Preamble/thanks.tex b/000_Preamble/thanks.tex new file mode 100755 index 0000000..16ae8f1 --- /dev/null +++ b/000_Preamble/thanks.tex @@ -0,0 +1,19 @@ +\chapter*{\HIGHLIGHT{\glsknowledgmentsname}} +\addcontentsline{toc}{chapter}{\glsknowledgmentsname} + +I would like to thank to everyone that help during this research work, and that make it possible to accomplish. + +I would like to express my gratitude to my advisers Prof. Pedro António Amado Assunção, Prof. Sérgio Manuel Maciel de Faria, Doctor Erhan Ekmekcioglu and Prof. Ahmet Kondoz for their support, guidance and availability that were essential for the proper conduct of this research work. + +I would like to thank Funda\c{c}\~{a}o para a Ci\^{e}ncia e a Tecnologia (FCT), which provided +the financial support for this PhD, under the PhD grant SFRH/BD/86368/2012. +I am also grateful to the Instituto de Telecomunicações and to the Institute for Digital Technologies, which provided the computer resources and laboratory facilities with excellent conditions for my research work. + +%I would like to acknowledge my research colleagues ... \HERE +%of Instituto de Telecomunicações~-~Leiria, Sylvain Marcelino, Lino Ferreira, Nelson Francisco, Luís Pinto, Luís Lucas, Anderson and Auridélia Moura, for the friendship and the wonderful work environment. I also want to express my gratitude for my graduation and non-graduation friends, for the moments outside the working environment, friendship and support, which were important along this research work. + +I would like to thank to all communities around the world that contribute for the development of many free and open-source tools used in this research work, namely the GNU/Linux, KDE software, Ubuntu and Gentoo communities. I also acknowledge the partnership of Luis Lucas in the Calyp software, and his contribution to the Remoterun scripts and the general computer management, which improved our developer capabilities and ease the experimental evaluations. + +Finally, a special thanks to my parents, António and Maria, and my sister Marta to whom I owe all that I have become. I also would like to express a special gratitude to Bianca Pires for her support and love during this journey. + + diff --git a/010_Introduction/HEVCTransmissionStack.pdf b/010_Introduction/HEVCTransmissionStack.pdf new file mode 100644 index 0000000..d26c152 Binary files /dev/null and b/010_Introduction/HEVCTransmissionStack.pdf differ diff --git a/010_Introduction/HEVCTransmissionStack.tikz b/010_Introduction/HEVCTransmissionStack.tikz new file mode 100644 index 0000000..8466a4c --- /dev/null +++ b/010_Introduction/HEVCTransmissionStack.tikz @@ -0,0 +1,66 @@ +\usetikzlibrary{positioning,calc} +\usetikzlibrary{arrows,matrix} +\usetikzlibrary{fit} +\usetikzlibrary{intersections} + +\def\scale{1} + + +\begin{tikzpicture}[ + font=\footnotesize, + auto, + node distance=1cm, + scale=\scale, + every node/.style={scale=\scale}, + template/.style={text centered}, + block/.style={template, rectangle, draw, text width=1.8cm}, + arrow/.style={template, draw, -latex}, + point/.style={template, draw, fill=black, circle,inner sep=0, minimum size=3} +] + +\tikzstyle{mainBlock} = [block] +\tikzstyle{secondBlock} = [block, minimum height=2cm] + + + +\node[secondBlock, anchor=north west] +(MPEG2) {MPEG-2 Transport Stream}; + +\node[secondBlock, text width=1.2cm, right=0 of MPEG2] +(RTP) {RTP}; + +\node[secondBlock, minimum height=1cm, right=0 of RTP.north east, anchor=north west] +(DASH) {MPEG DASH}; + +\node[secondBlock, minimum height=1cm, right=0 of DASH] +(Prog) {Progressive Download}; + +\node[secondBlock, inner sep=0, fit={(DASH) (Prog)}, minimum height=1cm, below=0 of DASH.south west, anchor=north west] +(ISOMediaFormat) {ISO Base Media \\ File Format}; + +\node[secondBlock, right=0 of Prog.north east, anchor=north west] +(MPEG2Systems) {MPEG-2 Systems (TS/PS)}; + +\node[mainBlock, inner sep=0, fit={(MPEG2) (MPEG2Systems)}, above=0 of MPEG2.north west, anchor=south west, minimum height=1cm, +label=center:HEVC / MPEG-H Part 2 / ITU-T Rec H.265] +(HEVC) {}; + +% \node [draw](chapter1) {\textit{Chapter 1}}; +% \node [draw,right=of chapter1] (paper1) {\textit{Paper 1}}; +% +% \node [draw, below=of chapter1](chapter2) {\textit{Chapter 2}}; +% \node [draw,right=of chapter2] (paper2) {\textit{Paper 2}}; +% +% \node [draw,below=of paper2] (paper3) {\textit{Paper 3}}; +% \node [draw,below=of paper3] (paper4) {\textit{Paper 4}}; +% +% \path let +% \p1=(paper3.north), \p2=(paper4.south), \n1 = {\y1-\y2-\pgflinewidth} +% in +% node [draw, below=of chapter2,minimum height=\n1](chapter3) {\textit{Chapter 3}}; + + +\end{tikzpicture} + + + diff --git a/010_Introduction/HEVCTransmissionStackOverview.pdf b/010_Introduction/HEVCTransmissionStackOverview.pdf new file mode 100644 index 0000000..6d53bc4 Binary files /dev/null and b/010_Introduction/HEVCTransmissionStackOverview.pdf differ diff --git a/010_Introduction/Introduction.tex b/010_Introduction/Introduction.tex new file mode 100755 index 0000000..0e40761 --- /dev/null +++ b/010_Introduction/Introduction.tex @@ -0,0 +1,97 @@ +\graphicspath{{010_Introduction}} + +\glsreset{hevc} +\chapter{Introduction} +\label{Chapter:Intro} + +\section{Context and motivation} + +\glsreset{h264} + +The developments in video compression and networking technologies over the recent years have made the deployment of many applications over the internet possible such as video conferencing, video streaming, video publishing, among others. The increasing diversity of services, the demands for high quality multimedia and the growing popularity of \ac{uhd} formats (\emph{e.g.}, $4k\times2k$ or $8k\times4k$ resolution and beyond) are pushing technologies forward and opening new research directions. +Moreover, the emergent video formats and increased resolutions are creating even stronger needs for coding efficiency superior to the MPEG-4/\ac{h264}~\cite{H264AVCStandard2003,H264OverviewJul2003} capabilities. +The need is even stronger when higher resolutions are needed by stereoscopic and multi-view videos, due to the increased number of camera views that need to be delivered simultaneously. Moreover, the traffic generated by mobile multimedia applications and the video-on-demand services of ever increasing popularity impose new challenges to the existing networks. +Thus, a new coding standard was developed by the ITU-T \ac{vceg} and the \ac{iso}/\ac{iec} \ac{mpeg} standardisation organisations, referred to as H.265 - \ac{hevc}~\cite{H265StandardFeb2018,HEVCOverviewDez2012}. + +\glsunset{h264} + +The \ac{hevc} is foreseen as the main video coding framework to extend current services towards higher resolution formats delivered with better compression efficiency. Similar to \ac{h264}, \ac{hevc} is a hybrid video codec based on predictive transform coding and an entropy coding step. The coding flexibility and efficiency of \ac{hevc}, results from the use of new coding tools and block structures, which enables new block partitions\cite{HEVCBlock2012}, improved prediction modes \cite{IntraCodingHEVC2012,AMPCodingHEVCJun2012} and new high-level features~\cite{HEVCHighLevelReferenceDez2012}, such as explicit reference picture management and new parameter sets. However, it is known that higher coding efficiency leads to lower robustness to transmission errors and data loss. Summarising, the new coding tools in \ac{hevc} bring certain compression efficiency~\cite{HEVCComparisonDec2012}, but also has some other disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and inherent reduction in error robustness. + +\begin{figure}[tb] +\centering +%\includegraphics[width=0.5\columnwidth]{./010_Introduction/HEVCTransmissionStack} +\includegraphics[width=0.7\columnwidth]{./010_Introduction/HEVCTransmissionStackOverview} +\caption{System layer stack compatible with \ac{hevc} video transmission~\cite{HEVCLayerIntegrationDez2012}.} +\label{fig:HEVCStack} +\end{figure} + +The error robustness characteristics of \ac{hevc} were analysed in \cite{StudyHEVCErrorsDez2012} confirming that, in general, the \ac{hevc} presents poor error resilience performance, despite its superior coding efficiency in comparison with previous standards. +Thus, when \ac{hevc} streams incur in transmission losses, this results in significant degradation of both objective and subjective quality~\cite{QoEHEVCMay2014,SubjectiveEvalSep2013}. +This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes of \ac{hevc}, which are selected according to a \mbox{rate-distortion~(R-D)} optimisation criteria, always assuming \mbox{error-free} transmission~\cite{ROPE2007}. A possible approach to encode robust video streams is to increase the error resilience of \ac{hevc} by including \mbox{error-aware} mechanisms in the coding process. As a result, the effect of error propagation due to prediction mismatch at the decoding side, will be limited. + +The widespread use of \ac{hevc} in broadcast and multimedia delivery services, will be based on transport technologies such as RTP~\cite{HEVCStreamMay2012}, \mbox{MPEG-2 TS}~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016}. +Recent developments also enable combinations of different technologies, e.g, \mbox{HTTP-based} streaming and \mbox{MPEG-2 TS}, to increase the flexibility of over-the-top (OTT) video services\cite{Broadcast2017}. For content delivery over heterogeneous networks, the MPEG Media Transport (MMT)~\cite{ATSCMMT2016} is another possible technology to be used with the \ac{hevc} standard. Besides the transport technologies, +the overall error robustness of video transmission may also take advantage of the increasing flexibility of multimedia systems to cope with heterogeneous networks, by using auxiliary streams to aid the video decoder to increase the quality of recovered lost slices~\cite{ErrorResilienceHeterogenousNetworks2016,HeterogeneousStreaming2016}. Moreover, error resilience techniques may also take advantage of the multi-stream based approach, where each stream allows different levels of robustness to errors~\mbox{\cite{HEVCHTTPStreamAug2016,PerfMediaTransport2017}}. + + +%Despite the different characteristics of the existing services and applications, the users' \ac{qoe} consistently depends on how the compressed streams are adapted to the diverse networking technologies, as well as on how the transmission errors and/or packet losses are handled at the application level ~\cite{HEVCStreamMay2012,HEVCLayerIntegrationDez2012,StudyHEVCErrorsDez2012}. As mentioned before, the \ac{hevc} standard is expected to replace the \ac{h264} in future video services. + + +%The system interface of \ac{hevc} is an important component +%in the media access chain, and a prerequisite for its successful +%deployment in various consumer electronics applications. We +%consider four broad categories of systems: broadcast delivery +%over digital television and Internet Protocol Television (IPTV) +%channels including video-on-demand (VoD), streaming over +%the Internet using the Hypertext Transport Protocol (HTTP), +%real-time communications such as video conferencing, and +%store-forward delivery based on a file or a physical memory +%delivered to the end user (such as: camcorded videos or Blu- +%ray Disks). To this end, different system technologies have +%been standardized. These technologies can be summarized as +%follows. + + + + +In summary, it is an emergent research problem to increase the adaptability of \ac{hevc} compressed streams for diverse network conditions and its robustness against transmission errors, in order to provide acceptable levels of \ac{qoe} to the end users at all times. The \ac{hevc} reference codec is still under development by multiple proponents from around the world, and there are open questions in terms of error resilience and techniques to increase the concealment performance on the decoder side. Although the \ac{ec} methods have been developed during the past years for \ac{h264}~\cite{2DRecursiveRefinement2010} and recently for \ac{hevc}~\cite{HEVCMCBlockAug2013}, a combined and optimised approach between encoder and decoder is yet to be developed, which would improve the users’ \ac{qoe} by exploiting shared information between the encoder and the decoder sides. + +\section{\HIGHLIGHT{Main objectives and contributions}} + +This research plan addresses robust error resilience and efficient error concealment for the forthcoming multimedia services, based on the recent \ac{hevc} standard. Thus, this work focuses on novel techniques to increase delivered quality of \ac{hevc} bitstreams in the following respects: +\begin{enumerate} +\item \textbf{evaluate the error robustness of \ac{hevc}:} + +\item \textbf{source coding level:} At the encoder side, novel techniques to extend the existing error robustness features of the \ac{hevc} standard will be investigated in order to increase the received video quality over error-prone networks. The new coding features introduced in the standard increase the compression ratio, resulting in more complex dependencies that lead to increased error propagation rate. + +\item \textbf{error concealment of the received video:} At the decoder side, the error concealment techniques improve the perceived video quality by reconstructing the missing information. The paradigm of error concealment should be evaluated at the encoder side, as part of the encoding process, in order to evaluate the concealment performance and share used information with the decoder to improve the overall performance of the transmission system. + +\item \textbf{jointly error resilience and concealment:} the ultimate goal of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet higher quality demands for different video contents characteristics and network conditions. +\end{enumerate} + +%Finally, the ultimate goal of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet a set of \ac{qos}/\ac{qoe} requirements. +%Since the ultimate quality judgement is made by end users, subjective tests will constitute a part of the research work in order to validate the objective gains reported as a result of the deployment of various proposed and implemented methods. Given the importance of the \ac{hevc} standard in the near future, the simulation scenarios will be defined based on practical service requirements, such as, bit rate allocation, transmission delay and required QoS. + +\section{\HIGHLIGHT{Outline of this thesis}} + +This thesis is organised as follows. The current chapter introduces the main research topics of this thesis, describing the motivations and objectives to achieve with this work, as well as the original contributions of this thesis. + +The first two chapters review the \mbox{state-of-the-art} concepts related with this research work. Chapter~\ref{Chapter:HEVCReview} presents an overview of the \ac{hevc} standard, with some details in the concepts explored in this work, e.g., high-level features and temporal predictions. Chapter~\ref{Chapter:ErrorResilienceReview} reviews some of the work regarding robust video transmission proposed over the last years, since they provide relevant knowledge to the main objectives of this work. + +Chapter~\ref{Chapter:HEVCErrorResilience} firstly presents an evaluation study of the performance of the \ac{hevc} standard under errors for various coding configuration. The study aims at finding the relevant tools introduced in the standard that highly affects the error robustness. Secondly, a new method is proposed to decrease the vulnerability of \ac{hevc} temporal prediction and decrease error propagation, leading to higher video quality in error-prone conditions. + +Chapter~\ref{Chapter:ErrorResilienceTechniques} proposes a novel error resilience approach in the context of the \ac{hevc} standard. The motivation for this works is related with the poor error robustness performance show in Chapter~\ref{Chapter:HEVCErrorResilience}. Thus, a new method to distribute the use of reference frames was proposed to reduce the error propagation and improve the \ac{ec} performance. Moreover, to reduce the effect of mismatch motion predictions (also shown in Chapter~\ref{Chapter:HEVCErrorResilience}) a method based on selective motion vector redundancies is proposed. This method is able to improve the decoded video quality using a low amount of overhead. + +Chapter~\ref{Chapter:ConcealmentAwareTechniques} addresses the problem of efficient \ac{ec} in the \ac{hevc}. This work was motivated by the requirements for more accurate \ac{ec} algorithms which are able to reduce the error propagation. Different \ac{ec} algorithms were taken into consideration in order to develop a concealment-aware coding scheme. The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled to the decoder. Such optimal EC\ac{ec} modes are found by simulating transmission losses followed by a \ac{rd} optimisation of the signalling rate and \ac{ec} distortion cost. +A generalised saliency-weighted distortion is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. When data loss occurs the decoder uses the signalling information is used at the decoder, in case of data loss, to improve the reconstruction quality. The simulation results show that the proposed method achieves consistent quality gains in comparison with other reference methods and previous works. + + +Chapter~\ref{Chapter:EndToEndRobustSystem} ... \HERE + +Chapter~\ref{Chapter:Conclusions} concludes this dissertation and presents some suggestions for future work. + +Appendix~\ref{Appen:Sequences} illustrates the original test signals used in simulations throughout the thesis. This includes a partial set of the \ac{hevc} test material and other external test material. +Appendix~\ref{Appen:Papers} presents a summary of the published papers, describing the contributions of the research work of this thesis. + + + diff --git a/021_VideoCoding/Figures/GeneralVideoSystem.tikz b/021_VideoCoding/Figures/GeneralVideoSystem.tikz new file mode 100644 index 0000000..d5095de --- /dev/null +++ b/021_VideoCoding/Figures/GeneralVideoSystem.tikz @@ -0,0 +1,30 @@ +\usetikzlibrary{arrows,matrix} +% \usepackage{amssymb, amsmath, mathptmx} + +\begin{tikzpicture}[font=\scriptsize,node distance=1.5cm,scale=0.9] + + +\tikzstyle{simpleBlock}=[centered, text centered] +\tikzstyle{block}=[simpleBlock, draw, minimum size=1cm, minimum width=2.4cm, thick] +\tikzstyle{arrow}=[draw,-latex,thick] +\tikzstyle{point} = [draw, fill=black, circle,inner sep=1pt] + + + +\node[block] (PreProc) {Pre-processing}; +\node[block,right=of PreProc] (Encoding) {Encoding}; +\node[block,right=of Encoding, yshift=-1cm, text width=2cm] (Transmission) {Transmission or Storage}; +\node[block,left=of Transmission, yshift=-1cm] (Decoding) {Decoding}; +\node[block,left=of Decoding] (PostProc) {Post-processing}; + +\node[simpleBlock,left=of PreProc, text width=1.5cm] (Input) {\textbf{Input}}; +\node[simpleBlock,left=of PostProc, text width=1.5cm] (Output) {\textbf{Output}}; + +\path[arrow] (Input) -- (PreProc); +\path[arrow] (PreProc) -- (Encoding); +\path[arrow] (Encoding) -| (Transmission); +\path[arrow] (Transmission) |- (Decoding); +\path[arrow] (Decoding) -- (PostProc); +\path[arrow] (PostProc) -- (Output); + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/AssymetricBlockPartition.pdf b/021_VideoCoding/Figures/HEVC/AssymetricBlockPartition.pdf new file mode 100644 index 0000000..181b36e Binary files /dev/null and b/021_VideoCoding/Figures/HEVC/AssymetricBlockPartition.pdf differ diff --git a/021_VideoCoding/Figures/HEVC/AssymetricBlockPartition.tikz b/021_VideoCoding/Figures/HEVC/AssymetricBlockPartition.tikz new file mode 100644 index 0000000..0a4b513 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/AssymetricBlockPartition.tikz @@ -0,0 +1,81 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{0.7} + +\begin{tikzpicture}[scale=\scale,every node/.style={scale=\scale},on grid ] + + \definecolor{LCUFill}{rgb}{0,0.5,1} + \definecolor{DrawColor}{rgb}{0,0,0.5} + \definecolor{LCUFill}{rgb}{1,1,1} + \definecolor{DrawColor}{rgb}{0,0,0} + + \tikzstyle{TextStyle}=[scale=1.5, text centered] + + \tikzstyle{LCU}=[draw=DrawColor, fill=LCUFill,thick] + \tikzstyle{CU}=[draw=DrawColor,thick] + \tikzstyle{PU}=[draw=DrawColor, densely dashed] + +% \newcommand{\gridsize}{70} +% \draw[help lines] (-\gridsize,-\gridsize) grid ++(\gridsize*2,\gridsize*2); + + + \begin{scope}[xshift=0] + \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 + \draw[CU] (0,0) rectangle ++(1,4); % 64x64 + \draw[CU] (1,0) rectangle ++(3,4); % 64x64 + \node[TextStyle] at (2.5,2) {$\frac{3N}{4}\times N$}; + \end{scope} + + \begin{scope}[xshift=5cm] + \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 + \draw[CU] (0,0) rectangle ++(3,4); % 64x64 + \draw[CU] (3,0) rectangle ++(1,4); % 64x64 + \node[TextStyle] at (1.5,2) {$\frac{3N}{4}\times N$}; + \end{scope} + + \begin{scope}[xshift=10cm] + \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 + \draw[CU] (0,0) rectangle ++(4,3); % 64x64 + \draw[CU] (0,3) rectangle ++(4,1); % 64x64 + \node[TextStyle] at (2,1.5) {$N\times\frac{3N}{4}$}; + \end{scope} + + \begin{scope}[xshift=15cm] + \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 + \draw[CU] (0,0) rectangle ++(4,1); % 64x64 + \draw[CU] (0,1) rectangle ++(4,3); % 64x64 + \node[TextStyle] at (2,2.5) {$N\times\frac{3N}{4}$}; + \end{scope} + +% \draw[CU] (0,0) rectangle ++(16,16); % 64x64 +% +% \node[TextStyle] at (4,12) {$32\times32$}; +% \draw[CU] (0,0) rectangle ++(8,8); % 32x32 +% \draw[CU] (8,8) rectangle ++(8,8); % 32x32 +% +% \node[TextStyle] at (10,14) {$16\times16$}; +% \draw[CU] (0,0) rectangle ++(4,4); % 16x16 +% \draw[CU] (4,4) rectangle ++(4,4); % 16x16 +% +% \draw[CU] (8,8) rectangle ++(4,4); % 16x16 +% \draw[CU] (12,12) rectangle ++(4,4); % 16x16 +% +% \node[TextStyle] at (13,11) {$8\times8$}; +% \draw[CU] (12,8) rectangle ++(2,2); % 8x8 +% \draw[CU] (14,10) rectangle ++(2,2); % 8x8 + + + + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/BitstreamSyntax/.directory b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/.directory new file mode 100644 index 0000000..28b696c --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/.directory @@ -0,0 +1,4 @@ +[Dolphin] +Timestamp=2013,6,23,21,48,12 +Version=3 +ViewMode=1 diff --git a/021_VideoCoding/Figures/HEVC/BitstreamSyntax/BitstreamSyntax.pdf b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/BitstreamSyntax.pdf new file mode 100644 index 0000000..ac7040e Binary files /dev/null and b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/BitstreamSyntax.pdf differ diff --git a/021_VideoCoding/Figures/HEVC/BitstreamSyntax/BitstreamSyntax.tikz b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/BitstreamSyntax.tikz new file mode 100644 index 0000000..6d3195b --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/BitstreamSyntax.tikz @@ -0,0 +1,56 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{1} + +\begin{tikzpicture}[yscale=0.5, scale=\scale,every node/.style={scale=\scale},on grid ] + +\definecolor{LCUFill}{rgb}{0,0.5,1} +\definecolor{DrawColor}{rgb}{0,0,0.5} +\definecolor{LCUFill}{rgb}{1,1,1} +\definecolor{DrawColor}{rgb}{0,0,0} + +\tikzstyle{TextStyle}=[scale=.7, text centered] +\tikzstyle{Square}=[draw=DrawColor,thick] +\tikzstyle{Byte}=[draw=DrawColor,thick, dashed] + + +\begin{scope}[xshift=0, yshift=-5cm] + \begin{scope}[xshift=0, yshift=0cm] + \node[TextStyle] at (1,.5) {Slice header}; + \draw[Square] (0,0) rectangle ++(2,1cm); + \draw[Square] (2,0) rectangle ++(7,1cm); + + \foreach \x in {1,...,3} + { + \draw[Square,thin] (2*\x+.1*\x,.1) rectangle ++(2,0.8cm); + \node[TextStyle] at (1+2*\x+.1*\x,.1+0.8/2) {CTU}; + } + \node[TextStyle] at (1+2*3+.1*3+1.35,.1+0.8/2) {...}; + + \coordinate (EdgeL) at (2+.1,.1); + \coordinate (EdgeR) at (4+.1,.1); + \end{scope} + \begin{scope}[xshift=.2cm, yshift=-2cm, scale=1] + \draw[Square] (0,0) rectangle ++(8,1cm); + + \foreach \x in {1,7,13} + \draw[draw=black] (\x*.5,0) -- ++(0,.5); + + \draw[Square,dashed] (EdgeL) -- (0,1); + \draw[Square,dashed] (EdgeR) -- (8,1); + + \end{scope} +\end{scope} + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/BitstreamSyntax/CodingUnits.pdf b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/CodingUnits.pdf new file mode 100644 index 0000000..0208788 Binary files /dev/null and b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/CodingUnits.pdf differ diff --git a/021_VideoCoding/Figures/HEVC/BitstreamSyntax/CodingUnits.tikz b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/CodingUnits.tikz new file mode 100644 index 0000000..8348a1f --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/CodingUnits.tikz @@ -0,0 +1,129 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{1} + +\begin{tikzpicture}[yscale=0.5, scale=\scale,every node/.style={scale=\scale},on grid ] + +\definecolor{LCUFill}{rgb}{0,0.5,1} +\definecolor{DrawColor}{rgb}{0,0,0.5} +\definecolor{LCUFill}{rgb}{1,1,1} +\definecolor{DrawColor}{rgb}{0,0,0} + +\tikzstyle{TextStyle}=[scale=.5, text centered] +\tikzstyle{Square}=[draw=DrawColor,thick] +\tikzstyle{Connection}=[Square,dashed] +\tikzstyle{Byte}=[draw=DrawColor,thick, dashed] + +\begin{scope}[xshift=0, yshift=-5cm] + + \begin{scope}[xshift=0, yshift=0cm] + \node[TextStyle] at (1,.5) {Slice header}; + \draw[Square] (0,0) rectangle ++(2,1cm); + \draw[Square] (2,0) rectangle ++(7,1cm); + + \node[TextStyle] at (2+.25,0.5) {...}; + \foreach \x in {0,...,2} + { + \draw[Square,thin] (2.5+2*\x,0) rectangle ++(2,1cm); + \node[TextStyle] at (2.5+2*\x+1,.5) {CTU$_{n+\x}$}; + } + \node[TextStyle] at (2.5+2*3+.25,.1+0.8/2) {...}; + + \coordinate (EdgeL) at (2+.5,0); + \coordinate (EdgeR) at (2+.5+2,0); + \end{scope} + + \begin{scope}[xshift=.25*1cm, yshift=-2.5*1cm, scale=1] + \draw[Connection] (EdgeL) -- (0,1); + \draw[Connection] (EdgeR) -- (8,1); + + \draw[Square] (0,0) rectangle ++(8,1cm); + \draw[Square] (0,0) rectangle ++(2,1cm); +% \draw[Square] (1.5,0) rectangle ++(2.5,1cm); + +% \node[TextStyle] at (.75,.5) {CTU Position}; + \node[TextStyle] at (2/2,.5) {SAO (optional)}; + \node[TextStyle] at (6/2+2,.5) {Coding quatree (CT)}; + + \coordinate (EdgeL) at (2,0); + \coordinate (EdgeR) at (8,0); + \end{scope} + + \begin{scope}[xshift=.25*2cm, yshift=-2.5*2cm, scale=1] + \coordinate (BraceT) at (.35,1.2); + + \draw[Connection] (EdgeL) -- (0,1); + \draw[Connection] (EdgeR) -- (8,1); + + \draw[Square] (0,0) rectangle ++(8,1cm); + + \draw[Square] (0,0) rectangle ++(1.5,1cm); + \node[TextStyle,scale=.95] at (1.5/2,.5) {split\_cu\_flag==1}; + + \draw[Square] (1.5,0) rectangle ++(8/4-1.5/4,1cm); + \draw[Square] (1.5+8/4-1.5/4,0) rectangle ++(8/4-1.5/4,1cm); + \draw[Square] (1.5+2*8/4-2*1.5/4,0) rectangle ++(8/4-1.5/4,1cm); + + \node[TextStyle] at (1.5+8/8-1.5/8,.5) {Coding quatree}; + \node[TextStyle] at (1.5+8/4-1.5/4+8/8-1.5/8,.5) {Coding quatree}; + \node[TextStyle] at (1.5+2*8/4-2*1.5/4+8/8-1.5/8,.5) {Coding quatree}; + \node[TextStyle] at (1.5+3*8/4-3*1.5/4+8/8-1.5/8,.5) {Coding quatree}; + + \end{scope} + + \begin{scope}[xshift=.25*2cm, yshift=-2.5*2.7cm, scale=1] + \coordinate (BraceB) at (.35,-.2); + + \draw[Square] (0,0) rectangle ++(8,1cm); + + \draw[Square] (0,0) rectangle ++(1.5,1cm); + \node[TextStyle,scale=.95] at (1.5/2,.5) {split\_cu\_flag==0}; + + \node[TextStyle] at (1.5+8/2-1.5/2,.5) {Coding unit}; + + \coordinate (EdgeL) at (1.5,0); + \coordinate (EdgeR) at (8,0); + \end{scope} + + \draw [thick, decoration={brace, mirror, raise=0.5cm }, decorate] (BraceT) -- (BraceB) ; +\end{scope} + +\begin{scope}[xshift=.8cm, yshift=-14cm] + + \begin{scope}[xshift=.5*0cm, yshift=-2.5*0, scale=1] + + \draw[Connection] (EdgeL) -- (0,1); + \draw[Connection] (EdgeR) -- (8,1); + + \coordinate (BraceB) at (.35,-.2); + + \draw[Square] (0,0) rectangle ++(8,1cm); + \node[TextStyle] at (.25,.5) {...}; + + \draw[Square] (0.5,0) rectangle ++(1.8,1cm); + \node[TextStyle] at (1.8/2+0.5,.5) {Prediction unit$_{1}$}; + \node[TextStyle] at (.5+1.8+.3,.5) {...}; + \draw[Square] (0.5+1.8+0.6,0) rectangle ++(1.8,1cm); + \node[TextStyle] at (1.8/2+0.5+1.8+0.6,.5) {Prediction unit$_{n}$}; + + \node[TextStyle] at (.5+1.8*2+.3*3,.5) {...}; + \draw[Square] (0.5+1.8*2+0.6*2,0) rectangle ++(8-0.5-1.8*2-0.6*2,1cm); + \node[TextStyle] at (1.8/2+0.5+1.8+0.6+2.8,.5) {Transform tree}; + + \end{scope} + +\end{scope} + + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/BitstreamSyntax/NALUnit.pdf b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/NALUnit.pdf new file mode 100644 index 0000000..0e741ff Binary files /dev/null and b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/NALUnit.pdf differ diff --git a/021_VideoCoding/Figures/HEVC/BitstreamSyntax/NALUnit.tikz b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/NALUnit.tikz new file mode 100644 index 0000000..09d1466 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/NALUnit.tikz @@ -0,0 +1,56 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{1} + +\begin{tikzpicture}[yscale=0.5, scale=\scale,every node/.style={scale=\scale},on grid ] + +\definecolor{LCUFill}{rgb}{0,0.5,1} +\definecolor{DrawColor}{rgb}{0,0,0.5} +\definecolor{LCUFill}{rgb}{1,1,1} +\definecolor{DrawColor}{rgb}{0,0,0} + +\tikzstyle{TextStyle}=[scale=.5, text centered] +\tikzstyle{Square}=[draw=DrawColor,thick] +\tikzstyle{Byte}=[draw=DrawColor,thick, dashed] + +\begin{scope}[xshift=0, yshift=0] + \begin{scope}[xshift=0, yshift=0] + \draw[Square] (0,0) rectangle ++(2,1cm); + \draw[Square] (2,0) rectangle ++(7,1cm); + \node[TextStyle] at (2/2,.5) {NAL header}; + \node[TextStyle] at (2+7/2,.5) {Slice data}; + \coordinate (EdgeL) at (0,0); + \coordinate (EdgeR) at (2,0); + \end{scope} + + \begin{scope}[xshift=.2cm, yshift=-2cm, scale=1] + \draw[Square] (0,0) rectangle ++(8,1cm); + + \foreach \x in {1,7,13} + \draw[draw=black] (\x*.5,0) -- ++(0,.5); + + \draw[Square,dashed] (EdgeL) -- (0,1); + \draw[Square,dashed] (EdgeR) -- (8,1); + + \node[TextStyle] at (.25,.5) {F}; + \node[TextStyle] at (6*.25+0.5,.5) {NAL type (6)}; + \node[TextStyle] at (7*.25+3.25,.5) {Layer ID (6)}; + \node[TextStyle] at (3*.25+6.5,.5) {TIDP (3)}; + + \node[TextStyle] at (8/2,-0.5) {2 bytes header}; + + \end{scope} +\end{scope} + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/BitstreamSyntax/Slice.pdf b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/Slice.pdf new file mode 100644 index 0000000..3b0e9f2 Binary files /dev/null and b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/Slice.pdf differ diff --git a/021_VideoCoding/Figures/HEVC/BitstreamSyntax/Slice.tikz b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/Slice.tikz new file mode 100644 index 0000000..134c7ac --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/BitstreamSyntax/Slice.tikz @@ -0,0 +1,52 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{1} + +\begin{tikzpicture}[yscale=0.5, scale=\scale,every node/.style={scale=\scale},on grid ] + +\definecolor{LCUFill}{rgb}{0,0.5,1} +\definecolor{DrawColor}{rgb}{0,0,0.5} +\definecolor{LCUFill}{rgb}{1,1,1} +\definecolor{DrawColor}{rgb}{0,0,0} + +\tikzstyle{TextStyle}=[scale=.5, text centered] +\tikzstyle{Square}=[draw=DrawColor,thick] +\tikzstyle{Connection}=[Square,dashed] +\tikzstyle{Byte}=[draw=DrawColor,thick, dashed] + +\begin{scope}[xshift=0, yshift=-5cm] + + \begin{scope}[xshift=0, yshift=0cm] + \node[TextStyle] at (1,.5) {Slice header}; + \draw[Square] (0,0) rectangle ++(2,1cm); + \draw[Square] (2,0) rectangle ++(7,1cm); + + \node[TextStyle] at (2+.25,0.5) {...}; + \foreach \x in {0,...,2} + { + \draw[Square,thin] (2.5+2*\x,0) rectangle ++(2,1cm); + \node[TextStyle] at (2.5+2*\x+1,.5) {CTU$_{n+\x}$}; + } + \node[TextStyle] at (2.5+2*3+.25,.1+0.8/2) {...}; + + \coordinate (EdgeL) at (2+.5,0); + \coordinate (EdgeR) at (2+.5+2,0); + \end{scope} + + +\end{scope} + + + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/BlockPartitions.pdf b/021_VideoCoding/Figures/HEVC/BlockPartitions.pdf new file mode 100644 index 0000000..3ffeef7 Binary files /dev/null and b/021_VideoCoding/Figures/HEVC/BlockPartitions.pdf differ diff --git a/021_VideoCoding/Figures/HEVC/BlockPartitions.tikz b/021_VideoCoding/Figures/HEVC/BlockPartitions.tikz new file mode 100644 index 0000000..661a7cf --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/BlockPartitions.tikz @@ -0,0 +1,71 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{0.4} + +\begin{tikzpicture}[scale=\scale,every node/.style={scale=\scale},on grid ] + + \definecolor{LCUFill}{rgb}{0,0.5,1} + \definecolor{DrawColor}{rgb}{0,0,0.5} + \definecolor{LCUFill}{rgb}{1,1,1} + \definecolor{DrawColor}{rgb}{0,0,0} + + \tikzstyle{TextStyle}=[scale=1.8, text centered] + + \tikzstyle{CTU}=[draw=DrawColor, fill=LCUFill, very thick] + \tikzstyle{CU}=[draw=DrawColor, thin] + \tikzstyle{PU}=[draw=DrawColor, thin, densely dashed] + + \node[TextStyle] at (8,16.5) {$64\times64$ - CTU}; + \draw[CTU] (0,0) rectangle ++(16,16); % 64x64 + + %% 32x32 + \draw[CU] (0,0) rectangle ++(8,8); % 32x32 + \draw[CU] (8,8) rectangle ++(8,8); % 32x32 + + %% Q1 + \draw[CU] (0,8) rectangle ++(4,4); % 16x16 + \draw[CU] (4,12) rectangle ++(4,4); % 16x16 + \draw[CU] (4,12) rectangle ++(2,2); % 8x8 + \draw[CU] (6,14) rectangle ++(2,2); % 8x8 + + %% Q3 + \draw[CU] (0,0) rectangle ++(4,4); % 16x16 + \draw[CU] (4,4) rectangle ++(4,4); % 16x16 + \draw[CU] (4,4) rectangle ++(2,2); % 8x8 + \draw[CU] (6,6) rectangle ++(2,2); % 8x8 + + \draw[PU] (4,4) rectangle ++(1,1); % 4x4 + \draw[PU] (5,5) rectangle ++(1,1); % 4x4 + + \node[TextStyle] at (12,12) {$32\times32$}; + \node[TextStyle] at (2,14) {$16\times16$}; + \node[TextStyle] at (5,13) {$8\times8$}; + + \node[TextStyle] (CULabel) at (9.5,6) {CU}; + \draw[black, -latex] (CULabel) to (7,7); + + \node[TextStyle] (PULabel) at (2.5,5.5) {PU}; + \draw[black, -latex] (PULabel) to (4.5,4.5); + + +% %H.264 +% \node[TextStyle] (PULabel) at (10.5,4.5) {\textcolor{red}{MB}}; +% \draw[black, -latex] (PULabel) to (13,3); +% \draw[CU, draw=red] (12,0) rectangle ++(4,4); % 16x16 +% \draw[CU, draw=red] (14,0) rectangle ++(2,2); % 8x8 +% \draw[CU, draw=red] (12,2) rectangle ++(2,2); % 8x8 +% \draw[CU, draw=red] (15,0) rectangle ++(1,1); % 4x4 +% \draw[CU, draw=red] (14,1) rectangle ++(1,1); % 4x4 + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/BlockPartitionsTree.tikz b/021_VideoCoding/Figures/HEVC/BlockPartitionsTree.tikz new file mode 100644 index 0000000..47fa440 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/BlockPartitionsTree.tikz @@ -0,0 +1,39 @@ +\tikzset{ + treenode/.style = {align=center, inner sep=0pt, text centered, font=\sffamily}, + node_split/.style = {treenode, circle,inner sep=1pt, white, font=\sffamily\bfseries, draw=black, fill=black}, + node_cu/.style = {treenode, circle, draw, thick, font=\sffamily\bfseries\scriptsize, text width=.95em} +} + +\begin{tikzpicture}[->,>=stealth', + level 1/.style={level distance = 1.4cm, sibling distance=1.65cm}, + level 2/.style={level distance = 1.4cm, sibling distance=.85cm}, + level 3/.style={level distance = 1.4cm, sibling distance=.55cm}, +] + +\node [node_split] {} +child{ node [node_split] {} + child{ node [node_cu] {1} } + child{ node [node_split] {} + child{ node [node_cu] {2} } + child{ node [node_cu] {3} } + child{ node [node_cu] {4} } + child{ node [node_cu] {5} } + } + child{ node [node_cu] {5} } + child{ node [node_cu] {6} } +} +child{ node [node_cu] {7} } +child{ node [node_split] {} + child{ node [node_cu] {8} } + child{ node [node_split] {} + child{ node [node_cu] {9} } + child{ node [node_cu] {10} } + child{ node [node_cu] {11} } + child{ node [node_cu] {12} } + } + child{ node [node_cu] {13} } + child{ node [node_cu] {14} } +} +child{ node [node_cu] {15} } +; +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/BlockPartitionsWithTree.tikz b/021_VideoCoding/Figures/HEVC/BlockPartitionsWithTree.tikz new file mode 100644 index 0000000..6dd35e6 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/BlockPartitionsWithTree.tikz @@ -0,0 +1,109 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{1} + +\begin{tikzpicture}[scale=\scale,on grid, ->,>=stealth'] + +\begin{scope}[scale=0.4] + + \definecolor{LCUFill}{rgb}{0,0.5,1} + \definecolor{DrawColor}{rgb}{0,0,0.5} + \definecolor{LCUFill}{rgb}{1,1,1} + \definecolor{DrawColor}{rgb}{0,0,0} + + \tikzstyle{TextStyle}=[scale=.75,text centered] + + \tikzstyle{CTU}=[draw=DrawColor, fill=LCUFill, very thick] + \tikzstyle{CU}=[draw=DrawColor, thin] + \tikzstyle{PU}=[draw=DrawColor, thin, densely dashed] + + \node[TextStyle] at (8,16.5) {$64\times64$ - CTU}; + \draw[CTU] (0,0) rectangle ++(16,16); % 64x64 + + %% 32x32 + \draw[CU] (0,0) rectangle ++(8,8); % 32x32 + \draw[CU] (8,8) rectangle ++(8,8); % 32x32 + + %% Q1 + \draw[CU] (0,8) rectangle ++(4,4); % 16x16 + \draw[CU] (4,12) rectangle ++(4,4); % 16x16 + \draw[CU] (4,12) rectangle ++(2,2); % 8x8 + \draw[CU] (6,14) rectangle ++(2,2); % 8x8 + + %% Q3 + \draw[CU] (0,0) rectangle ++(4,4); % 16x16 + \draw[CU] (4,4) rectangle ++(4,4); % 16x16 + \draw[CU] (4,4) rectangle ++(2,2); % 8x8 + \draw[CU] (6,6) rectangle ++(2,2); % 8x8 + + \draw[PU] (4,4) rectangle ++(1,1); % 4x4 + \draw[PU] (5,5) rectangle ++(1,1); % 4x4 + + \node[TextStyle] at (12,12) {$32\times32$}; + \node[TextStyle] at (2,14) {$16\times16$}; + \node[TextStyle] at (5,13) {$8\times8$}; + + \node[TextStyle] (CULabel) at (9.5,6) {CU}; + \draw[black, -latex] (CULabel) to (7,7); + + \node[TextStyle] (PULabel) at (2.5,5.5) {PU}; + \draw[black, -latex] (PULabel) to (4.5,4.5); +\end{scope} + +% %H.264 +% \node[TextStyle] (PULabel) at (10.5,4.5) {\textcolor{red}{MB}}; +% \draw[black, -latex] (PULabel) to (13,3); +% \draw[CU, draw=red] (12,0) rectangle ++(4,4); % 16x16 +% \draw[CU, draw=red] (14,0) rectangle ++(2,2); % 8x8 +% \draw[CU, draw=red] (12,2) rectangle ++(2,2); % 8x8 +% \draw[CU, draw=red] (15,0) rectangle ++(1,1); % 4x4 +% \draw[CU, draw=red] (14,1) rectangle ++(1,1); % 4x4 + +\tikzset{ + level 1/.style={level distance = 1.6cm, sibling distance=1.6cm}, + level 2/.style={level distance = 1.6cm, sibling distance=.7cm}, + level 3/.style={level distance = 1.6cm, sibling distance=.55cm}, + treenode/.style = {align=center, inner sep=0pt, text centered, font=\sffamily}, + node_split/.style = {treenode, circle,inner sep=1pt, white, font=\sffamily\bfseries, draw=black, fill=black}, + node_cu/.style = {treenode, circle, draw, thick, font=\sffamily\bfseries\scriptsize, text width=.95em} +} + +\node [node_split,scale=1/\scale] at (11cm,15*.4cm) {} +child{ node [node_split] {} + child{ node [node_cu] {1} } + child{ node [node_split] {} + child{ node [node_cu] {2} } + child{ node [node_cu] {3} } + child{ node [node_cu] {4} } + child{ node [node_cu] {5} } + } + child{ node [node_cu] {5} } + child{ node [node_cu] {6} } +} +child{ node [node_cu] {7} } +child{ node [node_split] {} + child{ node [node_cu] {8} } + child{ node [node_split] {} + child{ node [node_cu] {9} } + child{ node [node_cu] {10} } + child{ node [node_cu] {11} } + child{ node [node_cu] {12} } + } + child{ node [node_cu] {13} } + child{ node [node_cu] {14} } +} +child{ node [node_cu] {15} } +; + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/HEVCBlockDiagram.tikz b/021_VideoCoding/Figures/HEVC/HEVCBlockDiagram.tikz new file mode 100644 index 0000000..a0df5a7 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/HEVCBlockDiagram.tikz @@ -0,0 +1,91 @@ +\usetikzlibrary{arrows,matrix} +% \usepackage{amssymb, amsmath, mathptmx} + +\begin{tikzpicture}[font=\scriptsize,node distance=1cm,scale=1.2] + +\tikzstyle{sumBlock}=[draw,circle, inner sep=0cm, minimum size=0.1cm] +\tikzstyle{processBlock}=[draw, fill=\TikZColorFill, inner sep=0.1cm, minimum size=0.7cm,text centered] +\tikzstyle{proposedBlock}=[processBlock, fill=gray!30] +\tikzstyle{blockArrow}=[draw,-latex] +\tikzstyle{point} = [draw, fill=black, circle,inner sep=1pt] + +\coordinate (sumPoint) at (0,0); +\node[sumBlock] (ResidueDiff) at (sumPoint) {$+$}; + + +% \begin{scope}[xshift=-2cm, yshift=3.2cm, scale=1.5] +% \def\spacing{.2} +% \node at ((.5+3*\spacing*.5,-.5) (Input) {}; +% \foreach \i in {0,...,3} +% \draw[thick,fill=white] (\spacing*\i,-.15*\i) rectangle ++(1,1); +% % \node[text width=.7cm, text centered] at (1.6,1.1) {Video Stream}; +% % \node[text width=.7cm, text centered] at (1.1,.1) {Input video}; +% \end{scope} + + +\node[left of=ResidueDiff,text width=1cm,node distance=2.5cm] (Input) {\textbf{Input video}}; +\path[blockArrow] (Input) -- (ResidueDiff) node [point, midway] (p0) {}; + +\node[processBlock,right=of ResidueDiff] (Transf) {$T$}; +\node[processBlock,right=of Transf] (Quant) {$Q$}; +\node[processBlock,right=of Quant] (VLC) {CABAC}; + +\node[right=of VLC, text centered,text width=1.5cm] (Output) {\textbf{HEVC bitstream}}; +\path[blockArrow] (VLC) -- (Output); + + +\path[blockArrow] (ResidueDiff) -- (Transf); +\path[blockArrow] (Transf) -- (Quant); +\path[blockArrow] (Quant) -- (VLC) node [point,midway] (QuantFrame) {}; + +\node[processBlock,below=.8 of QuantFrame] (InvQuant) {$Q^{-1}$}; +\node[processBlock,right=0.7 of InvQuant] (InvTransf) {$T^{-1}$}; +\node[sumBlock,below=.5 of InvTransf] (ReconResidue) {$+$}; +\node[processBlock,text width=1.9cm, below=.8 of ReconResidue] (Filters) {Deblocking \& SAO filters}; + +\path[blockArrow] (QuantFrame) -- (InvQuant); +\path[blockArrow] (InvQuant) -- (InvTransf); +\path[blockArrow] (InvTransf) -- (ReconResidue); +\path[blockArrow] (ReconResidue) -- (Filters) node[point,midway] (p1) {}; + +\node[left=.4 of InvQuant, coordinate,yshift=0.5cm] (x1) {}; +\path[blockArrow,-] (x1 -| ResidueDiff) node[point] () {} -- (x1); +\path[blockArrow] (x1) |- (ReconResidue); + +\node [processBlock,text width=1.5cm] (Intra) at (2.2,-1.8) {Intra prediction}; +\node [processBlock,text width=1.5cm,below=1.2 of Intra] (Inter) {Inter prediction}; + +\path[blockArrow] (p1) -| (Intra); + +\node [point,left=.8 of Intra] (z1) {}; +\node [point,left=.8 of Inter] (z2) {}; + +\path[blockArrow,-] (Intra) -- (z1); +\path[blockArrow,-] (Inter) -- (z2); + +\node [coordinate,left of=z1,below of=z1,node distance=0.15cm] (w1) {}; +\node [coordinate,left of=z2, above of=z2, node distance=0.15cm] (w2) {}; + +\path (z1) -- (z2) node [coordinate, midway] (w) {}; +\path (w) -- (w -| sumPoint) node [point] (z) {}; + +\draw[-latex,densely dashed] (w1) to [bend left=60] (w2); +\draw [thick,blockArrow] (z1)+(0.1,-0.1) -- (z); + +\path[blockArrow] (z) -- node[pos=0.96,right] {-} (ResidueDiff); + +\node[processBlock,below=of Filters,text width=2cm] (RefBuffer) {Decoded Picture Buffer}; + +\node[processBlock,text width=1.5cm] at(Inter |- RefBuffer) (MotionSearch) {Motion Estimation}; + +\path[blockArrow] (Filters) -- (RefBuffer) node[point,pos=0.4] (p2) {}; +\path[blockArrow] (RefBuffer) -- (MotionSearch); +\path[blockArrow] (MotionSearch) -- (Inter); +\path[blockArrow] (p0) |- (MotionSearch); + + +\node[right of=p2, text centered,text width=1.5cm, node distance=2.3cm] (Output) {\textbf{Video output}}; +\path[blockArrow] (p2) -- (Output); + + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/IntraPrediction.pdf b/021_VideoCoding/Figures/HEVC/IntraPrediction.pdf new file mode 100644 index 0000000..ad37c32 Binary files /dev/null and b/021_VideoCoding/Figures/HEVC/IntraPrediction.pdf differ diff --git a/021_VideoCoding/Figures/HEVC/IntraPredictions.tikz b/021_VideoCoding/Figures/HEVC/IntraPredictions.tikz new file mode 100644 index 0000000..8312e1c --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/IntraPredictions.tikz @@ -0,0 +1,98 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{.1} + +\begin{tikzpicture}[scale=\scale,every node/.style={scale=\scale},on grid ] + + \definecolor{LCUFill}{rgb}{0,0.5,1} + \definecolor{DrawColor}{rgb}{0,0,0.5} + \definecolor{LCUFill}{rgb}{1,1,1} + \definecolor{DrawColor}{rgb}{0,0,0} + + \tikzstyle{TextStyle}=[scale=2, text centered] + + \tikzstyle{LCU}=[draw=DrawColor, fill=LCUFill, very thick] + \tikzstyle{CU}=[draw=DrawColor, thin] + \tikzstyle{PU}=[draw=DrawColor, thin, densely dashed] + + \tikzstyle arrowstyle=[scale=1] + \tikzstyle{IntraArrow} =[draw=black, shorten <=0, postaction={decorate,decoration={markings, mark=at position .3 with {\arrow[arrowstyle]{stealth}}}} ] + + +% \newcommand{\gridsize}{70} +% \draw[help lines] (-\gridsize,-\gridsize) grid ++(\gridsize*2,\gridsize*2); + + \draw[draw=black, thick] (-31,-31) rectangle (31,31); + + \coordinate (Center) at (0,0); + +% \foreach \x in {-32,...,31} +% \draw[draw=black] (\x-.5,31.5) rectangle ++(1,1); +% \foreach \y in {-31,...,32} +% \draw[draw=black] (-32-.5,\y-.5) rectangle ++(1,1); + + \newcounter{intra} + \setcounter{intra}{1} + \foreach \y in {-31,-26,-21,-17,-13,-9,-5,-2,0,2,5,9,13,17,21,26} + { + \addtocounter{intra}{1} + \node[scale=3.5, align=center] at (-32,\y) {\theintra}; + \draw[IntraArrow] (-31,\y) to (Center); + } + \addtocounter{intra}{1} + \node[scale=3.5, align=center] at (-32,32) {\theintra}; + \draw[IntraArrow] (-31,31) to (Center); + \foreach \x in {-26,-21,-17,-13,-9,-5,-2,0,2,5,9,13,17,21,26,31} + { + \addtocounter{intra}{1} + \node[scale=3.5, align=center] at (\x,32) {\theintra}; + \draw[IntraArrow] (\x,31) to (Center); + } + +% \foreach \x in {-31,-26,-21,-17,-13,-9,-5,-2,0,2,5,9,13,17,21,26,31} +% { +% \draw[IntraArrow] (Center) to (\x,-31); +% } +% \foreach \y in {-26,-21,-17,-13,-9,-5,-2,0,2,5,9,13,17,21,26,31} +% { +% \draw[IntraArrow] (Center) to (31,\y); +% } + +% \end{scope} +% \foreach \y in {0,...,31} +% \draw[black] (-1,\y) rectangle ++(1,1); + + +% \node[scale=6, text width=2cm, align=left] at (8,-1) {Planar}; +% \node[scale=6, text width=2cm, align=left] at (8,-4) { DC}; + + +% % H.264 +% \foreach \x in {-31,-15,0,15,32} +% \draw[ red,-latex] (Center) to (\x,-32); +% +% \foreach \y in {-15,0,15} +% \draw[red,-latex] (Center) to (32,\y); + +% \foreach \x in {-31,-15,0,15,31} +% \draw[IntraArrow, red] (\x,31) to (Center); +% +% \foreach \y in {-15,0,15} +% \draw[IntraArrow, red] (-31,\y) to (Center); +% +% \node[scale=6, text width=2cm, align=left,red] at (8,-1) {Planar}; +% \node[scale=6, text width=2cm, align=left,red] at (8,-4) { DC}; + + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/IntraPredictionsExample.tikz b/021_VideoCoding/Figures/HEVC/IntraPredictionsExample.tikz new file mode 100644 index 0000000..95c44a5 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/IntraPredictionsExample.tikz @@ -0,0 +1,52 @@ + +\newcommand{\scale}{.4} +\begin{tikzpicture}[auto, on grid, scale=\scale] + \def\BlockSize{8} + \tikzstyle{TextStyle}=[text centered, font=\Huge] + + \tikzstyle{PU} =[draw, scale=\scale, minimum height=\BlockSize cm, minimum width=\BlockSize cm, font=\Huge] + \tikzstyle{Neighbour} =[draw, scale=\scale, minimum height=1 cm, minimum width=\BlockSize cm, font=\Large, fill=gray!60] + \tikzstyle{Arrow} =[-latex,thick, draw=black] + \tikzstyle{Pixel} =[circle, rounded corners ,fill=lightgray, thick, scale=0.3] + \tikzstyle{PixelNeighbour} =[Pixel, fill=gray] + + \node[PU, anchor=north west] at (0,0) { }; % PU +% \node[Neighbour, anchor=south west] at (0,0) { top }; % Top +% \node[Neighbour, anchor=south west] at (\BlockSize,0) { top-right }; % Top-right +% \node[Neighbour, anchor=south east, rotate=90] at (0,0) { left }; % Left +% \node[Neighbour, anchor=south east, rotate=90] at (0,-\BlockSize) { bottom-left }; % Bottom-left +% \draw[Neighbour, scale=1/\scale] (0,0) rectangle ++(-1cm,1cm); + + + % Pixels + \foreach \x in {0,1,...,7} + \foreach \y in {-1,...,-8} + { + \node at (\x+0.5,\y+0.5) [Pixel]() {}; + } + \foreach \x in {-1} + \foreach \y in {0,...,-9} + { + \node at (\x+0.5,\y+0.5) [PixelNeighbour]() {}; + } + \foreach \x in {0,1,...,12} + \foreach \y in {0} + { + \node at (\x+0.5,\y+0.5) [PixelNeighbour]() {}; + } + + \path[Arrow, shorten >= 48 pt] (1+.5,.5) -- ++( -3 , -\BlockSize ); + \path[Arrow, shorten >= 25 pt] (2+.5,.5) -- ++( -3 , -\BlockSize ); + \foreach \x in {3,...,10} + { + \path[Arrow] (\x +.5,.5) -- ++(-3, -\BlockSize); + } + \node at (15,-6) [scale=.4] (PredLabel) {Predicted Pixels}; + \node[below=.3cm of PredLabel, scale=.4] (RefLabel) {Reference Pixels}; + \node[Pixel, left=.8 cm of PredLabel] (Pred) {}; + \node[PixelNeighbour, left=.8 cm of RefLabel] (Ref) {}; + + + +\end{tikzpicture} + diff --git a/021_VideoCoding/Figures/HEVC/MergeModeHEVC.tikz b/021_VideoCoding/Figures/HEVC/MergeModeHEVC.tikz new file mode 100644 index 0000000..342b918 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/MergeModeHEVC.tikz @@ -0,0 +1,36 @@ +\begin{tikzpicture} + +\newcommand{\scale}{1} + +\tikzstyle{BlockMain} =[draw=black, thick] +\tikzstyle{Block} =[draw=black, thick, loosely dashed] +\tikzstyle{Label} =[scale=1/\scale,font=\small] + +\newcommand{\MainBlockSize}{5} +\newcommand{\NeighbourBlockSize}{.5} + +\draw[BlockMain] (0,0) rectangle ++(\MainBlockSize,\MainBlockSize); +% \node[Label,text width=\MainBlockSize+25,text centered] at (\MainBlockSize/2,\MainBlockSize/2) {Current PU}; + +\draw[Block] (0,0) rectangle ++(-2*\NeighbourBlockSize,-2*\NeighbourBlockSize); +\node[Label] at (-\NeighbourBlockSize,-\NeighbourBlockSize) {$A_0$}; + +\draw[Block] (0,0) rectangle ++(-2*\NeighbourBlockSize,2*\NeighbourBlockSize); +\node[Label] at (-\NeighbourBlockSize,\NeighbourBlockSize) {$A_1$}; + +\draw[Block] (\MainBlockSize,\MainBlockSize) rectangle ++(2*\NeighbourBlockSize,2*\NeighbourBlockSize); +\node[Label] at (\MainBlockSize + \NeighbourBlockSize,\MainBlockSize + \NeighbourBlockSize) {$B_0$}; + +\draw[Block] (\MainBlockSize,\MainBlockSize) rectangle ++(-2*\NeighbourBlockSize,2*\NeighbourBlockSize); +\node[Label] at (\MainBlockSize-\NeighbourBlockSize,\MainBlockSize + \NeighbourBlockSize) {$B_1$}; + +\draw[Block] (0,\MainBlockSize) rectangle ++(-2*\NeighbourBlockSize,2*\NeighbourBlockSize); +\node[Label] at (-\NeighbourBlockSize,\MainBlockSize + \NeighbourBlockSize) {$B_2$}; + +\draw[Block] (\MainBlockSize,0) rectangle ++(2*\NeighbourBlockSize,-2*\NeighbourBlockSize); +\node[Label] at (\MainBlockSize + \NeighbourBlockSize,-\NeighbourBlockSize) {$T_{0}$}; + +\draw[Block] (\MainBlockSize/2,\MainBlockSize/2) rectangle ++(2*\NeighbourBlockSize,-2*\NeighbourBlockSize); +\node[Label] at (\MainBlockSize/2 + \NeighbourBlockSize,\MainBlockSize/2 -\NeighbourBlockSize) {$T_{1}$}; + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/MergeModeHEVCTwoFrames.tikz b/021_VideoCoding/Figures/HEVC/MergeModeHEVCTwoFrames.tikz new file mode 100644 index 0000000..e39905f --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/MergeModeHEVCTwoFrames.tikz @@ -0,0 +1,36 @@ +\begin{tikzpicture} +\newcommand{\xfshift}{8cm} +\newcommand{\scale}{1} + +\tikzstyle{BlockMain} =[draw=black, thick, fill=\TikZColorFillDark] +\tikzstyle{Block} =[draw=black, thick, loosely dashed] +\tikzstyle{Label} =[scale=1/\scale,font=\small] + +\newcommand{\MainBlockSize}{2} +\newcommand{\NeighbourBlockSize}{.4} + +\begin{scope}[yshift=1*\TikZyfshift, xshift=1*\xfshift, style={yslant=\TikZyslant,xscale=\TikZxscale}] + \draw[very thick] (-\MainBlockSize,-\MainBlockSize) rectangle ++(3*\MainBlockSize, 3*\MainBlockSize); + \draw[BlockMain] (0,0) rectangle ++(\MainBlockSize,\MainBlockSize); + \draw[Block] (0,0) rectangle ++(-2*\NeighbourBlockSize,-2*\NeighbourBlockSize); + \node[Label] at (-\NeighbourBlockSize,-\NeighbourBlockSize) {$A_0$}; + \draw[Block] (0,0) rectangle ++(-2*\NeighbourBlockSize,2*\NeighbourBlockSize); + \node[Label] at (-\NeighbourBlockSize,\NeighbourBlockSize) {$A_1$}; + \draw[Block] (\MainBlockSize,\MainBlockSize) rectangle ++(2*\NeighbourBlockSize,2*\NeighbourBlockSize); + \node[Label] at (\MainBlockSize + \NeighbourBlockSize,\MainBlockSize + \NeighbourBlockSize) {$B_0$}; + \draw[Block] (\MainBlockSize,\MainBlockSize) rectangle ++(-2*\NeighbourBlockSize,2*\NeighbourBlockSize); + \node[Label] at (\MainBlockSize-\NeighbourBlockSize,\MainBlockSize + \NeighbourBlockSize) {$B_1$}; + \draw[Block] (0,\MainBlockSize) rectangle ++(-2*\NeighbourBlockSize,2*\NeighbourBlockSize); + \node[Label] at (-\NeighbourBlockSize,\MainBlockSize + \NeighbourBlockSize) {$B_2$}; +\end{scope} + +\begin{scope}[yshift=0*\TikZyfshift, xshift=0*\xfshift, style={yslant=\TikZyslant,xscale=\TikZxscale}] + \draw[very thick] (-\MainBlockSize,-\MainBlockSize) rectangle ++(3*\MainBlockSize, 3*\MainBlockSize); + \draw[BlockMain] (0,0) rectangle ++(\MainBlockSize,\MainBlockSize); + \draw[Block] (\MainBlockSize,0) rectangle ++(2*\NeighbourBlockSize,-2*\NeighbourBlockSize); + \node[Label] at (\MainBlockSize + \NeighbourBlockSize,-\NeighbourBlockSize) {$T_{0}$}; + \draw[Block] (\MainBlockSize/2,\MainBlockSize/2) rectangle ++(2*\NeighbourBlockSize,-2*\NeighbourBlockSize); + \node[Label] at (\MainBlockSize/2 + \NeighbourBlockSize,\MainBlockSize/2 -\NeighbourBlockSize) {$T_{1}$}; +\end{scope} + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/MotionMergeMode.tikz b/021_VideoCoding/Figures/HEVC/MotionMergeMode.tikz new file mode 100644 index 0000000..f92e048 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/MotionMergeMode.tikz @@ -0,0 +1,37 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\begin{tikzpicture} + +\newcommand{\scale}{0.06} +\begin{scope}[xshift=3cm,yshift=3cm,node distance = 2.5cm, auto,every node/.style={minimum size=\scale,scale=\scale},scale=\scale] + +\newcommand{\ScaleBlocks}{3} + +\tikzstyle{BlockMain} =[draw=black, thick] +\tikzstyle{Block} =[draw=black, thick, loosely dashed] +\tikzstyle{Label} =[scale=1/\scale,font=\small] + +\newcommand{\MainBlockSize}{30} + + \draw[BlockMain] (0,0) rectangle ++(\MainBlockSize,\MainBlockSize); + \node[Label,text width=\MainBlockSize+25,text centered] at (\MainBlockSize/2,\MainBlockSize/2) {Current Block}; + + + \draw[Block] (0,0) rectangle ++(-10*\ScaleBlocks,-10*\ScaleBlocks); + \node[Label] at (-5*\ScaleBlocks,-5*\ScaleBlocks) {$A_0$}; + \draw[Block] (0,0) rectangle ++(-10*\ScaleBlocks,10*\ScaleBlocks); + \node[Label] at (-5*\ScaleBlocks,5*\ScaleBlocks) {$A_1$}; + + \draw[Block] (\MainBlockSize,\MainBlockSize) rectangle ++(10*\ScaleBlocks,10*\ScaleBlocks); + \node[Label] at (\MainBlockSize+5*\ScaleBlocks,\MainBlockSize+5*\ScaleBlocks) {$B_0$}; + \draw[Block] (\MainBlockSize,\MainBlockSize) rectangle ++(-10*\ScaleBlocks,10*\ScaleBlocks); + \node[Label] at (\MainBlockSize-5*\ScaleBlocks,\MainBlockSize+5*\ScaleBlocks) {$B_1$}; + \draw[Block] (0,\MainBlockSize) rectangle ++(-10*\ScaleBlocks,10*\ScaleBlocks); + \node[Label] at (-5*\ScaleBlocks,\MainBlockSize+5*\ScaleBlocks) {$B_2$}; + + +\end{scope} +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/MotionSkipMode.tikz b/021_VideoCoding/Figures/HEVC/MotionSkipMode.tikz new file mode 100644 index 0000000..36cd86e --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/MotionSkipMode.tikz @@ -0,0 +1,48 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + + +\begin{tikzpicture} + +% \draw[draw=black] (0,0) rectangle ++(21.58cm,27.94cm); +\newcommand{\scale}{0.06} + +\begin{scope}[xshift=3cm,yshift=3cm,node distance = 2.5cm, auto,every node/.style={minimum size=\scale,scale=\scale},scale=\scale] +\newcommand{\ScaleBlocks}{3} + +\tikzstyle{BlockMain} =[draw=black, thick] +\tikzstyle{Block} =[draw=black, thick, loosely dashed] +\tikzstyle{Label} =[scale=1/\scale,font=\small] + +\newcommand{\MainBlockSize}{30} + + \draw[Block,opacity=0] (0,0) rectangle ++(-10*\ScaleBlocks,-5*\ScaleBlocks); +% \draw[Block,opacity=0] (0-10*\ScaleBlocks,\MainBlockSize) rectangle ++(-5*\ScaleBlocks,10*\ScaleBlocks); +% \draw[Block,opacity=0] (\MainBlockSize,\MainBlockSize) rectangle ++(5*\ScaleBlocks,10*\ScaleBlocks); + + \draw[BlockMain] (0,0) rectangle ++(\MainBlockSize,\MainBlockSize); + \node[Label,text width=\MainBlockSize+20,text centered] at (\MainBlockSize/2,\MainBlockSize/2) {Current Block}; + + + \draw[Block] (0,0) rectangle ++(-10*\ScaleBlocks,10*\ScaleBlocks); + \node[Label] at (-5*\ScaleBlocks,5*\ScaleBlocks) {$MV_A$}; + + \draw[Block] (\MainBlockSize,\MainBlockSize) rectangle ++(-10*\ScaleBlocks,10*\ScaleBlocks); + \node[Label] at (\MainBlockSize-5*\ScaleBlocks,\MainBlockSize+5*\ScaleBlocks) {$MV_B$}; + + \draw[Block] (\MainBlockSize,\MainBlockSize) rectangle ++(10*\ScaleBlocks,10*\ScaleBlocks); + \node[Label] at (\MainBlockSize+5*\ScaleBlocks,\MainBlockSize+5*\ScaleBlocks) {$MV_{C}$}; + + + \draw[Block] (0,\MainBlockSize) rectangle ++(-10*\ScaleBlocks,10*\ScaleBlocks); + \node[Label] at (-5*\ScaleBlocks,\MainBlockSize+5*\ScaleBlocks) {$MV_{D}$}; + +% \draw[Block] (\MainBlockSize,0) rectangle ++(10*\ScaleBlocks,-10*\ScaleBlocks); +% \node[Label] at (\MainBlockSize+5*\ScaleBlocks,-5*\ScaleBlocks) {$T_{BR}$}; +% \draw[Block] (32,32) rectangle ++(-10*\ScaleBlocks,10*\ScaleBlocks); +% \node[Label] at (32-5*\ScaleBlocks,32+5*\ScaleBlocks) {$T_{CTR}$}; + +\end{scope} +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/NALHeader.tikz b/021_VideoCoding/Figures/HEVC/NALHeader.tikz new file mode 100644 index 0000000..8e9be30 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/NALHeader.tikz @@ -0,0 +1,29 @@ +\newcommand\bithead[1]{\bitbox[]{#1}{\scriptsize #1}} +\newcommand{\colorbitbox}[3]{% + \rlap{\bitbox{#2}{\color{#1}\rule{\width}{\height}}}% + \bitbox{#2}{#3}} + +% \begin{figure*} +% \centering + + \begin{tikzpicture} +% \begin{scope} +% \coordinate (A) at (0cm,0); +% \coordinate (B) at (7.73cm,0); +% \draw[transform canvas={xshift = -3.5cm, yshift = 0.1cm}, <->] (A) -- (B) node[midway,fill=white] {{Frame 1}}; +% \end{scope} + +% \begin{scope} +% \coordinate (A) at (0cm,0); +% \coordinate (B) at (7.73cm,0); +% \draw[transform canvas={xshift = 0.04cm - 3.5cm + 7.73cm + 0.65cm, yshift = 0.1cm}, <->] (A) -- (B) node[midway,fill=white] {{Frame N}}; +% \end{scope} + \end{tikzpicture} + + \begin{bytefield}[bitwidth=20pt]{50} + \bitheader{0-31} \\ + \colorbitbox{gray!20}{1}{F} & \colorbitbox{gray!60}{6}{NAL Type} & \colorbitbox{gray!20}{6}{Reserved} & \colorbitbox{gray!60}{3}{TIDP} + \end{bytefield} + +% \caption{MRP stream frame-based structure.}\label{fig:mrp-bitstream} +% \end{figure*} diff --git a/021_VideoCoding/Figures/HEVC/NALUnitHeader.tikz b/021_VideoCoding/Figures/HEVC/NALUnitHeader.tikz new file mode 100644 index 0000000..da826b0 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/NALUnitHeader.tikz @@ -0,0 +1,60 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{1} + +\begin{tikzpicture}[scale=\scale,every node/.style={scale=\scale},on grid ] + +\definecolor{LCUFill}{rgb}{0,0.5,1} +\definecolor{DrawColor}{rgb}{0,0,0.5} +\definecolor{LCUFill}{rgb}{1,1,1} +\definecolor{DrawColor}{rgb}{0,0,0} + +\tikzstyle{TextStyle}=[scale=.5, text centered] +\tikzstyle{Square}=[draw=DrawColor,thick] +\tikzstyle{Byte}=[draw=DrawColor,thick, dashed] + + +% \begin{scope}[xshift=0, yshift=0] +% \draw[Square] (0,0) rectangle ++(2,1cm); +% \draw[Square] (2,0) rectangle ++(7,1cm); +% \node[TextStyle] at (2/2,.5) {NAL header}; +% \node[TextStyle] at (2+7/2,.5) {Slice data}; +% \coordinate (EdgeL) at (0,0); +% \coordinate (EdgeR) at (2,0); +% \end{scope} + + + + \foreach \x in {0,1,...,15} + \node at (\x cm,1cm) {$\x$}; + + \node at (0 cm,0cm) {F}; + +% \draw[Square] (0,0) rectangle ++(8,1cm); +% +% \foreach \x in {1,7,13} +% \draw[draw=black] (\x*.5,0) -- ++(0,.5); +% +% % \draw[Square,dashed] (EdgeL) -- (0,1); +% % \draw[Square,dashed] (EdgeR) -- (8,1); +% +% \node[TextStyle] at (.25,.5) {F}; +% \node[TextStyle] at (6*.25+0.5,.5) {NAL type (6)}; +% \node[TextStyle] at (7*.25+3.25,.5) {Layer ID (6)}; +% \node[TextStyle] at (3*.25+6.5,.5) {TIDP (3)}; + + \node[TextStyle] at (8/2,-0.5) {2 bytes header}; + + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/PUAsymetric.tikz b/021_VideoCoding/Figures/HEVC/PUAsymetric.tikz new file mode 100644 index 0000000..5aa518f --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/PUAsymetric.tikz @@ -0,0 +1,63 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{0.65} + +\begin{tikzpicture}[scale=\scale,every node/.style={scale=\scale},on grid ] + + \definecolor{LCUFill}{rgb}{0,0.5,1} + \definecolor{DrawColor}{rgb}{0,0,0.5} + \definecolor{LCUFill}{rgb}{1,1,1} + \definecolor{DrawColor}{rgb}{0,0,0} + + \tikzstyle{TextStyle}=[scale=1.5, text centered] + + \tikzstyle{LCU}=[draw=DrawColor, fill=LCUFill,thick] + \tikzstyle{CU}=[draw=DrawColor,thick] + \tikzstyle{PU}=[draw=DrawColor, densely dashed] + +% \newcommand{\gridsize}{70} +% \draw[help lines] (-\gridsize,-\gridsize) grid ++(\gridsize*2,\gridsize*2); + + +\begin{scope}[xshift=0] + \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 + \draw[CU] (0,0) rectangle ++(1,4); % 64x64 + \draw[CU] (1,0) rectangle ++(3,4); % 64x64 + \node[TextStyle] at (2,-.8) {$\frac{3N}{4}\times N$ (L)}; +\end{scope} + +\begin{scope}[xshift=5cm] + \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 + \draw[CU] (0,0) rectangle ++(3,4); % 64x64 + \draw[CU] (3,0) rectangle ++(1,4); % 64x64 + \node[TextStyle] at (2,-.8) {$\frac{3N}{4}\times N$ (R)}; +\end{scope} + +\begin{scope}[xshift=10cm] + \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 + \draw[CU] (0,0) rectangle ++(4,3); % 64x64 + \draw[CU] (0,3) rectangle ++(4,1); % 64x64 + \node[TextStyle] at (2,-.8) {$N\times\frac{3N}{4}$ (T)}; +\end{scope} + +\begin{scope}[xshift=15cm] + \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 + \draw[CU] (0,0) rectangle ++(4,1); % 64x64 + \draw[CU] (0,1) rectangle ++(4,3); % 64x64 + \node[TextStyle] at (2,-.8) {$N\times\frac{3N}{4}$ (B)}; +\end{scope} + + + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/PUSymetric.tikz b/021_VideoCoding/Figures/HEVC/PUSymetric.tikz new file mode 100644 index 0000000..a464f84 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/PUSymetric.tikz @@ -0,0 +1,59 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{0.65} + +\begin{tikzpicture}[scale=\scale,every node/.style={scale=\scale},on grid ] + + \definecolor{LCUFill}{rgb}{0,0.5,1} + \definecolor{DrawColor}{rgb}{0,0,0.5} + \definecolor{LCUFill}{rgb}{1,1,1} + \definecolor{DrawColor}{rgb}{0,0,0} + + \tikzstyle{TextStyle}=[scale=1.5, text centered] + + \tikzstyle{LCU}=[draw=DrawColor, fill=LCUFill,thick] + \tikzstyle{CU}=[draw=DrawColor,thick] + \tikzstyle{PU}=[draw=DrawColor, densely dashed] + +% \newcommand{\gridsize}{70} +% \draw[help lines] (-\gridsize,-\gridsize) grid ++(\gridsize*2,\gridsize*2); + + +\begin{scope}[xshift=0] + \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 + \node[TextStyle] at (2,-.8) {$N\times N$}; +\end{scope} + +\begin{scope}[xshift=10cm] + \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 + \draw[CU] (0,0) rectangle ++(2,4); % 64x64 + \node[TextStyle] at (2,-.8) {$\frac{N}{2}\times N$}; +\end{scope} + +\begin{scope}[xshift=5cm] + \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 + \draw[CU] (0,0) rectangle ++(4,2); % 64x64 + \node[TextStyle] at (2,-.8) {$N\times\frac{N}{2}$}; +\end{scope} + +\begin{scope}[xshift=15cm] + \draw[LCU] (0,0) rectangle ++(4,4); % 64x64 + \draw[CU] (0,0) rectangle ++(2,2); % 64x64 + \draw[CU] (2,2) rectangle ++(2,2); % 64x64 + \node[TextStyle] at (2,-.8) {$\frac{N}{2}\times\frac{N}{2}$}; +\end{scope} + + + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/PictureTypes_Leading_Trailing.tikz b/021_VideoCoding/Figures/HEVC/PictureTypes_Leading_Trailing.tikz new file mode 100644 index 0000000..079a7d8 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/PictureTypes_Leading_Trailing.tikz @@ -0,0 +1,34 @@ +\begin{tikzpicture}[auto] +\def\yshift{1.2cm} +\tikzstyle{frame}=[thick,draw, fill=\TikZColorFill, yslant=0,xscale=1,minimum size=1cm] +\tikzstyle{dependency}=[thick,draw,-latex, shorten <=3 pt, shorten >=3 pt] +\tikzstyle{label}=[text centered, font=\small] +\node[frame] at (-6 ,0*\yshift) (P0) {$P_0$}; +\node[frame] at (-4.5 ,2*\yshift) (B3) {$B_3$}; +\node[frame] at (-3 ,1*\yshift) (B2) {$B_2$}; +\node[frame] at (-1.5 ,2*\yshift) (B4) {$B_4$}; +\node[frame] at (0 ,0*\yshift) (I1) {$I_1$}; +\node[frame] at (6 ,0*\yshift) (P5) {$P_5$}; +\node[frame] at (3 ,1*\yshift) (B6) {$B_6$}; +\node[frame] at (1.5 ,2*\yshift) (B7) {$B_7$}; +\node[frame] at (4.5 ,2*\yshift) (B8) {$B_8$}; +\node[frame, white] at (-3,-.05) (B2A) {}; +\path[dependency] (P0) -- (B3); +\path[dependency] (P0) -- (B2); +\path[dependency] (B2) -- (B3); +\path[dependency] (B2) -- (B4); +\path[dependency] (I1) -- (B4); +\path[dependency] (I1) -- (B2); +\path[dependency] (I1) -- (P5); +\path[dependency] (I1) -- (B6); +\path[dependency] (I1) -- (B7); +\path[dependency] (B6) -- (B7); +\path[dependency] (B6) -- (B8); +\path[dependency] (P5) -- (B6); +\path[dependency] (P5) -- (B8); +\node[label,below=.1 of I1] {CRA}; +\node[draw,dashed,inner sep=5pt, fit=(B3) (B2) (B4) (B2A)] (Leading) {}; +\node[label,below=.1 of Leading] {Leading}; +\node[draw,dashed,inner sep=5pt, fit=(B7) (B6) (B8) (P5)] (Trailing) {}; +\node[label,below=.1 of Trailing] {Trailing}; +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/PictureTypes_TSA.tikz b/021_VideoCoding/Figures/HEVC/PictureTypes_TSA.tikz new file mode 100644 index 0000000..3ff7340 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/PictureTypes_TSA.tikz @@ -0,0 +1,29 @@ +\begin{tikzpicture}[auto] +\def\yshift{1.2cm} +\tikzstyle{frame}=[thick,draw, fill=\TikZColorFill, yslant=0,xscale=1,minimum size=1cm] +\tikzstyle{dependency}=[thick,draw,-latex, shorten <=3 pt, shorten >=3 pt] +\tikzstyle{label}=[text centered, font=\small] +\node[frame] at (-6 ,0*\yshift) (P0) {$P_0$}; +\node[frame] at (-3 ,1*\yshift) (P1) {$P_1$}; +\node[frame] at (-4.5 ,2*\yshift) (P2) {$P_2$}; +\node[frame] at (-1.5 ,2*\yshift) (P3) {$P_3$}; +\node[frame] at (0 ,0*\yshift) (P4) {$P_4$}; +\node[frame] at (1.5 ,2*\yshift) (P5) {$P_5$}; +\node[frame] at (3 ,1*\yshift) (P6) {$P_6$}; +\node[frame] at (4.5 ,2*\yshift) (P7) {$P_7$}; +\node[frame] at (6 ,0*\yshift) (P8) {$P_8$}; +% \node[frame, white] at (-3,-.05) (AX) {}; + +\path[dependency] (P0) -- (P1); +\path[dependency] (P0) -- (P2); +\path[dependency] (P0) -- (P4); +\path[dependency] (P1) -- (P3); +\path[dependency] (P2) -- (P3); +\path[dependency] (P4) -- (P5); +\path[dependency] (P4) -- (P6); +\path[dependency] (P4) -- (P8); +\path[dependency] (P6) -- (P7); +% \node[draw,dashed,inner sep=5pt, fit=(B3) (B2) (B4) (B2A)] (Leading) {}; +% \node[label,below=.1 of Leading] {Leading}; +\node[label,right=.1 of P6, yshift=-.3cm] {TSA}; +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/Slices/SliceStructuring.pdf b/021_VideoCoding/Figures/HEVC/Slices/SliceStructuring.pdf new file mode 100644 index 0000000..cd6b65c Binary files /dev/null and b/021_VideoCoding/Figures/HEVC/Slices/SliceStructuring.pdf differ diff --git a/021_VideoCoding/Figures/HEVC/Slices/SliceStructuring.tikz b/021_VideoCoding/Figures/HEVC/Slices/SliceStructuring.tikz new file mode 100755 index 0000000..4d34873 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/Slices/SliceStructuring.tikz @@ -0,0 +1,12 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + +\newcommand{\width}{6cm} +\newcommand{\vdisp}{6cm} + +\begin{tikzpicture} + \node[inner sep=0] at (0,0) {\includegraphics[width=\width]{Slices}}; + \node[inner sep=0] at (\width+.5cm,0) {\includegraphics[width=\width]{Tiles}}; + \node[inner sep=0] at (2*\width+1cm,0) {\includegraphics[width=\width]{WPP}}; +\end{tikzpicture} \ No newline at end of file diff --git a/021_VideoCoding/Figures/HEVC/Slices/Slices.pdf b/021_VideoCoding/Figures/HEVC/Slices/Slices.pdf new file mode 100644 index 0000000..a2315fc Binary files /dev/null and b/021_VideoCoding/Figures/HEVC/Slices/Slices.pdf differ diff --git a/021_VideoCoding/Figures/HEVC/Slices/Tiles.pdf b/021_VideoCoding/Figures/HEVC/Slices/Tiles.pdf new file mode 100644 index 0000000..15578aa --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/Slices/Tiles.pdf @@ -0,0 +1,135 @@ +%PDF-1.3 +1 0 obj +<< +/Pages 2 0 R +/Type /Catalog +>> +endobj +2 0 obj +<< +/Type /Pages +/Kids [ 3 0 R ] +/Count 1 +>> +endobj +3 0 obj +<< +/Type /Page +/Parent 2 0 R +/Resources << +/XObject << /Im0 8 0 R >> +/ProcSet 6 0 R >> +/MediaBox [0 0 784 671] +/CropBox [0 0 784 671] +/Contents 4 0 R +/Thumb 11 0 R +>> +endobj +4 0 obj +<< +/Length 5 0 R +>> +stream +q +784 0 0 671 0 0 cm +/Im0 Do +Q +endstream +endobj +5 0 obj +31 +endobj +6 0 obj +[ /PDF /Text /ImageC ] +endobj +7 0 obj +<< +>> +endobj +8 0 obj +<< +/Type /XObject +/Subtype /Image +/Name /Im0 +/Filter [ /RunLengthDecode ] +/Width 784 +/Height 671 +/ColorSpace 10 0 R +/BitsPerComponent 8 +/Length 9 0 R +>> +stream + !"!!!!!!!!!!!!!!!!" "! !!!!!!!! !!!!!!!!! !!!!!!!! "!!!!!!!!! !" !!!!!!!!" !"!!!!!!!!!" !"!!!!!!!!!! !" !!!!!!!!"!"! " "!!!!!!!!"!" !!!!!!!!!! !" !!"!"!!!!!!!! " "! !!!"!!!!!!!!" "!"! "!!!!!!!! !!!!!!!!! !! "!!!" !"!!!" "!!!"!!!" "!!!" !"!!!"! !! "!"!!!" !!! !"! ! "!!!"!!! !"! ! "!!! " !!!" !!" !!!" !!! !"!" !!!!!! !"!" !!! "!!! !" ! !! "! ! ! !"!" !!"!"! "!"!"! "!"!"!"!! "!!!""! " !""! " !"!"!" !! ! !" !"!!" !"!!" ! "!!"!"!"!"!!"!"!! !!" ! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! "! ! ! !"! ! !! !"!! ! !! !! !!!! "! ! ! !"! ! !!" "!" !!! !! !!!!" ! ! !!"!" ! "!" !!"! ! " ! ! ! !! !!!" ! ! !!"!" ! "!" !! "!!! !!!! !"!"! "!"!!!"! !!!!!!! !"!"! "!"!!! !!!!!!! ! !"! !!!! !"!"! ! ! ! !!!!! ! !"! !!! !!!!!! !"! !!" !!!!" !! ! " !!!! !"! !!" !!! !"!!!!!!!"!!"!"! "!!! ! "!" "!"!!!!!"!!"!"! "!!!"!"!" ! !!!!!!!" !!!"!!!!!"!""!!!!!!!" !!!"!!!! ! !!!!!!!"! ! ! !!!!! ! !!"!!!!!!!"! ! ! !!!!! "!!!!!!!"!!"!! !!!! !!"!!!!!!"!!"!! !!!!!"!!!!!!" !!! "!!!!!" !!! !!!!!" !!! "!!!!"!!!!!!!!" ! !!!!! "!!!!!!!" ! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"!!!!"!! !"!!!!! !!!!"!!!!"!! !!!"!!"!!!!!" !! !"! !! !!!" !!!!"!!!!!" !! !"! !! !! !"!! !!! !!!!"!!"!!!" !"!!! ! ! !!"!!" !!"!!"!!!" !"!!!" "! "! !!!" !!! !! !!!"!! !!"!" !!"" ! !! !! !!!"!! !!"! "!!!" ! !!!"!!"!!"! !!! ! !!!!!!!"!!"!!"! !!! !! !!!" !!!! ! ! !!! " !!! ! !" ! "!" ! "!!! ! ! !!! " !!!"!!!"!"!!!"! !"!!"! "!! !!! ! !! ! !!" !!"! !"!!"! "!! !! !" !! ! !!!!! ! ! "!!! !"!" !"!!" !!!! ! ! "!!!!!"! !!!!"!" !"! "!" ! !"!!!"!"! !!! ! !!!!"!" !"! "!" ! !"!!" !"! "!"!!! ! !!!"! ! !" !" !! ! ! !!!"!"!! " !!"! ! !" !" !! ! ! !!! "! ! !"!!" ! "!!!! "! !!"!! ! !!!!!!!! "! !!"!! ! !!! !" !!! "!" !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" "!"! " ! !!! !! ! !"! " "!!"!! !!" " "!! !!" !!!!! !" !!" !""!"!!! !"!!"! !"!"!!"!! !"!"!!"!"! !"!"!! ! ! "!!"! !" "!! !"!!!"!" !"!!! !! !"!!!"!"!!!"! "!"!!"!" !!! !"!!!"!!"! !!!!! !"!! "!!!" "!!!! "!!!"! !!! " !!!!!"! "! !" !"!!! "! !! !!"! !""! "! !!!"!! "! !" ! "!! ! !"!!" !!! !"!""!"" !! ! !!! "!! !!! !! ! ! ! ! "! !! !! !!!!"!!" ! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" ! ! ! !" ! ! !! ! "!!!!!!!"! " !"! ! ! !! " !"!!!!!!! ! !! !!!!! ! !!!!!! !"!"!" !"! ! !!!! !"! "!"!"! !!!!!!"!"!"!!"! !!!!!"!!!!!!!" !! ! !"!"!" !!!!!"! ! ! "! !" !!!!! !" !"!"! ! !" !!!!"! !"!!!!!!!" ! !"! !"!" "! "!"!!!!!"! ! ! ! !"! ! !"!!!!!! !" "!!!!!!"! "! !"!" !!!!! !"! !"! !!!!"!""!!!!!!!"!# !" !!!!!"!!"!"!!!!!!!"!! !!!!!"! !! ! !!!!!!! ! !!!!!"! !"!"!!!! ! "!!"!!!!! !! !!!!!"!! ! !!!!!!"!"!!!!! " !"!"!!!!!!"!!" "!!!!! !"!"! !!!!!" !!!!!!! !!" !! !!!" " !" !!!!!!"!!!!!!!!!!!!!"!!! !!!!" !!!!!!!!"! !!!!!!"!!!!!!!"!!!!! "!!!!!!"!"!!!!!"! !!!!!!"!!!!! !!!!!!!!!!! !!!!!!!!!!!!"!!!!!!! !!!!!! "!!!!!!!!!!! !!!!!!! !!!!!!!!!!! !!!!!!!!!!! "!!!!" "!!!!!!! !!!!!!!!!! "!!!!!! !!!!!!! !!!!!!!!!!"!!!!! !!!!!!!!!!!!!!!!!"!"!!"!"!!! !!!!!!!!! !!!!"!!!! !!! "!!!" ! "!!"!!!"!!"!! !"" !!"!!!! !!" !!"!!!"!!!!"!!" !" !! !! "! !" !!!" !!! !! ! !!""!!!! !!"!!!!"!"!!!"!! !!!!!!!!"!! !"!"!"!!! "!!! "!!"!!!! "! !!"!"!!!! !#!"! !"!!! !!! !" !!! "!"!!# ! ! "!! !"!!" "!"! !! !!" "!! ! !!"!!"!!!"! !"!!! !!"!!! " !"! ! ! "!!"!!!!"!"!! ! !" "!!"! !""!!!!!!!"!!!! "! " "!! !"!! # !! !"! "!!!!! ! !"!! !!"!"!!"!!! !!!!! !!"!!"! " !!!"!" !!! !!"!!! !!"!! ! !"!!! ! !"! ! !"!!!!! ! ! !" !"!"!!!!!"!" !! "! !"!"!" ! ! ! ! !"!!!"!"! !" ! !!!! ! !! !" !"!!! !!!" !!! ! !"!!"!!"! !!!"! ! !!"!" ! "!"!!!!"!! ! !" !!!"!!!!!! "!"!!" !!!" !!! "!" !!! !! "!" ! !! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!" ! !!! !! ! ! ! "! !!"!! ! ! ! !!! "!!!"!!! ! !"!!" !!! !!"! "!"! !""! "! !!!"!!"!!"! "! !" !"!! "! "!!!! "!!!"! !!! " !! !"!!!" ! "! !!! !!!!"!"! !!!!! !"!!!!!!!!!!" !!!!!!!!!"!!"!!!!!!!!!"!!!!!!!!!" "!"!!!!!!!!!! !!!!!!!!!" "!!!!!!!!! "!! !!!!!!!!! "!!!!!!!!" !! "!!!!!!!!! ""!!!!!!!! ! "!!!!!!!!! " ! !!!!!!!!" !!" !!!!!!!! ! "!!!!!!!!!!!!!!!!!!! "! !!!!!" ! ! !"!"!" "!!! !!! ! !"!!!!! "! ! #!!!!!"! ! "! ! "!!!! "!"! ! !" !" !" !!!!"!"!!"!!!" "! !"! !"! ! !!" !"!"!! !" !!"!!" ! ! " !"! !" " "!"!"!! !!" !"! ! ! !"! " !! !"!"!!" "!!!!" "!"!!!!! !!!!!!!!!! !"!!!!!!"! "!!!!! !!!!!"!"!!!!!! " !"!!!!!"! !!!!!! ! !!!!!! "! !!!!!"! "!!!!!! !"!"! !!!!!!"! ! !!!!!!"! !"!"!!!!! !"!" " !!!!! !"! " "!!!!!! "!"!" !" !!!!! ! !!!!!! ! !!!!!"! ! "! !" !"!!!!! ! "! !!!!!!" " !" !" !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !"!"!!! !!!!!!!!" " "!!!!!!! !" !!! ! !!!!!! !" !" !!"!"!!!!! !! !! !"!!!!!!!! !"!!!!!!!!!"!!!!!!!!! "!"!!!!!!! " ! ! !!!!!!!"! "!! ! !!!!!!! !!!!! #! "!!!!!!!!! !"! "! !!!!!!!!!" ! "!"! !" !!!!!!!!!! !"!!"!!!!!!!! ! !!! !!!!!!!!! "!! "!!!!!!!!!! !!!"!!!!!!!!"! !!"! !!!!!!!!" !!"!!! !!!!!!!!"!!!!!!!!! !! "! !!!!!!!!"! !"!" ! !"! "!!!!!!!!!! ! !"!"!!!!!!!!!" !!!!!!!!! !"!"!"!!!!!!!!! ! "!!!!!!!!!"!"!!!!!!!!! !!!!!!!!!! ! !!!!!!!!!!!!!!!!!!!"!!!!!!!!!!! " !!!!!!!!!" " "!!!!!!!!!!!! ! !!!!!!!!!!"!"! "!!!!!!!!"!"!"!"!!!!!!!!!! ! " !!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"! !!!!!"!"!!!!"!!!!! !!!!! " !!!! !"!!!!" ! !!!!"!" !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !"!! !! !" ! "! !!!!!!!!! "!!!!!!!!!! "!!!!!!!!" !!! !! !!!!!!!!!"!!!!!!!"!"! " !!!!!!!!"! "!!!!!!!!!" ! "! !!!!!!"!! "! "!"!!!!!!! !! !! ! " "!!!!!!! !"!! ! !" ! "!! !! !"!!!!! "! !!"! !!"!!"!!!!!!!!!!! !! !!!!!!!!!! "! ! "!!" "! !!!!!!!!!! "! !!!" "! !!!!!!!!!!! !!!!!!!!!!!!! !!"! !!! !!!!!!!!!!! !!"!!"!!!!!!!!!! !!!!!!!!!!!!! !"!!!!!!!!!!!!!!!!! ! !" !!!!!!!!!!"!" #! !!" ! ! ! !!!!!!!!! !"!" !"!!"!" " "!!!!!!!!!" ! ! "!"! !"! "!! "!!!!!!!!!! ! ! !!! !! !!!!!!!! !" !!! "!!"!!!!!!!!!!"!!!" # !!"!!!!!!!!! !!!" "!! !!!!!!!!!!!!"! !!!!!!!!!!"! "!"! ! !!!!!!!!!"! !" !!! "!!!!!!!!!! !"!!! !"!!!!!!!!!"!!"!!"! !"!"!!!!!!!!! !"!!"!" !!"!" "!"!!!!!!!! ! ! "! "!!!"!" !!!!!!!!! "!"!"!!!"!" !!!!!!!!!!" ! !!!!!!!! !!!! ! ! !!!!!!!!!!!! !!!!!! ! ! ! !!!!!"! !!!!!!" !"!!!!!! !"!"!!!!!! " !"!!!!!"!" "!"!" ! !!!! !" "! !!!!!! "! ! !!!!!!" !"!"!!!!!! !!!! "! !"!" !!!!! ! "!!!!! !"!!!!!" ! !"!!!! !"!!!!!! ! ! " "!!!!!! !"!!!!! "! !!!!!"!!!!!! ! ! ! !!!!!!!!!!!!!!!"! "!!!!" ! !!!!!!"!!!!!!!!!! "!!!!!! !! " ! !" "!!" ! !!" !!"!! ! ! "! !! !!"!"! !" " "!"!" ! # !! "! !! !" !! "! "!!!! "!"! !"!" ! !! !!" !"!!!! !" ! !"!!"! ! #!!!!!!"! !!!" ! !"!!!!!" "! !" ! ! "! !!!!!"! !"!!! !!!"! ! !!!!!! "!"!!!!!!!!" "! !!!!!!!! "!"! !!!!!!!!!" !!!!!!!!!" !" "!!!!!!!!" !! !"!!!!!!!!! ! "!!!!!!!!"! !" !!!!!!!!! "!!!!!!!!!!"!!!!!!!!! "! !"!!!!!!!!!" "!!!!!!!!! "! "!"!!!!!!!!" ! !!!!!!!!!" !"!!!!!!!!!!!!!!!!!!!"!! !"!!!"!" !"!!! !"! !!!! !"!!!" !"!!!"! "!"!! ! ! "! "!!!!"!"! ! "!! !"!"!"! "! !"!! "!"!!! !"!"! !"! !"! !"" !!"!!!! !! " !" !!"!"!"!!"!! ! ! !!" " "!!!" !!! ! ! !! ! ! "!! " "!!!!! !" "! !! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"! "! !"! !"!!!!!!!!!!!"!"!"! ! !!!!!"!"!!!!!! ! !! "! !!!" !"!"!"!"!!!!!!"!"! !"! "! !!!!""!"!!!!!!!"! !"!" ! ! !!!!! !" "! "! !"!"! !!!!!!"!" "!!" !"!!!!! !"! !"!!!!!!"! !"! "!!"! !"!!!!" !" !"!"! ! ! ! ! ! !!!!"!"!"! "!" ! "!" ! !!!!"!"! !" ! !!!!!!"!"! !!!!!! " !! "!!!!!!! !"!!"!"!! !!!! "!!" "!!!!!!"!!!!!!!" " ! !!!!!!!#! ""!"!!!!!!" ! !!!!!!"!! !"" !!!!! !"! !!!!!!! !"!"! !!!!! !!!!!! "!!"!"!!!!!"!"!!!!!!!"!""!"!! !!!!! !!!!!!!!!"! !!!!! !!!!!!!"! "!!!!!!! !!!!!!"! !!!!!!!!!!!!!!!!!!!!!!!!" "!!!!!!" !!!!!!!!!!"! !!!!!!!"!!!!!" !!!!!" !!!!!!!!!!!!!!!!!!!!!!!"!!!!!! !!!!!!!!!!!"!!!!!!!!!!!"!!!!!! !!!! !!!!!!!!!!! !!!!!!!"!!!! !"!!!!!!!!!!!"!"!!!!!!!! "!!!"!!! !!!!! !!!!"!!!!!! !!! !"!!!"!! "!"! !! !" ! !"! !"! !! !! !!!" "!!"! !"! !!"! !"!!!!!! ! !!" !"!!"!! !"!!!!"!!!!!!! !!!!!!! !!!"!!! "!!" !!!! ! !!!!! !"" "!!!!!!! !!!"!!"! !"!!" !! "!! !!"!"!! " !!!"! !"!!!!!!! "!!! !!! !"!! "! !" !!!" !!!"!" !!!"!!"!!! !!!!! ! !!! " !!"! ! ! "! !" !! !!!!"!!! "!"!! !! !!!!!!" !!!"!! ! !!!"!!!!!!"!! !! !!"!"!!!!" !!! ! "! ! !"!"!"!!!!!!! !!!! "!!"! ! " ! ! !"! !!!!" !"! "!!"!" !! ! ! "! ! !""!"! !!" !!! !!!!"! "!"! !!"! ! "!"!!! "!" "! "! "!" "!" "! !"!! !" !" ! "!!" " "!"! "!" !!! !"!" !"!!!"! !"!"! !"!!!! "! !" ! !" ! ! !! ! ! !"!!"!"!!!!"! ! !!!" "! !" ! ! "! !" !""!""!" ! "!!!!" "!"!"!!"!! !"!!"!"!"!"! !!" !!!!"!"!! !! "! "!!!"!"!" "!!! "!" !!!!"! ! ! "!"!"!!!"! " !!!!!" !"! ! ! ! !!! !"!!" "!" "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"!"!!"!"!!!" ! !! ! !!" !"!!" ! "!!!""!! " !"! ""!" !!"!"! "!!"!"!"!! "!! "! !! "!!" !!" !!!" !!! !"!!" !!!!! !" ! !! "!"!!!" !!! !"!! ! "!!! "!!!" !! "!!!" !"!!!!" "!!!" ! !"!!!"! !!!!!!!!!!!!!!!!!!"!!!!!!!!!!"!!!!!!!!!!!!!!!!" "! ! !!!!!! !" "! ! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! "! ! ! !"! ! !! !"!! ! !! !! !!!!"! ! " !"! !!"!!" "!" !!! !! !!!!" ! ! !!"!" ! "!" !!"! ! " ! ! ! !! !!! ! !! !!" ! "!! "!!! !!!! !"!"! "!"!!!"! !!!!!!! !!"! !! "!!! !!!!!!! ! !"! !!!! !"!"! ! ! ! !!!!!"!"!! " ! "! ! !"!"! !!! !!!!!! !"! !!" !!!!" !! ! " !!!!! ! !! " !!! !"!!!!!!!"!!"!"! "!!! ! "!" "!"!!!! "!"! !!"!!!!"!"!" ! !!!!!!!" !!!"!!!!!"!""!!!!!!! "!" ! !!!! ! !!!!!!!"! ! ! !!!!! ! !!"!!!!!!!!!"!"!!!! "!!!!!!!"!!"!! !!!! !!"!!!!!! ""!"!!" !!!!!"!!!!!!" !!! "!!!!!" !!! !!!!! "!!!"!!!"!!!!!!!!" ! !!!!! "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"!!!!"!! !"!!!!! !!!"!!!"! !!!!"!!"!!!!!" !! !"! !! !!!" !!!!"!!!!! "!!!! "!"!! !"!! !!! !!!!"!!"!!!" !"!!! ! ! !!"!!" !! ! "!! !!" !!!" "! "! !!!" !!! !! !!!"!! !!"!" !!"" ! !!!!"! !! "!!"!!!"! "!!!" ! !!!"!!"!!"! !!! ! !!!!!!! " ! !! " "! !! !! !!!" !!!! ! ! !!! " !!! ! !" ! "!" ! "!!! "!"!!"!"!" !"!!"!!!"!"!!!"! !"!!"! "!! !!! ! !! ! !!" !!! ! !!!!! !! !" !! ! !!!!! ! ! "!!! !"!" !"!!" !!!"!"!!!"!!!!!"! !!!!"!" !"! "!" ! !"!!!"!"! !!! ! !!!! !"!"!!"! "!!" !"! "!"!!! ! !!!"! ! !" !" !! ! ! !!!"!"!! " !!"!"! "!! ! !!! "! ! !"!!" ! "!!!! "! !!"!! ! !!!!!!!!! ! !"! !! !" !!! "!" !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !! "!"!"!!! !!"!" ! ! !!!!! ! !!!"!! ! !"!"!!!!!! !"!!! "!! !!!" !!! "! ! !!! !!!" " !"!!!!!" !!!!"!!!" "! !!!"!!!" !!!!!! !! "!!!"!!!"!" ! !!!!!! "!!!!! !!"!"! !!!"!!! " !"!!!!! ! !!!"! ! !!!!! !! ! !!!!! " !!!"!" !!!!!"!!!!!!"! !" !!!!!" " !" !!!" "!!! "!!!" !!!!!!"! !"!" !!!!!! !!!" " "!!!" ! !"!!!!!! !"!! !!!!!!!! !!!! ! !"!!!!!! !!"!"!!!!!!"!!!!!!" "!!!!!! "!!" !" "!!!!!!" !!! !!! " !" !!!!!! ! " "!"!!!!! !!!!!!! +endstream +endobj +9 0 obj +63847 +endobj +10 0 obj +/DeviceRGB +endobj +11 0 obj +<< +/Filter [ /RunLengthDecode ] +/Width 106 +/Height 91 +/ColorSpace 10 0 R +/BitsPerComponent 8 +/Length 12 0 R +>> +stream +^!xm!6DKx SS!Yo@o vv @n uu ~@o vv @o vv @o)tv|>{}x,sv >{{-sv @ovns+tvpzns}@o vv @o vv @o vv @o vv BcvvsER|ww`x@o vv @n vv @o vv @p vv @n$uv?wrwp(uu(yvsy>~|~(ut/{~?rx#uu#~@nvv~@p vv @o vv @o uv Bq vr }DF@A?.BA@BI@A? /IEFGFHY" !  !Yuopsorf>rpk$.IEFI:nxvu|A!rwvxvwqc[ vv  uu ] vv }> vv vv vJ vv Zyzh vv \^ vv \n vv W| vv vv! vv vs- voA vo@ voA voA voAltx voAp& voAfu{ voAqYsk voAci4 voA voA" voAE uo@us voAYuoqsopoql B|uvwol8CA@D*1sospot[\tmcDF@AB:1C@A?# "hqonu@>BAB.?A@DCD.Bq!qv @o vu @o vv @o!vv @o`g]+tvah]>{|,st+{>{v.st,y @o~*tv ~@o vv @o vv @o vv @o vv @o vv FMywwtqAlvv@o vv @o vv @o vv @o#uv @pf|tbz,sv!fyub >}|,st-|?x~y/su){ @n'tv@p vv @o vv @n vv Bq"xx"?/xnoplMqoqj%D@AB=9CA@D$jqoqMlponvK@A@AF@ALeqnopjhponplNA@FA@?Cg +endstream +endobj +12 0 obj +8751 +endobj +13 0 obj +endobj +14 0 obj +8751 +endobj +15 0 obj +<< +>> +endobj +16 0 obj +8751 +endobj +17 0 obj +<< +/Title (fig-2) +/CreationDate (D:20130623174624) +/ModDate (D:20130623174624) +/Producer (ImageMagick 6.7.7-10 2013-02-25 Q16 http://www.imagemagick.org) +>> +endobj +xref +0 18 +0000000000 65535 f +0000000010 00000 n +0000000059 00000 n +0000000118 00000 n +0000000300 00000 n +0000000383 00000 n +0000000401 00000 n +0000000439 00000 n +0000000460 00000 n +0000064493 00000 n +0000064514 00000 n +0000064541 00000 n +0000073437 00000 n +0000073458 00000 n +0000073474 00000 n +0000073495 00000 n +0000073517 00000 n +0000073538 00000 n +trailer +<< +/Size 18 +/Info 17 0 R +/Root 1 0 R +>> +startxref +73711 +%%EOF diff --git a/021_VideoCoding/Figures/HEVC/Slices/WPP.pdf b/021_VideoCoding/Figures/HEVC/Slices/WPP.pdf new file mode 100644 index 0000000..803fce9 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/Slices/WPP.pdf @@ -0,0 +1,135 @@ +%PDF-1.3 +1 0 obj +<< +/Pages 2 0 R +/Type /Catalog +>> +endobj +2 0 obj +<< +/Type /Pages +/Kids [ 3 0 R ] +/Count 1 +>> +endobj +3 0 obj +<< +/Type /Page +/Parent 2 0 R +/Resources << +/XObject << /Im0 8 0 R >> +/ProcSet 6 0 R >> +/MediaBox [0 0 1084 666] +/CropBox [0 0 1084 666] +/Contents 4 0 R +/Thumb 11 0 R +>> +endobj +4 0 obj +<< +/Length 5 0 R +>> +stream +q +1084 0 0 666 0 0 cm +/Im0 Do +Q +endstream +endobj +5 0 obj +32 +endobj +6 0 obj +[ /PDF /Text /ImageC ] +endobj +7 0 obj +<< +>> +endobj +8 0 obj +<< +/Type /XObject +/Subtype /Image +/Name /Im0 +/Filter [ /RunLengthDecode ] +/Width 1084 +/Height 666 +/ColorSpace 10 0 R +/BitsPerComponent 8 +/Length 9 0 R +>> +stream + !"!!!!!!!!!!!!!!!!!!"!!!"!!! "! !"!!!!! "!"!"! !!!!! "!"! !" !!! "! !!!! !!!"! !!!" "!!!" !" "!!!" "!!!" ! !"!!!"! !!!!! !! !!!! !" !"! ! !"!!!!!! !"!! !!!! " ! ! !!! !! "" !!"! !"! " ! " !!! !!! !"! !"! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !"! ! !"!! ! !!!!!!!!!"! "!!!!!!!!!!!"!! "! " ! !!!"!!!!!!!!!" !!! !" ! !!! !!!!!!!!!"! !"! !!"!!!!!!!!" !!!"!!" !!!!!!!!!" ! !"!!!!! ! !!!!!!!!! !" !"!! "!!!!"!!!!!!!! "! !"!!! !!!!"! "!!!"! ! "! !!"! !! !"! ! ! !! !"! ! ! !!! ! !"!" !! ! !! !! !! !! !"! !"! !!"! !"!"!!!!! "!! !! !"!"!""!! !!!! !!"!"! "!!!! ! ! !!" ! "!!!! ! ! !!" ! "!!!!!!" !"!"!!!!!"!"!"! !!!!! "!!" !"!!!!!!!! ! "!" !" !"! !! !" ! ! ! ! ! !!" !!! ! "! !! ! "!"!" !!!! " !" !! ! !!!!! " !" !! ! !!!!! ! !"!"!" ! !!!! ! " ! !!"! !"!!!!!"!"! !! "! ! !"!" !!!! " !!!!" !"!!!"!"!!"!"! " ! !"! ! "!"! "!"! ! !!"!"!! "!! ! "!! " ! " !" ! ! !!!!"!" !"!! !"!"! !!!!!"!" !"!! !"!"! !!!!!" "!" " !"!! ! "! " ! " !!!! "! !"!"!"! !" " ! !"!!!!!"! !"!"!"!! ! !!!! ! "! !!! ! "! " ! "!! "! ! !"!"!!"!"! !! ! !" !"!!! ! !!"!" ! !!!"! !! !!"!!!!"! !! !!"!!!! !" ! ! !!!!"! "!! "!!!!! ! !"!!"!!!!!"!" !!!!" " !! !"!! ! "! ! ! ! !!" ! !"!!"! !" ! "!! "!! !"!!"!" !""!!!!"!!" " !!!!!"!!" " !!!!"!""!"! !!! "! ! ! !"! "!" !!!!"!"! !" !!!!! !" ! !"!!! !" !" " "!"!" !!" ! !"!! ! !" "! !"! " !!!" "!!"!"! !!!!"! !" !"!"!!!!"! !" !"!"!!!! !"!"!!"!!!!" !! !"!!!! ! "!" " "!!!! ! !" !"!!!"! !! ! !! ! ! !! "!"!"!"!!!! !" !! ! ! !!!! ! !! !!!!! ! !! !!!! !! ! !!!! " !! !!!"!!"!!!!! " ! ! !!!! " !! "!"!!!!!!!!! !! !" !!!!!!!!!!!!!!!!!"!!!! !"! !"!!!!"!! !!!!!"! !!!! !!!"!!!!!!!!"!!!!!!!!!!!!!!!!!! !"!!!!!"!!!!!!!!!!! !"! !!!" !"!!"! "!!" !! "!!!! !!!!! ! " !!!! ! " !!!! " !!!!! !!!!!"! !!!!!"! !" !!!!"!!!!"!"! !" "!!!!"! "!!!!! !!!!! !!!!!"!!!!!!!!!! !" !!!!! !"!"!!! !!! !! ! ! "!" !""!!! "!!!!!!"!!!!!"!!!!" ! !!!!!"!!!!"! !!!!!"!" ! "! !" !!!! " !! " !! ! " "! " " !" "!!!!! !!!!!! !!!!! !!!!! !!!!!!!!!! "!!!!! !" !!"! "! ! !!! !"!!!"!"!! " "!"!!!!"!!!!!"! !!!!"! !!!!!"!!!!"!!!!!!" "!!!! ! # !!"!" " !" ! !!!" !!! !! "!"! "! "!"!"! !!!!!"!!!!!!!!!!!!!!!!"! !!!!!!!!!"! !!!! !!" !"!"! "!!!!" !!! !! " !"!" !! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"!" !"!!!!!!" "!!"!"!"!"!!"!!!!"!!" !! !!!"!!!!!"!!!!! !!!!!!!"!!"!!! "!!!!"!"!"!!!!!!! ! !"! " "!! !""!!! ! "! !!! !! !!" !! !! !!" !!" !"!"!! !! "!! !"!" !"!!!" !"! ! !! !"! " ! !"!!!!!!!!" !"!"! !! !!!!"!!!" "! !"!"!"!! !!"!"!"!! !!! !!!!"!!!!!!"!!"! "!!!!! ! # !"!" ! ! !" !"!!!!!!!" !"! !" !"! !!! !"!!"!!!"! !!! ! !! !"! !!! ! !! !"! !!"! !"!" "!!"!! "!! "!!!"! "! !!!! !" !! "! !" ! !"!!!!!"!"! "!"!! !!" "" !!!!"!"!!! !"!"!!!" ! "! !! "! "!!" ! "! !! "! "!!! !!" " !! "!! !!! "! !!" "! !! !!"!!! !"!"!" !!!!!!!"!! !!" !! !!""!!!!!!!"!!" !!!! !!"!" !!!! !!"!!"!"!"!!! !!" "!! !" "!"!! !"!! !!"!!! ! ! ! "! !!!!! !! " ""!"!!"! !" !! "!!!! " !" !!! !!!"!"!!!" !! !!"!"!!!" !! !!"! !!!"! " "!! "!! !!"!"!!"! !!" "!"!" !!!" ! " !!!!!!!!" "!! ! "! !!!!!"!! "!!!"! !!"!!!!! "!!"!!!!! "!!!"!"!!!!!! ! !!!" !!!"! !!!"!!! ! !!! !! !! !"!!! !" !! ! ! " ! !!!!"!!"!"!!"!! !!!!" "!!!!"!!!!" "!!!!"!! "! ! "!! "!" !"!!!"!!" !!!!!! !!! !! ! !!!"!"!! !" !! "! !" ! !" ! "!! "!" !!" "!"!"!! !!!"! "!"! ! ! !"! !! !" !"! " ! ! ! !" !! !" !"! " ! ! ! !" !!"!"! "!" ! !" "! "! !"!"!! "!"! "! !! ! !" !"! "!!" " !" !!!!" ! "!" !"!!!!!! "! !"!"!"!! "! !" ! !!"!"!"!" "!" !"! ""!"! !"! !" ! !"!!"! " !!!"! ! "! !""!"! "! " !!"! ! "! !""!"! "! " ! !"!!" !"! ! !"!!"!"! " !!"! !"! !"!!!" ! "! " !! ! "!" " !!! !"!"!!! !! !! !!" !! !! "!"! ! "!! "! !" !" !!! ! "! " !"!! !" !" !!!" ! ! "! !!"!!!" ! ! "! !!"!!! "!!"! !"!" "!"! !" !" ! "!!"!"!" "!"!!! "!"!"" " !!" ! ! !"!!!" ! ! ! !!"!!""!!"! "!"!"!"!"!! "!"! !! !!" !!"! "!!! "! ! !!! "! ! !!!"!!! " !!"!!"! !!! !"!"!"!"!"! !!"! "!"!!!!!!!!!!" !!!!!!!!!! !!!!!!!!!!!!!!!!!!"!!!!!!!!!! "!" !!!!!!!!! !"!"!!!!!!!!!! ! ! !!!!!!!!!!"!"!"!!!!!!!!!!" !!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!""!" !! !!!!"!! ! " !!!!!" !!!"!"!!!" "!"!!! "!"!" "! "!!!"!#!!! " !!! ! "!!! !!!" ! ! "!!! "!!!" ! ! !!!"!"! !!!"! !"!"!" !"!!!!!" "!" !"!!! " ! !!! ! !!!"!!! "!"!!! ! "!! !"!!"!!"! "!"!!!!! ! !!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"! !!!!!!!"!"!"!! ! !" ! " " !!!!!!"! ! "!!!!!! " "! !"!!!!!!"!"! !! !!!!"! !!!!!!! ! " "! !!! !! !!! !!!!!!!"!"! ! "! !!! "!!"!" !! !!!!!!! !!! ! !!! !"!" !!!!! !" ! !!!" " "! "!"!" !!!!!!"! "! " !" !!!!" !!! "! !"!!!!!!!"! !"! !"!! !!"!" !! "!!!" ! ! "!"!!!"! ! "! ! ! !! ! "! ! ! !! " " !!" !!!!! "! "!!!!"!! "!" !" !!"!!! !"!!!"!"! !!!" !!" !!!"!!! "!!!!!! "!!!!!! " "! !! "!!!!"! ! ! "!"! !!!!"!"! !" !" "! !"! ! !"!"!"!"! " ! "!! "!"! ! !!" ! !" !"!!! ! "!! !"! "! !!!!!! !"! "! !!!!!!"! !!!! !" ! " ! "! !!!! !!"!" !"! !!!" "! !"!"!! !!"!!!! !"!!"!"!!! !!"!!!"!"!"!" !!!!!"!"!"!" !!!!"! ! ! !!!!"! ! "! ! "! !!! ! ! ! " !" !! ! " ! !"! ! "!" ! "! "! !!!! !!!!!!"! !! !!!!"! !! !!!!! !!!!!! !!! !" !! !! ! "!!" "!! ! !"!"! !"!" !" !!"! !"! !!"!!"! !!!!"! !!" ! !!!!"! !!" ! !!!!!"! !!!!!!"! !!!" !" ! !"!" !!"!"! ! !! !" !! !"!!"!!! "!"! !"" !"!!!!! "! !!!!! "! !!!!!" !! "!!!!! ! " !" !!! "! !"! !" "! !!!" "!!" ! !"!"!!" "!!! !"!" !!!!!"! !!!!!!"! !!!!!!"!!" " !!!! ! !! ! !"! !"!!!"!"!!" ! !" "!!" !! ! ! ! ! !! !" !"!!!!!"!"!!!!!!"!"!!!!!!!"! !!!! ! ! "! "!!!! "!! "!! " !"!"!"! !"!"!"!! !!!" !!!!!"!!!!!"!!!!!"!!!!!! " " !! " "! ! "!"! !!!! !!" ! "! "! ! !! !! !"!!!! !"!!!!! !"!!!!!!!!!!"! " ! !" "! " ! "! !!!!!"!! !"!!! "! !! !!!!!!!!!!!!!!!!!!"!!!!!! ! ! "! !!!!!! !"!!!!!!!!!!!!!!!!!!!!!!"!!!!!!"!!!!!! !"!"! " !! !!!! "!!"! !!!!!!!!!!!!!!!" !!!! !!"! " ! ! !!!!!!! "!! !!!!"!!"! !!!!"!!!!!"!!!!!! "!!!! ! # !"!""! ! !" !"!!!!!!"!"! "!"! !!!# ! !!! !!!!!!!!!!!!!!!"!!!!! !" !" ! "! !" ! !"!!!!!!"!"!"! ! "!!!!! !! "!!!!!! !!!!!! !!!!!! !!!!!"!" !!!!!!"! !"!" !!"!!!!"!!"!!!!!!"!!!!!!"!!!!!!"!"!!!!! ! "! !!!!!!!"! " !!! !" !! !"!!!! ! !!!! ! !!!!!!!!!"! "! " !!!!!!! !!!! !!"!"!!!!!! "!!!!!!! "!!!!!!! !!!!!!"!!!!!!!! !!! !!!!!!! !"!!!!!!"!!!!!!!"!!!!"!""!"!!"!! !"!!"!"!!!!!!"!! !!"! !"!"! !! !! ! !! "!" !!! !! !!! "!!! !! !!! "!!!!" ! ! " !"! !! " ! !"!!!!!!!" !" !! !!" "!! ! !!!" !"!!" ! !!!"!! "!!!!!!"!! "!!!!! "!"! !!! !!" !!! ! !" !"!!!!!!!"!!"!!# !"!" "!"!"!! ! "!"!!!"!! !!!"! !" !!! ! !!!"! !" !!! ! !!!# "!"!! !"!! ! !! !" ! !"!!!!!! "! !!! ! "! ! ! !!!!"!!! "!"!! " !!"! !!!"!!! " !!"! !!!"!!!!" " !"!!!" !!!!"!" !!!!!!! !"!! ! ! ! !"!! !"!" !" !"! !"! ! !"! !"! !!" " "!!" !"!" ! ! !! !"!!" !"!" ! ! !! !"!!! "! !! !!" !!! ! "! !!!!!! "!" ! " ! ! "! " !! ! " !! "!" !"!!!" ! !"! " "!! ! !"!"!!! "!"! ! ! !!!" ! !" !"!!! "!"! ! ! !!!" ! !" !"!!! !"! "! !!!"!"!"!" # !!! " !!!!!!"!"!"!"!#!!"!!! "!"!! !!"!!!"!! !!!"!! !!!"! ! ! !!! "! ! ! !!!!!!!!! ! !! ! "!!! !! !!! !! !"!" !"! ! !! !"!"! ! !! !"!"! !!"! "!"!"!!! ! !!!"!"!!! !! !!"! ! ! !!"!" !! "! ! !! ! !"! " !!"!"!!!"!"!! " !"!"! ! !"! ! " ! !"!!!!!!!! ! !" !"!!!!!!! !" ! !"!!!!!!"!" !!!!!!!! "! !!!!!!! " !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!""!!!!! !! !!! "" " "!"!"!!!" ! "!" !"!"!!!!"!!"!"! !!"! !!! !!!"!"! !!!" !!! "!!! !!! ! !" !!!"! !!!" !!!"!" !!!!! "!!!" !!!" !"!!!" !!! !!! !!! !!! "! "!!"!" !""!!!!"!!!!! !!!! !!"!!! !" !! !!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ! ! ! ! ! !"!!!!! "! !" !! !! ! " !!!!!!"!""!! !!!!! !"!"!"! !!! !!!!!! "! !!!"!"! !!!!!!"!!"! !"!!! ! "!" ! !!! !" !!!"!!! "!"!!!!"! ! !"!"! "!!!! " "!!!!! "!!! !! !!"!"!!!!! "!"! !!"!! !"! !"!!!" !"!! !!!!" ! "! !! " !"!!" ! !!!"! ! "!! ! ! !"! !! !!!! !!!!! ! "! !!!#! !!"!!!" "! !" !" " " !"!"!!"!!!!!!!! " ! "!"!"! !" !" ! !" !! ! !"! "!"!" # " ! " "! !! ! "!!" !" !" !" ! !" !" "!!!" ! "!"! " ! " !"!"" !! "!! !!!!"! !"!"!! !"!! !! ! " "! "! !" ! !!" !"!"!!"!!!"!"!" ! "!" ! " !!"!! !!!"! !" "! " " ! " " !" "!!"!!"!!! " !" !!!" ! ! !"!! !"!" !"!"!! "!!" ! "!" !"!!"!!! " "!"! "!"! ! " !"! !" "!! !!" !!" !"!" ! "!"!" !!!!!"! !"!!!! !!"!! "!!" !! !"!" "" !! ! !!!"! ! ! " !!"!" !"!!"!!!!" ! " ! ! " " !!""!!! "!!!!" "!!! !"!" !! ! !"!" "!!"!!"!"!!!" !! !"! ! ! !!!"!"! "! ! !! !! ! "!"! !!! !"!" !!" "!! !!"!!!" !!! !!!" !# !!"!!!" !!! "!!"!!! !!! " ! "! !!!!"!"!!!"!!"!!!! ! !!! "!"!"!! ! !!""!!!!!! ! "!"! !!!!!!!!!!!! ! ! !!!"!!!!!!"!! !!!!!!"!!!!!!!"! !!"! !!!" !" " !"!!!!!! ! "!"!!!! ! "! !!!!!! !"! ! ""!!!"!" "!! !!!!!!!"!!!!!! !" !!" !!!!!!! ! !!" !"!!!"!!! "!!!!! "!!!!!"! !" !!"! "!"!!!!!!! "! !!"!!!" "!!!!!!!!!!!!" !" ! "!" ! ! ! !!!!!!! !" ! !! !"! " !"!!"!"!"!!"!!!!!!!!! "!"!"!""! !"!"!!!!!!!"!!"!!!!! ! !!!!! !!!!! ! !!"!" ! "!" !!!!!!!!"!!!!! ! !!!!!!!"!"!!!!!"!!!!!!!"!! !!" !!! !!! !!!! "!!!!!"!!!!"!!!!!!! !! !! !!! !!" !!!!"!!! !!!" !"!! ! !!"!" ! "!" !!!!!!!"!"!! !"!!" !!"!! ! !! !!!"!"!! !!! "!!! ""!! "!"!"!""! !"!"!!!!!!! "! "! "!! ! !! !" ! ! !!!!!"!"! !!!"! !!" !" ! "!" ! ! ! !!!!!!!! !"!!" ! !! "! ! "!!! "!!! !!"!!!" " !! ! "!"! !" !!"! "!"!!!!!! !""!"!! " !""! "! "!!" ! ! !"! !!!!"!!!"!! !! "!"!!!"!"!"!! !" !!" !!!!!!! " "!!"! !!"!!" ! "! !"! !"!"! ! !"!" !!! " "! "!! ! ! "! !"!"! !!!! " ! ! "! !!!!!! !" "!! ! ! ! !! !" !"! !!! ! ! "!!!!" !! "!!!! !!!! !!"!!"!"!!"!""!"!"! !"!"! !!"!" ! !!"!"!!!! !! !! !!!!""!!!"!" !!"!!! "" !! "! ! ! ! !" ! ! ! !! !! " !"! ! !! !"!"!! !!" ! !! !!! "!!! !! !!!!" ! "!" !!! !! ! !!"!! ! !!" !! !"! ! ! !"!!!""!"!!"" !!"!!!"!"!" "! !! " !! !"!! !"!"!!!!" !"!" !"!!"! "! ! " ! !!!" ! "!"!" ! ! "!"! !"!"!"!!" ! ! ! !!! " !" !" "! ! !! !"! "!!! "! "! !"! !"!!!"! "!"!!!! " !" !"!!"!!"!!! "! " !"!"!!!" !!!"! " !" " " ! "!! "! " ! ! !!! !!!!!!!!!! ! "!"! !!!!" ! "! !!!!! "!"! !!!!!! "!!!!!"! ! !"!!!! !" !!!!!!"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"! !!!" " " "!!! !!!" "!!!"! !!!" "!!! !!!!! "! !!!" "! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! "!"! !! !"!!!" !!!!!!!!!!!! !! !!!! "!"! !! !"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!""!" !" !!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!! !"!"!!!!!!!!!!!!!!!!!! "! +endstream +endobj +9 0 obj +73491 +endobj +10 0 obj +/DeviceRGB +endobj +11 0 obj +<< +/Filter [ /RunLengthDecode ] +/Width 106 +/Height 65 +/ColorSpace 10 0 R +/BitsPerComponent 8 +/Length 12 0 R +>> +stream +ufn!^#+)(#!{l E{>LJLB'AFrQy;-)(D<ނ߮ +endstream +endobj +12 0 obj +4992 +endobj +13 0 obj +endobj +14 0 obj +4992 +endobj +15 0 obj +<< +>> +endobj +16 0 obj +4992 +endobj +17 0 obj +<< +/Title (fig-3) +/CreationDate (D:20130623174628) +/ModDate (D:20130623174628) +/Producer (ImageMagick 6.7.7-10 2013-02-25 Q16 http://www.imagemagick.org) +>> +endobj +xref +0 18 +0000000000 65535 f +0000000010 00000 n +0000000059 00000 n +0000000118 00000 n +0000000302 00000 n +0000000386 00000 n +0000000404 00000 n +0000000442 00000 n +0000000463 00000 n +0000074141 00000 n +0000074162 00000 n +0000074189 00000 n +0000079326 00000 n +0000079347 00000 n +0000079363 00000 n +0000079384 00000 n +0000079406 00000 n +0000079427 00000 n +trailer +<< +/Size 18 +/Info 17 0 R +/Root 1 0 R +>> +startxref +79600 +%%EOF diff --git a/021_VideoCoding/Figures/HEVC/Tiles_Slices_Example_1.tikz b/021_VideoCoding/Figures/HEVC/Tiles_Slices_Example_1.tikz new file mode 100644 index 0000000..07331bd --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/Tiles_Slices_Example_1.tikz @@ -0,0 +1,51 @@ +% \usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +% decorations.pathreplacing,decorations.pathmorphing,shapes,% +% matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\newcommand{\scale}{.65} + +\begin{tikzpicture}[auto, on grid, scale=\scale,every node/.style={scale=\scale} ] + +\tikzstyle{CTU}=[draw, text centered, minimum size=1cm,anchor=south west] +\tikzstyle{Tile}=[draw, line width=1.5pt] +\tikzstyle{TileLabel}=[text centered,fill=white,inner sep=0,font=\large] + +% \newcounter{maincounter} + +\setcounter{maincounter}{1} +\foreach \x in {1,4,7} +{ + \node[TileLabel] at (\x + .5,1.5cm) {Tile \#\themaincounter}; + \addtocounter{maincounter}{1} +} + +\setcounter{maincounter}{1} +\foreach \y in {0,-1,...,-8} +\foreach \x in {0,1,2} +{ + \node[CTU] at (\x,\y) {\themaincounter}; + \addtocounter{maincounter}{1} +} +\foreach \y in {0,-1,...,-8} +\foreach \x in {3,4,5} +{ + \node[CTU] at (\x,\y) {\themaincounter}; + \addtocounter{maincounter}{1} +} +\foreach \y in {0,-1,...,-8} +\foreach \x in {6,7,8} +{ + \node[CTU] at (\x,\y) {\themaincounter}; + \addtocounter{maincounter}{1} +} + +\path[Tile] (3,1) -- ++(0,-9); +\path[Tile] (6,1) -- ++(0,-9); + +\draw[draw=red,densely dashed,black!70!green] (-.1,1+.1) rectangle (9.1,-8-.1); +\node[black!70!green,TileLabel] at (9/2,-8.5) {Slice \#1}; + +\draw[white] (-.4,2) rectangle (9.4,-9); + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/Tiles_Slices_Example_2.tikz b/021_VideoCoding/Figures/HEVC/Tiles_Slices_Example_2.tikz new file mode 100644 index 0000000..e943389 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/Tiles_Slices_Example_2.tikz @@ -0,0 +1,81 @@ +% \usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +% decorations.pathreplacing,decorations.pathmorphing,shapes,% +% matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\newcommand{\scale}{.65} + +\begin{tikzpicture}[auto, on grid, scale=\scale,every node/.style={scale=\scale} ] + +\tikzstyle{CTU}=[draw, text centered, minimum size=1cm,anchor=south west] +\tikzstyle{Tile}=[draw, line width=1.5pt] +\tikzstyle{TileLabel}=[text centered,fill=white,inner sep=2pt,font=\large] + +\tikzstyle{SlicePattern}=[pattern color=gray!70] +\tikzstyle{Slice1}=[SlicePattern, pattern=north west lines] +\tikzstyle{Slice2}=[SlicePattern, pattern=dots] +\tikzstyle{Slice3}=[SlicePattern, pattern=bricks] +\tikzstyle{Slice4}=[SlicePattern, pattern=crosshatch, step=1pt] +\tikzstyle{Slice5}=[SlicePattern, pattern=north east lines] + +% \newcounter{maincounter} + +\setcounter{maincounter}{1} +\foreach \x in {1,4,7} +{ + \node[TileLabel] at (\x + .5,1.5cm) {Tile \#\themaincounter}; + \addtocounter{maincounter}{1} +} + +\setcounter{maincounter}{1} +\foreach \y in {0,-1,...,-8} +\foreach \x in {0,1,2} +{ + \ifthenelse{\themaincounter < 14} + { + \node[CTU] at (\x,\y) {\themaincounter}; + } + { + \node[CTU,Slice1] at (\x,\y) {\themaincounter}; + } + \addtocounter{maincounter}{1} +} +\foreach \y in {0,-1,...,-8} +\foreach \x in {3,4,5} +{ + \ifthenelse{\themaincounter < 44} + { + \node[CTU,Slice2] at (\x,\y) {\themaincounter}; + } + { + \node[CTU,Slice3] at (\x,\y) {\themaincounter}; + } + \addtocounter{maincounter}{1} +} +\foreach \y in {0,-1,...,-8} +\foreach \x in {6,7,8} +{ + \ifthenelse{\themaincounter < 69} + { + \node[CTU,Slice4] at (\x,\y) {\themaincounter}; + } + { + \node[CTU,Slice5] at (\x,\y) {\themaincounter}; + } + \addtocounter{maincounter}{1} +} +\path[Tile] (3,1) -- ++(0,-9); +\path[Tile] (6,1) -- ++(0,-9); + +\node[black!70!green,TileLabel] at (.5*9/3,-1) {Slice \#1}; +\node[black!70!green,TileLabel] at (.5*9/3,-6) {Slice \#2}; +\node[black!70!green,TileLabel] at (1.5*9/3,-2) {Slice \#3}; +\node[black!70!green,TileLabel] at (1.5*9/3,-7) {Slice \#4}; +\node[black!70!green,TileLabel] at (2.5*9/3,-1) {Slice \#5}; +\node[black!70!green,TileLabel] at (2.5*9/3,-6) {Slice \#6}; + + + +\draw[white] (-.4,2) rectangle (9.4,-9); + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/WPP.pdf b/021_VideoCoding/Figures/HEVC/WPP.pdf new file mode 100644 index 0000000..803fce9 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/WPP.pdf @@ -0,0 +1,135 @@ +%PDF-1.3 +1 0 obj +<< +/Pages 2 0 R +/Type /Catalog +>> +endobj +2 0 obj +<< +/Type /Pages +/Kids [ 3 0 R ] +/Count 1 +>> +endobj +3 0 obj +<< +/Type /Page +/Parent 2 0 R +/Resources << +/XObject << /Im0 8 0 R >> +/ProcSet 6 0 R >> +/MediaBox [0 0 1084 666] +/CropBox [0 0 1084 666] +/Contents 4 0 R +/Thumb 11 0 R +>> +endobj +4 0 obj +<< +/Length 5 0 R +>> +stream +q +1084 0 0 666 0 0 cm +/Im0 Do +Q +endstream +endobj +5 0 obj +32 +endobj +6 0 obj +[ /PDF /Text /ImageC ] +endobj +7 0 obj +<< +>> +endobj +8 0 obj +<< +/Type /XObject +/Subtype /Image +/Name /Im0 +/Filter [ /RunLengthDecode ] +/Width 1084 +/Height 666 +/ColorSpace 10 0 R +/BitsPerComponent 8 +/Length 9 0 R +>> +stream + !"!!!!!!!!!!!!!!!!!!"!!!"!!! "! !"!!!!! "!"!"! !!!!! "!"! !" !!! "! !!!! !!!"! !!!" "!!!" !" "!!!" "!!!" ! !"!!!"! !!!!! !! !!!! !" !"! ! !"!!!!!! !"!! !!!! " ! ! !!! !! "" !!"! !"! " ! " !!! !!! !"! !"! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !"! ! !"!! ! !!!!!!!!!"! "!!!!!!!!!!!"!! "! " ! !!!"!!!!!!!!!" !!! !" ! !!! !!!!!!!!!"! !"! !!"!!!!!!!!" !!!"!!" !!!!!!!!!" ! !"!!!!! ! !!!!!!!!! !" !"!! "!!!!"!!!!!!!! "! !"!!! !!!!"! "!!!"! ! "! !!"! !! !"! ! ! !! !"! ! ! !!! ! !"!" !! ! !! !! !! !! !"! !"! !!"! !"!"!!!!! "!! !! !"!"!""!! !!!! !!"!"! "!!!! ! ! !!" ! "!!!! ! ! !!" ! "!!!!!!" !"!"!!!!!"!"!"! !!!!! "!!" !"!!!!!!!! ! "!" !" !"! !! !" ! ! ! ! ! !!" !!! ! "! !! ! "!"!" !!!! " !" !! ! !!!!! " !" !! ! !!!!! ! !"!"!" ! !!!! ! " ! !!"! !"!!!!!"!"! !! "! ! !"!" !!!! " !!!!" !"!!!"!"!!"!"! " ! !"! ! "!"! "!"! ! !!"!"!! "!! ! "!! " ! " !" ! ! !!!!"!" !"!! !"!"! !!!!!"!" !"!! !"!"! !!!!!" "!" " !"!! ! "! " ! " !!!! "! !"!"!"! !" " ! !"!!!!!"! !"!"!"!! ! !!!! ! "! !!! ! "! " ! "!! "! ! !"!"!!"!"! !! ! !" !"!!! ! !!"!" ! !!!"! !! !!"!!!!"! !! !!"!!!! !" ! ! !!!!"! "!! "!!!!! ! !"!!"!!!!!"!" !!!!" " !! !"!! ! "! ! ! ! !!" ! !"!!"! !" ! "!! "!! !"!!"!" !""!!!!"!!" " !!!!!"!!" " !!!!"!""!"! !!! "! ! ! !"! "!" !!!!"!"! !" !!!!! !" ! !"!!! !" !" " "!"!" !!" ! !"!! ! !" "! !"! " !!!" "!!"!"! !!!!"! !" !"!"!!!!"! !" !"!"!!!! !"!"!!"!!!!" !! !"!!!! ! "!" " "!!!! ! !" !"!!!"! !! ! !! ! ! !! "!"!"!"!!!! !" !! ! ! !!!! ! !! !!!!! ! !! !!!! !! ! !!!! " !! !!!"!!"!!!!! " ! ! !!!! " !! "!"!!!!!!!!! !! !" !!!!!!!!!!!!!!!!!"!!!! !"! !"!!!!"!! !!!!!"! !!!! !!!"!!!!!!!!"!!!!!!!!!!!!!!!!!! !"!!!!!"!!!!!!!!!!! !"! !!!" !"!!"! "!!" !! "!!!! !!!!! ! " !!!! ! " !!!! " !!!!! !!!!!"! !!!!!"! !" !!!!"!!!!"!"! !" "!!!!"! "!!!!! !!!!! !!!!!"!!!!!!!!!! !" !!!!! !"!"!!! !!! !! ! ! "!" !""!!! "!!!!!!"!!!!!"!!!!" ! !!!!!"!!!!"! !!!!!"!" ! "! !" !!!! " !! " !! ! " "! " " !" "!!!!! !!!!!! !!!!! !!!!! !!!!!!!!!! "!!!!! !" !!"! "! ! !!! !"!!!"!"!! " "!"!!!!"!!!!!"! !!!!"! !!!!!"!!!!"!!!!!!" "!!!! ! # !!"!" " !" ! !!!" !!! !! "!"! "! "!"!"! !!!!!"!!!!!!!!!!!!!!!!"! !!!!!!!!!"! !!!! !!" !"!"! "!!!!" !!! !! " !"!" !! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"!" !"!!!!!!" "!!"!"!"!"!!"!!!!"!!" !! !!!"!!!!!"!!!!! !!!!!!!"!!"!!! "!!!!"!"!"!!!!!!! ! !"! " "!! !""!!! ! "! !!! !! !!" !! !! !!" !!" !"!"!! !! "!! !"!" !"!!!" !"! ! !! !"! " ! !"!!!!!!!!" !"!"! !! !!!!"!!!" "! !"!"!"!! !!"!"!"!! !!! !!!!"!!!!!!"!!"! "!!!!! ! # !"!" ! ! !" !"!!!!!!!" !"! !" !"! !!! !"!!"!!!"! !!! ! !! !"! !!! ! !! !"! !!"! !"!" "!!"!! "!! "!!!"! "! !!!! !" !! "! !" ! !"!!!!!"!"! "!"!! !!" "" !!!!"!"!!! !"!"!!!" ! "! !! "! "!!" ! "! !! "! "!!! !!" " !! "!! !!! "! !!" "! !! !!"!!! !"!"!" !!!!!!!"!! !!" !! !!""!!!!!!!"!!" !!!! !!"!" !!!! !!"!!"!"!"!!! !!" "!! !" "!"!! !"!! !!"!!! ! ! ! "! !!!!! !! " ""!"!!"! !" !! "!!!! " !" !!! !!!"!"!!!" !! !!"!"!!!" !! !!"! !!!"! " "!! "!! !!"!"!!"! !!" "!"!" !!!" ! " !!!!!!!!" "!! ! "! !!!!!"!! "!!!"! !!"!!!!! "!!"!!!!! "!!!"!"!!!!!! ! !!!" !!!"! !!!"!!! ! !!! !! !! !"!!! !" !! ! ! " ! !!!!"!!"!"!!"!! !!!!" "!!!!"!!!!" "!!!!"!! "! ! "!! "!" !"!!!"!!" !!!!!! !!! !! ! !!!"!"!! !" !! "! !" ! !" ! "!! "!" !!" "!"!"!! !!!"! "!"! ! ! !"! !! !" !"! " ! ! ! !" !! !" !"! " ! ! ! !" !!"!"! "!" ! !" "! "! !"!"!! "!"! "! !! ! !" !"! "!!" " !" !!!!" ! "!" !"!!!!!! "! !"!"!"!! "! !" ! !!"!"!"!" "!" !"! ""!"! !"! !" ! !"!!"! " !!!"! ! "! !""!"! "! " !!"! ! "! !""!"! "! " ! !"!!" !"! ! !"!!"!"! " !!"! !"! !"!!!" ! "! " !! ! "!" " !!! !"!"!!! !! !! !!" !! !! "!"! ! "!! "! !" !" !!! ! "! " !"!! !" !" !!!" ! ! "! !!"!!!" ! ! "! !!"!!! "!!"! !"!" "!"! !" !" ! "!!"!"!" "!"!!! "!"!"" " !!" ! ! !"!!!" ! ! ! !!"!!""!!"! "!"!"!"!"!! "!"! !! !!" !!"! "!!! "! ! !!! "! ! !!!"!!! " !!"!!"! !!! !"!"!"!"!"! !!"! "!"!!!!!!!!!!" !!!!!!!!!! !!!!!!!!!!!!!!!!!!"!!!!!!!!!! "!" !!!!!!!!! !"!"!!!!!!!!!! ! ! !!!!!!!!!!"!"!"!!!!!!!!!!" !!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!""!" !! !!!!"!! ! " !!!!!" !!!"!"!!!" "!"!!! "!"!" "! "!!!"!#!!! " !!! ! "!!! !!!" ! ! "!!! "!!!" ! ! !!!"!"! !!!"! !"!"!" !"!!!!!" "!" !"!!! " ! !!! ! !!!"!!! "!"!!! ! "!! !"!!"!!"! "!"!!!!! ! !!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"! !!!!!!!"!"!"!! ! !" ! " " !!!!!!"! ! "!!!!!! " "! !"!!!!!!"!"! !! !!!!"! !!!!!!! ! " "! !!! !! !!! !!!!!!!"!"! ! "! !!! "!!"!" !! !!!!!!! !!! ! !!! !"!" !!!!! !" ! !!!" " "! "!"!" !!!!!!"! "! " !" !!!!" !!! "! !"!!!!!!!"! !"! !"!! !!"!" !! "!!!" ! ! "!"!!!"! ! "! ! ! !! ! "! ! ! !! " " !!" !!!!! "! "!!!!"!! "!" !" !!"!!! !"!!!"!"! !!!" !!" !!!"!!! "!!!!!! "!!!!!! " "! !! "!!!!"! ! ! "!"! !!!!"!"! !" !" "! !"! ! !"!"!"!"! " ! "!! "!"! ! !!" ! !" !"!!! ! "!! !"! "! !!!!!! !"! "! !!!!!!"! !!!! !" ! " ! "! !!!! !!"!" !"! !!!" "! !"!"!! !!"!!!! !"!!"!"!!! !!"!!!"!"!"!" !!!!!"!"!"!" !!!!"! ! ! !!!!"! ! "! ! "! !!! ! ! ! " !" !! ! " ! !"! ! "!" ! "! "! !!!! !!!!!!"! !! !!!!"! !! !!!!! !!!!!! !!! !" !! !! ! "!!" "!! ! !"!"! !"!" !" !!"! !"! !!"!!"! !!!!"! !!" ! !!!!"! !!" ! !!!!!"! !!!!!!"! !!!" !" ! !"!" !!"!"! ! !! !" !! !"!!"!!! "!"! !"" !"!!!!! "! !!!!! "! !!!!!" !! "!!!!! ! " !" !!! "! !"! !" "! !!!" "!!" ! !"!"!!" "!!! !"!" !!!!!"! !!!!!!"! !!!!!!"!!" " !!!! ! !! ! !"! !"!!!"!"!!" ! !" "!!" !! ! ! ! ! !! !" !"!!!!!"!"!!!!!!"!"!!!!!!!"! !!!! ! ! "! "!!!! "!! "!! " !"!"!"! !"!"!"!! !!!" !!!!!"!!!!!"!!!!!"!!!!!! " " !! " "! ! "!"! !!!! !!" ! "! "! ! !! !! !"!!!! !"!!!!! !"!!!!!!!!!!"! " ! !" "! " ! "! !!!!!"!! !"!!! "! !! !!!!!!!!!!!!!!!!!!"!!!!!! ! ! "! !!!!!! !"!!!!!!!!!!!!!!!!!!!!!!"!!!!!!"!!!!!! !"!"! " !! !!!! "!!"! !!!!!!!!!!!!!!!" !!!! !!"! " ! ! !!!!!!! "!! !!!!"!!"! !!!!"!!!!!"!!!!!! "!!!! ! # !"!""! ! !" !"!!!!!!"!"! "!"! !!!# ! !!! !!!!!!!!!!!!!!!"!!!!! !" !" ! "! !" ! !"!!!!!!"!"!"! ! "!!!!! !! "!!!!!! !!!!!! !!!!!! !!!!!"!" !!!!!!"! !"!" !!"!!!!"!!"!!!!!!"!!!!!!"!!!!!!"!"!!!!! ! "! !!!!!!!"! " !!! !" !! !"!!!! ! !!!! ! !!!!!!!!!"! "! " !!!!!!! !!!! !!"!"!!!!!! "!!!!!!! "!!!!!!! !!!!!!"!!!!!!!! !!! !!!!!!! !"!!!!!!"!!!!!!!"!!!!"!""!"!!"!! !"!!"!"!!!!!!"!! !!"! !"!"! !! !! ! !! "!" !!! !! !!! "!!! !! !!! "!!!!" ! ! " !"! !! " ! !"!!!!!!!" !" !! !!" "!! ! !!!" !"!!" ! !!!"!! "!!!!!!"!! "!!!!! "!"! !!! !!" !!! ! !" !"!!!!!!!"!!"!!# !"!" "!"!"!! ! "!"!!!"!! !!!"! !" !!! ! !!!"! !" !!! ! !!!# "!"!! !"!! ! !! !" ! !"!!!!!! "! !!! ! "! ! ! !!!!"!!! "!"!! " !!"! !!!"!!! " !!"! !!!"!!!!" " !"!!!" !!!!"!" !!!!!!! !"!! ! ! ! !"!! !"!" !" !"! !"! ! !"! !"! !!" " "!!" !"!" ! ! !! !"!!" !"!" ! ! !! !"!!! "! !! !!" !!! ! "! !!!!!! "!" ! " ! ! "! " !! ! " !! "!" !"!!!" ! !"! " "!! ! !"!"!!! "!"! ! ! !!!" ! !" !"!!! "!"! ! ! !!!" ! !" !"!!! !"! "! !!!"!"!"!" # !!! " !!!!!!"!"!"!"!#!!"!!! "!"!! !!"!!!"!! !!!"!! !!!"! ! ! !!! "! ! ! !!!!!!!!! ! !! ! "!!! !! !!! !! !"!" !"! ! !! !"!"! ! !! !"!"! !!"! "!"!"!!! ! !!!"!"!!! !! !!"! ! ! !!"!" !! "! ! !! ! !"! " !!"!"!!!"!"!! " !"!"! ! !"! ! " ! !"!!!!!!!! ! !" !"!!!!!!! !" ! !"!!!!!!"!" !!!!!!!! "! !!!!!!! " !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!""!!!!! !! !!! "" " "!"!"!!!" ! "!" !"!"!!!!"!!"!"! !!"! !!! !!!"!"! !!!" !!! "!!! !!! ! !" !!!"! !!!" !!!"!" !!!!! "!!!" !!!" !"!!!" !!! !!! !!! !!! "! "!!"!" !""!!!!"!!!!! !!!! !!"!!! !" !! !!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ! ! ! ! ! !"!!!!! "! !" !! !! ! " !!!!!!"!""!! !!!!! !"!"!"! !!! !!!!!! "! !!!"!"! !!!!!!"!!"! !"!!! ! "!" ! !!! !" !!!"!!! "!"!!!!"! ! !"!"! "!!!! " "!!!!! "!!! !! !!"!"!!!!! "!"! !!"!! !"! !"!!!" !"!! !!!!" ! "! !! " !"!!" ! !!!"! ! "!! ! ! !"! !! !!!! !!!!! ! "! !!!#! !!"!!!" "! !" !" " " !"!"!!"!!!!!!!! " ! "!"!"! !" !" ! !" !! ! !"! "!"!" # " ! " "! !! ! "!!" !" !" !" ! !" !" "!!!" ! "!"! " ! " !"!"" !! "!! !!!!"! !"!"!! !"!! !! ! " "! "! !" ! !!" !"!"!!"!!!"!"!" ! "!" ! " !!"!! !!!"! !" "! " " ! " " !" "!!"!!"!!! " !" !!!" ! ! !"!! !"!" !"!"!! "!!" ! "!" !"!!"!!! " "!"! "!"! ! " !"! !" "!! !!" !!" !"!" ! "!"!" !!!!!"! !"!!!! !!"!! "!!" !! !"!" "" !! ! !!!"! ! ! " !!"!" !"!!"!!!!" ! " ! ! " " !!""!!! "!!!!" "!!! !"!" !! ! !"!" "!!"!!"!"!!!" !! !"! ! ! !!!"!"! "! ! !! !! ! "!"! !!! !"!" !!" "!! !!"!!!" !!! !!!" !# !!"!!!" !!! "!!"!!! !!! " ! "! !!!!"!"!!!"!!"!!!! ! !!! "!"!"!! ! !!""!!!!!! ! "!"! !!!!!!!!!!!! ! ! !!!"!!!!!!"!! !!!!!!"!!!!!!!"! !!"! !!!" !" " !"!!!!!! ! "!"!!!! ! "! !!!!!! !"! ! ""!!!"!" "!! !!!!!!!"!!!!!! !" !!" !!!!!!! ! !!" !"!!!"!!! "!!!!! "!!!!!"! !" !!"! "!"!!!!!!! "! !!"!!!" "!!!!!!!!!!!!" !" ! "!" ! ! ! !!!!!!! !" ! !! !"! " !"!!"!"!"!!"!!!!!!!!! "!"!"!""! !"!"!!!!!!!"!!"!!!!! ! !!!!! !!!!! ! !!"!" ! "!" !!!!!!!!"!!!!! ! !!!!!!!"!"!!!!!"!!!!!!!"!! !!" !!! !!! !!!! "!!!!!"!!!!"!!!!!!! !! !! !!! !!" !!!!"!!! !!!" !"!! ! !!"!" ! "!" !!!!!!!"!"!! !"!!" !!"!! ! !! !!!"!"!! !!! "!!! ""!! "!"!"!""! !"!"!!!!!!! "! "! "!! ! !! !" ! ! !!!!!"!"! !!!"! !!" !" ! "!" ! ! ! !!!!!!!! !"!!" ! !! "! ! "!!! "!!! !!"!!!" " !! ! "!"! !" !!"! "!"!!!!!! !""!"!! " !""! "! "!!" ! ! !"! !!!!"!!!"!! !! "!"!!!"!"!"!! !" !!" !!!!!!! " "!!"! !!"!!" ! "! !"! !"!"! ! !"!" !!! " "! "!! ! ! "! !"!"! !!!! " ! ! "! !!!!!! !" "!! ! ! ! !! !" !"! !!! ! ! "!!!!" !! "!!!! !!!! !!"!!"!"!!"!""!"!"! !"!"! !!"!" ! !!"!"!!!! !! !! !!!!""!!!"!" !!"!!! "" !! "! ! ! ! !" ! ! ! !! !! " !"! ! !! !"!"!! !!" ! !! !!! "!!! !! !!!!" ! "!" !!! !! ! !!"!! ! !!" !! !"! ! ! !"!!!""!"!!"" !!"!!!"!"!" "! !! " !! !"!! !"!"!!!!" !"!" !"!!"! "! ! " ! !!!" ! "!"!" ! ! "!"! !"!"!"!!" ! ! ! !!! " !" !" "! ! !! !"! "!!! "! "! !"! !"!!!"! "!"!!!! " !" !"!!"!!"!!! "! " !"!"!!!" !!!"! " !" " " ! "!! "! " ! ! !!! !!!!!!!!!! ! "!"! !!!!" ! "! !!!!! "!"! !!!!!! "!!!!!"! ! !"!!!! !" !!!!!!"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"! !!!" " " "!!! !!!" "!!!"! !!!" "!!! !!!!! "! !!!" "! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! "!"! !! !"!!!" !!!!!!!!!!!! !! !!!! "!"! !! !"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!""!" !" !!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!! !"!"!!!!!!!!!!!!!!!!!! "! +endstream +endobj +9 0 obj +73491 +endobj +10 0 obj +/DeviceRGB +endobj +11 0 obj +<< +/Filter [ /RunLengthDecode ] +/Width 106 +/Height 65 +/ColorSpace 10 0 R +/BitsPerComponent 8 +/Length 12 0 R +>> +stream +ufn!^#+)(#!{l E{>LJLB'AFrQy;-)(D<ނ߮ +endstream +endobj +12 0 obj +4992 +endobj +13 0 obj +endobj +14 0 obj +4992 +endobj +15 0 obj +<< +>> +endobj +16 0 obj +4992 +endobj +17 0 obj +<< +/Title (fig-3) +/CreationDate (D:20130623174628) +/ModDate (D:20130623174628) +/Producer (ImageMagick 6.7.7-10 2013-02-25 Q16 http://www.imagemagick.org) +>> +endobj +xref +0 18 +0000000000 65535 f +0000000010 00000 n +0000000059 00000 n +0000000118 00000 n +0000000302 00000 n +0000000386 00000 n +0000000404 00000 n +0000000442 00000 n +0000000463 00000 n +0000074141 00000 n +0000074162 00000 n +0000074189 00000 n +0000079326 00000 n +0000079347 00000 n +0000079363 00000 n +0000079384 00000 n +0000079406 00000 n +0000079427 00000 n +trailer +<< +/Size 18 +/Info 17 0 R +/Root 1 0 R +>> +startxref +79600 +%%EOF diff --git a/021_VideoCoding/Figures/HEVC/WPP.tikz b/021_VideoCoding/Figures/HEVC/WPP.tikz new file mode 100644 index 0000000..f470b5d --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/WPP.tikz @@ -0,0 +1,62 @@ +% \usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +% decorations.pathreplacing,decorations.pathmorphing,shapes,% +% matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\newcommand{\scale}{.68} + +\begin{tikzpicture}[auto, on grid, scale=\scale,every node/.style={scale=\scale} ] + +\tikzstyle{CTU}=[draw,text centered, minimum size=1cm,anchor=south west] +\tikzstyle{CTUNum}=[text centered, minimum size=1cm,anchor=south west] +\tikzstyle{Tile}=[draw, line width=1.5pt] +\tikzstyle{ThreadLabel}=[text centered,fill=white,inner sep=0,font=\large] + +\draw[CTU,very thick] (0,1) rectangle ++(12,-8); + + +%\newcounter{maincounter} + +\setcounter{maincounter}{1} +\foreach \x in {0,1,...,9} +{ + \node[CTU] at (\x, 0) {\themaincounter}; \addtocounter{maincounter}{1} +} +\draw[->, >=latex, black, line width=2pt] (10,.5) -- ++(1,0); + +\setcounter{maincounter}{3} +\foreach \x in {0,1,...,7} +{ + \node[CTU] at (\x, -1) {\themaincounter}; \addtocounter{maincounter}{1} +} +\draw[->, >=latex, black, line width=2pt] (8,-.5) -- ++(1,0); + +\setcounter{maincounter}{5} +\foreach \x in {0,1,...,5} +{ + \node[CTU] at (\x, -2) {\themaincounter}; \addtocounter{maincounter}{1} +} +\draw[->, >=latex, black, line width=2pt] (6,-1.5) -- ++(1,0); + +\setcounter{maincounter}{7} +\foreach \x in {0,...,3} +{ + \node[CTU] at (\x, -3) {\themaincounter}; \addtocounter{maincounter}{1} +} +\draw[->, >=latex, black, line width=2pt] (4,-2.5) -- ++(1,0); + +\setcounter{maincounter}{9} +\foreach \x in {0,...,1} +{ + \node[CTU] at (\x, -4) {\themaincounter}; \addtocounter{maincounter}{1} +} +\draw[->, >=latex, black, line width=2pt] (2,-3.5) -- ++(1,0); + +\setcounter{maincounter}{1} +\foreach \y in {0,...,-4} +{ + \node[ThreadLabel] at (-2, \y+.4) {Thread \#\themaincounter}; \addtocounter{maincounter}{1} +} +\node[ThreadLabel,rotate=90] at (-2, -5+.5) {...}; + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/hevc_review_figures/.directory b/021_VideoCoding/Figures/HEVC/hevc_review_figures/.directory new file mode 100644 index 0000000..0aca798 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/hevc_review_figures/.directory @@ -0,0 +1,4 @@ +[Dolphin] +Timestamp=2013,6,23,17,50,12 +Version=3 +ViewMode=1 diff --git a/021_VideoCoding/Figures/HEVC/hevc_review_figures/.pdf b/021_VideoCoding/Figures/HEVC/hevc_review_figures/.pdf new file mode 100644 index 0000000..23fc7e6 Binary files /dev/null and b/021_VideoCoding/Figures/HEVC/hevc_review_figures/.pdf differ diff --git a/021_VideoCoding/Figures/HEVC/hevc_review_figures/BlockPartitions.tikz b/021_VideoCoding/Figures/HEVC/hevc_review_figures/BlockPartitions.tikz new file mode 100644 index 0000000..0b2535c --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/hevc_review_figures/BlockPartitions.tikz @@ -0,0 +1,69 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{0.3} + +\begin{tikzpicture}[scale=\scale,every node/.style={scale=\scale},on grid ] + + \definecolor{LCUFill}{rgb}{0,0.5,1} + \definecolor{DrawColor}{rgb}{0,0,0.5} + \definecolor{LCUFill}{rgb}{1,1,1} + \definecolor{DrawColor}{rgb}{0,0,0} + + \tikzstyle{TextStyle}=[scale=1.8, text centered] + + \tikzstyle{LCU}=[draw=DrawColor, fill=LCUFill, very thick] + \tikzstyle{CU}=[draw=DrawColor, thin] + \tikzstyle{PU}=[draw=DrawColor, thin, densely dashed] + +% \newcommand{\gridsize}{70} +% \draw[help lines] (-\gridsize,-\gridsize) grid ++(\gridsize*2,\gridsize*2); + + \node[TextStyle] at (8,16.5) {$64\times64$ - LCU}; + \draw[LCU] (0,0) rectangle ++(16,16); % 64x64 + + \node[TextStyle] at (4,12) {$32\times32$}; + \draw[CU] (0,0) rectangle ++(8,8); % 32x32 + \draw[CU] (8,8) rectangle ++(8,8); % 32x32 + + \node[TextStyle] at (10,14) {$16\times16$}; + \draw[CU] (0,0) rectangle ++(4,4); % 16x16 + \draw[CU] (4,4) rectangle ++(4,4); % 16x16 + + \draw[CU] (8,8) rectangle ++(4,4); % 16x16 + \draw[CU] (12,12) rectangle ++(4,4); % 16x16 + + \node[TextStyle] at (13,11) {$8\times8$}; + \draw[CU] (12,8) rectangle ++(2,2); % 8x8 + \draw[CU] (14,10) rectangle ++(2,2); % 8x8 + + \node[TextStyle] (CULabel) at (5.5,9) {CU}; + \draw[black, -latex] (CULabel) to (7,7); + \draw[CU] (4,4) rectangle ++(2,2); % 8x8 + \draw[CU] (6,6) rectangle ++(2,2); % 8x8 + + \node[TextStyle] (PULabel) at (2.5,5.5) {PU}; + \draw[black, -latex] (PULabel) to (4.5,4.5); + \draw[PU] (4,4) rectangle ++(1,1); % 4x4 + \draw[PU] (5,5) rectangle ++(1,1); % 4x4 + + %H.264 + \node[TextStyle] (PULabel) at (10.5,4.5) {\textcolor{red}{MB}}; + \draw[black, -latex] (PULabel) to (13,3); + \draw[CU, draw=red] (12,0) rectangle ++(4,4); % 16x16 + \draw[CU, draw=red] (14,0) rectangle ++(2,2); % 8x8 + \draw[CU, draw=red] (12,2) rectangle ++(2,2); % 8x8 + \draw[CU, draw=red] (15,0) rectangle ++(1,1); % 4x4 + \draw[CU, draw=red] (14,1) rectangle ++(1,1); % 4x4 + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/hevc_review_figures/EntropyCodingHEVC.pdf b/021_VideoCoding/Figures/HEVC/hevc_review_figures/EntropyCodingHEVC.pdf new file mode 100644 index 0000000..2f7a514 Binary files /dev/null and b/021_VideoCoding/Figures/HEVC/hevc_review_figures/EntropyCodingHEVC.pdf differ diff --git a/021_VideoCoding/Figures/HEVC/hevc_review_figures/EntropyCodingHEVC.tikz b/021_VideoCoding/Figures/HEVC/hevc_review_figures/EntropyCodingHEVC.tikz new file mode 100755 index 0000000..c42b230 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/hevc_review_figures/EntropyCodingHEVC.tikz @@ -0,0 +1,40 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + +\def\MVE{\mbox{MVE}} +\def\MVBASE{\mbox{BASE\_MC}} +\def\INTERVIEW{\mbox{INTERVIEW\_COPY}} +\def\COMBMC{\mbox{COMB\_MC}} +\def\COMBMCDISP{\mbox{COMB\_MC\_Disp}} + +% Define block styles +\tikzstyle{block} = [rectangle, draw, text width=5cm, text centered, rounded corners, minimum height=1.5cm] +\tikzstyle{node_circle} = [circle, draw, text centered] + +\tikzstyle{decision} = [diamond, draw, text centered, node distance=3cm] +\tikzstyle{line} = [draw, -latex'] + +\newcommand{\scale}{0.75} + +\begin{tikzpicture}[scale=\scale,node distance = 2.5cm, auto,every node/.style={scale=\scale}] + + + \node [block] at (0,0) (A) {Decode Split Info\\ \textbf{CABAC}}; + \node [block,below of=A] (B) {Decode Mode Info \\ \textbf{CABAC}}; + \node [block,below of=B] (C) {Decode Prediction Info \\ \textbf{CABAC}}; + + \node [block,below of=C, left of=C, xshift=-.5cm, yshift=-.5cm] (D1) {Decode Residual Info \\ \textbf{CABAC}}; + + \node [block,below of=C, right of=C, xshift=.5cm, yshift=-.5cm] (D2) {CABAC Bypass}; + \node [block,below of=D2] (E) {Decode Residual Info \\ \textbf{CAVLC}}; + + + \path [line] (A) -- (B); + \path [line] (B) -- (C); + \path [line] (C) -- node[above, xshift=-.5cm] {HEB} (D1); + \path [line] (C) -- node[above, xshift=.5cm] {HTB} (D2); + \path [line] (D2) -- (E); + + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/hevc_review_figures/HEVCBlockDiagram.pdf b/021_VideoCoding/Figures/HEVC/hevc_review_figures/HEVCBlockDiagram.pdf new file mode 100644 index 0000000..de7454e Binary files /dev/null and b/021_VideoCoding/Figures/HEVC/hevc_review_figures/HEVCBlockDiagram.pdf differ diff --git a/021_VideoCoding/Figures/HEVC/hevc_review_figures/HEVCBlockDiagram.tikz b/021_VideoCoding/Figures/HEVC/hevc_review_figures/HEVCBlockDiagram.tikz new file mode 100644 index 0000000..1919503 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/hevc_review_figures/HEVCBlockDiagram.tikz @@ -0,0 +1,92 @@ +\newcommand{\scale}{0.7} +\usetikzlibrary{arrows,matrix} +% \usepackage{amssymb, amsmath, mathptmx} + +\begin{tikzpicture}[font=\scriptsize,node distance=1.5cm,scale=0.9] + +\tikzstyle{sumBlock}=[draw,circle, inner sep=0cm, minimum size=0.1cm] +\tikzstyle{processBlock}=[draw, inner sep=0.1cm, minimum size=0.7cm,text centered] +\tikzstyle{proposedBlock}=[processBlock, fill=gray!30] +\tikzstyle{blockArrow}=[draw,-latex] +\tikzstyle{point} = [draw, fill=black, circle,inner sep=1pt] + +\coordinate (sumPoint) at (0,0); +\node[sumBlock] (ResidueDiff) at (sumPoint) {$+$}; + + +% \begin{scope}[xshift=-2cm, yshift=3.2cm, scale=1.5] +% \def\spacing{.2} +% \node at ((.5+3*\spacing*.5,-.5) (Input) {}; +% \foreach \i in {0,...,3} +% \draw[thick,fill=white] (\spacing*\i,-.15*\i) rectangle ++(1,1); +% % \node[text width=.7cm, text centered] at (1.6,1.1) {Video Stream}; +% % \node[text width=.7cm, text centered] at (1.1,.1) {Input video}; +% \end{scope} + + +\node[left of=ResidueDiff,text width=1cm,node distance=2.5cm] (Input) {\textbf{Input video}}; +\path[blockArrow] (Input) -- (ResidueDiff) node [point, midway] (p0) {}; + +\node[processBlock,right of=ResidueDiff] (Transf) {$T$}; +\node[processBlock,right of=Transf] (Quant) {$Q$}; +\node[processBlock,right of=Quant,node distance=2.5cm] (VLC) {CABAC}; + +\node[right of=VLC, text centered,text width=1.5cm, node distance=2.3cm] (Output) {\textbf{HEVC bitstream}}; +\path[blockArrow] (VLC) -- (Output); + + +\path[blockArrow] (ResidueDiff) -- (Transf); +\path[blockArrow] (Transf) -- (Quant); +\path[blockArrow] (Quant) -- (VLC) node [point,midway] (QuantFrame) {}; + +\node[processBlock,below of=QuantFrame,node distance=1.2cm] (InvQuant) {$Q^{-1}$}; +\node[processBlock,right of=InvQuant] (InvTransf) {$T^{-1}$}; +\node[sumBlock,below of=InvTransf, node distance=1cm] (ReconResidue) {$+$}; +\node[processBlock,text width=1.9cm, below of=ReconResidue] (Filters) {Deblocking \& SAO filters}; + +\path[blockArrow] (QuantFrame) -- (InvQuant); +\path[blockArrow] (InvQuant) -- (InvTransf); +\path[blockArrow] (InvTransf) -- (ReconResidue); +\path[blockArrow] (ReconResidue) -- (Filters) node[point,midway] (p1) {}; + +\node[left of=InvQuant, coordinate, node distance=1cm,yshift=0.5cm] (x1) {}; +\path[blockArrow,-] (x1 -| ResidueDiff) node[point] () {} -- (x1); +\path[blockArrow] (x1) |- (ReconResidue); + +\node [processBlock,text width=1.5cm] (Intra) at (2,-2) {Intra prediction}; +\node [processBlock,text width=1.5cm,below of=Intra, node distance=1.8cm] (Inter) {Inter prediction}; + +\path[blockArrow] (p1) -| (Intra); + +\node [point,left of=Intra,node distance=1.2cm] (z1) {}; +\node [point,left of=Inter,node distance=1.2cm] (z2) {}; + +\path[blockArrow,-] (Intra) -- (z1); +\path[blockArrow,-] (Inter) -- (z2); + +\node [coordinate,left of=z1,below of=z1,node distance=0.15cm] (w1) {}; +\node [coordinate,left of=z2, above of=z2, node distance=0.15cm] (w2) {}; + +\path (z1) -- (z2) node [coordinate, midway] (w) {}; +\path (w) -- (w -| sumPoint) node [point] (z) {}; + +\draw[-latex,densely dashed] (w1) to [bend left=60] (w2); +\draw [thick,blockArrow] (z1)+(0.1,-0.1) -- (z); + +\path[blockArrow] (z) -- node[pos=0.96,right] {-} (ResidueDiff); + +\node[processBlock,below of=Filters,text width=1.5cm, node distance=1.5cm] (RefBuffer) {Reference Picture Buffer}; + +\node[processBlock,text width=1.5cm] at(Inter |- RefBuffer) (MotionSearch) {Motion Estimation}; + +\path[blockArrow] (Filters) -- (RefBuffer) node[point,pos=0.4] (p2) {}; +\path[blockArrow] (RefBuffer) -- (MotionSearch); +\path[blockArrow] (MotionSearch) -- (Inter); +\path[blockArrow] (p0) |- (MotionSearch); + + +\node[right of=p2, text centered,text width=1.5cm, node distance=2.3cm] (Output) {\textbf{Video output}}; +\path[blockArrow] (p2) -- (Output); + + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/hevc_review_figures/IntraPredictions.pdf b/021_VideoCoding/Figures/HEVC/hevc_review_figures/IntraPredictions.pdf new file mode 100644 index 0000000..11ba00d Binary files /dev/null and b/021_VideoCoding/Figures/HEVC/hevc_review_figures/IntraPredictions.pdf differ diff --git a/021_VideoCoding/Figures/HEVC/hevc_review_figures/IntraPredictions.tikz b/021_VideoCoding/Figures/HEVC/hevc_review_figures/IntraPredictions.tikz new file mode 100644 index 0000000..0340675 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/hevc_review_figures/IntraPredictions.tikz @@ -0,0 +1,93 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{.12} + +\begin{tikzpicture}[scale=\scale,every node/.style={scale=\scale},on grid ] + + \definecolor{LCUFill}{rgb}{0,0.5,1} + \definecolor{DrawColor}{rgb}{0,0,0.5} + \definecolor{LCUFill}{rgb}{1,1,1} + \definecolor{DrawColor}{rgb}{0,0,0} + + \tikzstyle{TextStyle}=[scale=2, text centered] + + \tikzstyle{LCU}=[draw=DrawColor, fill=LCUFill, very thick] + \tikzstyle{CU}=[draw=DrawColor, thin] + \tikzstyle{PU}=[draw=DrawColor, thin, densely dashed] + + \tikzstyle arrowstyle=[scale=1] + \tikzstyle{IntraArrow} =[draw=black, shorten <=0, postaction={decorate,decoration={markings, mark=at position .3 with {\arrow[arrowstyle]{stealth}}}} ] + + +% \newcommand{\gridsize}{70} +% \draw[help lines] (-\gridsize,-\gridsize) grid ++(\gridsize*2,\gridsize*2); + + \draw[draw=black, thick] (-31,-31) rectangle (31,31); + + \coordinate (Center) at (0,0); + + \foreach \x in {-32,...,31} + \draw[draw=black] (\x-.5,31.5) rectangle ++(1,1); + \foreach \y in {-31,...,32} + \draw[draw=black] (-32-.5,\y-.5) rectangle ++(1,1); + + + + \foreach \x in {-31,-26,-21,-17,-13,-9,-5,-2,0,2,5,9,13,17,21,26,31} + { +% \pgfmathsetmacro{\intra}{\x+26} +% \node[scale=2.7, align=center] at (\x*1.1,32.5) {$\intra$}; + \draw[IntraArrow] (\x,31) to (Center); + } + \foreach \y in {-31,-26,-21,-17,-13,-9,-5,-2,0,2,5,9,13,17,21,26} + { + \draw[IntraArrow] (-31,\y) to (Center); + } + +% \foreach \x in {-31,-26,-21,-17,-13,-9,-5,-2,0,2,5,9,13,17,21,26,31} +% { +% \draw[IntraArrow] (Center) to (\x,-31); +% } +% \foreach \y in {-26,-21,-17,-13,-9,-5,-2,0,2,5,9,13,17,21,26,31} +% { +% \draw[IntraArrow] (Center) to (31,\y); +% } + +% \end{scope} +% \foreach \y in {0,...,31} +% \draw[black] (-1,\y) rectangle ++(1,1); + + +% \node[scale=6, text width=2cm, align=left] at (8,-1) {Planar}; +% \node[scale=6, text width=2cm, align=left] at (8,-4) { DC}; + + +% % H.264 +% \foreach \x in {-31,-15,0,15,32} +% \draw[ red,-latex] (Center) to (\x,-32); +% +% \foreach \y in {-15,0,15} +% \draw[red,-latex] (Center) to (32,\y); + + \foreach \x in {-31,-15,0,15,31} + \draw[IntraArrow, red] (\x,31) to (Center); + + \foreach \y in {-15,0,15} + \draw[IntraArrow, red] (-31,\y) to (Center); + + \node[scale=6, text width=2cm, align=left,red] at (8,-1) {Planar}; + \node[scale=6, text width=2cm, align=left,red] at (8,-4) { DC}; + + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/hevc_review_figures/PixelBasedMVE.tex b/021_VideoCoding/Figures/HEVC/hevc_review_figures/PixelBasedMVE.tex new file mode 100644 index 0000000..134a6d1 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/hevc_review_figures/PixelBasedMVE.tex @@ -0,0 +1,45 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{0.20} + +\begin{tikzpicture}[scale=\scale,every node/.style={scale=\scale},on grid ] + + \tikzstyle{LostMB}=[black, thick] + \tikzstyle{ExtrapolatedMB}=[black, dashed, thick] + \tikzstyle{TextNode}=[scale=4.2, text centered, text width=3.3cm] + \tikzstyle{SymbolNode}=[scale=5.2, text centered] + + \newcommand{\gridsize}{30} + %\draw[help lines] (-\gridsize,-\gridsize) grid ++(\gridsize*2,\gridsize*2); + + \draw[LostMB] (0,0) rectangle ++(8,8); + + \draw[ExtrapolatedMB] (6-16,6) rectangle ++(16cm,16cm); + \draw[ExtrapolatedMB] (4,4) rectangle ++(16cm,16cm); + \draw[ExtrapolatedMB] (6,-14) rectangle ++(16cm,16cm); + \draw[ExtrapolatedMB] (3-16,-14) rectangle ++(16cm,16cm); + + \node[TextNode](EMB1) at (5-16+16/2,7+16/2) {Extrapolated MB$_1$}; + \node[TextNode](EMB1) at (5+16/2,4+16/2) {Extrapolated MB$_2$}; + \node[TextNode](EMB1) at (6+16/2,-14+16/2) {Extrapolated MB$_4$}; + \node[TextNode](EMB1) at (3-16+16/2,-14+16/2) {Extrapolated MB$_3$}; + + \node[SymbolNode] at (2,1) {$\circ$}; + \node[SymbolNode] at (7,1) {$\circ$}; + \node[SymbolNode] at (7,7) {$\circ$}; + \node[SymbolNode] at (5,7) {$\diamond$}; + \node[SymbolNode,scale=0.7] at (2,4) {$\bigtriangleup$}; + \node[SymbolNode] at (-2,4) {$\star$}; + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HEVC/hevc_review_figures/RPS.jpg b/021_VideoCoding/Figures/HEVC/hevc_review_figures/RPS.jpg new file mode 100644 index 0000000..6564684 Binary files /dev/null and b/021_VideoCoding/Figures/HEVC/hevc_review_figures/RPS.jpg differ diff --git a/021_VideoCoding/Figures/HEVC/hevc_review_figures/build_figures b/021_VideoCoding/Figures/HEVC/hevc_review_figures/build_figures new file mode 100755 index 0000000..043d120 --- /dev/null +++ b/021_VideoCoding/Figures/HEVC/hevc_review_figures/build_figures @@ -0,0 +1,45 @@ +#!/bin/bash + +currdir=$(pwd) +tikz_files=$( find . -name '*.tikz' ); + +for file_list in $tikz_files; do + + dirname="${file_list%/*}" + file="${file_list##*/}" + + cd ${dirname} + +cat > build.tex <<_EOF_ +\documentclass{article} + +\usepackage{tikz} +\usetikzlibrary{shapes,arrows} + + +\usepackage{verbatim} +\usepackage[active,tightpage]{preview} + +\PreviewEnvironment{tikzpicture} +\setlength\PreviewBorder{5pt} +\begin{document} +\pagestyle{empty} +\begin{figure}[p] +\centering +\input{ $file } +\end{figure} +\end{document} + +_EOF_ + + pdflatex build.tex + + rm build.aux + rm build.log + rm build.tex + + mv build.pdf ${file%%.*}.pdf + + cd ${currdir} + +done \ No newline at end of file diff --git a/021_VideoCoding/Figures/HybridCodingBlockDiagram.png b/021_VideoCoding/Figures/HybridCodingBlockDiagram.png new file mode 100644 index 0000000..ac17d8f Binary files /dev/null and b/021_VideoCoding/Figures/HybridCodingBlockDiagram.png differ diff --git a/021_VideoCoding/Figures/HybridCodingBlockDiagram.tikz b/021_VideoCoding/Figures/HybridCodingBlockDiagram.tikz new file mode 100644 index 0000000..7a2b0dc --- /dev/null +++ b/021_VideoCoding/Figures/HybridCodingBlockDiagram.tikz @@ -0,0 +1,120 @@ +\usetikzlibrary{arrows,matrix} +% \usepackage{amssymb, amsmath, mathptmx} + +\begin{tikzpicture}[font=\scriptsize,node distance=1.1cm,scale=1] + + + +\tikzstyle{simpleBlock}=[inner sep=0.1cm, minimum size=.9cm,text centered, minimum width=1cm, text width=1.2cm] +\tikzstyle{processBlock}=[simpleBlock, thick, draw] +\tikzstyle{sumBlock}=[draw,circle, inner sep=0cm, minimum size=0.4cm] +\tikzstyle{proposedBlock}=[processBlock, fill=gray!30] +\tikzstyle{blockArrow}=[draw,-latex] +\tikzstyle{point} = [draw, fill=black, circle,inner sep=1pt] + +\coordinate (sumPoint) at (0,0); +\node[sumBlock] (ResidueDiff) at (sumPoint) {$+$}; + +\node[simpleBlock, inner sep=0.2cm, left of=ResidueDiff,text width=1cm,node distance=2.8cm] (Input) {\textbf{Input video}}; +\path[blockArrow] (Input) -- (ResidueDiff) node [point, midway] (p0) {}; + +\node[processBlock,right=2.5 of ResidueDiff, text width=1.2cm] (Transf) {$T+Q$}; +% \node[processBlock,right=of Transf, text width=.7cm] (Quant) {$Q$}; +\node[processBlock,right=3.2 of Transf, text width=1.5cm] (VLC) {Entropy Encoder}; + + +\path[blockArrow] (ResidueDiff) -- (Transf); +% \path[blockArrow] (Transf) -- (Quant); +\path[blockArrow] (Transf) -- (VLC) node [point,midway] (QuantFrame) {}; + + +% \node[simpleBlock, right=of VLC, text width=1.5cm] (Output) {\textbf{HEVC bitstream}}; +% \path[blockArrow] (VLC) -- (Output); + +\node[processBlock, below=.3 of QuantFrame, text width=1.5cm] (InvQuant) {$Q^{-1} + T^{-1}$}; +% \node[processBlock, below=.3 of InvQuant, text width=.7cm] (InvTransf) {$T^{-1}$}; +\node[sumBlock, below=.3 of InvQuant] (ReconResidue) {$+$}; +\node[processBlock, left=.8 of ReconResidue,text width=1cm, yshift=-.4cm] (RefBuffer) {Frame Buffer}; + +\path[blockArrow] (QuantFrame) -- (InvQuant); +% \path[blockArrow] (InvQuant) -- (InvTransf); +\path[blockArrow] (InvQuant) -- (ReconResidue); +\path[blockArrow] (ReconResidue) |- (RefBuffer); + +\node[below=.3 of Transf, coordinate] (x1) {}; +\path[blockArrow,-] (x1 -| ResidueDiff) node[point] () {} -- (x1); +\path[blockArrow] (x1) -- (ReconResidue); + +\node [left=1.4 of RefBuffer,coordinate] (z0) {}; +\node [processBlock,text width=1.4cm, above=.25 of z0] (Intra) {Intra prediction}; +\node [processBlock,text width=1.4cm, below=.25 of z0] (Inter) {Inter prediction}; + +\path[blockArrow] (RefBuffer.160) -- (Intra.0); +\path[blockArrow] (RefBuffer.200) -- (Inter.0); + +\node [point,left=.4 of Intra] (z1) {}; +\node [point,left=.4 of Inter] (z2) {}; +\path[blockArrow,-] (Intra) -- (z1); +\path[blockArrow,-] (Inter) -- (z2); + +\node [coordinate,left of=z1,below of=z1,node distance=0.15cm] (w1) {}; +\node [coordinate,left of=z2, above of=z2, node distance=0.15cm] (w2) {}; +\path (z1) -- (z2) node [coordinate, midway] (w) {}; +\path (w) -- (w -| sumPoint) node [point] (z) {}; +\draw[-latex,densely dashed] (w1) to [bend left=60] (w2); +\draw [thick,blockArrow] (z1)+(0.1,-0.1) -- (z); +\path[blockArrow] (z) -- node[pos=0.96,right] {-} (ResidueDiff); + + +\node [below=.4 of Inter,coordinate] (inter0) {}; +\node [below=.4 of Inter, xshift=0.5cm, coordinate] (inter1) {}; +\path[blockArrow,-] (p0) |- (inter0); +\path[blockArrow] (inter0) -- (Inter); + +\path[blockArrow,-] (Intra.south -| inter1) -- (Inter.north -| inter1); +\path[blockArrow,-] (Inter.south -| inter1) -- (inter1); +\path[blockArrow] (inter1) -| (VLC); + + +\node[processBlock, below=4 of VLC, text width=1.5cm] (VLCDec) {Entropy Decoder}; +\node[processBlock, left=.6 of VLCDec, text width=1.5cm] (InvQuantDec) {$Q^{-1} + T^{-1}$}; +\node[sumBlock, left=.6 of InvQuantDec] (ReconSumDec) {$+$}; +\node[simpleBlock, inner sep=0.2cm, text width=1.2cm,node distance=2.5cm] (Output) at (Input |- VLCDec) {\textbf{Output video}} ; + +\path[blockArrow] (VLCDec) -- (InvQuantDec); +\path[blockArrow] (InvQuantDec) -- (ReconSumDec); +\path[blockArrow] (ReconSumDec) -- (Output) node [pos=.3] (predPosition) {}; + +\node [point] at (p0 |- ReconSumDec) (outputPosition) {}; + +\node [below=1.8 of predPosition, coordinate] (z0) {}; +\node [processBlock, left=1.5 of z0, text width=1cm] (RefBufferDec) {Frame Buffer}; +\node [processBlock,text width=1.4cm, above=.25 of z0] (IntraDec) {Intra prediction}; +\node [processBlock,text width=1.4cm, below=.25 of z0] (InterDec) {Inter prediction}; + +\path[blockArrow] (RefBufferDec.20) -- (IntraDec.180); +\path[blockArrow] (RefBufferDec.-20) -- (InterDec.180); + +\node [point,right=.5 of IntraDec] (z1) {}; +\node [point,right=.5 of InterDec] (z2) {}; +\path[blockArrow,-] (IntraDec) -- (z1); +\path[blockArrow,-] (InterDec) -- (z2); + +\node [coordinate,right of=z1,below of=z1,node distance=0.15cm] (w1) {}; +\node [coordinate,right of=z2, above of=z2, node distance=0.15cm] (w2) {}; +\path (z1) -- (z2) node [coordinate, midway] (w) {}; +\path (w) -- (w -| ReconSumDec) node [point] (z) {}; +\draw[-latex,densely dashed] (w1) to [bend right=60] (w2); +\draw [thick,blockArrow] (z1)+(-0.1,-0.1) -- (z); +\path[blockArrow] (z) -- node[pos=0.96,left] {-} (ReconSumDec); + +\path[blockArrow] (outputPosition) |- (RefBufferDec); + + +\path (VLC) -- (VLCDec) node [coordinate, midway] (stopos) {}; +\node [processBlock,text width=1.8cm, minimum height=1.2cm, right=1.5 of stopos] (Transmission) {Storage or Transmission}; +\path[blockArrow] (VLC) -| (Transmission); +\path[blockArrow] (Transmission) |- (VLCDec); + + +\end{tikzpicture} diff --git a/021_VideoCoding/Figures/HybridCodingBlockDiagramPrePost.tikz b/021_VideoCoding/Figures/HybridCodingBlockDiagramPrePost.tikz new file mode 100644 index 0000000..1c23e13 --- /dev/null +++ b/021_VideoCoding/Figures/HybridCodingBlockDiagramPrePost.tikz @@ -0,0 +1,139 @@ +\usetikzlibrary{arrows,matrix} +% \usepackage{amssymb, amsmath, mathptmx} + +\begin{tikzpicture}[font=\scriptsize,node distance=1.1cm,scale=.7] + + + +\tikzstyle{simpleBlock}=[inner sep=0.1cm, minimum size=.9cm,text centered, minimum width=1cm, text width=1.2cm,scale=.9] +\tikzstyle{processBlock}=[simpleBlock, fill=\TikZColorFill, thick, draw] +\tikzstyle{sumBlock}=[draw,circle, inner sep=0cm, minimum size=0.4cm] +\tikzstyle{proposedBlock}=[processBlock, fill=gray!30] +\tikzstyle{blockArrow}=[draw,-latex] +\tikzstyle{point} = [draw, fill=black, circle,inner sep=1pt] + + +\node[sumBlock] (ResidueDiff) at (0,0) {$+$}; + + +\node[processBlock, left=.95 of ResidueDiff, text width=2.1cm] (PreProc) {Pre-processing}; + +\node[simpleBlock, inner sep=0.2cm, left=.5 of PreProc,text width=1cm,node distance=2.8cm] (Input) {\textbf{Input video}}; + +\path[blockArrow] (Input) -- (PreProc); +\path[blockArrow] (PreProc) -- (ResidueDiff) node [point, midway] (p0) {}; + + + +\node[processBlock,right=2.8 of ResidueDiff, text width=1.2cm] (Transf) {$T+Q$}; +% \node[processBlock,right=of Transf, text width=.7cm] (Quant) {$Q$}; +\node[processBlock,right=6.4 of ResidueDiff, text width=1.5cm] (VLC) {Entropy Encoder}; + + +\path[blockArrow] (ResidueDiff) -- (Transf); +% \path[blockArrow] (Transf) -- (Quant); +\path[blockArrow] (Transf) -- (VLC) node [point,midway] (QuantFrame) {}; + + +% \node[simpleBlock, right=of VLC, text width=1.5cm] (Output) {\textbf{HEVC bitstream}}; +% \path[blockArrow] (VLC) -- (Output); + +\node[processBlock, below=.6 of QuantFrame, text width=1.5cm] (InvQuant) {$Q^{-1} + T^{-1}$}; +% \node[processBlock, below=.3 of InvQuant, text width=.7cm] (InvTransf) {$T^{-1}$}; +\node[sumBlock, below=.6 of InvQuant] (ReconResidue) {$+$}; +\node[processBlock, left=.8 of ReconResidue,text width=1cm, yshift=-.6cm] (RefBuffer) {Frame Buffer}; + +\path[blockArrow] (QuantFrame) -- (InvQuant); +% \path[blockArrow] (InvQuant) -- (InvTransf); +\path[blockArrow] (InvQuant) -- (ReconResidue); +\path[blockArrow] (ReconResidue) |- (RefBuffer); + +\node[below=.5 of Transf, coordinate,xshift=-.7cm] (x1) {}; +\path[blockArrow,-] (x1 -| ResidueDiff) node[point] () {} -- (x1); +\path[blockArrow] (x1) -- (ReconResidue); + +\node [left=1.4 of RefBuffer,coordinate] (z0) {}; +\node [processBlock,text width=1.4cm, above=.25 of z0] (Intra) {Intra prediction}; +\node [processBlock,text width=1.4cm, below=.25 of z0] (Inter) {Inter prediction}; + +\path[blockArrow] (RefBuffer.160) -- (Intra.0); +\path[blockArrow] (RefBuffer.200) -- (Inter.0); + +\node [point,left=.4 of Intra] (z1) {}; +\node [point,left=.4 of Inter] (z2) {}; +\path[blockArrow,-] (Intra) -- (z1); +\path[blockArrow,-] (Inter) -- (z2); + +\node [coordinate,left of=z1,below of=z1,node distance=0.15cm] (w1) {}; +\node [coordinate,left of=z2, above of=z2, node distance=0.15cm] (w2) {}; +\path (z1) -- (z2) node [coordinate, midway] (w) {}; +\path (w) -- (w -| ResidueDiff) node [point] (z) {}; +\draw[-latex,densely dashed] (w1) to [bend left=60] (w2); +\draw [thick,blockArrow] (z1)+(0.1,-0.1) -- (z); +\path[blockArrow] (z) -- node[pos=0.96,right] {-} (ResidueDiff); + + +\node [below=.4 of Inter,coordinate] (inter0) {}; +\node [below=.4 of Inter, xshift=0.5cm, coordinate] (inter1) {}; +\path[blockArrow,-] (p0) |- (inter0); +\path[blockArrow] (inter0) -- (Inter); + +\path[blockArrow,-] (Intra.south -| inter1) -- (Inter.north -| inter1); +\path[blockArrow,-] (Inter.south -| inter1) -- (inter1); +\path[blockArrow] (inter1) -| (VLC); + + +\node[draw,dashed,inner sep=5pt, fit=(p0) (inter1) (VLC)] (Encoder) {}; +\node[simpleBlock, above=-.3 of Encoder] {Encoding}; + + +\node[processBlock, below=5 of VLC, text width=1.5cm] (VLCDec) {Entropy Decoder}; +\node[processBlock, left=.4 of VLCDec, text width=1.5cm] (InvQuantDec) {$Q^{-1} + T^{-1}$}; +\node[sumBlock, left=.4 of InvQuantDec] (ReconSumDec) {$+$}; + +\node[processBlock, text width=2.1cm] at(PreProc |- InvQuantDec) (PostProc) {Post-processing}; + +\node[simpleBlock, inner sep=0.2cm, text width=1.2cm,node distance=2.5cm] (Output) at (Input |- VLCDec) {\textbf{Output video}} ; + +\path[blockArrow] (VLCDec) -- (InvQuantDec); +\path[blockArrow] (InvQuantDec) -- (ReconSumDec); +\path[blockArrow] (ReconSumDec) -- (PostProc) node [pos=.3] (predPosition) {}; +\path[blockArrow] (PostProc) -- (Output); + +\node [point] at (p0 |- ReconSumDec) (outputPosition) {}; + +\node [below=1.8 of predPosition, coordinate] (z0) {}; +\node [processBlock, left=1.5 of z0, text width=1cm] (RefBufferDec) {Frame Buffer}; +\node [processBlock,text width=1.4cm, above=.25 of z0] (IntraDec) {Intra prediction}; +\node [processBlock,text width=1.4cm, below=.25 of z0] (InterDec) {Inter prediction}; + +\path[blockArrow] (RefBufferDec.20) -- (IntraDec.180); +\path[blockArrow] (RefBufferDec.-20) -- (InterDec.180); + +\node [point,right=.5 of IntraDec] (z1) {}; +\node [point,right=.5 of InterDec] (z2) {}; +\path[blockArrow,-] (IntraDec) -- (z1); +\path[blockArrow,-] (InterDec) -- (z2); + +\node [coordinate,right of=z1,below of=z1,node distance=0.15cm] (w1) {}; +\node [coordinate,right of=z2, above of=z2, node distance=0.15cm] (w2) {}; +\path (z1) -- (z2) node [coordinate, midway] (w) {}; +\path (w) -- (w -| ReconSumDec) node [point] (z) {}; +\draw[-latex,densely dashed] (w1) to [bend right=60] (w2); +\draw [thick,blockArrow] (z1)+(-0.1,-0.1) -- (z); +\path[blockArrow] (z) -- node[pos=0.96,left] {-} (ReconSumDec); + +\path[blockArrow] (outputPosition) |- (RefBufferDec); + + +\path (VLC) -- (VLCDec) node [coordinate, midway] (stopos) {}; +\node [processBlock,text width=1.8cm, minimum height=1.2cm, right=1.1 of stopos] (Transmission) {Storage or Transmission}; +\path[blockArrow] (VLC) -| (Transmission); +\path[blockArrow] (Transmission) |- (VLCDec); + + +\node[draw,dashed,inner sep=5pt, outer sep=0, fit=(outputPosition) (InterDec) (VLCDec)] (Decoder) {}; +\node[simpleBlock, above=-.3 of Decoder] {Decoding}; + + +\end{tikzpicture} diff --git a/021_VideoCoding/VideoCodingReview.tex b/021_VideoCoding/VideoCodingReview.tex new file mode 100755 index 0000000..b319654 --- /dev/null +++ b/021_VideoCoding/VideoCodingReview.tex @@ -0,0 +1,544 @@ + +% Figure folders +\renewcommand{\FigsPath}{./021_VideoCoding/Figures} + +% \chapter{\HIGHLIGHT{The \ac{hevc} Standard}} +\chapter[The High Efficiency Video Coding Standard]{The High Efficiency Video \\Coding Standard} +%\chapter{\HIGHLIGHT{The State-of-Art Video Coding}} +\label{Chapter:HEVCReview} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Introduction} + +\glsunset{h264} + +This chapter presents an overview of the H.265 - \ac{hevc}~\cite{H265StandardFeb2018} standard, focusing on the most relevant features of the encoding algorithm for the work developed in this thesis. The \ac{hevc} standard aims to address essentially all existing applications of \ac{h264}~\cite{H264AVCStandard2003}, extending it in two keys aspects: increased video resolution and increased use of parallel processing architectures. In order to increase the flexibility and follow the increase in the video resolution, the \ac{hevc} standard adopts a new block partition structure, as well as, new prediction modes. In this chapter these new techniques introduced are described, especially those which may have a significant impact in the error robustness characteristics of the \ac{hevc}. +%The standard also improves the intra~\cite{IntraCodingHEVC2012} and inter~\cite{MotionCodingHEVC2013} coding and includes new high-level features~\cite{HEVCHighLevelReferenceDez2012}. This chapter will describe main features and improvements introduced in the standard. + +This chapter is organized as follows. Section~\ref{Section2:GeneraVideoSystem} introduced the functional blocks of a generic video coding system, and Section~\ref{Section2:Overview} provides the overview of the \ac{hevc} standard features. Subsequently, Section~\ref{Section2:HighLevel} describes the high-level features which are used to carry relevant information regarding the bitstream, Section~\ref{Section2:CodingStructures} describes the coding structures used to divide the video frames into smaller partitions and Section~\ref{Section2:PredictionModes} describes the main predictions techniques, namely intra and inter coding. Finally, a briefly description of the transform and quantisation, entropy coding and in-loop filters is given in Sections~\ref{Section2:TransformQuant}, \ref{Section2:Entropy} and \ref{Section2:Filters}, respectively. Section~\ref{Section2:Summary} concludes this chapter. + +\clearpage +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{General hybrid video coding} +\label{Section2:GeneraVideoSystem} + +A digital video is a temporal succession of images, also referred to as pictures or frames, which usually presents a significant amount of temporal and spatial correlations. In general, video coding and distribution systems follow a generic set of tools: pre-processing, encoding, storage or transmission, decoding and post-processing. +The pre-processing unit is used adapt the raw video signal, in which if the format of raw videos from video cameras is analogue, it will be firstly digitized into RGB images and then converted into YUV representation~\cite{DataCompression1997}. In the next processing tool, the video data will be compressed by the respective encoding unit by reducing its redundancy. The encoding process is able to compress the raw video in a high compression ratio and maintains a good trade-off between video quality and bitrate through high efficient algorithms. +On the contrary, the output video for displaying can be acquired through decoding and post-processing modules. + +% \begin{figure}[tb] +% \centering +% \input{\FigsPath/GeneralVideoSystem.tikz} +% \caption{Block diagram of a generic video coding system.} +% \label{FIG2:GeneralVideoSystem} +% \end{figure} + + + +Since H.261, the hybrid video coding scheme has been the basic structure for all video coding standards and recommendations of ITU-T \ac{vceg} and \ac{iso}/\ac{iec} \ac{mpeg}. While the structure has not been changed, the algorithms represented by the building blocks have been refined and the applicable configuration for the algorithms has become more and more flexible over years. The coding scheme is called hybrid as it combines temporal prediction between pictures of the video sequence with transform coding techniques for the prediction error \cite{VideoCodingFundamentals2015} as shown in Figure~\ref{Figure2:HybridCodec}~\cite{BookLucas2017}. + +\begin{figure}[tb] +\centering +\input{\FigsPath/HybridCodingBlockDiagramPrePost.tikz} +\caption{Block diagram of a generic video coding system, including the inner blocks of a generic hybrid encoding and decoding scheme.} +\label{Figure2:HybridCodec} +\end{figure} + +As aforementioned, the main purpose of video coding is to compress the video by reducing its redundancy. +In general, there are two kinds of redundancies existing in video signal: spatial and temporal. +On the one hand, the spatial redundancy is exploited through intra frame prediction, of which the mechanism is to find the best matching block for current one from its neighbours. This is due to a high correlation between neighbouring pixels inside a frame. As result, in intra frame prediction a block is reconstructed by the blocks that has already been encoded and stored in the bitstream for output. On the other hand, inter frame prediction is designed to deal with the temporal redundancy existed between consecutive frames such as the frame in the past or in the future. Temporal redundancy is normally referred as the correlation between two adjacent frames since they usually have similar backgrounds or a same object with a relative motion. Thus, the purpose of inter-frame prediction is to search for the best matching block for the current one in the reference frames, ranging from past to future frames. After the best matching candidate for the current block is found by either intra- or inter-frame prediction, only the difference between the original block and the predicted one needs to be encoded. +As seen from Figure~\ref{Figure2:HybridCodec}, these differences (also called residual signal) will be further processed and converted into the coefficients of transform domain (T). In general, redundant information can be eliminated through transforming. Subsequently, the transformed coefficients are quantized (Q) to reduce the range of the signal and save the number of bits required for signalling. Finally, the quantized transform coefficients together with the prediction information are compressed to form an output bitstream for transmission by the entropy encoder, as shown in Figure~\ref{Figure2:HybridCodec}. +On contrary, the decoding part performs the inverse operations on the bitstream to reconstruct the video for playing. Subsequently, the aforementioned blocks are described in the context of \ac{hevc} standard. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Overview the standard features} +\label{Section2:Overview} + +As for all the previous coding standards developed by JCT-VC, in \ac{hevc}~\cite{HEVCOverviewJul2012,HEVCOverviewDez2012,BookHEVC2014} only the bitstream structure and the coding syntax are standardised. +Included are the bitstream constraints, the definition of the semantic meaning of syntax elements and the decoding process, such that every decoder conforming to the standard generates the same output when given a correct bitstream. +Such limitation of scope in the standard allows maximal freedom for developers and enables specific optimisations to cope with various applications’ requirements (\emph{e.g.}, balancing the compression ratio and the coding complexity). Therefore, in order to be considered as an \ac{hevc} encoder, the algorithm needs to ensure that the produced bitstream complies with the standard specifications. + +Figure \ref{fig:HEVCBlockDiagram} illustrates the block diagram of the hybrid \ac{hevc} encoder. It follows the same overall architecture as its predecessors starting from H.261. +The encoder uses both intra- and inter-picture prediction to exploit spatial and temporal redundancies, respectively. Subsequently, the prediction residue transformed by linear spatial transform, which is then quantised, and entropy coded. This allows to reduce the bitstream redundancy to a minimum. +The resulting bitstream is combined with control information, which signals the prediction chosen, motion vectors, filtering operations and bitstream headers. + +\begin{figure}[tb] +\centering +% \includegraphics[width=0.75\columnwidth]{\FigsPath/HEVC/HEVCBlockDiagram} +\input{\FigsPath/HEVC/HEVCBlockDiagram.tikz} +\caption{Generic \ac{hevc} standard architecture.} +\label{fig:HEVCBlockDiagram} +\end{figure} + + +As depicted in Figure \ref{fig:HEVCBlockDiagram}, each picture can be compressed with intra coding algorithms or inter prediction using previously encoded and reconstructed pictures kept in the \ac{dpb} as reference. +The encoding process follows a block-based approach as in previous standards with the larger units referred to as \ac{ctu} with a maximum size of $64\times64$. In order to improve the overall quality of the reconstructed pictures and smooth the coding artefacts, \ac{hevc} uses the deblocking and \ac{sao} filters. +The encoder has a built-in decoder to generate the reference pictures. In the decoding loop, the residual information is delivered to the inverse quantisation and transform modules. The finally reconstructed samples are obtained by adding the predicted signal and applying the filtering operations. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{High-level syntax} +\label{Section2:HighLevel} + +This section describes the high-level components of the \ac{hevc} standard that provide signalling information and enable bitstream partitioning. +\ac{hevc} inherits several high-level features from \ac{h264}, e.g., the \ac{nal} units, parameter sets, the use of \ac{poc}, and \ac{sei} messages for auxiliary data. +Some high-level features that exist in \ac{h264} were not included in \ac{hevc}, for example \ac{fmo}, redundant slices, arbitrary slice order, data partitioning, and switching slices. +Several new high-level features are introduced, such as the \ac{vps}, the tiles and wavefront tools for parallel processing, the dependent slices for reduced delay, and a new reference picture management concept. +Moreover, new pictures types were introduced new picture types to increase the random-access flexibility, and temporal sub-layer access pictures. + +\subsection{Network Abstraction Layer} + +Similar to \ac{h264}~\cite{H264AVCStandard2003}, the \ac{hevc} standard used a bitstream based on \glsfirst{nal} units, which are useful to deal with lossy network environments, as it is useful for bitstream partitioning and synchronization \cite{H264OverIPJul2003}. +The \ac{nal} provides the ability to map the \ac{vcl} data that represents video frames on various transport layers, such as \ac{rtp}, MPEG-2 Systems and MPEG DASH. \Gls{nal} units are classified into two categories: (i) \ac{vcl} and (ii) non-\ac{vcl}, as they carry coded video frames or associated data, respectively. + +In the \ac{hevc} standard two bytes are allocated to describe the content of each \ac{nal} unit. These bytes are used to identify which type of \ac{nal} is being decoded and are transmitted in a fixed format form without variable length codes. This allow extra error robustness but increases the bit allocation. Figure~\ref{Figure2:NalHeader} show the structure of the \ac{hevc} \ac{nal} unit header. The first bit \textit{F}, referred to as \textit{forbiden\_bit}, is always zero in order to ensure that a start code can be detected across different protocol standards. This is followed by the \ac{nal} unit type which identifies its content, enabling the decoder to access if it is required for a particular decoding purpose, or if it can be discarded. Another syntax element defined by the \ac{hevc} standard in the \ac{nal} unit header is the TIDP which marks the temporal layer to which the unit belongs. This allows for temporal scalability (layers ranging from 0 to 6) by immediately discarding the \ac{nal} unit if it belongs to a layer higher than the desired one. Finally, six bits are reserved for future extensions, preventing the creating of extra \gls{nal} units for scalable and multiview extensions as it was done in \ac{h264}. + +\begin{figure}[tb] +\centering +\newcommand\bithead[1]{\bitbox[]{#1}{\scriptsize #1}} +\newcommand{\colorbitbox}[3]{% + \rlap{\bitbox{#2}{\color{#1}\rule{\width}{\height}}}% + \bitbox{#2}{#3}} +\begin{bytefield}[bitwidth=20pt]{16} +% \bitheader{0-15} \\ + \colorbitbox{\TikZColorFill}{1}{F} & \colorbitbox{gray!60}{6}{NAL Type} & \colorbitbox{\TikZColorFill}{6}{Reserved} & \colorbitbox{gray!60}{3}{TIDP} +\end{bytefield} +\caption{Two bytes \ac{nal} unit header in \ac{hevc}.} +\label{Figure2:NalHeader} +\end{figure} + +\subsection{Parameters set} + +The encoder uses a set of signalling packets as a robust method to share data with the decoder, referred to as parameter set units. This information was introduced in previous standards to deal with vulnerabilities in the transport of signalling information between the encoder and decoder, which compromised the streaming applications. +Within this approach the set of parameters are included in the bitstream and are repeated as many times as required by the application. This structure is inherited from the \ac{h264} standard\cite{H264OverIPJul2003} with modifications to cope with new coding tools. +The \ac{hevc} includes three sets of parameters, \ac{sps}, \ac{pps} and the newly introduced \glsfirst{vps}\cite{HEVCHighLevelReferenceDez2012}. +% \ac{sps} include sequence related information, that include the decoder operation point, flags to enable or disable certain tools within a given profile. \Gls{pps} include picture related information, that may change for every picture, e.g., \ac{qp} and tilling information. + +The new set of parameters introduced in the \ac{hevc} standard (\ac{vps}) contains information related to the different layers of the video, avoiding duplications like in the \ac{h264} and its extensions. \Gls{vps} aims to provide signalling for every layer, guaranteeing a clean and extendible standard, capable of supporting multiple layers. Moreover, \ac{vps} carries information for session negotiation (\emph{e.g.} profile, level and tier). The other sets are the \ac{sps} and \ac{pps}. \Glspl{sps} are related to the whole sequence (\emph{i.e.}, should affect all coded slices), and contain information related with the decoder operation point, as well as, flags to control optional tools and scalability control. \Glspl{pps} carry information that may change for every picture, such as, initial \ac{qp}, flags for picture related tools and tilling information. + + +\subsection{Picture types} + +\begin{table}[tb] + \centering + \caption{Picture categories in \ac{hevc} standard.}\label{Table2:PictureTypes} + \TABLESVSPACE\small + \begin{tabular}{m{1.3cm}|m{5.2cm}|m{6.3cm}} + \toprule + \multicolumn{3}{c}{\textbf{Random-access points pictures}} \\ + \midrule + IDR & + Instantaneous decoding refresh & + All leading pictures can be decoded \\ + \midrule + CRA & + Clean random access & + Pictures in the display order can be decoded\\ + \midrule + BLA & + Broken link access & + Leading pictures cannot be decoded \\ + + \midrule \midrule + \multicolumn{3}{c}{\textbf{Leading pictures}} \\ + \midrule + RADL & \multicolumn{2}{l}{Random access decodable picture}\\ + \midrule + RASL & \multicolumn{2}{l}{Random access skipped picture}\\ + + \midrule\midrule + \multicolumn{3}{c}{\textbf{Temporal sub-layer access pictures}} \\ + \midrule + TSA & + Temporal sub-layer access & + Allows switching to a given temporal layer or higher \\ + \midrule + STSA & + Step-wise temporal sub-layer access & + Allows switching to a given layer only \\ + \bottomrule + \end{tabular} +\end{table} + +The \ac{hevc} standard introduces several pictures types as show in Table~\ref{Table2:PictureTypes}. These pictures are divided in the following categories: (i) random-access points, (ii) leading pictures and (iii) temporal sub-layer access pictures~\cite{HEVCHighLevelReferenceDez2012,BookHEVC2014}. + +\subsubsection{Random-Access Point} + +The \ac{rap} pictures are used to provide points in the bitstream where it is possible to start the decoding process. The standard defines that a conforming bitstream must start with a \ac{rap} picture. These frames must belong to temporal sub-layer 0 and be coded using intra coding techniques, i.e., must not using previous decoded frames as reference. One should not that frames may be compressed with solely intra prediction and not be marked as random-access points. It is always possible to decode from the \ac{rap} frame and onwards to output any subsequent pictures in output order even if all pictures that precede the \ac{rap} in decoding order are discarded from the bitstream. + +The \ac{idr} frame is an intra coded picture that completely refresh the decoding process and starts a new conforming video stream. This means that neither the \ac{idr} nor any other frame following in the decoding order have dependencies to a prior frame. \ac{idr} frames are allowed to have leading pictures, but they must be decodable, thus creating a true random-access point where all subsequent frames in the decoding order can be decoder and displayed. + +In order to increase the coding efficiency, the \ac{hevc} standard introduces a new \ac{rap} picture, referred to as \ac{cra} picture. Theses frames do not refresh the decoding process and allows leading pictures to depend on frames that precede the \ac{cra} picture in the decoding order. This enables a more efficient prediction structure, while maintaining an access point to begin the decoding process. The introduction of \ac{cra} frames in \ac{hevc} were able to improve the coding efficiency by up to 6\% \cite{HEVCCRAPics2011}. + +The \ac{hevc} supports bitstream splicing, which means taking a particular bitstream starting from a \ac{rap} (both \ac{idr} and \ac{cra} frames) and introduce it in another bitstream at a random-access instance. Therefore, whenever this is performed for a bitstream starting from a \ac{cra} frame, the leading frames associated might not be correctly decoded. Therefore, in order to make splicing operation straightforward the \ac{nal} units of the \ac{cra} picture are changed to \ac{nal} units of type \ac{bla}. In case of bitstream splicing starting from an \ac{idr} frame the leading pictures does not have dependencies to frames prior the \ac{rap}, thus converting it to \ac{bla} is not required. + +\subsubsection{Leading and trailing pictures} + +\begin{figure}[tb] +\centering +\input{\FigsPath/HEVC/PictureTypes_Leading_Trailing.tikz} +\caption{Leading and trailing pictures associated with a random-access point.} +\label{Figure2:LeadingTralingPics} +\end{figure} + +Leading pictures correspond to those frame that follow a particular \ac{rap} frame in the decoding order but precedes it in the output order. +A trailing picture is a frame that follows a particular \ac{rap} frame in both decoding and output order and must not depend on any leading frames nor any trailing frame belonging to a previous \ac{rap}. +Figure~\ref{Figure2:LeadingTralingPics} shows examples of leading and trailing pictures. In the figure, frames are illustrated in the output order and the number associated with the frame type (i.e., I, P and B) represents the decoding order. In this example frame $I_1$ corresponds to a \ac{cra} picture. +Leading and trailing pictures are considered to be associated with the closest previous \ac{rap} picture in decoding order, such as frame $I_1$ in Figure~\ref{Figure2:LeadingTralingPics}. +Also, all leading frame of a \ac{rap} frame must precede in decoding order all trailing frames that are associated with the same \ac{rap}. This means that the following order is imposed by the \ac{hevc} standard: 1) \ac{rap} picture, 2) associated leading pictures and 3) associated trailing pictures. + +There is two types of leading pictures as shown in Table~\ref{Table2:PictureTypes}. On the one hand, the \ac{radl} pictures only depend on the associated \ac{rap} pictures and do not have any prior dependencies to trailing pictures associated with previous \ac{rap} frame. These frames can be correctly decoded. On the other hand, \ac{rasl} picture may have dependencies to a prior trailing picture. Therefore, when a random-access is performed to the associated \ac{rap} frame, these frames cannot be correctly decoded, and should be skipped. + +\subsubsection{Temporal sub-layer switching pictures} + +\begin{figure}[tb] +\centering +% \includegraphics[width=0.75\columnwidth]{\FigsPath/HEVC/Pictures_TSA} +\input{\FigsPath/HEVC/PictureTypes_TSA.tikz} +\caption{Temporal sub-layer access pictures example.} +\label{Figure2:TSAExample} +\end{figure} + +A \ac{tsa} picture is a trailing picture that marks a temporal sub-layer switching point. When a decoder is decoding a sub-set of the temporal sub-layers in the bitstream and encounters a \ac{tsa} picture type of the temporal sub-layer just above the maximum temporal sub-layer it is decoding, it is possible for the decoder to switch up to and decode any number of additional temporal sub-layers. For example, picture $P_6$ in Figure~\ref{Figure2:TSAExample} can use the \ac{tsa} frame type since it only depends on frame belonging to temporal sub-layer 0 for prediction, as well as, any subsequent frame predicted from the \ac{tsa} frame $P_6$. In the example of figure if the decoder was only decoding the temporal sub-layer 0 it can after $P_6$ starts decoding both sub-layer 0 and 1 or decode all the available three sub-layers. Although theses frame allows higher flexibility in terms of sub-layer switching, they severely constrain the prediction of trailing the \ac{tsa} frame. + +To reduce the constrains in frame prediction due to sub-layer switching, \ac{hevc} introduces the \ac{stsa} picture. These pictures have similar purpose as the \ac{tsa} frames but they only guarantee that frames in the same temporal sub-layer as the \ac{stsa} frame in the decoding order are able to be correctly reconstructed. This is guaranteed by preventing any dependency to frame preceding the \ac{stsa} to be used as reference in its temporal sub-layer. +One example of a \ac{stsa} frame is given in also given in Figure~\ref{Figure2:TSAExample} in frame $P_2$. This frame can be used to switch to the sub-layer 1 as it does not have dependencies to any prior frame in the same sub-layer ($P_2$ only depends on $I_0$). However, it cannot be classified as \ac{tsa} picture since $P_3$ has dependencies to prior frame in the decoding order to $P_1$. +Summarising, \ac{stsa} frames can be used to switch to a particular sub-layer as the \ac{tsa} can be used to switch to any sub-layer. + +\subsection{Picture partitioning} + + +The high-level segmentation of a picture in \ac{hevc} is achieved through four techniques: regular slices, dependent slices, tiles and \glsfirst{wpp}. Picture partitioning normally serves one or more of the following three purposes: +\begin{description} + \item[Error robustness:] partitioning the picture into smaller self-contained units in + order to gain error robustness by the ability to re-synchronize both the parsing and decoding + processes in case of data losses. + \item[Network adaptation:] adapt to the network constraint of \ac{mtu} size, found for example in \ac{ip} networks. + Such packetisation scheme restricts the maximum number of payload bits within a slice regardless of the size of the coded frame. To keep each slice within this limit and minimise the packetisation overhead, a variable number of coding units is used for each slice. + \item[Parallel processing:] partitioning the coded frame into units which can be processed in parallel. This is achieved by dividing the coding units in such way that they can be encoded and decoded independently of each other’s. +\end{description} + +\subsubsection{Slices} + +The \ac{hevc} standard preserves the slice structure present in the \ac{h264} standard. A slice may comprise either the entire frame or a section of it, and all the associated data (\emph{i.e.}, entropy symbols, prediction and residue information) can be independently decoded. Each slice is transported in a different \ac{nal} unit. As dependencies across slice boundaries are disabled, each slice can be independently reconstructed, regardless if other slices were lost or incorrectly decoded. Slice partitioning is the only tool that can be used for parallel processing in a virtually identical form as in \ac{h264}. As they are independent from each other, it is straightforward to process multiple slices in parallel, without inter-process communication (except for inter-frame prediction). Although this is a simple approach, it incurs in substantial coding overhead due to the higher number of slice headers, and reduction of causal neighbours for prediction (due to lack of predictions across slice boundaries). Another disadvantage of using slices for parallel processing is that they are also used to meet the \gls{mtu} size constrains. Therefore, it might be impossible to meet both requirements using only one technique. + +Another partitioning tool in \ac{hevc} standard is the dependent slices. These slices allow the partitioning of the frame at coding unit boundaries without breaking intra-frame predictions. Moreover, as they have a smaller slice header they are more efficient in terms of signalling overhead than regular slices. Dependent slices are normally used to reduce the end-to-end delay by allowing part of the slice to transmitted while the rest of it are still being processed. + +\begin{figure}[tb] +\centering +\begin{tabular}{x{.45\columnwidth}x{.45\columnwidth}} +\input{\FigsPath/HEVC/Tiles_Slices_Example_1.tikz} & +\input{\FigsPath/HEVC/Tiles_Slices_Example_2.tikz} \\ +(a) & (b) \\ +\end{tabular} +\caption{Examples of the interaction between tiles and slices.} +\label{Figure2:HEVCTiles} +\end{figure} + + +\subsubsection{Tiles} + +A new data partition has been included in the \ac{hevc} standard, referred to as Tiles\cite{HEVCTilesDec2013}. Tiles enable the partition of a coded frame into a smaller group of \acs{ctu} using horizontal and/or vertical boundaries. Figure~\ref{Figure2:HEVCTiles} shows two examples of slice partitioning using tiles. In both cases the frame is partitioned into three tiles by using two vertical boundaries. By defining tiles using only boundaries it allows for an efficient signalling with a small overhead. These boundaries are defined in the \ac{pps} \ac{nal} units. + +Although the \ac{hevc} standard enables the use of both slices and tiles at the same time, it impose constrains in order to reduce the implementation complexity. One of two cases must be used: (i) all \acs{ctu} within a tile must belong to the same slice, or (ii) all \acs{ctu} within a slice must belong to the same tile. These cases are illustrated in Figure~\ref{Figure2:HEVCTiles} (a) and (b), respectively. + +The use of tiles provide several advantages. Specifically, they improve frame partitioning for parallel processing~\cite{WPPTilesParallelHEVC2017} when compared to slices, by reducing the required overhead. Moreover, they also improve the \ac{mtu} size matching and reduce the line buffer memory (by dividing the frame as in Figure~\ref{Figure2:HEVCTiles}). Finally, as tiles allow flexible partitioning,they enable the definition of \acs{roi} for asymmetric video coding. + + +\subsubsection{Wavefront Parallel Processing} + +As mentioned before, the use of either slices or tiles to process each frame using parallel threads require the use of different entropy coding contexts for each slice/tile, in order to make them independently decodable. In consequence the compression efficiency decreases. In order to overcome this issue, the concept of \ac{wpp} was introduced, which enables coding efficient parallel processing~\cite{HEVCParallelCSVT2016}. + +\begin{figure}[tb] + \centering + \input{\FigsPath/HEVC/WPP.tikz} + \caption{Illustration of the wavefront parallel processing.} + \label{Figure2:WPP} +\end{figure} + +In \ac{wpp}, each slice is divided into single rows of \acs{ctu} which can be processed in parallel. Entropy decoding and prediction are enabled across \acs{ctu} of different rows. Therefore, higher compression efficiency can be achieved. In order to re-use the same context in the whole slice each row requires a two \ac{ctu} delay in order to make sure the required dependencies have already been coded/decoded. Figure~\ref{Figure2:WPP} illustrates an example of \ac{wpp} using at least five different threads. Each square corresponds to a \ac{ctu} and the numbers corresponds to the time instance in which a \ac{ctu} may start being processed. Using this staggered start (which appears like a wavefront as represented in the figure), the parallelisation may use as many threads as the number of \acs{ctu} rows available in the video frame. Although, the number of threads does not affect the coding efficiency, the required inter-process communication substantially increases. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Coding structures} +\label{Section2:CodingStructures} + +\glsreset{ctu} + +Despite the fact that the \ac{hevc} standard follows a traditional hybrid video coding architecture, it introduces significant changes in the data structures when compared to the previous standards (e.g., \ac{h264}). +In \ac{h264} each frame is divided into units so-called \acs{mb}, which are composed by a $16\times16$ block of luma samples and two $8\times8$ blocks of chroma samples, assuming video data in YUV 4:2:0 sub-sampling. The \ac{mb} is considered the basic processing unit in the \ac{h264}, which define if the containing pixels are compressed with intra or inter prediction techniques. Each \ac{mb} could also be partitioned into sub-blocks which should follow the same prediction type. + +\begin{figure}[tb] + \centering + \input{\FigsPath/HEVC/BlockPartitionsWithTree.tikz} + \caption{Representation of a CTU ($64\times64$) and its partitioning into CUs and PUs (left); corresponding quad-tree with coding order (right)} + \label{Figure2:HEVCCodingStructure} +\end{figure} + +In \ac{hevc} each picture is divided into blocks so-called \acs{ctu}, which are composed by the luma and chroma \acs{ctb} and corresponding signalling~\cite{BlockPartHEVC2012}. The maximum \ac{ctu} size is configured by the encoder and signalled to the decoder, and should be one of the following: $64\times64$, $32\times32$ and $16\times16$. The use of coding units with larger size in \ac{hevc} was adopted to achieve higher coding efficiency especially in high resolution video. Each \ac{ctu} can be recursively partitioned into four smaller units, referred to as \acs{cu}, until it reaches the minimum unit size, which cannot be inferior to $8\times8$ pixels. Similar to the maximum \ac{ctu} size the minimum size is also defined at the encoding time. Figure~\ref{Figure2:HEVCCodingStructure} illustrates an example of a \ac{ctu} partitioning into \acs{cu}. The partitioning illustrated in the figure leads to the quad-tree structure represented on the right side with the corresponding coding order. + +\begin{figure}[tb] + \centering + \begin{tabular}{x{.8\columnwidth}x{.1\columnwidth}} + \input{\FigsPath/HEVC/PUSymetric.tikz} & (a)\\ + \input{\FigsPath/HEVC/PUAsymetric.tikz} & (b)\\ + \end{tabular} + \caption{Partitioning modes of an inter coded \ac{cu} into \acs{pu}~\cite{HEVCBlock2012}.} + \label{Figure2:InterPuPartitions} +\end{figure} + +The \ac{cu} is the basic processing unit in \ac{hevc} for prediction. It is further partitioned into two types of units, specifically \ac{pu} and \ac{tu}, which are used for prediction and transform operations, respectively. Each \ac{pu} is composed by \acs{pb}, and each \ac{tu} is composed by \acs{tb}. +The decision whether to use intra-frame or inter-frame prediction modes is applied at the \ac{cu} level. However, depending on the partitioning of the \ac{cu} into \acs{pu} different intra or inter coding modes can be used for each \ac{pu}. + +When the intra prediction mode is signalled for a \ac{cu}, the size of the \ac{pb} need to be the same as the \ac{cb}, except when the smaller \ac{cb} is selected. In this case further splitting into four \ac{pb} of smaller size. For example, when the smaller \ac{cb} size is $8\times8$, the encoder may decide to divide it into four \acs{pb} of $4\times4$ pixels, each one having its own intra prediction mode. This is signalled using an one bit flag. +The use of small \acs{pb} is useful for complex regions that required a fine-granularity prediction. + +When inter prediction mode is signalled, \ac{hevc} defines a higher range of partitioning schemes, allowing the \ac{cu} to be divided into one, two or four \acs{pu}. The splitting into four \acs{pu} is only allowed when the \ac{cu} size is equal to the minimum. Otherwise, the splitting should be carried out at the \ac{cu} level which leads to the same partition tree. Figure~\ref{Figure2:InterPuPartitions} illustrates the possible \ac{cu} partitioning into \acs{pu}. One should note that the partition applied to the \ac{cu} is also applied to divide the \ac{cb} into \acs{pb}. +Figure~\ref{Figure2:InterPuPartitions}~(a) shows the symmetric partitions. The $N \times N$ corresponds to the case where the \ac{cu} is not further divided. Then, the standard defines the horizontal and vertical division into two \acs{pu}, corresponding to the $\frac{N}{2}\times N$ and $N \times \frac{N}{2}$ cases, respectively. Finally, the $\frac{N}{2} \times \frac{N}{2}$ case result in four \acs{pu}. +The partitions on Figure~\ref{Figure2:InterPuPartitions}~(b) correspond to the asymmetric partitioning, which is one of the main improvements introduced in \ac{hevc}, as it allow for higher flexibility when performing motion estimation and compensation, leading to higher compression efficiency~\cite{HEVCBlock2012}. These modes are only allowed for \acs{cu} with $16\times16$ pixels or larger. + +%The main key feature of \ac{hevc} is the new block partition structure, which achieves more flexibility and efficiency. Four different block concepts have been introduced in the standard: \ac{ctu}, \ac{cu}, \ac{pu}, and \ac{tu}, all of which have different roles. Figure \ref{fig:HEVCBlockStructure} illustrates an example of a CTU partitioning in HEVC, and it is compared to the \ac{h264} \ac{mb} structure. CTUs are defined as the high-level partition within each slice and can be as large as 64$\times$64, which can be recursively partitioned into four smaller blocks referred to as CUs. The CU is the basic units for the intra/inter prediction and can be as small as 8$\times$8. Subsequently, the CU are divided into up to four PUs which are used for inter prediction. For intra coding PU is not used. Finally, the CU is divided into a transform unit tree, which may include several TUs, in a quad-tree structure. Transform units are the basic units for the transform operations applied to the coding residue. +% +%Each CU may be split into two or four Pus resulting in either symmetric or asymmetric partitions. Asymmetric partitioning of the CU into two PUs enables more efficient motion estimation since it allows more flexible partitioning of the picture~\cite{BlockPartHEVC2012}. This scheme is able to improve the coding performance by up to $1\%$ when compared with rectangular symmetric partitioning. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Prediction methods} +\label{Section2:PredictionModes} + +Prediction methods are responsible for an important role in image and video coding standards, due to their ability to reduce the signal redundancy based on the previously encoded pixels. +This section describes the prediction techniques used in the \ac{hevc} standard. +These techniques include the directional intra prediction for efficiently reduce spatial redundancy and inter-frame motion compensation for remove the temporal redundancies. + +\subsection{Intra-frame} + +In previous standards, intra-frame prediction was used to efficiently reduce the spatial redundancy in video frames. Intra predicted blocks are obtained using the previously encoded pixels around the current \ac{pu}. \ac{hevc} adopts three types of intra-frame prediction: directional, Planar and DC~\cite{IntraCodingHEVC2012,HEVCIntraperformance2012}. Figure~\ref{Figure2:IntraModesHEVC} shows the available intra prediction mode available in \ac{hevc}. As shown, the predictions are applied to square blocks only. The example in the figure corresponds to \ac{pu} of $8\times8$ pixels, but the standard allows block sizes ranging from $4\times4$ to $32\times32$ +The encoder tests all the available intra prediction modes and signals the best mode to the decoder. The decoder must generate an identical prediction block, based on the reconstructed pixels. + +The reconstructed reference samples used in the prediction process belong to +the neighbouring blocks at left-down, left, top-left, top and top-right positions, as +can be noticeable in Figure~\ref{Figure2:IntraModesHEVC}. Given a \ac{pu} with size $N \times N$, intra prediction requires a top neighbouring row of $2N$ pixels, a left column of $2N$ pixels plus the top-left neighbouring pixel. +Due to unit boundaries, e.g., slice or tile boundaries, neighbouring pixels might not be all available to apply the intra prediction. In addition, when constrained intra prediction is enabled, reference samples belonging to inter-predicted \acs{pu} are not used in order to remove error propagation from potentially erroneously reconstructed reference frames. +In order to overcome this issue and being able to apply all possible modes, \ac{hevc} uses reference sample substitution to replace the unavailable reference samples with the closer available ones, so that all intra prediction modes can be used. + +\begin{figure}[tb] + \centering + %\begin{tabular}{x{.5\columnwidth}x{.5\columnwidth}} + %\input{\FigsPath/HEVC/IntraPredictions.tikz} & + %\input{\FigsPath/HEVC/IntraPredictionsExample.tikz} \\ + %\end{tabular} + %\caption{List of the available \ac{hevc} angular intra prediction modes (left) and example of the angular mode 29 (right).} + \includegraphics[width=0.9\columnwidth]{\FigsPath/HEVC/IntraPrediction} + \caption{Example of \ac{hevc} intra prediction modes for a $8\times8$ block~\cite{BookHEVC2014}.} + \label{Figure2:IntraModesHEVC} +\end{figure} + +\subsubsection{Angular modes} + +Directional intra prediction was first introduced in the \ac{h264} standard~\cite{H264OverviewJul2003}, to estimate regions with a structured texture or directional edge. +In its process, directional prediction projects the reconstructed samples in the block neighbourhood along a specific direction. The \ac{hevc} improves the number of available direction from 8 in \ac{h264} to 33 possible directions~\cite{IntraCodingHEVC2012}. The result of the projections in each possible direction is illustrated in Figure~\ref{Figure2:IntraModesHEVC}. + +The angular modes are designed to provide a dense coverage near the horizontal and vertical directions. It is not noticeable in the Figure~\ref{Figure2:IntraModesHEVC}, but the angular directions are more coarse as it gets closer to the diagonals. This reflects the observed statistical prevalence of the angles and the prediction efficiency. + +In order to improve the efficiency of angular prediction the when the reference samples need to be projected a bilinear interpolation is used from the two closest pixels using a $1/32$ accuracy. + +\subsubsection{Planar and DC} + +In contrast, with the angular mode which cover those regions with strong direction edges, \ac{hevc} also implements the Planar and DC modes targeting smooth regions. These modes are useful for smooth regions as they result in a linear or constant prediction. + +The DC mode uses the average or the neighbouring reference samples to generate a constant prediction for the current \ac{pu}. +The Planar mode predicts the \ac{pu} through linear predictions from vertical and horizontal slopes derived from the neighbouring reference samples. As the Planar mode used linear surface for prediction it prevents the discontinuities at \ac{pu} boundaries. Both of these modes are also illustrated in Figure~\ref{Figure2:IntraModesHEVC}. + + +\subsubsection{Sample smoothing} + +In \ac{hevc} sample smoothing is applied at two distinct occasions two improve the overall performance of the prediction accuracy. Firstly, the reference samples are filtered and secondly the \ac{pu} is filtered after prediction. This improves the previous standard \ac{h264} where only reference samples were filtered. + +In \ac{hevc} the reference sample smoothing is conditionally applied, based on the \ac{pu} and intra prediction mode. Similar to \ac{h264}, a three tap $\left[1\ 2\ 1\right]/4$ smoothing filter is applied, except for \ac{pu} sizes of $4\times4$. The prediction results from Planar mode is also filtered when the \ac{pu} size is greater than $8\times8$, and it is not used for the DC mode as it would not be useful. + +The second filter in \ac{hevc} is the boundary smoothing. This filter aims at removing the discontinuities along block boundaries due to intra prediction. The application of this filter is more exclusive, and it is only applied to the DC and angular mode 10 and 26 (exactly the horizontal and vertical directions, respectively). A two-tap filter $\left[3\ 1\right]/4$ fed by the original samples and the adjacent reference sample. For the DC mode this filter is applied to both the top row and the left-most column. In case of the angular mode 10 and 26 only the left-most column and the top row are filtered, respectively. Only the mentioned boundaries are filtered, as they are the most likely to introduce discontinuities. The result of the boundary smoothing can be noticed in Figure~\ref{Figure2:IntraModesHEVC} for the aforementioned modes. + +%\subsubsection{Rough Mode Decision} +%\cite{HEVCROughModeDecision} + +\subsection{Inter-frame} + +The efficiency of video coding algorithms consistently relies on inter-frame prediction techniques used to reduce temporal redundancy. The underlying idea of inter-frame prediction is to estimate the current frame from one or more previously encoded reference frames using block-based motion compensated. The most common technique used for motion estimation in \ac{hevc}, as well as in \ac{h264}, is the boundary matching algorithm. In the \ac{hevc} standard an \ac{amvp} for efficient motion vector signalling, which defines the Merge Mode and differential coding of the motion information. + +\subsubsection{Motion estimation and compensation} + +Motion estimation consists in searching for the current block with the highest similarity with the target block to be predicted (\ac{pu} in \ac{hevc}), using block-matching algorithms over the previously encoded frames, i.e. the reference frames. Typically, reference frames correspond to past or future temporal frames. The frame encoding order determines which reference frames are selected and whether future frames are available. Typically, video frame type P normally are predicted from the past as B frame are predicted from frames both in past and future. +Block-matching consist on subtracting the current block from one of equal size in the reference frame in a pre-defined window. The reference block that results in the lowest difference is selected for prediction. +In order to signal the selected block to be used for motion compensation the different between the target and the reference block positions are computed to obtain a two-dimensional vector, referred to as \ac{mv}. This vector is signalled to the decoder, in order to apply the same prediction. + +Similar to \ac{h264} the \ac{hevc} allows for either one or two \ac{mv} to be selected to compensate the target \ac{pu}. This results in uni-predictive and bi-predictive coding, respectively. \ac{hevc} also allows for weighted prediction, which consists in scaling and offset operations the predicted block using determined factors which are explicitly sent to the decoder. +The motion vectors are obtained from frames included in the reference frame list. Each frame is identified by an index, therefore to fully describe inter-frame prediction a combination of two parameters is required: \ac{mv} and reference index. + +In recent standards motion compensation has been performed using a variable block size approach. In \ac{h264} block sizes ranging from $4\times4$ up to $16\times$ pixels were allowed. In \ac{hevc} a significant improvement was introduced by extending the block sizes up to $64\times64$ pixels. As mentioned before it also allows asymmetric rectangular block sizes. +By using larger prediction units, \ac{hevc} is able to save bits in the signalling of the selected \ac{mv}, because less \ac{mv} are required. + +In order to increase the efficiency of motion estimation using fractional \acs{mv} are used. This has been proven to improve accuracy of motion compensation. To accomplish this, frame interpolation is performed to generated non-integer pixels in the reference frame. Both \ac{h264} and \ac{hevc} standards use \acs{mv} up to quarter-pixel accuracy for luma component. For chroma samples, the \ac{mv} accuracy is determined according to the chroma sampling format, which for 4:2:0 results in eighth-pixel accuracy. In contrast, with the two-stage approach used in \ac{h264}, \ac{hevc} uses a single separable approach for the interpolation process without intermediate rounding operations. This improves the precision and simplifies the architecture of the fractional interpolation. + +The block matching-based motion estimation is one of the most computationally complex procedures in the \ac{mv} encoder. Normally, the optimal solution for motion estimation is provided by the full-search algorithm, which tests all the possible candidate blocks in the search area. Although the full-search algorithm achieves the optimal performance, it requires the highest computational complexity. Therefore, sub-optimal algorithms have been proposed over the years to reduce the motion estimation complexity and decrease encoding time\cite{HEVCME2012,HEVCME2017,HEVCME2014}. +For example, the HM reference software implements a fast algorithm based on Test Zone Search ~\cite{HEVCME2012}. +Since motion estimation is a non-normative procedure, encoders may implement any alternative search algorithms that suits its needs (i.e., compression ratios or encoding time). + + +\subsubsection{Reference picture management} + +Another improvement in \ac{hevc} is related with the reference picture management scheme. \ac{hevc} introduces the \ac{rpset} concept~\cite{HEVCHighLevelReferenceDez2012}, which defines how previously decoded pictures are managed in the \ac{dpb}. Decoded frame in the \ac{dpb} are mark grouped in one of the following categories: (i) \emph{short-term reference}, (ii) \emph{long-term reference} and (iii) \emph{unused for reference}. Once a frame is marked as \emph{unused for reference} it is no longer used for motion compensation and it will be discarded from the \ac{dpb} after being displayed/outputted. +In \ac{h264} a sliding windows approach was used to implicitly manage the reference picture list. This process discards a given frame whenever the \ac{dpb} has the number of defined reference frames (which is signalled by the encoder). Moreover, \ac{h264} used an explicitly \ac{mmco} to send commands to the decoder and change the normal reference picture management process. + +\begin{table}[tb] + \centering + \caption{Reference Picture Set example.}\label{Table2:ReferencePictureListExample} + \small + \TABLESVSPACE + \begin{tabular}{x{1.5cm}|ccc} + \toprule + Frame & \multicolumn{3}{c}{\ac{rpset} (\{frame, is used\}) } \\ + \midrule + $I_1$ & -- & & \\ + $P_5$ & \{$I_1$, 1\} & & \\ + $B_6$ & \{$I_1$, 1\} & \{$P_5$, 1\} & \\ + $B_7$ & \{$I_1$, 1\} & \{$B_6$, 1\} & \{$P_5$, 0\}\\ + $B_8$ & \{$B_6$, 1\} & \{$P_5$, 1\} &\\ + \bottomrule + \end{tabular} +\end{table} + +In the \ac{hevc} standard instead of a signalling relative changes, the status of the \ac{dpb} is signalled for every slice. +The most fundamental difference with the \ac{rpset} concept compared to \ac{mmco}/sliding window of \ac{h264} is that for each particular slice a complete set of the reference frames that are used by the current or any subsequent frames are be provided. +Thus, a complete set of all frames that must be kept in the \ac{dpb} (in the present and future) are signalled. +This is different from the \ac{h264} scheme where only relative changes to the \ac{dpb} are signalled. With the \ac{rpset} concept, no information from earlier frames in decoding order is needed to maintain the correct status of the \ac{dpb}. Table~\ref{Table2:ReferencePictureListExample} shows the \ac{rpset} for some frames illustrated in Figure~\ref{Figure2:LeadingTralingPics}. As expected for the \ac{cra} frame ($I_2$) nothing is signalled. For the remaining frames, two types of signalled are used: frame number and a flag to mark if it is currently used. This is required because any frame not listed in a given \ac{rpset} will be discarded and subsequent frames cannot use it as reference. + + + +The \ac{rpset} concept in \ac{hevc} provides a basic level of error robustness to the reference picture management. In \ac{h264} whenever a reference frame is lost the decoder can look at the value of the +\emph{FrameNum} (incremental value associated with each reference frame) to detect the loss event. However, as gaps in \emph{FrameNum} are allowed the decoded might not be able to decide the missing frame was intentional or if it resulted from a loss event in the transmission. Moreover, even if the decoded if able to detect the loss event it might not be able to correctly compute the value the \ac{poc} associated with the concealed frame. Using the \ac{rps} concept in the \ac{hevc} the decoder is always able to detect loss events and correctly identify the missing frames, preventing it from use wrong frames as reference. + + +\subsubsection{Merge mode} + +\begin{figure}[tb] +\centering +\input{\FigsPath/HEVC/MergeModeHEVC.tikz} +%\input{\FigsPath/HEVC/MergeModeHEVCTwoFrames.tikz} +\caption{Illustration of the \ac{mv} prediction candidates in the AMVP.} +\label{Figure2:MVCandidates} +\end{figure} + +The Merge mode was introduced in \ac{hevc} standard, as a new technique to derive motion information from spatial and temporal neighbouring blocks~\cite{MotionCodingHEVC2013}. This technique extends the concepts of the direct and skip modes used in \ac{h264}~\cite{H264OverviewJul2003}, using a more sophisticated approach with two key differences. +Firstly, the Merge Mode allows for more \ac{mv} candidates. Thus, an index value is included to select one out of several \ac{mv} candidates. Secondly, it explicitly identifies the reference picture index used for the selected candidate, as well as, the reference picture list. This results in an improved flexibility, compared with the predefined values used in \ac{h264}. The \ac{mv} candidates allowed in the Merge Mode\cite{MotionCodingHEVC2013} are illustrated in Figure \ref{Figure2:MVCandidates}. The figure illustrates the spatial candidates' positions ($A_0$, $A_1$, $B_0$, $B_1$, $B_2$). +Moreover, it also uses a temporal motion vector candidate, derived from the co-located position on the temporal adjacent frame at positions $T_0$ and $T_1$. From this \ac{mv} candidate list the unavailable candidates are not accounted and duplicated candidates (i.e. with the same motion information) are removed. +In the Merge Mode, a $C-1$ number of candidates are used, selected from the aforementioned order and one temporal \ac{mv} candidate is selected to fulfil the total number of \ac{mv} ($C$). The value of $C$ is defined at the slice header and a flag controlling the use of temporal \ac{mv} candidates is sent at the \ac{pps}-level. +For B-slices, additional \ac{mv} candidates are generated by choosing two existing candidates according to a predefined +order for reference picture list 0 and list 1. A total number of twelve combination are specified in \ac{hevc} which are listed in ~\cite{HEVCOverviewDez2012}. For P-slices if the number of candidates is less than $C$ zero motion vectors are included to fill the remaining entries. + +The Merge Mode allows for the signalling of a \ac{mv} candidate and the residual information. In \ac{hevc}, the Skip mode, i.e., transmitting only the \ac{mv} without residual information, is treated as a special case of the Merge Mode. In this case only a skip flag is enabled, and the corresponding merge index is transmitted. + + +\subsubsection{Motion vector prediction} + +The \ac{hevc} also improves the performance of the motion vector coding for those cased where the Merge Mode is not used. \ac{hevc} always use a difference encoding approach for efficiently represents the motion information. +It considers the same \ac{mv} candidate of the Merge mode, illustrated in Figure~\ref{Figure2:MVCandidates}, to derive the motion vector predictors from which the difference vector is obtained. +From these \ac{mv} candidates only two are selected based on the following: +\begin{enumerate} +\item Left candidate: first available from $A_0$ and $A_1$. +\item Top candidate: first available from $B_0$, $B_1$ and $B_2$. +\item Temporal candidate: first available from $T_0$ and $T_1$. +\end{enumerate} +Therefore, the use of the temporal \ac{mv} candidate is limited to those cases where the spatial neighbours are not able to provide two valid candidates. + +\ac{hevc} only allows a much lower number of candidates to be used in the motion vector prediction process, since it can still send a motion vector difference. Moreover, the encoder will perform motion estimation for each candidate, which increase the encoding complexity for every possibility tested. For those cases where the reference index is not the same in the current \ac{pu} and the neighbouring one, a scaling version of the \ac{mv} is used. + +%In the \ac{h264} standard, along different motion compensated coding techniques~\cite{H264OverviewJul2003}, the so-called P\_SKIP mode was introduced. In this mode, neither the quantised residual information, nor the \ac{mv} is transmitted. This mode is useful for large areas with no change or just constant motion, and it is only enabled for $16\times16$ blocks, \emph{i.e.}, the whole \ac{mb}. The \ac{mv} for the P\_SKIP mode is derived based on the surrounding neighbour MBs\cite{H264MotionVectorPrediction2003}. When no neighbours are available, or the left or top \acs{mv} are equal to zero, a zero \ac{mv} is used. Otherwise, the \ac{mv} is derived from the median value of the following three neighbours: left, above and above right, as shown in Figure \ref{SkipMergeMode}~(a) by $MV_A$, $MV_B$ and $MV_C$, respectively. Whenever the $MV_C$ is not available, the $MV_D$ is used instead. +% + +%In contrast with the \ac{h264} standard, the \ac{hevc} standard allows the usage of a temporal \ac{mv} predictor, as well as, more spatial candidates to predict the current motion vector. Despite the improvements in the flexibility and coding efficiency, the increased levels of dependencies between the \acs{mv} make the motion information more prone to errors. Especially the introduction of the temporal candidate in the set of possible \acs{mv} increases the dependencies between subsequent frames. This leads to higher quality degradation in the presence of transmission errors, when compared to \ac{h264}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Transform and quantisation} +\label{Section2:TransformQuant} + + +\glsreset{tb} + +Similarly, to the previous standards, \ac{hevc} uses a transform followed by a quantisation to compress the residual signal. +The residual block is partitioned into multiple square blocks, referred to as \ac{tb}. + +\subsubsection{Core transform} + +A 2D transform of the residual information is obtained by applying 1D transforms in both the horizontal and vertical directions. The core transform matrices are obtained from the \ac{dct} basis functions to define a integer matrix of $32\times32$ points. Sub-sampled versions of this matrix are used to derive the remaining transform matrix sizes down to $4\times4$. + +A special case is applied for the residual coding of a $4\times4$ intra coded block. In this case an alternative integer transform based on the \ac{dst}. +The motivation for \ac{dst} is related to the statistics of the residual signal of these blocks, which tend to present higher residue as the distance to the boundary used for prediction (i.e, top or left boundary) increases. +Experiments have shown that DST provides up to 1\% of bitrate savings for the compression of $4\times4$ intra predicted blocks, without significantly increase the computational complexity \cite{DCTDSTComp2013}. + +\subsubsection{Quantisation} + +\ac{hevc} standard follows the same principle introduced in \ac{h264} for the quantisation method. After obtaining the residual transform a quantisation is applied using a uniform-reconstruction quantiser scheme based on the \ac{qp}. The standard supports \ac{qp} values ranging from 0 to 51, and a increase of 6 in the \ac{qp} corresponds to doubling the quantisation step. This procedure is the main responsible to introduce the coding errors as it performs many-to-one mapping, which cannot be reversed. + + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Entropy coding} +\label{Section2:Entropy} + +In contrast to \ac{h264} standard, which allows two entropy coding methods, \ac{hevc} only uses the \ac{cabac} \cite{HEVCCabac2003}. \Gls{cabac} is an arithmetic coding method that only uses binary symbols, considering different probability models for each symbol. Entropy coding efficiency is closely linked with the context model selected. Therefore, this was carefully designed in \ac{hevc} standard, extending the functionality presented in \ac{h264} +For example, the \ac{hevc} standard exploits the depth of the partition tree or residual transform tree in order to derive the context models for several syntax elements. +Regarding transform coefficient coding, \ac{cabac} uses a scanning method to firstly organise the coefficients and then transmits the position of the last non-zero transform coefficient. +Moreover, it transmits, a significance map, the sign bits and the levels for the transform coefficients. +Three coefficient scanning methods are available: the diagonal up-right, the horizontal and the vertical scan. +The coefficient scanning is implicitly selected and always performed in $4\times4$ sub-blocks for all transform block sizes. +% Details about the context models selection and the compression of each syntax element of \ac{hevc} can be found in [1]. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{In-loop filters} +\label{Section2:Filters} + +The \ac{hevc} standard uses two filtering procedures over the reconstructed pixels before writing them into the frame buffer, namely the Deblocking Filter and the \ac{sao} filter. +The main purpose of deblocking filter is to reduce blocking artefacts caused by the block-based coding tools, operating similarly to the deblocking filter of the \ac{h264} standard. +DBF is applied to the samples adjacent to the \ac{pu} or \ac{tu} boundaries, that are aligned on a $8\times8$ grid. Such grid-based restriction reduces the computational complexity and improves parallel-processing +implementation which is one of the main goals of \ac{hevc}. Three strength levels can be used by filter depending on the coded block characteristics. + +The \ac{sao} is a non-linear filter, that is adaptively applied to all samples of the image, after the deblocking filter. In its operation, the \ac{sao} filter modifies the samples +by adding an offset value, extracted from lookup tables transmitted by the encoder. For each \ac{ctu}, the encoder decides whether to apply the \ac{sao} filter or not, and if used, one of the two available filter types is applied, namely the band offset or the edge offset. In the band offset mode, the added offset value depends on the sample amplitude, while the edge offset uses the gradient to classify and derive the offset value. + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Summary} +\label{Section2:Summary} + +In this chapter we described the main features of the \glsfirst{hevc} standard, giving a special focus on the high-level features, coding structures and newly introduced prediction modes. + +At the high-level \ac{hevc} introduces new \ac{nal} units types, not only for new signalling features but also for the new pictures types introduced. \ac{hevc} introduces new picture types to handle different types of random-access strategies, which allow more flexibility and higher coding efficiency, when compared to the random-access in previous standard such \ac{h264}. + +Another major improvement comes from the inter-frame prediction. On the one hand new high-level reference picture management was introduced, which increase the error robustness, as it explicitly defines the active reference frame and the \ac{dpb} status. On the other hand it introduces the \ac{amvp} and the Merge mode which efficiently reduce the required bits or motion signalling. + +In general, \ac{hevc} achieves higher coding performance than its predecessors. Previous studies reported an average bitrate saving of approximately 50\% for the same subjective quality. Such amount of bitrate saving is normally related with increased prediction structures with higher dependencies. Such dependencies are more sensible to data loss, which will increase the error propagation to subsequent coding structures. Therefore, it is emergent to evaluate the impact of the transmission errors in the \ac{hevc} and develop efficient techniques to cope with data loss and reduce the error propagation. + + + + diff --git a/022_ErrorResilience/ErrorResilienceCategories.tex b/022_ErrorResilience/ErrorResilienceCategories.tex new file mode 100644 index 0000000..7340610 --- /dev/null +++ b/022_ErrorResilience/ErrorResilienceCategories.tex @@ -0,0 +1,53 @@ +\TABLESVSPACE +\footnotesize +\begin{tabular}{m{3.2cm}m{5.8cm}m{5.2cm}} + \toprule + \centering\let\newline\\\arraybackslash\hspace{0pt} \textbf{Category} & \centering\let\newline\\\arraybackslash\hspace{0pt} \textbf{Examples} & + \centering\let\newline\\\arraybackslash\hspace{0pt} \textbf{Advantage} \\ \hline + Localization & + \begin{tabular}{@{}l@{}} + $\bullet$ End-to-end distortion\\ + $\bullet$ Adaptive intra refresh\\ + $\bullet$ Reference picture selection\\ + $\bullet$ Leaky predictions\\ + \end{tabular}& + Reduces error propagation \\ + \midrule + Data partitioning and redundancy & + \begin{tabular}{@{}l@{}} + $\bullet$ Slice classification\\ + $\bullet$ Flexible macroblock order\\ + $\bullet$ Redundant pictures \\ + $\bullet$ Multiple-description coding \\ + \end{tabular}& + Unequal error protection, transport priority and multi-channel transmission \\ + \midrule + Error concealment by post-processing & + \begin{tabular}{@{}l@{}} + $\bullet$ Linear interpolation \\ + $\bullet$ Boundary matching\\ + $\bullet$ Motion vector extrapolation \\ + \end{tabular}& + Does not require extra information and adapts to every stream \\ + \midrule + Concealment-aware & + \begin{tabular}{@{}l@{}} + $\bullet$ Error concealment R-D optimisation\\ + \end{tabular}& + Increases the error concealment performance \\ + \bottomrule +\end{tabular} + +%Data partitioning & +%Enables unequal error protection and transport priority & +%\begin{tabular}{@{}l@{}} +%$\bullet$ Slice classification\\ +%$\bullet$ Flexible macroblock order\\ +%\end{tabular}\\ +%\midrule +%Redundant coding & Increase the decoder robustness & +%\begin{tabular}{@{}l@{}} +%$\bullet$ Redundant pictures \\ +%$\bullet$ Multiple-description coding \\ +%$\bullet$ Reversible variable-length coding \\ +%\end{tabular}\\ diff --git a/022_ErrorResilience/Figures/BMAAlgorithm.pdf b/022_ErrorResilience/Figures/BMAAlgorithm.pdf new file mode 100755 index 0000000..d8fe601 Binary files /dev/null and b/022_ErrorResilience/Figures/BMAAlgorithm.pdf differ diff --git a/022_ErrorResilience/Figures/BMAAlgorithm.tex b/022_ErrorResilience/Figures/BMAAlgorithm.tex new file mode 100755 index 0000000..e0f4437 --- /dev/null +++ b/022_ErrorResilience/Figures/BMAAlgorithm.tex @@ -0,0 +1,120 @@ +%User Variables +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric} +\usetikzlibrary{decorations.pathreplacing,decorations.pathmorphing,shapes} +\usetikzlibrary{matrix,shapes.symbols,plotmarks,decorations.markings,shadows} + +\newcommand{\block}{1cm} +\newcommand{\scale}{0.7} + +\begin{tikzpicture}[scale=\scale,every node/.style={minimum size=0.6*\scale*\block}, inner sep=0pt,on grid] + + \definecolor{lightgray}{rgb}{0.80,0.80,0.80}; + \definecolor{darkgray}{rgb}{0.40,0.40,0.40}; + + \tikzstyle{arrow}=[draw, <->,shorten <=1pt,shorten >=1pt,>=stealth',thick] + \tikzstyle{pel}=[circle, rounded corners ,draw=lightgray!80, fill=lightgray!80, thick, scale=0.5] + \tikzstyle{outerpel}=[pel, fill=gray, draw=gray!90, thick] + \tikzstyle{innerpel}=[pel, fill=black!90, draw=black!90, thick] + + % Image pixels + \foreach \x in {-2,...,9} + \foreach \y in {-2,...,9} + { + \node at (\x+0.5,\y+0.5) [pel]() {}; + } + + + % outer boundary + \foreach \x in {0,...,7} + \foreach \y in {-1,8} + { + \node at (\x+0.5,\y+0.5) [outerpel](ref_tp) {}; + } + \foreach \x in {-1,8} + \foreach \y in {0,...,7} + { + \node at (\x+0.5,\y+0.5) [outerpel](ref_tp) {}; + } + + % inner boundary + \foreach \x in {0,...,7} + \foreach \y in {0,7} + { + \node at (\x+0.5,\y+0.5) [innerpel](ref_tp) {}; + } + \foreach \x in {0,7} + \foreach \y in {1,...,6} + { + \node at (\x+0.5,\y+0.5) [innerpel](ref_tp) {}; + } + + +% \foreach \x in {0,8} +% \foreach \y in {1,...,2,4,5,...,7} +% { +% \node at (\x+0.5,\y+0.5) [pel](ref_lft) {}; +% } + + + % Center +% \draw[draw=black,thick, opacity=1] (\block,\block) rectangle ++(7*\block,7*\block); + \node[anchor=center,scale=\scale, text centered] at (4,4) {Lost Block}; + + \draw[draw=black, thick] (0*\block,-1) -- ++(0,10*\block); + \draw[draw=black, thick] (8*\block,-1) -- ++(0,10*\block); + \draw[draw=black, thick] (-1*\block,0*\block) -- ++(10*\block,0); + \draw[draw=black, thick] (-1*\block,8*\block) -- ++(10*\block,0); + +% \draw[draw=black, thick] (8*\block-8+4,-1) -- (8*\block-8+4,10); + + % Lost pixel + %\node at (4*\block+\block/2, 3*\block+\block/2) [pel, fill=black, draw=black, thick](lostpel) [label=north east: $lp$] {}; + + + % Reference pixels + %\node at (4*\block+\block/2, 8*\block+\block/2) [pel, fill=black, draw=black, thick](ref_tp) [label=north west: $p_1$] {}; + %\node at (0*\block+\block/2, 3*\block+\block/2) [pel, fill=black, draw=black, thick](ref_lft) [label=north west: $p_2$] {}; + %\node at (4*\block+\block/2, 0*\block+\block/2) [pel, fill=black, draw=black, thick](ref_bottm) [label=south east: $p_3$] {}; + %\node at (8*\block+\block/2, 3*\block+\block/2) [pel, fill=black, draw=black, thick](ref_rght) [label=north east: $p_4$] {} ; + + %\path [arrow] (lostpel) -- node[left] { $d_1$} (ref_tp); + %\path [arrow] (lostpel) -- node[above] { $d_2$} (ref_lft); + %\path [arrow] (lostpel) -- node[left] { $d_3$} (ref_bottm); + %\path [arrow] (lostpel) -- node[above] { $d_4$} (ref_rght); + + + + \node[anchor=west,scale=\scale] at (-1,10.1) (pixel_label) {Pixel}; + \node at (-1.6,8.35) (pel) {}; + \path [-latex,anchor=center] (pixel_label) edge (pel); + + \node[anchor=west,scale=\scale, text centered] at (1,-2.2) (inner_pixel_label) {Internal Boundary}; + \node at (4-0.4,0.6) (inner_pel) {}; + \path [-latex,anchor=center] (inner_pixel_label) edge (inner_pel); + + + + \node[anchor=west,scale=\scale, text centered] at (4.5,10.2) (outer_pixel_label) {External Boundary}; + \node at (6-0.5,8.4) (outer_pel) {}; + \path [-latex,anchor=center] (outer_pixel_label) edge (outer_pel); + + \node[anchor=west,scale=\scale, text centered] at (1.7,9.1) (ref_pixel_label) { \Large $f_{(x,y,t)}$ }; + \node at (1.4,7.4) (ref_pel) {}; + \path [-latex,anchor=center] (ref_pixel_label) edge (ref_pel); + + \node[anchor=center,scale=\scale, text centered] at (4,9) (ref_pixel_label) { \Large T }; + \node[anchor=center,scale=\scale, text centered] at (4,-1) (ref_pixel_label) { \Large B }; + \node[anchor=center,scale=\scale, text centered] at (-1,4) (ref_pixel_label) { \Large L }; + \node[anchor=center,scale=\scale, text centered] at (9,4) (ref_pixel_label) { \Large R }; + + + + %\node[anchor=west,scale=\scale] at (5.8,6.0) (edge_label) {Contour}; + %\path [-latex] (edge_label.south) edge (7.15,5.2); + +%\node at (6.5,-1.3) (edge_start) {}; +%\node at (10.3,8.1) (edge_end) {}; + %\path [-,thick] (edge_start.north) edge[out=80, in=190] (edge_end.west); + + +\end{tikzpicture} diff --git a/022_ErrorResilience/Figures/BiDirMotionCopy.tex b/022_ErrorResilience/Figures/BiDirMotionCopy.tex new file mode 100755 index 0000000..c8b59bf --- /dev/null +++ b/022_ErrorResilience/Figures/BiDirMotionCopy.tex @@ -0,0 +1,129 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + + +\newcommand{\scale}{1} +\newcommand{\TikZyslant}{-0.2} +\newcommand{\xslant}{0} +\newcommand{\xframeshift}{5cm} +\newcommand{\yframeshift}{0cm} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + + + + +\begin{tikzpicture}[scale=\scale,every node/.style={minimum size=1cm},on grid ] + + \tikzstyle{pel}=[rectangle,draw=black, thick] + \tikzstyle{refblock}=[rectangle,draw=darkgray, thick] + \tikzstyle{projectedblock}=[rectangle,draw=black, thick,dashed] + \tikzstyle{refmv_linestyle}=[-,very thick] + \tikzstyle{refmv}=[refmv_linestyle,decoration={markings,mark=at position 1 with {\arrow[ultra thick,scale=1]{latex}}}, + postaction={decorate}] + + \tikzstyle{arrow}=[draw, ->,>=stealth',thick,,decoration={markings,mark=at position 1 with {\arrow[ultra thick,scale=1]{stealth}}}, + postaction={decorate}] + + \tikzstyle{every node}=[font=\large] + + \tikzstyle{frame_contour}=[ultra thick,fill=white,draw=black,fill opacity=.7] + + \newcommand{\DrawFrame}{ + %outside line + \draw[thick,fill=white,draw=black,fill opacity=.7](0,0) rectangle (3,3); + \foreach \x in {0,2} + \foreach \y in {0,2} + \draw[black,loosely dashed] (\x,\y) +(0,0) rectangle ++(1,1); + \foreach \x in {1} + \foreach \y in {1} + \draw[black,loosely dashed] (\x,\y) +(0,0) rectangle ++(1,1); + } + + \begin{scope}[ yshift=0, xshift=0] + + % Frame t+1 + \begin{scope}[ yshift=2*\yframeshift, xshift=2*\xframeshift, + style={yslant=\TikZyslant,xslant=\xslant}] + + \coordinate (Bottom_V1) at (2.5,0); + + \DrawFrame %Draw a frame + + \coordinate (RefBlock) at (1,2); + \draw[refblock] (RefBlock) rectangle ++(1,-1); + \draw[refmv,style={yslant=-\TikZyslant,xslant=-\xslant},fill=red,draw=red] (RefBlock) to ++(-2,0.2); + + \node[scale=0.8] at (-0.7,2.25) { $mv_{t+1}$ }; + + \end{scope} + + + % Frame t-1 + \begin{scope}[ yshift=0, xshift=0, style={yslant=\TikZyslant,xslant=\xslant}] + + + \coordinate (TM2B3) at (1-0.8,2-0.8); + \coordinate (TM2B4) at (3+0.3,2-0.4); + + % Draw Motion vectors + %\draw[refmv] (T0B3) to (TM2B3); + + \DrawFrame; %Draw a frame + + \coordinate (RefBlock) at (1,2); + \draw[refblock] (RefBlock) rectangle ++(1,-1); + \draw[refmv,style={yslant=-\TikZyslant,xslant=-\xslant},fill=blue,draw=blue] (RefBlock) to ++(-2,-0.4); + + \node[scale=0.8] at (-0.7,1.8) { $mv_{t-1}$ }; + + + \end{scope} + + + + + % Frame t + \begin{scope}[ yshift=1*\yframeshift, xshift=1*\xframeshift, + style={yslant=\TikZyslant,xslant=\xslant}] + + %Define Points + \coordinate (TM1B1) at (1-0.7,4-0.8); + \coordinate (TM1B2) at (3-0.6,4-0.5); + + \DrawFrame; %Draw a frame + + \coordinate (RefBlock) at (1,2); + \node[scale=0.6] at (1.5,1.5) { LB }; + + + \draw[refblock] (RefBlock) rectangle ++(1,-1); + \draw[refmv,style={yslant=-\TikZyslant,xslant=-\xslant},fill=red,draw=red] (RefBlock) to ++(-2,0.2); + \node[scale=0.8] at (-0.7,2.25) { $mv_{t+1}$ }; + \draw[refmv,style={yslant=-\TikZyslant,xslant=-\xslant},fill=blue,draw=blue] (RefBlock) to ++(-2,-0.4); + \node[scale=0.8] at (-0.7,0.9) { $mv_{t-1}$ }; + \end{scope} + + % Frame number + \begin{scope}[xshift=1.5cm,yshift=3.2cm] + \tikzstyle{frame_label}=[anchor=center,scale=0.9] + \node[frame_label] at (0*\xframeshift,\yframeshift*3) { $f_{t-1}$ }; + \node[frame_label] at (1*\xframeshift,\yframeshift*3) { $f_{t}$ }; + \node[frame_label] at (2*\xframeshift,\yframeshift*3) { $f_{t+1}$ }; + \end{scope} + + \end{scope} + + + + +\end{tikzpicture} diff --git a/022_ErrorResilience/Figures/BiDirMotionCopy2011.pdf b/022_ErrorResilience/Figures/BiDirMotionCopy2011.pdf new file mode 100755 index 0000000..87909a4 Binary files /dev/null and b/022_ErrorResilience/Figures/BiDirMotionCopy2011.pdf differ diff --git a/022_ErrorResilience/Figures/DBFDPCon2009.pdf b/022_ErrorResilience/Figures/DBFDPCon2009.pdf new file mode 100755 index 0000000..c223fda Binary files /dev/null and b/022_ErrorResilience/Figures/DBFDPCon2009.pdf differ diff --git a/022_ErrorResilience/Figures/DFDPMethod.tex b/022_ErrorResilience/Figures/DFDPMethod.tex new file mode 100755 index 0000000..716033b --- /dev/null +++ b/022_ErrorResilience/Figures/DFDPMethod.tex @@ -0,0 +1,97 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{0.29} + +\begin{tikzpicture}[scale=\scale,every node/.style={scale=\scale},on grid ] + + \tikzstyle{Frame}=[black] + + \tikzstyle{TextNode}=[scale=2, text centered, text width=3.3cm] + \tikzstyle{SymbolNode}=[scale=5.2, text centered] + \tikzstyle{line} = [draw, -latex'] + + \newcommand{\gridsize}{30} + %\draw[help lines] (-\gridsize,-\gridsize) grid ++(\gridsize*2,\gridsize*2); + + % Frames Left and names + \draw[Frame] (0,0) rectangle ++(8,6); + \draw[Frame] (14,0) rectangle ++(8,6); + \node[TextNode,scale=1.3] at (6.5,1) {$f^{l}_{t-1}$}; + \node[TextNode,scale=1.3] at (6.5+14,1) {$f^{l}_{t}$}; + + % Objects + \draw[black] (2.5,4) circle (1.4 cm and 1 cm); + \draw[black] (2.5+14+1.4,4-0.7) circle (1.4 cm and 1 cm); + + % Temporal change detection + \node[TextNode,rectangle,draw,very thin] (TCD) at (14/2+8/2,-1.5) {Temporal change detection}; + \path [line] (8/2,0) |- (TCD.west); + \path [line] (8/2+14,0) |- (TCD.east); + + % Temporal changes and objects + \draw[Frame] (14/2,-9.5) rectangle ++(8,6); + \draw[fill=gray,draw=gray] (14/2+2.5,-9.5+4) circle (1.4 cm and 1 cm); + \draw[fill=gray,draw=gray] (14/2+2.5+1.4,-9.5+4-0.7) circle (1.4 cm and 1 cm); + + \path[line] (TCD) -- ++(0,-2); + + + \node[TextNode,text width=5cm] (TCDName) at (22,-5) {Change dectection map from the left view \\ $M^{l}_{t-1 \rightarrow t}$}; + \path [line,->,shorten >= 2pt] (17.5,-5.2) -- (14/2+8,-9.5+6/2); + + \node[circle,draw,very thin,scale=1.5] (Sum) at (14/2+8/2,-9.5-3.5/2) {DC}; + + % Right temporal change + \draw[Frame,dashed] (14/2,-19) rectangle ++(8,6); + \draw[draw, densely dashed] (14/2+2.5+2,-19+4) circle (1.4 cm and 1 cm); + \draw[draw, densely dashed,fill=white] (14/2+2.5+1.4+2,-19+4-0.7) circle (1.4 cm and 1 cm); + + \path [line] (14/2+8/2,-9.5) -- (Sum); + \path[line] (Sum) -- (14/2+8/2,-10-9+6); + + \node[TextNode,text width=2.6cm] (DCLabel) at (19,-10) {Disparity Compensation}; + \path [line,->, shorten >=2pt] (DCLabel) -- (Sum); + + + % Disparity Map estimation and temporal change compensation + \node[TextNode,rectangle,draw,very thin, text width=2.2cm] (Disp) at (3,-9.5-3.5/2) {Disparity map}; + \path [line] (Disp) -- (Sum); + \path [line] (3,0) -- (Disp); + \path [line] (3,-26+6) -- (Disp); + + \node[TextNode,text width=5cm] (RTCDName) at (22,-15) {Predicted frame difference for thr right view \\ $\Delta f^{r}_{t-1 \rightarrow t}$}; + \path [line,->,shorten >= 2pt ] (17.5,-15) -- (14/2+8,-19+6/2); + + + % Frames Right and names + \draw[Frame] (0,-26) rectangle ++(8,6); + \draw[Frame] (14,-26) rectangle ++(8,6); + \node[TextNode,scale=1.3] at (6.5,1-26) {$f^{r}_{t-1}$}; + \node[TextNode,scale=1.3] at (6.5+14,1-26) {$f^{r}_{t}$}; + + % Right Sum + \node[circle,draw,very thin,scale=2] (Sum) at (14/2+8/2,-26+6/2) {$+$}; + \path [line] (8,-26+6/2) -- (Sum); + \path [line] (8/2+14/2,-10-9) -- (Sum); + \path [line] (Sum) -- (14,-26+6/2); + + + \draw[black] (2.5+2,-26+4) circle (1.4 cm and 1 cm); + \draw[black, densely dashed] (2.5+14+1.4+2,4-0.7-26) circle (1.4 cm and 1 cm); + + + \draw[draw=white] (-5,-26.5) rectangle ++(14+8+5+5,26.5+6+0.5); + + +\end{tikzpicture} diff --git a/022_ErrorResilience/Figures/EdgesDirectionSDV.pdf b/022_ErrorResilience/Figures/EdgesDirectionSDV.pdf new file mode 100755 index 0000000..ce1b460 Binary files /dev/null and b/022_ErrorResilience/Figures/EdgesDirectionSDV.pdf differ diff --git a/022_ErrorResilience/Figures/EdgesInterception.pdf b/022_ErrorResilience/Figures/EdgesInterception.pdf new file mode 100755 index 0000000..48c15bf Binary files /dev/null and b/022_ErrorResilience/Figures/EdgesInterception.pdf differ diff --git a/022_ErrorResilience/Figures/EdgesInterception_a.pdf b/022_ErrorResilience/Figures/EdgesInterception_a.pdf new file mode 100755 index 0000000..0a3b5f9 Binary files /dev/null and b/022_ErrorResilience/Figures/EdgesInterception_a.pdf differ diff --git a/022_ErrorResilience/Figures/EdgesInterception_b.pdf b/022_ErrorResilience/Figures/EdgesInterception_b.pdf new file mode 100755 index 0000000..db8d337 Binary files /dev/null and b/022_ErrorResilience/Figures/EdgesInterception_b.pdf differ diff --git a/022_ErrorResilience/Figures/EdgesInterception_c.pdf b/022_ErrorResilience/Figures/EdgesInterception_c.pdf new file mode 100755 index 0000000..1e5d80d Binary files /dev/null and b/022_ErrorResilience/Figures/EdgesInterception_c.pdf differ diff --git a/022_ErrorResilience/Figures/ErrorPropagation.png b/022_ErrorResilience/Figures/ErrorPropagation.png new file mode 100644 index 0000000..66be15e Binary files /dev/null and b/022_ErrorResilience/Figures/ErrorPropagation.png differ diff --git a/022_ErrorResilience/Figures/ErrorPropagation_1.png b/022_ErrorResilience/Figures/ErrorPropagation_1.png new file mode 100644 index 0000000..5a32dec Binary files /dev/null and b/022_ErrorResilience/Figures/ErrorPropagation_1.png differ diff --git a/022_ErrorResilience/Figures/FMO_Slices_Type.png b/022_ErrorResilience/Figures/FMO_Slices_Type.png new file mode 100644 index 0000000..d4a3211 Binary files /dev/null and b/022_ErrorResilience/Figures/FMO_Slices_Type.png differ diff --git a/022_ErrorResilience/Figures/FlowchartIlluminationStereo.pdf b/022_ErrorResilience/Figures/FlowchartIlluminationStereo.pdf new file mode 100755 index 0000000..a0b5ec7 Binary files /dev/null and b/022_ErrorResilience/Figures/FlowchartIlluminationStereo.pdf differ diff --git a/022_ErrorResilience/Figures/HEVCTransmissionStack.pdf b/022_ErrorResilience/Figures/HEVCTransmissionStack.pdf new file mode 100644 index 0000000..d26c152 Binary files /dev/null and b/022_ErrorResilience/Figures/HEVCTransmissionStack.pdf differ diff --git a/022_ErrorResilience/Figures/HEVCTransmissionStackOverview.pdf b/022_ErrorResilience/Figures/HEVCTransmissionStackOverview.pdf new file mode 100644 index 0000000..6d53bc4 Binary files /dev/null and b/022_ErrorResilience/Figures/HEVCTransmissionStackOverview.pdf differ diff --git a/022_ErrorResilience/Figures/IVSMID2010.pdf b/022_ErrorResilience/Figures/IVSMID2010.pdf new file mode 100755 index 0000000..ef2e6c4 Binary files /dev/null and b/022_ErrorResilience/Figures/IVSMID2010.pdf differ diff --git a/022_ErrorResilience/Figures/LinearInterpolation.tex b/022_ErrorResilience/Figures/LinearInterpolation.tex new file mode 100755 index 0000000..aa529d9 --- /dev/null +++ b/022_ErrorResilience/Figures/LinearInterpolation.tex @@ -0,0 +1,68 @@ +%User Variables + +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric} +\usetikzlibrary{decorations.pathreplacing,decorations.pathmorphing,shapes} +\usetikzlibrary{matrix,shapes.symbols,plotmarks,decorations.markings,shadows} + +\newcommand{\block}{1cm} +\newcommand{\scale}{0.8} + +\begin{tikzpicture}[scale=\scale,every node/.style={minimum size=0.6*\scale*\block}, inner sep=0pt,on grid] + + \definecolor{lightgray}{rgb}{0.80,0.80,0.80}; + \definecolor{darkgray}{rgb}{0.40,0.40,0.40}; + + \tikzstyle{arrow}=[draw, <->,shorten <=1pt,shorten >=1pt,>=stealth',thick] + \tikzstyle{pel}=[circle, rounded corners ,draw=lightgray, fill=lightgray, thick, scale=0.5] + + + % Neighbor pixels + \foreach \x in {-2,...,10} + \foreach \y in {-1,...,9} + { + \node at (\x+0.5,\y+0.5) [pel]() {}; + } + + % \foreach \x in {0,8} + % \foreach \y in {1,...,2,4,5,...,7} + % { + % \node at (\x+0.5,\y+0.5) [pel](ref_lft) {}; + % } + + + % Center + \draw[draw=black, dashed,thick, opacity=1] (\block+4,\block+4) rectangle ++(7*\block-8,7*\block-8); + \node[anchor=west,scale=\scale] at (1*\block+0.1*\block,8*\block+3) {Lost region ($R_L$)}; + + + % Lost pixel + \node at (4*\block+\block/2, 3*\block+\block/2) + [pel, fill=black, draw=black, thick](lostpel) [label=north east: $lp$] {}; + + + % Reference pixels + \node at (4*\block+\block/2, 8*\block+\block/2) [pel, fill=black, draw=black, thick](ref_tp) [label=north west: $p_1$] {}; + \node at (0*\block+\block/2, 3*\block+\block/2) [pel, fill=black, draw=black, thick](ref_lft) [label=north west: $p_2$] {}; + \node at (4*\block+\block/2, 0*\block+\block/2) [pel, fill=black, draw=black, thick](ref_bottm) [label=south east: $p_3$] {}; + \node at (8*\block+\block/2, 3*\block+\block/2) [pel, fill=black, draw=black, thick](ref_rght) [label=north east: $p_4$] {} ; + + \path [arrow] (lostpel) -- node[left] { $d_1$} (ref_tp); + \path [arrow] (lostpel) -- node[above] { $d_2$} (ref_lft); + \path [arrow] (lostpel) -- node[left] { $d_3$} (ref_bottm); + \path [arrow] (lostpel) -- node[above] { $d_4$} (ref_rght); + + + + \node[anchor=west,scale=\scale] at (-1.4,10) (pixel_label) {Pixel}; + \node at (-1.5,8.4) (pel) {}; + \path [-latex,anchor=center] (pixel_label) edge (pel); + + \node[anchor=west,scale=\scale] at (5.8,6.0) (edge_label) {Contour}; + \path [-latex] (edge_label.south) edge (7.15,5.2); + + \node at (6.5,-1.3) (edge_start) {}; + \node at (10.3,8.1) (edge_end) {}; + \path [-,thick] (edge_start.north) edge[out=80, in=190] (edge_end.west); + + +\end{tikzpicture} diff --git a/022_ErrorResilience/Figures/MDC.png b/022_ErrorResilience/Figures/MDC.png new file mode 100644 index 0000000..ba055d2 Binary files /dev/null and b/022_ErrorResilience/Figures/MDC.png differ diff --git a/022_ErrorResilience/Figures/MVE.pdf b/022_ErrorResilience/Figures/MVE.pdf new file mode 100755 index 0000000..91ea915 Binary files /dev/null and b/022_ErrorResilience/Figures/MVE.pdf differ diff --git a/022_ErrorResilience/Figures/MVE.vsd b/022_ErrorResilience/Figures/MVE.vsd new file mode 100755 index 0000000..4664e07 Binary files /dev/null and b/022_ErrorResilience/Figures/MVE.vsd differ diff --git a/022_ErrorResilience/Figures/MaximallySmoothimageRecovery.pdf b/022_ErrorResilience/Figures/MaximallySmoothimageRecovery.pdf new file mode 100755 index 0000000..45bc7c3 Binary files /dev/null and b/022_ErrorResilience/Figures/MaximallySmoothimageRecovery.pdf differ diff --git a/022_ErrorResilience/Figures/ModAverageAlgorithm.pdf b/022_ErrorResilience/Figures/ModAverageAlgorithm.pdf new file mode 100755 index 0000000..9aa6177 Binary files /dev/null and b/022_ErrorResilience/Figures/ModAverageAlgorithm.pdf differ diff --git a/022_ErrorResilience/Figures/ModAverageAlgorithm/ModAverageAlgorithm.pdf b/022_ErrorResilience/Figures/ModAverageAlgorithm/ModAverageAlgorithm.pdf new file mode 100755 index 0000000..9aa6177 Binary files /dev/null and b/022_ErrorResilience/Figures/ModAverageAlgorithm/ModAverageAlgorithm.pdf differ diff --git a/022_ErrorResilience/Figures/ModAverageAlgorithm/ModAverageAlgorithm_A.tex b/022_ErrorResilience/Figures/ModAverageAlgorithm/ModAverageAlgorithm_A.tex new file mode 100755 index 0000000..2a33402 --- /dev/null +++ b/022_ErrorResilience/Figures/ModAverageAlgorithm/ModAverageAlgorithm_A.tex @@ -0,0 +1,47 @@ +% \usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +% decorations.pathreplacing,decorations.pathmorphing,shapes,% +% matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + +\newcommand{\scale}{1.2} + +\begin{tikzpicture}[node distance = 2.5cm, auto,every node/.style={minimum size=\scale,scale=\scale},scale=\scale] + % Place nodes + +%\draw[help lines] (-5,-5) grid ++(10,10); + + \draw[draw=black,thick] (0,0) rectangle ++(1,1); + \node[scale=0.6] at (0.5,0.5) {$LB_i$}; + + \draw[draw=black,thick] (1,0) rectangle ++(1,1); + \node[scale=0.6] at (1.5,0.5) {$LB_{i+1}$}; + + \draw[draw=black,thick] (-1,0) rectangle ++(1,1); + \node[scale=0.6] at (-0.5,0.5) {$LB_{i-1}$}; + + + \draw[draw=black] (0,0) -- ++(-1.2,0); + \draw[draw=black] (0,0) -- ++(0,-1); + \draw[draw=black] (0,1) -- ++(-1.2,0); + \draw[draw=black] (0,1) -- ++(0,1); + \draw[draw=black] (1,0) -- ++(0,-1.0); + + \draw[draw=black] (1,0) -- ++(1.2,0); + \draw[draw=black] (1,1) -- ++(1.2,0); + \draw[draw=black] (1,1) -- ++(0,1.0); + + \draw[draw=black] (0,1) rectangle ++(1,0.5); + \node[scale=0.6] at (0.5,1.25) {\tiny $MV_1$}; + + \draw[draw=black] (0,0) rectangle ++(1,-0.5); + \node[scale=0.6] at (0.5,-0.25) {\tiny $MV_2$}; + + + + + \node[text centered, inner sep=1.5pt, scale=0.8] (TB1) at (1.5,1.7) {\tiny $TB_1$}; + \draw[draw=black,-stealth'] (TB1) -- (1,1.25); + + \node[text centered, inner sep=1.5pt, scale=0.8] (TB2) at (1.5,-0.7) {\tiny $TB_2$}; + \draw[draw=black,-stealth'] (TB2) -- (1,-0.25); + +\end{tikzpicture} diff --git a/022_ErrorResilience/Figures/ModAverageAlgorithm/ModAverageAlgorithm_B.tex b/022_ErrorResilience/Figures/ModAverageAlgorithm/ModAverageAlgorithm_B.tex new file mode 100755 index 0000000..c337333 --- /dev/null +++ b/022_ErrorResilience/Figures/ModAverageAlgorithm/ModAverageAlgorithm_B.tex @@ -0,0 +1,55 @@ +% \usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +% decorations.pathreplacing,decorations.pathmorphing,shapes,% +% matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + +\newcommand{\scale}{1.2} + +\begin{tikzpicture}[node distance = 2.5cm, auto,every node/.style={minimum size=\scale,scale=\scale},scale=\scale] + % Place nodes + + \draw[draw=black,thick] (0,0) rectangle ++(1,1); + \node[scale=0.6] at (0.5,0.5) {$LB_i$}; + + \draw[draw=black,thick] (1,0) rectangle ++(1,1); + \node[scale=0.6] at (1.5,0.5) {$LB_{i+1}$}; + + \draw[draw=black,thick] (-1,0) rectangle ++(1,1); + \node[scale=0.6] at (-0.5,0.5) {$LB_{i-1}$}; + + \draw[draw=black] (0,0) -- ++(-1.2,0); + \draw[draw=black] (0,0) -- ++(0,-1); + \draw[draw=black] (0,1) -- ++(-1.2,0); + \draw[draw=black] (0,1) -- ++(0,1); + \draw[draw=black] (1,0) -- ++(0,-1); + + \draw[draw=black] (1,0) -- ++(1.2,0); + \draw[draw=black] (1,1) -- ++(1.2,0); + \draw[draw=black] (1,1) -- ++(0,1); + + \draw[draw=black] (0,1) rectangle ++(0.5,0.5); + \node[scale=0.6] at (0.25,1.25) {\tiny $MV_1$}; + + \node[text centered, inner sep=1.5pt, scale=0.8] (TB2) at (-0.5,+1.7) {\tiny $STB_1$}; + \draw[draw=black,-stealth'] (TB2) -- (0,1.25); + + \draw[draw=black] (0.5,1) rectangle ++(0.5,0.5); + \node[scale=0.6] at (0.75,1.25) {\tiny $MV_2$}; + + \node[text centered, inner sep=1.5pt, scale=0.8] (TB1) at (1.5,1.7) {\tiny $STB_2$}; + \draw[draw=black,-stealth'] (TB1) -- (1,1.25); + + \draw[draw=black] (0,0) rectangle ++(0.5,-0.5); + \node[scale=0.6] at (0.25,-0.25) {\tiny $MV_3$}; + + \node[text centered, inner sep=1.5pt, scale=0.8] (TB2) at (-0.5,-0.7) {\tiny $STB_3$}; + \draw[draw=black,-stealth'] (TB2) -- (0,-.25); + + \draw[draw=black] (0.5,0) rectangle ++(0.5,-0.5); + \node[scale=0.6] at (0.75,-0.25) {\tiny $MV_4$}; + + \node[text centered, inner sep=1.5pt, scale=0.8] (TB1) at (1.5,-.7) {\tiny $STB_4$}; + \draw[draw=black,-stealth'] (TB1) -- (1,-.25); + + + +\end{tikzpicture} diff --git a/022_ErrorResilience/Figures/ModAverageAlgorithm/ModAverageAlgorithm_C.tex b/022_ErrorResilience/Figures/ModAverageAlgorithm/ModAverageAlgorithm_C.tex new file mode 100755 index 0000000..5438640 --- /dev/null +++ b/022_ErrorResilience/Figures/ModAverageAlgorithm/ModAverageAlgorithm_C.tex @@ -0,0 +1,58 @@ +% \usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +% decorations.pathreplacing,decorations.pathmorphing,shapes,% +% matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + +\newcommand{\scale}{1.2} + +\begin{tikzpicture}[node distance = 2.5cm, auto,every node/.style={minimum size=\scale,scale=\scale},scale=\scale] + % Place nodes + + + + \draw[draw=black] (0,0) -- ++(-1.2,0); + \draw[draw=black] (0,0) -- ++(0,-1); + \draw[draw=black] (0,1) -- ++(-1.2,0); + \draw[draw=black] (0,1) -- ++(0,1); + \draw[draw=black] (1,0) -- ++(0,-1); + + \draw[draw=black] (1,0) -- ++(1.2,0); + \draw[draw=black] (1,1) -- ++(1.2,0); + \draw[draw=black] (1,1) -- ++(0,1); + + \draw[draw=black,fill=white] (-0.25,1) rectangle ++(0.5,0.5); + \node[scale=0.6] at (0,1.25) {\tiny $MV_1$}; + + \node[text centered, inner sep=1.5pt, scale=0.8] (TB2) at (-0.5,+1.7) {\tiny $ASTB_1$}; + \draw[draw=black,-stealth'] (TB2) -- (-0.25,1.25); + + \draw[draw=black,fill=white] (0.75,1) rectangle ++(0.5,0.5); + \node[scale=0.6] at (1,1.25) {\tiny $MV_2$}; + + \node[text centered, inner sep=1.5pt, scale=0.8] (TB1) at (1.5,1.7) {\tiny $ASTB_2$}; + \draw[draw=black,-stealth'] (TB1) -- (1.25,1.25); + + \draw[draw=black,fill=white] (-0.25,0) rectangle ++(0.5,-0.5); + \node[scale=0.6] at (0,-0.25) {\tiny $MV_3$}; + + \node[text centered, inner sep=1.5pt, scale=0.8] (TB2) at (-0.5,-0.7) {\tiny $ASTB_3$}; + \draw[draw=black,-stealth'] (TB2) -- (-0.25,-.25); + + \draw[draw=black,fill=white] (0.75,0) rectangle ++(0.5,-0.5); + \node[scale=0.6] at (1,-0.25) {\tiny $MV_4$}; + + \node[text centered, inner sep=1.5pt, scale=0.8] (TB1) at (1.5,-.7) {\tiny $ASTB_4$}; + \draw[draw=black,-stealth'] (TB1) -- (1.25,-.25); + + \draw[draw=black,thick] (0,0) rectangle ++(1,1); + \node[scale=0.6] at (0.5,0.5) {$LB_i$}; + + \draw[draw=black,thick] (1,0) rectangle ++(1,1); + \node[scale=0.6] at (1.5,0.5) {$LB_{i+1}$}; + + \draw[draw=black,thick] (-1,0) rectangle ++(1,1); + \node[scale=0.6] at (-0.5,0.5) {$LB_{i-1}$}; + + + + +\end{tikzpicture} diff --git a/022_ErrorResilience/Figures/ModEdgeOriented.pdf b/022_ErrorResilience/Figures/ModEdgeOriented.pdf new file mode 100755 index 0000000..eeba4d8 Binary files /dev/null and b/022_ErrorResilience/Figures/ModEdgeOriented.pdf differ diff --git a/022_ErrorResilience/Figures/MotionDisparityProjection.pdf b/022_ErrorResilience/Figures/MotionDisparityProjection.pdf new file mode 100755 index 0000000..611a002 Binary files /dev/null and b/022_ErrorResilience/Figures/MotionDisparityProjection.pdf differ diff --git a/022_ErrorResilience/Figures/OcclusionsDetection.pdf b/022_ErrorResilience/Figures/OcclusionsDetection.pdf new file mode 100755 index 0000000..6c89769 Binary files /dev/null and b/022_ErrorResilience/Figures/OcclusionsDetection.pdf differ diff --git a/022_ErrorResilience/Figures/POCSBlockDiagram.pdf b/022_ErrorResilience/Figures/POCSBlockDiagram.pdf new file mode 100755 index 0000000..f2b1d92 Binary files /dev/null and b/022_ErrorResilience/Figures/POCSBlockDiagram.pdf differ diff --git a/022_ErrorResilience/Figures/PixelBasedMVE.pdf b/022_ErrorResilience/Figures/PixelBasedMVE.pdf new file mode 100755 index 0000000..b49786b Binary files /dev/null and b/022_ErrorResilience/Figures/PixelBasedMVE.pdf differ diff --git a/022_ErrorResilience/Figures/PixelBasedMVE.tex b/022_ErrorResilience/Figures/PixelBasedMVE.tex new file mode 100755 index 0000000..5ecd948 --- /dev/null +++ b/022_ErrorResilience/Figures/PixelBasedMVE.tex @@ -0,0 +1,45 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + +\newcommand{\scale}{0.25} + +\begin{tikzpicture}[scale=\scale,every node/.style={scale=\scale},on grid ] + + \tikzstyle{LostMB}=[black, thick] + \tikzstyle{ExtrapolatedMB}=[black, dashed, thick] + \tikzstyle{TextNode}=[scale=4.2, text centered, text width=3.3cm] + \tikzstyle{SymbolNode}=[scale=5.2, text centered] + + \newcommand{\gridsize}{30} + %\draw[help lines] (-\gridsize,-\gridsize) grid ++(\gridsize*2,\gridsize*2); + + \draw[LostMB] (0,0) rectangle ++(8,8); + + \draw[ExtrapolatedMB] (6-16,6) rectangle ++(16cm,16cm); + \draw[ExtrapolatedMB] (4,4) rectangle ++(16cm,16cm); + \draw[ExtrapolatedMB] (6,-14) rectangle ++(16cm,16cm); + \draw[ExtrapolatedMB] (3-16,-14) rectangle ++(16cm,16cm); + + \node[TextNode](EMB1) at (5-16+16/2,7+16/2) {Extrapolated PU$_1$}; + \node[TextNode](EMB1) at (5+16/2,4+16/2) {Extrapolated PU$_2$}; + \node[TextNode](EMB1) at (6+16/2,-14+16/2) {Extrapolated PU$_4$}; + \node[TextNode](EMB1) at (3-16+16/2,-14+16/2) {Extrapolated PU$_3$}; + + \node[SymbolNode] at (2,1) {$\circ$}; + \node[SymbolNode] at (7,1) {$\circ$}; + \node[SymbolNode] at (7,7) {$\circ$}; + \node[SymbolNode] at (5,7) {$\diamond$}; + \node[SymbolNode,scale=0.7] at (2,4) {$\bigtriangleup$}; + \node[SymbolNode] at (-2,4) {$\star$}; + +\end{tikzpicture} diff --git a/022_ErrorResilience/Figures/ROILeakyPrediction.png b/022_ErrorResilience/Figures/ROILeakyPrediction.png new file mode 100644 index 0000000..882cef1 Binary files /dev/null and b/022_ErrorResilience/Figures/ROILeakyPrediction.png differ diff --git a/022_ErrorResilience/Figures/ROILeakyPrediction_1.png b/022_ErrorResilience/Figures/ROILeakyPrediction_1.png new file mode 100644 index 0000000..4d007e9 Binary files /dev/null and b/022_ErrorResilience/Figures/ROILeakyPrediction_1.png differ diff --git a/022_ErrorResilience/Figures/ROI_IntraRefresh.png b/022_ErrorResilience/Figures/ROI_IntraRefresh.png new file mode 100644 index 0000000..1dc5fa9 Binary files /dev/null and b/022_ErrorResilience/Figures/ROI_IntraRefresh.png differ diff --git a/022_ErrorResilience/Figures/ROI_IntraRefresh_1.png b/022_ErrorResilience/Figures/ROI_IntraRefresh_1.png new file mode 100644 index 0000000..fca2cf2 Binary files /dev/null and b/022_ErrorResilience/Figures/ROI_IntraRefresh_1.png differ diff --git a/022_ErrorResilience/Figures/RefPicSelection.png b/022_ErrorResilience/Figures/RefPicSelection.png new file mode 100644 index 0000000..4abfaef Binary files /dev/null and b/022_ErrorResilience/Figures/RefPicSelection.png differ diff --git a/022_ErrorResilience/Figures/RefinedMotionCopy.tex b/022_ErrorResilience/Figures/RefinedMotionCopy.tex new file mode 100755 index 0000000..f0f7e00 --- /dev/null +++ b/022_ErrorResilience/Figures/RefinedMotionCopy.tex @@ -0,0 +1,60 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + +\def\MVE{\mbox{MVE}} +\def\MVBASE{\mbox{BASE\_MC}} +\def\INTERVIEW{\mbox{INTERVIEW\_COPY}} +\def\COMBMC{\mbox{COMB\_MC}} +\def\COMBMCDISP{\mbox{COMB\_MC\_Disp}} + +% Define block styles +\tikzstyle{block} = [rectangle, draw, text width=6cm, text centered, minimum height=1.5cm, node distance=2.3cm] +\tikzstyle{node_circle} = [circle, draw, text centered] +\tikzstyle{decision} = [diamond, draw, text badly centered, inner sep=1pt, text width=2cm, node distance=2.8cm] +\tikzstyle{line} = [draw, -latex'] + +\newcommand{\scale}{0.7} + +\begin{tikzpicture}[node distance = 2.5cm, auto,every node/.style={minimum size=\scale,scale=\scale},scale=\scale] + % Place nodes + + \node [decision] at (0,0) (Indentify) {\small Identifying corrupted frame}; + + \node [block,ellipse, text width=2.5cm,above of=Indentify,yshift=0.7cm] (Start) {Frame Start}; + + \path [line] (Start) -- (Indentify); + + \node [block,below of=Indentify,yshift=-0.7cm] (Divide) {The frame is divided into $4\times4$ blocks}; + + \path [line] (Indentify) -- (Divide); + + \node [block,below of=Divide] (Normalization) {Motion vector normalization}; + + \path [line] (Divide) -- (Normalization); + + \node [block, below of=Normalization] (Differences) {Calculating motion vector differences (MVD)}; + + \path [line] (Normalization) -- (Differences); + + \node [block, right of=Indentify, xshift=5cm,yshift=1.4cm] (Recursive) {Recursive motion vector difference and refinement area selection}; + + \path [line] (Differences.south) -- +(0,-1) -- +(7.8/2,-1) -- +(7.8/2,11+0.8) -| (Recursive.north); + + \node [block, below of=Recursive] (Iterative) {Iterative counting the occurrences of MVD}; + + \path [line] (Recursive) -- (Iterative); + + \node [block,below of=Iterative] (Selection) {The MVD with maximum occurrence is individually selected in x and y directions}; + + \path [line] (Iterative) -- (Selection); + + \node [block,below of=Selection] (FinalMV) {Final motion vector is derived by adding the refined MVD}; + + \path [line] (Selection) -- (FinalMV); + + \node [block,ellipse, text width=2.5cm,below of=FinalMV, yshift=0cm] (End) {End of frame}; + + \path [line] (FinalMV) -- (End); + +\end{tikzpicture} diff --git a/022_ErrorResilience/Figures/SpatialTemporalErrorPropagation.tikz b/022_ErrorResilience/Figures/SpatialTemporalErrorPropagation.tikz new file mode 100644 index 0000000..28ca1d1 --- /dev/null +++ b/022_ErrorResilience/Figures/SpatialTemporalErrorPropagation.tikz @@ -0,0 +1,177 @@ + + +\newcommand{\scale}{1} + +\newcommand{\frameScale}{.3} +% \newcommand{\TikZxscale}{0.55} +\newcommand{\xfshift}{8cm} +% \newcommand{\TikZyslant}{0.4} +% \newcommand{\TikZyfshift}{0.05cm} + +\definecolor{colorFrameBackground}{rgb}{.82,.89,.95} +% \definecolor{colorErrorsRegion}{rgb}{.163,.547,.757} +\definecolor{colorErrors}{rgb}{.16,.40,.40} +\definecolor{colorErrorsRegion}{rgb}{.209,.705,.740} + + +\usetikzlibrary{patterns} + +\makeatletter +\pgfdeclareshape{test shape} +{ + \inheritsavedanchors[from=rectangle] + \inheritanchor[from=rectangle]{center} + + \backgroundpath + { + \def\my@color{.}% + \ifx\tikz@strokecolor\pgfutil@empty + \else + \let\my@color\tikz@strokecolor + \fi + \pgfsetfillpattern{north east lines}{\my@color} + \pgfpathrectanglecorners{\pgfpoint{0}{0}}{\pgfpoint{1cm}{1cm}} + \pgfusepath{fill} + } +} +\makeatother + +\newcommand{\FillFrame}[1] +{ + \draw[fill=#1,fill opacity=1, very thick] (0,0) rectangle (8+1,8+1); % Opacity +% \draw[thick] (0,0) rectangle (#2+1,#2+1); +% \foreach \x in {1,...,#2} +% \draw (\x,0) -- ++(0,#2+1); +% \foreach \y in {1,...,#2} +% \draw (0,\y) -- ++(#2+1,0); +} + +\newcommand{\DrawFrame}[1] +{ + \draw[fill=#1,fill opacity=1, very thick] (0,0) rectangle (8+1,8+1); % Opacity +% \draw[thick] (0,0) rectangle (#2+1,#2+1); +% \foreach \x in {1,...,#2} +% \draw (\x,0) -- ++(0,#2+1); +% \foreach \y in {1,...,#2} +% \draw (0,\y) -- ++(#2+1,0); +} + +\newcommand{\DrawBlock}[3] +{ + \DrawBlockWithSize{#1}{#2}{#3}{1}{1} +} +\newcommand{\DrawBlockWithSize}[5] +{ + \draw[draw=#1, fill=#1] (#2,#3) rectangle ++(#4 cm,#5 cm); +} + + +\begin{tikzpicture}[use Hobby shortcut,closed=true] + +\tikzstyle{refmv_line}=[thick,draw=black] +\tikzstyle{refmv}=[refmv_line,-latex,densely dashed] +\tikzstyle{refmv_0}=[refmv,] +\tikzstyle{refmv_1}=[refmv_0] + +\tikzstyle{errorBlock}=[draw=colorErrors, fill=colorErrors] +\tikzstyle{errorPropagation}=[draw=colorErrorsRegion, fill=colorErrorsRegion] + +\coordinate (ErrorPosition) at (4,4); + + +\def\errorPosX{3} +\def\errorPosY{6} + +\begin{scope}[scale=\frameScale*.95, yshift=2*\TikZyfshift, xshift=-2*\xfshift] + \DrawFrame{colorFrameBackground} +% \node at (12cm/2,-1.5cm) { $f_{t}$ }; + + \draw[errorBlock] (\errorPosX,\errorPosY) rectangle ++(1cm,1cm); + + \foreach \x in {\errorPosX,...,7} + \draw[errorPropagation] (\x+1,\errorPosY) rectangle ++(1 cm,1 cm); + \foreach \y in {0.1,1,2,3,4,5} + \foreach \x in {0.1,1,2,3,4,5,6,7,8} + \draw[errorPropagation] (\x,\y) rectangle ++(1 cm,1 cm); + + \draw (0,\errorPosY) -- ++(\errorPosX + 1,0) -- ++(0,1) -- ++(8-\errorPosX,0); + +\end{scope} + +\begin{scope}[scale=\frameScale, yshift=2*\TikZyfshift, xshift=0*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] + \DrawFrame{colorFrameBackground} + \node at (12cm/2,-1.5cm) { $f_{t}$ }; + + \draw[errorBlock] (\errorPosX,\errorPosY) rectangle ++(1cm,1cm); +\end{scope} + +\begin{scope}[scale=\frameScale, yshift=2*\TikZyfshift, xshift=1*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] + \DrawFrame{colorFrameBackground} + \node at (12cm/2,-1.5cm) { $f_{t+1}$ }; +% \begin{scope}[yshift=3.8cm, xshift=2.4cm] +% \draw[erros] plot[scale=.3,smooth, tension=.7] coordinates {(0,0) (.5,3) (2.5,4) (5,3.5) (7.5,4) (8.5,3) (8.5,1) (10,-1.5) (3.5,0) (.5,-1.5) (0,0)}; +% \end{scope} + + \draw[errorPropagation] (\errorPosX,\errorPosY) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX,\errorPosY -1) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +1,\errorPosY) rectangle ++(1cm,1cm); + +\end{scope} + +\begin{scope}[scale=\frameScale, yshift=2*\TikZyfshift, xshift=2*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] + \DrawFrame{colorFrameBackground} + \node at (12cm/2,-1.5cm) { $f_{t+2}$ }; + + \draw[errorPropagation] (\errorPosX,\errorPosY) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +1,\errorPosY) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +2,\errorPosY) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX,\errorPosY -1) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX+1,\errorPosY -1) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX+1,\errorPosY -2) rectangle ++(1cm,1cm); +\end{scope} + +\begin{scope}[scale=\frameScale, yshift=3*\TikZyfshift, xshift=3*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] + \DrawFrame{colorFrameBackground} + \node at (12cm/2,-1.5cm) { $f_{t+3}$ }; +% \begin{scope}[yshift=3.1cm, xshift=1.9cm] +% \draw[erros] plot[scale=.5,smooth, tension=.7] coordinates {(0,0) (.5,3) (2.5,4) (5,3.5) (7.5,4) (8.5,3) (8.5,1) (10,-1.5) (3.5,0) (.5,-1.5) (0,0)}; +% \end{scope} + + \draw[errorPropagation] (\errorPosX ,\errorPosY) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +1,\errorPosY) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +2,\errorPosY) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX ,\errorPosY -1) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +1,\errorPosY -1) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +2,\errorPosY -1) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +0,\errorPosY -2) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +1,\errorPosY -2) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +0,\errorPosY -3) rectangle ++(1cm,1cm); + +\end{scope} + +\begin{scope}[scale=\frameScale, yshift=4*\TikZyfshift, xshift=4*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] + \DrawFrame{colorFrameBackground} + \node at (12cm/2,-1.5cm) { $f_{t+4}$ }; +% \begin{scope}[yshift=3cm, xshift=1.8cm] +% \draw[erros] plot[scale=.6,smooth, tension=.7] coordinates {(0,0) (.5,3) (2.5,4) (5,3.5) (7.5,4) (8.5,3) (8.5,1) (10,-1.5) (3.5,0) (.5,-1.5) (0,0)}; +% \end{scope} + \draw[errorPropagation] (\errorPosX ,\errorPosY) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +1,\errorPosY) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +2,\errorPosY) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +3,\errorPosY) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX ,\errorPosY -1) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +1,\errorPosY -1) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +2,\errorPosY -1) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +3,\errorPosY -1) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +4,\errorPosY -1) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +0,\errorPosY -2) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +1,\errorPosY -2) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +2,\errorPosY -2) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +0,\errorPosY -3) rectangle ++(1cm,1cm); + \draw[errorPropagation] (\errorPosX +2,\errorPosY -3) rectangle ++(1cm,1cm); + +\end{scope} + +\path[draw, -latex, thick] (-.5cm,-.5cm) -- ++(12.5cm,0) node[pos=1, yshift=-.2cm] {$t$}; + +\end{tikzpicture} diff --git a/022_ErrorResilience/Figures/TemporalErrorPropagation.tikz b/022_ErrorResilience/Figures/TemporalErrorPropagation.tikz new file mode 100644 index 0000000..4a9d0ce --- /dev/null +++ b/022_ErrorResilience/Figures/TemporalErrorPropagation.tikz @@ -0,0 +1,156 @@ + + +\newcommand{\scale}{1} + +\newcommand{\frameScale}{.3} +% \newcommand{\TikZxscale}{0.55} +\newcommand{\xfshift}{8cm} +% \newcommand{\TikZyslant}{0.4} +% \newcommand{\TikZyfshift}{0.05cm} + +\definecolor{colorFrameBackground}{rgb}{.82,.89,.95} +% \definecolor{colorErrorsRegion}{rgb}{.163,.547,.757} +\definecolor{colorErrorsRegion}{rgb}{.209,.705,.740} + + +\usetikzlibrary{patterns} + +\makeatletter +\pgfdeclareshape{test shape} +{ + \inheritsavedanchors[from=rectangle] + \inheritanchor[from=rectangle]{center} + + \backgroundpath + { + \def\my@color{.}% + \ifx\tikz@strokecolor\pgfutil@empty + \else + \let\my@color\tikz@strokecolor + \fi + \pgfsetfillpattern{north east lines}{\my@color} + \pgfpathrectanglecorners{\pgfpoint{0}{0}}{\pgfpoint{1cm}{1cm}} + \pgfusepath{fill} + } +} +\makeatother + +\newcommand{\DrawFrame}[1] +{ + \draw[fill=#1,fill opacity=1, thick] (0,0) rectangle (10+1,10+1); % Opacity +% \draw[thick] (0,0) rectangle (#2+1,#2+1); +% \foreach \x in {1,...,#2} +% \draw (\x,0) -- ++(0,#2+1); +% \foreach \y in {1,...,#2} +% \draw (0,\y) -- ++(#2+1,0); +} + +\newcommand{\DrawBlock}[3] +{ + \DrawBlockWithSize{#1}{#2}{#3}{1}{1} +} +\newcommand{\DrawBlockWithSize}[5] +{ + \draw[draw=#1, fill=#1] (#2,#3) rectangle ++(#4 cm,#5 cm); +} + + +\begin{tikzpicture}[use Hobby shortcut,closed=true] + +\tikzstyle{refmv_line}=[thick,draw=black] +\tikzstyle{refmv}=[refmv_line,-latex,densely dashed] +\tikzstyle{refmv_0}=[refmv,] +\tikzstyle{refmv_1}=[refmv_0] +\tikzstyle{erros}=[draw=colorErrorsRegion, fill=colorErrorsRegion] + +\begin{scope}[scale=\frameScale, yshift=2*\TikZyfshift, xshift=0*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] + \DrawFrame{colorFrameBackground} + \node at (12cm/2,-1.5cm) { $f_{t}$ }; + \DrawBlockWithSize{colorErrorsRegion}{4}{6}{1}{1} +\end{scope} + +\begin{scope}[scale=\frameScale, yshift=2*\TikZyfshift, xshift=1*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] + \DrawFrame{colorFrameBackground} + \node at (12cm/2,-1.5cm) { $f_{t+1}$ }; +% \begin{scope}[yshift=3.8cm, xshift=2.4cm] +% \draw[erros] plot[scale=.3,smooth, tension=.7] coordinates {(0,0) (.5,3) (2.5,4) (5,3.5) (7.5,4) (8.5,3) (8.5,1) (10,-1.5) (3.5,0) (.5,-1.5) (0,0)}; +% \end{scope} + + \DrawBlockWithSize{colorErrorsRegion}{4}{6}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{4}{5}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{5}{6}{1}{1} +\end{scope} + +\begin{scope}[scale=\frameScale, yshift=2*\TikZyfshift, xshift=2*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] + \DrawFrame{colorFrameBackground} + \node at (12cm/2,-1.5cm) { $f_{t+2}$ }; +% \begin{scope}[yshift=3.5cm, xshift=2.3cm] +% \draw[erros] plot[scale=.4,smooth, tension=.7] coordinates {(0,0) (.5,3) (2.5,4) (5,3.5) (7.5,4) (8.5,3) (8.5,1) (10,-1.5) (3.5,0) (.5,-1.5) (0,0)}; +% \end{scope} + \DrawBlockWithSize{colorErrorsRegion}{4}{6}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{4}{5}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{4}{4}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{5}{6}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{5}{5}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{5}{4}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{5}{3}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{6}{6}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{6}{5}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{6}{4}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{7}{6}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{7}{5}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{7}{4}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{8}{5}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{8}{3}{1}{1} +\end{scope} + +\begin{scope}[scale=\frameScale, yshift=3*\TikZyfshift, xshift=3*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] + \DrawFrame{colorFrameBackground} + \node at (12cm/2,-1.5cm) { $f_{t+3}$ }; +% \begin{scope}[yshift=3.1cm, xshift=1.9cm] +% \draw[erros] plot[scale=.5,smooth, tension=.7] coordinates {(0,0) (.5,3) (2.5,4) (5,3.5) (7.5,4) (8.5,3) (8.5,1) (10,-1.5) (3.5,0) (.5,-1.5) (0,0)}; +% \end{scope} + \DrawBlockWithSize{colorErrorsRegion}{4}{6}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{4}{5}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{4}{4}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{5}{6}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{5}{5}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{5}{4}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{5}{3}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{6}{6}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{6}{5}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{6}{4}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{7}{6}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{7}{5}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{7}{4}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{8}{5}{1}{1} +% \DrawBlockWithSize{colorErrorsRegion}{8}{3}{1}{1} +\end{scope} + +\begin{scope}[scale=\frameScale, yshift=4*\TikZyfshift, xshift=4*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] + \DrawFrame{colorFrameBackground} + \node at (12cm/2,-1.5cm) { $f_{t+4}$ }; +% \begin{scope}[yshift=3cm, xshift=1.8cm] +% \draw[erros] plot[scale=.6,smooth, tension=.7] coordinates {(0,0) (.5,3) (2.5,4) (5,3.5) (7.5,4) (8.5,3) (8.5,1) (10,-1.5) (3.5,0) (.5,-1.5) (0,0)}; +% \end{scope} + \DrawBlockWithSize{colorErrorsRegion}{4}{6}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{4}{5}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{4}{4}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{5}{6}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{5}{5}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{5}{4}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{5}{3}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{6}{6}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{6}{5}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{6}{4}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{7}{6}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{7}{5}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{7}{4}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{8}{5}{1}{1} + \DrawBlockWithSize{colorErrorsRegion}{8}{3}{1}{1} + +\end{scope} + +\path[draw, -latex, thick] (-.5cm,-.5cm) -- ++(12.5cm,0) node[pos=1, yshift=-.2cm] {$t$}; + +\end{tikzpicture} diff --git a/022_ErrorResilience/RobustTransmissionReview.tex b/022_ErrorResilience/RobustTransmissionReview.tex new file mode 100755 index 0000000..974c7d8 --- /dev/null +++ b/022_ErrorResilience/RobustTransmissionReview.tex @@ -0,0 +1,533 @@ +% Figure folders +\newcommand{\ErrorResilienceReviewFigs}{./022_ErrorResilience/Figures} +\newcommand{\TransmissionResiliencefigs}{./022_ErrorResilience/Figures} +\newcommand{\ErrorConcealmentReviewFigs}{./022_ErrorResilience/Figures} + + +\chapter{Robust Video Coding: A Review} +\label{Chapter:ErrorResilienceReview} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Introduction} + +In general, digital communications through networks are prone to transmission errors and data losses. +These errors can be handled in different ways, such as retransmission, error correction, and \ac{ec} \cite{ErrorResilience2000,PixelInterlacingIntraConcealment2011}. +This chapter reviews the main robust video coding techniques that can be applied at the video encoder/decoder to reduce the error propagation and increase the decode video quality. Several techniques have been proposed in the past to deal with the problem of robust video transmission, especially for video coding standard with high temporal dependencies, such as \ac{h264} and more recently \ac{hevc}. These techniques normally address the spatio-temporal dependencies in video coders which provide higher coding gains~\cite{ROPE2007} but lead to lower error robustness. In this chapter the error resilience and concealment schemes are divided in different categories depending on they functionality and algorithms used. Although different each approach follows a different techniques they all aim at striking the most efficient balance between coding efficiency and decoded video quality. + +This chapter is organised as follows: Section~\ref{Section22:Localisation} describes the localisation techniques, which reduce the temporal and spatial dependencies in coded bitstreams. Section~\ref{Section22:DataPartitionRedundant} provides a overview of methods based on data partition and classification and bitstream redundancy. These techniques ease the introduction of extra information in order to recover from lost data. Section~\ref{Section22:Concealment} covers the techniques used at the decoder-side as post-processing tools to reduce the effect of errors and improve visual quality and smoothness. Section~\ref{Section22:ConcealmentAware} provides a description of methods use the \ac{ec} techniques to provide improved error robustness at the encoder-side. Finally, Section~\ref{Section22:Summary} concludes this chapter by providing a overall comparison between the different approaches. + +\clearpage +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Categories of robust video coding techniques} + +%This chapter provides a review of the existing techniques used for robust video transmission under error-prone networks. +In order to increase the robustness of video transmission, robust coding mechanisms can be employed in a transmission chain to mitigate the artefacts caused by data loss. These techniques can be divided into several categories depending on the functional characteristics and the main problem they address~\cite{ErrorResilienceAug2005}. In this chapter we divide the robust video coding techniques into four categories: +\begin{itemize} + \item Localization techniques; + \item Data partitioning and redundant coding; + \item Error concealment by post-processing; + \item Concealment-aware error resilience. +\end{itemize} + +\begin{table}[!b] + \centering + \caption{Advantages of error robustness techniques by category.} + \label{tab:errorresilience} + \input{022_ErrorResilience/ErrorResilienceCategories.tex} +\end{table} + +Table~\ref{tab:errorresilience} provides a summary of the techniques covered in this section and the key advantages associated with each class of techniques. +Localisation techniques are able to reduce the spatial and temporal dependencies in the bitstreams~\cite{ErrorResilienceModeHEVC2016,ErrorResilienceModeHEVC2017}. +Data partitioning groups the coded data into different categories depending on their relative importance. +Redundant coding makes use of redundant blocks of data to correct the transmission errors and achieve robust decoding in error prone conditions. +The problem of robust transmission may also be addressed from the decoder side, by using post-processing \ac{ec} methods to ease the effects of data loss. +Finally, concealment-aware techniques might be adopted at the encoder to increase the performance of \ac{ec} algorithms and increase the reconstruction quality of lost data at the decoder~\cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ConcealmentIntraDataHidng2009,ConcealmentSignalling2008}. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Localisation techniques} +\label{Section22:Localisation} + +In general, video coding standard the compression efficiency is achieved by exploiting both the spatial and temporal redundancy of video sources. Since there is high correlation between neighbouring information, predictive coding schemes are very effective and obtain high compression gains exploiting such correlations. +The drawback of applying such techniques is the fact that the compressed symbols becomes very dependent on each other's which highly increase the error propagation. +Localization techniques break the predictive coding loop in order to reduce the likelihood of errors affect large regions of the video stream, both in the spatial and temporal domains. +As expected since localization techniques break the predictive coding, it reduces the efficiency of such methods. This leads to lower compression ratios and increasing bitrate. + +\begin{figure}[tb] + \centering + \includegraphics[width=0.7\columnwidth]{\TransmissionResiliencefigs/ErrorPropagation_1.png} + \caption{Examples of spatial and temporal error propagation and how it can be reduce using localisation techniques \cite{ErrorResilienceAug2005}.} + \label{fig2:ErrorPropagation} +\end{figure} + +This category of error resilience techniques are able to reduce the error propagation at two levels, i.e., temporal and spatial levels as shown in Figure~\ref{fig2:ErrorPropagation}. +As shown in the figure spatial techniques break the dependencies within a single frame, as temporal error propagation can be reduce by selectively breaking the dependencies across different frames. + +\subsection{Error resilient rate-distortion optimisation} + +A common localization technique addresses the error resilience problem of the rate-distortion optimisation on video encoders. Normally, such optimisation is equivalent to the minimisation of the Lagrangian cost given by: +\begin{equation} + J = D + \lambda R, +\end{equation} +where $D$ is the distortion of all pixels, $R$ the required bitrate, and $\lambda$ is the Lagrangian multiplier. +In \cite{ROPEJun2000,ROPE2007,ROPERefGenJan2010} a rate distortion optimisation for error resilience was presented. +In order to take into consideration possible network losses a recursive optimal per-pixel estimation (ROPE) of the end-to-end distortion is applied at the encoder-side. +Let $f_n^i$ denote the original value of pixel $i$ in frame $n$, $\hat f_n^i$ denote its encoder reconstruction and $tilde f_n^i$ the decoder reconstruction after error concealment if needed. +Since data loss may occur in the transmission channel, $tilde f_n^i$ is considered a random variable at the encoder. +The expected mean square error of each pixel is given by: +\begin{align} +E\left\{d_n^i\right\} &= E\left\{ \left( f_n^i - \tilde f_n^i\right)^2 \right\} \nonumber \\ + &= (f_n^i)^2 - 2 f_n^i E\left\{ \tilde f_n^i \right\} + E\left\{ \left(\tilde f_n^i\right)^2 \right\} +\end{align} +Then to take into consideration the channel losses packet loss probability ($p$) must be used in the distortion measurement. +This leads to the following for intra-coded pixels: +\begin{align}\label{EQ2:E2EDIntra} +E\left\{ \tilde f_n^i \right\} &= \left(1-p\right) \hat f_n^i + p E\left\{ \tilde f_{n-1}^i \right\} \nonumber \\ +E\left\{\left( \tilde f_n^i \right)^2 \right\} &= \left(1-p\right) \left( \hat f_n^i \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\}, +\end{align} +and inter-coded pixels: +\begin{align}\label{EQ2:E2EDInter} +E\left\{ \tilde f_n^i \right\} &= \left(1-p\right) \left( \hat e_n^i + E\left\{ \tilde f_{n-1}^{j} \right\} \right) + p E\left\{ \tilde f_{n-1}^i \right\} \nonumber \\ +E\left\{\left( \tilde f_n^i \right)^2 \right\} &= \left(1-p\right) \left( \left( \hat e_n^i \right)^2 + 2 \hat e_n^i E\left\{ \tilde f_{n-1}^{j} \right\} + E\left\{ \left( \tilde f_{n-1}^{j} \right)^2 \right\} \right) + p E\left\{ \left( \tilde f_{n-1}^i \right)^2 \right\} +\end{align} +where the inter-coded pixel $i$ is predicted from pixel $j$ in the previous frame. +The prediction error $e_n^i$, is quantized to the value $\hat e_n^i$, which is transmitted to the decoder along with the motion information. +Equations \ref{EQ2:E2EDIntra} and \ref{EQ2:E2EDInter} allow to measure the distortion taking into consideration the compression error but also the possible of frame loss and consequent \ac{ec} distortion. +This technique was used in \cite{ROPEJun2000} to switch between intra and inter coding modes reducing the temporal dependencies, and, thus achieving superior error robustness in video transmission. In \cite{AdvancesROPEHE2642007} this technique was optimized to quarter pixel estimation in \ac{h264} standard. Moreover, in \cite{ROPEBurst2009} this method is extended for burst losses by introducing the probability of burst losses into the original end-to-end estimation. This shown to improve the quality under frame loss. +In \cite{EndToEndDistortionConcealment2014} the distortion estimation assumes a temporal \ac{ec} at the decoder-side. +This increases the performance of the proposed as it does not rely of frame-copy which leads to lower reconstruction quality. More advanced \ac{ec} method were also consider in the study presented in~\cite{EndToEndDistortionConcealment2014}. + +In order to increase the error robustness of the \ac{hevc} standard distortion estimation was also used in the \ac{rd} optimisation \cite{ErrorResilienceModeHEVC2016} showing significant quality improvements comparing with the reference \ac{hevc} implementation. +The method in \cite{SSIMModeling2015} proposes the estimation of the end-to-end distortion using the \ac{ssim} \cite{QualitySSIMRef2002} due to its trade-off between complexity and efficiency to evaluate the perceptual quality \cite{VisualQualityMetrics2011}. + +\subsection{Intra-refreshing} + +In order to reduce the spatial dependencies one can introduce refresh points into the bitstream through intra-coded blocks \cite{IntraRefresh1999,IntraRefreshAttentionCM2007,IntraRefreshRefSelectionJan2009,IntraRefreshICIP2009,SandroIntraRefresh2011,JointIntraRefresh2014,IntraRefreshFeedback2015,IntraRefreshRPS2017}. +These techniques are fairly efficient to promote the error robustness of bitstreams, which avoid long error propagation at the temporal level. +The main feature of these techniques is to increase the number of intra-coded blocks as they have no dependencies to previous frames, however at the expense of higher amount of bits~\cite{IntraRefreshAttentionCM2007}. +Therefore, different alternatives have been proposed optimising the number of intra-coded blocks introduced by reducing it the refresh regions. +Such refresh regions can be introduced randomly, periodically, based on the motion information or on the end-to-end distortion estimation. +Moreover, whenever is possible the encoder can use a feedback channel to decide where to introduce refresh regions~\cite{IntraRefresh1999,IntraRefreshFeedback2015}. + +In the \ac{h264} reference software it was introduced a non-normative technique for random intra refreshing which introduces cyclic refresh not in raster scan order. Instead a random pattern is applied. This technique guarantees that all \acs{mb} are refreshed at some point, therefore errors do not propagate indefinitely. However, this technique refresh all \ac{mb} exactly the same number of times without consideration for rate distortion optimisation neither the region containing the \acs{mb}. +In \cite{IntraRefreshICIP2008} a rate control scheme is used to decide if a given \ac{mb} should be refreshed using a intra-coded block. The decision is performed based on the ratio of the Lagrangian cost between the best intra and inter mode. Whenever this ration is lower than a threshold a intra mode is selected. Using this guarantees that intra refreshing only forces an intra prediction mode if its Lagrangian cost is closer to optimal coding method. In \cite{IntraRefreshICIP2009} this is further extended to adapt the intra refresh algorithm to the network conditions, i.e., the threshold value is obtained from the bitrate and the \ac{plr} value. This technique automatically increases the number of intra-coded block as the bitrate and the packet loss increases. In case of \ac{hevc} intra refresh is also being used to reduce error propagation \cite{HEVCIntraRefreshICIP2017}. It was proposed to use joint intra-inter prediction to introduce soft-refresh points in the bitstream. Although such regions are not correctly decoded in case of errors (as happens with intra-refresh), it significantly decreases the bitrate increasing due the amount of intra-coded blocks. When compared with traditional intra-refresh bitrate saving of up to 12\% are achieved for a 5\% \gls{plr}. + + +\begin{figure}[tb] + \centering + \includegraphics[width=0.7\columnwidth]{\TransmissionResiliencefigs/ROI_IntraRefresh_1.png} + \caption{Examples of spatial and temporal error propagation and how it can be reduce using localisation techniques \cite{IntraRefreshAttentionCM2007}.} + \label{fig2:IntraRefreshAttention} +\end{figure} + +A different approach introduces refresh points based on the video contents. In~\cite{IntraRefreshAttentionCM2007} the error resilience of video streams is increased combining rate control and attention based models to define \acs{roi}. A weighting factor is used in the Lagrangian optimization to increase the number of intra blocks areas determined by the attention models. Using the aforementioned method lower refresh points are required to achieve high error resilience in the \ac{roi}. +A more recent approach was proposed in \cite{IntraRefreshPerceptual2010} using a perceptual noticeable difference model to determine an estimation of noticeable distortion. Then, using a visual attention model presented in [XX] intra refresh blocks are introduce in the bitstream. This method is able to improve the results of a previous work \cite{IntraRefresh2005}, especially for high motion sequences. This is due to the fact the previous approach only used an estimation of the error propagation without accounting for perceptual information. + +This was be also exploited for the \ac{hevc} standard \cite{HEVCROIErrorResilience2015} in combination with a feedback channel. Based on the received information the encoder can introduce intra \acs{cu} in the \ac{roi} areas. Experimental results reveal an increasing on the achieved video quality when higher number of intra \acs{cu} are used. Moreover, applying the proposed method to the \ac{roi} exclusively lower bitrate overhead is introduced leading to higher rate distortion gains. + +% \HIGHLIGHT{INTRA REFRESHING} + +\subsection{Leaky-predictions} + +Directly using reconstructed past frames as reference frames for predictive coding, may result in substantial error propagation whenever the reference frames are lost. +In order to reduce error propagation one can employ leaky predictions \cite{LeakyPred2009,ROILeakyPred2009}, which scales down reconstructed frames. +This leads to an exponential decay of error propagation, which leads to higher error robustness. +In order to generate such reference frames the following can be applied +\begin{equation}\label{EQ2:LeakyPredConstant} + \check f_n^i = \alpha \hat f_n^i + (1- \alpha)C, +\end{equation} +where $\check f_n^i$ is a reference frame pixel, $\alpha$ is the leak factor, and $C$ is a constant value. +The use of leaky predictions has been used in the past for error control especially in scalable video coding \cite{LeakyPredScalable2002}, where $C$ in Equation~(\ref{EQ2:LeakyPredConstant}) is replace by a lower layer error-free reconstruction. In case of single layer coding, the use of a constant value has significant impact on the coding efficiency leading to higher bitrate as shown in \cite{LeakyPredConstant1973}. + +In order to overcome the coding efficiency problem weighted prediction can be used with similar advantages. Since \ac{h264} motion compensation can be applied from two individual previously encoded frames and combined together to generate a single prediction for the current frame \cite{WeightedBiPred2005}. +In case of leaky prediction instead of applying motion compensation is applied once, but with an interpolated frame obtained as follow: +\begin{equation}\label{EQ2:LeakyPredDual} + \check f_n^i = \alpha \hat f_n^i + (1- \alpha) \hat f_{n-1}^i. +\end{equation} +Using $\check f_n^i$ as reference frame for prediction significantly decrease error propagation if only one of the original coded frames is lost (i.e., $\hat f_{n}^i$ or $\hat f_{n-1}^i$). + +\begin{figure}[tb] + \centering + \begin{tabular}{x{4cm}x{4cm}x{4cm}} + \multicolumn{3}{c}{\includegraphics[width=12cm]{\TransmissionResiliencefigs/ROILeakyPrediction.png}} \\ + $f_{n-1}$ & $f_{n}$ & $f_{n+1}$ \\ + \end{tabular} + \caption{Example of error propagation with a \ac{roi} (shaded area indicates error, and the arrows indicate temporal predictions to previous frame) \cite{ROILeakyPred2009}.} + \label{fig2:ErrorPropagationROI} +\end{figure} + + +In \cite{ROILeakyPred2009} in order to increase the error robustness with lower impact on the coding efficiency leaky prediction is confined to a pre-defined \ac{roi}. +Figure \ref{fig2:ErrorPropagationROI} shows an example of error propagation within two different regions: \ac{roi} region and background. +As show in the figure, since the \ac{roi} regions have predictions from regions in the background error propagation across regions occurs. +Therefore, the method in \cite{ROILeakyPred2009} uses a weighted prediction for the \ac{roi} region obtained as follows: 1) a globally optimal prediction obtained by unrestricted motion estimation, 2) a prediction obtained by restricting motion estimation within the \ac{roi} region. This limits the impact on \ac{roi} from errors that affect background region. Moreover, since the leaky predictions are confined to a specific regions higher coding efficiency is achieved when compared with other methods. +Equation (\ref{EQ2:LeakyPredDual}) results in a finite impulse response based weighting of reference frames, as previously weighted frames are not considered. In \cite{ROPERefGenJan2010} an infinitive impulse response based scheme is proposed to increase the decay in error propagation. To accomplish that, in Equation (\ref{EQ2:LeakyPredDual}) the use of the previous encoded frame ($\hat f_{n-1}^i.$) is replaced by the previous interpolated frame ($\check f_n^i$). +The idea of combining multiple predictions was also been used in the \ac{hevc} standard by combining it with end-to-end distortion to select the optimal predictions. + +\subsection{Reference picture selection} + +In order to reduce the temporal dependencies and reduce the error propagation reference picture selection has been widely used in the past \cite{ResilienceDualFrameBuffer2004,ErrorResilienceRPSFeb2009,RPSErrorsFeb2009,MVConcatenationAug2011}. +In \cite{ResilienceDualFrameBuffer2004}, an optimisation algorithm was used to select between short and long term references based on the end-to-end distortion obtained using the \ac{rope} method, in order to increase the prediction distance and reduce error propagation. +This method can also be used in combination with feedback channels to improve the references frame selection. +In the decoder side, an \ac{ec} method was later used to take advantage of such prediction structure \cite{ConcealmentDualFrameBuffer2005}. + +\begin{figure}[b] +\centering +\includegraphics[width=0.95\columnwidth]{\TransmissionResiliencefigs/RefPicSelection.png} +\caption{Examples of reference picture selection in error resilience video transcoding \cite{ErrorResilienceRPSFeb2009}.} +\label{fig2:ReferencePictureSelection} +\end{figure} + +A feedback channel was also used in \cite{ErrorResilienceRPSFeb2009}, to reduce error propagation at end-user decoders, by dynamically selecting the correctly received frames as references. Figure~\ref{fig2:ReferencePictureSelection} shows an example of error resilience transcoding, where based on network feedback the original temporal predictions (represented by the solid arrows) are transcoded to use the correctly received frame, i.e., prediction represented by the dashed arrow. +A more advanced scheme may be considered, where only the regions affected by mismatch predictions are process in the compressed domain, and subsequently transcoded, thus limit the dependencies to lost frames. +This was further exploited in \cite{MVConcatenationAug2011}. To limit complexity motion vector concatenation is employed, in order to obtain new inter-predictions to error-free frames. +In \cite{ChannelAdaptiveVideoStreaming2002} feedback-based is combined with path diversity to select the reference frames that are more likely to be received at the decoder. +In the proposed approach, the feedback is not directly used to select the reference frames. +Instead, the reference frames are selected through an end-to-end distortion estimation, taking into account the source distortion and the channel distortion (concealment distortion is also taken into account in channel distortion). +In \cite{ErrorResilienceModeHEVC2017} a \ac{roi} is defined based on the motion objects is used to limit the reference picture selection based on a feedback channel. +Moreover, intra refresh is also used based on the temporal distance to the previous intra-refresh frame. + +In~\cite{IntraRefreshRefSelectionJan2009} an algorithm to introduce intra refresh points at the macroblock level was proposed for the \ac{h264} standard. This method takes into consideration the distortion when no concealment is applied, the distortion when frame-copy concealment is applied and the packet loss conditions, in order to select the refresh blocks. Those refresh blocks are encoded using either intra prediction or inter prediction through a secondary algorithm to select the most reliable reference frame (i.e., frame with less total expected distortion). + +In the methods mentioned above the dependencies between the frames are analysed and reduced in order to decrease the error propagation in cases of data loss. Since the existing video coding standards use intense predictive coding, these methods present an efficient approach in highly erroneous conditions. +% However, the method in~\cite{RDReferenceSelectionJul2012} uses more complex distortion estimations, and in ideal conditions multiple loop encoders are needed to select the reference frames optimally. +Although these closed-loop techniques may offer a reliable control of the decoder distortion, the feedback required from the receiver side and multipath networking are not always available neither feasible in real-time and unidirectional channels (e.g. broadcast). + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Data partition and redundant coding} +\label{Section22:DataPartitionRedundant} + +In a compressed bitstream different symbols have different relevance in terms of error propagation, therefore the impact on the decoded quality is dependent on lost symbols. Therefore, data partition techniques have been developed to group different parts of the compressed bitstream according to their importance to the decoded quality. Then, some of those groups can be protected more efficiently than others using \ac{uep}. These techniques are normally associated with \ac{fec} algorithms. By applying multi-levels of \ac{fec}, different levels of error protection are achieved. + +In a single channel transmission, the important groups can be protected with stronger channel codes than the remaining groups. In case of multi-channel the important groups can be routed over the reliable channels. Moreover, to achieve higher quality in multi-channel transmission one can use extra information to increase the error resilience of video streams, and be able to decode then in case of errors. Redundancy can be added in an explicit form, e.g., redundant picture, or it can be implicit, e.g., multiple description coding \cite{MDC2005}. + + +\subsection{Flexible Macroblock Ordering} + +The \ac{h264}~\cite{H264OverviewJul2003} introduced an error resilience tool named \ac{fmo} technique~\cite{H264FMO2006}. This technique allows each \ac{mb} to be assigned freely to a specific slice group (i.e., a set of slices), organized in a mixed-up fashion. The standard supports up to eight slices groups in each picture, and \acs{mb} within a group can be assigned to a several slices in the default raster scan order. +The case where only one slice groups is used corresponds to the case where the \ac{fmo} tools is disabled. +\ac{fmo} increases the error resilience by partitioning the bitstream into several groups. Results show that by using \ac{fmo} it is possible to enhance the subjective and objective quality especially for high packet losses with at cost of an affordable overhead~\cite{H264FMO2006}. + +\ac{h264} \ac{fmo} support six types of slice groups. Figure~\ref{fig2:FMOTypes} shows \ac{fmo} types 0 to 5. In the \ac{fmo} Type 0 each slice groups has a maximum number of \acs{mb} before the next slice group begins. However, the same slice groups can appear more than once in a frame. \ac{fmo} Type 1 corresponds to scattered slices, where the \acs{mb} are assigned to each slice groups with a fixed formula. In \ref{fig2:FMOTypes} it is represented the \ac{fmo} type 1 case with two slice groups resulting in a checkerboard pattern. In the \ac{fmo} Type 2 slice groups are defined by rectangular shapes defined by the top-left and bottom-right corners. Although the standard allows slice groups to overlap each other, each \ac{mb} may only belong to one slice group. \ac{fmo} Type 3, 4 and 5 are special types each allow the configuration to change on every frame. It only allows two slice groups, in order to keep the signalling overhead affordable. The last \ac{fmo} type is the most general case, where the slice groups are explicitly defined using high level packets (i.e., signalling \ac{nal} units), allow more flexibility at the cost of higher signalling overhead. + +\begin{figure}[tb] + \centering + \includegraphics[width=0.8\columnwidth]{\TransmissionResiliencefigs/FMO_Slices_Type.png} + \caption{Different types of flexible macroblock reordering \cite{H264FMO2006}.} + \label{fig2:FMOTypes} +\end{figure} + + +In the past \ac{fmo} has been used widely used to increase the error robustness of video transmissions~ \cite{FMOErrorResilience2007,FMOResilience2010,FMOResilience2011,FMOROI2012,MotionEnergyDataPart2014} in \ac{h264} streams. In ~\cite{FMOErrorResilience2007} \ac{fmo} is used to increase the \ac{ec} performance and thus reduce the error propagation in H.264. The method divides the \acs{mb} into the two slice groups based on the \ac{ec} distortion using a two-pass approach. Then, the slice groups which has more impact of the \ac{ec} distortion is transmitted using a reliable channel, and it is assumed to be received error-free. Results show that the proposed approach is able to outperform the dispersed mode (\ac{fmo} type 1). +In \cite{FMOResilience2010} slice classification is also used based on the weighted sum of the motion vector information and the rate-distortion cost. As in the previous approach, the \acs{mb} are ranked based on the importance and distributed in the two groups until the same amount of \acs{mb} are presented in the two slice groups. This was further improved in \cite{FMOResilience2011} by applying a prediction based on the previous frame to estimate the mapping of the \acs{mb} into the slice groups. This reduces the computational complexity of the \ac{mb} grouping while achieving similar error resilience performance. +The method in \cite{FMOROI2012} divides the video into two regions which are then grouped in different slice groups. In order to classify the different \ac{mb} spatial information is devised from the \ac{mb} bit allocation, and temporal information is extracted based on the \ac{ec} distortion. This technique is able to reduce the number of erroneous \ac{mb} up to 82\% achieving quality of up to 3.97\%, measured with \ac{psnr}, when comparing to the case where \ac{fmo} is not used. +In \cite{MotionEnergyDataPart2014} a method based on motion energy is also used to classify \acs{mb} into different groups by . +In order to classify a given \ac{mb} into a groups its motion energy is compared with the motion energy of its neighbours and the average motion energy of the entire frame. This allows to obtain a dynamic threshold which is then used to determine the \ac{mb} importance. This method is compared with other \ac{uep} error protection schemes showing quality improvements under different \acs{plr}. + +In \cite{FMORedudantSlices_TB2007} a new \ac{fmo} type is proposed combining the type 1 and 3 in order to obtain a checkerboard pattern in a box-out scanning order, thus increasing the scattering of lost \acs{mb}. Results show that the proposed \ac{fmo} type is able to outperform the existing types under single and burst loss events. +The method described in \cite{FMOTypeROI2014} introduces a new type in order to efficiently divide the video contents in two regions with different levels of importance, based on the \ac{fmo} Type 6. Then, a spiral protection is applied to scan the \acs{mb} grouping then in two groups. Moreover, by proposing a new \ac{fmo} type they are able to signal several \ac{roi} areas achieving lower overhead than using the original \ac{fmo} Type 6. + + +\subsection{Unequal error protection} + +\glsreset{uep} + +Video signals normally have different types of content, both at the temporal and spatial level. Therefore, there is the possibility of categorising it into different groups. Moreover, it is well-know that different regions of a video bitstream are affected differently by transmission errors. +This motivated the use of \Gls{uep} techniques in video streaming, which allows different parts of bitstream to be better protected against errors. This not only ensures that those regions will be delivered with higher quality, but also leads to a reduction of the total overhead required to increase the error robustness of video streams. + +\ac{uep} has been used for a long time to increase the error robustness of video transmission. A wide variety of unequal loss protection schemes have been developed in the literature to optimize the performance of video communication over networks with packet loss. It has been observed that, within a \ac{gop}, different frame types (e.g. I, P, or B frames) have different levels of contribution to the reconstructed video quality. This has been exploited for unequal error protection in video transmission as shown in~\cite{UEP_TM2005,UEPMotion_TM2006}. + +In \cite{UEPLayered_TB1999} an evaluation of layered video was described. In order to increase the error robustness, redundant packets are used which ease the recovering of lost ones. In order to reduce the overhead for error protection each video layer was given a different importance and protected using different levels of redundancy, thus guaranteeing that lower level packets have a higher probability of being decoded without errors. This is a simple approach not dependent on the network capabilities, neither requires a deep knowledge of the video signals. A lightweight model is used to assign packets priorities to be used by error protection algorithms~\cite{UEP_TCE2011}. This way an \ac{uep} can be achieved. The model is based on a weighted sum of four components: (i) a slice type parameter giving higher importance to refresh and reference frames; (ii) a binary flag indicating where a network packet transports video related headers, i.e., \ac{nal} or slice header; (iii) a parameter indicating the distance to the packet containing the next slice; (iv) a parameter indicating the distance to the next refresh frame. The encoder-side is able to extract these parameters based on the \gls{nal} header, which makes it suitable to low-end encoders or even to prioritise each packet at a streaming level. + +However, \ac{uep} techniques usually require a deep video analysis which is difficult its integration in every transmission scenario. +For instance, one can use the motion information to classify different frames or even different regions within a frame, e.g., \ac{mb} in \ac{h264}\cite{UEPMotion_TM2006}. The motion information is used to determine the motion energy which is given by the L2-norm of the \acs{mv} within a determined region. +In~\cite{UEPMotionEnergy_WCNC2013} the motion energy of each frame is computed and compared with neighbouring frames. To achieve this the energy of co-located \acs{mb} in subsequent frames are compared, and higher importance is given to frames with higher motion energy differences. This method was compared with equal error protection revealing higher objective and subjective quality revealing the relation between motion information and impact of errors. +This work was extended in order to use a two levels unequal protection~\cite{UEPMotion_ICSPCC2012,UEPMotion_ICSPCS2013}. Not only \ac{mb} and frame importance is taken into consideration but also the combined importance of different frames are used to determine the importance of each \ac{gop}. + +In \cite{Zhang20131390} the transmission distortion is estimated at the packet level, to assess the impact of each packet in the error propagation. Then, different priorities are applied to each packet allow different levels of error protection. This was also be used to schedule the packet transmission through multi path networks\cite{PacketScheduling2010}. +In order to improve the classification of different regions of a video stream a perceptual algorithm can be used. This increases the correlation between the classification and subjective impact of errors. +In \cite{UEPSaliency_TCSVT2010} an method based on a combination of two parameters: a spatial parameter based on a perceptual attention model and a temporal parameters based on the frame position within \ac{gop}. The parameter is then used for \ac{fec} assignment, and experimental results show a significant quality improvement both used objective metrics and subjective evaluation. + +In scalable video coding scheme, the layered structure of the bit stream provides a natural framework for \ac{uep} \cite{UEPScalable_TCE2008,UEPScalable_TCSVT2010,UEPScalable_ICSPICES2015,UEP_TM2018}. +In~\cite{UEPScalable_TCSVT2010} an algorithm to estimate the overall distortion of reconstructed frame in the decoder is proposed for scalable video. At the encoder-side different enhancement layer truncation, error propagation and \ac{ec} at taken into account to perform the distortion estimation. The method performs a recursively computation of the total expected distortion in the decoder at the picture-level for each layer in the prediction hierarchy. This is used to perform a cross-layer optimisation and apply different levels of error protection. +The work in~\cite{UEPLayerVideo_TM2014} proposes a technique for finding the optimal coding rates for coded bit-streams at the encoder-side, in order to optimise the necessary \ac{fec} codes at the enhancement layer which aid the decoding of erroneous base-layer video. This not only improves the overall decoded video quality but also prevents that the correctly received enhancement layers are discarded due to missing dependencies. + +In the context of \ac{hevc} there are also methods to achieve unequal error protection. Since techniques like \ac{fmo} are not available, different approaches were taken into account \cite{HEVCUepLte_IWCMCC2015,UEPVideoDegradation_ISCAS2017,UEP_TM2018}.The work in~\cite{SliceImportance_ICHNRSR2015} proposes a resource allocation scheme based on prioritisation for \ac{lte} Advanced networks~\cite{HEVCUepLte_IWCMCC2015}. The prioritisation is performed at the slice level (i.e., \ac{nal} unit level) based on the inter-frame predictions. Such predictions are analysed and a ranking is given based on the amount of pixels predicted based on the current slice. In~\cite{UEPVideoDegradation_ISCAS2017} a quality degradation model is proposed for video encoded using the \ac{hevc} standard. The quality degradation is obtained by simulation error events and measuring the quality using the \ac{psnr} metric. From the empirical results a model can be devised to predict the quality under different events. This model is used to control the amount of redundant codes are applied at the network level. The work in~\cite{UEP_TM2018} proposes the use of \gls{uep} for video storage in cloud environments in the context of the scalable extension of the \ac{hevc} standard. The proposed method not only takes into consideration the layer information, but also considers the picture type (i.e., I, P or B-frame) to achieve a cross-optimisation. + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%\section{Redundant Coding} +\subsection{Redundant pictures} + +Redundant pictures is a well-known technique adopted in the \ac{h264} standard \cite{RedudantPicsJan2009}, which allows for different representations of the same picture to be transmitted using different encoding parameters. +For example, the primary picture may be coded with fine quantization, while the redundant one with coarse quantisation at lower bitrate. +One the one hand if the primary picture is received, the redundant information is discarded. +One the other hand if errors occur and the primary picture is lost, the redundant one is used to provide a lower level of reconstructed quality and contain the error propagation. + +Zhu \emph{et al.} investigated the paradigm of redundant picture coding in \ac{h264} standard~\cite{RedudantPicsJan2009}. +Different approaches were investigated in order to achieve a mixed method based on both reference picture selection to obtain a clean decoding without error propagation and a hierarchical allocation of the redundant pictures to increase the error robustness. +Redundant pictures achieve good performance under error prone conditions, since they introduce secondary pictures that can be decoded when the primary picture is not available. + +Moreover, redundant pictures might also work as localisation techniques when different reference frames are selected. However, such approaches generate large amount of redundant data and overheads. In~\cite{RedundantMVsJune2008} a newer approach was proposed to reduce the amount of information in the redundant pictures of the \ac{h264} standard, by introducing only redundant \acs{mv}. This approach increases the overall system performance by presenting an efficient trade-off between the resultant bitrate and the error robustness. + +\subsection{Multiple description coding} + +\begin{figure}[tb] + \centering + \includegraphics[width=0.7\columnwidth]{\TransmissionResiliencefigs/MDC.png} + \caption{Architecture of multiple description system \cite{MDC2005}.} + \label{fig2:MDCSystem} +\end{figure} + +In a \ac{mdc} scheme different bitstreams, named descriptions, are encoded such that a acceptable level of quality is achieve with one of the bitstreams isolated~\cite{MDC2005}. +Figure \ref{fig2:MDCSystem} illustrates the a basic architecture of encoder and decoder used by the \ac{mdc} technique. +The encoder-side creates two descriptions, i.e., $R_1$ and $R_2$, which shall be transmitted using two different channels, reducing the probability of both being lost at the same time. +As shown in the figure three situation may happen at the decoder-side: both descriptions are correctly received and decoder using the central decoder, or either one of the descriptions are not received and then one of the side decoders is used. +When both bitstreams are received, the central decoder combines the information carried by each description achieving higher quality. +In the \ac{mdc} encoder the redundancy can be controlled by changing the correlation between descriptions. Higher correlation leads to higher redundancy but also higher quality in the output of the side decoders. +In contrast with redundant pictures the redundant streams do not carry the same information which means that if both are received the decoded quality is increased. + +Different types of multiple description coding \cite{MDCHierarBFeb2012} can also be used to improve the quality under transmission errors in multiple path networks \cite{MDCCorreiaJun2012}. Although these techniques can be very effective in multi-path networks, a significant amount of overhead is required to create the different descriptions. + +%\HERE + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Error concealment by post-processing} +\label{Section22:Concealment} + +\glsreset{ec} + +In general, the \ac{ec} operation is a post processing step in the decoder side and it is the last step to handle errors in the communication chain\cite{ConcealmentReview1998}. Thus, concealment of missing data should take place within the video decoder to minimise perceived effects of lost data. In summary, to enhance the 3D video quality and depth perception, the design of the transmission system must involve the development of adequate \ac{ec} tools. This section reviews several \ac{ec} methods based on spatial and temporal redundancies. + + +\subsection{Spatial error concealment techniques} +% \label{SEC3:SpatialConcealment} +% +% This section introduces several concealment techniques that exploit the spatial redundancy of images to recover the lost data. To recover lost information, the methods introduced in this section entirely rely on neighbouring image regions received without errors. +% Consequently, spatial error concealment is mostly useful in the case where information of the neighbouring frames are not available. + +%\subsection{Pixel level concealment} + + +A typical approach to conceal missing regions in an image is based on linear interpolation of neighbouring pixels \cite{ErrorConcSpatial1995,ConcTCE2001}. Figure~\ref{FIG3:pixel_interpolation} shows the estimation process of a lost pixel ($lp$), within the lost region ($R_L$), using the four closest pixels ($p_i$) as reference, and the distance between the lost pixel and the reference ones ($d_i$). The pixel value ($lp$) is recovered using a weighted average, as follows: +\begin{equation} +\label{eq:pixel_interpolation} +lp = \left. \sum_{i}^n{ \frac{p_i}{d_i} } \middle/ \sum_{i}{ \frac{1}{d_i} } \right. . +\end{equation} +%\begin{equation} +%\label{eq:pixel_interpolation} +%\normalsize lp = \frac{\sum_i{ \frac{p_i}{d_i} } }{ \sum_{i}{ \frac{1}{d_i} } }. +%\end{equation} + + +Liner interpolation is a simple yet effective method to recover lost regions in digital images. However, as each lost pixel is recovered by giving the same importance to each reference, the image discontinuities are not taken into account. Thus, some pixels used as reference may belong to distinct regions of the image, without significant correlation with the lost region itself. +This leads to a poor estimation of lost pixels, resulting in noticeable artefacts in the reconstructed image. For example, in Figure~\ref{FIG3:pixel_interpolation} the lost pixel ($lp$) and the pixel $p_4$ are separated by an edge. Therefore, $p_4$ may not be suitable to recover the lost pixel value. So, to ensure a correct concealment of the lost region, the edges and image transitions must be considered, by selecting correct pixels to be used in the interpolation function. To overcome this problem, a directional interpolation can be performed, by recovering the lost pixels based on those within the same region, according to the edge directions. + +\begin{figure}[tb] +\centering +\input{\ErrorConcealmentReviewFigs/LinearInterpolation} +\caption{ Lost pixel interpolation from neighbouring error-free pixels. } +\label{FIG3:pixel_interpolation} +\end{figure} + +The directional interpolation improves the performance of the pixel recovery, by estimating the lost edges using either correctly received or previously recovered pixels. In order to obtain the best direction to interpolate, edges are calculated using the image gradient, for example applying the Sobel operator \cite{SignalImageVideoProcessing2012}. This operator gives the first-order approximation of the gradient of the image. This is widely used in image processing to locate the image edges, and uses an horizontal and vertical mask given by: +\begin{equation} +G_x = +\begin{bmatrix} +-1 & 0 & 1 \\ +-2 & 0 & 2 \\ +-1 & 0 & 1 +\end{bmatrix},\ \ +G_y = +\begin{bmatrix} +-1 & -2 & -1 \\ + 0 & 0 & 0 \\ + 1 & 2 & 1 +\end{bmatrix}. +\end{equation} +The convolution between the image pixels and the Sobel matrixes results in both horizontal ($g_x$) and vertical ($g_y$) gradients, from which the magnitude and the angular direction are computed using, +\begin{equation} +\begin{aligned} +G &= \sqrt{g^2_x + g^2_y}, \\ +\theta &= tan^{-1} \left( \frac{g_y}{g_x} \right).\\ +\end{aligned} +\end{equation} + +The Sobel operator is an efficient algorithm to detect edges, as it can reach extreme values at the image edges, so it can be used in pixel interpolation. The usage of this operator for \ac{ec} was evaluated in \cite{IntraConc2010,FineDirectTCE2006,SpatialEdgeConsecutiveBlocks2004,SpatialEdgeModConsecutiveBlocks2005,SpatialInterpolation2008}, and it is characterised as a fast and efficient method due to its small kernel \cite{GradEstimation2009}. + +% \abbrev{FC}{Frame-Copy} +% \abbrev{MC}{Motion-Copy} + +\subsection{Temporal error concealment techniques} + +\subsubsection{Motion-copy} + +The simplest way to recover missing frames in the temporal domain is using the \ac{fc} or the \ac{mc} method. These two basic concealment methods were proposed for the H.264 reference software in \cite{H264RefConc2006}. The \ac{fc} method consists in repeating (copying) the temporally closest frame in order to fill the gap of the missing frame. The \ac{mc} method uses the motion information of the closest frame to recover the missing one, thus the missing pixels are recovered through motion compensation. The experimental results showed that the \mbox{motion-copy} method achieves higher quality in the reconstructed frames in most of the cases. However, the complexity increases with the implementation of the \ac{mc} method, when compared with the \ac{fc} method. + +In \cite{2DRecursiveRefinement2010} a whole frame loss error concealment algorithm was proposed to improve the performance of the \ac{mc} method, since no refinement is performed in the \acs{mv} obtained from the closest decoded frame. Moreover, in this work the \acs{mv} between consecutive frames are assumed to have a strong correlation, so the motion information of several preceding frames is taken into account to estimate the lost one. Then, the motion vector differences between the vectors of successive frame pairs are recursively calculated, as well as the next refinement area is determined, until it becomes unchanged. This is shown to improve the performance of the \ac{mc} algorithm, achieving gains up $0.47\ dB$ (\ac{psnr}) for a \ac{plr} of $20\%$. + + +\subsubsection{Boundary match algorithms} +% \abbrev{ICMV}{Interpolated Candidate Motion Vectors} + +The recovery of missing blocks in video frames may rely on the available neighbouring regions, to find the suitable \ac{mv} to reconstruct the missing pixels. +A low complexity approach may use the boundary information of surrounding blocks to search for the motion vector based on a spatial smoothness constraint on boundaries of a lost \ac{mb} \cite{BMAAlgorithm2008}. Figure~\ref{FIG3:BMAPixelInformation} represents the boundary information used to estimate the \ac{mv} for the lost block from a set of different \acs{mv} candidates. +The \ac{bma} uses the available \acs{mv} provided by the top (T), bottom (B), left (L) and right (R) blocks (see Figure~\ref{FIG3:BMAPixelInformation}) to fetch the pixels from the previous decoded frame. +Then, the optimal \ac{mv} is selected according to the sum of differences, between pixels on the internal boundary in the recovered block and the corresponding ones on the external boundary. +These pixels are marked in the figure. + +\begin{figure}[tb] + \centering + %\includegraphics[width=0.45\columnwidth]{\ErrorConcealmentReviewFigs/BMAAlgorithm} + \input{\ErrorConcealmentReviewFigs/BMAAlgorithm} + \caption{Illustration of the boundary match algorithm.} + \label{FIG3:BMAPixelInformation} +\end{figure} + +In \cite{BMAAlgorithm2006}, error concealment based on the boundary matching algorithm is improved, by introducing the concept of the interpolated candidate motion vectors. +The candidate \acs{mv} are calculated with an interpolation of the motion vectors of the neighbouring block in the missing frame and the region around the corresponding block in the previous frame. +The interpolated candidate \acs{mv} are created based on the change of magnitude and angle of the top, bottom, left and right neighbouring \acs{mv} of the lost block from the previous frame to the current one. +Then, these modifications are applied to the \ac{mv} of the co-located block with the lost one in the previous frame. +A more generalised proposal is to apply such transformation not only to the closest neighbours but also to a wider area of blocks. +The concept of interpolated candidate \acs{mv} can be applied to any method based on \ac{bma}, improving its performance, by introducing more accurate candidates. + +Another improvement to \ac{bma} was obtained with the outer boundary matching algorithm\cite{OBMAMethod2008}, also known as decoder \ac{mv} estimation algorithm in \cite{DMVE2000}. +In this method the distortion is calculated by the difference of two outer boundaries of reconstructed block, instead of internal and external boundary in \ac{bma}. +Although \ac{bma} and outer \ac{bma} have the same design principle of minimising the mismatch distortion, in later the distortion function is measured in the outer boundaries, \emph{i.e.}, external boundaries of the reconstructed block. It was observed in the experimental results that outer \ac{bma} has better performance in video recovery \cite{OBMAMethod2008} than conventional \ac{bma}. The performance improvements are explained by the usage of the outer boundary in the distortion functions, which incorporates the edge information within the neighbourhood of the lost block in the decision of the most accurate \ac{mv} to recover the missing pixels \cite{OBMAMethod2008}. Moreover, in the method presented in \cite{OBMAMethod2008} further \ac{mv} candidates were introduced, a refined local search is performed and multiple boundary layers, \emph{i.e.}, increase the overlapped region to measure the distortion and to select the final MV, were tested. The experimental results revealed that the usage of a single layer match gives better performance than multiple layers, while selective search gives excellent trade-off between the image recovery and computation complexity + + +% \abbrev{OBMA}{Outer Boundary Matching Algorithm} +% \abbrev{DMVE}{Decoder Motion Vector Estimation} + + + +% In \cite{DynamicBMA2010} a novel approach to measure the spatial distortion on boundaries of the reconstructed block is presented. Instead of using only horizontal and vertical directions, diagonal and anti-diagonal are also tested, and the best direction is chosen from the reference frame. Since block boundaries may change abruptly, instead of minimising the distortion function the proposed method aims to approximate this distortion to an optimal difference. This optimal difference are determined in the neighbourhood of the co-located block in the reference frame. +% In \cite{BMAAlgorithm2008} a novel distortion function was proposed, to choose among different candidate MVs. The distortion function exploits both spatial and temporal smooth constraints. The temporal function is based on the outer boundary distortion, as in \cite{OBMAMethod2008}. +% The spatial distortion aims to reduce the gradient variances in boundaries of the reconstructed block, so the image edges may be guaranteed. Several experimental results were obtained for different loss events, covering from isolated block lost to the loss of a complete row of blocks. The proposed method is able to achieve higher quality in the reconstructed frame when comparing with the \ac{bma} method. Moreover, subjective analysis reveal that this method recover the edges in the lost image. However, the method uses complex mathematical operations, which may not be feasible in real time applications. + +\subsubsection{Motion vector extrapolation} + +As mentioned before, the \ac{mc} method is a simple way of concealment a missing frame using motion information from previously decoded frames. Although this method can provide an acceptable performance comparing with the \ac{fc}, as shown in \cite{H264RefConc2006}, more complex algorithms that use the assumption of motion continuity across consecutive frames are proposed to conceal, not only missing blocks, but also whole frames in compressed video. + +% \abbrev{MMA}{Multi-frame Motion vector Averaging} + +% To recover the entire missing frame a multi-frame motion vector averaging (MMA) algorithm was proposed in \cite{MMATempConc2003}. The method uses \acs{mv} of a few past frames (from $t-1$ to $t-L$), in order to estimate the forward \ac{mv} for each pixel in the last received frame. Then, each pixel of the last frame is projected onto the missing one. More specifically, the method proposed in \cite{MMATempConc2003} recovers the missing frame at instant $t$ using the following steps: +% \begin{enumerate} +% \item Generation of a \ac{mv} history based on the previously decoded frames from $t-1$ to $t-L$, with $L \in \left[ 2,5 \right]$, in order to understand which pixels were used to predict the ones of the frame at $t-1$. +% \item The \ac{mv} history is used to obtain a set of forward MV, from the frame $t-1$ to the missing one (at instant $t$). For each pixel, the forward \ac{mv} is measured by averaging the \ac{mv} history. +% \item A spatial regularisation of the \ac{mv} field is performed, by applying a median filter with a kernel size of $12$, to approximate an \mbox{edge-preserving} regularisation \cite{MedianFilterRegul2000}. +% \item The reconstruction of the missing frame is performed, at a half-pixel level, through the previously obtained forward MVs. The overlapped pixels are averaged, in order to obtain the final pixel value. +% \item The unfilled regions are recovered, with the median value of a $7\times7$ window +% \item Finally, the fully reconstructed frame at a half-pixel level is filtered using a mean filter with a $2\times2$ kernel, and down sampled. +% \end{enumerate} + +% This method usually produces relatively high quality in the reconstructed frame, but in low motion cases it behaves worse than the \ac{fc} method under a few conditions, as indicated by the experimental results in \cite{MMATempConc2003}. This phenomenon happens because the oldest reference frames usually do not have \ac{mv} similar to the missing one. Moreover, if $L$ increases, the complexity of the proposed algorithm may not be useful to be used in error concealment, due to complexity and memory requirements to determine the \ac{mv} history. + +\begin{figure}[t] +\centering +\includegraphics[width=0.9\columnwidth]{\ErrorConcealmentReviewFigs/MVE} +\caption{\ac{mve} extrapolation with the corresponding extrapolated macroblock, based on the method presented in \cite{BlockBasedMVE2002}.} +\label{FIG3:MVEExtrapolation} +\end{figure} + +\ac{mve} \cite{BlockBasedMVE2002,BlockBasedMVEOverllaped2011} methods are a simple but efficient way to reconstruct the \acs{mv} of a lost frame, and they are able to achieve good performance. In the method presented in \cite{BlockBasedMVE2002}, \acs{mv} are extrapolated from the last decoded frame \mbox{(reference frame)} onto the missing one. Firstly, each \ac{mb} in the reference frame is projected to the missing one with its own direction given by the corresponding extrapolated motion vector ($mv'$). An example of motion vector extrapolation is shown in Figure~\ref{FIG3:MVEExtrapolation}, where the vectors of the frame $t-1$ are projected onto the missing frame at instant $t$. Note that, in the reference frame ($f_{t-1}$) the motion vectors may point to different frames ($f_{t-2}$~and~$f_{t-3}$), previously decoded, so \ac{mv} extrapolation must take into account the difference between the original temporal distance and the projected one. +Secondly, to improve the accuracy of \ac{mve}, the damaged region is divided into $4$ blocks ($8\times8$ pixels). Then, according to the matching area between the extrapolated block and the damaged $8\times8$ region, the best \ac{mv} is chosen. +In order words, the \ac{mv} for the each block is obtained from the extrapolated block that achieves a higher number of overlapped pixels with the missing $8\times8$ region. Finally, the reconstructed \acs{mv} are used to fetch the missing pixels from the previously decoded frame ($f_{t-1}$). Although this method is able to overcome the disadvantage of incorrect \ac{mb} displacement, the choice of the best \ac{mv} is not very effective \cite{PixelBasedMVE2004}. In \cite{BlockBasedMVE2002} the \ac{mve} is used to extrapolate the \ac{mb} of the reference frame. +This results in overlapped regions, which are recovered using a weighted average of the multiple pixels candidates. +The weight coefficients assume a maximum value in the center of the extrapolated region, and a minimum value at the boundaries. + +Since the choice of the best candidate \ac{mv} relies on the overlapped area of the extrapolated block with the lost region, the method may give erroneously estimated \ac{mv} in large motion scenes. The example of Figure~\ref{FIG3:MVEPixelBased} shows a lost $8\times8$ block (in the centre of the figure) and four possible extrapolated \acs{pu} that can be used to estimate lost the motion information. As illustrated in the figure, although the largest overlapped area belongs to the PU$_2$, the overlapped pixels correspond to a small portion of the whole missing block. Moreover, the majority of the lost block's area was not covered by any extrapolated \ac{pu}. Thus, motion recovery using only the information available in PU$_2$ will not properly characterise the motion information for all lost pixels. + +To overcome the previous limitation and to achieve smoother and finer results, a \ac{pmve} algorithm was proposed in \cite{PixelBasedMVE2004}. This motion vector extrapolation method is divided into two cases: +\begin{enumerate} + \item The pixels in the lost region that are covered by at least one extrapolated block, such as circle points in Figure~\ref{FIG3:MVEPixelBased}, are recovered using the average of the \acs{mv} of the overlapped block. + \item Those pixels which are not covered by any extrapolation, such as the triangle point in Figure~\ref{FIG3:MVEPixelBased}, the \ac{mv} of the co-located pixel in the previous frame is used. +\end{enumerate} + +\begin{figure}[t] +\centering +\input{\ErrorConcealmentReviewFigs/PixelBasedMVE} +\caption{Pixel based motion vector extrapolation \cite{PixelBasedMVE2004}.} +\label{FIG3:MVEPixelBased} +\vspace{3mm} +\end{figure} + +Then, if the estimated \ac{mv} is $(MV_x,MV_y)$, then the estimation of the missing pixel, $p_m(x,y)$, is obtained by: +\begin{equation}\label{EQ:MotionComp} + p_m(x,y) = p_r(x+MV_x,y+MV_y), +\end{equation} +where $p_r$ refers to the pixels in the previous frame. This pixel-based \ac{mve} provides similar results than the block-based \ac{mve} algorithm in low motion scenarios. Nevertheless, it is able to overcome the previous method in large motion scenes. +An extension to the previous approach was considered in \cite{PixelBasedMVE2004}, by using the \acs{mv} of subsequent available frames. Thus, to estimate the lost motion information, forward and backward extrapolations are performed, in order to obtain two candidates to each lost pixel. Then, the lost pixel is estimated using the Equation~\ref{EQ:MotionComp}, using the \ac{mv} determined by the average of two \ac{mv} candidates. The experimental results revealed that the bi-directional \ac{pmve} outperforms the isolated use of forward or backward \ac{pmve}. Moreover, the bi-directional \ac{pmve} achieves gains up to $1.85$ dB and $1.32$ dB (\ac{psnr}) comparing with the work in \cite{MMATempConc2003} and \ac{mve} \cite{BlockBasedMVE2002}. + +In \cite{2DConcHybridMotion2010} a \ac{hmve} algorithm was proposed, to provide more accuracy to the estimated motion information. The method aims to improve the accuracy of the \acs{mv} of the \ac{pmve} method, by using the extrapolated vectors at the pixel and block level. Moreover, the proposed method discards the \ac{mv} that were wrongly extrapolated in order to obtain an accurate MV. In this concealment method, firstly a pixel based motion extrapolation is performed and the pixels is organised into three groups. Group A contains the pixels that are overlapped by at least one extrapolated block (both circle and diamond points in Figure~\ref{FIG3:MVEPixelBased}). +Group B includes the pixels that are not covered by any extrapolated block, but belong to a block that is covered, \emph{e.g.}, in Figure~\ref{FIG3:MVEPixelBased} the triangle point was not covered by any extrapolated \ac{mb} but it belongs to $8\times8$ block that is covered by 4 blocks. +The remaining pixels are included in group C (star pixels in the figure). Then, through the block-based \ac{mve} method described in \cite{BlockBasedMVE2002} two new \acs{mv} are determined: the first one corresponds to the \ac{mv} of the most overlapped block with the current one, and second one is obtained by the weighted average value of the \acs{mv} corresponding to different overlapped blocks. + +The \ac{hmve} method aims to find the more accurate \ac{mv} from the \ac{mv} set composed described above. To achieve this, the \ac{mv} set is further refined using the distance between each pair of \acs{mv}, in order to discard the wrongly extrapolated \acs{mv}. Finally, the pixels of the lost regions (belonging to the groups A and B) are obtained through motion compensation, using the average of the remaining vectors in the \ac{mv} set. The pixels of the group C, although the \ac{mv} set is empty, the same process is applied using the \ac{mv} of the co-located pixel in the previous frame. +This algorithm is able to outperform the \ac{pmve} for different packet loss probabilities, as shown by the results in \cite{2DConcHybridMotion2010}. The method is used to conceal both I (INTRA) and P frames in video sequence compressed with the \ac{h264}, achieving more $1.09$ dB (\ac{psnr}) in the erroneous frames. + +%\HIGHLIGHT{HEVC} +In the case of HEVC, there are also a few works worth to be mentioned. The works presented for the \ac{hevc} standard are mainly based on temporal \ac{ec} approaches. +In~\cite{HEVCMCBlockAug2013} the block partitions of the neighbouring frames were used to improve the performance motion vector extrapolation. Using this the reconstructed frame keeps the boundaries smooth, leading to higher objective and visual quality. +Moreover, in order to select only the reliable motion information, the residue information can be exploited, as shown in~\cite{HEVCMCResidueDec2013}. + +Overall, the aforementioned methods rely on post-processing without any assistance from the encoder-side, which may lead to inaccurate recovery of the lost information. Thus, in order to achieve higher reconstruction quality further information need to be shared with the decoder to minimise the residual distortion of \ac{ec}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Concealment-aware error resilience} +\label{Section22:ConcealmentAware} + +The last category of techniques for robust video transmission address those algorithms that help the \ac{ec} process at the decoder-side. Although such techniques might overlap with concepts described above, such as, data partition or redundant information, they have the main goal of improving the \ac{ec} performance. This group of algorithms combine the error resilience capabilities with \ac{ec} methods to achieve an improved overall performance. + +One of the approaches that can be used to improve the \ac{ec} performance is to estimate, at the encoding time, the \ac{ec} distortion introduced by the decoder in case of data loss. +In \cite{ConcealmentDrivenSliceOrdering2008} the \ac{ec} distortion estimation is used for slice classification and \ac{uep}. Such distortion is also used to optimise motion estimation~\cite{ConcealmentAwareME2006}, by including the distortion of the subsequent frame (assuming it is lost and recovered) in the \ac{rd} cost optimisation of the current frame. This was also used in the context of the \ac{hevc} to improve its error robustness as shown in \cite{ErrorResilienceModeHEVC2016}. This kinda of approaches take advantage of location techniques and data partitioning to improve the reconstruction quality achieved by the \ac{ec} algorithms. Normally, they use a single \ac{ec} method that is implemented both at the encoder and decoder, based on temporal algorithms, e.g., \ac{mc} and \ac{mve}. + +Another approach is the use \ac{ec} motion vectors which are \acs{mv} that might be carried by intra-coded units for the purpose of concealing errors. When an error occurs in a given unit, the concealment motion vector from the neighbouring units can be used to form a prediction from previously received frames. This concept was exploited in \cite{ConcealmentIntraDataHidng2009} by using data-hiding to carry additional motion information in the \ac{dct} coefficients. This technique not only improves the reconstruction quality but also shift the algorithm complexity from the decoder to the encoder-side. + +A different approach to improve the \ac{ec} performance was presented in \cite{PixelInterlacingIntraConcealment2011}. Pixel interlacing is used to divide the frame into different packets in order to enhance the performance of intra-frame \ac{ec} using linear interpolation. This technique reduces the probabilities of burst losses, therefore, the neighbouring pixels surrounding the lost regions are available to be used as reference for the concealment operation. +The above methods facilitate the \ac{ec} performance, either through slice reordering or based on \acs{mv}, they always rely on the assumption that a fixed \ac{ec} method is used at the decoder. +In \cite{ResilienceRECAP_ICASSP2009} a low-resolution video is transmitted alongside the main bitstream in order to be used as reference for concealment. The redundant stream is used by the decoder to improve the estimation accuracy of motion information. This method was extended in \cite{SaliencyErrorConcealment_TM2013} by using asymmetric coding of the redundant information based on Itti's attention model~\cite{SaliencyItti1998}. + +The work presented in \cite{ConcealmentSignalling2008,BurstConcealmentSignalling2008} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder, using \ac{roi} to identify the most sensitive image area. +Although such an approach achieves reasonably good performance, it relies on the spatial neighbouring information, which may not be available in case of full frame loss. Moreover, it requires transmission of the \ac{roi} map to the decoder, which increases the overall bitrate. +More recently a EC signalling methods was proposed for scalable video \cite{SignallingRobustScalable2015}, in order to select the best frame to be used to replace the missing one. + +Summarising, \ac{ec}-aware error techniques allow the combination of efficient algorithms, e.g., redundant pictures or error concealment-aware motion estimation or embedded information, with algorithms at the decoder which take advantages of such extra information. Such techniques are able to achieve quality improvements in comparison with the application of error recovery approaches a the decoder on its own. Although effective techniques have been proposed in the past, the exploration of dynamic techniques which explore the potential of different error concealment techniques with adaptable decoder have not been fully exploited. + + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% Section +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Summary} +\label{Section22:Summary} + +This chapter presented an overview of the different error resilience techniques proposed over the last years, based on they key features. Different approaches were described covering techniques for robust video transmission from dependency reduction at the encoder-side to error concealment by post-processing at the decoder. + +The techniques described in this chapter provide relevant insights for the methods proposed in this work. In order to achieve increase the error robustness of the \ac{hevc} bitstrems and improved reconstruction quality in case of errors the different algorithms presented before are considered and combined. In Chapter~\ref{Chapter:HEVCErrorResilience} localisation techniques are used to decrease motion vector dependencies in \ac{hevc} standard achieving higher quality at cost of small coding efficiency loss. Chapter~\ref{Chapter:ErrorResilienceTechniques} takes advantage of reference picture selection scheme in combination with motion vector redundancies to improve \ac{hevc} streaming. +Chapter~\ref{Chapter:ConcealmentAwareTechniques} proposes an error-concealment aware for increase reconstruction quality. +Finally, Chapter~\ref{Chapter:EndToEndRobustSystem} proposes an end-to-end robust video transmission scheme by combining the error resilience and error concealment based on content, network conditions and application. +% + + + + + + diff --git a/031_ErrorResilienceEvaluation/DynamicTmvp/DynamicTMVP.pdf b/031_ErrorResilienceEvaluation/DynamicTmvp/DynamicTMVP.pdf new file mode 100644 index 0000000..7f0fca8 Binary files /dev/null and b/031_ErrorResilienceEvaluation/DynamicTmvp/DynamicTMVP.pdf differ diff --git a/031_ErrorResilienceEvaluation/DynamicTmvp/DynamicTMVP.tikz b/031_ErrorResilienceEvaluation/DynamicTmvp/DynamicTMVP.tikz new file mode 100644 index 0000000..55f0d9f --- /dev/null +++ b/031_ErrorResilienceEvaluation/DynamicTmvp/DynamicTMVP.tikz @@ -0,0 +1,141 @@ + + +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + +\begin{tikzpicture} + +% \draw[draw=black] (0,0) rectangle ++(21.58cm,27.94cm); + +\newcommand{\scale}{0.8} +% \newcommand{\TikZyslant}{-0.2} +\newcommand{\xslant}{0} +\newcommand{\xframeshift}{6.5cm} +\newcommand{\yframeshift}{0.05cm} + + +\begin{scope}[scale=\scale] + + \tikzstyle{frame_label}=[anchor=center,scale=0.9/\scale,font=\small] + + \tikzstyle{pel}=[rectangle,draw=black, thick] + \tikzstyle{refblock}=[rectangle,draw=black!65, thick] + \tikzstyle{projectedblock}=[rectangle,draw=black, thick,dashed] + \tikzstyle{refmv}=[-latex,very thick,scale=1] + + \newcommand{\DrawFrame}{ + %outside line + \fill[white,fill opacity=.2] (0,0) rectangle (5,5); % Opacity + \draw[black, very thick] (0,0) rectangle (5,5); + + \foreach \x in {0,...,4} + \foreach \y in {0,...,4} + { + \draw[black,loosely dashed] (\x,\y) +(0,0) rectangle ++(1,1); + } + } + + % Frame t + \begin{scope}[ yshift=3*\TikZyfshift, xshift=2*\xframeshift, + style={yslant=\TikZyslant,xscale=\TikZxscale}] + + \DrawFrame + + \node[frame_label,scale=.75] at (2.4,-.4) { (1) }; + \node[frame_label,scale=.75] at (-.5,1.6) { (3) }; + \node[frame_label,scale=.75] at (-.5,3.6) { (2) }; + + \coordinate (TB1) at (1,4); + \draw[black,very thick] (TB1) +(0,0) rectangle ++(1,-1); + + \coordinate (TB2) at (1,2); + \draw[black,very thick] (TB2) +(0,0) rectangle ++(1,-1); + + \coordinate (TB3) at (3,1); + \draw[black,very thick] (TB3) +(0,0) rectangle ++(1,-1); + + \end{scope} + + % Frame t-1 + \begin{scope}[ yshift=2*\TikZyfshift, xshift=1*\xframeshift, + style={yslant=\TikZyslant,xscale=\TikZxscale}] + + \coordinate (T0B1) at (1,4); + \coordinate (T0B2) at (2,4); + \coordinate (T0B3) at (3,4); + \coordinate (T0B4) at (1,3); + \coordinate (T0B5) at (1,2); + \coordinate (T0B6) at (3,1); + + % Draw temporal Dependencies + \coordinate (MVRefT02T1) at (1.5,3.5); + \draw[refmv,dashed] (TB1)++(0.5,-0.5) to (MVRefT02T1); + \coordinate (MVRef1B7) at (3.5,0.5); + \draw[refmv] (TB3)+(0.5,-0.5) to (MVRef1B7); + \coordinate (MVRef1B6) at (1.5,1.5); + \draw[refmv,dashed] (TB2)+(0.5,-0.5) to (MVRef1B6); + + + \DrawFrame + + \draw[black,very thick] (T0B1) +(0,0) rectangle ++(1,-1); + \draw[black,very thick] (T0B2) +(0,0) rectangle ++(1,-1); + \draw[black,very thick] (T0B3) +(0,0) rectangle ++(1,-1); + \draw[black,very thick] (T0B4) +(0,0) rectangle ++(1,-1); + \draw[black,very thick] (T0B5) +(0,0) rectangle ++(1,-1); + \draw[black,very thick] (T0B6) +(0,0) rectangle ++(1,-1); + + % Draw spatial Dependencies + \coordinate (MVRef1B2) at (1.5,3.5); + \draw[refmv] (T0B2)+(0.5,-0.5) to (MVRef1B2); + \coordinate (MVRef1B3) at (2.5,3.5); + \draw[refmv] (T0B3)+(0.5,-0.5) to (MVRef1B3); + \coordinate (MVRef1B4) at (1.5,3.5); + \draw[refmv] (T0B4)+(0.5,-0.5) to (MVRef1B4); + \coordinate (MVRef1B5) at (1.5,2.5); + \draw[refmv] (T0B5)+(0.5,-0.5) to (MVRef1B5); + + \end{scope} + + + % Frame t-2 + \begin{scope}[ yshift=1*\TikZyfshift, xshift=0*\xframeshift, + style={yslant=\TikZyslant,xscale=\TikZxscale}] + + %Define Points + \coordinate (TM1B1) at (1,4); + \coordinate (MVRef1B1) at (1.5,3.5); + \coordinate (TM1B2) at (3,4); + + % Draw Dependencies + \draw[refmv] (T0B1)++(0.5,-0.5) to (MVRef1B1); + + \DrawFrame + + % Draw block + \draw[black,very thick] (TM1B1) rectangle ++(1,-1); + + \end{scope} + + % Frame number + \begin{scope}[xshift=2.5cm,yshift=5.3cm] + \node[frame_label] at (0*\xframeshift,0) { $f_{t-2}$ }; + \node[frame_label] at (1*\xframeshift,0) { $f_{t-1}$ }; + \node[frame_label] at (2*\xframeshift,0) { $f_{t}$ }; + \end{scope} + + + \begin{scope}[xshift=0.5cm,yshift=-1.8cm] + \draw[refmv] (0,0) to (1.2,0); + \fill[black] (1.25,0) node[frame_label, right, scale=.8] {Valid \ac{mv} dependency}; + \end{scope} + + \begin{scope}[xshift=0.5cm,yshift=-2.7cm] + \draw[refmv,dashed] (0,0) to (1.2,0); + \fill[black] (1.25,0) node[frame_label, right, scale=.8] {Disabled \ac{mv} dependency}; + \end{scope} + +\end{scope} + +\end{tikzpicture} diff --git a/031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingle.pdf b/031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingle.pdf new file mode 100644 index 0000000..5c90e8c Binary files /dev/null and b/031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingle.pdf differ diff --git a/031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingle.plt b/031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingle.plt new file mode 100644 index 0000000..5238e4d --- /dev/null +++ b/031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingle.plt @@ -0,0 +1,1214 @@ +set terminal postscript eps monochrome font 'TimesNewRoman,16' + +set grid +set size {0.75,0.75} +set xlabel "Frame Number" font ",18" offset character 0, 0.4, 0 +set xtics center font ",16" # offset character 0, 0, 0 +set ylabel "PSNR [dB]" font ",18" offset character 0.9, 0, 0 +set ytics font ",16" +set key spacing 1 font "TimesNewRoman,16" top right width 3 + +set title font ",20" offset character 0, -0.7, 0 + +set style line 1 lc 4 lt 4 lw 3 pt 10 ps 1 +#set style line 2 lc rgbcolor "#006400" lt 2 lw 3 pt 3 ps 1 +set style line 2 lc 2 lt 2 lw 3 pt 3 ps 1 +set style line 3 lc 3 lt 4 lw 3 pt 4 ps 1 +set style line 4 lc 1 lt 1 lw 3 pt 2 ps 1 +set style line 5 lc 7 lt 3 lw 3 pt 6 ps 1 +set style line 6 lc 8 lt 6 lw 3 pt 8 ps 1 +set style line 7 lc 2 lt 7 lw 3 pt 8 ps 1 +set style line 8 lc 3 lt 8 lw 3 pt 9 ps 1 +set style line 9 lc 4 lt 9 lw 3 pt 10 ps 1 + +set xrange[0:30] +set xtics 0,4,30 +set output 'plot_lowdelay_P_main_seq_kendo_c3_rate_1M000bps.eps' +set title 'lowdelay_P_main Kendo \@ 1M000bps' +plot '-' using 1:2 title 'Ref' w lp ls 1, '-' using 1:2 title 'TIDR' w lp ls 2, '-' using 1:2 title 'Proposed' w lp ls 3 + 0 45.10 45.18 45.33 + 1 43.47 44.82 44.85 + 2 43.16 44.67 44.62 + 3 43.66 44.72 44.62 + 4 29.00 32.88 31.59 + 5 26.01 32.72 31.10 + 6 23.32 32.54 30.54 + 7 22.46 33.11 30.68 + 8 21.85 32.45 29.86 + 9 21.50 32.42 29.64 + 10 21.20 32.55 29.64 + 11 20.95 32.06 29.16 + 12 20.87 31.68 28.82 + 13 20.67 31.62 28.68 + 14 20.83 31.40 28.45 + 15 20.94 31.29 28.28 + 16 20.86 31.45 28.30 + 17 20.82 31.20 28.03 + 18 20.82 30.99 27.82 + 19 20.81 31.00 27.78 + 20 21.12 31.04 27.73 + 21 21.23 30.95 27.64 + 22 20.91 30.72 27.43 + 23 21.02 30.57 27.30 + 24 21.48 30.55 27.26 + 25 21.39 30.33 27.10 + 26 21.35 30.12 26.89 + 27 21.52 29.97 26.78 + 28 21.21 29.71 26.56 + 29 21.25 29.59 26.44 + 30 21.53 29.47 26.31 + 31 21.56 29.39 26.24 + 32 47.07 46.41 47.15 + 33 44.43 45.43 45.70 + 34 43.97 45.20 45.17 + 35 43.80 44.95 44.87 + 36 43.67 44.88 44.50 + 37 43.29 44.69 44.15 + 38 43.19 44.59 43.89 + 39 43.10 44.51 43.72 + 40 43.22 44.53 43.72 + 41 43.29 44.53 43.78 + 42 43.37 44.55 43.82 + 43 43.55 44.60 44.00 + 44 43.69 44.64 44.06 + 45 43.81 44.73 44.20 + 46 43.93 44.77 44.22 + 47 44.00 44.82 44.37 + 48 44.16 44.89 44.46 + 49 44.27 44.93 44.58 + 50 44.34 44.95 44.64 + 51 44.45 44.98 44.68 + 52 44.50 45.01 44.69 + 53 44.53 45.08 44.73 + 54 44.56 45.04 44.85 + 55 44.54 45.06 44.84 + 56 44.57 45.04 44.82 + 57 44.56 45.12 44.91 + 58 44.59 45.12 44.90 + 59 44.63 45.12 44.92 + 60 44.67 45.14 44.96 + 61 44.62 45.16 44.90 + 62 44.60 45.15 44.84 + 63 44.61 45.13 44.90 + 64 47.25 46.53 47.17 + 65 45.17 45.75 46.25 + 66 44.92 45.58 45.94 + 67 44.78 45.53 45.73 + 68 44.76 45.47 45.58 + 69 44.73 45.44 45.43 + 70 44.67 45.41 45.34 + 71 44.61 45.43 45.26 + 72 44.58 45.33 45.18 + 73 44.45 45.25 45.05 + 74 44.35 45.25 44.94 + 75 44.23 45.20 44.88 + 76 44.10 45.10 44.72 + 77 44.00 45.02 44.62 + 78 43.91 44.94 44.54 + 79 43.87 44.93 44.51 + 80 43.82 44.88 44.48 + 81 43.77 44.86 44.46 + 82 43.76 44.82 44.51 + 83 43.87 44.87 44.56 + 84 43.79 44.86 44.49 + 85 43.79 44.83 44.50 + 86 43.79 44.75 44.50 + 87 43.71 44.75 44.48 + 88 43.62 44.65 44.39 + 89 43.58 44.61 44.38 + 90 43.47 44.53 44.29 + 91 43.39 44.48 44.23 + 92 43.31 44.41 44.12 + 93 43.18 44.36 44.00 + 94 43.06 44.26 43.81 + 95 42.91 44.12 43.60 +e + 0 45.10 45.18 45.33 + 1 43.47 44.82 44.85 + 2 43.16 44.67 44.62 + 3 43.66 44.72 44.62 + 4 29.00 32.88 31.59 + 5 26.01 32.72 31.10 + 6 23.32 32.54 30.54 + 7 22.46 33.11 30.68 + 8 22.42 32.60 30.21 + 9 22.17 32.36 29.96 + 10 22.38 32.21 29.79 + 11 22.13 31.88 29.42 + 12 21.99 31.65 29.16 + 13 22.24 31.42 28.91 + 14 22.29 31.28 28.73 + 15 22.12 31.05 28.51 + 16 22.10 30.90 28.39 + 17 22.13 31.17 28.54 + 18 22.00 30.96 28.31 + 19 21.96 30.87 28.21 + 20 22.23 30.80 28.09 + 21 22.33 30.65 27.94 + 22 22.47 31.10 28.12 + 23 22.94 31.02 28.03 + 24 22.97 30.93 27.94 + 25 23.47 30.84 27.87 + 26 23.51 30.74 27.74 + 27 23.88 30.60 27.60 + 28 23.89 30.44 27.41 + 29 23.89 30.21 27.17 + 30 24.00 29.99 26.94 + 31 24.01 29.79 26.73 + 32 47.07 46.41 47.15 + 33 44.45 45.42 45.66 + 34 43.95 45.23 45.11 + 35 43.78 44.98 44.72 + 36 43.69 44.91 44.50 + 37 43.29 44.72 44.11 + 38 43.18 44.60 43.83 + 39 43.11 44.53 43.68 + 40 43.19 44.53 43.70 + 41 43.26 44.52 43.75 + 42 43.33 44.54 43.81 + 43 43.53 44.59 43.93 + 44 43.70 44.66 44.01 + 45 43.79 44.77 44.18 + 46 43.90 44.82 44.26 + 47 44.00 44.89 44.37 + 48 44.15 44.93 44.45 + 49 44.26 44.95 44.53 + 50 44.34 44.95 44.59 + 51 44.44 44.99 44.63 + 52 44.47 45.02 44.70 + 53 44.51 45.08 44.74 + 54 44.57 45.02 44.81 + 55 44.54 45.05 44.76 + 56 44.53 45.06 44.72 + 57 44.56 45.13 44.79 + 58 44.57 45.12 44.82 + 59 44.62 45.12 44.86 + 60 44.64 45.19 44.95 + 61 44.61 45.17 44.88 + 62 44.56 45.16 44.85 + 63 44.57 45.17 44.88 + 64 47.27 46.55 47.17 + 65 45.18 45.77 46.24 + 66 44.92 45.59 45.95 + 67 44.80 45.54 45.73 + 68 44.76 45.49 45.59 + 69 44.74 45.46 45.45 + 70 44.70 45.41 45.37 + 71 44.63 45.45 45.33 + 72 44.56 45.37 45.22 + 73 44.42 45.26 45.09 + 74 44.36 45.28 45.00 + 75 44.24 45.23 44.94 + 76 44.12 45.11 44.79 + 77 43.99 45.03 44.71 + 78 43.89 44.94 44.56 + 79 43.86 44.93 44.50 + 80 43.76 44.89 44.49 + 81 43.73 44.85 44.54 + 82 43.77 44.84 44.54 + 83 43.88 44.88 44.65 + 84 43.76 44.87 44.58 + 85 43.80 44.85 44.57 + 86 43.83 44.80 44.57 + 87 43.77 44.80 44.56 + 88 43.60 44.71 44.46 + 89 43.62 44.61 44.45 + 90 43.50 44.57 44.36 + 91 43.42 44.49 44.26 + 92 43.35 44.41 44.19 + 93 43.24 44.37 44.02 + 94 43.10 44.24 43.82 + 95 42.92 44.09 43.54 +e + 0 45.10 45.18 45.33 + 1 43.47 44.82 44.85 + 2 43.15 44.67 44.61 + 3 43.66 44.73 44.61 + 4 28.65 32.53 31.10 + 5 25.09 31.62 29.92 + 6 23.92 31.37 29.54 + 7 23.52 30.61 28.74 + 8 23.46 30.88 28.88 + 9 23.39 30.72 28.74 + 10 23.19 30.29 28.30 + 11 23.17 30.19 28.06 + 12 22.96 29.64 27.52 + 13 23.09 29.56 27.39 + 14 23.43 29.65 27.36 + 15 23.68 29.53 27.24 + 16 23.59 29.42 27.10 + 17 23.43 29.12 26.83 + 18 23.25 28.70 26.43 + 19 22.99 28.42 26.19 + 20 23.83 28.94 26.52 + 21 24.54 29.28 26.74 + 22 24.77 29.35 26.75 + 23 24.72 29.53 26.87 + 24 24.81 29.54 26.84 + 25 24.78 29.46 26.76 + 26 24.74 29.34 26.60 + 27 24.64 29.08 26.33 + 28 24.57 28.93 26.17 + 29 24.54 28.78 26.01 + 30 24.51 28.88 26.07 + 31 24.48 28.72 25.92 + 32 47.07 46.41 47.15 + 33 44.42 45.43 45.68 + 34 43.94 45.21 45.12 + 35 43.75 44.96 44.82 + 36 43.65 44.93 44.52 + 37 43.26 44.72 44.13 + 38 43.15 44.57 43.82 + 39 43.08 44.54 43.66 + 40 43.18 44.54 43.67 + 41 43.23 44.53 43.81 + 42 43.32 44.58 43.81 + 43 43.48 44.61 43.96 + 44 43.65 44.65 44.01 + 45 43.79 44.73 44.17 + 46 43.89 44.80 44.27 + 47 44.00 44.84 44.38 + 48 44.15 44.91 44.47 + 49 44.27 44.95 44.56 + 50 44.35 44.95 44.56 + 51 44.44 44.96 44.58 + 52 44.45 45.01 44.67 + 53 44.49 45.07 44.73 + 54 44.53 45.02 44.75 + 55 44.55 45.04 44.72 + 56 44.56 45.03 44.68 + 57 44.54 45.09 44.78 + 58 44.55 45.11 44.79 + 59 44.61 45.12 44.80 + 60 44.66 45.17 44.85 + 61 44.59 45.17 44.85 + 62 44.59 45.15 44.79 + 63 44.59 45.15 44.84 + 64 47.25 46.55 47.17 + 65 45.16 45.76 46.23 + 66 44.91 45.57 45.92 + 67 44.77 45.54 45.73 + 68 44.72 45.48 45.61 + 69 44.69 45.45 45.45 + 70 44.67 45.41 45.37 + 71 44.60 45.43 45.27 + 72 44.55 45.31 45.17 + 73 44.43 45.23 45.03 + 74 44.34 45.25 44.96 + 75 44.22 45.21 44.89 + 76 44.13 45.10 44.79 + 77 43.99 45.08 44.70 + 78 43.88 44.97 44.55 + 79 43.83 44.95 44.55 + 80 43.80 44.88 44.54 + 81 43.76 44.86 44.51 + 82 43.78 44.85 44.51 + 83 43.86 44.84 44.55 + 84 43.75 44.83 44.47 + 85 43.77 44.81 44.46 + 86 43.82 44.75 44.48 + 87 43.71 44.80 44.50 + 88 43.59 44.67 44.46 + 89 43.59 44.64 44.41 + 90 43.48 44.55 44.35 + 91 43.39 44.48 44.20 + 92 43.30 44.43 44.13 + 93 43.18 44.31 43.95 + 94 43.04 44.29 43.77 + 95 42.90 44.11 43.53 +e + + +set xrange[0:30] +set xtics 0,4,30 +set output 'plot_lowdelay_P_main_seq_kendo_c3_rate_4M000bps.eps' +set title 'lowdelay_P_main Kendo \@ 4M000bps' +plot '-' using 1:2 title 'Ref' w lp ls 1, '-' using 1:2 title 'TIDR' w lp ls 2, '-' using 1:2 title 'Proposed' w lp ls 3 + 0 47.39 46.46 47.27 + 1 44.87 45.50 46.24 + 2 45.36 45.51 46.17 + 3 46.00 45.72 46.35 + 4 29.78 34.26 32.83 + 5 26.20 35.33 33.34 + 6 23.40 34.28 31.53 + 7 22.77 33.96 30.63 + 8 22.46 33.15 29.67 + 9 21.65 31.63 28.25 + 10 20.97 30.81 27.35 + 11 21.23 30.80 27.26 + 12 20.93 30.70 27.17 + 13 21.19 30.64 27.03 + 14 21.35 30.55 26.92 + 15 21.18 30.43 26.77 + 16 21.32 30.21 26.54 + 17 21.29 30.11 26.41 + 18 21.35 30.05 26.26 + 19 21.23 29.98 26.17 + 20 21.07 29.90 26.16 + 21 21.20 29.75 26.09 + 22 21.21 29.67 26.04 + 23 21.23 30.20 26.53 + 24 21.29 30.11 26.51 + 25 21.27 29.92 26.40 + 26 21.48 29.85 26.36 + 27 21.73 29.67 26.24 + 28 21.64 29.58 26.17 + 29 21.66 29.45 26.04 + 30 21.90 29.40 26.00 + 31 22.13 29.34 25.98 + 32 49.12 47.82 48.53 + 33 46.49 46.21 47.09 + 34 46.54 46.17 47.02 + 35 46.54 46.14 46.90 + 36 46.54 46.18 46.83 + 37 46.43 46.13 46.76 + 38 46.43 46.10 46.72 + 39 46.46 46.13 46.73 + 40 46.45 46.11 46.71 + 41 46.53 46.16 46.71 + 42 46.54 46.15 46.70 + 43 46.65 46.22 46.82 + 44 46.65 46.25 46.80 + 45 46.69 46.27 46.86 + 46 46.74 46.34 46.89 + 47 46.74 46.38 46.94 + 48 46.83 46.40 46.98 + 49 46.89 46.43 47.02 + 50 46.91 46.46 47.01 + 51 46.95 46.42 47.06 + 52 46.92 46.46 47.06 + 53 46.93 46.49 47.05 + 54 46.95 46.41 47.09 + 55 46.95 46.44 47.09 + 56 46.95 46.47 47.03 + 57 46.95 46.47 47.07 + 58 46.92 46.43 47.06 + 59 46.94 46.48 47.08 + 60 46.99 46.52 47.13 + 61 46.97 46.50 47.09 + 62 46.87 46.51 47.06 + 63 46.96 46.51 47.08 + 64 49.27 47.96 48.61 + 65 46.79 46.43 47.24 + 66 46.87 46.41 47.18 + 67 46.92 46.46 47.14 + 68 46.91 46.47 47.16 + 69 46.94 46.49 47.09 + 70 46.92 46.50 47.13 + 71 46.87 46.47 47.09 + 72 46.86 46.45 47.08 + 73 46.79 46.41 47.02 + 74 46.77 46.45 47.03 + 75 46.75 46.39 47.00 + 76 46.70 46.36 46.94 + 77 46.69 46.34 46.98 + 78 46.62 46.25 46.92 + 79 46.65 46.28 46.98 + 80 46.59 46.22 46.96 + 81 46.58 46.19 46.91 + 82 46.61 46.15 46.93 + 83 46.60 46.19 46.98 + 84 46.52 46.15 46.97 + 85 46.51 46.09 46.88 + 86 46.50 46.07 46.89 + 87 46.45 46.01 46.85 + 88 46.37 45.96 46.82 + 89 46.39 45.94 46.79 + 90 46.32 45.90 46.80 + 91 46.29 45.88 46.79 + 92 46.29 45.84 46.78 + 93 46.25 45.82 46.67 + 94 46.23 45.78 46.64 + 95 46.16 45.76 46.60 +e + 0 47.39 46.46 47.27 + 1 44.87 45.50 46.24 + 2 45.36 45.51 46.17 + 3 46.00 45.72 46.35 + 4 29.78 34.26 32.83 + 5 26.20 35.33 33.34 + 6 23.40 34.28 31.53 + 7 22.77 33.96 30.63 + 8 22.85 33.23 30.08 + 9 22.38 33.14 30.03 + 10 22.74 32.84 29.76 + 11 22.91 32.68 29.59 + 12 22.98 31.22 28.45 + 13 23.35 31.02 28.24 + 14 23.51 30.96 28.20 + 15 23.27 30.75 28.03 + 16 23.41 30.63 27.92 + 17 23.51 30.54 27.80 + 18 23.73 30.42 27.56 + 19 23.85 30.30 27.38 + 20 23.96 30.21 27.36 + 21 23.80 29.95 27.17 + 22 23.78 29.92 27.10 + 23 23.85 29.94 27.13 + 24 23.85 29.94 27.13 + 25 23.79 29.73 26.95 + 26 23.82 29.70 26.90 + 27 23.88 29.56 26.81 + 28 23.93 29.43 26.71 + 29 24.28 29.41 26.72 + 30 24.16 28.92 26.35 + 31 23.60 28.93 26.40 + 32 49.14 47.82 48.53 + 33 46.50 46.22 47.07 + 34 46.53 46.19 47.01 + 35 46.54 46.16 46.92 + 36 46.53 46.19 46.83 + 37 46.43 46.13 46.74 + 38 46.41 46.13 46.74 + 39 46.47 46.11 46.71 + 40 46.43 46.11 46.73 + 41 46.50 46.15 46.71 + 42 46.53 46.17 46.71 + 43 46.64 46.23 46.82 + 44 46.67 46.23 46.80 + 45 46.70 46.27 46.85 + 46 46.73 46.31 46.89 + 47 46.75 46.36 46.94 + 48 46.82 46.40 46.98 + 49 46.87 46.44 47.00 + 50 46.93 46.41 47.01 + 51 46.94 46.41 47.03 + 52 46.93 46.45 47.05 + 53 46.94 46.49 47.08 + 54 46.93 46.44 47.10 + 55 46.95 46.49 47.06 + 56 46.95 46.44 47.05 + 57 46.97 46.49 47.09 + 58 46.95 46.43 47.03 + 59 46.94 46.48 47.07 + 60 47.01 46.52 47.11 + 61 46.98 46.53 47.09 + 62 46.90 46.49 47.09 + 63 46.97 46.48 47.09 + 64 49.29 47.96 48.60 + 65 46.79 46.44 47.23 + 66 46.87 46.41 47.15 + 67 46.93 46.46 47.12 + 68 46.92 46.46 47.14 + 69 46.94 46.46 47.07 + 70 46.92 46.49 47.14 + 71 46.86 46.48 47.10 + 72 46.86 46.44 47.07 + 73 46.77 46.41 47.04 + 74 46.77 46.43 46.98 + 75 46.74 46.44 46.96 + 76 46.70 46.35 46.92 + 77 46.71 46.35 46.97 + 78 46.61 46.27 46.94 + 79 46.66 46.29 46.97 + 80 46.60 46.21 46.90 + 81 46.57 46.18 46.90 + 82 46.60 46.17 46.94 + 83 46.63 46.16 46.92 + 84 46.53 46.15 46.92 + 85 46.53 46.08 46.88 + 86 46.50 46.06 46.89 + 87 46.46 46.02 46.84 + 88 46.38 45.95 46.80 + 89 46.40 45.93 46.79 + 90 46.31 45.91 46.81 + 91 46.29 45.89 46.77 + 92 46.30 45.86 46.76 + 93 46.27 45.82 46.65 + 94 46.23 45.80 46.62 + 95 46.17 45.78 46.57 +e + 0 47.39 46.46 47.27 + 1 44.87 45.50 46.24 + 2 45.36 45.50 46.17 + 3 45.99 45.74 46.36 + 4 29.20 33.96 32.69 + 5 27.11 35.65 33.87 + 6 26.34 36.53 34.36 + 7 27.06 36.94 34.55 + 8 26.62 35.33 33.14 + 9 26.41 34.90 32.75 + 10 26.60 35.22 32.87 + 11 26.55 35.05 32.51 + 12 27.20 34.93 32.37 + 13 27.27 34.74 32.13 + 14 27.46 34.39 31.77 + 15 27.49 34.74 32.16 + 16 27.99 34.70 32.24 + 17 28.06 34.59 32.04 + 18 28.34 34.38 31.94 + 19 28.46 34.49 32.04 + 20 28.55 34.25 31.82 + 21 28.39 33.95 31.59 + 22 28.56 33.89 31.57 + 23 28.75 33.81 31.53 + 24 28.85 33.72 31.48 + 25 28.95 33.60 31.40 + 26 29.12 33.44 31.24 + 27 29.24 33.36 31.18 + 28 29.21 33.18 31.04 + 29 29.28 33.10 30.98 + 30 29.33 33.10 30.95 + 31 29.33 33.00 30.91 + 32 49.13 47.83 48.55 + 33 46.49 46.22 47.09 + 34 46.55 46.16 47.01 + 35 46.52 46.15 46.96 + 36 46.52 46.17 46.83 + 37 46.40 46.13 46.77 + 38 46.43 46.10 46.71 + 39 46.46 46.12 46.71 + 40 46.45 46.10 46.69 + 41 46.51 46.15 46.72 + 42 46.53 46.16 46.76 + 43 46.64 46.21 46.82 + 44 46.66 46.24 46.83 + 45 46.67 46.27 46.87 + 46 46.73 46.32 46.87 + 47 46.76 46.37 46.98 + 48 46.82 46.42 47.02 + 49 46.88 46.42 47.04 + 50 46.91 46.42 47.04 + 51 46.93 46.40 47.05 + 52 46.91 46.46 47.08 + 53 46.92 46.46 47.08 + 54 46.95 46.41 47.08 + 55 46.97 46.45 47.05 + 56 46.94 46.46 47.05 + 57 46.95 46.46 47.09 + 58 46.92 46.45 47.07 + 59 46.94 46.49 47.08 + 60 47.01 46.49 47.10 + 61 46.97 46.50 47.10 + 62 46.88 46.47 47.07 + 63 46.97 46.49 47.13 + 64 49.29 47.99 48.61 + 65 46.79 46.43 47.25 + 66 46.86 46.43 47.18 + 67 46.92 46.46 47.13 + 68 46.91 46.46 47.16 + 69 46.93 46.46 47.14 + 70 46.91 46.50 47.15 + 71 46.85 46.47 47.10 + 72 46.86 46.44 47.14 + 73 46.78 46.41 47.06 + 74 46.76 46.45 47.04 + 75 46.74 46.39 46.99 + 76 46.69 46.35 46.97 + 77 46.71 46.31 46.98 + 78 46.61 46.25 46.93 + 79 46.66 46.26 46.96 + 80 46.57 46.22 46.93 + 81 46.57 46.18 46.91 + 82 46.61 46.15 46.95 + 83 46.59 46.17 46.98 + 84 46.54 46.15 46.96 + 85 46.51 46.07 46.93 + 86 46.51 46.03 46.91 + 87 46.46 46.01 46.87 + 88 46.38 45.97 46.82 + 89 46.37 45.90 46.83 + 90 46.32 45.90 46.80 + 91 46.26 45.88 46.79 + 92 46.29 45.85 46.79 + 93 46.25 45.82 46.66 + 94 46.22 45.82 46.65 + 95 46.17 45.75 46.57 +e + + +set xrange[0:30] +set xtics 0,4,30 +set output 'plot_lowdelay_P_main_seq_bqsquare_rate_1M000bps.eps' +set title 'lowdelay_P_main BQSquare \@ 1M000bps' +plot '-' using 1:2 title 'Ref' w lp ls 1, '-' using 1:2 title 'TIDR' w lp ls 2, '-' using 1:2 title 'Proposed' w lp ls 3 + 0 39.53 42.26 42.80 + 1 35.83 42.08 42.70 + 2 35.16 41.99 42.67 + 3 34.61 42.02 42.59 + 4 29.62 39.04 38.96 + 5 26.90 39.01 38.95 + 6 24.50 39.01 38.97 + 7 23.19 38.75 38.67 + 8 22.16 38.75 38.71 + 9 21.21 38.63 38.51 + 10 20.49 38.44 38.37 + 11 20.22 38.35 38.29 + 12 19.84 38.32 38.18 + 13 19.57 38.16 37.95 + 14 19.09 38.05 37.82 + 15 18.84 37.90 37.65 + 16 18.59 37.75 37.42 + 17 18.36 37.70 37.44 + 18 18.33 37.57 37.29 + 19 18.25 37.46 37.12 + 20 18.14 37.32 36.94 + 21 18.12 37.16 36.78 + 22 18.13 36.99 36.62 + 23 18.13 36.92 36.46 + 24 18.15 36.72 36.26 + 25 18.15 36.54 36.08 + 26 18.17 36.39 35.90 + 27 18.15 36.26 35.74 + 28 18.14 36.13 35.55 + 29 18.16 35.99 35.44 + 30 18.19 35.89 35.27 + 31 18.19 35.75 35.10 + 32 39.01 41.92 42.60 + 33 34.91 41.74 42.49 + 34 34.42 41.71 42.48 + 35 33.76 41.65 42.42 + 36 33.14 41.62 42.30 + 37 32.81 41.51 42.22 + 38 32.54 41.37 42.08 + 39 32.35 41.30 41.97 + 40 32.27 41.16 41.90 + 41 32.24 41.09 41.82 + 42 32.06 40.96 41.76 + 43 32.00 40.87 41.65 + 44 32.00 40.77 41.58 + 45 31.92 40.62 41.48 + 46 31.97 40.54 41.37 + 47 31.90 40.41 41.28 + 48 31.78 40.31 41.14 + 49 31.84 40.16 41.10 + 50 31.80 40.06 40.99 + 51 31.70 39.97 40.92 + 52 31.66 39.84 40.82 + 53 31.59 39.75 40.68 + 54 31.57 39.61 40.56 + 55 31.59 39.55 40.46 + 56 31.46 39.40 40.36 + 57 31.44 39.32 40.26 + 58 31.46 39.27 40.20 + 59 31.45 39.19 40.11 + 60 31.53 39.08 40.01 + 61 31.52 39.01 39.99 + 62 31.43 38.90 39.89 + 63 31.42 38.85 39.85 + 64 39.12 41.70 42.47 + 65 34.50 41.55 42.37 + 66 33.57 41.51 42.34 + 67 32.92 41.52 42.31 + 68 32.53 41.39 42.23 + 69 32.15 41.32 42.14 + 70 31.94 41.24 42.10 + 71 31.78 41.17 42.06 + 72 31.60 41.11 41.95 + 73 31.49 40.96 41.87 + 74 31.49 40.87 41.71 + 75 31.33 40.74 41.64 + 76 31.22 40.59 41.53 + 77 31.29 40.51 41.43 + 78 31.22 40.43 41.32 + 79 31.20 40.30 41.23 + 80 31.24 40.14 41.12 + 81 31.19 40.04 41.00 + 82 31.19 39.91 40.92 + 83 31.28 39.79 40.83 + 84 31.18 39.71 40.71 + 85 31.20 39.58 40.60 + 86 31.19 39.54 40.44 + 87 31.10 39.37 40.37 + 88 31.11 39.28 40.33 + 89 31.16 39.20 40.22 + 90 31.11 39.07 40.14 + 91 31.19 39.03 40.04 + 92 31.22 38.99 39.96 + 93 31.12 38.87 39.88 + 94 31.20 38.80 39.77 + 95 31.19 38.75 39.69 +e + 0 39.53 42.26 42.80 + 1 35.83 42.08 42.70 + 2 35.16 41.99 42.67 + 3 34.61 42.02 42.59 + 4 29.62 39.04 38.96 + 5 26.90 39.01 38.95 + 6 24.50 39.01 38.97 + 7 23.19 38.75 38.67 + 8 23.32 38.47 38.43 + 9 23.43 38.03 37.84 + 10 23.42 37.62 37.42 + 11 23.53 37.37 37.12 + 12 23.58 37.11 36.81 + 13 23.57 36.90 36.58 + 14 23.58 36.49 36.19 + 15 23.66 36.55 36.20 + 16 23.66 36.36 35.99 + 17 23.71 36.17 35.83 + 18 23.70 36.02 35.65 + 19 23.76 35.87 35.54 + 20 23.78 35.66 35.37 + 21 23.76 35.43 35.15 + 22 23.79 35.28 34.96 + 23 23.87 35.07 34.70 + 24 23.93 34.95 34.54 + 25 23.94 34.81 34.43 + 26 23.92 34.65 34.27 + 27 23.95 34.47 34.06 + 28 23.89 34.36 33.91 + 29 23.89 34.22 33.79 + 30 23.86 34.07 33.61 + 31 23.83 34.03 33.42 + 32 39.05 41.89 42.54 + 33 34.93 41.70 42.45 + 34 34.40 41.67 42.41 + 35 33.78 41.64 42.34 + 36 33.15 41.56 42.22 + 37 32.82 41.42 42.15 + 38 32.52 41.30 42.07 + 39 32.38 41.22 41.98 + 40 32.21 41.10 41.87 + 41 32.25 41.00 41.81 + 42 32.10 40.90 41.78 + 43 32.04 40.83 41.63 + 44 32.06 40.71 41.59 + 45 31.97 40.61 41.50 + 46 32.00 40.58 41.38 + 47 31.94 40.46 41.31 + 48 31.77 40.36 41.19 + 49 31.88 40.22 41.15 + 50 31.80 40.10 41.00 + 51 31.73 39.98 40.95 + 52 31.68 39.86 40.80 + 53 31.60 39.75 40.69 + 54 31.50 39.64 40.57 + 55 31.50 39.60 40.48 + 56 31.47 39.49 40.35 + 57 31.50 39.35 40.29 + 58 31.48 39.30 40.21 + 59 31.43 39.21 40.11 + 60 31.46 39.15 40.08 + 61 31.42 39.07 40.06 + 62 31.39 38.97 39.96 + 63 31.43 38.88 39.89 + 64 39.12 41.70 42.47 + 65 34.44 41.55 42.37 + 66 33.61 41.51 42.33 + 67 32.93 41.52 42.30 + 68 32.51 41.38 42.24 + 69 32.12 41.32 42.14 + 70 31.95 41.23 42.11 + 71 31.81 41.18 42.07 + 72 31.63 41.09 41.95 + 73 31.61 40.94 41.89 + 74 31.53 40.84 41.75 + 75 31.43 40.73 41.65 + 76 31.36 40.58 41.55 + 77 31.39 40.48 41.43 + 78 31.34 40.37 41.32 + 79 31.29 40.22 41.22 + 80 31.28 40.09 41.13 + 81 31.18 40.00 40.99 + 82 31.17 39.88 40.91 + 83 31.25 39.83 40.83 + 84 31.18 39.72 40.69 + 85 31.12 39.62 40.61 + 86 31.17 39.55 40.44 + 87 31.12 39.38 40.33 + 88 31.09 39.30 40.33 + 89 31.11 39.20 40.22 + 90 31.13 39.06 40.12 + 91 31.13 39.05 40.07 + 92 31.15 38.97 39.98 + 93 31.14 38.87 39.89 + 94 31.15 38.80 39.79 + 95 31.20 38.76 39.73 +e + 0 39.53 42.26 42.80 + 1 35.83 42.08 42.70 + 2 35.16 41.99 42.67 + 3 34.60 42.02 42.58 + 4 29.76 38.82 38.71 + 5 27.28 38.75 38.65 + 6 26.52 38.65 38.51 + 7 26.27 38.36 38.23 + 8 26.09 38.21 38.10 + 9 25.98 37.84 37.69 + 10 26.06 37.56 37.41 + 11 26.00 37.25 37.06 + 12 25.89 37.10 36.89 + 13 25.84 36.85 36.60 + 14 25.94 36.39 36.15 + 15 25.94 36.14 35.83 + 16 26.06 35.82 35.48 + 17 25.99 35.39 35.06 + 18 26.00 35.30 35.00 + 19 26.10 35.19 34.87 + 20 26.05 35.04 34.68 + 21 25.98 34.87 34.50 + 22 25.97 34.71 34.30 + 23 25.82 34.54 34.15 + 24 25.81 34.46 34.00 + 25 25.78 34.32 33.85 + 26 25.76 34.15 33.68 + 27 25.77 34.03 33.63 + 28 25.74 33.88 33.42 + 29 25.73 33.73 33.24 + 30 25.66 33.60 33.07 + 31 25.67 33.51 32.94 + 32 39.00 41.90 42.60 + 33 34.87 41.75 42.50 + 34 34.38 41.71 42.51 + 35 33.75 41.68 42.42 + 36 33.17 41.62 42.29 + 37 32.79 41.49 42.24 + 38 32.52 41.36 42.11 + 39 32.29 41.28 41.98 + 40 32.19 41.18 41.91 + 41 32.13 41.10 41.87 + 42 32.01 40.99 41.80 + 43 32.02 40.94 41.69 + 44 31.99 40.84 41.65 + 45 31.85 40.71 41.54 + 46 31.95 40.64 41.42 + 47 31.87 40.51 41.31 + 48 31.72 40.39 41.18 + 49 31.74 40.31 41.15 + 50 31.73 40.21 41.01 + 51 31.68 40.10 40.95 + 52 31.63 39.98 40.86 + 53 31.62 39.87 40.71 + 54 31.49 39.69 40.58 + 55 31.51 39.66 40.50 + 56 31.50 39.53 40.39 + 57 31.45 39.44 40.33 + 58 31.40 39.33 40.25 + 59 31.32 39.26 40.15 + 60 31.36 39.17 40.06 + 61 31.32 39.08 40.01 + 62 31.28 39.00 39.93 + 63 31.31 38.92 39.85 + 64 39.13 41.65 42.42 + 65 34.44 41.53 42.34 + 66 33.55 41.51 42.31 + 67 32.90 41.50 42.27 + 68 32.48 41.38 42.22 + 69 32.13 41.29 42.09 + 70 31.94 41.23 42.03 + 71 31.80 41.19 42.00 + 72 31.63 41.14 41.88 + 73 31.51 40.99 41.81 + 74 31.46 40.87 41.64 + 75 31.34 40.74 41.58 + 76 31.29 40.61 41.47 + 77 31.29 40.48 41.36 + 78 31.24 40.42 41.24 + 79 31.20 40.29 41.13 + 80 31.22 40.16 41.01 + 81 31.25 40.09 40.90 + 82 31.28 39.99 40.77 + 83 31.27 39.91 40.71 + 84 31.19 39.77 40.57 + 85 31.19 39.64 40.46 + 86 31.28 39.56 40.32 + 87 31.21 39.43 40.23 + 88 31.23 39.34 40.20 + 89 31.19 39.20 40.10 + 90 31.18 39.05 40.03 + 91 31.18 39.00 39.94 + 92 31.15 38.96 39.86 + 93 31.10 38.83 39.77 + 94 31.17 38.75 39.67 + 95 31.15 38.76 39.62 +e + + +set xrange[0:30] +set xtics 0,4,30 +set output 'plot_lowdelay_P_main_seq_bqsquare_rate_4M000bps.eps' +set title 'lowdelay_P_main BQSquare \@ 4M000bps' +plot '-' using 1:2 title 'Ref' w lp ls 1, '-' using 1:2 title 'TIDR' w lp ls 2, '-' using 1:2 title 'Proposed' w lp ls 3 + 0 44.97 46.16 46.50 + 1 41.14 45.12 45.87 + 2 40.87 44.98 45.74 + 3 40.59 44.81 45.48 + 4 31.31 39.20 39.10 + 5 28.14 39.17 39.14 + 6 25.40 38.85 38.78 + 7 23.40 38.61 38.58 + 8 21.55 38.44 38.36 + 9 20.31 38.40 38.26 + 10 19.51 38.08 37.91 + 11 18.76 37.80 37.59 + 12 18.20 37.73 37.43 + 13 17.87 37.61 37.20 + 14 17.50 37.55 36.99 + 15 17.22 37.35 36.68 + 16 17.10 37.27 36.44 + 17 16.98 37.11 36.08 + 18 16.82 36.97 35.86 + 19 16.64 36.83 35.62 + 20 16.40 36.66 35.36 + 21 16.24 36.56 35.21 + 22 16.11 36.47 35.04 + 23 15.95 36.35 34.85 + 24 15.80 36.20 34.62 + 25 15.72 36.08 34.41 + 26 15.59 35.87 34.25 + 27 15.44 35.69 34.03 + 28 15.32 35.53 33.84 + 29 15.23 35.30 33.68 + 30 15.06 35.10 33.47 + 31 14.76 34.90 33.31 + 32 44.25 45.47 45.75 + 33 40.82 44.60 45.26 + 34 40.24 44.47 45.10 + 35 39.91 44.25 44.88 + 36 39.70 44.04 44.70 + 37 39.54 43.77 44.51 + 38 39.35 43.64 44.33 + 39 39.24 43.42 44.17 + 40 39.29 43.20 43.94 + 41 39.35 43.11 43.89 + 42 39.19 43.01 43.80 + 43 39.33 42.91 43.64 + 44 39.26 42.83 43.57 + 45 39.06 42.66 43.50 + 46 39.20 42.65 43.37 + 47 39.22 42.55 43.29 + 48 39.06 42.46 43.19 + 49 39.17 42.39 43.14 + 50 39.07 42.27 43.04 + 51 39.03 42.32 42.97 + 52 39.04 42.24 42.88 + 53 39.00 42.19 42.79 + 54 38.96 42.17 42.81 + 55 38.98 42.14 42.74 + 56 38.91 42.10 42.72 + 57 38.95 42.07 42.69 + 58 38.83 42.07 42.63 + 59 38.82 42.00 42.58 + 60 38.89 41.91 42.53 + 61 38.89 41.95 42.59 + 62 38.89 41.89 42.46 + 63 38.90 41.85 42.48 + 64 43.84 45.09 45.53 + 65 40.22 44.26 45.01 + 66 39.62 44.07 44.83 + 67 39.35 43.94 44.67 + 68 39.17 43.60 44.46 + 69 38.94 43.38 44.19 + 70 38.95 43.19 43.99 + 71 38.88 43.05 43.86 + 72 38.82 42.93 43.63 + 73 38.85 42.77 43.52 + 74 38.82 42.62 43.36 + 75 38.69 42.47 43.20 + 76 38.68 42.27 43.04 + 77 38.69 42.19 42.92 + 78 38.59 42.10 42.79 + 79 38.59 42.00 42.75 + 80 38.66 41.93 42.73 + 81 38.58 41.90 42.68 + 82 38.66 41.85 42.58 + 83 38.73 41.83 42.57 + 84 38.59 41.81 42.46 + 85 38.55 41.76 42.52 + 86 38.62 41.79 42.46 + 87 38.53 41.69 42.48 + 88 38.51 41.69 42.44 + 89 38.54 41.69 42.34 + 90 38.55 41.63 42.31 + 91 38.53 41.74 42.32 + 92 38.48 41.74 42.28 + 93 38.47 41.62 42.20 + 94 38.49 41.66 42.21 + 95 38.51 41.70 42.24 +e + 0 44.97 46.16 46.50 + 1 41.14 45.12 45.87 + 2 40.87 44.98 45.74 + 3 40.59 44.81 45.48 + 4 31.31 39.20 39.10 + 5 28.14 39.17 39.14 + 6 25.40 38.85 38.78 + 7 23.40 38.61 38.58 + 8 23.51 38.38 38.36 + 9 23.70 38.25 38.18 + 10 23.81 37.81 37.70 + 11 23.92 37.49 37.37 + 12 23.96 37.16 37.02 + 13 24.04 36.76 36.64 + 14 24.05 36.45 36.34 + 15 24.12 36.16 35.96 + 16 24.21 35.91 35.64 + 17 24.26 35.73 35.48 + 18 24.29 35.53 35.24 + 19 24.34 35.34 35.02 + 20 24.40 35.11 34.78 + 21 24.41 34.91 34.60 + 22 24.44 34.75 34.41 + 23 24.47 34.52 34.17 + 24 24.48 34.37 33.99 + 25 24.47 34.15 33.80 + 26 24.46 33.96 33.67 + 27 24.46 33.83 33.53 + 28 24.47 33.67 33.36 + 29 24.42 33.50 33.19 + 30 24.40 33.30 33.01 + 31 24.41 33.19 32.85 + 32 44.25 45.47 45.75 + 33 40.80 44.62 45.26 + 34 40.29 44.48 45.09 + 35 39.94 44.25 44.89 + 36 39.69 44.05 44.71 + 37 39.55 43.75 44.50 + 38 39.36 43.61 44.30 + 39 39.24 43.39 44.17 + 40 39.26 43.24 43.95 + 41 39.46 43.15 43.87 + 42 39.20 43.05 43.77 + 43 39.33 42.96 43.63 + 44 39.33 42.82 43.59 + 45 39.14 42.74 43.48 + 46 39.22 42.72 43.38 + 47 39.16 42.65 43.29 + 48 39.03 42.54 43.13 + 49 39.17 42.46 43.09 + 50 39.11 42.39 42.96 + 51 39.05 42.35 42.92 + 52 39.01 42.31 42.94 + 53 38.95 42.27 42.86 + 54 38.94 42.20 42.79 + 55 38.94 42.18 42.72 + 56 38.89 42.10 42.69 + 57 38.94 42.03 42.57 + 58 38.82 42.01 42.63 + 59 38.80 41.92 42.55 + 60 38.88 41.93 42.55 + 61 38.86 41.90 42.60 + 62 38.84 41.86 42.49 + 63 38.87 41.89 42.53 + 64 43.84 45.09 45.53 + 65 40.19 44.27 45.01 + 66 39.63 44.09 44.81 + 67 39.35 43.94 44.66 + 68 39.17 43.61 44.45 + 69 38.93 43.38 44.15 + 70 38.92 43.20 43.93 + 71 38.87 43.07 43.81 + 72 38.75 42.94 43.66 + 73 38.86 42.72 43.52 + 74 38.79 42.58 43.32 + 75 38.70 42.47 43.20 + 76 38.71 42.30 43.03 + 77 38.71 42.17 42.89 + 78 38.61 42.13 42.82 + 79 38.59 42.00 42.82 + 80 38.64 41.98 42.78 + 81 38.60 41.90 42.71 + 82 38.68 41.85 42.61 + 83 38.71 41.81 42.59 + 84 38.58 41.78 42.47 + 85 38.53 41.79 42.53 + 86 38.60 41.80 42.43 + 87 38.56 41.71 42.44 + 88 38.56 41.68 42.38 + 89 38.50 41.73 42.37 + 90 38.57 41.65 42.35 + 91 38.53 41.69 42.33 + 92 38.49 41.66 42.38 + 93 38.45 41.59 42.29 + 94 38.55 41.60 42.25 + 95 38.53 41.68 42.28 +e + 0 44.97 46.16 46.50 + 1 41.14 45.12 45.87 + 2 40.87 44.98 45.74 + 3 40.60 44.81 45.48 + 4 31.38 39.48 39.30 + 5 29.79 39.35 39.14 + 6 28.19 39.07 38.91 + 7 26.78 38.70 38.46 + 8 25.26 38.33 38.08 + 9 23.60 38.02 37.66 + 10 22.18 37.80 37.34 + 11 21.43 37.57 37.05 + 12 20.96 37.37 36.78 + 13 20.39 37.14 36.41 + 14 19.65 36.79 36.01 + 15 19.43 36.68 35.82 + 16 19.36 36.58 35.68 + 17 19.36 36.43 35.51 + 18 19.45 36.19 35.27 + 19 19.03 36.07 35.04 + 20 18.93 35.85 34.76 + 21 18.91 35.68 34.52 + 22 18.86 35.54 34.39 + 23 18.93 35.37 34.27 + 24 19.02 35.21 34.14 + 25 19.11 35.07 34.01 + 26 19.15 34.83 33.87 + 27 19.20 34.59 33.69 + 28 19.26 34.40 33.50 + 29 19.32 34.22 33.37 + 30 19.35 34.03 33.20 + 31 19.40 33.89 33.06 + 32 44.25 45.47 45.75 + 33 40.83 44.61 45.26 + 34 40.24 44.48 45.11 + 35 39.91 44.26 44.92 + 36 39.67 44.08 44.70 + 37 39.49 43.78 44.51 + 38 39.32 43.64 44.32 + 39 39.26 43.44 44.16 + 40 39.26 43.25 43.97 + 41 39.43 43.14 43.93 + 42 39.17 43.06 43.83 + 43 39.30 42.94 43.71 + 44 39.31 42.84 43.63 + 45 39.09 42.71 43.50 + 46 39.21 42.66 43.41 + 47 39.18 42.63 43.36 + 48 39.04 42.51 43.21 + 49 39.18 42.42 43.19 + 50 39.11 42.38 43.09 + 51 39.05 42.31 43.00 + 52 39.04 42.29 42.99 + 53 39.00 42.23 42.87 + 54 38.97 42.18 42.76 + 55 38.89 42.22 42.74 + 56 38.87 42.15 42.70 + 57 38.89 42.01 42.63 + 58 38.79 41.99 42.63 + 59 38.79 41.94 42.57 + 60 38.84 41.86 42.56 + 61 38.87 41.85 42.55 + 62 38.78 41.84 42.47 + 63 38.85 41.86 42.48 + 64 43.80 45.09 45.47 + 65 40.14 44.27 45.02 + 66 39.60 44.07 44.84 + 67 39.31 43.94 44.64 + 68 39.12 43.63 44.43 + 69 38.92 43.34 44.10 + 70 38.92 43.19 43.89 + 71 38.83 43.04 43.74 + 72 38.73 42.91 43.56 + 73 38.80 42.70 43.44 + 74 38.80 42.57 43.28 + 75 38.64 42.47 43.21 + 76 38.65 42.25 43.04 + 77 38.68 42.18 42.94 + 78 38.54 42.09 42.82 + 79 38.55 42.05 42.80 + 80 38.65 41.94 42.73 + 81 38.56 41.88 42.69 + 82 38.62 41.82 42.62 + 83 38.68 41.80 42.60 + 84 38.55 41.82 42.57 + 85 38.50 41.75 42.57 + 86 38.55 41.75 42.47 + 87 38.47 41.67 42.49 + 88 38.48 41.65 42.44 + 89 38.49 41.61 42.34 + 90 38.52 41.60 42.32 + 91 38.50 41.65 42.31 + 92 38.47 41.68 42.31 + 93 38.40 41.67 42.32 + 94 38.50 41.65 42.24 + 95 38.51 41.71 42.19 +e + diff --git a/031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingle.tex b/031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingle.tex new file mode 100644 index 0000000..d616c07 --- /dev/null +++ b/031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingle.tex @@ -0,0 +1,273 @@ +\newcommand{\ft}[0]{\footnotesize} +\newcommand{\pltw}{0.6\columnwidth} +\newcommand{\plth}{0.5\textheight} +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +set multiplot layout 2,1 +set xlabel '\TitleFont Frame Number' +set key top right Right +set key width -8 + +set xrange[0:31] + +set title '\TitleFont Kendo' +plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \TIDR\cite{DynamicTMVP2011}' w lp ls 2, '-' using 1:2 title '\KeyFont \PROP' w lp ls 3 + 0 47.39 46.46 47.27 + 1 44.87 45.50 46.24 + 2 45.36 45.51 46.17 + 3 46.00 45.72 46.35 + 4 29.78 34.26 32.83 + 5 26.20 35.33 33.34 + 6 23.40 34.28 31.53 + 7 22.77 33.96 30.63 + 8 22.46 33.15 29.67 + 9 21.65 31.63 28.25 + 10 20.97 30.81 27.35 + 11 21.23 30.80 27.26 + 12 20.93 30.70 27.17 + 13 21.19 30.64 27.03 + 14 21.35 30.55 26.92 + 15 21.18 30.43 26.77 + 16 21.32 30.21 26.54 + 17 21.29 30.11 26.41 + 18 21.35 30.05 26.26 + 19 21.23 29.98 26.17 + 20 21.07 29.90 26.16 + 21 21.20 29.75 26.09 + 22 21.21 29.67 26.04 + 23 21.23 30.20 26.53 + 24 21.29 30.11 26.51 + 25 21.27 29.92 26.40 + 26 21.48 29.85 26.36 + 27 21.73 29.67 26.24 + 28 21.64 29.58 26.17 + 29 21.66 29.45 26.04 + 30 21.90 29.40 26.00 + 31 22.13 29.34 25.98 + 32 49.12 47.82 48.53 + 33 46.49 46.21 47.09 + 34 46.54 46.17 47.02 + 35 46.54 46.14 46.90 + 36 46.54 46.18 46.83 + 37 46.43 46.13 46.76 + 38 46.43 46.10 46.72 + 39 46.46 46.13 46.73 + 40 46.45 46.11 46.71 +e + 0 47.39 46.46 47.27 + 1 44.87 45.50 46.24 + 2 45.36 45.51 46.17 + 3 46.00 45.72 46.35 + 4 29.78 34.26 32.83 + 5 26.20 35.33 33.34 + 6 23.40 34.28 31.53 + 7 22.77 33.96 30.63 + 8 22.85 33.23 30.08 + 9 22.38 33.14 30.03 + 10 22.74 32.84 29.76 + 11 22.91 32.68 29.59 + 12 22.98 31.22 28.45 + 13 23.35 31.02 28.24 + 14 23.51 30.96 28.20 + 15 23.27 30.75 28.03 + 16 23.41 30.63 27.92 + 17 23.51 30.54 27.80 + 18 23.73 30.42 27.56 + 19 23.85 30.30 27.38 + 20 23.96 30.21 27.36 + 21 23.80 29.95 27.17 + 22 23.78 29.92 27.10 + 23 23.85 29.94 27.13 + 24 23.85 29.94 27.13 + 25 23.79 29.73 26.95 + 26 23.82 29.70 26.90 + 27 23.88 29.56 26.81 + 28 23.93 29.43 26.71 + 29 24.28 29.41 26.72 + 30 24.16 28.92 26.35 + 31 23.60 28.93 26.40 + 32 49.14 47.82 48.53 + 33 46.50 46.22 47.07 + 34 46.53 46.19 47.01 + 35 46.54 46.16 46.92 + 36 46.53 46.19 46.83 + 37 46.43 46.13 46.74 + 38 46.41 46.13 46.74 + 39 46.47 46.11 46.71 + 40 46.43 46.11 46.73 +e + 0 47.39 46.46 47.27 + 1 44.87 45.50 46.24 + 2 45.36 45.50 46.17 + 3 45.99 45.74 46.36 + 4 29.20 33.96 32.69 + 5 27.11 35.65 33.87 + 6 26.34 36.53 34.36 + 7 27.06 36.94 34.55 + 8 26.62 35.33 33.14 + 9 26.41 34.90 32.75 + 10 26.60 35.22 32.87 + 11 26.55 35.05 32.51 + 12 27.20 34.93 32.37 + 13 27.27 34.74 32.13 + 14 27.46 34.39 31.77 + 15 27.49 34.74 32.16 + 16 27.99 34.70 32.24 + 17 28.06 34.59 32.04 + 18 28.34 34.38 31.94 + 19 28.46 34.49 32.04 + 20 28.55 34.25 31.82 + 21 28.39 33.95 31.59 + 22 28.56 33.89 31.57 + 23 28.75 33.81 31.53 + 24 28.85 33.72 31.48 + 25 28.95 33.60 31.40 + 26 29.12 33.44 31.24 + 27 29.24 33.36 31.18 + 28 29.21 33.18 31.04 + 29 29.28 33.10 30.98 + 30 29.33 33.10 30.95 + 31 29.33 33.00 30.91 + 32 49.13 47.83 48.55 + 33 46.49 46.22 47.09 + 34 46.55 46.16 47.01 + 35 46.52 46.15 46.96 + 36 46.52 46.17 46.83 + 37 46.40 46.13 46.77 + 38 46.43 46.10 46.71 + 39 46.46 46.12 46.71 + 40 46.45 46.10 46.69 +e + +set title '\TitleFont BQSquare' +plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \TIDR\cite{DynamicTMVP2011}' w lp ls 2, '-' using 1:2 title '\KeyFont \PROP' w lp ls 3 + 0 39.53 42.26 42.80 + 1 35.83 42.08 42.70 + 2 35.16 41.99 42.67 + 3 34.61 42.02 42.59 + 4 29.62 39.04 38.96 + 5 26.90 39.01 38.95 + 6 24.50 39.01 38.97 + 7 23.19 38.75 38.67 + 8 22.16 38.75 38.71 + 9 21.21 38.63 38.51 + 10 20.49 38.44 38.37 + 11 20.22 38.35 38.29 + 12 19.84 38.32 38.18 + 13 19.57 38.16 37.95 + 14 19.09 38.05 37.82 + 15 18.84 37.90 37.65 + 16 18.59 37.75 37.42 + 17 18.36 37.70 37.44 + 18 18.33 37.57 37.29 + 19 18.25 37.46 37.12 + 20 18.14 37.32 36.94 + 21 18.12 37.16 36.78 + 22 18.13 36.99 36.62 + 23 18.13 36.92 36.46 + 24 18.15 36.72 36.26 + 25 18.15 36.54 36.08 + 26 18.17 36.39 35.90 + 27 18.15 36.26 35.74 + 28 18.14 36.13 35.55 + 29 18.16 35.99 35.44 + 30 18.19 35.89 35.27 + 31 18.19 35.75 35.10 + 32 39.01 41.92 42.60 + 33 34.91 41.74 42.49 + 34 34.42 41.71 42.48 + 35 33.76 41.65 42.42 + 36 33.14 41.62 42.30 + 37 32.81 41.51 42.22 + 38 32.54 41.37 42.08 + 39 32.35 41.30 41.97 + 40 32.27 41.16 41.90 +e + 0 39.53 42.26 42.80 + 1 35.83 42.08 42.70 + 2 35.16 41.99 42.67 + 3 34.61 42.02 42.59 + 4 29.62 39.04 38.96 + 5 26.90 39.01 38.95 + 6 24.50 39.01 38.97 + 7 23.19 38.75 38.67 + 8 23.32 38.47 38.43 + 9 23.43 38.03 37.84 + 10 23.42 37.62 37.42 + 11 23.53 37.37 37.12 + 12 23.58 37.11 36.81 + 13 23.57 36.90 36.58 + 14 23.58 36.49 36.19 + 15 23.66 36.55 36.20 + 16 23.66 36.36 35.99 + 17 23.71 36.17 35.83 + 18 23.70 36.02 35.65 + 19 23.76 35.87 35.54 + 20 23.78 35.66 35.37 + 21 23.76 35.43 35.15 + 22 23.79 35.28 34.96 + 23 23.87 35.07 34.70 + 24 23.93 34.95 34.54 + 25 23.94 34.81 34.43 + 26 23.92 34.65 34.27 + 27 23.95 34.47 34.06 + 28 23.89 34.36 33.91 + 29 23.89 34.22 33.79 + 30 23.86 34.07 33.61 + 31 23.83 34.03 33.42 + 32 39.05 41.89 42.54 + 33 34.93 41.70 42.45 + 34 34.40 41.67 42.41 + 35 33.78 41.64 42.34 + 36 33.15 41.56 42.22 + 37 32.82 41.42 42.15 + 38 32.52 41.30 42.07 + 39 32.38 41.22 41.98 + 40 32.21 41.10 41.87 +e + 0 39.53 42.26 42.80 + 1 35.83 42.08 42.70 + 2 35.16 41.99 42.67 + 3 34.60 42.02 42.58 + 4 29.76 38.82 38.71 + 5 27.28 38.75 38.65 + 6 26.52 38.65 38.51 + 7 26.27 38.36 38.23 + 8 26.09 38.21 38.10 + 9 25.98 37.84 37.69 + 10 26.06 37.56 37.41 + 11 26.00 37.25 37.06 + 12 25.89 37.10 36.89 + 13 25.84 36.85 36.60 + 14 25.94 36.39 36.15 + 15 25.94 36.14 35.83 + 16 26.06 35.82 35.48 + 17 25.99 35.39 35.06 + 18 26.00 35.30 35.00 + 19 26.10 35.19 34.87 + 20 26.05 35.04 34.68 + 21 25.98 34.87 34.50 + 22 25.97 34.71 34.30 + 23 25.82 34.54 34.15 + 24 25.81 34.46 34.00 + 25 25.78 34.32 33.85 + 26 25.76 34.15 33.68 + 27 25.77 34.03 33.63 + 28 25.74 33.88 33.42 + 29 25.73 33.73 33.24 + 30 25.66 33.60 33.07 + 31 25.67 33.51 32.94 + 32 39.00 41.90 42.60 + 33 34.87 41.75 42.50 + 34 34.38 41.71 42.51 + 35 33.75 41.68 42.42 + 36 33.17 41.62 42.29 + 37 32.79 41.49 42.24 + 38 32.52 41.36 42.11 + 39 32.29 41.28 41.98 + 40 32.19 41.18 41.91 +e + + +\end{gnuplot} + diff --git a/031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingleKendoFrame4.pdf b/031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingleKendoFrame4.pdf new file mode 100644 index 0000000..9db3854 Binary files /dev/null and b/031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingleKendoFrame4.pdf differ diff --git a/031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNR.tex b/031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNR.tex new file mode 100644 index 0000000..d11ddc7 --- /dev/null +++ b/031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNR.tex @@ -0,0 +1,43 @@ +\TABLESVSPACE +\newcommand{\nlines}{2} +\small +\footnotesize +\begin{tabular}{p{3cm}|x{3.5cm}|x{1.8cm}|x{1.8cm}|x{1.8cm}} +\toprule +\textbf{Sequence} & \textbf{TMVP configuration} & \textbf{Error free} & \textbf{1\% loss} & \textbf{5\% loss} \\ +\midrule +\multirow{\nlines}{*}{\vbox{Basketball Drill}} + & \TIDR~\cite{DynamicTMVP2011} & -0.011 & 0.092 & 0.365\\ + & \PROP & -0.054 & 0.218 & 0.823\\ +% & Without TMVP & -0.094 & 0.574 & 2.165\\ +\hline +\multirow{\nlines}{*}{\vbox{Book Arrival}} + & \TIDR~\cite{DynamicTMVP2011} & -0.003 & 0.099 & 0.428\\ + & \PROP & -0.012 & 0.359 & 1.436\\ +% & Without TMVP & -0.021 & 0.641 & 2.494\\\ +\hline +\multirow{\nlines}{*}{\vbox{BQSquare}} + & \TIDR~\cite{DynamicTMVP2011} & -0.005 & 0.418 & 1.572\\ + & \PROP & -0.032 & 0.539 & 1.967\\ +% & Without TMVP & -0.033 & 1.328 & 4.897\\ +\hline +\multirow{\nlines}{*}{\vbox{Kendo}} + & \TIDR~\cite{DynamicTMVP2011} & -0.008 & 0.150 & 0.618\\ + & \PROP & -0.031 & 0.380 & 1.539\\ +% & Without TMVP & -0.035 & 0.569 & 2.277\\ +\hline +\multirow{\nlines}{*}{\vbox{Race Horses}} + & \TIDR~\cite{DynamicTMVP2011} & -0.010 & 0.179 & 0.689\\ + & \PROP & -0.040 & 0.236 & 0.909\\ +% & Without TMVP & -0.050 & 0.776 & 2.796\\ +\hline +\multirow{\nlines}{*}{\vbox{Tennis}} + & \TIDR~\cite{DynamicTMVP2011} & -0.003 & 0.099 & 0.494\\ + & \PROP & -0.041 & 0.214 & 0.932\\ +% & Without TMVP & -0.035 & 0.599 & 2.488\\ +\midrule +\multirow{\nlines}{*}{\vbox{Average}} + & \TIDR~\cite{DynamicTMVP2011} & -0.006 & 0.171 & 0.695 \\ + & \PROP & -0.034 & 0.342 & 1.339 \\ +\bottomrule +\end{tabular} diff --git a/031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNRSTD.tex b/031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNRSTD.tex new file mode 100644 index 0000000..b46f89d --- /dev/null +++ b/031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNRSTD.tex @@ -0,0 +1,42 @@ +\TABLESVSPACE +\newcommand{\nlines}{3} +\small +\footnotesize +\begin{tabular}{p{2.8cm}|x{3cm}|x{1.8cm}|x{1.8cm}|x{1.8cm}} +\toprule +\multirow{2}{*}{\textbf{Sequence}} & +\multirow{2}{*}{\vbox{\centering \textbf{TMVP configuration}}} & +\multicolumn{3}{c}{\textbf{Packet loss rate}}\\ +& & \textbf{1\%} & \textbf{3\%} & \textbf{5\%} \\ +\hline +\multirow{\nlines}{*}{\vbox{Basketball Drill}} % + & \REF & 5.09 & 7.51 & 7.70 \\ + & \TIDR~\cite{DynamicTMVP2011} & 5.00 & 7.30 & 7.52 \\ + & \PROP & 4.78 & 7.03 & 7.24 \\ +\hline +\multirow{\nlines}{*}{\vbox{Book Arrival}}% + & \REF & 5.45 & 8.74 & 9.08 \\ + & \TIDR~\cite{DynamicTMVP2011} & 5.24 & 8.48 & 8.88 \\ + & \PROP & 4.84 & 7.83 & 8.26 \\ +\hline +\multirow{\nlines}{*}{\vbox{BQSquare}} % + & \REF & 5.98 & 8.86 & 9.14 \\ + & \TIDR~\cite{DynamicTMVP2011} & 5.20 & 7.51 & 8.01 \\ + & \PROP & 4.94 & 7.28 & 7.68 \\ +\hline +\multirow{\nlines}{*}{\vbox{Kendo}}% + & \REF & 7.56 & 11.58 & 12.05 \\ + & \TIDR~\cite{DynamicTMVP2011} & 7.25 & 11.00 & 11.55 \\ + & \PROP & 6.62 & 10.06 & 10.66 \\ +\hline +\multirow{\nlines}{*}{\vbox{Race Horses}} % + & \REF & 5.74 & 8.31 & 8.55 \\ + & \TIDR~\cite{DynamicTMVP2011} & 5.47 & 7.78 & 8.07 \\ + & \PROP & 5.29 & 7.63 & 7.88 \\ +\hline +\multirow{\nlines}{*}{\vbox{Tennis}} % + & \REF & 6.70 & 9.55 & 10.07 \\ + & \TIDR~\cite{DynamicTMVP2011} & 6.55 & 9.25 & 9.83 \\ + & \PROP & 6.42 & 9.11 & 9.67 \\ +\bottomrule +\end{tabular} diff --git a/031_ErrorResilienceEvaluation/DynamicTmvp/build_figures b/031_ErrorResilienceEvaluation/DynamicTmvp/build_figures new file mode 100755 index 0000000..1f8b507 --- /dev/null +++ b/031_ErrorResilienceEvaluation/DynamicTmvp/build_figures @@ -0,0 +1,55 @@ +#!/bin/bash + +currdir=$(pwd) +tikz_files=$( find . -name '*.tikz' ); + +for file_list in $tikz_files; do + + dirname="${file_list%/*}" + file="${file_list##*/}" + + cd ${dirname} + + +# % set up externalization +# \usetikzlibrary{external} +# \tikzset{external/system call={latex \tikzexternalcheckshellescape -halt-on-error +# -interaction=batchmode -jobname "\image" "\texsource"; +# dvips -o "\image".ps "\image".dvi; +# ps2eps "\image.ps"}} +# \tikzexternalize + +cat > build.tex <<_EOF_ +\documentclass{article} + +\usepackage{tikz} +\usetikzlibrary{shapes,arrows} + + +\usepackage{verbatim} +\usepackage[active,tightpage]{preview} + +\PreviewEnvironment{tikzpicture} +\setlength\PreviewBorder{5pt} +\begin{document} +\pagestyle{empty} +\begin{figure}[p] +\centering +\input{ $file } +\end{figure} +\end{document} + +_EOF_ + +# latex --shell-escape build.tex + pdflatex build.tex + + rm build.aux + rm build.log + rm build.tex + + mv build.pdf ${file%%.*}.pdf + + cd ${currdir} + +done \ No newline at end of file diff --git a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex new file mode 100644 index 0000000..50cda82 --- /dev/null +++ b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex @@ -0,0 +1,374 @@ +% Figure folders + +\newcommand{\HEVCRobustnessEvaluation}{./031_ErrorResilienceEvaluation/QualityEvaluation} +\newcommand{\DynamicTMVP}{031_ErrorResilienceEvaluation/DynamicTmvp} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% CHAPTER +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\chapter{A Method to Increase the Error Robustness of HEVC} +\label{Chapter:HEVCErrorResilience} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Introduction} + +The \ac{hevc} standard is expected be used by the same range of applications as its predecessors (\emph{e.g.}, H.264/AVC), such as, broadcasting, internet streaming, two-way communications, and storage~\cite{HEVCLayerIntegrationDez2012}. +More advanced prediction structures, as presented in Section~\ref{Chapter:HEVCReview}, allow bitrate savings, but also to some disadvantages, such as increased complexity~\cite{PerformanceComplexityDez2012} and reduced error robustness. Therefore, it is important to evaluate the error robustness of the current standard in order to check for its weaknesses. This can provide relevant insights on future directions on how to improve the standard robustness to transmission losses. + +The results provided by this experimental study allowed to improve the \ac{hevc} encoder in order to avoid ... + +In this chapter an error resilience evaluation of the \ac{hevc} is presented in Section~\ref{Section3:ErrorResilienceEvaluation}. The evaluation covers different coding configuration to evaluate the individual impact of some of the new coding features introduced, e.g., increased partition size, intra-refreshing, slice partitioning and advanced motion vector prediction. In Section~\ref{Section3:DynamicMVRefreshHEVC} a novel method to increase the error robustness of the \ac{hevc} \ac{mv} coding is presented and evaluated against existing techniques. Finally, Section~\ref{Section3:Summary} concludes this chapter. + +\clearpage +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% SECTION +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Evaluation study of the error robustness of HEVC} +\label{Section3:ErrorResilienceEvaluation} + +\subsection{Motivation and objectives} + +As mentioned before the \ac{hevc} standard provide improvements in terms of coding efficiency, however at the cost of more advanced predictions structures, which increases complexity and bitstream dependencies. +Although the complexity may not pose significant problems due to the rapid evolution of hardware technologies, reduced error robustness strongly affects quality under packet loss conditions. + +A critical aspect related to the performance of robust video streaming methods is the error detection accuracy. +Since \ac{hevc} streaming is expected to use known transport technologies, such as RTP~\cite{HEVCStreamMay2012}, MPEG-2 TS~\cite{HEVCLayerIntegrationDez2012}, or more recent standards as MPEG-DASH~\cite{HEVCHTTPStreamAug2016} and MPEG Media Transport\cite{ATSCMMT2016}, their error detection capabilities have significant impact on the overall system performance. +In general, the error detection operation at the transport layers is decoupled from the video coding layer. +The encoding side implements the robust coding methods, while the efficient recovery of lost video data is implemented in the decoder. +Between them, an inter-layer signalling ensures that the decoder is given the necessary information to identify the lost slices/frames in the video data. Therefore, stream robustness can be investigated on its own, assuming any possible form of error detection and streaming technologies. This is the main focus of the study presented in this chapter. + +The are some studies presented in the past regarding the error robustness of the \ac{hevc} standard which are worth to be mention as they provide relevant insights for the research of carried out in this research work. In~\cite{StudyHEVCErrorsDez2012} the both the coding efficiency and error robustness of the \ac{hevc} are evaluated, and compared against the \ac{h264}. Experimental results show that \ac{hevc} has reduced error robustness despite its increased coding efficiency. A subjective analysis is presented in~\cite{SubjectiveEvalSep2013,QoEHEVCMay2014} revealing that packet loss ratios higher than $3\%$ have a significant effect on the perceived video quality. In both works it is concluded that the robustness of \ac{hevc} decreases for high motion sequences. Moreover, in~\cite{DynamicTMVP2011} a the vulnerability of \ac{mv} prediction is presented and its impact of the error robustness is evaluated, showing its weak error resilience in contrast with the coding improvements. Summarising, the decreasing in error robustness is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes of \ac{hevc}, which are selected according to a rate distortion optimisation criteria, assuming error-free transmission\cite{ROPE2007}. + +This section presents an error robustness evaluation of the \ac{hevc} standard. This study focus on the novel techniques introduced extending the generic studies presented in previous works mentioned above. This study may provide relevant insights for future directions where the error robustness problem should be tackled in the \ac{hevc} standard. +The following cases studies are covered in this experimental evaluation: +\begin{enumerate} + \item different maximum \ac{cu} sizes to evaluate the impact of higher larger frame partitioning; +% \item experiments with and without filter operations, i.e., deblocking filter and \ac{sao}, in order to evaluate their impact on decoding video quality; + \item results for difference distances between intra frames, i.e., evaluate the impact of introducing different amount of refresh points to the overall quality; + \item different slice partitioning schemes by varying the number of bytes per slice; + \item different \ac{mv} coding schemes using different types of \ac{mv} candidates, with an analysis at the spatial and temporal level. +\end{enumerate} + +\subsection{Characterisation of the experiments} + +\input{031_ErrorResilienceEvaluation/TablesSequences.tex} + +The experimental setup used in this work to evaluate the error robustness of the \ac{hevc} standard used thirteen test sequences, which were chosen to cover different types of motion and texture complexity. +% Each sequence comprises 240 frames and the whole set covers. + +Table~\ref{TableTestSequences} presents a summary of the main characteristics of the video sequences. Appendix~\ref{Appen:Sequences} shows a list of the first frame of each test sequence. +In order to evaluation the spatial and temporal complexity of the video sequences the following metrics \cite{ITUTP9112008} were used: +\begin{itemize} + \item {\ac{si})} based on Sobel filter. Each video frame ($f_t$) is firstly filtered, then the standard deviation is computed. Finally, the maximum value over time is chosen to represent the \ac{si} metric. This process can be represented in the following equation: + \begin{equation} + SI = max_{time}\Big\{ std_{space} \big( Sobel\left(f_t\right) \big) \Big\}. + \end{equation} + \item {\ac{ti}} based upon the motion difference feature, which is the difference between the pixel values at the same location in space but in consecutive time instances. \ac{ti} is computed as the maximum over time of the standard deviation over space. More motion in adjacent frames will result in higher values of \ac{ti}. This process can be represented by the following: + \begin{equation} + TI = max_{time}\Big\{ std_{space} \big( f_{t}(x,y) - f_{t-1}(x,y) \big) \Big\}. + \end{equation} +\end{itemize} +The values of the metrics described above are represented in the Table~\ref{TableTestSequences} (see second column). + +The experimental results for this study were obtained by using the HM reference software, version $16.2$. +The sequences were encoded with an IDR period of $16$ frames and all the coding modes enabled, following two recommended configurations: Low-Delay (P-frames with 4 references frames) and Random-Access (B-frames with 2 reference frames in each list)~\cite{HEVCCommonTests2012}. +In order to achieve high quality images for all coded sequences (i.e., \ac{psnr} around 40 dB) different bitrates were used. +%The following bitrates were selected to achieve high quality (i.e., around 40 dB): 1.0 Mbits/s (Kendo), 1.5 Mbits/s (Book Arrival), 4.0 Mbits/s (BQSquare, Park Scene and Race Horses) and 12.0 Mbits/s (People on Street). + +% At the decoder side, error concealment is based on motion compensation from previously decoded frames. +% The motion field of a missing frame is obtained applying \ac{mv} extrapolation from the two closest neighbours. +% Afterwards, the final \acs{mv} are selected based on the residual energy of their original prediction~\cite{HEVCMCResidueDec2013}. + +To simulate the lossy transmission environment, each frame is encoded into several slices, each one packetized into one \ac{nal} unit, which are then transmitted as the payload of independent packets. +In our experiments a whole packet is lost whenever an error occurs, originating a lost slice. +Random packet loss was simulated using a two-state Markov model for \ac{plr} of 1\%, 3\%, 5\%, 10\%. +For each test condition, 50 runs were performed and the average \ac{psnr} was measured. +The filtering operations were disabled at slices boundaries in order to keep the slices self-contained. The remaining configurations have been kept at their default values as given in the common test conditions~\cite{HEVCCommonTests2012}. + +%\subsection{Experimental results} + +\subsection{Evaluation of the impact of the partition units size} + + + +The \ac{hevc} standard introduced new partitioning schemes to allow coding improvements in high resolution videos. These partition structures, named \glsfirst{cu}, are allowed to be as large as $64\times64$ pixels. In the first set of experiments the impact of the maximum \ac{cu} size in the decoded video quality is investigated. In this experiment the maximum \ac{cu} size is set to three different sizes, i.e., 16$\times$16, 32$\times$32 and 64$\times$64. The lower size is targeting the \ac{mb} size used in the H.264/AVC. + +\begin{table}[!t] + \centering + \caption{Average \ac{psnr} for different maximum CU size.} + \label{Table3:Quality_BlockSize} + \TABLESVSPACE + \input{./031_ErrorResilienceEvaluation/QualityEvaluation/TableCUSizes.tex} + \vspace{2mm} +\end{table} + +Table~\ref{Table3:Quality_BlockSize} shows the average \ac{psnr} obtained for the three condition tested, for error-free and 4 different packet loss ratios. Results for the error-free scenario shows a consistent quality increasing as the maximum \ac{cu} size increases, to a maximum of 1.41 dB for the Tennis sequence when comparing the maximum \ac{cu} size of $64\times64$ with $16\times16$. This confirms the principle used by the \ac{hevc} standard, that larger coding units lead to higher coding efficiency. In case of data loss, results indicate that larger coding units should be maintained. Even though larger frame partitions lead to larger coding dependencies, it will not affect the robustness to packet loss and error propagation. This is mainly due to the fact that error propagation occurs mostly in complex and moving regions, which normally are coded using smaller partition units. + +\begin{figure}[t] + \centering + % \small + \renewcommand{\FIGSIZE}{0.35\textwidth} + % \resizebox{\columnwidth}{!}{ + \begin{tabular}{y{.45\columnwidth}y{.45\columnwidth}} + \includegraphics[width=\FIGSIZE]{\HEVCRobustnessEvaluation/Bask_Partitions_16.png} & + \includegraphics[width=\FIGSIZE]{\HEVCRobustnessEvaluation/Bask_Partitions_64.png} \\ + (a) $16\times16$ & (b) $64\times64$ \\ + \end{tabular} + % } + \caption{Illustration of the block partition and prediction mode for a region of the Basketball Drill sequence.} + \label{Figure3:BlockPartitionBask} +\end{figure} + +Figure~\ref{Figure3:BlockPartitionBask} illustrates an example of the frame partitions for a region of the Basketball Drill sequence for two cases: (i) maximum \ac{cu} size of $64\times64$ and (i) maximum \ac{cu} size of $16\times16$. As shown, in the moving regions, e.g., moving player, the partitions obtained for both cases have a similar pattern, mainly composed of \acs{cu} with small size. The differences in the size of the \acs{cu} are only noticeable in the background, which is less prone to errors. Therefore, the overall error robustness is not affected by reducing the maximum \ac{cu} size. One may conclude that the use of larger coding units is able to improve the overall \ac{hevc} efficiency in error-free and error prone applications. + + +% \subsection{Impact of filtering operation} +% +% The \ac{hevc} introduced new in-loop filter operations, extending the deblocking filter of the H.264/AVC. One of the filtering operations present in the standard is the \glsfirst{sao}. As mentioned before, the filter is able to reduce the reconstruction distortion by conditionally adding an offset value to each sample after the application of the deblocking filter. In this sub-section the impact of the \ac{sao} is evaluated, by comparing the quality obtained under packet loss with the filter enabled and disabled. +% +% +% \begin{table}[!t] +% \centering +% \caption{Average \ac{psnr} for different filtering configurations.} +% \label{TAB3:QualitySao} +% \input{\HEVCRobustnessEvaluation/TableFilters.tex} +% \end{table} +% +% Table~\ref{TAB3:QualitySao} shows the average decoded quality for the two tested scenario. The case where the filter is enabled is used as reference for comparison and the absolute value of the \ac{psnr} is shown. For the case where the \ac{sao} is disabled the quality differences are illustrated. +% Results confirm the superiority of enabling the the \ac{sao} both in error-free but also in data loss environments. Although only a small quality impact occurs in the case where the filter is disabled is, + + +\subsection{Evaluation of the impact of intra refreshing} + +\begin{figure}[!t] + \centering +% \renewcommand{\FIGSIZE}{0.49\textwidth} +% \includegraphics[width=\FIGSIZE,page=1]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot} +% \includegraphics[width=\FIGSIZE,page=2]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot} +% \includegraphics[width=\FIGSIZE,page=3]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot} +% \includegraphics[width=\FIGSIZE,page=4]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot} +% \includegraphics[width=\FIGSIZE,page=5]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot} +% \includegraphics[width=\FIGSIZE,page=6]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot} +% \includegraphics[width=\FIGSIZE,page=7]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot} +% \includegraphics[width=\FIGSIZE,page=8]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot} +% \includegraphics[width=\FIGSIZE,page=9]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot} +% \includegraphics[width=\FIGSIZE,page=10]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot} +% \includegraphics[width=\FIGSIZE,page=11]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot} +% \includegraphics[width=\FIGSIZE,page=12]{\HEVCRobustnessEvaluation/PSNRvsIntra_Plot} + \input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex.bak} + \caption{Decoded video quality for different intra periods, i.e., distance between intra-frames.} +% \caption{Decoded video quality for different intra periods, i.e., distance between intra-frames (Complementary results in Figure~\ref{Appendix:Figure3:QualityvsIntra}).} + \label{Figure3:QualityvsIntra} +\end{figure} + +In this section the relation between the distance of subsequent intra-coded frames, i.e., period of the intra frames, and the quality under errors is evaluated. As the intra-coded frames have no dependencies to prior coded pictures, they introduce a refresh point into the bitstream. This allows the decoded to recover from slice loss and continue a error-free decoding. + +Results on Figure~\ref{Figure3:QualityvsIntra} shows the average decoded video quality for different distance of intra frames. The figure covers packet loss ratios between 3\% and 5\%. Results indicate that has the distance between intra frames increases the decoded video quality significantly decreases. This is especially noticeable for an intra period higher than 32 frames, indicating that it is not suitable for video streaming under error-prone conditions. + + + +One should also be noted from the results of Figure~\ref{Figure3:QualityvsIntra} that using a small distance between intra frames is not always more efficient. Since intra frames require higher amount of bytes, the overall coding efficiency is reduced, resulting in an overall quality decreasing even with a 5\% f \ac{plr}. This is clear noticeable for the Four People sequence. From the results for Park Scene sequence (and Cactus sequence in Figure~\ref{Appendix:Figure3:QualityvsIntra}) it can be concluded that using sixteen frames between refresh points (i.e., intra frames) achieves an acceptable efficiency under errors, without compromising the coding efficiency. + +\subsection{Evaluation for using different slice partitioning schemes} + +The partitioning of the \acs{ctu} into different slices introduce flexibility into video bitstream, by introducing spatial refresh points and avoiding the error propagation across different \acs{ctu}. In this section the amount of bytes per slices is evaluated, since it the most efficient partitioning scheme in the \ac{hevc} standard that suits the network requirements of fixed packaging. As a reference for comparison a single slice per frame is also evaluated. + + +\begin{figure}[!t] +\centering +\input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsSlices.tex.bak} +\caption{Decoded video quality for different slice partitioning schemes.} + +%\input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsSlices.tex} +%\caption{Decoded video quality for different slice partitioning schemes (Complementary results in Figure~\ref{Appendix:Figure3:SlicePartitioning}).} +\label{Figure3:SlicePartitioning} +\end{figure} + +Figure~\ref{Figure3:SlicePartitioning} shows the average \ac{psnr} for three difference slice partitioning schemes. Results reveals a clear impact of the slice partitioning on the error robustness of the \ac{hevc}, as the quality degradation as the \ac{plr} increases varies for each scheme. +Firstly it is clearly notice the disadvantage of using a single slice per frame as it leads to the lower quality when comparing the test cases studied. This is due to the lack of refresh points within a frame, resulting in loss events that affects the whole frame which will propagate to the subsequent inter predicted frames. +Secondly, comparing slices of different sizes, i.e., comparing the use of 1200 and 2400 bytes per slice, reveals that using small slices leads to higher quality which is able to overcome the inherent degradation on coding efficiency due to higher amount of slice headers. + + + +\subsection{Motion vector coding impact on error propagation} + +\begin{figure}[!t] +\centering +\renewcommand{\FIGSIZE}{0.75\textwidth} +% \includegraphics[width=\FIGSIZE,page=1]{\HEVCRobustnessEvaluation/PSNRvsLossNoTMVPMergeSpatial_Plot} +% \includegraphics[width=\FIGSIZE,page=9]{\HEVCRobustnessEvaluation/PSNRvsLossNoTMVPMergeSpatial_Plot} +\input{031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsMergeCand.tex} +\caption{Average quality using different number of \ac{mv} candidates.} +\label{Figure3:AvgPSNRMergeCandidates} +\end{figure} + + +In \ac{hevc}, new motion vector prediction modes are introduced, such as the \ac{amvp} and Merge Mode~\cite{MotionCodingHEVC2011}. In this section the the error robustness of the Merge Mode is evaluated, which predicts \acs{mv} for the current block from its neighbours, which obviously leads to prediction mismatches at the decoder side in case of packet loss. + +\subsubsection{Spatial \ac{mv} candidates} + + +The first set of experiments aimed to evaluate the impact of using several \ac{mv} candidates in the Merge Mode. As mentioned before, \ac{hevc} allows up to five candidates to be used, which may increase the amount of spatial dependencies. +In order to check the impact of the \ac{mv} candidates, different coding configurations were used, by reducing the number of available candidates. Figure~\ref{Figure3:AvgPSNRMergeCandidates} shows the average quality results for the Basketball Drill sequence under different packet loss conditions. The results show that, besides the amount of \ac{mv} candidates, the error resilience of the \ac{hevc} is not compromised. + +\begin{table}[t] +\newcommand{\somespaces}{} +\newcommand{\colwidth}{1.8cm} +\centering +\caption{Relation between the number of available \ac{mv} candidates and its usage} \label{Table3:MergeIdxRatio} +\footnotesize +\TABLESVSPACE +\begin{tabular}{x{3.3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} +\toprule +\multirow{2}{*}{\vbox{\centering \textbf{Max. number of candidates}}} & \multicolumn{5}{c}{\textbf{Candidate selected (ratio)}} \\ +\cline{2-6} +& 1 & 2 & 3 & 4 & 5 \\ +\hline +3 \acs{mv} & $66.17\%$ & 22.79$\%$ & 11.04$\%$ & - & - \\ +\hline +5 \acs{mv} & 64.54$\%$ & 22.24$\%$ & 7.38$\%$ & 3.37$\%$ & 2.47$\%$ \\ +\bottomrule +\end{tabular} +\end{table} + +The occurrence rate for each spatial \ac{mv} candidate was analysed in order to check if they are uniformly used. Table~\ref{Table3:MergeIdxRatio} shows the use of each \ac{mv} candidate when three and five candidates are available. The results indicate that most predictions use the first candidate and minor differences occur by adding 2 or more \acs{mv}. +Since the mostly used \acs{mv} are the initial ones, changing the number of available candidates does not significantly affects the error resilience. +Moreover, the slice partitioning scheme used (\emph{i.e.}, 1200 bytes per slice) adds more refresh points in each frame reducing the spatial error propagation due to mismatch \ac{mv} predictions. +% \HERE + + +\subsubsection{Temporal \ac{mv} candidates} + + +\begin{figure}[!t] +\centering +\renewcommand{\FIGSIZE}{0.76\textwidth} +% \includegraphics[width=\FIGSIZE,page=2]{\HEVCRobustnessEvaluation/PSNRvsFrameSingleFrameTMVP} +\input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP} +\caption{Error propagation for Kendo sequence when Frame~\#6 is lost for the Low-Delay configuration.} +\label{Figure3:PSNRvsFrameErrorFreeConc} +\end{figure} + +The second set of experiments aimed to find the influence of losing temporal motion predictors on the error resilience of the \ac{hevc} standard. +In order to perform this test, a single frame is lost, and the error propagation is evaluated for two configurations: with the temporal \ac{mv} candidate enabled and disabled. +Figure~\ref{Figure3:PSNRvsFrameErrorFreeConc} illustrates the error propagation, when a single frame is lost. For this particular experiment a intra-refreshing period of 32 frames was used to evaluate a long error propagation period. +The results show that when the \ac{tmvp} mode is enabled, the loss of Frame \#6 leads to significant reduction in the reconstruction quality of the subsequent frames. +% This is true even for sequences with low motion activity as per Table~\ref{tab:sequences}. +On the contrary, when the temporal \ac{mv} candidate is disabled, motion information on subsequent frames is not affected by the loss of a single frame, since it is not temporal dependent, achieving significant quality increasing. This is more noticeable in the Low-Delay configuration since more error propagation occurs in P-frames and it is propagated for several frames until an intra frame is decoded. + +\begin{table}[!t] +\centering +%\caption{Average \ac{psnr} using different motion vector candidates.} +\caption{\ac{psnr} under random packet loss.} \label{Table3:AvgPSNRWithoutTMVP} +\TABLESVSPACE +\input{./031_ErrorResilienceEvaluation/QualityEvaluation/TableMVPs.tex} +\end{table} + +In order to further evaluate the impact of the temporal \ac{mv} dependencies, the \ac{hevc} streams were subject to random packet losses. +Table~\ref{Table3:AvgPSNRWithoutTMVP} presents the average quality results, measured using the \ac{psnr}. +As expected, the results indicate that in error free environment, \ac{hevc} achieves higher quality by enabling the \ac{tmvp} candidate, since it presents higher compression efficiency due to more \ac{mv} predictors in the Merge Mode. This is true for all common test conditions, \emph{i.e.}, both Low Delay and Random Access. +However, in the presence of errors, the quality significantly decreases due to low error robustness when temporal \ac{mv} candidates are used, resulting in an average of 2.65 dB of \ac{psnr} increasing when temporal \ac{mv} predictor is disabled across all configuration tested. +The results also show that the negative impact of using temporal \ac{mv} dependencies decreases as the number of B-frames increases. The average gain of disabling the temporal \ac{mv} predictor is 4.51 dB (Low Delay) and 0.78 dB (Random Access). +This also confirms the previous results of Figure~\ref{Figure3:PSNRvsFrameErrorFreeConc}. +A higher negative impact is obtained when only P-frames are used, since subsequent frames are temporally closer, leading to more accurate temporal \ac{mv} candidates. +Moreover, the results also indicate that for higher loss ratios the negative impact of using the \ac{tmvp} does not increase as both configuration are subject to high quality degradation. + + +\subsection{Discussion and conclusions} + +The results presented in the previous sub-sections show that \ac{hevc} has poor robustness against transmission errors, resulting in significant quality degradation, though dependent on the video content and coding options. As shown coding options as slice partition and intra refreshing period have high correlation with the decoded video quality under errors, which shall be taken into consideration when designing a broadcast application. + +One of the main improvements of the \ac{hevc} standard is related with the high coding units which does not have an impact of its error robustness, so shall be configured to achieve the highest coding performance. Another improvement in the standard was the Merge Mode, which significantly improves the flexibility in coding and signalling the motion information. One the one hand the spatial \ac{mv} candidates do not significantly decrease the error robustness. On the other hand, the introduction of temporal \ac{mv} candidates significantly contribute to increase the temporal dependencies, leading to more severe quality degradation in the presence of slice loss. + +Based on these evidences, the next section presents an efficient method to improve the error robustness of \ac{hevc} based on the underlying principle of reducing the mismatch in \ac{mv} predictions caused by temporal dependencies. + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% SECTION +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Dynamic motion vector refreshing} +\label{Section3:DynamicMVRefreshHEVC} + +%\subsection{Problem formulation} + +The aforementioned results revealed a lack of error resilience in the \ac{hevc} standard motion prediction. Based on this evidence, this section proposes a method to make intelligent use of temporal \ac{mv} candidates during the motion estimation process, in order to decrease the temporal dependency, and improve the error resiliency without penalising the \ac{rd} performance. + +\subsection{Proposed method} + +In this section a new approach to deal with temporal motion vector dependencies is proposed, aiming at reducing the amount of temporally dependent \acs{mv}. This is accomplished by modifying the rate distortion optimisation process in \ac{hevc}. Since the number of \acs{mv} that can be independently decoded increases, the error recovery percentage is improved after a frame loss event. + +In order to overcome in the limitation of the method presented in \cite{DynamicTMVP2011}, a new method that breaks the motion information dependencies between frames is proposed, which is used in the Merge Mode. Instead of disabling the temporal motion candidate at the frame level, the proposed method aims to remove the dependencies at the block (CU) level. To achieve this, the temporal dependency of all \ac{mv} candidates is analysed for each block, in order to decide on the suitable candidates to be used to predict the current \ac{mv}. Based on the analysis of the temporal dependency, some temporal candidates are marked as unsuitable for the prediction of the current \ac{mv}, and are not selected in the merge modes. In the proposed approach, a temporal \ac{mv} candidate is marked as unsuitable based on the following criteria: +\begin{enumerate} + \item if it is encoded based on another temporal \ac{mv} candidate from a previous encoded frame; + \item if it is predicted using a spatial neighbour that was previously encoded using a temporal \ac{mv} candidate. +\end{enumerate} + +\begin{figure}[t] + \centering + % \includegraphics[width=0.6\columnwidth]{\DynamicTMVP/DynamicTMVP} + \input{./031_ErrorResilienceEvaluation/DynamicTmvp/DynamicTMVP.tikz} + \caption{Proposed approach to use the temporal motion vector candidate in the \ac{hevc} encoder.} + \label{ProposedDynamicMethod} + \FIGSVSPACE +\end{figure} + +Figure \ref{ProposedDynamicMethod} can be used to explain the proposed method, illustrating different blocks and different \ac{mv} dependencies represented with arrows. Note that the squares illustrate a generic block size; the representation of the temporal dependencies is simplified, and in some cases may not correspond to the exact co-located block in the previous frame. In frame $f_{t-1}$ there are several \acs{mv} encoded using spatial predictions and one \ac{mv} encoded using a temporal prediction (arrow pointing to $f_{t-2}$). In frame $f_{t}$, the \ac{mv} corresponding to the block (1) may use the temporal \ac{mv} candidate, since the co-located block in $f_{t-1}$ is not temporally dependent. However, the \ac{mv} prediction marked with the arrow (2) is not allowed in the proposed scheme to avoid propagation from temporal dependencies. Moreover, the block corresponding to the arrow (3) cannot use the temporal \ac{mv} candidate since it already depends on a \ac{mv} that was previously encoded using the temporal candidate. The proposed method reduces the temporal dependencies of the \glspl{mv} at the block level, providing improved error robustness to the encoded streams. Using this approach, the dependencies are selectively removed, improving the error recovery, without fully disabling the temporal \gls{mv} candidates for a given frame. + +\subsection{Performance evaluation} +\label{sec:ProposedMethodEvaluation} + +\begin{figure}[tb] + \centering + \input{031_ErrorResilienceEvaluation/DynamicTmvp/PSNRvsFrameSingle.tex} + \caption{Quality obtained with the different approaches tested in the \ac{hevc} when the frame \#5 is missing for Kendo and BQSquare sequences.} + \label{Figure3:ProposedMethodEvaluationSingle} +\end{figure} + + +In this section the performance of the proposed scheme (\PROP) is evaluated against two other approaches: reference \ac{hevc} with TMVP enabled and the \TIDR method as proposed in \cite{DynamicTMVP2011}. To evaluate the proposed method the same experimental setup as presented in Section \ref{Section3:ErrorResilienceEvaluation} was used. + +\subsubsection{Error propagation under single loss events} + +Firstly, the performance of the proposed method was measured under a single loss event. Figure \ref{Figure3:ProposedMethodEvaluationSingle} shows the error propagation for various tested methods. The results show a significant quality difference between the proposed method, the \TIDR method, and the reference \ac{hevc}. +When the temporal \ac{mv} candidate is disabled (proposed and \TIDR method) more error robustness can be achieved. +The results also show the relevance of using the proposed method, as the \TIDR is only able to recover from the frame loss after the frame 8 (\emph{i.e.}, the refresh frame). + + + +\subsubsection{Quality evaluation under random errors} + +\begin{table}[tbp] + \centering + \caption{Average standard deviation of the decoded video quality (PSNR).} \label{Table3:AverageStdPSNR} + \TABLESVSPACE + \input{./031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNRSTD.tex} +\end{table} + +Secondly, further tests were performed to evaluate the effectiveness of the proposed error resiliency method under realistic packet loss patterns. As in previous experiments, each coded frame is transmitted in one packet. +Table~\ref{Table3:AverageStdPSNR} illustrates the average standard deviation of the PSNR, for three of the sequences tested. The results show that increasing the packet loss rate leads to higher quality deviations, and the impact of errors in the subjective quality experience is higher. However, reducing the temporal dependencies of the \acs{mv} using the proposed method decreases the quality differences variation, achieving the lowest value for all scenarios tested. The proposed method is able to reduce the quality difference variations up to 0.94 dB in Kendo sequence (3$\%$ of packet loss) compared to the~\TIDR~method. + + +\begin{table}[tbp] +\centering +\caption{Bjontegaard's average \ac{psnr} (dB) differences for all sequences tested at different frame loss conditions.} \label{Table3:AverageResultsProposed} +\input{031_ErrorResilienceEvaluation/DynamicTmvp/TableDynamicTMVPPSNR.tex} +\end{table} + +Table~\ref{Table3:AverageResultsProposed} shows the Bjontegaard's average \ac{psnr} increase \cite{BJResults2001}. The results are obtained taking as reference the standard \ac{hevc} encoder. +% The results are obtained taking as reference the standard \ac{hevc} encoder, in order to evaluate the quality difference when temporal \ac{mv} candidates are disabled in both error free and under error-prone conditions. +The results presented in Table~\ref{Table3:AverageResultsProposed} show that the proposed method presents practically the same rate-distortion performance when compared to \TIDR method. +In general, the average quality, in an error-free scenario, is only decreased by 0.028 dB in comparison to the \TIDR method and by 0.034 dB in comparison to the reference encoder. +However, the proposed method widely outperforms the reference \TIDR technique~\cite{DynamicTMVP2011} in terms of error resiliency performance, achieving in average higher overall quality under error-prone conditions. +The effectiveness of the proposed method increases at higher packet loss rates (\emph{e.g.},~$5\%$), achieving an average gain of 0.573 dB, and a maximum gain of 1 dB (PSNR) for the Book Arrival sequence, when compared to the \TIDR technique. Higher gains are achieved when compared with the reference \ac{hevc} encoder, as shown in Table~\ref{Table3:AverageResultsProposed}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% SECTION +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Summary} +\label{Section3:Summary} + +In this chapter, the error resilience performance of the \ac{hevc} standard was analysed under error prone conditions. Results revealed that new coding tool introduced in \ac{hevc} lead to higher error propagation. This reveals that new techniques should be employed within \ac{hevc} encoder and decoder to increase its resilience to errors, achieving higher overall performance under broadcast environments. +Therefore, a new approach to deal with a drawback in \ac{mv} coding was proposed, addressing \ac{mv} dependencies at the block level. The proposed method selectively disables the temporal \ac{mv} candidates, in order to break the dependencies across more than one frame. The experimental evaluation shows that the proposed method increases error robustness and outperforms the existing state-of-the-art method and the standard. The results also show that the proposed method is able to provide a good trade-off between the coding performance and error resilience. The results demonstrate that selective usage of temporal \ac{mv} candidates is an efficient method to enhance error resilience in \ac{hevc}. diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/.directory b/031_ErrorResilienceEvaluation/QualityEvaluation/.directory new file mode 100644 index 0000000..1d9570f --- /dev/null +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/.directory @@ -0,0 +1,4 @@ +[Dolphin] +Timestamp=2018,6,20,16,47,0 +Version=4 +ViewMode=1 diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/.pdf b/031_ErrorResilienceEvaluation/QualityEvaluation/.pdf new file mode 100644 index 0000000..92a964e Binary files /dev/null and b/031_ErrorResilienceEvaluation/QualityEvaluation/.pdf differ diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/Bask_Partitions_16.png b/031_ErrorResilienceEvaluation/QualityEvaluation/Bask_Partitions_16.png new file mode 100644 index 0000000..14921bc Binary files /dev/null and b/031_ErrorResilienceEvaluation/QualityEvaluation/Bask_Partitions_16.png differ diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/Bask_Partitions_64.png b/031_ErrorResilienceEvaluation/QualityEvaluation/Bask_Partitions_64.png new file mode 100644 index 0000000..6a00ebf Binary files /dev/null and b/031_ErrorResilienceEvaluation/QualityEvaluation/Bask_Partitions_64.png differ diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP.pdf b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP.pdf new file mode 100644 index 0000000..a8dd183 Binary files /dev/null and b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP.pdf differ diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP.plt b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP.plt new file mode 100644 index 0000000..245267e --- /dev/null +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP.plt @@ -0,0 +1,427 @@ +set terminal postscript eps monochrome font 'TimesNewRoman,16' + +set grid +set size {0.75,0.75} +set xlabel "Frame Number" font ",18" offset character 0, 0.4, 0 +set xtics center font ",16" # offset character 0, 0, 0 +set ylabel "PSNR [dB]" font ",18" offset character 0.9, 0, 0 +set ytics font ",16" +set key spacing 1 font "TimesNewRoman,16" top right width -2 + +set title font ",20" offset character 0, -0.7, 0 + +set style line 1 lc 4 lt 4 lw 3 pt 10 ps 1 +#set style line 2 lc rgbcolor "#006400" lt 2 lw 3 pt 3 ps 1 +set style line 2 lc 2 lt 2 lw 3 pt 3 ps 1 +set style line 3 lc 3 lt 4 lw 3 pt 4 ps 1 +set style line 4 lc 1 lt 1 lw 3 pt 2 ps 1 +set style line 5 lc 7 lt 3 lw 3 pt 6 ps 1 +set style line 6 lc 8 lt 6 lw 3 pt 8 ps 1 +set style line 7 lc 2 lt 7 lw 3 pt 8 ps 1 +set style line 8 lc 3 lt 8 lw 3 pt 9 ps 1 +set style line 9 lc 4 lt 9 lw 3 pt 10 ps 1 + +set xrange[4:4+12] +set xtics 4,2,4+12 +set output 'plot_QualityVsFrameSingle_PSNR_lowdelay_P_4R_slice_single_seq_basketball_drill_rate_single_4.eps' +set title 'Basketball Drill' +plot '-' using 1:2 title 'With TMVP' w lp ls 1, '-' using 1:2 title 'Without TMVP' w lp ls 2 + 0 42.08 44.10 44.86 + 1 42.08 44.10 44.86 + 2 40.24 43.19 44.01 + 3 40.24 42.98 43.90 + 4 41.24 43.64 44.47 + 5 40.21 42.99 43.77 + 6 28.09 35.75 36.85 + 7 24.16 31.81 33.79 + 8 23.04 30.94 33.15 + 9 22.56 30.73 33.00 + 10 22.86 30.95 33.37 + 11 23.07 31.23 33.69 + 12 23.43 31.41 33.92 + 13 23.61 31.72 33.97 + 14 23.83 32.02 34.21 + 15 24.10 32.08 34.28 + 16 42.02 43.95 44.65 + 17 40.04 42.95 43.31 +e + 0 42.08 44.10 44.86 + 1 42.08 44.10 44.86 + 2 40.24 43.19 44.01 + 3 40.24 42.96 43.94 + 4 41.26 43.62 44.49 + 5 40.22 42.95 43.82 + 6 28.15 35.99 37.12 + 7 29.64 36.35 38.27 + 8 30.59 37.98 41.00 + 9 31.02 38.11 41.02 + 10 31.45 38.21 41.32 + 11 31.66 38.34 41.38 + 12 31.44 38.56 41.85 + 13 31.70 38.40 41.48 + 14 31.70 38.40 41.46 + 15 31.45 38.25 40.97 + 16 42.02 43.95 44.65 + 17 40.06 42.88 43.27 +e + + +set xrange[4:4+12] +set xtics 4,2,4+12 +set output 'plot_QualityVsFrameSingle_PSNR_lowdelay_P_4R_slice_single_seq_book_arrival_c6_rate_single_4.eps' +set title 'Book Arrival' +plot '-' using 1:2 title 'With TMVP' w lp ls 1, '-' using 1:2 title 'Without TMVP' w lp ls 2 + 0 41.35 41.07 43.41 + 1 40.37 40.66 43.21 + 2 40.07 40.60 43.19 + 3 40.14 40.52 43.15 + 4 40.48 40.57 43.16 + 5 40.17 40.49 43.14 + 6 35.59 40.37 42.94 + 7 32.10 39.90 42.41 + 8 30.09 39.88 42.23 + 9 29.30 39.61 41.94 + 10 29.03 39.56 41.86 + 11 28.77 39.40 41.67 + 12 29.13 40.01 42.01 + 13 29.23 39.81 41.99 + 14 29.40 39.76 41.97 + 15 29.69 39.77 41.94 + 16 41.30 40.81 43.36 + 17 40.20 40.40 43.05 +e + 0 41.35 41.07 43.41 + 1 40.37 40.66 43.21 + 2 40.04 40.60 43.19 + 3 40.15 40.53 43.14 + 4 40.47 40.56 43.16 + 5 40.17 40.49 43.13 + 6 35.61 40.36 42.94 + 7 36.51 40.27 42.85 + 8 37.20 40.60 43.07 + 9 36.86 40.44 42.97 + 10 36.95 40.41 42.99 + 11 36.74 40.24 42.91 + 12 37.37 40.89 43.38 + 13 36.92 40.57 43.22 + 14 37.06 40.49 43.26 + 15 36.98 40.45 43.23 + 16 41.30 40.81 43.36 + 17 40.20 40.40 43.04 +e + + +set xrange[4:4+12] +set xtics 4,2,4+12 +set output 'plot_QualityVsFrameSingle_PSNR_lowdelay_P_4R_slice_single_seq_bqsquare_rate_single_4.eps' +set title 'BQSquare' +plot '-' using 1:2 title 'With TMVP' w lp ls 1, '-' using 1:2 title 'Without TMVP' w lp ls 2 + 0 41.82 43.90 44.56 + 1 38.85 43.50 44.33 + 2 39.91 43.54 44.35 + 3 38.69 43.46 44.18 + 4 42.75 44.77 45.24 + 5 40.69 44.14 44.81 + 6 33.35 43.93 44.57 + 7 24.72 43.28 43.25 + 8 22.76 42.88 42.50 + 9 20.71 42.18 40.49 + 10 19.72 41.31 39.10 + 11 18.79 40.76 39.31 + 12 17.92 40.65 39.28 + 13 17.81 40.31 38.92 + 14 17.53 39.82 38.33 + 15 17.30 39.35 37.84 + 16 41.75 43.96 44.43 + 17 39.55 43.49 44.20 +e + 0 41.82 43.90 44.56 + 1 38.85 43.50 44.33 + 2 39.89 43.54 44.36 + 3 38.67 43.47 44.20 + 4 42.79 44.71 45.25 + 5 40.68 44.15 44.79 + 6 33.35 43.91 44.56 + 7 34.23 43.82 44.42 + 8 34.64 43.81 44.41 + 9 34.84 43.64 44.28 + 10 35.26 43.69 44.30 + 11 35.29 43.60 44.19 + 12 36.07 44.12 44.70 + 13 35.54 43.82 44.47 + 14 35.81 43.78 44.35 + 15 35.47 43.65 44.30 + 16 41.75 43.96 44.43 + 17 39.57 43.49 44.20 +e + + +set xrange[4:4+12] +set xtics 4,2,4+12 +set output 'plot_QualityVsFrameSingle_PSNR_lowdelay_P_4R_slice_single_seq_four_people_rate_single_4.eps' +set title 'Four People' +plot '-' using 1:2 title 'With TMVP' w lp ls 1, '-' using 1:2 title 'Without TMVP' w lp ls 2 + 0 41.47 44.76 45.95 + 1 40.44 44.50 45.75 + 2 39.95 44.53 45.74 + 3 40.10 44.42 45.65 + 4 40.33 44.45 45.72 + 5 39.96 44.37 45.61 + 6 38.79 44.31 45.49 + 7 35.59 44.04 45.00 + 8 33.48 43.92 44.59 + 9 32.26 43.67 44.13 + 10 31.40 43.47 43.69 + 11 30.73 43.22 43.32 + 12 30.65 43.50 43.27 + 13 30.05 43.20 42.98 + 14 29.54 43.10 42.78 + 15 29.22 42.98 42.57 + 16 41.44 44.84 46.05 + 17 40.38 44.54 45.87 +e + 0 41.47 44.76 45.95 + 1 40.44 44.50 45.75 + 2 39.90 44.53 45.73 + 3 40.10 44.43 45.67 + 4 40.33 44.44 45.71 + 5 39.97 44.37 45.60 + 6 38.78 44.28 45.43 + 7 38.88 44.22 45.36 + 8 39.72 44.44 45.66 + 9 39.46 44.38 45.59 + 10 39.39 44.37 45.54 + 11 39.29 44.32 45.54 + 12 40.34 44.72 45.93 + 13 39.82 44.59 45.83 + 14 39.84 44.58 45.78 + 15 39.63 44.51 45.73 + 16 41.44 44.84 46.05 + 17 40.37 44.54 45.87 +e + + +set xrange[4:4+12] +set xtics 4,2,4+12 +set output 'plot_QualityVsFrameSingle_PSNR_lowdelay_P_4R_slice_single_seq_kimono_rate_single_4.eps' +set title 'Kimono' +plot '-' using 1:2 title 'With TMVP' w lp ls 1, '-' using 1:2 title 'Without TMVP' w lp ls 2 + 0 42.69 43.90 45.51 + 1 41.72 42.60 44.39 + 2 42.19 43.10 44.75 + 3 42.34 43.37 45.11 + 4 43.28 44.86 46.63 + 5 42.70 43.93 45.85 + 6 30.31 40.90 42.81 + 7 25.52 38.85 40.45 + 8 23.88 37.59 39.96 + 9 22.55 36.35 38.91 + 10 21.92 35.96 38.85 + 11 20.83 35.17 38.24 + 12 20.57 35.26 38.64 + 13 20.32 34.81 38.21 + 14 20.21 34.79 38.40 + 15 20.00 34.65 38.20 + 16 42.53 43.87 45.41 + 17 42.67 43.93 45.50 +e + 0 42.69 43.90 45.51 + 1 41.72 42.60 44.39 + 2 42.19 43.10 44.72 + 3 42.31 43.35 45.14 + 4 43.27 44.87 46.61 + 5 42.68 43.91 45.81 + 6 30.20 40.86 42.83 + 7 31.22 41.70 43.49 + 8 31.42 42.43 44.15 + 9 31.67 42.21 43.99 + 10 32.51 42.93 44.62 + 11 32.63 42.42 44.21 + 12 32.95 43.66 45.17 + 13 32.99 42.71 44.46 + 14 33.31 43.42 45.01 + 15 33.26 42.87 44.58 + 16 42.53 43.87 45.41 + 17 42.67 43.94 45.49 +e + + +set xrange[4:4+12] +set xtics 4,2,4+12 +set output 'plot_QualityVsFrameSingle_PSNR_lowdelay_P_4R_slice_single_seq_park_rate_single_4.eps' +set title 'Park Scene' +plot '-' using 1:2 title 'With TMVP' w lp ls 1, '-' using 1:2 title 'Without TMVP' w lp ls 2 + 0 40.67 42.36 43.38 + 1 38.61 41.30 42.89 + 2 38.92 41.20 42.76 + 3 38.98 41.24 42.74 + 4 40.37 41.98 43.23 + 5 39.28 41.40 42.93 + 6 33.37 40.26 41.94 + 7 26.85 37.34 40.39 + 8 24.84 36.08 40.02 + 9 23.89 35.09 39.09 + 10 23.29 34.66 38.74 + 11 22.85 34.28 38.39 + 12 22.52 34.27 38.50 + 13 22.20 33.88 38.09 + 14 21.89 33.59 37.83 + 15 21.70 33.32 37.50 + 16 40.68 42.36 43.40 + 17 39.17 41.44 43.01 +e + 0 40.67 42.36 43.38 + 1 38.61 41.30 42.89 + 2 38.91 41.20 42.76 + 3 38.96 41.23 42.75 + 4 40.35 41.99 43.22 + 5 39.25 41.41 42.91 + 6 33.34 40.25 41.87 + 7 34.01 40.32 42.08 + 8 34.53 41.32 42.94 + 9 34.37 40.78 42.59 + 10 34.69 40.91 42.66 + 11 34.63 40.69 42.49 + 12 35.28 41.72 43.29 + 13 34.93 41.09 42.93 + 14 35.11 41.14 42.87 + 15 35.03 40.93 42.73 + 16 40.68 42.36 43.40 + 17 39.17 41.44 43.01 +e + + +set xrange[4:4+12] +set xtics 4,2,4+12 +set output 'plot_QualityVsFrameSingle_PSNR_lowdelay_P_4R_slice_single_seq_people_rate_single_4.eps' +set title 'People on Street' +plot '-' using 1:2 title 'With TMVP' w lp ls 1, '-' using 1:2 title 'Without TMVP' w lp ls 2 + 0 38.19 42.44 42.94 + 1 35.07 41.98 42.52 + 2 35.61 41.79 42.43 + 3 34.53 41.50 42.20 + 4 38.49 42.51 42.97 + 5 36.58 42.21 42.69 + 6 24.34 40.83 41.88 + 7 20.83 39.29 40.61 + 8 19.09 38.38 39.35 + 9 18.17 37.60 38.47 + 10 17.58 37.04 37.93 + 11 17.12 36.65 37.52 + 12 16.98 36.72 37.39 + 13 16.73 36.43 37.06 + 14 16.54 36.22 36.86 + 15 16.36 36.03 36.58 + 16 38.16 42.45 42.94 + 17 35.99 42.09 42.56 +e + 0 38.19 42.44 42.94 + 1 35.07 41.98 42.52 + 2 35.59 41.76 42.38 + 3 34.44 41.45 42.14 + 4 38.43 42.47 42.95 + 5 36.52 42.16 42.67 + 6 24.41 40.77 41.89 + 7 25.59 40.88 41.89 + 8 26.24 41.27 42.19 + 9 26.59 41.13 42.04 + 10 26.85 41.04 41.99 + 11 27.03 40.93 41.88 + 12 27.65 41.54 42.32 + 13 27.58 41.30 42.12 + 14 27.80 41.22 42.07 + 15 27.78 41.08 41.98 + 16 38.16 42.45 42.94 + 17 36.01 42.09 42.57 +e + + +set xrange[4:4+12] +set xtics 4,2,4+12 +set output 'plot_QualityVsFrameSingle_PSNR_lowdelay_P_4R_slice_single_seq_race_rate_single_4.eps' +set title 'Race Horses' +plot '-' using 1:2 title 'With TMVP' w lp ls 1, '-' using 1:2 title 'Without TMVP' w lp ls 2 + 0 40.88 42.15 42.89 + 1 38.67 40.46 41.59 + 2 40.70 41.68 42.63 + 3 38.32 40.45 41.51 + 4 43.57 44.35 44.85 + 5 40.26 41.90 42.86 + 6 22.05 29.84 32.10 + 7 18.13 26.39 28.37 + 8 17.02 25.27 27.32 + 9 16.23 24.41 25.99 + 10 15.95 24.72 25.82 + 11 15.70 24.60 25.64 + 12 15.58 24.38 25.57 + 13 15.44 24.34 25.50 + 14 15.17 24.18 25.23 + 15 15.15 24.09 24.95 + 16 41.07 42.46 43.15 + 17 39.48 41.17 42.09 +e + 0 40.88 42.15 42.89 + 1 38.67 40.46 41.59 + 2 40.68 41.70 42.59 + 3 38.29 40.49 41.58 + 4 43.58 44.34 44.82 + 5 40.25 41.87 42.81 + 6 21.87 29.77 32.00 + 7 22.82 30.24 32.68 + 8 22.88 30.11 32.38 + 9 23.04 30.24 32.53 + 10 23.43 30.46 32.72 + 11 23.79 30.83 32.76 + 12 24.23 31.40 33.14 + 13 24.43 31.80 33.24 + 14 24.79 32.04 33.43 + 15 25.00 32.07 33.47 + 16 41.07 42.46 43.15 + 17 39.48 41.16 42.08 +e + + +set xrange[4:4+12] +set xtics 4,2,4+12 +set output 'plot_QualityVsFrameSingle_PSNR_lowdelay_P_4R_slice_single_seq_tennis_rate_single_4.eps' +set title 'Tennis' +plot '-' using 1:2 title 'With TMVP' w lp ls 1, '-' using 1:2 title 'Without TMVP' w lp ls 2 + 0 40.70 43.53 44.69 + 1 37.86 42.51 43.33 + 2 37.29 42.13 42.96 + 3 36.41 41.80 42.48 + 4 39.79 43.03 43.95 + 5 38.28 42.34 43.18 + 6 22.56 37.86 38.81 + 7 18.13 34.28 36.70 + 8 16.74 33.00 35.72 + 9 16.49 32.86 35.58 + 10 16.66 32.85 35.69 + 11 16.99 33.22 35.87 + 12 17.97 34.14 36.43 + 13 18.37 34.51 36.96 + 14 17.98 33.86 36.90 + 15 17.63 33.42 36.48 + 16 40.61 43.40 44.62 + 17 37.94 42.30 43.22 +e + 0 40.70 43.53 44.69 + 1 37.86 42.51 43.33 + 2 37.19 42.13 42.92 + 3 36.32 41.77 42.33 + 4 39.75 43.00 43.90 + 5 38.23 42.31 43.13 + 6 22.32 37.71 38.77 + 7 24.13 38.61 40.28 + 8 24.66 39.82 41.33 + 9 25.37 39.87 41.41 + 10 26.09 40.16 41.68 + 11 27.77 40.94 41.96 + 12 31.73 42.60 43.89 + 13 32.13 42.23 43.16 + 14 29.61 41.00 42.48 + 15 27.46 40.08 41.92 + 16 40.61 43.40 44.62 + 17 37.93 42.28 43.25 +e + diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP.tex b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP.tex new file mode 100644 index 0000000..c288cc1 --- /dev/null +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsFrameSingleFrameTMVP.tex @@ -0,0 +1,51 @@ +\newcommand{\pltw}{0.6\columnwidth} +\newcommand{\plth}{0.25\textheight} +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +set xlabel '\TitleFont Frame Number' +set xrange[4:16] +set xtics 0,2,0+31 +set key width 1 + +#set title '\TitleFont Basketball Drill' +plot '-' using 1:2 title '\KeyFont With TMVP' w lp ls 1, '-' using 1:2 title '\KeyFont Without TMVP' w lp ls 2 +0 42.08 44.10 44.86 +1 42.08 44.10 44.86 +2 40.24 43.19 44.01 +3 40.24 42.98 43.90 +4 41.24 43.64 44.47 +5 40.21 42.99 43.77 +6 28.09 35.75 36.85 +7 24.16 31.81 33.79 +8 23.04 30.94 33.15 +9 22.56 30.73 33.00 +10 22.86 30.95 33.37 +11 23.07 31.23 33.69 +12 23.43 31.41 33.92 +13 23.61 31.72 33.97 +14 23.83 32.02 34.21 +15 24.10 32.08 34.28 +16 42.02 43.95 44.65 +17 40.04 42.95 43.31 +e +0 42.08 44.10 44.86 +1 42.08 44.10 44.86 +2 40.24 43.19 44.01 +3 40.24 42.96 43.94 +4 41.26 43.62 44.49 +5 40.22 42.95 43.82 +6 28.15 35.99 37.12 +7 29.64 36.35 38.27 +8 30.59 37.98 41.00 +9 31.02 38.11 41.02 +10 31.45 38.21 41.32 +11 31.66 38.34 41.38 +12 31.44 38.56 41.85 +13 31.70 38.40 41.48 +14 31.70 38.40 41.46 +15 31.45 38.25 40.97 +16 42.02 43.95 44.65 +17 40.06 42.88 43.27 +e + +\end{gnuplot} diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex new file mode 100644 index 0000000..d44eb2d --- /dev/null +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex @@ -0,0 +1,183 @@ + +\ifx\extraresults\undefined + +%\newcommand{\pltw}{0.95\columnwidth} +%\newcommand{\plth}{0.6\textheight} + +\newcommand{\pltw}{0.7\columnwidth} +\newcommand{\plth}{0.65\textheight} + +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +#set multiplot layout 3,2 +set multiplot layout 2,1 +set xlabel '\TitleFont Distance between intra-frames' +set xrange[8:128] +set xtics 8,24,128 +set key width 0 +set key top right + + + +set yrange[32:40] +set ytics 0,2,60 +set title '\TitleFont Four People' +plot '-' using 1:2 title '\KeyFont PLR=3\%' w lp ls 1, '-' using 1:2 title '\KeyFont PLR=5\%' w lp ls 2, '-' using 1:2 title '\KeyFont PLR=10\%' w lp ls 3 +8 38.133 +16 39.406 +32 39.348 +64 38.679 +128 36.869 +e +8 37.945 +16 39.005 +32 38.656 +64 37.630 +128 34.773 +e +8 37.553 +16 38.115 +32 37.104 +64 35.689 +128 32.253 +e + +set ytics 1,2,60 +set yrange[29:39] +set title '\TitleFont Park Scene' +plot '-' using 1:2 title '\KeyFont PLR=3\%' w lp ls 1, '-' using 1:2 title '\KeyFont PLR=5\%' w lp ls 2, '-' using 1:2 title '\KeyFont PLR=10\%' w lp ls 3 +8 38.358 +16 38.385 +32 37.981 +64 36.489 +128 33.335 +e +8 38.166 +16 38.069 +32 37.657 +64 35.836 +128 31.918 +e +8 37.747 +16 37.561 +32 37.237 +64 34.992 +128 30.270 +e + +\end{gnuplot} + +\else + +\newcommand{\pltw}{0.7\columnwidth} +\newcommand{\plth}{.55\textheight} + +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +#set multiplot layout 3,2 +set multiplot layout 3,1 +set xlabel '\TitleFont Distance between intra-frames' +set xrange[8:128] +set xtics 8,24,128 +set key width 0 +set key top right + +# set yrange[28:38] +# set ytics 0,2,60 +# set title '\TitleFont Basketball Drill' +# plot '-' using 1:2 title '\KeyFont PLR=3\%' w lp ls 1, '-' using 1:2 title '\KeyFont PLR=5\%' w lp ls 2, '-' using 1:2 title '\KeyFont PLR=10\%' w lp ls 3 +# 8 37.109 +# 16 36.705 +# 32 35.293 +# 64 33.595 +# 128 31.678 +# e +# 8 36.239 +# 16 35.526 +# 32 33.974 +# 64 32.148 +# 128 30.154 +# e +# 8 34.801 +# 16 33.680 +# 32 31.793 +# 64 30.041 +# 128 28.417 +# e + +set key bottom left +set key Right +set yrange[20:38] +set ytics 0,4,60 +set title '\TitleFont BQSquare' +plot '-' using 1:2 title '\KeyFont PLR=3\%' w lp ls 1, '-' using 1:2 title '\KeyFont PLR=5\%' w lp ls 2, '-' using 1:2 title '\KeyFont PLR=10\%' w lp ls 3 +8 37.119 +16 36.973 +32 35.490 +64 32.657 +128 28.359 +e +8 36.368 +16 35.779 +32 33.757 +64 29.928 +128 24.355 +e +8 34.702 +16 33.624 +32 30.857 +64 26.249 +128 19.927 +e + +set key bottom left +set key Right +set yrange[32:38] +set ytics 32,1,38 +set title '\TitleFont Cactus' +plot '-' using 1:2 title '\KeyFont PLR=3\%' w lp ls 1, '-' using 1:2 title '\KeyFont PLR=5\%' w lp ls 2, '-' using 1:2 title '\KeyFont PLR=10\%' w lp ls 3 +8 37.405 +16 37.400 +32 37.126 +64 36.228 +128 34.402 +e +8 37.319 +16 37.270 +32 36.972 +64 35.899 +128 33.694 +e +8 37.123 +16 36.930 +32 36.771 +64 35.507 +128 32.936 +e + +set ytics 19,4,60 +set yrange[19:35] +set title '\TitleFont People On Street' +plot '-' using 1:2 title '\KeyFont PLR=3\%' w lp ls 1, '-' using 1:2 title '\KeyFont PLR=5\%' w lp ls 2, '-' using 1:2 title '\KeyFont PLR=10\%' w lp ls 3 +8 34.174 +16 33.795 +32 32.959 +64 30.096 +128 25.043 +e +8 33.830 +16 33.225 +32 32.469 +64 29.101 +128 23.017 +e +8 33.188 +16 32.404 +32 31.783 +64 27.646 +128 20.037 +e + +\end{gnuplot} + +\fi diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex.bak b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex.bak new file mode 100644 index 0000000..529bc2e --- /dev/null +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex.bak @@ -0,0 +1,158 @@ + +\newcommand{\pltw}{0.95\columnwidth} +\newcommand{\plth}{0.65\textheight} + +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +set multiplot layout 3,2 +#set multiplot layout 3,1 +set xlabel '\TitleFont Distance between intra-frames' +set xrange[8:128] +set xtics 8,24,128 +set key width 0 +set key top right + +set yrange[28:38] +set ytics 0,2,60 +set title '\TitleFont Basketball Drill' +plot '-' using 1:2 title '\KeyFont PLR=3\%' w lp ls 1, '-' using 1:2 title '\KeyFont PLR=5\%' w lp ls 2, '-' using 1:2 title '\KeyFont PLR=10\%' w lp ls 3 +8 37.109 +16 36.705 +32 35.293 +64 33.595 +128 31.678 +e +8 36.239 +16 35.526 +32 33.974 +64 32.148 +128 30.154 +e +8 34.801 +16 33.680 +32 31.793 +64 30.041 +128 28.417 +e + +set key bottom left +set key Right +set yrange[20:38] +set ytics 0,4,60 +set title '\TitleFont BQSquare' +plot '-' using 1:2 title '\KeyFont PLR=3\%' w lp ls 1, '-' using 1:2 title '\KeyFont PLR=5\%' w lp ls 2, '-' using 1:2 title '\KeyFont PLR=10\%' w lp ls 3 +8 37.119 +16 36.973 +32 35.490 +64 32.657 +128 28.359 +e +8 36.368 +16 35.779 +32 33.757 +64 29.928 +128 24.355 +e +8 34.702 +16 33.624 +32 30.857 +64 26.249 +128 19.927 +e + +set key bottom left +set key Right +set yrange[32:38] +set ytics 32,1,38 +set title '\TitleFont Cactus' +plot '-' using 1:2 title '\KeyFont PLR=3\%' w lp ls 1, '-' using 1:2 title '\KeyFont PLR=5\%' w lp ls 2, '-' using 1:2 title '\KeyFont PLR=10\%' w lp ls 3 +8 37.405 +16 37.400 +32 37.126 +64 36.228 +128 34.402 +e +8 37.319 +16 37.270 +32 36.972 +64 35.899 +128 33.694 +e +8 37.123 +16 36.930 +32 36.771 +64 35.507 +128 32.936 +e + +set yrange[32:40] +set ytics 0,2,60 +set title '\TitleFont Four People' +plot '-' using 1:2 title '\KeyFont PLR=3\%' w lp ls 1, '-' using 1:2 title '\KeyFont PLR=5\%' w lp ls 2, '-' using 1:2 title '\KeyFont PLR=10\%' w lp ls 3 +8 38.133 +16 39.406 +32 39.348 +64 38.679 +128 36.869 +e +8 37.945 +16 39.005 +32 38.656 +64 37.630 +128 34.773 +e +8 37.553 +16 38.115 +32 37.104 +64 35.689 +128 32.253 +e + +set ytics 1,2,60 +set yrange[29:39] +set title '\TitleFont Park Scene' +plot '-' using 1:2 title '\KeyFont PLR=3\%' w lp ls 1, '-' using 1:2 title '\KeyFont PLR=5\%' w lp ls 2, '-' using 1:2 title '\KeyFont PLR=10\%' w lp ls 3 +8 38.358 +16 38.385 +32 37.981 +64 36.489 +128 33.335 +e +8 38.166 +16 38.069 +32 37.657 +64 35.836 +128 31.918 +e +8 37.747 +16 37.561 +32 37.237 +64 34.992 +128 30.270 +e + +set ytics 19,4,60 +set yrange[19:35] +set title '\TitleFont People On Street' +plot '-' using 1:2 title '\KeyFont PLR=3\%' w lp ls 1, '-' using 1:2 title '\KeyFont PLR=5\%' w lp ls 2, '-' using 1:2 title '\KeyFont PLR=10\%' w lp ls 3 +8 34.174 +16 33.795 +32 32.959 +64 30.096 +128 25.043 +e +8 33.830 +16 33.225 +32 32.469 +64 29.101 +128 23.017 +e +8 33.188 +16 32.404 +32 31.783 +64 27.646 +128 20.037 +e + +\end{gnuplot} + diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra_Plot.pdf b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra_Plot.pdf new file mode 100644 index 0000000..d3d1eb3 Binary files /dev/null and b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra_Plot.pdf differ diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra_Plot.plt b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra_Plot.plt new file mode 100644 index 0000000..63db3ba --- /dev/null +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra_Plot.plt @@ -0,0 +1,294 @@ +set terminal postscript eps monochrome font 'TimesNewRoman,18' + +set grid +set size {0.65,0.65} +set xlabel "Distance between intra frames" center # offset character 0, 0.1, 0 +set xtics center +set ylabel "PSNR (dB)" offset character 0.9, 0, 0 +set ytics center +set key spacing 1 font "TimesNewRoman,16" top right width 4 + +set title offset character 0, -0.7, 0 + +set style line 1 lc 4 lt 4 lw 3 pt 10 ps 1 +# set style line 2 lc rgbcolor "#006400" lt 2 lw 3 pt 3 ps 1 +set style line 2 lc 2 lt 2 lw 3 pt 3 ps 1 +set style line 3 lc 3 lt 4 lw 3 pt 4 ps 1 +set style line 4 lc 1 lt 1 lw 3 pt 2 ps 1 +set style line 5 lc 7 lt 3 lw 3 pt 6 ps 1 +set style line 6 lc 8 lt 6 lw 3 pt 8 ps 1 +set style line 7 lc 2 lt 7 lw 3 pt 8 ps 1 +set style line 8 lc 3 lt 8 lw 3 pt 9 ps 1 +set style line 9 lc 4 lt 9 lw 3 pt 10 ps 1 + +set xrange[8:128] +set xtics 8,8,128 + +set output 'plot_PSNRvsIntra_Plot_seq_basketball_drill_NoTMVP_ConcealmentMVE_rate_single.eps' +set title 'Basketball Drill' +plot '-' using 1:2 title 'PLR=3%' w lp ls 1, '-' using 1:2 title 'PLR=5%' w lp ls 2, '-' using 1:2 title 'PLR=10%' w lp ls 3 +8 37.109 +16 36.705 +32 35.293 +64 33.595 +128 31.678 +e +8 36.239 +16 35.526 +32 33.974 +64 32.148 +128 30.154 +e +8 34.801 +16 33.680 +32 31.793 +64 30.041 +128 28.417 +e + +set output 'plot_PSNRvsIntra_Plot_seq_book_arrival_c6_NoTMVP_ConcealmentMVE_rate_single.eps' +set title 'Book Arrival' +plot '-' using 1:2 title 'PLR=3%' w lp ls 1, '-' using 1:2 title 'PLR=5%' w lp ls 2, '-' using 1:2 title 'PLR=10%' w lp ls 3 +8 37.709 +16 36.334 +32 34.301 +64 31.863 +128 28.203 +e +8 36.630 +16 34.694 +32 31.776 +64 28.455 +128 24.761 +e +8 34.163 +16 31.357 +32 28.227 +64 24.431 +128 21.233 +e + +set output 'plot_PSNRvsIntra_Plot_seq_bqsquare_NoTMVP_ConcealmentMVE_rate_single.eps' +set title 'BQSquare' +plot '-' using 1:2 title 'PLR=3%' w lp ls 1, '-' using 1:2 title 'PLR=5%' w lp ls 2, '-' using 1:2 title 'PLR=10%' w lp ls 3 +8 37.119 +16 36.973 +32 35.490 +64 32.657 +128 28.359 +e +8 36.368 +16 35.779 +32 33.757 +64 29.928 +128 24.355 +e +8 34.702 +16 33.624 +32 30.857 +64 26.249 +128 19.927 +e + +set yrange[32:38] +set ytics 32,1,38 +set output 'plot_PSNRvsIntra_Plot_seq_cactus_NoTMVP_ConcealmentMVE_rate_single.eps' +set title 'Cactus' +plot '-' using 1:2 title 'PLR=3%' w lp ls 1, '-' using 1:2 title 'PLR=5%' w lp ls 2, '-' using 1:2 title 'PLR=10%' w lp ls 3 +8 37.405 +16 37.400 +32 37.126 +64 36.228 +128 34.402 +e +8 37.319 +16 37.270 +32 36.972 +64 35.899 +128 33.694 +e +8 37.123 +16 36.930 +32 36.771 +64 35.507 +128 32.936 +e + +set autoscale +set ytics autofreq +set output 'plot_PSNRvsIntra_Plot_seq_four_people_NoTMVP_ConcealmentMVE_rate_single.eps' +set title 'Four People' +plot '-' using 1:2 title 'PLR=3%' w lp ls 1, '-' using 1:2 title 'PLR=5%' w lp ls 2, '-' using 1:2 title 'PLR=10%' w lp ls 3 +8 38.133 +16 39.406 +32 39.348 +64 38.679 +128 36.869 +e +8 37.945 +16 39.005 +32 38.656 +64 37.630 +128 34.773 +e +8 37.553 +16 38.115 +32 37.104 +64 35.689 +128 32.253 +e + +set output 'plot_PSNRvsIntra_Plot_seq_kendo_c3_NoTMVP_ConcealmentMVE_rate_single.eps' +set title 'Kendo' +plot '-' using 1:2 title 'PLR=3%' w lp ls 1, '-' using 1:2 title 'PLR=5%' w lp ls 2, '-' using 1:2 title 'PLR=10%' w lp ls 3 +8 38.161 +16 37.264 +32 35.115 +64 33.059 +128 30.010 +e +8 36.845 +16 34.900 +32 31.838 +64 29.280 +128 25.823 +e +8 33.502 +16 30.713 +32 27.469 +64 24.951 +128 22.402 +e + +set output 'plot_PSNRvsIntra_Plot_seq_kimono_NoTMVP_ConcealmentMVE_rate_single.eps' +set title 'Kimono' +plot '-' using 1:2 title 'PLR=3%' w lp ls 1, '-' using 1:2 title 'PLR=5%' w lp ls 2, '-' using 1:2 title 'PLR=10%' w lp ls 3 +8 41.925 +16 41.587 +32 41.041 +64 39.534 +128 37.000 +e +8 41.625 +16 41.109 +32 40.590 +64 38.628 +128 35.291 +e +8 41.202 +16 40.637 +32 40.168 +64 37.814 +128 33.701 +e + +set output 'plot_PSNRvsIntra_Plot_seq_kristen_sara_NoTMVP_ConcealmentMVE_rate_single.eps' +set title 'Kristen Sara' +plot '-' using 1:2 title 'PLR=3%' w lp ls 1, '-' using 1:2 title 'PLR=5%' w lp ls 2, '-' using 1:2 title 'PLR=10%' w lp ls 3 +8 38.824 +16 39.840 +32 39.876 +64 39.172 +128 37.537 +e +8 38.625 +16 39.395 +32 39.057 +64 37.632 +128 35.424 +e +8 38.147 +16 38.567 +32 37.519 +64 35.222 +128 32.004 +e + +set output 'plot_PSNRvsIntra_Plot_seq_park_NoTMVP_ConcealmentMVE_rate_single.eps' +set title 'Park Scene' +plot '-' using 1:2 title 'PLR=3%' w lp ls 1, '-' using 1:2 title 'PLR=5%' w lp ls 2, '-' using 1:2 title 'PLR=10%' w lp ls 3 +8 38.358 +16 38.385 +32 37.981 +64 36.489 +128 33.335 +e +8 38.166 +16 38.069 +32 37.657 +64 35.836 +128 31.918 +e +8 37.747 +16 37.561 +32 37.237 +64 34.992 +128 30.270 +e + +set output 'plot_PSNRvsIntra_Plot_seq_people_NoTMVP_ConcealmentMVE_rate_single.eps' +set title 'People On Street' +plot '-' using 1:2 title 'PLR=3%' w lp ls 1, '-' using 1:2 title 'PLR=5%' w lp ls 2, '-' using 1:2 title 'PLR=10%' w lp ls 3 +8 34.174 +16 33.795 +32 32.959 +64 30.096 +128 25.043 +e +8 33.830 +16 33.225 +32 32.469 +64 29.101 +128 23.017 +e +8 33.188 +16 32.404 +32 31.783 +64 27.646 +128 20.037 +e + +set output 'plot_PSNRvsIntra_Plot_seq_race_NoTMVP_ConcealmentMVE_rate_single.eps' +set title 'Race Horses' +plot '-' using 1:2 title 'PLR=3%' w lp ls 1, '-' using 1:2 title 'PLR=5%' w lp ls 2, '-' using 1:2 title 'PLR=10%' w lp ls 3 +8 38.139 +16 37.661 +32 36.793 +64 35.866 +128 33.616 +e +8 37.691 +16 37.118 +32 36.090 +64 35.143 +128 32.239 +e +8 36.881 +16 36.218 +32 35.110 +64 34.175 +128 30.279 +e + +set output 'plot_PSNRvsIntra_Plot_seq_tennis_NoTMVP_ConcealmentMVE_rate_single.eps' +set title 'Tennis' +plot '-' using 1:2 title 'PLR=3%' w lp ls 1, '-' using 1:2 title 'PLR=5%' w lp ls 2, '-' using 1:2 title 'PLR=10%' w lp ls 3 +8 38.120 +16 37.515 +32 36.697 +64 34.934 +128 33.778 +e +8 37.357 +16 36.545 +32 35.767 +64 33.817 +128 32.510 +e +8 36.104 +16 35.064 +32 34.339 +64 31.954 +128 30.726 +e + diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsLossMergeCandidates.pdf b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsLossMergeCandidates.pdf new file mode 100644 index 0000000..acdc070 Binary files /dev/null and b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsLossMergeCandidates.pdf differ diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsLossNoTMVPMergeSpatial_Plot.pdf b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsLossNoTMVPMergeSpatial_Plot.pdf new file mode 100644 index 0000000..daa74d8 Binary files /dev/null and b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsLossNoTMVPMergeSpatial_Plot.pdf differ diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsLossNoTMVPMergeSpatial_Plot.plt b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsLossNoTMVPMergeSpatial_Plot.plt new file mode 100644 index 0000000..3c3a5bb --- /dev/null +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsLossNoTMVPMergeSpatial_Plot.plt @@ -0,0 +1,67 @@ +set terminal postscript eps monochrome font 'TimesNewRoman,16' + +set grid +set size {0.75,0.75} +set xlabel "Packet Loss Ratio [%]" font ",18" offset character 0, 0.4, 0 +set xtics center font ",16" # offset character 0, 0, 0 +set ylabel "PSNR [dB]" font ",18" offset character 0.9, 0, 0 +set ytics font ",16" +set key spacing 1 font "TimesNewRoman,16" top right width -4 + +set title font ",20" offset character 0, -0.7, 0 + +set style line 1 lc 4 lt 4 lw 3 pt 10 ps 1 +# set style line 2 lc rgbcolor "#006400" lt 2 lw 3 pt 3 ps 1 +set style line 2 lc 2 lt 2 lw 3 pt 3 ps 1 +set style line 3 lc 3 lt 4 lw 3 pt 4 ps 1 +set style line 4 lc 1 lt 1 lw 3 pt 2 ps 1 +set style line 5 lc 7 lt 3 lw 3 pt 6 ps 1 +set style line 6 lc 8 lt 6 lw 3 pt 8 ps 1 +set style line 7 lc 2 lt 7 lw 3 pt 8 ps 1 +set style line 8 lc 3 lt 8 lw 3 pt 9 ps 1 +set style line 9 lc 4 lt 9 lw 3 pt 10 ps 1 + +set output 'plot_PSNRvsLossNoTMVPMergeSpatial_Plot_lowdelay_P_4R_slice_1200B_seq_basketball_drill_rate_single.eps' +set title 'Low-delay - Basketball Drill - Fixed Bitrate' +plot '-' using 1:2 title '1 Merge Cand' w lp ls 1, '-' using 1:2 title '3 Merge Cand' w lp ls 2, '-' using 1:2 title '5 Merge Cand' w lp ls 3 +0.0 39.860 +1.0 38.401 +3.0 36.517 +5.0 35.428 +10.0 33.623 +e +0.0 39.890 +1.0 38.472 +3.0 36.564 +5.0 35.580 +10.0 33.562 +e +0.0 39.900 +1.0 38.462 +3.0 36.705 +5.0 35.526 +10.0 33.680 +e + +set output 'plot_PSNRvsLossNoTMVPMergeSpatial_Plot_lowdelay_P_4R_slice_1200B_seq_people_rate_single.eps' +set title 'Low-delay - People On Street - Fixed Bitrate' +plot '-' using 1:2 title '1 Merge Cand' w lp ls 1, '-' using 1:2 title '3 Merge Cand' w lp ls 2, '-' using 1:2 title '5 Merge Cand' w lp ls 3 +0.0 35.840 +1.0 34.696 +3.0 33.731 +5.0 33.188 +10.0 32.357 +e +0.0 35.890 +1.0 34.694 +3.0 33.767 +5.0 33.294 +10.0 32.375 +e +0.0 35.900 +1.0 34.736 +3.0 33.795 +5.0 33.225 +10.0 32.404 +e + diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsLossSlices_Plot.pdf b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsLossSlices_Plot.pdf new file mode 100644 index 0000000..debaf50 Binary files /dev/null and b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsLossSlices_Plot.pdf differ diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsMergeCand.tex b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsMergeCand.tex new file mode 100644 index 0000000..f2e5cef --- /dev/null +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsMergeCand.tex @@ -0,0 +1,63 @@ +\newcommand{\pltw}{0.75\columnwidth} +\newcommand{\plth}{0.5\textheight} +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] + +load 'carreira_phd_thesis_gnuplot_template.cfg' + +set size {1, 1} +set multiplot layout 2,1 + +set xlabel '\PLRLABEL' offset character 0, 0, 0 +set ylabel '\PSNRLABEL' offset character -1, 0, 0 +# set xrange[8:128] +# set xtics 8,24,128 font ",2" +set ytics 0,1,40 + +set key width 0 + +set yrange[33:40] +set title '\TitleFont Basketball Drill' +plot '-' using 1:2 title '\KeyFont 1 Merge Cand' w lp ls 1, '-' using 1:2 title '\KeyFont 3 Merge Cand' w lp ls 2, '-' using 1:2 title '\KeyFont 5 Merge Cand' w lp ls 3 +0.0 39.860 +1.0 38.401 +3.0 36.517 +5.0 35.428 +10.0 33.623 +e +0.0 39.890 +1.0 38.472 +3.0 36.564 +5.0 35.580 +10.0 33.562 +e +0.0 39.900 +1.0 38.462 +3.0 36.705 +5.0 35.526 +10.0 33.680 +e + +set yrange[32:36] +set title '\TitleFont People On Street' +plot '-' using 1:2 title '\KeyFont 1 Merge Cand' w lp ls 1, '-' using 1:2 title '\KeyFont 3 Merge Cand' w lp ls 2, '-' using 1:2 title '\KeyFont 5 Merge Cand' w lp ls 3 +0.0 35.840 +1.0 34.696 +3.0 33.731 +5.0 33.188 +10.0 32.357 +e +0.0 35.890 +1.0 34.694 +3.0 33.767 +5.0 33.294 +10.0 32.375 +e +0.0 35.900 +1.0 34.736 +3.0 33.795 +5.0 33.225 +10.0 32.404 +e + +\end{gnuplot} + diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsSlices.tex b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsSlices.tex new file mode 100644 index 0000000..7380441 --- /dev/null +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsSlices.tex @@ -0,0 +1,298 @@ +\ifx\extraresults\undefined +\newcommand{\pltw}{0.7\columnwidth} +\newcommand{\plth}{0.65\textheight} +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +set multiplot layout 2,1 + +set key width -3 + + +set yrange[36:44] +set ytics 36,2,60 +set title '\TitleFont Kimono' +plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +0.0 42.940 +1.0 42.180 +3.0 40.589 +5.0 39.315 +10.0 36.789 +e +0.0 42.880 +1.0 42.247 +3.0 41.587 +5.0 41.109 +10.0 40.637 +e +0.0 42.900 +1.0 42.140 +3.0 41.143 +5.0 40.650 +10.0 39.841 +e + +set yrange[30:42] +set ytics 30,2,60 +set title '\TitleFont Race Horses' +plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +0.0 40.030 +1.0 38.846 +3.0 36.403 +5.0 34.608 +10.0 30.944 +e +0.0 39.870 +1.0 38.724 +3.0 37.661 +5.0 37.118 +10.0 36.218 +e +0.0 39.920 +1.0 38.269 +3.0 36.467 +5.0 35.580 +10.0 34.204 +e + + +\end{gnuplot} + +\else + +\newcommand{\pltw}{0.7\columnwidth} +\newcommand{\plth}{.95\textheight} + +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +set multiplot layout 4,1 + +set key width -3 + +set yrange[28:42] +set ytics 0,2,60 +set title '\TitleFont Basketball Drill' +plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +0.0 40.080 +1.0 39.198 +3.0 37.351 +5.0 35.953 +10.0 33.024 +e +0.0 39.900 +1.0 38.462 +3.0 36.705 +5.0 35.526 +10.0 33.680 +e +0.0 39.970 +1.0 37.988 +3.0 34.884 +5.0 32.921 +10.0 29.059 +e + + +# set title '\TitleFont Book Arrival' +# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +# 0.0 40.390 +# 1.0 39.635 +# 3.0 37.954 +# 5.0 36.821 +# 10.0 34.185 +# e +# 0.0 40.280 +# 1.0 38.818 +# 3.0 36.334 +# 5.0 34.694 +# 10.0 31.357 +# e +# 0.0 40.320 +# 1.0 39.093 +# 3.0 36.939 +# 5.0 34.280 +# 10.0 30.948 +# e + +set yrange[31:41] +set ytics 31,2,60 +set title '\TitleFont BQSquare' +plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +0.0 39.430 +1.0 38.946 +3.0 37.851 +5.0 36.991 +10.0 35.078 +e +0.0 39.300 +1.0 38.486 +3.0 36.973 +5.0 35.779 +10.0 33.624 +e +0.0 39.350 +1.0 38.468 +3.0 36.429 +5.0 34.475 +10.0 31.059 +e + + +# set title '\TitleFont Cactus' +# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +# 0.0 38.060 +# 1.0 37.545 +# 3.0 36.519 +# 5.0 35.651 +# 10.0 33.521 +# e +# 0.0 38.000 +# 1.0 37.707 +# 3.0 37.400 +# 5.0 37.270 +# 10.0 36.930 +# e +# 0.0 38.030 +# 1.0 37.719 +# 3.0 37.329 +# 5.0 37.122 +# 10.0 36.666 +# e + + +# set title '\TitleFont Four People' +# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +# 0.0 40.270 +# 1.0 40.081 +# 3.0 39.678 +# 5.0 39.272 +# 10.0 38.230 +# e +# 0.0 40.130 +# 1.0 39.862 +# 3.0 39.406 +# 5.0 39.005 +# 10.0 38.115 +# e +# 0.0 40.180 +# 1.0 39.947 +# 3.0 39.328 +# 5.0 38.753 +# 10.0 37.494 +# e + + +# set title '\TitleFont Kendo' +# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +# 0.0 41.320 +# 1.0 40.405 +# 3.0 38.475 +# 5.0 37.084 +# 10.0 34.076 +# e +# 0.0 41.150 +# 1.0 39.809 +# 3.0 37.264 +# 5.0 34.900 +# 10.0 30.713 +# e +# 0.0 41.240 +# 1.0 40.124 +# 3.0 37.752 +# 5.0 35.788 +# 10.0 32.144 +# e + +# set title '\TitleFont Kristen Sara' +# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +# 0.0 40.650 +# 1.0 40.488 +# 3.0 40.121 +# 5.0 39.782 +# 10.0 38.861 +# e +# 0.0 40.500 +# 1.0 40.328 +# 3.0 39.840 +# 5.0 39.395 +# 10.0 38.567 +# e +# 0.0 40.550 +# 1.0 40.316 +# 3.0 39.861 +# 5.0 39.241 +# 10.0 38.111 +# e + +set yrange[34:42] +set ytics 34,2,60 +set title '\TitleFont Park Scene' +plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +0.0 39.570 +1.0 39.033 +3.0 37.859 +5.0 36.923 +10.0 34.945 +e +0.0 39.470 +1.0 38.905 +3.0 38.385 +5.0 38.069 +10.0 37.561 +e +0.0 39.510 +1.0 38.934 +3.0 38.304 +5.0 37.856 +10.0 37.133 +e + +set yrange[29:37] +set ytics 29,2,60 +set title '\TitleFont People On Street' +plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +0.0 36.140 +1.0 35.257 +3.0 33.619 +5.0 32.391 +10.0 29.845 +e +0.0 35.900 +1.0 34.736 +3.0 33.795 +5.0 33.225 +10.0 32.404 +e +0.0 35.970 +1.0 34.583 +3.0 33.228 +5.0 32.490 +10.0 31.323 +e + +# +# set title '\TitleFont Tennis' +# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +# 0.0 40.900 +# 1.0 39.773 +# 3.0 37.491 +# 5.0 35.896 +# 10.0 32.303 +# e +# 0.0 40.680 +# 1.0 39.242 +# 3.0 37.515 +# 5.0 36.545 +# 10.0 35.064 +# e +# 0.0 40.770 +# 1.0 38.369 +# 3.0 35.105 +# 5.0 33.079 +# 10.0 29.667 +# e + + + +\end{gnuplot} + +\fi diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsSlices.tex.bak b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsSlices.tex.bak new file mode 100644 index 0000000..078bc63 --- /dev/null +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsSlices.tex.bak @@ -0,0 +1,283 @@ +\newcommand{\ft}[0]{\footnotesize} +\newcommand{\pltw}{0.95\columnwidth} +\newcommand{\plth}{0.6\textheight} +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +set multiplot layout 3,2 + +set key width -3 + +set yrange[28:42] +set ytics 0,2,60 +set title '\TitleFont Basketball Drill' +plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +0.0 40.080 +1.0 39.198 +3.0 37.351 +5.0 35.953 +10.0 33.024 +e +0.0 39.900 +1.0 38.462 +3.0 36.705 +5.0 35.526 +10.0 33.680 +e +0.0 39.970 +1.0 37.988 +3.0 34.884 +5.0 32.921 +10.0 29.059 +e + + +# set title '\TitleFont Book Arrival' +# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +# 0.0 40.390 +# 1.0 39.635 +# 3.0 37.954 +# 5.0 36.821 +# 10.0 34.185 +# e +# 0.0 40.280 +# 1.0 38.818 +# 3.0 36.334 +# 5.0 34.694 +# 10.0 31.357 +# e +# 0.0 40.320 +# 1.0 39.093 +# 3.0 36.939 +# 5.0 34.280 +# 10.0 30.948 +# e + +set yrange[31:41] +set ytics 31,2,60 +set title '\TitleFont BQSquare' +plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +0.0 39.430 +1.0 38.946 +3.0 37.851 +5.0 36.991 +10.0 35.078 +e +0.0 39.300 +1.0 38.486 +3.0 36.973 +5.0 35.779 +10.0 33.624 +e +0.0 39.350 +1.0 38.468 +3.0 36.429 +5.0 34.475 +10.0 31.059 +e + + +# set title '\TitleFont Cactus' +# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +# 0.0 38.060 +# 1.0 37.545 +# 3.0 36.519 +# 5.0 35.651 +# 10.0 33.521 +# e +# 0.0 38.000 +# 1.0 37.707 +# 3.0 37.400 +# 5.0 37.270 +# 10.0 36.930 +# e +# 0.0 38.030 +# 1.0 37.719 +# 3.0 37.329 +# 5.0 37.122 +# 10.0 36.666 +# e + + +# set title '\TitleFont Four People' +# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +# 0.0 40.270 +# 1.0 40.081 +# 3.0 39.678 +# 5.0 39.272 +# 10.0 38.230 +# e +# 0.0 40.130 +# 1.0 39.862 +# 3.0 39.406 +# 5.0 39.005 +# 10.0 38.115 +# e +# 0.0 40.180 +# 1.0 39.947 +# 3.0 39.328 +# 5.0 38.753 +# 10.0 37.494 +# e + + +# set title '\TitleFont Kendo' +# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +# 0.0 41.320 +# 1.0 40.405 +# 3.0 38.475 +# 5.0 37.084 +# 10.0 34.076 +# e +# 0.0 41.150 +# 1.0 39.809 +# 3.0 37.264 +# 5.0 34.900 +# 10.0 30.713 +# e +# 0.0 41.240 +# 1.0 40.124 +# 3.0 37.752 +# 5.0 35.788 +# 10.0 32.144 +# e + +set yrange[36:44] +set ytics 36,2,60 +set title '\TitleFont Kimono' +plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +0.0 42.940 +1.0 42.180 +3.0 40.589 +5.0 39.315 +10.0 36.789 +e +0.0 42.880 +1.0 42.247 +3.0 41.587 +5.0 41.109 +10.0 40.637 +e +0.0 42.900 +1.0 42.140 +3.0 41.143 +5.0 40.650 +10.0 39.841 +e + + +# set title '\TitleFont Kristen Sara' +# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +# 0.0 40.650 +# 1.0 40.488 +# 3.0 40.121 +# 5.0 39.782 +# 10.0 38.861 +# e +# 0.0 40.500 +# 1.0 40.328 +# 3.0 39.840 +# 5.0 39.395 +# 10.0 38.567 +# e +# 0.0 40.550 +# 1.0 40.316 +# 3.0 39.861 +# 5.0 39.241 +# 10.0 38.111 +# e + +set yrange[34:42] +set ytics 34,2,60 +set title '\TitleFont Park Scene' +plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +0.0 39.570 +1.0 39.033 +3.0 37.859 +5.0 36.923 +10.0 34.945 +e +0.0 39.470 +1.0 38.905 +3.0 38.385 +5.0 38.069 +10.0 37.561 +e +0.0 39.510 +1.0 38.934 +3.0 38.304 +5.0 37.856 +10.0 37.133 +e + +set yrange[29:37] +set ytics 29,2,60 +set title '\TitleFont People On Street' +plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +0.0 36.140 +1.0 35.257 +3.0 33.619 +5.0 32.391 +10.0 29.845 +e +0.0 35.900 +1.0 34.736 +3.0 33.795 +5.0 33.225 +10.0 32.404 +e +0.0 35.970 +1.0 34.583 +3.0 33.228 +5.0 32.490 +10.0 31.323 +e + +set yrange[30:42] +set ytics 30,2,60 +set title '\TitleFont Race Horses' +plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +0.0 40.030 +1.0 38.846 +3.0 36.403 +5.0 34.608 +10.0 30.944 +e +0.0 39.870 +1.0 38.724 +3.0 37.661 +5.0 37.118 +10.0 36.218 +e +0.0 39.920 +1.0 38.269 +3.0 36.467 +5.0 35.580 +10.0 34.204 +e + +# +# set title '\TitleFont Tennis' +# plot '-' using 1:2 title '\KeyFont Single Slice/Frame' w lp ls 1, '-' using 1:2 title '\KeyFont 1200 Bytes/Slice' w lp ls 2, '-' using 1:2 title '\KeyFont 2400 Bytes/Slice' w lp ls 3 +# 0.0 40.900 +# 1.0 39.773 +# 3.0 37.491 +# 5.0 35.896 +# 10.0 32.303 +# e +# 0.0 40.680 +# 1.0 39.242 +# 3.0 37.515 +# 5.0 36.545 +# 10.0 35.064 +# e +# 0.0 40.770 +# 1.0 38.369 +# 3.0 35.105 +# 5.0 33.079 +# 10.0 29.667 +# e + + + +\end{gnuplot} + diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/TableCUSizes.tex b/031_ErrorResilienceEvaluation/QualityEvaluation/TableCUSizes.tex new file mode 100644 index 0000000..fe6caaa --- /dev/null +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/TableCUSizes.tex @@ -0,0 +1,73 @@ +%\small +\footnotesize +\newcommand{\colwidth}{1.5cm} +\newcommand\mergedlines{3} +\begin{tabular}{p{3.1cm}|x{2.5cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} +\toprule +\multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & +% \multirow{2}{*}{\vbox{\centering \textbf{Condition}}} & +\multirow{2}{*}{\vbox{\centering \textbf{Maximum\ \ CU size}}} & +\multirow{2}{*}{\vbox{\centering \textbf{Error free}}} & +\multicolumn{3}{c}{\textbf{Packet loss rate}}\\ +& & & \textbf{1\%} & \textbf{5\%} &\textbf{10\%}\\ +\hline +\multirow{\mergedlines}{*}{Basketball Drill} +& \BLOCKSIXTEEN & 39.25 & 37.82 & 34.66 & 32.67 \\ +& \BLOCKTHIRTYTWO & +0.47 & +0.38 & +0.66 & +0.63 \\ +& \BLOCKSIXTYFOUR & +0.65 & +0.64 & +0.87 & +1.01 \\ +\hline +\multirow{\mergedlines}{*}{Book Arrival} +& \BLOCKSIXTEEN & 39.85 & 38.11 & 33.63 & 30.60 \\ +& \BLOCKTHIRTYTWO & +0.25 & +0.19 & +0.21 & -0.38 \\ +& \BLOCKSIXTYFOUR & +0.43 & +0.71 & +1.06 & +0.75 \\ +\hline +\multirow{\mergedlines}{*}{BQSquare} +& \BLOCKSIXTEEN & 38.99 & 38.11 & 35.04 & 32.75 \\ +& \BLOCKTHIRTYTWO & +0.22 & +0.17 & +0.33 & +0.34 \\ +& \BLOCKSIXTYFOUR & +0.31 & +0.37 & +0.74 & +0.88 \\ +\hline +\multirow{\mergedlines}{*}{Cactus} +& \BLOCKSIXTEEN & 37.50 & 37.19 & 36.72 & 36.45 \\ +& \BLOCKTHIRTYTWO & +0.39 & +0.38 & +0.41 & +0.39 \\ +& \BLOCKSIXTYFOUR & +0.50 & +0.52 & +0.55 & +0.48 \\ +\hline +\multirow{\mergedlines}{*}{Four People} +& \BLOCKSIXTEEN & 39.54 & 39.26 & 38.42 & 37.54 \\ +& \BLOCKTHIRTYTWO & +0.44 & +0.43 & +0.38 & +0.34 \\ +& \BLOCKSIXTYFOUR & +0.59 & +0.60 & +0.59 & +0.57 \\ +\hline +\multirow{\mergedlines}{*}{Kendo} +& \BLOCKSIXTEEN & 39.76 & 38.47 & 33.90 & 29.57 \\ +& \BLOCKTHIRTYTWO & +0.99 & +1.04 & +0.74 & +1.01 \\ +& \BLOCKSIXTYFOUR & +1.39 & +1.34 & +1.00 & +1.14 \\ +\hline +\multirow{\mergedlines}{*}{Kimono} +& \BLOCKSIXTEEN & 42.42 & 41.69 & 40.64 & 40.13 \\ +& \BLOCKTHIRTYTWO & +0.36 & +0.39 & +0.32 & +0.33 \\ +& \BLOCKSIXTYFOUR & +0.46 & +0.56 & +0.47 & +0.51 \\ +\hline +\multirow{\mergedlines}{*}{Park Scene} +& \BLOCKSIXTEEN & 39.03 & 38.49 & 37.64 & 37.18 \\ +& \BLOCKTHIRTYTWO & +0.33 & +0.26 & +0.38 & +0.30 \\ +& \BLOCKSIXTYFOUR & +0.44 & +0.41 & +0.43 & +0.38 \\ +\hline +\multirow{\mergedlines}{*}{People On Street} +& \BLOCKSIXTEEN & 35.20 & 34.05 & 32.66 & 31.83 \\ +& \BLOCKTHIRTYTWO & +0.37 & +0.31 & +0.34 & +0.26 \\ +& \BLOCKSIXTYFOUR & +0.70 & +0.69 & +0.57 & +0.57 \\ +\hline +\multirow{\mergedlines}{*}{Race Horses} +& \BLOCKSIXTEEN & 39.19 & 37.92 & 36.33 & 35.56 \\ +& \BLOCKTHIRTYTWO & +0.51 & +0.55 & +0.49 & +0.47 \\ +& \BLOCKSIXTYFOUR & +0.68 & +0.81 & +0.79 & +0.66 \\ +\hline +\multirow{\mergedlines}{*}{Tennis} +& \BLOCKSIXTEEN & 39.27 & 37.91 & 35.39 & 34.04 \\ +& \BLOCKTHIRTYTWO & +0.95 & +0.83 & +0.73 & +0.58 \\ +& \BLOCKSIXTYFOUR & +1.41 & +1.33 & +1.15 & +1.02 \\ +\hline +\multirow{2}{*}{\textbf{\vbox{Average Difference}}} +& \textbf{\BLOCKTHIRTYTWO} & \textbf{+0.48} & \textbf{+0.45} & \textbf{+0.45} & \textbf{+0.39} \\ +& \textbf{\BLOCKSIXTYFOUR} & \textbf{+0.69} & \textbf{+0.73} & \textbf{+0.75} & \textbf{+0.72} \\ +\bottomrule +\end{tabular} diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/TableErrorResilience.tex b/031_ErrorResilienceEvaluation/QualityEvaluation/TableErrorResilience.tex new file mode 100644 index 0000000..139597f --- /dev/null +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/TableErrorResilience.tex @@ -0,0 +1,2 @@ + + diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/TableFilters.tex b/031_ErrorResilienceEvaluation/QualityEvaluation/TableFilters.tex new file mode 100644 index 0000000..948a447 --- /dev/null +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/TableFilters.tex @@ -0,0 +1,59 @@ +\small +\newcommand{\colwidth}{1.45cm} +\newcommand\mergedlines{2} +\begin{tabular}{p{3cm}|p{2cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} +\toprule +\multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & +\multirow{2}{*}{\vbox{\centering \textbf{\ac{sao}}}} & +\multirow{2}{*}{\vbox{\centering \textbf{Error Free}}} & +\multicolumn{4}{c}{\textbf{Packet loss rate}}\\ +& & & \textbf{1\%} & \textbf{3\%} & \textbf{5\%} &\textbf{10\%}\\ +\hline +\multirow{\mergedlines}{*}{Basketball Drill} +& \WITHSAO & 39.90 & 38.46 & 36.71 & 35.53 & 33.68 \\ +& \NOSAO & -0.20 & -0.21 & -0.25 & -0.17 & -0.25 \\ +\hline +\multirow{\mergedlines}{*}{Book Arrival} +& \WITHSAO & 40.28 & 38.82 & 36.33 & 34.69 & 31.36 \\ +& \NOSAO & -0.12 & -0.03 & +0.14 & -0.28 & -0.03 \\ +\hline +\multirow{\mergedlines}{*}{BQSquare} +& \WITHSAO & 39.30 & 38.49 & 36.97 & 35.78 & 33.62 \\ +& \NOSAO & -0.07 & -0.04 & -0.07 & -0.16 & +0.02 \\ +\hline +\multirow{\mergedlines}{*}{Cactus} +& \WITHSAO & 38.00 & 37.71 & 37.40 & 37.27 & 36.93 \\ +& \NOSAO & -0.26 & -0.27 & -0.27 & -0.28 & -0.23 \\ +\hline +\multirow{\mergedlines}{*}{Four People} +& \WITHSAO & 40.13 & 39.86 & 39.41 & 39.01 & 38.12 \\ +& \NOSAO & -0.15 & -0.15 & -0.16 & -0.11 & -0.08 \\ +\hline +\multirow{\mergedlines}{*}{Kendo} +& \WITHSAO & 41.15 & 39.81 & 37.26 & 34.90 & 30.71 \\ +& \NOSAO & -0.10 & -0.08 & -0.13 & -0.08 & -0.09 \\ +\hline +\multirow{\mergedlines}{*}{Kimono} +& \WITHSAO & 42.88 & 42.25 & 41.59 & 41.11 & 40.64 \\ +& \NOSAO & -0.13 & -0.10 & -0.09 & -0.02 & -0.06 \\ +\hline +\multirow{\mergedlines}{*}{Park Scene} +& \WITHSAO & 39.47 & 38.90 & 38.38 & 38.07 & 37.56 \\ +& \NOSAO & -0.21 & -0.21 & -0.18 & -0.19 & -0.17 \\ +\hline +\multirow{\mergedlines}{*}{People On Street} +& \WITHSAO & 35.90 & 34.74 & 33.79 & 33.22 & 32.40 \\ +& \NOSAO & -0.22 & -0.20 & -0.16 & -0.12 & -0.19 \\ +\hline +\multirow{\mergedlines}{*}{Race Horses} +& \WITHSAO & 39.87 & 38.72 & 37.66 & 37.12 & 36.22 \\ +& \NOSAO & -0.28 & -0.28 & -0.25 & -0.27 & -0.20 \\ +\hline +\multirow{\mergedlines}{*}{Tennis} +& \WITHSAO & 40.68 & 39.24 & 37.51 & 36.54 & 35.06 \\ +& \NOSAO & -0.11 & -0.13 & -0.08 & -0.11 & -0.05 \\ +\hline +\multicolumn{2}{c|}{\textbf{Average Difference}} +& \textbf{-0.17} & \textbf{-0.15} & \textbf{-0.14} & \textbf{-0.16} & \textbf{-0.12} \\ +\bottomrule +\end{tabular} diff --git a/031_ErrorResilienceEvaluation/QualityEvaluation/TableMVPs.tex b/031_ErrorResilienceEvaluation/QualityEvaluation/TableMVPs.tex new file mode 100644 index 0000000..6440104 --- /dev/null +++ b/031_ErrorResilienceEvaluation/QualityEvaluation/TableMVPs.tex @@ -0,0 +1,121 @@ +% \small +% \newcommand{\colwidth}{1.3cm} +% \newcommand\mergedlines{3} +% \begin{tabular}{p{2.8cm}|p{2.9cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} +% \toprule +% \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & +% \multirow{2}{*}{\vbox{\centering \textbf{Condition}}} & +% \multirow{2}{*}{\vbox{\centering \textbf{Error Free}}} & +% \multicolumn{4}{c}{\textbf{Packet loss rate}}\\ +% & & & \textbf{1\%} & \textbf{3\%} & \textbf{5\%} &\textbf{10\%}\\ +% \hline +% \multirow{\mergedlines}{*}{Basketball Drill} +% & \REF & 39.90 & 38.46 & 36.71 & 35.53 & 33.68 \\ +% & \MERGEONE & -0.04 & -0.06 & -0.19 & -0.10 & -0.06 \\ +% & \TMVP & +0.02 & -0.50 & -1.16 & -1.31 & -1.17 \\ +% \hline +% \multirow{\mergedlines}{*}{Book Arrival} +% & \REF & 40.28 & 38.82 & 36.33 & 34.69 & 31.36 \\ +% & \MERGEONE & -0.02 & -0.09 & +0.10 & -0.00 & -0.05 \\ +% & \TMVP & +0.02 & -0.71 & -1.38 & -1.95 & -2.56 \\ +% \hline +% \multirow{\mergedlines}{*}{BQSquare} +% & \REF & 39.30 & 38.49 & 36.97 & 35.78 & 33.62 \\ +% & \MERGEONE & -0.02 & -0.02 & -0.01 & -0.10 & +0.01 \\ +% & \TMVP & +0.04 & -2.11 & -3.73 & -4.56 & -5.16 \\ +% \hline +% \multirow{\mergedlines}{*}{Cactus} +% & \REF & 38.00 & 37.71 & 37.40 & 37.27 & 36.93 \\ +% & \MERGEONE & -0.01 & -0.02 & -0.00 & -0.04 & +0.03 \\ +% & \TMVP & +0.02 & -0.04 & -0.09 & -0.13 & -0.03 \\ +% \hline +% \multirow{\mergedlines}{*}{Four People} +% & \REF & 40.13 & 39.86 & 39.41 & 39.01 & 38.12 \\ +% & \MERGEONE & -0.01 & -0.03 & -0.04 & +0.05 & +0.02 \\ +% & \TMVP & +0.02 & -0.15 & -0.58 & -0.96 & -1.58 \\ +% \hline +% \multirow{\mergedlines}{*}{Kendo} +% & \REF & 41.15 & 39.81 & 37.26 & 34.90 & 30.71 \\ +% & \MERGEONE & -0.04 & +0.01 & -0.16 & -0.02 & +0.08 \\ +% & \TMVP & +0.05 & -0.60 & -1.87 & -2.80 & -3.97 \\ +% \hline +% \multirow{\mergedlines}{*}{Kimono} +% & \REF & 42.88 & 42.25 & 41.59 & 41.11 & 40.64 \\ +% & \MERGEONE & -0.01 & -0.01 & -0.01 & +0.03 & -0.00 \\ +% & \TMVP & +0.02 & -0.36 & -0.48 & -0.43 & -0.40 \\ +% \hline +% \multirow{\mergedlines}{*}{Park Scene} +% & \REF & 39.47 & 38.90 & 38.38 & 38.07 & 37.56 \\ +% & \MERGEONE & -0.01 & -0.01 & -0.01 & +0.05 & -0.02 \\ +% & \TMVP & +0.03 & -0.18 & -0.27 & -0.26 & -0.25 \\ +% \hline +% \multirow{\mergedlines}{*}{People} +% & \REF & 35.90 & 34.74 & 33.79 & 33.22 & 32.40 \\ +% & \MERGEONE & -0.06 & -0.04 & -0.06 & -0.04 & -0.05 \\ +% & \TMVP & +0.09 & -0.29 & -0.45 & -0.43 & -0.42 \\ +% \hline +% \multirow{\mergedlines}{*}{Race Horses} +% & \REF & 39.87 & 38.72 & 37.66 & 37.12 & 36.22 \\ +% & \MERGEONE & -0.01 & -0.04 & -0.00 & +0.00 & -0.01 \\ +% & \TMVP & +0.04 & -0.53 & -0.72 & -0.71 & -0.50 \\ +% \hline +% \multirow{\mergedlines}{*}{Tennis} +% & \REF & 40.68 & 39.24 & 37.51 & 36.54 & 35.06 \\ +% & \MERGEONE & -0.02 & -0.07 & -0.02 & -0.01 & -0.04 \\ +% & \TMVP & +0.05 & -0.53 & -0.91 & -1.03 & -0.90 \\ +% \hline +% \multirow{2}{*}{\vbox{\textbf{Average Difference}}} +% & \textbf{\MERGEONE} & \textbf{-0.02} & \textbf{-0.03} & \textbf{-0.04} & \textbf{-0.02} & \textbf{-0.01} \\ +% & \textbf{\TMVP} & \textbf{+0.04} & \textbf{-0.55} & \textbf{-1.06} & \textbf{-1.33} & \textbf{-1.54} \\ +% +% \bottomrule +% \end{tabular} + + +\small +\footnotesize +\newcommand{\nlines}{2} +\newcommand{\colwidth}{1.5cm} +\begin{tabular}{p{2.8cm}|x{2.3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} +\toprule +\multirow{2}{*}{\centering\textbf{Sequence}} & \multirow{2}{*}{\vbox{\centering\textbf{TMVP}}} & \multirow{2}{*}{\vbox{\centering\textbf{Error Free}}} & \multicolumn{3}{c}{\textbf{Packet loss rate}} \\ +& & & \textbf{1\%} & \textbf{5\%} & \textbf{10\%} \\ +\hline +\multirow{\nlines}{*}{\vbox{Basketbal Drill}} +& Enabled & 38.69 & 34.73 & 27.92 & 25.41 \\ +% \cline{2-6} +& Disabled & -0.04 & +1.97 & +3.93 & +3.43 \\ +\hline +\multirow{\nlines}{*}{\vbox{Book Arrival}} +& Enabled & 40.62 & 36.78 & 30.78 & 27.29 \\ +% \cline{2-6} +& Disabled & -0.01 & +1.98 & +3.32 & +3.33 \\ +\hline +\multirow{\nlines}{*}{\vbox{BQSquare}} +& Enabled & 39.64 & 31.99 & 23.45 & 20.95 \\ +% \cline{2-6} +& Disabled & -0.05 & +6.14 & +10.6 & +10.7 \\ +\hline +\multirow{\nlines}{*}{\vbox{Kendo}} +& Enabled & 43.31 & 38.80 & 29.80 & 25.45 \\ +% \cline{2-6} +& Disabled & -0.04 & +2.04 & +4.51 & +4.98 \\ +\hline +\multirow{\nlines}{*}{\vbox{Park Scene}} +& Enabled & 36.89 & 33.78 & 28.09 & 25.77 \\ +% \cline{2-6} +& Disabled & -0.06 & +2.33 & +5.72 & +6.05 \\ +\hline +\multirow{\nlines}{*}{\vbox{Race Horses}} +& Enabled & 37.06 & 29.64 & 22.92 & 20.35 \\ +% \cline{2-6} +& Disabled & -0.07 & +3.72 & +4.78 & +4.19 \\ +\hline +\multirow{\nlines}{*}{\vbox{Tennis}} +& Enabled & 41.61 & 33.74 & 25.73 & 22.68 \\ +% \cline{2-6} +& Disabled & -0.04 & +3.10 & +4.01 & +3.52 \\ +\hline +\multicolumn{2}{c|}{\textbf{Average difference}} & \textbf{-0.04} & \textbf{+3.04} & \textbf{+5.32} & \textbf{+5.17} \\ +\bottomrule +\end{tabular} diff --git a/031_ErrorResilienceEvaluation/TablesSequences.tex b/031_ErrorResilienceEvaluation/TablesSequences.tex new file mode 100644 index 0000000..2b922f0 --- /dev/null +++ b/031_ErrorResilienceEvaluation/TablesSequences.tex @@ -0,0 +1,48 @@ +\begin{table}[t] +\centering +\caption{Test sequences used in the experiments.} +\label{TableTestSequences} +\TABLESVSPACE +\footnotesize +%\begin{tabular}{p{2.8cm}|x{2.2cm}|x{2cm}|m{7cm}} +%\toprule +%\centering \textbf{Sequence} & \textbf{Resolution} & \textbf{SI/TI} & \centering\let\newline\\\arraybackslash\hspace{0pt} \textbf{Description} \\ +%\hline +%Basketball Drill & $832\times480$ 50 fps & 33.4 / 14.4 & High motion with several basketball players \\ \hline +%Book Arrival & $1024\times768$ 30 fps & 28.4 / 21.7 & Low translational motion with two moving persons \\ \hline +%BQSquare & $416\times240$ 60 fps & 63.2 / 11.5 & Moderate outside motion with moving camera capturing from high point \\ \hline +%Bosphorus & $3840\times2160$ 120 fps & 13.4 / 3.8 & Boat shipping at low motion with moderate complex background \\ \hline +%Cactus & $1920\times1080$ 50 fps & 30.5 / 11.4 & Several objects with high details with moderate motion \\ \hline +%Four People & $1280\times720$ 60 fps & 31.3 / 6.9 & Four people talking of the desk and passing objects to each other \\ \hline +%Jockey & $3840\times2160$ 120 fps & 11.5 / 16.2 & High motion with one horse rider \\ \hline +%Kendo & $1024\times768$ 30 fps & 19.6 / 16.1 & Moderate motion with two moving persons, and moving camera \\ \hline +%Kimono & $1920\times1080$ 24 fps & 23.4 / 32.5 & Capture of a women moving in a forest with moderate motion \\ \hline +%% Kristen and Sara & $1280\times720$ 60 fps & 25.6 / 6.30 & Two persons talking to each other with moderate motion \\ \hline +%Park Scene & $1920\times1080$ 24 fps & 31.3 / 11.6 & Moderate motion with cyclists passing across the scene \\ \hline +%People On Street & $2560\times1600$ 24 fps & 29.6 / 9.90 & With point capture of people moving; high motion and texture complexity \\ \hline +%Race Horses & $832\times480$ 30 fps & 43.7 / 24.4 & Moderate motion with several horse riders \\ \hline +%% Tennis & $1920\times1080$ 30 fps & 41.2 / 10.7 & High motion with one moving person in the scene \\ \hline +%Traffic & $2560\times1600$ 30 fps & 28.4 / 22.3 & Elevated camera capturing highway with several cars with moderate speed \\ +%\bottomrule +%\end{tabular} + \begin{tabular}{m{3.5cm}|x{1.9cm}|x{1.4cm}|m{6.8cm}} + \toprule + \textbf{Sequence} & \textbf{SI/TI} & \textbf{Bitrate (Mbps)} & \centering\let\newline\\\arraybackslash\hspace{0pt} \textbf{Description} \\ \hline + {Basketball Drill \newline $832\times480@50Hz$} & 33.4 / 14.4 & 4.50 & High motion with several basket ball players \\ \hline + {Book Arrival \newline $1024\times768@30Hz$} & 28.4 / 21.7 & 1.50 & Moderate translational motion with two moving persons \\ \hline + {Bosphorus \newline $3840\times2160@120Hz$} & 13.4 / 3.8 & 12.00 & Boat shipping at low motion with moderate complex background \\ \hline + {BQSquare \newline $416\times240@60Hz$} & 63.2 / 11.5 & 4.00 & Moderate outside motion with moving camera capturing from high point \\ \hline + {Cactus \newline $1920\times1080@50Hz$} & 30.5 / 11.4 & 20.00 & Several objects with high details with moderate motion \\ \hline + {Four People \newline $1280\times720@60Hz$} & 31.3 / 6.90 & 2.00 & Four people talking of the desk and passing objects to each other \\ \hline + {Jockey \newline $3840\times2160@120Hz$} & 11.5 / 16.2 & 12.00 & High motion with one horse rider \\ \hline + {Kendo \newline $1024\times768@30Hz$} & 19.6 / 16.1 & 0.60 & Moderate motion with two moving persons, and moving camera \\ \hline + {Kimono \newline $1920\times1080@24Hz$} & 23.4 / 32.5 & 8.00 & Capture of a women moving in a forest with moderate motion \\ \hline + {Kristen and Sara \newline $1280\times720@60Hz$} & 25.6 / 6.30 & 15.00 & Two persons talking to each other with moderate motion \\ \hline + {Park Scene \newline $1920\times1080@24Hz$} & 31.3 / 11.6 & 10.00 & Moderate motion with cyclists passing across the scene \\ \hline + {People on Street \newline $2560\times1600@24Hz$} & 40.0 / 25.4 & 12.00 & With point capture of people moving; high motion and texture complexity \\ \hline + {Race Horses \newline $832\times480@30Hz$} & 43.7 / 24.4 & 8.00 & High motion with several horse riders \\ \hline + {Tennis \newline $1920\times1080@30Hz$} & 20.3 / 45.3 & 5.00 & High motion with one moving person in the scene \\ \hline + {Traffic \newline $2560\times1600@30Hz$} & 28.4 / 22.3 & 6.00 & Elevated camera capturing highway with several cars with moderate speed \\ + \bottomrule +\end{tabular} +\end{table} diff --git a/041_ImprovedErrorResilience/Figures/BlockDiagram.pdf b/041_ImprovedErrorResilience/Figures/BlockDiagram.pdf new file mode 100644 index 0000000..24dc992 Binary files /dev/null and b/041_ImprovedErrorResilience/Figures/BlockDiagram.pdf differ diff --git a/041_ImprovedErrorResilience/Figures/BlockDiagram.tikz b/041_ImprovedErrorResilience/Figures/BlockDiagram.tikz new file mode 100644 index 0000000..9179d8d --- /dev/null +++ b/041_ImprovedErrorResilience/Figures/BlockDiagram.tikz @@ -0,0 +1,72 @@ + +\usetikzlibrary{arrows,matrix} +\def\scale{1} +\begin{tikzpicture}[ + font=\footnotesize, + node distance=1.2cm, + scale=\scale, + every node/.style={scale=\scale}, + template/.style={thick, text centered}, + block/.style={template, rectangle, text width=1.3cm, ,inner sep=3, minimum height=.6cm}, + arrow/.style={template, draw, -latex}, + point/.style={template, draw, fill=black, circle,inner sep=0, minimum size=3}] +\tikzstyle{sumBlock}=[template, draw,circle, inner sep=0cm, minimum size=0.1cm] +\tikzstyle{processBlock}=[template, draw, inner sep=0.2cm, minimum size=1cm,text width=1.5cm,fill=\TikZColorFill] +\tikzstyle{methodBlock}=[processBlock, fill=\TikZColorFillProp] +\tikzstyle{blockArrow}=[arrow,draw,-latex] +\tikzstyle{point} = [draw, fill=black, circle,inner sep=1pt] +\coordinate (sumPoint) at (0,0); +\node[sumBlock] (ResidueDiff) at (sumPoint) {$+$}; +\node[left=0.6cm of ResidueDiff] (Input) {\footnotesize $f_t$}; +\path[blockArrow] (Input) -- (ResidueDiff); +\node[processBlock,right=of ResidueDiff] (TQ) {$T + Q$}; +\node[processBlock,right=1.8cm of TQ] (VLC) {Entropy Coding}; +\path[blockArrow] (ResidueDiff) -- (TQ); +\path[blockArrow] (TQ) -- (VLC) node [point,pos=0.4] (p1) {}; +\node[processBlock,below=of p1] (InvTQ) { $Q^{-1} + T^{-1}$}; +\path[blockArrow] (p1) -- (InvTQ); +\node[sumBlock,below=.8cm of InvTQ] (ReconResidue) {$+$}; +\path[blockArrow] (InvTQ) -- (ReconResidue); +\node[processBlock,text width=1.8cm,below=1cm of ResidueDiff] (Pred) {Prediction}; +\path[blockArrow] (Pred) -- node[pos=0.96,right] {-} node [point,pos=0.4] (p2) {} (ResidueDiff); +\path[blockArrow] (p2) -- (p2 -| TQ) |- (ReconResidue); +\node[processBlock,below=.8cm of ReconResidue] (FrameBuffer) {Frame buffer}; +\path[blockArrow] (ReconResidue) -- (FrameBuffer); +\node[processBlock,text width=1.8cm,below=.4cm of Pred] (MotionSearch) {Motion estimation}; +\node[methodBlock,text width=1.8cm] at (MotionSearch |- FrameBuffer) (RefSelection) {Reference picture selection}; +\node[right of=ReconResidue, below of=ReconResidue, node distance=0.3cm, text centered] () {\footnotesize $\hat f_t$}; +\path[blockArrow] (FrameBuffer.180) -- (RefSelection.0 |- FrameBuffer.180 ); +\path[blockArrow] (RefSelection) -- (MotionSearch); +\path[blockArrow] (MotionSearch) -- (Pred); +\node[methodBlock, right of=VLC, node distance=5.3cm] (Mux) {MUX}; +\node[right=1.3cm of Mux] (Output) {}; +\path[blockArrow] (VLC) -- (Mux) node [point,pos=0.5] (p3) {} node [pos=0.2] (p4) {}; +\path[blockArrow] (Mux) -- (Output) node [pos=0.8] (p5) {}; +\node[methodBlock,text width=1.8cm,below=1cm of p3] (Parser) {Stream parser}; +\node[methodBlock,text width=1.8cm,below of=Parser,node distance=2.8cm] (MVSelection) {MV selection \& ranking}; +\node[methodBlock] at (MVSelection -| Mux) (MVCoding) {MV coding}; +\path[blockArrow] (p3) -- (Parser); +\path[blockArrow] (Parser) -- (MVSelection.north) node[pos=0.6,right,text width=1.4cm] () {MV info} ; +\path[blockArrow] (MVSelection) -- (MVCoding); +\path[blockArrow] (MVCoding) -- (Mux) node[pos=0.7,right,scale=1] () {SEI}; + +\newcommand{\upboundary}{40} +\newcommand{\downboundary}{-50} +\node[left=.1cm of Input,yshift=\upboundary] (i0) {}; +\node[yshift=\downboundary] at (FrameBuffer -| i0) (i1) {}; +\path[template,draw,dashed] (i0) -- (i1); + +\node[right=0.1cm of VLC,yshift=\upboundary] (b0) {}; +\node[yshift=\downboundary] at (FrameBuffer -| b0) (b1) {}; +\path[template,draw,dashed] (b0) -- (b1); + +\node[right=0.3cm of Mux,yshift=\upboundary] (o0) {}; +\node[yshift=\downboundary] at (FrameBuffer -| o0) (o1) {}; +\path[template,draw,dashed] (o0) -- (o1); +\path (i1) -- node[midway,yshift=10] () {\footnotesize \textbf{Encoding Stage}} (b1); +\path (b1) -- node[midway,yshift=10] () {\footnotesize \textbf{Streaming Stage}} (o1); +\path (p4) -- (p5) node[pos=0.6,yshift=40, text centered,text width=1.8cm] (HEVCLabel) {\textbf{HEVC Stream}} ; +\path[template,draw, -stealth] (HEVCLabel.-15) -- (p5); +\path[template,draw, -stealth] (HEVCLabel.195) -- (p4); + +\end{tikzpicture} diff --git a/041_ImprovedErrorResilience/Figures/ExponentialWeight.pdf b/041_ImprovedErrorResilience/Figures/ExponentialWeight.pdf new file mode 100644 index 0000000..d915b66 Binary files /dev/null and b/041_ImprovedErrorResilience/Figures/ExponentialWeight.pdf differ diff --git a/041_ImprovedErrorResilience/Figures/MVDependencies.tikz b/041_ImprovedErrorResilience/Figures/MVDependencies.tikz new file mode 100644 index 0000000..efaa9d3 --- /dev/null +++ b/041_ImprovedErrorResilience/Figures/MVDependencies.tikz @@ -0,0 +1,98 @@ +\def\MVDEPA{\mbox{(a)}\xspace} +\def\MVDEPB{\mbox{(b)}\xspace} +\def\MVDEPC{\mbox{(c)}\xspace} + +\newcommand{\scale}{1} +\newcommand{\xfshift}{4.5cm} +\begin{tikzpicture}[scale=\scale] +\tikzstyle{block}=[thick] +\tikzstyle{refmv}=[-latex,very thick,scale=1] +\tikzstyle{MvLabel}=[right, font=\footnotesize] +\newcommand{\DrawFrameMVDependency} +{ + \newcommand{\fsize}{4cm} + \fill[fill=white,fill opacity=.6] (0,0) rectangle (\fsize,\fsize); % Opacity + \draw[thick] (0,0) rectangle (\fsize,\fsize); + \foreach \x in {1,...,4} + \draw[loosely dashed] (\x,0) -- ++(0,\fsize); + \foreach \y in {1,...,4} + \draw[loosely dashed] (0,\y) -- ++(\fsize,0); +} +% Frame t +\begin{scope}[yshift=2*\TikZyfshift, xshift=2*\xfshift, style={yslant=\TikZyslant,xscale=\TikZxscale}] + \coordinate (T0B1) at (1,3); + \coordinate (T0B20) at (2,3); + \coordinate (T0B21) at (2,2.5); + \coordinate (T0B22) at (2.5,3); + \coordinate (T0B3) at (3,3); + \coordinate (T0B4) at (1,2); + \coordinate (T0B5) at (2,1); + \coordinate (T0B6) at (3,1); + \DrawFrameMVDependency + \draw[block] (T0B1) +(0,0) rectangle ++(1,-1); + \draw[block] (T0B20) rectangle ++(0.5,-0.5); + \draw[block] (T0B21) rectangle ++(0.5,-0.5); + \draw[block] (T0B22) rectangle ++(0.5,-1); + \draw[block] (T0B3) +(0,0) rectangle ++(1,-1); + \draw[block] (T0B5) +(0,0) rectangle ++(1,-1); + \draw[block] (T0B6) +(0,0) rectangle ++(1,-1); + \draw[refmv] (T0B20)+(0.25,-0.25) -- ++(-0.5,-0.5); + \draw[refmv] (T0B21)+(0.25,-0.25) -- ++(-0.5,0); + \draw[refmv] (T0B22)+(0.25,-0.5) -- ++(-0.25,-0.25); + \draw[refmv] (T0B3)+(0.5,-0.5) -- ++(-0.25,-0.5); + \draw[refmv] (T0B6)+(0.5,-0.5) -- ++(-0.5,-0.5); + \node[font=\scriptsize] () at (1.35,2.75) {\MVDEPA}; + \node[font=\scriptsize] () at (2.35,0.75) {\MVDEPB}; + \node at (4cm/2,4.7cm) { $f_{0}$ }; +\end{scope} +% Frame t-1 +\begin{scope}[ yshift=2*\TikZyfshift, xshift=1*\xfshift,style={yslant=\TikZyslant,xscale=\TikZxscale}] + \coordinate (MVRef1B1) at (1.5,2.5); + \draw[refmv,dashed] (T0B1)++(0.5,-0.5) to (MVRef1B1); + \draw[refmv,-,shorten >=35] (T0B1)++(0.5,-0.5) to (MVRef1B1); + \coordinate (MVRef1B2) at (2.75,0.25); + \draw[refmv,dashed] (T0B5)++(0.5,-0.5) to (MVRef1B2); + \draw[refmv,-,shorten >=18] (T0B5)++(0.5,-0.5) to (MVRef1B2); + \DrawFrameMVDependency + \coordinate (T0B1) at (1,3); + \draw[block] (T0B1) rectangle ++(1,-1); + \coordinate (T0B2) at (2,3); + \draw[block] (T0B2) rectangle ++(0.5,-0.5); + \draw[refmv] (T0B2)+(0.25,-0.25) -- ++(-0.5,-0.5); + \coordinate (T0B3) at (2.5,3); + \draw[refmv] (T0B3)+(0.25,-0.25) -- ++(-0.25,-0.25); + \draw[block] (T0B3) rectangle ++(0.5,-0.5); + \coordinate (T0B4) at (2,2.5); + \draw[block] (T0B4) rectangle ++(0.5,-0.5); + \draw[refmv] (T0B4)+(0.25,-0.25) -- ++(0.25,0.25); + \coordinate (T0B5) at (2.5,2.5); + \draw[block] (T0B5) rectangle ++(0.5,-0.5); + \draw[refmv] (T0B5)+(0.25,-0.25) -- ++(-0.25,-0.25); + \coordinate (T0B6) at (1,2); + \draw[block] (T0B6) +(0,0) rectangle ++(0.5,-0.5); + \draw[refmv] (T0B6)+(0.25,-0.25) -- ++(0.5,0.5); + \draw[block] (T0B6) ++(0.5,0) rectangle ++(0.5,-0.5); + \draw[refmv] (T0B6)+(0.75,-0.25) -- ++(0.25,-0.25); + \coordinate (T0B7) at (2.5,0.5); + \draw[block] (T0B7) rectangle ++(0.5,-0.5); + \node[font=\scriptsize] () at (1.35,2.75) {\MVDEPC}; + \node at (4cm/2,4.7cm) { $f_{-1}$ }; +\end{scope} +% Frame t-2 +\begin{scope}[ yshift=1*\TikZyfshift, xshift=0*\xfshift, style={yslant=\TikZyslant,xscale=\TikZxscale}] + %Define Points + \coordinate (TM1B1) at (1,3); + \coordinate (MVRef1B1) at (1.5,2.5); + % Draw Dependencies + \draw[refmv,dashed] (T0B1)++(0.5,-0.5) to (MVRef1B1); + \draw[refmv,-,shorten >=35] (T0B1)++(0.5,-0.5) to (MVRef1B1); + \DrawFrameMVDependency + % Draw block + \draw[block] (TM1B1) rectangle ++(1,-1); + \node at (4cm/2,4.7cm) { $f_{-2}$ }; +\end{scope} +\begin{scope}[xshift=0cm,yshift=-1.2cm] + \draw[refmv] (0,0) to (1.2,0); + \fill[black] (1.25,0) node[MvLabel]{MV dependency}; +\end{scope} +\end{tikzpicture} diff --git a/041_ImprovedErrorResilience/Figures/RefPicSelection.tikz b/041_ImprovedErrorResilience/Figures/RefPicSelection.tikz new file mode 100644 index 0000000..0c86d80 --- /dev/null +++ b/041_ImprovedErrorResilience/Figures/RefPicSelection.tikz @@ -0,0 +1,78 @@ +\newcommand{\scale}{1} +\newcommand{\xfshift}{4cm} +\begin{tikzpicture} +\begin{scope}[scale=\scale] +\tikzstyle{refmv_line}=[thick,draw=black] +\tikzstyle{refmv}=[refmv_line,-latex,densely dashed] +\tikzstyle{refmv_0}=[refmv,] +\tikzstyle{refmv_1}=[refmv_0] +\newcommand{\DrawFrame}[2] +{ + \fill[fill=#1,fill opacity=.6] (0,0) rectangle (#2+1,#2+1); % Opacity + \draw[thick] (0,0) rectangle (#2+1,#2+1); + \foreach \x in {1,...,#2} + \draw (\x,0) -- ++(0,#2+1); + \foreach \y in {1,...,#2} + \draw (0,\y) -- ++(#2+1,0); +} +% Frame t +\begin{scope}[yshift=2*\TikZyfshift, xshift=2*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] + \coordinate (T0B1) at (0.5-0.25,2.5+0.25); + \coordinate (T0B2) at (1.5,2.5); + \coordinate (T0B3) at (2.5,2.5); + \coordinate (T0B4) at (0.5,1.5); + \coordinate (T0B5) at (1.5,1.5); + \coordinate (T0B6) at (2.5,1.5); + \coordinate (T0B7) at (0.5,0.5); + \coordinate (T0B8) at (1.5,0.5); + \coordinate (T0B9) at (2.5,0.5); + \DrawFrame{white}{2} + \foreach \x in {0,2} + \foreach \y in {0,2} + \draw[fill=gray!40,fill opacity=0.6] (\x,\y) +(0,0) rectangle ++(1,1); + \draw[fill=gray!40,fill opacity=0.6] (1,1) +(0,0) rectangle ++(1,1); + \begin{scope}[yshift=2cm, xshift=0,scale=0.5] + \foreach \x in {0,1} + \foreach \y in {0,1} + \draw (\x,\y) +(0,0) rectangle ++(1,1); + \begin{scope}[ yshift=1cm, xshift=1cm,scale=0.5] + \foreach \x in {0,1} + \foreach \y in {0,1} + \draw (\x,\y) +(0,0) rectangle ++(1,1); + \end{scope} + \end{scope} + \node[font=\footnotesize, text width=3cm, text centered] () at (1.5,-0.6) {Current frame ($C$)}; + \node[scale=0.6, text width=2cm, text centered] (CTULabel) at (2.5,0.5) {CTU}; +\end{scope} +% Frame t-1 +\begin{scope}[ yshift=1*\TikZyfshift, xshift=1*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] + \coordinate (TM1B1) at (0.5,2.5); + \coordinate (TM1B3) at (2.5-0.3,2.5); + \coordinate (TM1B5) at (1.5,1.5); + % Draw Motion vectors + \draw[refmv] (T0B1) to (TM1B1); + \draw[refmv_line,shorten >=28] (T0B1) to (TM1B1); + \draw[refmv] (T0B3) to (TM1B3); + \draw[refmv_line,shorten >=15] (T0B3) to (TM1B3); + \draw[refmv] (T0B5) to (TM1B5); + \draw[refmv_line,shorten >=25] (T0B5) to (TM1B5); + \DrawFrame{gray!40}{2} + \node[font=\footnotesize, text width=3cm, text centered] () at (1.5,-0.6) {Ref. frame 1 ($r_1$)}; +\end{scope} +% Frame t-2 +\begin{scope}[ yshift=0*\TikZyfshift, xshift=0*\xfshift, yslant=\TikZyslant,xscale=\TikZxscale] + \coordinate (TM2B4) at (0.5-0.2,1.5+0.2); + \coordinate (TM2B6) at (2.5-0.1,1.5-0.1); + \coordinate (TM2B8) at (1.5-0.3,0.5+0.3); + \coordinate (TM2) at (1.5,1.5); + \draw[refmv] (T0B4) to (TM2B4); + \draw[refmv_line,shorten >=40,shorten <=79] (T0B4) to (TM2B4); + \draw[refmv_line,shorten >=117] (T0B4) to (TM2B4); + \draw[refmv] (T0B8) to (TM2B8); + \draw[refmv_line,shorten >=27,shorten <=80] (T0B8) to (TM2B8); + \draw[refmv_line,shorten >=105] (T0B8) to (TM2B8); + \DrawFrame{white}{2} + \node[font=\footnotesize, text width=3cm, text centered] () at (1.5,-0.6) {Ref. frame 2 ($r_2$)}; +\end{scope} +\end{scope} +\end{tikzpicture} diff --git a/041_ImprovedErrorResilience/ImprovedErrorResilience.tex b/041_ImprovedErrorResilience/ImprovedErrorResilience.tex new file mode 100644 index 0000000..bf6359a --- /dev/null +++ b/041_ImprovedErrorResilience/ImprovedErrorResilience.tex @@ -0,0 +1,454 @@ +% Figure folders +\newcommand{\ImprovedErrorResilience}{./041_ImprovedErrorResilience} +\graphicspath{{\ImprovedErrorResilience/Results/}} + + +\chapter{A Two-stage Approach for Robust Coding and Streaming} +% \chapter{Error Resilience Techniques for \ac{hevc} Standard} +%\chapter{Two-stage Robust Video Coding} +\label{Chapter:ErrorResilienceTechniques} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Introduction} + +\glsreset{mv} +\glsreset{rd} + +The increased compression ratios achieved by the \ac{hevc} standard lead to reduced robustness of coded streams, with increased susceptibility to network errors and consequent video quality degradation, as shown in Chapter~\ref{Chapter:HEVCErrorResilience}. +Therefore, this chapter proposes a two-stage approach method to improve the error robustness of \ac{hevc} streaming, by reducing temporal error propagation in case of frame loss. +The prediction mismatch that occurs at the decoder after frame loss is reduced through the following two stages: (i) at the encoding stage, the reference pictures are dynamically selected based on constraining conditions and Lagrangian optimisation, which distributes the use of reference pictures, by reducing the number of \acs{pu} that depend on a single reference; (ii) at the streaming stage, a \ac{mv} prioritisation algorithm, based on spatial dependencies, selects an optimal sub-set of \acs{mv} to be transmitted, redundantly, as side information to reduce mismatched \ac{mv} predictions at the decoder. + +This chapter is organized as follows. Section~\ref{Section4:Problem} presents the problem formulation and motivation for the method proposed in this chapter. Section~\ref{Section4:Introduction} provides an overview of the proposed method, describing the functional blocks that compose the two-stage approach for robust coding and streaming. Section~\ref{Section4:RefPicSel} describes the encoding stage of the proposed method, that is used to selected the optimal reference picture for inter-prediction. Section~\ref{Section4:MVSelection} describes the streaming stage used to select a sub-set of \acs{mv} to reduce the mismatch motion predictions. Section~\ref{Section4:Performance} provides a performance evaluation of the proposed method, illustrating the advantages of each stage and how they are combined together to achieve superior error robustness in \ac{hevc} standard. Finally, Section~\ref{Section4:Summary} concludes this chapter. + +\clearpage +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Problem formulation} +\label{Section4:Problem} + + +The error robustness characteristics of \ac{hevc} analysed in Chapter~\ref{Chapter:HEVCErrorResilience} confirmed that, in general, \ac{hevc} presents poor error resilience performance. Thus, \ac{hevc} bitstreams subject to transmission losses result in significant degradation of both objective and subjective quality. This is mostly due to the strong decoding dependencies imposed by the highly complex prediction modes of \ac{hevc}, which are selected according to a \ac{rd} optimisation criteria, assuming error-free transmission~\cite{ROPE2007}. Therefore, high compression efficiency is achieved, but resilience to transmission errors is penalised. This compromises the error robustness of video decoders which leads to poor performance under error conditions. Moreover, the use of advanced \ac{mv} prediction in \ac{hevc} \cite{MotionCodingHEVC2011} also contributes to increase the spatial and temporal dependencies to a great extent. Although the spatial MV candidates are used more often, temporally predicted \acs{mv} are responsible for a greater impact on error propagation. + +\input{./041_ImprovedErrorResilience/Results/AverageRefPicUsage.tex} + +As mentioned before the video decoder may rely of a distributed use of references frames to achieve higher video quality, by experienced lower error propagation and by optimising the error concealment algorithms to extract information from correctly decoded reference frames. In order to evaluate how the reference \ac{rd} optimisation presented in the reference \ac{hevc} encoder different test sequences were encoded using a configuration with two reference frames, referred to as $r_1$ and $r_2$. Figure~\ref{FIG4:ReferenceUsageRatio} illustrates the average usage ratio of reference $r_1$ for different sequences. The remaining corresponds to those cases where reference $r_2$ were selected. +Results in Figure~\ref{FIG4:ReferenceUsageRatio} shows that the reference $r_1$ is selected in most of the cases when two references are available. This is mainly true for sequences with higher motion. For those cases with lower motion but high spatial complexity, e.g., BQSquare sequence (see \ac{ti} in Table~\ref{TableTestSequences}), both frames are selected for temporal prediction, with $r_1$ being used for 58\% of cases and $r_2$ for 42\%. This reveals that even a configuration with two references the encoder will mainly use the temporally closer one as the reference for inter-frame predictions. This will lead to a higher number of mismatch predictions in case of frame loss and consequently to higher error propagation. + +As mentioned in Chapter~\ref{Chapter:HEVCErrorResilience}, another technique that contributes to increased error propagation is the use of advanced \ac{mv} prediction in \ac{hevc} \cite{MotionCodingHEVC2011}. This contributes to increase the spatial and temporal dependencies to a great extent. Although the spatial \ac{mv} candidates are used more often, temporally predicted \acs{mv} are responsible for a greater impact on error propagation, as they propagate errors in the \ac{mv} decoding to across different time instances. + +In general, increased error robustness in video streaming can be jointly achieved at two different stages: (i) while encoding live video, by selecting robust coding modes and (ii) after encoding, by adding robust features to the compressed streams. While in the former this is accomplished by optimising coding parameters and decisions to increase robustness, in the latter some type of compressed domain processing must be used to increase error robustness of pre-encoded streams, e.g., error resilient transcoding. Inevitably, in both cases this is achieved at the expense of some loss in coding efficiency, therefore the challenge is to find an optimum trade-off between robustness and bitrate overhead. In this chapter a two-fold approach is used to overcome the two problems mentioned above. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Proposed architecture} +\label{Section4:Introduction} + +The goal of the proposed method is to reduce the error propagation in decoded video in case of frame loss. This is accomplished by constraining inter-frame prediction dependencies, in order to minimise the distortion $D_t$ given by +\begin{equation} +D_t = \tilde f_t - \hat f_t, \text{when } \tilde f_{t-1} \neq \hat f_{t-1}, +\label{eq1} +\end{equation} +where $\hat f_t$ and $\tilde f_t$ are the encoded and decoded frames, respectively, at instant $t$. + +Figure \ref{fig:proposedMethod} illustrates the block diagram of the two-stage method proposed in this work. +At the encoding stage, a reference frame selection mechanism is used for each \ac{pu} within a coding tree unit (\ac{ctu}) forcing the use of different reference frames in adjacent \acs{ctu}. The decision criterion contributes to reduce the amount of predictions from a single reference frame, and forces to interleave different reference frames along all encoded \acs{pu}, based on a \ac{rd} optimised criterion. +At the streaming stage, the coded stream is parsed to extract the \ac{mv} information, which allows ranking \acs{mv} according to a relevance criterion. Then, a sub-set of these \acs{mv} is selected to be transmitted as side information. The amount of selected \acs{mv} is not fixed and can be optimised for different content and end-network characteristics. +Note that the streaming stage is an independent and complementary subsystem to the encoder. Thus, it is able to operate at network nodes, including the streaming server on pre-encoded streams. + +\begin{figure}[tb] +\centering +% \includegraphics[width=0.85\columnwidth]{041_ImprovedErrorResilience/Figures/BlockDiagram} +\input{041_ImprovedErrorResilience/Figures/BlockDiagram.tikz} +\caption{Functional blocks of the proposed method (\textsl{gray} blocks) in the context of the \ac{hevc} coding and streaming.} +\label{fig:proposedMethod} +\end{figure} + +In case of frame loss these two stages have the common objective of helping the decoder to recover from mismatched predictions at the expense of a small increase in bitrate. +This is equivalent to reducing the error propagation by using two techniques: (i) reducing the mismatch of inter-frame predictions, by encoding each \ac{pu} from a different reference frame and (ii) reducing the mismatch of \ac{mv} predictions by sending the most important \acs{mv} as side information. These are the two stages of the proposed method described in the following sections. + + + +\section{Encoding stage: Reference frame selection} +\label{Section4:RefPicSel} + + +The reference picture selection mechanism attempts to uniformly distribute the use of different reference pictures across the \ac{ctu} of a coded video frame. In case of frame loss, the error propagation in the decoder is reduced because the number of \acs{ctu} not predicted from the lost frame decreases and thus error concealment benefits from more data correctly decoded. Note that standard compliance is maintained. +The underlying principle of this method is to minimise the probability of using the same reference picture for neighbouring \acs{ctu}, which tends to result in a checkerboard structure, where a different reference frame is used for predicting every other \ac{ctu} within the current frame. +For instance, in a coding structure with two reference frames, as shown in Figure~\ref{fig:checkerboardRefSelection}, the reference frames $r_{1}$ and $r_{2}$ are used to predict the same number of \acs{ctu} in current frame~($C$). +If each reference frame is used to predict half of the pixels in the current frame, when any of these references is lost, the subsequent predicted frame is only partially distorted by mismatched predictions. +This is because only half of its predictions depend on the missing reference frame, which limits the overall error propagation. +Moreover, since the motion information of the predicted frame is encoded with reference not only to the missing frame but also to the past ones, those \acs{mv} that cross over a missing reference frame, pointing to the previous ones, are still useful for reconstructing correct \acs{mv} and also for error concealment. + +In the limit, a straightforward implementation of the previous example leads to a simple interleaved uniform distribution of the reference frames over all \acs{ctu} in the current frame, forcing the checkerboard structure shown in Figure \ref{fig:checkerboardRefSelection}. In the case of two reference frames, 50\% of the \acs{ctu} would use one reference frame, while the other 50\% would use the other reference. +However, such solution would not take into consideration coding efficiency, since the selection process of the reference frame would not be \ac{rd} optimised for each \ac{pu} within a \ac{ctu}. + +Therefore, a more efficient approach to improve error robustness was devised to dynamically choose the coding mode for each \ac{pu}, that ensures a balanced use of all possible reference frames $F=\{r_1,r_2,...r_S\}$ across the \acs{ctu} of the current frame $C$, also optimising the \ac{rd} cost. +This leads to the overall goal of finding the optimum R-D coding mode for each \ac{pu}, as the one which tends to minimise the standard deviation of the reference frames use count $\mathbf{N}_C(r_i)$, considering all encoded \acs{pu} of the current frame $C$, as follows: +\begin{equation} +\sigma_C=std\Big(\mathbf{N}_C(r_i) \Big), i=1, 2,\cdots S. +\label{eq:StdX} +\end{equation} +In this case $\sigma_C$ measures the deviation from the checkerboard solution, where $\sigma_C = 0$, since all possible reference frames $r_i\in F$ are uniformly used throughout the \acs{pu} of $C$. + +\begin{figure}[t] +\centering +\input{\ImprovedErrorResilience/Figures/RefPicSelection.tikz} +\caption{Example of the reference picture selection applied to the frame $C$ based on the checkerboard at \ac{ctu} level (\acs{pu} represented by small partitions).} +\label{fig:checkerboardRefSelection} +\end{figure} + +To attain the goal defined above, the best coding mode for each \ac{pu} is chosen through a Lagrangian \ac{rd} optimisation that tends to minimise $\sigma_C$. +This is done by penalising (or benefiting) the \ac{rd} cost of those coding modes which intend to use reference frames that have already been used more (or less) frequently than their average use in previous \acs{pu} of the current frame. +For each \ac{pu}, this mechanism has the effect of reducing (or increasing) the probability of using each possible reference frame $r_i\in F$, according to the number of previous \acs{pu} that have used each one. +Such approach allows to select the reference frames, by independently processing each \ac{pu} in order to minimise the global standard deviation given by~Equation~(\ref{eq:StdX}). Thus, the optimum coding mode $\phi^*$, selected from a set of possible coding modes $M$, is derived based on the following Lagrangian optimisation: +\begin{align} +\phi^* &= arg \min_{\phi \in M} J(\phi) \\ +J(\phi) &= \Big( D(\phi) + \lambda \times R(\phi) \Big) \times e^{W(r_{\phi})} +\label{eq:Lagrangian} +\end{align} +where $D(\phi)$ and $R(\phi)$ are the distortion and rate, respectively, associated with the coding mode $\phi$. +The exponential weighting factor $e^{W(r_{\phi})}$ is used to penalise (or benefit) the Lagrangian cost of a given coding mode $\phi$ associated to reference frame $r_\phi$. +$W(r_{\phi})$ is obtained according to the global and local deviation between the number of times that $r_\phi$ was used and the average use of all $r_i \in F$ across the \acs{pu} of current frame $C$, as follows, +\begin{equation}\label{eq:KFactor} +W(r_{\phi}) = \Big( \Delta_G(r_{\phi}) + \Delta_L(r_{\phi}) \Big)\times 2^{-T_{ID}} \times \gamma +\end{equation} +where $T_{ID}$ is given by the temporal hierarchy in \ac{hevc}, and $\gamma$ is a parameter to control the slope of ${W(r_\phi)}$. +The global and local deviations, $\Delta_G (r_\phi)$ and $\Delta_L (r_\phi)$, respectively, are computed as follows, +\begin{equation}\label{eq:KFactorXValue} +\Delta_\zeta(r_{\phi}) = \mathbf{N}_\zeta(r_\phi) - \frac{1}{S} \sum_{i=1}^{S}\mathbf{N}_\zeta(r_i), \hspace{2mm} \zeta=G,L. +\end{equation} +For $\zeta=G$, function $\Delta_G(r_\phi)$ measures the difference between the number of times that reference frame $r_\phi$ was used and the global average use of all $r_i \in F$ in the \acs{pu} of $C$ that were encoded before the current \ac{pu}. $\mathbf{N}_G(r_i)$ is the number of times that each reference frame $r_i$ was used across all previously encoded \acs{pu}. For $\zeta=L$, function $\Delta_L(r_\phi)$ only considers the three top-left neighbouring \acs{ctu} for counting the number of times each reference frame was used, which is expressed by $\mathbf{N}_L(r_i)$. Overall, Equation (\ref{eq:KFactorXValue}) is used to adjust the penalty weight defined by Equation (\ref{eq:KFactor}) in order to achieve an approximately uniform use of all possible reference frames. This corresponds to the general objective of minimising the values of $\Delta_\zeta(r_{\phi})$. +The optimum is reached when all reference frames are used to encode the same amount of \acs{pu} (i.e., $\Delta_G(r_{\phi})=\Delta_L(r_{\phi})=0$), and the original Lagrangian cost is not affected, i.e., $e^{W(r_{\phi})}=1$ in Equation (\ref{eq:Lagrangian}). +Note that an exponential cost increase is used in Equation (\ref{eq:Lagrangian}) in order to increase the penalty associated to any reference frame that tends to be used much more often than the average. + +\begin{figure}[t] +\centering +\includegraphics[width=0.8\columnwidth]{\ImprovedErrorResilience/Figures/ExponentialWeight} +\caption{Exponential weight as a function of the global and local deviations} +\label{fig:weightfct} +\end{figure} + +In Equation (\ref{eq:KFactor}), the $\gamma$ constant controls the slope of $W(r_\phi)$, leading to higher weights as the value of $\gamma$ increases. +On the one hand a very high value for $\gamma$ leads to the checkerboard pattern in the reference picture usage, as previously described. +On the other hand, a low value of $\gamma$ reduces the importance of the choice of the reference frames in the Lagrangian cost. +Therefore, the impact of the proposed method on the coding efficiency can be controlled through $\gamma$. Another way of controlling the impact of the proposed method is the use of the $T_{ID}$. This allows using lower weights in higher temporal layers, since such frames cannot be used as references for others with lower temporal ID, thus having less impact on error propagation. +Therefore, there is a reduction of the impact of the proposed method when hierarchical coding configuration is adopted without compromising the error robustness. + +As an example, Figure~\ref{fig:weightfct} illustrates the exponential weight used in the proposed method for different values of $\gamma$ and $T_{ID}=0$. The horizontal axis represents the sum of $\Delta_G(r_\phi)$~plus~$\Delta_L(r_\phi)$. +For instance, for $\gamma=5$, when a reference frame $r_\phi$ is used to predict 3 more \acs{ctu} ($\Delta_G(r_\phi) + \Delta_L(r_\phi)=3$) than the other reference frames (all \acs{pu} within those \acs{ctu} use the same reference), then the Lagrangian cost is increased by $50\%$ ($e^{W(r_{\phi})}\approx1.5$). + + +\begin{figure*}[tb] + \centering + \newcommand{\figsize}{0.22\textwidth} + \resizebox{\textwidth}{!}{ + \begin{tabular}{y{.24\textwidth}y{.24\textwidth}y{.24\textwidth}y{.24\textwidth}} + \small Original frame & \small \REF & \small \CHKB & \small \EXP \\ + \includegraphics[width=\figsize]{SelectionExample/ORG} & + \includegraphics[width=\figsize]{SelectionExample/REF} & + \includegraphics[width=\figsize]{SelectionExample/CHKB} & + \includegraphics[width=\figsize]{SelectionExample/EXP} \\ + & \footnotesize $r_{1}$: $61.8\%$ | $r_{2}$: $25.7\%$ & \footnotesize $r_{1}$: $42.2\%$ | $r_{2}$: $39.3\%$ & \footnotesize $r_{1}$: $45.22\%$ | $r_{2}$: $43.2\%$ \\ + \end{tabular} + } + \caption{Example of the partitions and references pictures selected using the different encoding methods (the dark gray squares indicate that the \ac{mv} is using the reference $r_{1}$, light~gray squares corresponds to the reference $r_{2}$ and unfilled regions are intra-coded blocks).} + \label{fig:RefPicSelectionExample} + \FIGSVSPACE +\end{figure*} + +Figure \ref{fig:RefPicSelectionExample} illustrates the distribution of reference frames achieved by the proposed method (\EXP) across different \acs{pu}. This result is compared against a fixed checkerboard approach (\CHKB) and the reference \ac{hevc} (\REF). The dark-gray and light-gray squares indicate the reference frame in use, either $r_1$ or $r_2$, respectively. The white squares indicate intra-coded \acs{pu}. Below each method, the percentages of the image areas predicted by each reference frame are shown. As shown in this figure, the proposed method (\EXP) is able to distribute the use of the reference frames closely to the \CHKB method, i.e., $r_2$ is used as reference for approximately the same amount of \acs{pu} as $r_1$ (see use percentages of the predicted image). However, the predicted regions in the proposed method do not exhibit the same regular pattern as in \CHKB method. + +It is noticeable in the figure that for several \acs{ctu} the \EXP method is able to keep the same partitions and reference frames when compared to the optimal encoding (\REF). This can be observed, for example in the small partitions on the left, where the \EXP method uses the same reference frame as \REF.Thus, \EXP is expected to achieve better coding efficiency than \CHKB, since more optimal coding modes may be chosen. + +Summarising, the underlying principle of Equations~(\ref{eq:Lagrangian})~and~(\ref{eq:KFactor}) is that the cost of using a given reference frame increases with the number of \acs{pu} using it for prediction. However, the reference frames are not hard-selected without taking into account the R-D cost. In general, it is more difficult to reach a uniform use of reference frames in complex sequences than simpler ones. +Note that standard compatibility is maintained, since the proposed optimisation method does not require any syntax change. Moreover, as the proposed approach is applied at the Lagrangian computation already present in the \ac{hevc} optimisation, the complexity of the encoder is not significantly affected. + + +\section{Streaming stage: \ac{mv} selection} +\label{Section4:MVSelection} + +Since \ac{hevc} uses differential and predictive \ac{mv} coding, whenever a \ac{mv} is lost the subsequent ones will also be affected, until a refresh point is reached to break the dependency chain. This increases the error propagation whenever the temporal \ac{mv} candidate is used in the Merge Mode. +To increase the number of possible refresh points and improve error robustness, a reduced number of \acs{mv} are selected to be transmitted as side information. The selection of these redundant \acs{mv} is based on a trade-off that attempts to maximise the image area covered by such \acs{mv} using the lowest number of bits to encode them. + +\def\MVDEPA{\mbox{(a)}\xspace} +\def\MVDEPB{\mbox{(b)}\xspace} +\def\MVDEPC{\mbox{(c)}\xspace} + +Figure~\ref{fig:mvdependencies} illustrates how \acs{mv} have different importance in error propagation. +In this figure, each block corresponds to a \ac{pu}, which has an associated \ac{mv}. The arrows point to the \ac{pu} containing the \ac{mv} used as predictor. In the current frame $f_0$, two \acs{pu} have temporally dependent \acs{mv}. +While the \ac{mv} of \ac{pu} \MVDEPA is used to predict a total of four other \acs{mv}, the \ac{mv} of \ac{pu} \MVDEPB is only used to predict one single \ac{mv}. Thus, in case of data loss, the \ac{mv} of \ac{pu} \MVDEPA has more impact on error propagation and quality degradation than the \ac{mv} of \ac{pu} \MVDEPB, because of its higher number of dependent \acs{mv}. +For the this reason, in the \ac{mv} selection process, the proposed method assigns higher importance to the \ac{mv} of \ac{pu} \MVDEPA than \ac{pu} \MVDEPB. Moreover in Figure~\ref{fig:mvdependencies}, it can be observed that although the number of \acs{mv} predicted from the \acs{mv} of \ac{pu} \MVDEPC is higher than \ac{pu} \MVDEPA (6 and 4 \acs{mv}, respectively), +the image area dependent from the former is smaller that the latter. This is due to the variable size of \acs{pu}. +Therefore, the image area covered by all dependent \acs{pu} is separately considered in the \ac{mv} selection process. + +\begin{figure}[tb] + \centering + \input{\ImprovedErrorResilience/Figures/MVDependencies.tikz} + \caption{Example of temporal and spatial \ac{mv} dependencies in \ac{hevc}.} + \label{fig:mvdependencies} +\end{figure} + + +Based on the evidence mentioned above, the best \acs{mv} to be encoded redundantly as side information for each frame are selected through the following procedure: +\begin{enumerate} +\item +the \acs{mv} with temporal dependencies are firstly selected as the most important set: $V$ = \{\emph{\ac{mv}: \ac{mv} is temporally dependent from others}\}. +\item +the $MVs \in V$ are ranked according to the number of spatial dependencies associated to each one. +\item +the optimal sub-set $U \subset V$ is found as the best trade-off between the image area covered by the dependent \acs{pu} and bitrate overhead, i.e., the one that maximises the cost function in Equation (\ref{eq:MVSelectOpt}). +\end{enumerate} + +Considering $V$ the ordered set of \acs{mv} encoded for the current frame, i.e., $V=\{MV_1, MV_2, \cdots MV_s\}$ and $U \subset V$ always starting in the first vector, i.e., \mbox{$U=\{MV_1, MV_2, \cdots MV_n\}$} with $n\leq s$, the best set of redundant \acs{mv} ($U^*)$ is selected based on the optimisation procedure given by Equation (\ref{eq:MVSelectOpt}). Note that any value of $n$ corresponds to a unique subset $U$. + +\begin{equation}\label{eq:MVSelectOpt} +\newcommand{\argmax}{\operatornamewithlimits{arg\ max}} +n^* = \argmax_{n\leq s} \left\{ \sum_{i=1}^n A_{D}(i) - \alpha \frac{t_C-t_I}{T_{IDR}} R_U \right\}, +\end{equation} +where $R_U$ is the amount of bits required to encode the $n$ \acs{mv} of $U$ and $A_{D}(i)$ is the total area of the \acs{pu} whose \acs{mv} are dependently encoded from $\ac{mv}_i$: +\begin{equation} +A_{D}(i) = \sum_{j=1}^{DV_i} ( w_j\times h_j ),\hspace{2mm} i = 1, 2 \cdots s, +\end{equation} +where $DV_i$ is the number of \acs{mv} dependent from $MV_i$ and $w_j$ and $h_j$ are the width and height of the corresponding \ac{pu}. +The weighting factor used to control the amount of redundancy $R_U$ is used to increase the cost of the overhead rate as the duration of error propagation decreases. +It is given by the ratio between the distance from the current frame ($t_c$) to the next \ac{idr} ($t_I$) and the \ac{idr} period ($T_{IDR}$), combined with the $\alpha$ parameter. Therefore, for higher values of $\alpha$, less redundant \acs{mv} are selected. + +As pointed out before, this streaming stage may be implemented at any point of the network, to increase robustness without fully decoding the stream. Note that the parsing operation represented in Figure~\ref{fig:proposedMethod} does not decode the video frames, introducing only one frame delay because the \ac{mv} selection is performed on a frame-by-frame basis. + +To maintain compatibility with the \ac{hevc} standard, the redundant motion information is transmitted using the \ac{sei} \ac{nal} units \cite{HEVCHighLevelReferenceDez2012}. Thus, this information can be multiplexed into the coded bitstream without affecting the compressed video. +The subset of selected \acs{mv} is independently encoded, ensuring that no reference information is needed in the decoder to properly decode the redundant \acs{mv}. In this work arithmetic coding~\cite{ArithmeticCodingJun1994} was used for the redundant \acs{mv}, but any other entropy coding method can be used. + + +\section{Performance evaluation} +\label{Section4:Performance} + +In this section the robustness of the proposed method is evaluated and discussed for two cases: (i) only using the encoding stage and (ii) using the streaming stage in addition to the encoding stage. In the former case, identified as ``\EXP'', only the dynamic reference picture selection method described in Section~\ref{Section4:RefPicSel} is evaluated. In the latter case, identified as ``\DYN'', besides the previous method of reference picture selection, redundant \acs{mv} are encoded as described in Section~\ref{Section4:MVSelection}, resulting in a combination of both. + +As references for comparison, three other methods are used: (i) a fixed checkerboard approach (\CHKB), (ii) the reference picture generation method proposed in \cite{ROPERefGenJan2010}, identified as ``\AVG'', and (iii) long-term reference frames (\LONG). The \LONG method uses only the key frames as reference, i.e., in the Random-Access configuration only one out of eight frames is used as reference frame, allowing a more robust transmission, as the reference frames are less likely to be hit by errors. + +The experimental setup used in this section is the same as the one previously described in Section~\ref{Section3:ErrorResilienceEvaluation}. The evaluation of the encoding-stage is carried out separately from the streaming-stage. + +\subsection{Encoding stage} + + +The performance of the encoding stage is evaluated on its own in order to consider only the improvements achieved by the dynamic reference frame selection method, when reference frames were reconstructed with distortion (e.g. as a result of non-perfect error concealment). +Since the encoding stage does not deal with errors in temporal \ac{mv} predictions (this is handled by the streaming-stage of the proposed method), in this evaluation the temporal \ac{mv} predictions were not used. This is necessary to avoid masking the results with the errors caused by wrong \ac{mv} predictions. + +The performance of the proposed encoding stage (\EXP) was evaluated using $\gamma = 5$. This allows to increase the Lagrangian cost by about $50\%$, when a reference frame is used for three or more \acs{ctu} in comparison to the usage of the other ones (see Figure~\ref{fig:weightfct}). Further results for a wide range of $\gamma$ values are also presented. + +\subsubsection{Performance under error-free conditions} + +Table~\ref{tab:bd_error_free} shows the Bjontegaard's Delta PSNR (BD-PSNR) values compared to the reference \ac{hevc} encoder for different coding configurations in error-free decoding. Four QPs were used from the common test conditions: 22, 27, 32, 37. +The aim is to evaluate the \ac{rd} penalty incurred by the~\EXP method in comparison with \CHKB, as well as the other reference methods. +The use of long-term reference pictures (\LONG) is only tested for the case of Random-Access, where key frames are available, while method \cite{ROPERefGenJan2010} is tested for the Low-Delay configuration. +The results in Table~\ref{tab:bd_error_free} indicate a small loss of coding efficiency when the proposed reference picture selection approach is used under error-free conditions. As expected, this is due to the sub-optimal \ac{rd} optimisation of the proposed method, because the cost function also includes the cost associated to the choice of the reference frame, in addition to the rate and distortion. + +\begin{table}[t] +\centering +\newcommand{\resultswidth}{1.5cm} +\caption{BD-PSNR in error-free conditions.} +\TABLESVSPACE +\footnotesize +\label{tab:bd_error_free} +\begin{tabular}{p{3cm}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}} +\toprule +\multirow{2}{*}{\centering\textbf{Sequence}} & \multicolumn{3}{c|}{\textbf{Low-Delay}} & \multicolumn{3}{c}{\textbf{Random-Access}} \\ +& \AVG & \CHKB & \EXP & \LONG & \CHKB & \EXP \\ \hline +Basketball Drill & -1.14 & -1.15 & -0.51 & -0.85 & -0.29 & -0.22 \\ \hline +Kendo & -2.10 & -1.39 & -0.99 & -0.91 & -0.39 & -0.28 \\ \hline +Park Scene & -1.79 & -1.38 & -0.81 & -0.68 & -0.38 & -0.28 \\ \hline +Traffic & -1.91 & -1.21 & -0.71 & -0.55 & -0.36 & -0.21 \\ \hline +Bosphorus & -0.97 & -0.95 & -1.10 & -0.64 & -0.20 & -0.20 \\ \hline +\textbf{Average} & \textbf{-1.58} & \textbf{-1.22} & \textbf{-0.82} & \textbf{-0.73} & \textbf{-0.32} & \textbf{-0.24} \\ +\bottomrule +\end{tabular} +\end{table} + +While the fixed checkerboard approach leads to an average quality reduction of $1.22$~dB for the Low-Delay ($0.32$~dB for Random-Access), the proposed method only loses $0.82$~dB and $0.24$~dB in the Low-Delay and Random-Access configurations, respectively. In comparison to \LONG and the method in~\cite{ROPERefGenJan2010}, the proposed method is still more efficient, consistently achieving better results. +Overall, the loss in the quality compared to the reference \ac{hevc} is considered acceptable, given the increase in robustness obtained in lossy transmission, as discussed in the following sections. + +%\subsubsection{Reference picture selection results} + + +\subsubsection{Quality evaluation under error-prone conditions} + +The robustness achieved by the proposed method was evaluated by comparison of the error propagation resulting from a single loss event in the Low-Delay configuration. + +Figure~\ref{fig:RecoveredExample} shows the visual impact of error propagation on a region of the decoded Frame \#10 when Frame \#9 is affected by errors, for Basketball Drill and Race Horses sequences (the PSNR corresponds to the entire frame). As shown in this figure, the proposed method (\EXP) is able to reduce the impact of mismatch decoding in comparison with the method~in~\cite{ROPERefGenJan2010}, revealing the higher efficiency achieving by distributing the use of the references frames. + +\begin{figure}[tb] + \centering + \footnotesize + \newcommand{\CurrFigSize}{0.23\columnwidth} + \begin{tabular}{y{.23\columnwidth}y{.23\columnwidth}|y{.23\columnwidth}y{.23\columnwidth}} + \multicolumn{2}{c}{Basketball Drill} & \multicolumn{2}{|c}{Race Horses} \\ + \AVG & \EXP & \AVG & \EXP \\ + \includegraphics[width=\CurrFigSize]{Figure_6_1} & + \includegraphics[width=\CurrFigSize]{Figure_6_2} & + \includegraphics[width=\CurrFigSize]{Figure_6_3} & + \includegraphics[width=\CurrFigSize]{Figure_6_4} \\ + PSNR = 31.93 dB & PSNR = 33.83 dB & PSNR = 24.12 dB & PSNR = 31.93 dB \\ + \end{tabular} + \caption{Zoom of recovered Frame \#10 after a loss event at Frame \#9 for Basketball Drill and Race Horses sequence.} + \label{fig:RecoveredExample} +\end{figure} + +\begin{figure}[t] + \renewcommand{\FIGSIZE}{0.8\columnwidth} + \centering +% \includegraphics[width=\FIGSIZE]{\ImprovedErrorResilience/Results/PSNRversusFrameKendo} + \input{041_ImprovedErrorResilience/Results/PSNRversusFrameKendo.tex} + \caption{Error propagation when errors affect Frame \#7 in Kendo sequence with an \ac{idr} period of 32 frames.} + \label{fig:PSNRvsFrameForRefPicSel} +\end{figure} + + +\input{\ImprovedErrorResilience/Results/TableResultsRandomLossRefPicSelection.tex} + +Figure~\ref{fig:PSNRvsFrameForRefPicSel}~shows the PSNR over a GOP with an \ac{idr} period of 32 frames, encoded at the same constant bitrate. +This figure shows that all methods are able to outperform the reference \ac{hevc} (\REF), gaining approximately $3$~dB in reconstruction quality. +Since the proposed method is more efficient than the others for error-free transmission (see~Table~\ref{tab:bd_error_free}), and it also provides better error robustness, it can be considered as an efficient solution to better deal with transmission errors. +The similar quality levels achieved by \CHKB and \EXP reveal that dynamic selection of the reference frames is more robust because a better trade-off between coding efficiency and error resilience is attained. +Overall, the method proposed for the encoding stage is able to increase the error robustness of \ac{hevc} without severely compromising the coding efficiency. + +Further tests were run to evaluate the effectiveness of the proposed method under various packet loss rates. +For each test condition, 50 trials were performed and the average quality obtained across all trials for both coding configurations are shown in Table~\ref{tab:AverageQualityCHKBEXPLowDelay}~and~\ref{tab:AverageQualityCHKBEXPRandomAccess}, for the Low-Delay and Random-Access, respectively. +In both tables the absolute PSNR value is shown for the reference \ac{hevc} case (\REF), while the PSNR difference is presented for the other cases. +Results show that the proposed method is able to outperform other methods, by increasing the robustness of \ac{hevc}. An average gain up to $1.47$~dB is obtained for the Low-Delay configuration (see Table~\ref{tab:AverageQualityCHKBEXPLowDelay}), in comparison to the \REF, while the fixed checkerboard pattern approach (\CHKB) only achieves up to $1.33$~dB on average. +In \cite{ROPERefGenJan2010}, the use of interpolated references in the encoding loop reveals to reduce error propagation, at a cost of high degradation of the coding efficiency in the error-free environment, thus the overall performance is lower (see results for ``\AVG''). + + +The proposed method is also able to outperform the \ac{hevc} standard when an hierarchical coding structure is used (see results for Random-Access configuration in Table~\ref{tab:AverageQualityCHKBEXPRandomAccess}), which inherently limits the error propagation. +In comparison to the reference \ac{hevc}, quality improvements up to $1.42$~dB at PLR=$10\%$ are obtained (Kendo), while the average gain across all test sequences is 1~dB. +When comparing results for both cases, it is noticeable that the proposed method is able to outperform the reference methods both when \ac{idr} frame loss is allowed or not (case 1 and 2, respectively). +As the \ac{plr} increases the gains of the proposed method decreases when comparing with the case where the \ac{idr} frames delivered error-free (i.e., case 1), due to the higher error propagation of the missing \ac{idr} frame. +Taking into account the above results and the spatio-temporal information shown in Table~\ref{TableTestSequences}, one can observe that the performance of the proposed method is lower for sequences with high spatial details and low temporal activity (see results for BQSquare and Park Scene), which indicates higher effectiveness for video sequences with higher motion activity. + +\begin{figure}[!b] + \renewcommand{\FIGSIZE}{0.8\columnwidth} + \centering + %\includegraphics[width=\FIGSIZE,page=1]{\ImprovedErrorResilience/Results/PSNRversusGama} + %\includegraphics[width=\FIGSIZE,page=2]{\ImprovedErrorResilience/Results/PSNRversusGama} + \input{./041_ImprovedErrorResilience/Results/PSNRversusGama.tex} + \caption{Influence of the $\gamma$ parameter in the average quality gains, comparing with \REF, for different packet loss ratios.} + \label{fig:QualityVsGamma} +\end{figure} + +\subsubsection{Influence of parameter $\gamma$} + + + +The influence of the $\gamma$ parameter used in Equation (\ref{eq:KFactor}) was also studied. +Figure \ref{fig:QualityVsGamma} shows the average quality gains in comparison to the reference \ac{hevc} (\REF) across all test sequences, when using different values of $\gamma$ in the proposed method (\EXP). +For the error-free case, the results in this figure show that increasing $\gamma$ leads to a decrease in the coding efficiency, especially for the Low-Delay configuration. The reason is that higher values of $\gamma$ lead to higher values of $W(r_{\phi})$, therefore the reference picture selection is more constrained by the exponential weights. Since $T_{ID}$ is constant for all frames in the Low-Delay, higher weights are used for all frames ($T_{ID}$ is used to reduce the weights in case of hierarchical coding) and the encoder is not able to select the best reference frame in terms of R-D optimisation. +In the case where packet loss occurs, higher values of $\gamma$ lead to higher error resilience. Since higher weights used in the \ac{rd} cost, the encoder is constrained to distribute the use of the reference frames, reducing the error propagation. +For very high values (e.g., $\gamma=1000$) the encoder is forced to uniformly select the reference frames, resulting in a fixed checkerboard pattern. +Overall, using the $\gamma$ parameter in the proposed method, the impact of the reference frame, selected by the \ac{rd} optimisation, can be controlled, and thus a better trade-off between higher error resilience and loss of coding efficiency can be achieved. + + +\subsubsection{Complexity overhead} + +The implementation of the proposed method has slightly increased the complexity of the standard non-robust encoder. Therefore, the computational complexity has also been considered as a performance metric in this research, which is measured as the encoding time in a controlled hardware platform. The reference used for comparison with other methods is the \ac{hevc} reference software encoding time, running on the same platform with the same coding configuration. +The relative complexity increase of the robust encoder using \EXP, \CHKB and the method presented in \cite{ROPERefGenJan2010} is shown in Table~\ref{tab:cpu_time}. + +\begin{table}[b] + \newcommand{\resultswidth}{2.2cm} + \centering + \caption{Relative CPU times of tested methods.}\TABLESVSPACE + \label{tab:cpu_time} + \footnotesize + \begin{tabular}{p{4cm}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}} + \toprule + \multirow{2}{*}{\centering\textbf{Sequence}} & \multicolumn{3}{c}{\textbf{Method}} \\ + & \AVG & \CHKB & \EXP \\ \hline + Basketball Drill & +4.19\% & +2.13\% & +2.93\% \\ \hline + Kendo & +4.96\% & +1.83\% & +2.05\% \\ \hline + Park Scene & +4.98\% & +3.90\% & +4.04\% \\ \hline + Traffic & +8.65\% & +2.28\% & +3.05\% \\ + \bottomrule + \end{tabular} +\end{table} + + +As shown in this table, the method presented in \cite{ROPERefGenJan2010} is the most time consuming because it requires post-processing of the encoded frames in order to interpolate new references. +The proposed scheme, using a reference picture selection, \EXP presents a lower complexity increase than \cite{ROPERefGenJan2010}, while the fixed checkerboard scheme (\CHKB) presents a slightly lower complexity than \EXP. + +\subsection{Streaming stage} + + + + +In this section the coding performance that results from using redundant \ac{mv} (streaming stage) in combination with the reference picture selection scheme (encoding stage) is presented. In the experimental tests the temporal \ac{mv} candidates are enabled and redundant \acs{mv} are added into the stream, as described in Section \ref{Section4:MVSelection}. This is referred to as ``\DYN'', and compared to cases where a fixed amount of redundant \acs{mv} per frame are transmitted (i.e., the most important 10\%, 30\% and 50\% $\ac{mv} \in V$). Different $\alpha$ values are used for different redundancy ratios. All streams were encoded with the same total bitrate, including the redundant bits, in order to make a fair quality comparison. + + + + +% \begin{figure}[tbp] +% \centering +% \newcommand{\CurrFigSize}{0.45\columnwidth} +% +% \caption{Decoded video quality for different amounts of redundancy, selected using a fixed approach and the proposed method (Continuation).} +% \label{fig:QualityVsRedundancy\ac{mv}SelectionContinuation} +% \end{figure} + +\subsubsection{Dynamic MV selection performance} + +\begin{figure}[!b] + \centering + \newcommand{\CurrFigSize}{0.49\columnwidth} +% \includegraphics[width=\CurrFigSize,page=1]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf} +% \includegraphics[width=\CurrFigSize,page=2]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf} + \includegraphics[width=\CurrFigSize,page=3]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf} + \includegraphics[width=\CurrFigSize,page=4]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf} + \includegraphics[width=\CurrFigSize,page=5]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf} + \includegraphics[width=\CurrFigSize,page=6]{\ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf} + \caption{Decoded video quality for different amounts of redundancy, selected using a fixed approach and the proposed method.} + \label{fig:QualityVsRedundancyMVSelection} +\end{figure} + + +In the case where variable redundancy is allowed, \acs{mv} are selected using the proposed dynamic method (Dyn) and a fixed approach (Fixed). +Figure~\ref{fig:QualityVsRedundancyMVSelection} illustrates the relation between the quality obtained and the amount of redundancy. +The results indicate that for approximately the same amount of redundant bits, dynamic selection of \acs{mv} lead to higher performance in all test sequences. For instance, the proposed method is able to gain up to $1.5$~dB for the BQSquare sequence. Moreover, the results are consistent for different \ac{plr}. + + + +\subsubsection{Quality evaluation} + +\begin{table}[!t] + \centering + \caption{Average quality using redundant \ac{mv} selected by the proposed approach.} + \TABLESVSPACE + \label{tab:AverageQualityRedundancies} + \TABLESVSPACE + \input{\ImprovedErrorResilience/Results/TableResultsRandomLossMvRedundancies.tex} +\end{table} + + +Table~\ref{tab:AverageQualityRedundancies} presents the average PSNR of decoded video for different \ac{plr} (columns~4~to~8) and for various percentages of \ac{mv} redundancy (third column). +It can be confirmed from the results that, for each sequence, as $\alpha$ decreases more redundant \acs{mv} are used, resulting in higher objective image quality for all tested sequences. +This indicates that redundant \acs{mv} are able to minimise the error propagation. This is accomplished with an acceptable increase in the bitrate, since only a sub-set of \acs{mv} is transmitted. +The use of redundant \acs{mv} reveals better results for \ac{plr} as low as $1\%$, achieving a maximum gain of $3.93$~dB for \LOSTALL ($7.82$~dB for \KEEPINTRA) for the BQSquare sequence at PLR=$5\%$. + +In summary, the proposed method is able to select the most relevant motion \ac{mv} information and when combined with a reference picture selection scheme the problem of mismatch \ac{mv} predictions is mitigated. +Moreover, the dynamic approach devised to select the best \acs{mv} also contribute to increase the overall performance. +Thus, using the proposed two-stage approach to recover temporal predictions consistently improves the error robustness of \ac{hevc}. + +\section{Summary} +\label{Section4:Summary} + +In this chapter a two-stage approach is proposed to increase the error robustness of \ac{hevc} streaming and reduce the error propagation in case of packet loss. +A constrained coding approach was devised to select reference frames and dynamically distribute temporal dependencies. +This is jointly used with a controlled amount of side information in coded streams, comprising a small set of the most relevant \acs{mv}, to minimise \ac{mv} mismatch at the decoder in the presence of frame loss. +As can be concluded from the results, this method contributes to reduce the temporal dependencies with consistent gains for different \ac{plr} and coding configurations, only incurring in a small drop of coding efficiency. +Overall, the proposed approach is an effective method for coping with video transmission over error-prone networks. + + + + + + diff --git a/041_ImprovedErrorResilience/Results/.directory b/041_ImprovedErrorResilience/Results/.directory new file mode 100644 index 0000000..e85cf4a --- /dev/null +++ b/041_ImprovedErrorResilience/Results/.directory @@ -0,0 +1,4 @@ +[Dolphin] +Timestamp=2018,6,20,15,49,31 +Version=4 +ViewMode=1 diff --git a/041_ImprovedErrorResilience/Results/AverageRefPicUsage.pdf b/041_ImprovedErrorResilience/Results/AverageRefPicUsage.pdf new file mode 100644 index 0000000..f69dba8 Binary files /dev/null and b/041_ImprovedErrorResilience/Results/AverageRefPicUsage.pdf differ diff --git a/041_ImprovedErrorResilience/Results/AverageRefPicUsage.plt b/041_ImprovedErrorResilience/Results/AverageRefPicUsage.plt new file mode 100644 index 0000000..daa233b --- /dev/null +++ b/041_ImprovedErrorResilience/Results/AverageRefPicUsage.plt @@ -0,0 +1,44 @@ +set terminal postscript eps monochrome font 'TimesNewRoman,16' + +set size {0.75,0.75} +set xlabel "Sequence" font ",18" offset character 0, 0.4, 0 +set xtics center font ",16" # offset character 0, 0, 0 +set ylabel "Reference Usage Ratio (%)" font ",18" offset character 0.9, 0, 0 +set ytics font ",16" + + + +set style line 1 lc 4 lt 4 lw 3 pt 10 ps 1 +#set style line 2 lc rgbcolor "#006400" lt 2 lw 3 pt 3 ps 1 +set style line 2 lc 2 lt 2 lw 3 pt 3 ps 1 +set style line 3 lc 3 lt 4 lw 3 pt 4 ps 1 +set style line 4 lc 1 lt 1 lw 3 pt 2 ps 1 +set style line 5 lc 7 lt 3 lw 3 pt 6 ps 1 +set style line 6 lc 8 lt 6 lw 3 pt 8 ps 1 +set style line 7 lc 2 lt 7 lw 3 pt 8 ps 1 +set style line 8 lc 3 lt 8 lw 3 pt 9 ps 1 +set style line 9 lc 4 lt 9 lw 3 pt 10 ps 1 + + +set style histogram clustered gap 1.5 +set grid y +set style data histograms +set style fill solid +set boxwidth 1 +set key off +set xtics rotate by 30 offset 0,-1 +set bmargin 5 + +set yrange[0.4:1.0] +set ytics 0.4,.1,1.0 +set output 'plot_AverageRefPicUsage_lowdelay_P_2Ref_main_16_rate_2M000bps_0.eps' +plot '-' using 3:xtic(2) linecolor 9 +0 "Kendo" 0.839689 +1 "Book Arrival" 0.873156 +2 "Basketball Drill" 0.836044 +3 "Race Horses" 0.885911 +4 "BQSquare" 0.5816 +5 "Tennis" 0.865467 +6 "Park Scene" 0.849022 +e + diff --git a/041_ImprovedErrorResilience/Results/AverageRefPicUsage.tex b/041_ImprovedErrorResilience/Results/AverageRefPicUsage.tex new file mode 100644 index 0000000..fe2ce54 --- /dev/null +++ b/041_ImprovedErrorResilience/Results/AverageRefPicUsage.tex @@ -0,0 +1,37 @@ +\begin{figure}[tb] +\centering +\newcommand{\pltw}{0.85\columnwidth} +\newcommand{\plth}{0.25\textheight} +\begin{gnuplot}[terminal=epslatex,terminaloptions={font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +unset grid +unset xlabel +set ylabel '\TitleFont Reference Usage (\%)' + +set style histogram clustered +#set style histogram rowstacked + +set grid y +set style histogram gap 1.5 +set style data histograms +set style fill solid +set boxwidth 1 +set key off +set xtics right rotate by 20 offset .5,-.1 +set bmargin 5 +set style line 20 lc rgb '#696969' lt 1 lw 2 pi -1 ps 1.0 +set yrange[50:100] +set ytics 40,10,100 +plot '-' using ($3*100):xtic(2) ls 20 +0 "\\TitleFont Book Arrival" 0.873156 +1 "\\TitleFont BQSquare" 0.5816 +2 "\\TitleFont Basketball Drill" 0.836044 +3 "\\TitleFont Kendo" 0.839689 +4 "\\TitleFont Park Scene" 0.849022 +5 "\\TitleFont Race Horses" 0.885911 +6 "\\TitleFont Tennis" 0.865467 +e +\end{gnuplot} +\caption{Average usage of the reference frame 1 ($r_1$) for different sequences.} +\label{FIG4:ReferenceUsageRatio} +\end{figure} diff --git a/041_ImprovedErrorResilience/Results/Figure_6_1.pdf b/041_ImprovedErrorResilience/Results/Figure_6_1.pdf new file mode 100644 index 0000000..3351792 Binary files /dev/null and b/041_ImprovedErrorResilience/Results/Figure_6_1.pdf differ diff --git a/041_ImprovedErrorResilience/Results/Figure_6_2.pdf b/041_ImprovedErrorResilience/Results/Figure_6_2.pdf new file mode 100644 index 0000000..e1cfecb Binary files /dev/null and b/041_ImprovedErrorResilience/Results/Figure_6_2.pdf differ diff --git a/041_ImprovedErrorResilience/Results/Figure_6_3.pdf b/041_ImprovedErrorResilience/Results/Figure_6_3.pdf new file mode 100644 index 0000000..6201576 Binary files /dev/null and b/041_ImprovedErrorResilience/Results/Figure_6_3.pdf differ diff --git a/041_ImprovedErrorResilience/Results/Figure_6_4.pdf b/041_ImprovedErrorResilience/Results/Figure_6_4.pdf new file mode 100644 index 0000000..e7a80ce Binary files /dev/null and b/041_ImprovedErrorResilience/Results/Figure_6_4.pdf differ diff --git a/041_ImprovedErrorResilience/Results/PSNRversusFrameKendo.pdf b/041_ImprovedErrorResilience/Results/PSNRversusFrameKendo.pdf new file mode 100644 index 0000000..bcdde40 Binary files /dev/null and b/041_ImprovedErrorResilience/Results/PSNRversusFrameKendo.pdf differ diff --git a/041_ImprovedErrorResilience/Results/PSNRversusFrameKendo.tex b/041_ImprovedErrorResilience/Results/PSNRversusFrameKendo.tex new file mode 100644 index 0000000..bd9eb8f --- /dev/null +++ b/041_ImprovedErrorResilience/Results/PSNRversusFrameKendo.tex @@ -0,0 +1,178 @@ +\newcommand{\pltw}{0.75\columnwidth} +\newcommand{\plth}{0.3\textheight} +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +set xlabel '\TitleFont Frame Number' +set xrange[2:0+31] +set xtics 0,2,0+31 +set yrange[33:45] +set ytics 33,2,45 +set key spacing 1.4 bottom right width 6 box +plot '-' using 1:2 title '\KeyFont \EXP' w lp ls 1, '-' using 1:2 title '\KeyFont \CHKB' w lp ls 2, '-' using 1:2 title '\KeyFont \AVG' w lp ls 3, '-' using 1:2 title '\KeyFont \REF' w lp ls 4 + 0 43.01 44.05 43.66 + 1 41.71 43.89 43.36 + 2 40.83 43.77 43.16 + 3 41.39 43.81 43.15 + 4 42.64 43.99 43.44 + 5 41.52 43.74 43.07 + 6 41.50 43.72 42.98 + 7 36.44 43.60 42.80 + 8 42.37 44.15 43.63 + 9 40.49 43.68 42.86 + 10 41.69 43.89 43.27 + 11 40.69 43.56 42.55 + 12 43.33 44.41 43.96 + 13 40.74 43.48 42.61 + 14 41.55 43.79 42.92 + 15 40.86 43.64 42.59 + 16 43.38 44.33 43.97 + 17 41.36 43.54 42.78 + 18 41.76 43.70 43.00 + 19 41.37 43.46 42.75 + 20 43.10 44.25 43.80 + 21 41.02 43.57 42.64 + 22 41.62 43.65 42.85 + 23 41.30 43.69 42.69 + 24 43.08 44.25 43.78 + 25 41.33 43.67 42.72 + 26 41.71 43.90 42.98 + 27 41.34 43.79 42.86 + 28 43.19 44.33 43.89 + 29 41.42 43.76 42.94 + 30 41.62 43.90 43.06 + 31 41.57 43.88 43.03 + 32 43.21 44.27 43.80 + 33 42.26 44.06 43.42 + 34 42.17 43.91 43.23 + 35 41.95 43.79 42.96 + 36 43.36 44.25 43.54 + 37 41.28 43.54 42.30 + 38 41.64 43.57 42.41 +e + 0 43.01 44.05 43.66 + 1 41.71 43.89 43.36 + 2 40.76 43.77 43.14 + 3 41.36 43.78 43.15 + 4 42.65 43.95 43.44 + 5 41.50 43.79 43.04 + 6 41.53 43.73 43.03 + 7 37.34 43.60 42.75 + 8 41.64 44.13 43.53 + 9 39.48 43.64 42.74 + 10 40.94 43.90 43.18 + 11 40.22 43.50 42.46 + 12 42.30 44.36 43.92 + 13 40.97 43.55 42.72 + 14 41.60 43.72 42.78 + 15 40.73 43.52 42.42 + 16 42.89 44.35 43.94 + 17 41.05 43.66 42.77 + 18 41.54 43.74 42.84 + 19 41.34 43.65 42.82 + 20 43.04 44.18 43.61 + 21 41.13 43.51 42.55 + 22 41.58 43.72 42.84 + 23 41.12 43.69 42.69 + 24 42.89 44.26 43.62 + 25 41.49 43.81 42.82 + 26 41.76 43.97 42.99 + 27 41.31 43.91 42.92 + 28 43.19 44.40 43.84 + 29 41.36 43.83 42.89 + 30 41.41 43.80 42.84 + 31 41.38 43.80 42.74 + 32 43.21 44.27 43.80 + 33 42.30 44.08 43.44 + 34 42.18 43.96 43.22 + 35 41.96 43.81 43.00 + 36 43.36 44.16 43.48 +e + 0 43.01 44.05 43.66 + 1 41.71 43.89 43.36 + 2 40.80 43.77 43.15 + 3 41.41 43.81 43.14 + 4 42.71 43.98 43.47 + 5 41.39 43.73 42.99 + 6 41.57 43.70 42.84 + 7 35.60 43.24 41.69 + 8 41.02 44.13 43.48 + 9 39.06 43.40 42.29 + 10 40.06 43.59 42.66 + 11 38.91 43.02 41.76 + 12 41.66 44.26 43.70 + 13 39.59 43.25 42.18 + 14 39.50 43.13 41.82 + 15 38.84 42.76 41.28 + 16 41.39 43.95 43.19 + 17 39.50 42.92 41.51 + 18 39.41 43.09 41.45 + 19 39.15 42.80 41.15 + 20 40.83 43.76 42.87 + 21 39.69 42.92 41.51 + 22 39.78 42.91 41.39 + 23 39.44 42.96 41.47 + 24 41.51 43.81 42.89 + 25 39.73 43.01 41.87 + 26 40.21 42.99 41.77 + 27 39.61 42.73 41.59 + 28 41.79 44.00 43.08 + 29 39.69 43.02 41.93 + 30 39.86 42.88 41.67 + 31 39.61 42.86 41.77 + 32 43.21 44.27 43.80 + 33 41.97 44.06 43.41 + 34 41.89 43.89 43.09 + 35 41.43 43.67 42.67 + 36 43.19 44.09 43.42 + 37 40.66 43.41 42.00 + 38 40.92 43.39 41.92 + 39 40.31 43.07 41.44 +e + 0 43.01 44.05 43.66 + 1 41.71 43.89 43.36 + 2 40.83 43.78 43.17 + 3 41.33 43.79 43.12 + 4 42.58 43.98 43.42 + 5 41.85 43.88 43.17 + 6 41.55 43.79 43.02 + 7 34.55 43.16 41.33 + 8 36.21 43.72 42.06 + 9 36.75 43.69 42.04 + 10 37.41 43.77 42.07 + 11 37.40 43.68 41.91 + 12 38.79 44.21 42.81 + 13 38.41 43.94 42.43 + 14 39.02 44.05 42.62 + 15 38.88 43.85 42.29 + 16 39.69 44.44 43.49 + 17 39.01 44.09 42.95 + 18 39.55 44.09 42.99 + 19 39.37 43.81 42.64 + 20 40.24 44.45 43.52 + 21 39.61 44.09 43.04 + 22 39.65 43.99 42.93 + 23 39.42 43.91 42.69 + 24 40.38 44.42 43.51 + 25 39.77 44.18 43.18 + 26 39.92 44.25 43.19 + 27 39.83 44.14 43.03 + 28 40.60 44.66 43.90 + 29 40.08 44.38 43.49 + 30 40.00 44.30 43.35 + 31 39.92 44.27 43.20 + 32 43.21 44.27 43.80 + 33 42.52 44.08 43.51 + 34 42.55 44.02 43.39 + 35 42.17 43.88 43.08 + 36 44.01 44.50 44.13 + 37 42.68 44.19 43.54 + 38 42.64 44.10 43.32 + 39 42.44 43.99 43.12 + 40 44.21 44.64 44.27 + 41 43.26 44.41 43.88 + 42 43.35 44.44 43.87 + 43 43.13 44.43 43.77 + 44 44.52 44.86 44.61 + 45 43.55 44.72 44.30 +e +\end{gnuplot} diff --git a/041_ImprovedErrorResilience/Results/PSNRversusGama.pdf b/041_ImprovedErrorResilience/Results/PSNRversusGama.pdf new file mode 100644 index 0000000..bb21c44 Binary files /dev/null and b/041_ImprovedErrorResilience/Results/PSNRversusGama.pdf differ diff --git a/041_ImprovedErrorResilience/Results/PSNRversusGama.tex b/041_ImprovedErrorResilience/Results/PSNRversusGama.tex new file mode 100644 index 0000000..6a7a840 --- /dev/null +++ b/041_ImprovedErrorResilience/Results/PSNRversusGama.tex @@ -0,0 +1,166 @@ +\newcommand{\ft}[0]{\footnotesize} +\newcommand{\pltw}{0.75\columnwidth} +\newcommand{\plth}{0.6\textheight} +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' + +set size {1, 1} +set multiplot layout 2,1 + +set key width 0 + +set grid +set xlabel '\TitleFont $\gamma$' font ",18" offset character 0, 0.4, 0 +set xtics center font ",16" # offset character 0, 0, 0 +set ylabel '\TitleFont $\Delta$ PSNR (dB)' font ",18" offset character 0.9, 0, 0 +set ytics font ",16" +set key spacing 1.2 top right width -1 + + +set key top left Right +set xrange[0.01:1000] +set xtics 0.01, 10, 1000 +set logscale x 10 +set mxtics 10 +set grid mxtics + +set yrange[-1:2.5] +set title '\TitleFont Low-Delay Configuration' +plot '-' using 1:2 title '\KeyFont Error-free' w lp ls 1, '-' using 1:2 title '\KeyFont 1.0\% PLR' w lp ls 2, '-' using 1:2 title '\KeyFont 3.0\% PLR' w lp ls 3, '-' using 1:2 title '\KeyFont 5.0\% PLR' w lp ls 4, '-' using 1:2 title '\KeyFont 10.0\% PLR' w lp ls 5 +.001 -.034 +.01 -.034 +.1 -.230 +.5 -.514 +2 -.634 +5 -.730 +10 -.786 +50 -.850 +100 -.856 +200 -.800 +500 -.746 +1000 -.726 +e +.001 .0226 +.01 .0226 +.1 .1544 +.5 .1982 +2 .2394 +5 .2336 +10 .2682 +50 .2954 +100 .3130 +200 .3268 +500 .2600 +1000 .2508 +e +.001 .1310 +.01 .1310 +.1 .4150 +.5 .8454 +2 .9018 +5 1.1026 +10 1.1578 +50 1.2952 +100 1.3686 +200 1.2270 +500 1.1162 +1000 1.0374 +e +.001 .1324 +.01 .1324 +.1 .5328 +.5 .9914 +2 1.2386 +5 1.4932 +10 1.5908 +50 1.7704 +100 1.8158 +200 1.6648 +500 1.4716 +1000 1.440 +e +.001 .1040 +.01 .1040 +.1 .5454 +.5 1.1746 +2 1.3558 +5 1.6724 +10 1.7774 +50 2.0048 +100 2.1462 +200 1.9868 +500 1.7622 +1000 1.6894 +e + +set yrange[-0.3:1.5] +set title '\TitleFont Random-Access Configuration' +plot '-' using 1:2 title '\KeyFont Error-free' w lp ls 1, '-' using 1:2 title '\KeyFont 1.0\% PLR' w lp ls 2, '-' using 1:2 title '\KeyFont 3.0\% PLR' w lp ls 3, '-' using 1:2 title '\KeyFont 5.0\% PLR' w lp ls 4, '-' using 1:2 title '\KeyFont 10.0\% PLR' w lp ls 5 +.001 -.012 +.01 -.012 +.1 -.058 +.5 -.120 +2 -.154 +5 -.184 +10 -.202 +50 -.232 +100 -.218 +200 -.196 +500 -.164 +1000 -.152 +e +.001 .3212 +.01 .3212 +.1 .3704 +.5 .3034 +2 .3372 +5 .4130 +10 .3452 +50 .3946 +100 .2884 +200 .2966 +500 .2190 +1000 .2200 +e +.001 .4324 +.01 .4324 +.1 .4978 +.5 .6868 +2 .6594 +5 .7054 +10 .7196 +50 .740 +100 .763 +200 .7274 +500 .6362 +1000 .5356 +e +.001 .6524 +.01 .6524 +.1 .7198 +.5 .8208 +2 .9694 +5 .9504 +10 .9846 +50 1.0522 +100 1.0454 +200 .8868 +500 .8400 +1000 .7678 +e +.001 .6082 +.01 .6082 +.1 .7476 +.5 .9630 +2 .9544 +5 1.0054 +10 1.0874 +50 1.0688 +100 1.0326 +200 .9138 +500 .8126 +1000 .8434 +e + +\end{gnuplot} + diff --git a/041_ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf b/041_ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf new file mode 100644 index 0000000..ee708bb Binary files /dev/null and b/041_ImprovedErrorResilience/Results/PSNRvsRedundancy.pdf differ diff --git a/041_ImprovedErrorResilience/Results/RefPicSelectionPSNRDifferentialvsGamaAverageThesis.pdf b/041_ImprovedErrorResilience/Results/RefPicSelectionPSNRDifferentialvsGamaAverageThesis.pdf new file mode 100644 index 0000000..675dcf9 Binary files /dev/null and b/041_ImprovedErrorResilience/Results/RefPicSelectionPSNRDifferentialvsGamaAverageThesis.pdf differ diff --git a/041_ImprovedErrorResilience/Results/RefPicSelectionPSNRDifferentialvsGamaAverageThesis.plt b/041_ImprovedErrorResilience/Results/RefPicSelectionPSNRDifferentialvsGamaAverageThesis.plt new file mode 100644 index 0000000..c1c9c69 --- /dev/null +++ b/041_ImprovedErrorResilience/Results/RefPicSelectionPSNRDifferentialvsGamaAverageThesis.plt @@ -0,0 +1,169 @@ +set terminal postscript eps monochrome font 'TimesNewRoman,16' + +set grid +set size {0.75,0.75} +set xlabel "Bitrate [kbits/s]" font ",18" offset character 0, 0.4, 0 +set xtics center font ",16" # offset character 0, 0, 0 +set ylabel "PSNR [dB]" font ",18" offset character 0.9, 0, 0 +set ytics font ",16" +set key spacing 1 font "TimesNewRoman,16" top right width 0 + +set title font ",20" offset character 0, -0.7, 0 + +set style line 1 lc 4 lt 4 lw 3 pt 10 ps 1 +# set style line 2 lc rgbcolor "#006400" lt 2 lw 3 pt 3 ps 1 +set style line 2 lc 2 lt 2 lw 3 pt 3 ps 1 +set style line 3 lc 3 lt 4 lw 3 pt 4 ps 1 +set style line 4 lc 1 lt 1 lw 3 pt 2 ps 1 +set style line 5 lc 7 lt 3 lw 3 pt 6 ps 1 +set style line 6 lc 8 lt 6 lw 3 pt 8 ps 1 +set style line 7 lc 2 lt 7 lw 3 pt 8 ps 1 +set style line 8 lc 3 lt 8 lw 3 pt 9 ps 1 +set style line 9 lc 4 lt 9 lw 3 pt 10 ps 1 + +set output 'plot_RefPicSelectionPSNRDifferentialvsGamaAverageThesis_common_lowdelay_P_main_rate_single.eps' +set title 'Low-Delay configuration' +set xtics 0.01, 10, 1000 + set logscale x 10 + set mxtics 10 + set grid mxtics +plot '-' using 1:2 title 'Error-free' w lp ls 1, '-' using 1:2 title '1.0% PLR' w lp ls 2, '-' using 1:2 title '3.0% PLR' w lp ls 3, '-' using 1:2 title '5.0% PLR' w lp ls 4, '-' using 1:2 title '10.0% PLR' w lp ls 5 +.001 -.034 +.01 -.034 +.1 -.230 +.5 -.514 +2 -.634 +5 -.730 +10 -.786 +50 -.850 +100 -.856 +200 -.800 +500 -.746 +1000 -.726 +e +.001 .0226 +.01 .0226 +.1 .1544 +.5 .1982 +2 .2394 +5 .2336 +10 .2682 +50 .2954 +100 .3130 +200 .3268 +500 .2600 +1000 .2508 +e +.001 .1310 +.01 .1310 +.1 .4150 +.5 .8454 +2 .9018 +5 1.1026 +10 1.1578 +50 1.2952 +100 1.3686 +200 1.2270 +500 1.1162 +1000 1.0374 +e +.001 .1324 +.01 .1324 +.1 .5328 +.5 .9914 +2 1.2386 +5 1.4932 +10 1.5908 +50 1.7704 +100 1.8158 +200 1.6648 +500 1.4716 +1000 1.440 +e +.001 .1040 +.01 .1040 +.1 .5454 +.5 1.1746 +2 1.3558 +5 1.6724 +10 1.7774 +50 2.0048 +100 2.1462 +200 1.9868 +500 1.7622 +1000 1.6894 +e + +set output 'plot_RefPicSelectionPSNRDifferentialvsGamaAverageThesis_common_randomaccess_main_rate_single.eps' +set title 'Random-Access configuration' +set xtics 0.01, 10, 1000 + set logscale x 10 + set mxtics 10 + set grid mxtics +plot '-' using 1:2 title 'Error-free' w lp ls 1, '-' using 1:2 title '1.0% PLR' w lp ls 2, '-' using 1:2 title '3.0% PLR' w lp ls 3, '-' using 1:2 title '5.0% PLR' w lp ls 4, '-' using 1:2 title '10.0% PLR' w lp ls 5 +.001 -.012 +.01 -.012 +.1 -.058 +.5 -.120 +2 -.154 +5 -.184 +10 -.202 +50 -.232 +100 -.218 +200 -.196 +500 -.164 +1000 -.152 +e +.001 .3212 +.01 .3212 +.1 .3704 +.5 .3034 +2 .3372 +5 .4130 +10 .3452 +50 .3946 +100 .2884 +200 .2966 +500 .2190 +1000 .2200 +e +.001 .4324 +.01 .4324 +.1 .4978 +.5 .6868 +2 .6594 +5 .7054 +10 .7196 +50 .740 +100 .763 +200 .7274 +500 .6362 +1000 .5356 +e +.001 .6524 +.01 .6524 +.1 .7198 +.5 .8208 +2 .9694 +5 .9504 +10 .9846 +50 1.0522 +100 1.0454 +200 .8868 +500 .8400 +1000 .7678 +e +.001 .6082 +.01 .6082 +.1 .7476 +.5 .9630 +2 .9544 +5 1.0054 +10 1.0874 +50 1.0688 +100 1.0326 +200 .9138 +500 .8126 +1000 .8434 +e + diff --git a/041_ImprovedErrorResilience/Results/SelectionExample/CHKB.png b/041_ImprovedErrorResilience/Results/SelectionExample/CHKB.png new file mode 100644 index 0000000..a698d3d Binary files /dev/null and b/041_ImprovedErrorResilience/Results/SelectionExample/CHKB.png differ diff --git a/041_ImprovedErrorResilience/Results/SelectionExample/EXP.png b/041_ImprovedErrorResilience/Results/SelectionExample/EXP.png new file mode 100644 index 0000000..5e77fb6 Binary files /dev/null and b/041_ImprovedErrorResilience/Results/SelectionExample/EXP.png differ diff --git a/041_ImprovedErrorResilience/Results/SelectionExample/ORG.png b/041_ImprovedErrorResilience/Results/SelectionExample/ORG.png new file mode 100644 index 0000000..0c75001 Binary files /dev/null and b/041_ImprovedErrorResilience/Results/SelectionExample/ORG.png differ diff --git a/041_ImprovedErrorResilience/Results/SelectionExample/REF.png b/041_ImprovedErrorResilience/Results/SelectionExample/REF.png new file mode 100644 index 0000000..b6e9db4 Binary files /dev/null and b/041_ImprovedErrorResilience/Results/SelectionExample/REF.png differ diff --git a/041_ImprovedErrorResilience/Results/TableResultsRandomLossMvRedundancies.tex b/041_ImprovedErrorResilience/Results/TableResultsRandomLossMvRedundancies.tex new file mode 100644 index 0000000..0986926 --- /dev/null +++ b/041_ImprovedErrorResilience/Results/TableResultsRandomLossMvRedundancies.tex @@ -0,0 +1,67 @@ +\resizebox{\columnwidth}{!}{ +\newcommand\nlines{4} +\newcommand{\resultswidth}{1.1cm} +\small +\footnotesize +\begin{tabular}{x{2.0cm}|x{3.2cm}|x{1.8cm}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}} +\toprule +\multirow{3}{*}{\centering\textbf{Sequence}} & \multirow{3}{*}{\vbox{\centering\textbf{Method}}} & \multirow{3}{*}{\vbox{\centering\textbf{Overhead ratio ($\%$)}}} & \multicolumn{5}{c}{\textbf{Packet loss rate}} \\ +& & & \multicolumn{2}{c|}{\textbf{\LOSTALL}} & \multicolumn{3}{c}{\textbf{\KEEPINTRA}} \\ +& & & \textbf{1\%} & \textbf{5\%} & \textbf{1\%} & \textbf{5\%} & \textbf{10\%} \\ +\hline +\multirow{\nlines}{*}{\vbox{Basketball Drill }} +& \NOREDUND & -- & 30.41 & 23.31 & 34.76 & 28.56 & 25.89 \\ +& \OPTREDUND{1.5} & 2.33 & +0.68 & +0.41 & +0.58 & +1.60 & +1.35 \\ +& \OPTREDUND{0.8} & 4.52 & +0.96 & +0.96 & +0.95 & +2.31 & +2.03 \\ +& \OPTREDUND{0.6} & 5.85 & +1.12 & +1.16 & +0.93 & +2.58 & +2.14 \\ +\hline +\multirow{\nlines}{*}{\vbox{Book Arrival }} +& \NOREDUND & -- & 33.63 & 25.59 & 37.47 & 31.60 & 28.27 \\ +& \OPTREDUND{1.5} & 1.81 & +0.38 & +0.14 & +0.02 & +0.70 & +0.61 \\ +& \OPTREDUND{0.8} & 2.95 & +0.23 & +1.15 & +0.49 & +0.76 & +1.06 \\ +& \OPTREDUND{0.6} & 3.92 & +0.44 & +1.80 & +0.34 & +1.53 & +1.60 \\ +\hline +%\multirow{\nlines}{*}{\vbox{Bosphorus }} +%& \NOREDUND & -- & 40.58 & 38.86 & 41.32 & 40.14 & 39.19 \\ +%& \OPTREDUND{1.5} & 1.46 & +0.54 & +1.21 & +0.27 & +0.62 & +0.75 \\ +%& \OPTREDUND{0.8} & 3.05 & +0.73 & +1.63 & +0.31 & +0.85 & +1.09 \\ +%& \OPTREDUND{0.6} & 3.94 & +0.75 & +1.70 & +0.33 & +0.92 & +1.17 \\ +%\hline +\multirow{\nlines}{*}{\vbox{BQSquare }} +& \NOREDUND & -- & 28.66 & 20.22 & 31.00 & 23.09 & 20.21 \\ +& \OPTREDUND{1.5} & 5.35 & +3.04 & +3.94 & +4.05 & +6.19 & +5.86 \\ +& \OPTREDUND{0.8} & 9.69 & +3.23 & +3.80 & +5.03 & +7.38 & +7.20 \\ +& \OPTREDUND{0.6} & 12.38 & +3.54 & +3.93 & +5.13 & +7.82 & +7.52 \\ +\hline +\multirow{\nlines}{*}{\vbox{Kendo }} +& \NOREDUND & -- & 36.21 & 28.38 & 38.23 & 29.58 & 25.52 \\ +& \OPTREDUND{1.5} & 2.36 & +0.78 & +1.16 & +0.36 & +1.58 & +1.69 \\ +& \OPTREDUND{0.8} & 4.03 & +0.94 & +1.32 & +0.79 & +2.46 & +3.03 \\ +& \OPTREDUND{0.6} & 5.30 & +0.86 & +1.08 & +0.97 & +2.84 & +3.10 \\ +\hline +\multirow{\nlines}{*}{\vbox{Park Scene }} +& \NOREDUND & -- & 28.60 & 23.85 & 33.02 & 28.26 & 25.98 \\ +& \OPTREDUND{1.5} & 3.88 & +0.53 & +0.72 & +1.05 & +2.06 & +2.09 \\ +& \OPTREDUND{0.8} & 8.58 & +0.79 & +0.96 & +1.55 & +3.19 & +3.29 \\ +& \OPTREDUND{0.6} & 11.22 & +0.73 & +1.05 & +1.58 & +3.61 & +3.73 \\ +\hline +\multirow{\nlines}{*}{\vbox{Race Horses }} +& \NOREDUND & -- & 26.87 & 21.78 & 29.67 & 23.23 & 20.48 \\ +& \OPTREDUND{1.5} & 3.43 & +2.46 & +1.45 & +2.28 & +3.11 & +2.82 \\ +& \OPTREDUND{0.8} & 6.08 & +2.76 & +2.23 & +2.88 & +3.96 & +3.53 \\ +& \OPTREDUND{0.6} & 7.58 & +2.93 & +2.27 & +2.89 & +4.22 & +3.79 \\ +\hline +\multirow{\nlines}{*}{\vbox{Tennis }} +& \NOREDUND & -- & 31.27 & 24.81 & 34.33 & 27.09 & 23.96 \\ +& \OPTREDUND{1.5} & 1.63 & +0.92 & +0.69 & +0.87 & +1.42 & +1.20 \\ +& \OPTREDUND{0.8} & 3.12 & +1.26 & +0.76 & +1.33 & +1.93 & +1.73 \\ +& \OPTREDUND{0.6} & 4.10 & +0.97 & +1.00 & +1.45 & +2.17 & +2.07 \\ +\hline +\multirow{\nlines}{*}{\vbox{Traffic }} +& \NOREDUND & -- & 27.38 & 22.22 & 33.92 & 27.92 & 24.86 \\ +& \OPTREDUND{1.5} & 2.09 & +0.30 & +0.48 & +0.95 & +1.61 & +1.66 \\ +& \OPTREDUND{0.8} & 5.67 & +0.82 & +0.95 & +1.82 & +3.59 & +3.67 \\ +& \OPTREDUND{0.6} & 7.98 & +0.96 & +0.87 & +2.07 & +3.89 & +4.12 \\ +\bottomrule +\end{tabular} +} diff --git a/041_ImprovedErrorResilience/Results/TableResultsRandomLossRefPicSelection.tex b/041_ImprovedErrorResilience/Results/TableResultsRandomLossRefPicSelection.tex new file mode 100644 index 0000000..2a0c12f --- /dev/null +++ b/041_ImprovedErrorResilience/Results/TableResultsRandomLossRefPicSelection.tex @@ -0,0 +1,188 @@ +\begin{table}[!t] +\newcommand\nlines{4} +\newcommand{\resultswidth}{1.3cm} +\centering +\caption{Average quality (PSNR): Low-Delay.} +\TABLESVSPACE +\small +\footnotesize +\label{tab:AverageQualityCHKBEXPLowDelay} +\resizebox{\columnwidth}{!}{ +\begin{tabular}{p{2.2cm}|x{2.2cm}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}} +\toprule +\multirow{3}{*}{\centering\textbf{Sequence}} & \multirow{3}{*}{\vbox{\centering\textbf{Method}}} & \multicolumn{6}{c}{\textbf{Packet loss rate}} \\ +& & \multicolumn{2}{c|}{\textbf{\LOSTALL}} & \multicolumn{4}{c}{\textbf{\KEEPINTRA}} \\ +& & \textbf{1\%} & \textbf{5\%} & \textbf{1\%} & \textbf{3\%} & \textbf{5\%} & \textbf{10\%} \\ +\hline +\multirow{\nlines}{*}{\vbox{Basketball Drill }} +& \REF & 32.25 & 24.43 & 36.52 & 33.70 & 31.70 & 28.51 \\ +& \AVG & -1.05 & +0.50 & -1.95 & +0.12 & +1.32 & +2.90 \\ +& \CHKB & -0.39 & +0.60 & -0.09 & +0.69 & +0.97 & +1.34 \\ +& \EXP & +0.08 & +0.86 & +0.45 & +1.26 & +1.54 & +1.78 \\ +\hline +\multirow{\nlines}{*}{\vbox{Book Arrival }} +& \REF & 34.47 & 26.95 & 38.53 & 35.82 & 33.63 & 30.67 \\ +& \AVG & -1.84 & -0.72 & -1.76 & -0.64 & -0.05 & -0.04 \\ +& \CHKB & -0.08 & +0.81 & +0.49 & +1.33 & +1.66 & +2.15 \\ +& \EXP & -0.37 & +0.56 & +0.35 & +1.17 & +1.41 & +1.55 \\ +\hline +\multirow{\nlines}{*}{\vbox{Cactus }} +& \REF & 34.37 & 31.94 & 37.04 & 35.67 & 34.51 & 32.65 \\ +& \AVG & -0.13 & +0.20 & +0.31 & +0.46 & +0.35 & +0.83 \\ +& \CHKB & -0.34 & -0.12 & +0.02 & +0.37 & +0.71 & +1.00 \\ +& \EXP & -0.23 & +0.04 & +0.06 & +0.21 & +0.43 & +0.54 \\ +\hline +\multirow{\nlines}{*}{\vbox{Bosphorus }} +& \REF & 31.79 & 24.00 & 42.19 & 41.31 & 40.56 & 38.74 \\ +& \AVG & -0.09 & -1.04 & -0.34 & -0.15 & +0.09 & +0.67 \\ +& \CHKB & +0.16 & -0.49 & -0.38 & -0.13 & +0.13 & +0.72 \\ +& \EXP & +0.23 & -0.31 & -0.47 & -0.18 & +0.07 & +0.69 \\ +\hline +\multirow{\nlines}{*}{\vbox{BQSquare }} +& \REF & 32.47 & 25.11 & 38.20 & 36.08 & 34.57 & 31.93 \\ +& \AVG & -4.42 & -3.79 & -4.42 & -3.79 & -3.73 & -3.74 \\ +& \CHKB & -0.38 & +0.63 & -0.12 & +0.29 & +0.44 & +0.67 \\ +& \EXP & +0.74 & +1.37 & -0.14 & +0.66 & +0.88 & +1.35 \\ +\hline +%\multirow{\nlines}{*}{\vbox{Jockey }} +%& \REF & 33.52 & 26.57 & 37.91 & 33.23 & 30.75 & 27.53 \\ +%& \AVG & +1.29 & -0.03 & +2.10 & +3.21 & +2.84 & +1.00 \\ +%& \CHKB & +2.16 & +0.19 & +1.47 & +2.03 & +2.20 & +2.05 \\ +%& \EXP & +2.17 & +0.23 & +1.68 & +2.41 & +2.30 & +2.16 \\ +%\hline +\multirow{\nlines}{*}{\vbox{Kendo }} +& \REF & 38.49 & 30.89 & 40.45 & 36.79 & 33.80 & 30.35 \\ +& \AVG & -1.82 & +0.07 & -1.82 & +0.10 & +1.34 & +1.30 \\ +& \CHKB & +0.09 & +1.16 & +0.24 & +1.38 & +2.27 & +2.15 \\ +& \EXP & +0.51 & +1.33 & +0.19 & +1.34 & +2.26 & +2.26 \\ +\hline +\multirow{\nlines}{*}{\vbox{Park Scene }} +& \REF & 29.81 & 25.29 & 36.06 & 34.83 & 33.64 & 31.63 \\ +& \AVG & -2.29 & -1.63 & -3.49 & -3.40 & -3.34 & -3.45 \\ +& \CHKB & -0.05 & +0.10 & -0.69 & -0.20 & +0.22 & +0.71 \\ +& \EXP & -0.08 & -0.04 & -0.50 & -0.05 & +0.29 & +0.86 \\ +\hline +\multirow{\nlines}{*}{\vbox{People on Street }} +& \REF & 26.07 & 21.36 & 32.42 & 29.61 & 27.85 & 25.00 \\ +& \AVG & +0.26 & +0.82 & -0.66 & +0.42 & +0.83 & +1.26 \\ +& \CHKB & +1.05 & +0.43 & +0.43 & +0.95 & +1.13 & +1.38 \\ +& \EXP & +0.94 & +0.34 & +0.58 & +1.02 & +1.17 & +1.42 \\ +\hline +\multirow{\nlines}{*}{\vbox{Race Horses }} +& \REF & 29.34 & 23.88 & 33.00 & 29.41 & 27.49 & 24.55 \\ +& \AVG & -0.10 & +1.25 & +0.60 & +2.71 & +3.31 & +3.88 \\ +& \CHKB & +0.69 & +0.64 & +0.46 & +1.35 & +1.47 & +1.64 \\ +& \EXP & +1.11 & +0.84 & +0.79 & +1.80 & +1.94 & +2.11 \\ +\hline +\multirow{\nlines}{*}{\vbox{Tennis }} +& \REF & 32.50 & 25.69 & 36.50 & 31.90 & 29.67 & 26.15 \\ +& \AVG & -1.04 & +0.12 & -1.74 & -0.72 & -0.08 & +1.71 \\ +& \CHKB & +0.48 & +0.45 & +0.69 & +1.19 & +0.81 & +1.17 \\ +& \EXP & +1.15 & +1.17 & +1.20 & +2.03 & +1.95 & +2.12 \\ +% \hline +% \multirow{3}{*}{\vbox{\textbf{Average} }} +% & \textbf{\AVG} & \textbf{-0.95} & \textbf{-0.34} & \textbf{-1.16} & \textbf{-0.15} & \textbf{+0.25} & \textbf{+0.59} \\ +% & \textbf{\CHKB} & \textbf{+0.30} & \textbf{+0.38} & \textbf{+0.17} & \textbf{+0.78} & \textbf{+1.04} & \textbf{+1.33} \\ +% & \textbf{\EXP} & \textbf{+0.50} & \textbf{+0.53} & \textbf{+0.33} & \textbf{+1.00} & \textbf{+1.22} & \textbf{+1.47} \\ +\bottomrule +\end{tabular} +} +\end{table} + + +\begin{table}[!t] +\newcommand{\resultswidth}{1.3cm} +\newcommand\nlines{4} +\centering +\caption{Average quality (PSNR): Random-Access.} +\TABLESVSPACE +\small +\footnotesize +\label{tab:AverageQualityCHKBEXPRandomAccess} +\resizebox{\columnwidth}{!}{ +\begin{tabular}{p{2.2cm}|x{2.2cm}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}} +\toprule +\multirow{3}{*}{\centering\textbf{Sequence}} & \multirow{3}{*}{\vbox{\centering\textbf{Method}}} & \multicolumn{6}{c}{\textbf{Packet loss rate}} \\ +& & \multicolumn{2}{c|}{\textbf{\LOSTALL}} & \multicolumn{4}{c}{\textbf{\KEEPINTRA}} \\ +& & \textbf{1\%} & \textbf{5\%} & \textbf{1\%} & \textbf{3\%} & \textbf{5\%} & \textbf{10\%} \\ +\hline +\multirow{\nlines}{*}{\vbox{Basketball Drill }} +& \REF & 31.36 & 25.07 & 36.90 & 33.54 & 31.05 & 27.88 \\ +& \LONG & +0.09 & -0.12 & -0.02 & +0.31 & +0.57 & +0.54 \\ +& \CHKB & +0.12 & +0.20 & +0.15 & +0.46 & +0.60 & +0.69 \\ +& \EXP & +0.39 & +0.37 & +0.26 & +0.57 & +0.79 & +0.87 \\ +\hline +\multirow{\nlines}{*}{\vbox{Book Arrival }} +& \REF & 33.39 & 25.48 & 38.71 & 35.90 & 33.01 & 29.53 \\ +& \LONG & -0.14 & +1.01 & +0.18 & +0.38 & +0.96 & +0.94 \\ +& \CHKB & +0.30 & +0.10 & +0.07 & +0.06 & +0.31 & +0.70 \\ +& \EXP & +0.26 & +1.16 & +0.34 & +0.28 & +0.67 & +0.97 \\ +\hline +\multirow{\nlines}{*}{\vbox{Cactus }} +& \REF & 34.58 & 32.36 & 36.92 & 35.08 & 33.92 & 32.12 \\ +& \LONG & -0.32 & -0.10 & -0.34 & -0.32 & -0.30 & -0.30 \\ +& \CHKB & +0.15 & +0.22 & +0.21 & +0.36 & +0.42 & +0.53 \\ +& \EXP & +0.19 & +0.13 & +0.19 & +0.34 & +0.46 & +0.50 \\ +\hline +\multirow{\nlines}{*}{\vbox{Bosphorus }} +& \REF & 32.43 & 24.86 & 41.64 & 39.70 & 38.28 & 36.05 \\ +& \LONG & +0.09 & -0.87 & -0.60 & -0.62 & -0.61 & -0.66 \\ +& \CHKB & -0.22 & +0.37 & +0.19 & +0.43 & +0.55 & +0.63 \\ +& \EXP & +0.21 & +0.49 & +0.18 & +0.47 & +0.59 & +0.69 \\ +\hline +\multirow{\nlines}{*}{\vbox{BQSquare }} +& \REF & 33.85 & 23.63 & 37.85 & 33.45 & 30.44 & 27.10 \\ +& \LONG & -0.24 & +0.46 & -0.75 & -0.06 & -0.04 & -0.41 \\ +& \CHKB & +0.32 & +0.62 & +0.66 & +0.65 & +1.06 & +1.05 \\ +& \EXP & +0.42 & +0.31 & +0.50 & +1.11 & +0.93 & +0.90 \\ +\hline +%\multirow{\nlines}{*}{\vbox{Jockey }} +%& \REF & 33.86 & 26.52 & 36.80 & 31.82 & 29.27 & 26.21 \\ +%& \LONG & +1.22 & +0.45 & +0.70 & +0.50 & +0.25 & +0.05 \\ +%& \CHKB & +1.00 & +0.79 & +0.74 & +1.08 & +1.12 & +0.96 \\ +%& \EXP & +1.08 & +0.86 & +0.85 & +1.30 & +1.16 & +1.13 \\ +%\hline +\multirow{\nlines}{*}{\vbox{Kendo }} +& \REF & 36.81 & 28.11 & 39.78 & 34.69 & 30.73 & 26.62 \\ +& \LONG & +1.04 & +0.97 & +0.59 & +1.43 & +1.44 & +0.80 \\ +& \CHKB & +0.55 & +1.09 & +0.24 & +0.62 & +1.07 & +0.98 \\ +& \EXP & +0.84 & +1.65 & +0.37 & +0.95 & +1.53 & +1.42 \\ +\hline +\multirow{\nlines}{*}{\vbox{Park Scene }} +& \REF & 30.03 & 25.37 & 36.74 & 32.20 & 29.60 & 26.35 \\ +& \LONG & -0.15 & -0.27 & -0.56 & -0.44 & -0.39 & -0.12 \\ +& \CHKB & +0.24 & +0.16 & +0.17 & +0.51 & +0.56 & +0.58 \\ +& \EXP & +0.36 & +0.31 & +0.65 & +0.95 & +0.93 & +1.04 \\ +\hline +\multirow{\nlines}{*}{\vbox{People on Street }} +& \REF & 26.04 & 20.91 & 30.86 & 27.21 & 25.17 & 22.39 \\ +& \LONG & +0.26 & +0.14 & -0.22 & -0.27 & -0.30 & -0.27 \\ +& \CHKB & +0.56 & +0.81 & +0.56 & +0.64 & +0.68 & +0.68 \\ +& \EXP & +0.74 & +0.83 & +0.76 & +0.87 & +0.96 & +0.93 \\ +\hline +\multirow{\nlines}{*}{\vbox{Race Horses }} +& \REF & 30.43 & 24.11 & 33.22 & 29.14 & 26.84 & 23.93 \\ +& \LONG & -0.17 & -0.20 & +0.09 & +0.50 & +0.56 & +0.34 \\ +& \CHKB & +0.50 & +0.43 & +0.29 & +0.69 & +0.83 & +0.89 \\ +& \EXP & +0.50 & +0.38 & +0.54 & +1.08 & +1.10 & +1.08 \\ +\hline +\multirow{\nlines}{*}{\vbox{Tennis }} +& \REF & 31.89 & 25.89 & 36.74 & 32.20 & 29.60 & 26.35 \\ +& \LONG & +1.12 & +0.58 & +0.09 & +1.55 & +1.61 & +1.25 \\ +& \CHKB & +0.16 & +0.12 & +0.17 & +0.51 & +0.56 & +0.58 \\ +& \EXP & +1.46 & +1.09 & +0.65 & +0.95 & +0.93 & +1.04 \\ +% \hline +% \multirow{\nlines}{*}{\vbox{Traffic }} +% & \REF & 28.60 & 23.47 & 37.29 & 34.52 & 32.54 & 29.60 \\ +% & \LONG & -0.09 & -0.31 & -0.37 & -0.32 & -0.34 & -0.24 \\ +% & \CHKB & -0.05 & -0.07 & +0.16 & +0.54 & +0.63 & +0.96 \\ +% & \EXP & -0.09 & +0.06 & +0.41 & +0.61 & +0.76 & +0.95 \\ +% \hline +% \multirow{3}{*}{\vbox{\textbf{Average} }} +% & \textbf{\LONG} & \textbf{+0.23} & \textbf{+0.15} & \textbf{-0.10} & \textbf{+0.22} & \textbf{+0.28} & \textbf{+0.16} \\ +% & \textbf{\CHKB} & \textbf{+0.30} & \textbf{+0.40} & \textbf{+0.29} & \textbf{+0.53} & \textbf{+0.69} & \textbf{+0.78} \\ +% & \textbf{\EXP} & \textbf{+0.53} & \textbf{+0.64} & \textbf{+0.43} & \textbf{+0.75} & \textbf{+0.87} & \textbf{+0.94} \\ +\bottomrule +\end{tabular} +} +\end{table} diff --git a/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex b/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex new file mode 100644 index 0000000..09466c7 --- /dev/null +++ b/042_ConcealmentAwareEncoding/ConcealmentAwareEncoding.tex @@ -0,0 +1,751 @@ +% Figure folders +\newcommand{\ConcealmentAwareDir}{./042_ConcealmentAwareEncoding} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% CHAPTER +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\chapter{Error Concealment-Aware Coding for Robust Video Transmission} +\label{Chapter:ConcealmentAwareTechniques} +%\vspace{-6mm} +%\clearpage + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Introduction} + +\glsreset{ec} +\glsreset{rd} + +In this chapter an error concealment-aware encoding scheme is proposed to improve the quality of decoded video in broadcast environments prone to transmission errors and data loss. The method aims to improve the quality of recovered lost frames by optimising the efficiency of the \ac{ec} beforehand at the encoder-side. +The proposed scheme is based on a scalable coding approach where the best \ac{ec} methods to be used at the decoder are optimally determined at the encoder and signalled to the decoder through \ac{sei} messages. Such optimal \ac{ec} modes are found by simulating transmission losses followed by a lagrangian optimisation of the \textit{signalling rate - \ac{ec} distortion} cost. A generalised saliency-weighted distortion is used and the residue between coded frames and their EC substitutes is encoded using a rate-controlled enhancement layer. When data loss occurs the decoder uses the signalling information is used at the decoder, in case of data loss, to improve the reconstruction quality. + +%In this chapter a novel error concealment-aware video encoding scheme is proposed to increase the robustness of \ac{hevc} streams over transmission channels prone to errors or data loss, such as broadcasting networks. +%The method aims to improve the quality of recovered lost frames by optimising the efficiency of the error concealment (\ac{ec}) beforehand at the encoder-side. The proposed approach is based on finding the optimal \ac{ec} mode, in \ac{rd} sense, to be used at the decoder side. The process includes data loss simulation and minimisation of a lagrangian cost based on saliency-weighted distortion and signalling rate. Furthermore the residue +%between the coded frame and its error reconstructed version is also encoded using a rate-controlled enhancement scalable layer. + +This chapter is organized as follows. Section~\ref{Section5:Problem} presents the problem formulation and motivation for the method proposed in this chapter. In Section~\ref{Section5:ECCandidates} does an initial evaluation of different error concealment schemes in order to evaluate the relevant ones to be used in the proposed error concealment-aware coding scheme. +Section~\ref{Section5:OverviewProposedMethod} provides an overview of the architecture of the proposed method and Section~\ref{Section5:ProposedMethod} describe the proposed method in detail. +Section~\ref{Section5:ECSignallingPerformance} presents a discussion of the performance of the proposed \ac{ec} optimisation and signalling, and Section~\ref{Section5:OverallPerformance} presents an overall performance of the proposed method. +Finally, Section~\ref{Section5:Summary} concludes this chapter. + +\clearpage +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Problem formulation} +\label{Section5:Problem} + +\glsreset{mve} + +In Chapter~\ref{Chapter:ErrorResilienceReview} described different \ac{ec} algorithms that can effectively reconstruct lost data, leading to lower error propagation. However, as mentioned before different algorithms might not present the same accuracy for every lost frame or slice. Moreover, selection at the \ac{mb}/\ac{pu} level is not straightforward. Approaches in the past proposed the combination of spatial and temporal \ac{ec} strategies to improve the reconstruction quality~\cite{HybridConcTB2008}. Regarding temporal \ac{ec}, there are hybrid algorithms to optimise the \ac{mve} at the pixel and block levels \cite{HybridConcealmentTB2008,2DConcHybridMotion2010}. Regarding the \ac{hevc}, existing algorithms also aim to improve the reconstruction accuracy of the motion-based recovery by using the information of correctly received neighbours, e.g., block partitions~\cite{HEVCMCBlockAug2013} and prediction residue~\cite{HEVCMCResidueDec2013}. Although such methods are able to improve the reconstruction quality and reduce the artefacts, decoder-based decisions are still not fully accurate, as they do not have enough reference information to ease the decision process. Therefore, higher performance can be achieved by performing encoding-side estimations and decisions based on the error-free information. This approach not only improves quality but also reduces the complexity of the video decoders. + +In this chapter a robust coding driven by \ac{ec} methods with the objective of enhancing the quality of reconstructed video at decoders receiving erroneous streams from the transmission network. The proposed approach combines error resilience tools at the encoder-side with \ac{ec} at the decoder-side. Different approaches were proposed in the past to that address the problem of \ac{ec} at the encoder-side \cite{ConcealmentDrivenSliceOrdering2008,ConcealmentAwareME2006,ErrorResilienceModeHEVC2016}. Moreover, extra information was used to aid the \ac{ec} operation at the decoder-side, either by sending redundant data \cite{ResilienceRECAP_ICASSP2009,SaliencyErrorConcealment_TM2013} or by embedding data into the bitstream \cite{ConcealmentIntraDataHidng2009}. Although the above methods facilitate the EC performance either through slice reordering or based on \acs{mv}, they always rely on the assumption that a fixed \ac{ec} method is used at the decoder. The work presented in \cite{ConcealmentSignalling2008,SignallingRobustScalable2015} proposes testing several methods with transmission of extra symbols to indicate the best method to be applied by the decoder. Although such approaches achieves reasonably good performance, it relies on the spatial neighbouring information or multi-layer video, which may not be available in all cases. + +To advance one step further, in this chapter an EC-aware encoding scheme is proposed to achieve higher flexibility in terms of the error concealment approaches which leads to higher recovered video quality. +This is accomplished by selecting at the encoder, among several \ac{ec} algorithms with different characteristics, the one that yields the lowest reconstruction distortion at the decoder and also minimises the \ac{rd} cost considering the necessary signalling rate. Thus, the proposed approach outperforms similar previous methods \cite{ConcealmentAwareME2006,ErrorResilienceModeHEVC2016} by selecting the best error concealment method without affecting the encoder optimisation. +To further improve the quality of recovered frames, an enhancement layer is encoded with the mismatch residue between the transmitted and recovered frames. This is a dynamic optimisation process, which is more efficient than traditional approaches based on redundant pictures \cite{RedudantPicsJan2009,VPxResiliencePCS2016}, which transmit the whole frame as redundancy. +In contrast with previous work presented in Chapter~\ref{Chapter:ErrorResilienceTechniques} that addressed the error resilience problem by optimising the coding dependencies, the approach followed in this work considers both \ac{ec} performance and the rate required for signalling the best \ac{ec} mode to the decoder. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Traditional error concealment algorithms} +\label{Section5:ECCandidates} + +In order to proposed a novel error concealment-aware technique different algorithms were taken into consideration. This section provides a description and evaluation of the error concealment candidates used in the proposed method. Firstly, a detailed description of each approach is provided followed by a performance comparison between each method. + +\subsection{Motion-compensation algorithms} +%\subsection{Motion-compensated reconstruction} + +The most common algorithms to recover missing frames used the correctly received \acs{mv} from the neighbouring frames. This has been widely used in the past for \ac{h264} standard \cite{H264RefConc2006,BlockBasedMVE2002,BlockBasedMVEOverllaped2011} and also in the \ac{hevc} standard~\cite{HEVCMCBlockAug2013,HEVCMCResidueDec2013,DownhillHEVCCon_ICCE2017}. + +Let's consider the example of Figure~\ref{FIG5:MotionVectorExtrapolation}, where a missing frame ($f_0$) at time instance $t_0$ is reconstructed based the information present in the neighbouring frames, $f_{-1}$ and $f_{-2}$. +One of the common approaches, i.e., motion-copy based algorithms reconstructs the missing frame through motion compensation directly using the co-located vectors in the closest neighbour frame ($f_{-1}$). The \acs{mv} for the motion compensation ($v_0$) are obtained by copying the \acs{mv} $v_{-1}$ from the same position in $f_{-1}$, as shown by the example (1) in the figure. + +\begin{figure}[b] + \centering + \input{\ConcealmentAwareDir/Figures/MotionVectorExtrapolation.tikz} + \caption{Representation of the video frame with associated original and extrapolated motion vectors represented at full and dashed arrows, respectively.} + \label{FIG5:MotionVectorExtrapolation} +\end{figure} + +An alternative approach reconstructs the missing frame by using motion compensation through the correctly received \acs{mv}, of frame $f_{-1}$ at time instant $t_{-1}$, according to the following procedure: +The \acs{mv} ($v_{-1}$) of the neighbouring frame ($f_{-1}$) are extrapolated to compute the set of \acs{mv} ($v_{0}$) for the missing frame as follows: +\begin{equation} +% v^\zeta_0 = \frac{ t_0 - t_{C} }{ t_{-1} - t_{R} } \times v^\zeta_{-1},\ \zeta=s,r +v_0 = \frac{ t_0 - t_{-1} }{ t_{-1} - t_{R} } \times v_{-1}, +\end{equation} +where $t_{R}$ is the time instant of the reference frame pointed by the original \acs{mv} $v_{-1}$. Then, the \ac{mv} associated to the block $b_{-1}(x,y)$ at spatial position $(x,y)$ in $f_{-1}$ may be used to recover the block $b^{'}_{{0}}(x^{'},y^{'})$ in $f_0$, at position $(x^{'},y^{'})$ obtained from $v_0$ components as follows: +\begin{align} +x^{'} &= x - v_{0x} \\ +y^{'} &= y - v_{0y} +\end{align} +This is illustrated in Figure~\ref{FIG5:MotionVectorExtrapolation} by the example (2). This method is referred to as \glsfirst{mve}. + +Finally, as a base for comparison the default \ac{ec} implemented in the reference \ac{hevc} implementation the \ac{fc} algorithm is also evaluated. This method consists in replacing the lost frame with a copy of the previous decoded one, which can also be described as a motion compensated reconstruction using only zero \acs{mv}. + +\subsection{Algorithms based on motion estimation} + +In this sub-section describe different method that can be used to estimate a motion-field, which can be used to reconstruct the missing information. The obtained motion-field is used in combination with the \ac{mve} method to recover the lost frame. + +Optical flow estimation algorithms are used to estimate a 2-D \ac{mv}, $v=(v_x, v_y)$ for each pixel, such as, the following energy is minimised: +\begin{equation} +E(x,y) = \sum_{x} \sum_{y} \bigg| f_{0}\big(x,y\big) - f_{1}\big( x + v_x(x,y), y+v_y(x,y) \big) \bigg|^2, +\end{equation} +resulting an unique motion field given by $v$. +Besides using a full search algorithm where all block-based approach is used to search for the best \acs{mv} based on the sum of absolute errors, in this work three different approaches were tested to evaluate they effectiveness to be used in \ac{ec} algorithms, namely: +\begin{enumerate} + \item a method based on the total variation based on the minimization of the \mbox{L-1} norm~\cite{DualFlow2016}; + \item a method which approximates the neighbouring information of a given pixel by quadratic polynomials on both frames~\cite{PolyFlow2003}; + \item a method based on a combination of descriptor matching and large displacement estimation~\cite{DeepFlow2013}. +\end{enumerate} + + +\subsubsection{Motion estimation based on total variation and regularisation (\textit{DualFlow})} + +In this method the motion field is estimated based on the minimization of the total variation obtained using the L-1 norm \cite{DualFlow2007,DualFlow2016} and taking into account the Horn and Schunck smoothness condition~\cite{HornSchunckOpticalFlow1981}. This condition forces the estimated field to be regular across neighbouring pixels by penalising deviations in a quadratic way. +Using this method, for each pixel $\mathbf{p}$, the motion field $v(\mathbf{p})$ is obtained by minimising the following energy function, +\begin{equation}\label{EQ5:DualFlow} +E(v) = +\sum_{\mathbf{p} \in \Omega} \Big( \big| f_{-1}(\mathbf{p}) - f_{-2}( \mathbf{p} + v(\mathbf{p}))\big| + \big|\nabla v(\mathbf{p})\big| \Big)%d\mathbf{p},\ \mathbf{p} \in \Omega +\end{equation} +where $f_{-1}$ and $f_{-2}$ are the two previous frames at time instances $t_{-1}$ and $t_{-2}$, as represented in Figure~\ref{FIG5:MotionVectorExtrapolation}. In the equation, $\Omega$ defines the sets of pixels of those frames. +$|\nabla v(\mathbf{p})|$ is a weighting factor that penalises high variation of the optical flow which effectively disallow discontinuities. This is based on the assumption that neighbouring regions belong to the same object, thus such regions should have similar motion. + + +\subsubsection{Motion estimation based on polynomial expansion (\textit{PolyFlow})} + +In this method the motion field is estimated based on a polynomial expansion a neighbourhood of a pixel~\cite{PolyFlow2003}. +In this method only quadratic polynomials are considered, which gives the local signal model, expressed in a local coordinate system as +\begin{equation}\label{EQ5:PolyFlow} +f(\mathbf{x}) \sim \mathbf{x}^TA_1\mathbf{x}+\mathbf{b}_1^T\mathbf{x}+C_1 +\end{equation} +where $\mathbf{A}$ is a symmetric matrix, $\mathbf{B}$ a vector and $C$ a scalar. The coefficients are +estimated from a weighted least squares fit to the signal values in the neighbourhood. + +Since the result of polynomial expansion is that each neighbourhood is approximated by a polynomial, we start by analysing what happens if a polynomial undergoes an ideal translation. Let's consider the signal $f_1(\mathbf{x})$ represented by Equation (\ref{EQ5:PolyFlow}) and a signal $f_2$ which corresponds to a displacement of $f_1$ given by $\mathbf{d}$. This can be expressed by the following: +\begin{align} +f_2(\mathbf{x}) +&= f_1(\mathbf{x}+\mathbf{d}) \nonumber \\ +&= (\mathbf{x}+\mathbf{d})^TA_1(\mathbf{x}+\mathbf{d})+\mathbf{b}_1^T(\mathbf{x}+\mathbf{d})+C_1 \\ +&= \mathbf{x}^TA_2\mathbf{x}+\mathbf{b}_2^T\mathbf{x}+C_2, \nonumber +\end{align} +which results in the following relations, +\begin{align} +\mathbf{A}_2 &= A_1 \nonumber \\ +\mathbf{b}_2 &= \mathbf{b} + 2 \mathbf{A}_1\mathbf{d}, \\ +C_2 &= \mathbf{d}^T\mathbf{A}_1\mathbf{d}+\mathbf{b}_1\mathbf{d}+C_1. \nonumber +\end{align} +This allows us to find the displacement $\mathbf{d}$ for any signal dimensions by solving the above equations. Detailed implementations of the polynomial expansion and solving can be found in~\cite{NormalizedDiffConvolution1993,farneback2002polynomial}. + +\subsubsection{Large displacement optical flow with deep matching (\textit{DeepFlow})} + +In this method the motion field is estimated based on the large displacement optical flow presented in \cite{LargeDisplacementOptFlow_TPAMI2011}, which extends the total variation methods, as the one expressed by Equation (\ref{EQ5:DualFlow}), including descriptor matching. The correspondences between descriptions are used to apply a coarse-to-fine wrapping, thus avoiding local minimum values. +The matching algorithm is built upon a multi-stage architecture with six layers with different patches sizes, interleaving convolutions and max-pooling \cite{DeepFlow2013}. The combination of both approaches leads towards a better matching. + +In \cite{DeepFlow2013} to improve the estimated motion field accuracy, the following extension to the method proposed in~\cite{LargeDisplacementOptFlow_TPAMI2011} were introduced: (i) description matching was used in the minimisation problem; (ii) a normalisation parameter was introduced to down-height the impact of regions with high spatial changes; (iii) different weights are used for each layers of optimisation. Figure~\ref{FIG5:DeepFlowDiagram} shows the blocks that are included in the estimation process. As shown in the figure a multi-size optimisation is applied by using different patches sizes. This is used to estimate an initial motion field, referred to as $v'$ . + +\begin{figure}[t] + \centering + \includegraphics[width=.95\columnwidth]{\ConcealmentAwareDir/Figures/DeepFlowBlockDiagram} + \caption{Block diagram of the deep matching algorithm \cite{DeepFlow2013}.} + \label{FIG5:DeepFlowDiagram} + \FIGSVSPACE +\end{figure} + +Subsequently, the optical flow is obtained by minimising an energy function, which can be described by the following: +\begin{equation} +E(v) = +\sum_{\mathbf{p} \in \Omega} \Big( E_D(\mathbf{p}) + \alpha E_S(\mathbf{p}) + \beta E_M(\mathbf{p}) \Big), +\end{equation} +where $E_D$ is the data term, $E_S$ is a smoothing term and $E_M$ is the matching term. +The data term is obtained from the difference between matching pixels in both video frames. +The smoothness term is a robust penalisation of the gradient of the flow as used in the Equation (\ref{EQ5:DualFlow}), i.e., $E_S(\mathbf{p})=|\nabla v(\mathbf{p})|$. Finally, the matching term ($E_M$) is used to encourage the flow estimation to be similar to the precomputed motion field~($v'$). + + +\subsection{Experimental evaluation} + +In this section a performance evaluation of the different \ac{ec} schemes previously described is carried out. This evaluation aims at finding the suitable methods to be used for efficiently reconstruct the missing frames in case of data loss. In this evaluation the default \ac{ec} method of the reference \ac{hevc} implementation is used as a reference for comparison. Then, two method based on the correctly received \acs{mv} are tested, i.e., \ac{mc} and \ac{mve}. Moreover, the four different approaches based on a estimated motion field are compared: (i) full search based on the minimisation of the sum of absolute errors (\FULLSEARCH), (ii) optical flow based a duality between total variation and smoothness term (\DUALFLOW), (iii) optical flow based on polynomial approximations (\POLYFLOW) and (iv) optical flow based on deep matching (\DEEPFLOW). + +Table~\ref{TAB5:ECPerformaceComparison} shows the average quality results obtained using the \ac{psnr} metric for different methods. The absolute value is shown for the frame-copy method (\FC), and differential values are shown for the remaining methods. +Results reveal that the methods \MC and \MVE are able to overcome \FC which does not use any \acs{mv}, achieving average quality gains of up to 1.41 dB and 1.46 dB, respectively. Moreover, by extrapolating the \acs{mv} from the correctly receiving frame to the lost time instance (\MVE) is able to outperform simple motion-copy (\MC). + +\begin{table}[tb] + \centering + \caption{Average quality (\ac{psnr}) using different error concealment algorithms} + \label{TAB5:ECPerformaceComparison} + \TABLESVSPACE + \input{\ConcealmentAwareDir/Results/PerformanceConcealmentMethods.tex} +\end{table} + +Results for \ac{ec} methods based motion estimation (see columns 5 to 8 of Table~\ref{TAB5:ECPerformaceComparison}) reveal a overall superior performance when comparing with methods which rely of correctly received \acs{mv}. +Results for \FULLSEARCH method reveals that only minimising the sum of absolute errors between two consecutive frames might not always be directly related with objects' motions. Thus, this may results in a inaccurate motion estimation to be used for missing frames reconstruction. Finally, results from Table~\ref{TAB5:ECPerformaceComparison} reveal that the \DEEPFLOW algorithm leads to the most accurate motion field to be used for \ac{ec}, in comparison with the other methods. It is able to achieve an average quality gain of 1.79 dB, and up to 3.39 dB for Book Arrival sequence, where \POLYFLOW only achieves 3.05 dB. + +\subsection{Discussion and conclusions} + +Summarising, the analysis of the previous results indicates that all methods cope with lost frames and manage to recover the missing data with acceptable quality. Nevertheless, it is noticeable that the recovering lost frames with an estimated motion field leads to higher quality, especially using the \DEEPFLOW method. However, as the other methods tested are also able to achieve acceptable video quality they shall not be neglected. Thus, in order to achieve higher reconstruction quality different approaches shall be considered. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Proposed architecture} +\label{Section5:OverviewProposedMethod} + +\glsreset{ec} + +This section presents an overview of the proposed method architecture describing the functional blocks presented providing an introduction for the subsequent detailed descriptions. + +Figure~\ref{FIG5:BlockDiagram} illustrates the architecture of the proposed encoding scheme. +After the video encoder, it includes data loss simulation at the \ac{nal} for every slice. Subsequently, error concealment is applied to find the best \ac{ec} methods that should be used to reconstruct the lost slices in case of transmission errors. +Saliency-weighted \ac{ec} mode decision is used to select the optimal \ac{ec} mode from the best trade-off between \ac{ec} distortion, using the original input video ($f$) as reference, and the number of bits required for signalling. +Then the optimal mode is sent to the decoder along with the residue formed between the error-free decoded frame and the error concealed one, using the scalable extension of the \ac{hevc} standard. + +\begin{figure}[tb] +\centering +%\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram} + \input{042_ConcealmentAwareEncoding/Figures/BlockDiagram.tikz} +\caption{Architecture of the error concealment-aware encoding scheme.} +\label{FIG5:BlockDiagram} +\FIGSVSPACE +\end{figure} + + +For the \ac{ec} mode decision optimization the missing slices are reconstructed, as they would be in the decoder, but using different \ac{ec} methods based on motion estimation and extrapolation techniques, to selected the best one at the encoder-side. +In this process, each \ac{ctu} is partitioned to form a quadtree structure of \ac{ecu} that are used to test the different \ac{ec} candidate modes. +Then, the best trade-off between \ac{ecu} partition and signalling is found through \ac{rd} optimisation. +In order to achieve an efficient use of the signalling overhead a saliency-weighted procedure is embedded in the \ac{rd} optimisation based on a spatio-temporal saliency, where more signalling bits are allocated to regions with higher saliency values. +Finally, the signalling information is transmitted to the decoder by using the \ac{sei} \ac{nal} unit type~\cite{HEVCHighLevelReferenceDez2012}. +Since in general, the optimal \ac{ec} mode is still not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable enhancement layer. +This residual information contains the difference between the compressed ($\hat f_t$) and the \ac{ec} frames ($\tilde f$). + +The transmitted information is used to assist the \ac{ec} operation at the decoder to recover erroneous frames with high accuracy, by using the best \ac{ec} mode and also residual information to reduce error propagation. +The remaining of this section describes in detail the building blocks of the proposed encoding architecture, i.e., the \ac{ec} candidate modes, the saliency-weighted optimal \ac{ec} mode decision, signalling and enhancement layer coding. + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%\section{Saliency-weighted error concealment mode decision} +\section{Error concealment mode decision} +\label{Section5:ProposedMethod} + +In this section the saliency-weighted \ac{ec} mode decision algorithm is described. Firstly, the list of \ac{ec} candidates from the previously described methods are enumerated. Then the \ac{rd} optimisation is proposed based on a saliency detection to decrease the overall bitrate without compromising the reconstruction quality. Moreover, the signalling coding strategy is also described. Finally, the introducing of \ac{ec} residue as auxiliary information to improve the reconstruction quality is described. + + +\subsection{Error concealment candidates} + + + +A set of four \ac{ec} candidate modes (M1...M4) is used in the optimisation process that selects one of them as best for each particular \ac{ecu}. Each \ac{ec} mode corresponds to a different \ac{ec} method, as shown in Figure~\ref{FIG5:concealmentcandidates}, namely: +\begin{itemize} + \item M1, M2: These two methods are based on extrapolation of an estimated motion field; + \item M3: This method is based on \ac{mv} extrapolation of previously received \acs{mv}; + \item M4: This method uses the co-located motion information from the closest neighbour; +\end{itemize} + +\begin{figure}[tb] +\centering +%\includegraphics[width=0.7\columnwidth]{\ConcealmentAwareDir/Figures/ConcealmentCandidates} + \input{\ConcealmentAwareDir/Figures/ConcealmentCandidates.tikz} +\caption{\ac{ec} candidates used for optimization.} +\label{FIG5:concealmentcandidates} +\FIGSVSPACE +\end{figure} + +The first two methods M1, M2 rely on an estimated motion field, using the method \DEEPFLOW \cite{DeepFlow2013}, to recover the lost frame. Then, it is extrapolated at the pixel level, i.e., one \ac{mv} is assigned to each pixel, and \ac{cu} level, i.e., one \ac{mv} is assigned to the entire block, at the missing frame instant. This results in two different reconstructions of the missing frame, where the former may increase the accuracy of the pixel extrapolation, but the latter has the advantage of keeping the block structure and generate a smoother reconstruction. + +The method M3 is based on a traditional \glsfirst{mve}, and finally M4 reconstructs the missing frames using a \glsfirst{mc} algorithm. + + +\subsection{Selection optimization} + +As described above, in the sub-section~\ref{Section5:ECCandidates}, the proposed method includes four \ac{ec} candidates, which result in different recovered frames for each lost frame (simulated). +The lost frame is partitioned into quadtree structures of \ac{ecu}, each one forming a set of variable size \acs{ecu}. +Then, finding the optimal \ac{ec} method for each \ac{ecu} is formulated as a rate-distortion cost minimisation. To solve the aforementioned problem, a Lagrangian optimisation is used~\cite{LagragianMultiplier1982} to achieve the best trade-off between reconstruction quality and signalling overhead. +Furthermore, image saliency driven by visual attention modelling is used in the optimisation process to weight the cost of the overhead information of each \ac{ecu}, according to the visual relevance of its content. + +% \HIGHLIGHT{% +% The optimal depth $Level$ ($l^*$) that determines the best \ac{ecu} size is obtained by minimising the following lagragian cost.} +% \begin{equation}\label{eq:optdepth} +% \newcommand{\argmin}{\operatornamewithlimits{arg\ min}} +% l^* = \argmin_{l=0..3} \left\{ D(l) + \left(1+\frac{t_0-t_I}{T_{I}}\right) \lambda R(l) \right\}, +% % l^* = \argmin_{l=1..3} \left\{ D(l) + \lambda \left(1-\frac{t_I-t_c}{T_{\ac{ird}}}\right) R(l) \right\}, +% \end{equation} +% \HIGHLIGHT{where $R(l)$ represents the number of overhead bits required to encode \ac{ec} mode, and $D(l)$ is mean square error of the \ac{ctu} formed by $4^l$ ECUs. +% } +% \HIGHLIGHT{% +% In equation \ref{eq:optdepth}, besides the parameter $\lambda$ used to trade-off the cost of distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(l)$ according to the duration of the error propagation decreases. This is given by the ratio between the temporal distance from the current frame ($t_0$) to the next refresh frame ($t_I$) and the period of refresh frames ($T_{I}$). +% } +% \begin{align} +% (ecm^*, l^*) &= arg \min_{\phi \in M} J(ecm, l), \\ +% J(ecm,l) &= D(ecm,l) + \left(1+\frac{t_0-t_I}{T_{I}}\right) \lambda R(ecm,l), +% \label{eq:Lagrangian} +% \end{align} + +The optimal \ac{ec} mode $m_{l,i}^*$, for the $i_{th}$ \ac{ecu} which size is defined by the quadtree partitioning level $l$, is obtained by minimising the following lagrangian cost: +\begin{equation}\label{eq:optdepth} +\small +\newcommand{\argmin}{\operatornamewithlimits{arg\ min}} +% (m^*,l^*) = \argmin_{m,l} \left\{ D(m,l) + \left(1+\frac{t_0-t_I}{T_{I}}\right) \lambda R(m,l) \right\}, +% l^* = \argmin_{l=1..3} \left\{ D(l) + \lambda \left(1-\frac{t_I-t_c}{T_{\ac{ird}}}\right) R(l) \right\}, +m_{l,i}^* = \argmin\limits_{\footnotesize \substack{l\in\left\{0...3\right\} \\ i\in\left\{1...4^l\right\}}}\left\{ D(m_{l,i}) + \left(1+\frac{t_0-t_I}{T_{I}}\right) \lambda R(m_{l,i}) \right\}, +\end{equation} +where $R(m_{l,i})$ represents the number of overhead bits required to encode the \ac{ec} mode, and $D(m_{l,i})$ is sum of absolute error of the \ac{ecu}. + + +In equation \ref{eq:optdepth}, besides the parameter $\lambda$ used to trade-off the cost of distortion and overhead rate in the minimisation process, there is also a weighting factor to control the cost of $R(m_{l,i})$ according to the duration of the error propagation. This is given by the ratio between the temporal distance from the current frame ($t_0$) to the next refresh frame ($t_I$) and the period of refresh frames ($T_{I}$). This weight varies between $\left[0;1 \right]$, achieving the lowest value for intra-coded frames and the maximum for the frame located immediately before an intra-coded one. +Moreover, a dynamic value for $\lambda$ is used to adjust the amount of overhead across the frame, according to the visual relevance of each image region given by a saliency map. + +\subsection{Saliency detection} + +The saliency maps were computed based on the well-known Itti's attention model \cite{SaliencyItti1998}, which applies a hierarchical decomposition based on a set of spatial features and then combines them with center-surround weighting to obtain the saliency map. +Since the Itti's model only produces saliency maps for still images, in this work it is further combined with a temporal feature. This is obtained by estimating the optical flow between two consecutive frames, based on the algorithm presented in \cite{HornSchunckOpticalFlow1981}. The final saliency map is obtained by combining the spatial and temporal components, $S_{s}$ and $S_{t}$ respectively, as follows, +\begin{equation}\label{eq:saliency} +S = \rho S_{t} + (1-\rho) S_{s} +\end{equation} +The parameter $\rho$ is used to give higher relevance to the temporal component, since motion is correlated with error propagation, producing longer artefacts and consequently higher impact in visual attention. In this work the temporal component was given weight slightly higher than 50\%, i.e. $\rho=0.6$. + +On the one hand, higher values of saliency (i.e. regions with higher visual relevance) should result in lower values of $\lambda$, which tend to increase the overhead rate value on those regions, thus reducing \ac{ec} distortion at the decoder in case of loss. On the other hand, higher values of $\lambda$ should reduce the overhead, allowing higher \ac{ec} distortion at the decoder in case of loss. In order to achieve the aforementioned goal the following definition for $\lambda$ is used: +\begin{equation}\label{eq:lambdasaliency} +% \lambda = \lambda_{0} - \frac{S_{\ac{ctu}} - \mean S}{\Delta S}, \Delta S > 0, +% \lambda = \lambda_{0} + \frac{\Delta S}{S_{\ac{ctu}} - \mean S}, +% \lambda = \lambda_{0} e^{\frac{\mean S - S_{\ac{ctu}}}{\Delta S}}, \Delta S > 0, +\lambda=\begin{cases} +\lambda_{0} e^{\frac{\mean S - s}{\Delta S}}, & \text{if $\Delta S > 0$}.\\ +\lambda_{0}, & \text{no saliency used}. +\end{cases} +\end{equation} +where $s$ is the saliency value of the current \ac{ctu}, $\mean S$ is the average saliency value and $\Delta S$ is the difference between the maximum and minimum. +One should note that the method is not dependent on the existence of a saliency map. If the saliency map is constant or non-existent a fixed value of $\lambda$ is used, which means that no distinction is made across image regions. + + +\subsection{Signalling the optimal EC modes} + +\begin{figure}[tb] +\centering +\includegraphics[width=0.8\columnwidth]{\ConcealmentAwareDir/Figures/ConcealmentSignalling} +% \input{\ConcealmentAwareDir/Figures/ConcealmentSignalling.tikz} +\caption{Payload structure of the \ac{sei} \acs{nal}} % to signal the optimal error concealment method.} +% Symbols transmitted through \ac{sei} messages to signal the optimal error concealment method.} +\label{FIG5:signalling} +\end{figure} + +% \begin{figure}[tb] +% \begin{algorithmic} +% \Procedure{encodectuecm}{$level$} +% \If{$level=3] (p2) -- (p2 -| LossSim.north); +\path[arrow, shorten <=3] (p2 -| LossSim.north) -| (ConcDiff) node[right, pos=1, yshift=-.3cm] {$\hat f$}; + + +%% EC mode coding +\node [propBlockNoDraw, anchor=north west, text width=1.4cm, minimum height=1cm] at (ConcOpt.north -| Scalable.west) (Signalling) {EC Mode Coding}; +\node [refBlockNoDraw, text width=.2cm, right=-.025 of Signalling.south east, anchor=south west, text centered, minimum height=1cm] (SignallingNAL) {N\\[-2pt]A\\[-2pt]L}; +\DrawEdgesTwoNodes{Signalling}{SignallingNAL}; % Draw border +\path[arrow,-,densely dotted] (SignallingNAL.south west) -- (SignallingNAL.north west); % Draw inside lines +\path[arrow] (ConcOpt.east |- Signalling) -- (Signalling) node[above, pos=.45, yshift=-1] {\scriptsize M1...M4}; + +%% Input +\node[text width=.5cm, text centered] at (-1.7,4.5) (Input) {Input video}; +\path[arrow] (Input) |- (Encoder); +\path[arrow] (Input |- ConcOpt) -- (ConcOpt) node[point] at (Input |- ConcOpt) {}; + +%% TS Mux and output +\path (Encoder.south) -- (Encoder |- Signalling.north) node[midway] (middleRef) {}; % Find middle ref +\node [refBlock, right=8.5 of middleRef, text width=.7cm, minimum height=5.5cm] (NAL) {TS Mux}; +\node [right of=NAL, node distance=1.5cm, coordinate] (Out) {}; + +\path[arrow] (EncoderNAL) -- (NAL.west |- EncoderNAL) node[above,pos=1, xshift=-.4cm] {VCL$_0$}; +\path[arrow] (ScalableNAL) -- (NAL.west |- Scalable) node[above,pos=1, xshift=-.4cm] {VCL$_1$}; +\path[arrow] (SignallingNAL) -- (NAL.west |- Signalling) node[above,pos=1, xshift=-.4cm] {SEI}; +\path[arrow] (NAL) -- (Out) node[above,pos=1, xshift=-.2cm, text width=1.3cm] {Transport Stream}; + + +\end{tikzpicture} + + diff --git a/042_ConcealmentAwareEncoding/Figures/ConcealmentCandidates.pdf b/042_ConcealmentAwareEncoding/Figures/ConcealmentCandidates.pdf new file mode 100644 index 0000000..d58ded7 Binary files /dev/null and b/042_ConcealmentAwareEncoding/Figures/ConcealmentCandidates.pdf differ diff --git a/042_ConcealmentAwareEncoding/Figures/ConcealmentCandidates.tikz b/042_ConcealmentAwareEncoding/Figures/ConcealmentCandidates.tikz new file mode 100755 index 0000000..00b72cf --- /dev/null +++ b/042_ConcealmentAwareEncoding/Figures/ConcealmentCandidates.tikz @@ -0,0 +1,97 @@ +\usetikzlibrary{calc,trees,positioning,arrows,chains,shapes.geometric,% +decorations.pathreplacing,decorations.pathmorphing,shapes,% +matrix,shapes.symbols,plotmarks,decorations.markings,shadows,backgrounds} + +\def\MVE{\mbox{MVE}} +\def\MVBASE{\mbox{BASE\_MC}} +\def\INTERVIEW{\mbox{INTERVIEW\_COPY}} +\def\COMBMC{\mbox{COMB\_MC}} +\def\COMBMCDISP{\mbox{COMB\_MC\_Disp}} + +\newcommand{\scale}{0.4} + +% Define block styles +% \tikzstyle{simple_block} = [rectangle, text width=1.4cm, text centered, minimum height=1cm,node distance = 1.7cm] +% \tikzstyle{block} = [simple_block,draw,fill=gray!10] +% \tikzstyle{node_circle} = [circle, draw, text centered] +% +% \tikzstyle{decision} = [diamond, draw, text centered, node distance=3cm] +% \tikzstyle{line} = [draw, -latex', thick] + + +\def\scale{1} + + + + +\begin{tikzpicture}[ + font=\footnotesize, + auto, + every node/.style={scale=\scale}, + template/.style={thick, text centered}, + block/.style={template, draw,fill=\TikZColorFill, rectangle, text width=2.2cm, text centered, minimum height=.9cm, node distance=1cm}, + labels/.style={template, scale=1.2}, + line/.style={template, draw, -latex', thick}, + point/.style={template, draw, fill=black, circle,inner sep=0, minimum size=3} +] + +\tikzset{ + between/.style args={#1 and #2}{ + at = ($(#1)!0.5!(#2)$) + } +} + + +\node [block] (BMVE) {Block-based \\ MVE}; +\node [block, right=of BMVE] (PMVE) {Pixel-based \\ MVE}; +\node [block, right=of PMVE] (MVE) {MVE}; +\node [block, right=of MVE] (MC) {MC}; + +\node[coordinate, yshift=1cm, between=BMVE and PMVE] (p1) {}; +\node[coordinate, yshift=1cm, between=MVE and MC] (p2) {}; + +\node [block, above=.6cm of p1, text width=3.7cm] (MotionEst) {Optical flow estimation}; +\path [line] (MotionEst) |- (BMVE |- p1) -- (BMVE); +\path [line] (MotionEst) |- (PMVE |- p1) -- (PMVE); + + +\node [block, text width=2cm] at (MotionEst -| p2) (MotionExtraction) {MV parser}; +\path [line] (MotionExtraction) |- (MVE |- p2) -- (MVE); +\path [line] (MotionExtraction) |- (MC |- p2) -- (MC); + +\node[above=0cm of p2, xshift=.3cm] {$v_{-1}$}; + +\node[coordinate, between=PMVE and MVE] (p3) {}; +\node[labels,above=3.4 of p3] (FTM1) {$f_{-1}$}; +\node[labels] at (FTM1 -| MotionEst.140) (FTM2) {$f_{-2}$}; + + +\path [line] (FTM2) -- (MotionEst.140); +\path [line] (FTM1) -- ($(FTM1 |- MotionExtraction)+(0,1)$) -| (MotionEst.40); +\path [line] (FTM1) -- ($(FTM1 |- MotionExtraction)+(0,1)$) -| (MotionExtraction.north); + + + +\newcommand{\MDistance}{.6cm} +\node [below=\MDistance of BMVE] (M1) {M1}; +\path [line] (BMVE) -- (M1); +\node [below=\MDistance of PMVE] (M2) {M2}; +\path [line] (PMVE) -- (M2); +\node [below=\MDistance of MVE] (M3) {M3}; +\path [line] (MVE) -- (M3); +\node [below=\MDistance of MC] (M4) {M4}; +\path [line] (MC) -- (M4); + + + +% \path (BMVE.west) -- (MC.east) node[pos=0.5] (p2) {}; +% \node [block, below of=p2, text width=7.5cm,node distance=1.8cm] (Select) {Concealment method selection and signalling}; + +% \path[line] (BMVE.south) -- (Select.north -| BMVE); +% \path[line] (PMVE.south) -- (Select.north -| PMVE); +% \path[line] (MVE.south) -- (Select.north -| MVE); +% \path[line] (MC.south) -- (Select.north -| MC); + + +\end{tikzpicture} + diff --git a/042_ConcealmentAwareEncoding/Figures/ConcealmentSignalling.pdf b/042_ConcealmentAwareEncoding/Figures/ConcealmentSignalling.pdf new file mode 100644 index 0000000..22e531c Binary files /dev/null and b/042_ConcealmentAwareEncoding/Figures/ConcealmentSignalling.pdf differ diff --git a/042_ConcealmentAwareEncoding/Figures/ConcealmentSignalling.tikz b/042_ConcealmentAwareEncoding/Figures/ConcealmentSignalling.tikz new file mode 100755 index 0000000..8576e1c --- /dev/null +++ b/042_ConcealmentAwareEncoding/Figures/ConcealmentSignalling.tikz @@ -0,0 +1,44 @@ +\newcommand{\scale}{1} + + +\begin{tikzpicture}[scale=\scale, auto] + % Place nodes + +% Define block styles +\tikzstyle{block} = [rectangle, fill=white, text centered, minimum height=0.6cm, inner sep=1,font=\scriptsize] +\tikzstyle{blockDepth} = [draw,block,minimum width=0.75cm] +\tikzstyle{blockEC} = [draw,block,minimum width=0.5cm] +\tikzstyle{blockPoints} = [block, minimum width=0.3cm] +\tikzstyle{text_labels} = [anchor=center,font=\footnotesize,minimum height=0.1cm] + +\tikzstyle{neighbour} = [right=-.01 of Node,] + + +\node [blockPoints] (Node) at (0,0) {...}; +\node [neighbour, blockDepth] (Node) {\textit{split}=1}; +\node [neighbour, blockDepth] (Node) {\textit{split}=0}; +\node [neighbour, blockEC] (Node) {\textit{ecm}}; +\node [neighbour, blockDepth] (Node) {\textit{split}=0}; +\node [neighbour, blockEC] (Node) {\textit{ecm}}; +\node [neighbour, blockPoints] (Node) {...}; +\node [neighbour, blockDepth] (Node) {\textit{split}=1}; +\node [neighbour, blockPoints] (Node) {...}; +\node [neighbour, blockDepth] (Node) {\textit{split}=1}; +\node [neighbour, blockEC] (Node) {\textit{ecm}}; +\node [neighbour, blockEC] (Node) {\textit{ecm}}; +\node [neighbour, blockEC] (Node) {\textit{ecm}}; +\node [neighbour, blockEC] (Node) {\textit{ecm}}; +\node [neighbour, blockPoints] (Node) {...}; + +\draw[text_labels, anchor=center,latex-latex] (.15,-.4) -- ++(5*.75cm+6*.5cm+2*.3cm+.1cm,0) node [pos=0.5,below,yshift=0cm] (VLC) {CTU}; + +\node[draw,dashed, minimum height=1.3cm, minimum width=5*.75cm+6*.5cm+4*.3cm+2*.1cm, anchor=south west] (DashedBox) at (-.2cm,-.9cm) {}; +\node[text_labels, text centered, above=0cm of DashedBox] {Supplemental Enhancement Information (SEI)}; + +\begin{scope}[xshift=-.15cm,yshift=-1.2cm] + \node[text_labels, anchor=west, text width=7cm, node distance=0.6cm] at (0,0) (D) {\textit{split}: flag to mark quadtree splitting}; + \node[text_labels, anchor=west, text width=7cm, below of=D,node distance=0.4cm] (EC) {\textit{ecm}: error concealment method selected for each ECU}; +\end{scope} + + +\end{tikzpicture} diff --git a/042_ConcealmentAwareEncoding/Figures/DecFrames/Bask_MVE_F6.png b/042_ConcealmentAwareEncoding/Figures/DecFrames/Bask_MVE_F6.png new file mode 100644 index 0000000..c138c98 Binary files /dev/null and b/042_ConcealmentAwareEncoding/Figures/DecFrames/Bask_MVE_F6.png differ diff --git a/042_ConcealmentAwareEncoding/Figures/DecFrames/Bask_PropSign_F6.png b/042_ConcealmentAwareEncoding/Figures/DecFrames/Bask_PropSign_F6.png new file mode 100644 index 0000000..d34ad0b Binary files /dev/null and b/042_ConcealmentAwareEncoding/Figures/DecFrames/Bask_PropSign_F6.png differ diff --git a/042_ConcealmentAwareEncoding/Figures/DecFrames/People_MVE_F6.png b/042_ConcealmentAwareEncoding/Figures/DecFrames/People_MVE_F6.png new file mode 100644 index 0000000..d350780 Binary files /dev/null and b/042_ConcealmentAwareEncoding/Figures/DecFrames/People_MVE_F6.png differ diff --git a/042_ConcealmentAwareEncoding/Figures/DecFrames/People_PropSign_F6.png b/042_ConcealmentAwareEncoding/Figures/DecFrames/People_PropSign_F6.png new file mode 100644 index 0000000..56b5c9e Binary files /dev/null and b/042_ConcealmentAwareEncoding/Figures/DecFrames/People_PropSign_F6.png differ diff --git a/042_ConcealmentAwareEncoding/Figures/DeepFlowBlockDiagram.png b/042_ConcealmentAwareEncoding/Figures/DeepFlowBlockDiagram.png new file mode 100644 index 0000000..4e0c1f3 Binary files /dev/null and b/042_ConcealmentAwareEncoding/Figures/DeepFlowBlockDiagram.png differ diff --git a/042_ConcealmentAwareEncoding/Figures/MotionVectorExtrapolation.tikz b/042_ConcealmentAwareEncoding/Figures/MotionVectorExtrapolation.tikz new file mode 100755 index 0000000..4cf2381 --- /dev/null +++ b/042_ConcealmentAwareEncoding/Figures/MotionVectorExtrapolation.tikz @@ -0,0 +1,207 @@ + +\usetikzlibrary{calc} +\usetikzlibrary{snakes} +\usetikzlibrary{arrows} +\usetikzlibrary{positioning} +\usetikzlibrary{decorations} +\usetikzlibrary{decorations.markings} +\usetikzlibrary{arrows.meta} + +\newcommand{\scale}{0.65} +% \newcommand{\TikZyslant}{-0.2} +% \newcommand{\TikZyfshift}{0.05cm} +\newcommand{\yframeshift}{0.05cm} +\newcommand{\xframeshift}{7cm} + + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfdeclarelayer{L0} +\pgfdeclarelayer{L1} +\pgfdeclarelayer{L2} +\pgfdeclarelayer{L3} +\pgfsetlayers{background,L0,L1,L2,L3,main,foreground} + + + + +\begin{tikzpicture}[scale=\scale,every node/.style={minimum size=1cm},on grid ] + +\tikzstyle{pel}=[rectangle,draw=black, thick] +\tikzstyle{refblock}=[rectangle,draw=darkgray, thick] +\tikzstyle{projectedblock}=[rectangle,draw=black, thick,dashed] +\tikzstyle{refmv_linestyle}=[-, thick] + +\tikzstyle{refmv}=[black!60!green, refmv_linestyle,decoration={markings,mark=at position 1 with {\arrow[ultra thick,scale=1]{latex}}},postaction={decorate}] +\tikzstyle{extrapolated_mv}=[refmv_linestyle,dashed, blue, decoration={markings,mark=at position 1 with {\arrow[ultra thick,scale=1]{latex}}},postaction={decorate}] + +\tikzstyle{arrow}=[draw, ->,>=stealth',thick,,decoration={markings,mark=at position 1 with {\arrow[ultra thick,scale=0.5]{stealth}}},postaction={decorate}] + +\tikzstyle{every node}=[font=\large] + +\tikzstyle{frame_contour}=[ultra thick,fill=white,draw=black,fill opacity=.7] + +\newcommand{\DrawFrame}[2] +{ + \fill[fill=#1,fill opacity=.6] (0,0) rectangle (#2+1,#2+1); % Opacity + \draw[thick] (0,0) rectangle (#2+1,#2+1); + \foreach \x in {1,...,#2} + \draw[dotted] (\x,0) -- ++(0,#2+1); + \foreach \y in {1,...,#2} + \draw[dotted] (0,\y) -- ++(#2+1,0); +} + +% Frame t +\begin{pgfonlayer}{L1} +\begin{scope}[ yshift=3*\yframeshift, xshift=3*\xframeshift, +style={yslant=\TikZyslant,xscale=\TikZxscale}] + + + %Draw a frame + \draw[very thick,fill=white,draw=black,fill opacity=.7,dashed](0,0) rectangle (5,5); + \foreach \x in {0,2,4} + \foreach \y in {0,2,4} + \draw[black, dotted] (\x,\y) +(0,0) rectangle ++(1,1); + \foreach \x in {1,3} + \foreach \y in {1,3} + \draw[black, dotted] (\x,\y) +(0,0) rectangle ++(1,1); + + \coordinate (TB1) at (1+0.7,4+0.8); + \draw[projectedblock] (TB1) rectangle ++(1,-1); + + \coordinate (TB2) at (3,4); + \draw[projectedblock] (TB2) rectangle ++(1,-1); + + \coordinate (TB3) at (3+0.2/2,2-0.8/2); + \draw[projectedblock] (TB3) rectangle ++(1,-1); + + \coordinate (TB4) at (3,2); + \draw[projectedblock] (TB4) rectangle ++(1,-1); + +\end{scope} +\end{pgfonlayer} + + +% % Frame t+1 +% \begin{pgfonlayer}{L0} +% \begin{scope}[ yshift=4*\yframeshift, xshift=4*\xframeshift, +% style={yslant=\TikZyslant,xscale=\TikZxscale}] +% +% +% \coordinate (TP1B1) at (1,4); +% \coordinate (TP1B2) at (3,4); +% \coordinate (TP1B3) at (1,2); +% \coordinate (TP1B4) at (3,2); +% +% %Draw a frame +% \DrawFrame{white}{4} +% +% % \draw[black,very thick] (TP1B2) +(0,0) rectangle ++(1,-1); +% \draw[black,very thick] (TP1B4) +(0,0) rectangle ++(1,-1); +% +% \end{scope} +% \end{pgfonlayer} + +% Frame t-1 +\begin{pgfonlayer}{L1} +\begin{scope}[ yshift=2*\TikZyfshift, xshift=2*\xframeshift, +style={yslant=\TikZyslant,xscale=\TikZxscale}] + + \coordinate (T0B1) at (1,4); + \coordinate (T0B2) at (3,4); + \coordinate (T0B3) at (1,2); + \coordinate (T0B4) at (3,2); + + \coordinate (TM1AuxB3) at (1,2); + + \coordinate (Bottom_V1) at (2.5,0); + + \draw[extrapolated_mv] (TB1) to (T0B1); + + % Draw Motion vectors + \coordinate (TM1B0) at (3+0.2,2-0.8); + \coordinate (TM1B0C) at (3+0.2/2,2-0.8/2); +% \draw[refmv] (TP1B4) to (TM1B0); + \draw[refblock] (TM1B0) rectangle ++(1,-1); + \draw[extrapolated_mv] (TB3) to (TM1B0); + + \draw[refblock] (TM1B0C) rectangle ++(1,-1); + \draw[extrapolated_mv] (TB4) to (TM1B0C); + + + \coordinate (TM1B1) at (3-0.6,4-0.5); + \draw[extrapolated_mv] (TB2) to (TM1B1); + + \DrawFrame{white}{4} + + \draw[black,very thick] (T0B1) +(0,0) rectangle ++(1,-1); + \draw[black,very thick] (T0B2) +(0,0) rectangle ++(1,-1); + +\end{scope} +\end{pgfonlayer} + +% Frame t-2 +\begin{pgfonlayer}{L2} +\begin{scope}[ yshift=1*\TikZyfshift, xshift=1*\xframeshift, +style={yslant=\TikZyslant,xscale=\TikZxscale}] + + %Define Points + \coordinate (TM2B1) at (1-0.7,4-0.8); + \coordinate (TM2B2) at (3-0.6,4-0.5); + + % Draw Motion vectors + \draw[refmv] (T0B1) to (TM2B1); + \draw[refmv] (T0B2) to (TM2B2); + + \DrawFrame{white}{4} + + \coordinate (TM2B4_aux) at (3+0.3/2,2-0.4/2); + + % Draw block + \draw[refblock] (TM2B1) rectangle ++(1,-1); + \draw[refblock] (TM2B2) rectangle ++(1,-1); + +\end{scope} +\end{pgfonlayer} + + +%MVs name +\node[scale=0.5] at (3*\xframeshift-1.0cm,4.9) {(2)}; +\node[scale=0.5] at (3*\xframeshift-1.0cm,3) {(1)}; +% \node[scale=0.5] at (3*\xframeshift-2.0cm,2) {(2)}; +% \node[scale=0.5] at (3*\xframeshift-1.0cm,0.3) {(4)}; + + \tikzstyle{frame_label}=[anchor=center,scale=0.8] +% \node[frame_label,scale=0.95] (LostFrameLabel) at (4*\xframeshift-0.9cm,-1.8) {Lost frame }; +% \path[draw,-latex,thick] (LostFrameLabel) edge[out=90, in=0] (3*\xframeshift+5cm*\TikZxscale,0cm); + +% Frame number +\begin{scope}[xshift=2.5cm*\TikZxscale,yshift=5.5cm] +% \node[frame_label] at (0*\xframeshift,\yframeshift*3) { $f_{-3}, v_{-3}$ }; + \node[frame_label] at (1*\xframeshift,\yframeshift*3) { $f_{-2}, v_{-2}$ }; + \node[frame_label] at (2*\xframeshift,\yframeshift*3) { $f_{-1,} v_{-1}$ }; + \node[frame_label] at (3*\xframeshift,\yframeshift*3) { $f_{0}, v_{0}$ }; +% \node[frame_label] at (4*\xframeshift,\yframeshift*4) { $f_{1}, v_{1}$ }; +\end{scope} + +% Frame number +\begin{scope}[xshift=2.5cm*\TikZxscale,yshift=-1.2cm] +% \node[frame_label] at (0*\xframeshift,\yframeshift*3) { $t_{-3}$ }; + \node[frame_label] at (1*\xframeshift,\yframeshift*3) { $t_{-2}$ }; + \node[frame_label] at (2*\xframeshift,\yframeshift*3) { $t_{-1}$ }; + \node[frame_label] at (3*\xframeshift,\yframeshift*3) { $t_{0}$ }; +% \node[frame_label] at (4*\xframeshift,\yframeshift*4) { $t_{1}$ }; +\end{scope} + +% \begin{scope}[yshift=-3.2cm,xshift=-0.5cm] +% \begin{scope} +% \draw[refmv] (0,0) to (1.5,0); +% \fill (2cm,0) node[frame_label,scale=0.95,right]{Original Motion Vector}; +% \end{scope} +% \begin{scope}[xshift=14cm] +% \draw[extrapolated_mv] (0,0) to (1.5,0); +% \fill (2cm,0) node[frame_label,scale=0.95,right]{Extrapolated Motion Vector}; +% \end{scope} +% \end{scope} + +\end{tikzpicture} diff --git a/042_ConcealmentAwareEncoding/Figures/ReconstructedFrames.pdf b/042_ConcealmentAwareEncoding/Figures/ReconstructedFrames.pdf new file mode 100644 index 0000000..f68c442 Binary files /dev/null and b/042_ConcealmentAwareEncoding/Figures/ReconstructedFrames.pdf differ diff --git a/042_ConcealmentAwareEncoding/Results/ECMQualitySingleloss.tex b/042_ConcealmentAwareEncoding/Results/ECMQualitySingleloss.tex new file mode 100644 index 0000000..a20cbd4 --- /dev/null +++ b/042_ConcealmentAwareEncoding/Results/ECMQualitySingleloss.tex @@ -0,0 +1,1045 @@ +\begin{figure}[tbp] +\centering +\renewcommand{\FIGSIZE}{0.47\columnwidth} +% \includegraphics[width=\FIGSIZE,page=1]{\ConcealmentAwareDir/Figures/QualityVsFrameSingle} +% \includegraphics[width=\FIGSIZE,page=2]{\ConcealmentAwareDir/Figures/QualityVsFrameSingle} +% \includegraphics[width=\FIGSIZE,page=5]{\ConcealmentAwareDir/Figures/QualityVsFrameSingle} +% \includegraphics[width=\FIGSIZE,page=7]{\ConcealmentAwareDir/Figures/QualityVsFrameSingle} +% \includegraphics[width=\FIGSIZE,page=8]{\ConcealmentAwareDir/Figures/QualityVsFrameSingle} +% \includegraphics[width=\FIGSIZE,page=9]{\ConcealmentAwareDir/Figures/QualityVsFrameSingle} + +\newcommand{\pltw}{\columnwidth} +\newcommand{\plth}{0.45\textheight} +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +set multiplot layout 2,2 +set xlabel '\TitleFont Frame Number' +set xrange[5:15] +set xtics 5,2,15 +set ytics 4 right +set key width 6 + +set title '\TitleFont Basketball Drill' +plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \MC' w lp ls 2, '-' using 1:2 title '\KeyFont \DEEPFLOW' w lp ls 3, '-' using 1:2 title '\KeyFont \PROP' w lp ls 4 + 0 42.08 44.10 44.86 + 1 42.08 44.10 44.86 + 2 40.24 43.19 44.01 + 3 40.24 42.96 43.94 + 4 41.26 43.62 44.49 + 5 40.22 42.95 43.82 + 6 23.48 32.53 33.19 + 7 24.18 32.52 33.42 + 8 25.22 32.95 35.63 + 9 25.35 33.20 35.74 + 10 25.55 33.28 35.81 + 11 25.68 33.46 35.83 + 12 25.52 33.72 35.98 + 13 26.28 33.80 36.03 + 14 26.48 33.85 36.00 + 15 26.64 33.91 36.05 + 16 42.02 43.95 44.65 + 17 40.06 42.88 43.27 + 18 40.42 43.06 43.40 + 19 40.22 42.85 43.05 +e + 0 42.08 44.10 44.86 + 1 42.08 44.10 44.86 + 2 40.24 43.19 44.01 + 3 40.24 42.96 43.94 + 4 41.26 43.62 44.49 + 5 40.22 42.95 43.82 + 6 28.15 35.99 37.12 + 7 29.64 36.35 38.27 + 8 30.59 37.98 41.00 + 9 31.02 38.11 41.02 + 10 31.45 38.21 41.32 + 11 31.66 38.34 41.38 + 12 31.44 38.56 41.85 + 13 31.70 38.40 41.48 + 14 31.70 38.40 41.46 + 15 31.45 38.25 40.97 + 16 42.02 43.95 44.65 + 17 40.06 42.88 43.27 + 18 40.42 43.06 43.40 + 19 40.22 42.85 43.05 +e + 0 42.08 44.10 44.86 + 1 42.08 44.10 44.86 + 2 40.24 43.19 44.01 + 3 40.24 42.96 43.94 + 4 41.26 43.62 44.49 + 5 40.22 42.95 43.82 + 6 29.36 38.01 38.43 + 7 31.64 38.73 39.79 + 8 32.82 40.75 42.40 + 9 33.55 40.88 42.30 + 10 34.07 40.98 42.41 + 11 34.19 41.11 42.30 + 12 34.04 41.72 42.94 + 13 34.48 41.41 42.47 + 14 34.66 41.53 42.51 + 15 34.34 41.39 42.19 + 16 42.02 43.95 44.65 + 17 40.06 42.88 43.27 + 18 40.42 43.06 43.40 + 19 40.22 42.85 43.05 +e + 0 42.08 44.10 44.86 + 1 42.08 44.10 44.86 + 2 40.24 43.19 44.01 + 3 40.24 42.96 43.94 + 4 41.26 43.62 44.49 + 5 40.22 42.95 43.82 + 6 30.80 38.69 39.15 + 7 32.90 39.17 40.20 + 8 34.13 41.03 42.54 + 9 35.10 41.11 42.36 + 10 35.62 41.17 42.45 + 11 35.79 41.25 42.38 + 12 35.37 41.75 42.92 + 13 35.82 41.46 42.44 + 14 36.03 41.57 42.46 + 15 35.48 41.43 42.11 + 16 42.02 43.95 44.65 + 17 40.06 42.88 43.27 + 18 40.42 43.06 43.40 + 19 40.22 42.85 43.05 +e + +set title '\TitleFont Book Arrival' +plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \MC' w lp ls 2, '-' using 1:2 title '\KeyFont \DEEPFLOW' w lp ls 3, '-' using 1:2 title '\KeyFont \PROP' w lp ls 4 + 0 41.35 41.07 43.41 + 1 40.37 40.66 43.21 + 2 40.04 40.60 43.19 + 3 40.15 40.53 43.14 + 4 40.47 40.56 43.16 + 5 40.17 40.49 43.13 + 6 31.41 39.80 42.55 + 7 31.75 39.76 42.45 + 8 32.15 40.16 42.62 + 9 32.06 40.03 42.52 + 10 32.19 40.00 42.55 + 11 32.07 39.86 42.47 + 12 32.25 40.49 42.92 + 13 32.20 40.20 42.79 + 14 32.35 40.15 42.85 + 15 32.57 40.15 42.82 + 16 41.30 40.81 43.36 + 17 40.20 40.40 43.04 + 18 40.26 40.32 43.08 + 19 39.97 40.25 42.90 +e + 0 41.35 41.07 43.41 + 1 40.37 40.66 43.21 + 2 40.04 40.60 43.19 + 3 40.15 40.53 43.14 + 4 40.47 40.56 43.16 + 5 40.17 40.49 43.13 + 6 35.61 40.36 42.94 + 7 36.51 40.27 42.85 + 8 37.20 40.60 43.07 + 9 36.86 40.44 42.97 + 10 36.95 40.41 42.99 + 11 36.74 40.24 42.91 + 12 37.37 40.89 43.38 + 13 36.92 40.57 43.22 + 14 37.06 40.49 43.26 + 15 36.98 40.45 43.23 + 16 41.30 40.81 43.36 + 17 40.20 40.40 43.04 + 18 40.26 40.32 43.08 + 19 39.97 40.25 42.90 +e + 0 41.35 41.07 43.41 + 1 40.37 40.66 43.21 + 2 40.04 40.60 43.19 + 3 40.15 40.53 43.14 + 4 40.47 40.56 43.16 + 5 40.17 40.49 43.13 + 6 35.96 40.39 42.93 + 7 36.81 40.29 42.82 + 8 37.60 40.65 43.01 + 9 37.23 40.49 42.92 + 10 37.24 40.44 42.92 + 11 37.01 40.27 42.84 + 12 37.74 40.92 43.32 + 13 37.24 40.60 43.16 + 14 37.36 40.52 43.22 + 15 37.33 40.49 43.18 + 16 41.30 40.81 43.36 + 17 40.20 40.40 43.04 + 18 40.26 40.32 43.08 + 19 39.97 40.25 42.90 +e + 0 41.35 41.07 43.41 + 1 40.37 40.66 43.21 + 2 40.04 40.60 43.19 + 3 40.15 40.53 43.14 + 4 40.47 40.56 43.16 + 5 40.17 40.49 43.13 + 6 36.43 40.36 42.95 + 7 37.33 40.26 42.85 + 8 38.06 40.63 43.06 + 9 37.61 40.47 42.96 + 10 37.60 40.43 42.98 + 11 37.33 40.27 42.90 + 12 38.21 40.92 43.38 + 13 37.69 40.60 43.22 + 14 37.80 40.51 43.26 + 15 37.73 40.48 43.22 + 16 41.30 40.81 43.36 + 17 40.20 40.40 43.04 + 18 40.26 40.32 43.08 + 19 39.97 40.25 42.90 +e + +set title '\TitleFont Kimono' +plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \MC' w lp ls 2, '-' using 1:2 title '\KeyFont \DEEPFLOW' w lp ls 3, '-' using 1:2 title '\KeyFont \PROP' w lp ls 4 + 0 42.69 43.90 45.51 + 1 41.72 42.60 44.39 + 2 42.19 43.10 44.72 + 3 42.31 43.35 45.14 + 4 43.27 44.87 46.61 + 5 42.68 43.91 45.81 + 6 25.90 38.71 40.32 + 7 26.96 39.66 41.18 + 8 27.49 40.37 41.88 + 9 27.75 40.39 41.98 + 10 28.23 41.11 42.50 + 11 28.36 40.90 42.35 + 12 28.83 41.95 43.19 + 13 28.89 41.38 42.89 + 14 29.21 42.00 43.47 + 15 29.23 41.65 43.20 + 16 42.53 43.87 45.41 + 17 42.67 43.94 45.49 + 18 43.28 44.77 46.53 + 19 42.80 43.94 45.85 +e + 0 42.69 43.90 45.51 + 1 41.72 42.60 44.39 + 2 42.19 43.10 44.72 + 3 42.31 43.35 45.14 + 4 43.27 44.87 46.61 + 5 42.68 43.91 45.81 + 6 30.20 40.86 42.83 + 7 31.22 41.70 43.49 + 8 31.42 42.43 44.15 + 9 31.67 42.21 43.99 + 10 32.51 42.93 44.62 + 11 32.63 42.42 44.21 + 12 32.95 43.66 45.17 + 13 32.99 42.71 44.46 + 14 33.31 43.42 45.01 + 15 33.26 42.87 44.58 + 16 42.53 43.87 45.41 + 17 42.67 43.94 45.49 + 18 43.28 44.77 46.53 + 19 42.80 43.94 45.85 +e + 0 42.69 43.90 45.51 + 1 41.72 42.60 44.39 + 2 42.19 43.10 44.72 + 3 42.31 43.35 45.14 + 4 43.27 44.87 46.61 + 5 42.68 43.91 45.81 + 6 31.35 41.17 43.08 + 7 32.27 41.91 43.66 + 8 32.89 42.59 44.32 + 9 33.12 42.34 44.14 + 10 33.56 43.05 44.78 + 11 33.61 42.50 44.34 + 12 33.94 43.73 45.30 + 13 33.82 42.76 44.56 + 14 34.07 43.45 45.12 + 15 34.04 42.89 44.66 + 16 42.53 43.87 45.41 + 17 42.67 43.94 45.49 + 18 43.28 44.77 46.53 + 19 42.80 43.94 45.85 +e + 0 42.69 43.90 45.51 + 1 41.72 42.60 44.39 + 2 42.19 43.10 44.72 + 3 42.31 43.35 45.14 + 4 43.27 44.87 46.61 + 5 42.68 43.91 45.81 + 6 32.26 41.20 43.06 + 7 33.07 41.92 43.68 + 8 33.62 42.61 44.31 + 9 33.82 42.35 44.10 + 10 34.33 43.08 44.70 + 11 34.40 42.54 44.29 + 12 34.69 43.75 45.22 + 13 34.61 42.78 44.50 + 14 34.98 43.36 44.99 + 15 34.94 42.74 44.54 + 16 42.53 43.87 45.41 + 17 42.67 43.94 45.49 + 18 43.28 44.77 46.53 + 19 42.80 43.94 45.85 +e + +set title '\TitleFont People on Street' +plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \MC' w lp ls 2, '-' using 1:2 title '\KeyFont \DEEPFLOW' w lp ls 3, '-' using 1:2 title '\KeyFont \PROP' w lp ls 4 + 0 38.19 42.44 42.94 + 1 35.07 41.98 42.52 + 2 35.59 41.76 42.38 + 3 34.44 41.45 42.14 + 4 38.43 42.47 42.95 + 5 36.52 42.16 42.67 + 6 21.03 39.51 40.76 + 7 21.83 39.76 40.86 + 8 22.33 40.19 41.22 + 9 22.55 40.10 41.15 + 10 22.79 40.07 41.13 + 11 22.95 39.99 41.08 + 12 23.34 40.57 41.50 + 13 23.38 40.40 41.34 + 14 23.50 40.35 41.31 + 15 23.57 40.24 41.22 + 16 38.16 42.45 42.94 + 17 36.01 42.09 42.57 + 18 36.02 41.93 42.43 + 19 35.50 41.69 42.28 +e + 0 38.19 42.44 42.94 + 1 35.07 41.98 42.52 + 2 35.59 41.76 42.38 + 3 34.44 41.45 42.14 + 4 38.43 42.47 42.95 + 5 36.52 42.16 42.67 + 6 24.41 40.77 41.89 + 7 25.59 40.88 41.89 + 8 26.24 41.27 42.19 + 9 26.59 41.13 42.04 + 10 26.85 41.04 41.99 + 11 27.03 40.93 41.88 + 12 27.65 41.54 42.32 + 13 27.58 41.30 42.12 + 14 27.80 41.22 42.07 + 15 27.78 41.08 41.98 + 16 38.16 42.45 42.94 + 17 36.01 42.09 42.57 + 18 36.02 41.93 42.43 + 19 35.50 41.69 42.28 +e + 0 38.19 42.44 42.94 + 1 35.07 41.98 42.52 + 2 35.59 41.76 42.38 + 3 34.44 41.45 42.14 + 4 38.43 42.47 42.95 + 5 36.52 42.16 42.67 + 6 25.80 41.08 42.06 + 7 26.99 41.14 42.02 + 8 27.80 41.53 42.34 + 9 28.05 41.33 42.17 + 10 28.29 41.23 42.10 + 11 28.43 41.10 41.98 + 12 29.11 41.71 42.40 + 13 28.93 41.45 42.20 + 14 29.14 41.37 42.14 + 15 29.19 41.22 42.05 + 16 38.16 42.45 42.94 + 17 36.01 42.09 42.57 + 18 36.02 41.93 42.43 + 19 35.50 41.69 42.28 +e + 0 38.19 42.44 42.94 + 1 35.07 41.98 42.52 + 2 35.59 41.76 42.38 + 3 34.44 41.45 42.14 + 4 38.43 42.47 42.95 + 5 36.52 42.16 42.67 + 6 27.41 41.29 42.12 + 7 28.64 41.31 42.09 + 8 29.45 41.65 42.37 + 9 29.63 41.45 42.21 + 10 29.87 41.35 42.14 + 11 30.01 41.22 42.02 + 12 30.87 41.83 42.44 + 13 30.57 41.57 42.24 + 14 30.73 41.48 42.19 + 15 30.65 41.32 42.09 + 16 38.16 42.45 42.94 + 17 36.01 42.09 42.57 + 18 36.02 41.93 42.43 + 19 35.50 41.69 42.28 +e + +# set ytics 4 +# set title '\TitleFont Race Horses' +# plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \MC' w lp ls 2, '-' using 1:2 title '\KeyFont \DEEPFLOW' w lp ls 3, '-' using 1:2 title '\KeyFont \PROP' w lp ls 4 +# 0 40.88 42.15 42.89 +# 1 38.67 40.46 41.59 +# 2 40.68 41.70 42.59 +# 3 38.29 40.49 41.58 +# 4 43.58 44.34 44.82 +# 5 40.25 41.87 42.81 +# 6 17.85 26.55 28.48 +# 7 18.71 27.28 29.03 +# 8 18.73 27.37 29.15 +# 9 18.85 27.55 29.25 +# 10 19.13 27.82 29.35 +# 11 19.31 28.23 29.48 +# 12 19.56 28.62 29.68 +# 13 19.73 28.88 29.79 +# 14 19.75 29.08 29.86 +# 15 19.85 29.14 29.96 +# 16 41.07 42.46 43.15 +# 17 39.48 41.16 42.08 +# 18 40.05 41.54 42.28 +# 19 39.62 41.35 42.12 +# e +# 0 40.88 42.15 42.89 +# 1 38.67 40.46 41.59 +# 2 40.68 41.70 42.59 +# 3 38.29 40.49 41.58 +# 4 43.58 44.34 44.82 +# 5 40.25 41.87 42.81 +# 6 21.87 29.77 32.00 +# 7 22.82 30.24 32.68 +# 8 22.88 30.11 32.38 +# 9 23.04 30.24 32.53 +# 10 23.43 30.46 32.72 +# 11 23.79 30.83 32.76 +# 12 24.23 31.40 33.14 +# 13 24.43 31.80 33.24 +# 14 24.79 32.04 33.43 +# 15 25.00 32.07 33.47 +# 16 41.07 42.46 43.15 +# 17 39.48 41.16 42.08 +# 18 40.05 41.54 42.28 +# 19 39.62 41.35 42.12 +# e +# 0 40.88 42.15 42.89 +# 1 38.67 40.46 41.59 +# 2 40.68 41.70 42.59 +# 3 38.29 40.49 41.58 +# 4 43.58 44.34 44.82 +# 5 40.25 41.87 42.81 +# 6 22.96 31.27 33.08 +# 7 24.02 31.96 33.73 +# 8 24.08 31.77 33.29 +# 9 24.25 31.99 33.44 +# 10 24.58 32.26 33.55 +# 11 24.86 32.65 33.57 +# 12 25.33 33.40 33.91 +# 13 25.49 33.74 33.88 +# 14 25.78 33.97 34.03 +# 15 25.98 34.03 34.10 +# 16 41.07 42.46 43.15 +# 17 39.48 41.16 42.08 +# 18 40.05 41.54 42.28 +# 19 39.62 41.35 42.12 +# e +# 0 40.88 42.15 42.89 +# 1 38.67 40.46 41.59 +# 2 40.68 41.70 42.59 +# 3 38.29 40.49 41.58 +# 4 43.58 44.34 44.82 +# 5 40.25 41.87 42.81 +# 6 24.07 32.55 33.82 +# 7 25.18 33.09 34.46 +# 8 25.28 32.94 34.06 +# 9 25.48 33.21 34.19 +# 10 25.78 33.49 34.26 +# 11 26.06 33.89 34.29 +# 12 26.43 34.52 34.68 +# 13 26.59 34.81 34.64 +# 14 26.85 35.12 34.80 +# 15 27.06 35.21 34.92 +# 16 41.07 42.46 43.15 +# 17 39.48 41.16 42.08 +# 18 40.05 41.54 42.28 +# 19 39.62 41.35 42.12 +# e +# +# set key bottom right +# set ytics 4 +# set title '\TitleFont Tennis' +# plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \MC' w lp ls 2, '-' using 1:2 title '\KeyFont \DEEPFLOW' w lp ls 3, '-' using 1:2 title '\KeyFont \PROP' w lp ls 4 +# 0 40.70 43.53 44.69 +# 1 37.86 42.51 43.33 +# 2 37.19 42.13 42.92 +# 3 36.32 41.77 42.33 +# 4 39.75 43.00 43.90 +# 5 38.23 42.31 43.13 +# 6 19.56 35.56 37.70 +# 7 21.24 36.62 39.36 +# 8 22.70 38.18 40.69 +# 9 23.96 38.79 41.27 +# 10 24.75 39.27 41.55 +# 11 25.85 40.01 41.86 +# 12 29.17 41.81 43.56 +# 13 29.81 41.30 42.56 +# 14 28.42 40.37 42.12 +# 15 26.56 39.61 41.53 +# 16 40.61 43.40 44.62 +# 17 37.93 42.28 43.25 +# 18 37.78 41.83 42.70 +# 19 36.70 41.39 42.01 +# e +# 0 40.70 43.53 44.69 +# 1 37.86 42.51 43.33 +# 2 37.19 42.13 42.92 +# 3 36.32 41.77 42.33 +# 4 39.75 43.00 43.90 +# 5 38.23 42.31 43.13 +# 6 22.32 37.71 38.77 +# 7 24.13 38.61 40.28 +# 8 24.66 39.82 41.33 +# 9 25.37 39.87 41.41 +# 10 26.09 40.16 41.68 +# 11 27.77 40.94 41.96 +# 12 31.73 42.60 43.89 +# 13 32.13 42.23 43.16 +# 14 29.61 41.00 42.48 +# 15 27.46 40.08 41.92 +# 16 40.61 43.40 44.62 +# 17 37.93 42.28 43.25 +# 18 37.78 41.83 42.70 +# 19 36.70 41.39 42.01 +# e +# 0 40.70 43.53 44.69 +# 1 37.86 42.51 43.33 +# 2 37.19 42.13 42.92 +# 3 36.32 41.77 42.33 +# 4 39.75 43.00 43.90 +# 5 38.23 42.31 43.13 +# 6 24.77 38.50 38.80 +# 7 26.92 39.27 40.19 +# 8 27.53 40.17 41.05 +# 9 28.14 40.14 41.15 +# 10 28.96 40.37 41.53 +# 11 31.48 41.25 42.10 +# 12 35.56 42.65 43.95 +# 13 35.12 41.98 43.04 +# 14 31.29 41.10 42.06 +# 15 29.84 40.41 41.30 +# 16 40.61 43.40 44.62 +# 17 37.93 42.28 43.25 +# 18 37.78 41.83 42.70 +# 19 36.70 41.39 42.01 +# e +# 0 40.70 43.53 44.69 +# 1 37.86 42.51 43.33 +# 2 37.19 42.13 42.92 +# 3 36.32 41.77 42.33 +# 4 39.75 43.00 43.90 +# 5 38.23 42.31 43.13 +# 6 26.80 39.28 39.60 +# 7 28.58 39.95 40.91 +# 8 28.83 40.73 41.73 +# 9 29.32 40.56 41.66 +# 10 30.04 40.73 41.84 +# 11 32.20 41.37 42.18 +# 12 35.87 42.76 43.94 +# 13 36.20 42.36 43.21 +# 14 32.62 41.51 42.63 +# 15 31.31 40.78 42.12 +# 16 40.61 43.40 44.62 +# 17 37.93 42.28 43.25 +# 18 37.78 41.83 42.70 +# 19 36.70 41.39 42.01 +# e +\end{gnuplot} +\vspace{-8mm} +\caption{Error propagation after losing the inter-coded frame $\#6$.} %for Basketball Drill and People On Street sequences.} +\label{FIG5:singlelossInter} +\FIGSVSPACE +\vspace{5mm} +% \end{figure} +% \begin{figure}[!t] +% \centering +% \renewcommand{\FIGSIZE}{0.47\columnwidth} +% \includegraphics[width=\FIGSIZE,page=1]{\ConcealmentAwareDir/Figures/QualityVsFrameSingleIntra} +% \includegraphics[width=\FIGSIZE,page=2]{\ConcealmentAwareDir/Figures/QualityVsFrameSingleIntra} +% \includegraphics[width=\FIGSIZE,page=5]{\ConcealmentAwareDir/Figures/QualityVsFrameSingleIntra} +% \includegraphics[width=\FIGSIZE,page=7]{\ConcealmentAwareDir/Figures/QualityVsFrameSingleIntra} +% \includegraphics[width=\FIGSIZE,page=8]{\ConcealmentAwareDir/Figures/QualityVsFrameSingleIntra} +% \includegraphics[width=\FIGSIZE,page=9]{\ConcealmentAwareDir/Figures/QualityVsFrameSingleIntra} +%\newcommand{\pltw}{\columnwidth} +%\newcommand{\plth}{0.45\textheight} +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +set multiplot layout 2,2 +set xlabel '\TitleFont Frame Number' +set xrange[14:30] +set xtics 14,2,40 +set ytics 4 right +set key width 6 + +set title '\TitleFont Basketball Drill' +plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \MC' w lp ls 2, '-' using 1:2 title '\KeyFont \DEEPFLOW' w lp ls 3, '-' using 1:2 title '\KeyFont \PROP' w lp ls 4 + 14 40.94 43.32 43.96 + 15 40.49 43.10 43.56 + 16 24.28 34.14 31.38 + 17 24.45 33.67 31.00 + 18 24.56 33.91 30.93 + 19 24.64 34.10 30.88 + 20 24.59 34.17 30.91 + 21 24.79 34.55 30.99 + 22 25.00 34.52 30.95 + 23 24.81 34.12 30.90 + 24 24.73 33.41 30.70 + 25 24.60 33.15 30.56 + 26 24.67 32.86 30.71 + 27 24.57 32.19 30.51 + 28 24.63 32.27 30.81 + 29 24.66 32.17 30.78 + 30 24.68 32.34 30.90 +e + 14 40.94 43.32 43.96 + 15 40.49 43.10 43.56 + 16 28.87 37.08 36.42 + 17 29.26 36.53 35.77 + 18 28.95 36.57 35.60 + 19 29.17 36.95 35.53 + 20 29.36 37.27 35.82 + 21 29.70 38.06 35.88 + 22 29.98 38.17 35.99 + 23 29.61 37.59 35.62 + 24 29.41 36.40 35.54 + 25 29.31 36.04 35.27 + 26 29.27 35.75 35.18 + 27 29.01 34.66 34.61 + 28 28.80 34.61 34.62 + 29 28.75 34.45 34.43 + 30 28.85 34.77 34.51 +e + 14 40.94 43.32 43.96 + 15 40.49 43.10 43.56 + 16 31.22 40.07 38.02 + 17 31.79 40.17 37.49 + 18 32.29 40.57 37.47 + 19 32.43 40.51 37.38 + 20 32.37 41.17 37.64 + 21 32.49 40.94 37.29 + 22 32.57 40.77 37.27 + 23 32.48 40.52 36.90 + 24 32.48 40.32 37.09 + 25 32.24 39.91 36.57 + 26 32.33 39.74 36.41 + 27 32.31 39.16 36.10 + 28 32.67 39.52 36.28 + 29 32.64 39.20 36.07 + 30 32.84 39.26 36.08 +e + 14 40.94 43.32 43.96 + 15 40.49 43.10 43.56 + 16 32.49 40.29 38.40 + 17 32.89 40.52 38.07 + 18 33.42 40.82 37.94 + 19 33.57 40.73 37.91 + 20 33.50 41.37 38.39 + 21 33.68 41.13 38.28 + 22 33.86 41.05 38.40 + 23 33.79 40.81 38.17 + 24 33.94 41.12 38.81 + 25 33.83 40.76 38.48 + 26 34.01 40.75 38.31 + 27 34.03 40.41 38.00 + 28 34.50 40.84 38.28 + 29 34.39 40.45 37.99 + 30 34.57 40.50 37.92 +e + +set title 'Book Arrival' +plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \MC' w lp ls 2, '-' using 1:2 title '\KeyFont \DEEPFLOW' w lp ls 3, '-' using 1:2 title '\KeyFont \PROP' w lp ls 4 + 14 40.57 40.61 43.38 + 15 40.31 40.56 43.31 + 16 27.24 39.00 41.32 + 17 27.14 39.04 41.18 + 18 27.38 39.18 41.20 + 19 27.51 39.18 41.00 + 20 27.31 39.81 41.70 + 21 26.93 39.46 41.52 + 22 26.67 39.33 41.42 + 23 26.38 39.24 41.31 + 24 26.51 39.65 41.71 + 25 26.31 39.34 41.51 + 26 26.20 39.30 41.37 + 27 26.16 39.15 41.28 + 28 26.27 39.64 41.64 + 29 26.04 39.32 41.39 + 30 25.77 39.26 41.31 +e + 14 40.57 40.61 43.38 + 15 40.31 40.56 43.31 + 16 30.59 39.65 42.09 + 17 30.96 39.64 42.05 + 18 30.91 39.69 42.30 + 19 30.81 39.61 42.16 + 20 29.06 40.01 42.60 + 21 28.87 39.58 42.36 + 22 28.80 39.36 42.22 + 23 28.67 39.19 42.10 + 24 28.77 39.73 42.50 + 25 28.53 39.42 42.23 + 26 28.55 39.36 42.00 + 27 28.50 39.16 41.85 + 28 28.86 39.59 42.22 + 29 28.81 39.21 41.89 + 30 28.93 39.07 41.77 +e + 14 40.57 40.61 43.38 + 15 40.31 40.56 43.31 + 16 31.66 39.79 42.39 + 17 31.96 39.84 42.46 + 18 32.03 39.91 42.60 + 19 32.21 39.84 42.49 + 20 31.87 40.35 42.79 + 21 31.58 39.97 42.56 + 22 31.53 39.90 42.43 + 23 31.43 39.82 42.35 + 24 31.56 40.17 42.75 + 25 31.18 39.81 42.48 + 26 31.20 39.75 42.30 + 27 31.05 39.56 42.12 + 28 31.45 40.06 42.51 + 29 31.43 39.71 42.16 + 30 31.45 39.63 42.06 +e + 14 40.57 40.61 43.38 + 15 40.31 40.56 43.31 + 16 33.08 39.88 42.63 + 17 33.33 39.90 42.59 + 18 33.60 39.93 42.66 + 19 33.70 39.87 42.53 + 20 33.21 40.35 42.89 + 21 32.78 40.00 42.66 + 22 32.47 39.90 42.53 + 23 32.37 39.81 42.41 + 24 32.21 40.12 42.76 + 25 31.92 39.76 42.48 + 26 31.88 39.71 42.27 + 27 31.64 39.52 42.11 + 28 32.07 40.02 42.51 + 29 32.04 39.68 42.19 + 30 32.13 39.62 42.06 +e + +set yrange [24:48] +set title 'Kimono' +plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \MC' w lp ls 2, '-' using 1:2 title '\KeyFont \DEEPFLOW' w lp ls 3, '-' using 1:2 title '\KeyFont \PROP' w lp ls 4 + 14 43.27 44.84 46.71 + 15 42.66 44.04 46.02 + 16 25.91 38.97 40.65 + 17 26.10 39.26 40.95 + 18 26.58 39.80 41.62 + 19 26.82 39.67 41.58 + 20 27.38 40.77 42.60 + 21 27.36 40.21 42.20 + 22 27.51 40.70 42.38 + 23 27.54 40.65 42.38 + 24 28.18 41.51 43.06 + 25 28.33 41.17 42.73 + 26 28.46 41.50 42.95 + 27 28.50 41.34 42.79 + 28 28.55 42.00 43.54 + 29 28.66 41.57 43.15 + 30 28.74 41.93 43.58 +e + 14 43.27 44.84 46.71 + 15 42.66 44.04 46.02 + 16 31.89 41.36 43.53 + 17 31.84 41.42 43.42 + 18 32.32 41.80 43.65 + 19 31.81 41.45 43.18 + 20 33.56 42.64 44.26 + 21 33.04 41.79 43.61 + 22 33.71 42.29 43.87 + 23 34.03 42.13 43.72 + 24 34.93 42.98 44.18 + 25 34.95 42.37 43.74 + 26 35.88 42.72 44.01 + 27 36.02 42.43 43.76 + 28 36.54 43.27 44.53 + 29 36.36 42.64 44.01 + 30 36.45 43.12 44.32 +e + 14 43.27 44.84 46.71 + 15 42.66 44.04 46.02 + 16 33.15 41.64 43.67 + 17 33.40 41.66 43.48 + 18 34.50 42.01 43.80 + 19 34.71 41.61 43.37 + 20 35.36 42.76 44.21 + 21 35.32 41.91 43.59 + 22 35.60 42.39 43.77 + 23 35.64 42.19 43.54 + 24 36.42 43.10 43.99 + 25 36.42 42.47 43.54 + 26 36.91 43.04 43.83 + 27 36.80 42.65 43.53 + 28 37.22 43.57 44.23 + 29 37.02 42.88 43.62 + 30 37.14 43.38 43.92 +e + 14 43.27 44.84 46.71 + 15 42.66 44.04 46.02 + 16 34.03 41.62 43.64 + 17 34.31 41.69 43.56 + 18 35.25 42.05 43.97 + 19 35.47 41.67 43.62 + 20 36.09 42.95 44.66 + 21 35.99 42.10 44.08 + 22 36.32 42.63 44.30 + 23 36.28 42.49 44.18 + 24 37.08 43.41 44.67 + 25 37.03 42.74 44.14 + 26 37.42 43.25 44.39 + 27 37.24 42.87 44.07 + 28 37.59 43.75 44.94 + 29 37.31 42.94 44.32 + 30 37.40 43.45 44.78 +e + +set autoscale +set title 'People on Street' +plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \MC' w lp ls 2, '-' using 1:2 title '\KeyFont \DEEPFLOW' w lp ls 3, '-' using 1:2 title '\KeyFont \PROP' w lp ls 4 + 14 36.26 41.81 42.45 + 15 35.77 41.64 42.34 + 16 21.34 39.46 40.64 + 17 21.48 39.29 40.48 + 18 21.64 39.21 40.36 + 19 21.74 39.11 40.37 + 20 21.96 39.48 40.64 + 21 21.90 39.28 40.47 + 22 21.93 39.24 40.42 + 23 21.95 39.17 40.33 + 24 22.15 39.52 40.66 + 25 22.10 39.35 40.50 + 26 22.09 39.30 40.46 + 27 22.02 39.16 40.34 + 28 22.24 39.57 40.67 + 29 22.09 39.39 40.50 + 30 22.07 39.34 40.45 +e + 14 36.26 41.81 42.45 + 15 35.77 41.64 42.34 + 16 24.49 40.61 41.57 + 17 24.73 40.38 41.33 + 18 24.97 40.25 41.22 + 19 25.00 40.12 41.23 + 20 25.44 40.46 41.45 + 21 25.37 40.25 41.27 + 22 25.37 40.18 41.17 + 23 25.41 40.09 41.09 + 24 25.91 40.46 41.39 + 25 25.73 40.26 41.20 + 26 25.81 40.18 41.13 + 27 25.73 40.04 41.02 + 28 26.13 40.48 41.35 + 29 25.94 40.29 41.17 + 30 26.01 40.22 41.12 +e + 14 36.26 41.81 42.45 + 15 35.77 41.64 42.34 + 16 25.99 40.88 41.75 + 17 26.26 40.61 41.49 + 18 26.53 40.48 41.37 + 19 26.62 40.32 41.35 + 20 27.19 40.67 41.58 + 21 27.05 40.44 41.39 + 22 27.09 40.37 41.30 + 23 27.05 40.27 41.22 + 24 27.61 40.64 41.53 + 25 27.45 40.45 41.35 + 26 27.50 40.37 41.27 + 27 27.38 40.24 41.15 + 28 27.93 40.68 41.50 + 29 27.66 40.49 41.31 + 30 27.72 40.43 41.25 +e + 14 36.26 41.81 42.45 + 15 35.77 41.64 42.34 + 16 27.12 40.92 41.70 + 17 27.37 40.66 41.47 + 18 27.62 40.52 41.35 + 19 27.67 40.38 41.34 + 20 28.27 40.72 41.54 + 21 28.11 40.49 41.36 + 22 28.16 40.42 41.26 + 23 28.11 40.31 41.16 + 24 28.78 40.67 41.45 + 25 28.64 40.47 41.26 + 26 28.76 40.37 41.17 + 27 28.65 40.23 41.05 + 28 29.32 40.66 41.37 + 29 29.00 40.47 41.18 + 30 29.10 40.40 41.13 +e + +# set title 'Race Horses' +# plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \MC' w lp ls 2, '-' using 1:2 title '\KeyFont \DEEPFLOW' w lp ls 3, '-' using 1:2 title '\KeyFont \PROP' w lp ls 4 +# 14 39.96 41.59 42.35 +# 15 39.58 41.33 42.19 +# 16 16.93 25.53 27.56 +# 17 17.02 25.39 27.74 +# 18 17.12 25.32 27.97 +# 19 17.18 25.03 28.11 +# 20 17.25 24.81 28.33 +# 21 17.36 24.79 28.47 +# 22 17.42 24.80 28.72 +# 23 17.43 24.83 28.90 +# 24 17.44 24.97 29.21 +# 25 17.37 25.15 29.22 +# 26 17.47 25.24 29.46 +# 27 17.52 25.50 29.58 +# 28 17.49 25.48 29.88 +# 29 17.38 25.36 29.92 +# 30 17.50 25.49 30.28 +# e +# 14 39.96 41.59 42.35 +# 15 39.58 41.33 42.19 +# 16 23.31 32.07 33.65 +# 17 23.58 32.39 34.23 +# 18 23.68 32.24 34.10 +# 19 24.07 31.80 34.10 +# 20 24.32 32.09 34.37 +# 21 24.48 31.91 34.07 +# 22 24.65 32.19 34.13 +# 23 24.79 32.14 34.14 +# 24 25.03 32.19 34.61 +# 25 24.94 32.29 34.58 +# 26 25.12 32.31 34.75 +# 27 25.23 32.35 34.88 +# 28 25.43 32.49 34.93 +# 29 25.54 31.91 34.58 +# 30 25.75 31.70 34.53 +# e +# 14 39.96 41.59 42.35 +# 15 39.58 41.33 42.19 +# 16 24.47 34.05 35.43 +# 17 24.91 34.82 36.16 +# 18 25.05 34.83 36.00 +# 19 25.40 34.66 35.82 +# 20 25.65 35.04 36.08 +# 21 25.71 34.77 35.59 +# 22 25.89 35.16 35.57 +# 23 26.07 35.40 35.58 +# 24 26.34 35.81 36.07 +# 25 26.33 35.70 35.99 +# 26 26.56 35.76 36.21 +# 27 26.66 35.86 36.27 +# 28 26.86 36.44 36.39 +# 29 27.00 35.72 35.91 +# 30 27.16 35.74 35.90 +# e +# 14 39.96 41.59 42.35 +# 15 39.58 41.33 42.19 +# 16 25.65 35.22 35.61 +# 17 26.11 35.61 36.20 +# 18 26.17 35.34 35.90 +# 19 26.47 35.06 35.68 +# 20 26.71 35.46 35.82 +# 21 26.79 35.09 35.30 +# 22 27.03 35.49 35.35 +# 23 27.19 35.75 35.41 +# 24 27.47 36.32 35.89 +# 25 27.45 36.11 35.85 +# 26 27.72 36.17 35.99 +# 27 27.79 36.24 36.14 +# 28 28.10 36.55 36.13 +# 29 27.99 35.64 35.64 +# 30 28.25 35.47 35.57 +# e +# +# set yrange [16:42] +# set title 'Tennis' +# plot '-' using 1:2 title '\KeyFont \REF' w lp ls 1, '-' using 1:2 title '\KeyFont \MC' w lp ls 2, '-' using 1:2 title '\KeyFont \DEEPFLOW' w lp ls 3, '-' using 1:2 title '\KeyFont \PROP' w lp ls 4 +# 14 38.43 42.39 43.26 +# 15 37.85 42.04 42.99 +# 16 19.55 35.52 38.13 +# 17 19.07 34.94 37.80 +# 18 18.68 34.70 37.67 +# 19 18.58 34.48 37.46 +# 20 18.81 35.10 38.03 +# 21 19.11 35.23 37.95 +# 22 19.94 35.63 38.13 +# 23 20.33 35.65 38.33 +# 24 20.41 35.91 38.65 +# 25 19.92 35.56 38.30 +# 26 19.73 35.34 38.18 +# 27 19.54 35.27 38.14 +# 28 19.37 35.40 38.48 +# 29 19.56 35.54 38.35 +# 30 20.18 35.83 38.52 +# e +# 14 38.43 42.39 43.26 +# 15 37.85 42.04 42.99 +# 16 21.78 37.69 39.35 +# 17 21.10 37.01 38.74 +# 18 20.61 36.58 38.40 +# 19 20.14 36.03 37.90 +# 20 20.43 36.88 38.36 +# 21 21.19 37.31 38.46 +# 22 22.84 38.09 38.93 +# 23 23.90 38.43 39.50 +# 24 22.76 38.10 39.63 +# 25 21.71 37.36 38.95 +# 26 21.31 37.00 38.68 +# 27 21.07 36.79 38.48 +# 28 21.06 37.15 38.74 +# 29 21.69 37.54 38.74 +# 30 22.69 37.95 39.13 +# e +# 14 38.43 42.39 43.26 +# 15 37.85 42.04 42.99 +# 16 25.27 38.56 39.36 +# 17 25.46 38.22 38.89 +# 18 25.84 38.06 38.62 +# 19 25.81 37.94 38.41 +# 20 26.88 39.18 39.45 +# 21 27.47 39.25 39.61 +# 22 28.40 39.68 40.02 +# 23 27.79 39.29 40.03 +# 24 26.26 39.19 40.22 +# 25 26.10 38.95 39.49 +# 26 26.23 38.76 39.31 +# 27 26.34 38.70 39.17 +# 28 26.95 39.45 39.87 +# 29 27.30 39.58 39.86 +# 30 27.89 39.84 40.08 +# e +# 14 38.43 42.39 43.26 +# 15 37.85 42.04 42.99 +# 16 27.84 39.70 40.17 +# 17 27.69 39.10 39.53 +# 18 27.73 38.89 39.15 +# 19 27.42 38.76 38.76 +# 20 28.50 39.75 39.60 +# 21 29.40 39.94 39.79 +# 22 30.95 40.36 40.31 +# 23 31.44 40.28 40.66 +# 24 29.63 40.11 40.82 +# 25 28.71 39.63 40.10 +# 26 28.56 39.43 39.76 +# 27 28.29 39.40 39.54 +# 28 28.92 40.04 40.04 +# 29 29.45 40.17 40.10 +# 30 30.56 40.46 40.42 +# e +\end{gnuplot} +\vspace{-8mm} +\caption{Error propagation after losing intra-coded frame $\#16$.} % for Kimono and Tennis sequences.} +\label{FIG5:singlelossIntra} +\FIGSVSPACE +\end{figure} + +\begin{figure}[!t] +\centering +\newcommand{\figsize}{0.3\columnwidth} +%\resizebox{\columnwidth}{!}{ +\begin{tabular}{y{.4\columnwidth}y{.4\columnwidth}} +\includegraphics[width=\figsize]{\ConcealmentAwareDir/Figures/DecFrames/Bask_MVE_F6} & +\includegraphics[width=\figsize]{\ConcealmentAwareDir/Figures/DecFrames/Bask_PropSign_F6} \\ +\small \MC (28.15~dB) & \small \PROP (30.99~dB) \\ %[10pt] +\end{tabular} +%} +\caption{Recovered frame \#6 of Basketball Drill} +\label{FIG5:RecoveredExample} +\FIGSVSPACE +\end{figure} diff --git a/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrame.pdf b/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrame.pdf new file mode 100644 index 0000000..cf82ad6 Binary files /dev/null and b/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrame.pdf differ diff --git a/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrame.plt b/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrame.plt new file mode 100644 index 0000000..27c2877 --- /dev/null +++ b/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrame.plt @@ -0,0 +1,446 @@ +set terminal postscript eps monochrome font 'TimesNewRoman,16' + +set grid +set size {0.75,0.75} +set xlabel "Bitrate [kbits/s]" font ",18" offset character 0, 0.4, 0 +set xtics center font ",16" # offset character 0, 0, 0 +set ylabel "PSNR [dB]" font ",18" offset character 0.9, 0, 0 +set ytics font ",16" +set key spacing 1 font "TimesNewRoman,16" top right width 0 + +set title font ",20" offset character 0, -0.7, 0 + +set style line 1 lc 4 lt 4 lw 3 pt 10 ps 1 +#set style line 2 lc rgbcolor "#006400" lt 2 lw 3 pt 3 ps 1 +set style line 2 lc 2 lt 2 lw 3 pt 3 ps 1 +set style line 3 lc 3 lt 4 lw 3 pt 4 ps 1 +set style line 4 lc 1 lt 1 lw 3 pt 2 ps 1 +set style line 5 lc 7 lt 3 lw 3 pt 6 ps 1 +set style line 6 lc 8 lt 6 lw 3 pt 8 ps 1 +set style line 7 lc 2 lt 7 lw 3 pt 8 ps 1 +set style line 8 lc 3 lt 8 lw 3 pt 9 ps 1 +set style line 9 lc 4 lt 9 lw 3 pt 10 ps 1 + +set xrange[6:6+27] +set xtics 6,2,6+27 +set output 'plot_QualityVsFrameSingle_PSNR_lowdelay_P_4R_slice_single_seq_basketball_drill_rate_single_6.eps' +set title 'Basketball Drill' +plot '-' using 1:2 title 'MC' w lp ls 1, '-' using 1:2 title '16x16' w lp ls 2, '-' using 1:2 title 'Proposed' w lp ls 3 + 2 40.24 43.19 44.01 + 3 40.24 42.96 43.94 + 4 41.26 43.62 44.49 + 5 40.22 42.95 43.82 + 6 41.34 43.71 44.62 + 7 39.72 42.83 43.48 + 8 42.17 44.27 45.07 + 9 41.20 43.56 44.39 + 10 40.87 43.35 44.17 + 11 40.78 43.20 43.86 + 12 41.78 43.98 44.83 + 13 40.72 43.32 43.96 + 14 40.94 43.32 43.96 + 15 40.49 43.10 43.56 + 16 28.87 37.08 36.42 + 17 29.26 36.53 35.77 + 18 28.95 36.57 35.60 + 19 29.17 36.95 35.53 + 20 29.36 37.27 35.82 + 21 29.70 38.06 35.88 + 22 29.98 38.17 35.99 + 23 29.61 37.59 35.62 + 24 29.41 36.40 35.54 + 25 29.31 36.04 35.27 + 26 29.27 35.75 35.18 + 27 29.01 34.66 34.61 + 28 28.80 34.61 34.62 + 29 28.75 34.45 34.43 + 30 28.85 34.77 34.51 + 31 28.91 34.68 34.37 + 32 42.09 44.07 44.58 + 33 40.03 42.78 43.23 +e + 2 40.24 43.19 44.01 + 3 40.24 42.96 43.94 + 4 41.26 43.62 44.49 + 5 40.22 42.95 43.82 + 6 41.34 43.71 44.62 + 7 39.72 42.83 43.48 + 8 42.17 44.27 45.07 + 9 41.20 43.56 44.39 + 10 40.87 43.35 44.17 + 11 40.78 43.20 43.86 + 12 41.78 43.98 44.83 + 13 40.72 43.32 43.96 + 14 40.94 43.32 43.96 + 15 40.49 43.10 43.56 + 16 32.14 40.15 38.22 + 17 32.73 40.47 37.88 + 18 33.20 40.69 37.77 + 19 33.41 40.65 37.77 + 20 33.30 41.23 38.26 + 21 33.53 41.04 38.15 + 22 33.73 40.95 38.33 + 23 33.75 40.72 38.11 + 24 33.95 41.02 38.65 + 25 33.79 40.69 38.31 + 26 33.92 40.64 38.16 + 27 33.92 40.36 37.86 + 28 34.39 40.73 38.13 + 29 34.26 40.35 37.85 + 30 34.44 40.40 37.77 + 31 34.43 40.16 37.59 + 32 42.09 44.07 44.58 + 33 40.03 42.78 43.23 +e + 2 40.24 43.19 44.01 + 3 40.24 42.96 43.94 + 4 41.26 43.62 44.49 + 5 40.22 42.95 43.82 + 6 41.34 43.71 44.62 + 7 39.72 42.83 43.48 + 8 42.17 44.27 45.07 + 9 41.20 43.56 44.39 + 10 40.87 43.35 44.17 + 11 40.78 43.20 43.86 + 12 41.78 43.98 44.83 + 13 40.72 43.32 43.96 + 14 40.94 43.32 43.96 + 15 40.49 43.10 43.56 + 16 31.62 40.07 37.61 + 17 32.33 40.31 37.21 + 18 32.85 40.64 37.20 + 19 33.01 40.57 37.21 + 20 33.07 41.23 37.70 + 21 33.30 40.99 37.60 + 22 33.22 40.80 37.50 + 23 33.04 40.54 37.20 + 24 33.18 40.68 37.06 + 25 32.94 40.35 36.68 + 26 33.06 40.29 36.25 + 27 33.02 39.91 35.78 + 28 33.41 40.25 35.77 + 29 33.38 39.87 35.51 + 30 33.64 39.94 35.43 + 31 33.63 39.68 35.24 + 32 42.09 44.07 44.58 + 33 40.03 42.78 43.23 +e + + +set xrange[6:6+27] +set xtics 6,2,6+27 +set output 'plot_QualityVsFrameSingle_PSNR_lowdelay_P_4R_slice_single_seq_park_rate_single_6.eps' +set title 'Park Scene' +plot '-' using 1:2 title 'MC' w lp ls 1, '-' using 1:2 title '16x16' w lp ls 2, '-' using 1:2 title 'Proposed' w lp ls 3 + 2 38.91 41.20 42.76 + 3 38.96 41.23 42.75 + 4 40.35 41.99 43.22 + 5 39.25 41.41 42.91 + 6 39.73 41.63 43.00 + 7 39.04 41.28 42.78 + 8 40.94 42.43 43.62 + 9 39.40 41.65 43.14 + 10 39.84 41.74 43.16 + 11 39.16 41.42 42.93 + 12 41.10 42.60 43.80 + 13 39.61 41.79 43.36 + 14 39.88 41.83 43.29 + 15 39.37 41.57 43.11 + 16 34.11 40.55 42.26 + 17 34.00 40.42 42.22 + 18 34.21 40.45 42.19 + 19 34.09 40.25 42.05 + 20 34.60 40.79 42.45 + 21 34.42 40.28 42.20 + 22 34.52 40.29 42.14 + 23 34.46 40.15 42.00 + 24 35.06 40.85 42.50 + 25 34.72 40.36 42.21 + 26 34.83 40.36 42.15 + 27 34.70 40.18 42.02 + 28 35.34 40.91 42.50 + 29 34.95 40.39 42.19 + 30 35.05 40.33 42.08 + 31 34.88 40.17 41.92 + 32 40.65 42.35 43.30 + 33 39.13 41.45 42.87 +e + 2 38.91 41.20 42.76 + 3 38.96 41.23 42.75 + 4 40.35 41.99 43.22 + 5 39.25 41.41 42.91 + 6 39.73 41.63 43.00 + 7 39.04 41.28 42.78 + 8 40.94 42.43 43.62 + 9 39.40 41.65 43.14 + 10 39.84 41.74 43.16 + 11 39.16 41.42 42.93 + 12 41.10 42.60 43.80 + 13 39.61 41.79 43.36 + 14 39.88 41.83 43.29 + 15 39.37 41.57 43.11 + 16 35.24 40.31 42.15 + 17 35.06 40.16 42.09 + 18 35.27 40.19 42.06 + 19 35.14 40.01 41.96 + 20 35.60 40.54 42.36 + 21 35.35 40.07 42.12 + 22 35.48 40.09 42.05 + 23 35.37 39.96 41.92 + 24 36.07 40.64 42.41 + 25 35.62 40.19 42.13 + 26 35.74 40.19 42.08 + 27 35.55 40.02 41.94 + 28 36.28 40.72 42.42 + 29 35.82 40.21 42.11 + 30 35.87 40.17 41.99 + 31 35.66 40.01 41.84 + 32 40.65 42.35 43.30 + 33 39.13 41.45 42.87 +e + 2 38.91 41.20 42.76 + 3 38.96 41.23 42.75 + 4 40.35 41.99 43.22 + 5 39.25 41.41 42.91 + 6 39.73 41.63 43.00 + 7 39.04 41.28 42.78 + 8 40.94 42.43 43.62 + 9 39.40 41.65 43.14 + 10 39.84 41.74 43.16 + 11 39.16 41.42 42.93 + 12 41.10 42.60 43.80 + 13 39.61 41.79 43.36 + 14 39.88 41.83 43.29 + 15 39.37 41.57 43.11 + 16 34.95 40.22 42.06 + 17 34.81 40.09 42.03 + 18 35.01 40.14 42.03 + 19 34.89 39.97 41.93 + 20 35.35 40.50 42.33 + 21 35.12 40.05 42.09 + 22 35.24 40.06 42.02 + 23 35.13 39.93 41.90 + 24 35.79 40.61 42.38 + 25 35.38 40.16 42.10 + 26 35.48 40.17 42.05 + 27 35.31 39.99 41.91 + 28 36.00 40.70 42.39 + 29 35.57 40.19 42.08 + 30 35.63 40.15 41.96 + 31 35.42 39.99 41.81 + 32 40.65 42.35 43.30 + 33 39.13 41.45 42.87 +e + + +set xrange[6:6+27] +set xtics 6,2,6+27 +set output 'plot_QualityVsFrameSingle_PSNR_randomaccess_slice_single_seq_basketball_drill_rate_single_6.eps' +set title 'Basketball Drill' +plot '-' using 1:2 title 'MC' w lp ls 1, '-' using 1:2 title '16x16' w lp ls 2, '-' using 1:2 title 'Proposed' w lp ls 3 + 2 41.18 43.75 44.63 + 3 40.53 43.36 44.16 + 4 41.35 43.60 44.45 + 5 40.79 43.47 44.23 + 6 42.06 44.05 44.97 + 7 41.08 43.81 44.56 + 8 43.30 44.97 45.91 + 9 30.08 36.75 37.84 + 10 27.48 33.98 35.08 + 11 24.76 31.63 31.95 + 12 23.13 30.37 30.69 + 13 22.38 29.56 29.75 + 14 21.46 28.86 28.63 + 15 20.79 28.24 27.58 + 16 19.17 27.10 26.30 + 17 20.17 27.80 27.43 + 18 20.62 27.94 27.78 + 19 21.03 28.21 28.08 + 20 21.22 28.32 28.22 + 21 21.56 28.45 28.86 + 22 21.72 28.45 29.22 + 23 21.79 28.37 29.46 + 24 21.60 28.12 29.36 + 25 23.65 30.14 31.44 + 26 25.69 32.47 33.81 + 27 27.34 34.34 35.01 + 28 28.70 35.59 35.72 + 29 29.55 36.05 36.21 + 30 30.86 37.23 36.97 + 31 33.49 39.97 39.43 + 32 42.53 44.31 44.90 + 33 40.36 43.06 43.41 +e + 2 41.18 43.75 44.63 + 3 40.53 43.36 44.16 + 4 41.35 43.60 44.45 + 5 40.79 43.47 44.23 + 6 42.06 44.05 44.97 + 7 41.08 43.81 44.56 + 8 43.30 44.97 45.91 + 9 30.63 36.72 37.81 + 10 28.07 34.04 35.15 + 11 25.26 31.69 32.02 + 12 23.60 30.43 30.79 + 13 22.80 29.65 29.85 + 14 21.87 28.97 28.72 + 15 21.25 28.38 27.67 + 16 19.73 27.22 26.40 + 17 20.77 28.00 27.53 + 18 21.27 28.22 27.94 + 19 21.73 28.55 28.25 + 20 21.94 28.73 28.41 + 21 22.27 28.89 29.07 + 22 22.45 28.94 29.47 + 23 22.51 28.92 29.66 + 24 22.28 28.71 29.54 + 25 24.37 30.72 31.62 + 26 26.51 33.18 34.07 + 27 28.19 35.05 35.25 + 28 29.53 36.31 35.98 + 29 30.45 36.56 36.43 + 30 31.84 37.85 37.26 + 31 34.46 40.34 39.78 + 32 42.53 44.31 44.90 + 33 40.36 43.06 43.41 +e + 2 41.18 43.75 44.63 + 3 40.53 43.36 44.16 + 4 41.35 43.60 44.45 + 5 40.79 43.47 44.23 + 6 42.06 44.05 44.97 + 7 41.08 43.81 44.56 + 8 43.30 44.97 45.91 + 9 30.48 36.59 37.75 + 10 27.84 33.83 35.06 + 11 25.12 31.68 31.92 + 12 23.46 30.47 30.68 + 13 22.77 29.63 29.78 + 14 21.82 28.91 28.68 + 15 21.15 28.33 27.64 + 16 19.56 27.16 26.35 + 17 20.63 27.92 27.45 + 18 21.13 28.17 27.88 + 19 21.57 28.52 28.20 + 20 21.77 28.70 28.35 + 21 22.21 28.95 29.03 + 22 22.44 29.06 29.43 + 23 22.52 29.11 29.63 + 24 22.29 29.06 29.52 + 25 24.34 30.93 31.59 + 26 26.42 33.45 34.00 + 27 28.07 35.30 35.17 + 28 29.35 36.59 35.87 + 29 30.28 36.66 36.34 + 30 31.73 37.96 37.23 + 31 34.35 40.36 39.74 + 32 42.53 44.31 44.90 + 33 40.36 43.06 43.41 +e + + +set xrange[6:6+27] +set xtics 6,2,6+27 +set output 'plot_QualityVsFrameSingle_PSNR_randomaccess_slice_single_seq_park_rate_single_6.eps' +set title 'Park Scene' +plot '-' using 1:2 title 'MC' w lp ls 1, '-' using 1:2 title '16x16' w lp ls 2, '-' using 1:2 title 'Proposed' w lp ls 3 + 2 39.63 41.81 43.47 + 3 39.22 41.77 43.39 + 4 39.84 41.86 43.47 + 5 39.47 41.98 43.56 + 6 40.59 42.36 43.88 + 7 40.20 42.19 43.82 + 8 41.52 42.94 44.11 + 9 34.88 41.38 43.19 + 10 32.71 40.76 42.87 + 11 30.87 39.88 42.27 + 12 29.07 39.19 41.75 + 13 28.16 38.42 41.27 + 14 27.01 37.64 40.66 + 15 25.86 36.77 39.95 + 16 23.90 35.38 38.45 + 17 24.58 35.93 38.74 + 18 24.85 36.18 38.84 + 19 25.08 36.22 38.55 + 20 25.18 36.06 37.78 + 21 25.41 36.40 38.42 + 22 25.44 36.65 38.83 + 23 25.46 36.72 38.87 + 24 25.43 37.11 39.67 + 25 27.03 37.93 40.37 + 26 27.74 38.50 40.74 + 27 28.83 39.14 41.21 + 28 30.27 40.13 42.16 + 29 31.34 40.41 42.34 + 30 32.52 40.88 42.54 + 31 34.49 41.29 42.83 + 32 40.96 42.63 43.62 + 33 39.55 42.03 43.51 +e + 2 39.63 41.81 43.47 + 3 39.22 41.77 43.39 + 4 39.84 41.86 43.47 + 5 39.47 41.98 43.56 + 6 40.59 42.36 43.88 + 7 40.20 42.19 43.82 + 8 41.52 42.94 44.11 + 9 35.95 41.51 43.23 + 10 34.35 41.15 43.03 + 11 32.58 40.36 42.48 + 12 30.95 39.79 42.07 + 13 30.14 39.05 41.61 + 14 29.19 38.45 41.14 + 15 27.97 37.58 40.47 + 16 25.85 36.23 39.00 + 17 26.47 36.81 39.62 + 18 26.73 37.03 39.79 + 19 26.91 37.16 39.92 + 20 27.00 37.29 39.97 + 21 27.18 37.35 40.06 + 22 27.25 37.44 40.10 + 23 27.24 37.42 40.09 + 24 27.19 37.54 40.00 + 25 28.87 38.41 41.00 + 26 29.57 38.99 41.35 + 27 30.55 39.59 41.79 + 28 31.98 40.42 42.37 + 29 32.70 40.63 42.48 + 30 33.63 41.13 42.70 + 31 35.31 41.46 42.94 + 32 40.96 42.63 43.62 + 33 39.55 42.03 43.51 +e + 2 39.63 41.81 43.47 + 3 39.22 41.77 43.39 + 4 39.84 41.86 43.47 + 5 39.47 41.98 43.56 + 6 40.59 42.36 43.88 + 7 40.20 42.19 43.82 + 8 41.52 42.94 44.11 + 9 35.70 41.49 43.21 + 10 34.04 41.07 42.99 + 11 32.20 40.24 42.43 + 12 30.54 39.61 41.98 + 13 29.73 38.89 41.53 + 14 28.76 38.25 41.04 + 15 27.54 37.37 40.36 + 16 25.45 36.02 38.84 + 17 26.08 36.60 39.50 + 18 26.37 36.81 39.68 + 19 26.55 36.93 39.82 + 20 26.64 37.01 39.88 + 21 26.83 37.13 39.98 + 22 26.91 37.26 40.01 + 23 26.91 37.25 40.02 + 24 26.86 37.37 39.91 + 25 28.53 38.25 40.93 + 26 29.25 38.85 41.28 + 27 30.25 39.48 41.72 + 28 31.67 40.33 42.32 + 29 32.46 40.57 42.43 + 30 33.43 41.08 42.66 + 31 35.16 41.42 42.90 + 32 40.96 42.63 43.62 + 33 39.55 42.03 43.51 +e + diff --git a/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameLowDelay.tex b/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameLowDelay.tex new file mode 100644 index 0000000..2d67ed8 --- /dev/null +++ b/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameLowDelay.tex @@ -0,0 +1,112 @@ +\newcommand{\pltw}{0.75\columnwidth} +\newcommand{\plth}{0.3\textheight} +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +set xlabel '\TitleFont Frame Number' +set xrange[6:6+27] +set xtics 6,2,6+27 +set key bottom left width 1 +#set title '\TitleFont Basketball Drill' +plot '-' using 1:2 title '\KeyFont \MC' w lp ls 1, '-' using 1:2 title '\KeyFont \emph{16x16}' w lp ls 2, '-' using 1:2 title '\KeyFont \PROP' w lp ls 3 + 2 40.24 43.19 44.01 + 3 40.24 42.96 43.94 + 4 41.26 43.62 44.49 + 5 40.22 42.95 43.82 + 6 41.34 43.71 44.62 + 7 39.72 42.83 43.48 + 8 42.17 44.27 45.07 + 9 41.20 43.56 44.39 + 10 40.87 43.35 44.17 + 11 40.78 43.20 43.86 + 12 41.78 43.98 44.83 + 13 40.72 43.32 43.96 + 14 40.94 43.32 43.96 + 15 40.49 43.10 43.56 + 16 28.87 37.08 36.42 + 17 29.26 36.53 35.77 + 18 28.95 36.57 35.60 + 19 29.17 36.95 35.53 + 20 29.36 37.27 35.82 + 21 29.70 38.06 35.88 + 22 29.98 38.17 35.99 + 23 29.61 37.59 35.62 + 24 29.41 36.40 35.54 + 25 29.31 36.04 35.27 + 26 29.27 35.75 35.18 + 27 29.01 34.66 34.61 + 28 28.80 34.61 34.62 + 29 28.75 34.45 34.43 + 30 28.85 34.77 34.51 + 31 28.91 34.68 34.37 + 32 42.09 44.07 44.58 + 33 40.03 42.78 43.23 +e + 2 40.24 43.19 44.01 + 3 40.24 42.96 43.94 + 4 41.26 43.62 44.49 + 5 40.22 42.95 43.82 + 6 41.34 43.71 44.62 + 7 39.72 42.83 43.48 + 8 42.17 44.27 45.07 + 9 41.20 43.56 44.39 + 10 40.87 43.35 44.17 + 11 40.78 43.20 43.86 + 12 41.78 43.98 44.83 + 13 40.72 43.32 43.96 + 14 40.94 43.32 43.96 + 15 40.49 43.10 43.56 + 16 32.14 40.15 38.22 + 17 32.73 40.47 37.88 + 18 33.20 40.69 37.77 + 19 33.41 40.65 37.77 + 20 33.30 41.23 38.26 + 21 33.53 41.04 38.15 + 22 33.73 40.95 38.33 + 23 33.75 40.72 38.11 + 24 33.95 41.02 38.65 + 25 33.79 40.69 38.31 + 26 33.92 40.64 38.16 + 27 33.92 40.36 37.86 + 28 34.39 40.73 38.13 + 29 34.26 40.35 37.85 + 30 34.44 40.40 37.77 + 31 34.43 40.16 37.59 + 32 42.09 44.07 44.58 + 33 40.03 42.78 43.23 +e + 2 40.24 43.19 44.01 + 3 40.24 42.96 43.94 + 4 41.26 43.62 44.49 + 5 40.22 42.95 43.82 + 6 41.34 43.71 44.62 + 7 39.72 42.83 43.48 + 8 42.17 44.27 45.07 + 9 41.20 43.56 44.39 + 10 40.87 43.35 44.17 + 11 40.78 43.20 43.86 + 12 41.78 43.98 44.83 + 13 40.72 43.32 43.96 + 14 40.94 43.32 43.96 + 15 40.49 43.10 43.56 + 16 31.62 40.07 37.61 + 17 32.33 40.31 37.21 + 18 32.85 40.64 37.20 + 19 33.01 40.57 37.21 + 20 33.07 41.23 37.70 + 21 33.30 40.99 37.60 + 22 33.22 40.80 37.50 + 23 33.04 40.54 37.20 + 24 33.18 40.68 37.06 + 25 32.94 40.35 36.68 + 26 33.06 40.29 36.25 + 27 33.02 39.91 35.78 + 28 33.41 40.25 35.77 + 29 33.38 39.87 35.51 + 30 33.64 39.94 35.43 + 31 33.63 39.68 35.24 + 32 42.09 44.07 44.58 + 33 40.03 42.78 43.23 +e + + +\end{gnuplot} diff --git a/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameRandomAccess.tex b/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameRandomAccess.tex new file mode 100644 index 0000000..72cebc9 --- /dev/null +++ b/042_ConcealmentAwareEncoding/Results/IntraEC_QualityVsFrameRandomAccess.tex @@ -0,0 +1,112 @@ +%\newcommand{\pltw}{0.75\columnwidth} +%\newcommand{\plth}{0.3\textheight} +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +set xlabel '\TitleFont Frame Number' +set xrange[6:6+27] +set xtics 6,2,6+27 +set key bottom left width 1 +#set title '\TitleFont Park Scene' +plot '-' using 1:2 title '\KeyFont \MC' w lp ls 1, '-' using 1:2 title '\KeyFont \emph{16x16}' w lp ls 2, '-' using 1:2 title '\KeyFont \PROP' w lp ls 3 + 2 39.63 41.81 43.47 + 3 39.22 41.77 43.39 + 4 39.84 41.86 43.47 + 5 39.47 41.98 43.56 + 6 40.59 42.36 43.88 + 7 40.20 42.19 43.82 + 8 41.52 42.94 44.11 + 9 34.88 41.38 43.19 + 10 32.71 40.76 42.87 + 11 30.87 39.88 42.27 + 12 29.07 39.19 41.75 + 13 28.16 38.42 41.27 + 14 27.01 37.64 40.66 + 15 25.86 36.77 39.95 + 16 23.90 35.38 38.45 + 17 24.58 35.93 38.74 + 18 24.85 36.18 38.84 + 19 25.08 36.22 38.55 + 20 25.18 36.06 37.78 + 21 25.41 36.40 38.42 + 22 25.44 36.65 38.83 + 23 25.46 36.72 38.87 + 24 25.43 37.11 39.67 + 25 27.03 37.93 40.37 + 26 27.74 38.50 40.74 + 27 28.83 39.14 41.21 + 28 30.27 40.13 42.16 + 29 31.34 40.41 42.34 + 30 32.52 40.88 42.54 + 31 34.49 41.29 42.83 + 32 40.96 42.63 43.62 + 33 39.55 42.03 43.51 +e + 2 39.63 41.81 43.47 + 3 39.22 41.77 43.39 + 4 39.84 41.86 43.47 + 5 39.47 41.98 43.56 + 6 40.59 42.36 43.88 + 7 40.20 42.19 43.82 + 8 41.52 42.94 44.11 + 9 35.95 41.51 43.23 + 10 34.35 41.15 43.03 + 11 32.58 40.36 42.48 + 12 30.95 39.79 42.07 + 13 30.14 39.05 41.61 + 14 29.19 38.45 41.14 + 15 27.97 37.58 40.47 + 16 25.85 36.23 39.00 + 17 26.47 36.81 39.62 + 18 26.73 37.03 39.79 + 19 26.91 37.16 39.92 + 20 27.00 37.29 39.97 + 21 27.18 37.35 40.06 + 22 27.25 37.44 40.10 + 23 27.24 37.42 40.09 + 24 27.19 37.54 40.00 + 25 28.87 38.41 41.00 + 26 29.57 38.99 41.35 + 27 30.55 39.59 41.79 + 28 31.98 40.42 42.37 + 29 32.70 40.63 42.48 + 30 33.63 41.13 42.70 + 31 35.31 41.46 42.94 + 32 40.96 42.63 43.62 + 33 39.55 42.03 43.51 +e + 2 39.63 41.81 43.47 + 3 39.22 41.77 43.39 + 4 39.84 41.86 43.47 + 5 39.47 41.98 43.56 + 6 40.59 42.36 43.88 + 7 40.20 42.19 43.82 + 8 41.52 42.94 44.11 + 9 35.70 41.49 43.21 + 10 34.04 41.07 42.99 + 11 32.20 40.24 42.43 + 12 30.54 39.61 41.98 + 13 29.73 38.89 41.53 + 14 28.76 38.25 41.04 + 15 27.54 37.37 40.36 + 16 25.45 36.02 38.84 + 17 26.08 36.60 39.50 + 18 26.37 36.81 39.68 + 19 26.55 36.93 39.82 + 20 26.64 37.01 39.88 + 21 26.83 37.13 39.98 + 22 26.91 37.26 40.01 + 23 26.91 37.25 40.02 + 24 26.86 37.37 39.91 + 25 28.53 38.25 40.93 + 26 29.25 38.85 41.28 + 27 30.25 39.48 41.72 + 28 31.67 40.33 42.32 + 29 32.46 40.57 42.43 + 30 33.43 41.08 42.66 + 31 35.16 41.42 42.90 + 32 40.96 42.63 43.62 + 33 39.55 42.03 43.51 +e + + +\end{gnuplot} diff --git a/042_ConcealmentAwareEncoding/Results/PSNRversusFrameKendo.tex b/042_ConcealmentAwareEncoding/Results/PSNRversusFrameKendo.tex new file mode 100644 index 0000000..bd9eb8f --- /dev/null +++ b/042_ConcealmentAwareEncoding/Results/PSNRversusFrameKendo.tex @@ -0,0 +1,178 @@ +\newcommand{\pltw}{0.75\columnwidth} +\newcommand{\plth}{0.3\textheight} +\begin{gnuplot}[terminal=epslatex,terminaloptions={monochrome font ",\PlotFont" size \convertlen{\pltw},\convertlen{\plth}}] +load 'carreira_phd_thesis_gnuplot_template.cfg' +set xlabel '\TitleFont Frame Number' +set xrange[2:0+31] +set xtics 0,2,0+31 +set yrange[33:45] +set ytics 33,2,45 +set key spacing 1.4 bottom right width 6 box +plot '-' using 1:2 title '\KeyFont \EXP' w lp ls 1, '-' using 1:2 title '\KeyFont \CHKB' w lp ls 2, '-' using 1:2 title '\KeyFont \AVG' w lp ls 3, '-' using 1:2 title '\KeyFont \REF' w lp ls 4 + 0 43.01 44.05 43.66 + 1 41.71 43.89 43.36 + 2 40.83 43.77 43.16 + 3 41.39 43.81 43.15 + 4 42.64 43.99 43.44 + 5 41.52 43.74 43.07 + 6 41.50 43.72 42.98 + 7 36.44 43.60 42.80 + 8 42.37 44.15 43.63 + 9 40.49 43.68 42.86 + 10 41.69 43.89 43.27 + 11 40.69 43.56 42.55 + 12 43.33 44.41 43.96 + 13 40.74 43.48 42.61 + 14 41.55 43.79 42.92 + 15 40.86 43.64 42.59 + 16 43.38 44.33 43.97 + 17 41.36 43.54 42.78 + 18 41.76 43.70 43.00 + 19 41.37 43.46 42.75 + 20 43.10 44.25 43.80 + 21 41.02 43.57 42.64 + 22 41.62 43.65 42.85 + 23 41.30 43.69 42.69 + 24 43.08 44.25 43.78 + 25 41.33 43.67 42.72 + 26 41.71 43.90 42.98 + 27 41.34 43.79 42.86 + 28 43.19 44.33 43.89 + 29 41.42 43.76 42.94 + 30 41.62 43.90 43.06 + 31 41.57 43.88 43.03 + 32 43.21 44.27 43.80 + 33 42.26 44.06 43.42 + 34 42.17 43.91 43.23 + 35 41.95 43.79 42.96 + 36 43.36 44.25 43.54 + 37 41.28 43.54 42.30 + 38 41.64 43.57 42.41 +e + 0 43.01 44.05 43.66 + 1 41.71 43.89 43.36 + 2 40.76 43.77 43.14 + 3 41.36 43.78 43.15 + 4 42.65 43.95 43.44 + 5 41.50 43.79 43.04 + 6 41.53 43.73 43.03 + 7 37.34 43.60 42.75 + 8 41.64 44.13 43.53 + 9 39.48 43.64 42.74 + 10 40.94 43.90 43.18 + 11 40.22 43.50 42.46 + 12 42.30 44.36 43.92 + 13 40.97 43.55 42.72 + 14 41.60 43.72 42.78 + 15 40.73 43.52 42.42 + 16 42.89 44.35 43.94 + 17 41.05 43.66 42.77 + 18 41.54 43.74 42.84 + 19 41.34 43.65 42.82 + 20 43.04 44.18 43.61 + 21 41.13 43.51 42.55 + 22 41.58 43.72 42.84 + 23 41.12 43.69 42.69 + 24 42.89 44.26 43.62 + 25 41.49 43.81 42.82 + 26 41.76 43.97 42.99 + 27 41.31 43.91 42.92 + 28 43.19 44.40 43.84 + 29 41.36 43.83 42.89 + 30 41.41 43.80 42.84 + 31 41.38 43.80 42.74 + 32 43.21 44.27 43.80 + 33 42.30 44.08 43.44 + 34 42.18 43.96 43.22 + 35 41.96 43.81 43.00 + 36 43.36 44.16 43.48 +e + 0 43.01 44.05 43.66 + 1 41.71 43.89 43.36 + 2 40.80 43.77 43.15 + 3 41.41 43.81 43.14 + 4 42.71 43.98 43.47 + 5 41.39 43.73 42.99 + 6 41.57 43.70 42.84 + 7 35.60 43.24 41.69 + 8 41.02 44.13 43.48 + 9 39.06 43.40 42.29 + 10 40.06 43.59 42.66 + 11 38.91 43.02 41.76 + 12 41.66 44.26 43.70 + 13 39.59 43.25 42.18 + 14 39.50 43.13 41.82 + 15 38.84 42.76 41.28 + 16 41.39 43.95 43.19 + 17 39.50 42.92 41.51 + 18 39.41 43.09 41.45 + 19 39.15 42.80 41.15 + 20 40.83 43.76 42.87 + 21 39.69 42.92 41.51 + 22 39.78 42.91 41.39 + 23 39.44 42.96 41.47 + 24 41.51 43.81 42.89 + 25 39.73 43.01 41.87 + 26 40.21 42.99 41.77 + 27 39.61 42.73 41.59 + 28 41.79 44.00 43.08 + 29 39.69 43.02 41.93 + 30 39.86 42.88 41.67 + 31 39.61 42.86 41.77 + 32 43.21 44.27 43.80 + 33 41.97 44.06 43.41 + 34 41.89 43.89 43.09 + 35 41.43 43.67 42.67 + 36 43.19 44.09 43.42 + 37 40.66 43.41 42.00 + 38 40.92 43.39 41.92 + 39 40.31 43.07 41.44 +e + 0 43.01 44.05 43.66 + 1 41.71 43.89 43.36 + 2 40.83 43.78 43.17 + 3 41.33 43.79 43.12 + 4 42.58 43.98 43.42 + 5 41.85 43.88 43.17 + 6 41.55 43.79 43.02 + 7 34.55 43.16 41.33 + 8 36.21 43.72 42.06 + 9 36.75 43.69 42.04 + 10 37.41 43.77 42.07 + 11 37.40 43.68 41.91 + 12 38.79 44.21 42.81 + 13 38.41 43.94 42.43 + 14 39.02 44.05 42.62 + 15 38.88 43.85 42.29 + 16 39.69 44.44 43.49 + 17 39.01 44.09 42.95 + 18 39.55 44.09 42.99 + 19 39.37 43.81 42.64 + 20 40.24 44.45 43.52 + 21 39.61 44.09 43.04 + 22 39.65 43.99 42.93 + 23 39.42 43.91 42.69 + 24 40.38 44.42 43.51 + 25 39.77 44.18 43.18 + 26 39.92 44.25 43.19 + 27 39.83 44.14 43.03 + 28 40.60 44.66 43.90 + 29 40.08 44.38 43.49 + 30 40.00 44.30 43.35 + 31 39.92 44.27 43.20 + 32 43.21 44.27 43.80 + 33 42.52 44.08 43.51 + 34 42.55 44.02 43.39 + 35 42.17 43.88 43.08 + 36 44.01 44.50 44.13 + 37 42.68 44.19 43.54 + 38 42.64 44.10 43.32 + 39 42.44 43.99 43.12 + 40 44.21 44.64 44.27 + 41 43.26 44.41 43.88 + 42 43.35 44.44 43.87 + 43 43.13 44.43 43.77 + 44 44.52 44.86 44.61 + 45 43.55 44.72 44.30 +e +\end{gnuplot} diff --git a/042_ConcealmentAwareEncoding/Results/PerformanceConcealmentMethods.tex b/042_ConcealmentAwareEncoding/Results/PerformanceConcealmentMethods.tex new file mode 100644 index 0000000..259c9ed --- /dev/null +++ b/042_ConcealmentAwareEncoding/Results/PerformanceConcealmentMethods.tex @@ -0,0 +1,47 @@ +%\small +%\newcommand{\colwidth}{1.6cm} +%\newcommand\mergedlines{1} +%\centering +%\begin{tabular}{p{2.7cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} +%\toprule +%\centering \textbf{Sequence} & \REF & \MVE & \FULLSEARCH & \DUALFLOW & \POLYFLOW & \DEEPFLOW \\ +%Basketball Drill & 28.22 & +2.43 & +2.61 & +2.71 & +3.08 & +3.33 \\ \hline +%Book Arrival & 31.01 & +2.50 & +2.31 & +2.81 & +3.05 & +3.39 \\ \hline +%BQSquare & 31.36 & +2.28 & +0.43 & +1.38 & +1.93 & +1.88 \\ \hline +%Cactus & 31.93 & +0.91 & +0.77 & +0.96 & +1.05 & +1.08 \\ \hline +%Four People & 37.12 & +1.24 & +0.33 & +1.38 & +1.44 & +1.44 \\ \hline +%Kendo & 31.57 & +2.34 & +2.04 & +2.80 & +2.78 & +3.05 \\ \hline +%Kimono & 32.24 & +0.12 & +0.10 & +0.13 & +0.13 & +0.14 \\ \hline +%Park Scene & 30.42 & +1.30 & +1.21 & +1.23 & +1.50 & +1.57 \\ \hline +%People on Street & 25.05 & +0.40 & +0.36 & +0.56 & +0.61 & +0.61 \\ \hline +%Race Horses & 24.19 & +0.68 & +0.66 & +0.66 & +0.76 & +0.81 \\ \hline +%Tennis & 27.08 & +1.34 & +1.17 & +1.65 & +0.74 & +2.37 \\ \hline +%\textbf{Average} & \textbf{+30.02} & \textbf{+1.41} & \textbf{+1.09} & \textbf{+1.48} & \textbf{+1.55} & \textbf{+1.79} \\ +%\bottomrule +%\end{tabular} + + + +\small +\footnotesize +\newcommand{\colwidth}{1.35cm} +\newcommand\mergedlines{1} +\centering +\begin{tabular}{p{2.7cm}|x{1.25cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} +\toprule +\centering \textbf{Sequence} & \FC & \MC & \MVE & \FULLSEARCHLONG & \DUALFLOWLONG & \POLYFLOWLONG & \DEEPFLOWLONG \\ +\hline +Basketball Drill & 28.22 & +2.43 & +2.56 & +2.61 & +2.71 & +3.08 & +3.33 \\ \hline +Book Arrival & 31.01 & +2.50 & +2.49 & +2.31 & +2.81 & +3.05 & +3.39 \\ \hline +BQSquare & 31.36 & +2.28 & +2.32 & +0.43 & +1.38 & +1.93 & +1.88 \\ \hline +Cactus & 31.93 & +0.91 & +0.96 & +0.77 & +0.96 & +1.05 & +1.08 \\ \hline +Four People & 37.12 & +1.24 & +1.26 & +0.33 & +1.38 & +1.44 & +1.44 \\ \hline +Kendo & 31.57 & +2.34 & +2.36 & +2.04 & +2.80 & +2.78 & +3.05 \\ \hline +Kimono & 32.24 & +0.12 & +0.12 & +0.10 & +0.13 & +0.13 & +0.14 \\ \hline +Park Scene & 30.42 & +1.30 & +1.38 & +1.21 & +1.23 & +1.50 & +1.57 \\ \hline +People on Street & 25.05 & +0.40 & +0.43 & +0.36 & +0.56 & +0.61 & +0.61 \\ \hline +Race Horses & 24.19 & +0.68 & +0.67 & +0.66 & +0.66 & +0.76 & +0.81 \\ \hline +Tennis & 27.08 & +1.34 & +1.56 & +1.17 & +1.65 & +0.74 & +2.37 \\ \hline +\textbf{Average} & \textbf{--} & \textbf{+1.41} & \textbf{+1.46} & \textbf{+1.09} & \textbf{+1.48} & \textbf{+1.55} & \textbf{+1.79} \\ +\bottomrule +\end{tabular} diff --git a/042_ConcealmentAwareEncoding/Results/QualityVsFrameSingle.pdf b/042_ConcealmentAwareEncoding/Results/QualityVsFrameSingle.pdf new file mode 100644 index 0000000..ecaa494 Binary files /dev/null and b/042_ConcealmentAwareEncoding/Results/QualityVsFrameSingle.pdf differ diff --git a/042_ConcealmentAwareEncoding/Results/QualityVsFrameSingleIntra.pdf b/042_ConcealmentAwareEncoding/Results/QualityVsFrameSingleIntra.pdf new file mode 100644 index 0000000..e3381bf Binary files /dev/null and b/042_ConcealmentAwareEncoding/Results/QualityVsFrameSingleIntra.pdf differ diff --git a/042_ConcealmentAwareEncoding/Results/TableOveralPerformance.tex b/042_ConcealmentAwareEncoding/Results/TableOveralPerformance.tex new file mode 100644 index 0000000..b4fbf06 --- /dev/null +++ b/042_ConcealmentAwareEncoding/Results/TableOveralPerformance.tex @@ -0,0 +1,141 @@ + +%% Whole +% \begin{table}[t] +% \centering +% \caption{Average PSNR for different methods and different overhead ratios at the same total bitrate.} +% \label{tab:concealmentResidueRandomLoss} +% % \begin{adjustbox}{angle=90} +% \newcommand{\colwidth}{1.1cm} +% \begin{tabular}{p{2cm} +% |x{\colwidth}x{\colwidth}x{\colwidth}x{\colwidth}x{\colwidth} +% |x{\colwidth}x{\colwidth}x{\colwidth}x{\colwidth}x{\colwidth}} +% \toprule +% \multirow{3}{*}{\vbox{\centering \textbf{Sequence}}} +% % & \multicolumn{4}{c||}{\textbf{Error-free}} +% & \multicolumn{5}{c|}{\textbf{PLR=3\%}} & \multicolumn{5}{c}{\textbf{PLR=10\%}}\\ +% & \multirow{2}{*}{\vbox{\centering \REF}} & \multirow{2}{*}{\vbox{\centering \MC}} +% & \multirow{2}{*}{\vbox{\centering \AVG}} & \multicolumn{2}{c|}{\PROP} +% & \multirow{2}{*}{\vbox{\centering \REF}} & \multirow{2}{*}{\vbox{\centering \MC}} +% & \multirow{2}{*}{\vbox{\centering \AVG}} & \multicolumn{2}{c}{\PROP} \\ +% & & & & $10\%$ & $20\%$ & & & & $10\%$ & $20\%$ \\ +% \midrule +% & \multicolumn{10}{c}{Low-delay configuration} \\ [2pt] +% \hline +% Basketball Drill & 33.31 & +2.10 & +3.86 & +4.62 & +4.56 & 28.22 & +2.34 & +5.32 & +6.26 & +6.90 \\ \hline +% Book Arrival & 36.21 & +1.33 & +2.07 & +2.71 & +2.99 & 31.01 & +2.48 & +4.49 & +5.60 & +6.31 \\ \hline +% BQSquare & 35.64 & +1.47 & +1.18 & +2.01 & +1.77 & 31.36 & +2.50 & +2.40 & +3.33 & +3.55 \\ \hline +% Four People & 39.16 & +0.47 & -0.65 & +0.40 & -0.06 & 37.12 & +1.24 & -0.99 & +1.82 & +1.51 \\ \hline +% Kimono & 36.19 & +1.41 & -0.54 & +3.09 & +3.47 & 32.24 & +0.12 & -1.92 & -0.04 & -0.21 \\ \hline +% Park Scene & 34.54 & +2.06 & +2.01 & +3.22 & +3.15 & 30.42 & +1.22 & +1.56 & +2.92 & +3.64 \\ \hline +% People & 29.48 & +1.50 & +1.35 & +2.64 & +3.08 & 25.05 & +0.40 & +0.74 & +0.97 & +2.09 \\ \hline +% Race Horses & 29.08 & +2.34 & +4.52 & +5.35 & +6.02 & 24.19 & +0.63 & +2.53 & +2.75 & +3.74 \\ \hline +% Tennis & 32.71 & +1.32 & +3.13 & +3.81 & +4.97 & 27.08 & +1.30 & +3.51 & +4.58 & +6.61 \\ \hline +% \textbf{Average} & \textbf{--} & \textbf{+1.55} & \textbf{+1.88} & \textbf{+3.10} & \textbf{+3.33} +% & \textbf{--} & \textbf{+1.36} & \textbf{+1.96} & \textbf{+3.13} & \textbf{+3.79} \\ +% \midrule +% & \multicolumn{10}{c}{Random-access configuration} \\ [2pt] +% \hline +% Basketball Drill & 35.40 & +0.17 & +0.20 & +1.39 & +1.53 & 29.92 & +0.14 & +1.45 & +1.84 & +2.42 \\ \hline +% Book Arrival & 37.26 & +0.05 & +0.72 & +1.31 & +1.28 & 32.27 & +0.13 & +2.58 & +2.74 & +2.91 \\ \hline +% BQSquare & 34.32 & -0.06 & +0.51 & +1.46 & +1.59 & 28.13 & -0.01 & +2.74 & +2.06 & +2.85 \\ \hline +% Four People & 39.92 & +0.02 & -0.39 & +0.29 & +0.02 & 37.81 & +0.09 & -0.05 & +1.46 & +1.68 \\ \hline +% Kimono & 36.98 & +0.01 & -2.44 & +0.34 & +0.00 & 33.09 & +0.00 & -0.03 & -1.15 & -2.02 \\ \hline +% Park Scene & 35.18 & +0.06 & -1.27 & +1.11 & +1.13 & 30.93 & +0.01 & +0.73 & +0.83 & +1.02 \\ \hline +% People on Street & 29.21 & +0.03 & -0.53 & +0.58 & +0.95 & 24.57 & +0.00 & +1.70 & +0.09 & +0.35 \\ \hline +% Race Horses & 30.89 & +0.09 & -0.34 & +1.38 & +1.42 & 25.71 & +0.01 & +1.30 & +0.58 & +0.71 \\ \hline +% Tennis & 33.86 & +0.16 & +0.18 & +0.98 & +1.48 & 27.74 & +0.07 & +1.35 & +1.17 & +2.09 \\ \hline +% \textbf{Average} & \textbf{--} & \textbf{+0.06} & \textbf{-0.37} & \textbf{+0.98} & \textbf{+1.05} +% & \textbf{--} & \textbf{+0.05} & \textbf{+1.31} & \textbf{+1.07} & \textbf{+1.33} \\ +% \bottomrule +% \end{tabular} +% % \end{adjustbox} +% \end{table} + + +%% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% %% +%% Split + +\begin{table}[!t] +\centering +\caption{Average quality under 3\% of PLR for different methods and overhead ratios.} +\label{tab:concealmentResidueRandomLoss_PLR3} +\small +\footnotesize +\newcommand{\colwidth}{2cm} +\begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} +\toprule +\multirow{3}{*}{\vbox{\centering \textbf{Sequence}}} +& \multirow{2}{*}{\vbox{\centering \REF}} & \multirow{2}{*}{\vbox{\centering \MC}} & \multirow{2}{*}{\vbox{\centering \AVG}} & \multicolumn{2}{c}{\PROP} \\ +& & & & $10\%$ & $20\%$ \\ +\midrule +& \multicolumn{5}{c}{Low-delay configuration} \\ [2pt] +\hline +Basketball Drill & 33.31 & +2.10 & +3.86 & +4.62 & +4.56 \\ \hline +Book Arrival & 36.21 & +1.33 & +2.07 & +2.71 & +2.99 \\ \hline +BQSquare & 35.64 & +1.47 & +1.18 & +2.01 & +1.77 \\ \hline +Four People & 39.16 & +0.47 & -0.65 & +0.40 & -0.06 \\ \hline +Kimono & 36.19 & +1.41 & -0.54 & +3.09 & +3.47 \\ \hline +Park Scene & 34.54 & +2.06 & +2.01 & +3.22 & +3.15 \\ \hline +People & 29.48 & +1.50 & +1.35 & +2.64 & +3.08 \\ \hline +Race Horses & 29.08 & +2.34 & +4.52 & +5.35 & +6.02 \\ \hline +Tennis & 32.71 & +1.32 & +3.13 & +3.81 & +4.97 \\ \hline +\textbf{Average} & \textbf{--} & \textbf{+1.55} & \textbf{+1.88} & \textbf{+3.10} & \textbf{+3.33} \\ +\midrule +& \multicolumn{5}{c}{Random-access configuration} \\ [2pt] +\hline +Basketball Drill & 35.40 & +0.17 & +0.20 & +1.39 & +1.53 \\ \hline +Book Arrival & 37.26 & +0.05 & +0.72 & +1.31 & +1.28 \\ \hline +BQSquare & 34.32 & -0.06 & +0.51 & +1.46 & +1.59 \\ \hline +Four People & 39.92 & +0.02 & -0.39 & +0.29 & +0.02 \\ \hline +Kimono & 36.98 & +0.01 & -2.44 & +0.34 & +0.00 \\ \hline +Park Scene & 35.18 & +0.06 & -1.27 & +1.11 & +1.13 \\ \hline +People on Street & 29.21 & +0.03 & -0.53 & +0.58 & +0.95 \\ \hline +Race Horses & 30.89 & +0.09 & -0.34 & +1.38 & +1.42 \\ \hline +Tennis & 33.86 & +0.16 & +0.18 & +0.98 & +1.48 \\ \hline +\textbf{Average} & \textbf{--} & \textbf{+0.06} & \textbf{-0.37} & \textbf{+0.98} & \textbf{+1.05} \\ +\bottomrule +\end{tabular} +\end{table} + + +\begin{table}[!t] +\centering +\caption{Average quality under 10\% of PLR.} +\label{tab:concealmentResidueRandomLoss_PLR10} +\small +\footnotesize +\newcommand{\colwidth}{2cm} +\begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} +\toprule +\multirow{3}{*}{\vbox{\centering \textbf{Sequence}}} +& \multirow{2}{*}{\vbox{\centering \REF}} & \multirow{2}{*}{\vbox{\centering \MC}} & \multirow{2}{*}{\vbox{\centering \AVG}} & \multicolumn{2}{c}{\PROP} \\ +& & & & $10\%$ & $20\%$\\ +\midrule +& \multicolumn{5}{c}{Low-delay configuration} \\ [2pt] +\hline +Basketball Drill & 28.22 & +2.34 & +5.32 & +6.26 & +6.90 \\ \hline +Book Arrival & 31.01 & +2.48 & +4.49 & +5.60 & +6.31 \\ \hline +BQSquare & 31.36 & +2.50 & +2.40 & +3.33 & +3.55 \\ \hline +Four People & 37.12 & +1.24 & -0.99 & +1.82 & +1.51 \\ \hline +Kimono & 32.24 & +0.12 & -1.92 & -0.04 & -0.21 \\ \hline +Park Scene & 30.42 & +1.22 & +1.56 & +2.92 & +3.64 \\ \hline +People & 25.05 & +0.40 & +0.74 & +0.97 & +2.09 \\ \hline +Race Horses & 24.19 & +0.63 & +2.53 & +2.75 & +3.74 \\ \hline +Tennis & 27.08 & +1.30 & +3.51 & +4.58 & +6.61 \\ \hline +\textbf{Average} & \textbf{--} & \textbf{+1.36} & \textbf{+1.96} & \textbf{+3.13} & \textbf{+3.79} \\ +\midrule +& \multicolumn{5}{c}{Random-access configuration} \\ [2pt] +\hline +Basketball Drill & 29.92 & +0.14 & +1.45 & +1.84 & +2.42 \\ \hline +Book Arrival & 32.27 & +0.13 & +2.58 & +2.74 & +2.91 \\ \hline +BQSquare & 28.13 & -0.01 & +2.74 & +2.06 & +2.85 \\ \hline +Four People & 37.81 & +0.09 & -0.05 & +1.46 & +1.68 \\ \hline +Kimono & 33.09 & +0.00 & -0.03 & -1.15 & -2.02 \\ \hline +Park Scene & 30.93 & +0.01 & +0.73 & +0.83 & +1.02 \\ \hline +People on Street & 24.57 & +0.00 & +1.70 & +0.09 & +0.35 \\ \hline +Race Horses & 25.71 & +0.01 & +1.30 & +0.58 & +0.71 \\ \hline +Tennis & 27.74 & +0.07 & +1.35 & +1.17 & +2.09 \\ \hline +\textbf{Average} & \textbf{--} & \textbf{+0.05} & \textbf{+1.31} & \textbf{+1.07} & \textbf{+1.33} \\ +\bottomrule +\end{tabular} +\end{table} diff --git a/042_ConcealmentAwareEncoding/Results/TableSignallingPerformance.tex b/042_ConcealmentAwareEncoding/Results/TableSignallingPerformance.tex new file mode 100644 index 0000000..345e959 --- /dev/null +++ b/042_ConcealmentAwareEncoding/Results/TableSignallingPerformance.tex @@ -0,0 +1,173 @@ +% % \begin{table}[tbp] +% \begin{sidewaystable} +% \newcommand{\somespaces}{} +% \centering +% \caption{Average quality in frame loss conditions for Low-Delay configuration.} +% \label{TAB4:SignallingRandomLoss} +% \TABLESVSPACE +% \small +% \renewcommand*{\arraystretch}{1.25} +% % \resizebox{\textwidth}{!}{ +% \newcommand{\smallWdith}{1.9cm} +% \newcommand{\colwidth}{2.3cm} +% \begin{tabular}{p{3.2cm}|x{\smallWdith}x{\smallWdith}x{\colwidth}x{\colwidth}|x{\smallWdith}x{\smallWdith}x{\colwidth}x{\colwidth}} +% \toprule +% \multirow{3}{*}{\vbox{\textbf{Sequence}}} & \multicolumn{4}{c|}{\textbf{PLR=3\%}} & \multicolumn{4}{c}{\textbf{PLR=10\%}}\\ +% & \REF & \MC & \FIXEDLAMBDALONG & \DYNLAMBDALONG & \REF & \MC & \FIXEDLAMBDALONG & \DYNLAMBDALONG \\ \hline +% Basketball Drill & 28.41 (29.25) & +2.93 (+3.22) & +5.28 (+5.02) & +2.64 (+5.01) & 24.64 (22.89) & +2.63 (+3.27) & +4.93 (+5.66) & +4.82 (+5.66) \\ \hline +% Book Arrival & 30.69 (33.53) & +2.31 (+1.56) & +3.80 (+2.61) & +3.73 (+3.17) & 28.29 (27.28) & +2.65 (+2.82) & +4.59 (+4.91) & +4.51 (+5.86) \\ \hline +% BQSquare & 31.82 (35.39) & +1.97 (+1.59) & +2.21 (+1.75) & +2.19 (+2.29) & 28.06 (30.91) & +2.59 (+2.52) & +3.14 (+2.97) & +3.11 (+3.82) \\ \hline +% Four People & 34.54 (37.03) & +1.46 (+0.75) & +1.91 (+0.96) & +1.89 (+1.14) & 33.07 (34.04) & +1.85 (+1.86) & +2.59 (+2.52) & +2.56 (+2.83) \\ \hline +% Kimono & 32.39 (33.86) & +1.66 (+1.76) & +2.32 (+2.43) & +2.30 (+2.65) & 27.98 (29.54) & +0.14 (+0.14) & +0.19 (+0.19) & +0.19 (+0.50) \\ \hline +% Park Scene & 31.66 (31.76) & +2.28 (+2.89) & +2.96 (+4.03) & +2.93 (+4.27) & 27.76 (26.82) & +1.32 (+1.53) & +1.93 (+2.42) & +1.90 (+2.81) \\ \hline +% People on Street & 26.72 (27.24) & +1.59 (+1.78) & +2.96 (+3.21) & +2.88 (+3.35) & 22.53 (22.57) & +0.40 (+0.45) & +0.85 (+0.91) & +0.83 (+1.06) \\ \hline +% Race Horses & 25.97 (27.81) & +2.55 (+2.73) & +3.80 (+3.98) & +3.69 (+4.29) & 21.55 (22.63) & +0.68 (+0.76) & +1.12 (+1.23) & +1.07 (+1.50) \\ \hline +% Tennis & 28.47 (30.79) & +1.66 (+1.27) & +4.39 (+3.52) & +4.27 (+3.62) & 23.77 (24.71) & +1.41 (+1.13) & +4.34 (+3.79) & +4.19 (+3.84) \\ \hline +% \textbf{Average} & \textbf{--} & \textbf{+2.04 (+1.95)} & \textbf{+3.29 (+3.06)} & \textbf{+2.95 (+3.31)} & \textbf{--} & \textbf{+1.52 (+1.61)} & \textbf{+2.63 (+2.73)} & \textbf{+2.58 (+3.10)} \\ +% \bottomrule +% \end{tabular} +% % } +% % \end{table} +% \end{sidewaystable} + + +% \begin{table}[tb] +% \newcommand{\somespaces}{} +% \centering +% % \caption{Average quality in frame loss conditions for Low-Delay configuration.} +% \caption{Average decoded quality for 3\% of PLR.} +% \label{tab:randomloss_plr3} +% \small +% % \renewcommand*{\arraystretch}{1.25} +% % \resizebox{\textwidth}{!}{ +% \newcommand{\colwidth}{2.3cm} +% \begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} +% \toprule +% % \multirow{3}{*}{\vbox{\textbf{Sequence}}} & \multicolumn{4}{c|}{\textbf{PLR=3\%}} & \multicolumn{4}{c}{\textbf{PLR=10\%}}\\ +% & \REF & \MC & \FIXEDLAMBDALONG & \DYNLAMBDALONG \\ \hline +% Basketball Drill & 28.41 (29.25) & +2.93 (+3.22) & +5.28 (+5.02) & +2.64 (+5.01) \\ \hline +% Book Arrival & 30.69 (33.53) & +2.31 (+1.56) & +3.80 (+2.61) & +3.73 (+3.17) \\ \hline +% BQSquare & 31.82 (35.39) & +1.97 (+1.59) & +2.21 (+1.75) & +2.19 (+2.29) \\ \hline +% Four People & 34.54 (37.03) & +1.46 (+0.75) & +1.91 (+0.96) & +1.89 (+1.14) \\ \hline +% Kimono & 32.39 (33.86) & +1.66 (+1.76) & +2.32 (+2.43) & +2.30 (+2.65) \\ \hline +% Park Scene & 31.66 (31.76) & +2.28 (+2.89) & +2.96 (+4.03) & +2.93 (+4.27) \\ \hline +% People on Street & 26.72 (27.24) & +1.59 (+1.78) & +2.96 (+3.21) & +2.88 (+3.35) \\ \hline +% Race Horses & 25.97 (27.81) & +2.55 (+2.73) & +3.80 (+3.98) & +3.69 (+4.29) \\ \hline +% Tennis & 28.47 (30.79) & +1.66 (+1.27) & +4.39 (+3.52) & +4.27 (+3.62) \\ \hline +% \textbf{Average} & \textbf{--} & \textbf{+2.04 (+1.95)} & \textbf{+3.29 (+3.06)} & \textbf{+2.95 (+3.31)} \\ +% \bottomrule +% \end{tabular} +% % } +% \end{table} +% \begin{table}[tb] +% \newcommand{\somespaces}{} +% \centering +% \caption{Average decoded quality for 10\% of PLR.} +% \label{tab:randomloss_plr10} +% \small +% % \renewcommand*{\arraystretch}{1.25} +% % \resizebox{\textwidth}{!}{ +% \newcommand{\colwidth}{2.3cm} +% \begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} +% \toprule +% % \multirow{3}{*}{\vbox{\textbf{Sequence}}} & \multicolumn{4}{c|}{\textbf{PLR=3\%}} & \multicolumn{4}{c}{\textbf{PLR=10\%}}\\ +% & \REF & \MC & \FIXEDLAMBDALONG & \DYNLAMBDALONG \\ \hline +% Basketball Drill & 24.64 (22.89) & +2.63 (+3.27) & +4.93 (+5.66) & +4.82 (+5.66) \\ \hline +% Book Arrival & 28.29 (27.28) & +2.65 (+2.82) & +4.59 (+4.91) & +4.51 (+5.86) \\ \hline +% BQSquare & 28.06 (30.91) & +2.59 (+2.52) & +3.14 (+2.97) & +3.11 (+3.82) \\ \hline +% Four People & 33.07 (34.04) & +1.85 (+1.86) & +2.59 (+2.52) & +2.56 (+2.83) \\ \hline +% Kimono & 27.98 (29.54) & +0.14 (+0.14) & +0.19 (+0.19) & +0.19 (+0.50) \\ \hline +% Park Scene & 27.76 (26.82) & +1.32 (+1.53) & +1.93 (+2.42) & +1.90 (+2.81) \\ \hline +% People on Street & 22.53 (22.57) & +0.40 (+0.45) & +0.85 (+0.91) & +0.83 (+1.06) \\ \hline +% Race Horses & 21.55 (22.63) & +0.68 (+0.76) & +1.12 (+1.23) & +1.07 (+1.50) \\ \hline +% Tennis & 23.77 (24.71) & +1.41 (+1.13) & +4.34 (+3.79) & +4.19 (+3.84) \\ \hline +% \textbf{Average} & \textbf{--} & \textbf{+1.52 (+1.61)} & \textbf{+2.63 (+2.73)} & \textbf{+2.58 (+3.10)} \\ +% \bottomrule +% \end{tabular} +% % } +% \end{table} + + +\begin{sidewaystable} +\newcommand{\somespaces}{} +\centering +\caption{Average quality (\ac{psnr} and \ac{wpsnr}) of the affected frames for Low-Delay configuration.} +\label{TAB4:SignallingPSNR} +\TABLESVSPACE +\small +\footnotesize +\renewcommand*{\arraystretch}{1.8} +\newcommand{\colwidth}{1.3cm} +\newcommand{\propcolwidth}{2cm} +\begin{tabular}{p{3cm}|x{\colwidth}x{\colwidth}x{\colwidth}x{\propcolwidth}x{\propcolwidth}|x{\colwidth}x{\colwidth}x{\colwidth}x{\propcolwidth}x{\propcolwidth}} +\toprule +\multirow{3}{*}{\vbox{\textbf{Sequence}}} & \multicolumn{5}{c|}{\textbf{PLR=3\%}} & \multicolumn{5}{c}{\textbf{PLR=10\%}}\\ +& \REF & \MC & \DEEPFLOWLONG & \FIXEDLAMBDALONG & \DYNLAMBDALONG & \REF & \MC & \DEEPFLOWLONG & \FIXEDLAMBDALONG & \DYNLAMBDALONG \\ \hline +Basketball Drill & 28.41 (29.25) & +2.93 (+3.22) & +3.25 (+3.24) & +5.28 (+5.02) & +2.64 (+5.01) & 24.64 (22.89) & +2.63 (+3.27) & +3.09 (+3.25) & +4.93 (+5.66) & +4.82 (+5.66) \\ \hline +Book Arrival & 30.69 (33.53) & +2.31 (+1.56) & +3.11 (+1.48) & +3.80 (+2.61) & +3.73 (+3.17) & 28.29 (27.28) & +2.65 (+2.82) & +1.47 (+2.74) & +4.59 (+4.91) & +4.51 (+5.86) \\ \hline +BQSquare & 31.82 (35.39) & +1.97 (+1.59) & +1.23 (+0.90) & +2.21 (+1.75) & +2.19 (+2.29) & 28.06 (30.91) & +2.59 (+2.52) & +1.92 (+1.43) & +3.14 (+2.97) & +3.11 (+3.82) \\ \hline +Four People & 34.54 (37.03) & +1.46 (+0.75) & +1.87 (+0.73) & +1.91 (+0.96) & +1.89 (+1.14) & 33.07 (34.04) & +1.85 (+1.86) & +2.70 (+1.88) & +2.59 (+2.52) & +2.56 (+2.83) \\ \hline +Kimono & 32.39 (33.86) & +1.66 (+1.76) & +1.88 (+1.87) & +2.32 (+2.43) & +2.30 (+2.65) & 27.98 (29.54) & +0.14 (+0.14) & +0.29 (+0.13) & +0.19 (+0.19) & +0.19 (+0.50) \\ \hline +Park Scene & 31.66 (31.76) & +2.28 (+2.89) & +2.15 (+2.39) & +2.96 (+4.03) & +2.93 (+4.27) & 27.76 (26.82) & +1.32 (+1.53) & +1.25 (+1.31) & +1.93 (+2.42) & +1.90 (+2.81) \\ \hline +People on Street & 26.72 (27.24) & +1.59 (+1.78) & +2.15 (+2.36) & +2.96 (+3.21) & +2.88 (+3.35) & 22.53 (22.57) & +0.40 (+0.45) & +1.25 (+0.61) & +0.85 (+0.91) & +0.83 (+1.06) \\ \hline +Race Horses & 25.97 (27.81) & +2.55 (+2.73) & +2.59 (+2.57) & +3.80 (+3.98) & +3.69 (+4.29) & 21.55 (22.63) & +0.68 (+0.76) & +0.76 (+0.72) & +1.12 (+1.23) & +1.07 (+1.50) \\ \hline +Tennis & 28.47 (30.79) & +1.66 (+1.27) & +2.02 (+1.53) & +4.39 (+3.52) & +4.27 (+3.62) & 23.77 (24.71) & +1.41 (+1.13) & +1.82 (+1.45) & +4.34 (+3.79) & +4.19 (+3.84) \\ \hline +\textbf{Average} & \textbf{--} & \textbf{+2.04 (+1.95)} & \textbf{+2.25 (+1.90)} & \textbf{+3.29 (+3.06)} & \textbf{+2.95 (+3.31)} & \textbf{--} & \textbf{+1.52 (+1.61)} & \textbf{+1.62 (+1.50)} & \textbf{+2.63 (+2.73)} & \textbf{+2.58 (+3.10)} \\ +\bottomrule +\end{tabular} +\end{sidewaystable} + +\begin{sidewaystable} +\newcommand{\somespaces}{} +\centering +\caption{Average \ac{fsim} of the entire video sequence in frame loss conditions (scaled between $[0-100]$).} +\label{TAB4:SignallingFSIM} +\TABLESVSPACE +\small +\footnotesize +\renewcommand*{\arraystretch}{1.8} +\newcommand{\colwidth}{1.3cm} +\newcommand{\propcolwidth}{2cm} +\begin{tabular}{p{3cm}|x{\colwidth}x{\colwidth}x{\colwidth}x{\propcolwidth}x{\propcolwidth}|x{\colwidth}x{\colwidth}x{\colwidth}x{\propcolwidth}x{\propcolwidth}} +\toprule +\multirow{3}{*}{\vbox{\textbf{Sequence}}} & \multicolumn{5}{c|}{\textbf{PLR=3\%}} & \multicolumn{5}{c}{\textbf{PLR=10\%}}\\ +& \REF & \MC & \DEEPFLOWLONG & \FIXEDLAMBDALONG & \DYNLAMBDALONG & \REF & \MC & \DEEPFLOWLONG & \FIXEDLAMBDALONG & \DYNLAMBDALONG \\ \hline +Basketball Drill & 96.96 & +1.13 & +1.16 & +1.58 & +1.59 & 92.74 & +2.38 & +2.50 & +3.60 & +3.65 \\ \hline +Book Arrival & 98.61 & +0.36 & +0.41 & +0.57 & +0.60 & 96.66 & +1.05 & +1.21 & +1.74 & +1.75 \\ \hline +BQSquare & 98.03 & +0.39 & +0.28 & +0.41 & +0.41 & 96.31 & +1.15 & +0.81 & +1.30 & +1.31 \\ \hline +Four People & 99.41 & +0.09 & +0.09 & +0.11 & +0.11 & 98.97 & +0.31 & +0.32 & +0.39 & +0.60 \\ \hline +Kimono & 98.54 & +0.48 & +0.52 & +0.59 & +0.62 & 95.79 & +0.09 & +0.12 & +0.12 & +0.15 \\ \hline +Park Scene & 98.74 & +0.58 & +0.55 & +0.69 & +0.71 & 96.41 & +0.83 & +0.79 & +1.03 & +1.10 \\ \hline +People on Street & 98.19 & +0.67 & +0.73 & +0.92 & +0.94 & 94.65 & +0.45 & +0.51 & +0.66 & +0.70 \\ \hline +Race Horses & 95.16 & +1.78 & +1.80 & +2.28 & +2.29 & 88.25 & +1.18 & +1.21 & +1.62 & +1.68 \\ \hline +Tennis & 96.64 & +0.98 & +0.98 & +1.69 & +1.64 & 91.02 & +2.20 & +2.29 & +4.27 & +4.32 \\ \hline +\textbf{Average} & \textbf{--} & \textbf{+0.72} & \textbf{+0.73} & \textbf{+0.98} & \textbf{+0.99} & \textbf{--} & \textbf{+1.07} & \textbf{+1.09} & \textbf{+1.64} & \textbf{+1.70} \\ +\bottomrule +\end{tabular} +\end{sidewaystable} + + +\begin{table}[tb] +\newcommand{\somespaces}{} +\centering +\caption{Error concealment signalling overhead.}\label{tab:redundancy_signalling} +\TABLESVSPACE +\footnotesize +\begin{tabular}{p{3.6cm}|x{3.8cm}|x{3.8cm}} +\toprule +\multirow{3}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{2}{c}{\textbf{Bits used for signalling (\%)}} \\ +& \FIXEDLAMBDALONG & \DYNLAMBDALONG \\ +\hline +Basketball Drill & 2.88 & 2.55 \\ \hline +Book Arrival & 4.03 & 3.35 \\ \hline +BQSquare & 0.33 & 0.24 \\ \hline +Four People & 3.06 & 3.18 \\ \hline +Kimono & 1.75 & 1.30 \\ \hline +Park Scene & 1.84 & 1.37 \\ \hline +People on Street & 8.39 & 5.67 \\ \hline +Race Horses & 1.84 & 1.21 \\ \hline +Tennis & 12.15 & 7.46 \\ \hline +\textbf{Average} & \textbf{+4.04} & \textbf{+2.91 (-28\%)} \\ +\bottomrule +\end{tabular} +\end{table} diff --git a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex new file mode 100644 index 0000000..97945db --- /dev/null +++ b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex @@ -0,0 +1,72 @@ +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% CHAPTER +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\chapter{\HIGHLIGHT{Joint error resilience-concealment for robust video streaming}} +\label{Chapter:EndToEndRobustSystem} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Introduction} + + +In this chapter the error resilience and error concealment approaches proposed in the previous chapters, namely Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, are combined to achieved an higher decoder video quality across different network conditions and applications. +This chapter presents a combined evaluation of different approaches and proposes a method to efficiently select the optimal one in terms of reconstructed quality. + +This chapter is organised as follows. Section~\ref{Section6:Problem} presents the problem formulation and motivation for the method proposed in this chapter. \HIGHLIGHT{...} +Section~\ref{Section6:Summary} concludes this chapter. + +\clearpage +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Problem formulation} +\label{Section6:Problem} + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Performance evaluation under different network condition} + +\subsection{Test conditions} + +\subsection{Experimental results} + +\subsection{Discussion and conclusions} + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Proposed architecture} +\label{Section6:ProposedMethod} + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Error robustness optimisation} + +\section{Optimisation method} + +\subsection{Performance evaluation} + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Error concealment optimisation based on MV differences} + + + +\subsection{Experimental results} + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% SECTION %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Summary} +\label{Section6:Summary} \ No newline at end of file diff --git a/043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz b/043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz new file mode 100644 index 0000000..2f61cc1 --- /dev/null +++ b/043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz @@ -0,0 +1,95 @@ + +\usetikzlibrary{arrows,matrix} +\usetikzlibrary{fit} +\usetikzlibrary{intersections} +\usetikzlibrary{intersections} +% \usetkzobj{all} + +\pgfdeclarelayer{background} +\pgfdeclarelayer{foreground} +\pgfsetlayers{background,main,foreground} + + +\def\scale{1} + + +\begin{tikzpicture}[ + font=\scriptsize, + auto, + node distance=1cm, + scale=\scale, + every node/.style={scale=\scale}, + template/.style={thick, text centered}, + block/.style={template, rectangle, text width=1.3cm, ,inner sep=3, minimum height=.6cm}, + arrow/.style={template, draw, -latex}, + point/.style={template, draw, fill=black, circle,inner sep=0, minimum size=3} +] + +% Define extra block styles +\tikzstyle{sumBlock} = [template, draw, circle, inner sep=0cm, minimum size=0.3cm] +\tikzstyle{propBlock} = [block, draw, fill=blue!15] +\tikzstyle{refBlock} = [block, draw, fill=gray!10] +\tikzstyle{propBlockNoDraw} = [block, fill=\TikZColorFillProp] +\tikzstyle{refBlockNoDraw} = [block, fill=\TikZColorFill] + + +\newcommand{\DrawEdgesTwoNodes}[2]{\path[arrow,-] (#1.south west) |- (#2.north east) |- (#1.south west);} + +%% Encoder/Decoder +\node [refBlockNoDraw, text width=1.1cm, minimum height=.6cm] (Encoder) at (0,0) {Encoder}; +\node [refBlockNoDraw, text width=0.6cm, minimum height=.6cm, right=-0.03 of Encoder.south east, anchor=south west, text centered] (EncoderNAL) {NAL}; +\node [refBlockNoDraw, text width=1.1cm, minimum height=.6cm, above=-0.03 of Encoder.north west, anchor=south west] (Decoder) {Decoder}; +\path[arrow,-] (Encoder.south west) |- (Decoder.north east) |- (EncoderNAL.north east) |- (Encoder.south west); +\path[arrow,-,densely dotted] (EncoderNAL.south west -| Decoder.south east) -- (Decoder.south east); % Draw inside lines +\path[arrow,-,densely dotted] (Decoder.south west |- EncoderNAL.north west) -- (EncoderNAL.north west); % Draw inside lines + + +%% NALU Loss and EC optimisation +\node [propBlock, above=2.2 of Encoder.west, anchor=south west, text width=3.8cm, minimum height=2.4cm] (ConcOpt) {Saliency-weighted EC Mode Decision}; +\node [propBlock, below=.8 of ConcOpt.south east, anchor=north east, text width=1.7cm, minimum height=.9cm] (LossSim) {NALU Loss Simulation}; +\path[arrow] (EncoderNAL -| LossSim.south) -- (LossSim) node[point] at (EncoderNAL -| LossSim) {};; +\path[arrow] (Decoder) -- (ConcOpt.south -| Decoder) node[pos=.6,coordinate] (p2) {} node[point] at (p2) {}; +\path[arrow] (LossSim) -- (ConcOpt.south -| LossSim); + + +%% EL encoder +\node [propBlockNoDraw, right=4cm of ConcOpt.south, anchor=south west, text width=1.4cm, minimum height=1cm] (Scalable) {EL\\Encoder}; +\node [refBlockNoDraw, text width=.2cm, right=-.025 of Scalable.south east, anchor=south west, text centered, minimum height=1cm] (ScalableNAL) {N\\[-2pt]A\\[-2pt]L}; +\DrawEdgesTwoNodes{Scalable}{ScalableNAL}; % Draw border +\path[arrow,-,densely dotted] (ScalableNAL.south west) -- (ScalableNAL.north west); % Draw inside lines + + +%% Diff +\node [sumBlock, left=.75 of Scalable] (ConcDiff) {$-$}; +\path[arrow] (ConcDiff) -- (Scalable) node[above, pos=0, xshift=.2cm] {$e^c$}; +\path[arrow] (ConcOpt.east |- ConcDiff) -- (ConcDiff) node[above, pos=1, xshift=-.3cm] {$\tilde f$}; +\path[arrow,-, shorten >=3] (p2) -- (p2 -| LossSim.north); +\path[arrow, shorten <=3] (p2 -| LossSim.north) -| (ConcDiff) node[right, pos=1, yshift=-.3cm] {$\hat f$}; + + +%% EC mode coding +\node [propBlockNoDraw, anchor=north west, text width=1.4cm, minimum height=1cm] at (ConcOpt.north -| Scalable.west) (Signalling) {EC Mode Coding}; +\node [refBlockNoDraw, text width=.2cm, right=-.025 of Signalling.south east, anchor=south west, text centered, minimum height=1cm] (SignallingNAL) {N\\[-2pt]A\\[-2pt]L}; +\DrawEdgesTwoNodes{Signalling}{SignallingNAL}; % Draw border +\path[arrow,-,densely dotted] (SignallingNAL.south west) -- (SignallingNAL.north west); % Draw inside lines +\path[arrow] (ConcOpt.east |- Signalling) -- (Signalling) node[above, pos=.45, yshift=-1] {\scriptsize M1...M4}; + +%% Input +\node[text width=.5cm, text centered] at (-1.7,4.5) (Input) {Input video}; +\path[arrow] (Input) |- (Encoder); +\path[arrow] (Input |- ConcOpt) -- (ConcOpt) node[point] at (Input |- ConcOpt) {}; + +%% TS Mux and output +\path (Encoder.south) -- (Encoder |- Signalling.north) node[midway] (middleRef) {}; % Find middle ref +\node [refBlock, right=8.5 of middleRef, text width=.7cm, minimum height=5.5cm] (NAL) {TS Mux}; +\node [right of=NAL, node distance=1.5cm, coordinate] (Out) {}; + +\path[arrow] (EncoderNAL) -- (NAL.west |- EncoderNAL) node[above,pos=1, xshift=-.4cm] {VCL$_0$}; +\path[arrow] (ScalableNAL) -- (NAL.west |- Scalable) node[above,pos=1, xshift=-.4cm] {VCL$_1$}; +\path[arrow] (SignallingNAL) -- (NAL.west |- Signalling) node[above,pos=1, xshift=-.4cm] {SEI}; +\path[arrow] (NAL) -- (Out) node[above,pos=1, xshift=-.2cm, text width=1.3cm] {Transport Stream}; + + +\end{tikzpicture} + + diff --git a/050_Conclusions/Conclusion.tex b/050_Conclusions/Conclusion.tex new file mode 100755 index 0000000..83060d8 --- /dev/null +++ b/050_Conclusions/Conclusion.tex @@ -0,0 +1,37 @@ + +\chapter{\HIGHLIGHT{Conclusion and future work}} +\label{Chapter:Conclusions} + +This chapter concludes this thesis, presenting some conclusions about this research work and some future research perspectives in the area of video transmission, error resilience and error concealment. + +%\vspace{10mm} +\section{Conclusions} + +% This section presents a summary of the of the chapters presented before and a discussion of the contributions presented. +% +% Chapter~\ref{CHAP:3DVideo} presented a review on the main technologies for 3D video systems related with the formats for representation, coding methods and proposed storage and transmission systems. This chapter also describes the main features of the MVC extension of the H.264/MVC standard, which is the coding standard for the most 3D applications \mbox{(\emph{e.g.}, Blu-ray)}. Moreover, as this research focuses in transmission errors and error concealment, a review of some concealment methods proposed over the last years was presented in the Chapter~\ref{CHAP:Concealment}. +% +% The research work presented in this thesis leaded +% %different contributions +% to the development of an advanced 3D video transmission system. More specifically, our research focused on the study of the subjective impacts of transmission errors in 3D video, when the auxiliary view is subject to frame loss. These results were presented in Chapter~\ref{CHAP:SubjectiveEval}. The results demonstrated that the disparity is intrinsically related with the quality of the concealed 3D video, and that errors have higher effects in those sequences with higher disparity. Moreover, for longer error bursts, the results showed that it is better to switch into 2D visualisation rather than trying to restore the 3D effect by using frame concealment methods. +% +% The results presented in the Chapter~\ref{CHAP:SubjectiveEval} leaded to further experiments in order to evaluate the objective quality of the transmitted 3D video using different objective metrics. A transmission framework was presented in Chapter~\ref{CHAP:HierarchicalDVBT3D} and the quality evaluation was performed. The quality models, devised from the analysis of the quality results, are a novel contribution to the research community and also for the network planners. Through those models, estimated values of the quality using different metrics can be obtained from the packet loss ratio of the transmission channel. This can be easily measured at the receiver side. The quality models are able to cover different video quality features, as the signal-to-noise relation, the structural similarity and a more specific feature of the 3D video, the stereo perception. +% The tools developed are part of the first contribution of this research work, which resulted in an efficient 3D receiver to cope with transmission errors in 3D video streams, delivered using the MPEG-2 TS encapsulation. Moreover, as part of the receiver side, an efficient 3D video decoder was achieved, that is able to detect and conceal the missing frames using different concealment techniques. This set of tools provides relevant knowledge, for further investigations in quality evaluation and error concealment. +% +% The second contribution of this research work was the development of efficient concealment methods, able to recover the motion field for missing frames in 3D video. Assuming the hierarchical transmission system, as presented in Chapter~\ref{CHAP:HierarchicalDVBT3D}, the proposed methods are an efficient alternative to the existing ones. They are mainly based on the information presented in the error-free view (\emph{i.e.}, the base view), since these information can provide more accurate clues to estimate the loss data in the auxiliary view, as described in Chapter~\ref{CHAP:PropFrameLossConc}. The presented results showed the effectiveness of the proposed techniques, when compared with conventional motion-based algorithms, achieving an average gain of $1.89\ dB$ in random frame loss and up to $6\ dB$ of gain in the objective quality of the reconstructed frame. Moreover, an improvement to the proposed method was devised, and a motion based decision approach was incorporated, combining both methods, thus allowing to improve the quality of the reconstructed frames up $2\ dB$. +% +% One may conclude that the research on alternative frame loss concealment techniques may bring new insights to achieve a more efficiently video decoding in the presence of error prone channels. Moreover, the subjective study presented in this dissertation provided relevant subjective factors that may be used in future developments of error concealment techniques. New representation formats for 3D video and novel coding approaches are being proposed, leading to new error concealment problems, and different ways to recovery the missing information, thus, the research work in this area should continue. + + +\section{Future work} + +% In this research work, the technologies and the subjective quality impact of errors in 3D video were studied. Nevertheless, there are still a number of topics that require further investigation, and may lead to more efficient 3D video decoders, namely: +% \begin{itemize} +% \item \textbf{3D video quality models extension to the channel level.} The 3D video quality models presented in Chapter~\ref{CHAP:HierarchicalDVBT3D} revealed to match with the experimental data. Thus, the combination with the hierarchical DVB-T channel models could extend the proposed models to a new level, where the quality of the 3D video could be estimated from the channel conditions, instead of the packet loss. This would extend the relevance of the proposed models to the network planners and developers. +% +% \item \textbf{Extension of the proposed method to a multi-view scenario.} The proposed concealment methods, described in Chapter~\ref{CHAP:PropFrameLossConc}, revealed good performance to conceal missing frames in 3D video (\emph{i.e.}, stereoscopic video). Therefore, the same techniques are expected to have similar behaviour under a multi-view scenario, where, besides the base view frames, other auxiliary view frames that may provide motion information clues to estimate the lost frames' motion field. +% +% \item \textbf{Extend the concealment algorithm to MVD coding scenario.} Although the proposed concealment methods did not focus in the recovery of the depth information, they could be used to recover texture in multi-view plus depth video signals. In this case, besides disparity information, depth maps are available to extract relevant information, for example, to detect occlusions and errors in the disparity map. The performance of the proposed method could also be investigated to reconstruct the lost depth map. +% +% \item \textbf{Performance evaluation of the concealment methods in a MDC.} The multi-description coding approach is being investigated over the last years, as a novel alternative for the emergent multi-path networks. The algorithms presented in the proposed method have great flexibility and could be applied under different coding scenarios. Moreover, since MDC introduces new possibilities to error concealment by using the redundant information in each description, such characteristic will be used to investigate how different types of redundancy can be jointly used with the proposed concealment techniques to enhance the quality of received views. +% \end{itemize} diff --git a/060_Appendix/Papers.tex b/060_Appendix/Papers.tex new file mode 100755 index 0000000..4a37523 --- /dev/null +++ b/060_Appendix/Papers.tex @@ -0,0 +1,44 @@ + +\chapter{Published papers} +\label{Appen:Papers} + +This appendix presents a list of the published papers that resulted from the research work done during this thesis proposal. +Submitted papers are also presented. + +\section{Journal publications} +\begin{enumerate} +\item J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, +``A two-stage approach for robust \ac{hevc} coding and streaming,'' \emph{IEEE + Transactions on Circuits and Systems for Video Technology}, Apr. 2017. + +\item J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, +``Error Concealment-Aware Encoding for Robust Video Transmission,'' \emph{IEEE + Transactions on Broadcasting}, \textbf{(Submitted)}. +\end{enumerate} +\section{Conference publications} +\begin{enumerate} +\item +J.~Carreira, V.~D. Silva, E.~Ekmekcioglu, A.~Kondoz, P.~Assuncao, and S.~Faria, + ``Dynamic motion vector refreshing for enhanced error resilience in HEVC,'' + in \emph{22nd European Signal Processing Conference (EUSIPCO)}, Sep. 2014, + pp. 281--285. +\item +J.~Carreira, E.~Ekmekcioglu, A.~Kondoz, P.~Assuncao, S.~Faria, and V.~D. Silva, + ``Selective motion vector redundancies for improved error resilience in + {HEVC},'' in \emph{IEEE International Conference on Image Processing (ICIP)}, + Oct. 2014, pp. 2457--2461. +\item +J.~Carreira, S.~Faria, P.~Assuncao, E.~Ekmekcioglu, and A.~Kondoz, ``Error + resilience analysis of motion vector prediction in HEVC,'' in + \emph{Conference on Telecommunications (Conftele)}, Oct. 2015, pp. 1--4. +\item +J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, A.~Kondoz, and H.Lim, + ``Reference picture selection using checkerboard pattern for resilient video + coding,'' in \emph{IEEE International Conference on Visual Communications and + Image Processing (VCIP)}, Dec. 2015, pp. 1--5. +\item +J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, ``A robust + video encoding scheme to enhance error concealment of intra frames,'' in + \emph{IEEE International Conference on Circuits and Systems (ISCAS)}, May + 2017, pp. 1--5. +\end{enumerate} diff --git a/060_Appendix/Results.tex b/060_Appendix/Results.tex new file mode 100755 index 0000000..c3fff89 --- /dev/null +++ b/060_Appendix/Results.tex @@ -0,0 +1,21 @@ + +\chapter{Complementary Results} +\label{Appen:Results} + +This appendix presents complementary results for some of the figures presented throughout this thesis. + + +\begin{figure}[!b] +\centering +\input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsIntra.tex} +\caption{Complementary results for Figure~\ref{Figure3:QualityvsIntra}.} +\label{Appendix:Figure3:QualityvsIntra} +\end{figure} + +\begin{figure}[tbp] +\centering +\input{./031_ErrorResilienceEvaluation/QualityEvaluation/PSNRvsSlices.tex} +\caption{Complementary results for Figure~\ref{Figure3:SlicePartitioning}.} +\label{Appendix:Figure3:SlicePartitioning} +\end{figure} + diff --git a/060_Appendix/Sequences.tex b/060_Appendix/Sequences.tex new file mode 100755 index 0000000..f3f8822 --- /dev/null +++ b/060_Appendix/Sequences.tex @@ -0,0 +1,96 @@ + +\newcommand{\AppenAFigs}{./060_Appendix/Sequences} +\newcommand{\SeuqenceSize}{0.35\textheight} + + +\chapter{Test video sequences} +\label{Appen:Sequences} + +This appendix illustrates the original test signals used in simulations throughout the thesis. This includes a partial set of the \ac{hevc} test material and other external test material. + + +\begin{figure}[!b] +\centering +\includegraphics[width=\SeuqenceSize]{\AppenAFigs/BasketballDrill_832x480_50_FRAME_100.jpg} +\caption{Basketball Drill, $832\times480$, 50 fps, 240 frames.} +\vspace{5mm} +\includegraphics[width=\SeuqenceSize]{\AppenAFigs/BookArrival_1024x768_30_FRAME_100.jpg} +\caption{Book Arrival, $1024\times768$, 30 fps, 240 frames.} +\end{figure} + +% \begin{figure}[tbp] +% \centering +% +% \end{figure} + +\begin{figure}[tbp] +\centering +\includegraphics[width=\SeuqenceSize]{\AppenAFigs/BQSquare_416x240_60_FRAME_100.jpg} +\caption{BQSquare, $416\times240$, 60 fps, 240 frames.} +\end{figure} + +\begin{figure}[tbp] +\centering +\includegraphics[width=\SeuqenceSize]{\AppenAFigs/Bosphorus_3840x2160_120_FRAME_100.jpg} +\caption{Bosphorus, $3840\times2160$, 120 fps, 240 frames.} +\end{figure} + +\begin{figure}[tbp] +\centering +\includegraphics[width=\SeuqenceSize]{\AppenAFigs/Cactus_1920x1080_50_FRAME_100.jpg} +\caption{Cactus, $1920\times1080$, 50 fps, 500 frames.} +\end{figure} + +\begin{figure}[tbp] +\centering +\includegraphics[width=\SeuqenceSize]{\AppenAFigs/FourPeople_1280x720_60_FRAME_100.jpg} +\caption{Four People, $1280\times720$, 60 fps, 240 frames.} +\end{figure} + +\begin{figure}[tbp] +\centering +\includegraphics[width=\SeuqenceSize]{\AppenAFigs/Jockey_3840x2160_120_FRAME_100.jpg} +\caption{Jockey, $3840\times2160$, 120 fps, 240 frames.} +\end{figure} + +\begin{figure}[tbp] +\centering +\includegraphics[width=\SeuqenceSize]{\AppenAFigs/Kendo_1024x768_30_FRAME_100.jpg} +\caption{Kendo, $1024\times768$, 30 fps, 240 frames.} +\end{figure} + +\begin{figure}[tbp] +\centering +\includegraphics[width=\SeuqenceSize]{\AppenAFigs/Kimono1_1920x1080_24_FRAME_100.jpg} +\caption{Kimono, $1920\times1080$, 24 fps, 240 frames.} +\end{figure} + +\begin{figure}[tbp] +\centering +\includegraphics[width=\SeuqenceSize]{\AppenAFigs/ParkScene_1920x1080_24_FRAME_100.jpg} +\caption{Park Scene, $1920\times1080$, 24 fps, 240 frames.} +\end{figure} + +\begin{figure}[tbp] +\centering +\includegraphics[width=\SeuqenceSize]{\AppenAFigs/PeopleOnStreet_2560x1600_30_FRAME_100.jpg} +\caption{People On Street, $2560\times1600$, 24 fps, 150 frames.} +\end{figure} + +\begin{figure}[tbp] +\centering +\includegraphics[width=\SeuqenceSize]{\AppenAFigs/RaceHorses_832x480_30_FRAME_100.jpg} +\caption{Race Horses, $832\times480$, 30 fps, 240 frames.} +\end{figure} + +% \begin{figure}[tbp] +% \centering +% \includegraphics[width=\SeuqenceSize]{\AppenAFigs/Tennis_1920x1080_24_FRAME_100.jpg} +% \caption{Tennis, $1920\times1080$, 30 fps, 240 frames.} +% \end{figure} + +\begin{figure}[tbp] +\centering +\includegraphics[width=\SeuqenceSize]{\AppenAFigs/Traffic_2560x1600_30_FRAME_100.jpg} +\caption{Traffic, $2560\times1600$, 30 fps, 150 frames.} +\end{figure} diff --git a/060_Appendix/Sequences/BQSquare_416x240_60_FRAME_0.jpg b/060_Appendix/Sequences/BQSquare_416x240_60_FRAME_0.jpg new file mode 100644 index 0000000..d570c25 Binary files /dev/null and b/060_Appendix/Sequences/BQSquare_416x240_60_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/BQSquare_416x240_60_FRAME_100.jpg b/060_Appendix/Sequences/BQSquare_416x240_60_FRAME_100.jpg new file mode 100644 index 0000000..4b85698 Binary files /dev/null and b/060_Appendix/Sequences/BQSquare_416x240_60_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/BasketballDrill_832x480_50_FRAME_0.jpg b/060_Appendix/Sequences/BasketballDrill_832x480_50_FRAME_0.jpg new file mode 100644 index 0000000..42b7a74 Binary files /dev/null and b/060_Appendix/Sequences/BasketballDrill_832x480_50_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/BasketballDrill_832x480_50_FRAME_100.jpg b/060_Appendix/Sequences/BasketballDrill_832x480_50_FRAME_100.jpg new file mode 100644 index 0000000..e205293 Binary files /dev/null and b/060_Appendix/Sequences/BasketballDrill_832x480_50_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/BasketballDrive_1920x1080_50_FRAME_0.jpg b/060_Appendix/Sequences/BasketballDrive_1920x1080_50_FRAME_0.jpg new file mode 100644 index 0000000..e77a100 Binary files /dev/null and b/060_Appendix/Sequences/BasketballDrive_1920x1080_50_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/BasketballDrive_1920x1080_50_FRAME_100.jpg b/060_Appendix/Sequences/BasketballDrive_1920x1080_50_FRAME_100.jpg new file mode 100644 index 0000000..15ce8c4 Binary files /dev/null and b/060_Appendix/Sequences/BasketballDrive_1920x1080_50_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/BookArrival_1024x768_30_FRAME_0.jpg b/060_Appendix/Sequences/BookArrival_1024x768_30_FRAME_0.jpg new file mode 100644 index 0000000..ed018eb Binary files /dev/null and b/060_Appendix/Sequences/BookArrival_1024x768_30_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/BookArrival_1024x768_30_FRAME_100.jpg b/060_Appendix/Sequences/BookArrival_1024x768_30_FRAME_100.jpg new file mode 100644 index 0000000..614c11a Binary files /dev/null and b/060_Appendix/Sequences/BookArrival_1024x768_30_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/Bosphorus_3840x2160_120_FRAME_0.jpg b/060_Appendix/Sequences/Bosphorus_3840x2160_120_FRAME_0.jpg new file mode 100644 index 0000000..20bcf4b Binary files /dev/null and b/060_Appendix/Sequences/Bosphorus_3840x2160_120_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/Bosphorus_3840x2160_120_FRAME_100.jpg b/060_Appendix/Sequences/Bosphorus_3840x2160_120_FRAME_100.jpg new file mode 100644 index 0000000..9821236 Binary files /dev/null and b/060_Appendix/Sequences/Bosphorus_3840x2160_120_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/Cactus_1920x1080_50_FRAME_0.jpg b/060_Appendix/Sequences/Cactus_1920x1080_50_FRAME_0.jpg new file mode 100644 index 0000000..73071ec Binary files /dev/null and b/060_Appendix/Sequences/Cactus_1920x1080_50_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/Cactus_1920x1080_50_FRAME_100.jpg b/060_Appendix/Sequences/Cactus_1920x1080_50_FRAME_100.jpg new file mode 100644 index 0000000..564e2ac Binary files /dev/null and b/060_Appendix/Sequences/Cactus_1920x1080_50_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/FourPeople_1280x720_60_FRAME_0.jpg b/060_Appendix/Sequences/FourPeople_1280x720_60_FRAME_0.jpg new file mode 100644 index 0000000..5b82b03 Binary files /dev/null and b/060_Appendix/Sequences/FourPeople_1280x720_60_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/FourPeople_1280x720_60_FRAME_100.jpg b/060_Appendix/Sequences/FourPeople_1280x720_60_FRAME_100.jpg new file mode 100644 index 0000000..1760ba3 Binary files /dev/null and b/060_Appendix/Sequences/FourPeople_1280x720_60_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/Jockey_3840x2160_120_FRAME_0.jpg b/060_Appendix/Sequences/Jockey_3840x2160_120_FRAME_0.jpg new file mode 100644 index 0000000..2269d98 Binary files /dev/null and b/060_Appendix/Sequences/Jockey_3840x2160_120_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/Jockey_3840x2160_120_FRAME_100.jpg b/060_Appendix/Sequences/Jockey_3840x2160_120_FRAME_100.jpg new file mode 100644 index 0000000..ca6897a Binary files /dev/null and b/060_Appendix/Sequences/Jockey_3840x2160_120_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/Kendo_1024x768_30_FRAME_0.jpg b/060_Appendix/Sequences/Kendo_1024x768_30_FRAME_0.jpg new file mode 100644 index 0000000..c0a984f Binary files /dev/null and b/060_Appendix/Sequences/Kendo_1024x768_30_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/Kendo_1024x768_30_FRAME_100.jpg b/060_Appendix/Sequences/Kendo_1024x768_30_FRAME_100.jpg new file mode 100644 index 0000000..e1ec088 Binary files /dev/null and b/060_Appendix/Sequences/Kendo_1024x768_30_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/Kimono1_1920x1080_24_FRAME_0.jpg b/060_Appendix/Sequences/Kimono1_1920x1080_24_FRAME_0.jpg new file mode 100644 index 0000000..7f308b6 Binary files /dev/null and b/060_Appendix/Sequences/Kimono1_1920x1080_24_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/Kimono1_1920x1080_24_FRAME_100.jpg b/060_Appendix/Sequences/Kimono1_1920x1080_24_FRAME_100.jpg new file mode 100644 index 0000000..5f8b502 Binary files /dev/null and b/060_Appendix/Sequences/Kimono1_1920x1080_24_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/KristenAndSara_1280x720_60_FRAME_0.jpg b/060_Appendix/Sequences/KristenAndSara_1280x720_60_FRAME_0.jpg new file mode 100644 index 0000000..1c12103 Binary files /dev/null and b/060_Appendix/Sequences/KristenAndSara_1280x720_60_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/KristenAndSara_1280x720_60_FRAME_100.jpg b/060_Appendix/Sequences/KristenAndSara_1280x720_60_FRAME_100.jpg new file mode 100644 index 0000000..9c1da2a Binary files /dev/null and b/060_Appendix/Sequences/KristenAndSara_1280x720_60_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/ParkScene_1920x1080_24_FRAME_0.jpg b/060_Appendix/Sequences/ParkScene_1920x1080_24_FRAME_0.jpg new file mode 100644 index 0000000..e19f62f Binary files /dev/null and b/060_Appendix/Sequences/ParkScene_1920x1080_24_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/ParkScene_1920x1080_24_FRAME_100.jpg b/060_Appendix/Sequences/ParkScene_1920x1080_24_FRAME_100.jpg new file mode 100644 index 0000000..9cd6f94 Binary files /dev/null and b/060_Appendix/Sequences/ParkScene_1920x1080_24_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/PeopleOnStreet_2560x1600_30_FRAME_0.jpg b/060_Appendix/Sequences/PeopleOnStreet_2560x1600_30_FRAME_0.jpg new file mode 100644 index 0000000..dda91fd Binary files /dev/null and b/060_Appendix/Sequences/PeopleOnStreet_2560x1600_30_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/PeopleOnStreet_2560x1600_30_FRAME_100.jpg b/060_Appendix/Sequences/PeopleOnStreet_2560x1600_30_FRAME_100.jpg new file mode 100644 index 0000000..cbb2310 Binary files /dev/null and b/060_Appendix/Sequences/PeopleOnStreet_2560x1600_30_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/RaceHorses_832x480_30_FRAME_0.jpg b/060_Appendix/Sequences/RaceHorses_832x480_30_FRAME_0.jpg new file mode 100644 index 0000000..28ad51f Binary files /dev/null and b/060_Appendix/Sequences/RaceHorses_832x480_30_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/RaceHorses_832x480_30_FRAME_100.jpg b/060_Appendix/Sequences/RaceHorses_832x480_30_FRAME_100.jpg new file mode 100644 index 0000000..f2a8c58 Binary files /dev/null and b/060_Appendix/Sequences/RaceHorses_832x480_30_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/Tennis_1920x1080_24_FRAME_0.jpg b/060_Appendix/Sequences/Tennis_1920x1080_24_FRAME_0.jpg new file mode 100644 index 0000000..de98582 Binary files /dev/null and b/060_Appendix/Sequences/Tennis_1920x1080_24_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/Tennis_1920x1080_24_FRAME_100.jpg b/060_Appendix/Sequences/Tennis_1920x1080_24_FRAME_100.jpg new file mode 100644 index 0000000..6c6b7f0 Binary files /dev/null and b/060_Appendix/Sequences/Tennis_1920x1080_24_FRAME_100.jpg differ diff --git a/060_Appendix/Sequences/Traffic_2560x1600_30_FRAME_0.jpg b/060_Appendix/Sequences/Traffic_2560x1600_30_FRAME_0.jpg new file mode 100644 index 0000000..4f8577e Binary files /dev/null and b/060_Appendix/Sequences/Traffic_2560x1600_30_FRAME_0.jpg differ diff --git a/060_Appendix/Sequences/Traffic_2560x1600_30_FRAME_100.jpg b/060_Appendix/Sequences/Traffic_2560x1600_30_FRAME_100.jpg new file mode 100644 index 0000000..4ae0541 Binary files /dev/null and b/060_Appendix/Sequences/Traffic_2560x1600_30_FRAME_100.jpg differ diff --git a/060_Appendix/pub.bib b/060_Appendix/pub.bib new file mode 100755 index 0000000..8c305bf --- /dev/null +++ b/060_Appendix/pub.bib @@ -0,0 +1,163 @@ +% Encoding: ISO-8859-1 + +@INPROCEEDINGS{PCS2010, +author={Carreira, J. and Pinto, L. and Rodrigues, N. and Faria, S. and Assuncao, P.}, +booktitle={Picture Coding Symposium (PCS)}, +title={{Subjective assessment of frame loss concealment methods in 3D video}}, +year={2010}, +month=dec, +volume={}, +number={}, +pages={182-185}, +adress={Nagoya, Japan}, +abstract={This paper investigates the subjective impact resulting from different concealment methods for coping +with lost frames in 3D video communication systems. It is assumed that a high priority channel is assigned to +the main view and only the auxiliary view is subject to either transmission errors or packet loss, leading to +missing frames at the decoder output. Three methods are used for frame concealment under different loss ratios. +The results show that depth is well perceived by users and the subjective impact of frame loss not only depends on +the concealment method but also exhibits high correlation with the disparity of the original sequence. +It is also shown that under heavy loss conditions it is better to switch from 3D to 2D rather than presenting +concealed 3D video to users.}, +keywords={3D video communication systems;decoder output;depth perception;frame loss concealment methods; +loss ratios;packet loss;transmission errors;video coding;video communication;}, +doi={10.1109/PCS.2010.5702455}, +ISSN={},} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +@INPROCEEDINGS{QoMEX2011, +author={Pinto, L. and Carreira, J. and Faria, S. and Rodrigues, N. and Assuncao, P.}, +booktitle={Third International Workshop on Quality of Multimedia Experience (QoMEX)}, +title={{Subjective quality factors in packet 3D video}}, +year={2011}, +month=sep, +volume={}, +number={}, +pages={149-154}, +adress={Mechelen, Belgium}, +abstract={This paper presents an experimental study on subjective quality of 3D video under packet loss conditions. +A priority network is assumed, such that the base view of a stereoscopic video stream is delivered through a +guaranteed channel and the auxiliary view is subject to packet losses over a low priority channel. +Due to the packetisation scheme, one frame is lost whenever a packet is lost. Three frame concealment methods +are proposed to deal with frame losses and seven different 3DTV sequences are used. The subjective evaluation +study was carried out for different packet loss ratios and concealment methods, using shutter glasses with +synchronized display. The experimental results show that sequence disparity, type of loss concealment and packet +loss pattern should be jointly considered as relevant quality factors in 3D video over packet networks.}, +keywords={3D video quality;3DTV;packet 3D video;packet networks;stereoscopic video stream; +subjective quality factors;three-dimensional television;}, +doi={10.1109/QoMEX.2011.6065694}, +ISSN={},} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +@INPROCEEDINGS{3DTVCON2012, +author={Carreira, J. and Assuncao, P. and Rodrigues, N. and Faria, S.}, +booktitle={3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON)}, +title={{Frame loss concealment for 3D video decoders based on disparity-compensated motion field}}, +year={2012}, +month=oct, +volume={}, +number={}, +pages={1-4}, +adress={Zurich, Switzerland}, +abstract={This paper addresses the problem of frame loss concealment in 3D video decoders capable of handling +stereoscopic views compliant with H.264/MVC. A joint motion-disparity compensation method is proposed +to fully recover an estimated motion field for the lost frame in stereoscopic video, +by using a combination of inter-view disparity-compensated motion vectors and intra-view motion extrapolation. +The overall motion field is estimated from both the motion information of the co-located view, +through disparity compensation, and neighboring frames of the same view, through motion vector extrapolation. +The results show that the proposed method outperforms currently used methods such as frame-copy and motion-copy. +In video sequences, average PSNR gains up to 1.6 dB are obtained over motion-copy, while for individual +frames a maximum of 5 dB is achieved.}, +keywords={}, +doi={10.1109/3DTV.2012.6365468}, +ISSN={2161-2021},} + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +@INPROCEEDINGS{ConfTele2013, +author={Carreira, J. and Rodrigues, N. and Faria, S. and Assuncao, P.}, +booktitle={Conference on Telecommunications (Conftele)}, +title={{Frame Loss Concealment for H.264/AVC Stereo Video Decoders}}, +year={2013}, +month=may, +volume={}, +number={}, +pages={1-4}, +abstract={This paper deals with reconstruction of lost frames in H.264/AVC stereo video transmitted over error-prone networks. +The main goal of this research work is to develop an efficient concealment method to recover lost frames in stereoscopic video. +The proposed method combines the use of both motion and disparity vectors to reconstruct missing frames in the auxiliary view of stereoscopic video. +A disparity-compensated motion field is firstly used to fill the largest possible missing regions, and then the remaining unrecovered +regions are reconstructed using either temporally extrapolated motion vectors or disparity vectors. A decision algorithm based on the +temporally co-located base view frame is devised to efficiently adapt the proposed concealment scheme to different video sequences and +different loss events. The experimental results reveal that the proposed method consistently outperforms the existing methods that only +exploit the motion and \mbox{inter-view} similarities. The performance increases for higher bitrates and video sequences with higher +motion achieving PSNR gains up to 2.61 dB.}, +} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +@INPROCEEDINGS{EUSIPCO2014, +author={J. Carreira and V. D. Silva and E. Ekmekcioglu and A. Kondoz and P. Assuncao and S. Faria}, +booktitle={22nd European Signal Processing Conference (EUSIPCO)}, +title={Dynamic motion vector refreshing for enhanced error resilience in {HEVC}}, +year={2014}, +pages={281-285}, +keywords={data compression;motion estimation;video coding;H.264-AVC standard;HEVC coding technique;HEVC standard;compression efficiency;dynamic motion vector refreshing;enhanced error resiliency;error prone condition;error propagation;motion estimation process;motion estimation tool;motion information;rate-distortion performance;reference method;temporal \ac{mv} candidate;temporal dependency;temporal motion vector candidate;video quality;Encoding;Loss measurement;Resilience;Robustness;Standards;Vectors;Video coding;HEVC;error resilience;video coding}, +month=sep,} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +@INPROCEEDINGS{ICIP2014, +author={J. Carreira and E. Ekmekcioglu and A. Kondoz and P. Assuncao and S. Faria and V. De Silva}, +booktitle={IEEE International Conference on Image Processing (ICIP)}, +title={Selective motion vector redundancies for improved error resilience in {HEVC}}, +year={2014}, +pages={2457-2461}, +keywords={image motion analysis;vectors;video coding;HEVC standard;data loss;error resilience performance;high efficiency video coding;improved error resilience;mismatched \ac{mv} predictions;motion vector coding dependencies;reduced overhead;redundant coding;selective motion vector redundancies;subsequent dependent coding units;transmission errors;Bit rate;Encoding;Redundancy;Resilience;Standards;Vectors;Video coding;HEVC;error resilience;motion vectors redundancy}, +doi={10.1109/ICIP.2014.7025497}, +month=oct,} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +@INPROCEEDINGS{ConfTele2015, +author={J. Carreira and S. Faria and P. Assuncao and E. Ekmekcioglu and A. Kondoz}, +booktitle={Conference on Telecommunications (Conftele)}, +title={Error resilience analysis of motion vector prediction in {HEVC}}, +year={2015}, +pages={1-4}, +month=oct,} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +@INPROCEEDINGS{VCIP2015, +author={J. Carreira and P. Assuncao and S. Faria and E. Ekmekcioglu and A. Kondoz and H.Lim}, +booktitle={IEEE International Conference on Visual Communications and Image Processing (VCIP)}, +title={Reference picture selection using checkerboard pattern for resilient video coding}, +year={2015}, +pages={1-5}, +month=dec,} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +@Article{TransactionsCircuitsSystems2017, + author = {J. Carreira and P. Assuncao and S. Faria and E. Ekmekcioglu and A. Kondoz}, + title = {A Two-stage Approach for Robust \ac{hevc} Coding and Streaming}, + journal = {IEEE Transactions on Circuits and Systems for Video Technology}, + year = {2017}, + month = apr, + doi = {10.1109/TCSVT.2017.2691471}, +} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +@INPROCEEDINGS{ISCAS2017, +author={J. Carreira and P. Assuncao and S. Faria and E. Ekmekcioglu and A. Kondoz}, +booktitle={IEEE International Conference on Circuits and Systems (ISCAS)}, +title={A Robust Video Encoding Scheme to Enhance Error Concealment of Intra Frames}, +year={2017}, +pages={1-5}, +month=may,} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% diff --git a/IEEEtran.bst b/IEEEtran.bst new file mode 100644 index 0000000..b639ff8 --- /dev/null +++ b/IEEEtran.bst @@ -0,0 +1,2419 @@ +%% +%% IEEEtran.bst +%% BibTeX Bibliography Style file for IEEE Journals and Conferences (unsorted) +%% Version 1.12 (2007/01/11) +%% +%% Copyright (c) 2003-2007 Michael Shell +%% +%% Original starting code base and algorithms obtained from the output of +%% Patrick W. Daly's makebst package as well as from prior versions of +%% IEEE BibTeX styles: +%% +%% 1. Howard Trickey and Oren Patashnik's ieeetr.bst (1985/1988) +%% 2. Silvano Balemi and Richard H. Roy's IEEEbib.bst (1993) +%% +%% Support sites: +%% http://www.michaelshell.org/tex/ieeetran/ +%% http://www.ctan.org/tex-archive/macros/latex/contrib/IEEEtran/ +%% and/or +%% http://www.ieee.org/ +%% +%% For use with BibTeX version 0.99a or later +%% +%% This is a numerical citation style. +%% +%%************************************************************************* +%% Legal Notice: +%% This code is offered as-is without any warranty either expressed or +%% implied; without even the implied warranty of MERCHANTABILITY or +%% FITNESS FOR A PARTICULAR PURPOSE! +%% User assumes all risk. +%% In no event shall IEEE or any contributor to this code be liable for +%% any damages or losses, including, but not limited to, incidental, +%% consequential, or any other damages, resulting from the use or misuse +%% of any information contained here. +%% +%% All comments are the opinions of their respective authors and are not +%% necessarily endorsed by the IEEE. +%% +%% This work is distributed under the LaTeX Project Public License (LPPL) +%% ( http://www.latex-project.org/ ) version 1.3, and may be freely used, +%% distributed and modified. A copy of the LPPL, version 1.3, is included +%% in the base LaTeX documentation of all distributions of LaTeX released +%% 2003/12/01 or later. +%% Retain all contribution notices and credits. +%% ** Modified files should be clearly indicated as such, including ** +%% ** renaming them and changing author support contact information. ** +%% +%% File list of work: IEEEabrv.bib, IEEEfull.bib, IEEEexample.bib, +%% IEEEtran.bst, IEEEtranS.bst, IEEEtranSA.bst, +%% IEEEtranN.bst, IEEEtranSN.bst, IEEEtran_bst_HOWTO.pdf +%%************************************************************************* +% +% +% Changelog: +% +% 1.00 (2002/08/13) Initial release +% +% 1.10 (2002/09/27) +% 1. Corrected minor bug for improperly formed warning message when a +% book was not given a title. Thanks to Ming Kin Lai for reporting this. +% 2. Added support for CTLname_format_string and CTLname_latex_cmd fields +% in the BST control entry type. +% +% 1.11 (2003/04/02) +% 1. Fixed bug with URLs containing underscores when using url.sty. Thanks +% to Ming Kin Lai for reporting this. +% +% 1.12 (2007/01/11) +% 1. Fixed bug with unwanted comma before "et al." when an entry contained +% more than two author names. Thanks to Pallav Gupta for reporting this. +% 2. Fixed bug with anomalous closing quote in tech reports that have a +% type, but without a number or address. Thanks to Mehrdad Mirreza for +% reporting this. +% 3. Use braces in \providecommand in begin.bib to better support +% latex2html. TeX style length assignments OK with recent versions +% of latex2html - 1.71 (2002/2/1) or later is strongly recommended. +% Use of the language field still causes trouble with latex2html. +% Thanks to Federico Beffa for reporting this. +% 4. Added IEEEtran.bst ID and version comment string to .bbl output. +% 5. Provide a \BIBdecl hook that allows the user to execute commands +% just prior to the first entry. +% 6. Use default urlstyle (is using url.sty) of "same" rather than rm to +% better work with a wider variety of bibliography styles. +% 7. Changed month abbreviations from Sept., July and June to Sep., Jul., +% and Jun., respectively, as IEEE now does. Thanks to Moritz Borgmann +% for reporting this. +% 8. Control entry types should not be considered when calculating longest +% label width. +% 9. Added alias www for electronic/online. +% 10. Added CTLname_url_prefix control entry type. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% DEFAULTS FOR THE CONTROLS OF THE BST STYLE %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +% These are the defaults for the user adjustable controls. The values used +% here can be overridden by the user via IEEEtranBSTCTL entry type. + +% NOTE: The recommended LaTeX command to invoke a control entry type is: +% +%\makeatletter +%\def\bstctlcite{\@ifnextchar[{\@bstctlcite}{\@bstctlcite[@auxout]}} +%\def\@bstctlcite[#1]#2{\@bsphack +% \@for\@citeb:=#2\do{% +% \edef\@citeb{\expandafter\@firstofone\@citeb}% +% \if@filesw\immediate\write\csname #1\endcsname{\string\citation{\@citeb}}\fi}% +% \@esphack} +%\makeatother +% +% It is called at the start of the document, before the first \cite, like: +% \bstctlcite{IEEEexample:BSTcontrol} +% +% IEEEtran.cls V1.6 and later does provide this command. + + + +% #0 turns off the display of the number for articles. +% #1 enables +FUNCTION {default.is.use.number.for.article} { #1 } + + +% #0 turns off the display of the paper and type fields in @inproceedings. +% #1 enables +FUNCTION {default.is.use.paper} { #1 } + + +% #0 turns off the forced use of "et al." +% #1 enables +FUNCTION {default.is.forced.et.al} { #0 } + +% The maximum number of names that can be present beyond which an "et al." +% usage is forced. Be sure that num.names.shown.with.forced.et.al (below) +% is not greater than this value! +% Note: There are many instances of references in IEEE journals which have +% a very large number of authors as well as instances in which "et al." is +% used profusely. +FUNCTION {default.max.num.names.before.forced.et.al} { #10 } + +% The number of names that will be shown with a forced "et al.". +% Must be less than or equal to max.num.names.before.forced.et.al +FUNCTION {default.num.names.shown.with.forced.et.al} { #1 } + + +% #0 turns off the alternate interword spacing for entries with URLs. +% #1 enables +FUNCTION {default.is.use.alt.interword.spacing} { #1 } + +% If alternate interword spacing for entries with URLs is enabled, this is +% the interword spacing stretch factor that will be used. For example, the +% default "4" here means that the interword spacing in entries with URLs can +% stretch to four times normal. Does not have to be an integer. Note that +% the value specified here can be overridden by the user in their LaTeX +% code via a command such as: +% "\providecommand\BIBentryALTinterwordstretchfactor{1.5}" in addition to +% that via the IEEEtranBSTCTL entry type. +FUNCTION {default.ALTinterwordstretchfactor} { "4" } + + +% #0 turns off the "dashification" of repeated (i.e., identical to those +% of the previous entry) names. IEEE normally does this. +% #1 enables +FUNCTION {default.is.dash.repeated.names} { #1 } + + +% The default name format control string. +FUNCTION {default.name.format.string}{ "{f.~}{vv~}{ll}{, jj}" } + + +% The default LaTeX font command for the names. +FUNCTION {default.name.latex.cmd}{ "" } + + +% The default URL prefix. +% FUNCTION {default.name.url.prefix}{ "[Online]. Available:" } +FUNCTION {default.name.url.prefix}{ "" } + + +% Other controls that cannot be accessed via IEEEtranBSTCTL entry type. + +% #0 turns off the terminal startup banner/completed message so as to +% operate more quietly. +% #1 enables +FUNCTION {is.print.banners.to.terminal} { #1 } + + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% FILE VERSION AND BANNER %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +FUNCTION{bst.file.version} { "1.12" } +FUNCTION{bst.file.date} { "2007/01/11" } +FUNCTION{bst.file.website} { "http://www.michaelshell.org/tex/ieeetran/bibtex/" } + +FUNCTION {banner.message} +{ is.print.banners.to.terminal + { "-- IEEEtran.bst version" " " * bst.file.version * + " (" * bst.file.date * ") " * "by Michael Shell." * + top$ + "-- " bst.file.website * + top$ + "-- See the " quote$ * "IEEEtran_bst_HOWTO.pdf" * quote$ * " manual for usage information." * + top$ + } + { skip$ } + if$ +} + +FUNCTION {completed.message} +{ is.print.banners.to.terminal + { "" + top$ + "Done." + top$ + } + { skip$ } + if$ +} + + + + +%%%%%%%%%%%%%%%%%%%%%% +%% STRING CONSTANTS %% +%%%%%%%%%%%%%%%%%%%%%% + +FUNCTION {bbl.and}{ "and" } +FUNCTION {bbl.etal}{ "et~al." } +FUNCTION {bbl.editors}{ "eds." } +FUNCTION {bbl.editor}{ "ed." } +FUNCTION {bbl.edition}{ "ed." } +FUNCTION {bbl.volume}{ "vol." } +FUNCTION {bbl.of}{ "of" } +FUNCTION {bbl.number}{ "no." } +FUNCTION {bbl.in}{ "in" } +FUNCTION {bbl.pages}{ "pp." } +FUNCTION {bbl.page}{ "p." } +FUNCTION {bbl.chapter}{ "ch." } +FUNCTION {bbl.paper}{ "paper" } +FUNCTION {bbl.part}{ "pt." } +FUNCTION {bbl.patent}{ "Patent" } +FUNCTION {bbl.patentUS}{ "U.S." } +FUNCTION {bbl.revision}{ "Rev." } +FUNCTION {bbl.series}{ "ser." } +FUNCTION {bbl.standard}{ "Std." } +FUNCTION {bbl.techrep}{ "Tech. Rep." } +FUNCTION {bbl.mthesis}{ "Master's thesis" } +FUNCTION {bbl.phdthesis}{ "Ph.D. dissertation" } +FUNCTION {bbl.st}{ "st" } +FUNCTION {bbl.nd}{ "nd" } +FUNCTION {bbl.rd}{ "rd" } +FUNCTION {bbl.th}{ "th" } + + +% This is the LaTeX spacer that is used when a larger than normal space +% is called for (such as just before the address:publisher). +FUNCTION {large.space} { "\hskip 1em plus 0.5em minus 0.4em\relax " } + +% The LaTeX code for dashes that are used to represent repeated names. +% Note: Some older IEEE journals used something like +% "\rule{0.275in}{0.5pt}\," which is fairly thick and runs right along +% the baseline. However, IEEE now uses a thinner, above baseline, +% six dash long sequence. +FUNCTION {repeated.name.dashes} { "------" } + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% PREDEFINED STRING MACROS %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +MACRO {jan} {"Jan."} +MACRO {feb} {"Feb."} +MACRO {mar} {"Mar."} +MACRO {apr} {"Apr."} +MACRO {may} {"May"} +MACRO {jun} {"Jun."} +MACRO {jul} {"Jul."} +MACRO {aug} {"Aug."} +MACRO {sep} {"Sep."} +MACRO {oct} {"Oct."} +MACRO {nov} {"Nov."} +MACRO {dec} {"Dec."} + + + +%%%%%%%%%%%%%%%%%% +%% ENTRY FIELDS %% +%%%%%%%%%%%%%%%%%% + +ENTRY + { address + assignee + author + booktitle + chapter + day + dayfiled + edition + editor + howpublished + institution + intype + journal + key + language + month + monthfiled + nationality + note + number + organization + pages + paper + publisher + school + series + revision + title + type + url + volume + year + yearfiled + CTLuse_article_number + CTLuse_paper + CTLuse_forced_etal + CTLmax_names_forced_etal + CTLnames_show_etal + CTLuse_alt_spacing + CTLalt_stretch_factor + CTLdash_repeated_names + CTLname_format_string + CTLname_latex_cmd + CTLname_url_prefix + } + {} + { label } + + + + +%%%%%%%%%%%%%%%%%%%%%%% +%% INTEGER VARIABLES %% +%%%%%%%%%%%%%%%%%%%%%%% + +INTEGERS { prev.status.punct this.status.punct punct.std + punct.no punct.comma punct.period + prev.status.space this.status.space space.std + space.no space.normal space.large + prev.status.quote this.status.quote quote.std + quote.no quote.close + prev.status.nline this.status.nline nline.std + nline.no nline.newblock + status.cap cap.std + cap.no cap.yes} + +INTEGERS { longest.label.width multiresult nameptr namesleft number.label numnames } + +INTEGERS { is.use.number.for.article + is.use.paper + is.forced.et.al + max.num.names.before.forced.et.al + num.names.shown.with.forced.et.al + is.use.alt.interword.spacing + is.dash.repeated.names} + + +%%%%%%%%%%%%%%%%%%%%%% +%% STRING VARIABLES %% +%%%%%%%%%%%%%%%%%%%%%% + +STRINGS { bibinfo + longest.label + oldname + s + t + ALTinterwordstretchfactor + name.format.string + name.latex.cmd + name.url.prefix} + + + + +%%%%%%%%%%%%%%%%%%%%%%%%% +%% LOW LEVEL FUNCTIONS %% +%%%%%%%%%%%%%%%%%%%%%%%%% + +FUNCTION {initialize.controls} +{ default.is.use.number.for.article 'is.use.number.for.article := + default.is.use.paper 'is.use.paper := + default.is.forced.et.al 'is.forced.et.al := + default.max.num.names.before.forced.et.al 'max.num.names.before.forced.et.al := + default.num.names.shown.with.forced.et.al 'num.names.shown.with.forced.et.al := + default.is.use.alt.interword.spacing 'is.use.alt.interword.spacing := + default.is.dash.repeated.names 'is.dash.repeated.names := + default.ALTinterwordstretchfactor 'ALTinterwordstretchfactor := + default.name.format.string 'name.format.string := + default.name.latex.cmd 'name.latex.cmd := + default.name.url.prefix 'name.url.prefix := +} + + +% This IEEEtran.bst features a very powerful and flexible mechanism for +% controlling the capitalization, punctuation, spacing, quotation, and +% newlines of the formatted entry fields. (Note: IEEEtran.bst does not need +% or use the newline/newblock feature, but it has been implemented for +% possible future use.) The output states of IEEEtran.bst consist of +% multiple independent attributes and, as such, can be thought of as being +% vectors, rather than the simple scalar values ("before.all", +% "mid.sentence", etc.) used in most other .bst files. +% +% The more flexible and complex design used here was motivated in part by +% IEEE's rather unusual bibliography style. For example, IEEE ends the +% previous field item with a period and large space prior to the publisher +% address; the @electronic entry types use periods as inter-item punctuation +% rather than the commas used by the other entry types; and URLs are never +% followed by periods even though they are the last item in the entry. +% Although it is possible to accommodate these features with the conventional +% output state system, the seemingly endless exceptions make for convoluted, +% unreliable and difficult to maintain code. +% +% IEEEtran.bst's output state system can be easily understood via a simple +% illustration of two most recently formatted entry fields (on the stack): +% +% CURRENT_ITEM +% "PREVIOUS_ITEM +% +% which, in this example, is to eventually appear in the bibliography as: +% +% "PREVIOUS_ITEM," CURRENT_ITEM +% +% It is the job of the output routine to take the previous item off of the +% stack (while leaving the current item at the top of the stack), apply its +% trailing punctuation (including closing quote marks) and spacing, and then +% to write the result to BibTeX's output buffer: +% +% "PREVIOUS_ITEM," +% +% Punctuation (and spacing) between items is often determined by both of the +% items rather than just the first one. The presence of quotation marks +% further complicates the situation because, in standard English, trailing +% punctuation marks are supposed to be contained within the quotes. +% +% IEEEtran.bst maintains two output state (aka "status") vectors which +% correspond to the previous and current (aka "this") items. Each vector +% consists of several independent attributes which track punctuation, +% spacing, quotation, and newlines. Capitalization status is handled by a +% separate scalar because the format routines, not the output routine, +% handle capitalization and, therefore, there is no need to maintain the +% capitalization attribute for both the "previous" and "this" items. +% +% When a format routine adds a new item, it copies the current output status +% vector to the previous output status vector and (usually) resets the +% current (this) output status vector to a "standard status" vector. Using a +% "standard status" vector in this way allows us to redefine what we mean by +% "standard status" at the start of each entry handler and reuse the same +% format routines under the various inter-item separation schemes. For +% example, the standard status vector for the @book entry type may use +% commas for item separators, while the @electronic type may use periods, +% yet both entry handlers exploit many of the exact same format routines. +% +% Because format routines have write access to the output status vector of +% the previous item, they can override the punctuation choices of the +% previous format routine! Therefore, it becomes trivial to implement rules +% such as "Always use a period and a large space before the publisher." By +% pushing the generation of the closing quote mark to the output routine, we +% avoid all the problems caused by having to close a quote before having all +% the information required to determine what the punctuation should be. +% +% The IEEEtran.bst output state system can easily be expanded if needed. +% For instance, it is easy to add a "space.tie" attribute value if the +% bibliography rules mandate that two items have to be joined with an +% unbreakable space. + +FUNCTION {initialize.status.constants} +{ #0 'punct.no := + #1 'punct.comma := + #2 'punct.period := + #0 'space.no := + #1 'space.normal := + #2 'space.large := + #0 'quote.no := + #1 'quote.close := + #0 'cap.no := + #1 'cap.yes := + #0 'nline.no := + #1 'nline.newblock := +} + +FUNCTION {std.status.using.comma} +{ punct.comma 'punct.std := + space.normal 'space.std := + quote.no 'quote.std := + nline.no 'nline.std := + cap.no 'cap.std := +} + +FUNCTION {std.status.using.period} +{ punct.period 'punct.std := + space.normal 'space.std := + quote.no 'quote.std := + nline.no 'nline.std := + cap.yes 'cap.std := +} + +FUNCTION {initialize.prev.this.status} +{ punct.no 'prev.status.punct := + space.no 'prev.status.space := + quote.no 'prev.status.quote := + nline.no 'prev.status.nline := + punct.no 'this.status.punct := + space.no 'this.status.space := + quote.no 'this.status.quote := + nline.no 'this.status.nline := + cap.yes 'status.cap := +} + +FUNCTION {this.status.std} +{ punct.std 'this.status.punct := + space.std 'this.status.space := + quote.std 'this.status.quote := + nline.std 'this.status.nline := +} + +FUNCTION {cap.status.std}{ cap.std 'status.cap := } + +FUNCTION {this.to.prev.status} +{ this.status.punct 'prev.status.punct := + this.status.space 'prev.status.space := + this.status.quote 'prev.status.quote := + this.status.nline 'prev.status.nline := +} + + +FUNCTION {not} +{ { #0 } + { #1 } + if$ +} + +FUNCTION {and} +{ { skip$ } + { pop$ #0 } + if$ +} + +FUNCTION {or} +{ { pop$ #1 } + { skip$ } + if$ +} + + +% convert the strings "yes" or "no" to #1 or #0 respectively +FUNCTION {yes.no.to.int} +{ "l" change.case$ duplicate$ + "yes" = + { pop$ #1 } + { duplicate$ "no" = + { pop$ #0 } + { "unknown boolean " quote$ * swap$ * quote$ * + " in " * cite$ * warning$ + #0 + } + if$ + } + if$ +} + + +% pushes true if the single char string on the stack is in the +% range of "0" to "9" +FUNCTION {is.num} +{ chr.to.int$ + duplicate$ "0" chr.to.int$ < not + swap$ "9" chr.to.int$ > not and +} + +% multiplies the integer on the stack by a factor of 10 +FUNCTION {bump.int.mag} +{ #0 'multiresult := + { duplicate$ #0 > } + { #1 - + multiresult #10 + + 'multiresult := + } + while$ +pop$ +multiresult +} + +% converts a single character string on the stack to an integer +FUNCTION {char.to.integer} +{ duplicate$ + is.num + { chr.to.int$ "0" chr.to.int$ - } + {"noninteger character " quote$ * swap$ * quote$ * + " in integer field of " * cite$ * warning$ + #0 + } + if$ +} + +% converts a string on the stack to an integer +FUNCTION {string.to.integer} +{ duplicate$ text.length$ 'namesleft := + #1 'nameptr := + #0 'numnames := + { nameptr namesleft > not } + { duplicate$ nameptr #1 substring$ + char.to.integer numnames bump.int.mag + + 'numnames := + nameptr #1 + + 'nameptr := + } + while$ +pop$ +numnames +} + + + + +% The output routines write out the *next* to the top (previous) item on the +% stack, adding punctuation and such as needed. Since IEEEtran.bst maintains +% the output status for the top two items on the stack, these output +% routines have to consider the previous output status (which corresponds to +% the item that is being output). Full independent control of punctuation, +% closing quote marks, spacing, and newblock is provided. +% +% "output.nonnull" does not check for the presence of a previous empty +% item. +% +% "output" does check for the presence of a previous empty item and will +% remove an empty item rather than outputing it. +% +% "output.warn" is like "output", but will issue a warning if it detects +% an empty item. + +FUNCTION {output.nonnull} +{ swap$ + prev.status.punct punct.comma = + { "," * } + { skip$ } + if$ + prev.status.punct punct.period = + { add.period$ } + { skip$ } + if$ + prev.status.quote quote.close = + { "''" * } + { skip$ } + if$ + prev.status.space space.normal = + { " " * } + { skip$ } + if$ + prev.status.space space.large = + { large.space * } + { skip$ } + if$ + write$ + prev.status.nline nline.newblock = + { newline$ "\newblock " write$ } + { skip$ } + if$ +} + +FUNCTION {output} +{ duplicate$ empty$ + 'pop$ + 'output.nonnull + if$ +} + +FUNCTION {output.warn} +{ 't := + duplicate$ empty$ + { pop$ "empty " t * " in " * cite$ * warning$ } + 'output.nonnull + if$ +} + +% "fin.entry" is the output routine that handles the last item of the entry +% (which will be on the top of the stack when "fin.entry" is called). + +FUNCTION {fin.entry} +{ this.status.punct punct.no = + { skip$ } + { add.period$ } + if$ + this.status.quote quote.close = + { "''" * } + { skip$ } + if$ +write$ +newline$ +} + + +FUNCTION {is.last.char.not.punct} +{ duplicate$ + "}" * add.period$ + #-1 #1 substring$ "." = +} + +FUNCTION {is.multiple.pages} +{ 't := + #0 'multiresult := + { multiresult not + t empty$ not + and + } + { t #1 #1 substring$ + duplicate$ "-" = + swap$ duplicate$ "," = + swap$ "+" = + or or + { #1 'multiresult := } + { t #2 global.max$ substring$ 't := } + if$ + } + while$ + multiresult +} + +FUNCTION {capitalize}{ "u" change.case$ "t" change.case$ } + +FUNCTION {emphasize} +{ duplicate$ empty$ + { pop$ "" } + { "\emph{" swap$ * "}" * } + if$ +} + +FUNCTION {do.name.latex.cmd} +{ name.latex.cmd + empty$ + { skip$ } + { name.latex.cmd "{" * swap$ * "}" * } + if$ +} + +% IEEEtran.bst uses its own \BIBforeignlanguage command which directly +% invokes the TeX hyphenation patterns without the need of the Babel +% package. Babel does a lot more than switch hyphenation patterns and +% its loading can cause unintended effects in many class files (such as +% IEEEtran.cls). +FUNCTION {select.language} +{ duplicate$ empty$ 'pop$ + { language empty$ 'skip$ + { "\BIBforeignlanguage{" language * "}{" * swap$ * "}" * } + if$ + } + if$ +} + +FUNCTION {tie.or.space.prefix} +{ duplicate$ text.length$ #3 < + { "~" } + { " " } + if$ + swap$ +} + +FUNCTION {get.bbl.editor} +{ editor num.names$ #1 > 'bbl.editors 'bbl.editor if$ } + +FUNCTION {space.word}{ " " swap$ * " " * } + + +% Field Conditioners, Converters, Checkers and External Interfaces + +FUNCTION {empty.field.to.null.string} +{ duplicate$ empty$ + { pop$ "" } + { skip$ } + if$ +} + +FUNCTION {either.or.check} +{ empty$ + { pop$ } + { "can't use both " swap$ * " fields in " * cite$ * warning$ } + if$ +} + +FUNCTION {empty.entry.warn} +{ author empty$ title empty$ howpublished empty$ + month empty$ year empty$ note empty$ url empty$ + and and and and and and + { "all relevant fields are empty in " cite$ * warning$ } + 'skip$ + if$ +} + + +% The bibinfo system provides a way for the electronic parsing/acquisition +% of a bibliography's contents as is done by ReVTeX. For example, a field +% could be entered into the bibliography as: +% \bibinfo{volume}{2} +% Only the "2" would show up in the document, but the LaTeX \bibinfo command +% could do additional things with the information. IEEEtran.bst does provide +% a \bibinfo command via "\providecommand{\bibinfo}[2]{#2}". However, it is +% currently not used as the bogus bibinfo functions defined here output the +% entry values directly without the \bibinfo wrapper. The bibinfo functions +% themselves (and the calls to them) are retained for possible future use. +% +% bibinfo.check avoids acting on missing fields while bibinfo.warn will +% issue a warning message if a missing field is detected. Prior to calling +% the bibinfo functions, the user should push the field value and then its +% name string, in that order. + +FUNCTION {bibinfo.check} +{ swap$ duplicate$ missing$ + { pop$ pop$ "" } + { duplicate$ empty$ + { swap$ pop$ } + { swap$ pop$ } + if$ + } + if$ +} + +FUNCTION {bibinfo.warn} +{ swap$ duplicate$ missing$ + { swap$ "missing " swap$ * " in " * cite$ * warning$ pop$ "" } + { duplicate$ empty$ + { swap$ "empty " swap$ * " in " * cite$ * warning$ } + { swap$ pop$ } + if$ + } + if$ +} + + +% IEEE separates large numbers with more than 4 digits into groups of +% three. IEEE uses a small space to separate these number groups. +% Typical applications include patent and page numbers. + +% number of consecutive digits required to trigger the group separation. +FUNCTION {large.number.trigger}{ #5 } + +% For numbers longer than the trigger, this is the blocksize of the groups. +% The blocksize must be less than the trigger threshold, and 2 * blocksize +% must be greater than the trigger threshold (can't do more than one +% separation on the initial trigger). +FUNCTION {large.number.blocksize}{ #3 } + +% What is actually inserted between the number groups. +FUNCTION {large.number.separator}{ "\," } + +% So as to save on integer variables by reusing existing ones, numnames +% holds the current number of consecutive digits read and nameptr holds +% the number that will trigger an inserted space. +FUNCTION {large.number.separate} +{ 't := + "" + #0 'numnames := + large.number.trigger 'nameptr := + { t empty$ not } + { t #-1 #1 substring$ is.num + { numnames #1 + 'numnames := } + { #0 'numnames := + large.number.trigger 'nameptr := + } + if$ + t #-1 #1 substring$ swap$ * + t #-2 global.max$ substring$ 't := + numnames nameptr = + { duplicate$ #1 nameptr large.number.blocksize - substring$ swap$ + nameptr large.number.blocksize - #1 + global.max$ substring$ + large.number.separator swap$ * * + nameptr large.number.blocksize - 'numnames := + large.number.blocksize #1 + 'nameptr := + } + { skip$ } + if$ + } + while$ +} + +% Converts all single dashes "-" to double dashes "--". +FUNCTION {n.dashify} +{ large.number.separate + 't := + "" + { t empty$ not } + { t #1 #1 substring$ "-" = + { t #1 #2 substring$ "--" = not + { "--" * + t #2 global.max$ substring$ 't := + } + { { t #1 #1 substring$ "-" = } + { "-" * + t #2 global.max$ substring$ 't := + } + while$ + } + if$ + } + { t #1 #1 substring$ * + t #2 global.max$ substring$ 't := + } + if$ + } + while$ +} + + +% This function detects entries with names that are identical to that of +% the previous entry and replaces the repeated names with dashes (if the +% "is.dash.repeated.names" user control is nonzero). +FUNCTION {name.or.dash} +{ 's := + oldname empty$ + { s 'oldname := s } + { s oldname = + { is.dash.repeated.names + { repeated.name.dashes } + { s 'oldname := s } + if$ + } + { s 'oldname := s } + if$ + } + if$ +} + +% Converts the number string on the top of the stack to +% "numerical ordinal form" (e.g., "7" to "7th"). There is +% no artificial limit to the upper bound of the numbers as the +% least significant digit always determines the ordinal form. +FUNCTION {num.to.ordinal} +{ duplicate$ #-1 #1 substring$ "1" = + { bbl.st * } + { duplicate$ #-1 #1 substring$ "2" = + { bbl.nd * } + { duplicate$ #-1 #1 substring$ "3" = + { bbl.rd * } + { bbl.th * } + if$ + } + if$ + } + if$ +} + +% If the string on the top of the stack begins with a number, +% (e.g., 11th) then replace the string with the leading number +% it contains. Otherwise retain the string as-is. s holds the +% extracted number, t holds the part of the string that remains +% to be scanned. +FUNCTION {extract.num} +{ duplicate$ 't := + "" 's := + { t empty$ not } + { t #1 #1 substring$ + t #2 global.max$ substring$ 't := + duplicate$ is.num + { s swap$ * 's := } + { pop$ "" 't := } + if$ + } + while$ + s empty$ + 'skip$ + { pop$ s } + if$ +} + +% Converts the word number string on the top of the stack to +% Arabic string form. Will be successful up to "tenth". +FUNCTION {word.to.num} +{ duplicate$ "l" change.case$ 's := + s "first" = + { pop$ "1" } + { skip$ } + if$ + s "second" = + { pop$ "2" } + { skip$ } + if$ + s "third" = + { pop$ "3" } + { skip$ } + if$ + s "fourth" = + { pop$ "4" } + { skip$ } + if$ + s "fifth" = + { pop$ "5" } + { skip$ } + if$ + s "sixth" = + { pop$ "6" } + { skip$ } + if$ + s "seventh" = + { pop$ "7" } + { skip$ } + if$ + s "eighth" = + { pop$ "8" } + { skip$ } + if$ + s "ninth" = + { pop$ "9" } + { skip$ } + if$ + s "tenth" = + { pop$ "10" } + { skip$ } + if$ +} + + +% Converts the string on the top of the stack to numerical +% ordinal (e.g., "11th") form. +FUNCTION {convert.edition} +{ duplicate$ empty$ 'skip$ + { duplicate$ #1 #1 substring$ is.num + { extract.num + num.to.ordinal + } + { word.to.num + duplicate$ #1 #1 substring$ is.num + { num.to.ordinal } + { "edition ordinal word " quote$ * edition * quote$ * + " may be too high (or improper) for conversion" * " in " * cite$ * warning$ + } + if$ + } + if$ + } + if$ +} + + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% LATEX BIBLIOGRAPHY CODE %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +FUNCTION {start.entry} +{ newline$ + "\bibitem{" write$ + cite$ write$ + "}" write$ + newline$ + "" + initialize.prev.this.status +} + +% Here we write out all the LaTeX code that we will need. The most involved +% code sequences are those that control the alternate interword spacing and +% foreign language hyphenation patterns. The heavy use of \providecommand +% gives users a way to override the defaults. Special thanks to Javier Bezos, +% Johannes Braams, Robin Fairbairns, Heiko Oberdiek, Donald Arseneau and all +% the other gurus on comp.text.tex for their help and advice on the topic of +% \selectlanguage, Babel and BibTeX. +FUNCTION {begin.bib} +{ "% Generated by IEEEtran.bst, version: " bst.file.version * " (" * bst.file.date * ")" * + write$ newline$ + preamble$ empty$ 'skip$ + { preamble$ write$ newline$ } + if$ + "\begin{thebibliography}{" longest.label * "}" * + write$ newline$ + % "\providecommand{\url}[1]{#1}" - CARREIRA + "\def\url#1{}" + write$ newline$ + "\csname url@samestyle\endcsname" + write$ newline$ + "\providecommand{\newblock}{\relax}" + write$ newline$ + "\providecommand{\bibinfo}[2]{#2}" + write$ newline$ + "\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}" + write$ newline$ + "\providecommand{\BIBentryALTinterwordstretchfactor}{" + ALTinterwordstretchfactor * "}" * + write$ newline$ + "\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus " + write$ newline$ + "\BIBentryALTinterwordstretchfactor\fontdimen3\font minus \fontdimen4\font\relax}" + write$ newline$ + "\providecommand{\BIBforeignlanguage}[2]{{%" + write$ newline$ + "\expandafter\ifx\csname l@#1\endcsname\relax" + write$ newline$ + "\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%" + write$ newline$ + "\typeout{** loaded for the language `#1'. Using the pattern for}%" + write$ newline$ + "\typeout{** the default language instead.}%" + write$ newline$ + "\else" + write$ newline$ + "\language=\csname l@#1\endcsname" + write$ newline$ + "\fi" + write$ newline$ + "#2}}" + write$ newline$ + "\providecommand{\BIBdecl}{\relax}" + write$ newline$ + "\BIBdecl" + write$ newline$ +} + +FUNCTION {end.bib} +{ newline$ "\end{thebibliography}" write$ newline$ } + +FUNCTION {if.url.alt.interword.spacing} +{ is.use.alt.interword.spacing + {url empty$ 'skip$ {"\BIBentryALTinterwordspacing" write$ newline$} if$} + { skip$ } + if$ +} + +FUNCTION {if.url.std.interword.spacing} +{ is.use.alt.interword.spacing + {url empty$ 'skip$ {"\BIBentrySTDinterwordspacing" write$ newline$} if$} + { skip$ } + if$ +} + + + + +%%%%%%%%%%%%%%%%%%%%%%%% +%% LONGEST LABEL PASS %% +%%%%%%%%%%%%%%%%%%%%%%%% + +FUNCTION {initialize.longest.label} +{ "" 'longest.label := + #1 'number.label := + #0 'longest.label.width := +} + +FUNCTION {longest.label.pass} +{ type$ "ieeetranbstctl" = + { skip$ } + { number.label int.to.str$ 'label := + number.label #1 + 'number.label := + label width$ longest.label.width > + { label 'longest.label := + label width$ 'longest.label.width := + } + { skip$ } + if$ + } + if$ +} + + + + +%%%%%%%%%%%%%%%%%%%%% +%% FORMAT HANDLERS %% +%%%%%%%%%%%%%%%%%%%%% + +%% Lower Level Formats (used by higher level formats) + +FUNCTION {format.address.org.or.pub.date} +{ 't := + "" + year empty$ + { "empty year in " cite$ * warning$ } + { skip$ } + if$ + address empty$ t empty$ and + year empty$ and month empty$ and + { skip$ } + { this.to.prev.status + this.status.std + cap.status.std + address "address" bibinfo.check * + t empty$ + { skip$ } + { punct.period 'prev.status.punct := + space.large 'prev.status.space := + address empty$ + { skip$ } + { ": " * } + if$ + t * + } + if$ + year empty$ month empty$ and + { skip$ } + { t empty$ address empty$ and + { skip$ } + { ", " * } + if$ + month empty$ + { year empty$ + { skip$ } + { year "year" bibinfo.check * } + if$ + } + { month "month" bibinfo.check * + year empty$ + { skip$ } + { " " * year "year" bibinfo.check * } + if$ + } + if$ + } + if$ + } + if$ +} + + +FUNCTION {format.names} +{ 'bibinfo := + duplicate$ empty$ 'skip$ { + this.to.prev.status + this.status.std + 's := + "" 't := + #1 'nameptr := + s num.names$ 'numnames := + numnames 'namesleft := + { namesleft #0 > } + { s nameptr + name.format.string + format.name$ + bibinfo bibinfo.check + 't := + nameptr #1 > + { nameptr num.names.shown.with.forced.et.al #1 + = + numnames max.num.names.before.forced.et.al > + is.forced.et.al and and + { "others" 't := + #1 'namesleft := + } + { skip$ } + if$ + namesleft #1 > + { ", " * t do.name.latex.cmd * } + { s nameptr "{ll}" format.name$ duplicate$ "others" = + { 't := } + { pop$ } + if$ + t "others" = + { " " * bbl.etal emphasize * } + { numnames #2 > + { "," * } + { skip$ } + if$ + bbl.and + space.word * t do.name.latex.cmd * + } + if$ + } + if$ + } + { t do.name.latex.cmd } + if$ + nameptr #1 + 'nameptr := + namesleft #1 - 'namesleft := + } + while$ + cap.status.std + } if$ +} + + + + +%% Higher Level Formats + +%% addresses/locations + +FUNCTION {format.address} +{ address duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + } + if$ +} + + + +%% author/editor names + +FUNCTION {format.authors}{ author "author" format.names } + +FUNCTION {format.editors} +{ editor "editor" format.names duplicate$ empty$ 'skip$ + { ", " * + get.bbl.editor + capitalize + * + } + if$ +} + + + +%% date + +FUNCTION {format.date} +{ + month "month" bibinfo.check duplicate$ empty$ + year "year" bibinfo.check duplicate$ empty$ + { swap$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + "there's a month but no year in " cite$ * warning$ } + if$ + * + } + { this.to.prev.status + this.status.std + cap.status.std + swap$ 'skip$ + { + swap$ + " " * swap$ + } + if$ + * + } + if$ +} + +FUNCTION {format.date.electronic} +{ month "month" bibinfo.check duplicate$ empty$ + year "year" bibinfo.check duplicate$ empty$ + { swap$ + { pop$ } + { "there's a month but no year in " cite$ * warning$ + pop$ ")" * "(" swap$ * + this.to.prev.status + punct.no 'this.status.punct := + space.normal 'this.status.space := + quote.no 'this.status.quote := + cap.yes 'status.cap := + } + if$ + } + { swap$ + { swap$ pop$ ")" * "(" swap$ * } + { "(" swap$ * ", " * swap$ * ")" * } + if$ + this.to.prev.status + punct.no 'this.status.punct := + space.normal 'this.status.space := + quote.no 'this.status.quote := + cap.yes 'status.cap := + } + if$ +} + + + +%% edition/title + +% Note: IEEE considers the edition to be closely associated with +% the title of a book. So, in IEEEtran.bst the edition is normally handled +% within the formatting of the title. The format.edition function is +% retained here for possible future use. +FUNCTION {format.edition} +{ edition duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + convert.edition + status.cap + { "t" } + { "l" } + if$ change.case$ + "edition" bibinfo.check + "~" * bbl.edition * + cap.status.std + } + if$ +} + +% This is used to format the booktitle of a conference proceedings. +% Here we use the "intype" field to provide the user a way to +% override the word "in" (e.g., with things like "presented at") +% Use of intype stops the emphasis of the booktitle to indicate that +% we no longer mean the written conference proceedings, but the +% conference itself. +FUNCTION {format.in.booktitle} +{ booktitle "booktitle" bibinfo.check duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + select.language + intype missing$ + { emphasize + bbl.in " " * + } + { intype " " * } + if$ + swap$ * + cap.status.std + } + if$ +} + +% This is used to format the booktitle of collection. +% Here the "intype" field is not supported, but "edition" is. +FUNCTION {format.in.booktitle.edition} +{ booktitle "booktitle" bibinfo.check duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + select.language + emphasize + edition empty$ 'skip$ + { ", " * + edition + convert.edition + "l" change.case$ + * "~" * bbl.edition * + } + if$ + bbl.in " " * swap$ * + cap.status.std + } + if$ +} + +FUNCTION {format.article.title} +{ title duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + "t" change.case$ + } + if$ + "title" bibinfo.check + duplicate$ empty$ 'skip$ + { quote.close 'this.status.quote := + is.last.char.not.punct + { punct.std 'this.status.punct := } + { punct.no 'this.status.punct := } + if$ + select.language + "``" swap$ * + cap.status.std + } + if$ +} + +FUNCTION {format.article.title.electronic} +{ title duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + "t" change.case$ + } + if$ + "title" bibinfo.check + duplicate$ empty$ + { skip$ } + { select.language } + if$ +} + +FUNCTION {format.book.title.edition} +{ title "title" bibinfo.check + duplicate$ empty$ + { "empty title in " cite$ * warning$ } + { this.to.prev.status + this.status.std + select.language + emphasize + edition empty$ 'skip$ + { ", " * + edition + convert.edition + status.cap + { "t" } + { "l" } + if$ + change.case$ + * "~" * bbl.edition * + } + if$ + cap.status.std + } + if$ +} + +FUNCTION {format.book.title} +{ title "title" bibinfo.check + duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + select.language + emphasize + } + if$ +} + + + +%% journal + +FUNCTION {format.journal} +{ journal duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + select.language + emphasize + } + if$ +} + + + +%% how published + +FUNCTION {format.howpublished} +{ howpublished duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + } + if$ +} + + + +%% institutions/organization/publishers/school + +FUNCTION {format.institution} +{ institution duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + } + if$ +} + +FUNCTION {format.organization} +{ organization duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + } + if$ +} + +FUNCTION {format.address.publisher.date} +{ publisher "publisher" bibinfo.warn format.address.org.or.pub.date } + +FUNCTION {format.address.publisher.date.nowarn} +{ publisher "publisher" bibinfo.check format.address.org.or.pub.date } + +FUNCTION {format.address.organization.date} +{ organization "organization" bibinfo.check format.address.org.or.pub.date } + +FUNCTION {format.school} +{ school duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + } + if$ +} + + + +%% volume/number/series/chapter/pages + +FUNCTION {format.volume} +{ volume empty.field.to.null.string + duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + bbl.volume + status.cap + { capitalize } + { skip$ } + if$ + swap$ tie.or.space.prefix + "volume" bibinfo.check + * * + cap.status.std + } + if$ +} + +FUNCTION {format.number} +{ number empty.field.to.null.string + duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + status.cap + { bbl.number capitalize } + { bbl.number } + if$ + swap$ tie.or.space.prefix + "number" bibinfo.check + * * + cap.status.std + } + if$ +} + +FUNCTION {format.number.if.use.for.article} +{ is.use.number.for.article + { format.number } + { "" } + if$ +} + +% IEEE does not seem to tie the series so closely with the volume +% and number as is done in other bibliography styles. Instead the +% series is treated somewhat like an extension of the title. +FUNCTION {format.series} +{ series empty$ + { "" } + { this.to.prev.status + this.status.std + bbl.series " " * + series "series" bibinfo.check * + cap.status.std + } + if$ +} + + +FUNCTION {format.chapter} +{ chapter empty$ + { "" } + { this.to.prev.status + this.status.std + type empty$ + { bbl.chapter } + { type "l" change.case$ + "type" bibinfo.check + } + if$ + chapter tie.or.space.prefix + "chapter" bibinfo.check + * * + cap.status.std + } + if$ +} + + +% The intended use of format.paper is for paper numbers of inproceedings. +% The paper type can be overridden via the type field. +% We allow the type to be displayed even if the paper number is absent +% for things like "postdeadline paper" +FUNCTION {format.paper} +{ is.use.paper + { paper empty$ + { type empty$ + { "" } + { this.to.prev.status + this.status.std + type "type" bibinfo.check + cap.status.std + } + if$ + } + { this.to.prev.status + this.status.std + type empty$ + { bbl.paper } + { type "type" bibinfo.check } + if$ + " " * paper + "paper" bibinfo.check + * + cap.status.std + } + if$ + } + { "" } + if$ +} + + +FUNCTION {format.pages} +{ pages duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + duplicate$ is.multiple.pages + { + bbl.pages swap$ + n.dashify + } + { + bbl.page swap$ + } + if$ + tie.or.space.prefix + "pages" bibinfo.check + * * + cap.status.std + } + if$ +} + + + +%% technical report number + +FUNCTION {format.tech.report.number} +{ number "number" bibinfo.check + this.to.prev.status + this.status.std + cap.status.std + type duplicate$ empty$ + { pop$ + bbl.techrep + } + { skip$ } + if$ + "type" bibinfo.check + swap$ duplicate$ empty$ + { pop$ } + { tie.or.space.prefix * * } + if$ +} + + + +%% note + +FUNCTION {format.note} +{ note empty$ + { "" } + { this.to.prev.status + this.status.std + punct.period 'this.status.punct := + note #1 #1 substring$ + duplicate$ "{" = + { skip$ } + { status.cap + { "u" } + { "l" } + if$ + change.case$ + } + if$ + note #2 global.max$ substring$ * "note" bibinfo.check + cap.yes 'status.cap := + } + if$ +} + + + +%% patent + +FUNCTION {format.patent.date} +{ this.to.prev.status + this.status.std + year empty$ + { monthfiled duplicate$ empty$ + { "monthfiled" bibinfo.check pop$ "" } + { "monthfiled" bibinfo.check } + if$ + dayfiled duplicate$ empty$ + { "dayfiled" bibinfo.check pop$ "" * } + { "dayfiled" bibinfo.check + monthfiled empty$ + { "dayfiled without a monthfiled in " cite$ * warning$ + * + } + { " " swap$ * * } + if$ + } + if$ + yearfiled empty$ + { "no year or yearfiled in " cite$ * warning$ } + { yearfiled "yearfiled" bibinfo.check + swap$ + duplicate$ empty$ + { pop$ } + { ", " * swap$ * } + if$ + } + if$ + } + { month duplicate$ empty$ + { "month" bibinfo.check pop$ "" } + { "month" bibinfo.check } + if$ + day duplicate$ empty$ + { "day" bibinfo.check pop$ "" * } + { "day" bibinfo.check + month empty$ + { "day without a month in " cite$ * warning$ + * + } + { " " swap$ * * } + if$ + } + if$ + year "year" bibinfo.check + swap$ + duplicate$ empty$ + { pop$ } + { ", " * swap$ * } + if$ + } + if$ + cap.status.std +} + +FUNCTION {format.patent.nationality.type.number} +{ this.to.prev.status + this.status.std + nationality duplicate$ empty$ + { "nationality" bibinfo.warn pop$ "" } + { "nationality" bibinfo.check + duplicate$ "l" change.case$ "united states" = + { pop$ bbl.patentUS } + { skip$ } + if$ + " " * + } + if$ + type empty$ + { bbl.patent "type" bibinfo.check } + { type "type" bibinfo.check } + if$ + * + number duplicate$ empty$ + { "number" bibinfo.warn pop$ } + { "number" bibinfo.check + large.number.separate + swap$ " " * swap$ * + } + if$ + cap.status.std +} + + + +%% standard + +FUNCTION {format.organization.institution.standard.type.number} +{ this.to.prev.status + this.status.std + organization duplicate$ empty$ + { pop$ + institution duplicate$ empty$ + { "institution" bibinfo.warn } + { "institution" bibinfo.warn " " * } + if$ + } + { "organization" bibinfo.warn " " * } + if$ + type empty$ + { bbl.standard "type" bibinfo.check } + { type "type" bibinfo.check } + if$ + * + number duplicate$ empty$ + { "number" bibinfo.check pop$ } + { "number" bibinfo.check + large.number.separate + swap$ " " * swap$ * + } + if$ + cap.status.std +} + +FUNCTION {format.revision} +{ revision empty$ + { "" } + { this.to.prev.status + this.status.std + bbl.revision + revision tie.or.space.prefix + "revision" bibinfo.check + * * + cap.status.std + } + if$ +} + + +%% thesis + +FUNCTION {format.master.thesis.type} +{ this.to.prev.status + this.status.std + type empty$ + { + bbl.mthesis + } + { + type "type" bibinfo.check + } + if$ +cap.status.std +} + +FUNCTION {format.phd.thesis.type} +{ this.to.prev.status + this.status.std + type empty$ + { + bbl.phdthesis + } + { + type "type" bibinfo.check + } + if$ +cap.status.std +} + + + +%% URL + +FUNCTION {format.url} +{ url empty$ + { "" } + { this.to.prev.status + this.status.std + cap.yes 'status.cap := + name.url.prefix " " * + "\url{" * url * "}" * + punct.no 'this.status.punct := + punct.period 'prev.status.punct := + space.normal 'this.status.space := + space.normal 'prev.status.space := + quote.no 'this.status.quote := + } + if$ +} + + + + +%%%%%%%%%%%%%%%%%%%% +%% ENTRY HANDLERS %% +%%%%%%%%%%%%%%%%%%%% + + +% Note: In many journals, IEEE (or the authors) tend not to show the number +% for articles, so the display of the number is controlled here by the +% switch "is.use.number.for.article" +FUNCTION {article} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors "author" output.warn + name.or.dash + format.article.title "title" output.warn + format.journal "journal" bibinfo.check "journal" output.warn + format.volume output + format.number.if.use.for.article output + format.pages output + format.date "year" output.warn + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {book} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + author empty$ + { format.editors "author and editor" output.warn } + { format.authors output.nonnull } + if$ + name.or.dash + format.book.title.edition output + format.series output + author empty$ + { skip$ } + { format.editors output } + if$ + format.address.publisher.date output + format.volume output + format.number output + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {booklet} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors output + name.or.dash + format.article.title "title" output.warn + format.howpublished "howpublished" bibinfo.check output + format.organization "organization" bibinfo.check output + format.address "address" bibinfo.check output + format.date output + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {electronic} +{ std.status.using.period + start.entry + if.url.alt.interword.spacing + format.authors output + name.or.dash + format.date.electronic output + format.article.title.electronic output + format.howpublished "howpublished" bibinfo.check output + format.organization "organization" bibinfo.check output + format.address "address" bibinfo.check output + format.note output + format.url output + fin.entry + empty.entry.warn + if.url.std.interword.spacing +} + +FUNCTION {inbook} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + author empty$ + { format.editors "author and editor" output.warn } + { format.authors output.nonnull } + if$ + name.or.dash + format.book.title.edition output + format.series output + format.address.publisher.date output + format.volume output + format.number output + format.chapter output + format.pages output + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {incollection} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors "author" output.warn + name.or.dash + format.article.title "title" output.warn + format.in.booktitle.edition "booktitle" output.warn + format.series output + format.editors output + format.address.publisher.date.nowarn output + format.volume output + format.number output + format.chapter output + format.pages output + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {inproceedings} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors "author" output.warn + name.or.dash + format.article.title "title" output.warn + format.in.booktitle "booktitle" output.warn + format.series output + format.editors output + format.volume output + format.number output + publisher empty$ + { format.address.organization.date output } + { format.organization "organization" bibinfo.check output + format.address.publisher.date output + } + if$ + format.paper output + format.pages output + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {manual} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors output + name.or.dash + format.book.title.edition "title" output.warn + format.howpublished "howpublished" bibinfo.check output + format.organization "organization" bibinfo.check output + format.address "address" bibinfo.check output + format.date output + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {mastersthesis} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors "author" output.warn + name.or.dash + format.article.title "title" output.warn + format.master.thesis.type output.nonnull + format.school "school" bibinfo.warn output + format.address "address" bibinfo.check output + format.date "year" output.warn + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {misc} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors output + name.or.dash + format.article.title output + format.howpublished "howpublished" bibinfo.check output + format.organization "organization" bibinfo.check output + format.address "address" bibinfo.check output + format.pages output + format.date output + format.note output + format.url output + fin.entry + empty.entry.warn + if.url.std.interword.spacing +} + +FUNCTION {patent} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors output + name.or.dash + format.article.title output + format.patent.nationality.type.number output + format.patent.date output + format.note output + format.url output + fin.entry + empty.entry.warn + if.url.std.interword.spacing +} + +FUNCTION {periodical} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.editors output + name.or.dash + format.book.title "title" output.warn + format.series output + format.volume output + format.number output + format.organization "organization" bibinfo.check output + format.date "year" output.warn + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {phdthesis} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors "author" output.warn + name.or.dash + format.article.title "title" output.warn + format.phd.thesis.type output.nonnull + format.school "school" bibinfo.warn output + format.address "address" bibinfo.check output + format.date "year" output.warn + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {proceedings} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.editors output + name.or.dash + format.book.title "title" output.warn + format.series output + format.volume output + format.number output + publisher empty$ + { format.address.organization.date output } + { format.organization "organization" bibinfo.check output + format.address.publisher.date output + } + if$ + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {standard} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors output + name.or.dash + format.book.title "title" output.warn + format.howpublished "howpublished" bibinfo.check output + format.organization.institution.standard.type.number output + format.revision output + format.date output + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {techreport} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors "author" output.warn + name.or.dash + format.article.title "title" output.warn + format.howpublished "howpublished" bibinfo.check output + format.institution "institution" bibinfo.warn output + format.address "address" bibinfo.check output + format.tech.report.number output.nonnull + format.date "year" output.warn + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {unpublished} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors "author" output.warn + name.or.dash + format.article.title "title" output.warn + format.date output + format.note "note" output.warn + format.url output + fin.entry + if.url.std.interword.spacing +} + + +% The special entry type which provides the user interface to the +% BST controls +FUNCTION {IEEEtranBSTCTL} +{ is.print.banners.to.terminal + { "** IEEEtran BST control entry " quote$ * cite$ * quote$ * " detected." * + top$ + } + { skip$ } + if$ + CTLuse_article_number + empty$ + { skip$ } + { CTLuse_article_number + yes.no.to.int + 'is.use.number.for.article := + } + if$ + CTLuse_paper + empty$ + { skip$ } + { CTLuse_paper + yes.no.to.int + 'is.use.paper := + } + if$ + CTLuse_forced_etal + empty$ + { skip$ } + { CTLuse_forced_etal + yes.no.to.int + 'is.forced.et.al := + } + if$ + CTLmax_names_forced_etal + empty$ + { skip$ } + { CTLmax_names_forced_etal + string.to.integer + 'max.num.names.before.forced.et.al := + } + if$ + CTLnames_show_etal + empty$ + { skip$ } + { CTLnames_show_etal + string.to.integer + 'num.names.shown.with.forced.et.al := + } + if$ + CTLuse_alt_spacing + empty$ + { skip$ } + { CTLuse_alt_spacing + yes.no.to.int + 'is.use.alt.interword.spacing := + } + if$ + CTLalt_stretch_factor + empty$ + { skip$ } + { CTLalt_stretch_factor + 'ALTinterwordstretchfactor := + "\renewcommand{\BIBentryALTinterwordstretchfactor}{" + ALTinterwordstretchfactor * "}" * + write$ newline$ + } + if$ + CTLdash_repeated_names + empty$ + { skip$ } + { CTLdash_repeated_names + yes.no.to.int + 'is.dash.repeated.names := + } + if$ + CTLname_format_string + empty$ + { skip$ } + { CTLname_format_string + 'name.format.string := + } + if$ + CTLname_latex_cmd + empty$ + { skip$ } + { CTLname_latex_cmd + 'name.latex.cmd := + } + if$ + CTLname_url_prefix + missing$ + { skip$ } + { CTLname_url_prefix + 'name.url.prefix := + } + if$ + + + num.names.shown.with.forced.et.al max.num.names.before.forced.et.al > + { "CTLnames_show_etal cannot be greater than CTLmax_names_forced_etal in " cite$ * warning$ + max.num.names.before.forced.et.al 'num.names.shown.with.forced.et.al := + } + { skip$ } + if$ +} + + +%%%%%%%%%%%%%%%%%%% +%% ENTRY ALIASES %% +%%%%%%%%%%%%%%%%%%% +FUNCTION {conference}{inproceedings} +FUNCTION {online}{electronic} +FUNCTION {internet}{electronic} +FUNCTION {webpage}{electronic} +FUNCTION {www}{electronic} +FUNCTION {default.type}{misc} + + + +%%%%%%%%%%%%%%%%%% +%% MAIN PROGRAM %% +%%%%%%%%%%%%%%%%%% + +READ + +EXECUTE {initialize.controls} +EXECUTE {initialize.status.constants} +EXECUTE {banner.message} + +EXECUTE {initialize.longest.label} +ITERATE {longest.label.pass} + +EXECUTE {begin.bib} +ITERATE {call.type$} +EXECUTE {end.bib} + +EXECUTE{completed.message} + + +%% That's all folks, mds. diff --git a/TikzThesisTemplate.pgs b/TikzThesisTemplate.pgs new file mode 100644 index 0000000..244c9c5 --- /dev/null +++ b/TikzThesisTemplate.pgs @@ -0,0 +1,17 @@ +\documentclass[phdthesis,en]{estgleiria} + +\usepackage{carreira_phd_thesis} +\usepackage{balance} + +\begin{document} + +\begin{figure}[t] +\centering +<> +\caption{This is the caption.} +\end{figure} + +\balance +The increasing diversity of multimedia applications and services, and the emergence of Ultra-HD formats (\emph{e.g.}, \mbox{4k} or \mbox{8k} resolution) are continuously reinforcing the need for high efficiency video coding efficiency, particularly in + +\end{document} diff --git a/build_plts.bash b/build_plts.bash new file mode 100755 index 0000000..f0bf8ca --- /dev/null +++ b/build_plts.bash @@ -0,0 +1,26 @@ +#!/bin/bash + +curr_dir=$(pwd) + +for plt in $(find . -name "*.plt") +do + plt_folder=$( dirname $plt ) + plt_name=$( basename $plt ) + plt_name_no_ext=${plt_name%.*} + + cd $plt_folder + + gnuplot -e "load '$plt_name'" + file_list= + for f in *.eps + do + ps2pdf -dEPSCrop "${f%%.*}.eps" "${f%%.*}.pdf" + rm "${f%%.*}.eps" + file_list="$file_list ${f%%.*}.pdf" + done + run_echo gs -q -sPAPERSIZE=letter -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile="${plt_name_no_ext}.pdf" $file_list + rm $file_list + + cd $curr_dir +done + diff --git a/carreira_phd_thesis.pdf b/carreira_phd_thesis.pdf new file mode 100644 index 0000000..b4b5182 Binary files /dev/null and b/carreira_phd_thesis.pdf differ diff --git a/carreira_phd_thesis.sty b/carreira_phd_thesis.sty new file mode 100644 index 0000000..e30fd26 --- /dev/null +++ b/carreira_phd_thesis.sty @@ -0,0 +1,120 @@ +\usepackage[english]{babel} + +% *** MISC *** +\usepackage{verbatim} +\usepackage{subfiles} +\usepackage{xspace} +\usepackage{pdfpages} +\usepackage[colorlinks=false,hidelinks,pdfpagelabels]{hyperref} + +% *** Minitoc *** +%\usepackage[nohints,tight]{minitoc} +%\setcounter{minitocdepth}{2} +% Generates mini tables of contents per chapter + +% JEM: The following fixes minitoc to not be multiple-spaced. +%\let\oldmtc@verse\mtc@verse +%\renewcommand{\mtc@verse}[1]{\oldmtc@verse{#1}\setlength{\baselineskip}{\z@}} + + +% *** CITATION PACKAGES *** +\usepackage{cite} +\usepackage{bibentry} + + +% *** GRAPHICS RELATED PACKAGES *** +\usepackage{drawstack} + +\usepackage{epstopdf} +\usepackage{dblfloatfix} +\usepackage[tight,footnotesize]{subfigure} +\usepackage{pgfgantt} +\usepackage{etex} + +\usepackage{bytefield} % Create bitstrem image +\newsavebox{\bytefieldbox} + +%% Gnuplot +\usepackage[subfolder]{gnuplottex} +\usepackage{xparse} +\ExplSyntaxOn +\DeclareExpandableDocumentCommand{\convertlen}{ O{cm} m } +{ +\dim_to_decimal_in_unit:nn { #2 } { 1 #1 } cm +} +\ExplSyntaxOff +\newcommand{\PlotFont}[0]{7} +\newcommand{\TitleFont}[0]{\footnotesize} +\newcommand{\TicsFont}[0]{\scriptsize} +\newcommand{\KeyFont}[0]{\scriptsize} +\newcommand{\PLRLABEL}{\footnotesize Packet Loss Ratio (\%)} +\newcommand{\PSNRLABEL}{\footnotesize PSNR (dB)} + +%% TikZ +\usepackage{tikz} +\usepackage{pgfplots} +\usetikzlibrary{fit} +\usetikzlibrary{calc} +\usetikzlibrary{snakes} +\usetikzlibrary{arrows} +\usetikzlibrary{positioning} +\usetikzlibrary{decorations} +\usetikzlibrary{graphs} +\usetikzlibrary{trees} +\usetikzlibrary{decorations.markings} +\def\TikZColorFill{gray!20} +\def\TikZColorFillDark{gray!80} +\def\TikZColorFillProp{blue!15} +\def\TikZyslant{-0.2} +\def\TikZxscale{0.75} +\def\TikZyfshift{0.05cm} +\newcounter{maincounter} + +% *** TABLE PACKAGES *** +\usepackage{tabularx, colortbl, booktabs, array, multirow} +\newcolumntype{x}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}m{#1}} +\newcolumntype{y}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}b{#1}} + + +% *** FLOAT PACKAGES *** +% Coloca sempre as figuras dentro das secções a que pertencem +\usepackage{float} +\usepackage[section]{placeins} + + +% *** MATH PACKAGES *** +\usepackage{array,mathtools} +\usepackage{mdwmath,amsmath} +\usepackage{array,mdwmath,amsmath,commath} +\newcommand*\mean[1]{\bar{#1}} + + +% *** PAGE ROTATION PACKAGES *** +\usepackage{pdflscape} +\usepackage{adjustbox} +\usepackage{rotating} + + +% *** ABREVIATIONS *** +\usepackage[acronym,toc,nomain,nonumberlist]{glossaries} +%\newcommand{\customnewacronym}[4][]{\newacronym[#1,description=#4]{#2}{#3}{\MakeLowercase{#4}}} +\newcommand{\customnewacronym}[4][]{\newacronym[#1,description=#4]{#2}{\mbox{#3}}{#4}} +\setglossarystyle{long} +\newcommand{\ac}[1]{\gls{#1}} +\newcommand{\acs}[1]{\glspl{#1}} +\loadglsentries{carreira_phd_thesis_abr.sty} +\makeglossaries + + +% *** FLOAT SIZES AND PATH *** +\newcommand{\FigsPath}{} +\newcommand{\FIGSIZE}{0.7\columnwidth} +\newcommand{\FIGSIZEDOUBLE}{0.45\columnwidth} +\newcommand{\FIGSIZETABLE}{0.5\columnwidth} +\newcommand{\TABLESVSPACE}{\vspace{2mm}} +\newcommand{\FIGSVSPACE}{} +\newcommand{\optbold}[1]{\textbf{#1}} + +% %Options: Sonny, Lenny, Glenn, Conny, Rejne, Bjarne, Bjornstrup +% \usepackage[Bjornstrup]{fncychap} + diff --git a/carreira_phd_thesis.tex b/carreira_phd_thesis.tex new file mode 100644 index 0000000..1bd2779 --- /dev/null +++ b/carreira_phd_thesis.tex @@ -0,0 +1,91 @@ +\documentclass[phdthesis,en]{estgleiria} +% \documentclass[a4paper,twoside]{ociamthesis} + +%\pdfcompresslevel=0 +%\pdfobjcompresslevel=0 +% \usepackage{showframe} + +\usepackage{carreira_phd_thesis} + +% correct bad hyphenation here +\hyphenation{op-tical net-works semi-conduc-tor} + +%This selects the language defined in the class of the document +\selectoriginallanguage + +% \newcommand{\HIGHLIGHT}[1]{\textcolor{red}{#1}} +\newcommand{\HIGHLIGHT}[1]{\textcolor{red}{#1}} +\newcommand{\HERE}{\textcolor{red}{CHECK HERE!}} + +%\includeonly{010_Introduction/Introduction} +%\includeonly{022_ErrorResilience/RobustTransmissionReview} +%\includeonly{031_ErrorResilienceEvaluation/ErrorResilienceEvaluation} +%\includeonly{041_ImprovedErrorResilience/ImprovedErrorResilience} +%\includeonly{042_ConcealmentAwareEncoding/ConcealmentAwareEncoding} +%\includeonly{043_EndToEndRobustSystem/EndToEndRobustSystem} + +\begin{document} + +\title{Error Resilience and Concealment Techniques \\ for High Efficiency Video Coding} + +\department{Department of Electronic Engineering} + +\course{Electronic Engineering PhD} +\foreigncourse{Electronic Engineering PhD} + + +\author{Jo\~{a}o Filipe}{Monteiro Carreira} +\date{7}{2018} + +\titlefigs{\includegraphics[width=10cm]{lborolodonlogo.png}} + +% make the title area +\maketitle + +\frontmatter +% PDF exports of Word Documents available (Exported August 2012) +% Thesis Access Form +\clearpage +\includepdf[pages=-, pagecommand=]{000_Preamble/LU/access.pdf} +% Certificate of Originality +%\includepdf[pages=-, pagecommand=, templatesize={5in}{10in}]{000_Preamble/LU/origin.pdf} + +\input{000_Preamble/dedic} +\input{000_Preamble/thanks} +\input{000_Preamble/abstract} + +%\dominitoc +\tableofcontents +\listoffigures +\listoftables +\printglossary[title=List of Abbreviations] + + +\mainmatter +% \pagenumbering{gobble} + +%\setcounter{mtc}{6} +\include{010_Introduction/Introduction} + +\include{021_VideoCoding/VideoCodingReview} +\include{022_ErrorResilience/RobustTransmissionReview} +\include{031_ErrorResilienceEvaluation/ErrorResilienceEvaluation} +\include{041_ImprovedErrorResilience/ImprovedErrorResilience} +\include{042_ConcealmentAwareEncoding/ConcealmentAwareEncoding} +\include{043_EndToEndRobustSystem/EndToEndRobustSystem} + +\include{050_Conclusions/Conclusion} + +\backmatter + +\appendix +\include{060_Appendix/Sequences} +\include{060_Appendix/Papers} +%\def\extraresults{0} +%\include{060_Appendix/Results} + +\input{References/References.tex} + +% that's all folks +\end{document} + diff --git a/carreira_phd_thesis_abr.sty b/carreira_phd_thesis_abr.sty new file mode 100644 index 0000000..4a03456 --- /dev/null +++ b/carreira_phd_thesis_abr.sty @@ -0,0 +1,200 @@ +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% Generic +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\customnewacronym{uhd}{UHD}{Ultra-HD} +\customnewacronym{qoe}{QoE}{Quality of Experience} \glsunset{qoe} +\customnewacronym{qos}{QoS}{Quality of Service} \glsunset{qos} +\customnewacronym[longplural={frames per second},description=Frames per Second]{fps}{FPS}{frames per second} +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% Network +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\customnewacronym{rtp}{RTP}{Real-time Transport Protocol} \glsunset{rtp} +\customnewacronym{ip}{IP}{Internet Protocol} \glsunset{ip} +\customnewacronym{lte}{LTE}{Long Term Evolution} +\customnewacronym{mtu}{MTU}{Maximum Transfer Unit} +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% Video Coding +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\customnewacronym{mpeg}{MPEG}{Moving Picture Experts Group} \glsunset{mpeg} +\customnewacronym{vceg}{VCEG}{Video Coding Experts Group} \glsunset{vceg} + +\customnewacronym{iso}{ISO}{International Organization for Standardization} \glsunset{iso} +\customnewacronym{iec}{IEC}{International Electrotechnical Commission} \glsunset{iec} + +\customnewacronym{gop}{GOP}{Group of Pictures} +\customnewacronym{qp}{QP}{Quantized Parameter} +\customnewacronym{rd}{R-D}{\mbox{Rate-Distortion}} + +\customnewacronym{nal}{NAL}{Network Abstraction Layer} +\customnewacronym{vcl}{VCL}{Video Coding Layer} +\customnewacronym{poc}{POC}{Picture Order Count} +\customnewacronym{sei}{SEI}{Supplemental Enhancement Information} + +\customnewacronym{mv}{MV}{Motion Vector} + +\customnewacronym{dpb}{DPB}{Decoded Picture Buffer} + +\customnewacronym{dct}{DCT}{Discrete Cosine Transform} +\customnewacronym{dst}{DST}{Discrete Sine Transform} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% H.264 +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\customnewacronym{h264}{H.264/AVC}{H.264 - Advanced Video Coding} \glsunset{h264} +\customnewacronym{mb}{MB}{Macroblock} +\customnewacronym{fmo}{FMO}{Flexible Macroblock Ordering} +\customnewacronym{mmco}{MMCO}{Memory Management Control Operation} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% HEVC +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\customnewacronym{hevc}{HEVC}{High Efficiency Video Coding} + +\customnewacronym{vps}{VPS}{Video paramenter set} +\customnewacronym{sps}{SPS}{Sequence Paramenter Set} +\customnewacronym{pps}{PPS}{Picture Paramenter Set} +\customnewacronym{rpset}{RPSet}{Reference Picture Set} + +\customnewacronym{rap}{RAP}{Random-Access Point} +\customnewacronym{bla}{BLA}{Broken Link Access} +\customnewacronym{cra}{CRA}{Clean Random Access} +\customnewacronym{idr}{IDR}{Instantaneous Decoding Refresh} +\customnewacronym{radl}{RADL}{Random access decodable leading picture} +\customnewacronym{rasl}{RASL}{Random access skipped leading picture} +\customnewacronym{tsa}{TSA}{Temporal Sub-layer Access} +\customnewacronym{stsa}{STSA}{Step-wise Temporal Sub-layer Access} + + +\customnewacronym{ctu}{CTU}{Coding Tree Unit} +\customnewacronym{ctb}{CTB}{Coding Tree Block} +\customnewacronym{cu}{CU}{Coding Unit} +\customnewacronym{cb}{CB}{Coding Block} +\customnewacronym{pu}{PU}{Prediction Unit} +\customnewacronym{pb}{PU}{Prediction Block} +\customnewacronym{tu}{TU}{Transform Unit} +\customnewacronym{tb}{TB}{Transform Block} + +\customnewacronym{sao}{SAO}{Sample Adaptive Offset} +\customnewacronym{amvp}{AMVP}{Advanced Motion Vector Prediction} + +\customnewacronym{cabac}{CABAC}{Context Adaptive Binary Arithmetic Coding} + +\customnewacronym{wpp}{WPP}{Wavefront Parallel Processing} + + +\customnewacronym{tmvp}{TMVP}{Temporal Motion Vector Predictor} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% Resilience / Concealment +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\customnewacronym{ec}{EC}{Error Concealment} +\customnewacronym{fec}{FEC}{Forward Error Correction} +\customnewacronym{rir}{RIR}{Random Intra Refreshing} +\customnewacronym[longplural={Regions of Interest}]{roi}{ROI}{Region of Interest} +\customnewacronym{rope}{ROPE}{Recursive Optimal Per-Pixel Estimation} +\customnewacronym{rps}{RPS}{Reference Picture Selection} +\customnewacronym{uep}{UEP}{Unequal Error protection} +\customnewacronym{mdc}{MDC}{Multiple Description Coding} + +\customnewacronym{fc}{FC}{Frame-Copy} +\customnewacronym{mc}{MC}{Motion-Copy} +\customnewacronym{bma}{BMA}{Boundary Match Algorithms} +\customnewacronym{mve}{MVE}{Motion Vector Extrapolation} +\customnewacronym{pmve}{PMVE}{Pixel-based Motion Vector Extrapolation} +\customnewacronym{hmve}{HMVE}{Hybrid Motion Vector Extrapolation} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% Misc. +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\customnewacronym{si}{SI}{Spatial Information} +\customnewacronym{ti}{TI}{Temporal Information} + +\customnewacronym{plr}{PLR}{Packet Loss Ratio} +\customnewacronym{mse}{MSE}{Mean Square Error} +\customnewacronym{psnr}{PSNR}{Peak Signal-to-Noise Ratio} +\customnewacronym{wpsnr}{WPSNR}{Weighted Peak Signal-to-Noise Ratio} \glsunset{wpsnr} +\customnewacronym{ssim}{SSIM}{Structural Similarity Index Metric} +\customnewacronym{fsim}{FSIM}{Feature Similarity Index} + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% Experiemental Results +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\customnewacronym{ecu}{ECU}{Error Concealment Units} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% Loss types +\def\LOSTALL{$IDR_{Loss}$} +\def\KEEPINTRA{$IDR_{NoLoss}$} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% Reference methods +\def\REF{\mbox{\textit{{Ref}}}} +\def\TIDR{\mbox{\textit{TIDR}}\xspace} +\def\AVG{\mbox{\emph{Ref~\cite{ROPERefGenJan2010}}}\xspace} +\def\LONG{\mbox{\textit{Long}}\xspace} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% Constrains +\def\NOMotion{\mbox{\textit{LostMotion}}\xspace} +\def\NOCONS{\mbox{\textit{Ref}}\xspace} +% \def\BLOCKSIXTEEN{\mbox{\textit{MaxCUSize16}}\xspace} +% \def\BLOCKTHIRTYTWO{\mbox{\textit{MaxCUSize32}}\xspace} +% \def\BLOCKSIXTYFOUR{\mbox{\textit{MaxCUSize64}}\xspace} +\def\BLOCKSIXTEEN{16$\times$16} +\def\BLOCKTHIRTYTWO{32$\times$32} +\def\BLOCKSIXTYFOUR{64$\times$64} +\def\MERGE{\mbox{\textit{SingleCandidate}}\xspace} +\def\MERGETHREE{\mbox{\textit{ThreeCandidates}}\xspace} +\def\MERGEONE{\mbox{\textit{SingleCandidate}}\xspace} +\def\TMVP{\mbox{\textit{TMVP}}\xspace} +\def\NOTMVP{\mbox{\textit{NoTMVP}}\xspace} +\def\WITHSAO{Enabed} +\def\NOSAO{Disabled} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% error concealment +\def\FC{\mbox{\textit{{FC}}}\xspace} +\def\MC{\mbox{\textit{{MC}}}\xspace} +\def\MVE{\mbox{\textit{{MVE}}}\xspace} + +\def\FULLSEARCHLONG{\textit{Full Search}} +\def\DUALFLOWLONG{\textit{Dual Flow}} +\def\POLYFLOWLONG{\textit{Poly Flow}} +\def\DEEPFLOWLONG{\textit{Deep Flow}} + +\def\FULLSEARCH{\mbox{\textit{FullSearch}}\xspace} +\def\DUALFLOW{\mbox{\textit{DualFlow}}\xspace} +\def\POLYFLOW{\mbox{\textit{PolyFlow}}\xspace} +\def\DEEPFLOW{\mbox{\textit{DeepFlow}}\xspace} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% Proposed methods +\def\PROP{\mbox{\textit{{Proposed}}}\xspace} + +\def\CHKB{\mbox{\textit{Chkb}}\xspace} +\def\EXP{\mbox{\textit{Prop}}\xspace} +\def\NOREDUND{\mbox{Without MVs}\xspace} +\def\DYN{\mbox{\textit{Prop-MV}}\xspace} +\newcommand\REDUND[1]{\mbox{#1\%\ of MVs}\xspace} +\newcommand\OPTREDUND[1]{\mbox{\DYN ($\alpha$=#1)}\xspace} + +\def\FIXEPARTSIZE{\mbox{\textit{{Fixed Part}}}\xspace} +\def\FIXEDLAMBDA{\textit{\mbox{NoSaliency}}\xspace} +\def\DYNLAMBDA{\textit{\mbox{Saliency}}\xspace} + + +%% Long Labels +\def\FIXEDLAMBDALONG{\FIXEDLAMBDA \mbox{\textit{(Constant $\lambda$)}}\xspace} +\def\DYNLAMBDALONG{\DYNLAMBDA \mbox{\textit{(Adaptive $\lambda$)}}\xspace} + + + + + + + + + + diff --git a/carreira_phd_thesis_gnuplot_template.cfg b/carreira_phd_thesis_gnuplot_template.cfg new file mode 100644 index 0000000..53769bc --- /dev/null +++ b/carreira_phd_thesis_gnuplot_template.cfg @@ -0,0 +1,24 @@ +# Gnuplot template +reset +set size {1,1} +set grid +set title offset character 0,-.5,0 +set key box top right spacing 1.3 +set style line 1 lc 4 lt 4 lw 3 pt 10 ps 1 +# set style line 2 lc rgbcolor "#006400" lt 2 lw 3 pt 3 ps 1 +set style line 2 lc 2 lt 2 lw 3 pt 3 ps 1 +set style line 3 lc 3 lt 4 lw 3 pt 4 ps 1 +set style line 4 lc 1 lt 1 lw 3 pt 2 ps 1 +set style line 5 lc 7 lt 3 lw 3 pt 6 ps 1 +set style line 6 lc 8 lt 6 lw 3 pt 8 ps 1 +set style line 7 lc 2 lt 7 lw 3 pt 8 ps 1 +set style line 8 lc 3 lt 8 lw 3 pt 9 ps 1 +set style line 9 lc 4 lt 9 lw 3 pt 10 ps 1 + +#set tics format '\footnotesize $%h$' offset 0.7,0 +set tics format '\scriptsize $%h$' +# set ytics offset character -.5,0,0 + +set xlabel '\PLRLABEL' +set ylabel '\PSNRLABEL' + diff --git a/clean_files.bash b/clean_files.bash new file mode 100755 index 0000000..d2b17aa --- /dev/null +++ b/clean_files.bash @@ -0,0 +1,32 @@ +#!/bin/bash + +function clean() +{ +PATTERN_LIST=" +*.acn +*.acr +*.aux +*.alg +*.gnuploterrors +*.bbl +*.blg +*.fls +*.ist +*.log +*.lof +*lot +*.maf +*.mtc* +*.out +*.toc +*.synctex.gz +*_latexmk +" +for p in $PATTERN_LIST +do + find . -iname "${p}" -exec rm {} \; +done +[[ -d gnuplottex ]] &&rm -R gnuplottex +} + +clean diff --git a/estgleiria.cls b/estgleiria.cls new file mode 100644 index 0000000..9dad236 --- /dev/null +++ b/estgleiria.cls @@ -0,0 +1,550 @@ +%% +%% This is file `estgleiria.cls', +%% +%% This is the `estgleiria' document class used for composition of LaTeX +%% monographs (master thesis) satisfying the requirements of the Coordination +%% of Master Course in Engenharia Electrotecnica - Telecomunicacoes at Escola +%% Superior de Tecnologia e Gestao of Instituto Politecnico de Leiria. +%% +%% Copyright (C) 2011 Luis F. R. Lucas +%% Copyright (C) 2018 Joao F. M. Carreira +%% +%% This program is free software; you can redistribute it and/or modify +%% it under the terms of the GNU General Public License version 3 as +%% published by the Free Software Foundation. +%% +%% This program is distributed in the hope that it will be useful, +%% but WITHOUT ANY WARRANTY; without even the implied warranty of +%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +%% GNU General Public License version 3 for more details. +%% +%% You should have received a copy of the GNU General Public License +%% version 3 along with this package (see COPYING file). +%% If not, see . +%% +%% Author(s): +%% Luis F. R. Lucas +%% Joao F. M. Carreira +%% +%% \CharacterTable +%% {Upper-case \A\B\C\D\E\F\G\H\I\J\K\L\M\N\O\P\Q\R\S\T\U\V\W\X\Y\Z +%% Lower-case \a\b\c\d\e\f\g\h\i\j\k\l\m\n\o\p\q\r\s\t\u\v\w\x\y\z +%% Digits \0\1\2\3\4\5\6\7\8\9 +%% Exclamation \! Double quote \" Hash (number) \# +%% Dollar \$ Percent \% Ampersand \& +%% Acute accent \' Left paren \( Right paren \) +%% Asterisk \* Plus \+ Comma \, +%% Minus \- Point \. Solidus \/ +%% Colon \: Semicolon \; Less than \< +%% Equals \= Greater than \> Question mark \? +%% Commercial at \@ Left bracket \[ Backslash \\ +%% Right bracket \] Circumflex \^ Underscore \_ +%% Grave accent \` Left brace \{ Vertical bar \| +%% Right brace \} Tilde \~} +%% +\def\fileversion{v1.0} +\def\filedate{2011/06/01} +\NeedsTeXFormat{LaTeX2e}[1995/12/01] +\ProvidesClass{estgleiria}[\filedate\ \fileversion\ IPL Dissertations and Thesis] +\LoadClass[12pt,a4paper,twoside]{book} +\RequirePackage{hyphenat} +\RequirePackage{lastpage} +\RequirePackage[english]{babel} +\RequirePackage{ifthen} +\RequirePackage{graphicx} +\RequirePackage{cite} +\RequirePackage{setspace} +\RequirePackage{tabularx} +\RequirePackage{eqparbox} +\RequirePackage{ltxcmds} +% \RequirePackage{nomencl} +% \RequirePackage[a4paper,bindingoffset=0.0cm,vcentering=true,top=3cm,bottom=2.5cm,left=3.0cm,right=2cm]{geometry} +\RequirePackage[a4paper,bindingoffset=0.0cm,vcentering=true,top=3cm,bottom=2.5cm,left=3.5cm,right=2cm]{geometry} +% \RequirePackage[includeheadfoot,margin=3.5cm]{geometry} + +\def\EstgTeX{{\rm E\kern-.05em{\sc S\kern-.025em T\kern-.025em +G\kern-.025em}}\kern-.08em +T\kern-.1667em\lower.5ex\hbox{E}\kern-.125emX\spacefactor1000} +\RequirePackage{fancyhdr} +\RequirePackage{titlesec} + +% %Options: Sonny, Lenny, Glenn, Conny, Rejne, Bjarne, Bjornstrup +% \RequirePackage[Sonny]{fncychap} + +\newcommand\selectoriginallanguage{\selectlanguage{portuguese}} +\newcommand\glsknowledgmentsname{Agradecimentos} +\newcommand\listabbreviationname{Lista de Abreviaturas} +\newcommand\name@bib{Bibliografia} +\newcommand\contentsnameestg{\'Indice} +\newcommand\secondpagecourse{\local@course} +\newcommand\secondpagedate{\local@monthname\ de \number\year} + +\DeclareOption{pt}{% +\renewcommand\selectoriginallanguage{\selectlanguage{portuguese}} +\renewcommand\contentsname{\'Indice} +\renewcommand\glsknowledgmentsname{Agradecimentos} +\renewcommand\listabbreviationname{Lista de Abreviaturas} +\renewcommand\name@bib{Bibliografia} +\renewcommand\secondpagecourse{\local@course} +\renewcommand\secondpagedate{\local@monthname\ de \number\year} +} + +\DeclareOption{en}{% +\renewcommand\selectoriginallanguage{\selectlanguage{english}} +\renewcommand\contentsnameestg{Contents} +\renewcommand\glsknowledgmentsname{Acknowledgments} +\renewcommand\listabbreviationname{List of Abbreviations} +\renewcommand\name@bib{Bibliography} +\renewcommand\secondpagecourse{\foreign@course} +\renewcommand\secondpagedate{\foreign@monthname\ \number\year} +} + + +\headsep 1cm + +\renewcommand\tableofcontents{% + \cleardoublepage + \if@twocolumn + \@restonecoltrue\onecolumn + \else + \@restonecolfalse + \fi + \chapter*{\contentsnameestg + \@mkboth{% + \MakeUppercase\contentsnameestg}{\MakeUppercase\contentsnameestg}}% + \@starttoc{toc}% + \if@restonecol\twocolumn\fi + \addcontentsline{toc}{chapter}{\contentsnameestg} +} + +\newcommand{\properpagestyle}{\pagestyle{fancy}} +\renewcommand{\chaptermark}[1]% + {\markboth{\thechapter.\ #1}{}} +\renewcommand{\sectionmark}[1]% + {\markright{\thesection.\ #1}} +\renewcommand{\headrulewidth}{0.5pt} +\renewcommand{\footrulewidth}{0pt} +\fancyhf{} +\fancyhead[LE,RO]{\nouppercase{\footnotesize\scshape\thepage}} +\fancyhead[LO]{\nouppercase{\footnotesize\scshape\rightmark}} +\fancyhead[RE]{\nouppercase{\footnotesize\scshape\leftmark}} + +\def\cleardoublepage{\clearpage\if@twoside \ifodd\c@page\else +\hbox{} \vspace*{\fill} \vspace{\fill} \thispagestyle{empty} +\newpage +\if@twocolumn\hbox{}\newpage\fi\fi\fi} + +\newboolean{maledoc} +\setboolean{maledoc}{false} + +\def\insersecondtitle{} + +\newcommand\secondtitle{% +\def\insersecondtitle{\cleardoublepage\@mainmatterfalse\thispagestyle{empty}\makefrontpage}% +} + + +\DeclareOption{msc}{% + \newcommand{\@degree}{MSc} + \newcommand{\@degreename}{Mestrado} + \newcommand{\local@degname}{Mestre} + \newcommand{\foreign@degname}{Master} + \newcommand\local@doctype{\uppercase{Disserta{\c c}{\~ a}o} } + \newcommand\foreign@doctype{\local@doctype} + \DeclareOption{pt}{% + \renewcommand\foreign@doctype{\local@doctype}% + \renewcommand\foreign@degname{}% + } + \DeclareOption{en}{\renewcommand\foreign@doctype{Dissertation}} + \secondtitle +} +\DeclareOption{grad}{% + \newcommand{\@degree}{Eng.} + \newcommand{\@degreename}{Licenciatura} + \newcommand{\local@degname}{Licenciado} + \newcommand{\local@doctype}{Projecto} + \newcommand{\foreign@doctype}{Project} +} +\DeclareOption{report}{% + \newcommand{\@degreename}{Licenciatura} + \newcommand\local@doctype{Relat\'orio} + \newcommand\foreign@doctype{Report} +} + +\DeclareOption{phdconfirmation}{% + \newcommand{\@degree}{PhD} + \newcommand{\@degreename}{PhD} + \newcommand{\local@degname}{PhD} + \newcommand{\foreign@degname}{PhD} + \newcommand\local@doctype{Confirmation Report} + \newcommand\foreign@doctype{Confirmation Report} +} + +\DeclareOption{phdthesis}{% + \newcommand{\@degree}{PhD} + \newcommand{\@degreename}{PhD} + \newcommand{\local@degname}{PhD} + \newcommand{\foreign@degname}{PhD} + \newcommand\local@doctype{PhD Thesis} + \newcommand\foreign@doctype{PhD Thesis} +} + +\onehalfspacing +\DeclareOption{doublespacing}{% + \doublespacing +} +\ProcessOptions\relax +\newcommand\department[1]{% + \global\def\local@deptname{#1}% +} +\newcommand\course[1]{% + \global\def\local@course{#1}% +} +\newcommand\foreigncourse[1]{% + \global\def\foreign@course{#1}% +} +\renewcommand\title[1]{% + \global\def\local@title{#1}% +} +\newcommand\foreigntitle[1]{% + \global\def\foreign@title{#1}% +} +\newcommand\covertext[1]{% + \global\def\local@covertext{#1}% +} +\newcommand\foreigncovertext[1]{% + \global\def\foreign@covertext{#1}% +} +\renewcommand\author[2]{% + \global\def\@authname{#1} + \global\def\@authsurn{#2} +} +\renewcommand\date[2]{% + \month=#1 + \year=#2 +} +\newcommand\local@monthname{\ifcase\month\or + janeiro\or fevereiro\or mar{\c c}o\or abril\or maio\or junho\or + julho\or agosto\or setembro\or outubro\or novembro\or dezembro\fi} +\newcommand\foreign@monthname{\ifcase\month\or + January\or February\or March\or April\or May\or June\or + July\or August\or September\or October\or November\or December\fi} + + +\newcommand\ENAbstract[2]{% +\selectlanguage{english} +\chapter*{Abstract} +\addcontentsline{toc}{chapter}{Abstract} +#1 + +\vspace{10mm} +\textbf{Keywords:} #2 +} + +\renewcommand\frontmatter{% + \insersecondtitle + \cleardoublepage + \setcounter{page}{1} + \pagenumbering{roman} + \pagestyle{plain} + \titleformat{\chapter}[display] + {\normalfont\Large\bfseries} + {\sc\chaptertitlename\ \thechapter}{15pt} + {\Huge \thispagestyle{empty}} + [\vspace{5mm}\titlerule] +} +\renewcommand\mainmatter{% + \estg@mainBegin + \cleardoublepage + \@mainmattertrue +% \titleformat{\chapter}[display] +% {\normalfont\Large\bfseries} +% {\sc\chaptertitlename\ \thechapter}{15pt} +% {\Huge \thispagestyle{empty}} +% [\vspace{1cm}\titlerule\minitoc\glsresetall] + \titleformat{\chapter}[display]{\normalfont\Huge\bfseries}{\sc \chaptertitlename\ \thechapter}{20pt}{\Huge \thispagestyle{empty}}[\vspace{5mm}] + \properpagestyle + \pagenumbering{arabic}} + +\renewcommand\backmatter{% + \cleardoublepage + \addcontentsline{toc}{chapter}{\name@bib} + \titleformat{\chapter}[display] + {\normalfont\Large\bfseries\raggedleft} + {\sc\chaptertitlename\ \thechapter}{15pt} + {\Huge \thispagestyle{empty}} + [\vspace{1cm}\titlerule] + } + +\newcommand\titlefigs[1]{% + \def\inserttitlefigs{#1}% +} + +% \renewcommand\maketitle{% +% \pagestyle{empty} +% \begin{titlepage} +% \vspace*{1.5mm} +% \setlength\parskip{1mm} +% \begin{center} +% \inserttitlefigs +% \end{center} +% \vspace{1cm} +% \begin{center} +% \doublespacing +% \nohyphens{\normalsize%\uppercase\expandafter{\local@degname} +% \uppercase{\uppercase\expandafter{\local@doctype} }}\par +% \nohyphens{\large\expandafter{\local@course}}\par +% \vspace*{2cm} +% \nohyphens{\Large\bf\expandafter{\local@title}}\par +% \vspace{3cm} +% \large\rm +% \vspace*{0.5cm} +% \nohyphens{\sc\@authname\ \@authsurn}\par +% \end{center} +% +% \vspace*{2.1cm} +% \begin{flushright} +% \begin{minipage}{8.45cm} +% \end{minipage}\par +% \vspace*{7.5mm} +% \end{flushright} +% \vspace*{\fill} +% \begin{center} +% % \local@cityname, \local@monthname\ de \number\year +% \foreign@monthname\ \number\year +% \end{center} +% \end{titlepage} +% \global\let\maketitle\relax% +% \global\let\and\relax} + +\renewcommand\maketitle{% + \pagestyle{empty} + \begin{titlepage} + \vspace*{1.5mm} + \setlength\parskip{1mm} + \begin{center} + \inserttitlefigs + \end{center} + \vspace{.5cm} + \begin{center} + \doublespacing + \nohyphens{\Large\bf\expandafter{\local@title}}\par + \vspace{1.5cm} + \large\rm +% by \par + \sc\@authname\ \@authsurn\par + \vspace{1.5cm} + \bfseries A Doctoral Thesis + \end{center} + \vspace*{1cm} + \begin{flushright} + \begin{minipage}{8.6cm} + Submitted in partial fulfilment of the requirements for the award of Doctor of Philosophy of Loughborough University in London + \end{minipage}\par + \vspace*{7.5mm} + \end{flushright} + \vspace*{\fill} + \begin{center} + % \local@cityname, \local@monthname\ de \number\year + \foreign@monthname\ \number\year \par + \vspace{2cm} + \copyright\ \number\year\ by Jo\~{a}o Carreira + \end{center} + \end{titlepage} + \global\let\maketitle\relax% + \global\let\and\relax} + + \newcommand\makefrontpage{% + + \vspace*{1.5mm} + \setlength\parskip{1mm} + \begin{center} + \inserttitlefigs + \end{center} + \vspace*{1cm} + \begin{center} + \doublespacing + \nohyphens{\normalsize%\uppercase\expandafter{\local@degname} + \uppercase{\uppercase\expandafter{\foreign@degname} \uppercase\expandafter{\foreign@doctype} }}\par + \nohyphens{\large \expandafter{\secondpagecourse}}\par + \vspace*{2cm} + \nohyphens{\Large\bf\expandafter{\local@title}}\par + \vspace{3cm} + \large\rm + \vspace*{0.5cm} + \nohyphens{\sc\@authname\ \@authsurn}\par + \end{center} + + \vspace*{1cm} + \vspace*{4mm} + \frontpage@maintext + \vspace*{12mm} + \noindent \par + \begin{center} + \local@cityname, \secondpagedate + \end{center} + +%\begin{center} +%\includegraphics[width=6cm]{IT_logo_ad_hoc.pdf} +%\end{center} + + + } + +\newcommand\dedication[1]{ + \gdef\@dedic{#1} + \cleardoublepage + \thispagestyle{empty} + \vspace*{\fill} + \begin{flushright} + \begin{minipage}{60mm} + \raggedleft \it \normalsize \@dedic + \end{minipage} + \end{flushright} + \vspace*{\stretch{3}} + \cleardoublepage} + + +\renewcommand\listoffigures{% + \estg@hasLof + \if@twocolumn + \@restonecoltrue\onecolumn + \else + \@restonecolfalse + \fi + \chapter*{\listfigurename}% + \addcontentsline{toc}{chapter}{\listfigurename}% + \@mkboth{\MakeUppercase\listfigurename}% + {\MakeUppercase\listfigurename}% + \@starttoc{lof}% + \if@restonecol\twocolumn\fi + } +\renewcommand\listoftables{% + \if@twocolumn + \@restonecoltrue\onecolumn + \else + \@restonecolfalse + \fi + \chapter*{\listtablename}% + \addcontentsline{toc}{chapter}{\listtablename}% + \@mkboth{% + \MakeUppercase\listtablename}% + {\MakeUppercase\listtablename}% + \@starttoc{lot}% + \if@restonecol\twocolumn\fi + } +\newcommand\printloabbreviations{% +\renewcommand\glossaryname{\listabbreviationname}% +\@input@{\jobname.lab}} +\def\makeloabbreviations{% + \newwrite\@labfile + \immediate\openout\@labfile=\jobname.abx + \def\abbrev{\@bsphack\begingroup + \@sanitize + \@wrlab}\typeout + {Writing index of abbreviations file \jobname.abx}% + \let\makeloabbreviations\@empty +} +\@onlypreamble\makeloabbreviations +\AtBeginDocument{% +\@ifpackageloaded{hyperref}{% + \def\@wrlab#1#2{% + \protected@write\@labfile{}% + {\string\indexentry{[#1] #2|hyperpage}{\thepage}}% + \endgroup + \@esphack}}{% + \def\@wrlab#1#2{% + \protected@write\@labfile{}% + {\string\indexentry{[#1] #2}{\arabic{page}}}% + \endgroup + \@esphack}}} +\AtBeginDocument{% + \@ifpackageloaded{hyperref}{ + \def\estg@bibEnd{% + \immediate\write\@auxout{% + \string\newlabel{bib:end}{{}{\arabic{page}}{}{page.\arabic{page}}{}}}}% + \def\estg@bibBegin{% + \immediate\write\@auxout{% + \string\newlabel{bib:begin}{{}{\arabic{page}}{}{page.\arabic{page}}{}}}}% + \def\estg@mainBegin{% + \immediate\write\@auxout{% + \string\newlabel{front:pageno}{{}{\Roman{page}}{}{page.\roman{page}}{}}}}% + \def\estg@hasLof{% + \immediate\write\@auxout{% + \string\newlabel{cat:lofflag}{{}{:~il.;}{}{page.\roman{page}}{}}}}% + }{% + \def\estg@bibEnd{% + \immediate\write\@auxout{% + \string\newlabel{bib:end}{{}{\arabic{page}}{}}}}% + \def\estg@bibBegin{% + \immediate\write\@auxout{% + \string\newlabel{bib:begin}{{}{\arabic{page}}{}}}}% + \def\estg@mainBegin{% + \immediate\write\@auxout{% + \string\newlabel{front:pageno}{{}{\Roman{page}}{}}}}% + \def\estg@hasLof{% + \immediate\write\@auxout{% + \string\newlabel{cat:lofflag}{{}{:~il.;}{}}}}% + }% +} + +\newenvironment{theglossary}{% + \if@twocolumn% + \@restonecoltrue\onecolumn% + \else% + \@restonecolfalse% + \fi% + \@mkboth{\MakeUppercase\glossaryname}% + {\MakeUppercase\glossaryname}% + \chapter*{\glossaryname}% + \addcontentsline{toc}{chapter}{\glossaryname} + \list{} + {\setlength{\listparindent}{0in}% + \setlength{\labelwidth}{1.0in}% + \setlength{\leftmargin}{1.5in}% + \setlength{\labelsep}{0.5in}% + \setlength{\itemindent}{0in}}% + \sloppy}% + {\if@restonecol\twocolumn\fi% +\endlist} +\renewenvironment{theindex}{% + \if@twocolumn + \@restonecolfalse + \else + \@restonecoltrue + \fi + \twocolumn[\@makeschapterhead{\indexname}]% + \@mkboth{\MakeUppercase\indexname}% + {\MakeUppercase\indexname}% + \thispagestyle{plain}\parindent\z@ + \addcontentsline{toc}{chapter}{\indexname} + \parskip\z@ \@plus .3\p@\relax + \columnseprule \z@ + \columnsep 35\p@ + \let\item\@idxitem} + {\if@restonecol\onecolumn\else\clearpage\fi} +\newcommand\glossaryname{Glossary} +\newcommand\local@advisorstring{Orientador} +\newcommand\foreign@advisorstring{Advisor} +\newcommand\local@universityname{Instituto Polit{\'e}cnico de Leiria} +\newcommand\local@deptstring{Departamento} +\newcommand\foreign@deptstring{Department} +% \newcommand\local@cityname{Leiria} +\newcommand\local@cityname{Surrey} +\newcommand\local@countryname{Portugal} +\newcommand\frontpage@maintext{ + \noindent\sloppy\nohyphens{\local@covertext}% +} +\newcommand\frontpage@bottomtext{% + \begin{center} + %\local@cityname, \foreign@monthname\ \number\year + \end{center} + +\begin{center} + \includegraphics[width=6cm]{figs/IT_logo_ad_hoc.pdf} + \end{center} +} + +%% +%% +%% End of file `estgleiria.cls'. diff --git a/lborolodonlogo.png b/lborolodonlogo.png new file mode 100644 index 0000000..ac07ab9 Binary files /dev/null and b/lborolodonlogo.png differ