diff --git a/000_Preamble/abstract.tex b/000_Preamble/abstract.tex index 4836fa8..2f3699a 100755 --- a/000_Preamble/abstract.tex +++ b/000_Preamble/abstract.tex @@ -1,27 +1,28 @@ \selectlanguage{english} -\chapter*{\HIGHLIGHT{Abstract}} -\addcontentsline{toc}{chapter}{Abstract} +\chapter*{Abstract \VERSION{0}} +\addcontentsline{toc}{chapter}{Abstract \VERSION{0}} -% The increasing diversity of services, the demands for high quality multimedia and the growing popularity of Ultra-HD formats are pushing technologies forward and opening new research directions. -% The recently developments in video coding techniques resulted in the novel standard, referred to as High Efficiency Video Coding (HEVC). -% The high level of compression efficiency achieved by \ac{hevc} coding techniques decreases the error resilience performance under error prone conditions. -% -% In the scope of this research work the error resilience of the \ac{hevc} standard was firstly analysed. The results reveal the lack of robustness of the motion vector (MV) coding, leading to quality degradation due to error propagation. It is also shown that the temporal dependency of motion information is comparatively higher than that in the H.264/AVC standard. -% Based on this evidence, this paper proposes a method to make intelligent use of temporal motion vector (MV) candidates during the motion estimation process, in order to decrease the temporal dependency, and improve the error resiliency without penalising the rate-distortion performance. -% The simulation results show that the proposed method improves the error resilience under tested conditions, compared to the reference method that always enables temporal \ac{mv} candidates. -% -% Moreover, a method based on the prediction dependency of \acs{mv}is proposed, to select the most relevant ones for redundant coding with reduced overhead. -% The \ac{mv} dependencies are analysed in the encoder to prioritise the motion information , based on the number of subsequent dependent coding units. -% Then a subset of prioritised \acs{mv}selected with the previous is transmitted as redundancy, to reduce the use and propagation of mismatched \ac{mv} predictions in case of transmission errors or data loss. -% The simulation results show that the proposed \acs{mv} selection method can effectively identify the most relevant motion field, achieving improved error robustness with a reduced redundancy overhead. -% -% Further research will be pursed to improve the error robustness of the \ac{hevc} standard and develop novel concealment methods using combined info between the encoder and decoder-side. +\vspace{-6mm} -%The increasing diversity of services and the demand for ever higher quality multimedia at wider resolutions, which leads to the growing popularity of Ultra-HD (UHD) format, are pushing technologies forward and opening new research directions. The recent developments in video coding techniques have resulted in a novel standard called \ac{hevc} that is the next breakthrough in multimedia compression after the globally adopted MPEG-4/\ac{h264} standard. However, the significantly improved levels of compression efficiency achieved by the \ac{hevc} standard lead to an inherent reduction in the error resilience performance under error prone conditions. It should be aimed at ensuring an adequate level of protection against network and transmission induced impairments (e.g., packet losses), without sacrificing the compression efficiency thanks to the novel block level operations done by \ac{hevc}. -% -%My research work takes the \ac{hevc} standard to its core and builds new techniques on top of it to enhance its error resilience performance, and develop new processing techniques to improve decoder-side error concealment performance. The first scope of this research work is to make an extensive analysis of various error resilience techniques developed for the \ac{hevc} standard, but also for the previous video coding standards (most notably \ac{h264}); identify their shortcomings and address those shortcomings. Up to date work has revealed that the \ac{mv} coding technique lacks robustness, which leads to quality degradation due to error propagation in lossy transmission environments. It has also been seen that the temporal dependency of motion information is comparatively higher than that in the \ac{h264} standard. Based on this evidence, a method was developed to make intelligent use of temporal \ac{mv} candidates during the motion estimation process, in order to decrease the temporal dependency, and improve the error resiliency without penalising the rate-distortion performance. The simulation results have shown that the proposed method improves the error resilience under tested conditions, compared to the reference method that always enables temporal \acs{mv} candidates. Another proposed method has exploited the prediction dependency of \glspl{mv}in order to select the most relevant ones for redundant coding with reduced overhead. Simulation results have shown that the proposed \gls{mv} selection method can effectively identify the most relevant motion fields, achieving improved error robustness with a significantly reduced redundancy overhead. -% -%The second scope of this research work is to develop new processing techniques for the concealment of the coding elements lost during the transmission and delivery system, using combined information between the encoder and decoder-sides. Error concealment techniques mitigate the detrimental effects of network impairments on the perceptual multimedia quality and the performance these techniques determine the quality of a system directly. The third and the ultimate aim of this research work is to strike a balance between the designed error resilience and error concealment techniques so as to meet a set of QoS/QoE requirements associated with current and future multimedia-based services (e.g., real-time multi-party video conferencing, Ultra-HD video on demand, etc.), such as bandwidth and delay. It is envisaged to produce a set of configuration profiles for chosen services based around the developed tools for \ac{hevc}. +This thesis investigates the problem of robust coding and \ac{ec} in highly efficient video coders. +Firstly, a simulation study of the error robustness of the \ac{hevc} is presented, focusing on the new tools adopted. Experimental results reveal the lack of protection against errors, which leads to significant quality degradation. +Based on this evidence, a new method to reduce the temporal dependencies of \ac{mv} is proposed, improving the decoded video quality without compromising the compression efficiency. + +The second contribution of this thesis is a two-stage approach for reducing the mismatch temporal predictions. +At the encoder stage, the reference pictures are dynamically distributed based on a constrained Lagrangian optimization, thus reducing the number of predictions from a single reference. +At the streaming stage, a prioritization algorithm, based on spatial dependencies, selects a reduced set of MVs to be transmitted, as side information, to reduce mismatched MV predictions at the decoder. +Results show that this method achieves higher quality that the reference HEVC for low-packet-loss rates, also outperforms existing techniques. + +The problem of EC-aware video coding is also investigated to enhance the overall error robustness. +A new approach based on a scalable coding and optimally EC selection is proposed, where the optimal EC modes are found by simulating transmission losses, followed by a saliency-weighted optimisation of the signalling~rate/EC~distortion cost. +Moreover, EC residual information is encoded using a rate-controlled \ac{el}. +Both are transmitted to the decoder to be used in case of data loss. +Simulation results show that the proposed method achieves consistent quality gains in comparison with reference methods, which can be increased by using the EL information. + + +Finally, an adaptive error resilience scheme is proposed to dynamically predict the video stream that achieves the highest decoded quality for a particular loss case. +A pre-trained neural network selects among the various video streams encoded with different levels of compression efficiency and error protection, based on different input parameters that include information from the video signal, the coded stream and the transmission network. +Results show the advantage of using a dynamic approach to achieve a better trade-off between coding efficiency and error robustness. diff --git a/000_Preamble/thanks.tex b/000_Preamble/thanks.tex index 0609779..0aa14e9 100755 --- a/000_Preamble/thanks.tex +++ b/000_Preamble/thanks.tex @@ -1,19 +1,19 @@ \chapter*{\glsknowledgmentsname} \addcontentsline{toc}{chapter}{\glsknowledgmentsname} -Firstly, I would like to express my gratitude to my advisors, Dr. Pedro António Amado Assunção, Dr. Sérgio Manuel Maciel de Faria, Dr Erhan Ekmekcioglu and Dr. Ahmet Kondoz for the tireless support of my PhD studies and related research, for their motivation, patience, and immense knowledge. I am thankful for their guidance and availability that were essential for the proper +Firstly, I would like to express my gratitude to my advisors, Dr. Pedro Ant\'{o}nio Amado Assun\c{c}\~{a}o, Dr. S\'{e}rgio Manuel Maciel de Faria, Dr Erhan Ekmekcioglu and Dr. Ahmet Kondoz for the tireless support of my PhD studies and related research, for their motivation, patience, and immense knowledge. I am thankful for their guidance and availability that were essential for the proper conduct of this research work. I would also like to acknowledge Dr. Varuna de Silva for the initial support on this research~work. I would like to thank Funda\c{c}\~{a}o para a Ci\^{e}ncia e a Tecnologia (FCT), which provided the financial support for this PhD, under the PhD grant SFRH/BD/86368/2012. -I am also grateful to the Instituto de Telecomunicações and to the Institute for Digital Technologies, which provided the computer resources and laboratory facilities with excellent conditions for my research work. +I am also grateful to the Instituto de Telecomunica\c{c}\~{o}es and to the Institute for Digital Technologies, which provided the computer resources and laboratory facilities with excellent conditions for my research work. -I would like to acknowledge my research colleagues of Instituto de Telecomunicações~-~Leiria, in special Lucas, João, Ricardo, Nuno and Rodolfo for the friendship and the wonderful work environment, for sharing new ideas, and also for the moments outside the laboratory. +I would like to acknowledge my research colleagues, both in Portugal and United Kingdom, for the friendship and the wonderful work environment, for sharing new ideas, and also for the moments outside the laboratory. I also thank all my friends, that accompanied me during this journey, for the distracting and social moments. -I must thank all communities around the world that contribute for the development of many free and open-source tools used in this research work, namely the GNU/Linux, KDE software, Ubuntu and Gentoo communities. I also acknowledge the partnership of Luis Lucas in the Calyp software, and his contribution to the Remoterun scripts and the general computer management, which improved our developer capabilities and ease the experimental evaluations. +I must thank all communities around the world that contribute for the development of many free and open-source tools used in this research work, namely the GNU/Linux, KDE software, Ubuntu and Gentoo communities. I also acknowledge the help of Luis Lucas and Jo\~{a}o Santos in the Calyp software, Remoterun scripts and general computer management, which improved our developing capabilities and ease the experimental evaluations. -A special thanks to my parents, António and Maria, and my sister Marta to whom I owe all that I have become. Also, I must a acknowledge my uncles and cousins living in the United Kingdom for the kind welcome into their home, which significantly ease my research experience abroad. +A special thanks to my parents, Ant\'{o}nio and Maria, and my sister Marta to whom I owe all that I have become. Also, I must acknowledge my uncles and cousins living in the United Kingdom for the kind welcome into their home, which significantly ease my research experience abroad. Last but not least, I also would like to express a special gratitude to Bianca Pires for her unconditional support and love during this journey. diff --git a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex index 555df2a..a64c531 100644 --- a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex +++ b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex @@ -15,6 +15,7 @@ \section{Introduction} The use of \ac{hevc} in different type of application and video services over networks that are prone to errors and packet losses, requires mechanisms to minimise the effect of such probabilistic events. This chapter starts by presenting an evaluation study about the error resilience of \ac{hevc} in Section~\ref{Section3:ErrorResilienceEvaluation} , with the aim of defining the weak elements of coded streams and the level of error robustness under different transmission loss conditions. The evaluation study covers different coding configuration to evaluate the individual impact of some new coding tools introduced, e.g., increased partition size, intra-refreshing, slice partitioning and advanced motion vector prediction. + In Section~\ref{Section3:DynamicMVRefreshHEVC} a novel method to increase the error robustness of the \ac{hevc} \ac{mv} coding is presented and compared to existing techniques. The proposed method introduces refresh point in the motion prediction used in the \ac{hevc}, which significantly reduces the impact of mismatch \ac{mv} predictions. Finally, Section~\ref{Section3:Summary} concludes the chapter. diff --git a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex index e5650fc..6502b3b 100644 --- a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex +++ b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex @@ -13,16 +13,16 @@ \glsunset{ec} \glsreset{nn} -The effectiveness of different error resilience techniques is rather variable depending on the video content, the transmitted bitrate, network conditions, among others. +The effectiveness of different error resilience techniques is rather variable depending on the video content, the transmitted bitrate, network conditions, among others. Therefore, it is not straightforward to select a single technique which achieves the highest efficiency for every possible scenario and also for the whole range of networking conditions. This chapter addresses this problem by proposing an adaptive network that optimises the efficiency of the techniques previously described in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}. At the encoder-side, a pre-trained neural network predicts the video stream that is able to achieve the highest decoded quality for a particular loss cases, among the various video streams encoded with different levels of compression efficiency and error robustness. -The selection is based on different input parameters that include information from the video signal, the coded stream and the transmission network. +The selection is based on different input parameters that include information from the video signal, the coded stream and the transmission network. Using this approach the overall performance of the proposed adaptive method significantly increases, yielding a better trade-off between compression efficiency and error robustness. -This chapter is organised as follows. +This chapter is organised as follows. Section~\ref{Section6:Problem} presents the problem addressed in this chapter and experimental evidence, as motivation for the method investigated. -Section~\ref{Section6:ProposedMethod} provides an overview of architecture that supports the implementation of the proposed method and Section~\ref{Section6:OptimisationMethod} describes the technical details. +Section~\ref{Section6:ProposedMethod} provides an overview of architecture that supports the implementation of the proposed method and Section~\ref{Section6:OptimisationMethod} describes the technical details. Section~\ref{Section6:ExperimentalResults} presents the performance evaluation and a discussion of results. Finally, Section~\ref{Section6:Summary} concludes the chapter. @@ -33,21 +33,21 @@ Finally, Section~\ref{Section6:Summary} concludes the chapter. \section{The problem of adaptive error robustness} \label{Section6:Problem} -In the scope of this thesis different robust coding techniques were investigated to increase the overall performance of video transmission under error-prone conditions. -Different algorithms were devised in the proposed methods, such as, reference frame selection, motion vector redundancies and error concealment signalling. +In the scope of this thesis different robust coding techniques were investigated to increase the overall performance of video transmission under error-prone conditions. +Different algorithms were devised in the proposed methods, such as, reference frame selection, motion vector redundancies and error concealment signalling. In previous chapters, these methods were evaluated against existing techniques, revealing superior performance and better efficiency to deal with error robustness problems in high efficient video coding. Overall, the performance results of these methods have demonstrated that they can efficiently increase the robustness of video streams against transmission errors and losses, leading to higher decoded video quality. -However, each method on its own does not adapt to different conditions. +However, each method on its own does not adapt to different conditions. While the reference frame selection algorithm presented in Chapter~\ref{Chapter:ErrorResilienceTechniques} introduces low overhead in comparison with existing techniques, the method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} achieves higher quality gains at cost of slightly higher bitrate increase. -Therefore, one may conclude that each method may be further optimised for different applications, video characteristics and network conditions. +Therefore, one may conclude that each method may be further optimised for different applications, video characteristics and network conditions. This chapter advances one step forward , by investigating a joint robust coding method using different techniques to optimise the robust coder across different conditions, such as, the input content and the network information. -This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and can be selected based on the current conditions. +This scheme can be used in transmission systems where different video bitstreams are available at the encoder-side and can be selected based on the current conditions. \subsection{Comparison of the proposed methods} -In order to address the above problem of adaptive robust coding, a simulation study was carried out to compare the error robustness performance of the previously proposed methods for different conditions. +In order to address the above problem of adaptive robust coding, a simulation study was carried out to compare the error robustness performance of the previously proposed methods for different conditions. This study extends the previous evaluations presented in Chapter~\ref{Chapter:ErrorResilienceTechniques}~and~\ref{Chapter:ConcealmentAwareTechniques}, as they target a wider range of test material and network conditions. Moreover, it aims at comparing the performance of different techniques across a variety of video contents, bitrates and loss conditions, in order to find relevant insights which affect the decoded video quality. Consequently, these relevant insights may be used to develop an adaptive robust coding method. @@ -55,17 +55,19 @@ Moreover, it aims at comparing the performance of different techniques across a The following methods were evaluated in the simulation study, namely: \begin{itemize} \item the default \ac{hevc} decoder with frame-copy \ac{ec} (\REF), as reference method; - \item the default \ac{hevc} encoder without robust coding-techniques and motion vector extrapolation \ac{ec} (\MVE); - \item the proposed reference frame selection scheme (\EXP); - \item the \EXP method in combination with an \ac{ec} optimisation with 10\% of bitrate allocated for the scalable enhancement layer (\EXPELTEN); - \item the same as above, with 20\% of bitrate allocated for the scalable enhancement layer (\EXPELTWEN). + \item the default \ac{hevc} encoder without robust coding-techniques and motion vector extrapolation \ac{ec} (\MEC); + \item the proposed reference frame selection scheme (\MEXP); + \item the \MEXP method in combination with an \ac{ec} optimisation with 10\% of bitrate allocated for the scalable enhancement layer (\MEXPELTEN); + \item the same as above, with 20\% of bitrate allocated for the scalable enhancement layer (\MEXPELTWEN). \end{itemize} These methods were selected as they are the main techniques proposed within this research work and cover two aspects of a robust video transmission: (i) robust video encoding by dynamically select the reference frames and (ii) robust video decoding by improving the error concealment performance. \input{043_EndToEndRobustSystem/Results/AdaptiveSequenceList.tex} -Table~\ref{Table6:TestMaterialCharacteristics} presents the list of test sequences used. These sequences were selected to cover a wide range of video content, with different levels of spatial and temporal complexity, as demonstrated by the values of \ac{si} and \ac{ti}~\cite{ITUTP9102008}, respectively. -Moreover, from the bitstream encoded with the reference implementation of the \ac{hevc} encoder, the percentage of the inter-coded blocks and the average \ac{mv} magnitude was extracted and presented in the table (columns 4 and 5, respectively). These values were extracted to compare them against the performance obtained with the proposed robust coding methods. +Table~\ref{Table6:TestMaterialCharacteristics} presents the list of test sequences used. +These sequences were selected to cover a wide range of video content, with different levels of spatial and temporal complexity, as demonstrated by the values of \ac{si} and \ac{ti}~\cite{ITUTP9102008}, respectively. +Moreover, from the bitstream encoded with the reference implementation of the \ac{hevc} encoder, the percentage of the inter-coded blocks and the average \ac{mv} magnitude was extracted and presented in the table (columns 4 and 5, respectively). +These parameters were considered as they might be related with the performance obtained with the proposed robust coding methods. The remaining configuration was kept as described in Section~\ref{Section3:ErrorResilienceEvaluation}. %The experimental methodology carried out in this section follows the same conditions described in Section~\ref{Section3:ErrorResilienceEvaluation}. @@ -79,11 +81,12 @@ The remaining configuration was kept as described in Section~\ref{Section3:Error \label{Figure6:AverageQualityMain} \end{figure} - -Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} shows the average decoded video quality \ac{psnr} obtained from 50 trials, for different loss conditions. -Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate listed in Table~\ref{Table6:TestMaterialCharacteristics}, referred to as main bitrate. +Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} shows the average decoded video quality \ac{psnr} obtained from 50 trials, for different \acs{plr}. +Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate listed in Table~\ref{Table6:TestMaterialCharacteristics}, referred to as main bitrate. Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained by reducing the main bitrate by 40\%. This is used to evaluate the impact of the network bitrate on the decoded video quality. The results shown in the figures corresponds to the same test sequences, in order to carry out an accurate comparison. + + \begin{figure}[!t] \centering \input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex} @@ -92,13 +95,14 @@ Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained by reducin \label{Figure6:AverageQualityReduction} \end{figure} -Firstly, comparing the results for different bitrates, it is noticeable that lower quality is obtained for the lower bitrates, which results from the higher compression ratios used. It is also noticed that lower quality gains, when comparing with the reference \ac{hevc} encoder (\REF), are obtained for the video streams encoded with a 40\% reduction of bitrate. This indicates that for lower bitrates the performance of the proposed methods decreases, especially for the \EXPELTWEN case, where 20\% of the bitrate is allocated for the enhancement layer. Therefore, for lower bitrates, using methods that have a lower impact on the coding efficiency (e.g., \EXP) or even disable the robust coding techniques (\MVE) may lead to higher performance. +Firstly, comparing the results for different bitrates, it is noticeable that lower quality is obtained for the lower bitrates, which results from the higher compression ratios used. It is also noticed that lower quality gains, when comparing with the reference \ac{hevc} encoder (\REF), are obtained for the video streams encoded with a 40\% reduction of bitrate. This indicates that for lower bitrates the performance of the proposed methods decreases, especially for the \MEXPELTWEN case, where 20\% of the bitrate is allocated for the enhancement layer. Therefore, for lower bitrates, using methods that have a lower impact on the coding efficiency (e.g., \MEXP) or even disable the robust coding techniques (\MEC) may lead to higher performance. -Secondly, comparing the results across different video content, i.e., different test sequences, one can notice that the performance of the different methods is not consistent. -For instance, in the result of Figure~\ref{Figure6:AverageQualityMain}, for Tennis sequence the \EXPELTWEN clearly outperforms the remaining methods, however, for the BQSquare sequence higher performance is achieved by using less bitrate allocated for the \ac{el} (i.e., 10\%). This is a consequence of the higher quality degradation due to higher motion in Tennis sequence (see results for TI and \ac{mv} size in Table~\ref{Table6:TestMaterialCharacteristics}) which required higher redundant information to increase the overall quality. +Secondly, comparing the results across different video content, i.e., different test sequences, one can notice that the performance of the different methods is not consistent. +For instance, in the result of Figure~\ref{Figure6:AverageQualityMain}, for Tennis sequence the \MEXPELTWEN clearly outperforms the remaining methods, however, for the BQSquare sequence higher performance is achieved by using less bitrate (i.e., 10\%) allocated for the \ac{el} (\MEXPELTEN). +This is a consequence of the higher quality degradation due to higher motion in Tennis sequence (see results for TI and \ac{mv} size in Table~\ref{Table6:TestMaterialCharacteristics}) which required higher redundant information to increase the overall quality. %\HIGHLIGHT{Mention SI!!} -Finally, comparing the results for all test sequences and both bitrates it can be clearly noticed that the \acs{plr} have a significant impact on the performance of the error robustness techniques. In case of the proposed reference frame selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions. For the \EXPELTWEN case, higher \acs{plr} are required to be able to outperform both the reference \ac{hevc} and the \ac{mve} method. +Finally, comparing the results for all test sequences and both bitrates it can be clearly noticed that the \acs{plr} have a significant impact on the performance of the error robustness techniques. In case of the proposed reference frame selection method (\EXP), quality gains in comparison with the \REF case are achieved for lower loss ratio, i.e., \acs{plr} as low as 2\% for all test conditions. For the \MEXPELTWEN case, higher \acs{plr} are required to be able to outperform both the reference \ac{hevc} and the \ac{mve} method. Summarising, the performance of the proposed methods is not similar across different video content and bitrates, opening research to adaptively choose the best method that should be applied to achieve the highest decoded video quality. In order to achieve increased efficiency and higher video quality, the best trade-off between auxiliary information and video bitrate can be found by selecting the optimal robust coding algorithm. @@ -116,7 +120,7 @@ The main objective of this robust coding scheme it to achieve an optimal trade-o \centering %\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram} \input{043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz} - \caption{Adaptive error robustness architecture for 4 streams (S1...S4).} + \caption{Adaptive error robustness architecture for 4 streams (\SMEC...\SMEXPELTWEN).} \label{Figure6:BlockDiagram} \FIGSVSPACE \end{figure} @@ -125,16 +129,16 @@ Figure~\ref{Figure6:BlockDiagram} illustrates the proposed architecture. Firstly, the input signals are coded using different levels of error protections. Then, an optimisation algorithm is used to decide which bitstream should be transmitted at any given instant, based on different parameters. The selected stream is expected to obtain the highest quality at the decoder-side for the given conditions. A reference video coder is used to compress the video signal using highly efficient coding techniques, obtaining the best \ac{rd} performance for error-free environments. This is used as the reference bitstream. -Subsequently, a transcoding operation is applied to this reference bitstream aided by the input video signal to efficiently select the reference frames used for inter prediction. +Subsequently, a transcoding operation is applied to this reference bitstream aided by the input video signal to efficiently select the reference frames used for inter prediction. The reference frame selection mechanism proposed in Chapter~\ref{Chapter:ErrorResilienceTechniques} is used for this purpose. Moreover, loss simulation and a saliency-based \ac{ec} optimisation are also used in the robust coding process, which enables the selection of the optimal \ac{ec} algorithm to be applied at the decoder. -Finally, since the optimal \ac{ec} mode might not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el} using two different bitrates. In this case 10\% and 20\% of the total bitrate were chosen, providing two different levels of error robustness. +Finally, since the optimal \ac{ec} mode might not be able to correctly reconstruct all the missing regions, extra residual information is multiplexed in the stream as a scalable \ac{el} using two different bitrates. In this case 10\% and 20\% of the total bitrate were chosen, providing two different levels of error robustness. The \ac{ec} optimisation and \ac{el} coding is performed as described in Chapter~\ref{Chapter:ConcealmentAwareTechniques}. -The proposed approach results in four different bitstreams with different characteristics (referred to as "S1" to "S4", respectively, as shown in Figure~\ref{Figure6:BlockDiagram}): -(i) S1: reference stream without error robust coding; -(ii) S2: robust video coding using reference frame selection; -(iii) S3: robust video coding using selective reference frames and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue; -(iv) S4: same as the previous bitstream but with 20\% bitrate allocated for the scalable~\ac{el}. +The proposed approach results in four different bitstreams with different characteristics (referred to as \SMEC to \SMEXPELTWEN, as shown in Figure~\ref{Figure6:BlockDiagram}): +(i) \SMEC: reference stream without error robust coding; +(ii) \SMEXP: robust video coding using reference frame selection; +(iii) \SMEXPELTEN: robust video coding using selective reference frames and optimal \ac{ec} method signalling plus 10\% of bitrate allocated for the scalable enhancement layer \ac{ec} residue; +(iv) \SMEXPELTWEN: same as the previous bitstream but with 20\% bitrate allocated for the scalable~\ac{el}. Finally, the video stream with the optimal error robustness is selected using an algorithm based on different parameters extracted from the video signal, the compressed bitstream and feedback from the transmission channel. From the input video signal, relevant information regarding the spatial and temporal complexity is computed using existing metrics. Moreover, further information is also extracted from the coded bitstream using a stream parser. @@ -142,7 +146,7 @@ Finally, the loss ratio of the transmission channel is also used. These parameters are fed into the optimisation block to select the best bitstream for transmission. The optimisation process is described in detail in the following section. -The main purpose of this architecture is to allow dynamic stream switching during real time transmission. +The main purpose of this architecture is to allow dynamic stream switching during real time transmission. To accomplish this without causing drift at the decoder, the switching procedure should take place at refresh points, i.e., \ac{idr} and \ac{cra} frames, which correspond to clean random access points. In the proposed scheme, \ac{idr} frames are introduced every 16 frames, resulting in between 1 to 3 switching points every second, depending on the frame rate of the video signal. @@ -154,7 +158,7 @@ To accomplish this without causing drift at the decoder, the switching procedure \label{Section6:OptimisationMethod} In this section the optimisation method used to select a robust video stream is presented. -The selection method is based on a shallow neural network which predicts the optimal stream based on different parameters. +The selection method is based on a shallow neural network which predicts the optimal stream based on different parameters. This section describes both the input parameters and the neural network used in the optimal stream selection process. Moreover, a description data set used to train and evaluate the neural network used is also provided. \subsection{Data set} @@ -163,7 +167,7 @@ This section describes both the input parameters and the neural network used in \def\NumCases{910\xspace} -The proposed method includes an optimisation stage that uses a neural network classifier, in order to select the optimal stream to be used for transmission. +The proposed method includes an optimisation stage that uses a neural network classifier, in order to select the optimal stream to be used for transmission. This stream is able to obtain the highest quality for a particular transmission scenario. Therefore, several test cases were simulated in order to obtain results for a wide range of conditions. These tests are mainly characterised by three elements: \begin{itemize} @@ -173,8 +177,8 @@ Therefore, several test cases were simulated in order to obtain results for a wi \end{itemize} To obtain the data set, 14 different test sequences with different scene content were used, as listed in Table~\ref{Table6:TestMaterialCharacteristics}. Those sequences were encoded with the main bitrate (see Table~\ref{Table6:TestMaterialCharacteristics}), as well as four different others, obtained by reducing the main bitrate by 10\%, 20\%, 30\% and 40\%. -Then, the following \acs{plr} were used to cover a wide range of loss ratios giving more detail to low \acs{plr}: 0\%, 0.5\%, 1.0\%, 1.2\%, 1.6\%, 2.0\%, 2.2\%, 2.4\%, 2.6\%, 2.8\%, 3.0\%, 5.0\% and 10.0\%. -This results in a total of \NumCases test conditions. +Then, the following \acs{plr} were used to cover a wide range of loss ratios giving more detail to low \acs{plr}: 0\%, 0.5\%, 1.0\%, 1.2\%, 1.6\%, 2.0\%, 2.2\%, 2.4\%, 2.6\%, 2.8\%, 3.0\%, 5.0\% and 10.0\%. +This results in a total of \NumCases test conditions. \subsection{Input parameters} @@ -182,7 +186,7 @@ This results in a total of \NumCases test conditions. \glsreset{ti} A set of different parameters is used to predict the optimal stream that achieves the highest quality in specific network conditions. -Based on these inputs the proposed scheme is able to select the appropriate robust coding scheme which should be used. +Based on these inputs the proposed scheme is able to select the appropriate robust coding scheme which should be used. These parameters aim at characterise three different aspects of the video transmission system; (i) input video (two parameters), (ii) video stream (three parameters) and (iii) transmission network (one parameter). %These parameters aim at covering three different aspects of the video transmission: two parameters from the input video, three parameters from the video stream and one parameter from the transmission network. From the input video the two parameters are measured in a frame by frame basis, while for the video stream information a stream parser is applied to the reference bitstream that was compressed only based optimal \ac{rd} optimisation without considering any robust coding techniques, as shown in Figure~\ref{Figure6:BlockDiagram}. Finally, the parameter related with the transmission network can be obtained using feedback information from the decoder. @@ -191,8 +195,8 @@ From the input video the two parameters are measured in a frame by frame basis, The characterisation of these three aspects of the video transmission results in a total of six parameters: \begin{enumerate} - -\item \textit{\ac{si}}: parameter expressing the amount of spatial detail, which consequently affects the error recovery operations, leading to higher quality degradation~\cite{ITUTP9102008}. + +\item \textit{\ac{si}}: parameter expressing the amount of spatial detail, which consequently affects the error recovery operations, leading to higher quality degradation~\cite{ITUTP9102008}. \item \textit{\ac{ti}}: parameter expressing the amount of difference between temporally adjacent frames~\cite{ITUTP9102008}. This has a direct relation to the impact of errors, and thus the performance of the error resilience techniques, as shown in the results of Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction}, which were discussed in Section~\ref{Section6:Problem}. @@ -201,12 +205,12 @@ This is an indication of the distribution of spatial and temporal dependencies. This is calculated as the ratio of inter-coded blocks over intra-coded ones. Normally, coded streams with higher number of temporal dependencies are more prone to error propagation, thus higher number of bits should be allocated for error protection \cite{UEPScalable_TCSVT2010}. -\item \textit{AVG\_MV}: for the inter-coded regions, the motion information is analysed and the average \ac{mv} magnitude is calculated to provide information about the motion intensity. +\item \textit{AVG\_MV}: for the inter-coded regions, the motion information is analysed and the average \ac{mv} magnitude is calculated to provide information about the motion intensity. As shown in previous studies, motion-based \ac{ec} algorithms are less efficient for video signals with higher motion intensity, as \acs{mv} are less predictable using algorithms, such as \ac{mc} or \ac{mve} \cite{HybridConcealmentTB2008}. \item \textit{Bitrate}: information regarding the average bit allocation per frames is calculated, since it will affect the video quality and the amount of bits that can be available for auxiliary information. This value is expressed in Mbps, as illustrated in Table~\ref{Table6:TestMaterialCharacteristics}. -\item \textit{\ac{plr}:} feedback information from the network to characterise the transmission losses, expressed by the percentage of packets loss. This is used since for higher loss ratio there is a clear increase in the quality degradation, which affects the performance of the robust coding techniques. +\item \textit{\ac{plr}:} feedback information from the network to characterise the transmission losses, expressed by the percentage of packets loss. This is used since for higher loss ratio there is a clear increase in the quality degradation, which affects the performance of the robust coding techniques. Transmission networks with higher loss ratios require a higher bit allocation for robust techniques, therefore the error resilience technique should adapt based on the value of the \ac{plr}. @@ -217,10 +221,10 @@ Transmission networks with higher loss ratios require a higher bit allocation fo %These parameters are obtained as described in Chapter~\ref{Chapter:HEVCErrorResilience}, and aim at evaluating the spatial complexity and motion complexity of the video scenes, which have relevant impact on the error robustness of the coded streams. %As shown, test material with higher \ac{si} or \ac{ti} suffer from higher quality degradation as the \ac{plr} increases, e.g., Book Arrival and Tennis sequences. -These parameters were compared against the optimal stream, selected from S1 to S4. -This aims at finding the correlation between the input parameters and the output variable, i.e., target stream. -In this study the streams are characterised by a number between 1 and 4. -Table~\ref{Table6:Correlation} shows the absolute Person Correlation coefficient obtained across all sequences, bitrates and \acs{plr}. +These parameters were compared against the optimal stream, selected from \SMEC to \SMEXPELTWEN. +This aims at finding the correlation between the input parameters and the output variable, i.e., target stream. +In this study the streams are characterised by a number between 1 and 4. +Table~\ref{Table6:Correlation} shows the absolute Person Correlation coefficient obtained across all sequences, bitrates and \acs{plr}. The table shows the results for the parameters previously described, as well as, a new parameter obtained from the average magnitude of the optical flow measured using the method in~\cite{DeepFlow2013} (\textit{OPT\_FLOW}). Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable. As both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method, as it achieves a higher Person Correlation coefficient. @@ -233,27 +237,19 @@ Results in Table~\ref{Table6:Correlation} show that the selected parameters are \begin{tabular}{p{4cm}x{4cm}} \toprule Parameter & Absolute Pearson Correlation \\ \hline - % \textit{SI} & -0.1553 \\ \hline - % \textit{TI} & 0.6079 \\ \hline - % \textit{Inter (percentage)} & -0.6487 \\ \hline - % \textit{OPT\_FLOW} & 0.6953 \\ \hline - % \textit{AVG\_MV} & 0.4647 \\ \hline - % \textit{Bitrate} & 0.3580 \\ \hline - % \textit{PLR} & 0.5755 \\ \hline - \textit{SI} & 0.3953 \\ \hline \textit{TI} & 0.6079 \\ \hline \textit{Inter (percentage)} & 0.6487 \\ \hline \textit{OPT\_FLOW} & 0.1553 \\ \hline \textit{AVG\_MV} & 0.4647 \\ \hline \textit{Bitrate} & 0.3580 \\ \hline - \textit{PLR} & 0.5755 \\ \hline - \end{tabular} + \textit{PLR} & 0.5755 \\ \hline + \end{tabular} \end{table} % Podia se citar este paper -% \cite{IntraRefreshICIP2009} +% \cite{IntraRefreshICIP2009} \subsection{Neural network} @@ -265,11 +261,11 @@ Mainly, a \ac{nn} consists of three different layers: the input layer which rece In the past \acs{nn} have been widely used in video processing applications such as, quality evaluation~\cite{VideoQualityNN_TNN2006}, video analysis~\cite{VideoAnalysisNN_2018} and video coding~\cite{RateDistortionNN_VCIP2017,HEVCComplexityNN_TIP2018}. In the proposed method a feed-forward network is used, which connects every neuron in a given layer to the neurons of the subsequent layer. -Figure~\ref{FIG6:NeuralNetwork} shows the diagram of the \ac{nn} used in the proposed method. +Figure~\ref{FIG6:NeuralNetwork} shows the diagram of the \ac{nn} used in the proposed method. The network contains and input and output layer, as well as, a single hidden layer. The input layer receives the six inputs, as described in the previous sub-section. These inputs are correlated with the optimisation variable being maximised, i.e., the output video quality. Subsequently, the network transforms the input data through a series of neurons in the hidden layer. -The output layer comprises four neurons corresponding to each available video stream from S1 to S4 (different error resilience techniques). +The output layer comprises four neurons corresponding to each available video stream from \SMEC to \SMEXPELTWEN (different error resilience techniques). Finally, the output with the highest value is selected and the corresponding stream is used for transmission, since it is the one expected to achieve the highest video quality for a particular scenario. \begin{figure}[tb] @@ -294,10 +290,10 @@ The remaining 254 conditions are used to evaluate the accuracy of the proposed m Table~\ref{Table6:ConfusionTraining} shows the confusion matrix of the \ac{nn} training phase. -In this matrix each column corresponds to the target result, i.e., the cases where the coded streams, S1 to S4, would lead to the highest decoded video quality. -Similarly, each row corresponds to the \ac{nn} selected by the trained networks. -Thus, those cases where the horizontal and vertical labels match, indicate that the network produced the correct output. -The remaining positions correspond to the erroneous outputs. +In this matrix each column corresponds to the target result, i.e., the cases where the coded streams, \SMEC to \SMEXPELTWEN, would lead to the highest decoded video quality. +Similarly, each row corresponds to the \ac{nn} selected by the trained networks. +Thus, those cases where the horizontal and vertical labels match, indicate that the network produced the correct output. +The remaining positions correspond to the erroneous outputs. The last line shows the accuracy for each target output. These results correspond to the dataset used to train the \ac{nn}. @@ -310,18 +306,18 @@ These results correspond to the dataset used to train the \ac{nn}. \begin{tabular}{x{.5cm}x{1cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}} & & \multicolumn{4}{c}{Target} \\ \multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} - & & S1 & S2 & S3 & S4 \\ \hline - & S1 & 122 & 17 & 3 & 0 \\ - \cline{2-6} & S2 & 13 & 80 & 19 & 2 \\ - \cline{2-6} & S3 & 0 & 12 & 96 & 9 \\ - \cline{2-6} & S4 & 0 & 4 & 11 & 202 \\ + & & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN \\ \hline + & \SMEC & 122 & 17 & 3 & 0 \\ + \cline{2-6} & \SMEXP & 13 & 80 & 19 & 2 \\ + \cline{2-6} & \SMEXPELTEN & 0 & 12 & 96 & 9 \\ + \cline{2-6} & \SMEXPELTWEN & 0 & 4 & 11 & 202 \\ \hline \multicolumn{2}{x{1.7cm}|}{Accuracy} & 90.3\% & 70.8\% & 74.4\% & 94.8\% \\ \end{tabular} \end{table} -Results in Table~\ref{Table6:ConfusionTraining} shows that streams S1 (i.e., without robust coding techniques) and S4 (i.e., using optimal \ac{ec} with 20\% of bitrate in the \ac{el}) lead to the highest quality in the majority of the test cases (are more often selected). -The \ac{nn} is able to achieve a high accuracy for S1 and S4 streams, correctly selecting the output stream in more than 90\% of cases. Regarding the remaining robust streams (i.e., S2 and S3), an acceptable accuracy of 70.8\% and 74.4\% is achieved, respectively. This reveals that the \ac{nn} used in the proposed method is able to predict the coded stream which leads to the highest video quality across different conditions with acceptable accuracy. +Results in Table~\ref{Table6:ConfusionTraining} shows that streams \SMEC (i.e., without robust coding techniques) and \SMEXPELTWEN (i.e., using optimal \ac{ec} with 20\% of bitrate in the \ac{el}) lead to the highest quality in the majority of the test cases (are more often selected). +The \ac{nn} is able to achieve a high accuracy for \SMEC and \SMEXPELTWEN streams, correctly selecting the output stream in more than 90\% of cases. Regarding the remaining robust streams (i.e., \SMEXP and \SMEXPELTEN), an acceptable accuracy of 70.8\% and 74.4\% is achieved, respectively. This reveals that the \ac{nn} used in the proposed method is able to predict the coded stream which leads to the highest video quality across different conditions with acceptable accuracy. \begin{table}[tb] \centering @@ -332,21 +328,21 @@ The \ac{nn} is able to achieve a high accuracy for S1 and S4 streams, correctly \begin{tabular}{x{.5cm}x{1cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}|x{1.3cm}} & & \multicolumn{4}{c}{Target} \\ \multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{3.8cm}{\centering Selected}}} - & & S1 & S2 & S3 & S4 \\ \hline - & S1 & 56 & 7 & 2 & 0 \\ - \cline{2-6} & S2 & 10 & 28 & 6 & 2 \\ - \cline{2-6} & S3 & 1 & 5 & 31 & 8 \\ - \cline{2-6} & S4 & 0 & 5 & 7 & 86 \\ + & & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN \\ \hline + & \SMEC & 56 & 7 & 2 & 0 \\ + \cline{2-6} & \SMEXP & 10 & 28 & 6 & 2 \\ + \cline{2-6} & \SMEXPELTEN & 1 & 5 & 31 & 8 \\ + \cline{2-6} & \SMEXPELTWEN & 0 & 5 & 7 & 86 \\ \hline \multicolumn{2}{x{1.7cm}|}{Accuracy} & 83.6\% & 62.2\% & 67.4\% & 89.6\% \\ %\bottomrule \end{tabular} \end{table} -To further evaluate the performance of the \ac{nn}, the confusion matrix of the data set used for testing is shown in the Table~\ref{Table6:ConfusionTest}. +To further evaluate the performance of the \ac{nn}, the confusion matrix of the data set used for testing is shown in the Table~\ref{Table6:ConfusionTest}. This table shows the same results as Table~\ref{Table6:ConfusionTraining}, but for different test cases, which were not used in the training phase. % Therefore, the trained network is not aware of the test cases used for validation. -Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 62.2\% with a maximum value of 89.6\% when the S4 is the target stream. Although a lower accuracy is obtained using the validation test cases, the overall accuracy is still acceptable, since it allows to select the optimal robust coding technique (i.e., S1 to S4) for the majority of the conditions. +Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 62.2\% with a maximum value of 89.6\% when the \SMEXPELTWEN is the target stream. Although a lower accuracy is obtained using the validation test cases, the overall accuracy is still acceptable, since it allows to select the optimal robust coding technique (i.e., \SMEC to \SMEXPELTWEN) for the majority of the conditions. In summary, the results presented in this sub-section reveal that the input parameters used in the proposed \ac{nn} are correlated with the decoded video quality, as they can effectively be used to perform data categorisation used in the selection of the optimal error robustness coding technique for the \ac{hevc} standard. \section{Performance evaluation} @@ -356,10 +352,10 @@ In summary, the results presented in this sub-section reveal that the input para In this section the performance of the proposed method is evaluated and discussed. The overall goal of the proposed method is to achieve improved quality using a combination of different robust coding techniques adaptively selected using a feed-forward \ac{nn}. -The error resilience techniques comprising the proposed method are used in the context of the \ac{hevc} standard and compared with the reference implementation (\REF). -The quality obtained with the proposed optimal error robustness scheme (\PROP) is compared against the reference \ac{hevc} (\REF) and the methods used to generate the streams S1 to S4. +The error resilience techniques comprising the proposed method are used in the context of the \ac{hevc} standard and compared with the reference implementation (\REF). +The quality obtained with the proposed optimal error robustness scheme (\PROP) is compared against the reference \ac{hevc} (\REF) and the methods used to generate the streams \SMEC to \SMEXPELTWEN. -\begin{figure}[!t] +\begin{figure}[!t] \centering \input{043_EndToEndRobustSystem/Results/PlotQualty_vs_PLR_Predicted.tex} \vspace{-2mm} @@ -367,12 +363,12 @@ The quality obtained with the proposed optimal error robustness scheme (\PROP) i \label{Figure6:ProposedQuality} \end{figure} -Figure~\ref{Figure6:ProposedQuality} shows the quality obtained for the proposed method and the \REF case for the main target bitrate shown in Table~\ref{TableTestSequences}. Moreover, the quality obtained with the target stream (\BEST) is also shown. +Figure~\ref{Figure6:ProposedQuality} shows the quality obtained for the proposed method and the \REF case for the main target bitrate shown in Table~\ref{TableTestSequences}. Moreover, the quality obtained with the target stream (\BEST) is also shown. One should note that the \BEST case corresponds to the optimal selection, which can only be performed after quality evaluation at the decoder, therefore it is not suitable for real time applications. -It is only used to establish the optimal case for comparison purposes. -The results in the figure allows to evaluate the impact in the decoded quality of the selection used in the proposed method and compare it with the optimal case. +It is only used to establish the optimal case for comparison purposes. +The results in the figure allows to evaluate the impact in the decoded quality of the selection used in the proposed method and compare it with the optimal case. -As shown in Figure~\ref{Figure6:ProposedQuality}, the proposed method clearly outperforms the reference \ac{hevc} for all \acs{plr}. +As shown in Figure~\ref{Figure6:ProposedQuality}, the proposed method clearly outperforms the reference \ac{hevc} for all \acs{plr}. Moreover, in the error-free case (\ac{plr}=0), it leads to imperceptible quality reduction, since the proposed approach is able to select the optimal stream, thus disabling any robust coding technique when the transmission network does not have packet losses. In case of data loss, the results show that the proposed method is able to match the quality obtained by the \BEST case in most cases, revealing that the proposed method is able to efficiently select the optimal stream for transmission. @@ -385,9 +381,8 @@ In case of data loss, the results show that the proposed method is able to match \begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} \toprule \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\ - & S1 & S2 & S3 & S4 & \PROP \\ \hline + & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN & \PROP \\ \hline Basketball Drill & +1.61 & +2.66 & +2.48 & +2.44 & +2.75 \\ \hline -%Basketball Drive & +2.17 & +3.46 & +3.22 & +3.63 & +3.69 \\ \hline Book Arrival & +0.96 & +1.66 & +1.43 & +1.59 & +1.74 \\ \hline BQSquare & +0.92 & +0.96 & +0.80 & +0.62 & +1.09 \\ \hline Cactus & +1.28 & +2.04 & +1.89 & +1.96 & +2.09 \\ \hline @@ -408,22 +403,22 @@ Tennis & +1.15 & +2.24 & +2.17 & +3.12 & +3.18 \\ \hline \draw[red, line width=1.2pt] (3,3.6) ellipse (2.3cm and 0.3cm); \draw[blue, line width=1.2pt] (6.7,1.45) ellipse (.85cm and 0.25cm); \draw[blue, line width=1.2pt] (6.7,4.10) ellipse (.85cm and 0.25cm); -\end{tikzpicture} +\end{tikzpicture} \end{table} Subsequently, a comparison between the different methods is performed, in order to evaluate the quality gains obtained with the proposed method in comparison with the different techniques. -%for every test case. -Table~\ref{Table6:AverageGains} shows the average quality gains in comparison with the reference \ac{hevc} obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams S1 to S4. The average quality gains are measured across all test cases, i.e., different bitrates and different loss ratios. +%for every test case. +Table~\ref{Table6:AverageGains} shows the average quality gains in comparison with the reference \ac{hevc} obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams \SMEC to \SMEXPELTWEN. The average quality gains are measured across all test cases, i.e., different bitrates and different loss ratios. -Results in Table~\ref{Table6:AverageGains} shows that the robust coded streams S1 to S4 are able to achieve acceptable average quality gains up to 1.51 dB (see results for S2). -However, since the implementation of robust coding techniques decreases the coding efficiency, it also leads to lower quality than the reference \ac{hevc}. This is noticeable in the results obtained for the Four People sequence in the streams S2 to S4 and for the Kristen and Sara sequence in the streams S3 and S4 (see red ellipses). +Results in Table~\ref{Table6:AverageGains} shows that the robust coded streams \SMEC to \SMEXPELTWEN are able to achieve acceptable average quality gains up to 1.51 dB (see results for \SMEXP). +However, since the implementation of robust coding techniques decreases the coding efficiency, it also leads to lower quality than the reference \ac{hevc}. This is noticeable in the results obtained for the Four People sequence in the streams \SMEXP to \SMEXPELTWEN and for the Kristen and Sara sequence in the streams \SMEXPELTEN and \SMEXPELTWEN (see red ellipses). The overall performance of such methods reveals that they are not suitable for transmission conditions with low \acs{plr}, e.g., lower than 3\% (higher number of conditions have \acs{plr} lower than 3\%). -On the contrary, the proposed method is always able to outperform the reference \ac{hevc} and achieve quality gains of at least 0.25 dB for the Four People sequence. +On the contrary, the proposed method is always able to outperform the reference \ac{hevc} and achieve quality gains of at least 0.25 dB for the Four People sequence. Moreover, for the Kimono and Race Horses sequences, it is able to achieve the highest quality gains, of up 3.41 dB and 4.39 dB, respectively (see blue ellipses). Summarising, by adaptively selecting the video stream, and consequently the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness, achieving an average quality gain across different bitrates and \acs{plr} of 1.88 dB. Finally, the performance evaluation of the proposed method is evaluated for a video sequence, which was not fed into the \ac{nn} training phase. This study aims at finding the accuracy of the optimal stream selection in case of a new video content. For this purpose the Basketball Drive sequence was randomly selected, and not used in previous simulations. -Table~\ref{Table6:AverageGainBasketBallDrive} shows the average quality results obtained with this sequence, similarly to Table~\ref{Table6:AverageGains}. -These results confirm that the proposed method is able to select the optimal robust stream to be used for transmission, even when new video content is transmitted. The proposed method achieves a quality gain of 3.55 dB in comparison with the reference \ac{hevc}, outperforming the quality achieved with the stream S4 (i.e., 3.45 dB). +Table~\ref{Table6:AverageGainBasketBallDrive} shows the average quality results obtained with this sequence, similarly to Table~\ref{Table6:AverageGains}. +These results confirm that the proposed method is able to select the optimal robust stream to be used for transmission, even when new video content is transmitted. The proposed method achieves a quality gain of 3.55 dB in comparison with the reference \ac{hevc}, outperforming the quality achieved with the stream \SMEXPELTWEN (i.e., 3.45 dB). This confirms the advantage of the proposed method to adaptively select among different streams with different levels of error protections. \begin{table}[tb] @@ -435,8 +430,8 @@ This confirms the advantage of the proposed method to adaptively select among di \begin{tabular}{p{3cm}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}|x{\colwidth}} \toprule \multirow{2}{*}{\vbox{\centering \textbf{Sequence}}} & \multicolumn{5}{c}{\textbf{Robust coded stream}} \\ - & S1 & S2 & S3 & S4 & \PROP \\ \hline - Basketball Drive & +2.10 & +2.68 & +3.36 & +3.45 & +3.55 \\ + & \SMEC & \SMEXP & \SMEXPELTEN & \SMEXPELTWEN & \PROP \\ \hline + Basketball Drive & +2.10 & +2.68 & +3.36 & +3.45 & +3.55 \\ \bottomrule \end{tabular} \end{table} @@ -447,7 +442,7 @@ This confirms the advantage of the proposed method to adaptively select among di \section{Summary} \label{Section6:Summary} -In this chapter an adaptive error robustness method is proposed, based on reference frame selection and \ac{ec}-aware optimisation. By adaptively selecting a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the reference techniques. +In this chapter an adaptive error robustness method is proposed, based on reference frame selection and \ac{ec}-aware optimisation. By adaptively selecting a different error robustness technique at the encoder-side using a pre-trained \ac{nn}, higher quality gains are achieved when comparing against the reference techniques. %Moreover, the proposed method is able to outperform those cases where a single error resilience technique is used. The overall results demonstrate that the proposed method is able to increase the reconstruction video quality in case of errors. Also, by adapting the transmitted stream to the network conditions, a better trade-off between coding efficiency and error robustness can be achieved. diff --git a/043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz b/043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz index 441995d..ca81da9 100644 --- a/043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz +++ b/043_EndToEndRobustSystem/Figures/JointlyRobustnessDiagram.tikz @@ -79,10 +79,10 @@ \tikzstyle{OutputLabels} = [pos=1, anchor=south east, text width=2cm, align=right] -\path[MainArrow] (Encoder) -- (Mux.south |- Encoder) node[OutputLabels] {{S1}}; -\path[MainArrow] (EXP) -- (Mux.south |- EXP) node[OutputLabels] {{S2}}; -\path[MainArrow] (Scalable1) -- (Mux.south |- Scalable1) node[OutputLabels] {{S3}}; -\path[MainArrow] (Scalable0) -- (Mux.south |- Scalable0) node[OutputLabels] {{S4}}; +\path[MainArrow] (Encoder) -- (Mux.south |- Encoder) node[OutputLabels] {\SMEC}; +\path[MainArrow] (EXP) -- (Mux.south |- EXP) node[OutputLabels] {\SMEXP}; +\path[MainArrow] (Scalable1) -- (Mux.south |- Scalable1) node[OutputLabels] {\SMEXPELTEN}; +\path[MainArrow] (Scalable0) -- (Mux.south |- Scalable0) node[OutputLabels] {\SMEXPELTWEN}; \path[MainArrow] (Mux) -- (Out) node[above, midway, text width=2cm, text centered] {Optimal \\ Robust Steam}; \node [text centered, text width=1.5cm, minimum height=1cm, below=.3 of Mux.east, xshift=1.9cm] (Channel) {Channel Conditions}; diff --git a/043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz b/043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz index 413d799..0ef103a 100644 --- a/043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz +++ b/043_EndToEndRobustSystem/Figures/NeuralNetwork.tikz @@ -39,7 +39,7 @@ \foreach \out in {1,...,4} \path[draw] (O-\out) -- (O-O) node[pos=.3,above] {\scriptsize S\out}; - \path[draw] (O-O) -- ++(1.5,0) node[pos=0, above, anchor=south west] {S1...S4}; + \path[draw] (O-O) -- ++(1.5,0) node[pos=0, above, anchor=south west] {\SMEC...\SMEXPELTWEN}; % Annotate the layers \node[annot,above of=H-1, node distance=.8cm] (hl) {Hidden}; diff --git a/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex b/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex index 5fb2973..e19b44a 100644 --- a/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex +++ b/043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex @@ -5,12 +5,12 @@ load 'carreira_phd_thesis_gnuplot_template.cfg' set multiplot layout 3,2 -set key bottom left Left reverse width 8.5 +set key bottom left Left reverse width 1 set ytics 1, 2,100 set yrange[29:41] set title '\TitleFont Basketball Drill' -plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MEC', '-' using 2:3 w lp ls 3 title '\KeyFont \MEXP', '-' using 2:3 w lp ls 4 title '\KeyFont \MEXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \MEXPELTWEN' "EF" 0.0 39.900 "0.5%" 0.5 38.057 "1.0%" 1.0 36.949 @@ -62,7 +62,7 @@ set autoscale y set ytics 1, 2,100 set yrange[33:41] set title '\TitleFont Book Arrival' -plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MEC', '-' using 2:3 w lp ls 3 title '\KeyFont \MEXP', '-' using 2:3 w lp ls 4 title '\KeyFont \MEXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \MEXPELTWEN' "EF" 0.0 40.280 "0.5%" 0.5 39.122 "1.0%" 1.0 38.578 @@ -113,7 +113,7 @@ set autoscale y set ytics 2, 2, 100 set yrange[32:40] set title '\TitleFont BQSquare' -plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MEC', '-' using 2:3 w lp ls 3 title '\KeyFont \MEXP', '-' using 2:3 w lp ls 4 title '\KeyFont \MEXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \MEXPELTWEN' "EF" 0.0 39.300 "0.5%" 0.5 38.381 "1.0%" 1.0 37.908 @@ -164,7 +164,7 @@ set autoscale y set ytics 1, 2,100 set yrange[33:42] set title '\TitleFont Kendo' -plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MEC', '-' using 2:3 w lp ls 3 title '\KeyFont \MEXP', '-' using 2:3 w lp ls 4 title '\KeyFont \MEXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \MEXPELTWEN' "EF" 0.0 41.150 "0.5%" 0.5 40.679 "1.0%" 1.0 39.544 @@ -216,7 +216,7 @@ set autoscale y set ytics 1, 2,100 set yrange[31:40] set title '\TitleFont Park Scene' -plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MEC', '-' using 2:3 w lp ls 3 title '\KeyFont \MEXP', '-' using 2:3 w lp ls 4 title '\KeyFont \MEXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \MEXPELTWEN' "EF" 0.0 39.470 "0.5%" 0.5 38.067 "1.0%" 1.0 36.969 @@ -268,7 +268,7 @@ set autoscale y set ytics 2, 2,100 set yrange[28:41] set title '\TitleFont Tennis' -plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MEC', '-' using 2:3 w lp ls 3 title '\KeyFont \MEXP', '-' using 2:3 w lp ls 4 title '\KeyFont \MEXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \MEXPELTWEN' "EF" 0.0 40.680 "0.5%" 0.5 38.243 "1.0%" 1.0 36.815 diff --git a/043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex b/043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex index 64ea211..36b6e70 100644 --- a/043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex +++ b/043_EndToEndRobustSystem/Results/Quality_PLR_Methods_Reduced.tex @@ -5,12 +5,12 @@ load 'carreira_phd_thesis_gnuplot_template.cfg' set multiplot layout 3,2 -set key bottom left Left reverse width 9 +set key bottom left Left reverse width 1 set ytics 1, 2, 100 set yrange[29:41] set title '\TitleFont Basketball Drill' -plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MEC', '-' using 2:3 w lp ls 3 title '\KeyFont \MEXP', '-' using 2:3 w lp ls 4 title '\KeyFont \MEXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \MEXPELTWEN' "EF" 0.0 37.570 "0.5%" 0.5 36.658 "1.0%" 1.0 35.927 @@ -47,7 +47,7 @@ e "3.0%" 3.0 35.991 "5.0%" 5.0 35.431 e -"EF" 0.0 36.130 +"EF" 0.0 36.130s "0.5%" 0.5 36.038 "1.0%" 1.0 35.975 "1.6%" 1.6 35.885 @@ -61,7 +61,7 @@ set autoscale y set ytics 1, 2, 100 set yrange[33:41] set title '\TitleFont Book Arrival' -plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MEC', '-' using 2:3 w lp ls 3 title '\KeyFont \MEXP', '-' using 2:3 w lp ls 4 title '\KeyFont \MEXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \MEXPELTWEN' "EF" 0.0 38.920 "0.5%" 0.5 38.641 "1.0%" 1.0 37.769 @@ -112,7 +112,7 @@ set autoscale y set ytics 2, 2,100 set yrange[32:40] set title '\TitleFont BQSquare' -plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MEC', '-' using 2:3 w lp ls 3 title '\KeyFont \MEXP', '-' using 2:3 w lp ls 4 title '\KeyFont \MEXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \MEXPELTWEN' "EF" 0.0 36.780 "0.5%" 0.5 36.370 "1.0%" 1.0 36.129 @@ -164,7 +164,7 @@ set autoscale y set ytics 1, 2, 100 set yrange[33:42] set title '\TitleFont Kendo' -plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MEC', '-' using 2:3 w lp ls 3 title '\KeyFont \MEXP', '-' using 2:3 w lp ls 4 title '\KeyFont \MEXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \MEXPELTWEN' "EF" 0.0 38.730 "0.5%" 0.5 38.365 "1.0%" 1.0 37.714 @@ -214,7 +214,7 @@ set autoscale y set ytics 1, 2,100 set yrange[31:40] set title '\TitleFont Park Scene' -plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MEC', '-' using 2:3 w lp ls 3 title '\KeyFont \MEXP', '-' using 2:3 w lp ls 4 title '\KeyFont \MEXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \MEXPELTWEN' "EF" 0.0 37.810 "0.5%" 0.5 36.969 "1.0%" 1.0 36.189 @@ -265,7 +265,7 @@ set autoscale y set ytics 2, 2,100 set yrange[28:41] set title '\TitleFont Tennis' -plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MVE', '-' using 2:3 w lp ls 3 title '\KeyFont \EXP', '-' using 2:3 w lp ls 4 title '\KeyFont \EXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \EXPELTWEN' +plot '-' using 2:3 w lp ls 1 title '\KeyFont \REF', '-' using 2:3 w lp ls 2 title '\KeyFont \MEC', '-' using 2:3 w lp ls 3 title '\KeyFont \MEXP', '-' using 2:3 w lp ls 4 title '\KeyFont \MEXPELTEN', '-' using 2:3 w lp ls 5 title '\KeyFont \MEXPELTWEN' "EF" 0.0 39.020 "0.5%" 0.5 37.569 "1.0%" 1.0 36.453 diff --git a/050_Conclusions/Conclusion.tex b/050_Conclusions/Conclusion.tex index 99a4df1..2d3940e 100755 --- a/050_Conclusions/Conclusion.tex +++ b/050_Conclusions/Conclusion.tex @@ -58,4 +58,24 @@ As a final remark, one may conclude that the use of novel methods, proposed in t This was jointly achieved by achieving implementing robust video coding techniques with efficient error concealment algorithms optimised at the encoder. -\subsection*{Future work} \ No newline at end of file +\subsection*{Future work} + +In this research work different methods were proposed to increase the error robustness of video streams and to enhance the quality achieved by \ac{ec} methods are the decoder. +In the context of this thesis there are also some suggestions for future research directions which can be used to strength the overall performance. + +In the context of robust coding techniques, one can combine existing error resilience techniques, e.g., intra-refreshing constrains, in the Lagrangian optimisation used in the reference picture selection proposed in Chapter~\ref{Chapter:HEVCErrorResilience}. +This way it would be possible to further prevent error propagation by introducing extra refresh points. +Moreover, one can also use more advanced constrains, such as, end-to-end based optimisations, to achieve a more efficient distribution of the use of reference frames. + +Regarding the \ac{ec}-aware method proposed in Chapter~\ref{Chapter:ConcealmentAwareTechniques} there are also some suggestions of future research directions. Firstly, as the proposed does not constrain the \ac{ec} candidate, one can use more advanced algorithms, in order to improve the reconstruction accuracy. This can be adopted either by replacing the existing methods, but also extending beyond four candidates. +Alternatively, to reduce the computational complexity overhead at the encoder, instead of measuring the \ac{rd} cost for each method, pre-trained classification algorithms could be used to predict the optimal \ac{ec} mode for each block. +As the complexity of the decoder is also a relevant issue, due to the increasing of mobile and battery-based devices, the optimal \ac{ec} selection should also take into account the end devices, thus penalising high complexity algorithms (e.g., algorithms requiring an estimated motion field) in case of constraining devices. +The performance of the adaptive selection of the robust coding technique could also be improved by combining different classification algorithms, based on more complex neural network architectures. + +The error robustness schemes proposed in this research work could be extended to cope with robust transmission of multi-view video, where error propagate in the temporal and inter-layer domains. +Finally, as a new coding standard is currently being developed, targeting the deployment of higher-quality video services and emerging applications such as 360-degree omnidirectional immersive multimedia and high-dynamic-range video, it is emergent to study the impact of errors in such video applications and propose efficient techniques to provide high quality video to the end users. + + + + + diff --git a/carreira_phd_thesis.tex b/carreira_phd_thesis.tex index 8091cc8..67572dc 100644 --- a/carreira_phd_thesis.tex +++ b/carreira_phd_thesis.tex @@ -20,6 +20,7 @@ \newcommand{\HERE}{\textcolor{red}{CHECK HERE!}} \newcommand{\VERSION}[1]{\textcolor{red}{(r#1)}} +%\includeonly{000_Preamble/abstract} %\includeonly{010_Introduction/Introduction} %\includeonly{021_VideoCoding/VideoCodingReview} %\includeonly{022_ErrorResilience/RobustTransmissionReview} @@ -40,7 +41,7 @@ \author{Jo\~{a}o Filipe}{Monteiro Carreira} -\date{7}{2018} +\date{9}{2018} \titlefigs{\includegraphics[width=10cm]{lborolodonlogo.png}} @@ -52,9 +53,9 @@ \frontmatter % Certificate of Originality %\clearpage -\includepdf[pagecommand=]{000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf} +\includepdf[pages=1,pagecommand=]{000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf} %\cleardoublepage -%\includepdf[pages=2, pagecommand=]{000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf} +\includepdf[pages=2, pagecommand=]{000_Preamble/LU/Certificate_Originality_Thesis_Access.pdf} \input{000_Preamble/dedic} \input{000_Preamble/thanks} diff --git a/carreira_phd_thesis_abr.sty b/carreira_phd_thesis_abr.sty index cd5ac73..4dda5f8 100644 --- a/carreira_phd_thesis_abr.sty +++ b/carreira_phd_thesis_abr.sty @@ -198,8 +198,19 @@ \def\FIXEDLAMBDA{\textit{\mbox{NoSaliency}}\xspace} \def\DYNLAMBDA{\textit{\mbox{Saliency}}\xspace} -\def\EXPELTEN{\textit{\mbox{Exp+EL10\%}}\xspace} -\def\EXPELTWEN{\textit{\mbox{Exp+EL20\%}}\xspace} + +%\def\MEXPELTEN{\textit{\mbox{Exp+EL10\%}}\xspace} +%\def\MEXPELTWEN{\textit{\mbox{Exp+EL20\%}}\xspace} +\def\MEC{\textit{\mbox{M1}}} +\def\MEXP{\textit{\mbox{M2}}} +\def\MEXPELTEN{\textit{\mbox{M3}}\xspace} +\def\MEXPELTWEN{\textit{\mbox{M4}}\xspace} + +\def\SMEC{{\mbox{S1}}\xspace} +\def\SMEXP{{\mbox{S2}}\xspace} +\def\SMEXPELTEN{{\mbox{S3}}\xspace} +\def\SMEXPELTWEN{{\mbox{S4}}\xspace} + \def\BEST{\textit{\mbox{Target}}\xspace}