From e7388f2314783e4a6a93db753a316ff5cb8b05a2 Mon Sep 17 00:00:00 2001 From: Joao Carreira Date: Tue, 18 Sep 2018 00:52:08 +0100 Subject: [PATCH] Final version 0 --- 000_Preamble/abstract.tex | 6 +- 021_VideoCoding/VideoCodingReview.tex | 15 ++-- .../ErrorResilienceEvaluation.tex | 41 +++++----- .../ImprovedErrorResilience.tex | 29 +++---- .../EndToEndRobustSystem.tex | 78 +++++++++---------- 050_Conclusions/Conclusion.tex | 6 +- 060_Appendix/Papers.tex | 7 +- carreira_phd_thesis.tex | 4 +- 8 files changed, 88 insertions(+), 98 deletions(-) diff --git a/000_Preamble/abstract.tex b/000_Preamble/abstract.tex index b4ee5ad..1026a8d 100755 --- a/000_Preamble/abstract.tex +++ b/000_Preamble/abstract.tex @@ -7,8 +7,7 @@ \vspace{-5mm} This thesis investigates the problem of robust coding and \ac{ec} in highly efficient video coders. -After a review of the current state-of-art, a simulation study of the error robustness of the \ac{hevc} is presented. -Experimental results reveal the lack of protection against errors, which leads to significant quality degradation. +After a review of the current state-of-art, a simulation study about error robustness, revealed that the \ac{hevc} has weak protection against transmission errors with significant impact on video quality degradation. Based on this evidence, the first contribution of this work is a new method to reduce the temporal dependencies between \acs{mv}, by improving the decoded video quality without compromising the compression efficiency. The second contribution of this thesis is a two-stage approach for reducing the mismatch of temporal predictions in case of video streams received with errors or lost data. @@ -22,7 +21,8 @@ Both are transmitted to the decoder to be used in case of data loss. Finally, an adaptive error resilience scheme is proposed to dynamically predict the video stream that achieves the highest decoded quality for a particular loss case. A pre-trained neural network selects among the various video streams, encoded with different levels of compression efficiency and error protection, based on different input parameters that include information from the video signal, the coded stream and the transmission network. -The new robust video coding method investigated in this thesis yield consistent quality gains in comparison with other existing methods and also the ones implemented in the \ac{hevc} reference software. + +Overall, the new robust video coding methods investigated in this thesis yield consistent quality gains in comparison with other existing methods and also the ones implemented in the \ac{hevc} reference software. Furthermore, the trade-off between coding efficiency and error robustness is also better in the proposed methods. %\vspace{2mm} diff --git a/021_VideoCoding/VideoCodingReview.tex b/021_VideoCoding/VideoCodingReview.tex index dde72c2..3f81850 100755 --- a/021_VideoCoding/VideoCodingReview.tex +++ b/021_VideoCoding/VideoCodingReview.tex @@ -182,6 +182,11 @@ The \ac{pps} is responsible to carry information that may change for every pictu \subsection{Picture types} +The \ac{hevc} standard supports several pictures types, as show in Table~\ref{Table2:PictureTypes}, which are classified in the following categories: (i) random-access points, (ii) leading pictures and (iii) temporal sub-layer access pictures~\cite{HEVCHighLevelReferenceDez2012,BookHEVC2014}. +The use of different types of pictures allows the \ac{hevc} stream to be more flexible, support a wider range of applications and achieve robust video communications. +They are important to perform random-access, which is a critical feature for channel switching, seek operations and dynamic streaming services. +Moreover, introducing random-access points into the bitstream increases its error robustness as they provide anchor points for a clean decoding after data loss or synchronisation problems. + \begin{table}[tb] \centering \caption{Picture categories in HEVC standard.}\label{Table2:PictureTypes} @@ -201,14 +206,14 @@ The \ac{pps} is responsible to carry information that may change for every pictu BLA & Broken link access & Leading pictures cannot be decoded \\ - + \midrule \midrule \multicolumn{3}{c}{\textbf{Leading pictures}} \\ \midrule RADL & \multicolumn{2}{l}{Random access decodable picture}\\ \midrule RASL & \multicolumn{2}{l}{Random access skipped picture}\\ - + \midrule\midrule \multicolumn{3}{c}{\textbf{Temporal sub-layer access pictures}} \\ \midrule @@ -223,12 +228,6 @@ The \ac{pps} is responsible to carry information that may change for every pictu \end{tabular} \end{table} -The \ac{hevc} standard supports several pictures types, as show in Table~\ref{Table2:PictureTypes}, which are classified in the following categories: (i) random-access points, (ii) leading pictures and (iii) temporal sub-layer access pictures~\cite{HEVCHighLevelReferenceDez2012,BookHEVC2014}. -The use of different types of pictures allows the \ac{hevc} stream to be more flexible, support a wider range of applications and achieve robust video communications. -They are important to perform random-access, which is a critical feature for channel switching, seek operations and dynamic streaming services. -Moreover, introducing random-access points into the bitstream increases its error robustness as they provide anchor points for a clean decoding after data loss or synchronisation problems. - - \subsubsection{Random-Access Point} The \ac{rap} are pictures used to provide access points in the bitstream with no dependencies, where it is possible to start the decoding process. The standard defines that a conforming bitstream must start with a \ac{rap} picture. diff --git a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex index 59121bd..07c3360 100644 --- a/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex +++ b/031_ErrorResilienceEvaluation/ErrorResilienceEvaluation.tex @@ -119,14 +119,8 @@ This is noticeable in the results of the BQSquare sequence (i.e., highest ratio Overall, comparing the results for the error-free and data loss cases, it is noticeable that similar quality gains are obtained,mostly due to the superior coding efficiency of larger \acs{cu}. In most cases the quality gains slightly decrease for increasing \acs{plr}. Therefore, one may conclude that the error robustness of \ac{hevc} streams is not affected by limiting the maximum \ac{cu} size. -%\todo{Complete discussion of partition unit size} - -%Results for the case of data loss indicate that using larger coding units leads to higher video quality. -% -%that larger coding units should be maintained. Even though larger frame partitions lead to larger coding dependencies, it will not affect the robustness to packet loss and error propagation. This is mainly due to the fact that error propagation occurs mostly in complex and moving regions, which normally are coded using smaller partition units. - -\begin{table}[tbp] +\begin{table}[!t] \centering \caption{Average \ac{psnr} for different maximum CU size.} \label{Table3:Quality_BlockSize} @@ -134,7 +128,8 @@ Therefore, one may conclude that the error robustness of \ac{hevc} streams is no \input{./031_ErrorResilienceEvaluation/QualityEvaluation/TableCUSizes.tex} % \vspace{6mm} \end{table} -\begin{figure}[tbp] + +\begin{figure}[!t] \centering \renewcommand{\FIGSIZE}{0.38\textwidth} % \resizebox{\columnwidth}{!}{ @@ -278,13 +273,6 @@ Moreover, the slice partitioning mode used only 1200 bytes per slice, which intr \subsubsection{Temporal \ac{mv} candidates} -The second set of experiments aimed to find the influence of losing temporal motion predictors on the error resilience of \ac{hevc}. -In order to perform this test, a single frame loss in enforced, and the error propagation is evaluated for two configurations: (i) with the temporal \ac{mv} candidate enabled and (ii) disabled. -Figure~\ref{Figure3:PSNRvsFrameErrorFreeConc} illustrates the error propagation, when a single frame is lost. For this particular experiment an intra-refreshing period of 32 frames was used to evaluate the impact on long error propagation periods. -The results show that when the \ac{tmvp} mode is enabled, the loss of Frame \#6 leads to significant reduction in the reconstruction quality of the subsequent frames. -On the contrary, when the temporal \ac{mv} candidate is disabled, the motion information on subsequent frames is not affected by the loss of a single frame and significantly higher quality is obtained. -This is the effect of breaking the temporal dependencies between \acs{mv}, which prevents erroneous \acs{mv} to propagate throughout the set of dependent frames. - \begin{figure}[!t] \centering \renewcommand{\FIGSIZE}{0.76\textwidth} @@ -295,6 +283,21 @@ This is the effect of breaking the temporal dependencies between \acs{mv}, which \end{figure} +The second set of experiments aimed to find the influence of losing temporal motion predictors on the error resilience of \ac{hevc}. +In order to perform this test, a single frame loss in enforced, and the error propagation is evaluated for two configurations: (i) with the temporal \ac{mv} candidate enabled and (ii) disabled. +Figure~\ref{Figure3:PSNRvsFrameErrorFreeConc} illustrates the error propagation, when a single frame is lost. For this particular experiment an intra-refreshing period of 32 frames was used to evaluate the impact on long error propagation periods. +The results show that when the \ac{tmvp} mode is enabled, the loss of Frame \#6 leads to significant reduction in the reconstruction quality of the subsequent frames. +On the contrary, when the temporal \ac{mv} candidate is disabled, the motion information on subsequent frames is not affected by the loss of a single frame and significantly higher quality is obtained. +This is the effect of breaking the temporal dependencies between \acs{mv}, which prevents erroneous \acs{mv} to propagate throughout the set of dependent frames. + +\begin{table}[!t] + \centering + %\caption{Average \ac{psnr} using different motion vector candidates.} + \caption{\ac{psnr} under random packet loss.} \label{Table3:AvgPSNRWithoutTMVP} + \TABLESVSPACE + \input{./031_ErrorResilienceEvaluation/QualityEvaluation/TableMVPs.tex} +\end{table} + In order to further evaluate the impact of the temporal \ac{mv} dependencies on the error propagation, the \ac{hevc} streams were subject to random packet losses. Table~\ref{Table3:AvgPSNRWithoutTMVP} presents the average quality results, measured using the \ac{psnr}. As expected, the results indicate that under error-free conditions higher quality is achieved by enabling the \ac{tmvp} candidate, since higher compression efficiency is obtained because more \ac{mv} predictors are used. @@ -305,13 +308,7 @@ The results also show that the negative impact of using temporal \ac{mv} depende A higher negative impact is obtained when only P-frames are used (\ac{ld} configuration), since subsequent frames are temporally closer, leading to more accurate temporal \ac{mv} candidates. Moreover, these results also indicate that for higher \ac{plr} the negative impact of using the \ac{tmvp} does not increase, as both configurations are subject to high quality degradation. -\begin{table}[!t] - \centering - %\caption{Average \ac{psnr} using different motion vector candidates.} - \caption{\ac{psnr} under random packet loss.} \label{Table3:AvgPSNRWithoutTMVP} - \TABLESVSPACE - \input{./031_ErrorResilienceEvaluation/QualityEvaluation/TableMVPs.tex} -\end{table} + \subsection*{Remarks on error robustness of HEVC} diff --git a/041_ImprovedErrorResilience/ImprovedErrorResilience.tex b/041_ImprovedErrorResilience/ImprovedErrorResilience.tex index f58a7d7..351f891 100644 --- a/041_ImprovedErrorResilience/ImprovedErrorResilience.tex +++ b/041_ImprovedErrorResilience/ImprovedErrorResilience.tex @@ -11,6 +11,7 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SECTION %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\vspace{-1cm} \section{Introduction} \glsreset{rd} @@ -365,10 +366,12 @@ In comparison to the reference \ac{hevc}, quality improvements up to $1.42$~dB When comparing results for both test cases, it is noticeable that the proposed method is able to outperform the reference methods for both cases, regardless whether \ac{idr} frame loss is allowed or not (\LOSTALL and \KEEPINTRA, respectively). As the \ac{plr} increases, the gains of the proposed method decrease in comparison with the case where the \ac{idr} frames are delivered error-free (i.e., case \KEEPINTRA), due to the higher error propagation of the missing \ac{idr} frame. Moreover, the quality gains are lower in the \LOSTALL case when comparing with \KEEPINTRA for the same \ac{plr}, for both the \ac{ld} and \ac{ra}. Taking into account the above results and the spatio-temporal information shown in Table~\ref{TableTestSequences}, one can observe that the performance of the proposed method is lower for sequences with high spatial details and low temporal activity (e.g., BQSquare and Park Scene), which indicates higher effectiveness for video content with higher motion activity. -%\HIGHLIGHT{complete with analysis of both cases} -\begin{figure}[!b] +\subsubsection{Influence of parameter $\gamma$} + + +\begin{figure}[tb] \renewcommand{\FIGSIZE}{0.8\columnwidth} \centering %\includegraphics[width=\FIGSIZE,page=1]{\ImprovedErrorResilience/Results/PSNRversusGama} @@ -378,10 +381,6 @@ Taking into account the above results and the spatio-temporal information shown \label{fig:QualityVsGamma} \end{figure} -\subsubsection{Influence of parameter $\gamma$} - - - The influence of the $\gamma$ parameter used in Equation (\ref{eq:KFactor}) on the video quality was also evaluated, in order to check how the cost constrains affect the performance of the proposed method. Figure~\ref{fig:QualityVsGamma} shows the average quality gains of the proposed method in comparison to the reference \ac{hevc} (\REF) across all test sequences, for different values of $\gamma$. For the error-free case (PLR=0\%), the results in this figure show that increasing $\gamma$ leads to a decrease in the coding efficiency, especially in the \ac{ld} configuration. @@ -498,6 +497,15 @@ For lower \acs{plr}, since fewer frames are affected by data losses, the use of \subsubsection{Quality evaluation} +\begin{table}[!t] + \centering + \caption{Average quality using redundant \ac{mv} selected by the proposed approach.} + \TABLESVSPACE + \label{tab:AverageQualityRedundancies} + \TABLESVSPACE + \input{\ImprovedErrorResilience/Results/TableResultsRandomLossMvRedundancies.tex} +\end{table} + Table~\ref{tab:AverageQualityRedundancies} presents the average \ac{psnr} of decoded video for different \ac{plr} (columns~4~to~8) and for various percentages of \ac{mv} redundancy (third column). The absolute quality values are shown for the reference case (\NOREDUND) and \ac{psnr} gains are shown for three different levels of redundancy introduced by the proposed method (\DYN). This study was also conducted for the two different cases, with and without \ac{idr} loss. @@ -510,14 +518,7 @@ In summary, the proposed method is able to select the most relevant motion \ac{m Moreover, the dynamic approach devised to select the best \acs{mv} also contributes to increase the overall performance. Thus, using the proposed two-stage approach to recover temporal predictions consistently improves the error robustness of \ac{hevc}. -\begin{table}[!t] - \centering - \caption{Average quality using redundant \ac{mv} selected by the proposed approach.} - \TABLESVSPACE - \label{tab:AverageQualityRedundancies} - \TABLESVSPACE - \input{\ImprovedErrorResilience/Results/TableResultsRandomLossMvRedundancies.tex} -\end{table} + \section{Summary} \label{Section4:Summary} diff --git a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex index 2f6ab09..d049ff0 100644 --- a/043_EndToEndRobustSystem/EndToEndRobustSystem.tex +++ b/043_EndToEndRobustSystem/EndToEndRobustSystem.tex @@ -75,18 +75,18 @@ The remaining configuration was kept as described in Section~\ref{Section3:Error \subsubsection{Preliminary simulation results and discussion} -\begin{figure}[!t] - \centering - \input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex} - \vspace{-4mm} - \caption{Average decoded video quality for the bitstreams encoded with the main bitrate (bps).} - \label{Figure6:AverageQualityMain} -\end{figure} Figures~\ref{Figure6:AverageQualityMain}~and~\ref{Figure6:AverageQualityReduction} show the average decoded video quality (\ac{psnr}) obtained for different \acs{plr}. Figure~\ref{Figure6:AverageQualityMain} shows the results obtained with the test sequence encoded using a constant target bitrate listed in Table~\ref{Table6:TestMaterialCharacteristics}, referred to as the main bitrate. Results on Figure~\ref{Figure6:AverageQualityReduction} were obtained by reducing the main bitrate by 40\%. This is used to evaluate the impact of the network bandwidth on the decoded video quality. The results shown in the figures correspond to the same test sequences, in order to allow an accurate and fair comparison. +\begin{figure}[!t] + \centering + \input{043_EndToEndRobustSystem/Results/Quality_PLR_Methods.tex} + \vspace{-4mm} + \caption{Average decoded video quality for the bitstreams encoded with the main bitrate (bps).} + \label{Figure6:AverageQualityMain} +\end{figure} \begin{figure}[!t] @@ -221,19 +221,7 @@ Transmission networks with higher loss ratios require a higher bit allocation fo \end{enumerate} -%To characterise the input video signal the following parameters are used: \ac{si} and \ac{ti}~\cite{ITUTP9102008}. -%These parameters are obtained as described in Chapter~\ref{Chapter:HEVCErrorResilience}, and aim at evaluating the spatial complexity and motion complexity of the video scenes, which have relevant impact on the error robustness of the coded streams. -%As shown, test material with higher \ac{si} or \ac{ti} suffer from higher quality degradation as the \ac{plr} increases, e.g., Book Arrival and Tennis sequences. - -These parameters were compared against the optimal stream, selected from \SMEC to \SMEXPELTWEN. -This aims at finding the correlation between the input parameters and the output variable, i.e., target stream. -In this study the streams are characterised by a number between 1 and 4. -Table~\ref{Table6:Correlation} shows the absolute Person Correlation coefficient obtained across all sequences, bitrates and \acs{plr}. -The table shows the results for the parameters previously described, as well as, a new parameter obtained from the average magnitude of the optical flow measured using the method in~\cite{DeepFlow2013} (\textit{OPT\_FLOW}). -Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable. -Since both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method, due to its higher Person Correlation coefficient. - -\begin{table}[b] +\begin{table}[tb] \centering \caption{Pearson Correlation coefficient between the different parameters and the optimal robust coding techniques (optimal stream).} \label{Table6:Correlation} @@ -252,9 +240,14 @@ Since both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar informatio \end{tabular} \end{table} +These parameters were compared against the optimal stream, selected from \SMEC to \SMEXPELTWEN. +This aims at finding the correlation between the input parameters and the output variable, i.e., target stream. +In this study the streams are characterised by a number between 1 and 4. +Table~\ref{Table6:Correlation} shows the absolute Person Correlation coefficient obtained across all sequences, bitrates and \acs{plr}. +The table shows the results for the parameters previously described, as well as, a new parameter obtained from the average magnitude of the optical flow measured using the method in~\cite{DeepFlow2013} (\textit{OPT\_FLOW}). +Results in Table~\ref{Table6:Correlation} show that the selected parameters are all related with the optimal streams, with the ratio of inter-coded blocks achieving the highest correlation. Moreover, one can also notice that the parameters related with the motion activity, i.e., \ac{ti} and \textit{AVG\_MV}, are also correlated with the output variable. +Since both \textit{AVG\_MV} and \textit{OPT\_FLOW} represent similar information (motion insensitive), only \textit{AVG\_MV} was used in the proposed method, due to its higher Person Correlation coefficient. -% Podia se citar este paper -% \cite{IntraRefreshICIP2009} \subsection{Neural network} @@ -266,13 +259,6 @@ Mainly, a \ac{nn} consists of three different layers: the input layer which rece In the past \acs{nn} have been widely used in video processing applications such as, quality evaluation~\cite{VideoQualityNN_TNN2006}, video analysis~\cite{VideoAnalysisNN_2018} and video coding~\cite{RateDistortionNN_VCIP2017,HEVCComplexityNN_TIP2018}. In the proposed method a feed-forward network is used, which connects every neuron in a given layer to the neurons of the subsequent layer. -Figure~\ref{FIG6:NeuralNetwork} shows the diagram of the \ac{nn} used in the proposed method. -The network contains an input and output layer, as well as, a single hidden layer. -The input layer receives the six inputs, as described in the previous sub-section. These inputs are correlated with the optimisation variable being maximised, i.e., the output video quality. -Subsequently, the network transforms the input data through a series of neurons in the hidden layer. -The output layer comprises four neurons corresponding to each available video stream from \SMEC to \SMEXPELTWEN (different error resilience techniques). -Finally, the output with the highest value is selected and the corresponding stream is used for transmission, since it is the one expected to achieve the highest video quality for a particular scenario. - \begin{figure}[tb] \centering %\includegraphics[width=.9\columnwidth]{\ConcealmentAwareDir/Figures/BlockDiagram} @@ -282,6 +268,13 @@ Finally, the output with the highest value is selected and the corresponding str \FIGSVSPACE \end{figure} +Figure~\ref{FIG6:NeuralNetwork} shows the diagram of the \ac{nn} used in the proposed method. +The network contains an input and output layer, as well as, a single hidden layer. +The input layer receives the six inputs, as described in the previous sub-section. These inputs are correlated with the optimisation variable being maximised, i.e., the output video quality. +Subsequently, the network transforms the input data through a series of neurons in the hidden layer. +The output layer comprises four neurons corresponding to each available video stream from \SMEC to \SMEXPELTWEN (different error resilience techniques). +Finally, the output with the highest value is selected and the corresponding stream is used for transmission, since it is the one expected to achieve the highest video quality for a particular scenario. + In the proposed approach a shallow \ac{nn} is used for simplicity and to avoid over-fitting, due to the limited test cases. By avoiding over-fitting, it is guaranteed that the proposed method would continue to work in future test conditions, not used for the network training. However, different network topologies with different number of neurons in each layer could also be used to predict between different error resilience techniques, as in the proposed method. @@ -349,15 +342,6 @@ The results from the validation set, shown in Table~\ref{Table6:ConfusionVal}, r \end{tabular} \end{table} - - -To further evaluate the performance of the \ac{nn}, the confusion matrix of the data set used for testing (test set) is shown in the Table~\ref{Table6:ConfusionTest}. -This table shows the same results as Table~\ref{Table6:ConfusionTraining}, but for different test cases, which were not used in the training phase. -% Therefore, the trained network is not aware of the test cases used for validation. -Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 50\% with a maximum value of 84.9\% when the \SMEXPELTWEN is the target stream. -Although a lower accuracy is obtained using the test set, the overall accuracy is still acceptable, since it allows to select the optimal robust coding technique (i.e., \SMEC to \SMEXPELTWEN) for the majority of the conditions. -In summary, the results presented in this sub-section reveal that the input parameters used in the proposed \ac{nn} are correlated with the decoded video quality, as they can effectively be used to perform data categorisation used in the selection of the optimal error robustness coding technique for the \ac{hevc} standard. - \begin{table}[tb] \centering \caption{Confusion matrix of the test set.} @@ -378,6 +362,15 @@ In summary, the results presented in this sub-section reveal that the input para \end{tabular} \end{table} +To further evaluate the performance of the \ac{nn}, the confusion matrix of the data set used for testing (test set) is shown in the Table~\ref{Table6:ConfusionTest}. +This table shows the same results as Table~\ref{Table6:ConfusionTraining}, but for different test cases, which were not used in the training phase. +% Therefore, the trained network is not aware of the test cases used for validation. +Results show that the proposed \ac{nn} is able to achieve an accuracy of at least 50\% with a maximum value of 84.9\% when the \SMEXPELTWEN is the target stream. +Although a lower accuracy is obtained using the test set, the overall accuracy is still acceptable, since it allows to select the optimal robust coding technique (i.e., \SMEC to \SMEXPELTWEN) for the majority of the conditions. +In summary, the results presented in this sub-section reveal that the input parameters used in the proposed \ac{nn} are correlated with the decoded video quality, as they can effectively be used to perform data categorisation used in the selection of the optimal error robustness coding technique for the \ac{hevc} standard. + + + \section{Performance evaluation} \label{Section6:ExperimentalResults} @@ -406,7 +399,7 @@ In case of data loss, the results show that the proposed method is able to match \begin{table}[tb] \centering -\caption{Average quality gains ($\Delta$PSNR) in comparison with the reference \ac{hevc} across different bitrates and loss ratios.} +\caption{Average quality gains ($\Delta$PSNR) in comparison with the reference \ac{hevc} across different bitrates and packet loss ratios.} \label{Table6:AverageGains} \TABLESVSPACE \newcommand{\colwidth}{2cm} @@ -439,13 +432,14 @@ Tennis & +1.18 & +2.11 & +2.43 & +3.28 & +3.28 \\ \hline Subsequently, a comparison between the different methods is performed, in order to evaluate the quality gains obtained with the proposed method in comparison with each of the different techniques. %for every test case. Table~\ref{Table6:AverageGains} shows the average quality gains in comparison with the reference \ac{hevc} for the test set. -These were obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams \SMEC to \SMEXPELTWEN. The average quality gains are measured across all test cases, i.e., test set with different bitrates and different packet loss ratios. +These were obtained with the proposed scheme and the four different techniques used in selection method, i.e., methods used in the streams \SMEC to \SMEXPELTWEN. +The average quality gains are measured across all test cases, i.e., test set with different bitrates and different packet loss ratios. Results in Table~\ref{Table6:AverageGains} show that the robust coded streams \SMEC to \SMEXPELTWEN are able to achieve acceptable average quality gains up to 1.50 dB (see results for \SMEXPELTEN). However, since the implementation of robust coding techniques decreases the coding efficiency, it also leads to lower quality than the reference \ac{hevc}. This is noticeable in the results obtained for the Four People sequence in the streams \SMEXP to \SMEXPELTWEN and for the Kristen and Sara sequence in the streams \SMEXPELTEN and \SMEXPELTWEN (see red ellipses). -The overall performance of such methods reveals that they are not suitable for transmission conditions with low \acs{plr}, e.g., lower than 3\% (higher number of conditions have \acs{plr} lower than 3\%). -On the contrary, the proposed method is always able to outperform the reference \ac{hevc} and achieve quality gains of at least 0.28 dB for the Four People sequence. +The overall performance of such methods reveals that they are not the best options for transmission conditions with low \acs{plr} (higher number of conditions have \acs{plr} lower than 3\%). +On the contrary, the proposed method is always capable of outperforming the reference \ac{hevc} and achieve quality gains from 0.28 dB to 3.28 dB. For the Kimono and Tennis sequences, it is able to achieve the highest quality gains, of up 3.07 dB and 3.28 dB, respectively (see blue ellipses). Summarising, by adaptively selecting the video stream, and implicitly the robust coding technique, the proposed method is able to achieve a better trade-off between coding efficiency and error robustness, achieving an average quality gain across different bitrates and \acs{plr} of 1.88 dB. diff --git a/050_Conclusions/Conclusion.tex b/050_Conclusions/Conclusion.tex index 9298f54..02c331b 100755 --- a/050_Conclusions/Conclusion.tex +++ b/050_Conclusions/Conclusion.tex @@ -15,7 +15,7 @@ In this thesis different methods were proposed to address this problem, by inves Firstly, an introduction of the main motivation for this research work was presented, enumerating its main objectives and contributions. -Then, the main functional blocks of the state-of-the-art \ac{hevc} were described. +Then, the main coding tools of the state-of-the-art \ac{hevc} were described. This description covered the high-level features, such as the new picture types and the slice partitioning schemes, the new block partitioning scheme, which allow multi-level hierarchical structures with size-independent representation, the main predictions techniques, including the newly introduced advanced motion vector prediction technique. This provided the necessary knowledge about the novel techniques in \ac{hevc}, which may have a relevant impact on the error robustness of the coded streams. The main algorithms proposed in the past to increase the error robustness of video streams were also presented, describing techniques based on error resilience \ac{rd} optimisation, redundant based approaches and \ac{ec}-based methods. @@ -39,9 +39,9 @@ Experimental results demonstrated that a clear improvement on decoded video qual Another investigation carried out in this thesis was focused on the reference frame distribution in highly efficient video coders. Since in general, the reference frame usage is unbalanced, a new method based on constrained \ac{rd} optimisation was proposed to equally distribute the use of multiple reference frames across neighbouring \acs{pu}. -An extension \HERE using selective \ac{mv} redundancies was also investigated, in order to recover from erroneous motion predictions. +Moreover, the use of selective \ac{mv} redundancies was also investigated, in order to recover from erroneous motion predictions. This resulted in a two-fold approach to reduce error propagation. -As part of the proposed method can be applied at the streaming-stage, using a low complexity parser, this method can be used to adapt pre-encoded streams to error-prone transmission channels. +As part of the proposed method can be applied at the streaming-stage, using a low complexity parser, it can be used to adapt pre-encoded streams to error-prone transmission channels. The proposed method was compared against existing techniques, and the simulation results revealed a consistently superior performance of using the two-fold approach for different packet loss ratios. To address the problem of robust video decoding and increase the quality of the reconstructed frames, \ac{ec}-aware resilience techniques were also investigated. diff --git a/060_Appendix/Papers.tex b/060_Appendix/Papers.tex index 1e9b85c..222c154 100755 --- a/060_Appendix/Papers.tex +++ b/060_Appendix/Papers.tex @@ -2,16 +2,15 @@ \chapter{Published papers} \label{Appen:Papers} -This appendix presents a list of the published papers that resulted from the research work done during this thesis proposal. -Submitted papers are also presented. +This appendix presents a list of the published papers that resulted from the research work done during this thesis. \section{Journal publications} \begin{description} \item[\namedlabel{Journal:TwoStage}{J1}] -J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, ``A two-stage approach for robust HEVC coding and streaming,'' \emph{IEEE Transactions on Circuits and Systems for Video Technology}, vol. 28, no. 8, pp. 1960-1973, Aug. 2018 (available since Apr. 2017). +J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, ``A two-stage approach for robust HEVC coding and streaming,'' \emph{IEEE Transactions on Circuits and Systems for Video Technology}, vol. 28, no. 8, pp. 1960-1973, Aug. 2018. \item[\namedlabel{Journal:ConcAware}{J2}] -J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, ``Error Concealment-Aware Encoding for Robust Video Transmission,'' \emph{IEEE Transactions on Broadcasting} (available since Aug. 2018). +J.~Carreira, P.~Assuncao, S.~Faria, E.~Ekmekcioglu, and A.~Kondoz, ``Error Concealment-Aware Encoding for Robust Video Transmission,'' \emph{IEEE Transactions on Broadcasting} (available in IEEEXplore since Aug. 2018). \end{description} \section{Conference publications} diff --git a/carreira_phd_thesis.tex b/carreira_phd_thesis.tex index ea58b1a..276046c 100644 --- a/carreira_phd_thesis.tex +++ b/carreira_phd_thesis.tex @@ -16,8 +16,8 @@ \selectoriginallanguage % \newcommand{\HIGHLIGHT}[1]{\textcolor{red}{#1}} -\newcommand{\HIGHLIGHT}[1]{\textcolor{red}{#1}} -\newcommand{\HERE}{\textcolor{red}{CHECK HERE!}} +%\newcommand{\HIGHLIGHT}[1]{\textcolor{red}{#1}} +%\newcommand{\HERE}{\textcolor{red}{CHECK HERE!}} \newcommand{\VERSION}[1]{\textcolor{red}{(r#1)}} % %\includeonly{000_Preamble/abstract}