From c00e9a65465c7a9410301c81ce40e25d58fed9ba Mon Sep 17 00:00:00 2001 From: Joao Carreira Date: Thu, 19 Jul 2018 17:56:35 +0100 Subject: [PATCH] Fix labels in 041_ImprovedErrorResilience --- .../ImprovedErrorResilience.tex | 29 ++++++++++--------- .../Results/PSNRversusFrameKendo.tex | 2 +- .../TableResultsRandomLossRefPicSelection.tex | 20 ++++++------- 3 files changed, 26 insertions(+), 25 deletions(-) diff --git a/041_ImprovedErrorResilience/ImprovedErrorResilience.tex b/041_ImprovedErrorResilience/ImprovedErrorResilience.tex index bf6359a..3eee35e 100644 --- a/041_ImprovedErrorResilience/ImprovedErrorResilience.tex +++ b/041_ImprovedErrorResilience/ImprovedErrorResilience.tex @@ -147,7 +147,7 @@ For instance, for $\gamma=5$, when a reference frame $r_\phi$ is used to predict \newcommand{\figsize}{0.22\textwidth} \resizebox{\textwidth}{!}{ \begin{tabular}{y{.24\textwidth}y{.24\textwidth}y{.24\textwidth}y{.24\textwidth}} - \small Original frame & \small \REF & \small \CHKB & \small \EXP \\ + \small Original frame & \small \REF & \small \CHKB & \small \PROP \\ \includegraphics[width=\figsize]{SelectionExample/ORG} & \includegraphics[width=\figsize]{SelectionExample/REF} & \includegraphics[width=\figsize]{SelectionExample/CHKB} & @@ -160,9 +160,9 @@ For instance, for $\gamma=5$, when a reference frame $r_\phi$ is used to predict \FIGSVSPACE \end{figure*} -Figure \ref{fig:RefPicSelectionExample} illustrates the distribution of reference frames achieved by the proposed method (\EXP) across different \acs{pu}. This result is compared against a fixed checkerboard approach (\CHKB) and the reference \ac{hevc} (\REF). The dark-gray and light-gray squares indicate the reference frame in use, either $r_1$ or $r_2$, respectively. The white squares indicate intra-coded \acs{pu}. Below each method, the percentages of the image areas predicted by each reference frame are shown. As shown in this figure, the proposed method (\EXP) is able to distribute the use of the reference frames closely to the \CHKB method, i.e., $r_2$ is used as reference for approximately the same amount of \acs{pu} as $r_1$ (see use percentages of the predicted image). However, the predicted regions in the proposed method do not exhibit the same regular pattern as in \CHKB method. +Figure \ref{fig:RefPicSelectionExample} illustrates the distribution of reference frames achieved by the proposed method across different \acs{pu}. This result is compared against a fixed checkerboard approach (\CHKB) and the reference \ac{hevc} (\REF). The dark-gray and light-gray squares indicate the reference frame in use, either $r_1$ or $r_2$, respectively. The white squares indicate intra-coded \acs{pu}. Below each method, the percentages of the image areas predicted by each reference frame are shown. As shown in this figure, the proposed method is able to distribute the use of the reference frames closely to the \CHKB method, i.e., $r_2$ is used as reference for approximately the same amount of \acs{pu} as $r_1$ (see use percentages of the predicted image). However, the predicted regions in the proposed method do not exhibit the same regular pattern as in \CHKB method. -It is noticeable in the figure that for several \acs{ctu} the \EXP method is able to keep the same partitions and reference frames when compared to the optimal encoding (\REF). This can be observed, for example in the small partitions on the left, where the \EXP method uses the same reference frame as \REF.Thus, \EXP is expected to achieve better coding efficiency than \CHKB, since more optimal coding modes may be chosen. +It is noticeable in the figure that for several \acs{ctu} the \PROP method is able to keep the same partitions and reference frames when compared to the optimal encoding (\REF). This can be observed, for example in the small partitions on the left, where the \PROP method uses the same reference frame as \REF.Thus, the proposed method is expected to achieve better coding efficiency than \CHKB, since more optimal coding modes may be chosen. Summarising, the underlying principle of Equations~(\ref{eq:Lagrangian})~and~(\ref{eq:KFactor}) is that the cost of using a given reference frame increases with the number of \acs{pu} using it for prediction. However, the reference frames are not hard-selected without taking into account the R-D cost. In general, it is more difficult to reach a uniform use of reference frames in complex sequences than simpler ones. Note that standard compatibility is maintained, since the proposed optimisation method does not require any syntax change. Moreover, as the proposed approach is applied at the Lagrangian computation already present in the \ac{hevc} optimisation, the complexity of the encoder is not significantly affected. @@ -226,7 +226,8 @@ The subset of selected \acs{mv} is independently encoded, ensuring that no refer \section{Performance evaluation} \label{Section4:Performance} -In this section the robustness of the proposed method is evaluated and discussed for two cases: (i) only using the encoding stage and (ii) using the streaming stage in addition to the encoding stage. In the former case, identified as ``\EXP'', only the dynamic reference picture selection method described in Section~\ref{Section4:RefPicSel} is evaluated. In the latter case, identified as ``\DYN'', besides the previous method of reference picture selection, redundant \acs{mv} are encoded as described in Section~\ref{Section4:MVSelection}, resulting in a combination of both. + +In this section the robustness of the proposed method is evaluated and discussed for two cases: (i) only using the encoding stage and (ii) using the streaming stage in addition to the encoding stage. In the former case, identified as ``\PROP'', only the dynamic reference picture selection method described in Section~\ref{Section4:RefPicSel} is evaluated. In the latter case, identified as ``\DYN'', besides the previous method of reference picture selection, redundant \acs{mv} are encoded as described in Section~\ref{Section4:MVSelection}, resulting in a combination of both. As references for comparison, three other methods are used: (i) a fixed checkerboard approach (\CHKB), (ii) the reference picture generation method proposed in \cite{ROPERefGenJan2010}, identified as ``\AVG'', and (iii) long-term reference frames (\LONG). The \LONG method uses only the key frames as reference, i.e., in the Random-Access configuration only one out of eight frames is used as reference frame, allowing a more robust transmission, as the reference frames are less likely to be hit by errors. @@ -238,12 +239,12 @@ The experimental setup used in this section is the same as the one previously de The performance of the encoding stage is evaluated on its own in order to consider only the improvements achieved by the dynamic reference frame selection method, when reference frames were reconstructed with distortion (e.g. as a result of non-perfect error concealment). Since the encoding stage does not deal with errors in temporal \ac{mv} predictions (this is handled by the streaming-stage of the proposed method), in this evaluation the temporal \ac{mv} predictions were not used. This is necessary to avoid masking the results with the errors caused by wrong \ac{mv} predictions. -The performance of the proposed encoding stage (\EXP) was evaluated using $\gamma = 5$. This allows to increase the Lagrangian cost by about $50\%$, when a reference frame is used for three or more \acs{ctu} in comparison to the usage of the other ones (see Figure~\ref{fig:weightfct}). Further results for a wide range of $\gamma$ values are also presented. +The performance of the proposed encoding stage (\PROP) was evaluated using $\gamma = 5$. This allows to increase the Lagrangian cost by about $50\%$, when a reference frame is used for three or more \acs{ctu} in comparison to the usage of the other ones (see Figure~\ref{fig:weightfct}). Further results for a wide range of $\gamma$ values are also presented. \subsubsection{Performance under error-free conditions} Table~\ref{tab:bd_error_free} shows the Bjontegaard's Delta PSNR (BD-PSNR) values compared to the reference \ac{hevc} encoder for different coding configurations in error-free decoding. Four QPs were used from the common test conditions: 22, 27, 32, 37. -The aim is to evaluate the \ac{rd} penalty incurred by the~\EXP method in comparison with \CHKB, as well as the other reference methods. +The aim is to evaluate the \ac{rd} penalty incurred by the proposed method in comparison with \CHKB, as well as the other reference methods. The use of long-term reference pictures (\LONG) is only tested for the case of Random-Access, where key frames are available, while method \cite{ROPERefGenJan2010} is tested for the Low-Delay configuration. The results in Table~\ref{tab:bd_error_free} indicate a small loss of coding efficiency when the proposed reference picture selection approach is used under error-free conditions. As expected, this is due to the sub-optimal \ac{rd} optimisation of the proposed method, because the cost function also includes the cost associated to the choice of the reference frame, in addition to the rate and distortion. @@ -257,7 +258,7 @@ The results in Table~\ref{tab:bd_error_free} indicate a small loss of coding ef \begin{tabular}{p{3cm}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}} \toprule \multirow{2}{*}{\centering\textbf{Sequence}} & \multicolumn{3}{c|}{\textbf{Low-Delay}} & \multicolumn{3}{c}{\textbf{Random-Access}} \\ -& \AVG & \CHKB & \EXP & \LONG & \CHKB & \EXP \\ \hline +& \AVG & \CHKB & \PROP & \LONG & \CHKB & \PROP \\ \hline Basketball Drill & -1.14 & -1.15 & -0.51 & -0.85 & -0.29 & -0.22 \\ \hline Kendo & -2.10 & -1.39 & -0.99 & -0.91 & -0.39 & -0.28 \\ \hline Park Scene & -1.79 & -1.38 & -0.81 & -0.68 & -0.38 & -0.28 \\ \hline @@ -278,7 +279,7 @@ Overall, the loss in the quality compared to the reference \ac{hevc} is consider The robustness achieved by the proposed method was evaluated by comparison of the error propagation resulting from a single loss event in the Low-Delay configuration. -Figure~\ref{fig:RecoveredExample} shows the visual impact of error propagation on a region of the decoded Frame \#10 when Frame \#9 is affected by errors, for Basketball Drill and Race Horses sequences (the PSNR corresponds to the entire frame). As shown in this figure, the proposed method (\EXP) is able to reduce the impact of mismatch decoding in comparison with the method~in~\cite{ROPERefGenJan2010}, revealing the higher efficiency achieving by distributing the use of the references frames. +Figure~\ref{fig:RecoveredExample} shows the visual impact of error propagation on a region of the decoded Frame \#10 when Frame \#9 is affected by errors, for Basketball Drill and Race Horses sequences (the PSNR corresponds to the entire frame). As shown in this figure, the proposed method is able to reduce the impact of mismatch decoding in comparison with the method~in~\cite{ROPERefGenJan2010}, revealing the higher efficiency achieving by distributing the use of the references frames. \begin{figure}[tb] \centering @@ -286,7 +287,7 @@ Figure~\ref{fig:RecoveredExample} shows the visual impact of error propagation o \newcommand{\CurrFigSize}{0.23\columnwidth} \begin{tabular}{y{.23\columnwidth}y{.23\columnwidth}|y{.23\columnwidth}y{.23\columnwidth}} \multicolumn{2}{c}{Basketball Drill} & \multicolumn{2}{|c}{Race Horses} \\ - \AVG & \EXP & \AVG & \EXP \\ + \AVG & \PROP & \AVG & \PROP \\ \includegraphics[width=\CurrFigSize]{Figure_6_1} & \includegraphics[width=\CurrFigSize]{Figure_6_2} & \includegraphics[width=\CurrFigSize]{Figure_6_3} & @@ -312,7 +313,7 @@ Figure~\ref{fig:RecoveredExample} shows the visual impact of error propagation o Figure~\ref{fig:PSNRvsFrameForRefPicSel}~shows the PSNR over a GOP with an \ac{idr} period of 32 frames, encoded at the same constant bitrate. This figure shows that all methods are able to outperform the reference \ac{hevc} (\REF), gaining approximately $3$~dB in reconstruction quality. Since the proposed method is more efficient than the others for error-free transmission (see~Table~\ref{tab:bd_error_free}), and it also provides better error robustness, it can be considered as an efficient solution to better deal with transmission errors. -The similar quality levels achieved by \CHKB and \EXP reveal that dynamic selection of the reference frames is more robust because a better trade-off between coding efficiency and error resilience is attained. +The similar quality levels achieved by \CHKB and \PROP reveal that dynamic selection of the reference frames is more robust because a better trade-off between coding efficiency and error resilience is attained. Overall, the method proposed for the encoding stage is able to increase the error robustness of \ac{hevc} without severely compromising the coding efficiency. Further tests were run to evaluate the effectiveness of the proposed method under various packet loss rates. @@ -343,7 +344,7 @@ Taking into account the above results and the spatio-temporal information shown The influence of the $\gamma$ parameter used in Equation (\ref{eq:KFactor}) was also studied. -Figure \ref{fig:QualityVsGamma} shows the average quality gains in comparison to the reference \ac{hevc} (\REF) across all test sequences, when using different values of $\gamma$ in the proposed method (\EXP). +Figure \ref{fig:QualityVsGamma} shows the average quality gains in comparison to the reference \ac{hevc} (\REF) across all test sequences, when using different values of $\gamma$ in the proposed method (\PROP). For the error-free case, the results in this figure show that increasing $\gamma$ leads to a decrease in the coding efficiency, especially for the Low-Delay configuration. The reason is that higher values of $\gamma$ lead to higher values of $W(r_{\phi})$, therefore the reference picture selection is more constrained by the exponential weights. Since $T_{ID}$ is constant for all frames in the Low-Delay, higher weights are used for all frames ($T_{ID}$ is used to reduce the weights in case of hierarchical coding) and the encoder is not able to select the best reference frame in terms of R-D optimisation. In the case where packet loss occurs, higher values of $\gamma$ lead to higher error resilience. Since higher weights used in the \ac{rd} cost, the encoder is constrained to distribute the use of the reference frames, reducing the error propagation. For very high values (e.g., $\gamma=1000$) the encoder is forced to uniformly select the reference frames, resulting in a fixed checkerboard pattern. @@ -353,7 +354,7 @@ Overall, using the $\gamma$ parameter in the proposed method, the impact of the \subsubsection{Complexity overhead} The implementation of the proposed method has slightly increased the complexity of the standard non-robust encoder. Therefore, the computational complexity has also been considered as a performance metric in this research, which is measured as the encoding time in a controlled hardware platform. The reference used for comparison with other methods is the \ac{hevc} reference software encoding time, running on the same platform with the same coding configuration. -The relative complexity increase of the robust encoder using \EXP, \CHKB and the method presented in \cite{ROPERefGenJan2010} is shown in Table~\ref{tab:cpu_time}. +The relative complexity increase of the robust encoder using \PROP, \CHKB and the method presented in \cite{ROPERefGenJan2010} is shown in Table~\ref{tab:cpu_time}. \begin{table}[b] \newcommand{\resultswidth}{2.2cm} @@ -364,7 +365,7 @@ The relative complexity increase of the robust encoder using \EXP, \CHKB and the \begin{tabular}{p{4cm}|x{\resultswidth}|x{\resultswidth}|x{\resultswidth}} \toprule \multirow{2}{*}{\centering\textbf{Sequence}} & \multicolumn{3}{c}{\textbf{Method}} \\ - & \AVG & \CHKB & \EXP \\ \hline + & \AVG & \CHKB & \PROP \\ \hline Basketball Drill & +4.19\% & +2.13\% & +2.93\% \\ \hline Kendo & +4.96\% & +1.83\% & +2.05\% \\ \hline Park Scene & +4.98\% & +3.90\% & +4.04\% \\ \hline @@ -375,7 +376,7 @@ The relative complexity increase of the robust encoder using \EXP, \CHKB and the As shown in this table, the method presented in \cite{ROPERefGenJan2010} is the most time consuming because it requires post-processing of the encoded frames in order to interpolate new references. -The proposed scheme, using a reference picture selection, \EXP presents a lower complexity increase than \cite{ROPERefGenJan2010}, while the fixed checkerboard scheme (\CHKB) presents a slightly lower complexity than \EXP. +The proposed scheme, using a reference picture selection, presents a lower complexity increase than \cite{ROPERefGenJan2010}, while the fixed checkerboard scheme (\CHKB) presents a slightly lower complexity \PROP method. \subsection{Streaming stage} diff --git a/041_ImprovedErrorResilience/Results/PSNRversusFrameKendo.tex b/041_ImprovedErrorResilience/Results/PSNRversusFrameKendo.tex index bd9eb8f..54d5efb 100644 --- a/041_ImprovedErrorResilience/Results/PSNRversusFrameKendo.tex +++ b/041_ImprovedErrorResilience/Results/PSNRversusFrameKendo.tex @@ -8,7 +8,7 @@ set xtics 0,2,0+31 set yrange[33:45] set ytics 33,2,45 set key spacing 1.4 bottom right width 6 box -plot '-' using 1:2 title '\KeyFont \EXP' w lp ls 1, '-' using 1:2 title '\KeyFont \CHKB' w lp ls 2, '-' using 1:2 title '\KeyFont \AVG' w lp ls 3, '-' using 1:2 title '\KeyFont \REF' w lp ls 4 +plot '-' using 1:2 title '\KeyFont \PROP' w lp ls 1, '-' using 1:2 title '\KeyFont \CHKB' w lp ls 2, '-' using 1:2 title '\KeyFont \AVG' w lp ls 3, '-' using 1:2 title '\KeyFont \REF' w lp ls 4 0 43.01 44.05 43.66 1 41.71 43.89 43.36 2 40.83 43.77 43.16 diff --git a/041_ImprovedErrorResilience/Results/TableResultsRandomLossRefPicSelection.tex b/041_ImprovedErrorResilience/Results/TableResultsRandomLossRefPicSelection.tex index 2a0c12f..b2710df 100644 --- a/041_ImprovedErrorResilience/Results/TableResultsRandomLossRefPicSelection.tex +++ b/041_ImprovedErrorResilience/Results/TableResultsRandomLossRefPicSelection.tex @@ -110,31 +110,31 @@ & \REF & 31.36 & 25.07 & 36.90 & 33.54 & 31.05 & 27.88 \\ & \LONG & +0.09 & -0.12 & -0.02 & +0.31 & +0.57 & +0.54 \\ & \CHKB & +0.12 & +0.20 & +0.15 & +0.46 & +0.60 & +0.69 \\ -& \EXP & +0.39 & +0.37 & +0.26 & +0.57 & +0.79 & +0.87 \\ +& \PROP & +0.39 & +0.37 & +0.26 & +0.57 & +0.79 & +0.87 \\ \hline \multirow{\nlines}{*}{\vbox{Book Arrival }} & \REF & 33.39 & 25.48 & 38.71 & 35.90 & 33.01 & 29.53 \\ & \LONG & -0.14 & +1.01 & +0.18 & +0.38 & +0.96 & +0.94 \\ & \CHKB & +0.30 & +0.10 & +0.07 & +0.06 & +0.31 & +0.70 \\ -& \EXP & +0.26 & +1.16 & +0.34 & +0.28 & +0.67 & +0.97 \\ +& \PROP & +0.26 & +1.16 & +0.34 & +0.28 & +0.67 & +0.97 \\ \hline \multirow{\nlines}{*}{\vbox{Cactus }} & \REF & 34.58 & 32.36 & 36.92 & 35.08 & 33.92 & 32.12 \\ & \LONG & -0.32 & -0.10 & -0.34 & -0.32 & -0.30 & -0.30 \\ & \CHKB & +0.15 & +0.22 & +0.21 & +0.36 & +0.42 & +0.53 \\ -& \EXP & +0.19 & +0.13 & +0.19 & +0.34 & +0.46 & +0.50 \\ +& \PROP & +0.19 & +0.13 & +0.19 & +0.34 & +0.46 & +0.50 \\ \hline \multirow{\nlines}{*}{\vbox{Bosphorus }} & \REF & 32.43 & 24.86 & 41.64 & 39.70 & 38.28 & 36.05 \\ & \LONG & +0.09 & -0.87 & -0.60 & -0.62 & -0.61 & -0.66 \\ & \CHKB & -0.22 & +0.37 & +0.19 & +0.43 & +0.55 & +0.63 \\ -& \EXP & +0.21 & +0.49 & +0.18 & +0.47 & +0.59 & +0.69 \\ +& \PROP & +0.21 & +0.49 & +0.18 & +0.47 & +0.59 & +0.69 \\ \hline \multirow{\nlines}{*}{\vbox{BQSquare }} & \REF & 33.85 & 23.63 & 37.85 & 33.45 & 30.44 & 27.10 \\ & \LONG & -0.24 & +0.46 & -0.75 & -0.06 & -0.04 & -0.41 \\ & \CHKB & +0.32 & +0.62 & +0.66 & +0.65 & +1.06 & +1.05 \\ -& \EXP & +0.42 & +0.31 & +0.50 & +1.11 & +0.93 & +0.90 \\ +& \PROP & +0.42 & +0.31 & +0.50 & +1.11 & +0.93 & +0.90 \\ \hline %\multirow{\nlines}{*}{\vbox{Jockey }} %& \REF & 33.86 & 26.52 & 36.80 & 31.82 & 29.27 & 26.21 \\ @@ -146,31 +146,31 @@ & \REF & 36.81 & 28.11 & 39.78 & 34.69 & 30.73 & 26.62 \\ & \LONG & +1.04 & +0.97 & +0.59 & +1.43 & +1.44 & +0.80 \\ & \CHKB & +0.55 & +1.09 & +0.24 & +0.62 & +1.07 & +0.98 \\ -& \EXP & +0.84 & +1.65 & +0.37 & +0.95 & +1.53 & +1.42 \\ +& \PROP & +0.84 & +1.65 & +0.37 & +0.95 & +1.53 & +1.42 \\ \hline \multirow{\nlines}{*}{\vbox{Park Scene }} & \REF & 30.03 & 25.37 & 36.74 & 32.20 & 29.60 & 26.35 \\ & \LONG & -0.15 & -0.27 & -0.56 & -0.44 & -0.39 & -0.12 \\ & \CHKB & +0.24 & +0.16 & +0.17 & +0.51 & +0.56 & +0.58 \\ -& \EXP & +0.36 & +0.31 & +0.65 & +0.95 & +0.93 & +1.04 \\ +& \PROP & +0.36 & +0.31 & +0.65 & +0.95 & +0.93 & +1.04 \\ \hline \multirow{\nlines}{*}{\vbox{People on Street }} & \REF & 26.04 & 20.91 & 30.86 & 27.21 & 25.17 & 22.39 \\ & \LONG & +0.26 & +0.14 & -0.22 & -0.27 & -0.30 & -0.27 \\ & \CHKB & +0.56 & +0.81 & +0.56 & +0.64 & +0.68 & +0.68 \\ -& \EXP & +0.74 & +0.83 & +0.76 & +0.87 & +0.96 & +0.93 \\ +& \PROP & +0.74 & +0.83 & +0.76 & +0.87 & +0.96 & +0.93 \\ \hline \multirow{\nlines}{*}{\vbox{Race Horses }} & \REF & 30.43 & 24.11 & 33.22 & 29.14 & 26.84 & 23.93 \\ & \LONG & -0.17 & -0.20 & +0.09 & +0.50 & +0.56 & +0.34 \\ & \CHKB & +0.50 & +0.43 & +0.29 & +0.69 & +0.83 & +0.89 \\ -& \EXP & +0.50 & +0.38 & +0.54 & +1.08 & +1.10 & +1.08 \\ +& \PROP & +0.50 & +0.38 & +0.54 & +1.08 & +1.10 & +1.08 \\ \hline \multirow{\nlines}{*}{\vbox{Tennis }} & \REF & 31.89 & 25.89 & 36.74 & 32.20 & 29.60 & 26.35 \\ & \LONG & +1.12 & +0.58 & +0.09 & +1.55 & +1.61 & +1.25 \\ & \CHKB & +0.16 & +0.12 & +0.17 & +0.51 & +0.56 & +0.58 \\ -& \EXP & +1.46 & +1.09 & +0.65 & +0.95 & +0.93 & +1.04 \\ +& \PROP & +1.46 & +1.09 & +0.65 & +0.95 & +0.93 & +1.04 \\ % \hline % \multirow{\nlines}{*}{\vbox{Traffic }} % & \REF & 28.60 & 23.47 & 37.29 & 34.52 & 32.54 & 29.60 \\