Multimodal Image Analysis using Nonlinear Spectral Representations
Non-linear operators are a prominent tool in state-of-the-art image processing algorithms. Thus, much research focus has been given to the spectral analysis of such operators, striving for both deeper understanding and better algorithm design. In this work, we discuss spectral representations of non-linear operators for the analysis and design of image processing algorithms.
We first present a novel local approach to the functional-based, mathematically-formulated spectral total variation (TV) transform. We define the spectral TV local scale signatures, a new descriptor using spectral TV comprehensive scale and location information. Signatures of significant objects are sparse and strong, and their locality allows for good object differentiation within an image. We thoroughly show and analyze their useful sensitivity and invariance properties. Our novel, unified, yet simple framework allows object differentiation by contrast, size or structure, which no previous spectral TV-based approach can produce. This object differentiation serves as the cornerstone for various image processing tasks and modalities. We achieve comparable or superior results for image fusion of thermal and visible images, or of medical images of different modalities; and for image manipulation using clustering and size differentiation.
We then turn to examine spectral representations for generic, black-box operators. As these operators are not functional-induced, and cannot be analytically formulated, they are very complex to characterize. Nevertheless, they play a dominant role in today’s image processing world. In fact, any nonlinear image processing algorithm of same-sized input and output (e.g. denoising or deblurring) can be seen as such a generic operator. Thus, their research carries wide potential implications.
Our non-linear spectral analysis of these operators focuses on generating and analyzing their eigenfunctions. That is, input images for which the output of the operator is proportional to the input. This has never been investigated before for such generic operators.
We introduce a generalized nonlinear power iteration for eigenproblem solution for generic operators. We propose simple adaptations of the well-known linear power iteration and of known concepts of linear eigenvalue analysis. We define eigenfunctions corresponding to large or small eigenvalues as the stable or unstable modes of the operator, respectively. That is, its most or least-suitable inputs. For denoisers (on which we focus here), stable modes achieve superior PSNR in noise removal, whereas unstable modes are strongly suppressed. Analogously to the linear case, stable modes show coarse structures and correspond to large eigenvalues, whereas unstable modes are textural with small eigenvalues. This analysis deepens our understanding of the nature of operators, and can be used in the future to construct spectral decompositions.
We first validate the method using the functional-based, mathematically-formulated TV denoiser. We then exemplify our findings using the classic EPLL and Non-local Means denoisers, and suggest an encryption-decryption application. We also apply our method for TV and EPLL deblurring operators. Finally, we apply our method to the new powerful tool of deep denoising neural nets, demonstrating it for DnCNN and FFDnet nets.
* PhD student under the supervision of Prof. Guy Gilboa.