1/29 Nonlocal methods for image processing Lecture note, Xiaoqun Zhang Oct 30, 2009
2/29 Outline 1 Local smoothing Filters 2 Nonlocal means filter 3 Nonlocal operators 4 Applications 5 References
3/29 General Model v(x) = u(x) + n(x), x Ω v(x) observed image u(x) true image n(x) i.i.d gaussian noise (white noise) Gaussian kernel x G h (x) = 1 x 2 4πh 2 e 4h 2
Local smoothing Filters 4/29 Outline 1 Local smoothing Filters 2 Nonlocal means filter 3 Nonlocal operators Denoising by nonlocal functionals Inverse problems by nonlocal regularization Nonlocal regularization with Bregmanized methods 4 Applications Compressive sampling Deconvolution Wavelet Inpainting 5 References
Local smoothing Filters 5/29 Linear low-pass filter Idea: average in a local spatial neighborhood GF h (v)(x) = G h v(x) = 1 v(y) exp y x 2 4h C(x) 2 y Ω dy where C(x) = 4πh 2 Pro: work well for harmonic function (homogenous region) Con: perform poorly on singular part, namely edge and texture
Local smoothing Filters 6/29 Anisotropic filter Idea: average only in the direction orthogonal to Dv(x)( v(x) x, v(y) y ). AF h (v)(x) = 1 C(x) t v(x + f Dv(x) Dv(x) t 2 ) exp h 2 dt where C(x) = 4πh 2. Pro: Avoid blurring effect of Gaussian filter, maintaining edges. Con: perform poorly on flat region, worse there than a Gaussian blur.
Local smoothing Filters Neighboring filter Spatial neighborhood B ρ (x) = {y Ω y x ρ} Gray-level neighborhood B ( x, h) = {y Ω v(y) v(x) ρ} for a given image v. Yaroslavsky filter Y NF h,ρ = 1 u(y)e u(y) u(x) 2 4h C(x) 2 dy Bilateral(SUSAN) filter SUSAN h,ρ = 1 C(x) B ρ(x) u(y)e u(y) u(x) 2 4h 2 e y x 2 4ρ 2 dy Behave like weighted heat equation, enhancing the edges 7/29
Local smoothing Filters 8/29 Denoising example
Nonlocal means filter 9/29 Outline 1 Local smoothing Filters 2 Nonlocal means filter 3 Nonlocal operators Denoising by nonlocal functionals Inverse problems by nonlocal regularization Nonlocal regularization with Bregmanized methods 4 Applications Compressive sampling Deconvolution Wavelet Inpainting 5 References
Nonlocal means filter 10/29 Nonlocal mean filter 1 Idea: Take advantage of high degree of redundancy of natural images.
Nonlocal means filter 11/29 Denoising formula where NLM(v)(x) := 1 w(x, y)v(y)dy, C(x) Ω w(x, y) = exp{ G a ( v(x + ) v(y + ) 2 )(0) }, C(x) = Ω 2h 2 0 w v (x, y)dy
Nonlocal means filter 12/29 Weight from clean image
Nonlocal means filter 13/29 Weight from noisy image
Nonlocal means filter 14/29 Example
Nonlocal means filter 15/29 Comparison with other methods
Nonlocal operators 16/29 Outline 1 Local smoothing Filters 2 Nonlocal means filter 3 Nonlocal operators Denoising by nonlocal functionals Inverse problems by nonlocal regularization Nonlocal regularization with Bregmanized methods 4 Applications Compressive sampling Deconvolution Wavelet Inpainting 5 References
Nonlocal operators Nonlocal operators 2 /Graph based Regularization Given a nonnegative and symmetric weight function w(x, y) for each pair of points (x, y) Ω Ω: Nonlocal gradient of an image u(x): w u(x, y) = (u(y) u(x)) w(x, y) : Ω Ω Ω Nonlocal divergence of a gradient filed p(x, y) : Ω Ω R is defined by < w u, p >= < u, div w p >, u(x), p(x, y) = div w p(x) = (p(x, y) p(y, x)) w(x, y)dy. Ω Nonlocal functionals of u: J NL/H 1(f) = 1 w u(x) 2 1 : w u(x, y) 2 4 Ω 4 x y J NL/T V (f) = w u(x) 1 : w u(x, y) 2. Ω x y 17/29
Nonlocal operators Denoising by nonlocal functionals Nonlocal H 1 regularization by non-local means Model:min J NL/H1 (u) + µ 2 u f 2 Euler-Lagrange equation: L w (u)u + µ(u f) = 0, where L w is unnormalized graph laplacian : L w (u) = w(x, y)(u(x) u(y)). Ω We can replace L w (u) by normalized graph laplacian 3 L 0 w = 1 C(x) L w = Id NLM w (u). Semi-explicit iteration: for a time step τ > 0, s = 1 + τ + τµ, α 1 = τ s, α 2 = τµ s : u k+1 = (1 α 1 )u k + α 1 NLM w (u k ) + α 2 f. 3 When N and h 0 0, then L 0 w converges to the continuous manifold Laplace - Beltrami operator. 18/29
Nonlocal operators Denoising by nonlocal functionals Nonlocal TV regularization by Chambolle s algorithm Model: min u J NL/T V,w (u) + µ 2 u f 2 Extension of Chambolle s projection method for Nonlocal TV: inf u sup p 1 sup p 1 Ω Ω Ω Ω < w u, p > + µ 2 u f 2, where the solution can be solved by a projected solution u = f 1 µ w p. and the dual variable p is obtained by < w u, p > + 1 2µ div wp 2. Algorithm: p n+1 = pn + τ w (div w p n µf) 1 + τ w (div w p n µf), τ > 0 19/29
Nonlocal operators Inverse problems by nonlocal regularization Deblurring by Nonlocal Means 4 Problem: f = Au + n, A linear operator, n Gaussian noise. Idea: Use initial blurry and noisy image f to compute the weight. 20/29 J NLM,w(f) := min u NLM f u 2 + λ 2 Au f 2 (1) which is equivalent to J NLM,w(f) := min L 0 w f (u) 2 + λ 2 Au f 2 (2) where L 0 w f is the normalized graph laplacian with the weight computed from f. Gradient descents flow: ((L 0 w f ) L 0 w f )u + λa (Au f) = 0 4 A. Buades, B. Coll, and J-M. Morel. 2006
Nonlocal operators Inverse problems by nonlocal regularization Image recovery via nonlocal operators Idea: Use a deblurred image to compute the weight. 1 Preprocessing: Compute a deblurred image via a fast method: u 0 = min 1 2 Au f 2 + δ u 2 u 0 = (A A + δ) 1 A f. where δ is chosen optimally by respecting the condition σ 2 = Au 0 f 2 where σ 2 is the noise level in blurry image. Compute the nonlocal weight w 0 by using u 0 as a reference image (set h 0 = σ 2 (A A + δ) 1 A 2.) 2 Nonlocal regularization with the fixed weight w 0 : min J w0 (u) + λ Au f 2 2 by gradient descent. 21/29
Nonlocal operators Inverse problems by nonlocal regularization Nonlocal regularization for inverse problems 22/29 Idea: nonlocal weight updating during nonlocal regularization by operator splitting. Model : Approximated Algorithm: min J w(u) (u) + λ Au v 2 u 2 v k+1 = u k + 1 µ A (f Au k ) w k+1 = w(v k+1 )(optional) (3) u k+1 = arg min J NL/T V,w k+1 + λµ 2 u vk+1 2 where u k+1 is solved by Chamobelle s method for NLTV.
Nonlocal operators Nonlocal regularization with Bregmanized methods Nonlocal regularization with Bregmanized methods 23/29 With/without weight updating: Algorithm: f k+1 = f k + f Au k v k+1 = u k + 1 µ A (f k+1 Au k ) w k+1 = w(v k+1 )(optional) u k+1 = arg min J NL/T V,w k+1 + λµ 2 u vk+1 2 (4)
Applications 24/29 Outline 1 Local smoothing Filters 2 Nonlocal means filter 3 Nonlocal operators Denoising by nonlocal functionals Inverse problems by nonlocal regularization Nonlocal regularization with Bregmanized methods 4 Applications Compressive sampling Deconvolution Wavelet Inpainting 5 References
Applications Compressive sampling Compressive sampling : Au = RF u 25/29 True Image Initial guess TV NLTV Figure: Data: 30% random Fourier measurements
Applications Deconvolution Deconvolution: Au = k u 26/29 True Image Blurry and noisy Image Fix weight Update weight Figure: 9 9 box average blur kernel, σ = 3
Applications Wavelet Inpainting Wavelet Inpainting: Au = RW u Original Received, PSNR= 17.51 TV, PSNR=28.64 NLTV, PSNR= 36.06 Figure: Block loss(including low-low frequencies loss). For both TV and NLTV, the initial guess is the received image 27/29
References 28/29 Outline 1 Local smoothing Filters 2 Nonlocal means filter 3 Nonlocal operators Denoising by nonlocal functionals Inverse problems by nonlocal regularization Nonlocal regularization with Bregmanized methods 4 Applications Compressive sampling Deconvolution Wavelet Inpainting 5 References
References References A Review of Image Denoising Algorithms, with a New One, A. Buades, B. Coll, J. M. Morel Multiscale Modeling and Simulation, Vol. 4, No. 2. (2005), pp. 490-530. Guy Gilboa and Stanley Osher, Nonlocal Operators with Applications to Image Processing, UCLA CAM report, July 2007 Image Recovery via Nonlocal Operators, Journal of Scientific Computing, Yifei Lou, Xiaoqun Zhang, Stanley Osher, Andrea Bertozzi Xiaoqun Zhang, Martin Burger, Xavier Bresson and Stanley Osher, Bregmanized Nonlocal Regularization for Deconvolution and Sparse Reconstruction, UCLA CAM Report, January 2009 Xiaoqun Zhang and Tony F. Chan, Wavelet Inpainting by Nonlocal Total Variation, UCLA CAM Report, July 2009 29/29