..

colorizing the prokudin-gorskii photo collection

background

sergei mikhailovich prokudin-gorskii (1863-1944) traveled across the russian empire and captured photographs of everything! back then, there were no ways to print colored images, however, he was convinced that it would be possible someday in the future. from people and sculptures to buildings and trains, he recorded everything down on three glass plates with different exposures (red, green, and blue).

overview

for this project, i use image processing techniques to take the digitized images prokudin-gorskii took, align them, and stack them together - basically recolorizing the images. this was done by the library of congress.

before performing any sort of alignement, an important step of this process is to clean up any visual arjpgacts possible. this is crucial as it will create inaccuracies in our results when we perform caclulations in the later steps. to do this, i first divide the glass plate images into three equal parts (r, g, b) and then crop the borders by 10%.

then, i will use one of the color channels (blue in my case) to act as an anchor while i shift the red and green channels around until they are all aligned.

1. single-scale search method

initially, i developed an algorithm which would exhaustively search a [-15, -15] pixels window of possible displacements. i would then calculate the normalized cross-correlation (ncc) to find the best score between the two images (blue and red/green) using the dot product of im1/norm(im1) and im2/norm(im2). after that, i would take the displacement with the best ncc score and shift the image with respect to the blue image to align it nicely.

this method works well with lower resolution images (such as the .jpg images), however, when it comes to the larger resolution images (.tif images), this method proves to be computationally expensive as the search window would be much greater.

2. multi-scale pyramid search method

multi-scale pyramid diagram

image pyramid diagram with scaling factor 1/2

source: pyimagesearch

to improve the efficiency of the algorithm, i implemented a new algorithm which uses an image pyramid. this algorithm basically scales down the original image multiple times to a coarser image where i can then use the single-scale search method and then slowly work our way up the pyramid shifting the images according to the displacement each time as i go.

this step is done recursively with a base case of threshold=300px*300px and the search window becomes [-2, 2] pixels after the coarsest layer. the reason is because we would have already traversed up to a finer, shifted image (due to the displacement * 2 which is the scaling factor) where we would then only have to search a smaller neighborhood. in doing so, it significantly speeds up the computation time from an average of more than one minute to less than five seconds.

results

three channels of monastery image of monastery before processing image of monastery after processing

monastery.jpg
red channel: (2, 3)
green channel: (2, -3)
total time elapsed: 0.48s

three channels of tobolsk image of tobolsk before processing image of tobolsk after processing

tobolsk.jpg
red channel: (3, 6)
green channel: (3, 3)
total time elapsed: 0.44s

three channels of cathedral image of cathedral before processing image of cathedral after processing

cathedral.jpg
red channel: (3, 12)
green channel: (2, 5)
total time elapsed: 0.43s

three channels of emir image of emir before processing image of emir after processing

emir.tif
red channel: (-305, 93)
green channel: (24, 49)
total time elapsed: 3.63s

three channels of church image of church before processing image of church after processing

church.tif
red channel: (-4, 58)
green channel: (4, 25)
total time elapsed: 3.45s

three channels of three generations image of three generations before processing image of three generations after processing

three_generations.tif
red channel: (11, 112)
green channel: (14, 53)
total time elapsed: 3.48s

three channels of melons image of melons before processing image of melons after processing

melons.tif
red channel: (13, 178)
green channel: (10, 81)
total time elapsed: 3.27s

three channels of onion church image of onion church before processing image of onion church after processing

onion_church.tif
red channel: (36, 108)
green channel: (26, 51)
total time elapsed: 3.28s

three channels of train image of train before processing image of train after processing

train.tif
red channel: (32, 87)
green channel: (5, 42)
total time elapsed: 3.25s

three channels of icon image of icon before processing image of icon after processing

icon.tif
red channel: (23, 89)
green channel: (17, 41)
total time elapsed: 3.28s

three channels of self portrait image of self portrait before processing image of self portrait after processing

self_portrait.tif
red channel: (37, 176)
green channel: (29, 78)
total time elapsed: 3.37s

three channels of harvesters image of harvesters before processing image of harvesters after processing

harvesters.tif
red channel: (13, 124)
green channel: (16, 59)
total time elapsed: 3.17s

three channels of sculpture image of sculpture before processing image of sculpture after processing

sculpture.tif
red channel: (-27, 140)
green channel: (-11, 33)
total time elapsed: 3.33s

three channels of lady image of lady before processing image of lady after processing

lady.tif
red channel: (12, 111)
green channel: (9, 51)
total time elapsed: 3.26s