Original
Reconstructed
For my final I chose to complete Gradient Domain Fusion and Lightfield Camera. I will first go over gradient domain fusion and then cover the lightfield camera after.
This project demonstrates a technique used to blend images into one another through the act of balancing the color values to make it almost seamless.
The preliminary stage of the project surrounds accurately implementing gradient domain fusion on a grayscale image. The result should essentially recreate the exact same image.
This involves solving the optimization of various minimized square differences of gradients across both dimensions.
minimize ( v(x+1,y)-v(x,y) - (s(x+1,y)-s(x,y)) )^2
minimize ( v(x,y+1)-v(x,y) - (s(x,y+1)-s(x,y)) )^2
minimize (v(1,1)-s(1,1))^2
These equations cover every pixel's gradient values as well as comparing the color values of the top 2 corners.
After initializing and filling in a sparse matrix and the constraints, we solve using least squares.
Results:
|
Original |
Reconstructed |
We can see that both the original sample and the reconstructed image are roughly the same, indicating a decent implementation of gradient domain fusion.
This technique essentially performs the same operation as the toy problem, but instead of comparing the gradients of itself, we will minimize the square differences of the gradients from one image with the gradients of another image.
This will put one image as the background, and another image to be inpainted within the background.
By comparing the gradients, we can get values that serve as a balance between both images, causing a more seamless blend/integration.
This can be done by performing a similar operation to the toy problem, while integrating a mask to serve as the location of the blending.
The images were downscaled to allow the process to run better.
Background |
Target |
Objective |
Mask |
Basic Blend |
Poisson Blend |
Background |
Target |
Objective |
Mask |
Basic Blend |
Poisson Blend |
Background |
Target |
Objective |
Mask |
Basic Blend |
Poisson Blend |
||
Slightly More Tight Mask |
Poisson Blend |
The penguin cases worked quite well. I think that this is a result of the colors and backgrounds of the two images already looking very similar, making it easier for such a blend to have a effect.
The case of soldier and the Team Fortress 2 map is not quite as successful as I hoped it would be. I thought it would be interesting to insert the POV of the soldier on a different map from the same game and see if it could blend in the FOV appropriately.
The masking does not entirely encompass the complex shape that cuts in and out of the canvas (under the rocket launcher).
The project spec also mentioned that the masking area could be quite generous so I tried playing around with to see how much I could mask in comparison to the larger background size.
The larger image to be blended seems to cause a significant amount of distortion in the surrounding area.
This task involves attempting to create a better implementation of converting a color to grayscale.
Certain images such as colorblind tests, when converted to grayscale, lose important contrast information leaving an image seemingly without the significant features.
We can retain some of this information by converting the images into HSV form and comparing the values of a neighboring pixel's saturation and value, with whatever one having the larger magnitude taking priority when selecting.
Results
Original |
rgb2gray |
Color2Gray |
This project involves 2 parts: depth refocusing, and aperture adjustment which play with effects on a set of images.
For this project we use specially taken images over a regularly spaced grid that are provided by the Stanford Light Field Archive.
Images were downscaled for the program to work more efficiently.
This concept involves shifting and averaging sets of images to create a combined image that is focused at different depths. We can see this concept at play if we take the average of all images. This will cause objects farther from the viewer to have a higher focus.
Using the compiled image library, we can create a 17x17 array based on the 289 images and revolve our refocusing around the center image at (8, 8).
Depending on the scale selected, we can view the image at varying focus levels.
Chess Results (Scale from -0.3 to 1.6):
Scale = -0.3 |
Scale = 0.46 |
Scale = 1.22 |
Scale = 1.6 |
Gif of Process
Lego Results (Scale from -1 to 4.4):
Scale = -1 |
Scale = 1.16 |
Scale = 3.32 |
Scale = 4.4 |
Gif of Process
In both examples, we can see that as the scale increases, the focus travels from objects in the back of the image, and towards the front.
For this task, we can manipulate the aperture of an image depending on the number of images sampled with more images leading to a larger aperture, and fewer images resulting in a smaller aperture.
This can be done by setting the depth to a constant scale value, and adding a radius variable that determines how many images will be skipped.
In both of the following examples, the depth scale is set to 0
Lego Results (Radii from 0 to 15):
Radius = 0 |
Radius = 6 |
Radius = 12 |
Radius = 15 |
Gif of Process
Lego Results (Radii from 0 to 15):
Radius = 0 |
Radius = 6 |
Radius = 12 |
Radius = 15 |
Gif of Process
I am thoroughly surprised by how uncomplex the operations and process is for creating a lightfield camera. I would imagine that a lot of the work definitely revolves around taking the photos.