Snakes for image splicing and blending (image compositing)

This is a technique used to composite multiple images into a single image, as for example, you would do when creating a panorama. This technique does not describe how you would determine the alignment of the individual images or whether or if you need to transform each image prior to compositing them. It's a technique that describes how, at the end, each image is composited together into a single image and done so in a way that the final image appears to bear no traces of the compositing.

Let's continue with the panorama example. When taking pictures for a panorama, it's common that the scene changes while the pictures are being taken. The most common situation is when there are people moving around in the scene. As a result, the pictures are not exactly identical in the areas that they overlap.

See the three smaller pictures below. They were taken in a busy corner in Mumbai, India. As I was taking the pictures, people were walking in and out of the picture and sometimes from one picture into another. In the common areas of the first and second picture, some people are in one picture but not in the other. The large picture below is the panorama created from the three smaller pictures. Notice that not all of the people are in the final panorama. This technique intelligently extracts out parts of each of the individual image to construct the panorama. So, while the panorama is not really a true-to-life wide-angle snapshot of the the busy scene, it definitely looks like it could have been.

The most common compositing algorithm, known as feathering, combines two images into one by averaging the color values from the two images to produce the color value in the final image. When the scene in the two images are different from each other, this results in an effect called ghosting. Consider what happens when a person is in one image but not the other (because the person walked away, for instance). Averaging the color values in the area around the person would create a ghostly image of the person in the final image.

Instead of averaging across the entire region that is common between two images, this splicing technique computes a curved line from top to bottom in the common region, and assembles the composite by taking the part of the first image to the left of the line and places it adjacent to the part of the second image to the right of the line.

The curved line is determined by linking parts of the overlapping region that are similar and avoiding areas that are different. Technically, the two sliced images do overlap a little; without some overlap, the splice line would be easy to see. The picture below shows a composite created from two images that were not aligned perfectly. Notice the ghosting effect - especially near the bottom middle of the picture (due to parallax). The red line represents the splice line. The two smaller images on the bottom show the two images spliced around the splice line that are combined to produce the improved composite.

The area of the two spliced images that overlap is the region bordered by the two blue lines. Notice that the region is not even in width. This is a refinement of the technique based on the following observation. Compositing regions that are not highly textured (e.g. the sky) is best done by averaging over large regions. Compositing regions that are highly textured (e.g. grass) is best done by averaging over smaller regions; averaging over larger regions would result in ghosting. Thus, the overlapping region in the example is wider in areas that are not highly textured and narrower in areas that are highly textured.

Details of how the splice line and region is computed can be found in the patent. The actual computation involves describing some of the principles as an error measure that is to be minimized. The minimization is done efficiently using dynamic programming.

What this technique does is find the most visually pleasing way to combine multiple images into a composite. In the above examples, the images were of the same scene (except for moving objects and misalignments). However, this technique also works on images that are actually different.

The slide below shows how a single image can be rearranged and spliced to create a new image that can be tiled side-to-side without showing a seam. The original image is cut down the middle to create two vertical half images. The left half image is placed on the right and the right half image is placed on the left - with some overlap. The splicing and blending technique is used to blend and composite the opposite edges of the two halves. Since the new left and right edges of the composite came from the same part of the original image, they can be put together without showing a seam. Repeating this process horizontally creates an image that can be tiled seamlessly in both directions.