Perspective Interpolation – Specularity and Refraction Problems

So, how about converting 2D to 3D or converting two perspective 3D into multi-perspective autostereoscopic… Technology certainly should easily make that possible, right?

The answer is a bit complicated. Because for some images it is quite possible to achieve excellent results. Unfortunately, for many images and scenes it truly is impossible to create accurate 3D from 2D and/or interpolate additional perspectives for autostereoscopic displays.

Case in point? Look at the animation below:

In the background painting there are tiny bits of highly reflective particles embedded in oil paint. These dots of light reflect bright points of light depending upon the perspective. They “come on” quickly as you change perspective because of the paint occlusion where you see them in one eye but not the other. Any program that interpolates views would not know what to do with a picture like this. Morph the dots of light? In real life, they don’t morph, they pop on with the light brightening as the perspective angle changes.

Now, take a close look at the glass gems. Notice how their specularity is influenced by the perspective position relative to the background?  Notice the refraction as you see the background through the transparent glass. Unless you modeled the gems in a 3D program and rendered them, there would be no way to interpolate with a pixel warping program what is going on with the look and texture of these gems as they change perspective.

What happens typically with a conversion is an abysmal mess for items with specularity and refraction. It looks 3D for sure – but in no way is representative of reality. And this is the conundrum. There is no uniformity or consistency with regards to 2D to 3D conversions or 2 perspective to multi perspective conversions. It is completely content based and the results are dependent upon the subject matter.

Binocular disparity and as this example demonstrates, binocular rivalry where one perspective contains elements not visible in the other perspective create monumental problems for conversion.

The solution? Shoot multiple perspectives. And this is the path that I have been forced to take to create consistent and uniform results. Indeed, fewer than 10 perspectives does not yield quality, uniform results in my humble opinion. Can fewer than 10 perspectives work? The answer is yes if what you are photographing has no specularity or refraction properties and the texture is smooth and uniform. But as an artist, I find that restriction way to limiting and live in a world that consists mostly of refractive material (water) and glass and gems and metals. Indeed, just look around and the world is filled with specular and refractive content.

Even portraits pose a problem because unless the person has extremely dry eyes, they glisten as the moisture that coats the eye creates specularity and refraction. Of course, if you don’t have a close up or reduce the resolution then it isn’t that noticeable. But here again, as an artist I find that too limiting.

I do not understand the willingness of people to ignore these problems. While it is true that in many cases specularity and refraction are subtle and nuanced. But given that 3D mimics the way we see real life, shouldn’t 3D be subtle and nuanced? Perhaps the gross over emphasized poke you in the eye effects are doing the potential of 3D a disservice?

That’s my view. But what do I know?

Filed under 3D, 3D Photography, autostereoscopic, S3D, stereopsis

3 responses to “Perspective Interpolation – Specularity and Refraction Problems”

1. Daniele

excellent post and nice example!!

This and many other things make real stereoscopic content so gorgeous.
Stereopsis provide not only information about the >wherewhatstereoscopic surface perception<.
The same is true for stereoscopic transparency perception. Try to convert semitransparent objects like smoke.

2. I agree with what Daniele said and I thank you for all the tips you’ve given in this post.I will definitely follow it to the letter. I actually like your view and I will support it.

3. Thanks Pete. My subsequent research to this article indicates that artificially created pixel warped perspectives can create spatial distortions with all sorts of negative ramifications when perceived with stereopsis by humans. Indeed, watching spatially shifting and distorted imagery over time could facilitate a type of adaptation that could even be problematical in unexpected ways. When we remove perception constancy and force adaptation then there is no “grounding” or fixed frame of reference. This induces problems with perception and interpretation of what we see. Optical illusions exploit this. When our perception references are screwed up, I wonder how optical illusions will be interpreted and if this will be a bad thing? Of course, we would have to look at a lot of pixel warped content for it to have any long term effect. But still, it is something to wonder about.