Category Archives: stereopsis

Is there a secret problem with depth maps?


It sounds like a great idea to utilize a depth map to extract information and control the depth depicted in an image or series of images. It sounds great for converting a 2D image into a 3D image. It sounds like a great tool for plenoptic cameras to interpolate the data into imagery with depth. Alpha channels are great to use for transparency mapping – so a depth map should be equally useful, shouldn’t it?

Take a look at this depth map:

icedepthmapThis is a depth map created from a plenoptic camera shot of a bunch of ice bits. It is a grayscale image with 256 shades of gray to depict the parts of the ice that are closer to the camera and the parts of the ice that are farther away from the camera. This information is used to adjust the depth of those bits that are closer and farther away by stretching or compressing pixels.

Now check out a rocking animation that uses motion parallax to depict the depth (items closer to you appear to move differently than items that are farther away).

ice

Right away you can notice a few errors in the depth map, and for complex images this is typical and can be edited and “corrected”. But there is something else. Take a close look at the parts of the image where the depth map is seemingly correct. Sure, you can see the depth but does it really look like ice? If you are like me, the answer is no. Ice reflects and scatters light in a way that is unique for each perspective. Indeed, there IS binocular rivalry where one eye sees light reflection and distortion that is not present in the other eye’s perspective. This disparity tells us something about the texture and makeup of what we are looking at. Stretching or compressing pixels eliminates this information and only provides depth cues relating to the spatial position of things. For most people, I suspect it is reasonable to assume that this creates a perception conflict in their brains. There is something perceptually wrong with the image above. It does not look like ice because the light coming off of the two perspectives looks the same. A depth map does not provide information regarding binocular rivalry and creates errors as a result. Errors that can’t be fixed. Herein you see the flaw in using a depth map. It throws away all of the binocular rivalry information. In other words, it throws away the information between perspectives that is different.

In my opinion, depth maps take the life out of an image. It removes important texture information which, I believe, is gleaned from how light shifts and changes and appears and disappears as you alter perspective.

This is the secret fundamental flaw with depth maps. Now you can subjectively look at the image above and deem it to be cool and otherwise amazing. That is all good and well, but the truth is that, compared with looking at the real ice, it is fundamentally lacking and does not depict what is seen when you look at the ice in real life.

So, people ask themselves if this is important and some will say yes and some will say no. And there are many examples where you could argue both points of view. I don’t have an argument with that. My position is only to point out that this flaw exists and it should not be ignored.

Advertisements

1 Comment

Filed under 1, 3D, 3D Photography, autostereoscopic, S3D, stereopsis, stereovision

New theory about stereo vision, 3D, stereopsis, binocular vision and depth perception


It might be time to expand the way we think of human visual perception.  What we “see” is a construct of our brain and how it processes the stream of data that is input from our senses. The vast amount of raw data that our brains receive from our eyes, set aside the data from our other senses for now, is not something that we typically think about. We open our eyes and see stuff.  We’ve spent a lot of time learning about the parts of the eye and the mechanics, but I’m not sure that teaches us very much about “seeing”.

Understanding computers gives us a new way to think about this, specifically the converting of data (the signals our eyes send to the brain) into conscious perception. We aren’t born with all of the “software” needed to perceive the signals coming from our eyes. “Software” is created over time as the brain interprets and learns cause and effect through experience. I believe the brain never stops tweaking that processing and makes all sorts of modifications in the same way that computer software has upgrades that provide desirable new features and ease of use functions and performance enhancements and so on.

What we see and how we perceive what we see is a function of the snapshot in time of the current version of our vision “software”. Maybe that’s a radical idea, but there is anecdotal evidence that this might be true. I became aware of it when I noticed that each time I looked at a 3D image of an African tribal mask that it looked different from what I remembered. It was the same picture, it had not changed but how I perceived the image did change.

The weird thing about the image of the mask was that I did not have the same reaction to a 2D image of it. The 2D image always looked the same. The 3D image always looked slightly different. In my experience, my brain seems to be much more aggressive at tweaking how I perceive images with depth than it is when I look at flat images.

Having said that, it isn’t noticeable for all 3D images. Images that are life size or larger than life size and ones that I have some level of interest in seem to change in a more noticeable way. I’m curious if other 3D enthusiasts have experienced this.

I think it might be more pronounced with a 3D image because it is an illusion with perception conflicts that the brain must reconcile in some way.

Leave a comment

Filed under 3D, Perception Conflicts, stereopsis

Seeing With The Brain.


Common sense tells us that we see with our eyes. Afterall, when we close our eyes we stop seeing. Right?

Well, when you think for a minute you realize that’s not true. There is this thing called the mind’s eye and dreaming and envisioning, etc. Truth is, the eyes are little more than data acquisition devices that feed the brain with information. Actually, to be more precise; the eyes stream flawed data to the brain with tons of errors and giant missing pieces of data.

The amount of processing the brain performs to make vision possible is staggering. Scientists have written that up to 1/4 of the entire brain is involved in vision processing and interpretation. How we see and what we see is influenced by everything we have seen before. It is also influenced by what we hear, what we smell, what we taste and what we touch. Don’t believe it? Well, science proves it. One example, off the top of my head, was demonstrated at an audio engineering society convention in New York City many years ago. There were rooms with different resolution video monitors and different speaker systems. As it turned out, the room deemed to have the highest video quality was not the one with the best video monitor, but the one with the best sound system.

Much of the time what we think we see really doesn’t match with reality. Much of what we see doesn’t even make it out of our subconscious. So, when 3D cinematographers obsess over camera spacing (inter-axial distance) and convergence and depth of field as it relates to eye geometry, they are misguided in my humble opinion. The brain is not limited to the geometry of the eye, or it’s limitations. If it was, we would have two big black circles where the eye has no receptors (where the optic nerve is connected).

Indeed, how we see and what we see varies greatly from person to person. Then, there are people with eye problems and vision impariment. People that can’t fuse and have double vision.  Who’s to say that in a room filled with 99 people who have strabismus and one person who can see with stereopsis that the people with strabismus wouldn’t be “normal” given that they represented the majority?

How the majority of people see is the result of evolution and natural selection. Human vision is not the best of what nature can create. There are examples of eyes that are superior to human eyes in terms of clarity, detail, color, focus, etc. In the near future, there will be machine to biological connections that might enhance or even replace our eyes with superior devices.

My point to this rambling is that it is a mistake to limit the way multi perspective imagery is created to analytics based solely on eye geometry and how the eyes work. As I begin my research into analyzing the brain and how it responds to multi perspective imagery, I hope that there are discoveries that enlighten and enrich our perception of the space between things and the importance of textures and reflective properties to the interpretation of the world around us.

There is more to it than this:

http://www.nlm.nih.gov/medlineplus/ency/imagepages/9708.htm

2 Comments

Filed under 3D, Perception Conflicts, stereopsis, stereovision, strabismus

Boston Globe, Tom Keane, The curse of 3-D movies, Jan. 2, 2011


Hi Tom,

Your article in today’s Globe where you celebrate your inability to see with stereovision is a fascinating study that I have seen before. For some reason, some people with strabismus believe that they have no disadvantage (and from the standpoint of your article you seem to think you have an advantage) when it comes to how they perceive the world. Nothing could be further from the truth. The ill effects of strabismus have been well documented by noted scientists including Oliver Sacks, Susan Barry, Dominick Maino, Frederick Brock and a host of others. I suggest you Google those names and do some reading. Surprisingly, your “lazy eye” condition might be improved by vision therapy and that might surely change your perception with regards to 3D.

The fact that a motion picture can be seen by an audience in 3D has nothing to do with whether or not the movie is good or bad. However, 3D can most certainly make a movie better as well as worse in the same way digital cameras and projection equipment can make a movie better as well as worse.

Instead of slamming the technology, how about slamming directors and producers for poor implementation of 3D? …And heralding directors and producers for good implementation of 3D? But you indicate you can’t see 3D, so how would you know if the 3D was good or bad?

3D is not a novelty. 3D is the way the majority of the population perceives the world and everyday life. It is one of the ways we distinguish “real” from “referential” imagery. This most certainly, in the hands of a skilled director/producer, can make for amazing motion picture making – and there are many scenes that have been produced in 3D that achieve amazing quality and “realness”.

I’m fine if you want to pan the 3D implementation of a film for having technical problems. However, I don’t see how you would be qualified since you don’t see with stereovision. Indeed, I submit that you are not qualified to write the article you wrote. It would be analogous to a deaf person panning the BSO because the conductor waved his arms without authority. That person might be able to see the string section moving their bows and feel the vibrations of the air around them but I think judging the performance in a widely circulated newspaper would be an overreach.

I encourage you to seek help for your eye alignment condition. A great resource is www.COVD.org

Your comment “…2-D versions of the same films are clearer and more engrossing.” is way off the mark for those who can see with stereovision. Nothing could be further from the truth. You do the public a disservice making that statement. Parents need to be encouraged to do everything possible for a child with lazy eye or strabismus as it can lead to a whole host of problems where ADD and learning disability can be misdiagnosed and a child needlessly medicated when all they required was vision therapy.

Seeing with 3D is a very big deal. I encourage you to find out if vision therapy could help you. In the meantime, I suggest you reconsider reviewing anything 3D until you are able to see and judge imagery with binocular disparity (stereopsis).

As to your dismissive tone with regards to Avatar, you really missed what happened there…

Good luck with vision therapy, Tom. I really mean that. I hope you gain stereovision and with it, the ability to write a meaningful article that provides an accurate perspective. Your disability absolutely does not provide you with ANY advantage. In this mindset, you are wrong in my humble opinion (and in the opinion of many others like you who have gained stereovision later in life). It is quite possible that you could overcome your disability through vision therapy. Please check it out.

Leave a comment

Filed under 3D, 3D Motion Picture, stereopsis, stereovision, strabismus, vision therapy

I am presenting a paper at SPIE January 25, 2011 at 5:30 PM Paper 7863-49


SPIE (the International Society for Optical Engineering)  See:
http://spie.org/x16218.xml is holding a conference on 3D imaging from Jan. 23 – 27 in San Francisco, CA. My paper and presentation: “Human perception considerations for 3D content creation” is about the problem of perception conflicts as they relate to 3D imagery and what to do about them.

I first started thinking about this when I saw an old lenticular photograph of Queen Elizabeth. The photograph could be viewed with stereopsis but the Queen looked like she was dead. Watching the movie Beowulf, while not in 3D, also gave me the creeps as the characters had a dead aspect to them. I noticed some 3D lenticular photographs of people presented with a doll-like character. I then started to notice things in 3D movies that didn’t seem right. When details disappeared into blackness or got blown out to white I noticed an uneasy feeling while looking at that part of the 3D presentation.

Indeed, every time something was presented in 3D that was atypical or not possible to see in the real world, I could detect a feeling of conflict present at some level in my subconsious and I started to manifest a sensitivity to it with regards to recognizing when it was happening.

All of these observations got me thinking about the various mechanisims that we use to see and interpret depth, space and texture. Certainly vergence is the primary mechanism, but as I became more aware of supporting clues like accommodation, motion, luminance dynamic range, binocular rivalry, field of view and so on, I came to a realization.  I realized that when non-vergence depth clues weren’t complementary that those elements or perceptions in conflict required suppression to continue viewing without some sort of physical effect occurring (typically unpleasant such as headache, nausea, etc.).

My paper is a start to the investigation of the importance of supporting perception cues as it relates to stereovision.

*Vergence is the simultaneous movement of both eyes in opposite directions to obtain fixation and the ability to see depth.

*Accommodation is the automatic adjustment in the focal length of the lens of the eye to permit retinal focus of images of objects at varying distances. It is achieved through the action of the ciliary muscles that change the shape of the lens of the eye.

Leave a comment

Filed under 3D, 3D HDR, 3D Health Issues, 3D Motion Picture, 3D Photography, autostereoscopic, binocular disparity, binocular rivalry, HDR, High Dynamic Range, Perception Conflicts, S3D, stereopsis, stereovision

The Role of Accommodation And Seeing Depth


As I read stereo filmmaker’s blogs and read lists and group postings, the comments center around: What are the rules? What is the right camera lens spacing? How do you avoid breaking the stereo wind0w. Too much parallax? What is the right camera toe-in? If you just have the right formula, then perfect 3D can be accomplished. Everyone wants to know the right thing to do.

If something doesn’t fit that mindset, it is quickly dismissed. For example, it is impossible to simulate accommodation (the eye muscles focusing on objects at different distances) when all of the visual content is on a single plane (the screen in a 3D theater, or the plane of the printed surface on a multi perspective lenticular. Well, we seem to be able to see 3D just fine anyway–so it must not be important. 

Common sense tells me that is wrong.

Since Whetstone’s observations in the 1800’s scientists have fixated on the notion that two eyes and binocular disparity is what seeing depth is all about. Common sense tells me that it is the most important component, but only because it facilitates the supporting capabilities to fully define seeing depth – which, in the overall scheme of things is about 70% of the process in my humble opinion.

Binocular disparity (a slightly different image seen from each eye) makes us aware that depth and space exist. Without that, the other supporting depth cues are considerably weaker. Just ask anyone who acquired stereopsis vision later in life. They will tell you that things like motion parallax did not give them a sense of depth – whereas someone having stereopsis vision all their life will see depth with one eye looking out the window of a moving car.

So it is with accommodation. I didn’t really “get” the importance of accommodation until one day I was walking directly under several electrical powerlines overhead. I looked up with the power lines going across my field of view from left to right where they simply appeared as horizontal lines overhead. I could determine that some power lines were higher (farther away) than others – but NOT because of binocular disparity. In this instance, there is little, if any, binocular disparity since they are horizontal lines with little perceived texture. Each eye is seeing a horizontal line. Indeed, if I draw a left eye set of horizontal lines and a right eye set of horizontal lines, the two images (with horizontal displacement) have NO depth when viewed with a stereoscope or similar viewing device.

It wasn’t motion parallax that made it possible to see depth. It was accommodation that clearly made it possible for me to see the various heights of the different power lines. The depth cue to me was equally powerful to that of motion parallax. And accommodation solved the problem of being able to see depth where binocular disparity provided no clues with regards to the height of the power lines.

So, it is no great leap to realize that where binocular disparity becomes weak as a sense of depth, other components of vision take over to fill in the gap. Seeing 3D is a system whereby the brain uses multiple senses to perceive and interpret a scene. When one or more of the senses conflict, and that conflict isn’t suppressed, it stands to reason that people might feel some level of discomfort no different from motion sickness or dizzyness. This HAPPENS when senses conflict and suppression of a conflicting sense is not suppressed.

Over time, most people – probably all people if so motivated – can overcome and suppress conflicting sensory input. And that process is adaptable. Take people at sea… they get their sea legs and then get their land legs. For some, it happens faster than for others.

Lets jump back to accommodation for a second. When we remove accommodation from our stereopsis vision, then it stands to reason that it isn’t a good idea to present imagery whereby accommodation would play a major role in seeing depth. For a 3D motion picture – that means AVOIDING large amounts of depth in a scene – or at least large amounts where there isn’t a specific area of attention.

Wait, that means it isn’t a good idea for things to stick way out of a 3D screen!

Yes, that’s right. Especially things with a horizontal orientation. And whenever there are scenes with a lot of depth then as many other depth cues should be implemented like motion parallax to help compensate for the supporting sense (accommodation) that is missing.

Once we start to understand that seeing with depth is a system of many perceptions, then we start to realize how to be better stereographers and 3D image artists.

1 Comment

Filed under stereopsis

Perspective Interpolation – Specularity and Refraction Problems


So, how about converting 2D to 3D or converting two perspective 3D into multi-perspective autostereoscopic… Technology certainly should easily make that possible, right?

The answer is a bit complicated. Because for some images it is quite possible to achieve excellent results. Unfortunately, for many images and scenes it truly is impossible to create accurate 3D from 2D and/or interpolate additional perspectives for autostereoscopic displays.

Case in point? Look at the animation below:

In the background painting there are tiny bits of highly reflective particles embedded in oil paint. These dots of light reflect bright points of light depending upon the perspective. They “come on” quickly as you change perspective because of the paint occlusion where you see them in one eye but not the other. Any program that interpolates views would not know what to do with a picture like this. Morph the dots of light? In real life, they don’t morph, they pop on with the light brightening as the perspective angle changes.

Now, take a close look at the glass gems. Notice how their specularity is influenced by the perspective position relative to the background?  Notice the refraction as you see the background through the transparent glass. Unless you modeled the gems in a 3D program and rendered them, there would be no way to interpolate with a pixel warping program what is going on with the look and texture of these gems as they change perspective.

What happens typically with a conversion is an abysmal mess for items with specularity and refraction. It looks 3D for sure – but in no way is representative of reality. And this is the conundrum. There is no uniformity or consistency with regards to 2D to 3D conversions or 2 perspective to multi perspective conversions. It is completely content based and the results are dependent upon the subject matter.

Binocular disparity and as this example demonstrates, binocular rivalry where one perspective contains elements not visible in the other perspective create monumental problems for conversion.

The solution? Shoot multiple perspectives. And this is the path that I have been forced to take to create consistent and uniform results. Indeed, fewer than 10 perspectives does not yield quality, uniform results in my humble opinion. Can fewer than 10 perspectives work? The answer is yes if what you are photographing has no specularity or refraction properties and the texture is smooth and uniform. But as an artist, I find that restriction way to limiting and live in a world that consists mostly of refractive material (water) and glass and gems and metals. Indeed, just look around and the world is filled with specular and refractive content.

Even portraits pose a problem because unless the person has extremely dry eyes, they glisten as the moisture that coats the eye creates specularity and refraction. Of course, if you don’t have a close up or reduce the resolution then it isn’t that noticeable. But here again, as an artist I find that too limiting.

I do not understand the willingness of people to ignore these problems. While it is true that in many cases specularity and refraction are subtle and nuanced. But given that 3D mimics the way we see real life, shouldn’t 3D be subtle and nuanced? Perhaps the gross over emphasized poke you in the eye effects are doing the potential of 3D a disservice?

That’s my view. But what do I know?

3 Comments

Filed under 3D, 3D Photography, autostereoscopic, S3D, stereopsis