Category Archives: stereovision

Is there a secret problem with depth maps?


It sounds like a great idea to utilize a depth map to extract information and control the depth depicted in an image or series of images. It sounds great for converting a 2D image into a 3D image. It sounds like a great tool for plenoptic cameras to interpolate the data into imagery with depth. Alpha channels are great to use for transparency mapping – so a depth map should be equally useful, shouldn’t it?

Take a look at this depth map:

icedepthmapThis is a depth map created from a plenoptic camera shot of a bunch of ice bits. It is a grayscale image with 256 shades of gray to depict the parts of the ice that are closer to the camera and the parts of the ice that are farther away from the camera. This information is used to adjust the depth of those bits that are closer and farther away by stretching or compressing pixels.

Now check out a rocking animation that uses motion parallax to depict the depth (items closer to you appear to move differently than items that are farther away).

ice

Right away you can notice a few errors in the depth map, and for complex images this is typical and can be edited and “corrected”. But there is something else. Take a close look at the parts of the image where the depth map is seemingly correct. Sure, you can see the depth but does it really look like ice? If you are like me, the answer is no. Ice reflects and scatters light in a way that is unique for each perspective. Indeed, there IS binocular rivalry where one eye sees light reflection and distortion that is not present in the other eye’s perspective. This disparity tells us something about the texture and makeup of what we are looking at. Stretching or compressing pixels eliminates this information and only provides depth cues relating to the spatial position of things. For most people, I suspect it is reasonable to assume that this creates a perception conflict in their brains. There is something perceptually wrong with the image above. It does not look like ice because the light coming off of the two perspectives looks the same. A depth map does not provide information regarding binocular rivalry and creates errors as a result. Errors that can’t be fixed. Herein you see the flaw in using a depth map. It throws away all of the binocular rivalry information. In other words, it throws away the information between perspectives that is different.

In my opinion, depth maps take the life out of an image. It removes important texture information which, I believe, is gleaned from how light shifts and changes and appears and disappears as you alter perspective.

This is the secret fundamental flaw with depth maps. Now you can subjectively look at the image above and deem it to be cool and otherwise amazing. That is all good and well, but the truth is that, compared with looking at the real ice, it is fundamentally lacking and does not depict what is seen when you look at the ice in real life.

So, people ask themselves if this is important and some will say yes and some will say no. And there are many examples where you could argue both points of view. I don’t have an argument with that. My position is only to point out that this flaw exists and it should not be ignored.

1 Comment

Filed under 1, 3D, 3D Photography, autostereoscopic, S3D, stereopsis, stereovision

Seeing With The Brain.


Common sense tells us that we see with our eyes. Afterall, when we close our eyes we stop seeing. Right?

Well, when you think for a minute you realize that’s not true. There is this thing called the mind’s eye and dreaming and envisioning, etc. Truth is, the eyes are little more than data acquisition devices that feed the brain with information. Actually, to be more precise; the eyes stream flawed data to the brain with tons of errors and giant missing pieces of data.

The amount of processing the brain performs to make vision possible is staggering. Scientists have written that up to 1/4 of the entire brain is involved in vision processing and interpretation. How we see and what we see is influenced by everything we have seen before. It is also influenced by what we hear, what we smell, what we taste and what we touch. Don’t believe it? Well, science proves it. One example, off the top of my head, was demonstrated at an audio engineering society convention in New York City many years ago. There were rooms with different resolution video monitors and different speaker systems. As it turned out, the room deemed to have the highest video quality was not the one with the best video monitor, but the one with the best sound system.

Much of the time what we think we see really doesn’t match with reality. Much of what we see doesn’t even make it out of our subconscious. So, when 3D cinematographers obsess over camera spacing (inter-axial distance) and convergence and depth of field as it relates to eye geometry, they are misguided in my humble opinion. The brain is not limited to the geometry of the eye, or it’s limitations. If it was, we would have two big black circles where the eye has no receptors (where the optic nerve is connected).

Indeed, how we see and what we see varies greatly from person to person. Then, there are people with eye problems and vision impariment. People that can’t fuse and have double vision.  Who’s to say that in a room filled with 99 people who have strabismus and one person who can see with stereopsis that the people with strabismus wouldn’t be “normal” given that they represented the majority?

How the majority of people see is the result of evolution and natural selection. Human vision is not the best of what nature can create. There are examples of eyes that are superior to human eyes in terms of clarity, detail, color, focus, etc. In the near future, there will be machine to biological connections that might enhance or even replace our eyes with superior devices.

My point to this rambling is that it is a mistake to limit the way multi perspective imagery is created to analytics based solely on eye geometry and how the eyes work. As I begin my research into analyzing the brain and how it responds to multi perspective imagery, I hope that there are discoveries that enlighten and enrich our perception of the space between things and the importance of textures and reflective properties to the interpretation of the world around us.

There is more to it than this:

http://www.nlm.nih.gov/medlineplus/ency/imagepages/9708.htm

2 Comments

Filed under 3D, Perception Conflicts, stereopsis, stereovision, strabismus

SPIE Stereoscopic Displays & Applications Conference


I just got back from a very interesting week of listening to scientists from around the world presenting papers on all things 3D.  I was also a presenter at the poster session a few days ago (January 25, 2011) where I talked about human perceptions as it relates to content creation for 3D.

Essentially, my theory is that 3D can’t be described in a single formula which, if followed, assures “perfect” 3D.  Yet this is the approach many 3D content producers are taking and one that many scientists are keen to say that they provide.

I listened and watched as scientists talked about formulas for 3D vision comfort (VC) as if VC was an obtainable and definable numerical value. There were formulas for parallax, screen size, viewing distance, lens to lens spacing for cameras, angular distortions, etc. and many claims with regards to things that define “quality 3D”.

Now, I totally agree that there are many things that can be measured with regards to the human eyes. And that information can be useful in terms of what is the “normal” use for the eyes. And there is plenty of evidence that convergence and accommodation are key components for people being able to “see” 3D. But for me, the real meat and potatoes has little to do with these outward measurements. We don’t see with our eyes anymore than we hear with our ears or smell with our nose or taste with our mouth or touch with our skin. Those things only help to facilitate the experience. The thing at work that actually DOES see and hear, smell, taste and touch is our brain.  Our brain takes in the information and processes it and provides the human perception experience.

The more clues we provide to the brain with regards to 3D, the better the experience. And especially when there are conflicting clues – we need to help offset that by countering with supporting cues.

I think it is commonsense that if you present a 3D image with convergence decoupled with accommodation and you add in camera shake (perceived motion visually, not matched to the inner ear), and you ask the eyes to diverge in a way that would never happen with regular viewing, and you throw in a dose of 2D to 3D conversion errors where the left eye has differences with the right eye and spice it up with a touch of vertical misalignment… There you have a recipe for a formula that probably equals VNC (vision NO comfort). AKA “headache inducer with a side order of vomiting”.

BUT

I submit that even with all of those poorly advised components, that a person so motivated to engage in learning to suppress the perception conflicts mentioned above and learn to overcome and train the muscles to accommodate “unnatural” viewing conditions… that person could watch the content and get something very positive out of it because the content is being processed and understood and given context and meaning by the brain which is incredibly adaptable to new experience.

Think about it, our early ancestors didn’t have books that required focus and convergence for an object held 18″ from our face (sometimes even closer).  Think about the resolving power and eye alignnment required to read the tiny text of a web page on an iPhone. Think about the increase in ambient noise we live with every day that would be deafening to our ancestors. Similarly, I suspect that certain foods that we eat in today’s world would have caused our ancestors to vomit violently. One of my favorite examples is the scene from the movie “Back to the Future” where Michael J. Fox’s character puts earphones on his 1950’s Dad and blasts distorted electric guitar sounds that struck terror and fear into the 1950’s character.

Technology is evolving rapidly, yet evolution takes hundreds and thousands of years. Fortunately, our brain has the ability to adapt. The key is understanding that adaptability and where it is “good” and where it might be “bad”.

Lets say you want to be a weight lifter. Well, I wouldn’t suggest signing up for an extended stay at the orbiting space station where astronauts are “training” their bones to lose density and their muscles to atrophy.  Yes, they are training their bones and muscles because they are exposing their bones and muscles to zero gravity. Well, if you expose yourself to stereoscopic material that has multiple perception conflicts, you are training your brain to ignore and/or accept that reality. If that is in preparation for the need to do that, then that might be a good thing. If not, then perhaps not a very good thing – especially if reversing the brain’s adaptation is difficult.

All of this has got me thinking a great deal about studying the brain in greater detail to learn more about how my imagery, and 3D in general, triggers brain activity and response and perception.

2 Comments

Filed under stereovision

Boston Globe, Tom Keane, The curse of 3-D movies, Jan. 2, 2011


Hi Tom,

Your article in today’s Globe where you celebrate your inability to see with stereovision is a fascinating study that I have seen before. For some reason, some people with strabismus believe that they have no disadvantage (and from the standpoint of your article you seem to think you have an advantage) when it comes to how they perceive the world. Nothing could be further from the truth. The ill effects of strabismus have been well documented by noted scientists including Oliver Sacks, Susan Barry, Dominick Maino, Frederick Brock and a host of others. I suggest you Google those names and do some reading. Surprisingly, your “lazy eye” condition might be improved by vision therapy and that might surely change your perception with regards to 3D.

The fact that a motion picture can be seen by an audience in 3D has nothing to do with whether or not the movie is good or bad. However, 3D can most certainly make a movie better as well as worse in the same way digital cameras and projection equipment can make a movie better as well as worse.

Instead of slamming the technology, how about slamming directors and producers for poor implementation of 3D? …And heralding directors and producers for good implementation of 3D? But you indicate you can’t see 3D, so how would you know if the 3D was good or bad?

3D is not a novelty. 3D is the way the majority of the population perceives the world and everyday life. It is one of the ways we distinguish “real” from “referential” imagery. This most certainly, in the hands of a skilled director/producer, can make for amazing motion picture making – and there are many scenes that have been produced in 3D that achieve amazing quality and “realness”.

I’m fine if you want to pan the 3D implementation of a film for having technical problems. However, I don’t see how you would be qualified since you don’t see with stereovision. Indeed, I submit that you are not qualified to write the article you wrote. It would be analogous to a deaf person panning the BSO because the conductor waved his arms without authority. That person might be able to see the string section moving their bows and feel the vibrations of the air around them but I think judging the performance in a widely circulated newspaper would be an overreach.

I encourage you to seek help for your eye alignment condition. A great resource is www.COVD.org

Your comment “…2-D versions of the same films are clearer and more engrossing.” is way off the mark for those who can see with stereovision. Nothing could be further from the truth. You do the public a disservice making that statement. Parents need to be encouraged to do everything possible for a child with lazy eye or strabismus as it can lead to a whole host of problems where ADD and learning disability can be misdiagnosed and a child needlessly medicated when all they required was vision therapy.

Seeing with 3D is a very big deal. I encourage you to find out if vision therapy could help you. In the meantime, I suggest you reconsider reviewing anything 3D until you are able to see and judge imagery with binocular disparity (stereopsis).

As to your dismissive tone with regards to Avatar, you really missed what happened there…

Good luck with vision therapy, Tom. I really mean that. I hope you gain stereovision and with it, the ability to write a meaningful article that provides an accurate perspective. Your disability absolutely does not provide you with ANY advantage. In this mindset, you are wrong in my humble opinion (and in the opinion of many others like you who have gained stereovision later in life). It is quite possible that you could overcome your disability through vision therapy. Please check it out.

Leave a comment

Filed under 3D, 3D Motion Picture, stereopsis, stereovision, strabismus, vision therapy

I am presenting a paper at SPIE January 25, 2011 at 5:30 PM Paper 7863-49


SPIE (the International Society for Optical Engineering)  See:
http://spie.org/x16218.xml is holding a conference on 3D imaging from Jan. 23 – 27 in San Francisco, CA. My paper and presentation: “Human perception considerations for 3D content creation” is about the problem of perception conflicts as they relate to 3D imagery and what to do about them.

I first started thinking about this when I saw an old lenticular photograph of Queen Elizabeth. The photograph could be viewed with stereopsis but the Queen looked like she was dead. Watching the movie Beowulf, while not in 3D, also gave me the creeps as the characters had a dead aspect to them. I noticed some 3D lenticular photographs of people presented with a doll-like character. I then started to notice things in 3D movies that didn’t seem right. When details disappeared into blackness or got blown out to white I noticed an uneasy feeling while looking at that part of the 3D presentation.

Indeed, every time something was presented in 3D that was atypical or not possible to see in the real world, I could detect a feeling of conflict present at some level in my subconsious and I started to manifest a sensitivity to it with regards to recognizing when it was happening.

All of these observations got me thinking about the various mechanisims that we use to see and interpret depth, space and texture. Certainly vergence is the primary mechanism, but as I became more aware of supporting clues like accommodation, motion, luminance dynamic range, binocular rivalry, field of view and so on, I came to a realization.  I realized that when non-vergence depth clues weren’t complementary that those elements or perceptions in conflict required suppression to continue viewing without some sort of physical effect occurring (typically unpleasant such as headache, nausea, etc.).

My paper is a start to the investigation of the importance of supporting perception cues as it relates to stereovision.

*Vergence is the simultaneous movement of both eyes in opposite directions to obtain fixation and the ability to see depth.

*Accommodation is the automatic adjustment in the focal length of the lens of the eye to permit retinal focus of images of objects at varying distances. It is achieved through the action of the ciliary muscles that change the shape of the lens of the eye.

Leave a comment

Filed under 3D, 3D HDR, 3D Health Issues, 3D Motion Picture, 3D Photography, autostereoscopic, binocular disparity, binocular rivalry, HDR, High Dynamic Range, Perception Conflicts, S3D, stereopsis, stereovision

Blog Neglect… Sorry!


I apologize for neglecting my blog but I’ve been having a debate over at the LinkedIN Stereoscopic 3-D Professionals Worldwide group. There is at least one person who thinks that focus and convergence don’t happen and/or isn’t important or something such as that which confused the heck out of me…

He made a bunch of empirical statements and I challenged him and ignored his flames and insults. I engaged him in a professional manner and the net result of what I got out of the exchange is that he thinks that because of eye geometry and eye spacing that somehow a special angle of projection alignment can create distortion free, error free 3D.

Folks, that is snake oil if I ever heard it.

To justify his case he said he has blurry vision for things up close and that somehow proves that rivalry or discomfort doesn’t exist. My opinion is that the brain is very plastic and adaptable and when we lose vision acuity, we get cataracs or experience eye damage and disease – that the brain adapts to the altered sensory input. I believe the brain is constantly adapting and interpreting sensory input and learns to discard conflicting information and make the most of what it is receiving.

But!

The brain also responds to GAINING sensory information and incorporates things like motion parallax cues and shadow cues and atmosphereic distortion cues and even an eyeglasses prescription changes to augment our perception. Perhaps vision exercises could also improve our ability to perceive things in an enhanced way. It is one of many issues I plan on taking up with Dr. Maino in the coming weeks.

Again, apologies for the delay posting content. I’ll have some interesting commentary on human perception coming soon… Stay tuned and please comment if you have something specific you are interested in exploring with me!

1 Comment

Filed under stereovision, vision therapy

3D – Present A Different Image To Each Eye – Simple, Right? Wrong!


Those of us with “normal” vision see the world with our two eyes in 3D all of the time. We see the space between things and perceive distance, size, texture, etc.

It’s different when you go to a 3D movie or look at one of my autostereoscopic photographs because you are looking at a flat surface and perceive 3D by way of an optical illusion. There are many subtle differences (and many not so subtle) between normal “seeing” and watching a 3D movie that are important to understand and consider. The biggest difference is that for normal viewing the 3D is “real”. You see an object in front of another object and you perceive the distance. When you reach out to touch the object, that perception is verified. A 3D movie on the other hand is created with an optical illusion. It is made possible because our eyes/brain has an amazing capability to decouple focus from convergence and see and perceive the illusion as if it were real.

Some of you are asking, “What does decouple focus from convergence mean?“.

For normal viewing our eyes focus and converge on the same point in space (the thing we are looking at). Just like separate camera lenses, each eye focuses on an object independently. Then our brain processes those two retinal images into a single image with depth. Because the focusing and merging is done separately, we can perform a trick whereby we fool the brain into processing retinal images that converge at a different point from focus. The only thing the brain cares about is the alignment, size and similarity of the images on each retina. It doesn’t matter what the focusing distance is. Our brain just totally disregards this disparity… or does it? Since most people seem to be able to perceive 3D looking at 3D movies, we just make that assumption. My guess is that our brain does enable our perception of focus distance but that we suppress the conflict that happens with a 3D movie in the same way we suppress other perception conflicts (think flying in an airplane for example where our inner ear conflicts with what we see). Perhaps some of us aren’t able to suppress this conflict as easily and manifest some sort of discomfort with the experience; like getting a headache or feeling nausea.

However, it is generally thought of as a “given” that this isn’t a big deal and that perception conflicts occur in nature and the brain just “handles it”. I think that is probably true – but I’m not a scientist or doctor and it would be nice to read that my opinion has some basis in true science.  Perhaps it does and someone will comment?

You still might be wondering how a 3D movie decouples focus from convergence. The fact is that it has to do that. The motion picture screen has to be the point of focus at all times. That is the source of the reflected light and the focus point of the projector in the back of the theater. The left eye and right eye images will be offset from each other based upon whether the objects depicted are in front of the screen or going into the screen. The eyes converge or diverge to align the objects on screen but the point of focus is always the screen surface. At this point, I just have to jump up and down and say THAT IS PRETTY AMAZING!!!  Our brain is really incredible and adaptable in a way that we don’t even have to think about.

Stay tuned for more… coming soon ;^)

3 Comments

Filed under 3D, 3D Motion Picture, S3D, stereovision