Tom Sharpless is one of the true pioneers of panoramic and VR photography, with a background of decades in computer programming. He has contributed a number of interesting thoughts, techniques, and ideas to the panoramic photography community, most notably the “Panini” projection which is perhaps the most natural-looking way to display extremely wide angle (120-170 degrees) scenes.
What is your background?
Medical imaging software engineer. In the last decade, I was a large scale integrated circuit design software engineer. I was supporting people designing chips. it’s a bit like making a movie. In the core team working on a big chip there’s probably about 600-800 engineers, divided into numerous specialist groups and they’re supported by tens of thousands of engineering workstation computers put into big farms to simulate elements of the design process. the chip has to be simulated exhaustively in 3 or 4 different levels of details before anyone will consider making a test chip because that costs a few million dollars to run. There are logic designers working in languages similar to cobol and c++ and they are good at modeling abstract logical elements. They put together little bits of logic verify that they are correct by running simulations.
I was largely supporting that level of design. The company I was working for, their speciality was one of the black arts. In the transition from pure logic to building a physical chip, the biggest problem there is getting all those idealized clocks to actually happen in a synchronized fashion because there are time delays and bugs and so on. ‘Clock Tree Insertion’ is one of the more mysterious elements of chip design. I didn’t do any of that but I did support logic designers in writing test beds for simulation at the logic level. The special expertise is how to insert instrumentation so you can diagnose what actually went wrong. There is a bit of my logic in one ATI graphics chip actually.
What was it like working with so many smart people?
it was fun, you don’t get asked stupid questions, you get asked things that make you think!
When did you start doing photography?
I never did any photography until I started doing panoramas in 2004. And then I was doing slit scan photography with a modified flatbed document scanner. I thought it would be an interesting hack. I wasn’t the first or last but I did figure it out all by myself. I hacked together a working slit camera out of a scanner and stuck a little TV lens in front of it, and it made unbelievable pictures! So I built a better one using a real 35mm camera lens, and I got the white balance calibration software working better and it made absolutely stunning pictures! you’ve seen panoscan for example, it’s the same idea. the pictures were super resolution, I was getting 30 or 50 megapixel cylindrical panoramas out of it in 2004.
My prettiest one, (pictured) was from an HP sensor but it has a bit of a crappy sensor so this one does not make as good pictures as this one:
this is made from a umax scanner, the kind that costs $49. the only modI faction was soldering ga couple wires that needed connectors. the two main functions I had to add to make it into a camera were putting an infrared blocker in front of the sensor, and a shutter so that I could get quality. the quality of flatbed sensors depends on them calibrating themselves before every scan by looking at a dark strip and white strip. the dark strip measures the noise and the white strip measures the sensitivity. the chip in the sensor as, in it, a realtime corrector to subtract the dark and white image from each pixel, and that gives them a remarkably clean response, better than any digital camera i’ve used in terms of photometric flatness. they’re noisier than slurs but the flatness of field is spectacular.
would you have put this on KS?
maybe but by the time I build the second one in 2008 I knew the rotating slit camera business was heading for oblivion. spheron had made their last new design two years before, panoscan was only making small improvements. I believe you can still buy a panoscan but i’m not sure about it. spheron hasn’t sold a slit camera in a long time.
these cameras have serious limitations. national semI made a scanner chip with wide range of adjustments between the motor stepping speed and video speed, they built the chip ike that because they were building it for so many scanner manufacturers. the customized chips that came later are useless because they’re slaves to the exact requirements of a particular scanner. also in no cases is the low level apI of the chip public! that’s why I could do it in the first place. some people have built cameras out of scanners without an apI but they’re missing a number of things. on my camera I can close the scanner and do a dark field image in order to subtract the noise. I have a white field calibration device that I use once per session which sets the brightness response correction and the white balance.
getting the black and white calibration was the hardest part and that’s what other people have had problems with. they’re not the best slit cameras that have ever been made out there, but I think they rank pretty high.
the biggest limit for me are 1. it’s very slow – each slit of pixels is a separate exposure! you have to do 10,000 exposures. so it takes minutes to scan an image. the longest scan i’ve made is 45 minutes. barely usable but is’a popular picture of mine.
The other limitation is that there is no anti-blooming circuitry in these cheap sensors. the chip in a good camera, I believe has about 60% of its circuitry is devoted to anti-blooming. Blooming is when the light hitting one pixel, leaks to neighboring pixels. CMOS is much less prone to blooming. on a CCD you see a big line when the sun hits it.
So, most of my outdoor pictures have vertical streaks in them because of blooming. you can’t take pictures of stuff with lots of specular reflections in it.
And the whole thing including the laptop hi-cad batteries ended up weighing around 35lbs, so I just sort of gave up. I started being much more interested in the pictures than the gear….. I was beginning to lose interestI in the hardware and I wanted to make panoramic pictures with people in them, and so…. I got an SLR and downloaded panotools, and being a soft are engineer, looked into panotools and was very impressed with it, and soon became aware that there was a major open source project based on panotools called Hugin that was pushing the state of the art. I signed up to be a hugin developer and learn d a great deal from that experience.
Are you still contributing to hugin?
not much anymore, i’m still on the list but not much. the last contribution I made was 4 years ago, putting in the gneral paninI projection. I use it all the time to reproject equirectangualr panos for printing. I made two of my own programs for that called PaninI and PaninI pro. But they aren’t as easy to make a nice paninI perspective view with those as it is with Hugin.
For those of you who haven’t heard it before, it’s not a sandwich and it’s not a sports car. it’s a method fro constructing an alternative projection that was developed in holland in thee 1600 and perfected in italy in the 1700’s. bruno postle and I called it paninI because we figured it out by looking at paints by Panini. 18th century italian painter and professor of perspective at the dutch academy in rome, and taught it to many great painters of the late 18th and early 19th century, paining mostly ruins in huge wide perspectives that you could tell were not real. they rearranged their ruins a lot and painted ruins that never were.
Can you explain how the PaninI projection compares visually to rectilinear or fisheye projection?
If you have seen the work of Piranesi, you’ll see what I mean. Hypothetical or made-up painting of ruins that are unbelievably spectacular, all in black and white. He made his living selling portfolios of 30-50 black and white lithographs of roman ruins or hypothetical buildings he made up himself or combinations thereof. They’re still selling well. a full set of piranesI lithographs will set you back 35 grand. The buildings he drew were so convincing that lots of archaeologists came to believe that it was how roman buildings were actually built and that was what they were going to find when they dug them up!
There is something about his perspective that is terrifically convincing. he really wanted it to look like that. things look taller than they really are.
I spent a lot of time allaying these paintings comparing angles measured in the paintings and angles in the actual buildings such as a church, and with the exception of one of the most interesting paintings, all the paintings I analyzed followed these mathematical rules very accurately.
The primary effect of the paninI projection is to compress the horizontal scale to something much smaller. so you can have a view that is nearly 180 degrees that doesn’t seem distorted. paninI projection solves the problem of showing extremely wide angles, and since most pictures are wider than they are high, this gives you a lot of freedom. and for architecture photography you need vertical lines to stay vertical, and paninI preserves vertical lines.
When I want to make the most convincing looking picture of an interior that is 170 degrees wide in reality, a 100% paninI projection is a bit too much but something like 70% works quite well for me.
The reason I like the paninI projection in hugin is that it vertically squeezes the top and bottom and for the purpose of eliminating the worst part of the paninI projection which is that horizontal lines above and below the equator are bulging a lot. in architecture that’s disturbing. in paintings they simply drew those lines straight. in a photo you can’t do that. the general paninI projection is nowhere near what a painter could do, but the squeezes at the top and bottom are nice adjustment so that horizontal lines are straighter.
Despite being 75 years old, the future is interesting and I’m involved in it. the next big tech development is 2.5d / 3d stitching – make the first spherical 3d panoramic content reliably. you can’t do it now. the big problem with spherical 3d, unless you do it with a single camera, only for stills of course, the stitcher is going to modify each spherical image according to what control points it finds and it beds them out of shape a bit. so not only you get the left-right disparity but you get a lot of other disparities due to the fact that the images are aligned differently, and that interferes with stereo viewing. so you need real 3d stitching which means the stitcher isn’t allowed to warp the control points until it knows the depths of the control points. so finding a good way to assign depths to control points is the first step. that will solve a lot of registration problems. it requires getting depth maps out of pictures before you’ve lined them up. it’s something that hasn’t been solved yet. when it is solved, it will be possible to churn out still and video 3d panos very reliably. as long as oculus keeps growing there can be a big market for that.
The next step after that is real 3d – true virtual reality with views that are not at the center of the camera array. that’s already being done in hollywood so it’s a matter of time before everyone else is doing that too. I’ve thought of a couple applications – one is taking picture of buildings where you can’t get far enough away to get the right perspective. or the building you’re interested in when you see it has telephone poles and wires in between. Some of those defects could be solved by 3d stitching, taking a bunch of images taken from different points, combining different pints of view, and you could see past wires and poles, or synthesize a new view as if you were standing much further away.