Methods To Sense The 3D Surface/Structure Of A Book

[steve1066d]

I finally had a change to play around with some laser scans.

Here's an example:

after:

Here's the process I'm using:
1. I put down some grid paper, and aligned the lasers so they were parallel. The actual angle doesn't matter. In fact, they don't even need to match.
2. I took a calibration scan with the lasers on.
3. I put a 2x4 block on the base. I use that to calculate the change in position for a the change in height.
4. I measured the distance from the camera to the base.
5. I put the book on the base, then took a picture with the lasers on.
6. Took a few pages with the lasers off.
7. Repeat steps 5-6 for the book.
8. Bring the scans into BSW, rotate and correct for keystoning (I didn't really try to get my camera level).
9. For now I enter the source DPI and the height to the camera.
10. I run the unwarp program. If it finds a laser scan it recaculates the warp. Otherwise it uses the last warp and renders the page.

Notes:
Its very helpful to have two hands to hold the book with, so I'm going to try using a foot control to take the picture.
I think the quality will be better if I were to hold the pages from the sides in the middle. Right now I'm assuming the page is curved directly up and down, which isn't quite the case. w
The only things that really matters for alignment is to make the lasers parallel, and to keep the book aligned with the grid.

Some tricks I"m using:
I detect if it is a laser image by looking at the average brightness of the image. Since the light only fires when the lasers are off, if there is a dark image, it is a laser image.
By measuring the change in height with the block, I avoid having to keep the angles precise.

[Mark Main]

Why do we need a grid? Right now I am thinking two lines are enough.

Re: IR cameras, yep, I'm familiar. But I don't want to be using IR because we'll likely be infringing on Google's patent, and that's not going to help us stay in business.

I'm not yet worried about the line quality at all. We can improve it somewhat with a baffle and/or pinhole. But I'll order a couple of red line generators anyway. Even if it is fuzzy, we can threshold it or use that information to produce a better line (it will get wider as the angle gets more oblique).

Google's patents probably mostly pertain to the software/math side, do you know what the patent #'s are? Even still staying away from IR will reduce costs, imo. So thats another benefit. IR just allows you to go faster since you can shoot one picture.

I wonder if you can get a glass filter that blocks everything but the nm range of the laser light, and a filter that blocks the laser light only... then you could basically do what google does without IR, project the laser light continuously and take both pictures at once. That would dramatically increase costs though for those filters.. heh.

I'm personally for doing it with minimal amount of pre-processing of the images as possible... but writing the software to accept everything from poor quality lines that have lots of bleed light and not positioned 100% accurately to perfect grids would be good.

as for the grids... I donno much about the algorithms involved so I can't say.. I just know that sensing depth usualy involves grids, like the Kinect sensor and google's dewarping tech both use grids... the work being done on the software for the scannable checkerboard to dewarp the page...

My thoughts is if you project a checkerboard pattern, a grid, the same algoritms that is used in that software might work?

It really boils down to the software, do we have an idea how to create it?

I found this on Google's Scanning Technology:

http://www.patentgenius.com/patent/7508978.html
http://www.freepatentsonline.com/7508978.pdf
http://www.npr.org/blogs/library/2009/0 ... 08978.html

I was feeling bad for that Assistant Professor, Yoshihiro Watanabe, working for Professor Masatoshi Ishikawa at the Ishikawa Komuro Laboratory in the University of Tokyo, and assumed this to be an example where the small time inventor gets squashed by the corporate giants (he's the one who made this: http://www.youtube.com/watch?v=tCOXC5PTJj8) But look at the dates.

The Tokyo video is done in March 17, 2010 and the Google Patient was applied for in March 24, 2009. So, I have to assume the Tokyo version is different and that they hope that it will someday be better,faster, and/or cheaper than Google's version?

As a side note to this: one day, when CDC cameras get better and better, one way to flip the pages faster is to allow for more excessively radical angles on the pages to be scanned rather than having them open wide for a moment in time. I would think that a imaging capable parabolic concentrator would be the ticket because it would allow light in from extreme angles while still allowing the image to be converted (as apposed to a compound parabolic concentrator CPC, which is a non-imaging lens due to being to extreme: http://en.wikipedia.org/wiki/File:Non-imaging_Compound_Parabolic_Concentrator_and_Parabolic_Concentrator.png) The only way to get the light shot into the book pages turning so fast and so narrow would then require lighting from all angles concentrated onto the book--this is how a doctor is able to operate on a patient and see--the operating table lights provide a shadowless environment because the light is focused in from a very broad range of angles. I used to work for a medical manufacturer years ago and we made operating table lights. Some less know information about them happens to work here on the scanners too. First of all, notice they all have those rounded prisms, so it's very similar to the Fresnel lens, but I think that the different between an operating light and a true Fresnel lens is that a Fresnel is an imaging lens but the operating table lights make their prisms way way more pronounced and they do this for a reason. Imagine a long narrow prism--now bend it into a circle; now do this again and again with smaller and smaller circles. Now design these prisms so they behave very similar to the Fresnel len design, but they are also very much independent prisms--but these prisms are designed at just the right angles to provide perfectly pure white light--an intense white light that is void of all infrared light and ultra-violet. This would certainly be good for books--you get massive bright white light without the heat focused on the book.

One other side note about the operating table lights--it's actually very important that the lights are pure white and void of infrared and ultraviolet and shadowless: first of all the doctor is working with a tiny hole cut into someone--so the beam is able to light up more inside the hole than you would imagine from a tiny hole simply because it's focused into the hole, and since it comes from wide angles, it's shadowless even with their hands and surgical instruments in there; when a doctor is trying to figure out if something is a certain shade of blue, red or purple... you want the light to be pure white, so color is important to the surgeon; and lastly, when you are focusing all this huge volume of light into a hole, if you did not filter out the UV and IR you could cause some serious problems--especially with the IR. Even raising the temperature of the skin by a couple degrees makes the incision scare more. Hope that I didn't put folks to sleep. Hehehe

[steve1066d]

I've made some progress.

I've modified the laser & lights to take two pictures automatically (one with the lasers, and one with the lights), and have a foot switch to activate it.

I made a timer circuit with a 555 chip to do the following:

The light is normally on, and if the foot switch it tapped it will take a picture normally.
If instead I hold down the switch, it will take a picture, then one second later it will turn off the light and turn on the lasers, then take another picture as soon as the camera recycles (about 1.5 seconds).

So I either take just the picture, or with another picture with the laser lines. The idea is that depending on the book, the warp could be consistent from one page to anther so you'd just need to take laser pictures every 10 pages or so. Or if you want to take laser pictures with every page that is easy to do too.

I'm currently playing with the best way to integrate the warping software into BSW.

[steve1066d]

Here's an example of how I'm using the modified hardware:

[daniel_reetz]

That is SO SLICK. I'm going to try to set my rig up tomorrow. Thanks for the inspiration!

[Shaknum]

Great work!!!

I just ran the last photo through scantailor and page 11 is perfect. Amazing! Page 10 is pretty far off, however, were you working just with the right page, or was this supposed to work well with both pages?

In any event, this is very exciting, I can tell you've put a lot of work into it, big congrats.

[eslavko]

Hello...

This is my 1'st post on this forum. I just wonder how laser 3D works if the page is squashed. For digitizing book should be ok. But for scanning the magazines and other stuff with thin paper might be problem(as only 2 lines are on the image). So I just think about solution's and look twice on google princype. I think it can be done without IR. As we need grayscale or line art output we can use similar princype as is done in anaglyph 3D. The light is masked with green filter so we scan book with only green light. (as near all fax machine). But over the book is another light source with red color and that project checkboard on the book. (for example old diaprojector). The software then separate RGB image to get image in green channel and checkboard on red channel. The separation can be done. I try to make photo of red and green LED in night. The red light is barely wisible on red channel of photo. If I get RAW output from camera then color separation is even better. So I think that same can be done with laser. So I check if I can separate red laser in same way. Doesn't work. The laser beam seems just to bright and saturate all colors.
And with checkboard the image can be automaticaly unwarped/unrotated/unkeystoned.
Similar can be done with laser but they should project more lines and we need to take two photos as laser beam can't be erased from image. (at least I can't)

[CCDeye]

Hi everybody.
I have easy idea 3D surface reconstruction. No lasers.

drawing_u235.jpg

P.S.
Sorry, i've got some problems with my English.

I like this idea.. Single camera shot, easy to reconstruct a 3D surface from the 2 images.
The bad thing is that you must sacrifice some image space on the top and the bottom, depending of the book thickness.
I may try this "at home" I'll try to scan the top image to find the book boundaries, store the float values as "height values", do the same on the bottom image and then use a dense flat vertex buffer (like on my terrain engine) DirectX and HLSL with vertex texture fetch to offset the vertices in the vertex shader in order to generate the 3D surface of the book or even better, without using vertex textures, lock the buffer and update the vertices on the CPU.
Edit:
One thing that could mess this setup up is, that the reflection on the mirror could come from the opposite side of the book.
My English isn't perfect either, but I'll try to explain.
If the the right page bottom right corner of the book is curved up, that curve could be registered on the both reflections. The top and the bottom. The computer then wouldn't know if the bottom right corner of the page is curved up, or the upper right corner.
I'm thinking of a solution like this for example :
Put a green LED to illuminate the top book side and for example red LED to illuminate the bottom side. Then when the computer scans the images it will look for a color threshold to distinguish the top and bottom "curvature".

[Shaknum]

It seems like there has been little action on this lately. Has any progress been made, or is the idea becoming less feasible?

[steve1066d]

For me, life has gotten in the way, but I'm hopeful I'll be able to turn my attention back to it soon.