Methods To Sense The 3D Surface/Structure Of A Book

[daniel_reetz]

Guys, sorry - I got all of these wired and working this morning, but I don't think I'm going to get to the post office today, which means they won't ship until Tuesday (after the national holiday on Monday). I'm still going to try to get them shipped today, but it seems unlikely. I'm just too busy now getting ready to travel internationally over the next month to spend a lot of time hunting for boxes.

Anonymous, if you could send me your snail mail address via email, that would be great.

[L0g1cM0del]

Ok so here is how I have a regular laser pointer setup using the arm piece of the clear protractor if anyone is interested.

Here is another picture of the lines it makes.

And then here it is showing the lines on a book. I had to take two pictures and overlay one on top of each other because I only have the one laser. For some reason, I had trouble with the bottom line. It wasn't as clear as the top one, that is maybe why it is fuzzier then the top. Anyway, if anyone wants to try and dewarp it here are the measurements (they are approximations because I measured while I was holding them in place. It was just a quick setup to show some results:

the camera is 20" overhead of the book.
Distance from the center of the book to laser stand is 23".
Edge of book to laser stand is 21".
Height from laser to book is 22".
Height from laser to base is 23".
Angle of laser downward is 45 degrees.
Page dimensions are 5 &1/2" wide x 7 & 3/4" tall

[copec]

Hi, my name is Chris Cope, I'm a much better lurker then someone who adds value in posting so forgive the sideways post here...and this is a really belated introduction: I have been DIY book scanning for a few years now. I had a scanner setup very similar to Reetz's scanner, but it is currently disassembled. I am moving on to version 3 of my book scanner. I don't really have a interest in scanning older or out of print books at this time, although in the future I want to build a library with both physical and digital copies, so this is more of a back-burner project. As of right now I mostly scan my college textbooks (been a part time mathematics major for ~6 years now...eeeeech!) by buying a used edition (now old edition) when the new edition comes out (for like $2-5 on amazon usually) and then just run it through the carbide tipped blade on my table saw and then a Fujitsu scanner...yeah thats cheating.

Anyways to be on topic, I have had it in the back of my mind that one day we would reach the point that we can elegantly scan books using some sort of high resolution 3d scan process and then doing all of the "post-production" as it were on the 3d model. It occurred to me as well that someone would be able to use the data from a Kinect in combination with a higher resolution camera to somehow do this. When I googled this I came across this thread and it finally led me to register on this site

As has been pretty well understood it seems to be more of a software problem then pulling together practical hardware. There are many ways to get a 3d model, what we need is a nice software package...which seems to definitely be the hard part.

I myself am an amateur photographer (who isn't these days), and have been experimenting using extension rings on ~80-150 mm of lens or so (with aps-c sized sensor) to make for a really small depth of field to allow me to focus stack images of book sized objects. Then something like helicon focus can make extremely high resolution 3d models. http://www.heliconsoft.com/heliconfocus.html

Theres no way to elegantly process these 3d objects into images without a lot of work, however.

edit: Anyways, I'm really interested in following your guy's work with the lasers. It seems like this is going to be the way to go!

[daniel_reetz]

Hey Chris, I'd love to see what you can do with a setup like that. Gerard tried doing focus stacking using a compact camera, and didn't end up with great results.

I, for one, am very very interested in computational photography style methods of book scanning. Glad you came and introduced yourself!

[andigit]

Worked on some code to figure out the 3D space of the laser plane.

You can browse around here...
https://tanoshi-apps.sourcerepo.com/red ... correction

The most interesting peace of code being:
https://tanoshi-apps.sourcerepo.com/red ... tTable.cpp

Basically this code figures out the 3D location of the laser plane in the image. I am still trying to figure out a good way to figure out the FOV of the image as this determines where the laser plane ends up in the picture.

Right know the origin is sitting directly above the base, as this seems to be the setup we are going for. However, if we really wanted too I can actually move the camera to any location and solve for the 3D space of the laser line using the same math

Oh the code is written in C++ using Qt. If any body wants to run it, qmake and compile. Double click and open one of the sample image from the app. I stuck some of the sample image in the repo, so please let me know if you don't want it there.

[Anonymous1]

@andigit, what are you going to do with the intersection of the laser plane and the book mesh? Steve's approach doesn't even care about the 3D model, and mine maps the 3D coordinates onto a 2D plane.

The main hurdle to get over is the actual dewarping part. What are you thinking of doing (just curious, as the more methods, the better)?

[andigit]

I'm going to slice it into smaller strips and flatten it. Meaning make all the normal of the polygon (0,0,-1), assuming the camera was facing (0,0,1). Then translate each polygon to the same z coordinate, stitch each peaces to gather. If I texture map this polygon with the original image, I will get a de-warped image. Of course in my program I'm not going to texture map a polygon but create a new image out of this information.

Essentially, the polygon that is flatten will tell me how much each strip of the image needs to be skewed, Horizontally and Vertically. In the end its exactly the same thing as what you and steve are doing. I'm just driving the amount of skewing based on 3D coordinate.

For example if I have a strip (this is unrealistic because you wouldn't be able to see this strip but for example).
P = [(0,0,0), (0,-1,0), (0,-1,-1), (0, 0, -1)] normal would be [0, -1, 0]
if I rotate this and make the normal 0,0,-1... I would get.
Q = [(0,0,0), (0, -1, 0), (1, -1, 0), (1,0,0)]
So Q is facing straight up at the camera.

If I plot Q... instead of having a narrow strip I should get a perfect square on the screen.

This tell me that the image area representing P needs to be stretched to fit Q to de-skew.

Does that make since?

[andigit]

Actually you want to rotate the polygon to the inverse of camera normal... Then technically I can take a picture from any angle and be able to de-skew the image.

[Anonymous1]

That seems like it would work.

[Anonymous1]

I've almost got the dewarping working well. Almost.

I used a median smoothing filter coupled with an erosion to remove virtually all of the noise I get, so it should work a bit better now. I also smoothed the book's lines so that the images don't come out as jittery. I'm having some problem with the top of the book (probably a bad camera value, but I'll see), but other than that, it works to some extent. Here's a set of debug images:

IMG_0885.JPG

Screenshot-Test-2.png

Screenshot-Test-1.png

Screenshot-Test.png

Screenshot-Test-4.png

I'm still battling that lens distortion, but I guess I'll have to live with it.

Here's the code (this link will always be current): https://github.com/Anonymous3D/Laser-based-Book-Dewarping/raw/master/dewarp.py