Dewraping with focus stacking

DIY Book Scanner Skunk Works. Share your crazy ideas and novel approaches. Home of the "3D structure of a book" thread.

Moderator: peterZ

User avatar
Gerard
Posts: 154
Joined: 17 Oct 2010, 07:15
Number of books owned: 0
Location: Berlin (Germany)

Dewraping with focus stacking

Post by Gerard »

Hi, i just wont to ask, how accurate (minimal step) it is possible to change the focus, from a series of images with different focus, it is maybe possible calculate a 3d map
and then dewrap the page(s)
if we have a stack (maybe a video with continues changing focus) it is possible to calculate the local sharpens (e.g. variance filter) and then searching for every pixel the focus image with the highest sharpens

if some one is interested in this i could try it, the software part (i don't have a chdk enabled camera)

http://www.google.com/url?sa=t&source=w ... EeUFVSzqpg
http://en.wikipedia.org/wiki/Focus_stacking
User avatar
daniel_reetz
Posts: 2812
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Dewraping with focus stacking

Post by daniel_reetz »

Two problems with this approach for DIY scanning:

Compact cameras have a large depth of focus (in focal region) so getting good defocus can be problematic.
Taking many pictures of one page can take as long as scanning on a flatbed, so it becomes less attractive.

That said, I can provide you with some example bracketed focus shots next week. Are you a programmer?
User avatar
daniel_reetz
Posts: 2812
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Dewraping with focus stacking

Post by daniel_reetz »

Another issue is that for non-textured areas (white space) focus and defocus are underdetermined/ill specified.

I don't say this to discourage you in any way, though, I am a computational photography hobbyist and have read literally hundreds of papers in computer vision, image processing, and computational photography looking for solutions like this. I'd be happy to see you try it.
User avatar
Gerard
Posts: 154
Joined: 17 Oct 2010, 07:15
Number of books owned: 0
Location: Berlin (Germany)

Re: Dewraping with focus stacking

Post by Gerard »

daniel_reetz wrote:Two problems with this approach for DIY scanning:

Compact cameras have a large depth of focus (in focal region) so getting good defocus can be problematic.
Taking many pictures of one page can take as long as scanning on a flatbed, so it becomes less attractive.
i think i would be enough to take this bracketed focus shots at the beginning or in an interval, calculation the deformation grid and using this on all images

maybe i miss the point, do we need for every page the 3d map?

or we could also use a reference picture with a grid,
daniel_reetz wrote: That said, I can provide you with some example bracketed focus shots next week.?
would be nice
daniel_reetz wrote:Are you a programmer
yes i hope i can contribute something, but i should invest the time more in the wood work
daniel_reetz wrote: Another issue is that for non-textured areas (white space) focus and defocus are underdetermined/ill specified.
yes i had this problem also with my scanner, blank page with small amount of text was blured (auto focus problem)
but in this case maybe it is ok to use the data from images before
daniel_reetz wrote: I don't say this to discourage you in any way, though, I am a computational photography hobbyist and have read literally hundreds of papers in computer vision, image processing, and computational photography looking for solutions like this. I'd be happy to see you try it.
i am wondering why no one build a 3d scanner with this, maybe it will fail, or not many people had contact to chdk, with manual focus it is not accurate and auto focus does not give you the singe defocus images
spamsickle
Posts: 596
Joined: 06 Jun 2009, 23:57

Re: Dewraping with focus stacking

Post by spamsickle »

Gerard wrote:Hi, i just wont to ask, how accurate (minimal step) it is possible to change the focus, from a series of images with different focus, it is maybe possible calculate a 3d map and then dewrap the page(s)
CHDK permits one to set the focus down to the millimeter, though in practice I think the limitations of the mechanics will make it less accurate than that.
Gerard wrote:if some one is interested in this i could try it, the software part (i don't have a chdk enabled camera)
It took me a couple of reads through this thread to understand what you were proposing. While I question how well this would work, I'm more than happy to provide test images. As far as I know, cameras don't capture focus distance in their EXIF data*, so I'm curious to begin with how you would build your 3D map. I suppose if we agree that each image in a stack is focused 1mm from the last image, and you really can determine which image is "most focused" at a given pixel (the second thing I'm curious about), you could build a relative depth map. Relating one pixel to its neighbors and resolving inconsistencies seems like the third hurdle.

*EDIT: I just tried an online EXIF viewer, which reported the focus for a sample book scan as "86 cm to infinity", which seems about right. I don't know what the precision is, but apparently this IS part of the EXIF data in some form. The Times regrets the error...

It also seems to me that, if the camera is centered over the book, the "corners" of the book are further away than the "center" of the book, by substantially more than the typical "curl" from gutter to edge. If one truly determines distance by using focus, is this going to be something that the dewarping step can handle?

Doubts notwithstanding, I have a public domain "Harvard Classic" which, when opened to the middle of the book, bows up a couple of centimeters at most. Would you prefer a 400-image stack at 1mm nominal difference, or a smaller stack with a larger "step" between images?
User avatar
Gerard
Posts: 154
Joined: 17 Oct 2010, 07:15
Number of books owned: 0
Location: Berlin (Germany)

Re: Dewraping with focus stacking

Post by Gerard »

spamsickle wrote:
Gerard wrote:Hi, i just wont to ask, how accurate (minimal step) it is possible to change the focus, from a series of images with different focus, it is maybe possible calculate a 3d map and then dewrap the page(s)
CHDK permits one to set the focus down to the millimeter, though in practice I think the limitations of the mechanics will make it less accurate than that.
i hope that it will be possible to interpolate the data by fitting functions in the data, only the step size should be fixed

spamsickle wrote:
Gerard wrote:if some one is interested in this i could try it, the software part (i don't have a chdk enabled camera)
It took me a couple of reads through this thread to understand what you were proposing. While I question how well this would work, I'm more than happy to provide test images. As far as I know, cameras don't capture focus distance in their EXIF data*, so I'm curious to begin with how you would build your 3D map. I suppose if we agree that each image in a stack is focused 1mm from the last image, and you really can determine which image is "most focused" at a given pixel (the second thing I'm curious about), you could build a relative depth map. Relating one pixel to its neighbors and resolving inconsistencies seems like the third hurdle.
i will not relay on the exif data, i think it is not imported to know the real distance, a relative distance of a pixel to an other pixel is maybe enough


spamsickle wrote: It also seems to me that, if the camera is centered over the book, the "corners" of the book are further away than the "center" of the book, by substantially more than the typical "curl" from gutter to edge. If one truly determines distance by using focus, is this going to be something that the dewarping step can handle?
my first scan was bad aligned, the page was not parallel to the camera, i had a warped image and the part of the page that was more far away was blurred, so i came up with the idea to use the blurring a way to measure the distance
spamsickle wrote: Doubts notwithstanding, I have a public domain "Harvard Classic" which, when opened to the middle of the book, bows up a couple of centimeters at most. Would you prefer a 400-image stack at 1mm nominal difference, or a smaller stack with a larger "step" between images?
i prefer the stack with the small distance (400-image)
spamsickle
Posts: 596
Joined: 06 Jun 2009, 23:57

Re: Dewraping with focus stacking

Post by spamsickle »

I haven't forgotten you. I took a couple of sequences yesterday -- let the autofocus calculate the distance (680 mm), then bracketed it between 480mm and 880mm. I couldn't see anything in the extreme images which was out of focus. Then I tried 380 to 1080 with a step of 5 mm between them, and the results were the same, visually. I think the "minimum focus" when not in macro mode is something like 510 mm, so maybe attempting to set it lower prevented the rest of the sequence from working. I'll shoot another sequence today, with 510 mm as the lower bound all the way to 10,000 mm or so, just to make sure the set_focus command is working as documented. I should have something for you in a day or two -- either a sequence of test images, or a lame excuse.
User avatar
Gerard
Posts: 154
Joined: 17 Oct 2010, 07:15
Number of books owned: 0
Location: Berlin (Germany)

Re: Dewraping with focus stacking

Post by Gerard »

thanks
i looking forward to the result
spamsickle
Posts: 596
Joined: 06 Jun 2009, 23:57

Re: Dewraping with focus stacking

Post by spamsickle »

Okay, Dan's comment about depth of field was spot on. Taking the pictures varying the focus over a small range yielded pictures which I couldn't distinguish from one another at all. What I ended up doing is generating 300 pictures, while varying the focus with CHDK from 510 mm to 2010 mm. Even then, the 2010 mm focus was only a little blurry, still easy to read. Makes me doubly annoyed at those autofocus pictures that come out unreadable when there isn't enough text in the center where it's focusing -- I think the camera must just be focusing at infinity when that happens.

So anyway, the pictures have been split into 3 zip files, because the site where I'm hosting them has a 200 MB limit on each file. Here they are:

http://www.4shared.com/file/NgNwnTmn/FocuStk1.html
http://www.4shared.com/file/znY-0Z22/FocuStk2.html
http://www.4shared.com/file/NEiRr1bo/FocuStk3.html

Good luck, and let me know what might work better if these turn out to be not what you need.
User avatar
Gerard
Posts: 154
Joined: 17 Oct 2010, 07:15
Number of books owned: 0
Location: Berlin (Germany)

Re: Dewraping with focus stacking

Post by Gerard »

ok my first test between 2 images,
the images have to be aligned, i've excepted this

the other problem seams to be: i see a lot of rectangles in the difference images, jpeg compression rectangles, i will try to eliminate this by averaging some neighbor pictures to one

i will try more pictures
Post Reply