Spreads for Post Processing
Moderator: peterZ
-
- Posts: 5
- Joined: 03 Mar 2015, 16:05
- Number of books owned: 5000
- Country: USA
Spreads for Post Processing
I want to thank everyone for literally “tons” of valuable information.
I have run into a snag using Spreads only for Post Processing.
I have multiple books scanned in as .jpg images and want to use Spreads to Post Process them.
I have been able to Spreads to work fine (as far as I know), but when I launch it in Post Process mode and point it to a directory with the .jpg images, it keeps returning that the directory was empty and not doing anything.
Is there a file structure or specific files that are needed for Spreads to process the images?
Thanks,
Rich
I have run into a snag using Spreads only for Post Processing.
I have multiple books scanned in as .jpg images and want to use Spreads to Post Process them.
I have been able to Spreads to work fine (as far as I know), but when I launch it in Post Process mode and point it to a directory with the .jpg images, it keeps returning that the directory was empty and not doing anything.
Is there a file structure or specific files that are needed for Spreads to process the images?
Thanks,
Rich
-
- Posts: 388
- Joined: 01 Jun 2014, 17:04
- Number of books owned: 1000
- Country: United States of America
Re: Spreads for Post Processing
Spreads builds up a fair amount of metadata about the files while it is scanning. But I'm not entirely sure how much is necessary when post-processing. Here is what to try first:
Make a book folder.
Inside of that, make a folder called 'data'.
Inside of that, make a folder called 'raw' and put all your .jpg images inside of it.
This may be enough for it to work properly. If not, then I or Johannes will have to look deeper into the code to see how to make it work.
-D
Make a book folder.
Inside of that, make a folder called 'data'.
Inside of that, make a folder called 'raw' and put all your .jpg images inside of it.
This may be enough for it to work properly. If not, then I or Johannes will have to look deeper into the code to see how to make it work.
-D
-
- Posts: 5
- Joined: 03 Mar 2015, 16:05
- Number of books owned: 5000
- Country: USA
Re: Spreads for Post Processing
Thanks for the information.
I had already tried that after several experimentations.
After running the spread postprocess {project directory} command, the application creates 4 files (bag-info.txt bagit.txt manifest-md5.txt tagmanifest.md5.txt) and moves the raw directory down several directories ( {project directory}/data/data/data/raw ) and then quits.
Here is the output:
~$ spread postprocess ./scans/FarCry2test1/
Workflow: Initializing workflow ./scans/FarCry2test1/
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
bagit: Copying path scans/FarCry2test1/data to paylod directory
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
spreads encountered an error:
Traceback (most recent call last):
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/main.py", line 318, in main
run()
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/main.py", line 308, in run
args.subcommand(config)
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/cli.py", line 352, in postprocess
workflow = spreads.workflow.Workflow(config=config, path=path)
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/workflow.py", line 431, in __init__
self.slug = util.slugify(unicode(self.path.name))
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/workflow.py", line 492, in slug
self.bag.info['spreads-slug'] = value
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/vendor/bagit.py", line 684, in __setitem__
self.save()
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/vendor/bagit.py", line 752, in save
with open(self._path, "wb") as fp:
IOError: [Errno 2] No such file or directory: u'/tmp/tmpIlEZQZ/bag-info.txt'
I think that you are right and that some metadata or information about the images is missing and it is confusing the system.
Thanks for the help.
I had already tried that after several experimentations.
After running the spread postprocess {project directory} command, the application creates 4 files (bag-info.txt bagit.txt manifest-md5.txt tagmanifest.md5.txt) and moves the raw directory down several directories ( {project directory}/data/data/data/raw ) and then quits.
Here is the output:
~$ spread postprocess ./scans/FarCry2test1/
Workflow: Initializing workflow ./scans/FarCry2test1/
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
bagit: Copying path scans/FarCry2test1/data to paylod directory
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
spreads encountered an error:
Traceback (most recent call last):
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/main.py", line 318, in main
run()
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/main.py", line 308, in run
args.subcommand(config)
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/cli.py", line 352, in postprocess
workflow = spreads.workflow.Workflow(config=config, path=path)
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/workflow.py", line 431, in __init__
self.slug = util.slugify(unicode(self.path.name))
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/workflow.py", line 492, in slug
self.bag.info['spreads-slug'] = value
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/vendor/bagit.py", line 684, in __setitem__
self.save()
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/vendor/bagit.py", line 752, in save
with open(self._path, "wb") as fp:
IOError: [Errno 2] No such file or directory: u'/tmp/tmpIlEZQZ/bag-info.txt'
I think that you are right and that some metadata or information about the images is missing and it is confusing the system.
Thanks for the help.
- jbaiter
- Posts: 98
- Joined: 17 Jun 2013, 16:42
- E-book readers owned: 2
- Number of books owned: 0
- Country: Germany
- Location: Munich, Germany
- Contact:
Re: Spreads for Post Processing
- What version of spreads are you using?
- Did you scan with spreads?
- What's the content of "./scans/FarCry2test1/" before you run postprocessing?
spreads: Command-line workflow assistant
-
- Posts: 5
- Joined: 03 Mar 2015, 16:05
- Number of books owned: 5000
- Country: USA
Re: Spreads for Post Processing
Hello.
Sorry it took forever for me to respond. I got “surged” at work and lost most of my free time.
The version of Spreads is reporting as 0.5git20150203.220d
I didn’t perform the scans with Spreads. I captured them with a sheet feed scanner and a flatbed scanner. So the images are all .jpg files.
The contents of the “./scan/FarCry2test1/” folder before running Spreads in postprocessing is this:
scans
|
data
|
raw
|
FarCry001.jpg
FarCry002.jpg
.
.
.
FarCry020.jpg
Make sense?
There are no other files in the folders other than the .jpg files.
Thanks for all the help.
Sorry it took forever for me to respond. I got “surged” at work and lost most of my free time.
The version of Spreads is reporting as 0.5git20150203.220d
I didn’t perform the scans with Spreads. I captured them with a sheet feed scanner and a flatbed scanner. So the images are all .jpg files.
The contents of the “./scan/FarCry2test1/” folder before running Spreads in postprocessing is this:
scans
|
data
|
raw
|
FarCry001.jpg
FarCry002.jpg
.
.
.
FarCry020.jpg
Make sense?
There are no other files in the folders other than the .jpg files.
Thanks for all the help.
- jbaiter
- Posts: 98
- Joined: 17 Jun 2013, 16:42
- E-book readers owned: 2
- Number of books owned: 0
- Country: Germany
- Location: Munich, Germany
- Contact:
Re: Spreads for Post Processing
Thank you, I think you found a bug I'll look more closely into it and hopefully write a test and a fix and get back to you!
spreads: Command-line workflow assistant
- jbaiter
- Posts: 98
- Joined: 17 Jun 2013, 16:42
- E-book readers owned: 2
- Number of books owned: 0
- Country: Germany
- Location: Munich, Germany
- Contact:
Re: Spreads for Post Processing
RrangerRich, this bug has already been fixed a few weeks back, you should just have to update to a newer version. (See here: https://github.com/DIYBookScanner/sprea ... cbee29962a).
Note, though, that the pages won't show up in the web interface and you won't be able to do anything with them. I have not yet finished the code for that, but it should be ready soon.
Note, though, that the pages won't show up in the web interface and you won't be able to do anything with them. I have not yet finished the code for that, but it should be ready soon.
spreads: Command-line workflow assistant
-
- Posts: 5
- Joined: 03 Mar 2015, 16:05
- Number of books owned: 5000
- Country: USA
Re: Spreads for Post Processing
Hello,
I updated to the latest nightly build. spreads 0.5git20150426.f8de
When I run the post processing command now, there are no thrown exceptions.
I get the following text:
Workflow: Initializing workflow ./scans/FarCry2test1/
bagit: Adding path /home/engineer/scans/FarCry2test1/bag-info.txt to payload
bagit: Adding path scans/FarCry2test1/config.yml to payload
Workflow: Starting postprocessing...%
Workflow: Running 'process' hooks
spreadsplug.autorotate: Rotating images
spreadsplug.scantailor: Generating ScanTailor configuration
spreadsplug.scantailor: /usr/bin/scantailor-cli --start-filter=1 --end-filter=5 --layout=1.5 -o=/tmp/tmpFGq0s9.ScanTailor --margins-top=2.5 --margins-right=2.5 --margins-bottom=2.5 --margins-left=2.5 /tmp/st-outlcujtf
.
.
.
The a good deal off text about the ScanTailor options. (Omitted for brevity)
.
.
.
spreadsplug.tesseract: Performing OCR%
spreadsplug.tesseract: Language is "osd"
bagit: Path scans/FarCry2test1/data/done is an empty directory , will be skipped.
bagit: Adding path /home/engineer/scans/FarCry2test1/bag-info.txt to payload
bagit: Adding path scans/FarCry2test1/pagemeta.json to payload
Workflow: Done with postprocessing!
A "data" directory and a "done" directory are created in the "data" directory and the "raw" directory is moved into the created "data" directory.
The "done" directory is empty.
I appreciate all the help and hope that I am not causing you extra work because of my own ineptitude.
Thanks,
Rich
I updated to the latest nightly build. spreads 0.5git20150426.f8de
When I run the post processing command now, there are no thrown exceptions.
I get the following text:
Workflow: Initializing workflow ./scans/FarCry2test1/
bagit: Adding path /home/engineer/scans/FarCry2test1/bag-info.txt to payload
bagit: Adding path scans/FarCry2test1/config.yml to payload
Workflow: Starting postprocessing...%
Workflow: Running 'process' hooks
spreadsplug.autorotate: Rotating images
spreadsplug.scantailor: Generating ScanTailor configuration
spreadsplug.scantailor: /usr/bin/scantailor-cli --start-filter=1 --end-filter=5 --layout=1.5 -o=/tmp/tmpFGq0s9.ScanTailor --margins-top=2.5 --margins-right=2.5 --margins-bottom=2.5 --margins-left=2.5 /tmp/st-outlcujtf
.
.
.
The a good deal off text about the ScanTailor options. (Omitted for brevity)
.
.
.
spreadsplug.tesseract: Performing OCR%
spreadsplug.tesseract: Language is "osd"
bagit: Path scans/FarCry2test1/data/done is an empty directory , will be skipped.
bagit: Adding path /home/engineer/scans/FarCry2test1/bag-info.txt to payload
bagit: Adding path scans/FarCry2test1/pagemeta.json to payload
Workflow: Done with postprocessing!
A "data" directory and a "done" directory are created in the "data" directory and the "raw" directory is moved into the created "data" directory.
The "done" directory is empty.
I appreciate all the help and hope that I am not causing you extra work because of my own ineptitude.
Thanks,
Rich
- jbaiter
- Posts: 98
- Joined: 17 Jun 2013, 16:42
- E-book readers owned: 2
- Number of books owned: 0
- Country: Germany
- Location: Munich, Germany
- Contact:
Re: Spreads for Post Processing
Rich, can you please upload the whole log somewhere? Ideally when running with the "--verbose" flag.
spreads: Command-line workflow assistant
-
- Posts: 5
- Joined: 03 Mar 2015, 16:05
- Number of books owned: 5000
- Country: USA
Re: Spreads for Post Processing
Hello.
Here are the commands that I ran and the resulting log files.
spread --loglevel degug --logfile ./FarCry2test1.log postprocess ./scans/FarCry2test1/
spread --loglevel debug --logfile ./FarCry2test1_out.log output ./scans/FarCry2test1/
Thanks,
Rich
Here are the commands that I ran and the resulting log files.
spread --loglevel degug --logfile ./FarCry2test1.log postprocess ./scans/FarCry2test1/
spread --loglevel debug --logfile ./FarCry2test1_out.log output ./scans/FarCry2test1/
Thanks,
Rich
- Attachments
-
- FarCry2test1_out.log
- Output log file
- (592 Bytes) Downloaded 512 times
-
- FarCry2test1.log
- Post Processing log file
- (1.34 KiB) Downloaded 517 times