There is one specific type of JPEG images that Scan Tailor doesn't support. That would be JPEGs with arithmetic compression. The lack of support is due to patent restrictions. I can suggest two options:
1. Don't scan to JPEG. Scan to some lossless format, like TIFF (usually lossless) or PNG (always lossless).
2. Convert your JPEGs to TIFF or PNG. Note that conversion to TIFF may just involve wrapping JPEG data into a TIFF container, so it might still not work.
When Scan Tailor asks you to enter DPIs manually, never enter arbitrary values. The video tutorial shows how to estimate the real DPI.