Scan Tailor won't accept anything less than 150 DPI, as in 98% of cases that would produce such poor results that you shouldn't even try. You can still fake a higher DPI, but you need to make it proportionally higher for other pages as well.
Anyway, the problem you describe is not about low DPIs. Pages ending up tiny is caused by them having exaggerated DPIs, for example a file tells its 600 DPI while in reality it's 100 or 150. If you inspect DPIs for each page in the "Fix DPI" dialog, you should be able to locate and fix such files there. Note that it doesn't change the files themselves, it just saves the correct values to Scan Tailor's project file. If you've got just two size groups, just apply the correct DPIs for each group - there is no point in trying to locate problematic pages individually if you can mass-fix them. And again, don't do it in the "Needs Fixing" tab, as it only contains obviously wrong pages. Do it in "All Pages" tab instead. BTW, if the "Fix DPI" dialog is not appearing automatically, there is an option to force it to appear in the "Project File" dialog.
When Scan Tailor asks you to enter DPIs manually, never enter arbitrary values. The video tutorial shows how to estimate the real DPI.