Commit Graph
132 Commits
Author SHA1 Message Date
James R. Barlow 0c4ee5af4e Add 'auto' output type for best-effort PDF/A without Ghostscript
- Add new '--output-type auto' option (now the default) that produces
  best-effort PDF/A without requiring Ghostscript
- When verapdf is available, use speculative PDF/A conversion
- Without verapdf, pass through as PDF/A if safe (input claims PDF/A
  or --force-ocr was used), otherwise output as regular PDF
- Make Ghostscript check conditional - only required for pdfa* output types
- Update soft error tests to explicitly use --output-type pdfa since they
  exercise Ghostscript failure modes
- Fix Tesseract OSD error handling to check both stdout and stderr for
  known non-fatal messages like "Too few characters"
2026-01-09 00:56:00 -08:00
James R. Barlow b2b6a7c4b1 Pass OMP_THREAD_LIMIT to Tesseract subprocesses instead of modifying parent env
Instead of setting OMP_THREAD_LIMIT in the parent process's environment,
calculate the thread limit in the validate hook and pass it through to
Tesseract subprocess calls via the env parameter. This avoids polluting
the parent process's environment while still controlling Tesseract's
thread usage.
2026-01-06 18:43:29 -08:00
James R. Barlow 7a4b98974c Integrate fpdf2 renderer and remove legacy hOCR renderer
- Update pipeline to use fpdf2 renderer as default
- Remove legacy hocrtransform PDF renderer (_font.py, _hocr.py,
  pdf_renderer.py)
- Update CLI and options for fpdf2 renderer
- Add fpdf2 dependency to pyproject.toml
- Update graft module for fpdf2 multi-page rendering
2026-01-06 13:45:44 -08:00
James R. Barlow 83a43408c2 Refactor tesseract thresholding to use enum type
Replace integer-based thresholding parameter with ThresholdingMethod
enum for improved type safety. The CLI still accepts the same string
values (auto, otsu, adaptive-otsu, sauvola) but internally uses a
strongly-typed enum. This makes the code more maintainable and catches
type errors at development time.
2025-12-27 13:32:56 -08:00
James R. Barlow 2cb0973540 Improve Ghostscript API/CLI definitions 2025-12-27 01:40:12 -08:00
James R. Barlow 0d6e0c4560 Merge branch 'main' into dev 2025-12-24 00:44:18 -08:00
James R. Barlow e613db6a82 Fix Ghostscript 10.6 JPEG corruption by repairing truncated images
Ghostscript 10.6 has a bug that truncates JPEG data by 1-15 bytes.
This adds detection and repair by comparing output images to input
images and restoring the original bytes when truncation is detected.

- Add warning when GS 10.6+ is used with PDF/A output
- Add _repair_gs106_jpeg_corruption() to fix damaged JPEGs after
  Ghostscript processing
- Add unit tests for the repair function
2025-12-23 14:56:24 -08:00
James R. Barlow e9bfce34f1 Fix ruff linting issues
- Use X | Y syntax in isinstance calls (UP038)
- Remove trailing whitespace from blank lines (W293)
2025-12-23 03:07:48 -08:00
James R. Barlow 16c2604a07 Remove lossy JBIG2 support, retain lossless JBIG2 only
Lossy JBIG2 has been removed due to well-documented risks of character
substitution errors (e.g., 6/8 confusion). The --jbig2-lossy and
--jbig2-page-group-size arguments are now deprecated and ignored with
a warning.

Changes:
- Remove jbig2_lossy and jbig2_page_group_size from OCROptions
- Simplify optimize.py to use single-image JBIG2 encoding only
  (no symbol dictionaries/JBIG2Globals)
- Remove convert_group() from jbig2enc.py
- Deprecate CLI args with warnings for backward compatibility
- Update documentation to explain lossless-only JBIG2
2025-12-23 02:45:07 -08:00
James R. Barlow 9ebba91466 Use plugin namespace access pattern throughout codebase
Migrate all code from flat accessor pattern (options.tesseract_timeout)
to the plugin namespace pattern (options.tesseract.timeout).

Key changes:
- Fix _get_plugin_options to raise AttributeError for unregistered
  namespaces instead of silently returning None
- Add _convert_value helper to convert PathLike to str for plugin
  model field compatibility
- Filter out _plugin_cache_* entries from JSON serialization to fix
  worker process serialization (test_simulate_oom_killer)
- Update tesseract_ocr.py, ghostscript.py, _validation_coordinator.py,
  and _pipelines/ocr.py to use options.tesseract.* and
  options.ghostscript.* accessors
- Update tests to use setup_plugin_infrastructure() for plugin
  model registration
2025-12-23 02:02:21 -08:00
James R. BarlowandClaude Opus 4.5 3e46b039ed feat: add use_cropbox parameter to align rasterizer APIs
Added use_cropbox parameter to rasterize_pdf_page hook to allow
choosing between MediaBox and CropBox rendering:

- Default is use_cropbox=False (MediaBox) for consistency with
  Ghostscript's existing behavior
- Ghostscript: passes -dUseCropBox when use_cropbox=True
- pypdfium: calculates crop values to expand from CropBox to MediaBox
  when use_cropbox=False

This aligns both rasterizers to produce the same output dimensions
by default, making the rasterizer choice transparent for page
geometry.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 12:29:17 -08:00
James R. BarlowandClaude Opus 4.5 ae783b4ae6 fix: add thread safety lock to pypdfium plugin
pypdfium2/PDFium is not thread-safe - concurrent calls from different
threads can crash or corrupt the process. Added a module-level lock to
serialize all pdfium operations.

PIL image processing and file I/O are done outside the lock since they
are thread-safe, minimizing lock contention.

For maximum parallelism, users can use process-based parallelism
(use_threads=False) where each process has its own pdfium instance.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 12:29:17 -08:00
James R. BarlowandClaude Opus 4.5 b9f488d65c test: add comprehensive tests for --rasterizer option
Add test_rasterizer.py with tests covering:
- Basic rasterizer option validation ('auto', 'ghostscript', 'pypdfium')
- Rasterizer + --rotate-pages interaction
- PDFs with nonstandard MediaBox/TrimBox/CropBox
- Direct hook tests verifying plugins respect the option

Also fix pluggy parameter passing: make 'options' a required parameter
(no default) in the hookspec so pluggy forwards it to implementations.
Update test plugins and test_rotation.py to pass the new parameter.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 12:29:17 -08:00
James R. BarlowandClaude Opus 4.5 ed813cec67 feat: add --rasterizer CLI option to select PDF rasterization backend
Add user control over which rasterizer is used for PDF page rendering:
- 'auto' (default): prefers pypdfium when available, falls back to Ghostscript
- 'pypdfium': force pypdfium2 (errors if not installed)
- 'ghostscript': force traditional Ghostscript rasterizer

Changes:
- Add rasterizer field with validation to OCROptions model
- Add --rasterizer CLI argument in the Advanced options group
- Update rasterize_pdf_page hookspec to pass options to plugins
- Update pypdfium plugin with check_options hook for availability check
- Update both plugins to respect the rasterizer option

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 12:29:17 -08:00
James R. BarlowandClaude Opus 4.5 938ce8e285 fix: make pypdfium plugin optional with Ghostscript fallback
- Remove check_options hook from pypdfium that raised error when
  pypdfium2 wasn't installed
- Return None from pypdfium's rasterize_pdf_page when pypdfium2 is
  unavailable, allowing the hook to fall through
- Restore Ghostscript's rasterize_pdf_page hook as fallback

This allows OCRmyPDF to work without pypdfium2 installed, using
Ghostscript for rasterization as before.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 12:29:17 -08:00
James R. Barlow cf3fb6e89b Fix raster_device settings for pypdfium rasterizer 2025-12-21 12:29:17 -08:00
James R. Barlowandaider 3482ea5fe5 refactor: Modularize rasterize_pdf_page into separate PDF, page, and image processing functions
Co-authored-by: aider (anthropic/claude-sonnet-4-20250514) <aider@aider.chat>
2025-12-21 12:29:17 -08:00
James R. Barlow e85c5bbb4d refactor: Simplify error message and code formatting in pypdfium plugin 2025-12-21 12:29:17 -08:00
James R. Barlowandaider 740b0bddc6 feat: add pypdfium2 rasterization plugin for OCRmyPDF
Co-authored-by: aider (anthropic/claude-sonnet-4-20250514) <aider@aider.chat>
2025-12-21 12:29:17 -08:00
James R. Barlowandaider f6fcdfa618 fix: allow 0 as valid value for jbig2_page_group_size
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:48 -08:00
James R. Barlowandaider 01ea6c2b8b feat: add ValidationCoordinator for cross-cutting validation
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:48 -08:00
James R. Barlowandaider f02d733d31 feat: add legacy field mapping to plugin registry
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:48 -08:00
James R. Barlowandaider 28d6ea0f10 feat: Add CLI generation methods to plugin option models
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:48 -08:00
James R. Barlowandaider 6913ec7cb8 feat: add PluginOptionRegistry for dynamic option models
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:48 -08:00
James R. Barlowandaider b7640bdb9c feat: implement plugin option models with backward compatibility
This commit introduces Pydantic models for plugin-specific options in OCRmyPDF, focusing on:
- Creating TesseractOptions, OptimizeOptions, and GhostscriptOptions
- Adding backward compatibility properties
- Maintaining existing CLI and API functionality
- Preparing for future plugin option registration system

The changes include:
- Added type-annotated option models with validation
- Updated OCROptions to include legacy fields
- Added backward compatibility properties for jbig2 options
- Prepared groundwork for dynamic plugin option handling

Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:48 -08:00
James R. Barlow 1f493ba789 refactor: post-AI code cleanup 2025-12-21 12:21:47 -08:00
James R. Barlowandaider 65b89cafde fix: use available_cpu_count fallback for jobs in hOCR
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:47 -08:00
James R. Barlowandaider 7b37f57b1c refactor: replace Namespace with OCROptions in plugins and validation
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:46 -08:00
James R. Barlow 414d80fc16 Deprecate semfree and don't auto activate it
Instead the standard executor will fall back to threads.

semfree caused test failures  with Py3.14:
https://github.com/ocrmypdf/OCRmyPDF/issues/1558

In retrospect and with emerging Python tech like freethreading, semfree is becoming less necessary. We can use threads for the time being.

A consequence is that performance may be lower on Lambda and Termux when we are using threads and not shelling out work.
2025-09-11 17:13:04 -07:00
James R. Barlow f974e3b3c1 ghostscript: change input filename order for 10.03.1
Ghostscript now expects the pdfa.ps file to precede other files. Fixes #1359.
2024-07-22 14:56:03 -07:00
James R. Barlow 214a333e2d Block Tesseract 5.4.0 2024-06-11 14:44:54 -07:00
James R. Barlow 59f6bc8306 More Tesseract-specific language checks to its plugin 2024-06-01 00:15:50 -07:00
James R. Barlow 5fdcb7602b Make downsampling large images that Tesseract would otherwise error on into default behavior
Fixes #1281
2024-04-07 13:43:20 -07:00
James R. Barlow 065bddbc6c Reformat with ruff format 2024-04-07 00:25:32 -07:00
James R. Barlow 686481982a Fix naming of hOCR rendered files 2024-03-22 13:27:20 -07:00
James R. Barlow 6a746a1cbb ruff linting/Python 3.10 cleanup 2024-02-14 12:41:51 -08:00
James R. Barlow 7d63355c3c Use hocr renderer for LTR languages 2024-02-12 01:41:41 -08:00
James R. Barlow f87fc7b12d gs: check for text-corrupting versions 2024-01-07 01:33:49 -08:00
James R. Barlow 103c3e0cd6 Drop Ghostscript CVE warning and adjust version down to 9.54
Most distros have probably now backported the CVE fix. Dropped version to 9.54 so RHEL 9 can use recent ocrmypdf.
2024-01-01 13:02:57 -08:00
James R. Barlow 1db8b0b943 Temporarily revert default renderer to sandwich while investigating regressions in macOS Preview 2023-12-23 16:19:44 -08:00
James R. Barlow 5b2f2e6290 Merge branch 'feature/modernhocr' 2023-12-03 15:17:02 -08:00
James R. Barlow f6e90a5934 hOCR renderer is now default 2023-12-02 19:58:00 -08:00
James R. Barlow c90d5cd84b Fix Ghostscript installation instructions and add warning 2023-11-29 14:10:04 -08:00
James R. Barlow fec53be841 Remove next major release deprecations 2023-11-21 00:47:51 -08:00
James R. Barlow d2dbea6cf8 Reorganize progress bars so they can be typed properly 2023-10-24 00:54:31 -07:00
James R. Barlow 46a279a49a Improve passing of arguments to workers
The executor system was built around passing only a single
argument to workers, which was
always PageContext. For other tasks, all actual arguments were packed in
a tuple, which meant we needed intermediate functions to unpack the
tuple.

The situation is now rationlized and resembles how Python handles
argument passing to familiar multiprocessing tools.
2023-10-24 00:54:31 -07:00
James R. Barlow 7ce9d08b2d Define progress bar plugins formally instead of "tqdm-like" 2023-10-24 00:54:31 -07:00
James R. Barlow 6127f7abd6 tqdm_kwargs to progress_kwargs 2023-10-24 00:54:31 -07:00
James R. Barlow 113a6b45bd ruff autofixes (mostly typing.* -> collections.abc.*) 2023-10-01 00:02:53 -07:00
James R. Barlow e9419d2c40 Update notes on concurrency 2023-09-30 17:05:58 -07:00