Commit Graph
849 Commits
Author SHA1 Message Date
James R. Barlow be425e7405 Refactor pdfinfo: split info.py into focused modules
Split the 1288-line info.py into smaller, single-responsibility modules:
- _types.py: Enums, type aliases, lookup dictionaries
- _contentstream.py: PDF content stream parsing, DPI calculation
- _image.py: ImageInfo class and image finding functions
- _worker.py: Concurrency/worker process handling
- info.py: PageInfo, PdfInfo classes (reduced to ~530 lines)

Public API unchanged - all existing imports continue to work.
2025-12-22 01:27:23 -08:00
James R. BarlowandClaude Opus 4.5 b4f9673364 Add unit tests for HocrParser, PdfTextRenderer, and OcrElement
Comprehensive test coverage for the new hocrtransform components:

- test_ocr_element.py: Tests for BoundingBox, Baseline, FontInfo,
  OcrElement dataclass methods (iter_by_class, find_by_class,
  get_text_recursive, words/lines/paragraphs properties)

- test_hocr_parser.py: Tests for parsing hOCR files including
  page/paragraph/line/word extraction, RTL text, rotated text,
  different line types (header, caption), font info, and edge cases

- test_pdf_renderer.py: Tests for PDF rendering including text
  extraction verification, page sizing, multi-line content,
  text direction, baseline handling, textangle rotation, word breaks,
  debug options, and image overlay

Also fixes x_font regex pattern to not capture trailing semicolons.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 17:05:49 -08:00
James R. Barlow e162361d28 Make rotation test more robust 2025-12-21 14:42:14 -08:00
James R. Barlow 22d00837e3 WIP box tests 2025-12-21 14:03:28 -08:00
James R. Barlow 0faba42d36 test: Don't save local files 2025-12-21 14:03:28 -08:00
James R. Barlow 41758766a1 Test and fix page box issues 2025-12-21 14:03:28 -08:00
James R. BarlowandClaude Opus 4.5 3e46b039ed feat: add use_cropbox parameter to align rasterizer APIs
Added use_cropbox parameter to rasterize_pdf_page hook to allow
choosing between MediaBox and CropBox rendering:

- Default is use_cropbox=False (MediaBox) for consistency with
  Ghostscript's existing behavior
- Ghostscript: passes -dUseCropBox when use_cropbox=True
- pypdfium: calculates crop values to expand from CropBox to MediaBox
  when use_cropbox=False

This aligns both rasterizers to produce the same output dimensions
by default, making the rasterizer choice transparent for page
geometry.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 12:29:17 -08:00
James R. BarlowandClaude Opus 4.5 ae783b4ae6 fix: add thread safety lock to pypdfium plugin
pypdfium2/PDFium is not thread-safe - concurrent calls from different
threads can crash or corrupt the process. Added a module-level lock to
serialize all pdfium operations.

PIL image processing and file I/O are done outside the lock since they
are thread-safe, minimizing lock contention.

For maximum parallelism, users can use process-based parallelism
(use_threads=False) where each process has its own pdfium instance.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 12:29:17 -08:00
James R. BarlowandClaude Opus 4.5 b9f488d65c test: add comprehensive tests for --rasterizer option
Add test_rasterizer.py with tests covering:
- Basic rasterizer option validation ('auto', 'ghostscript', 'pypdfium')
- Rasterizer + --rotate-pages interaction
- PDFs with nonstandard MediaBox/TrimBox/CropBox
- Direct hook tests verifying plugins respect the option

Also fix pluggy parameter passing: make 'options' a required parameter
(no default) in the hookspec so pluggy forwards it to implementations.
Update test plugins and test_rotation.py to pass the new parameter.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 12:29:17 -08:00
James R. BarlowandClaude Opus 4.5 0ad7f5fc13 feat: add dynamic nested access to plugin options
Completes Phase 5 of the CLI refactoring plan by enabling nested
plugin option access (e.g., options.tesseract.timeout) alongside
the legacy flat access (options.tesseract_timeout).

Changes:
- Add module-level plugin option model registry in _options.py
- Add __getattr__ to OCROptions for dynamic namespace access
- Register plugin models in setup_plugin_infrastructure()
- Add test for nested plugin option access

Plugin option instances are lazily created from flat field values
and cached for subsequent access.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-21 12:21:48 -08:00
James R. Barlowandaider 28d6ea0f10 feat: Add CLI generation methods to plugin option models
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:48 -08:00
James R. Barlowandaider 62ad37b276 refactor: centralize plugin manager setup
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:48 -08:00
James R. Barlow b1de6a6ad4 Add more cached tests 2025-12-21 12:21:48 -08:00
James R. Barlow b7737446e4 cli: push up imports 2025-12-21 12:21:48 -08:00
James R. Barlowandaider 42891346d1 fix: update test files to use new get_options_and_plugins function
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:48 -08:00
James R. Barlow ade3ecd5a1 fix: add error handling in hOCR pipeline 2025-12-21 12:21:47 -08:00
James R. Barlow 1f493ba789 refactor: post-AI code cleanup 2025-12-21 12:21:47 -08:00
James R. Barlowandaider 60182ac8a8 fix: update JSON serialization tests to match default values
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:47 -08:00
James R. Barlowandaider d77d63f1dc feat: add JSON serialization tests for OCROptions in multiprocessing
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:47 -08:00
James R. Barlowandaider e1216eddb0 test: modify test_two_languages to use list of languages
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:47 -08:00
James R. Barlowandaider f5bfd2fd3e Add compat function for pages_from_ranges
fix: handle Pydantic validation errors with correct exit code

Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-21 12:21:46 -08:00
James R. Barlowandaider 66a3e8508e feat: add comprehensive validators to OCROptions
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-13 11:42:23 -08:00
James R. Barlowandaider 7bb3a97208 refactor: update test_validation.py to use Pydantic model validation
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-12-13 11:42:23 -08:00
James R. Barlow d556014185 Remove language warning 2025-12-13 11:41:58 -08:00
James R. Barlow 057eaff36d Skip devnull testing on Windows
No longer seems to work - Windows Server 2025 change, perhaps? Doesn't really matter.
2025-11-10 16:57:30 -08:00
James R. Barlow 599fb1a1f6 Fix test_semfree (skip Python 3.14)
This feature is now deprecated and won't be fixed for Python 3.14. Instead we just use threads on platforms that don't support semaphores.

Closes #1558
2025-09-14 13:02:33 -07:00
James R. Barlow 414d80fc16 Deprecate semfree and don't auto activate it
Instead the standard executor will fall back to threads.

semfree caused test failures  with Py3.14:
https://github.com/ocrmypdf/OCRmyPDF/issues/1558

In retrospect and with emerging Python tech like freethreading, semfree is becoming less necessary. We can use threads for the time being.

A consequence is that performance may be lower on Lambda and Termux when we are using threads and not shelling out work.
2025-09-11 17:13:04 -07:00
James R. Barlow 7e7e2f2e91 Raw value in pdfa XML block uses upper case codes, so account for this 2025-09-08 12:46:26 -07:00
James R. Barlow 4fc0c3a0d5 Add watcher test, such as it is 2025-08-13 01:04:58 -07:00
James R. Barlow 175b743ffe Fix version test 2025-07-03 11:30:05 -07:00
James R. Barlow 45cf92f40b xfail Python logging bug in 3.13.3/4 2025-07-03 09:21:31 -07:00
James R. Barlow 3beabf55e7 Skip optimizing images with pre-blended soft masks
Fixes issue [Bug]: Optimized pdf not rendering with Quartz / Core Graphics #1536
2025-06-12 23:58:43 -07:00
James R. Barlow 6851ea7f11 Remove test since ghostscript error handling changed 2025-04-21 12:23:34 -07:00
James R. Barlow 32322a9fe9 Fix broken test_hocrtransform_matches_sandwich
Expect word similarity rather than exact match. Difference appears to be due to quote styles.

Thanks @QuLogic for reporting.
2025-02-09 13:57:50 -08:00
James R. Barlow 137b054f43 Adjust test again for older Ghostscript 2025-01-27 23:44:37 -08:00
James R. Barlow 65df44f670 Modify tests to deal with variety of Ghostscript versions 2025-01-09 02:14:29 -08:00
James R. Barlow 6edc749023 Fix error handling when PDF contains an invalid image with both ImageMask and ColorSpace set
Fixes #1453
2025-01-07 00:27:07 -08:00
Kara Engelhardt 636623ab49 graft: fix invisible text appearing after strip_invisible_text
strip_invisible_text resets the text render mode on each `BT` (begin text) command. However the text state is not actually reset for each text element, only for each page.

The pdf reference says:

> The text state operators can appear outside text objects, and the values they set
> are retained across text objects in a single content stream. Like other graphics
> state parameters, these parameters are initialized to their default values at the
> beginning of each page.
>
> -- https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/pdfreference1.7old.pdf#page=397

With the current implementation, a text object is only deleted if it contains a `3 Tr` command (setting the text rendering mode to invalid). However the rendering mode may be set once and then not changed for multiple text objects or set outside of a text object.
In that case only the first text object (which contains the `3 Tr`-command) is removed. This not only leaves the other text objects in the pdf, but also makes them visible, since the text object that contained the `3 Tr`-command is removed.

This PR updates `strip_invisible_text` to not reset the rendering mode for each object and to keep track of the rendering mode when the graphic state is pushed/popped.
2024-12-11 18:01:12 +01:00
James R. Barlow 9a075039b5 Remove empty test file 2024-12-02 11:23:35 -08:00
James R. Barlow fe89be5dc0 Fix test broken in commit 85d6fb8c 2024-11-27 15:44:12 -08:00
James R. Barlow a659f83d67 Remove invalid hyperlink annotations to satisfy Ghostscript 10.x during PDF/A conversion
Closes #1425
2024-11-16 19:02:10 -08:00
James R. Barlow dbd3c93757 Fix issue with unpickling HOCRResult
Fixes [Bug]: HOCRResult.from_json() not unpickling correctly #1427
2024-11-10 02:05:57 -08:00
James R. Barlow f77f701a50 Fix quadratic time performance regression on scanning pages 2024-10-27 21:57:53 -07:00
James R. Barlow 6f755321b8 Ignore unpaper warning message when checking version
Fixes #1409
2024-10-27 13:06:30 -07:00
James R. Barlow 18b59c57b4 Refactor our tests that check if we are in a container 2024-10-27 11:55:22 -07:00
Elliott Sales de AndradeandGitHub bb4c47e707 Fix broken test_rotate_page_level (#1382)
Before 42ff7fc842, `make_rotate_test`
always used `resources / 'typewriter.png'`, but after the change the
second call accidentally used just `resources`, which is a directory,
and fails to open.
2024-08-21 01:25:07 -07:00
James R. Barlow 59f6bc8306 More Tesseract-specific language checks to its plugin 2024-06-01 00:15:50 -07:00
James R. Barlow abf9729c61 Semfree test: accept pdfa conversion failed as a valid return code
Fixes #1316
2024-05-21 01:26:11 -07:00
James R. Barlow 7a8cc21e31 Add support for sidecar output to io.BytesIO
Closes #1252
2024-04-07 01:38:55 -07:00
James R. Barlow 065bddbc6c Reformat with ruff format 2024-04-07 00:25:32 -07:00