fpdf2 >= 2.8.7 emits a custom begincidchar Encoding CMap for CFF-based
CID fonts (e.g. NotoSansCJK). pdfminer.six returns <CMap: None> for such
CMaps, so text extraction yields empty output. Switch to pdftotext (poppler)
which handles the new encoding correctly.
The API previously clobbered PIL.Image.MAX_IMAGE_PIXELS unconditionally
on every call, so host applications (e.g. Paperless-NGX) that configured
the PIL limit before invoking ocrmypdf.ocr() saw their setting silently
overwritten with the 250 MP default. Make max_image_mpixels default to
None and only apply the override when the caller explicitly sets it.
The CLI default of 250 MP is unchanged.
Fixes#1665
The JPEG truncation bug (1-15 bytes) persists in Ghostscript 10.7.0.
Update the warning message to show the actual GS version instead of
hardcoding "10.6.x", and remove the stale date reference. Also make
the test warning filter match the new message format.
fpdf2's shape_text() produces RTL ligature glyphs (e.g. lam-alef) with
multi-character CMap entries whose character order gets reversed by the
bidi algorithm during text extraction, producing garbled output like
"سالح" instead of "سلاح".
For invisible text (the production OCR overlay path), bypass text shaping
and use encode_text() with pre-reversed strings. encode_text() maps
characters 1:1 in logical order, avoiding the ligature CMap issue. The
pre-reversal compensates for bidi reversal by text extractors. Since the
text is invisible (Tr=3), the lack of joining forms is harmless.
Add RTL text extraction tests that verify glyph stream order, ToUnicode
CMap 1:1 mappings, and correct logical order for Arabic (including
lam-alef ligature) and Hebrew scripts.
The API's 'language' param was silently dropped because OcrOptions uses
'languages' (plural). Map language->languages in create_options() and
_pdf_to_hocr(), coercing bare strings to lists and splitting '+'
separated codes to match CLI behavior.
_build_text_layer_ctm() returned None when text_rotation was 0,
skipping the origin translation needed for pages with non-zero
MediaBox origins (e.g. JSTOR PDFs with [0, 100, 595, 982]). This
caused the text layer to be offset by the page origin amount.
Always compute the full CTM and only return None when the result
is the identity matrix.
Fixes#1630
OcrOptions.tesseract_timeout defaulted to 0.0, which caused
subprocess.run(timeout=0) to immediately raise TimeoutExpired before
Tesseract could produce any output. The CLI was unaffected because
argparse defaults --tesseract-timeout to 180. Change the OcrOptions
default to None so the plugin's own default (180s) is used.
Fixes#1636
Fixes#1630 where --redo-ocr would shift OCR text vertically on PDFs
with non-zero mediabox origins (e.g., [0, 100, width, height+100]).
The bug occurred in _graft_fpdf2_text_layer where the Form XObject BBox
was set to the text layer's mediabox [0, 0, w, h] instead of the base
page's mediabox [0, 100, w, h+100]. This caused a coordinate mismatch
between the BBox and the transformation matrix, resulting in text being
positioned incorrectly.
The fix changes line 450 in _graft.py to use base_mediabox instead of
mediabox, making the fpdf2 renderer consistent with the sandwich renderer
which already used base_mediabox correctly.
This issue commonly affected:
- JSTOR PDFs (generated by iText with cropping)
- Cropped PDFs from various tools
- PDFs with non-standard coordinate systems
Added regression test that creates a PDF with offset mediabox origin
and verifies --redo-ocr preserves coordinates correctly.
Ghostscript may fail when asked to rasterize at very low DPI values
(below 10 on either axis). This adds a workaround that uses a minimum
of 10 DPI for the Ghostscript call, then resizes the output image to
match the dimensions that would have resulted from the original low
DPI request.
Fixes#1612
Use a Pydantic field validator to convert string input to list[str]
during OcrOptions construction, simplifying the type from
`str | list[str] | None` to `list[str] | None`. Security validation
(path injection check) now happens at construction rather than in
check_options_preprocessing().
Allow users to bypass the TaggedPDFError when processing Tagged PDFs
by setting --tagged-pdf-mode=ignore. This is useful when users know
they want to OCR a Tagged PDF despite the warning.
- 'default': Error if --mode is default, otherwise warn (current behavior)
- 'ignore': Always warn but continue processing (never error)
FlateDecode+DCTDecode compressed images are essentially deflated JPEGs,
typically created by OCRmyPDF's optimizer. This change ensures pdfinfo
correctly identifies them and should_visible_page_image_use_jpg treats
them as JPEG-origin images, allowing JPEG output when appropriate.
The test expected MediaBox preservation for pdfa output, but this only
works when verapdf is available for speculative PDF/A conversion.
Without verapdf (Linux/Windows CI), Ghostscript normalizes the MediaBox.
Also convert pikepdf.Array to list in assertions for clearer error
messages, avoiding pytest repr issues with pikepdf objects.
The pypdfium rasterizer was producing output images that differed by 1
pixel compared to Ghostscript due to floating-point precision issues in
dimension calculations.
Root cause:
- pypdfium used harmonic mean of x/y DPI to calculate a single scale
factor, losing the distinction between x and y DPI
- No DPI rounding like Ghostscript's 6-decimal precision
- Compound rounding errors when converting points to pixels
Solution:
1. Round DPI to 6 decimals to match Ghostscript's precision
2. Calculate expected output dimensions using separate x/y DPI values
3. Handle dimension swapping for 90°/270° rotations
4. Resize output image if off by 1-2 pixels (graceful correction)
This ensures pixel-perfect matching with Ghostscript while being
minimally invasive and only resizing when necessary.
Changes:
- Modified _render_page_to_bitmap() to calculate expected dimensions
- Modified _process_image_for_output() to correct small discrepancies
- Updated rasterize_pdf_page() to pass dimensions through pipeline
- Parametrized rotation tests to run with both rasterizers
All 45 rotation tests now pass with both pypdfium and ghostscript.
Fixes test_rotated_skew_timeout with pypdfium rasterizer.
Introduce a new --mode (-m) argument that consolidates the three
mutually exclusive OCR processing options into a single enum:
- default: Error if text is found (standard behavior)
- force: Rasterize all content and run OCR (replaces --force-ocr)
- skip: Skip pages with existing text (replaces --skip-text)
- redo: Re-OCR pages, stripping old text layer (replaces --redo-ocr)
The legacy flags --force-ocr, --skip-text, and --redo-ocr remain as
silent aliases for backward compatibility. Both CLI and API usage
continue to work unchanged.
Introduce --ocr-engine option to select between OCR engines:
- 'auto' (default): Uses Tesseract
- 'tesseract': Explicit Tesseract selection
- 'none': Skip OCR entirely (for PDF processing only)
Key changes:
- Extend OcrEngine ABC with generate_ocr() and supports_generate_ocr()
for direct OcrElement tree output (bypasses hOCR)
- Add get_ocr_engine(options) hook parameter for engine selection
- Implement NullOcrEngine for --ocr-engine none
- Export OcrElement, OcrClass, BoundingBox from ocrmypdf package
- Add ocr_tree support to grafting pipeline
This prepares the foundation for pluggable OCR engines while maintaining
full backward compatibility with existing Tesseract-based workflows.
- Check all required output files exist before declaring cache hit,
not just stderr.bin
- Add 'hocr' to list of cached output file types
- Fix timeout=0.0 causing immediate timeout on cache miss by treating
it as "no timeout"
- Add new '--output-type auto' option (now the default) that produces
best-effort PDF/A without requiring Ghostscript
- When verapdf is available, use speculative PDF/A conversion
- Without verapdf, pass through as PDF/A if safe (input claims PDF/A
or --force-ocr was used), otherwise output as regular PDF
- Make Ghostscript check conditional - only required for pdfa* output types
- Update soft error tests to explicitly use --output-type pdfa since they
exercise Ghostscript failure modes
- Fix Tesseract OSD error handling to check both stdout and stderr for
known non-fatal messages like "Too few characters"
Insert space characters between words in the fpdf2 renderer so PDF
readers like pdfminer.six can properly segment words during text
extraction. Some PDF readers rely on explicit space characters rather
than inferring word boundaries from positioning.
- Use itertools.pairwise to iterate consecutive word pairs
- Render space immediately after each word (content stream order matters)
- Skip space insertion between CJK words (no spaces in CJK text)
- Use 5% line height threshold to filter OCR noise
- Support RTL text direction
Replace pm.hook.method() calls with pm.method() calls to match the
refactored plugin manager that now uses composition over inheritance.
The hook attribute is no longer directly exposed; instead, type-safe
methods are provided directly on the plugin manager class.
Introduce a fast path for PDF/A conversion that uses pikepdf to add
PDF/A structures directly (sRGB ICC profile and XMP metadata), then
validates with verapdf. If validation passes, skip Ghostscript entirely.
If validation fails or verapdf is unavailable, fall back to the existing
Ghostscript conversion path.
New files:
- src/ocrmypdf/_exec/verapdf.py: CLI wrapper for verapdf validator
- tests/test_verapdf.py: Test suite for new functionality
Modified:
- pdfa.py: Add speculative_pdfa_conversion() and helpers
- _pipeline.py: Add try_speculative_pdfa() function
- _pipelines/_common.py: Integrate speculative path into postprocess()
- Add hOCR test fixtures for Latin, Arabic, CJK, Devanagari scripts
- Add tests for fpdf2 renderer, multi-font manager, system font provider
- Add multilingual rendering tests
- Update existing tests to use fpdf2 renderer
Ghostscript 10.6 has a bug that truncates JPEG data by 1-15 bytes.
This adds detection and repair by comparing output images to input
images and restoring the original bytes when truncation is detected.
- Add warning when GS 10.6+ is used with PDF/A output
- Add _repair_gs106_jpeg_corruption() to fix damaged JPEGs after
Ghostscript processing
- Add unit tests for the repair function
Lossy JBIG2 has been removed due to well-documented risks of character
substitution errors (e.g., 6/8 confusion). The --jbig2-lossy and
--jbig2-page-group-size arguments are now deprecated and ignored with
a warning.
Changes:
- Remove jbig2_lossy and jbig2_page_group_size from OCROptions
- Simplify optimize.py to use single-image JBIG2 encoding only
(no symbol dictionaries/JBIG2Globals)
- Remove convert_group() from jbig2enc.py
- Deprecate CLI args with warnings for backward compatibility
- Update documentation to explain lossless-only JBIG2
Migrate all code from flat accessor pattern (options.tesseract_timeout)
to the plugin namespace pattern (options.tesseract.timeout).
Key changes:
- Fix _get_plugin_options to raise AttributeError for unregistered
namespaces instead of silently returning None
- Add _convert_value helper to convert PathLike to str for plugin
model field compatibility
- Filter out _plugin_cache_* entries from JSON serialization to fix
worker process serialization (test_simulate_oom_killer)
- Update tesseract_ocr.py, ghostscript.py, _validation_coordinator.py,
and _pipelines/ocr.py to use options.tesseract.* and
options.ghostscript.* accessors
- Update tests to use setup_plugin_infrastructure() for plugin
model registration
- Update __getattr__ docstring to clarify that plugin models must be
registered for namespace access (e.g., options.tesseract.timeout)
- Update test_json_serialization.py to properly register TesseractOptions
before accessing plugin namespaces
- Worker processes now register plugin models for multiprocessing tests
- Exclude plugin cache keys from extra_attrs comparison in tests
Split the 1288-line info.py into smaller, single-responsibility modules:
- _types.py: Enums, type aliases, lookup dictionaries
- _contentstream.py: PDF content stream parsing, DPI calculation
- _image.py: ImageInfo class and image finding functions
- _worker.py: Concurrency/worker process handling
- info.py: PageInfo, PdfInfo classes (reduced to ~530 lines)
Public API unchanged - all existing imports continue to work.
Comprehensive test coverage for the new hocrtransform components:
- test_ocr_element.py: Tests for BoundingBox, Baseline, FontInfo,
OcrElement dataclass methods (iter_by_class, find_by_class,
get_text_recursive, words/lines/paragraphs properties)
- test_hocr_parser.py: Tests for parsing hOCR files including
page/paragraph/line/word extraction, RTL text, rotated text,
different line types (header, caption), font info, and edge cases
- test_pdf_renderer.py: Tests for PDF rendering including text
extraction verification, page sizing, multi-line content,
text direction, baseline handling, textangle rotation, word breaks,
debug options, and image overlay
Also fixes x_font regex pattern to not capture trailing semicolons.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
James R. Barlow
