The JPEG truncation bug (1-15 bytes) persists in Ghostscript 10.7.0.
Update the warning message to show the actual GS version instead of
hardcoding "10.6.x", and remove the stale date reference. Also make
the test warning filter match the new message format.
fpdf2's shape_text() produces RTL ligature glyphs (e.g. lam-alef) with
multi-character CMap entries whose character order gets reversed by the
bidi algorithm during text extraction, producing garbled output like
"سالح" instead of "سلاح".
For invisible text (the production OCR overlay path), bypass text shaping
and use encode_text() with pre-reversed strings. encode_text() maps
characters 1:1 in logical order, avoiding the ligature CMap issue. The
pre-reversal compensates for bidi reversal by text extractors. Since the
text is invisible (Tr=3), the lack of joining forms is harmless.
Add RTL text extraction tests that verify glyph stream order, ToUnicode
CMap 1:1 mappings, and correct logical order for Arabic (including
lam-alef ligature) and Hebrew scripts.
The inter-word Tz calculation stretched "word " to span from the current
word to the next, producing extreme horizontal scaling (300-500%) for
words far apart (e.g. in tables). Use per-word Tz instead — Td
positioning already handles inter-word gaps correctly.
Fixes#1635
Catch OSError (parent of both FileNotFoundError and
NotADirectoryError) in verapdf.available() so environments where
executing `verapdf` raises NotADirectoryError gracefully fall back
instead of crashing the pipeline. Fixes#1638.
Fixes#1642. Adds an early check in check_requested_output_file() that
raises OutputFileAccessError (exit code 5) if the destination file
already exists and --no-overwrite is set. The option is wired through
CLI, OcrOptions, and the Python API.
The jpg_quality and png_quality options default to None in the pydantic
model, but the fallback check only handled == 0. This caused a TypeError
when calling ocrmypdf.ocr() with optimize >= 2 without explicitly
setting quality values. Fixes#1641.
Replace hatch-vcs dynamic versioning with static version in _version.py
and pyproject.toml. Split CI into build.yml (test + stage draft release
on main) and release.yml (publish from draft on tag push). Docker images
are built on main pushes and re-tagged with the release version on tag
push without rebuilding.
The API's 'language' param was silently dropped because OcrOptions uses
'languages' (plural). Map language->languages in create_options() and
_pdf_to_hocr(), coercing bare strings to lists and splitting '+'
separated codes to match CLI behavior.
_build_text_layer_ctm() returned None when text_rotation was 0,
skipping the origin translation needed for pages with non-zero
MediaBox origins (e.g. JSTOR PDFs with [0, 100, 595, 982]). This
caused the text layer to be offset by the page origin amount.
Always compute the full CTM and only return None when the result
is the identity matrix.
Fixes#1630
OcrOptions.tesseract_timeout defaulted to 0.0, which caused
subprocess.run(timeout=0) to immediately raise TimeoutExpired before
Tesseract could produce any output. The CLI was unaffected because
argparse defaults --tesseract-timeout to 180. Change the OcrOptions
default to None so the plugin's own default (180s) is used.
Fixes#1636
encode_text() maps unicode characters one-by-one to glyph IDs without
any text shaping, producing incorrect output for complex scripts:
Arabic glyphs in wrong order without joining forms, Devanagari conjuncts
broken apart. Replace with shape_text() which runs HarfBuzz for proper
BiDi reordering, Arabic shaping, and Devanagari conjunct formation.
poppler/pdftotext does not carry Tz (horizontal scaling) across
BT/ET boundaries, causing words to appear on separate lines.
Replace per-word BT blocks (via fpdf2's cell/set_stretching API)
with a single BT block per line using raw PDF operators. Each
non-last word gets a trailing space with Tz calculated to span
exactly to the next word's start position.
When Tesseract completely fails to detect text rotation (no textangle,
slope=0), it produces garbage text in tall-narrow bounding boxes. Add
an aspect ratio plausibility check that compares the OCR bounding box
shape to the expected shape of the rendered text (accounting for
baseline slope). Lines where the ratio of aspect ratios is < 0.1 are
suppressed.
Uses a fast path (width >= height) to skip the expensive font
measurement for the common case of normal horizontal text.
Tesseract doesn't output the textangle hOCR attribute for 90-degree
rotated text. Instead, the rotation is encoded as extreme baseline
slope values (e.g., 462.2). Without detecting this, the renderer used
the axis-aligned bounding box height as line_size_height, producing
font sizes of 2000+ pt instead of ~10 pt.
Detect steep baseline slopes (|slope| > 1.0, i.e., > 45° from
horizontal) and extract the effective text rotation angle via atan().
The meaningless slope/intercept values are then replaced with
font-metrics-based defaults.
The previous fix (e62e73e4) only corrected text_rendering_mode →
text_mode in the single-page Fpdf2PdfRenderer, but the main OCR
pipeline uses Fpdf2MultiPageRenderer which still had the old
attribute name. Since fpdf2 has no text_rendering_mode property,
setting it silently created a no-op attribute while text_mode stayed
at FILL — so 3 Tr (invisible text) was never emitted.
Fixes#1631, #1632
Fixes#1630 where --redo-ocr would shift OCR text vertically on PDFs
with non-zero mediabox origins (e.g., [0, 100, width, height+100]).
The bug occurred in _graft_fpdf2_text_layer where the Form XObject BBox
was set to the text layer's mediabox [0, 0, w, h] instead of the base
page's mediabox [0, 100, w, h+100]. This caused a coordinate mismatch
between the BBox and the transformation matrix, resulting in text being
positioned incorrectly.
The fix changes line 450 in _graft.py to use base_mediabox instead of
mediabox, making the fpdf2 renderer consistent with the sandwich renderer
which already used base_mediabox correctly.
This issue commonly affected:
- JSTOR PDFs (generated by iText with cropping)
- Cropped PDFs from various tools
- PDFs with non-standard coordinate systems
Added regression test that creates a PDF with offset mediabox origin
and verifies --redo-ocr preserves coordinates correctly.
- Prioritize uv over pip throughout, with uv as the recommended installer
- Update repology badges: Debian 13, Ubuntu 24.04, Fedora 40/41
- Make Python 3.12 the default (3.11 still supported)
- Promote Homebrew as full-featured option for macOS and Linux
- Add dependency summary table aligned with maintainers.md
- Document uharfbuzz and fonts-noto requirements
- Remove outdated warnings and simplify 32-bit section
Ghostscript may fail when asked to rasterize at very low DPI values
(below 10 on either axis). This adds a workaround that uses a minimum
of 10 DPI for the Ghostscript call, then resizes the output image to
match the dimensions that would have resulted from the original low
DPI request.
Fixes#1612
Make filter_pdf_page, get_ocr_engine, and optimize_pdf return
non-optional types by handling None cases explicitly: raise errors
for required results, return sensible defaults for optional ones.
Use a Pydantic field validator to convert string input to list[str]
during OcrOptions construction, simplifying the type from
`str | list[str] | None` to `list[str] | None`. Security validation
(path injection check) now happens at construction rather than in
check_options_preprocessing().
Allow users to bypass the TaggedPDFError when processing Tagged PDFs
by setting --tagged-pdf-mode=ignore. This is useful when users know
they want to OCR a Tagged PDF despite the warning.
- 'default': Error if --mode is default, otherwise warn (current behavior)
- 'ignore': Always warn but continue processing (never error)
FlateDecode+DCTDecode compressed images are essentially deflated JPEGs,
typically created by OCRmyPDF's optimizer. This change ensures pdfinfo
correctly identifies them and should_visible_page_image_use_jpg treats
them as JPEG-origin images, allowing JPEG output when appropriate.
pypdfium was not converting images to the correct colorspace/mode based
on the raster_device parameter. For example, pngmono should produce a
1-bit image, but pypdfium was outputting full-color or grayscale images.
This caused images to be incorrectly promoted to PNG instead of being
preserved as CCITT/JBIG2 when using --force-ocr with pypdfium, because
the optimizer relies on the image being in the correct mode.
Changes:
- Render in grayscale for pngmono device (better input for 1-bit conversion)
- Add mode conversion to match Ghostscript's native device output:
- pngmono: convert to mode '1' (1-bit)
- pnggray/jpeggray: convert to mode 'L' (8-bit grayscale)
- png256: convert to mode 'P' (8-bit indexed)
- png16m/jpeg: convert to mode 'RGB'
Add new options: --mode, --ocr-engine, --rasterizer,
--continue-on-soft-render-error, --tesseract-non-ocr-timeout,
--tesseract-downsample-large-images, --tesseract-downsample-above,
--unpaper-args (fish), --plugin (fish).
Update --output-type to include 'auto' as default.
Update --pdf-renderer to include 'fpdf2' and mark hocr as deprecated.
Remove non-working options: --remove-background, --threshold.
The test expected MediaBox preservation for pdfa output, but this only
works when verapdf is available for speculative PDF/A conversion.
Without verapdf (Linux/Windows CI), Ghostscript normalizes the MediaBox.
Also convert pikepdf.Array to list in assertions for clearer error
messages, avoiding pytest repr issues with pikepdf objects.
multiprocessing.Queue is a factory function, not a type, so it cannot
be used with the runtime | union operator. Move Queue, UserInit, and
WorkerInit type aliases into TYPE_CHECKING block to avoid evaluation
at runtime.
Python 3.11 is now the minimum supported version. This aligns with
the codebase's use of StrEnum (introduced in 3.11) and removes
compatibility shims that were only needed for older versions.
Document the new v17 API style where OcrOptions can be passed directly
to ocr(). Mark the positional argument style as legacy API for <v17
compatibility. Update examples to use modern syntax.
Allow passing an OcrOptions object directly to ocr() as the first
positional argument, providing a cleaner API for programmatic use.
The old-style API with individual parameters remains fully supported.
The pypdfium rasterizer was producing output images that differed by 1
pixel compared to Ghostscript due to floating-point precision issues in
dimension calculations.
Root cause:
- pypdfium used harmonic mean of x/y DPI to calculate a single scale
factor, losing the distinction between x and y DPI
- No DPI rounding like Ghostscript's 6-decimal precision
- Compound rounding errors when converting points to pixels
Solution:
1. Round DPI to 6 decimals to match Ghostscript's precision
2. Calculate expected output dimensions using separate x/y DPI values
3. Handle dimension swapping for 90°/270° rotations
4. Resize output image if off by 1-2 pixels (graceful correction)
This ensures pixel-perfect matching with Ghostscript while being
minimally invasive and only resizing when necessary.
Changes:
- Modified _render_page_to_bitmap() to calculate expected dimensions
- Modified _process_image_for_output() to correct small discrepancies
- Updated rasterize_pdf_page() to pass dimensions through pipeline
- Parametrized rotation tests to run with both rasterizers
All 45 rotation tests now pass with both pypdfium and ghostscript.
Fixes test_rotated_skew_timeout with pypdfium rasterizer.
Images with [FlateDecode, DCTDecode] filter chain were incorrectly
being marked for additional FlateDecode compression, resulting in
double-compressed data and invalid output PDFs.
Add _already_flate_encoded() helper to check if an image already has
FlateDecode in its filter chain, and skip such images in
_find_deflatable_jpeg().
Migrate watcher.py and pdf_text_diff.py from typer to cyclopts for
CLI argument parsing. Update pyproject.toml to reflect the dependency
change in the watcher optional feature.
Establish clear separation between user-facing optional dependencies
and developer-only dependency groups:
**Optional Dependencies (user features):**
- watcher: File watching service for batch processing
- webservice: Streamlit-based web UI
- Installable via: uv sync --extra <name> or pip install ocrmypdf[name]
**Dependency Groups (developer tools):**
- test: Testing infrastructure (merged from test + extended_test)
- docs: Documentation building tools
- streamlit-dev: Enhanced Streamlit development tools
- dev: General development tools (mypy, ipykernel)
- Installable via: uv sync --group <name> (uv only, NOT pip)
Breaking changes for developers:
- pip install -e .[test] no longer works → use uv sync --group test
- pip install -e .[docs] no longer works → use uv sync --group docs
- pip install -e .[extended_test] removed → merged into test group
No breaking changes for end users:
- pip install ocrmypdf[watcher] still works
- pip install ocrmypdf[webservice] still works
Updated:
- CI/CD workflows to use uv sync --group test
- Docker images to exclude test dependencies
- Documentation to recommend uv with pip as fallback
- pyproject.toml with clear comments explaining both systems
Introduce a new --mode (-m) argument that consolidates the three
mutually exclusive OCR processing options into a single enum:
- default: Error if text is found (standard behavior)
- force: Rasterize all content and run OCR (replaces --force-ocr)
- skip: Skip pages with existing text (replaces --skip-text)
- redo: Re-OCR pages, stripping old text layer (replaces --redo-ocr)
The legacy flags --force-ocr, --skip-text, and --redo-ocr remain as
silent aliases for backward compatibility. Both CLI and API usage
continue to work unchanged.
Introduce --ocr-engine option to select between OCR engines:
- 'auto' (default): Uses Tesseract
- 'tesseract': Explicit Tesseract selection
- 'none': Skip OCR entirely (for PDF processing only)
Key changes:
- Extend OcrEngine ABC with generate_ocr() and supports_generate_ocr()
for direct OcrElement tree output (bypasses hOCR)
- Add get_ocr_engine(options) hook parameter for engine selection
- Implement NullOcrEngine for --ocr-engine none
- Export OcrElement, OcrClass, BoundingBox from ocrmypdf package
- Add ocr_tree support to grafting pipeline
This prepares the foundation for pluggable OCR engines while maintaining
full backward compatibility with existing Tesseract-based workflows.
pypdfium2 expects clockwise rotation values, but OCRmyPDF tracks
rotation in counter-clockwise. Negate the rotation value to fix.
Also refactor nested try/finally blocks to use contextlib.closing()
for cleaner resource management.
- Check all required output files exist before declaring cache hit,
not just stderr.bin
- Add 'hocr' to list of cached output file types
- Fix timeout=0.0 causing immediate timeout on cache miss by treating
it as "no timeout"
- Add new '--output-type auto' option (now the default) that produces
best-effort PDF/A without requiring Ghostscript
- When verapdf is available, use speculative PDF/A conversion
- Without verapdf, pass through as PDF/A if safe (input claims PDF/A
or --force-ocr was used), otherwise output as regular PDF
- Make Ghostscript check conditional - only required for pdfa* output types
- Update soft error tests to explicitly use --output-type pdfa since they
exercise Ghostscript failure modes
- Fix Tesseract OSD error handling to check both stdout and stderr for
known non-fatal messages like "Too few characters"
Insert space characters between words in the fpdf2 renderer so PDF
readers like pdfminer.six can properly segment words during text
extraction. Some PDF readers rely on explicit space characters rather
than inferring word boundaries from positioning.
- Use itertools.pairwise to iterate consecutive word pairs
- Render space immediately after each word (content stream order matters)
- Skip space insertion between CJK words (no spaces in CJK text)
- Use 5% line height threshold to filter OCR noise
- Support RTL text direction
When the user explicitly sets --pdfa-image-compression to something
other than 'auto', skip the speculative PDF/A conversion and use
Ghostscript instead. The speculative conversion (using pikepdf +
verapdf) doesn't apply image compression settings, so Ghostscript
is required to honor the user's compression preference.
Replace pm.hook.method() calls with pm.method() calls to match the
refactored plugin manager that now uses composition over inheritance.
The hook attribute is no longer directly exposed; instead, type-safe
methods are provided directly on the plugin manager class.
Introduce a fast path for PDF/A conversion that uses pikepdf to add
PDF/A structures directly (sRGB ICC profile and XMP metadata), then
validates with verapdf. If validation passes, skip Ghostscript entirely.
If validation fails or verapdf is unavailable, fall back to the existing
Ghostscript conversion path.
New files:
- src/ocrmypdf/_exec/verapdf.py: CLI wrapper for verapdf validator
- tests/test_verapdf.py: Test suite for new functionality
Modified:
- pdfa.py: Add speculative_pdfa_conversion() and helpers
- _pipeline.py: Add try_speculative_pdfa() function
- _pipelines/_common.py: Integrate speculative path into postprocess()
Replace inheritance from pluggy.PluginManager with composition pattern,
providing a type-safe interface for all 16 hooks defined in pluginspec.py.
The underlying pluggy manager is now accessible via the .pluggy property
for advanced use cases like set_blocked().
This change enables IDE autocomplete and type checking for all hook calls
while maintaining full backward compatibility with the plugin system.
Instead of setting OMP_THREAD_LIMIT in the parent process's environment,
calculate the thread limit in the validate hook and pass it through to
Tesseract subprocess calls via the env parameter. This avoids polluting
the parent process's environment while still controlling Tesseract's
thread usage.
- Add hOCR test fixtures for Latin, Arabic, CJK, Devanagari scripts
- Add tests for fpdf2 renderer, multi-font manager, system font provider
- Add multilingual rendering tests
- Update existing tests to use fpdf2 renderer
- Update pipeline to use fpdf2 renderer as default
- Remove legacy hocrtransform PDF renderer (_font.py, _hocr.py,
pdf_renderer.py)
- Update CLI and options for fpdf2 renderer
- Add fpdf2 dependency to pyproject.toml
- Update graft module for fpdf2 multi-page rendering
Implement new PDF renderer using fpdf2 library that provides:
- Multilingual text support via font module
- Proper baseline and rotation handling
- Multi-page rendering with efficient font embedding
- Invisible but selectable text layer
- Add font module with FontManager, FontProvider, MultiFontManager,
and SystemFontProvider for multilingual font support
- Add NotoSans-Regular.ttf for Latin text rendering
- Replace pdf.ttf with Occulta.ttf glyphless font
- Add script to generate new Occulta glyphless font
- System font discovery for CJK, Arabic, Devanagari scripts
Replace integer-based thresholding parameter with ThresholdingMethod
enum for improved type safety. The CLI still accepts the same string
values (auto, otsu, adaptive-otsu, sauvola) but internally uses a
strongly-typed enum. This makes the code more maintainable and catches
type errors at development time.
James R. Barlow