OCRmyPDF/docs/releasenotes/version17.md

% SPDX-FileCopyrightText: 2022 James R. Barlow
% SPDX-License-Identifier: CC-BY-SA-4.0

# v17

## v17.6.0

- When the optimizer encounters an image it cannot process (for example, an
  exotic colorspace that cannot be transcoded), it now logs a concise warning
  that the image was left unchanged rather than printing an alarming
  traceback. The output file was already valid in these cases; only the
  reporting was misleading. The full traceback is still available at debug
  verbosity (`-v 1`) ({issue}`846`).
- `--pdfa-image-compression=auto` (the default) now selects lossless image
  compression at `-O0` so Ghostscript no longer transcodes lossless images to
  JPEG during PDF/A generation. At `-O1` and above, `auto` continues to defer
  to Ghostscript's heuristic, which may recompress images lossily. `-O1` (the
  default level) is kept as a historical exception because coercing it to
  lossless can substantially bloat output; users who want guaranteed lossless
  image handling should pass `--pdfa-image-compression=lossless` or use `-O0`
  ({issue}`1124`).
- `--pdfa-image-compression=lossless` now passes existing JPEG images through
  unchanged rather than re-encoding them with a lossless codec. Re-encoding an
  already-lossy JPEG losslessly cannot recover quality and only inflates the
  file, so JPEGs are preserved while non-JPEG images are encoded losslessly.
- OCRmyPDF now validates and repairs malformed page-boundary boxes
  (``/MediaBox``, ``/CropBox``, ``/TrimBox``, ``/ArtBox``, ``/BleedBox``) in its
  input, following the PDF 2.0 specification. Coordinates written in invalid
  exponential notation are reinterpreted ({issue}`1398`); rectangles whose
  corners are given in reversed order are normalized, which previously crashed
  with ``NegativeDimensionError`` ({issue}`1526`); and a crop/trim/art/bleed box
  that falls outside the MediaBox is clamped to their intersection, or discarded
  when that intersection is empty, which previously produced an output with a
  zero-height effective page that some viewers refused to open ({issue}`1400`).
  When a box is discarded, clamped, or reinterpreted, OCRmyPDF logs a warning
  recommending visual inspection of the output. Thanks @ajdlinux for the initial
  fix in PR #1691.
- OCRmyPDF now discards an embedded Adobe full-text search index
  (``/Root/PieceInfo/SearchIndex``) from its output. This proprietary index,
  produced by Acrobat's "Embed Index" feature, is read only by Adobe Acrobat;
  other viewers ignore it and search the text on the fly. Because any change to
  a PDF invalidates the index, retaining it after OCRmyPDF rewrites the document
  would leave a stale index that returns incorrect search results in Acrobat.
  Modern viewers rebuild a search index on demand, so there is no loss of
  search capability.
- OCRmyPDF now discards embedded per-page thumbnail images (the optional
  ``/Thumb`` image XObject on a page) from its output. OCRmyPDF alters page
  appearance (deskew, clean, rasterize, re-render) and plugins may edit pages
  arbitrarily, so a retained thumbnail would be stale and no longer match its
  page. Embedded thumbnails are a navigation aid that modern viewers generate
  on demand, so there is no loss of functionality.
- Fixed a regression in OCR quality for PDFs that paint a 1-bit image mask
  (stencil) with a gray or colored fill color. Previously such pages were
  rasterized as 1-bit black-and-white before OCR, so Ghostscript dithered
  mid-tone text into an unreadable stipple and Tesseract failed to recognize
  it. The rasterizer now inspects the fill color used to paint a mask and
  promotes the page to grayscale or full color as needed, so the distinction
  is preserved for the OCR engine. This applies to both the Ghostscript and
  pypdfium rasterizers. {issue}`1688`
- The default 1-bit raster device for Ghostscript is now ``pngmonod``
  (error-diffusion) instead of ``pngmono`` (ordered dithering). It produces
  better input for OCR on faint or anti-aliased scans at negligible cost and
  no change to output file size, since the rasterized image is an
  intermediate that is discarded after OCR.
- When rasterizing pages with Ghostscript, OCRmyPDF now enables text and
  graphics anti-aliasing (``-dTextAlphaBits=4 -dGraphicsAlphaBits=4``) for the
  grayscale and color raster devices. Ghostscript 10.x renders aliased glyphs
  that OCR frequently misreads as extra word breaks or substituted characters;
  anti-aliasing materially improves OCR accuracy on the Ghostscript
  rasterization path, especially for small fonts at moderate resolution. The
  1-bit monochrome devices are unaffected, since they perform their own
  anti-aliased downscaling and older Ghostscript versions reject alpha-bit
  options on them. Note that the default rasterizer (``--rasterizer auto``)
  prefers pypdfium2, which already anti-aliases; this change benefits users who
  select ``--rasterizer ghostscript`` or do not have pypdfium2 installed.
  OCRmyPDF now also logs which rasterizer rendered each page at debug verbosity
  (``-v 1``), and the ``--rasterizer`` help text explains the OCR-quality
  trade-off, to make such reports easier to diagnose. {issue}`1439`
- When Tesseract reports a page with many diacritics, OCRmyPDF still logs its
  interpreted "lots of diacritics - possibly poor OCR" hint, but now also emits
  Tesseract's raw message at debug verbosity (``-v 1``) so the original wording
  is available for diagnosis. {issue}`1566`

## v17.5.0

- Added support for the ``end`` alias in ``--pages``, denoting the last page
  of the document. For example, ``--pages 3-end`` OCRs from page 3 through
  the final page. {issue}`1615`
- Added ``--ghostscript-jpeg-quality`` and ``--ghostscript-jpeg-maxdpi``
  advanced options for tuning Ghostscript's PDF/A output. The optimizer's
  ``--jpeg-quality`` remains the recommended file-size control.
- Fixed pypdfium2 rasterizer clipping content when the CropBox was smaller
  than the MediaBox (e.g. JSTOR or cropped PDFs). {issue}`1685`
- Fixed Form XObject cycle detection in the optimizer's image xref scan.
  Self-referential or DAG-shaped Form graphs (notably from PowerPoint
  exports) previously produced floods of recursion warnings and could hang
  for minutes. {issue}`1321`
- Tesseract config errors are now surfaced as ``TesseractConfigError`` with
  actionable guidance, instead of crashing later with a confusing
  ``FileNotFoundError`` on the missing hOCR output. {issue}`1687`
- Refreshed the Chinese README translation. Thanks @cislunarspace.
- Internal refactoring of the ``_exec`` and ``subprocess`` modules to
  separate probing from execution.
- CI dependency updates.

## v17.4.2

- Fixed Python API unconditionally overriding ``PIL.Image.MAX_IMAGE_PIXELS``
  when the caller did not explicitly set ``max_image_mpixels``. Host
  applications (e.g. Paperless-NGX) that configure the PIL limit before
  invoking ``ocrmypdf.ocr()`` now have their setting respected. The CLI
  default of 250 megapixels is unchanged. {issue}`1665`
- Updated uv.lock to avoid pinning a vulnerable version of Pillow. {issue}`1666`

## v17.4.1

- Fixed RTL text extraction order in the fpdf2 renderer. Arabic lam-alef
  ligatures and other multi-character CMap entries were garbled by the bidi
  algorithm during text extraction. {issue}`1655`
- Fixed ``work_folder`` not being set in ``PdfContext`` options when using
  the Python API. Thanks @bluebox-steven. {issue}`1613`
- Updated Ghostscript JPEG corruption warning to include the detected version
  number, confirming the bug persists in Ghostscript 10.7.0.
- Internal refactoring.
- CI dependency updates.

## v17.4.0

- Added ``--no-overwrite`` / ``-n`` option to prevent overwriting output files.
  If the destination file already exists, OCRmyPDF exits with code 5
  (``OutputFileAccessError``). {issue}`1642`
- Fixed text layer stretching in the fpdf2 renderer for widely-spaced words.
  The horizontal scaling (Tz) was incorrectly stretched to fill inter-word gaps
  instead of relying on Td positioning, causing text selection to highlight far
  beyond the actual word boundaries. {issue}`1635`
- Fixed ``optimize=2`` or ``optimize=3`` crash when using the Python API without
  explicitly setting ``jpg_quality`` or ``png_quality``. {issue}`1641`
- Fixed ``verapdf`` availability check crashing with ``NotADirectoryError`` on
  some platforms. {issue}`1638`

## v17.3.0

- Fixed Python API ignoring the ``language`` parameter, always defaulting to
  ``eng``. The API now correctly maps ``language`` to OcrOptions ``languages``
  and splits ``+``-separated codes (e.g. ``eng+deu``) to match CLI behavior.
  {issue}`1640`
- Fixed Python API producing empty OCR output because ``tesseract_timeout``
  defaulted to 0, causing Tesseract to time out immediately. The default is
  now ``None``, falling back to the plugin's 180-second timeout. {issue}`1636`
- Fixed OCR text layer displacement on PDFs with non-zero MediaBox origins
  (e.g. JSTOR or cropped PDFs). The coordinate transformation matrix is now
  always computed, not skipped when rotation is zero. {issue}`1630`
- Restored image overlay support (``--image``) for the hocrtransform tool,
  enabling sandwich PDF output with the fpdf2 renderer. {issue}`1634`
- Docker: updated Alpine base image to 3.23.
- Documentation restructured into per-major-version release notes files.
- Release process improvements.

## v17.2.0

- Fixed incorrect word spacing in poppler-based PDF viewers and tools (Evince,
  pdftotext, and others) where words on the same line appeared separated by
  double newlines. This works around a poppler bug where Tz (horizontal scaling)
  is not carried across BT/ET boundaries. {issue}`1632`
- Fixed OCR text layer being visible instead of invisible due to incorrect fpdf2
  text rendering mode attribute. This caused OCR text to appear when images were
  removed from the PDF. {issue}`1631`
- Fixed OCR text layer misalignment with non-zero mediabox origins, which
  affected cropped PDFs and JSTOR PDFs generated by iText. The ``--redo-ocr``
  mode would shift text vertically on these files. {issue}`1630`
- Fixed Ghostscript rasterization failure with very low DPI values (below 10).
  OCRmyPDF now renders at a minimum of 10 DPI and resizes the output to match
  the originally requested dimensions. {issue}`1612`

## v17.1.0

- Added `--tagged-pdf-mode` to allow skipping the TaggedPDF error message, if desired.
- Fixed an issue where deflated JPEGs (FlateDecode + DCTDecode) were counted as
  lossless images for the purpose of determining whether to compress to JPEG,
  causing file size inflation with some workflows (`--mode force` in particular).

## v17.0.1

- Fixed output file size inflation when using pypdfium as rasterizer and force-ocr
  mode.

## v17.0.0

**Breaking changes**

- **Plugin interface migration**: Plugin hooks now receive `OcrOptions` objects instead of
  `argparse.Namespace` objects. Most plugins will continue working due to duck-typing
  compatibility, but plugin developers should update their type hints from `Namespace`
  to `OcrOptions`.
- Built-in plugins no longer modify options in-place, improving immutability and
  code clarity.
- **Lossy JBIG2 removed**: The `--jbig2-lossy` and `--jbig2-page-group-size` options have been
  removed due to well-documented risks of character substitution errors. These options are now
  deprecated and will emit warnings if used. Only lossless JBIG2 compression is supported.
- **PDF/A output behavior change**: If neither Ghostscript nor verapdf is installed,
  `--output-type auto` (the new default) will produce a standard PDF instead of PDF/A. This is
  a change from previous versions where Ghostscript was required and PDF/A was always produced.
  This configuration is rare but users should be aware of the change.

**New features**

- **pypdfium2 rasterizer**: Added optional pypdfium2-based PDF rasterization plugin as an
  alternative to Ghostscript for page rendering. Use `--rasterizer pypdfium` to enable
  (requires `pip install pypdfium2`). The default `--rasterizer auto` prefers pypdfium when
  available and falls back to Ghostscript.
- **Pluggable OCR engines**: New `--ocr-engine` option allows selecting OCR engines:
  - `auto` (default): Uses Tesseract
  - `tesseract`: Explicit Tesseract selection
  - `none`: Skip OCR entirely for PDF processing-only workflows

  This prepares the foundation for future third-party OCR engine plugins.
- **Smart PDF/A conversion**: New `--output-type auto` (now the default) produces best-effort
  PDF/A output without requiring Ghostscript when the verapdf validator is available. Falls back
  to traditional Ghostscript conversion when needed.
- **verapdf integration**: Added optional verapdf validation for fast PDF/A conversion. When
  available, OCRmyPDF attempts speculative PDF/A conversion using pikepdf, validates with verapdf,
  and skips Ghostscript if validation passes.
- **Optional Ghostscript**: As a consequence of the changes above, Ghostscript is no longer a required dependency. It is optional.
- **fpdf2 text renderer**: Replaced legacy hOCR text renderer with new fpdf2-based implementation,
  providing better multilingual support and more accurate text positioning.
- **Improved Occulta glyphless font**: The new Occulta font provides better handling of
  zero-width markers and double-width CJK characters for accurate text layer positioning.
- **Expanded multilingual font support**: Added FontProvider infrastructure with language-aware
  font selection for Devanagari (Hindi, Sanskrit, Marathi, Nepali), CJK (Chinese, Japanese,
  Korean), Arabic script, and many other scripts. System font discovery reduces package size.
- **Simplified mode selection**: New `--mode` (`-m`) argument consolidates processing options:
  - `default`: Error if text is found (standard behavior)
  - `force`: Rasterize all content and run OCR (replaces `--force-ocr`)
  - `skip`: Skip pages with existing text (replaces `--skip-text`)
  - `redo`: Re-OCR pages, stripping old text layer (replaces `--redo-ocr`)

  Legacy flags remain as silent aliases for backward compatibility.

**API improvements**

- Centralized validation logic in the `OcrOptions` Pydantic model
- Removed scattered option mutation throughout the codebase
- Better type safety for plugin development
- Simplified plugin option handling
- New `OcrElement`, `OcrClass`, and `BoundingBox` exports for OCR engine plugin developers
- Extended `OcrEngine` ABC with `generate_ocr()` method for direct OCR tree output, eliding the need to translate a modern engine's output to hOCR or directly write to PDF.

**Bug fixes**

- Fixed double-compression of already-deflated JPEGs.
- Fixed tesseract_cache plugin to properly handle cache misses.
- Fixed handling of PDF page boxes (ArtBox, BleedBox) which were not being processed correctly.
- Added thread safety lock to pypdfium plugin for concurrent operations.
- Improved pdfminer.six compatibility with explicit word spacing.

**Documentation**

- Updated cookbook to replace deprecated `--tesseract-timeout 0` with `--ocr-engine none`.
- Added comprehensive plugin documentation for new OCR engine framework.

**Dependency changes**

- Requires: one of `pypdfium2` or `ghostscript` for PDF rasterization (PDF to image)
  - Preferred: both
- Requires: one of `verapdf` or `ghostscript` for PDF/A generation
  - Preferred: both
- Recommended: `pypdfium2` for PDF rasterization (new dependency)
- Recommended: `ghostscript` (used to be Required)
- Recommended: Noto fonts for improved OCR text positioning
- Optional: `verapdf` for fast PDF/A validation (new dependency)
- Requires: `fpdf2` for text layer rendering (new dependency)
- Recommended: replace `typer` with `cyclopts` in misc scripts (new dependency)
- See docs/maintainers.md for details.

**Migration guide for plugin developers**

- Update imports: `from ocrmypdf._options import OcrOptions`
- Update type hints: `def check_options(options: OcrOptions)` instead of `options: Namespace`
- Attribute access remains unchanged: `options.languages`, `options.output_type`, etc.
- Remove any in-place option modifications - compute values at point of use instead
- Most existing plugins will continue working without changes due to duck-typing