* Bump uv lock, fix bump_version
* Discover variable and per-language Noto fonts for the OCR text layer (#1652)
SystemFontProvider only matched static "-Regular.ttf/.otf" filenames, so
the variable fonts shipped by Homebrew casks and current Google Fonts
(e.g. NotoSansArabic[wdth,wght].ttf) were never found. Users who had
installed the font still got the glyphless Occulta fallback and a cryptic
"No font found" warning.
- Match variable fonts (Base[...]), -VF, and bare-family filenames via a
boundary-aware flexible search, escaping the glob-special brackets.
- Make CJK language-aware: the modern per-language Noto fonts (NotoSansSC
/TC/HK/JP/KR) are region subsets, so map each CJK language to its own
family and keep the full-coverage pan-CJK super font as a shared
fallback. Glyph coverage, not shape, is what matters for the invisible
text layer.
- Reword the missing-font warning to explain the consequence (searchable
but blank when highlighted) and name the language-specific font.
Adds a processing mode that removes the invisible (render mode 3) OCR text
layer in place. Unlike `--ocr-engine none --force-ocr`, it does not
rasterize the page, so images and visible content are preserved unchanged
and the output is smaller rather than larger. Options that require
rasterization or OCR (--deskew, --clean, --sidecar, etc.) are rejected.
Only invisible text is removed; text drawn as visible glyphs under an
opaque image (some OCR engines, and OCRmyPDF v2.2 and earlier) cannot be
removed this way, as documented.
Closes#1435.
When Tesseract reports a page with many diacritics, OCRmyPDF rewrites the
message to "lots of diacritics - possibly poor OCR", which hid the
original wording. Keep the interpreted hint but also emit Tesseract's raw
line at debug verbosity (-v 1) so users can see exactly what Tesseract
reported.
Closes#1566.
Ghostscript 10.x renders aliased glyphs that OCR frequently misreads as
extra word breaks or substituted characters. Enable text and graphics
anti-aliasing (-dTextAlphaBits=4 -dGraphicsAlphaBits=4) for the contone
raster devices, which empirically improves OCR accuracy on the
Ghostscript path, especially for small fonts at moderate DPI. The 1-bit
mono devices are excluded, since older Ghostscript rejects alpha bits on
them and pngmonod performs its own anti-aliased downscaling.
Also log which rasterizer rendered each page at debug verbosity and
clarify the --rasterizer help text, so quality reports are easier to
diagnose. The default rasterizer (auto) already prefers pypdfium2, which
anti-aliases; this primarily benefits --rasterizer ghostscript and
installs without pypdfium2.
Closes#1439.
The optimizer is best-effort: any image it cannot process is left
unchanged in the output, which remains valid. Previously, an extraction
failure (e.g. an exotic colorspace pikepdf cannot transcode) was logged
with log.exception, printing a full traceback at ERROR level that alarmed
users even though nothing was wrong with the output (issue #846).
Trap such failures with a concise warning that the image was left
unchanged and the output is still valid, and demote the traceback to
debug verbosity for diagnosis.
Ghostscript's `auto` image compression heuristic can transcode lossless
images to JPEG during PDF/A generation, which is surprising at
optimization levels that otherwise promise lossless-only operations
(issue #1124).
`--pdfa-image-compression=auto` (the default) now coerces to lossless at
-O0 so Ghostscript will not transcode lossless images to JPEG. -O1 and
above continue to defer to Ghostscript's heuristic; -O1 (the default
level) is kept as a historical exception because coercing it to lossless
substantially bloats output. Users wanting guaranteed lossless image
handling can pass --pdfa-image-compression=lossless or use -O0.
Also make `lossless` pass existing JPEGs through unchanged
(-dPassThroughJPEGImages=true) instead of re-encoding them with a
lossless codec, which only inflates already-lossy data.
Validate and repair the page-boundary boxes (MediaBox, CropBox, TrimBox,
ArtBox, BleedBox) of input PDFs in triage(), following PDF 2.0:
- Coerce coordinates written in invalid exponential notation, stored by
qpdf/pikepdf as strings (#1398).
- Normalize rectangles whose corners are given in reversed order, which
previously crashed with NegativeDimensionError (#1526).
- Clamp a crop/trim/art/bleed box that extends outside the MediaBox to
their intersection, or discard it when the intersection is empty, which
previously produced a zero-height effective page some viewers rejected
(#1400).
When a box is discarded, clamped, or reinterpreted, a warning recommends
visual inspection of the output. pdfinfo box reading shares the same
coercion helper so PdfInfo no longer crashes on malformed boxes.
Supersedes PR #1691. Thanks @ajdlinux.
Closes#1398, #1526, #1400
A tagged/structured PDF carries a logical structure tree
(/Root/StructTreeRoot, /MarkInfo) that maps marked page content to
semantic elements via MCIDs. When --force-ocr rasterizes pages or
--redo-ocr rewrites the text layer, those MCIDs are destroyed or
renumbered and the tree is left dangling. We cannot rebuild it to match
the new text, so discard it for force/redo modes, following the same
pattern as the thumbnail and search-index discards. --skip-text leaves
text pages untouched, so their structure is preserved.
Also broaden the default-mode "looks born-digital" stop signal to fire
on /StructTreeRoot, not just /MarkInfo/Marked, so structure-tree-only
PDFs are no longer silently OCR'd. The existing --tagged-pdf-mode ignore
escape hatch is unchanged.
A page may carry an optional /Thumb image XObject (ISO 32000-2 12.3.4),
a miniature rendering used only as a navigation aid. OCRmyPDF alters
page appearance (deskew, clean, rasterize, re-render) and plugins may
edit pages arbitrarily, so a retained thumbnail would be stale and no
longer match its page. Modern viewers generate thumbnails on demand, so
there is no loss of functionality.
OcrGrafter.finalize() now strips /Thumb from every page before saving,
alongside the existing search-index discard, covering both the OCR and
hOCR pipelines. Orphaned thumbnail XObjects are garbage-collected on save.
Adobe Acrobat can embed a proprietary full-text search index in the
document catalog at /Root/PieceInfo/SearchIndex. Only Acrobat reads it;
other viewers ignore it and search the text on the fly. Any change to a
PDF invalidates the index, so once OCRmyPDF rewrites the document the
retained index is stale and returns incorrect search results in Acrobat.
OcrGrafter.finalize() now discards it before saving (covering both the
OCR and hOCR pipelines), preserving any other PieceInfo owner data and
dropping an empty PieceInfo. Modern viewers rebuild a search index on
demand, so there is no loss of search capability.
Address final review findings: a mask painted inside a Form XObject now
inherits the fill color in effect at the form's Do operator (previously it
reset to black, missing gray/color promotion one indirection deep). The cs
operator now resets the fill color to black per PDF spec, so a stale color
set before cs cannot leak to a subsequently drawn mask.
Track the current PDF fill color on the graphics stack alongside the CTM and
record an Ink classification (mono/gray/color) per image-draw event. Image
masks are painted with the current fill color, so this enables later device
promotion. Color operators are tolerant of malformed operands to preserve
robustness on untrusted input.
When Ghostscript reports a DeviceN colorspace with an inappropriate
alternate, the resulting PDF/A may render blank in viewers such as Adobe
Reader (#1187). The error is gated on that Ghostscript warning, which is
the authoritative signal that the *output* is broken.
Previously the error message always told the user to "use
--color-conversion-strategy", which is confusing when they already set
one and it didn't help. Crucially, the warning persists for strategies
that don't actually normalize the colorspace -- notably
UseDeviceIndependentColor (confirmed in #1187) -- so silencing the error
for any non-default strategy would emit a silently-broken PDF/A.
Keep raising whenever Ghostscript still reports the warning, regardless
of strategy, but tailor the guidance: if no conversion was requested,
suggest RGB/CMYK/Gray; if a conversion was requested but the warning
persisted, say so and point at strategies that work or --output-type pdf.
Add unit tests (mocked Ghostscript) covering the default case, the
warning-persists-despite-strategy case for both an ineffective strategy
and a normally-effective one, and the no-warning happy path.
Closes#1615. The token 'end' (case-insensitive) is now accepted as an
alias for the document's last page, e.g. --pages 3-end. Resolution is
deferred until the page count is known from the input PDF.
PDFium does not support negative crop values to expand the render
area beyond the CropBox: such values only pad the output canvas with
white, leaving content outside the CropBox clipped. Set the in-memory
CropBox to the MediaBox before rendering instead. Reported in #1685.
Expose Ghostscript's -dJPEGQ and image downsampling switches as
advanced, plugin-scoped options for tuning PDF/A output, without
polluting the central OcrOptions registry. The optimizer's existing
--jpeg-quality remains the recommended JPEG quality control.
- GhostscriptOptions gains jpeg_quality and jpeg_maxdpi fields and CLI
args (advanced help text). jpeg_quality=0 is honored as Ghostscript's
maximum compression rather than being silently coerced to the default.
- _exec.ghostscript.generate_pdfa() forwards both values; when
jpeg_maxdpi is set, downsample threshold is pinned at 1.0.
- _get_plugin_options falls back to extra_attrs for namespaced fields
so plugins can own their options without registering them centrally.
- Documentation explains the rationale: Ghostscript is the legacy path
(pypdfium + verapdf is preferred in v17+), the optimizer is the
supported file-size lever, and lowering quality is almost always a
better trade than downsampling.
When Tesseract cannot find its 'hocr' or 'txt' config files in the
tessdata configs/ directory, it prints "read_params_file: Can't open"
warnings, exits 0, and produces no output. OCRmyPDF then crashed with a
confusing FileNotFoundError on the missing hOCR file (issue #1687).
Promote read_params_file warnings to TesseractConfigError with guidance
on the likely cause, and verify the expected output file exists after
Tesseract claims success as defense-in-depth for other silent-failure
modes.
The 2024 guard against runaway recursion in _find_image_xrefs_container
only deduplicated image xrefs, but Form XObject xrefs are never added to
include_xrefs/exclude_xrefs, so a self-referential or DAG-shaped Form
graph re-entered every branch until the depth limit fired -- producing
the reported flood of warnings (and minutes-long hangs) on PowerPoint
exports.
Thread a visited_forms set through the recursion so each Form XObject is
descended into at most once per document. With memoization in place the
depth limit is no longer a cycle defense, so demote its log to debug.
Add a regression test that synthesises a circular-Form PDF from the
existing formxobject.pdf fixture (no new binary fixture, no license
issues) and asserts zero "Recursion depth exceeded" warnings.
Split ocrmypdf.subprocess/__init__.py into three private submodules by
concern (_run, _version, _check) and reduce __init__ to re-exports.
Introduce ocrmypdf._exec._probe.ToolProbe to centralize the version()/
available() pattern each tool module was reimplementing, so the "is this
tool installed and suitable?" question is cleanly distinct from the
pure, picklable functions that do the work.
Also replace the ghostscript module-import log.addFilter() side effect
with an idempotent _ensure_log_filter_installed() called at the top of
each work function, so the DuplicateFilter is present in subprocess
workers without relying on import-time ordering.
Public API of ocrmypdf.subprocess is unchanged.
The API previously clobbered PIL.Image.MAX_IMAGE_PIXELS unconditionally
on every call, so host applications (e.g. Paperless-NGX) that configured
the PIL limit before invoking ocrmypdf.ocr() saw their setting silently
overwritten with the 250 MP default. Make max_image_mpixels default to
None and only apply the override when the caller explicitly sets it.
The CLI default of 250 MP is unchanged.
Fixes#1665
The JPEG truncation bug (1-15 bytes) persists in Ghostscript 10.7.0.
Update the warning message to show the actual GS version instead of
hardcoding "10.6.x", and remove the stale date reference. Also make
the test warning filter match the new message format.
fpdf2's shape_text() produces RTL ligature glyphs (e.g. lam-alef) with
multi-character CMap entries whose character order gets reversed by the
bidi algorithm during text extraction, producing garbled output like
"سالح" instead of "سلاح".
For invisible text (the production OCR overlay path), bypass text shaping
and use encode_text() with pre-reversed strings. encode_text() maps
characters 1:1 in logical order, avoiding the ligature CMap issue. The
pre-reversal compensates for bidi reversal by text extractors. Since the
text is invisible (Tr=3), the lack of joining forms is harmless.
Add RTL text extraction tests that verify glyph stream order, ToUnicode
CMap 1:1 mappings, and correct logical order for Arabic (including
lam-alef ligature) and Hebrew scripts.
The inter-word Tz calculation stretched "word " to span from the current
word to the next, producing extreme horizontal scaling (300-500%) for
words far apart (e.g. in tables). Use per-word Tz instead — Td
positioning already handles inter-word gaps correctly.
Fixes#1635
Catch OSError (parent of both FileNotFoundError and
NotADirectoryError) in verapdf.available() so environments where
executing `verapdf` raises NotADirectoryError gracefully fall back
instead of crashing the pipeline. Fixes#1638.
Fixes#1642. Adds an early check in check_requested_output_file() that
raises OutputFileAccessError (exit code 5) if the destination file
already exists and --no-overwrite is set. The option is wired through
CLI, OcrOptions, and the Python API.
The jpg_quality and png_quality options default to None in the pydantic
model, but the fallback check only handled == 0. This caused a TypeError
when calling ocrmypdf.ocr() with optimize >= 2 without explicitly
setting quality values. Fixes#1641.
Replace hatch-vcs dynamic versioning with static version in _version.py
and pyproject.toml. Split CI into build.yml (test + stage draft release
on main) and release.yml (publish from draft on tag push). Docker images
are built on main pushes and re-tagged with the release version on tag
push without rebuilding.
The API's 'language' param was silently dropped because OcrOptions uses
'languages' (plural). Map language->languages in create_options() and
_pdf_to_hocr(), coercing bare strings to lists and splitting '+'
separated codes to match CLI behavior.
_build_text_layer_ctm() returned None when text_rotation was 0,
skipping the origin translation needed for pages with non-zero
MediaBox origins (e.g. JSTOR PDFs with [0, 100, 595, 982]). This
caused the text layer to be offset by the page origin amount.
Always compute the full CTM and only return None when the result
is the identity matrix.
Fixes#1630
OcrOptions.tesseract_timeout defaulted to 0.0, which caused
subprocess.run(timeout=0) to immediately raise TimeoutExpired before
Tesseract could produce any output. The CLI was unaffected because
argparse defaults --tesseract-timeout to 180. Change the OcrOptions
default to None so the plugin's own default (180s) is used.
Fixes#1636
encode_text() maps unicode characters one-by-one to glyph IDs without
any text shaping, producing incorrect output for complex scripts:
Arabic glyphs in wrong order without joining forms, Devanagari conjuncts
broken apart. Replace with shape_text() which runs HarfBuzz for proper
BiDi reordering, Arabic shaping, and Devanagari conjunct formation.
jbarlowandGitHub
James R. Barlow