namespace_to_options() only copied argparse namespace keys that were
literal members of OcrOptions.model_fields. The CLI dest for both
--jpeg-quality and --jpg-quality was jpeg_quality, but the pydantic field
was named jpg_quality (jpeg_quality existed only as a compatibility
property, absent from model_fields). The value was silently dropped into
extra_attrs, and the optimizer always fell back to its own hardcoded
default regardless of the flag.
The same alias mismatch also affected the Python API: create_options()
uses the same model_fields-matching logic as namespace_to_options(), so
ocrmypdf.ocr(jpeg_quality=...) was silently dropped too - only the
canonical jpg_quality= kwarg worked.
Rather than patch around the mismatch, consolidate on a single canonical
name: OcrOptions.jpeg_quality (matching the primary --jpeg-quality CLI
flag and the already-consistent naming in OptimizeOptions). jpg_quality
becomes a deprecated compatibility property, and ocrmypdf.ocr(jpg_quality=)
is a deprecated alias that warns and forwards to jpeg_quality via a new
create_options() remap step. --jpg-quality remains a working (already
hidden) CLI alias with no code-path divergence, since it now shares an
argparse dest that matches the field name directly.
SystemFontProvider.NOTO_FONT_PATTERNS enumerates about two dozen Noto
families by name, and MultiFontManager only ever asked for those. Any
script outside that list fell through to the glyphless Occulta fallback
even when the correct font was installed, which is the common case on
macOS: it ships around a hundred script-specific Noto faces in
/System/Library/Fonts/Supplemental, almost none of which we knew how to
ask for. The reporter had the fonts and still got told to install them.
Add an optional GlyphSearchingFontProvider protocol (find_font_with_glyphs)
implemented by SystemFontProvider, BuiltinFontProvider and
ChainedFontProvider, and a new selection phase that uses it once the named
families fail. The system scan enumerates every Noto face, reusing the
existing variable-font/-Regular/-VF filename classification, and keeps
only the font it selects so a full scan does not retain every font file on
the system. Fonts found this way are remembered and retried by name for
later words. Capability detection is isinstance-based so third-party
providers keep working unchanged.
The warning itself was also unactionable: it named neither the characters
nor the script, which is why the reporter had to ask which package to
install. It now identifies what it could not render, e.g.
'Ꮳ' U+13E3 CHEROKEE LETTER TSA. A word mixing scripts that no single font
covers now gets a distinct message saying that installing fonts will not
help, rather than sending the user after fonts they already have.
Finally, the documented macOS install command was wrong: `brew install
font-noto` does not exist, since Homebrew has no single Noto package, only
a cask per family. The Fedora package name was also corrected, as
google-noto-fonts-common ships no actual fonts.
A malformed PDF may store a non-dictionary object (an array, name, or
other type) at /Resources or /Resources /XObject. The pdfinfo image
scanner iterated these with .items()/.as_dict() and probed them with the
`in` operator, which raise TypeError/ValueError on non-dictionary pikepdf
objects and crashed PdfInfo on otherwise-processable files. OCRmyPDF's
domain is messy, machine-generated PDFs, so scanning must tolerate this.
Guard _image_xobjects and _find_form_xobject_images with
isinstance(x, Dictionary) before iterating, treating a non-dictionary
/Resources or /XObject as "no image XObjects". This is the same
robustness class as the pdfa.py find_nonembedded_cid_fonts fix, applied
to the pdfinfo image scanner.
Follow-on to the non-dictionary /Font and /XObject guard. Replace the
_dict_entries() helper with an isinstance(..., pikepdf.Dictionary) check
at each resource lookup, which pikepdf's metaclass supports directly and
which reads as exactly the invariant being enforced. Iterate via
as_dict().values() so the values are typed and mypy stays clean once the
Any from the untyped resources argument is narrowed away.
Also guard _cid_font_is_embedded against a non-dictionary /FontDescriptor:
`key in descriptor` raises ValueError on a non-dict, which the caller's
except (AttributeError, TypeError, KeyError) does not catch. Such a font
now counts as non-embedded and is reported, so PDF/A conversion is refused
rather than risking Ghostscript corrupting a pre-existing CID text layer.
Adds a regression test for the FontDescriptor case (issue #1713).
A malformed PDF can store a non-dictionary object under a page's /Font or
/XObject resource. find_nonembedded_cid_fonts() iterated .values() on that
object outside the per-entry try/except, so scanning such a page raised
(TypeError/AttributeError depending on the pikepdf version) instead of
producing output. This surfaced as a PDF/A conversion crash.
Route both resource lookups through a small helper that returns an empty
list when the resource is missing or not a dictionary, so a garbage entry
is simply treated as having no fonts. Add a regression test covering a
non-dictionary /Font and /XObject.
Replaces os.path/open()/os.stat()/os.chmod() calls with their Path
method equivalents across src, tests, misc, and bin, wrapping str
variables in Path(...) where they must stay str for other uses (e.g.
subprocess argv, CLI-arg formatting). helpers.safe_symlink() now
decodes StrOrBytesPath to a str Path via os.fsdecode() upfront, same
pattern already used elsewhere for the str|bytes union.
Resolves the last 16 mypy errors in the project (src/ocrmypdf and
tests are now fully clean).
fpdf_renderer/renderer.py (9 errors):
- add_page(format=...): fpdf2's own stub types this param as str, but
its docstring and get_page_format() helper confirm a (width, height)
tuple is accepted too - the stub annotation on add_page() itself is
the outlier. Used cast() to match the documented/actual behavior.
- pdf.current_font is typed CoreFont | TTFFont | None, but this
renderer only ever registers fonts via add_font() with a TTF file
(see _register_font/set_font call sites) - it never falls back to
fpdf2's built-in CoreFont. Added assertions (isinstance(font,
TTFFont) where shape_text()/escape_text() are needed, which
CoreFont lacks; plain not-None elsewhere) documenting that
invariant instead of narrowing defensively for a case that can't
happen here.
tests/test_pdf_renderer.py (7 errors): the ToUnicode/glyph-extraction
test helpers used `.get(key, {})` (a plain dict literal default) then
called `.values()`/`.items()` on the result. pikepdf.Object doesn't
declare `values()` in its stub (only `keys()`), so this silently
degraded to Object's catch-all `__getattr__` returning another Object,
which then failed as "not callable". Switched to `.get(key,
Dictionary()).as_dict()`, which returns pikepdf's properly-typed
_ObjectMapping helper.
prek runs local hooks as plain execs against tools uv already provisions,
so ruff/mypy can never drift from the versions/config uv.lock pins
elsewhere and CI needs no separate hook-cache download.
- Add ruff and prek to the uv dev dependency group (ruff wasn't a
uv-managed dependency before; pre-commit silently vendored its own).
- Replace .pre-commit-config.yaml with prek.toml: keep the
pre-commit-hooks repo for generic file checks, convert ruff-format/
ruff-check to local `uv run ruff ...` hooks, and add a local mypy
hook that reports but never fails (87 pre-existing errors need a
separate cleanup before it can be made blocking).
- Add a `lint` job to CI that runs `prek run --all-files` and gate the
OS/Python test matrix on it so lint issues fail fast.
- Fix the ruff debt (format + lint) uncovered by actually running it,
since it was small and mechanical, so the new CI gate starts green.
The test asserted that --mode skip preserves the structure tree, but with
the default --output-type auto the output runs through Ghostscript PDF/A
conversion, which discards /StructTreeRoot on Ghostscript 10.x (9.x kept
it). This failed on macOS CI and locally while passing on the Ubuntu
runners' Ghostscript 9.55. Pin the test to --output-type pdf so it exercises
OCRmyPDF's own structure-tree handling without the version-dependent GS step,
and document the caveat in advanced.md.
Ghostscript's PDF/A conversion re-embeds non-embedded CID (CJK) fonts by
substituting a system font, which corrupts the character-to-Unicode
mapping and silently destroys an existing text layer -- commonly the OCR
layer Adobe Acrobat adds to scanned CJK documents.
Detect non-embedded CID/Type0 fonts before conversion: with
--output-type auto (the default) downgrade to a regular PDF and preserve
the text layer; with an explicit --output-type pdfa* stop with an error
rather than emit corrupted output. Simple non-embedded fonts (e.g. Latin)
are left alone -- Ghostscript substitutes them without corrupting the
text, and they are far too common to treat as conversion blockers.
Use --output-type pdf to keep the existing text layer, or --force-ocr to
rebuild it with embedded fonts.
Writing the output PDF to stdout (ocrmypdf in.pdf -) previously relied on
an honor system: no in-process code -- third-party libraries, plugins, or
stray print() calls -- was supposed to write to stdout, enforced only
indirectly. A single accidental write to fd 1 would silently corrupt the
output PDF.
Enforce this at the OS level. At CLI startup, before plugins load or any
worker process/thread starts, save the real stdout via os.dup() and point
fd 1 at stderr, so stray writes are diverted to stderr while only the final
"produce the PDF" step writes to the preserved descriptor. Exposed as the
opt-in public API function configure_stdout_protection(), mirroring
configure_logging(); it is not enabled inside ocr() so in-process library
users keep their own stdout.
Also fix check_requested_output_file() to test the preserved real stdout
for tty-ness, since after the redirect sys.stdout reports stderr's status.
Fold unreleased v17.7.2 notes into v17.8.0.
Some PDFs use a /Name dictionary key in /DocumentInfo whose bytes are not
valid UTF-8/PDFDocEncoding, e.g. a Latin-1 /Saks#e5r. Older pikepdf raised
UnicodeDecodeError while iterating such a block, crashing the pipeline
during PDF/A conversion. repair_docinfo_nuls is documented to log and
continue on a malformed DocumentInfo block, so catch UnicodeDecodeError
alongside TypeError.
Add a mock-based unit test that drives the decode-error branch (current
pikepdf surrogate-escapes instead of raising) and an end-to-end test over
the reporter's file, committed as docinfo_latin1_key.pdf.
A page whose MediaBox has a non-zero origin (e.g. from PDF Arranger crops)
was rendered blank by --force-ocr in v16.12.0, because fix_pagepdf_boxes
offset the CropBox with the wrong sign, pushing it entirely outside the
image-page MediaBox. The behavior was fixed in v17.0.0 but had no
end-to-end test: the existing box tests only assert that ocrmypdf runs,
which the blank-page bug passed (it exited 0 with a valid, empty PDF/A).
Add a test that renders the output and asserts the visible content
survives, parametrized over both the ghostscript and pypdfium rasterizers
since the bug reproduced regardless of rasterizer.
* Bump uv lock, fix bump_version
* Discover variable and per-language Noto fonts for the OCR text layer (#1652)
SystemFontProvider only matched static "-Regular.ttf/.otf" filenames, so
the variable fonts shipped by Homebrew casks and current Google Fonts
(e.g. NotoSansArabic[wdth,wght].ttf) were never found. Users who had
installed the font still got the glyphless Occulta fallback and a cryptic
"No font found" warning.
- Match variable fonts (Base[...]), -VF, and bare-family filenames via a
boundary-aware flexible search, escaping the glob-special brackets.
- Make CJK language-aware: the modern per-language Noto fonts (NotoSansSC
/TC/HK/JP/KR) are region subsets, so map each CJK language to its own
family and keep the full-coverage pan-CJK super font as a shared
fallback. Glyph coverage, not shape, is what matters for the invisible
text layer.
- Reword the missing-font warning to explain the consequence (searchable
but blank when highlighted) and name the language-specific font.
Adds a processing mode that removes the invisible (render mode 3) OCR text
layer in place. Unlike `--ocr-engine none --force-ocr`, it does not
rasterize the page, so images and visible content are preserved unchanged
and the output is smaller rather than larger. Options that require
rasterization or OCR (--deskew, --clean, --sidecar, etc.) are rejected.
Only invisible text is removed; text drawn as visible glyphs under an
opaque image (some OCR engines, and OCRmyPDF v2.2 and earlier) cannot be
removed this way, as documented.
Closes#1435.
When Tesseract reports a page with many diacritics, OCRmyPDF rewrites the
message to "lots of diacritics - possibly poor OCR", which hid the
original wording. Keep the interpreted hint but also emit Tesseract's raw
line at debug verbosity (-v 1) so users can see exactly what Tesseract
reported.
Closes#1566.
Ghostscript 10.x renders aliased glyphs that OCR frequently misreads as
extra word breaks or substituted characters. Enable text and graphics
anti-aliasing (-dTextAlphaBits=4 -dGraphicsAlphaBits=4) for the contone
raster devices, which empirically improves OCR accuracy on the
Ghostscript path, especially for small fonts at moderate DPI. The 1-bit
mono devices are excluded, since older Ghostscript rejects alpha bits on
them and pngmonod performs its own anti-aliased downscaling.
Also log which rasterizer rendered each page at debug verbosity and
clarify the --rasterizer help text, so quality reports are easier to
diagnose. The default rasterizer (auto) already prefers pypdfium2, which
anti-aliases; this primarily benefits --rasterizer ghostscript and
installs without pypdfium2.
Closes#1439.
The optimizer is best-effort: any image it cannot process is left
unchanged in the output, which remains valid. Previously, an extraction
failure (e.g. an exotic colorspace pikepdf cannot transcode) was logged
with log.exception, printing a full traceback at ERROR level that alarmed
users even though nothing was wrong with the output (issue #846).
Trap such failures with a concise warning that the image was left
unchanged and the output is still valid, and demote the traceback to
debug verbosity for diagnosis.
Ghostscript's `auto` image compression heuristic can transcode lossless
images to JPEG during PDF/A generation, which is surprising at
optimization levels that otherwise promise lossless-only operations
(issue #1124).
`--pdfa-image-compression=auto` (the default) now coerces to lossless at
-O0 so Ghostscript will not transcode lossless images to JPEG. -O1 and
above continue to defer to Ghostscript's heuristic; -O1 (the default
level) is kept as a historical exception because coercing it to lossless
substantially bloats output. Users wanting guaranteed lossless image
handling can pass --pdfa-image-compression=lossless or use -O0.
Also make `lossless` pass existing JPEGs through unchanged
(-dPassThroughJPEGImages=true) instead of re-encoding them with a
lossless codec, which only inflates already-lossy data.
Validate and repair the page-boundary boxes (MediaBox, CropBox, TrimBox,
ArtBox, BleedBox) of input PDFs in triage(), following PDF 2.0:
- Coerce coordinates written in invalid exponential notation, stored by
qpdf/pikepdf as strings (#1398).
- Normalize rectangles whose corners are given in reversed order, which
previously crashed with NegativeDimensionError (#1526).
- Clamp a crop/trim/art/bleed box that extends outside the MediaBox to
their intersection, or discard it when the intersection is empty, which
previously produced a zero-height effective page some viewers rejected
(#1400).
When a box is discarded, clamped, or reinterpreted, a warning recommends
visual inspection of the output. pdfinfo box reading shares the same
coercion helper so PdfInfo no longer crashes on malformed boxes.
Supersedes PR #1691. Thanks @ajdlinux.
Closes#1398, #1526, #1400
A tagged/structured PDF carries a logical structure tree
(/Root/StructTreeRoot, /MarkInfo) that maps marked page content to
semantic elements via MCIDs. When --force-ocr rasterizes pages or
--redo-ocr rewrites the text layer, those MCIDs are destroyed or
renumbered and the tree is left dangling. We cannot rebuild it to match
the new text, so discard it for force/redo modes, following the same
pattern as the thumbnail and search-index discards. --skip-text leaves
text pages untouched, so their structure is preserved.
Also broaden the default-mode "looks born-digital" stop signal to fire
on /StructTreeRoot, not just /MarkInfo/Marked, so structure-tree-only
PDFs are no longer silently OCR'd. The existing --tagged-pdf-mode ignore
escape hatch is unchanged.
A page may carry an optional /Thumb image XObject (ISO 32000-2 12.3.4),
a miniature rendering used only as a navigation aid. OCRmyPDF alters
page appearance (deskew, clean, rasterize, re-render) and plugins may
edit pages arbitrarily, so a retained thumbnail would be stale and no
longer match its page. Modern viewers generate thumbnails on demand, so
there is no loss of functionality.
OcrGrafter.finalize() now strips /Thumb from every page before saving,
alongside the existing search-index discard, covering both the OCR and
hOCR pipelines. Orphaned thumbnail XObjects are garbage-collected on save.
Adobe Acrobat can embed a proprietary full-text search index in the
document catalog at /Root/PieceInfo/SearchIndex. Only Acrobat reads it;
other viewers ignore it and search the text on the fly. Any change to a
PDF invalidates the index, so once OCRmyPDF rewrites the document the
retained index is stale and returns incorrect search results in Acrobat.
OcrGrafter.finalize() now discards it before saving (covering both the
OCR and hOCR pipelines), preserving any other PieceInfo owner data and
dropping an empty PieceInfo. Modern viewers rebuild a search index on
demand, so there is no loss of search capability.
test_gray_mask_ocrs_to_text loaded DejaVu, which only exists on the Linux
CI runners; the OSError fallback hardcoded a Linux-only path, so the test
failed with "cannot open resource" on macOS and Windows. Load the bundled
ocrmypdf.data/NotoSans-Regular.ttf via importlib.resources instead, which
is guaranteed present on every platform.
Address final review findings: a mask painted inside a Form XObject now
inherits the fill color in effect at the form's Do operator (previously it
reset to black, missing gray/color promotion one indirection deep). The cs
operator now resets the fill color to black per PDF spec, so a stale color
set before cs cannot leak to a subsequently drawn mask.
Track the current PDF fill color on the graphics stack alongside the CTM and
record an Ink classification (mono/gray/color) per image-draw event. Image
masks are painted with the current fill color, so this enables later device
promotion. Color operators are tolerant of malformed operands to preserve
robustness on untrusted input.
When Ghostscript reports a DeviceN colorspace with an inappropriate
alternate, the resulting PDF/A may render blank in viewers such as Adobe
Reader (#1187). The error is gated on that Ghostscript warning, which is
the authoritative signal that the *output* is broken.
Previously the error message always told the user to "use
--color-conversion-strategy", which is confusing when they already set
one and it didn't help. Crucially, the warning persists for strategies
that don't actually normalize the colorspace -- notably
UseDeviceIndependentColor (confirmed in #1187) -- so silencing the error
for any non-default strategy would emit a silently-broken PDF/A.
Keep raising whenever Ghostscript still reports the warning, regardless
of strategy, but tailor the guidance: if no conversion was requested,
suggest RGB/CMYK/Gray; if a conversion was requested but the warning
persisted, say so and point at strategies that work or --output-type pdf.
Add unit tests (mocked Ghostscript) covering the default case, the
warning-persists-despite-strategy case for both an ineffective strategy
and a normally-effective one, and the no-warning happy path.
Closes#1615. The token 'end' (case-insensitive) is now accepted as an
alias for the document's last page, e.g. --pages 3-end. Resolution is
deferred until the page count is known from the input PDF.
Expose Ghostscript's -dJPEGQ and image downsampling switches as
advanced, plugin-scoped options for tuning PDF/A output, without
polluting the central OcrOptions registry. The optimizer's existing
--jpeg-quality remains the recommended JPEG quality control.
- GhostscriptOptions gains jpeg_quality and jpeg_maxdpi fields and CLI
args (advanced help text). jpeg_quality=0 is honored as Ghostscript's
maximum compression rather than being silently coerced to the default.
- _exec.ghostscript.generate_pdfa() forwards both values; when
jpeg_maxdpi is set, downsample threshold is pinned at 1.0.
- _get_plugin_options falls back to extra_attrs for namespaced fields
so plugins can own their options without registering them centrally.
- Documentation explains the rationale: Ghostscript is the legacy path
(pypdfium + verapdf is preferred in v17+), the optimizer is the
supported file-size lever, and lowering quality is almost always a
better trade than downsampling.
When Tesseract cannot find its 'hocr' or 'txt' config files in the
tessdata configs/ directory, it prints "read_params_file: Can't open"
warnings, exits 0, and produces no output. OCRmyPDF then crashed with a
confusing FileNotFoundError on the missing hOCR file (issue #1687).
Promote read_params_file warnings to TesseractConfigError with guidance
on the likely cause, and verify the expected output file exists after
Tesseract claims success as defense-in-depth for other silent-failure
modes.
The 2024 guard against runaway recursion in _find_image_xrefs_container
only deduplicated image xrefs, but Form XObject xrefs are never added to
include_xrefs/exclude_xrefs, so a self-referential or DAG-shaped Form
graph re-entered every branch until the depth limit fired -- producing
the reported flood of warnings (and minutes-long hangs) on PowerPoint
exports.
Thread a visited_forms set through the recursion so each Form XObject is
descended into at most once per document. With memoization in place the
depth limit is no longer a cycle defense, so demote its log to debug.
Add a regression test that synthesises a circular-Form PDF from the
existing formxobject.pdf fixture (no new binary fixture, no license
issues) and asserts zero "Recursion depth exceeded" warnings.
fpdf2 >= 2.8.7 emits a custom begincidchar Encoding CMap for CFF-based
CID fonts (e.g. NotoSansCJK). pdfminer.six returns <CMap: None> for such
CMaps, so text extraction yields empty output. Switch to pdftotext (poppler)
which handles the new encoding correctly.
The API previously clobbered PIL.Image.MAX_IMAGE_PIXELS unconditionally
on every call, so host applications (e.g. Paperless-NGX) that configured
the PIL limit before invoking ocrmypdf.ocr() saw their setting silently
overwritten with the 250 MP default. Make max_image_mpixels default to
None and only apply the override when the caller explicitly sets it.
The CLI default of 250 MP is unchanged.
Fixes#1665
The JPEG truncation bug (1-15 bytes) persists in Ghostscript 10.7.0.
Update the warning message to show the actual GS version instead of
hardcoding "10.6.x", and remove the stale date reference. Also make
the test warning filter match the new message format.
fpdf2's shape_text() produces RTL ligature glyphs (e.g. lam-alef) with
multi-character CMap entries whose character order gets reversed by the
bidi algorithm during text extraction, producing garbled output like
"سالح" instead of "سلاح".
For invisible text (the production OCR overlay path), bypass text shaping
and use encode_text() with pre-reversed strings. encode_text() maps
characters 1:1 in logical order, avoiding the ligature CMap issue. The
pre-reversal compensates for bidi reversal by text extractors. Since the
text is invisible (Tr=3), the lack of joining forms is harmless.
Add RTL text extraction tests that verify glyph stream order, ToUnicode
CMap 1:1 mappings, and correct logical order for Arabic (including
lam-alef ligature) and Hebrew scripts.
The API's 'language' param was silently dropped because OcrOptions uses
'languages' (plural). Map language->languages in create_options() and
_pdf_to_hocr(), coercing bare strings to lists and splitting '+'
separated codes to match CLI behavior.
_build_text_layer_ctm() returned None when text_rotation was 0,
skipping the origin translation needed for pages with non-zero
MediaBox origins (e.g. JSTOR PDFs with [0, 100, 595, 982]). This
caused the text layer to be offset by the page origin amount.
Always compute the full CTM and only return None when the result
is the identity matrix.
Fixes#1630
OcrOptions.tesseract_timeout defaulted to 0.0, which caused
subprocess.run(timeout=0) to immediately raise TimeoutExpired before
Tesseract could produce any output. The CLI was unaffected because
argparse defaults --tesseract-timeout to 180. Change the OcrOptions
default to None so the plugin's own default (180s) is used.
Fixes#1636
James R. Barlow
Matt Van Horn
jbarlowandGitHub