namespace_to_options() only copied argparse namespace keys that were
literal members of OcrOptions.model_fields. The CLI dest for both
--jpeg-quality and --jpg-quality was jpeg_quality, but the pydantic field
was named jpg_quality (jpeg_quality existed only as a compatibility
property, absent from model_fields). The value was silently dropped into
extra_attrs, and the optimizer always fell back to its own hardcoded
default regardless of the flag.
The same alias mismatch also affected the Python API: create_options()
uses the same model_fields-matching logic as namespace_to_options(), so
ocrmypdf.ocr(jpeg_quality=...) was silently dropped too - only the
canonical jpg_quality= kwarg worked.
Rather than patch around the mismatch, consolidate on a single canonical
name: OcrOptions.jpeg_quality (matching the primary --jpeg-quality CLI
flag and the already-consistent naming in OptimizeOptions). jpg_quality
becomes a deprecated compatibility property, and ocrmypdf.ocr(jpg_quality=)
is a deprecated alias that warns and forwards to jpeg_quality via a new
create_options() remap step. --jpg-quality remains a working (already
hidden) CLI alias with no code-path divergence, since it now shares an
argparse dest that matches the field name directly.
SystemFontProvider.NOTO_FONT_PATTERNS enumerates about two dozen Noto
families by name, and MultiFontManager only ever asked for those. Any
script outside that list fell through to the glyphless Occulta fallback
even when the correct font was installed, which is the common case on
macOS: it ships around a hundred script-specific Noto faces in
/System/Library/Fonts/Supplemental, almost none of which we knew how to
ask for. The reporter had the fonts and still got told to install them.
Add an optional GlyphSearchingFontProvider protocol (find_font_with_glyphs)
implemented by SystemFontProvider, BuiltinFontProvider and
ChainedFontProvider, and a new selection phase that uses it once the named
families fail. The system scan enumerates every Noto face, reusing the
existing variable-font/-Regular/-VF filename classification, and keeps
only the font it selects so a full scan does not retain every font file on
the system. Fonts found this way are remembered and retried by name for
later words. Capability detection is isinstance-based so third-party
providers keep working unchanged.
The warning itself was also unactionable: it named neither the characters
nor the script, which is why the reporter had to ask which package to
install. It now identifies what it could not render, e.g.
'Ꮳ' U+13E3 CHEROKEE LETTER TSA. A word mixing scripts that no single font
covers now gets a distinct message saying that installing fonts will not
help, rather than sending the user after fonts they already have.
Finally, the documented macOS install command was wrong: `brew install
font-noto` does not exist, since Homebrew has no single Noto package, only
a cask per family. The Fedora package name was also corrected, as
google-noto-fonts-common ships no actual fonts.
A malformed PDF may store a non-dictionary object (an array, name, or
other type) at /Resources or /Resources /XObject. The pdfinfo image
scanner iterated these with .items()/.as_dict() and probed them with the
`in` operator, which raise TypeError/ValueError on non-dictionary pikepdf
objects and crashed PdfInfo on otherwise-processable files. OCRmyPDF's
domain is messy, machine-generated PDFs, so scanning must tolerate this.
Guard _image_xobjects and _find_form_xobject_images with
isinstance(x, Dictionary) before iterating, treating a non-dictionary
/Resources or /XObject as "no image XObjects". This is the same
robustness class as the pdfa.py find_nonembedded_cid_fonts fix, applied
to the pdfinfo image scanner.
Follow-on to the non-dictionary /Font and /XObject guard. Replace the
_dict_entries() helper with an isinstance(..., pikepdf.Dictionary) check
at each resource lookup, which pikepdf's metaclass supports directly and
which reads as exactly the invariant being enforced. Iterate via
as_dict().values() so the values are typed and mypy stays clean once the
Any from the untyped resources argument is narrowed away.
Also guard _cid_font_is_embedded against a non-dictionary /FontDescriptor:
`key in descriptor` raises ValueError on a non-dict, which the caller's
except (AttributeError, TypeError, KeyError) does not catch. Such a font
now counts as non-embedded and is reported, so PDF/A conversion is refused
rather than risking Ghostscript corrupting a pre-existing CID text layer.
Adds a regression test for the FontDescriptor case (issue #1713).
A malformed PDF can store a non-dictionary object under a page's /Font or
/XObject resource. find_nonembedded_cid_fonts() iterated .values() on that
object outside the per-entry try/except, so scanning such a page raised
(TypeError/AttributeError depending on the pikepdf version) instead of
producing output. This surfaced as a PDF/A conversion crash.
Route both resource lookups through a small helper that returns an empty
list when the resource is missing or not a dictionary, so a garbage entry
is simply treated as having no fonts. Add a regression test covering a
non-dictionary /Font and /XObject.
Replaces os.path/open()/os.stat()/os.chmod() calls with their Path
method equivalents across src, tests, misc, and bin, wrapping str
variables in Path(...) where they must stay str for other uses (e.g.
subprocess argv, CLI-arg formatting). helpers.safe_symlink() now
decodes StrOrBytesPath to a str Path via os.fsdecode() upfront, same
pattern already used elsewhere for the str|bytes union.
Pillow >=9.2 ships its own inline types (py.typed), which the separate
types-Pillow stub package shadows when both are installed. Since
types-Pillow lived in the `test` dependency group (not `dev`), a local
`uv sync --group dev --group test` env had it installed and mypy
silently preferred its stubs, while CI's lint job only syncs the
default `dev` group and used Pillow's own (correct) stubs - surfacing
5 real "Incompatible types in assignment" errors that never appeared
locally.
Fixed the underlying type errors in _pipeline.py and ghostscript.py:
`im` was inferred as ImageFile.ImageFile from `Image.open(...) as im`,
but later reassigned the result of `.resize()`/`.transpose()`, which
return the broader Image.Image - now declared explicitly as
`im: Image.Image` before the `with` block.
Fix the 13 errors that surfaced under mypy --check-untyped-defs so the
flag can be turned on permanently in pyproject.toml, and drop the
advisory exit-0 wrapper on the mypy pre-commit hook now that the tree is
clean.
- _plugin_manager: rename colliding loop vars (module/name were reused
with conflicting types) and guard spec/spec.loader from
spec_from_file_location; call __init__ via the class in __setstate__.
- __main__: pass Verbosity(options.verbose), not a bare int.
- subprocess/_check: widen package to str | Mapping[str, str] to match
_error_trailer's existing per-platform handling.
- optimize.main: annotate the standalone PdfContext(..., None, None) that
only ever reads context.options.
Resolves the last 16 mypy errors in the project (src/ocrmypdf and
tests are now fully clean).
fpdf_renderer/renderer.py (9 errors):
- add_page(format=...): fpdf2's own stub types this param as str, but
its docstring and get_page_format() helper confirm a (width, height)
tuple is accepted too - the stub annotation on add_page() itself is
the outlier. Used cast() to match the documented/actual behavior.
- pdf.current_font is typed CoreFont | TTFFont | None, but this
renderer only ever registers fonts via add_font() with a TTF file
(see _register_font/set_font call sites) - it never falls back to
fpdf2's built-in CoreFont. Added assertions (isinstance(font,
TTFFont) where shape_text()/escape_text() are needed, which
CoreFont lacks; plain not-None elsewhere) documenting that
invariant instead of narrowing defensively for a case that can't
happen here.
tests/test_pdf_renderer.py (7 errors): the ToUnicode/glyph-extraction
test helpers used `.get(key, {})` (a plain dict literal default) then
called `.values()`/`.items()` on the result. pikepdf.Object doesn't
declare `values()` in its stub (only `keys()`), so this silently
degraded to Object's catch-all `__getattr__` returning another Object,
which then failed as "not callable". Switched to `.get(key,
Dictionary()).as_dict()`, which returns pikepdf's properly-typed
_ObjectMapping helper.
Resolves all 11 remaining errors in _concurrent.py and
builtin_plugins/concurrency.py.
Root cause for 9 of the 11: setup_executor() was annotated to return
ocrmypdf's own Executor ABC as its second tuple element, but it
actually returns a concurrent.futures pool class (ThreadPoolExecutor
or ProcessPoolExecutor) - an already-existing FuturesExecutorClass
alias was defined for exactly this but never wired into the
signature. That wrong annotation made mypy check
`executor_class(initializer=..., initargs=...)` in _execute() against
Executor.__call__'s signature (which has entirely different
parameters and returns None), cascading into 8 further errors
(unexpected keyword args, "function does not return a value", "None
has no attribute __enter__/__exit__"). Fixing the one annotation
resolved all of them.
Also:
- Added a proper Queue[LogRecord | None] generic parameter (was bare
Queue, which needs an explicit type argument for mypy to infer
loq_queue's type across the use_threads branches).
- _concurrent.py: Executor.__call__'s task parameter defaults to
_task_noop (return type None) when the caller omits a task, but the
parameter is typed Callable[..., T] for an unbound per-call T. Used
cast() since task_finished's own no-op default already accepts Any,
so the mismatch is never actually exercised unsafely.
Resolves the remaining 22 pikepdf-related mypy errors in
pdfinfo/_image.py, pdfinfo/info.py, optimize.py, and _graft.py
(89 -> 27 errors remaining, all in the concurrency/fpdf_renderer
clusters).
Adds pikepdf_get_int()/pikepdf_get_bool() to helpers.py: safe
accessors for dict.get(key, default) results, whose static type is
the ambiguous `Object | int`/`Object | bool` and doesn't support
arithmetic/comparison against a plain int/bool.
Important correctness fix caught by the test suite: pikepdf only
returns pikepdf.Object wrappers under explicit_conversion() mode,
which this codebase never enables. By default (implicit mode), PDF
Integers/Booleans are already unboxed to native int/bool by the time
callers see them, so calling the `.as_int()`/`.as_bool()`/
`.as_decimal()` safe accessors unconditionally crashes with
AttributeError on the native-type case (test_oversized_page caught
this for UserUnit). Fixed by using int()/float() builtins, which work
polymorphically on both native numbers and pikepdf.Object (bool()
does not, so pikepdf_get_bool checks isinstance first).
Also:
- pdfinfo/info.py: pass page.obj (an Object) to
_process_content_streams() instead of page (a Page wrapper, not an
Object subtype).
- pdfinfo/_image.py: cast() around Matrix(array_object) - pikepdf's
stub omits the Object/Array constructor overload that the C++
implementation actually supports.
- optimize.py: cast() around Object.write()'s filter/decode_parms
args for the same reason.
- _graft.py: iterate parse_content_stream() via .operands/.operator
instead of tuple-unpacking, since ContentStreamInstruction supports
the legacy __getitem__-based iteration protocol but not __iter__,
which mypy doesn't statically recognize.
- pdfinfo/_worker.py: _pdf_pageinfo_concurrent's return type was
Sequence[PageInfo | None] but always returns a real list; narrowed
to match, fixing PdfInfo.pages' declared list[...] return type.
Reduces mypy errors in src/ocrmypdf and tests from 89 to 51.
- Replace the `deprecation` package with stdlib `warnings.deprecated`
(falling back to typing_extensions on <3.13); drop the dependency.
- Add pypdfium2/uharfbuzz/pi_heif to mypy's ignore_missing_imports
overrides (no upstream stubs); drop pluggy, which now ships py.typed.
- Add a tests.* mypy override so test functions aren't required to
annotate -> None.
Real bugs found and fixed along the way, not just annotations:
- OcrmypdfPluginManager had a `pluggy` property shadowing the `pluggy`
module import within its own class body, breaking every
`pluggy.PluginManager` annotation below it; renamed to
`pluggy_manager`.
- `_option_registry` was bolted onto OcrmypdfPluginManager from outside
and read via `getattr(..., None)` instead of being a declared
attribute; declared it properly.
- ValidationCoordinator.__init__ was typed to accept a raw
pluggy.PluginManager, but every caller passes the OcrmypdfPluginManager
wrapper.
- check_options_sidecar() did `options.output_file + '.txt'`, assuming
output_file is always a str; would raise a raw TypeError if ever hit
with a stream/bytes output. Added an explicit guard.
- is_file_writable() called Path(test_file), which raises TypeError on
a bytes path; fixed via os.fsdecode().
- copy_final() had a dead, unused `original_file` parameter; removed it.
- run_hocr_pipeline() constructed PdfContext with the raw, untriaged
input_file instead of the locally-copied origin_pdf, inconsistent
with the other two pipelines.
- _options.py had jbig2_threshold declared twice in the same model.
prek runs local hooks as plain execs against tools uv already provisions,
so ruff/mypy can never drift from the versions/config uv.lock pins
elsewhere and CI needs no separate hook-cache download.
- Add ruff and prek to the uv dev dependency group (ruff wasn't a
uv-managed dependency before; pre-commit silently vendored its own).
- Replace .pre-commit-config.yaml with prek.toml: keep the
pre-commit-hooks repo for generic file checks, convert ruff-format/
ruff-check to local `uv run ruff ...` hooks, and add a local mypy
hook that reports but never fails (87 pre-existing errors need a
separate cleanup before it can be made blocking).
- Add a `lint` job to CI that runs `prek run --all-files` and gate the
OS/Python test matrix on it so lint issues fail fast.
- Fix the ruff debt (format + lint) uncovered by actually running it,
since it was small and mechanical, so the new CI gate starts green.
Ghostscript's PDF/A conversion re-embeds non-embedded CID (CJK) fonts by
substituting a system font, which corrupts the character-to-Unicode
mapping and silently destroys an existing text layer -- commonly the OCR
layer Adobe Acrobat adds to scanned CJK documents.
Detect non-embedded CID/Type0 fonts before conversion: with
--output-type auto (the default) downgrade to a regular PDF and preserve
the text layer; with an explicit --output-type pdfa* stop with an error
rather than emit corrupted output. Simple non-embedded fonts (e.g. Latin)
are left alone -- Ghostscript substitutes them without corrupting the
text, and they are far too common to treat as conversion blockers.
Use --output-type pdf to keep the existing text layer, or --force-ocr to
rebuild it with embedded fonts.
Writing the output PDF to stdout (ocrmypdf in.pdf -) previously relied on
an honor system: no in-process code -- third-party libraries, plugins, or
stray print() calls -- was supposed to write to stdout, enforced only
indirectly. A single accidental write to fd 1 would silently corrupt the
output PDF.
Enforce this at the OS level. At CLI startup, before plugins load or any
worker process/thread starts, save the real stdout via os.dup() and point
fd 1 at stderr, so stray writes are diverted to stderr while only the final
"produce the PDF" step writes to the preserved descriptor. Exposed as the
opt-in public API function configure_stdout_protection(), mirroring
configure_logging(); it is not enabled inside ocr() so in-process library
users keep their own stdout.
Also fix check_requested_output_file() to test the preserved real stdout
for tty-ness, since after the redirect sys.stdout reports stderr's status.
Fold unreleased v17.7.2 notes into v17.8.0.
Some PDFs use a /Name dictionary key in /DocumentInfo whose bytes are not
valid UTF-8/PDFDocEncoding, e.g. a Latin-1 /Saks#e5r. Older pikepdf raised
UnicodeDecodeError while iterating such a block, crashing the pipeline
during PDF/A conversion. repair_docinfo_nuls is documented to log and
continue on a malformed DocumentInfo block, so catch UnicodeDecodeError
alongside TypeError.
Add a mock-based unit test that drives the decode-error branch (current
pikepdf surrogate-escapes instead of raising) and an end-to-end test over
the reporter's file, committed as docinfo_latin1_key.pdf.
Since v16.4.3, OCRmyPDF forced pdfminer's read buffer to 256 MiB to work
around a pdfminer bug that mishandled tokens split across the buffer
boundary (gh #1361). On Windows this caused a severe performance
regression (gh #1662): CPython's BufferedReader.read(n) eagerly allocates
an n-byte buffer on every read, so pdfminer's thousands of seek+read
cycles each paid a ~30 ms 256 MiB allocation (this allocation is lazy and
effectively free on Linux). For a typical PDF the "Scanning contents"
phase went from ~5s on Linux to ~60s on Windows.
The underlying pdfminer bug was fixed upstream in pdfminer.six 20250327
(pdfminer/pdfminer.six#1030), with a follow-up for tokens split across
streams in 20260107 (pdfminer/pdfminer.six#1158). Remove the monkeypatch
entirely and raise the minimum pdfminer.six to 20260107 so we rely on the
upstream fix instead.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Bump uv lock, fix bump_version
* Discover variable and per-language Noto fonts for the OCR text layer (#1652)
SystemFontProvider only matched static "-Regular.ttf/.otf" filenames, so
the variable fonts shipped by Homebrew casks and current Google Fonts
(e.g. NotoSansArabic[wdth,wght].ttf) were never found. Users who had
installed the font still got the glyphless Occulta fallback and a cryptic
"No font found" warning.
- Match variable fonts (Base[...]), -VF, and bare-family filenames via a
boundary-aware flexible search, escaping the glob-special brackets.
- Make CJK language-aware: the modern per-language Noto fonts (NotoSansSC
/TC/HK/JP/KR) are region subsets, so map each CJK language to its own
family and keep the full-coverage pan-CJK super font as a shared
fallback. Glyph coverage, not shape, is what matters for the invisible
text layer.
- Reword the missing-font warning to explain the consequence (searchable
but blank when highlighted) and name the language-specific font.
Adds a processing mode that removes the invisible (render mode 3) OCR text
layer in place. Unlike `--ocr-engine none --force-ocr`, it does not
rasterize the page, so images and visible content are preserved unchanged
and the output is smaller rather than larger. Options that require
rasterization or OCR (--deskew, --clean, --sidecar, etc.) are rejected.
Only invisible text is removed; text drawn as visible glyphs under an
opaque image (some OCR engines, and OCRmyPDF v2.2 and earlier) cannot be
removed this way, as documented.
Closes#1435.
When Tesseract reports a page with many diacritics, OCRmyPDF rewrites the
message to "lots of diacritics - possibly poor OCR", which hid the
original wording. Keep the interpreted hint but also emit Tesseract's raw
line at debug verbosity (-v 1) so users can see exactly what Tesseract
reported.
Closes#1566.
Ghostscript 10.x renders aliased glyphs that OCR frequently misreads as
extra word breaks or substituted characters. Enable text and graphics
anti-aliasing (-dTextAlphaBits=4 -dGraphicsAlphaBits=4) for the contone
raster devices, which empirically improves OCR accuracy on the
Ghostscript path, especially for small fonts at moderate DPI. The 1-bit
mono devices are excluded, since older Ghostscript rejects alpha bits on
them and pngmonod performs its own anti-aliased downscaling.
Also log which rasterizer rendered each page at debug verbosity and
clarify the --rasterizer help text, so quality reports are easier to
diagnose. The default rasterizer (auto) already prefers pypdfium2, which
anti-aliases; this primarily benefits --rasterizer ghostscript and
installs without pypdfium2.
Closes#1439.
The optimizer is best-effort: any image it cannot process is left
unchanged in the output, which remains valid. Previously, an extraction
failure (e.g. an exotic colorspace pikepdf cannot transcode) was logged
with log.exception, printing a full traceback at ERROR level that alarmed
users even though nothing was wrong with the output (issue #846).
Trap such failures with a concise warning that the image was left
unchanged and the output is still valid, and demote the traceback to
debug verbosity for diagnosis.
Ghostscript's `auto` image compression heuristic can transcode lossless
images to JPEG during PDF/A generation, which is surprising at
optimization levels that otherwise promise lossless-only operations
(issue #1124).
`--pdfa-image-compression=auto` (the default) now coerces to lossless at
-O0 so Ghostscript will not transcode lossless images to JPEG. -O1 and
above continue to defer to Ghostscript's heuristic; -O1 (the default
level) is kept as a historical exception because coercing it to lossless
substantially bloats output. Users wanting guaranteed lossless image
handling can pass --pdfa-image-compression=lossless or use -O0.
Also make `lossless` pass existing JPEGs through unchanged
(-dPassThroughJPEGImages=true) instead of re-encoding them with a
lossless codec, which only inflates already-lossy data.
Validate and repair the page-boundary boxes (MediaBox, CropBox, TrimBox,
ArtBox, BleedBox) of input PDFs in triage(), following PDF 2.0:
- Coerce coordinates written in invalid exponential notation, stored by
qpdf/pikepdf as strings (#1398).
- Normalize rectangles whose corners are given in reversed order, which
previously crashed with NegativeDimensionError (#1526).
- Clamp a crop/trim/art/bleed box that extends outside the MediaBox to
their intersection, or discard it when the intersection is empty, which
previously produced a zero-height effective page some viewers rejected
(#1400).
When a box is discarded, clamped, or reinterpreted, a warning recommends
visual inspection of the output. pdfinfo box reading shares the same
coercion helper so PdfInfo no longer crashes on malformed boxes.
Supersedes PR #1691. Thanks @ajdlinux.
Closes#1398, #1526, #1400
A tagged/structured PDF carries a logical structure tree
(/Root/StructTreeRoot, /MarkInfo) that maps marked page content to
semantic elements via MCIDs. When --force-ocr rasterizes pages or
--redo-ocr rewrites the text layer, those MCIDs are destroyed or
renumbered and the tree is left dangling. We cannot rebuild it to match
the new text, so discard it for force/redo modes, following the same
pattern as the thumbnail and search-index discards. --skip-text leaves
text pages untouched, so their structure is preserved.
Also broaden the default-mode "looks born-digital" stop signal to fire
on /StructTreeRoot, not just /MarkInfo/Marked, so structure-tree-only
PDFs are no longer silently OCR'd. The existing --tagged-pdf-mode ignore
escape hatch is unchanged.
A page may carry an optional /Thumb image XObject (ISO 32000-2 12.3.4),
a miniature rendering used only as a navigation aid. OCRmyPDF alters
page appearance (deskew, clean, rasterize, re-render) and plugins may
edit pages arbitrarily, so a retained thumbnail would be stale and no
longer match its page. Modern viewers generate thumbnails on demand, so
there is no loss of functionality.
OcrGrafter.finalize() now strips /Thumb from every page before saving,
alongside the existing search-index discard, covering both the OCR and
hOCR pipelines. Orphaned thumbnail XObjects are garbage-collected on save.
Adobe Acrobat can embed a proprietary full-text search index in the
document catalog at /Root/PieceInfo/SearchIndex. Only Acrobat reads it;
other viewers ignore it and search the text on the fly. Any change to a
PDF invalidates the index, so once OCRmyPDF rewrites the document the
retained index is stale and returns incorrect search results in Acrobat.
OcrGrafter.finalize() now discards it before saving (covering both the
OCR and hOCR pipelines), preserving any other PieceInfo owner data and
dropping an empty PieceInfo. Modern viewers rebuild a search index on
demand, so there is no loss of search capability.
Address final review findings: a mask painted inside a Form XObject now
inherits the fill color in effect at the form's Do operator (previously it
reset to black, missing gray/color promotion one indirection deep). The cs
operator now resets the fill color to black per PDF spec, so a stale color
set before cs cannot leak to a subsequently drawn mask.
Track the current PDF fill color on the graphics stack alongside the CTM and
record an Ink classification (mono/gray/color) per image-draw event. Image
masks are painted with the current fill color, so this enables later device
promotion. Color operators are tolerant of malformed operands to preserve
robustness on untrusted input.
James R. Barlow
Matt Van Horn
sokaiandGitHub
