prek runs local hooks as plain execs against tools uv already provisions,
so ruff/mypy can never drift from the versions/config uv.lock pins
elsewhere and CI needs no separate hook-cache download.
- Add ruff and prek to the uv dev dependency group (ruff wasn't a
uv-managed dependency before; pre-commit silently vendored its own).
- Replace .pre-commit-config.yaml with prek.toml: keep the
pre-commit-hooks repo for generic file checks, convert ruff-format/
ruff-check to local `uv run ruff ...` hooks, and add a local mypy
hook that reports but never fails (87 pre-existing errors need a
separate cleanup before it can be made blocking).
- Add a `lint` job to CI that runs `prek run --all-files` and gate the
OS/Python test matrix on it so lint issues fail fast.
- Fix the ruff debt (format + lint) uncovered by actually running it,
since it was small and mechanical, so the new CI gate starts green.
Ghostscript's PDF/A conversion re-embeds non-embedded CID (CJK) fonts by
substituting a system font, which corrupts the character-to-Unicode
mapping and silently destroys an existing text layer -- commonly the OCR
layer Adobe Acrobat adds to scanned CJK documents.
Detect non-embedded CID/Type0 fonts before conversion: with
--output-type auto (the default) downgrade to a regular PDF and preserve
the text layer; with an explicit --output-type pdfa* stop with an error
rather than emit corrupted output. Simple non-embedded fonts (e.g. Latin)
are left alone -- Ghostscript substitutes them without corrupting the
text, and they are far too common to treat as conversion blockers.
Use --output-type pdf to keep the existing text layer, or --force-ocr to
rebuild it with embedded fonts.
Writing the output PDF to stdout (ocrmypdf in.pdf -) previously relied on
an honor system: no in-process code -- third-party libraries, plugins, or
stray print() calls -- was supposed to write to stdout, enforced only
indirectly. A single accidental write to fd 1 would silently corrupt the
output PDF.
Enforce this at the OS level. At CLI startup, before plugins load or any
worker process/thread starts, save the real stdout via os.dup() and point
fd 1 at stderr, so stray writes are diverted to stderr while only the final
"produce the PDF" step writes to the preserved descriptor. Exposed as the
opt-in public API function configure_stdout_protection(), mirroring
configure_logging(); it is not enabled inside ocr() so in-process library
users keep their own stdout.
Also fix check_requested_output_file() to test the preserved real stdout
for tty-ness, since after the redirect sys.stdout reports stderr's status.
Fold unreleased v17.7.2 notes into v17.8.0.
Some PDFs use a /Name dictionary key in /DocumentInfo whose bytes are not
valid UTF-8/PDFDocEncoding, e.g. a Latin-1 /Saks#e5r. Older pikepdf raised
UnicodeDecodeError while iterating such a block, crashing the pipeline
during PDF/A conversion. repair_docinfo_nuls is documented to log and
continue on a malformed DocumentInfo block, so catch UnicodeDecodeError
alongside TypeError.
Add a mock-based unit test that drives the decode-error branch (current
pikepdf surrogate-escapes instead of raising) and an end-to-end test over
the reporter's file, committed as docinfo_latin1_key.pdf.
Since v16.4.3, OCRmyPDF forced pdfminer's read buffer to 256 MiB to work
around a pdfminer bug that mishandled tokens split across the buffer
boundary (gh #1361). On Windows this caused a severe performance
regression (gh #1662): CPython's BufferedReader.read(n) eagerly allocates
an n-byte buffer on every read, so pdfminer's thousands of seek+read
cycles each paid a ~30 ms 256 MiB allocation (this allocation is lazy and
effectively free on Linux). For a typical PDF the "Scanning contents"
phase went from ~5s on Linux to ~60s on Windows.
The underlying pdfminer bug was fixed upstream in pdfminer.six 20250327
(pdfminer/pdfminer.six#1030), with a follow-up for tokens split across
streams in 20260107 (pdfminer/pdfminer.six#1158). Remove the monkeypatch
entirely and raise the minimum pdfminer.six to 20260107 so we rely on the
upstream fix instead.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Bump uv lock, fix bump_version
* Discover variable and per-language Noto fonts for the OCR text layer (#1652)
SystemFontProvider only matched static "-Regular.ttf/.otf" filenames, so
the variable fonts shipped by Homebrew casks and current Google Fonts
(e.g. NotoSansArabic[wdth,wght].ttf) were never found. Users who had
installed the font still got the glyphless Occulta fallback and a cryptic
"No font found" warning.
- Match variable fonts (Base[...]), -VF, and bare-family filenames via a
boundary-aware flexible search, escaping the glob-special brackets.
- Make CJK language-aware: the modern per-language Noto fonts (NotoSansSC
/TC/HK/JP/KR) are region subsets, so map each CJK language to its own
family and keep the full-coverage pan-CJK super font as a shared
fallback. Glyph coverage, not shape, is what matters for the invisible
text layer.
- Reword the missing-font warning to explain the consequence (searchable
but blank when highlighted) and name the language-specific font.
Adds a processing mode that removes the invisible (render mode 3) OCR text
layer in place. Unlike `--ocr-engine none --force-ocr`, it does not
rasterize the page, so images and visible content are preserved unchanged
and the output is smaller rather than larger. Options that require
rasterization or OCR (--deskew, --clean, --sidecar, etc.) are rejected.
Only invisible text is removed; text drawn as visible glyphs under an
opaque image (some OCR engines, and OCRmyPDF v2.2 and earlier) cannot be
removed this way, as documented.
Closes#1435.
When Tesseract reports a page with many diacritics, OCRmyPDF rewrites the
message to "lots of diacritics - possibly poor OCR", which hid the
original wording. Keep the interpreted hint but also emit Tesseract's raw
line at debug verbosity (-v 1) so users can see exactly what Tesseract
reported.
Closes#1566.
Ghostscript 10.x renders aliased glyphs that OCR frequently misreads as
extra word breaks or substituted characters. Enable text and graphics
anti-aliasing (-dTextAlphaBits=4 -dGraphicsAlphaBits=4) for the contone
raster devices, which empirically improves OCR accuracy on the
Ghostscript path, especially for small fonts at moderate DPI. The 1-bit
mono devices are excluded, since older Ghostscript rejects alpha bits on
them and pngmonod performs its own anti-aliased downscaling.
Also log which rasterizer rendered each page at debug verbosity and
clarify the --rasterizer help text, so quality reports are easier to
diagnose. The default rasterizer (auto) already prefers pypdfium2, which
anti-aliases; this primarily benefits --rasterizer ghostscript and
installs without pypdfium2.
Closes#1439.
The optimizer is best-effort: any image it cannot process is left
unchanged in the output, which remains valid. Previously, an extraction
failure (e.g. an exotic colorspace pikepdf cannot transcode) was logged
with log.exception, printing a full traceback at ERROR level that alarmed
users even though nothing was wrong with the output (issue #846).
Trap such failures with a concise warning that the image was left
unchanged and the output is still valid, and demote the traceback to
debug verbosity for diagnosis.
Ghostscript's `auto` image compression heuristic can transcode lossless
images to JPEG during PDF/A generation, which is surprising at
optimization levels that otherwise promise lossless-only operations
(issue #1124).
`--pdfa-image-compression=auto` (the default) now coerces to lossless at
-O0 so Ghostscript will not transcode lossless images to JPEG. -O1 and
above continue to defer to Ghostscript's heuristic; -O1 (the default
level) is kept as a historical exception because coercing it to lossless
substantially bloats output. Users wanting guaranteed lossless image
handling can pass --pdfa-image-compression=lossless or use -O0.
Also make `lossless` pass existing JPEGs through unchanged
(-dPassThroughJPEGImages=true) instead of re-encoding them with a
lossless codec, which only inflates already-lossy data.
Validate and repair the page-boundary boxes (MediaBox, CropBox, TrimBox,
ArtBox, BleedBox) of input PDFs in triage(), following PDF 2.0:
- Coerce coordinates written in invalid exponential notation, stored by
qpdf/pikepdf as strings (#1398).
- Normalize rectangles whose corners are given in reversed order, which
previously crashed with NegativeDimensionError (#1526).
- Clamp a crop/trim/art/bleed box that extends outside the MediaBox to
their intersection, or discard it when the intersection is empty, which
previously produced a zero-height effective page some viewers rejected
(#1400).
When a box is discarded, clamped, or reinterpreted, a warning recommends
visual inspection of the output. pdfinfo box reading shares the same
coercion helper so PdfInfo no longer crashes on malformed boxes.
Supersedes PR #1691. Thanks @ajdlinux.
Closes#1398, #1526, #1400
A tagged/structured PDF carries a logical structure tree
(/Root/StructTreeRoot, /MarkInfo) that maps marked page content to
semantic elements via MCIDs. When --force-ocr rasterizes pages or
--redo-ocr rewrites the text layer, those MCIDs are destroyed or
renumbered and the tree is left dangling. We cannot rebuild it to match
the new text, so discard it for force/redo modes, following the same
pattern as the thumbnail and search-index discards. --skip-text leaves
text pages untouched, so their structure is preserved.
Also broaden the default-mode "looks born-digital" stop signal to fire
on /StructTreeRoot, not just /MarkInfo/Marked, so structure-tree-only
PDFs are no longer silently OCR'd. The existing --tagged-pdf-mode ignore
escape hatch is unchanged.
A page may carry an optional /Thumb image XObject (ISO 32000-2 12.3.4),
a miniature rendering used only as a navigation aid. OCRmyPDF alters
page appearance (deskew, clean, rasterize, re-render) and plugins may
edit pages arbitrarily, so a retained thumbnail would be stale and no
longer match its page. Modern viewers generate thumbnails on demand, so
there is no loss of functionality.
OcrGrafter.finalize() now strips /Thumb from every page before saving,
alongside the existing search-index discard, covering both the OCR and
hOCR pipelines. Orphaned thumbnail XObjects are garbage-collected on save.
Adobe Acrobat can embed a proprietary full-text search index in the
document catalog at /Root/PieceInfo/SearchIndex. Only Acrobat reads it;
other viewers ignore it and search the text on the fly. Any change to a
PDF invalidates the index, so once OCRmyPDF rewrites the document the
retained index is stale and returns incorrect search results in Acrobat.
OcrGrafter.finalize() now discards it before saving (covering both the
OCR and hOCR pipelines), preserving any other PieceInfo owner data and
dropping an empty PieceInfo. Modern viewers rebuild a search index on
demand, so there is no loss of search capability.
Address final review findings: a mask painted inside a Form XObject now
inherits the fill color in effect at the form's Do operator (previously it
reset to black, missing gray/color promotion one indirection deep). The cs
operator now resets the fill color to black per PDF spec, so a stale color
set before cs cannot leak to a subsequently drawn mask.
Track the current PDF fill color on the graphics stack alongside the CTM and
record an Ink classification (mono/gray/color) per image-draw event. Image
masks are painted with the current fill color, so this enables later device
promotion. Color operators are tolerant of malformed operands to preserve
robustness on untrusted input.
When Ghostscript reports a DeviceN colorspace with an inappropriate
alternate, the resulting PDF/A may render blank in viewers such as Adobe
Reader (#1187). The error is gated on that Ghostscript warning, which is
the authoritative signal that the *output* is broken.
Previously the error message always told the user to "use
--color-conversion-strategy", which is confusing when they already set
one and it didn't help. Crucially, the warning persists for strategies
that don't actually normalize the colorspace -- notably
UseDeviceIndependentColor (confirmed in #1187) -- so silencing the error
for any non-default strategy would emit a silently-broken PDF/A.
Keep raising whenever Ghostscript still reports the warning, regardless
of strategy, but tailor the guidance: if no conversion was requested,
suggest RGB/CMYK/Gray; if a conversion was requested but the warning
persisted, say so and point at strategies that work or --output-type pdf.
Add unit tests (mocked Ghostscript) covering the default case, the
warning-persists-despite-strategy case for both an ineffective strategy
and a normally-effective one, and the no-warning happy path.
Closes#1615. The token 'end' (case-insensitive) is now accepted as an
alias for the document's last page, e.g. --pages 3-end. Resolution is
deferred until the page count is known from the input PDF.
PDFium does not support negative crop values to expand the render
area beyond the CropBox: such values only pad the output canvas with
white, leaving content outside the CropBox clipped. Set the in-memory
CropBox to the MediaBox before rendering instead. Reported in #1685.
Expose Ghostscript's -dJPEGQ and image downsampling switches as
advanced, plugin-scoped options for tuning PDF/A output, without
polluting the central OcrOptions registry. The optimizer's existing
--jpeg-quality remains the recommended JPEG quality control.
- GhostscriptOptions gains jpeg_quality and jpeg_maxdpi fields and CLI
args (advanced help text). jpeg_quality=0 is honored as Ghostscript's
maximum compression rather than being silently coerced to the default.
- _exec.ghostscript.generate_pdfa() forwards both values; when
jpeg_maxdpi is set, downsample threshold is pinned at 1.0.
- _get_plugin_options falls back to extra_attrs for namespaced fields
so plugins can own their options without registering them centrally.
- Documentation explains the rationale: Ghostscript is the legacy path
(pypdfium + verapdf is preferred in v17+), the optimizer is the
supported file-size lever, and lowering quality is almost always a
better trade than downsampling.
When Tesseract cannot find its 'hocr' or 'txt' config files in the
tessdata configs/ directory, it prints "read_params_file: Can't open"
warnings, exits 0, and produces no output. OCRmyPDF then crashed with a
confusing FileNotFoundError on the missing hOCR file (issue #1687).
Promote read_params_file warnings to TesseractConfigError with guidance
on the likely cause, and verify the expected output file exists after
Tesseract claims success as defense-in-depth for other silent-failure
modes.
The 2024 guard against runaway recursion in _find_image_xrefs_container
only deduplicated image xrefs, but Form XObject xrefs are never added to
include_xrefs/exclude_xrefs, so a self-referential or DAG-shaped Form
graph re-entered every branch until the depth limit fired -- producing
the reported flood of warnings (and minutes-long hangs) on PowerPoint
exports.
Thread a visited_forms set through the recursion so each Form XObject is
descended into at most once per document. With memoization in place the
depth limit is no longer a cycle defense, so demote its log to debug.
Add a regression test that synthesises a circular-Form PDF from the
existing formxobject.pdf fixture (no new binary fixture, no license
issues) and asserts zero "Recursion depth exceeded" warnings.
Split ocrmypdf.subprocess/__init__.py into three private submodules by
concern (_run, _version, _check) and reduce __init__ to re-exports.
Introduce ocrmypdf._exec._probe.ToolProbe to centralize the version()/
available() pattern each tool module was reimplementing, so the "is this
tool installed and suitable?" question is cleanly distinct from the
pure, picklable functions that do the work.
Also replace the ghostscript module-import log.addFilter() side effect
with an idempotent _ensure_log_filter_installed() called at the top of
each work function, so the DuplicateFilter is present in subprocess
workers without relying on import-time ordering.
Public API of ocrmypdf.subprocess is unchanged.
The API previously clobbered PIL.Image.MAX_IMAGE_PIXELS unconditionally
on every call, so host applications (e.g. Paperless-NGX) that configured
the PIL limit before invoking ocrmypdf.ocr() saw their setting silently
overwritten with the 250 MP default. Make max_image_mpixels default to
None and only apply the override when the caller explicitly sets it.
The CLI default of 250 MP is unchanged.
Fixes#1665
The JPEG truncation bug (1-15 bytes) persists in Ghostscript 10.7.0.
Update the warning message to show the actual GS version instead of
hardcoding "10.6.x", and remove the stale date reference. Also make
the test warning filter match the new message format.
fpdf2's shape_text() produces RTL ligature glyphs (e.g. lam-alef) with
multi-character CMap entries whose character order gets reversed by the
bidi algorithm during text extraction, producing garbled output like
"سالح" instead of "سلاح".
For invisible text (the production OCR overlay path), bypass text shaping
and use encode_text() with pre-reversed strings. encode_text() maps
characters 1:1 in logical order, avoiding the ligature CMap issue. The
pre-reversal compensates for bidi reversal by text extractors. Since the
text is invisible (Tr=3), the lack of joining forms is harmless.
Add RTL text extraction tests that verify glyph stream order, ToUnicode
CMap 1:1 mappings, and correct logical order for Arabic (including
lam-alef ligature) and Hebrew scripts.
James R. Barlow
sokaiandGitHub
