Compare commits

...
52 Commits
Author SHA1 Message Date
James R. Barlow 8224d89bc6 v11.3.4 release notes 2020-11-18 11:57:28 -08:00
James R. Barlow a2bbbe2a26 v11.3.4 release notes 2020-11-18 11:56:29 -08:00
James R. Barlow 43f41863fa check_pdf: document how we handle linearization 2020-11-18 11:54:07 -08:00
James R. Barlow d71e50e83d Fix "readLinearizationData for file that is not linearized"
pikepdf 2.1.0 throws wrong type of exception in this case, so special-case it.

Closes #680
Closes #681
2020-11-18 11:52:17 -08:00
James R. Barlow 1f598da3c1 ghostscript: better docs and comments 2020-11-18 11:34:17 -08:00
James R. Barlow d0cdbd5e1c watcher: include uppercase .PDF too 2020-11-12 02:29:47 -08:00
James R. Barlow 5c56f61209 unpaper: type hints 2020-11-11 02:59:37 -08:00
James R. Barlow 9bec85470a Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-11-10 04:08:05 -08:00
James R. Barlow a03863a17d docs: fix link to docker image 2020-11-10 04:08:01 -08:00
James R. Barlow 22cd9b2364 docs: fix csv-table errors 2020-11-10 04:07:49 -08:00
pretentious7andGitHub 4fc7d6d93e fix typo "charcter" -> "character" (#673) 2020-11-09 16:53:02 -08:00
James R. Barlow 71f0e7f545 v11.3.3 release notes 2020-11-07 00:53:33 -08:00
James R. Barlow 895fddd85e Replace most uses of universal_newlines with text
The parameters are equivalent but the latter is better named. Since
Python 3.6 doesn't support text= we use our wrapper to add it in that
place.

This is for subprocess.run.
2020-11-07 00:48:08 -08:00
James R. Barlow 5a59e4d543 unpaper: don't use universal_newlines=True
There's no specific reason to do this. We can log binary output equally
 well.
2020-11-07 00:18:27 -08:00
James R. Barlow b51abf2249 azure: Fix indentation mistake 2020-11-04 12:19:35 -08:00
James R. Barlow 6d3f9ff15a api: rework ocr() slightly to simplify variable handling 2020-11-03 17:10:52 -08:00
James R. Barlow 5d1d1a712b docs: more details about macOS API changes
Due to fork->spawn
2020-11-03 17:09:58 -08:00
James R. Barlow 6d5f8133e0 docs: show ifmain guard in example 2020-11-03 15:28:33 -08:00
James R. Barlow 13018d3d5c ci: Extend test matrix to Python 3.9 2020-11-03 04:15:14 -08:00
James R. Barlow 14a85f9473 Fix pinned dependencies 2020-11-03 04:12:47 -08:00
James R. Barlow d22a1b3367 v11.3.2 release notes (2)
Since we never tagged it, fix other things.
2020-11-03 02:03:25 -08:00
James R. Barlow b913e5dfef ghostscript: don't repeat log in debug
Subprocess already does this for us.
2020-11-03 01:45:06 -08:00
James R. Barlow dd8a5a4c72 Fix log domain names
ocrmypdf.subprocess.subprocess.ghostscript -> ocrmypdf.subprocess.ghostscript
2020-11-03 01:44:35 -08:00
James R. Barlow 36e9a54f02 Remove extraneous page rotation
This was added in commit b5ccbfd but seems to have been ill-advised.
2020-11-03 01:34:28 -08:00
James R. Barlow 3707af3b74 Change pdf.root to pdf.Root 2020-11-03 01:30:31 -08:00
James R. Barlow ced7ad9164 unpaper: round off DPI 2020-11-03 01:14:57 -08:00
James R. Barlow 54bbbfdeb3 Fix UnboundLocalError when considering ImageMasks for optimization
Uncovered by test file in issue 667, although unrelated to that issue.
2020-11-03 01:08:14 -08:00
James R. Barlow 7f73a6ed1e Some Python 3.9 fixes 2020-11-03 00:45:47 -08:00
James R. Barlow dce206d3dc Fix pre-commit for Py3.9 2020-11-03 00:20:25 -08:00
James R. Barlow 9304c856cf Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-11-02 02:47:36 -08:00
James R. Barlow e5df98cbdf v11.3.2 release notes 2020-11-02 02:43:32 -08:00
James R. Barlow 19bf3aeb00 api: improve typing 2020-11-02 02:33:34 -08:00
James R. Barlow e86be0031c unpaper: fix process output handling
With the ocrmypdf.subprocess wrapper, logging the output here
is redundant and loses the page number context.
2020-11-02 01:07:41 -08:00
James R. Barlow 6425977998 unpaper: use pnm instead of png
Some users reported problems with PNG recently; try PNM.

Fixes #665
Fixes #667
2020-11-02 01:05:56 -08:00
James R. Barlow d57df2d980 subprocess: support programs that write their messages to stdout 2020-11-02 01:00:59 -08:00
James R. Barlow 664d0c7969 Document configure_debug_logging 2020-11-02 00:59:00 -08:00
James R. Barlow a354663ee1 Fix typo in API documentation 2020-11-02 00:58:28 -08:00
Graham MilnandGitHub b21b048ec4 Add macOS brew language support (#615)
Note `brew` command for installing additional languages on macOS.
2020-10-30 01:09:06 -07:00
James R. Barlow 709c65b41a v11.3.1 release notes 2020-10-27 23:11:11 -07:00
James R. Barlow 67f99c5bb7 Endorse pdfminer.six 20201018 2020-10-27 23:09:45 -07:00
James R. Barlow d55e673d9c Fix warning about --pdfa-image-compression argument at wrong times
Closes #663
2020-10-27 23:09:45 -07:00
James R. Barlow 21b90d2d14 Endorse pikepdf 2.x 2020-10-27 23:09:45 -07:00
Edward BettsandGitHub 2def7e3392 Use % for percentage in string format (#643) 2020-10-27 23:09:14 -07:00
James R. Barlow b0dcaa7512 v11.3.0 release notes 2020-10-24 03:19:32 -07:00
James R. Barlow e8285b1d10 Add test to confirm rasterize_pdf_page rotates correct 2020-10-24 03:10:59 -07:00
James R. Barlow 5ba56adb53 Fix page rotation issue (again)
Commit 1327ab3 introduced a fix for a regression, which was reported
in #581, #634. It appears that the actual cause of this issue was
default parameters to rasterize_pdf_page in pluggy not working as
expected, causing a default rotation=0 even when a rotation was needed.
As such the OCR image was generated with the wrong orientation,
causing the initial regression and fix in commit 1327ab3.

Now that the real problem is identified, it's apparent that the logic
prior to 1327ab3 was found and we can revert to 1327ab3 since it fixes
all known cases including #658.

This reverts 1327ab3 except for retaining improves to rotation output.
2020-10-24 02:45:21 -07:00
James R. Barlow ca735278e0 setup: Version pluggy better 2020-10-24 02:35:41 -07:00
James R. Barlow b5ccbfdf25 Fix hookspec of rasterize_pdf_page to remove default parameters 2020-10-24 02:35:18 -07:00
James R. Barlow 8c35d6e6e4 Fix debug log messages being suppressed from child processes 2020-10-22 02:20:06 -07:00
James R. Barlow d1e0c81eda Ensure worker_pdf is closed after gathering info in a thread
This is hacky, uses global state, but it does improve the situation for now.
2020-10-22 00:38:24 -07:00
James R. Barlow 10c8e4f8b4 Only create debug.log when running from command line
When used as a library ocrmypdf shouldn't make policy decisions, like where to
put a log file. Unsurprisingly, creating it causes problems for library users
because we deleted the temporary folder which held the log file and made no
effort to move it to a new location.

Also update the documentation to better described how an application should
handle this.

Closes #657
2020-10-20 01:29:36 -07:00
James R. Barlow 6be2242c21 Describe "OCR" step as "Image processing" when --tesseract-timeout=0
Fixes #647
2020-10-08 01:03:42 -07:00
34 changed files with 389 additions and 221 deletions
+1 -1
View File
@@ -19,5 +19,5 @@ repos:
rev: 19.10b0
hooks:
- id: black
language_version: python3.8
language_version: python
exclude: ^src/ocrmypdf/lib/_leptonica.py
+3
View File
@@ -92,6 +92,9 @@ apt-get install tesseract-ocr-chi-sim # Example: Install Chinese Simplified lan
# Arch Linux users
pacman -S tesseract-data-eng tesseract-data-deu # Example: Install the English and German language packs
# brew macOS users
brew install tesseract-lang
```
You can then pass the `-l LANG` argument to OCRmyPDF to give a hint as to what languages it should search for. Multiple languages can be requested.
+5 -2
View File
@@ -22,6 +22,8 @@ stages:
python.version: "3.7"
Python38:
python.version: "3.8"
Python39:
python.version: "3.9"
steps:
- task: UsePythonVersion@0
inputs:
@@ -59,6 +61,8 @@ stages:
python.version: "3.7"
Python38:
python.version: "3.8"
Python39:
python.version: "3.9"
steps:
- task: UsePythonVersion@0
inputs:
@@ -156,9 +160,8 @@ stages:
# versionSpec: "$(python.version)"
- bash: |
brew update
brew unlink python@2
brew upgrade python
echo "Using Python `python3 --version`"
echo "Using `python3 --version`"
displayName: "Update brew and Python"
- bash: |
brew install \
+11 -10
View File
@@ -20,7 +20,8 @@ and largely have the same functions.
import ocrmypdf
ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
if __name__ == '__main__': # To ensure correct behavior on Windows
ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
With a few exceptions, all of the command line arguments are available
and may be passed as equivalent keywords.
@@ -35,8 +36,9 @@ The :func:`ocrmypdf.ocr` function runs OCRmyPDF similar to command line
execution. To do this, it will:
- create a monitoring thread
- create worker processes (forking itself)
- manage the signal flags of worker processes
- create worker processes (on Linux, forking itself; on Windows and macOS, by
spawning)
- manage the signal flags of its worker processes
- execute other subprocesses (forking and executing other programs)
The Python process that calls ``ocrmypdf.ocr()`` must be sufficiently
@@ -47,9 +49,9 @@ There is no currently no option to manage how jobs are scheduled other
than the argument ``jobs=`` which will limit the number of worker
processes.
Forking a child process to call ``ocrmypdf.ocr()`` is suggested. That
Creating a child process to call ``ocrmypdf.ocr()`` is suggested. That
way your application will survive and remain interactive even if
OCRmyPDF does not.
OCRmyPDF fails for any reason.
Programs that call ``ocrmypdf.ocr()`` should also install a SIGBUS signal
handler (except on Windows), to raise an exception if access to a memory
@@ -57,11 +59,10 @@ mapped file fails. OCRmyPDF may use memory mapping.
.. warning::
On Windows, the script that calls ``ocrmypdf.ocr()`` must be protected
by an "ifmain" guard (``if __name__ == '__main__'``) or you must use
``ocrmypdf.ocr(...use_threads=True)``. If you do not take at least one
of these steps, Windows process semantics will prevent OCRmyPDF from working
correctly.
On Windows and macOS, the script that calls ``ocrmypdf.ocr()`` must be
protected by an "ifmain" guard (``if __name__ == '__main__'``). If you do
not take at least one of these steps, process semantics will prevent
OCRmyPDF from working correctly.
Logging
-------
+1 -1
View File
@@ -127,7 +127,7 @@ Users may need to customize the script to meet their requirements.
"OCR_ON_SUCCESS_DELETE", "This will delete the input file if the exit code is 0 (OK)"
"OCR_OUTPUT_DIRECTORY_YEAR_MONTH", "This will place files in the output in ``{output}/{year}/{month}/{filename}``"
"OCR_DESKEW", "Apply deskew to crooked input PDFs"
"OCR_JSON_SETTINGS", "A JSON string specifying any other arguments for ``ocrmypdf.ocr``, e.g. ``'OCR_JSON_SETTINGS={"rotate_pages": true}'``.
"OCR_JSON_SETTINGS", "A JSON string specifying any other arguments for ``ocrmypdf.ocr``, e.g. ``'OCR_JSON_SETTINGS={""rotate_pages"": true}'``."
"OCR_POLL_NEW_FILE_SECONDS", "Polling interval"
"OCR_LOGLEVEL", "Level of log messages to report"
+2
View File
@@ -1,3 +1,5 @@
.. _docker:
=====================
OCRmyPDF Docker image
=====================
+1 -1
View File
@@ -1,7 +1,7 @@
OCRmyPDF documentation
======================
OCRmyPDF adds an optical charcter recognition (OCR) text layer to scanned PDF
OCRmyPDF adds an optical character recognition (OCR) text layer to scanned PDF
files, allowing them to be searched.
PDF is the best format for storing and exchanging scanned documents.
+2 -2
View File
@@ -604,7 +604,7 @@ However, the OCR-to-text-layer functionality is available.
Docker
------
You can also :ref:`Install the Docker <docker-install>` container on Windows. Ensure that
You can also :ref:`Install the Docker <docker>` container on Windows. Ensure that
your command prompt can run the docker "hello world" container.
Installing on FreeBSD
@@ -630,7 +630,7 @@ Installing the Docker image
For some users, installing the Docker image will be easier than
installing all of OCRmyPDF's dependencies.
See `OCRmyPDF Docker Image <docker>`__ for more information.
See :ref:`docker` for more information.
Installing with Python pip
==========================
+58
View File
@@ -12,6 +12,64 @@ may be unreliable. Use the API to depend on precise behavior.
The public API may be useful in scripts that launch OCRmyPDF processes or that
wish to use some of its features for working with PDFs.
v11.3.4
=======
- Fixed an error message 'called readLinearizationData for file that is not
linearized' that may occur when pikepdf 2.1.0 is used. (Upgrading to pikepdf
2.1.1 also fixes the issue.)
- File watcher now automatically includes ``.PDF`` in addition to ``.pdf`` to
better support case sensitive file systems.
- Some documentation and comment improvements.
v11.3.3
=======
- If unpaper outputs non-UTF-8 data, quietly fix this rather than choke on the
conversion. (Possibly addresses #671.)
v11.3.2
=======
- Explicitly require pikepdf 2.0.0 or newer when running on Python 3.9. (There are
concerns about the stability of pybind11 2.5.x with Python 3.9, which is used in
pikepdf 1.x.)
- Fixed another issue related to page rotation.
- Fixed an issue where image marked as image masks were not properly considered
as optimization candidates.
- On some systems, unpaper seems to be unable to process the PNGs we offer it
as input. We now convert the input to PNM format, which unpaper always accepts.
Fixes #665 and #667.
- DPI sent to unpaper is now rounded to a more reasonable number of decimal digits.
- Debug and error messages from unpaper were being suppressed.
- Some documentation tweaks.
v11.3.1
=======
- Declare support for new versions: pdfminer.six 20201018 and pikepdf 2.x
- Fix warning related to ``--pdfa-image-compression`` that appears at the wrong
time.
v11.3.0
=======
- The "OCR" step is describing as "Image processing" in the output messages when
OCR is disabled, to better explain the application's behavior.
- Debug logs are now only created when run as a command line, and not when OCR
is performed for an API call. It is the calling application's responsibility
to set up logging.
- For PDFs with a low number of pages, we gathered information about the input PDF
in a thread rather than process (when there are more pages). When run as a
thread, we did not close the file handle to the working PDF, leaking one file
handle per call of ``ocrmypdf.ocr``.
- Fixed an issue where debug messages send by child worker processes did not match
the log settings of parent process, causing messages to be dropped. This affected
macOS and Windows only where the parent process is not forked.
- Fixed the hookspec of rasterize_pdf_page to remove default parameters that
were not handled in an expected way by pluggy.
- Fixed another issue with automatic page rotation (#658) due to the issue above.
v11.2.1
=======
+1 -1
View File
@@ -45,7 +45,7 @@ OCR_JSON_SETTINGS = json.loads(os.getenv('OCR_JSON_SETTINGS', '{}'))
POLL_NEW_FILE_SECONDS = int(os.getenv('OCR_POLL_NEW_FILE_SECONDS', '1'))
USE_POLLING = bool(os.getenv('OCR_USE_POLLING', ''))
LOGLEVEL = os.getenv('OCR_LOGLEVEL', 'INFO').upper()
PATTERNS = ['*.pdf']
PATTERNS = ['*.pdf', '*.PDF']
log = logging.getLogger('ocrmypdf-watcher')
+7 -7
View File
@@ -1,12 +1,12 @@
# requirements.txt can be used to replicate the developer's build environment
# setup.py lists a separate set of requirements that are looser to simplify
# installation
cffi == 1.14.0
cffi == 1.14.3
coloredlogs == 14.0 # technically optional
img2pdf == 0.3.6
pdfminer.six == 20200517
pikepdf == 1.16.1
img2pdf == 0.4.0
pdfminer.six == 20201018
pikepdf == 2.0.0
pluggy == 0.13.1
Pillow == 7.1.2
reportlab == 3.5.42
tqdm == 4.46.1
Pillow == 8.0.1
reportlab == 3.5.55
tqdm == 4.51.0
+4 -4
View File
@@ -63,7 +63,6 @@ setup(
python_requires=' >= 3.6',
setup_requires=[ # can be removed whenever we can drop pip 9 support
'cffi >= 1.9.1', # to build the leptonica module
'pytest-runner', # to enable python setup.py test
'setuptools_scm', # so that version will work
'setuptools_scm_git_archive', # enable version from github tarballs
],
@@ -73,10 +72,11 @@ setup(
'cffi >= 1.9.1', # must be a setup and install requirement
'coloredlogs >= 14.0', # strictly optional
'img2pdf >= 0.3.0, < 0.5', # pure Python, so track HEAD closely
'pdfminer.six >= 20191110, != 20200720, <= 20200726',
'pikepdf >= 1.14.0, < 2',
'pdfminer.six >= 20191110, != 20200720, <= 20201018',
"pikepdf >= 1.14.0, < 3 ; python_version < '3.9'",
"pikepdf >= 2.0.0 ; python_version >= '3.9'",
'Pillow >= 7.0.0',
'pluggy >= 0.13.0',
'pluggy >= 0.13.0, < 1.0',
'reportlab >= 3.3.0', # oldest released version with sane image handling
'tqdm >= 4',
],
+4 -3
View File
@@ -49,7 +49,7 @@ def process_sigbus(*args):
raise InputFileError("A worker process lost access to an input file")
def process_init(queue, user_init):
def process_init(queue, user_init, loglevel):
"""Initialize a process pool worker"""
# Ignore SIGINT (our parent process will kill us gracefully)
@@ -62,6 +62,7 @@ def process_init(queue, user_init):
# Reconfigure the root logger for this process to send all messages to a queue
h = logging.handlers.QueueHandler(queue)
root = logging.getLogger()
root.setLevel(loglevel)
root.handlers = []
root.addHandler(h)
@@ -69,7 +70,7 @@ def process_init(queue, user_init):
user_init()
def thread_init(_queue, user_init):
def thread_init(_queue, user_init, _loglevel):
# As a thread, block SIGBUS so the main thread deals with it...
if hasattr(signal, 'SIGBUS'):
signal.pthread_sigmask(signal.SIG_BLOCK, {signal.SIGBUS})
@@ -102,7 +103,7 @@ def exec_progress_pool(
pool = pool_class(
processes=max_workers,
initializer=initializer,
initargs=(log_queue, task_initializer),
initargs=(log_queue, task_initializer, logging.getLogger("").level),
)
try:
results = pool.imap_unordered(task, task_arguments)
+20 -18
View File
@@ -25,24 +25,24 @@ from ocrmypdf.subprocess import get_version, run
log = logging.getLogger(__name__)
missing_gs_error = """
---------------------------------------------------------------------
This error normally occurs when ocrmypdf find can't Ghostscript.
Please ensure Ghostscript is installed and its location is added to
the system PATH environment variable.
For details see:
https://ocrmypdf.readthedocs.io/en/latest/installation.html
---------------------------------------------------------------------
"""
_gswin = None
if os.name == 'nt':
_gswin = which('gswin64c')
if not _gswin:
_gswin = which('gswin32c')
if not _gswin:
raise MissingDependencyError(
"""
---------------------------------------------------------------------
This error normally occurs when ocrmypdf can't Ghostscript. Please
ensure Ghostscript is installed and its location is added to the
system PATH environment variable.
For details see:
https://ocrmypdf.readthedocs.io/en/latest/installation.html
---------------------------------------------------------------------
"""
)
raise MissingDependencyError(missing_gs_error)
_gswin = Path(_gswin).stem
GS = _gswin if _gswin else 'gs'
@@ -81,8 +81,8 @@ def rasterize_pdf(
raster_device: str,
raster_dpi: Resolution,
pageno: int = 1,
page_dpi: Resolution = None,
rotation: int = None,
page_dpi: Optional[Resolution] = None,
rotation: Optional[int] = None,
filter_vector: bool = False,
):
"""Rasterize one page of a PDF at resolution raster_dpi in canvas units."""
@@ -122,8 +122,6 @@ def rasterize_pdf(
stderr = p.stderr.decode(errors='replace')
if _gs_error_reported(stderr):
log.error(stderr)
elif stderr:
log.debug(stderr)
with Image.open(BytesIO(p.stdout)) as im:
if rotation is not None:
@@ -148,6 +146,9 @@ def generate_pdfa(
pdf_version: str = '1.5',
pdfa_part: str = '2',
):
# Ghostscript's compression is all or nothing. We can either force all images
# to JPEG, force all to Flate/PNG, or let it decide how to encode the images.
# In most case it's best to let it decide.
compression_args = []
if compression == 'jpeg':
compression_args = [
@@ -175,8 +176,9 @@ def generate_pdfa(
strategy = 'RGB' if version() >= '9.19' else '/RGB'
if version() == '9.23':
# 9.23: new feature JPEG passthrough is broken in some cases, best to
# disable it always
# 9.23: added JPEG passthrough as a new feature, but with a bug that
# incorrectly formats some images. Fixed as of 9.24. So we disable this
# feature for 9.23.
# https://bugs.ghostscript.com/show_bug.cgi?id=699216
compression_args.append('-dPassThroughJPEGImages=false')
+1 -3
View File
@@ -99,9 +99,7 @@ def get_languages():
args_tess = ['tesseract', '--list-langs']
try:
proc = run(
args_tess, universal_newlines=True, stdout=PIPE, stderr=STDOUT, check=True
)
proc = run(args_tess, text=True, stdout=PIPE, stderr=STDOUT, check=True)
output = proc.stdout
except CalledProcessError as e:
raise MissingDependencyError(lang_error(e.output)) from e
+42 -38
View File
@@ -13,10 +13,11 @@
import logging
import os
import shlex
from decimal import Decimal
from pathlib import Path
from subprocess import PIPE, STDOUT, CalledProcessError
from subprocess import PIPE, STDOUT
from tempfile import TemporaryDirectory
from typing import Tuple
from typing import List, Optional, Tuple, Union
from PIL import Image
@@ -24,10 +25,12 @@ from ocrmypdf.exceptions import MissingDependencyError, SubprocessOutputError
from ocrmypdf.subprocess import get_version
from ocrmypdf.subprocess import run as external_run
DecFloat = Union[Decimal, float]
log = logging.getLogger(__name__)
def version():
def version() -> str:
return get_version('unpaper')
@@ -53,23 +56,25 @@ def _setup_unpaper_io(tmpdir: Path, input_file: Path) -> Tuple[Path, Path]:
except KeyError:
raise MissingDependencyError(
"Failed to convert image to a supported format."
) from e
) from None
if im_modified or input_file.suffix != '.png':
input_png = tmpdir / 'input.png'
im.save(input_png, format='PNG', compress_level=1)
if im_modified or input_file.suffix != '.pnm':
input_pnm = tmpdir / 'input.pnm'
im.save(input_pnm, format='PPM')
else:
# No changes, PNG input, just use the file we already have
input_png = input_file
input_pnm = input_file
output_pnm = tmpdir / f'output{suffix}'
return input_png, output_pnm
return input_pnm, output_pnm
def run(input_file, output_file, dpi, mode_args):
args_unpaper = ['unpaper', '-v', '--dpi', str(dpi)] + mode_args
def run(
input_file: Path, output_file: Path, dpi: DecFloat, mode_args: List[str]
) -> None:
args_unpaper = ['unpaper', '-v', '--dpi', str(round(dpi, 6))] + mode_args
with TemporaryDirectory() as tmpdir:
input_png, output_pnm = _setup_unpaper_io(Path(tmpdir), input_file)
input_pnm, output_pnm = _setup_unpaper_io(Path(tmpdir), input_file)
# To prevent any shenanigans from accepting arbitrary parameters in
# --unpaper-args, we:
@@ -78,41 +83,40 @@ def run(input_file, output_file, dpi, mode_args):
# 3) append absolute paths for the input and output file
# This should ensure that a user cannot clobber some other file with
# their unpaper arguments (whether intentionally or otherwise)
args_unpaper.extend([os.fspath(input_png), os.fspath(output_pnm)])
args_unpaper.extend([os.fspath(input_pnm), os.fspath(output_pnm)])
external_run(
args_unpaper,
close_fds=True,
check=True,
stderr=STDOUT, # unpaper writes logging output to stdout and stderr
stdout=PIPE, # and cannot send file output to stdout
cwd=tmpdir,
logs_errors_to_stdout=True,
)
try:
proc = external_run(
args_unpaper,
check=True,
close_fds=True,
universal_newlines=True,
stderr=STDOUT, # unpaper writes logging output to stdout and stderr
cwd=tmpdir, # and cannot send file output to stdout
stdout=PIPE,
)
except CalledProcessError as e:
log.debug(e.stderr)
raise e from e
else:
log.debug(proc.stderr)
try:
with Image.open(output_pnm) as imout:
imout.save(output_file, dpi=(dpi, dpi))
except (FileNotFoundError, OSError):
raise SubprocessOutputError(
"unpaper: failed to produce the expected output file. "
+ " Called with: "
+ str(args_unpaper)
) from None
with Image.open(output_pnm) as imout:
imout.save(output_file, dpi=(dpi, dpi))
except (FileNotFoundError, OSError):
raise SubprocessOutputError(
"unpaper: failed to produce the expected output file. "
+ " Called with: "
+ str(args_unpaper)
) from None
def validate_custom_args(args: str):
def validate_custom_args(args: str) -> List[str]:
unpaper_args = shlex.split(args)
if any(('/' in arg or arg == '.' or arg == '..') for arg in unpaper_args):
raise ValueError('No filenames allowed in --unpaper-args')
return unpaper_args
def clean(input_file, output_file, dpi, unpaper_args=None):
def clean(
input_file: Path,
output_file: Path,
dpi: DecFloat,
unpaper_args: Optional[List[str]] = None,
):
default_args = [
'--layout',
'none',
+15 -17
View File
@@ -109,6 +109,7 @@ class OcrGrafter:
if textpdf and not self.font:
self.font, self.font_key = self._find_font(textpdf)
emplaced_page = False
content_rotation = self.pdfinfo[pageno].rotation
path_image = Path(image).resolve() if image else None
if path_image is not None and path_image != self.path_base:
@@ -122,24 +123,21 @@ class OcrGrafter:
local_image_page = self.pdf_base.pages[-1]
self.pdf_base.pages[pageno].emplace(local_image_page)
del self.pdf_base.pages[-1]
# The pdf_image_page will always be created with any /Rotate applied
# applied already
content_rotation = 0
emplaced_page = True
if content_rotation != 0:
# Text can be misaligned on a /Rotate'd page.
# That is because we rasterize pages with /Rotate applied,
# so that the OCR image text is upright and comes back upright.
text_misaligned = (autorotate_correction - content_rotation) % 360
log.debug(
f"Text rotation: (autorotate, content) -> text misalignment = "
f"({autorotate_correction}, {content_rotation}) -> {text_misaligned}"
)
else:
text_misaligned = 0
# Calculate if the text is misaligned compared to the content
if emplaced_page:
content_rotation = autorotate_correction
text_rotation = autorotate_correction
text_misaligned = (text_rotation - content_rotation) % 360
log.debug(
f"Text rotation: (text, autorotate, content) -> text misalignment = "
f"({text_rotation}, {autorotate_correction}, {content_rotation}) -> {text_misaligned}"
)
if textpdf and self.font:
# Graft the text layer onto this page, whether new or old
# Graft the text layer onto this page, whether new or old, possibly
# rotating the text layer by the amount is misaligned.
strip_old = self.context.options.redo_ocr
self._graft_text_layer(
page_num=pageno + 1,
@@ -151,14 +149,14 @@ class OcrGrafter:
strip_old_text=strip_old,
)
# Correct the page rotation
# Correct the overall page rotation if needed, now that the text and content
# are aligned
page_rotation = (content_rotation - autorotate_correction) % 360
self.pdf_base.pages[pageno].Rotate = page_rotation
log.debug(
f"Page rotation: (content, auto) -> page = "
f"({content_rotation}, {autorotate_correction}) -> {page_rotation}"
)
if self.emplacements % MAX_REPLACE_PAGES == 0:
self.save_and_reload()
+4 -2
View File
@@ -332,8 +332,10 @@ def rasterize_preview(input_file: Path, page_context: PageContext):
output_file=output_file,
raster_device='jpeggray',
raster_dpi=canvas_dpi,
page_dpi=page_dpi,
pageno=page_context.pageinfo.pageno + 1,
page_dpi=page_dpi,
rotation=0,
filter_vector=False,
)
return output_file
@@ -433,7 +435,7 @@ def rasterize(
device = colorspaces[device_idx]
log.debug(f"Rasterize with {device}")
log.debug(f"Rasterize with {device}, rotation {correction}")
# Produce the page image with square resolution or else deskew and OCR
# will not work properly.
+1 -1
View File
@@ -82,7 +82,7 @@ def _setup_plugins(
pm.register(module)
def get_plugin_manager(plugins: List[str], builtins=True):
def get_plugin_manager(plugins: List[Union[str, Path]], builtins=True):
pm = OcrmypdfPluginManager(
project_name='ocrmypdf',
setup_func=partial(_setup_plugins, plugins=plugins, builtins=builtins),
+24 -17
View File
@@ -55,6 +55,7 @@ from ocrmypdf._validation import (
)
from ocrmypdf.exceptions import ExitCode, ExitCodeException
from ocrmypdf.helpers import (
NeverRaise,
available_cpu_count,
check_pdf,
pikepdf_enable_mmap,
@@ -209,9 +210,10 @@ def exec_page_sync(page_context: PageContext):
if options.pdf_renderer == 'hocr':
(hocr_out, text_out) = ocr_engine_hocr(ocr_image_out, page_context)
ocr_out = render_hocr_page(hocr_out, page_context)
if options.pdf_renderer == 'sandwich':
elif options.pdf_renderer == 'sandwich':
(ocr_out, text_out) = ocr_engine_textonly_pdf(ocr_image_out, page_context)
else:
raise NotImplementedError(f"pdf_renderer {options.pdf_renderer}")
return PageResult(
pageno=page_context.pageno,
@@ -244,7 +246,8 @@ def exec_concurrent(context: PdfContext):
"""Execute the pipeline concurrently"""
# Run exec_page_sync on every page context
max_workers = min(len(context.pdfinfo), context.options.jobs)
options = context.options
max_workers = min(len(context.pdfinfo), options.jobs)
if max_workers > 1:
log.info("Start processing %d pages concurrently", max_workers)
@@ -267,14 +270,14 @@ def exec_concurrent(context: PdfContext):
tls.pageno = None
exec_progress_pool(
use_threads=context.options.use_threads,
use_threads=options.use_threads,
max_workers=max_workers,
tqdm_kwargs=dict(
total=(2 * len(context.pdfinfo)),
desc='OCR',
desc='OCR' if options.tesseract_timeout > 0 else 'Image processing',
unit='page',
unit_scale=0.5,
disable=not context.options.progress_bar,
disable=not options.progress_bar,
),
task_initializer=partial(worker_init, PIL.Image.MAX_IMAGE_PIXELS),
task=exec_page_sync,
@@ -283,10 +286,10 @@ def exec_concurrent(context: PdfContext):
)
# Output sidecar text
if context.options.sidecar:
if options.sidecar:
text = merge_sidecars(sidecars, context)
# Copy text file to destination
copy_final(text, context.options.sidecar, context)
copy_final(text, options.sidecar, context)
# Merge layers to one single pdf
pdf = ocrgraft.finalize()
@@ -296,16 +299,17 @@ def exec_concurrent(context: PdfContext):
pdf = post_process(pdf, context)
# Copy PDF file to destination
copy_final(pdf, context.options.output_file, context)
copy_final(pdf, options.output_file, context)
class NeverRaise(Exception):
"""An exception that is never raised"""
def configure_debug_logging(log_filename, prefix: str = ''):
"""
Create a debug log file at a specified location.
pass # pylint: disable=unnecessary-pass
def configure_debug_logging(log_filename, prefix=''):
Arguments:
log_filename: Where to the put the log file.
prefix: The logging domain prefix that should be sent to the log.
"""
log_file_handler = logging.FileHandler(log_filename, delay=True)
log_file_handler.setLevel(logging.DEBUG)
formatter = logging.Formatter(
@@ -328,9 +332,12 @@ def run_pipeline(options, *, plugin_manager, api=False):
work_folder = Path(mkdtemp(prefix="com.github.ocrmypdf."))
debug_log_handler = None
if (options.keep_temporary_files or options.verbose >= 1) and not os.environ.get(
'PYTEST_CURRENT_TEST', ''
if (
(options.keep_temporary_files or options.verbose >= 1)
and not os.environ.get('PYTEST_CURRENT_TEST', '')
and not api
):
# Debug log for command line interface only with verbose output
debug_log_handler = configure_debug_logging(Path(work_folder) / "debug.log")
pikepdf_enable_mmap()
+3 -3
View File
@@ -213,12 +213,12 @@ def check_options_optimizing(options):
def check_options_advanced(options):
if options.pdfa_image_compression != 'auto' and options.output_type.startswith(
if options.pdfa_image_compression != 'auto' and not options.output_type.startswith(
'pdfa'
):
log.warning(
"--pdfa-image-compression argument has no effect when "
"--output-type is not 'pdfa', 'pdfa-1', or 'pdfa-2'"
"--pdfa-image-compression argument only applies when "
"--output-type is one of 'pdfa', 'pdfa-1', or 'pdfa-2'"
)
+39 -22
View File
@@ -10,7 +10,8 @@ import os
import sys
from enum import IntEnum
from pathlib import Path
from typing import BinaryIO, Iterable, Union
from typing import AnyStr, BinaryIO, Iterable, Optional, Union
from warnings import warn
from ocrmypdf._logging import PageNumberFilter, TqdmConsole
from ocrmypdf._plugin_manager import get_plugin_manager
@@ -25,7 +26,8 @@ except ModuleNotFoundError:
coloredlogs = None
PathOrIO = Union[BinaryIO, os.PathLike, str, bytes]
StrPath = Union[os.PathLike, AnyStr]
PathOrIO = Union[BinaryIO, StrPath]
class Verbosity(IntEnum):
@@ -44,16 +46,28 @@ def configure_logging(
):
"""Set up logging.
Library users may wish to use this function if they want their log output to be
similar to ocrmypdf command line interface. If not used, the external application
should configure logging on its own.
Before calling :func:`ocrmypdf.ocr()`, you can use this function to
configure logging, if you want ocrmypdf's output to look like the ocrmypdf
command line interface. It will register log handlers, log filters, and
formatters, configure color logging to standard error, and adjust the log
levels of third party libraries. Details of this are fine-tuned and subject
to change. The ``verbosity`` argument is equivalent to the argument
``--verbose`` and applies those settings.
ocrmypdf will perform all of its logging under the ``"ocrmypdf"`` logging namespace.
In addition, ocrmypdf imports pdfminer, which logs under ``"pdfminer"``. A library
user may wish to configure both; note that pdfminer is extremely chatty at the log
level ``logging.INFO``.
If this function is not called, ocrmypdf will not configure logging, and it
is up to the caller of ``ocrmypdf.ocr()`` to set up logging as it wishes using
the Python standard library's logging module. If this function is called,
the caller may of course make further adjustments to logging.
Library users may perform additional configuration afterwards.
Regardless of whether this function is called, ocrmypdf will perform all of
its logging under the ``"ocrmypdf"`` logging namespace. In addition,
ocrmypdf imports pdfminer, which logs under ``"pdfminer"``. A library user
may wish to configure both; note that pdfminer is extremely chatty at the
log level ``logging.INFO``.
This function does not set up the ``debug.log`` log file that the command
line interface does at certain verbosity levels. Applications should configure
their own debug logging.
Args:
verbosity (Verbosity): Verbosity level.
@@ -189,7 +203,7 @@ def ocr( # pylint: disable=unused-argument
language: Iterable[str] = None,
image_dpi: int = None,
output_type=None,
sidecar: os.PathLike = None,
sidecar: Optional[StrPath] = None,
jobs: int = None,
use_threads: bool = None,
title: str = None,
@@ -226,7 +240,7 @@ def ocr( # pylint: disable=unused-argument
user_words: os.PathLike = None,
user_patterns: os.PathLike = None,
fast_web_view: float = None,
plugins: Iterable[Union[str, Path]] = None,
plugins: Iterable[StrPath] = None,
keep_temporary_files: bool = None,
progress_bar: bool = None,
**kwargs,
@@ -248,7 +262,7 @@ def ocr( # pylint: disable=unused-argument
read.
output_file: If a :class:`pathlib.Path`, ``str`` or ``bytes``, this is
interpreted as file system path to the output file. If the object
appears to be a writable stream (with methods such as ``.read()`` and
appears to be a writable stream (with methods such as ``.write()`` and
``.seek()``), the output will be written to this stream. If
``output_file`` is ``"-"``, the output will be written to ``sys.stdout``
(provided that standard output does not seem to be a terminal device).
@@ -285,15 +299,18 @@ def ocr( # pylint: disable=unused-argument
else:
plugins = list(plugins)
parser = get_parser()
_plugin_manager = get_plugin_manager(plugins)
_plugin_manager.hook.add_options(parser=parser) # pylint: disable=no-member
create_options_kwargs = {
k: v for k, v in locals().items() if not k.startswith('_') and k != 'kwargs'
}
# No new variable names should be assigned until these two steps are run
create_options_kwargs = {k: v for k, v in locals().items() if k != 'kwargs'}
create_options_kwargs.update(kwargs)
parser = get_parser()
create_options_kwargs['parser'] = parser
plugin_manager = get_plugin_manager(plugins)
plugin_manager.hook.add_options(parser=parser) # pylint: disable=no-member
if 'verbose' in kwargs:
warn("ocrmypdf.ocr(verbose=) is ignored. Use ocrmypdf.configure_logging().")
options = create_options(**create_options_kwargs)
check_options(options, _plugin_manager)
return run_pipeline(options=options, plugin_manager=_plugin_manager, api=True)
check_options(options, plugin_manager)
return run_pipeline(options=options, plugin_manager=plugin_manager, api=True)
+3 -3
View File
@@ -61,9 +61,9 @@ def rasterize_pdf_page(
raster_device,
raster_dpi,
pageno,
page_dpi=None,
rotation=None,
filter_vector=False,
page_dpi,
rotation,
filter_vector,
):
ghostscript.rasterize_pdf(
input_file,
@@ -121,9 +121,7 @@ def validate(pdfinfo, options):
os.environ['OMP_THREAD_LIMIT'] = str(tess_threads)
else:
tess_threads = int(os.environ['OMP_THREAD_LIMIT'])
if tess_threads > 1:
log.info("Using Tesseract OpenMP thread limit %d", tess_threads)
log.debug("Using Tesseract OpenMP thread limit %d", tess_threads)
class TesseractOcrEngine(OcrEngine):
+17 -5
View File
@@ -58,6 +58,10 @@ class Resolution(namedtuple('Resolution', ('x', 'y'))):
return f"Resolution({self.x}x{self.y} dpi)"
class NeverRaise(Exception):
"""An exception that is never raised"""
def safe_symlink(input_file: os.PathLike, soft_link_name: os.PathLike):
"""
Helper function: relinks soft symbolic link if necessary
@@ -186,17 +190,25 @@ def check_pdf(input_file: Path) -> bool:
log.warning(msg)
sio = StringIO()
linearize = None
linearize_msgs = ''
try:
# If linearization is missing entirely, we do not complain. We do
# complain if linearization is present but incorrect.
pdf.check_linearization(sio)
except RuntimeError:
pass
except (
getattr(pikepdf, 'ForeignObjectError')
if pikepdf.__version__ == '2.1.0' # This version may throw wrong exception
else NeverRaise
):
pass
else:
linearize = sio.getvalue()
if linearize:
log.warning(linearize)
linearize_msgs = sio.getvalue()
if linearize_msgs:
log.warning(linearize_msgs)
if not messages and not linearize:
if not messages and not linearize_msgs:
return True
return False
finally:
+24 -20
View File
@@ -115,25 +115,29 @@ def extract_image_jbig2(
and filtdp[0] != Name.JBIG2Decode
and jbig2enc.available()
):
try:
# Save any colorspace associated with the image, so that we
# will export a pure 1-bit PNG with no palette or ICC profile.
# Showing the palette or ICC to jbig2enc will cause it to perform
# colorspace transform to 1bpp, which will conflict the palette or
# ICC if it exists.
colorspace = pim.obj.ColorSpace
# Set to DeviceGray temporarily; we already in 1 bpc.
pim.obj.ColorSpace = pikepdf.Name.DeviceGray
imgname = root / f'{xref:08d}'
with imgname.open('wb') as f:
ext = pim.extract_to(stream=f)
imgname.rename(imgname.with_suffix(ext))
except pikepdf.UnsupportedImageTypeError:
return None
finally:
# Restore image colorspace after temporarily setting it to DeviceGray
pim.obj.ColorSpace = colorspace
return XrefExt(xref, ext)
# Save any colorspace associated with the image, so that we
# will export a pure 1-bit PNG with no palette or ICC profile.
# Showing the palette or ICC to jbig2enc will cause it to perform
# colorspace transform to 1bpp, which will conflict the palette or
# ICC if it exists.
colorspace = pim.obj.get(pikepdf.Name.ColorSpace, None)
if colorspace is not None or pim.image_mask:
try:
# Set to DeviceGray temporarily; we already in 1 bpc.
pim.obj.ColorSpace = pikepdf.Name.DeviceGray
imgname = root / f'{xref:08d}'
with imgname.open('wb') as f:
ext = pim.extract_to(stream=f)
imgname.rename(imgname.with_suffix(ext))
except pikepdf.UnsupportedImageTypeError:
return None
finally:
# Restore image colorspace after temporarily setting it to DeviceGray
if colorspace is not None:
pim.obj.ColorSpace = colorspace
else:
del pim.obj.ColorSpace
return XrefExt(xref, ext)
return None
@@ -613,7 +617,7 @@ def optimize(input_file: Path, output_file: Path, context, save_settings) -> Non
)
ratio = input_size / output_size
savings = 1 - output_size / input_size
log.info(f"Optimize ratio: {ratio:.2f} savings: {(100 * savings):.1f}%")
log.info(f"Optimize ratio: {ratio:.2f} savings: {(savings):.1%}")
if savings < 0:
log.info("Image optimization did not improve the file - discarded")
+25 -15
View File
@@ -630,6 +630,9 @@ worker_pdf = None
def _pdf_pageinfo_sync_init(infile):
global worker_pdf # pylint: disable=global-statement
pikepdf_enable_mmap()
# If this function is called as a thread initializer, we need a messy hack
# to close worker_pdf. If called as a process, it will be released when the
# process is terminated.
worker_pdf = pikepdf.open(infile)
@@ -643,6 +646,7 @@ def _pdf_pageinfo_sync(args):
def _pdf_pageinfo_concurrent(
pdf, infile, progbar, max_workers, check_pages, detailed_analysis=False
):
global worker_pdf # pylint: disable=global-statement
pages = [None] * len(pdf.pages)
def update_pageinfo(result, pbar):
@@ -663,17 +667,23 @@ def _pdf_pageinfo_concurrent(
# a separate process.
use_threads = True
exec_progress_pool(
use_threads=use_threads,
max_workers=n_workers,
tqdm_kwargs=dict(
total=total, desc="Scanning contents", unit='page', disable=not progbar
),
task_initializer=partial(_pdf_pageinfo_sync_init, infile),
task=_pdf_pageinfo_sync,
task_arguments=contexts,
task_finished=update_pageinfo,
)
try:
exec_progress_pool(
use_threads=use_threads,
max_workers=n_workers,
tqdm_kwargs=dict(
total=total, desc="Scanning contents", unit='page', disable=not progbar
),
task_initializer=partial(_pdf_pageinfo_sync_init, infile),
task=_pdf_pageinfo_sync,
task_arguments=contexts,
task_finished=update_pageinfo,
)
finally:
if worker_pdf and use_threads:
assert n_workers == 1, "Should have only one worker when threaded"
# This is messy, but if we ran in thread, close worker_pdf
worker_pdf.close()
return pages
@@ -808,12 +818,12 @@ class PdfInfo:
check_pages=check_pages,
detailed_analysis=detailed_analysis,
)
self._needs_rendering = pdf.root.get('/NeedsRendering', False)
self._needs_rendering = pdf.Root.get('/NeedsRendering', False)
self._has_acroform = False
if '/AcroForm' in pdf.root:
if len(pdf.root.AcroForm.get('/Fields', [])) > 0:
if '/AcroForm' in pdf.Root:
if len(pdf.Root.AcroForm.get('/Fields', [])) > 0:
self._has_acroform = True
elif '/XFA' in pdf.root.AcroForm:
elif '/XFA' in pdf.Root.AcroForm:
self._has_acroform = True
@property
+3 -3
View File
@@ -89,9 +89,9 @@ def rasterize_pdf_page(
raster_device: str,
raster_dpi: Resolution,
pageno: int,
page_dpi: Optional[Resolution] = None,
rotation: Optional[int] = None,
filter_vector: bool = False,
page_dpi: Optional[Resolution],
rotation: Optional[int],
filter_vector: bool,
) -> Path:
"""Rasterize one page of a PDF at resolution raster_dpi in canvas units.
+27 -10
View File
@@ -25,13 +25,21 @@ from ocrmypdf.exceptions import MissingDependencyError
log = logging.getLogger(__name__)
def run(args, *, env=None, **kwargs):
def run(args, *, env=None, logs_errors_to_stdout=False, **kwargs):
"""Wrapper around :py:func:`subprocess.run`
The main purpose of this wrapper is to log subprocess output in an orderly
fashion that indentifies the responsible subprocess. An additional
task is that this function goes to greater lengths to find possible Windows
locations of our dependencies when they are not on the system PATH.
Arguments should be identical to ``subprocess.run``, except for following:
Arguments:
logs_errors_to_stdout: If True, indicates that the process writes its error
messages to stdout rather than stderr, so stdout should be logged
if there is an error. If False, stderr is logged. Could be used with
stderr=STDOUT, stdout=PIPE for example.
"""
if not env:
env = os.environ
@@ -43,25 +51,34 @@ def run(args, *, env=None, **kwargs):
args = _fix_windows_args(program, args, env)
log.debug("Running: %s", args)
process_log = log.getChild('subprocess.' + os.path.basename(program))
if sys.version_info < (3, 7) and os.name == 'nt':
# Can't use close_fds=True on Windows with Python 3.6 or older
# https://bugs.python.org/issue19575, etc.
kwargs['close_fds'] = False
process_log = log.getChild(os.path.basename(program))
if sys.version_info < (3, 7):
if os.name == 'nt':
# Can't use close_fds=True on Windows with Python 3.6 or older
# https://bugs.python.org/issue19575, etc.
kwargs['close_fds'] = False
if 'text' in kwargs:
# Convert run(...text=) to run(...universal_newlines=) for Python 3.6
kwargs['universal_newlines'] = kwargs['text']
del kwargs['text']
stderr = None
stderr_name = 'stderr' if not logs_errors_to_stdout else 'stdout'
try:
proc = subprocess_run(args, env=env, **kwargs)
except CalledProcessError as e:
stderr = getattr(e, 'stderr', None)
stderr = getattr(e, stderr_name, None)
raise
else:
stderr = getattr(proc, 'stderr', None)
stderr = getattr(proc, stderr_name, None)
finally:
if process_log.isEnabledFor(logging.DEBUG) and stderr:
with suppress(AttributeError, UnicodeDecodeError):
stderr = stderr.decode('utf-8', 'replace')
process_log.debug("stderr = %s", stderr)
if logs_errors_to_stdout:
process_log.debug("stdout/stderr = %s", stderr)
else:
process_log.debug("stderr = %s", stderr)
return proc
@@ -107,7 +124,7 @@ def get_version(
proc = run(
args_prog,
close_fds=True,
universal_newlines=True,
text=True,
stdout=PIPE,
stderr=STDOUT,
check=True,
+2 -2
View File
@@ -128,7 +128,7 @@ def run_ocrmypdf_api(input_file, output_file, *args):
@pytest.helpers.register
def run_ocrmypdf(input_file, output_file, *args, universal_newlines=True):
def run_ocrmypdf(input_file, output_file, *args, text=True):
"Run ocrmypdf and let caller deal with results"
p_args = (
@@ -151,7 +151,7 @@ def run_ocrmypdf(input_file, output_file, *args, universal_newlines=True):
p_args,
stdout=PIPE,
stderr=PIPE,
universal_newlines=universal_newlines,
universal_newlines=text, # When dropping support for Python 3.6 change to text=
env=env,
check=False,
)
+2 -2
View File
@@ -598,7 +598,7 @@ def test_compression_preserved(ocrmypdf_exec, resources, image, outpdf):
stdout=PIPE,
stderr=PIPE,
stdin=input_stream,
universal_newlines=True,
universal_newlines=True, # When dropping support for Python 3.6 change to text=
check=False,
)
@@ -659,7 +659,7 @@ def test_compression_changed(ocrmypdf_exec, resources, image, compression, outpd
stdout=PIPE,
stderr=PIPE,
stdin=input_stream,
universal_newlines=True,
universal_newlines=True, # When dropping support for Python 3.6 change to text=
check=False,
)
assert p.returncode == ExitCode.ok, p.stderr
+2 -2
View File
@@ -302,8 +302,8 @@ def test_kodak_toc(resources, outpdf):
p = pikepdf.open(outpdf)
if pikepdf.Name.First in p.root.Outlines:
assert isinstance(p.root.Outlines.First, pikepdf.Dictionary)
if pikepdf.Name.First in p.Root.Outlines:
assert isinstance(p.Root.Outlines.First, pikepdf.Dictionary)
def test_metadata_fixup_warning(resources, outdir, caplog):
+32 -1
View File
@@ -15,6 +15,7 @@ from PIL import Image
from ocrmypdf import leptonica
from ocrmypdf._exec import ghostscript, tesseract
from ocrmypdf._plugin_manager import get_plugin_manager
from ocrmypdf.helpers import Resolution
from ocrmypdf.pdfinfo import PdfInfo
@@ -240,7 +241,7 @@ def test_rotate_page_level(image_angle, page_angle, resources, outdir):
'--rotate-pages',
'--rotate-pages-threshold',
'0.001',
universal_newlines=False,
text=False,
)
err = err.decode('utf-8', errors='replace')
assert p.returncode == 0, err
@@ -256,3 +257,33 @@ def test_tesseract_orientation(resources, tmp_path):
tesseract.get_orientation( # Test results of this are unreliable
tmp_path / '000001.png', engine_mode='3', timeout=10
)
def test_rasterize_rotates(resources, tmp_path):
pm = get_plugin_manager([])
img = tmp_path / 'img90.png'
pm.hook.rasterize_pdf_page(
input_file=resources / 'graph.pdf',
output_file=img,
raster_device='pngmono',
raster_dpi=Resolution(20, 20),
page_dpi=Resolution(20, 20),
pageno=1,
rotation=90,
filter_vector=False,
)
assert Image.open(img).size == (123, 151), "Image not rotated"
img = tmp_path / 'img180.png'
pm.hook.rasterize_pdf_page(
input_file=resources / 'graph.pdf',
output_file=img,
raster_device='pngmono',
raster_dpi=Resolution(20, 20),
page_dpi=Resolution(20, 20),
pageno=1,
rotation=180,
filter_vector=False,
)
assert Image.open(img).size == (151, 123), "Image not rotated"
+2 -2
View File
@@ -137,9 +137,9 @@ def test_report_file_size(tmp_path, caplog):
caplog.clear()
waste_of_space = b'Dummy' * 5000
pdf.root.Dummy = waste_of_space
pdf.Root.Dummy = waste_of_space
pdf.save(in_)
pdf.root.Dummy2 = waste_of_space + waste_of_space
pdf.Root.Dummy2 = waste_of_space + waste_of_space
pdf.save(out)
with patch('ocrmypdf._validation.jbig2enc.available', return_value=True), patch(