Compare commits

...
3470 Commits
Author SHA1 Message Date
James R. Barlow 944d99bdc1 Fix PROGRAM_NAME 2024-08-31 01:23:55 -07:00
James R. Barlow 5bb6e1c5d7 Modify GitHub release strategy to use sigstore 2024-08-31 01:17:51 -07:00
James R. Barlow 8d7a8f0f98 Update documentation to transition from setuptools to hatchling 2024-08-31 01:14:51 -07:00
James R. Barlow b9dd0a5e3c Use hatchling and hatch-vcs as build backend 2024-08-31 01:08:23 -07:00
James R. Barlow 6949ad2c5d pyproject: link changelog 2024-08-31 00:45:46 -07:00
James R. Barlow b3324c3b4e Don't assume /Mask is always a Stream
Fixes #1377
2024-08-31 00:37:52 -07:00
James R. Barlow b38cac6931 Drop wheel from build requires
Not usually needed anymore
2024-08-28 01:31:56 -07:00
Elliott Sales de AndradeandGitHub bb4c47e707 Fix broken test_rotate_page_level (#1382)
Before 42ff7fc842, `make_rotate_test`
always used `resources / 'typewriter.png'`, but after the change the
second call accidentally used just `resources`, which is a directory,
and fails to open.
2024-08-21 01:25:07 -07:00
James R. Barlow 5e1e2497ab Enable Python 3.13 experimental 2024-08-15 01:32:33 -07:00
James R. Barlow cd910fbf21 Improve "PDF/A conversion failed" message 2024-08-10 01:34:02 -07:00
James R. Barlow 1225269a4b Clarify opporutnities available with OCR_JSON_SETTINGS 2024-08-10 01:02:05 -07:00
James R. Barlow 3a75b20740 v16.4.3 release notes 2024-07-31 02:14:12 -07:00
James R. Barlow d35d008806 Increase pdfminer's bufsiz to mitigate token splitting issue
Fixes #1361
2024-07-31 02:11:47 -07:00
James R. Barlow f5662d5eb0 Consider Masks and stencil masks when calculating DPI
Fixes #1362
2024-07-29 15:46:30 -07:00
James R. Barlow 39010dd255 Handle incompatible jbig2.exe from TeX Live
Fixes #1363
2024-07-27 01:09:34 -07:00
James R. Barlow fbaad570c7 v16.4.2 release notes 2024-07-22 15:02:53 -07:00
James R. Barlow f974e3b3c1 ghostscript: change input filename order for 10.03.1
Ghostscript now expects the pdfa.ps file to precede other files. Fixes #1359.
2024-07-22 14:56:03 -07:00
James R. Barlow 46b49cc176 Suppress missing jbig2dec warning message
Windows users can't resolve it easily.
2024-07-22 14:54:40 -07:00
Johannes KalliauerandGitHub 5256e74d0c Update installation.rst "python -m venv .venv" (#1355) 2024-07-18 06:28:07 -07:00
James R. Barlow 621d6a0b89 Fix image size calculation when SMask dimensions do not match image
Closes [Bug]: Ghostscript rasterizing failed #1351
2024-07-16 13:36:48 -07:00
IrisandGitHub 08be7c8bbe update arch base-devel install command (#1354)
the '--needed' flag only installs the package if it isn't installed, otherwise it would reinstall it if already installed.
2024-07-15 13:28:36 -07:00
James R. Barlow 980a5472b6 Fix test failures due to 4dde378 2024-07-09 15:54:53 -07:00
James R. Barlow 51c618e357 Merge branch 'main' of github.com:ocrmypdf/OCRmyPDF 2024-07-09 14:46:58 -07:00
James R. Barlow 4dde3786c2 Fix KeyError '/Subtype'
Fixes #1350
2024-07-09 14:46:47 -07:00
James Barlow d544342602 Merge branch 'main' of https://github.com/ocrmypdf/OCRmyPDF 2024-07-04 22:59:33 -07:00
James R. Barlow fac91fca2a v16.4.1 release notes 2024-06-30 00:11:27 -07:00
James R. Barlow 6edf756849 optimize: trap Hifi..Error
Fixes [Bug]: NotImplementedError: not sure how to get colorspace #1315
2024-06-30 00:08:51 -07:00
James R. Barlow 4fb1bb4de6 pipeline: fix typo in message 2024-06-30 00:08:31 -07:00
James Barlow 6a8eb7daaa docs: page seg mode 2024-06-26 01:16:31 -07:00
James R. Barlow 0544d06c3d Fix calculation of image printed area (used in finding weighted DPI for OCR)
Fixes #1334
2024-06-21 15:13:51 -07:00
James R. Barlow 34c285c9ac v16.4.0 release notes (3) 2024-06-17 14:40:22 -07:00
James R. Barlow 2f53b27651 Disable progbar for linearizing when --no-progress-bar set
Fixes #1332
2024-06-14 14:18:33 -07:00
James R. Barlow 772677746b Update issue templates to improve data collection for 3rd party apps 2024-06-13 15:25:50 -07:00
James R. Barlow f0bad87ea6 Restore choco since winget isn't supported (still) 2024-06-13 00:40:05 -07:00
James R. Barlow 44e71f8c14 Attempt to deal with jbig2dec warnings 2024-06-13 00:27:33 -07:00
James R. Barlow 964b30ca26 v16.4.0 release notes (2) 2024-06-11 16:55:33 -07:00
James R. Barlow 214a333e2d Block Tesseract 5.4.0 2024-06-11 14:44:54 -07:00
James R. Barlow ec6401ab57 Merge branch 'pr/helkaluin/1300' 2024-06-09 15:34:19 -07:00
James R. Barlow cbc5e8ce8d Revert "Delete and de-list snap because it no longer works"
This reverts commit 3a721e6578.
2024-06-09 15:32:34 -07:00
James R. Barlow a1c4cfe8f1 Merge branch 'main' of github.com:ocrmypdf/OCRmyPDF 2024-06-08 01:37:19 -07:00
James R. Barlow 3a721e6578 Delete and de-list snap because it no longer works 2024-06-08 01:24:47 -07:00
Omid RahaandGitHub e6b716cdde Update docker.rst (#1327) 2024-06-08 01:20:51 -07:00
James R. Barlow 02c39998b8 Note that alpine is now available for arm 2024-06-08 01:20:25 -07:00
James R. Barlow 0774bc7f14 v16.4.0 release notes 2024-06-01 02:01:11 -07:00
James R. Barlow c6a98b3d0b Merge branch 'feature/alpine-arm' 2024-06-01 01:55:26 -07:00
James R. Barlow 981bbf1105 optimize: add a recursion guard to avoid chasing cyclic form xobjects 2024-06-01 01:52:21 -07:00
James R. Barlow 2b0c6cfd40 v16.4.0 release ntoes 2024-06-01 00:26:07 -07:00
James R. Barlow 59f6bc8306 More Tesseract-specific language checks to its plugin 2024-06-01 00:15:50 -07:00
James R. Barlow 653c4ffb45 hocr: accept multiple spaces in bounding boxes
Fixes #1322
2024-05-31 16:22:27 -07:00
James R. Barlow d947ca258e Prevent issuing equ and osd as languages 2024-05-25 01:17:57 -07:00
James R. Barlow d5ff7f7db9 batch: fix issues flagged by ruff 2024-05-21 01:52:57 -07:00
James R. Barlow 579cef3649 watcher: Ensure output files are .pdf 2024-05-21 01:51:30 -07:00
James R. Barlow cb2f090c60 v16.3.1 release notes 2024-05-21 01:39:30 -07:00
James R. Barlow f3d6387bca Fix "OCR" progress bar not matching actual progress 2024-05-21 01:35:14 -07:00
James R. Barlow abf9729c61 Semfree test: accept pdfa conversion failed as a valid return code
Fixes #1316
2024-05-21 01:26:11 -07:00
James R. Barlow 442e9c9f0d Add missing codecov token where missed & drop unneeded brew openssl 2024-05-19 01:07:38 -07:00
James R. Barlow 397fad249d v16.3.0 release notes 2024-05-19 00:50:59 -07:00
James R. Barlow 9a3c5a3f7c Add progressbar for metadata_fixup
Might take time for big files. Pdf.open() potentially is expensive as well, but QPDF doesn't give us progress feedback for that.

Closes Show progress during postprocessing #1313
2024-05-19 00:46:50 -07:00
James R. Barlow 950c700274 Fix Ghostscript PDF/A progressbar not displaying 2024-05-19 00:44:21 -07:00
James R. Barlow 26432c38a9 Raise exception if rotate pages threshold adjusted without --rotate-pages
Fixes Make usage of --rotate-pages-threshold clearer #1309
2024-05-18 23:49:27 -07:00
James R. Barlow 28be50136c hocr: If a line box's coords are invalid, log and error and don't render
Addresses [Bug]: Crash on multiple .pdf files #1312

Not actually a fix, but at least it will get us better diagnostics. Appears old Tesseract 4.x generates bad line boxes at times.
2024-05-18 23:32:18 -07:00
James R. Barlow 0c62f2de5d Issue template: check for EOL OS 2024-05-17 19:51:15 -07:00
James R. Barlow 5caf654f22 Add new codecov token 2024-05-11 01:03:41 -07:00
James R. Barlow 205593445e Change test to run on macos x64 and arm64 2024-05-11 00:13:08 -07:00
James R. Barlow f25fb8c63a Merge branch 'main' of github.com:ocrmypdf/OCRmyPDF 2024-05-08 00:39:27 -07:00
James R. Barlow 99c78650b6 Add better error message for PDFs with invalid CTMs
Closes #1303
2024-05-07 14:00:30 -07:00
helkaluin 69355886a8 Fix wrong env var for GS path in Snap 2024-04-26 16:45:04 +08:00
Ahmed AbdouandGitHub 08e89e2dbe Adding language install docs for archlinux (#1296)
Adding language install docs for archlinux
2024-04-24 14:46:05 -07:00
James R. Barlow 0e013df161 v16.2.0 release notes 2024-04-16 00:37:03 -07:00
James R. Barlow 9ba4e3ab46 Log unusual exceptions when trying to obtain a version
Fixes #1262
2024-04-07 14:39:08 -07:00
James R. Barlow 5fdcb7602b Make downsampling large images that Tesseract would otherwise error on into default behavior
Fixes #1281
2024-04-07 13:43:20 -07:00
James R. Barlow b4db1b741f optimize: fix handling of [/FlateDecode none] - type images
Closes #1271
2024-04-07 01:44:08 -07:00
James R. Barlow 7a8cc21e31 Add support for sidecar output to io.BytesIO
Closes #1252
2024-04-07 01:38:55 -07:00
James R. Barlow 0674829d8f Remove tool.black config 2024-04-07 00:36:52 -07:00
James R. Barlow 315aa0474b Merge branch 'main' of github.com:ocrmypdf/OCRmyPDF 2024-04-07 00:34:51 -07:00
Ben BeasleyandGitHub df3451e779 Update the typer[all] dependency to typer-slim[standard] (#1287)
In 0.12.1, Typer was significantly reorganized.

- `typer-slim` is the library (for `import typer`)
- `typer-slim[standard]` adds optional dependencies (currently `rich`
  and `shellingham`, basically equivalent to the old `typer[all]`)
- `typer-cli` is the `typer` command-line tool
- `typer` is now basically a metapackage that brings in *all of the
  above*, and it no longer has an `all` extra

Pip will warn about this and proceed,

```
WARNING: typer 0.12.1 does not provide the extra 'all'
```

but there are other tools that will fail hard when asked to resolve a
(now) nonexistent extra.

Since this project doesn’t need the `typer` command-line tool, it looks
like changing the dependency to `typer-slim[standard]` is the best way
forward.

See https://typer.tiangolo.com/release-notes/#0121 and
tiangolo/typer#785 for further discussion
and details.
2024-04-07 00:34:34 -07:00
akierigandGitHub 3ba42802d1 added Macports install information (#1286) 2024-04-07 00:33:57 -07:00
James R. Barlow d6342cb8c2 Add heif/heic input image support 2024-04-07 00:33:13 -07:00
James R. Barlow 065bddbc6c Reformat with ruff format 2024-04-07 00:25:32 -07:00
James R. Barlow 067f429dde Merge branch 'main' of github.com:ocrmypdf/OCRmyPDF 2024-03-26 15:34:00 -07:00
Daniel LovegroveandGitHub 6895c2d70f Fix Broken Documentation Links (#1275)
* Update URL for PDFMARK documentation

For reference, here is a link to the old PDF:
https://web.archive.org/web/20190806035303/https://www.adobe.com/content/dam/acom/en/devnet/acrobat/pdfs/pdfmark_reference.pdf

It appears Adobe converted the PDF into a webpage-based document, the
wording seems to almost identical b/w the PDF and the website.

* Fix cross-references to JBIG2 page

* Fix links for Fedora + Arch + HEAD revision install

Fedora 39 has been released, and the package tracker no longer includes
a release overview for Fedora 37 hence why it was removed here.
2024-03-22 14:38:52 -07:00
James R. Barlow 686481982a Fix naming of hOCR rendered files 2024-03-22 13:27:20 -07:00
James R. Barlow a9e1d19b78 v16.1.2 release notes 2024-03-20 12:56:13 -07:00
James R. Barlow f95aa63718 Merge branch 'main' of github.com:ocrmypdf/OCRmyPDF 2024-03-20 12:26:02 -07:00
James Barlow 855de287b2 Fix test suite failure with Ghostscript >= 10.3
Ghostscript is more picky about a specific case with SMask that cannot be converted to PDF/A

Details here
https://github.com/ArtifexSoftware/ghostpdl/commit/4dcfae36bb4dcbc4ef3b5e5afc98bcde0d6b9ddc
2024-03-19 17:20:33 -07:00
NilsRoandGitHub feeb9f213f batch example: added archive, small corrections and optimizations (#1277)
* Added archive, small corrections

Added a function to archive originals and avoid calling ocrmypdf if they are still is PDF/A.

* Added Copyright
2024-03-18 13:22:24 -07:00
Emiel MolenaarandGitHub e7eb8fa805 Update Dockerfile.alpine (#1268)
Use Alpine 3.19 as base image to ensure we get GhostScript 10.2.1 to eliminate serious regressions that corrupt PDFs with existing text.
2024-03-13 14:49:42 -07:00
James R. Barlow 8a747f005a pixels -> megapixels
Fixes #1265
2024-02-29 15:31:07 -08:00
James R. Barlow 16ab4a8b4e Fix error message about missing Python exec
Message is
unable to start container process: exec: "python": executable file not found in $PATH: unknown.

Closes #1260
2024-02-21 23:54:41 -08:00
James R. Barlow 8d30cff4ef Undo future annotations from watcher.py till Typer fixes its issue
Fixes #1258
2024-02-20 19:14:39 -08:00
James R. Barlow 59d5b0d1bd v16.1.1 release notes 2024-02-15 16:56:25 -08:00
James R. Barlow 9ec0745ab8 Try pypy3.10 2024-02-14 14:25:13 -08:00
James R. Barlow 3a3635f7f9 Python 3.10 cleanup, manual fixes 2024-02-14 12:48:17 -08:00
James R. Barlow 6a746a1cbb ruff linting/Python 3.10 cleanup 2024-02-14 12:41:51 -08:00
James R. Barlow 906c130f96 Update rust toml settings 2024-02-14 12:32:26 -08:00
James R. Barlow 4a78458821 v16.1.0 release notes 2024-02-12 01:46:21 -08:00
James R. Barlow fddf3ce2f4 Clarify warnings filter 2024-02-12 01:43:47 -08:00
James R. Barlow 353b34e695 Merge branch 'feature/pageboxes' 2024-02-12 01:41:56 -08:00
James R. Barlow 7d63355c3c Use hocr renderer for LTR languages 2024-02-12 01:41:41 -08:00
James R. Barlow 42ff7fc842 Fix handling of pages that are restored to correct orientation with /Rotate
Appears inversion of CTM was incorrect, introduced in commit 9898904
2024-02-12 01:32:26 -08:00
James R. Barlow 26470fe16a Suppress reportlab deprecation warning 2024-02-12 01:17:08 -08:00
James R. Barlow 3b9d4b7f0a Attempt to deal with oddball mediaboxes 2024-02-11 15:34:54 -08:00
James R. Barlow 11f53fe9a9 First cut at propagating page boxes
This would fix the immediate issue, but does not address an offset mediabox.
2024-02-11 15:34:54 -08:00
James R. Barlow 123c0c766f Mention pipx, install --user --upgrade
Closes #1249
2024-02-08 09:42:00 -08:00
James R. Barlow 6a9be2142e Advise Homebrew on Linux for Ubuntu 20.04 2024-02-07 19:52:50 -08:00
James R. Barlow 0bc350f55e Merge branch 'main' of github.com:ocrmypdf/OCRmyPDF 2024-02-06 01:28:10 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
7a6edf62ba Bump codecov/codecov-action from 3 to 4 (#1247)
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 3 to 4.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-05 03:55:13 -08:00
James R. Barlow 07b6f06f11 optimize: log images with unclear decode tables 2024-02-01 15:42:40 -08:00
nisbet-hubbardandGitHub 2005f622bb Update gs dependency & instructions for RHEL (#1228)
* Update gs depedency to 9.54

* Add instructions for RHEL

* Update instruction for using python3.11 on RHEL
2024-01-24 10:07:58 -08:00
James R. Barlow cca04fd799 Document ability to remove all OCR 2024-01-23 23:35:43 -08:00
MaxiandGitHub 75bf8e4ba2 Add autotools automake libtool and leptonica requirements (#1230)
These tools are required for a successful install of JBIG2 on Ubuntu
2024-01-09 11:58:17 -08:00
Anish SapkotaandGitHub daabb5b100 Minor english correction in Docs (#1229) 2024-01-08 10:18:26 -08:00
James R. Barlow 035ebea72f Change Windows build to use older Ghostscript 2024-01-07 01:56:38 -08:00
James R. Barlow a499956462 v16.0.4 release notes 2024-01-07 01:39:06 -08:00
James R. Barlow 74d2a156c4 Update cache 2024-01-07 01:35:05 -08:00
James R. Barlow f87fc7b12d gs: check for text-corrupting versions 2024-01-07 01:33:49 -08:00
James R. Barlow 602f5632cb hocr: fix unbound local 2024-01-07 01:33:36 -08:00
James R. Barlow 9fbbcf7599 Merge branch 'feature/hocrdebug' 2024-01-07 01:22:54 -08:00
James R. Barlow 9498f01f59 hocr: Pass text direction for standalone words 2024-01-06 16:21:21 -08:00
James R. Barlow 2c59aca5a1 Fix render_mode ON_TOP when previous content stream does not restore CTM 2024-01-06 16:20:55 -08:00
James R. Barlow 51301d69c9 Change default graft render mode back to underneath
It was mistakenly changed to on top at some point
2024-01-06 16:20:30 -08:00
James R. Barlow 7e608fd1df Use Courier as debug font 2024-01-06 14:24:25 -08:00
James R. Barlow ecc79315df Adjust issue template files description 2024-01-04 00:27:06 -08:00
James R. Barlow 14365d10b8 Skip testing oom killer on Python 3.12
Need to investigate further if there's a safe way to do this test.
2024-01-02 16:28:22 -08:00
James R. Barlow 5e5320020f v16.0.3 release notes 2024-01-02 02:49:54 -08:00
James R. Barlow 103c3e0cd6 Drop Ghostscript CVE warning and adjust version down to 9.54
Most distros have probably now backported the CVE fix. Dropped version to 9.54 so RHEL 9 can use recent ocrmypdf.
2024-01-01 13:02:57 -08:00
Anthony NabilandGitHub 7a1c89edd9 fixed a spelling mistake (#1222) 2023-12-30 15:53:22 -08:00
James R. Barlow a5ff3d2f42 Update instructions about custom language packages
Closes #1218
2023-12-27 16:46:10 -08:00
James R. Barlow b71d16dd96 Draw RTL text with reversed matrix instead of reversing characters 2023-12-24 01:34:03 -08:00
James R. Barlow fd593eb5e9 Reversing character order for RTL helps output 2023-12-24 01:24:44 -08:00
James R. Barlow a0b98abb94 Improve and externalize debug rendering settings 2023-12-24 00:10:09 -08:00
James R. Barlow 18353e1e94 Also fix space rendering 2023-12-24 00:09:47 -08:00
James R. Barlow 9adcad84da Overhaul debug rendering to support possible use of Helvetica for testing 2023-12-23 23:47:35 -08:00
James R. Barlow f2714586d8 Fix upside down glyphless font 2023-12-23 23:44:37 -08:00
James R. Barlow 0b6fb62967 Update issue template 2023-12-23 16:21:17 -08:00
James R. Barlow 1db8b0b943 Temporarily revert default renderer to sandwich while investigating regressions in macOS Preview 2023-12-23 16:19:44 -08:00
James R. Barlow f38aebb3d5 Update issue template 2023-12-23 15:46:43 -08:00
James R. Barlow 7162c36d37 Name artifacts explicitly 2023-12-20 13:53:31 -08:00
James R. Barlow f4d4ea46c8 Update artifact actions 2023-12-20 12:44:43 -08:00
James R. Barlow 2fd1a0f178 v16.0.1 release notes 2023-12-20 12:33:41 -08:00
James R. Barlow 73ed33a086 Tighten dependencies 2023-12-20 12:33:18 -08:00
James R. Barlow e6095a9949 Fix text rendering issue with new hOCR text renderer 2023-12-20 12:26:06 -08:00
James R. Barlow 16f05af401 Fix release notes - drop rc from version 2023-12-18 20:08:45 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1631afc878 Bump actions/setup-python from 4 to 5 (#1205)
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 5.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-17 15:35:13 -08:00
Robin RichtsfeldandGitHub 63d87fc440 Fix --fast-web-view documentation (#1206) 2023-12-17 14:54:38 -08:00
James R. Barlow 9489c01259 Skip test_encrypted on Py3.12 + macOS 2023-12-08 00:12:24 -08:00
James R. Barlow 30d92ad83f Fix build settings to adjust for dropping py39 2023-12-07 23:40:45 -08:00
James R. Barlow a4987733c4 Filter rl_safe_eval deprecation warning
Full message
eportlab/lib/rl_safe_eval.py:11: DeprecationWarning: ast.NameConstant is deprecated and will be removed in Python 3.14; use ast.Constant instead
    haveNameConstant = hasattr(ast,'NameConstant')

Warning triggered by reportlab-4.0.7 and Python 3.12
2023-12-07 23:40:23 -08:00
James R. Barlow 39eee05230 v16.0.0rc1 release notes
Fixes #1009, #1191, #1157
2023-12-03 15:44:34 -08:00
James R. Barlow 5b2f2e6290 Merge branch 'feature/modernhocr' 2023-12-03 15:17:02 -08:00
James R. Barlow 445617a1a5 Rebuild cache for hocr default case 2023-12-03 15:16:18 -08:00
James R. Barlow f6e90a5934 hOCR renderer is now default 2023-12-02 19:58:00 -08:00
James R. Barlow 43618e6b3f Move canvas API to pikepdf and import it 2023-12-02 19:42:35 -08:00
James R. Barlow e97f89de3b Refactor font so glyphless isn't as hard coded 2023-12-02 08:55:01 -08:00
James R. Barlow 11d3e32f1e Fix hocrtransform CLI 2023-12-02 08:08:29 -08:00
James R. Barlow 2affa83efe Remove code that attempted to manage xattrs out of output file
Feature requested in issue #1179, but caused #1195. On further review,
there is no platform independent way to manage extended attributes
and it is not clear copying them through is necessarily the sensible
thing to do.

Closes #1179.
2023-11-29 23:25:51 -08:00
James R. Barlow c90d5cd84b Fix Ghostscript installation instructions and add warning 2023-11-29 14:10:04 -08:00
James R. Barlow aacaba3d26 Ignore pypy for now 2023-11-21 01:05:23 -08:00
James R. Barlow fec53be841 Remove next major release deprecations 2023-11-21 00:47:51 -08:00
James R. Barlow 3f7b540f76 Drop Python 3.9 support 2023-11-21 00:46:00 -08:00
James R. Barlow d217856166 Make hocrdebug work, and try to handle CJK spacing better 2023-11-21 00:33:02 -08:00
James R. Barlow e2be457e9b Avoid divzero 2023-11-20 23:08:00 -08:00
James R. Barlow 4850f486d2 Make text API more like an accessor 2023-11-20 22:59:50 -08:00
James R. Barlow 729c7febd9 Fix placement of spaces in debug mode 2023-11-20 22:44:12 -08:00
James R. Barlow 6c6aca2f1e Refactor save_state 2023-11-20 22:29:21 -08:00
James R. Barlow c69823f496 Refactor; accumulate content stream as bytes rather than discrete pikepdf objects 2023-11-20 22:11:59 -08:00
James R. Barlow 73f8f6aac8 Add RTL output - seems to work, but debug does not 2023-11-20 20:28:07 -08:00
James R. Barlow d944254e45 hocr: typing cont'd 2023-11-20 17:07:52 -08:00
James R. Barlow f7ddffe554 hocr: typing 2023-11-20 16:52:55 -08:00
James R. Barlow 8a73ed5d5a Fix JBIG2 not updating progress bar 2023-11-20 16:25:30 -08:00
James R. Barlow 03669183d7 Rationalize canvas interface 2023-11-20 15:54:13 -08:00
James R. Barlow 74e101a2fa Improve canvas interface with chaining 2023-11-20 14:42:48 -08:00
James R. Barlow 532cf18ad3 Restructure hocrtransform submodule to avoid having everything in __init__ 2023-11-20 00:57:58 -08:00
James R. Barlow 0b90b697e2 More tidying 2023-11-20 00:43:43 -08:00
James R. Barlow 6be7c5f7c8 Fix colors and space box rendering 2023-11-20 00:30:54 -08:00
James R. Barlow db2e5132e6 Remove some obsolete parameters 2023-11-20 00:10:55 -08:00
James R. Barlow b14f6f778a Tidying new hOCR renderer 2023-11-19 23:51:27 -08:00
James R. Barlow 415de77457 imageops: fix annots since not using singledispatch anymore 2023-11-19 23:51:27 -08:00
James R. Barlow a9466c4f58 Improve word box positioning 2023-11-19 23:51:27 -08:00
James R. Barlow d9ae453a63 Significantly improvement overall 2023-11-19 23:51:27 -08:00
James R. Barlow 9841e09233 More adjustments 2023-11-19 23:51:27 -08:00
James R. Barlow 0ca314e066 Replace Rect with pikepdf.Rectangle, migrate line matrix to page 2023-11-19 23:51:27 -08:00
James R. Barlow d7680cae27 Correcting Matrix logic helps
The good: don't have to do inverse and intermediate transforms.

The bad: skew looks bad, partly because the hOCR coordinate system is inconsistent around skew?
2023-11-19 23:51:27 -08:00
James R. Barlow 491b6bdb1f Remove concept of HOCR_OK_LANGS 2023-11-19 23:51:27 -08:00
James R. Barlow c591f9601a Remove Latin hOCR test 2023-11-19 23:51:27 -08:00
James R. Barlow 8d1e75017e Remote reportlab backend and make reportlab a test-only dependency 2023-11-19 23:51:27 -08:00
James R. Barlow 94615f7ad4 hOCR now works for all languages 2023-11-19 23:51:27 -08:00
James R. Barlow e5df8e1315 Nearly pixel perfect 2023-11-19 23:51:27 -08:00
James R. Barlow d739b91aef Tidy up 2023-11-19 23:51:27 -08:00
James R. Barlow 686cfb2539 Add rendering of space between boxes 2023-11-19 23:51:27 -08:00
James R. Barlow 2633716bb7 Render interword spaces separately and avoid box overlap 2023-11-19 23:51:27 -08:00
James R. Barlow 0a07c0a44e Fix more things 2023-11-19 23:51:27 -08:00
James R. Barlow 2ca6e110ca Fix private accessors, rename pdf to canvas 2023-11-19 23:51:27 -08:00
James R. Barlow 334a07c839 Refactor debug printing 2023-11-19 23:51:27 -08:00
James R. Barlow a57c39358d Refactor: extract methods 2023-11-19 23:51:27 -08:00
James R. Barlow 30a0c315fb Further exploratory improvements 2023-11-19 23:51:27 -08:00
James R. Barlow b860f0d94c Make coordinate system more consistent 2023-11-19 23:51:27 -08:00
James R. Barlow 14f4c19f5a WIP improve text positioning (not there yet) 2023-11-19 23:51:27 -08:00
James R. Barlow 7ab5c55d46 More colors 2023-11-19 23:51:27 -08:00
James R. Barlow 8b6ecd5971 Fix line and rect drawing 2023-11-19 23:51:27 -08:00
James R. Barlow 7b0871ae4c Fix position errors; ignore non-glyphless font 2023-11-19 23:51:27 -08:00
James R. Barlow b73af7ce10 Fix dashes 2023-11-19 23:51:26 -08:00
James R. Barlow 60645717e2 Test pikepdf canvas - renders... something at this point 2023-11-19 23:51:26 -08:00
James R. Barlow 1cbf578538 Fix import of pdf.ttf 2023-11-19 23:51:26 -08:00
James R. Barlow e966c1fceb Add pdf.ttf 2023-11-19 23:51:26 -08:00
James R. Barlow d0133f8641 Create pikepdf backend renderer 2023-11-19 23:51:26 -08:00
James R. Barlow 6d30b497dc Refactor reportlab into generic backend 2023-11-19 23:51:26 -08:00
James R. Barlow f3b89e66eb Refactor reportlab backend out of hocrtransform 2023-11-19 23:51:26 -08:00
James R. Barlow 04154e207c hocrtransform: move to module 2023-11-19 23:51:26 -08:00
James R. Barlow 9898904be7 Fix pikepdf PdfMatrix deprecation warning; v15.4.3 release notes 2023-11-15 20:27:16 -08:00
James R. Barlow 27d5229842 Make logger names unique 2023-11-09 23:03:39 -08:00
James R. Barlow 4a9a575ef0 ghostscript: better comments 2023-11-09 22:39:49 -08:00
James R. Barlow 52fd9a630d v15.4.2 release notes 2023-11-09 22:35:51 -08:00
James R. Barlow a596ccf844 Raise exception if resulting PDF might appear blank in a known in some PDF viewers
Fixes #1187
2023-11-09 22:33:22 -08:00
James R. Barlow e7fa97731f ghostscript duplicate filter: filter within a window of previous messages 2023-11-09 22:32:39 -08:00
James R. Barlow 290aa28108 Fix error on attempt to write to debug log after removing debug log handler 2023-11-09 16:02:41 -08:00
James R. Barlow a95640ed9e v15.4.1 release notes
Closes #1185
Closes #1183
2023-11-07 23:57:30 -08:00
James R. Barlow f69267bb67 watcher: restore ability to read json from file or command line string 2023-11-07 18:05:29 -08:00
James R. Barlow e36d5a309f Make grafting a little bit more configurable 2023-11-05 14:01:01 -08:00
James R. Barlow 55566d9830 Fix watcher.py kwarg error 2023-11-05 13:58:24 -08:00
James R. Barlow f02ea20678 docs: plugin documentation missing key special members 2023-11-05 00:10:51 -07:00
James R. Barlow 372c22d42b docs: improve 2023-11-04 02:43:08 -07:00
James R. Barlow 949265bbd0 graft: improve typing and remove procset tracking
ProcSet is optional and deprecated in PDF 2.0, and does little anyway; so
we removed it.
2023-11-04 02:32:39 -07:00
James R. Barlow 916106733c Skip semfree unless on Linux 2023-10-30 00:33:21 -07:00
James R. Barlow 44bcafd3aa Add missing file header 2023-10-30 00:28:09 -07:00
James R. Barlow 71166f7be8 Make hocr API experimental for now
This commit can be reverted when we are ready to release a new version.
2023-10-30 00:07:10 -07:00
James R. Barlow 580252a1a0 Merge branch 'feature/gscan2pdf'
Reconcile release notes and copy_final() with new pipeline.
2023-10-30 00:01:28 -07:00
James R. Barlow c0b60dae6a build: add repository -y 2023-10-29 23:41:36 -07:00
James R. Barlow ae123fd209 Try to retain/copy xattrs 2023-10-28 01:42:06 -07:00
James R. Barlow 454ad0acc5 build/macos: add openssl 2023-10-28 01:41:28 -07:00
James R. Barlow 0c306ac328 v15.3.1 release ntoes 2023-10-28 00:49:37 -07:00
James R. Barlow 52d99732b1 Fix mistakes with watcher loglevel handling 2023-10-28 00:47:40 -07:00
James R. Barlow 5b5827983b Tweak documentation of --output-type 2023-10-26 23:57:08 -07:00
James R. Barlow 56f9bc311d Improve verbosity of colorspace selection 2023-10-25 00:38:56 -07:00
James R. Barlow eb17dc1ecf Fix pdf save settings at metadata_fixup 2023-10-25 00:13:17 -07:00
James R. Barlow 6f8115a052 Fix import of metadata_fixup 2023-10-25 00:06:26 -07:00
James R. Barlow aac913c666 tesseract: EAFP 2023-10-24 13:50:04 -07:00
James R. Barlow b5e73ac4e4 Drop check for obsolete .dockerinit file 2023-10-24 13:49:46 -07:00
James R. Barlow 9e98c90891 docs: note on docker performance 2023-10-24 13:34:26 -07:00
James R. Barlow ca2592c1d9 Update draft release notes 2023-10-24 00:56:00 -07:00
James R. Barlow a31f17bb9d Update comments and make worker functions private 2023-10-24 00:56:00 -07:00
James R. Barlow 1cb46afa94 Update pluginspec docs 2023-10-24 00:56:00 -07:00
James R. Barlow 5a759947dd Update release notes so far 2023-10-24 00:56:00 -07:00
James R. Barlow db3df13e95 Remove ocrmypdf._sync 2023-10-24 00:54:31 -07:00
James R. Barlow 2a8bc03167 optimize: typing 2023-10-24 00:54:31 -07:00
James R. Barlow d2297b39d0 info: clarify ICC -> components checking 2023-10-24 00:54:31 -07:00
James R. Barlow e4cd081d4d info: clarify pageinfo context management 2023-10-24 00:54:31 -07:00
James R. Barlow d2dbea6cf8 Reorganize progress bars so they can be typed properly 2023-10-24 00:54:31 -07:00
James R. Barlow 46a279a49a Improve passing of arguments to workers
The executor system was built around passing only a single
argument to workers, which was
always PageContext. For other tasks, all actual arguments were packed in
a tuple, which meant we needed intermediate functions to unpack the
tuple.

The situation is now rationlized and resembles how Python handles
argument passing to familiar multiprocessing tools.
2023-10-24 00:54:31 -07:00
James R. Barlow 299f0c4003 Update dep5 2023-10-24 00:54:31 -07:00
James R. Barlow 9ffb45f283 Remove public domain congress.jpg and replace with baiona_color.jpg
For reuse compliance we are phasing out public domain licenses
2023-10-24 00:54:31 -07:00
James R. Barlow cd61c4efd9 pngquant: remove unused ability to quantize a non-PNG
Covering testing showed this branch was never used, and when tested it didn't work.
2023-10-24 00:54:31 -07:00
James R. Barlow a06ab2a1c5 unpaper: Remove format conversion
Code is no longer reachable since we rasterize a 1/L/RGB image prior to this point.
2023-10-24 00:54:31 -07:00
James R. Barlow dfa4ebf1a6 Simplify function signature of extract_image_filter 2023-10-24 00:54:31 -07:00
James R. Barlow 58f388c69d optimize: better coverage 2023-10-24 00:54:31 -07:00
James R. Barlow 990b462a94 Fix coverage settings and cover semfree 2023-10-24 00:54:31 -07:00
James R. Barlow b928dc0808 Skip fewer tests 2023-10-24 00:54:31 -07:00
James R. Barlow 8916955f45 Convert many run_ocrmypdf -> run_ocrmypdf_api 2023-10-24 00:54:31 -07:00
James R. Barlow 82bef40aa6 Eliminate more run_ocrmypdf calls 2023-10-24 00:54:31 -07:00
James R. Barlow 1c45f32941 tests: replace many run_ocrmypdf -> run_ocrmypdf_api 2023-10-24 00:54:31 -07:00
James R. Barlow fadc0cf69b Replace cryptic test error messages with more informative ones 2023-10-24 00:54:31 -07:00
James R. Barlow 7ce9d08b2d Define progress bar plugins formally instead of "tqdm-like" 2023-10-24 00:54:31 -07:00
James R. Barlow eb3a51e33a Prefer pikepdf's newer Page.mediabox accessor over .MediaBox 2023-10-24 00:54:31 -07:00
James R. Barlow f3dd733773 optimize: explore page container as objects instead of page helpers 2023-10-24 00:54:31 -07:00
James R. Barlow 4dbc5e1dba Fix some typing issues 2023-10-24 00:54:31 -07:00
James R. Barlow c0637c287e vscode isn't ready for black py312, revert 2023-10-24 00:54:31 -07:00
James R. Barlow 6127f7abd6 tqdm_kwargs to progress_kwargs 2023-10-24 00:54:31 -07:00
James R. Barlow a4059762e6 Fix hocrtransform test to generate blank hocr 2023-10-24 00:54:31 -07:00
James R. Barlow 40afcd68a7 pluginspec: spacing 2023-10-24 00:54:31 -07:00
James R. Barlow f238e721ed Improve documentation of new public hOCR APIs 2023-10-24 00:54:31 -07:00
James R. Barlow 16eb5627a7 Fix unused imports and other trivia 2023-10-24 00:54:31 -07:00
James R. Barlow fbf0674189 hocr_to_ocr_pdf: handle missing hocr json file 2023-10-24 00:54:31 -07:00
James R. Barlow 62c4f65fc3 Remove duplicate thread local storage of page numbers 2023-10-24 00:54:31 -07:00
James R. Barlow e400112f32 Improve ._pipelines naming 2023-10-24 00:54:31 -07:00
James R. Barlow 7935914f55 Use empty .hocr file instead of dummy template for symmetry with sandwich 2023-10-24 00:54:31 -07:00
James R. Barlow ad3a1dbbad deps: update PyMuPDF req 2023-10-24 00:54:31 -07:00
James R. Barlow 0655f8e7ae Add py312 to black coverage 2023-10-24 00:54:30 -07:00
James R. Barlow 04a9372584 docs: some copyediting 2023-10-24 00:54:30 -07:00
James R. Barlow b9646b6f85 Enable multiprocessing freeze_support (for Windows) and enable forkserver
Since we use a thread for logging among other possibilities, we were
never able to use threads safely when forking. forkserver will do the job.
2023-10-24 00:54:30 -07:00
James R. Barlow 53c953a561 Fix use_threads logic for get_pdfinfo
Some debug code was level in place that forced pdfinfo to run with only
one worker when --use-threads was issued. That is how it ought to be,
since threaded pdfinfo workers just fight over the GIL and there is no
sense in parallelizing them.

Also, the user's --use-threads or --no-use-threads would be ignored in
the case of pdfinfo. By setting max_workers=1 we disabled worker processes.

This fixes how that decision is made (putting it in the relevant code, which
knows its constraints) and allows the user to influence the
thread/process decision again.
2023-10-24 00:54:30 -07:00
James R. Barlow c278fecb34 Rename post_process -> postprocess
For consistency with preprocess
2023-10-24 00:54:30 -07:00
James R. Barlow 23951c9e38 Working HOCR folder to PDF converter 2023-10-24 00:54:30 -07:00
James R. Barlow e8ae370ceb Eliminate api= kwarg and implicit creation of pluginmanager 2023-10-24 00:54:30 -07:00
James R. Barlow 67be4d1904 Refactor CLI exception handling 2023-10-24 00:54:30 -07:00
James R. Barlow 6f82097d14 Refactor setup_pipeline to decouple manage_work_folder 2023-10-24 00:54:30 -07:00
James R. Barlow fc6f959d21 Refactor debug log and work folder context cleanup 2023-10-24 00:54:30 -07:00
James R. Barlow e38d569d8f languages: kwargs are overkill 2023-10-24 00:54:30 -07:00
James R. Barlow 0856750ee2 Fix exit code error on Ghostscript failure 2023-10-24 00:54:30 -07:00
James R. Barlow 05721ba84a Fix error on no languages available 2023-10-24 00:54:30 -07:00
James R. Barlow 38c3422e5e Automatically set document language to OCR language 2023-10-24 00:54:30 -07:00
James R. Barlow d153a6f6df Refactor metadata handling 2023-10-24 00:54:30 -07:00
James R. Barlow 1a7738a925 Refactor -migrate metadata repair to new module 2023-10-24 00:54:30 -07:00
James R. Barlow 8985c0dfe9 Refactor setup_pipeline to context manager 2023-10-24 00:52:31 -07:00
James R. Barlow ebfe008432 Refactor logging record thread local storage 2023-10-24 00:52:31 -07:00
James R. Barlow 1f16eb6f50 Refactor main pipeline into discrete pipelines
- test_ghostscript_pdfa_failure fails
- haven't dealt with logging record factories

Further refactoring and duplicate elimination
2023-10-24 00:52:31 -07:00
James R. Barlow cbb0868ae3 Add hocr to ocr pdf pipeline 2023-10-24 00:52:31 -07:00
James R. Barlow 68bb38d0ad pdf_to_hocr: improve plugin handling 2023-10-24 00:52:31 -07:00
James R. Barlow 0443e87345 Introduce pdf_to_hocr API 2023-10-24 00:52:31 -07:00
James R. Barlow b3de5833d3 Refactor conversion of ocrmypdf.ocr() arguments to cmdline 2023-10-24 00:52:31 -07:00
James R. Barlow 95b14ee282 Refactor lossless reconstruction setter into separate function
Still messy but good enough as a start.
2023-10-24 00:52:31 -07:00
James R. Barlow 07b89e6a19 Plugin manager: set reasonable default when called without params 2023-10-24 00:52:31 -07:00
James R. Barlow 8991d2cb33 Refactor main pipeline and start hocr pipeline 2023-10-24 00:52:31 -07:00
James R. Barlow 86a20c4130 Refactor exec_page_sync to outputs 2023-10-24 00:52:31 -07:00
James R. Barlow 6827a6efe8 Refactor exec_page_sync -> extract _process_page 2023-10-24 00:52:31 -07:00
James R. Barlow c6b5332699 Test on release py312 not py312-rc 2023-10-21 15:17:07 -07:00
Michael FlaggandGitHub 68610046c6 Correct the archive dir name in Watched folders with Docker (#1173) 2023-10-21 14:59:33 -07:00
James R. Barlow 880326868d v15.3.0 release notes 2023-10-21 14:59:09 -07:00
James R. Barlow c6be3ba076 watcher: Improve parameter validation 2023-10-20 20:11:00 -07:00
James R. Barlow 0565cb0b10 misc/watcher.py: use Typer and dotenv to improve ease of use 2023-10-20 19:56:39 -07:00
James R. Barlow dc49906704 Improve wait_for_file_ready loop 2023-10-20 19:55:50 -07:00
James R. Barlow 93fda0dd00 Detect and warn about Tagged PDFs 2023-10-12 01:03:09 -07:00
James R. Barlow d4110e78cb Fix "pikepdf mmap disabled" spam on macOS 2023-10-11 01:24:29 -07:00
James R. Barlow 5285d68fcc Add check for filenotfound due to being inside a snap 2023-10-11 01:02:50 -07:00
James R. Barlow 2b0e149809 Update docker document and v15.2.0 release notes 2023-10-10 01:25:26 -07:00
James R. Barlow b7ce5b0d7d Don't build docker alpine aarch64 for now and add alias tag for Ubuntu 2023-10-09 17:36:08 -07:00
James R. Barlow ffd6a64ce9 Attempt at alpine on arm64 - not working
pip attempts to download musllinux_1_1 wheels and can't find pikepdf's which is only 1_2. Not clear how to fix.
2023-10-09 17:34:28 -07:00
James R. Barlow 5727f1e081 Test docker alpine build and ubuntu aliasing 2023-10-09 15:08:38 -07:00
James R. Barlow b75a7eca2a Alpine image: Fix missing OSD and path 2023-10-09 14:59:56 -07:00
James R. Barlow 2b01676434 docker docs: remove VirtualBox content and update a few explanations 2023-10-09 14:57:55 -07:00
James R. Barlow e11c386c58 Add Alpine dockerfile 2023-10-09 14:35:52 -07:00
James R. Barlow 9346d1f970 Simplify deep nested with-block 2023-10-09 02:30:30 -07:00
James R. Barlow 012cbef865 v15.1.0 release notes 2023-10-04 00:16:22 -07:00
James R. Barlow 0687568e1b sync: update documentation 2023-10-04 00:05:27 -07:00
James R. Barlow 3086cfc3d9 pipeline: documentation and tweak merge_sidecars 2023-10-04 00:05:19 -07:00
James R. Barlow 91a14660b3 Require Pillow >= 10.0.1 and drop shims for older versions 2023-10-04 00:04:28 -07:00
James R. Barlow 539f0ee0ce Document some missing CLI options to API 2023-10-03 23:54:24 -07:00
James R. Barlow 7172817cd6 Replace ExitStack with contextmanager 2023-10-02 16:52:06 -07:00
James R. Barlow d9cc759142 Fix typos in release notes 2023-10-02 02:49:01 -07:00
James R. Barlow 364799fc3e Fix typing for pdinfo.layout and modernize
Also changed handling of undefined chars to be more Liskov-consistent with parent class.
2023-10-01 01:18:22 -07:00
James R. Barlow f4c211fa2d Use Python 3.9-style type hinting for tuple[] and AbstractSet -> Set 2023-10-01 00:09:05 -07:00
James R. Barlow 113a6b45bd ruff autofixes (mostly typing.* -> collections.abc.*) 2023-10-01 00:02:53 -07:00
James R. Barlow e9419d2c40 Update notes on concurrency 2023-09-30 17:05:58 -07:00
James R. Barlow fb006ef39f v15.0.2 release notes 2023-09-29 00:56:09 -07:00
James R. Barlow 890b994403 docs: update to discuss some v15 features not yet documented 2023-09-27 00:40:05 -07:00
James R. Barlow 01bbf7d144 docs: mention new features in v15, fix 32-bit text again 2023-09-27 00:14:19 -07:00
James R. Barlow 468de5324a Add Python 3.12 to test matrix 2023-09-26 20:20:33 -07:00
James R. Barlow 072db75fa3 v15.0.1 release notes (again) 2023-09-26 20:09:04 -07:00
James R. Barlow 8519b3f625 docs: clarify situation around 32-bit support
Maintainers of ARM 32-bit in particular don't necessarily need to drop support....
2023-09-26 20:06:33 -07:00
James R. Barlow dd7c4f3eaa docs: update install notes for some things missed with v15 release 2023-09-26 20:04:30 -07:00
James R. Barlow c8e6f20f8d Change 32-bit message from error to warning 2023-09-26 20:03:06 -07:00
James R. Barlow 10530a8698 Change Ghostscript version skip to fail
Reported to fail on earlier versions than the check tested for.
2023-09-26 20:02:52 -07:00
James R. Barlow 207866abf5 v15.0.1 release notes 2023-09-26 15:56:53 -07:00
James R. Barlow 3829af16fb Fix bdist_wheel tag set to py38 2023-09-26 15:46:47 -07:00
James R. Barlow 24db31b4c5 Innocuous change to bump tag 2023-09-26 12:25:02 -07:00
James R. Barlow 8132a4ae10 Update release notes and files 2023-09-26 00:37:04 -07:00
James R. Barlow d5128c5cf5 Further improvements to image DPI calculation 2023-09-26 00:28:54 -07:00
James R. Barlow 270e31fa67 Fix pluginmanager typing 2023-09-25 14:42:48 -07:00
James R. Barlow 85e31d0a19 Tidy imports and line length 2023-09-25 01:04:01 -07:00
James R. Barlow ea36aedb5f Overhaul version checkers to prefer Version to str 2023-09-25 00:59:44 -07:00
James R. Barlow bd4d44e182 Remove Python 3.8 shim for missing str.removeprefix 2023-09-25 00:31:48 -07:00
James R. Barlow 8fcf358934 Rename pike local variable to pdf for consistency 2023-09-25 00:22:26 -07:00
James R. Barlow 47b0f28564 Minor documentation and typing fixes 2023-09-25 00:19:48 -07:00
James R. Barlow 7018e2b247 Refactor ghostscript error message deduplicating 2023-09-24 20:22:04 -07:00
James R. Barlow 8d12ecb798 Refactor docinfo repair code 2023-09-24 20:21:26 -07:00
James R. Barlow 0ab29ec0ba Update release notes 2023-09-24 01:30:12 -07:00
James R. Barlow 179714770a Update release ntoes 2023-09-24 01:24:00 -07:00
James R. Barlow f04f45545c ghostscript: fix missing type annotation 2023-09-24 01:19:56 -07:00
James R. Barlow a3a083c125 logging: Avoid possible multiplication by None 2023-09-23 15:08:55 -07:00
James R. Barlow d855f63985 Remove single dispatch version of calculate_downsample 2023-09-23 15:05:04 -07:00
James R. Barlow d4863cbf0f helpers: fix typing 2023-09-23 14:59:11 -07:00
James R. Barlow 7b8f081fbf Enable pikepdf mmap again 2023-09-23 14:50:56 -07:00
James R. Barlow 7d33039bcd Add comment re: reportlab shim 2023-09-21 00:27:45 -07:00
James R. Barlow fde886baf4 Revert "Drop reportlab warning shim"
This reverts commit 162a47f98e.
2023-09-21 00:26:50 -07:00
James R. Barlow 146da79c00 Regenerate test cache 2023-09-21 00:24:55 -07:00
James R. Barlow 2fc3b0d973 Move issue*.pdf into separate folder 2023-09-21 00:19:13 -07:00
James R. Barlow 5667424530 Lower pngquant version req 2023-09-21 00:13:03 -07:00
James R. Barlow 8add531ffd Drop Ubuntu 20.04 from build matrix 2023-09-21 00:12:07 -07:00
James R. Barlow 0388c23ae7 Merge branch 'feature/jbig2thresh' into v15 2023-09-21 00:07:05 -07:00
James R. Barlow 9b77daae7c Merge branch 'feature/snap22' into v15 2023-09-21 00:06:14 -07:00
James R. Barlow 3e1b3ec98d Merge branch 'feature/fix-raster-dpi-too-high' into v15 2023-09-21 00:05:56 -07:00
James R. Barlow 0f0ca6f517 Draft v15 release notes 2023-09-21 00:05:39 -07:00
James R. Barlow c93349c350 Convert workflow to trusted PyPI publisher 2023-09-21 00:05:39 -07:00
James R. Barlow 0c287929c2 Don't use really old Python for deliverable building 2023-09-21 00:05:39 -07:00
James R. Barlow 23a37fc35c Document that Ubuntu 22.04 and 20.04 both use Tesseract 4.1.1 2023-09-21 00:05:39 -07:00
James R. Barlow 162a47f98e Drop reportlab warning shim 2023-09-21 00:05:39 -07:00
James R. Barlow 0239f69912 Tigthen Python dependencies 2023-09-21 00:05:39 -07:00
James R. Barlow 2ad8961d0b Require pngquant 2.13.1 or newer 2023-09-21 00:05:39 -07:00
James R. Barlow eec8a2b574 Change minimum Ghostscript version to 9.55 2023-09-21 00:05:39 -07:00
James R. Barlow 6c78076bea Drop support for gswin32c / 32-bit Ghostscript for Windows 2023-09-21 00:05:39 -07:00
James R. Barlow 2637e84691 Complain about all 32-bit interpreters, not just windows 2023-09-21 00:05:39 -07:00
James R. Barlow e8c82ee4b6 Remove shim for img2pdf < 0.4.4 2023-09-21 00:05:39 -07:00
James R. Barlow de2bb5ce8c Remove tqdm dependency and TqdmConsole
Might be too aggressive? No deprecation warning....
2023-09-21 00:05:39 -07:00
James R. Barlow ec1c377532 Drop Python 3.8 too
(cherry picked from commit a8cdc5a191bdf1fe401e96c2dc9a5fba39b90d79)
2023-09-21 00:05:39 -07:00
James R. Barlow 173428e81a Added weighted DPI rendering
To address #1010 and other issues.
2023-09-21 00:04:50 -07:00
James R. Barlow 67ed29dcea Fix incorrect printed_area calculation 2023-09-21 00:02:02 -07:00
James R. Barlow 3454c050ed Introduce Resolution.take_min 2023-09-21 00:01:42 -07:00
James R. Barlow 5ee99b26e7 Update snap to use core22 2023-09-20 17:01:36 -07:00
James R. Barlow ac3aa67d8a Revise to PageResolutionProfile 2023-09-20 16:58:59 -07:00
James R. Barlow 1768a1eda9 Improve documentation for large image downsampling 2023-09-20 16:58:59 -07:00
James R. Barlow 5902fe45c1 Rename Resolution.mean to to_scalar 2023-09-20 16:58:59 -07:00
James R. Barlow 78981641f0 Introduce new functions to calculate DPI exceptions 2023-09-20 16:58:59 -07:00
James R. Barlow c77ae4b34c Change hookspec to migration parameters for generate_pdf to options object
Breaking change for PDF rendering plugins (although none are known to exist).
This provides better separation of Ghostscript specific concerns from
the generic plugin interface.
2023-09-20 15:40:18 -07:00
James R. Barlow b2cbbf0099 Migrate --pdfa-image-compression argument to being managed by Ghostscript plugin 2023-09-20 15:20:42 -07:00
James R. Barlow 6b6c34af01 Fix test artifact generated incorrectly
Test still passed either way
2023-09-20 15:20:13 -07:00
James R. Barlow be12f7a728 Make fish completion a bit smarter 2023-09-20 14:45:22 -07:00
James R. Barlow e3c813fc67 Added support for changing color conversion strategy 2023-09-20 01:08:15 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
35a1eaf62a Bump docker/setup-qemu-action from 2 to 3 (#1151)
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 2 to 3.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-18 12:06:38 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
d393d18c13 Bump docker/login-action from 2 to 3 (#1152)
Bumps [docker/login-action](https://github.com/docker/login-action) from 2 to 3.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-18 12:06:30 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
54e622ad10 Bump docker/setup-buildx-action from 2 to 3 (#1153)
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2 to 3.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-18 12:06:19 -07:00
James R. Barlow 330352aeed Update completions for jbig2 threshold 2023-09-17 14:47:46 -07:00
James R. Barlow ac2fc49208 Add support for adjusting jbig2 threshold 2023-09-17 14:44:40 -07:00
Srikar SundaramandGitHub 4bee7355e9 Change skip-ocr to skip-text (#1146) 2023-09-14 17:22:34 -07:00
dependabot[bot]andGitHub 86f2b1f9a7 Bump actions/checkout from 3 to 4
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-11 10:37:22 +00:00
James R. Barlow 3002409e49 Remove older installation caveats 2023-08-22 01:25:57 -07:00
James R. Barlow 0cf6828c20 Merge branch 'main' of github.com:ocrmypdf/OCRmyPDF 2023-08-19 13:05:06 -07:00
James R. Barlow 331c829b6e docs: remove obsolete reference to setuptools-scm-git-archive 2023-08-19 13:03:31 -07:00
James R. Barlow 06a5e0c3f6 docs: various updates 2023-08-14 00:13:17 -07:00
811f23381a Add installation instructions for Gentoo Linux to README.md (#1134)
* Add installation instructions for Gentoo Linux to README.md

* Correct installation instructions for macOS (nix) in README.md

* Move installation instructions for Gentoo Linux from README.md to docs/installation.rst

---------

Co-authored-by: Fonic <5cx5jbnmt@mozmail.com>
2023-08-13 01:21:14 -07:00
James R. Barlow a371655052 v14.4.0 docs and release notes 2023-08-12 01:55:00 -07:00
James R. Barlow a6ce35b13a Add argument to override digital signatures 2023-08-12 01:31:36 -07:00
James R. Barlow 45added738 Implement digital signature detection 2023-08-12 01:20:20 -07:00
James R. Barlow 6e20439c91 Remove oddball log.error/raise Exception() pattern
Instead present error message in exception.
2023-08-12 01:02:42 -07:00
James R. Barlow 72e056436c Add simple signature detection 2023-08-12 00:59:54 -07:00
James R. Barlow e02ba19097 Modernize pdfinfo usage of pikepdf to Name.X 2023-08-12 00:54:45 -07:00
James R. Barlow d3b858f994 Improve progress bar presentation/alignment 2023-08-11 01:57:25 -07:00
James R. Barlow 19045c4f21 Replace coloredlogs and tqdm with rich 2023-08-11 01:47:42 -07:00
James R. Barlow f4d89fe6cc Fix typo 2023-08-11 01:47:42 -07:00
Trenton HandGitHub ab85c0f5a9 Enables creation of a release and uploading the build assets to it (#1132) 2023-08-11 01:37:38 -07:00
James R. Barlow 32693b683d Revert "Add Python 3.12 prerelease to build matrix"
This reverts commit b5dc276ba1.

Would require building pikepdf from source on Python 3.12, which is
not worth the effort at this point.
2023-08-01 01:11:15 -07:00
James R. Barlow b5dc276ba1 Add Python 3.12 prerelease to build matrix 2023-08-01 00:32:11 -07:00
James R. Barlow 7c38c71794 gh-action-pypi-publish doesn't have a main - use release/v1 2023-06-20 03:11:46 -07:00
James R. Barlow a80e7a127b Fix some references to master not renamed to main 2023-06-20 01:37:26 -07:00
James R. Barlow cf3309555f Merge branch 'main' of github.com:ocrmypdf/OCRmyPDF 2023-06-20 01:12:53 -07:00
James R. Barlow f80dd0d86a v14.3.0 release notes 2023-06-20 01:12:45 -07:00
James R. Barlow 1ba2bce486 Merge branch 'docs' 2023-06-20 01:08:09 -07:00
050dd1f5a8 Allow title, subject, author, and keywords to be unset with an empty string argument (#1117)
Co-authored-by: Frederick D. Hansen <frederick.hansen@gmail.com>
2023-06-20 01:07:23 -07:00
James R. Barlow e44a57aec0 Try a screencast/terminal demo 2023-06-20 00:48:42 -07:00
James R. Barlow d94d2671c3 Merge branch 'abwiersma-fix_random_ordering_languages' 2023-06-19 23:27:03 -07:00
James R. Barlow 5124daa79f Fix test failures from preceding 2023-06-19 23:25:31 -07:00
James R. Barlow 7293847da7 Merge branch 'fix_random_ordering_languages' of github.com:abwiersma/OCRmyPDF into abwiersma-fix_random_ordering_languages 2023-06-19 23:21:04 -07:00
Abe 59fd0ac587 Fix randomly ordered languages from set() 2023-06-16 22:14:40 +02:00
James R. Barlow 90619b308c pipeline: improve documentation 2023-06-12 19:50:23 -07:00
James R. Barlow d0d49ce989 docs: Prefer long arguments 2023-06-12 01:00:01 -07:00
James R. Barlow bf0224faa4 Remove final bits of isort 2023-06-11 16:44:09 -07:00
James R. Barlow ae2f8ed8f1 pipeline: describe page/canvas DPI 2023-06-11 13:40:14 -07:00
James R. Barlow 14ac9b0560 docs: some reorganizing 2023-06-11 13:37:53 -07:00
James R. Barlow dbe6148d41 Update release notes
Closes #1083
Closes #1101
2023-06-03 00:27:42 -07:00
James R. Barlow 36d4c2dbbc Merge commit '6eadd65dfba3f983702087605d582f880efc020b' 2023-06-03 00:23:18 -07:00
James R. Barlow adbffb7bd9 Merge branch 'feature/soft-error' 2023-06-03 00:23:10 -07:00
James R. Barlow 05ecb6ca46 Resolve Windows errors with -dPDFSTOPONERROR 2023-06-02 02:48:38 -07:00
James R. Barlow 0a7b60cda5 Fix error on metadata block 2023-06-02 02:47:41 -07:00
James R. Barlow 5f211ecf6f Add test cases for soft errors 2023-06-02 02:47:41 -07:00
James R. Barlow 417ee067a2 Remove obsolete universal_newlines check 2023-06-02 02:47:41 -07:00
James R. Barlow c4649dabef Clarify optimization ratio
Fixes #1070
2023-06-02 01:34:07 -07:00
James R. Barlow 6eadd65dfb Log deskew angle
Closes #1101
2023-06-02 01:12:49 -07:00
James R. Barlow e8ed510543 Add stop on soft render errors and option to override 2023-06-01 23:49:34 -07:00
James R. Barlow 22d35c199d Add Windows package manager to list 2023-06-01 22:57:31 -07:00
James R. Barlow 5a82ad63c9 Modernize issue templates 2023-06-01 22:55:33 -07:00
James R. Barlow 4769a6c50b Merge branches 'master' and 'master' of github.com:ocrmypdf/OCRmyPDF 2023-06-01 16:53:25 -07:00
jbarlow83andGitHub 9004009adc Create bug-report-general.yml 2023-06-01 16:52:08 -07:00
James R. Barlow 11221f9912 Improve rasterization by including more fonts 2023-05-23 13:24:09 -07:00
James R. Barlow 177349cc84 v14.2.1 release notes 2023-05-23 13:21:46 -07:00
James R. Barlow 070c9772ce Add test to confirm we can find formx optimizable images 2023-05-23 00:33:54 -07:00
James R. Barlow 1bc09045a5 optimize: Add recursion into Form XObjects 2023-05-23 00:07:15 -07:00
James R. Barlow e46a18dd2f Refactor searching for optimizable images 2023-05-22 23:42:10 -07:00
James R. Barlow c64871c2ed pyproject: Make classifiers more generic 2023-05-22 23:37:42 -07:00
James R. Barlow de909fb99a Docker arm64 needs to build pycairo; add its dependencies 2023-05-10 01:31:31 -07:00
James R. Barlow 731b2fc477 Fix interaction with old Python and singledispatch and typing 2023-05-10 00:15:34 -07:00
James R. Barlow 214f6ec759 v14.2.0 release notes 2023-05-09 23:32:57 -07:00
James R. Barlow 080aa4dbd1 imageops: fix rare assert failure 2023-05-09 23:27:08 -07:00
James R. Barlow 7af5dcd4a4 Improve directions about install other languages 2023-05-05 13:35:27 -07:00
James R. Barlow fe9f52fbe7 Remove Ghostscript UserUnit warning
All supported Ghostscript versions have proper UserUnit support.
2023-04-29 00:42:38 -07:00
James R. Barlow fcbdeb8dbe Fix resampling attributeerror when Pillow < 9.1.0 2023-04-25 15:17:45 -07:00
James R. Barlow cb251a8d03 Merge branches 'feature/better-imageops' and 'feature/cleanup-fitz' 2023-04-16 01:41:46 -07:00
James R. Barlow 3731fdfd72 Add --tesseract-downsample-above X 2023-04-16 01:41:19 -07:00
James R. Barlow b2e6a6431e Tidy use of pikepdf.open without with 2023-04-15 20:17:44 -07:00
James R. Barlow 9ff1e56bf6 Tidy test_metadata - importorskip idiom 2023-04-15 17:56:13 -07:00
James R. Barlow 2b30f74fce Improve imageops
Fix issues and add better tests. Require hypothesis.
2023-04-15 17:48:33 -07:00
James R. Barlow 10f4c48e0b Fix mypy errors 2023-04-14 13:49:14 -07:00
James R. Barlow 37d5c086bb Fix things not caught by ruff 2023-04-14 13:44:47 -07:00
James R. Barlow 91830627e5 v14.1.0 2023-04-14 13:11:10 -07:00
James R. Barlow f2fc37b257 Merge feature to downsample very large images 2023-04-14 13:09:15 -07:00
James R. Barlow a99e40fa84 Update notes about ocrmypdf.ocr usage 2023-04-14 11:54:58 -07:00
James R. Barlow 9ce692a6f1 ruff: further fixes 2023-04-14 02:39:36 -07:00
James R. Barlow a3c49b8f31 ruff: more lint 2023-04-14 02:19:19 -07:00
James R. Barlow 33b70be7d5 ruff: more fixes, mainly missing docstrings 2023-04-14 02:16:38 -07:00
James R. Barlow 4924b11b6b Additional ruff fixes 2023-04-14 01:25:16 -07:00
James R. Barlow 1d0e4e7c9f ruff: fix from __future__ and drop isort 2023-04-14 00:49:02 -07:00
James R. Barlow 9b8d14d16e Accept most of ruff's delinting 2023-04-14 00:45:34 -07:00
James R. Barlow b7eb93eb79 Adopt ruff and fix prelim lints 2023-04-14 00:19:17 -07:00
James R. Barlow 42c0d0f48f Update pre-commit 2023-04-14 00:06:13 -07:00
James R. Barlow 5fce50ff7c Change linter to ruff 2023-04-14 00:05:43 -07:00
James R. Barlow 765ed4c386 Remove older linters 2023-04-14 00:03:12 -07:00
James R. Barlow 1b2849ec0a Add different timeout control for non-OCR
This is mainly to permit the use of --tesseract-timeout 0 as a way of
disabling OCR but still allowing deskew and orientation detection to work.

The better solution will be introducing an explicit argument to
"don't ocr my pdf", but that will take more rework.
2023-04-13 23:59:58 -07:00
James R. Barlow 4f604591b4 Convert to image size based 2023-04-13 23:49:12 -07:00
James R. Barlow 01dc8e23ff As you were 2023-04-13 23:27:58 -07:00
James R. Barlow b432770cfc Add different timeout control for non-OCR
This is mainly to permit the use of --tesseract-timeout 0 as a way of
disabling OCR but still allowing deskew and orientation detection to work.

The better solution will be introducing an explicit argument to
"don't ocr my pdf", but that will take more rework.
2023-04-13 22:58:12 -07:00
James R. Barlow 5502fb8d9f jobcontext: typo 2023-04-13 16:04:46 -07:00
林博仁(Buo-ren, Lin)andGitHub e66922b030 Fix snap package building (#1082) (#1089)
* Fix snap package building (#1082)

Currently the snap package no longer contains the main Python package
of OCRmyPDF due to the fact that [Snapcraft doesn't run the `pip install .`
command when the setup.py isn't detected in the source tree][1].  This
patch incorporates a workaround from [@cjp256 at the Snapcraft Forum][2]
to generate a setup.py file from the pyproject.toml to make the package
works again.

[1]: [Line 143 of snapcraft/python.py at 0e2cf91 · snapcore/snapcraft · GitHub](https://github.com/snapcore/snapcraft/blob/0e2cf91/snapcraft_legacy/plugins/v2/python.py#L143)
[2]: [Ian Johnson's reply in "Building a core20 python snap using pyproject.toml - snapcraft - snapcraft.io"](https://forum.snapcraft.io/t/building-a-core20-python-snap-using-pyproject-toml/22028/2)

Fixes #1082.

Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>

* Remove unnecessary cloning of own source tree to build the snap package

Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>

---------

Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>
2023-04-13 00:06:19 -07:00
James R. Barlow 00e9759b16 readme: improve 2023-04-12 23:56:50 -07:00
Kian-Meng AngandGitHub ba10c5345b Fix typos (#1087)
Found via `codespell -S tests,LICENSES -L flate`
2023-03-29 21:57:32 -07:00
James R. Barlow 8a5f94988a v14.0.4 release notes 2023-03-17 14:32:04 -07:00
Frederick RobinsonandGitHub aa73e3c69f handle case when candidate is None (#1066) 2023-03-17 14:27:27 -07:00
James R. Barlow 9d5fa05a00 v14.0.3 2023-02-15 12:00:26 -08:00
997380e567 Avoid deleting /dev/null when run as root (#1068)
Co-authored-by: Andrew Wood <andrew@fluidgravity.co.uk>
2023-01-31 16:20:24 -08:00
comzineandGitHub 2685f910b1 watcher: added setting RETRIES_LOADING_FILE to avoid giving up to early (#1063) 2023-01-25 17:36:54 -08:00
Natanael ArndtandGitHub bfcc586032 tesseract-osd is also required on fedora (#1064) 2023-01-25 17:36:23 -08:00
Lucas LarsonandGitHub 2d77b95fd9 substitute broken link (#1057) (#1058)
use `formulae.brew.sh` as seen in other repositories¹ instead of
`brewformulas.org`, which has been offline since 2020,² to fix #1057.

1. https://github.com/search?type=code&q=/shields.io\/homebrew/+language:reStructuredText
2. https://gitlab.com/zedtux/brewformulas.org/issues/223#note_322019672

Signed-off-by: Lucas Larson <LucasLarson@riseup.net>

Signed-off-by: Lucas Larson <LucasLarson@riseup.net>
2023-01-15 13:13:41 -08:00
James R. Barlow f072e91120 v14.0.2 2023-01-03 17:07:53 -08:00
James R. Barlow efa2bca8a3 Github Actions ignores ||, force it to do the right thing 2023-01-03 14:51:09 -08:00
James R. Barlow b039010d3e Ignore brew update errors 2023-01-03 14:36:21 -08:00
James R. Barlow 3f7cefcf5d macos - drop Python 3.10 since homebrew is having trouble 2023-01-02 13:08:42 -08:00
James R. Barlow 45f97d1565 macos build: brew first? 2023-01-02 12:36:28 -08:00
James R. Barlow 1281f8eb68 Workaround 2to3 linking error in macos build 2023-01-02 01:10:20 -08:00
James R. Barlow 9ef61864fb Update Python versions 2023-01-01 19:13:50 -08:00
James R. Barlow 90b2119ad3 pdfinfo: skip null FormX entries 2023-01-01 16:58:14 -08:00
James R. Barlow f0cc7f2230 docs: explicitly end win32 support 2023-01-01 16:58:14 -08:00
M.Yasoob Ullah Khalid ☺andGitHub d60a384aab Fixed the source installation instructions (#1048) 2022-12-22 02:54:53 -08:00
James R. Barlow 14a6093636 Remove old issue templates 2022-12-14 17:33:00 -08:00
James R. Barlow 54b42d73ab Merge remote-tracking branch 'origin/master' 2022-12-14 17:31:56 -08:00
M.Yasoob Ullah Khalid ☺andGitHub 9abed14f1f fixed interchanged words (#1039) 2022-12-14 17:24:55 -08:00
Doug RinckesandGitHub d09f61d4fe log completion message (#1044)
This logs the "done" message if neither delete nor archive options are set.
2022-12-14 17:24:41 -08:00
M.Yasoob Ullah Khalid ☺andGitHub 4a0130649e Fixed some wording (#1045) 2022-12-14 17:24:22 -08:00
jbarlow83andGitHub a0224d94ca Update issue templates 2022-12-14 17:23:57 -08:00
Carlos GarcésandGitHub 9e7b9de830 Fix tesseract documentation url (#1047) 2022-12-14 11:58:21 -08:00
James R. Barlow 08fc5fc01d Fix issue templates - Github doesn't support comments above 2022-10-24 14:55:08 -07:00
James R. Barlow 110c75cba2 Use actual TesseractVersion for comparison 2022-10-03 23:28:36 -07:00
James R. Barlow 5afca3f342 Use relaxed version parsing 2022-10-03 22:44:35 -07:00
James R. Barlow d9eb0ba7ef v14.0.1 release notes 2022-10-03 22:02:50 -07:00
James R. Barlow 46d0978a09 Update version scripts to support Ghostscript 10.0 2022-10-03 21:59:31 -07:00
James R. Barlow 4e35100978 Docker image: add jbig2dec 2022-10-03 20:08:28 -07:00
James R. Barlow 7bd0e43243 Merge branch 'feature/py311' 2022-09-21 02:42:57 -07:00
James R. Barlow 9cd97da5f2 dockerfile: toml 2022-09-21 02:20:28 -07:00
James R. Barlow d002703c41 docker: install gpg-agent 2022-09-21 01:04:34 -07:00
James R. Barlow b1fecf3b05 Temporarily disable PyPy 3.8 and 3.9 testing
Minor pikepdf issue?
2022-09-21 01:00:08 -07:00
James R. Barlow c2ccc7f29d Fix test failure due to new logging from pikepdf 2022-09-21 01:00:08 -07:00
James R. Barlow 36dfd12e2c Enable Python 3.11 testing on Linux 2022-09-21 00:59:44 -07:00
James R. Barlow 7da4e6ca7f Address some linter warnings 2022-09-21 00:05:12 -07:00
James R. Barlow 16fc52079d Drop setup.py - use pyproject.toml exclusively 2022-09-20 23:47:40 -07:00
James R. Barlow f37decf3b3 Remove old pdfminer.six shim 2022-09-20 23:02:17 -07:00
James R. Barlow 4ad4a13ef0 Simplify dependencies 2022-09-20 22:50:44 -07:00
James R. Barlow 6f61f69a8d v14.0.0 release notes 2022-09-20 22:47:07 -07:00
James R. Barlow 88831e8ab1 Merge branch 'feature/tess5' 2022-09-20 22:33:07 -07:00
James R. Barlow 2ebc36fcec Merge branch 'feature/drop-3.7' 2022-09-20 22:32:59 -07:00
James R. Barlow 1709e23701 pipeline: add further typing details 2022-09-19 20:19:15 -07:00
James R. Barlow 2e55cb5980 helpers: improve typing of Resolution 2022-09-19 20:03:54 -07:00
James R. Barlow 6dbaebdc0c Merge branch 'master' into feature/drop-3.7 2022-09-15 23:00:27 -07:00
James R. Barlow 5156fe7662 Test PyPy 3.8 and 3.9 2022-09-15 22:56:44 -07:00
James R. Barlow 2c99c89e45 ghostscript: document why we treat gs 9.56.0 differently 2022-09-15 22:51:04 -07:00
James R. Barlow 74286e7e1e Use tesseract 5 again in Docker image 2022-08-23 13:39:45 -07:00
James R. Barlow 2e937dee9f Refactor cache manifest creation 2022-08-19 00:19:38 -07:00
James R. Barlow 23f3830533 Merge remote-tracking branch 'origin/master' 2022-08-14 17:01:10 -07:00
belamuandGitHub 305e6594be add Fedora packages in jbig2 doc (#1006)
Packages needed to install jbig2 encoding are mentioned for Ubuntu systems but are missing for other systems. This commit adds the infomation in the documentation what the packages are called on Fedora systems.
2022-08-11 10:31:09 -07:00
James R. Barlow f4155dca77 tests: convert all uses of multipage.pdf to fixture 2022-08-11 01:13:10 -07:00
James R. Barlow 545cd031b0 Replace public domain graph.pdf and derivates with licensed version 2022-08-11 01:09:00 -07:00
James R. Barlow a1c7826336 dep5: mention mawi 2022-08-06 15:47:45 -07:00
James R. Barlow c5359bd990 jbig2 is from linn now 2022-08-06 15:36:20 -07:00
James R. Barlow 7f77308846 Remake palette.pdf using baiona-colormapped file 2022-08-06 15:35:47 -07:00
James R. Barlow 8e7c5f3001 dep5: correction, graph.pdf is actually public domain 2022-08-06 15:34:38 -07:00
James R. Barlow 79db985181 Improve encryption tests; drop some public domain resources
Generate the encrypted files we need and remove special test files we retained for this.

Replace jbig2.pdf based on congress.jpg with version based on ccitt.pdf.
2022-08-06 14:37:45 -07:00
James R. Barlow 7d23a661fc Track .reuse/dep5 2022-08-06 14:32:09 -07:00
James R. Barlow 99e94807c5 .gitignore: updates 2022-08-06 14:30:05 -07:00
James R. Barlow 8412de9344 Merge branch 'master' into feature/drop-3.7 2022-08-06 14:04:23 -07:00
James R. Barlow b458b1422b Merge remote-tracking branch 'origin/master' 2022-08-06 03:06:07 -07:00
James R. Barlow 76bd8cab13 Drop Ubuntu 18.04 content 2022-08-06 02:58:26 -07:00
James R. Barlow ef70c9499e Don't deprecate falsy filter_page_image 2022-08-05 01:07:46 -07:00
James R. Barlow 47dcb6fcd0 Revert "Remove deprecated falsy handling of filter_page_image"
This reverts commit c9389c7713.
2022-08-05 01:01:09 -07:00
James R. Barlow 88d2949e6b Abolish setup.cfg and migrate to pyproject.toml 2022-08-04 05:00:46 -07:00
James R. Barlow c9389c7713 Remove deprecated falsy handling of filter_page_image 2022-08-04 04:21:15 -07:00
James R. Barlow 4d2f499f97 Remove optional status of coloredlogs
Everything optional is a possible complication.
Better to remove the option.
2022-08-04 04:15:56 -07:00
James R. Barlow 4104904a1e Document reason for suppress some third party deprecation warnings 2022-08-04 04:13:02 -07:00
James R. Barlow 1a0a797ca6 Remove our @deprecated decorator and use standard package 2022-08-04 04:00:25 -07:00
James R. Barlow 670ce2b969 Remove support for non-callable version checker 2022-08-04 03:58:10 -07:00
James R. Barlow 7e97981114 hocrtx: unused imports 2022-08-04 03:56:44 -07:00
James R. Barlow 53db866ef9 Remove deprecated exception PdfMergeFailedError 2022-08-04 03:54:55 -07:00
James R. Barlow d591a3e059 resources readme: remove license and copyright info
Better to not repeat ourselves and present this info in example one place.
2022-08-04 03:42:22 -07:00
James R. Barlow 37c050aa4f readme: fix copyright 2022-08-04 03:41:45 -07:00
James R. Barlow 4b9ea40a0c spdx: move identifiers to files that support them
If the apparent license changed, take this commit as correct.
2022-08-04 03:26:54 -07:00
林博仁(Buo-ren, Lin)andGitHub 165432486b snap: add JBIG2 support (#1001)
This patch includes a jbig2enc build into the snap package, allow the
user to run OcrMyPDF with the --jbig2-lossy option or other options that trigger JBIG2 encoding.

Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>
2022-08-02 16:35:01 -07:00
James R. Barlow d619fac0bd unpaper: tidy file 2022-08-02 15:30:20 -07:00
James R. Barlow acc70036cc Set minimum Tesseract to 4.1.1 2022-08-02 15:20:29 -07:00
James R. Barlow 80b7cf6330 Update black target versions 2022-08-02 15:07:43 -07:00
James R. Barlow 8a8c06c79c Update pre-commit for py3.8+ 2022-08-02 15:05:43 -07:00
James R. Barlow 67773da309 Drop support for Ghostscript <9.50 2022-08-02 15:01:10 -07:00
James R. Barlow d5a9861d5c readme: freebsd calls it py-ocrmypdf now 2022-08-02 14:49:09 -07:00
James R. Barlow 9ffe829a10 Remove external importlib_metadata since Python 3.8 provides it directly 2022-08-02 14:48:11 -07:00
James R. Barlow 8a3b82e364 Make Python 3.8 minimum requirement 2022-08-02 14:46:01 -07:00
James R. Barlow 580822a6a2 Fix Windows ghostscript path scanning 2022-08-02 14:39:23 -07:00
James R. Barlow 9f3a52fd12 windows: compare Ghostscript versions properly 2022-08-01 13:58:24 -07:00
James R. Barlow 52e829d845 windows: tidy path shimming 2022-08-01 13:23:57 -07:00
James R. Barlow 2b2e5c271a Don't lru_cache versions
lru_cache could interfere with testing and we don't have evidence this
is slow enough to justify optimizing.
2022-08-01 13:23:31 -07:00
James R. Barlow 5fe3102e4e tests: new test to confirm correct printing of tesseract install advice 2022-08-01 12:31:37 -07:00
James R. Barlow 5b57520c98 tests: simplify some validation tests 2022-08-01 12:31:05 -07:00
James R. Barlow 30e4198f3a tests: fix test_validation when chi_sim not installed 2022-08-01 02:47:39 -07:00
James R. Barlow 8ca9fa4ade tesseract: typing 2022-08-01 02:47:21 -07:00
James R. Barlow c1f99c0bd5 v13.7.0 release notes 2022-08-01 02:07:10 -07:00
James R. Barlow ba372e5841 Reorganize validation to fix exception when Tesseract not installed
The existing logic would call an OCR plugin's get_languages function before
allowing the plugin to check if its dependencies were available. This caused
an exception if Tesseract was installed, when we were supposed to issue
an error message advising the user to install Tesseract.
2022-08-01 02:04:09 -07:00
James R. Barlow c7fcbe9075 Improve Linux install advice 2022-08-01 01:59:40 -07:00
James R. Barlow 42c983c86c readme: update to describe SPDX licensing 2022-07-30 17:09:56 -07:00
James R. Barlow 80ed2117cc Change to SPDX license tracking 2022-07-28 01:10:07 -07:00
James R. Barlow d22ea3fd04 release notes: a word 2022-07-26 00:18:03 -07:00
James R. Barlow 21fb6c82ca v13.6.2 release notes 2022-07-25 23:48:22 -07:00
James R. Barlow 27f7b9f255 Fix missing TypeAlias on <3.10 2022-07-25 16:32:55 -07:00
James R. Barlow 6f31a92ffb _windows: Avoid messy next()/StopIteration in generator 2022-07-23 15:40:05 -07:00
James R. Barlow da2276788c Fix _windows module typing 2022-07-23 15:32:14 -07:00
James R. Barlow dc6f1a266a Modernize type annotations 2022-07-23 00:39:24 -07:00
James R. Barlow 9c8ddd853d Typing adjustments 2022-07-23 00:07:50 -07:00
James R. Barlow 014d0302f2 Pre-commit: update 2022-07-22 23:49:51 -07:00
James R. Barlow 65568b3dbc Merge commit '05e2b6698dacce898abc356222124c7a1609f569' 2022-07-18 14:12:01 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
05e2b6698d build(deps): bump docker/setup-buildx-action from 1 to 2 (#994)
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 1 to 2.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v1...v2)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-18 14:10:45 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2f8e0f7d95 build(deps): bump actions/download-artifact from 2 to 3 (#995)
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 2 to 3.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-18 14:10:33 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
7e7553fc6b build(deps): bump actions/setup-python from 2 to 4 (#996)
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 2 to 4.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v2...v4)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-18 14:10:19 -07:00
James R. BarlowandH. Felix Wittmann 6b425aaebe Add shim for cancel_futures in older Pythons
Thanks @hfwittmann

Closes #993

Co-authored-by: H. Felix Wittmann <hfwittmann@users.noreply.github.com>
2022-07-17 16:02:45 -07:00
James R. Barlow 725af43bc3 docs: fix badges for debian 2022-07-12 14:56:00 -07:00
James R. Barlow 5c60309609 Merge remote-tracking branch 'origin/master' 2022-07-12 02:23:17 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
7d5cd55909 build(deps): bump codecov/codecov-action from 1 to 3 (#990)
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 1 to 3.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v1...v3)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-12 02:23:02 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
ec4a06fad2 build(deps): bump docker/login-action from 1 to 2 (#988)
Bumps [docker/login-action](https://github.com/docker/login-action) from 1 to 2.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v1...v2)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-12 02:22:53 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
48c6e2318e build(deps): bump docker/setup-qemu-action from 1 to 2 (#987)
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 1 to 2.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v1...v2)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-12 02:22:38 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2eafa5e070 build(deps): bump actions/upload-artifact from 2 to 3 (#989)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 2 to 3.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-12 02:21:27 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
6aa04d7569 build(deps): bump actions/checkout from 2 to 3 (#991)
Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-12 02:21:12 -07:00
James R. Barlow b9bffa97ba Fix ABCMeta typing 2022-07-12 02:20:22 -07:00
James R. Barlow 777ba99ccc v13.6.1 release notes 2022-07-12 02:11:26 -07:00
James R. Barlow 001b3324f1 Require setuptools-scm 7.0.5 to ensure sdists work alright 2022-07-12 02:09:37 -07:00
James R. Barlow b1f2d257e2 Typing improvements 2022-07-09 02:24:19 -07:00
James R. Barlow 24a08e5170 hocrtransform: suppress deprecation warning from importing reportlab 2022-07-07 02:09:47 -07:00
James R. Barlow adf97fd82c pipeline: eliminate useless test 2022-07-07 02:08:14 -07:00
James R. Barlow a60ea72517 docs: improve remarks about lossy JBIG2 and lossy image transformations 2022-07-04 23:01:20 -07:00
James R. Barlow 59f967cdcd Activate GHA dependabot 2022-07-04 22:43:19 -07:00
James R. Barlow 6e439ee89e Modernize setuptools usage and setuptools_scm 2022-07-04 02:26:14 -07:00
James R. Barlow da38e1b035 docs: adding missing plugins 2022-07-04 02:16:20 -07:00
James R. Barlow 28c60c4f82 v13.6.0 release notes 2022-07-03 15:35:22 -07:00
James R. Barlow a5efc4af9b unpaper: replace input pnm with png
Unpaper or its underlying libraries don't seem to accept pnms with an
odd integer width. Although it's not clear if this is the issue at all.

In any case, keeping the image a PNG works around the issue. unpaper
only accepted PNM input in the past, which is why we send it PNM.
Since it now accepts PNG, we might as well use PNG.

Unpaper can write PNG as output too, but this added a few seconds to
the test suite was not committed.

Related issues:

https://github.com/ocrmypdf/OCRmyPDF/issues/887

https://github.com/ocrmypdf/OCRmyPDF/issues/665

https://github.com/unpaper/unpaper/issues/82
2022-07-03 15:32:16 -07:00
James R. Barlow 1141235c42 Merge remote-tracking branch 'origin/master' 2022-06-24 01:05:08 -07:00
2b6b7a4975 Update README to show nix package manager install option (#904)
ocrmypdf now works on M1 Mac (aarch64-darwin)

Co-authored-by: xave <xavieking@gmail.com>
2022-06-19 01:47:10 -07:00
James R. Barlow b062c9e8c0 pluginspec: hint that optimize may need to implement initialize
[ci skip]
2022-06-19 01:35:30 -07:00
James R. Barlow ed632ae366 docs: update batch to avoid suggesting Docker volumes
[ci skip]
2022-06-19 01:01:39 -07:00
James R. Barlow af742229e7 docs: fix sentence fragment in batch page
Closes #980

[ci skip]
2022-06-19 00:41:58 -07:00
Alexander JaustandGitHub e2d998245d Fix type in cookbook.rst (#978)
Add missing dash in warning about `--clean-final` and `--remove-background` commands.
2022-06-19 00:32:35 -07:00
James R. Barlow 8c58e95c3a Add new initialize hookspec to make suppress plugins easier 2022-06-19 00:31:42 -07:00
James R. Barlow 61600111d3 test_pdfinfo: refactor by extracting fixtures 2022-06-18 16:29:57 -07:00
James R. Barlow e4c45e3d3b Plugins should ideally not import from ocrmypdf._* 2022-06-18 16:03:55 -07:00
Julius BullingerandGitHub 7cabbb125f watcher: Add an option to archive processed originals (#951)
* watcher: Add an option to archive processed originals

This adds a feature from existing OCRmyPDF watchdog Docker containers like meyay/ocrmypdf-batch and unze/ocrmypdf-watchdog. With this option, the input directory can be kept clean from already processed files, without losing the originals.

* docs: Improve watcher.py's Docker parameters documentation
2022-06-17 15:17:03 -07:00
James R. Barlow d8753dc790 Fix Windows issue if no exception 2022-06-13 01:46:59 -07:00
James R. Barlow ef43d7e016 v13.5.0 release notes 2022-06-13 01:30:27 -07:00
James R. Barlow 17a5b8b43c Refactor reporting of optimization failures 2022-06-13 01:30:15 -07:00
James R. Barlow 13d11e76e5 optimize plugin: solve linearization and "is optimization enabled?" issues 2022-06-13 00:59:41 -07:00
James R. Barlow 61069660a2 Move optimization options to plugin 2022-06-12 02:42:16 -07:00
James R. Barlow 685a06c93d Move optimize into a builtin plugin 2022-06-12 02:23:13 -07:00
James R. Barlow 6cdf68363a docs: copyright year 2022-06-12 00:31:01 -07:00
James R. Barlow 522ff3c21a Remove major version pins 2022-06-12 00:31:01 -07:00
James R. Barlow 10245dc954 Refactor a helper to use the Python API eventually 2022-06-12 00:31:01 -07:00
James R. Barlow 3d4f80639d Remove test that is now always skipped 2022-06-12 00:31:01 -07:00
James R. Barlow db9a22c9dd api: call enable_ansi_support unconditionally
coloredlogs documentation says this is (now?) permitted.

Also import it from the right place.
2022-06-12 00:31:01 -07:00
James R. Barlow 31683530f8 api/cli: replace private variable access with method 2022-06-12 00:31:01 -07:00
James R. Barlow 0e550a1c6d graft: use pikepdf's random name API instead of our own
Minor change to PDF output.
2022-06-12 00:31:01 -07:00
James R. Barlow b17fb61389 Configure pylint in pyproject and delint 2022-06-12 00:30:44 -07:00
James R. Barlow d640c2ded3 Tidy some if condition -> EAFP 2022-06-11 00:21:21 -07:00
James R. Barlow a0ac448d52 Tidy some old-school %s strings 2022-06-11 00:15:19 -07:00
James R. Barlow e3ba13e365 docs: update installation notes
Add Snap. Remove Mageia because I have no idea if it still works. Drop a
lot of old versions and old notes. Change pip3->pip except for old py2/py3 images.
Try to make things less version sensitive.
2022-06-10 01:33:05 -07:00
James R. Barlow 0cd04abc4e snap: add logo 2022-06-09 16:59:20 -07:00
Alexander LangankeandJames R. Barlow ee81f3968f Add snapcraft.yaml 2022-06-08 01:22:18 -07:00
James R. Barlow 21cacad93b unpaper: super syntax 2022-06-02 17:05:59 -07:00
James R. Barlow 3589f4e7d1 unpaper: fix unsaved change 2022-06-02 02:22:48 -07:00
James R. Barlow 1cdc2591e5 v13.4.7 release notes 2022-06-02 02:00:16 -07:00
James R. Barlow e05f9575a8 Merge remote-tracking branch 'origin/master' 2022-06-02 01:57:39 -07:00
James R. Barlow 10c703e119 unpaper: use TemporaryDirectory(ignore_cleanup_errors=True) where available
Fixes #974 when used in conjunction with Python 3.10.

Reviewed other uses of TemporaryDirectory in ocrmypdf and decided it
was not worth fixing them since neither are exactly production code.
2022-06-02 01:56:41 -07:00
James R. Barlow 0ac15dd0b2 Suppress libxmp DeprecationWarning during test 2022-06-01 00:46:16 -07:00
Robert SchützandGitHub 808b24d59f ignore PermissionError when calling os.nice() (#973) 2022-05-28 18:07:54 -07:00
James R. Barlow c082526dea Test pypy-3.8 instead of 3.7 2022-05-26 13:52:56 -07:00
James R. Barlow 33cdabaf65 tests: account for test that expected pngquant for windows 2022-05-26 13:52:22 -07:00
James R. Barlow 94f8e36601 ci: don't install pngquant for windows anymore 2022-05-26 13:01:58 -07:00
James R. Barlow 865002c7be v13.4.6 release notes 2022-05-26 00:59:14 -07:00
James R. Barlow 5d0cc0a092 tests: Extract some test fixtures for better clarity 2022-05-26 00:57:31 -07:00
James R. Barlow 6c427f82ea Add test case for corrupt ICC profiles 2022-05-26 00:41:19 -07:00
James R. Barlow e7a44ba87a info: adjust ICC warning message 2022-05-26 00:21:31 -07:00
James R. Barlow c311768452 Merge branch 'corrupt-icc' of https://github.com/oscherler/OCRmyPDF into oscherler-corrupt-icc 2022-05-26 00:14:47 -07:00
James R. Barlow f53fedee63 pre-commit: autoupdate 2022-05-25 15:21:55 -07:00
James R. Barlow 87838127b0 info: replace introspection with explicit f-string 2022-05-25 13:16:40 -07:00
Olivier Scherler 4db4df5c72 Log a warning instead of failing on images with a corrupt ICC profile. 2022-05-25 12:37:21 +02:00
James R. Barlow 11125c5367 v13.4.5 release notes 2022-05-24 16:52:43 -07:00
James R. Barlow e648411067 Remove pdfminer.six upper version restriction
Needing to update this every time has become more inconvenient than dealing
with the occasional breakage in a new release.
2022-05-24 16:51:38 -07:00
Ben BeasleyandGitHub 11365575d7 Allow pdfminer.six 20220524 (#971) 2022-05-24 16:33:44 -07:00
James R. Barlow 845cb5c40c docs: clarify that --pages and --skip-text exclusions apply to image processing and OCR
Closes #950
2022-05-16 13:20:47 -07:00
James R. Barlow b699e158be Fix references to old repo at jbarlow83/OCRmyPDF 2022-05-16 12:48:10 -07:00
leonnicolasandGitHub 603da52026 Fix small typo in update api.py (#963)
[ci skip]
2022-05-14 21:45:23 -07:00
James R. Barlow 8d0765a5e0 v13.4.4 release notes 2022-05-14 14:53:03 -07:00
James R. Barlow 1ca327e13b Update pdfminer.six version 2022-05-14 14:51:59 -07:00
James R. Barlow f504fd1875 Change Docker base to 22.04 2022-04-28 22:46:30 -07:00
James R. Barlow cf7c20ca16 v13.4.3 release notes 2022-04-14 20:19:31 -07:00
James R. Barlow b00fe3dc5d pytest.skip() - remove kwarg entirely, to avoid breaking older pytest and not getting warns from newer pytest 2022-04-14 20:15:00 -07:00
James R. Barlow e6aa3a4299 tests: explain why CacheOcrEngine needs lock 2022-04-05 16:16:51 -07:00
James R. Barlow 24f1b57288 Merge branch 'master' of github.com:ocrmypdf/OCRmyPDF 2022-04-05 16:04:45 -07:00
James R. Barlow 43302d7e12 Fix pytest.warns() on older pytest
Thanks @QuLogic
2022-04-05 16:02:50 -07:00
Christopher BeschandGitHub fed0226761 Add Gentoo Language Installation Instructions (#936) 2022-04-05 00:26:24 -07:00
Joseph MorrisandGitHub 27e22b4f07 Update jbig2.rst (#911)
Adding Ubuntu package names for dependencies I needed to save others time. The tequired package for leptonica is particularly confusing since configure just says "Error! Leptonica not detected." and there are multiple leptonica packages
2022-04-05 00:25:28 -07:00
James R. Barlow 79382a6039 Docker: Change base image to debian
Debian has latest qpdf.
2022-04-04 00:40:03 -07:00
James R. Barlow 7788d94c4a v13.4.2 release notes, take 2 2022-04-03 23:58:37 -07:00
James R. Barlow 33bfba8449 ghostscript: match actual reported version 2022-04-03 23:35:30 -07:00
James R. Barlow 1d0584c644 Add advisory note to release notes 2022-04-03 23:04:55 -07:00
James Barlow 84b9d4d021 Speculation: Ghostscript 9.56 new PDF interpreter breaks things 2022-04-03 21:14:19 -07:00
James Barlow 41efd3bf0f Fix Python "3.10" 2022-04-03 21:13:59 -07:00
James Barlow 776ada6713 Upgrade pre-commit and associated tools; various lints 2022-04-03 20:53:01 -07:00
James Barlow f3593c915d ci: test Python 3.10 2022-04-03 19:06:50 -07:00
James Barlow dfe31a2f6d Add lock to certain "with patch" cases
Switch to --use-threads seems to have broken tests that assumed they could
monkeypatch things. Although that's odd, since while we can have multiple
worker threads, we should never have
parallel tests in the same process.
2022-04-03 17:22:04 -07:00
James Barlow 0c43963d69 Fix pytest deprecation warnings 2022-04-03 13:30:58 -07:00
James Barlow f29fe7f23e Fix Pillow deprecation warnings 2022-04-03 13:30:50 -07:00
James R. Barlow 04996caac3 pdfminer 20220319 2022-03-22 22:36:06 -07:00
James R. Barlow 13917c051c Disable oom killer test for --use-threads 2022-03-13 01:02:28 -08:00
James R. Barlow 8182fe9c92 Make --use-threads default and update release notes
Re: issue Hanging on Random Files  #814
2022-03-13 00:37:06 -08:00
James R. Barlow 1950acfbda docs: proofread plugins 2022-02-20 00:56:43 -08:00
James R. Barlow fca6403083 Disallow pikepdf 5.0.0 2022-02-19 01:45:36 -08:00
James R. Barlow c4e2fce1ef v13.4.0 release notes (2) 2022-02-18 23:37:33 -08:00
James R. Barlow 3546479658 optimize: don't deflate JPEGs with fancy DecodeParms settings
This is overly cautious but will do for now.
2022-02-18 23:36:53 -08:00
James R. Barlow 72442fa3d0 Fix error messages when run with pikepdf 5.0.0
Appears that these are spurious errors from qpdf probing the /DecodeParms
dict on images that don't have it.

Not sure exactly why it happens but this can be safely ignored.
2022-02-18 23:35:44 -08:00
James R. Barlow 8f714b1375 v13.4.0 release notes 2022-02-18 23:08:48 -08:00
James R. Barlow cb05c1d122 pdfinfo: a few annotations 2022-02-14 23:31:53 -08:00
James R. Barlow b0ad07bc5f Merge branch 'master' of github.com:ocrmypdf/OCRmyPDF 2022-02-08 00:46:26 -08:00
James R. Barlow 514038d4ec optimize: recognize and produce [/FlateDecode /DCTDecode] images 2022-02-08 00:38:08 -08:00
James R. Barlow 50d76e7f6c optimize: remove comment about issue in Pillow that is now fixed 2022-02-07 22:18:20 -08:00
James R. Barlow 6c78a46285 optimize: remove inaccurate about ICCs
pikepdf will now get the ICC profile out and put it in the JPEG.
2022-02-07 19:42:13 -08:00
James R. Barlow 863d560632 optimize: clarify log message about skipping images with multiple filters 2022-02-06 22:41:10 -08:00
James R. Barlow 73934c854c release notes: typo 2022-02-06 22:39:32 -08:00
rdiezandGitHub 2be8eeec2c Fix spelling of 'ephemeral' (#908) 2022-02-03 10:33:51 -08:00
James R. Barlow 3dfde479e2 The world is not ready for := 2022-01-26 00:16:51 -08:00
James R. Barlow aea1862644 v13.3.0 release notes 2022-01-25 23:50:48 -08:00
James R. Barlow 3b406112d0 ghostscript: improve test coverage of error cases 2022-01-25 23:45:47 -08:00
James R. Barlow fcc4c2d371 ghostscript: improve error message if image cannot be opened 2022-01-25 23:12:51 -08:00
James R. Barlow 3de18ed612 tesseract: account for more tesseract 5 output differences 2022-01-25 00:19:49 -08:00
James R. Barlow 93cca42e20 optimize: don't try to optimize an image we can't save 2022-01-25 00:19:23 -08:00
James R. Barlow 2d0ac4707c Use better img2pdf settings where possible while supporting old versions
Fixes #894
2022-01-14 11:55:54 -08:00
James R. Barlow 7d208175cf unpaper: refactoring 2022-01-11 10:57:54 -08:00
James R. Barlow ea69e868ed unpaper: issue warning if image too large to clean 2022-01-11 10:44:38 -08:00
James R. Barlow beea603ab3 Revert "docs: add sphinx-panels"
This reverts commit 7966192d6e.
2022-01-04 11:44:20 -08:00
James R. Barlow 7966192d6e docs: add sphinx-panels 2022-01-04 11:43:28 -08:00
Anton GladkyandGitHub 5acbd7a252 Wrap exception on non-CMYK images into the log warning (#881) 2021-12-29 01:15:49 -08:00
Krasimir NedelchevandGitHub aed955ca8c Fix typo (#882) 2021-12-27 15:36:19 -08:00
James R. Barlow 298bdb8690 v13.2.0 release notes 2021-12-19 11:40:23 -08:00
James R. Barlow 1a58abcc6a concurrency: fix extra update of progressbar 2021-12-19 11:34:09 -08:00
James R. Barlow dbfceba020 Standardize ghostscript version default 2021-12-18 02:01:42 -08:00
James R. Barlow 0faa618c3c Replace deprecated distutils with packaging.version 2021-12-18 01:57:59 -08:00
James R. Barlow 7035002c03 Order of operations suppressed detailed Ghostscript missing error message 2021-12-18 01:27:39 -08:00
James R. Barlow f8fadaef41 _windows: remove use of deprecated distutils 2021-12-13 20:45:54 -08:00
James R. Barlow ee21bf9ef6 Update cache 2021-12-13 20:45:30 -08:00
James R. Barlow 190ca81951 v13.1.1 release notes 2021-12-10 21:49:04 -08:00
James R. Barlow d48254d477 Fix issue with attempting to deskew a blank page on Tesseract 5
Closes #868
2021-12-10 21:48:09 -08:00
James R. Barlow 1ec2ccca14 docs: add warning about multiproc on macOS 2021-12-10 17:42:50 -08:00
James R. Barlow e78f0cc56f v13.1.0 release notes 2021-12-06 20:00:53 -08:00
James R. Barlow 13af3252ff tests: simplify run_ocrmypdf API 2021-12-06 17:00:25 -08:00
James R. Barlow 0528867e0b config: yaml strings for versions 2021-12-06 16:59:44 -08:00
James R. Barlow 6910c48b81 Fix test_outputtype_none on Windows and cleanup docs 2021-12-06 15:38:38 -08:00
James R. Barlow 69aa3981c4 docs: Remove reference to long removed 'tesseract' renderer 2021-12-06 15:38:38 -08:00
James R. Barlow 9c1e5adfe6 docs: remove Ubuntu 16.04 install instructions
It's EOL.
2021-12-06 15:38:38 -08:00
James R. Barlow e642dd4b35 Fix kill signal on Windows 2021-12-06 15:38:32 -08:00
James R. Barlow 9de06f62ee Use Python executors instead of pools
ProcessPool/ThreadPool don't have the ability to notice when a child worker
was terminated. ProcessPoolExecutor and ThreadPoolExecutor do notice and
provide better error messages.

Add tests to check.
2021-12-06 15:38:27 -08:00
James R. Barlow 1414a8f5dc Tidy pyproject.toml 2021-12-06 15:38:27 -08:00
James R. Barlow 26badf2882 typing: small improvements 2021-12-06 15:38:27 -08:00
James R. Barlow 8f873aaa45 sync: typing improvements 2021-12-06 15:38:27 -08:00
James R. Barlow 8fdcb15b4e tests: improve typing and remove some legacy code 2021-12-06 15:38:27 -08:00
James R. Barlow 0323738ada ocrmypdf.fish: fix indents
[ci skip]
2021-12-06 15:38:27 -08:00
FPilleandJames R. Barlow aae5591f7e Update ocrmypdf.bash completion
Squashed commit of the following:

commit 974de2e8ccad7fd34694f2c3a7a17c64bb52cdab
Merge: a8d7f969 ee04aa72
Author: James R. Barlow <james@purplerock.ca>
Date:   Sat Dec 4 20:22:50 2021 -0800

    Merge branch 'update_bash-completion' of git://github.com/FPille/OCRmyPDF into FPille-update_bash-completion

commit ee04aa7225
Author: FPille <f.pille@gmail.com>
Date:   Thu Oct 14 11:09:23 2021 +0200

    update

commit 76f64537aa
Author: FPille <f.pille@gmail.com>
Date:   Thu Oct 14 11:04:10 2021 +0200

    updated and descriptions for arguments and choices added
    deprecated arguments removed
    bug fix: typo "_init_completion" instead of "_init_completions"

commit de9b93e852
Merge: c23374de 42713b77
Author: Frank <50119297+FPille@users.noreply.github.com>
Date:   Thu Oct 14 08:08:11 2021 +0200

    Merge branch 'jbarlow83:master' into master

commit c23374de81
Merge: 40b2ebcb c409fa58
Author: Frank <50119297+FPille@users.noreply.github.com>
Date:   Wed May 26 20:31:00 2021 +0200

    Merge branch 'jbarlow83:master' into master

commit 40b2ebcb37
Merge: 79c84eef 7e388f59
Author: Frank <50119297+FPille@users.noreply.github.com>
Date:   Sat Jun 1 11:09:07 2019 +0200

    Merge pull request #1 from jbarlow83/master

    update master
2021-12-06 15:38:26 -08:00
James R. Barlow 4c1ff1086c tess cache: don't include full platform - could be sensitive 2021-12-06 15:38:26 -08:00
James R. Barlow f91faf9795 Add new argument --tesseract-thresholding to control tesseract thresholding where available
Also add missing test for --tesseract-oem
2021-12-06 15:38:14 -08:00
James R. Barlow 793cc33a90 Whitespace 2021-12-04 16:07:34 -08:00
James R. Barlow fbd72efd45 build: typo 2021-12-04 01:41:31 -08:00
James R. Barlow 1115923995 build: address checksum error from choco 2021-12-04 01:26:38 -08:00
James R. Barlow 8478d67b28 Merge branch 'release/v13' of github.com:jbarlow83/OCRmyPDF into release/v13 2021-11-15 16:38:11 -08:00
James R. Barlow c75ff4687a Turning on Ghostscript interpolation changes this test
Seems acceptable. We don't normally use Ghostscript to downsample PDFs
like is happening in this test.
2021-11-15 16:36:24 -08:00
mara004andGitHub 312c1e51b5 [ci skip] minor corrections to maintainers.rst (#858) 2021-11-15 15:13:12 -08:00
James R. Barlow cfe2bb25ba Merge commit 'cd49e70154f82f54bf74fc5bb2586fe7e0358971' into release/v13 2021-11-15 00:33:34 -08:00
Tristan PorteriesandGitHub cd49e70154 ghostscript: force interpolation when rendering (#855)
Specifying option --oversample tends to introduce upsampling in rendering
by rasterizing page to an higher DPI.

This upsampling improves OCR results, but a correct choice of interpolation
method can increase even more the OCR quality.

Ghostscript seems to use a nearest interpolation as default choice for pdf.
This method doesn't average new introduced pixels with original pixels
resulting in an almost similar image but with more pixels.

Providing -dInterpolateControl=-1 force switching interpolation on.

In this commit the above option is passed to all ghostscript rendering
calls.

After testing, rendering a page at same DPI with interpolation
enabled does not introduce significant time overhead.

time (repeat 40 gs -dQUIET -dSAFER -dBATCH -dNOPAUSE -sDEVICE=png16m \
	-dFirstPage=1 -dLastPage=1 -r100.000000x100.000000 \
	-dInterpolateControl=-1 -o /dev/null -dAutoRotatePages=/None -f pzII.pdf)
7,66s user 0,33s system 99% cpu 8,012 total

time (repeat 40 gs -dQUIET -dSAFER -dBATCH -dNOPAUSE -sDEVICE=png16m \
	-dFirstPage=1 -dLastPage=1 -r100.000000x100.000000 \
        -o /dev/null -dAutoRotatePages=/None -f pzII.pdf)
7,42s user 0,39s system 99% cpu 7,808 total

Ghostscript interpolation control reference:
https://www.ghostscript.com/doc/current/Use.htm
2021-11-15 00:32:58 -08:00
James R. Barlow 7ce1692eef windows: default version to '0' when looking for Ghostscript
To avoid ValueError: max() arg is an empty sequence

As suggested by @meet1919 in #833.
2021-11-14 23:00:08 -08:00
James R. Barlow 7959f7628d pyproject: tell black to target py37 2021-11-14 15:49:01 -08:00
James R. Barlow 4634b20de5 Raise max-image-mpixels again
PDFs are quite likely to have a lot of pixels, e.g. large high resolution scans.
250 MP is a page of A0 sized paper scanned at 400 DPI,
should be enough in most cases.
2021-11-14 15:47:39 -08:00
James R. Barlow 3810e576ff optimize: fix mypy lint 2021-11-13 14:48:00 -08:00
James R. Barlow 01c7895044 pipeline: tidy 2021-11-13 14:47:49 -08:00
James R. Barlow fdc6aa03fb docs: new maintainer notes 2021-11-13 14:29:30 -08:00
James R. Barlow 25cc17ee03 v13 release notes (2) 2021-11-13 02:02:04 -08:00
James R. Barlow e8098a1475 Dockerfile: remove requirements/ 2021-11-13 01:57:17 -08:00
James R. Barlow 6b773883dc build: use latest pip and wheel in all cases 2021-11-13 01:57:03 -08:00
James R. Barlow 4ed9622335 v13 release notes 2021-11-13 01:37:38 -08:00
James R. Barlow acc9d58c39 Skip no language test for Tess 5 2021-11-13 01:37:27 -08:00
James R. Barlow 659e738f92 Remove some 'liblept' references we no longer need 2021-11-13 01:22:09 -08:00
James R. Barlow 7b3d7ca92a ghostscript: choco doesn't put Ghostscript on PATH anymore
It seems that chocolately doesn't put gswin[32,64]c on PATH anymore,
so compensate.
2021-11-13 01:18:12 -08:00
James R. Barlow e3126d2806 Adjust test to support Tesseract 5 working harder to find its files 2021-11-13 01:16:35 -08:00
James R. Barlow 45020a7fcd build: tweak CI 2021-11-13 00:56:49 -08:00
James R. Barlow f51164aff8 Upgrade test version of pymupdf 2021-11-13 00:53:41 -08:00
James R. Barlow 6f58a14351 pdfa: remove deprecated pkg_resources based access and tests 2021-11-13 00:52:03 -08:00
James R. Barlow 7ba04267b1 Remove shims to support for old versions of pikepdf < 4 2021-11-13 00:43:20 -08:00
James R. Barlow 9749564313 Remove requirements/*.txt - use pip install ocrmypdf[etc] instead 2021-11-13 00:31:42 -08:00
James R. Barlow 698e8791d7 Remove Python 3.6 specific unicode environment checks 2021-11-13 00:28:52 -08:00
James R. Barlow 380b981763 Remove most Python 3.6 special casing 2021-11-13 00:27:48 -08:00
James R. Barlow 5abfb14c2a Remove leptonica and cffi 2021-11-13 00:06:35 -08:00
James R. Barlow 036afc4d88 Update cache, related to previous apparently 2021-11-12 23:57:50 -08:00
James R. Barlow 59642a98b2 Disable --remove-background so we can remove leptonica 2021-11-12 23:56:52 -08:00
James R. Barlow f8c6be2e26 test_rotation: replace leptonica test with Pillow channel ops
New function is likely not as robust but seems capable of inexact image comparison.
2021-11-12 23:49:38 -08:00
James R. Barlow 42bf5476dd optimize: replace leptonica compdata with direct insert of JPEG
Confirmed that img2pdf just inserts JPEG verbatim. Never had to go through
the trouble we did.
2021-11-12 23:20:49 -08:00
James R. Barlow 30440104ba Remove --threshold argument
Tesseract is now included better thresholding (binarization) in v5. Users that have
thresholding issues should try that first. If we find further problems
this can be brought back as a plugin.
2021-11-12 20:09:55 -08:00
James R. Barlow b159e02110 Convert deskew to use degrees, since all our other angles are in degrees 2021-11-12 16:40:51 -08:00
James R. Barlow a55ab05d16 Replace leptonica deskew with tesseract find skew and pillow rotate
Also rebuild the cache.
2021-11-12 16:35:08 -08:00
James R. Barlow 25d046ae95 Modernize OrientationConfidence definition 2021-11-10 00:55:23 -08:00
James R. Barlow 01b0f76e36 Remove duplicate definition of OrientationConfidence 2021-11-10 00:55:07 -08:00
James R. Barlow d74d315e8b docs: update some old OS versions 2021-11-10 00:30:48 -08:00
James R. Barlow 8be9a68c5e v12.7.2 release notes 2021-11-04 00:20:25 -07:00
James R. Barlow 6c34d59836 tesseract: yet another version variant 2021-11-04 00:17:18 -07:00
James R. Barlow 386453d178 pdfa: replace read_binary() with files() 2021-10-31 02:01:11 -07:00
James R. Barlow 615a7561b5 tesseract: tidy some uses of str paths instead of Path 2021-10-31 02:01:05 -07:00
James R. Barlow c4c64c3ea0 pre-commit updates 2021-10-31 01:31:57 -07:00
James R. Barlow 21279f5784 Fix leaked file handle for output_type none 2021-10-28 02:50:17 -07:00
James R. Barlow a63a21a7fc helpers: remove shim for unsupported old version of pikepdf 2021-10-28 02:46:34 -07:00
James R. Barlow 1c4d5d79f7 Improve some error messages 2021-10-28 01:16:22 -07:00
James R. Barlow 644581ed3c v12.7.1 release notes 2021-10-27 01:20:06 -07:00
James R. Barlow 77f7621bbc batch.py: tidy 2021-10-15 15:03:40 -07:00
James R. Barlow 42713b77d7 v12.7.0 release notes 2021-10-12 13:39:49 -07:00
James R. Barlow 690f88119d Fix test failures on pikepdf 3.2.0 + pybind11 2.8.0
When compiled without pybind11 2.8.0, pikepdf supplies a shim to implement
pikepdf._ObjectMapping.values() which has subtly different semantics
from a true dict-like objects; in particular it supports
next(objectmap.values())
where a standard dict requires
next(iter(objectmap.values()).

pybind11 2.8.0 now implements .values() properly, meaning some misuses of
protocol  in ocrmypdf fail.

If pybind11 < 2.8.0, pikepdf will
continue to offer its shim. If pybind11 >= 2.8.0, pikepdf does not add its shim.

Consequently no changes were needed in pikepdf.

Closes #843
2021-10-12 13:38:52 -07:00
James R. Barlow 78f391536b Offer hint to user to use --max-image-mpixels after decompression bob error
Closes #801
2021-10-06 00:19:11 -07:00
mara004andGitHub 7bdd1828a9 [ci skip] docs/conf.py: add intersphinx mapping to make external links work (#838) 2021-10-04 00:34:59 -07:00
mara004andGitHub a8f513eeeb [ci skip] Update api.rst (#839) 2021-10-04 00:34:39 -07:00
fedeliallalineaandGitHub af18bc0684 fixs importlib.{metadata,resource} for new python version (#840)
Signed-off-by: Marco Genasci <fedeliallalinea@gmail.com>
2021-10-03 23:30:11 -07:00
James R. Barlow b621df6947 v12.6.0 release notes 2021-10-02 01:02:17 -07:00
James R. Barlow 313c9e7dc1 docs: add missing sphinx extensions 2021-09-26 23:41:25 -07:00
James R. Barlow 9d04795f7f docs: fix package version error 2021-09-26 23:41:17 -07:00
James R. Barlow 9a08e71e7f docs: show logo 2021-09-26 23:40:54 -07:00
James R. Barlow 790d3022f6 Implement --output-type=none to skip producing the PDF and use only the sidecar
Closes #787
2021-09-26 01:07:34 -07:00
James R. Barlow ec311af796 typing: subprocess 2021-09-22 17:18:59 -07:00
James R. Barlow c725bf79da flake8 delinting 2021-09-21 16:37:03 -07:00
James R. Barlow 9559f76fae optimize: fix typo in debug msg 2021-09-19 16:31:00 -07:00
James R. Barlow 45736b7c2b cli: clarify text to more accurately describe behavior of --jbig2-lossy 2021-09-16 16:03:39 -07:00
James R. Barlow 5629e960b9 v12.5.0 release notes update 2021-09-15 00:27:32 -07:00
James R. Barlow 79fd8d01a5 info: fix incorrect handling of inline images and other typing fixes 2021-09-15 00:26:15 -07:00
James R. Barlow 79fe7a0a85 graft: remove separate implementation of unparse_content_stream 2021-09-15 00:25:06 -07:00
James R. Barlow b4b32a35b5 optimize: fix typing consistency 2021-09-15 00:09:08 -07:00
James R. Barlow 4634b3db55 v12.5.0 release notes 2021-09-14 17:26:07 -07:00
James R. Barlow f5053158d4 hocrtransform: fix regression causing hocr text to be not rendered
Fixes #828
2021-09-14 17:23:09 -07:00
James R. Barlow dfa4ce1612 pyproject: configure mypy 2021-09-14 17:22:41 -07:00
James R. Barlow 585595a98e pre-commit: autoupdate 2021-09-14 17:22:29 -07:00
James R. Barlow f6396fbaac pre-commit: drop isort mirror 2021-09-14 17:20:43 -07:00
James R. Barlow 3859bae85e Merge branch 'feature/drop-pkg-resources' 2021-09-14 00:31:36 -07:00
James R. Barlow ee1a7baae7 requirements: drop setuptools
With importlib_* we no longer setuptools's pkg_resources.
2021-09-14 00:28:42 -07:00
James R. Barlow a4da05b66b docs: various fixes
As suggested by @Chealer

Closes #829, #830, #831, #832
2021-09-14 00:24:18 -07:00
James R. Barlow 4d67812d51 importlib helpers don't provide importlib.thing, but importlib_thing
Fix everywhere.
2021-09-14 00:15:07 -07:00
James R. Barlow 3534742ef9 ci: Enable build of feature branches 2021-09-13 01:35:14 -07:00
James R. Barlow 8bfd46c80d docs: update copyright 2021-09-13 01:10:49 -07:00
James R. Barlow cc6e9cecc0 Replace pkg_resources version lookup with importlib.metadata 2021-09-13 01:10:49 -07:00
James R. Barlow 208657f840 pdfa: replace pkg_resources with importlib.resources 2021-09-13 01:10:49 -07:00
James R. Barlow f3de980447 Introduce importlib-resources,metadata backports for Python < 3.9 2021-09-13 01:10:43 -07:00
James R. Barlow eb8992e58b Update release notes 2021-09-09 15:53:08 -07:00
James R. Barlow 72ad618ae6 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2021-09-09 15:25:54 -07:00
James R. Barlow f07d0c39bb Require pikepdf<3 on PyPy 3.6
Because cibuildwheel does not build wheels for PyPy 3.6 anymore,
so pikepdf does not offer one.
2021-09-09 15:25:38 -07:00
James R. Barlow 9c5c7d9be0 release notes: mention Py3.6 EOL 2021-09-08 00:04:21 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
0b19b084e2 build(deps): bump pillow from 8.2.0 to 8.3.2 in /requirements/main.txt (#825)
Bumps [pillow](https://github.com/python-pillow/Pillow) from 8.2.0 to 8.3.2.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/master/CHANGES.rst)
- [Commits](https://github.com/python-pillow/Pillow/compare/8.2.0...8.3.2)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-09-08 00:03:35 -07:00
James R. Barlow 9b4516af7a ci : add packages for PyPy build 2021-08-31 02:44:07 -07:00
James R. Barlow 1eb45de5c9 v12.4.0 release notes 2021-08-31 02:35:39 -07:00
James R. Barlow 390b9924f5 ci: Add PyPy to matrix 2021-08-31 02:34:12 -07:00
James R. Barlow c28858a099 leptonica: fix a PyPy-specific error
Error is:
TypeError: from_buffer() got a 'memoryview' object, which supports the buffer interface but cannot be rendered as a plain raw address on PyPy

PyPy is happy to access a bytes() copy of the memoryview.
2021-08-31 02:31:58 -07:00
James R. Barlow f00b3c00cd Merge remote-tracking branch 'origin/master' 2021-08-31 02:19:09 -07:00
James R. Barlow 4e4f0bfa1f graft: use faster unparse_content_stream if available 2021-08-31 02:16:25 -07:00
Elliott Sales de AndradeandGitHub 0a31acf888 Allow pluggy v1. (#822)
There are breaking changes, but I could not find any reference to them
in the code.
2021-08-31 02:15:43 -07:00
James R. Barlow b91096c615 hoctransform: fix deprecation warning 2021-08-28 02:11:33 -07:00
James R. Barlow 95d9e8d91a info: inconsistent types used in ContentsInfo.name_index
This broke PyPy but CPython is fine with it.
2021-08-28 00:18:14 -07:00
James R. Barlow cb6c1939e9 typing: fix runtime issues 2021-08-27 02:18:54 -07:00
James R. Barlow 3764ee872a typing: refactor namedtuples in info 2021-08-27 00:51:11 -07:00
James R. Barlow e402d5cb4b typing: fix path and numeric issues 2021-08-27 00:23:38 -07:00
James R. Barlow 53cd04799a typing: fix Pillow usage, leptonica 2021-08-27 00:06:38 -07:00
James R. Barlow f2545d4496 typing: remove deprecated abstractstaticmethod 2021-08-26 23:51:11 -07:00
James R. Barlow 9b81e76ed4 typing: fix issues; avoid some magic literals 2021-08-26 23:48:27 -07:00
James R. Barlow 0956fc81aa typing: improvements for concurrency files 2021-08-26 23:47:53 -07:00
James R. Barlow 72279e7759 typing: confirmed that _pages_from_ranges(not-str) is never used 2021-08-26 23:47:11 -07:00
James R. Barlow 6f9b948064 typing: fix some trivial issues 2021-08-26 23:46:44 -07:00
James R. Barlow 4eca0a165b pre-commit: pyupgrade modernizing 2021-08-26 18:04:38 -07:00
James R. Barlow 067e61e03a pre-commit: auto update 2021-08-26 18:00:37 -07:00
James R. Barlow 1b46481f7e pre-commit: add setup.cfg fmt 2021-08-26 17:59:40 -07:00
James R. Barlow d8d9c41abb v12.3.3 release notes 2021-08-21 18:06:14 -07:00
James R. Barlow a8cad72f72 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2021-08-21 17:38:19 -07:00
James R. Barlow 86c04305f4 readme: confirm Tesseract 5 support 2021-08-21 17:37:10 -07:00
James R. Barlow 0a110fac55 watcher: fix bool not working as expecting
Closes #821
2021-08-21 17:30:14 -07:00
James R. Barlow f8970ad862 Update CI to test Tesseract 5 and more Linux versions 2021-08-17 02:30:24 -07:00
James R. Barlow fcfc78b7ee Move tool config to pyproject 2021-08-17 01:46:22 -07:00
8bb244df24 [ci skip] Update jbig2.rst (#817)
Co-authored-by: jbarlow83 <jbarlow83@users.noreply.github.com>
2021-08-14 20:05:27 -07:00
James R. Barlow 8a1cb70479 build: adjust pull request again 2021-08-04 03:06:28 -07:00
James R. Barlow 2c579700d6 v12.3.2 release notes 2021-08-04 02:52:45 -07:00
James R. Barlow 87ff6c8301 webservice: tidy flask apis 2021-08-04 02:49:48 -07:00
James R. Barlow 5915259bee Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2021-08-04 02:49:23 -07:00
Elliott Sales de AndradeandGitHub 969e54f0e3 Allow flask 2 for webservice (#816)
* Allow flask 2 for webservice.

The breaking changes do not appear to affect it.

* Add shebang to webservice script.
2021-08-04 02:49:13 -07:00
Elliott Sales de AndradeandGitHub b923612323 Allow watchdog 2. (#815)
* Allow watchdog 2.

The breaking change was dropping support for macOS 10.12 and earlier,
which doesn't affect us.

* Add shebang to watcher script.
2021-08-04 02:48:25 -07:00
mara004andGitHub dc2b161306 docs: Add two commas (#809) 2021-08-04 02:47:34 -07:00
James R. Barlow ae49e3b6db build: don't try to docker build for third party users 2021-08-04 02:47:03 -07:00
James R. Barlow 0fc5067ab6 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2021-08-01 01:00:17 -07:00
James R. Barlow 5eb5b5ba73 v12.3.1 release notes 2021-08-01 01:00:14 -07:00
James R. Barlow aa10a70d70 Rebuild test cache due to hocr output change 2021-08-01 01:00:05 -07:00
James R. Barlow f99fd686de hocrtransform: ensure text is rendered in document order
Previously we rendered text objects based on their vertical position, but this confuses
some PDF viewers.

Closes #813.
2021-08-01 00:49:57 -07:00
James R. Barlow 22dd9314ea logo: add square version 2021-07-20 23:24:17 -07:00
Kai KnoblichandGitHub 73b8b88724 Update the FreeBSD specific parts of the documentation (#810)
The Python default version was changed from 3.7 to 3.8 at the end of April.  For the new quarterly branch, 2021Q3, Python 3.8 has also been the default version since July.
2021-07-20 13:35:19 -07:00
James R. Barlow 4863a8e521 docker: make base install packages in common 2021-07-15 23:33:12 -07:00
James R. Barlow 2366629774 docker: Fix timezone prompt 2021-07-15 23:29:53 -07:00
James R. Barlow 814ad36e51 dockerfile: try newer ubuntu 2021-07-15 19:06:46 -07:00
James R. Barlow 1c2adc3d89 v12.3.0 release notes 2021-07-14 02:38:23 -07:00
James R. Barlow 37923ffe52 Work around Pillow 8.3.1 DPI changes
Pillow decided against round-tripping DPI values.
https://github.com/python-pillow/Pillow/pull/5476

Fixes #802
2021-07-14 02:34:28 -07:00
James R. Barlow 773e28478c graft: don't use deprecated pikepdf APIs 2021-07-14 01:44:26 -07:00
James R. Barlow e6db071273 Merge remote-tracking branch 'origin/master' 2021-07-14 00:48:30 -07:00
mara004andGitHub de74b80335 docs: Fix two missing words (#805)
* Fix two missing words
2021-07-14 00:48:00 -07:00
James R. Barlow 5cba68b93d tests: Don't require symlink permissions on Windows
Some of tests required symlink permissions, which CI workers have but typical Windows
user accounts do not. Mostly these are just correctness tests.
2021-07-14 00:11:47 -07:00
mara004andGitHub d2d39de92f Update api.rst (#797)
fix excess 'no'
2021-07-05 14:12:51 -07:00
James R. Barlow 0b834411fe validation: mention ISO 639-2 to give people a clue about how to find the appropriate code 2021-06-28 15:19:59 -07:00
James R. Barlow c935ba070b Fix name of sphinx-rtd-theme 2021-06-23 00:43:30 -07:00
James R. Barlow e30fffa8a4 v12.2.0 release notes 2021-06-16 00:40:47 -07:00
James R. Barlow e760be9e19 Drop an obsolete documentation file 2021-06-16 00:39:51 -07:00
James R. Barlow 38280e77f8 Use sphinx-issues to "refactor" release notes 2021-06-16 00:39:40 -07:00
James R. Barlow 6b9b5cc5d5 setup.py shouldn't have a shebang 2021-06-16 00:10:10 -07:00
James R. Barlow 5f01c5e330 Fix another species of Tesseract version number breaking regex
Fixes #795
2021-06-16 00:09:03 -07:00
James R. Barlow 5d08303805 docs: plugins - show setup.cfg example 2021-06-14 02:15:52 -07:00
James R. Barlow 7965b1f930 docs: don't suggest unmaintained alternatives, update on GS 2021-06-14 01:08:07 -07:00
James R. Barlow 0a1216bf14 v12.1.0 release notes 2021-06-13 02:26:10 -07:00
James R. Barlow f10a0f7707 readme: we Docker arm now 2021-06-13 02:20:36 -07:00
James R. Barlow dc11802809 github templates: ask for more details on feature reqs 2021-06-13 02:20:25 -07:00
James R. Barlow 4cce0077d0 Merge branch 'ci' 2021-06-10 00:30:32 -07:00
James R. Barlow d293e05946 Adjust readthedocs to not use requirements files
Remove .yml since RTD considers it deprecated.
2021-06-10 00:28:38 -07:00
James R. Barlow 4030258bbc Modernize build system to use setup.cfg
For now, keep but deprecate the requirements/*.txt files.
2021-06-10 00:27:52 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
db388165a9 Bump pillow from 8.1.2 to 8.2.0 in /requirements (#793)
Bumps [pillow](https://github.com/python-pillow/Pillow) from 8.1.2 to 8.2.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/master/CHANGES.rst)
- [Commits](https://github.com/python-pillow/Pillow/compare/8.1.2...8.2.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-08 13:27:17 -07:00
James R. Barlow 3d6907f7f6 v12.0.3 release notes 2021-05-27 13:53:33 -07:00
James R. Barlow 701c3b371b docs: take a more vender neutral position on commercial OCR 2021-05-27 13:45:59 -07:00
James R. Barlow 684e5b4944 docs: mention ISO Latin-1 2021-05-27 13:42:17 -07:00
James R. Barlow a964080f77 validation: a word 2021-05-27 13:42:05 -07:00
James R. Barlow 3f72f16958 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2021-05-27 01:26:40 -07:00
James R. Barlow b4f2582766 Show ExitCodeException traceback in verbose mode 2021-05-27 01:26:25 -07:00
FrankandGitHub c77cc7c837 Update README.md (#785)
readme: mention LinuxUser article
2021-05-26 12:37:58 -07:00
James R. Barlow f3715daf15 Move HOCR_OK_LANGS to hocrtransform.py 2021-05-23 01:15:53 -07:00
James R. Barlow c87221a4e6 validation: add proper list of languages supported by hocr
Based on Latin-1 support in default PDF fonts.
2021-05-23 01:14:15 -07:00
James R. Barlow c409fa5825 v12.0.2 release notes 2021-05-18 23:22:11 -07:00
James R. Barlow 09c485bd88 Convert harmless Leptonica exception to warning
Closes Error when trying --remove-background on pdf #769
2021-05-18 23:19:51 -07:00
James R. Barlow 9d51a1b5ab Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2021-05-13 23:29:02 -07:00
James R. Barlow 43e7765efd Expand documentation for adding language packs
Closes #777
2021-05-13 23:24:54 -07:00
Matthias BraunandGitHub 352f009c77 Fix code of French language pack in languages.rst (#776)
* Fix code of French language pack in languages.rst

* Update languages.rst

Reviewed-by: jbarlow83 <jbarlow83@users.noreply.github.com>
2021-05-09 01:03:13 -07:00
James R. Barlow 399b5548ca v12.0.1 release notes 2021-04-26 01:18:57 -07:00
James R. Barlow 7b1e5b4f41 Fix "invalid version number" for untagged tesseract versions
Fixes #770
2021-04-26 01:18:07 -07:00
James R. Barlow 33e0b16174 v12.0.0 release notes 2021-04-23 00:04:21 -07:00
James R. Barlow a613722e96 Auto-register semfree.py if needed 2021-04-21 23:39:38 -07:00
James R. Barlow ad0126185f Rename awslambda -> semfree.py 2021-04-21 23:29:55 -07:00
James R. Barlow be45871d10 docker: add special hint for using docker 2021-04-21 23:18:29 -07:00
James R. Barlow 252221fd8b dockerfile: remove unnecessary copy of /app/src 2021-04-21 23:17:42 -07:00
James R. Barlow d25c49ba81 docs: remove incorrect value of rotate-pages-threshold from docs
Closes #762
2021-04-19 00:06:22 -07:00
James R. Barlow 5112e9e857 Redo dpi calc to avoid 'math.ulp' 2021-04-16 00:54:41 -07:00
James R. Barlow 757b72b0af Revert "Remove apparently unused portion of a test"
This reverts commit d89a633ba7.
2021-04-16 00:21:11 -07:00
James R. Barlow 75c5b92cb9 v12.0.0 update release notes 2021-04-15 23:49:58 -07:00
James R. Barlow d673126994 Fix ZeroDivisionError on files containing images drawn at scale 0
Fixes #761
2021-04-15 23:26:14 -07:00
James R. Barlow d89a633ba7 Remove apparently unused portion of a test 2021-04-15 23:25:18 -07:00
James R. Barlow 710d797299 Fix comment typos 2021-04-14 00:36:58 -07:00
James R. Barlow fc75254c60 Fix awslambda progressbar.update() error
Fixes #759
2021-04-13 14:37:30 -07:00
James R. Barlow f453e94f14 awslambda: better documentation 2021-04-13 13:16:35 -07:00
James R. Barlow 8f8aaa93ed Refactor removing log handlers 2021-04-13 13:16:21 -07:00
James R. Barlow a90b9e669f Add reminder to not mess with pool/listener order 2021-04-09 13:22:48 -07:00
James R. Barlow e4f69cc1d6 Maybe fix a deadlock on attempting sys.stderr.flush()
Closes #758
Closes #733
2021-04-09 01:59:30 -07:00
James R. Barlow a5852ba199 Remove parent process's log handlers properly 2021-04-08 21:39:18 -07:00
James R. Barlow 051b9da991 Remove undocumented/unused debug environment variables 2021-04-08 21:08:12 -07:00
James R. Barlow 139d9f9841 Shut up pikepdf mmap disabled message 2021-04-08 20:58:46 -07:00
James R. Barlow 532d65a355 ci: push on tag 2021-04-08 14:19:59 -07:00
James R. Barlow 9de38afb13 Fix Tesseract version for ubuntu 18.04 2021-04-08 13:07:13 -07:00
James R. Barlow 913c939dc9 docs: update wrt AWS 2021-04-08 01:51:44 -07:00
James R. Barlow 9db9a3d6ec helpers: improve test coverage of Resolution 2021-04-07 23:26:37 -07:00
James R. Barlow 8423bd549b helpers: don't trap exception on failure to unlink
If we can't unlink a file we expect to unlink, logging and moving on is
probably the wrong action.

Coverage never hits this line.
2021-04-07 23:16:10 -07:00
James R. Barlow 336d274a54 Drop remnants of support for Tesseract without has_textonly_pdf
Also improve Tesseract version checking so it can compare all of their
weird conventions.
2021-04-07 23:05:21 -07:00
James R. Barlow 906d77b389 tests: remove obsolete running_in_travis() 2021-04-07 02:25:10 -07:00
James R. Barlow 9416e850ff Remove another instance of helpers_namespace 2021-04-07 02:23:04 -07:00
James R. Barlow fd3248869c Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2021-04-07 02:18:14 -07:00
James R. Barlow 2a09a668f6 Delinting: unused args 2021-04-07 02:18:08 -07:00
James R. Barlow e788dde607 tests: eliminate unnecessary mmap 2021-04-07 02:11:31 -07:00
James R. Barlow 173a80864d Delinting 2021-04-07 02:09:45 -07:00
James R. Barlow aa115a8be3 Remove pytest_helpers_namespace 2021-04-07 01:56:51 -07:00
e0441c4aa1 Explicitly require setuptools, since pdfa.py imports pkg_resources (#755)
Co-authored-by: jbarlow83 <jbarlow83@users.noreply.github.com>
2021-04-06 02:23:44 -07:00
James R. Barlow a861c58da2 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2021-04-06 02:07:16 -07:00
James R. Barlow b1306bd7a8 tests: skip test_bash on Windows 2021-04-06 01:15:00 -07:00
James R. Barlow 3a6eb383dc Run all tests 2021-04-06 00:28:06 -07:00
James R. Barlow 6c942ecefd Update release notes 2021-04-06 00:17:36 -07:00
James R. Barlow a25f8ecc62 Remove Azure Pipelines 2021-04-06 00:17:29 -07:00
andkrauseandJames R. Barlow e1f4813d94 Dockerfile: support arm64
Thanks to @andkrause for submitting the initial PR that made this change
possible, @0x326 for review comments on the same PR, and both for their
patience in waiting for the rest of OCRmyPDF to catch up.

Co-authored-by: James R. Barlow

Closes #564
2021-04-06 00:09:04 -07:00
James R. Barlow af526f078d Activate codecov 2021-04-06 00:08:58 -07:00
James R. Barlow 16438c1312 Begin GitHub Actions migration 2021-04-06 00:08:13 -07:00
James R. Barlow dd6cb7ce20 Preparing release notes 2021-04-02 01:10:53 -07:00
James R. Barlow 653e2e23df Update deps for next release 2021-04-02 01:10:39 -07:00
James R. Barlow a2033698fa graft: use newer pikepdf style 2021-04-02 00:11:13 -07:00
James R. Barlow ec1d585d40 Merge branch 'feature/misc-breaking' 2021-04-01 16:51:04 -07:00
James R. Barlow a4e1f8e1f3 Merge branch 'feature/lambda' 2021-04-01 16:36:22 -07:00
James R. Barlow 2e155c31bf Ensure builtin module registration is deterministic 2021-04-01 16:30:42 -07:00
James R. Barlow e09ae9c68a Fix test suite failure if filter_pdf_page is missing 2021-04-01 16:25:40 -07:00
James R. Barlow c5bf1dd90d reqs: bump reqs to avoid reportlab issue 2021-04-01 16:01:23 -07:00
e96770c5e4 Updated documentation for windows and additional languages (#753)
Changed the documentation for how to add new languages on Windows.

Co-authored-by: jbarlow83 <jbarlow83@users.noreply.github.com>
2021-03-26 16:13:28 -07:00
James R. Barlow f4f0f3c022 v11.7.3 release notes 2021-03-20 23:30:42 -07:00
James R. Barlow 0a42934c08 Exclude Group 3 images from optimization 2021-03-20 23:28:21 -07:00
James R. Barlow d8f47768f9 v11.7.2 release notes 2021-03-19 00:31:38 -07:00
James R. Barlow c9594a4a5f Update pinned versions to avoid Pillow vulnerabilties
See https://github.com/python-pillow/Pillow/blob/master/CHANGES.rst
2021-03-19 00:31:27 -07:00
James R. Barlow 873f915212 docs: mention problems with Debian/Ubuntu and other tidying 2021-03-09 23:24:10 -08:00
James R. Barlow 079c162a96 Ensure sidecar is not input or output file 2021-03-05 00:29:42 -08:00
James R. Barlow 25c8c4656f Fix error message change 2021-03-03 01:02:59 -08:00
James R. Barlow ffcae9a1a0 v11.7.1 release notes 2021-03-03 00:46:35 -08:00
James R. Barlow 6e71fe1186 Clarify --unpaper-args errors 2021-03-03 00:44:21 -08:00
James R. Barlow 0885799010 Update docs for conda
Closes #743
2021-03-03 00:43:45 -08:00
James R. Barlow 8ffc99f648 optimize: log errors more loudly 2021-03-03 00:43:40 -08:00
James R. Barlow 2261c51eff Reactivate pngquant on windows 2021-02-26 01:19:03 -08:00
James R. Barlow 5c470778a3 v11.7.0 release notes 2021-02-26 00:29:52 -08:00
James R. Barlow 4124889f36 Don't generate PDF/A-1b with object streams
Acrobat insists that PDF/A-1b should not have object streams.
Other programs like veraPDF disagree with this restriction, but
we can accommodate Acrobat so we will.

Also add more tests around this.
2021-02-26 00:23:57 -08:00
James R. Barlow a23c22b0e8 helpers: tidy check_pdf 2021-02-25 22:51:53 -08:00
James R. Barlow dd1f5f7215 pyproject: black doesn't like py39 yet 2021-02-25 16:10:20 -08:00
Dima KuznetsovandGitHub 5e2206bae7 Allow --sidecar along --pages (#735) 2021-02-19 16:55:35 -08:00
James R. Barlow 079ee86d43 pyproject: also target py39 2021-02-18 01:48:56 -08:00
James R. Barlow 3692868004 v11.6.2 release notes 2021-02-15 01:48:14 -08:00
James R. Barlow 064f935699 Fix page rotation regression
Page size fixes in commit b26749 did accounted for a "kept" rotation,
but not a corrected rotation.

Fixes #730.
2021-02-15 01:47:09 -08:00
James R. Barlow 8770fff968 tests: remove unreliable/incomplete test 2021-02-15 01:05:08 -08:00
James R. Barlow 82de78b6b0 v11.6.1 release notes 2021-02-14 01:51:26 -08:00
James R. Barlow 2a52c6dec2 optimize: skip images with unusually small dimensions
They're unlikely to be handled well by our recompressors. It seems
that JBIG2 cannot handle very small widths.

Fixes #732
2021-02-14 01:43:25 -08:00
James R. Barlow 2898879be7 docker-compose: fix typo 2021-02-14 01:43:06 -08:00
James R. Barlow 18e613657c docker-compose: fix typo 2021-02-14 01:23:01 -08:00
James R. Barlow a48ca556c7 Add filter_pdf_page hook 2021-02-14 01:22:33 -08:00
James R. Barlow 9cba738b48 Remove deprecated code 2021-01-31 19:27:59 -08:00
James R. Barlow bccf2f423f Stricter parameter checking for many public functions 2021-01-31 19:27:25 -08:00
James R. Barlow 390fdf8c05 Package OCR in Form XObject
Should improve results in some situations where the initial content
stream is messy or not well-formed.
2021-01-31 19:27:25 -08:00
James R. Barlow 166de3086b Merge branch 'feature/colorstrategy' 2021-01-31 19:26:59 -08:00
James R. Barlow 206c675df6 docs: api 2021-01-31 19:26:35 -08:00
James R. Barlow 6c8f9223e9 Update awslambda to new pluginspec 2021-01-31 03:00:05 -08:00
James R. Barlow 85c6a974ca Fix calls to hook.get_executor 2021-01-31 02:46:44 -08:00
James R. Barlow dccdcfaa91 leptonica: tidy 2021-01-31 02:46:09 -08:00
James R. Barlow b1da09f141 Add plugin for setting logging console
So that we are not tied to tqdm.
2021-01-31 02:29:11 -08:00
James R. Barlow 42c84531e4 optimize: rewrite JPEG optimize to avoid use of tqdm and parallelize
For some reason JPEG optimization was not done in parallel, and was
perhaps never done in parallel. Strange oversight.
2021-01-31 02:18:46 -08:00
James R. Barlow a9ad805347 optimize: Remove shim for unsupported pikepdf version 2021-01-31 00:08:20 -08:00
James R. Barlow 16bda74974 Refactor - decouple progressbar from executor 2021-01-30 20:42:00 -08:00
James R. Barlow d274d88929 Refactor to eliminate global state in _concurrent 2021-01-30 17:36:30 -08:00
James R. Barlow 327df5cbbc Use ColorConversionStrategy "LeaveColorUnchanged"
Faster, still produces PDF/A
2021-01-26 15:44:41 -08:00
James R. Barlow 46d0632fe2 v11.6.0 release notes 2021-01-26 01:47:49 -08:00
James R. Barlow ef1e7a814e Delinting 2021-01-26 01:45:04 -08:00
James R. Barlow 1084724937 docs: improve API docs 2021-01-26 01:40:40 -08:00
James R. Barlow ecb0109d79 docs: fix rst formatting error 2021-01-26 01:29:45 -08:00
James R. Barlow 386cabff00 Make progress pool common rather than plugin-specific 2021-01-24 23:56:09 -08:00
James R. Barlow 3bd5054634 lambda: move to extra_plugins folder 2021-01-24 23:47:26 -08:00
James R. Barlow 6a8dd65aa2 lambda: more issues related to new executor semantics
Now all tests pass, except for:
-tests that check the progress bar
-tests where xdist may or may not load a _lambda_plugin by running
some other test first before a test in optimize
2021-01-24 23:46:40 -08:00
James R. Barlow 6083b4f0a7 lambda: don't overrun number of workers needed 2021-01-24 23:46:40 -08:00
James R. Barlow 1a3ce59476 lambda: Don't be paranoid about exception marshalling
It works
2021-01-24 23:46:40 -08:00
James R. Barlow c395436ba3 lambda: tidying, special casing use_threads 2021-01-24 23:46:40 -08:00
James R. Barlow 8d23d0b441 Operational lambda executor 2021-01-24 23:46:40 -08:00
James R. Barlow c6a2716cdb Temporary move into package 2021-01-24 23:46:40 -08:00
James R. Barlow 5545bae76f lambda_plugin.py: doesn't work since entry point needs to be in package 2021-01-24 23:46:33 -08:00
James R. Barlow 7bccb8c748 tests: fix concurrency 2021-01-24 23:46:33 -08:00
James R. Barlow 173c0d1274 concurrency: lock progress pool
For API sanity and to communicate expectations. One progress pool at
a time is plenty of complexity.
2021-01-24 23:46:33 -08:00
James R. Barlow 6953f32465 pdfinfo: remove some messy concurrency handling
We can cut down on the use of global variables and save opening
an extra copy of the Pdf when threaded.
2021-01-24 23:46:33 -08:00
James R. Barlow 26b4d9bb4b Refactor concurrency so that it is pluggable
However, this may not be the best idea because it involves global
state that could be overridden by a parallel call to ocrmypdf.ocr.
2021-01-24 23:46:29 -08:00
James R. Barlow 34e564cd7d Use queue.Queue instead of multiprocessing.Queue in threaded mode 2021-01-24 23:45:26 -08:00
James R. Barlow 504d5776d2 Refactor plugin manager to eliminate callback 2021-01-24 23:42:40 -08:00
James R. Barlow ee23976858 Re-sequence plugin installation 2021-01-24 23:42:28 -08:00
James R. Barlow f559316881 Insert setuptools plugins with ocrmypdf prefix 2021-01-24 23:42:09 -08:00
James R. Barlow 084610c242 Automate insertion of builtin modules 2021-01-24 23:41:50 -08:00
James R. Barlow 9ff627472b Update pre-commit 2021-01-24 23:40:59 -08:00
James R. Barlow 956310d1ec Import PageContext, PdfContext since they are referenced in pluginspec 2021-01-24 02:04:47 -08:00
James R. Barlow 1a982da442 tests: confirm that we produce pdf when optimization is off 2021-01-24 01:54:25 -08:00
James R. Barlow 4879a1f0de docs: no MS Store Python 2021-01-18 13:27:31 -08:00
James R. Barlow ce66bcc9c8 github: Ask how ocrmypdf was installed 2021-01-10 02:35:24 -08:00
James R. Barlow 1ebf3144af v11.5.0 release notes 2021-01-09 16:48:40 -08:00
James R. Barlow 7a1cccbc4e Fallback to LeptonicaErrorTrap_Redirect if ffi.callback fails
Might fix issue #709, Apple silicon support.
2021-01-09 16:47:11 -08:00
James R. Barlow ebacff1b39 tests: Fix debug logging test 2021-01-09 16:41:57 -08:00
James R. Barlow c7c447be66 Add test for configure_debug_logging
Since we can't directly test it
2021-01-09 16:02:12 -08:00
James R. Barlow 91aa175602 Consider text when determining page raster DPI
Previously if we found vectors of any sort on a page, we would bump
the DPI up to 400. We did nothing
about pages with text. As a result,
pages with a low image resolution
and printable text would have the text downgraded to image
resolution when --force-ocr was used.

We don't try to determine if the text is visible or invisible OCR text, since
that is a slower test. --redo-ocr would improve such cases anyway.
2021-01-09 16:01:49 -08:00
James R. Barlow b267494e4a Create raster PDF pages to match input page size
Previously we produced a raster image, then multiplied image width
by DPI to get the page size. However if there is rounding the
page size may not match exactly. In this modified approach we
constrain the page size to match.
2021-01-08 15:10:43 -08:00
James R. Barlow f687180ecc tests: tidy pdfinfo 2021-01-08 15:04:52 -08:00
James R. Barlow 6f4b38b103 ghostscript: tidy comments 2021-01-08 00:41:03 -08:00
James R. Barlow d32324859c v11.4.5 release notes 2021-01-06 11:42:28 -08:00
James R. Barlow 48222b87b5 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2021-01-06 03:59:40 -08:00
62e5edc72b fix unclosed file warnings. (#710)
Co-authored-by: Jonas Winkler <jonas.winkler@jpwinkler.de>
2021-01-06 03:59:28 -08:00
James R. Barlow 2846d46bb8 Remove .coveragerc and fold into setup.cfg 2021-01-06 03:58:18 -08:00
James R. Barlow 47ef1914d4 v11.4.4 release notes 2021-01-01 01:39:24 -08:00
James R. Barlow df157552f3 Make ocrmypdf.ocr take a threading lock 2021-01-01 01:37:09 -08:00
James R. Barlow 0b3a526049 Partial fix crash on 'userunit' None (#700)
Our method of getting data from pdfminer would silently consume a StopIteration
if pdfminer returned no processed pages, leading to odd error message.

We improve an error from pdfminer properly, and returning a more
descriptive error of our own.

It would be possible for ocrmypdf to repair the file before sending it to
pdfminer, but this seems to be rare enough that we won't do that yet.
2021-01-01 01:11:32 -08:00
James R. Barlow 1e80d412fa tesseract: fix typing of some optional arguments 2021-01-01 00:46:00 -08:00
James R. Barlow df6e106203 concurrent: simplify results loop 2021-01-01 00:44:46 -08:00
James R. Barlow bd0f005861 tests: tag tests that need pngquant, jbig2enc 2020-12-30 01:58:57 -08:00
James R. Barlow 6ba4b7b3f3 ci: temporarily disable pngquant on Windows
Looks like a packaging error, choco complains of bad hashes.
2020-12-30 01:40:56 -08:00
James R. Barlow 2c11349ee8 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-12-29 21:40:46 -08:00
James R. Barlow b0afef09ef v11.4.3 release notes 2020-12-29 21:40:35 -08:00
James R. Barlow 72fa347c38 tests: skip metadata test for two pikepdf versions that warn incorrectly 2020-12-29 01:47:52 -08:00
James R. Barlow 96d68c2413 pipeline: refactor metadata_fixup 2020-12-29 01:47:32 -08:00
James R. Barlow babc76fa74 tests: assert that most patched functions are called
We were not actually checking if functions we patched we called when
expected.
2020-12-28 23:58:33 -08:00
Tim GatesandGitHub dc06990e5d docs: fix simple typo, instsalled -> installed (#704)
There is a small typo in docs/installation.rst.

Should read `installed` rather than `instsalled`.
2020-12-28 15:28:34 -08:00
James R. Barlow 0ff0d2f8d1 Remove PDF/A overprint debug message
Since we currently log all of a process's output at debug it's
redundant to log this separate message.
2020-12-27 16:19:05 -08:00
James R. Barlow 81602cf420 Fix test not patching properly after Ghostscript polling change 2020-12-27 16:01:50 -08:00
James R. Barlow 607e2d7e81 v11.4.2 release notes 2020-12-27 03:29:35 -08:00
James R. Barlow b01d9e07e8 Deal with missing pthread_sigmask on Cygwin
Closes #701
2020-12-27 02:24:00 -08:00
James R. Barlow 91db94cf2e watcher: fix OCR_LOGLEVEL env var not processed
Closes #702
2020-12-27 02:02:44 -08:00
James R. Barlow 416df803d4 pdfinfo: stricter typing 2020-12-24 22:39:00 -08:00
James R. Barlow 037b96ca16 pdfinfo: refactor to eliminate RawPageInfo 2020-12-24 02:57:44 -08:00
James R. Barlow bb258fc99c pdfinfo: Refactor pageinfo dictionary into a class 2020-12-24 01:47:53 -08:00
James R. Barlow 4b8ccbe8cb v11.4.1 release notes 2020-12-22 01:41:15 -08:00
James R. Barlow ab1ff3331b misc: synology fix
Accept user-contributed fix. Not testable.

Close #690.
2020-12-22 01:38:41 -08:00
James R. Barlow 3675ae918c Fix certain invalid page ranges causing exception
Closes #686
2020-12-22 01:22:14 -08:00
James R. Barlow 0ba32b96b7 Revert "v11.4.0 release notes - remove change not actually implemented"
This reverts commit ad202693b3.
Temporary folder prefix was actually changed in commit f11bb53e.
2020-12-22 00:47:25 -08:00
James R. Barlow add64e4fa2 docs: com.github.ocrmypdf -> ocrmypdf.io 2020-12-22 00:46:42 -08:00
James R. Barlow 7fe2954ede Change wheel tag to py36, update package_data to include py.typed 2020-12-12 16:49:04 -08:00
James R. Barlow ad202693b3 v11.4.0 release notes - remove change not actually implemented
Remove a change that was pushed back to a future release.
2020-12-12 16:27:38 -08:00
James R. Barlow 594ef83551 v11.4.0 release notes 2020-12-11 15:09:49 -08:00
James R. Barlow 78b71618c1 Fix BufferedReader TypeError 2020-12-11 14:19:20 -08:00
James R. Barlow b8aa89e1ec Fix log message queue flooding on certain files
Fixes #692
2020-12-11 14:14:21 -08:00
James R. Barlow b4c1f66bc1 typing: tidy up 2020-12-11 14:14:21 -08:00
James R. Barlow 5172dbde8d subprocess: use more mypy-friendly syntax 2020-12-11 14:14:21 -08:00
James R. Barlow d2908640c6 pdfa: help mypy figure out a type 2020-12-11 14:14:21 -08:00
James R. Barlow 997bf7578d hocrtransform: fix exception if no div ocr_page object 2020-12-11 14:14:21 -08:00
James R. Barlow 043258242c hocrtransform: trivial typing 2020-12-11 14:14:21 -08:00
James R. Barlow 156d5d9a9c cli: typing 2020-12-11 14:14:21 -08:00
James R. Barlow 0b7e52fb5e api: parse cmdline in more type friendly way 2020-12-11 14:14:21 -08:00
James R. Barlow a5feef07d0 Declare ocrmypdf as typed 2020-12-11 14:14:21 -08:00
James R. Barlow f11bb53e61 Change prefix of temporary folders
Shouldn't really use a name that suggests a connection to GitHub.
2020-12-07 21:51:46 -08:00
James R. Barlow 68a57a7839 Add feature to generate hocr-pdf with visible debug text 2020-12-04 17:38:48 -08:00
James R. Barlow 4194430dc1 Begin next release notes 2020-12-04 13:28:04 -08:00
James R. Barlow a707c56fae docs: improve windows instructions 2020-12-04 13:21:54 -08:00
James R. Barlow 3cba50bfbd windows: look in registry for Tesseract and Ghostscript 2020-12-04 13:21:54 -08:00
James R. Barlow ed5e17d0a4 completions: consider *.PDF and some images too 2020-12-04 13:20:35 -08:00
James R. Barlow ce0e0ecd4d Decouple tqdm from progressbar setup 2020-12-04 13:20:28 -08:00
James R. Barlow 7e1223c12c ghostscript: add output tracing 2020-11-29 14:53:35 -08:00
James R. Barlow b83d7f6d1a subprocess: refactor and add run_polling_stderr 2020-11-29 14:36:03 -08:00
James R. Barlow 80e957908a tesseract: fix run call with logs_errors_to_stdout 2020-11-29 14:25:46 -08:00
James R. Barlow f0e7bea8ba docs: remove redundant statement 2020-11-27 13:54:36 -08:00
James R. Barlow 0cdb9bd04a docs: remove description of how OMP_THREAD_LIMIT is managed 2020-11-23 12:36:04 -08:00
James R. Barlow 8224d89bc6 v11.3.4 release notes 2020-11-18 11:57:28 -08:00
James R. Barlow a2bbbe2a26 v11.3.4 release notes 2020-11-18 11:56:29 -08:00
James R. Barlow 43f41863fa check_pdf: document how we handle linearization 2020-11-18 11:54:07 -08:00
James R. Barlow d71e50e83d Fix "readLinearizationData for file that is not linearized"
pikepdf 2.1.0 throws wrong type of exception in this case, so special-case it.

Closes #680
Closes #681
2020-11-18 11:52:17 -08:00
James R. Barlow 1f598da3c1 ghostscript: better docs and comments 2020-11-18 11:34:17 -08:00
James R. Barlow d0cdbd5e1c watcher: include uppercase .PDF too 2020-11-12 02:29:47 -08:00
James R. Barlow 5c56f61209 unpaper: type hints 2020-11-11 02:59:37 -08:00
James R. Barlow 9bec85470a Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-11-10 04:08:05 -08:00
James R. Barlow a03863a17d docs: fix link to docker image 2020-11-10 04:08:01 -08:00
James R. Barlow 22cd9b2364 docs: fix csv-table errors 2020-11-10 04:07:49 -08:00
pretentious7andGitHub 4fc7d6d93e fix typo "charcter" -> "character" (#673) 2020-11-09 16:53:02 -08:00
James R. Barlow 71f0e7f545 v11.3.3 release notes 2020-11-07 00:53:33 -08:00
James R. Barlow 895fddd85e Replace most uses of universal_newlines with text
The parameters are equivalent but the latter is better named. Since
Python 3.6 doesn't support text= we use our wrapper to add it in that
place.

This is for subprocess.run.
2020-11-07 00:48:08 -08:00
James R. Barlow 5a59e4d543 unpaper: don't use universal_newlines=True
There's no specific reason to do this. We can log binary output equally
 well.
2020-11-07 00:18:27 -08:00
James R. Barlow b51abf2249 azure: Fix indentation mistake 2020-11-04 12:19:35 -08:00
James R. Barlow 6d3f9ff15a api: rework ocr() slightly to simplify variable handling 2020-11-03 17:10:52 -08:00
James R. Barlow 5d1d1a712b docs: more details about macOS API changes
Due to fork->spawn
2020-11-03 17:09:58 -08:00
James R. Barlow 6d5f8133e0 docs: show ifmain guard in example 2020-11-03 15:28:33 -08:00
James R. Barlow 13018d3d5c ci: Extend test matrix to Python 3.9 2020-11-03 04:15:14 -08:00
James R. Barlow 14a85f9473 Fix pinned dependencies 2020-11-03 04:12:47 -08:00
James R. Barlow d22a1b3367 v11.3.2 release notes (2)
Since we never tagged it, fix other things.
2020-11-03 02:03:25 -08:00
James R. Barlow b913e5dfef ghostscript: don't repeat log in debug
Subprocess already does this for us.
2020-11-03 01:45:06 -08:00
James R. Barlow dd8a5a4c72 Fix log domain names
ocrmypdf.subprocess.subprocess.ghostscript -> ocrmypdf.subprocess.ghostscript
2020-11-03 01:44:35 -08:00
James R. Barlow 36e9a54f02 Remove extraneous page rotation
This was added in commit b5ccbfd but seems to have been ill-advised.
2020-11-03 01:34:28 -08:00
James R. Barlow 3707af3b74 Change pdf.root to pdf.Root 2020-11-03 01:30:31 -08:00
James R. Barlow ced7ad9164 unpaper: round off DPI 2020-11-03 01:14:57 -08:00
James R. Barlow 54bbbfdeb3 Fix UnboundLocalError when considering ImageMasks for optimization
Uncovered by test file in issue 667, although unrelated to that issue.
2020-11-03 01:08:14 -08:00
James R. Barlow 7f73a6ed1e Some Python 3.9 fixes 2020-11-03 00:45:47 -08:00
James R. Barlow dce206d3dc Fix pre-commit for Py3.9 2020-11-03 00:20:25 -08:00
James R. Barlow 9304c856cf Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-11-02 02:47:36 -08:00
James R. Barlow e5df98cbdf v11.3.2 release notes 2020-11-02 02:43:32 -08:00
James R. Barlow 19bf3aeb00 api: improve typing 2020-11-02 02:33:34 -08:00
James R. Barlow e86be0031c unpaper: fix process output handling
With the ocrmypdf.subprocess wrapper, logging the output here
is redundant and loses the page number context.
2020-11-02 01:07:41 -08:00
James R. Barlow 6425977998 unpaper: use pnm instead of png
Some users reported problems with PNG recently; try PNM.

Fixes #665
Fixes #667
2020-11-02 01:05:56 -08:00
James R. Barlow d57df2d980 subprocess: support programs that write their messages to stdout 2020-11-02 01:00:59 -08:00
James R. Barlow 664d0c7969 Document configure_debug_logging 2020-11-02 00:59:00 -08:00
James R. Barlow a354663ee1 Fix typo in API documentation 2020-11-02 00:58:28 -08:00
Graham MilnandGitHub b21b048ec4 Add macOS brew language support (#615)
Note `brew` command for installing additional languages on macOS.
2020-10-30 01:09:06 -07:00
James R. Barlow 709c65b41a v11.3.1 release notes 2020-10-27 23:11:11 -07:00
James R. Barlow 67f99c5bb7 Endorse pdfminer.six 20201018 2020-10-27 23:09:45 -07:00
James R. Barlow d55e673d9c Fix warning about --pdfa-image-compression argument at wrong times
Closes #663
2020-10-27 23:09:45 -07:00
James R. Barlow 21b90d2d14 Endorse pikepdf 2.x 2020-10-27 23:09:45 -07:00
Edward BettsandGitHub 2def7e3392 Use % for percentage in string format (#643) 2020-10-27 23:09:14 -07:00
James R. Barlow b0dcaa7512 v11.3.0 release notes 2020-10-24 03:19:32 -07:00
James R. Barlow e8285b1d10 Add test to confirm rasterize_pdf_page rotates correct 2020-10-24 03:10:59 -07:00
James R. Barlow 5ba56adb53 Fix page rotation issue (again)
Commit 1327ab3 introduced a fix for a regression, which was reported
in #581, #634. It appears that the actual cause of this issue was
default parameters to rasterize_pdf_page in pluggy not working as
expected, causing a default rotation=0 even when a rotation was needed.
As such the OCR image was generated with the wrong orientation,
causing the initial regression and fix in commit 1327ab3.

Now that the real problem is identified, it's apparent that the logic
prior to 1327ab3 was found and we can revert to 1327ab3 since it fixes
all known cases including #658.

This reverts 1327ab3 except for retaining improves to rotation output.
2020-10-24 02:45:21 -07:00
James R. Barlow ca735278e0 setup: Version pluggy better 2020-10-24 02:35:41 -07:00
James R. Barlow b5ccbfdf25 Fix hookspec of rasterize_pdf_page to remove default parameters 2020-10-24 02:35:18 -07:00
James R. Barlow 8c35d6e6e4 Fix debug log messages being suppressed from child processes 2020-10-22 02:20:06 -07:00
James R. Barlow d1e0c81eda Ensure worker_pdf is closed after gathering info in a thread
This is hacky, uses global state, but it does improve the situation for now.
2020-10-22 00:38:24 -07:00
James R. Barlow 10c8e4f8b4 Only create debug.log when running from command line
When used as a library ocrmypdf shouldn't make policy decisions, like where to
put a log file. Unsurprisingly, creating it causes problems for library users
because we deleted the temporary folder which held the log file and made no
effort to move it to a new location.

Also update the documentation to better described how an application should
handle this.

Closes #657
2020-10-20 01:29:36 -07:00
James R. Barlow 6be2242c21 Describe "OCR" step as "Image processing" when --tesseract-timeout=0
Fixes #647
2020-10-08 01:03:42 -07:00
James R. Barlow 204c9d6ae1 Fix inverted colors during JBIG2 optimization on paletted images
Fixes #640
2020-10-07 04:08:50 -07:00
James R. Barlow 6eb393590b v11.2.0 release notes
Change v11.1.3 to v11.2.0 since it contains functional changes.
2020-10-06 03:24:31 -07:00
James R. Barlow 07c6654057 v11.1.3 release notes 2020-10-06 03:22:48 -07:00
James R. Barlow 4e15eb8d14 Fix image optimization discarding image masks and soft masks associated with PNGs
Fixes #648
2020-10-06 03:20:54 -07:00
James R. Barlow 8b01ab8ad2 Better type checking on ocrmypdf.ocr(plugins=...) 2020-10-05 15:02:34 -07:00
James R. Barlow e0a522ad50 Document the example plugin 2020-10-05 15:01:44 -07:00
James R. Barlow a1a8788c5a Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-09-29 02:46:27 -07:00
James R. Barlow cccdc178c3 v11.1.2 release notes 2020-09-29 02:46:18 -07:00
James R. Barlow 4eacb3454f hOCR: write text in correct order
Fixes #642
2020-09-29 02:45:11 -07:00
Jimit DholakiaandGitHub 82b8b41e80 docs: Add 'unpaper' optional dependency for Ubuntu 18.04 (#639) 2020-09-25 11:54:31 -07:00
James R. Barlow 581c5020ab v11.1.1 release notes 2020-09-25 00:28:38 -07:00
James R. Barlow 3ef8872a1e pngquant driver: refactor, use streams instead of temporary files 2020-09-25 00:18:02 -07:00
James R. Barlow 28eec73eed Tighten unpaper-args validation to exclude . and ..
Just in case
2020-09-25 00:18:02 -07:00
James R. Barlow bfe4a5b329 Tidy a log message 2020-09-25 00:17:57 -07:00
James R. Barlow 29097837d6 Release notes typo 2020-09-19 00:49:36 -07:00
James R. Barlow a40361db3c Remove unpaper from macOS build
Homebrew seems to be having issues with its deps?
2020-09-17 03:38:48 -07:00
James R. Barlow 8b29e3cbab Merge commit '9a6cd95e5fe2826d40861229aaa0431b76e302e7' 2020-09-17 03:34:35 -07:00
James R. Barlow b170be120b v11.1.0 release notes 2020-09-17 03:21:06 -07:00
Suyash BeheraandGitHub 9a6cd95e5f load zlib before liblept on windows (#633)
fixes #631
2020-09-17 03:14:42 -07:00
James R. Barlow d464d3122e Use img2pdf to create optimized PNG images
Fixes #629, #620
2020-09-17 03:11:26 -07:00
James R. Barlow 1327ab37d4 Fix page rotation regression
Fixes #634, #581
2020-09-17 02:57:00 -07:00
James R. Barlow 67553fc5c6 Display page numbers in log messages when grafting 2020-09-17 01:20:50 -07:00
James R. Barlow 306a903854 Remove unused function log_page_orientations 2020-09-17 01:20:02 -07:00
James R. Barlow b93cf51c0f Disable pikepdf mmap
Infrequently we can reproduce this error:

terminating with uncaught exception of type std::runtime_error: pybind11_object_dealloc(): Tried to deallocate unregistered instance!

The error is probably related to pybind11 issue #2252 and a bunch of
other related issues. Until that is resolved in pybind11 and pikepdf
we will disable the pikepdf mmap interface.
2020-09-16 23:48:55 -07:00
James R. Barlow 6b994221c6 Remove Python 3.7 from build since homebrew removed it 2020-09-16 23:44:18 -07:00
James R. Barlow 8b5b02e0d8 Expand documentation of filter_page_image 2020-09-14 14:36:17 -07:00
James R. Barlow 624df9bb23 Extend example plugin with example of mono conversion 2020-09-14 14:35:50 -07:00
James R. Barlow fa06ea3600 v11.0.2 release notes 2020-09-08 02:38:57 -07:00
James R. Barlow 31994258fb metadata fixup: don't try to update original PDF's metadata with docinfo 2020-09-08 02:35:16 -07:00
James R. Barlow 1f15ecbca5 Add "Postprocessing" message as a hint for long Ghostscript runs 2020-09-08 02:34:10 -07:00
James R. Barlow bcf5657e5c Reorganize issue templates 2020-08-26 17:11:52 -07:00
jbarlow83andGitHub 2ae028bf38 Update issue templates 2020-08-26 17:03:09 -07:00
James R. Barlow b51a5887e5 v11.0.1 release notes 2020-08-17 23:25:31 -07:00
James R. Barlow fc523e837c Clarify that the GPL-3 portion of pdfa.py was removed
Removal was in 8c90f7c97.

pdfa.py now has no special licensing and falls unders the
"Files:  *" clause of debian/copyright.
2020-08-17 23:23:31 -07:00
James R. Barlow caeba76a61 Approve img2pdf 0.4 as it passes tests 2020-08-14 01:34:04 -07:00
James R. Barlow cd35216f21 setup: blacklist pdfminer.six 20200720
NotAllowedError is going to removed
https://github.com/pdfminer/pdfminer.six/commit/99f0c09869370d74eb6a27234284b894c33414f4
2020-08-12 13:10:08 -07:00
James R. Barlow e6a7b58863 Merge branch 'de-gpl' 2020-08-12 12:20:38 -07:00
James R. Barlow 56184a762f Issue template:Give stronger hints about sample input files 2020-08-12 12:12:37 -07:00
James R. Barlow 07ab98f5af docs: mention that Ghostscript PDF/A can swallow hyperlinks
Addresses #605
2020-08-12 12:12:00 -07:00
James R. Barlow 04fb1892b4 Don't ask for sample files anymore 2020-08-12 11:40:42 -07:00
James R. Barlow 173ce2f215 v10.3.3 release notes 2020-08-07 02:23:21 -07:00
James R. Barlow 9b641055e1 Fix KeyError: 'dpi' when using --threshold on image to PDF
Fixes #607
2020-08-07 02:21:02 -07:00
James R. Barlow 4fa28d7e74 v10.3.2 release notes 2020-08-05 01:36:56 -07:00
James R. Barlow bed74501fc Fix test breakage in validation
Broken in commit 4cc0dc
2020-08-05 01:35:26 -07:00
James R. Barlow 8c90f7c972 Replace GPLv3-derived PDF/A template with PostScript generator 2020-08-05 01:30:45 -07:00
James R. Barlow aa0ec40102 Change license of all GPLv3 files to MPL-2.0
https://github.com/jbarlow83/OCRmyPDF/issues/600
2020-08-05 00:44:42 -07:00
James R. Barlow 12c567ee10 Copyright cleanup: relicense example_plugin.py
The author is relicensing this file to MIT.
2020-08-05 00:15:33 -07:00
James R. Barlow d39778ce3a Clarify copyright status of misc/batch.py, synology.py
At the time these files were contributed there was no discussion
of the license that the authors wanted to use, but the project was
MIT licensed at the time. As such, these files deemed to be MIT licensed.

https://github.com/jbarlow83/OCRmyPDF/issues/600
2020-08-05 00:12:44 -07:00
James R. Barlow e824cdbc4e Change license of misc/watcher.py to MIT
The authors of this file all agreed to relicense it under the MIT license.

https://github.com/jbarlow83/OCRmyPDF/issues/600
2020-08-04 23:57:41 -07:00
James R. Barlow 1d91c09963 Clarify license status of misc/completion/* files
These files were contributed when the project license was GPLv3. On
discussion, all known authors of these files agreed to place them
under MIT license.

See https://github.com/jbarlow83/OCRmyPDF/issues/600
2020-08-04 23:53:56 -07:00
James R. Barlow e821ca46d5 Approve pdfminer.six 20200726 2020-08-03 16:04:16 -07:00
James R. Barlow a29e4952fb Document use of mmap 2020-08-03 16:03:54 -07:00
James R. Barlow 4cc0dc6b4a Additional size increase reasons 2020-08-03 16:03:29 -07:00
James R. Barlow 7263702de9 Remove gs.py (spoofers entirely removed) and update copyright 2020-07-29 16:31:47 -07:00
James R. Barlow 0287d91874 v10.3.1 release notes 2020-07-26 21:53:08 -07:00
James R. Barlow 436af55050 Approve pdfminer.six 20200720 2020-07-26 21:51:49 -07:00
James R. Barlow d6128e6937 Fix support for older versions of pdfminer.six (boxes_flow error) 2020-07-26 21:51:25 -07:00
James R. Barlow 642437e804 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-07-22 00:34:33 -07:00
James R. Barlow 4ce802fdb2 v10.3.0 release notes 2020-07-22 00:34:27 -07:00
James R. Barlow a672422b0b Enable pikepdf mmap in other contexts 2020-07-22 00:20:07 -07:00
James R. Barlow addc2cbad0 Enable pikepdf mmap and set up signal handlers 2020-07-22 00:19:50 -07:00
James R. Barlow 5f45f77b4e docs: plugins update 2020-07-21 23:53:30 -07:00
James R. Barlow 93f9bffb37 Merge branch 'feature/leptonica-179' 2020-07-20 21:23:53 -07:00
James R. Barlow 44149ad319 Disable test_error_trap for Leptonica < 1.79
Old error trap seems unreliable in the first place so difficult to set up
a test.
2020-07-20 21:12:00 -07:00
fcatusandGitHub d80d963cea pdfinfo: Replace list comp with gen expr'n 2020-07-20 02:21:58 -07:00
James R. Barlow 4da33b8050 Update debian/copyright from Debian, with fixes 2020-07-19 03:50:43 -07:00
James R. Barlow 5cbbff8472 For Leptonica 1.79+ use leptSetStderrHandler
Lock free and considerably less dangerous to stderr messages.
2020-07-19 03:40:33 -07:00
James R. Barlow fa6e47c277 Merge branch 'feature/optimize-cleanup' 2020-07-19 01:53:11 -07:00
James R. Barlow 4ea9cffebd Add locking to Leptonica error trap
To protect another thread from interfering with our redirection of
stderr.
2020-07-19 01:51:58 -07:00
James R. Barlow ae68edefc5 pipelines: fix Python 3.7/3.8 on macOS 2020-07-19 01:51:58 -07:00
James R. Barlow 1558e068f1 docs: explain firstresult hook behavior 2020-07-16 00:01:59 -07:00
James R. Barlow d2a9c413f8 docs: install notes for ARM64 2020-07-14 01:25:16 -07:00
James R. Barlow a510b21b20 optimize: add typing for Xref, remove fspath()'s 2020-07-09 14:06:41 -07:00
James R. Barlow e33ba07aa4 Update pre-commit settings 2020-07-08 23:45:53 -07:00
James R. Barlow 373f27832b optimize: improve typing of xref_exts 2020-07-07 22:41:29 -07:00
James R. Barlow b20a6e4c5d optimize: add type hints 2020-07-07 22:18:50 -07:00
James R. Barlow 49734d5456 optimize: fix incorrect to prevent re-optimizing JBIG2s 2020-07-07 21:52:11 -07:00
James R. Barlow 26a415c5dd docs: Note usage of OCR_JSON_SETTINGS for watcher 2020-07-07 21:26:57 -07:00
James R. Barlow 60be64a5f1 Fix debug.log missing pageno handler 2020-07-04 03:59:38 -07:00
James R. Barlow 190294634c docs: edit plugins 2020-07-03 16:16:01 -07:00
James R. Barlow 1722cb579d v10.2.1 release notes 2020-07-01 03:26:57 -07:00
James R. Barlow dc42beb6a8 More typing improvements
Typing fixes bugs.
2020-06-30 15:02:30 -07:00
James R. Barlow 378f543619 TextPositionTracker: set boxes_flow=None
We don't care about the order of lines in our analysis, and this is an
expensive calculation in pdfminer.
2020-06-30 04:20:58 -07:00
James R. Barlow 62924ee280 Improve API documentation 2020-06-30 04:20:14 -07:00
James R. Barlow 86a73191b0 Plugin manager: accept Path(plugin) 2020-06-30 04:17:30 -07:00
James R. Barlow 86875997b8 Fix more mypy errors 2020-06-29 02:17:14 -07:00
James R. Barlow b939584c7a quality: fixing typing issues 2020-06-29 01:45:45 -07:00
James R. Barlow bbd174071d readme: markdown cleanup 2020-06-29 01:45:27 -07:00
James R. Barlow e5b6fe1317 pyproject.toml: weird line wrapping? 2020-06-29 01:45:12 -07:00
James R. Barlow f15d9049eb install: add Mageia
Closes #586. Thanks to @yannick56
2020-06-26 23:28:26 -07:00
James R. Barlow 7630c93e5b install: drop Ubuntu 14.04 steps
Bit rot must have set in.
2020-06-26 23:27:42 -07:00
James R. Barlow 638d68aa8a docs: move Windows ahead of FreeBSD 2020-06-26 22:49:34 -07:00
James R. Barlow a92dde058a docs: promote one liner installs, reorg Windows 2020-06-26 22:47:44 -07:00
James R. Barlow 580f2ebb4b Python 3.9beta is now known to work (Fedora) 2020-06-26 00:06:58 -07:00
James R. Barlow 01cae7a584 docs: Update Fedora versions 2020-06-23 02:08:24 -07:00
James R. Barlow 66337813e6 Spell runslow correctly 2020-06-22 23:32:09 -07:00
James R. Barlow eb5a211e72 New hocrtransform test isn't platform stable - mark runslow 2020-06-22 16:59:59 -07:00
James R. Barlow 5142933120 v10.2.0 release notes
Closes #582, #584, #545
2020-06-22 16:37:51 -07:00
James R. Barlow 06ab114aa8 Update test cache 2020-06-22 16:31:34 -07:00
James R. Barlow 1257419465 test_hocrtransform: this test is worth not caching 2020-06-22 16:31:06 -07:00
James R. Barlow 30404f53f0 Add test to sanity check our pdf renderers 2020-06-22 16:18:38 -07:00
James R. Barlow 1ce8edbdfe hocrtransform: some text not included in output after Tesseract changes 2020-06-22 15:48:23 -07:00
James R. Barlow d4b704a0ae hocrtransform: refactor colors 2020-06-22 15:22:48 -07:00
James R. Barlow 2d64e1536d hocrtransform: refactor xpath manipulations 2020-06-22 14:44:34 -07:00
James R. Barlow c8b581ac31 hoctransform: remove deprecated element.getchildren()
Breaks Python 3.9.
2020-06-22 14:28:18 -07:00
James R. Barlow ad8dead7df Document that API accepts streams now 2020-06-22 14:27:27 -07:00
James R. Barlow c9bd87254e A few minor typing issues 2020-06-22 02:31:53 -07:00
James R. BarlowandJames R. Barlow f4cb424451 Support input/output streams at API level 2020-06-22 02:02:18 -07:00
James R. Barlow fef14778d5 Fix missing f-string in log message 2020-06-22 01:17:16 -07:00
James R. Barlow 86ec63f215 Decouple plugin manager forking from PdfContext/Pagecontext 2020-06-22 01:16:59 -07:00
James R. Barlow 5b10ec9d39 jobcontext.PdfContext: remove dead code, add annotations 2020-06-22 00:34:58 -07:00
James R. Barlow 800c75c4e5 Bump requirements (mainly for Docker's benefit) 2020-06-21 01:58:53 -07:00
James R. Barlow 24d64b04c3 Update Docker to Ubuntu 20.04 and jbig2-latest 2020-06-21 01:48:31 -07:00
James R. Barlow 48e2750551 Fix some tests that were failing in Docker 2020-06-21 01:48:13 -07:00
James R. Barlow e182c5f63e Update and sync .dockerignore, .gitignore
Also blacklist .* and whitelist the ones we want.
2020-06-21 01:25:59 -07:00
James R. Barlow 06d52326db Fix deleted path in .coveragerc 2020-06-21 01:24:23 -07:00
James R. Barlow ebfe4f0d29 Fix issue #582 - PDF/A acquires title "Untitled" after conversion 2020-06-20 02:01:16 -07:00
James R. Barlow ad22977c84 v10.1.1 release notes 2020-06-17 14:45:32 -07:00
James R. Barlow 6ac50646f0 Fix OMP_THREAD_LIMIT rounded down to 0 in some cases 2020-06-17 14:43:19 -07:00
James R. Barlow 24b6a4ad50 v10.1.0 notes 2020-06-16 00:55:28 -07:00
James R. Barlow e802896d4d unpaper: use PNG input where possible
Unpaper accepts PNG as input now, so avoid generating a huge
temporary PPM file if we can. If we
must create a PNG, compress it lightly to keep our temp usage down.
2020-06-16 00:50:18 -07:00
James R. Barlow 0b5a20e593 coverage: ignore type checking 2020-06-15 15:55:39 -07:00
James R. Barlow 642998ead6 sync: refactor preprocess image filtering 2020-06-15 15:26:41 -07:00
James R. Barlow 698aab4f75 Add a lot of type annotations 2020-06-15 15:20:50 -07:00
James R. Barlow 34231ac667 sync: refactor intermediate image production 2020-06-15 15:02:28 -07:00
James R. Barlow ddedf7cd2e For --clean-final, use same image as --clean if possible 2020-06-15 13:48:49 -07:00
James R. Barlow 9d127d354c docs: improve description of plugins 2020-06-15 12:51:49 -07:00
James R. Barlow 2d2a4894ab Some corrections to release notes 2020-06-15 12:51:28 -07:00
James R. Barlow 862861e3ca Fix error message in logging from repeated filtering
If logging somehow triggers PageNumberFilter multiple times, it would fail on the second occurrence.
2020-06-13 14:50:58 -07:00
James R. Barlow 892db88f0e test_two_languages: use narrower test 2020-06-12 14:33:02 -07:00
James R. Barlow eeb44f78cc Fix tests that failed on other platforms from previous fix 2020-06-12 12:59:46 -07:00
James R. Barlow 863835f660 v10.0.1 release notes 2020-06-12 12:11:21 -07:00
James R. Barlow 393c5a9ea4 Fix error on -l lang1+lang2 2020-06-12 12:10:29 -07:00
James R. Barlow c6b9a49cbb Fix tests that fail in CI 2020-06-10 17:08:00 -07:00
James R. Barlow 17a4831745 v10 release notes and dependencies 2020-06-10 14:27:47 -07:00
James R. Barlow 7caf1e85ff info: change "Scan" message 2020-06-10 12:11:37 -07:00
James R. Barlow f59a757e8b info: tidy handling of content streams 2020-06-10 12:09:24 -07:00
James R. Barlow 872bafad4b Reinstate quick test for text/no text
Partial revert of commit 991db17
2020-06-10 12:00:52 -07:00
James R. Barlow 8599400445 Only do page analysis on pages we will do OCR on 2020-06-10 11:33:27 -07:00
James R. Barlow b6eebadf05 Use pikepdf.open with block to manage PdfInfo 2020-06-10 11:32:46 -07:00
James R. Barlow a4e88eb8f0 Simplify plugin_manager pickling 2020-06-10 00:41:19 -07:00
James R. Barlow f6257c2183 subprocess: lru_cache version checks 2020-06-10 00:32:06 -07:00
James R. Barlow 64891c2fc3 Pre-release delinting 2020-06-09 15:27:14 -07:00
James R. Barlow fe156db41d Merge branch 'release/v10' into trialmerge 2020-06-09 15:12:56 -07:00
James R. Barlow 0f942fb714 Rename ocrmypdf.exec -> ocrmypdf._exec 2020-06-09 14:59:09 -07:00
James R. Barlow be8ca589d4 Move ocrmypdf.exec.run and friends to ocrmypdf.subprocess 2020-06-09 14:53:10 -07:00
James R. Barlow 3b6f6782f0 Remove tesseract_env, --tesseract-env 2020-06-09 00:39:53 -07:00
James R. Barlow 21c0e045cb Remove _OCRMYPDF_TEST_PATH environment variable 2020-06-09 00:30:13 -07:00
James R. Barlow ebbf68bd08 The big payoff: abolishing spoofing machinery 2020-06-09 00:08:20 -07:00
James R. Barlow 2059e916da Convert all ghostscript spoofs to test plugins 2020-06-09 00:00:25 -07:00
James R. Barlow c22f245606 Plugins must return not-None if they intend to stop builtin 2020-06-08 23:48:45 -07:00
James R. Barlow 7b9025f397 Convert generate_pdfa to plugin 2020-06-08 22:28:38 -07:00
James R. Barlow b109445215 Move Ghostscript rasterize_pdf to plugin 2020-06-08 17:10:27 -07:00
James R. Barlow fd1cd8e50a docs: explain --rotate-pages-threshold 2020-06-08 07:46:55 -07:00
James R. Barlow c6c70c2171 docs: Ubuntu 20.04 install instructions 2020-06-08 07:42:13 -07:00
James R. Barlow a9a473f2e5 Convert all tesseract cache usages to plugin 2020-06-05 17:55:18 -07:00
James R. Barlow 6268e2faff Begin replacing tests/spoof/tesseract_cache with plugin 2020-06-05 17:27:10 -07:00
James R. Barlow ec3f506500 Convert tesseract_badutf8 to plugin 2020-06-05 16:38:19 -07:00
James R. Barlow 00daa51a73 v9.8.2 release notes 2020-06-03 13:28:35 -07:00
James R. Barlow e60f4d3f43 docs: tidy Cygwin install 2020-06-03 13:27:05 -07:00
James R. Barlow 7460745f80 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-06-03 13:25:12 -07:00
James R. Barlow 5e14d5b0dd Fix test_report_file_size
Use more realistic test data
2020-06-03 13:24:55 -07:00
James R. Barlow d118132fa6 layout: look for text in XObjects too 2020-06-03 13:16:55 -07:00
5f47aac36f Add installation instructions for Windows/Cygwin64 (#571)
Co-authored-by: Jim Garrison <bitbucket@jhmg.net>
2020-06-03 13:16:23 -07:00
James R. Barlow c6b2fa8851 Remove unpaper spoof; no plugin needed 2020-06-02 02:42:14 -07:00
James R. Barlow 1b92f447c3 Convert tesseract_crash to plugin 2020-06-02 02:36:41 -07:00
James R. Barlow 82e7eb91d2 Tidy tesseract_noop 2020-06-02 01:50:02 -07:00
James R. Barlow 4f4ad0fb76 Convert tesseract_big_image_error to plugin 2020-06-02 01:49:47 -07:00
James R. Barlow 1d0b8641a0 Improve file size increase warning to account for changes to small files
Fixes #569
2020-06-02 00:35:59 -07:00
James R. Barlow daca919775 Mark pdfminer.six 20200517 as supported 2020-06-02 00:11:02 -07:00
James R. Barlow 1598f2f0e5 Abolish spoof_tesseract_noop 2020-06-01 03:07:53 -07:00
James R. Barlow 2b23f7ec73 tesseract_noop: begin implementing with plugin 2020-06-01 02:45:49 -07:00
James R. Barlow 6528234608 Fix tesseract_ocr.py errors 2020-06-01 02:27:27 -07:00
James R. Barlow 642ebc6098 Fix test that failed on Windows 2020-05-28 15:52:00 -07:00
James R. Barlow 74fdfeea3f v9.8.1 notes 2020-05-28 15:04:23 -07:00
James R. Barlow 3754185f56 Mark pdfminer.six 20200517 as supported 2020-05-28 15:01:51 -07:00
James R. Barlow df9f5157bd Fix shim_paths to account for unexpected files in Program Files\gs
Fixes #565
2020-05-28 14:58:41 -07:00
James R. Barlow aa060db5bc Refactor tesseract_env variable into the plugin
Removed all cases except one in api.py, which isn't worth solving because
it should be removed anyway.

This also fixes a logic error in the OMP_THREAD_LIMIT decision, api.py
did not use pass kwargs correctly so they never worked before.
2020-05-26 02:14:06 -07:00
James R. Barlow d43212d30b Refactor --language argument into set 2020-05-25 03:20:10 -07:00
James R. Barlow a0f9ca3a30 Move Tesseract options validation into plugin 2020-05-25 01:31:46 -07:00
James R. Barlow 0cefe886ec Update email 2020-05-19 16:12:36 -07:00
James R. Barlow f656c00f41 docs: Note about OCRmyPDF speed 2020-05-18 01:27:45 -07:00
James R. Barlow 03da34ee24 Test files needed! 2020-05-16 17:04:44 -07:00
James R. Barlow 9bccff4f88 Move Tesseract specific arguments to plugin 2020-05-16 03:24:31 -07:00
James R. Barlow 2bd586e093 Compare requested languages to OCR engine instead of tesseract directly
Also refactoring to facilitating validation needing the plugin manager.
2020-05-16 01:50:37 -07:00
James R. Barlow 9af94ac9b7 pipeline: use OCR engine abstraction instead of Tesseract 2020-05-16 01:28:56 -07:00
James R. Barlow 8174089c8b Begin transforming Tesseract into pluggable OCR engine 2020-05-14 03:54:21 -07:00
James R. Barlow 41eb54cc0a Standardize tesseract.generate_hocr and _pdf parameters 2020-05-14 03:23:25 -07:00
James R. Barlow 12a2f78c4d Fix validation of languages not using tesseract_env
And some related issues.
2020-05-14 03:19:22 -07:00
James R. Barlow d372f1f7fa Remove "skip page" from tesseract interface
Breaks tests/test_main.py::test_tesseract_missing_tessdata because
conftest.py does not update options.tesseract_env before testing options
for some reason, and tesseract.has_textonly_pdf raises an exception
instead of returning False as the test assumes.
2020-05-12 04:09:42 -07:00
James R. Barlow 6f5b75bcd0 Remove lru_cache on get_version
Does not play well with forking.
2020-05-12 03:51:48 -07:00
James R. Barlow a2d3e0b53e Convert remaining imports to absolute 2020-05-12 02:12:08 -07:00
James R. Barlow 7f67556995 ocrmypdf.__init__: Hide _HookimplMarker 2020-05-12 01:35:45 -07:00
James R. Barlow db8c37e58c Refactor ocrmypdf.exec.__init__.py 2020-05-12 01:34:10 -07:00
James R. Barlow a87c81a64f helpers: remove unnecessary isinstance test 2020-05-12 01:28:50 -07:00
James R. Barlow 4b986a5943 cli: make ArgumentParser._api_mode private 2020-05-12 01:28:36 -07:00
James R. Barlow 2fae9b655e Remove **kwargs from check_external_program; deprecated 2020-05-12 01:07:01 -07:00
James R. Barlow 2541f6cf89 Fix missing jbig2enc reported as error with -O3 instead of warning
Fixes #558
2020-05-12 01:05:57 -07:00
James R. Barlow 33b68454f3 watcher: cleanup getenv casting 2020-05-08 03:49:49 -07:00
James R. Barlow 977665d2b6 Delint some tests 2020-05-08 03:49:33 -07:00
James R. Barlow fd7497f00d Remove old function tesseract.v4() 2020-05-08 03:44:39 -07:00
James R. Barlow 790ff58f67 Add fix for bug in Windows Python 3.6/3.7
TypeError: argument of type 'WindowsPath' is not iterable
2020-05-07 22:19:21 -07:00
James R. Barlow 4b98ce391b docs: rename security->pdfsecurity so github won't misinterpret it 2020-05-07 03:54:27 -07:00
James R. Barlow 417dbd43f6 docs: plugin documentation 2020-05-07 03:53:37 -07:00
James R. Barlow 7a12908db9 Relocate example plugin 2020-05-07 03:27:39 -07:00
James R. Barlow 9462f0a28f graft: more refactoring 2020-05-07 02:59:24 -07:00
James R. Barlow e760622a5c graft: refactor 2020-05-07 02:03:42 -07:00
James R. Barlow 1b086f60a9 tesseract.py: api cleanup 2020-05-06 12:37:44 -07:00
James R. Barlow 85cbf94a6e Convert many uses of str paths to Path 2020-05-06 02:53:47 -07:00
James R. Barlow 6f4286e1b1 New hook: filter_page_image 2020-05-06 02:24:07 -07:00
James R. Barlow 39888ae8c9 Rename install_cli to add_options 2020-05-06 01:10:09 -07:00
James R. Barlow dd361ecd05 Support importing plugin by filename 2020-05-06 00:44:40 -07:00
James R. Barlow 32759c9025 Change argument from --plugins to --plugin 2020-05-06 00:43:40 -07:00
James R. Barlow 59440448ee Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-05-04 01:38:26 -07:00
Peter HoggandGitHub 51b54893ce docs: update Arch Linux install instructions (#540)
The python-pdfminer.six package is now available in the official Arch
repositories. The dependency will be automatically resolved when
installing the OCRmyPDF AUR package.
2020-05-04 01:37:58 -07:00
James R. Barlow 1f3665f614 docs: remove reference to brewfile 2020-05-03 16:10:26 -07:00
James R. Barlow 75c34b873a optimize: convert from executor to progress pool 2020-05-03 02:04:57 -07:00
James R. Barlow fe4296c53b safe_symlink: remove deprecated params 2020-05-03 00:53:47 -07:00
James R. Barlow c85278b31d Delinting 2020-05-03 00:53:29 -07:00
James R. Barlow 5dbc080fa0 Rename PDFContext->PdfContext 2020-05-02 04:32:46 -07:00
James R. Barlow e02f6c1e97 Support plugin invocation with API 2020-05-02 03:34:31 -07:00
James R. Barlow 8c9a8fc85c pluginspec: avoid circular reference 2020-05-02 03:32:55 -07:00
James R. Barlow 23d558ad8c Allow plugins to add command line arguments 2020-05-02 01:37:24 -07:00
James R. Barlow be107b4fed Set up filter_ocr_image hook 2020-05-01 02:56:41 -07:00
James R. Barlow 8d2535e327 Get pluggy to work with forking workers 2020-05-01 02:39:50 -07:00
James R. Barlow 5eb4fe0052 Refactor plugin setup to get_plugin_manager 2020-05-01 02:18:31 -07:00
James R. Barlow d8ff4485f8 Move samefile to helpers 2020-05-01 02:18:11 -07:00
James R. Barlow 82bce463ae Start pluggy-based plugin system 2020-05-01 02:15:23 -07:00
James R. Barlow 016dfd420c Add warning if problematic --tesseract-pagesegmode is selected
Fixes #549
2020-04-30 04:12:11 -07:00
James R. Barlow b59e761a14 v9.8.0 release notes 2020-04-28 02:40:17 -07:00
James R. Barlow 17cd655752 Don't utf-8 decode tesseract --print-parameters
Output not guaranteed to be UTF-8.

Fixes #543.
2020-04-28 02:37:17 -07:00
James R. Barlow b840b16c82 Remove tesseract_badutf8.py
Should have been removed in 9db01c7
2020-04-28 02:35:23 -07:00
James R. Barlow 8f5c95f0f4 Remove last vestiges of command line usage of qpdf - change to check_pdf 2020-04-26 05:33:26 -07:00
James R. Barlow 168fc60774 Update release notes with v10 changes 2020-04-26 05:14:59 -07:00
James R. Barlow c84d0f606d ghostscript: remove deprecated argument from generate_pdfa 2020-04-26 05:11:11 -07:00
James R. Barlow 8b54ce338f setup: remove deprecated message about removeal of --force parameter 2020-04-26 05:09:42 -07:00
James R. Barlow 18c4aa10bf Adjust number of workers for concurrent page scanning 2020-04-26 04:21:15 -07:00
James R. Barlow 991db17fde Remove Ghostscript-based text extraction
While faster than Python based methods, we've outgrown the limited
amount of information Ghostscript provides with this feature, and it
repeats an analysis we have to do anyway to learn what images are
present.
2020-04-26 04:02:07 -07:00
James R. Barlow 2c07515907 macOS - use spawn for multiprocessing
See bpo-33725. This is the default for 3.8, opt-in for 3.7 and older.
2020-04-26 03:49:40 -07:00
James R. Barlow 27a3b80376 Use once-per-worker pikepdf init 2020-04-26 03:49:20 -07:00
James R. Barlow 8c381a0227 Replace task_initargs with use of partial() 2020-04-26 03:49:20 -07:00
James R. Barlow 86145a8c76 Some wrong with forking worker_pdf, just open it once per page for now 2020-04-26 03:49:20 -07:00
James R. Barlow 7513f5425c Fix some broken tests 2020-04-26 03:49:20 -07:00
James R. Barlow af3c3c6466 Further refactoring of concurrency concerns 2020-04-26 03:49:20 -07:00
James R. Barlow db3e75e33e Refactor multiprocessing pool 2020-04-26 03:49:13 -07:00
James R. Barlow ce49fc26dd Do pikepdf.open() once instead of per worker 2020-04-26 03:42:13 -07:00
James R. Barlow d0d0a98dca First cut at concurrent page scan
Improvement appears on 168 page file. Needs refactoring
2020-04-26 03:42:13 -07:00
James R. Barlow 3834d1a0bf azure: use brew python instead 2020-04-26 00:58:38 -07:00
James R. Barlow 33e982b3fd azure: add certifi, openssl for macOS 2020-04-26 00:37:14 -07:00
James R. Barlow 43d650e78c Fix issue where only first PNG-style image would be optimized 2020-04-25 03:50:11 -07:00
James R. Barlow b4c65c5781 Update requirements 2020-04-25 03:49:34 -07:00
James R. Barlow d96867e6ab watcher: add polling and log level adjustment 2020-04-24 04:14:44 -07:00
James R. Barlow 0a5108e704 install: clarify that old ocrmypdf should be removed from Ubuntu 18.04
Closes #526
2020-04-24 04:14:19 -07:00
James R. Barlow 94c52a6fa3 Refactor 'xyres' into Resolution 2020-04-24 04:12:05 -07:00
James R. Barlow 57771f06a3 Refactor xy-pair for resolution to tuple 2020-04-16 15:38:33 -07:00
James R. Barlow 58abb5785c pytest picky about list vs tuple 2020-04-15 03:16:51 -07:00
James R. Barlow 509e75eaff v9.7.2 release notes 2020-04-15 02:56:46 -07:00
James R. Barlow 0c50eedb2a Support pdfminer.six 20200402 2020-04-15 02:55:22 -07:00
James R. Barlow 4581027246 Drop support for pdfminer.six 20181108
This version required a patch that has since been mainlined, and also did not
declare its dependency on chardet
correctly. We can remove both hacks now.
2020-04-15 02:50:36 -07:00
James R. Barlow 31b5f63f85 hocrtransform: cleanup/PEP8
Some API breaking changes.
2020-04-15 02:48:56 -07:00
James R. Barlow 957fb1494e pytest picky about list vs tuple 2020-04-15 02:26:20 -07:00
James R. Barlow 9e3e4f2687 Improve help text about aborting due to text 2020-04-15 02:17:55 -07:00
James R. Barlow 2155bcacb4 Loosen test language requirements - eng/deu 2020-04-15 00:30:38 -07:00
James R. Barlow 346da95899 Suppress loglevel since we have color now 2020-04-15 00:09:36 -07:00
James R. Barlow f4f7946a0c Add colored logs 2020-04-15 00:05:38 -07:00
James R. Barlow c2919f2e1c Reinstate logging of page numbers 2020-04-15 00:05:23 -07:00
James R. Barlow a63d624052 Improve logging of subprocess output 2020-04-15 00:04:43 -07:00
James R. Barlow af91489376 Remove safe_symlink log= warning 2020-04-14 23:59:33 -07:00
James R. Barlow d146d2b65c The Great Logging Refactor
Remove all instances of logger object being passed as parameters.
This was a holdover from ruffus, and complicated a lot of simple things.
2020-04-14 23:59:33 -07:00
James R. Barlow 4ff4ed24a8 Refactor Windows executable shims 2020-04-14 23:59:33 -07:00
James R. Barlow c38ff90081 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-04-14 23:55:01 -07:00
James R. Barlow 4c029e973f Fix isinstance(..,str) 2020-04-14 23:53:52 -07:00
Lars K.W. GohlkeandGitHub 21cf9029e8 docs: Set ownership when using docker image (#518) 2020-04-14 23:32:01 -07:00
James R. Barlow 4a640b8dcd Fix language argument not working as list
Fixes #523
2020-04-14 23:18:52 -07:00
James R. Barlow 9471bc8921 Fix versions with leading v, e.g. v5.0 2020-04-10 13:42:33 -07:00
James R. Barlow 7fe06c64fc v9.7.1 release notes 2020-04-10 13:00:19 -07:00
James R. Barlow d13d70fd56 Fix version checker failing for qpdf 10.0.0
Fixes #527
2020-04-10 13:00:19 -07:00
James R. Barlow 58ec56180a Add a few more type annotations to public APIs 2020-04-10 13:00:19 -07:00
James R. Barlow 32a88f1bad docs: warn that AWS Lambda doesn't work 2020-04-10 13:00:19 -07:00
James R. Barlow 99ef42940c docs: warn that Windows users should use an ifmain guard 2020-04-10 13:00:19 -07:00
jbarlow83andGitHub c152710617 Update issue templates 2020-04-04 15:41:53 -07:00
James R. Barlow 8de0f9b86f v9.7.0 release notes 2020-03-29 22:45:25 -07:00
James R. Barlow 23bc3d3a29 tests: workaround for Ghostscript 9.52 txtwrite problem 2020-03-29 22:45:16 -07:00
James R. Barlow 8307832ce9 tests: add force OCR to a file with text that Ghostscript doesn't see
For gs 9.52 support.

Also refactor use of pikepdf.open() to use with blocks.
2020-03-29 22:44:27 -07:00
James R. Barlow dd1cf567db watcher: Fix JSONDecodeError if OCR_JSON_SETTINGS not set
Fixes #516
2020-03-29 21:58:37 -07:00
James R. Barlow 2490be8490 Fix debug.log not being deleted on Windows (probably)
Fixes #515
2020-03-29 21:53:56 -07:00
James R. Barlow 85e6c6669a docs: Add username to WSL instructions
Fixes #519
2020-03-29 21:16:24 -07:00
James R. Barlow 00498282f5 validation: blacklist Ghostscript 9.51 too 2020-03-24 21:27:18 -07:00
James R. Barlow e4cc9fcba7 Wrong number of threads to use shown when OMP_THREAD_LIMIT is defined 2020-03-23 01:06:55 -07:00
James R. Barlow a4555b1dae Add halftone mask to leptonica 2020-03-18 23:09:39 -07:00
James R. Barlow f35a2303bb info.py: linearize O(n^2) search for use images on a page 2020-03-18 22:59:18 -07:00
James R. Barlow 82142fe5ef Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-03-16 04:11:03 -07:00
James R. Barlow 9be533b5f4 watcher: allow all parameters to ocrmypdf.pdf to be passed by JSON 2020-03-15 21:45:51 -07:00
James R. Barlow 99653fcd32 optimize: consider ICCBased 1 bit for optimization 2020-03-15 02:20:44 -07:00
James R. Barlow 5442c97ed8 Consult ICC profile when determining image colorspace 2020-03-11 04:03:09 -07:00
tlwhitecandGitHub 0165255bd9 fix install instructions for Ubunti 16.04 (#507)
`pip3` defaults to the system's outdated version which downloads wrong qpdf package.
2020-03-11 02:57:37 -07:00
James R. Barlow 378e4dae3b Expand documentation for subprocess.run() from test 2020-03-04 13:37:44 -08:00
James R. Barlow cdf5afa753 reqs: update pikepdf version 2020-03-03 11:56:10 -08:00
James R. Barlow a2deee4920 v9.6.1 release notes 2020-03-03 03:40:33 -08:00
James R. Barlow 1efa79cce2 Remove potentially non-free file logo.afdesign 2020-03-03 03:40:33 -08:00
James R. Barlow b3b61c152c Handle malformed DocumentInfo (#497)
User submitted a PDF in which /Trailer /Info pointed to the XMP metadata
block instead of a DocumentInfo dictionary. Fix and add test.
2020-03-03 03:27:01 -08:00
James R. Barlow e429c3d729 docs: install cleanup 2020-03-03 03:25:43 -08:00
James R. Barlow 8308b20096 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-03-03 02:25:58 -08:00
James R. Barlow 8b41f60b6e docs: docker prefers .yml not .yaml 2020-03-03 02:25:21 -08:00
AlexandGitHub d56f749017 Improve ocrmypdf.bash completions on macOS (#504)
Fixes #502
2020-03-03 02:22:01 -08:00
James R. Barlow 9f31774aa9 docs: document --pages 2020-03-03 02:15:48 -08:00
James R. Barlow 7d55f6e01f docs: extract example files from batch.rst 2020-03-03 02:15:35 -08:00
James R. Barlow c3bd2f296d docs: fix Docker syntax to use stdin/stdout properly 2020-03-03 02:14:50 -08:00
James R. Barlow e40c60d4d8 watcher: add self to copyright 2020-03-03 00:59:57 -08:00
James R. Barlow 3960232ae0 docs: more clarifications 2020-03-03 00:59:48 -08:00
James R. Barlow 5fbb3fc6ac Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-03-01 23:05:10 -08:00
deisiandGitHub 0b1db8fccd Fixes docker-compose.yaml file (#499)
Fixes https://github.com/jbarlow83/OCRmyPDF/issues/498
2020-03-01 23:01:39 -08:00
James R. Barlow 0417610f9b docs: some mild improvements 2020-02-25 22:23:58 -08:00
James R. Barlow 43a23e3695 Disable Travis 2020-02-25 22:22:57 -08:00
James R. Barlow c4ca572afd Merge branch 'pigmonkey-aur-manual' 2020-02-25 22:21:35 -08:00
Pig Monkey e04e4565a9 Demonstrate installing the AUR package without a helper
This describes how to use the AUR package on a minimal install, as per
the discussion in #494.

There may be formatting mistakes. I don't use RST myself, so I wrote the
instructions in Markdown, converted via Pandoc, and gave the output a
quick comparison against the rest of the installation docs.
2020-02-25 18:59:49 -08:00
knobixandGitHub 2391fb0be0 Update installation instructions for FreeBSD (#493)
Python 3.7 is the new default version since 2020Q1 which is reflected in
the new prefix (= py37-).

Also update the current available FreeBSD versions:

* FreeBSD 11.2-RELEASE has reached its End-of-Life in 2019Q4
* FreeBSD 12.1-RELEASE was also introduced in 2019Q4
2020-02-24 23:40:25 -08:00
James R. Barlow c16f79d51b docs: add Docker compose configuration for watchdog 2020-02-18 02:50:57 -08:00
James R. Barlow e3e888efde Readme: Add another heise article 2020-02-18 02:41:28 -08:00
James R. Barlow 84f7e34ace Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-02-18 02:11:40 -08:00
Ivan KuchinandGitHub 32e2175891 Docker image includes also French, Portuguese and Spanish (#491) 2020-02-18 02:10:01 -08:00
James R. Barlow 975abfde9a docs: archlinux install - yaourt is gone 2020-02-18 02:08:58 -08:00
James R. Barlow d929ed76c2 docs: typo 2020-02-12 00:07:24 -08:00
James R. Barlow 4a27124eab Simplify metadata for invalid xml in output
Removes possibly non-free resource enron1.pdf.
2020-02-12 00:07:18 -08:00
James R. Barlow 683ffb84e8 Update reqs 2020-02-10 01:20:33 -08:00
James R. Barlow 2f2602357b v9.6.0 notes updated 2020-02-10 01:20:12 -08:00
James R. Barlow a5f429f499 Merge branch 'pr479' 2020-02-10 01:10:30 -08:00
James R. Barlow bdb7f92131 ifmain -> main() 2020-02-10 01:10:12 -08:00
James R. Barlow 09f15ac4c0 v9.6.0 notes 2020-02-10 01:01:49 -08:00
James R. Barlow 4fdbf55c11 setup: approve pdfminer.six 20200124 2020-02-09 23:50:56 -08:00
James R. Barlow fe2b07652b docs: simplify/fix Ubuntu 18.04 install instructions 2020-02-09 23:48:53 -08:00
James R. Barlow f6d7aa6e33 Refactor page rotation and re-enable message at info level 2020-01-30 17:35:20 -08:00
James R. Barlow a005d14f91 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2020-01-30 16:24:16 -08:00
Matthias BraunandGitHub 6f66232d44 Fix typos, add instructions for training data (#477) 2020-01-30 16:24:41 -08:00
James R. Barlow b8a780d684 Wait for file based on pikepdf 2020-01-30 12:40:48 -08:00
James R. Barlow 82f393dd09 Order of events 2020-01-30 12:40:19 -08:00
James R. Barlow 4952af1604 watcher: some refactoring 2020-01-28 12:56:19 -08:00
James R. Barlow bcf77375c0 Fix grammar in output message 2020-01-28 07:33:28 -08:00
Ian Alexander 3eab161771 Update logging and env var extensibility 2020-01-20 10:45:28 -08:00
Ian Alexander b7f38e976b Watched folder bug fixes, new flags, and docs updates. 2020-01-20 00:20:29 -08:00
James R. Barlow a6567f2ae4 v9.5.0 release notes revised 2020-01-18 01:48:33 -08:00
James R. Barlow e860c56b75 Fix regression: metadata updates not taking effect 2020-01-17 23:01:37 -08:00
James R. Barlow 2e15d52895 v9.5.0 release notes 2020-01-17 03:11:33 -08:00
James R. Barlow ce97af5a79 Add OCR quality measurement API 2020-01-17 03:10:27 -08:00
James R. Barlow 3831c4cd4d Refactor metadata_fixup 2020-01-14 01:10:15 -08:00
James R. Barlow 61a2674317 Skip test that needs chmod when on Windows 2020-01-06 02:36:04 -08:00
James R. Barlow 9ad8cbf1f6 Fix assert that depends on POSIX-y file handling 2020-01-06 02:02:05 -08:00
James R. Barlow 123fde174d Don't use debug.log in pytest
pytest does not reset the state of logging if we install a file handler,
which will cause FileNotFoundError after the temporary folder is removed.

Semi-related:
https://github.com/pytest-dev/pytest/issues/5502
2020-01-06 01:46:19 -08:00
James R. Barlow fd991a2380 Allow pdfminer.six 20200104 and update recommended versions 2020-01-05 21:37:28 -08:00
James R. Barlow 6f5d77d930 Also generate log file in temp folder on verbose mode 2020-01-05 21:33:32 -08:00
James R. Barlow 5169ac633b docs: mention pdfgrep too 2020-01-05 21:32:36 -08:00
James R. Barlow 5b6ab1e003 lept: improve lib not found error message
Closes #471
2020-01-05 01:05:53 -08:00
James R. Barlow 8f984bf958 docs: add note on limitations of sidecar file 2020-01-04 16:43:13 -08:00
James R. Barlow 9c5f0d0ec6 Eliminate last use of PyPDF2 from test suite 2020-01-04 16:32:01 -08:00
James R. Barlow 32041c43e1 tests: improve tesseract coverage 2020-01-04 02:35:14 -08:00
James R. Barlow 599028bebb tesseract: don't explicitly set lstm_use_matrix
Apparently tesseract does this own its own as needed.
2020-01-04 01:17:33 -08:00
James R. Barlow 6faa8f7221 logging: always log process arguments and stderr when at debug
Also remove ad-hoc logging of this information.
2020-01-01 16:48:48 -08:00
James R. Barlow a4dc5e365f logging: fix incorrect usage: logging.Logger() 2020-01-01 16:47:36 -08:00
James R. Barlow e2a563cc76 logging: create a debug log when -k parameter is issued 2020-01-01 16:47:15 -08:00
James R. Barlow 1037d73efb tests: use smaller files for ghostscript 2019-12-31 17:20:28 -08:00
James R. Barlow aeb7b142a9 tests: skip tests not compatible with coverage
For reasons not entirely clear, stdout will get some data injected when
pytest-cov is running. Our tests that
check for clean stdout need to ignore this.

We check for an environment variable that is defined only when coverage is
running.
2019-12-31 17:10:51 -08:00
James R. Barlow 422ea9777e Remove session scope from fixtures
pytest seems to prepare os.environ in complex ways, so we want to ensure
these fixtures are not reused.
2019-12-31 17:09:23 -08:00
James R. Barlow 2f1c743227 Rewrite main pool loop
pytest-cov documentation recommends using explicit
management of multiprocessing.Pool rather than the context manager.
This is supposed to work better for collecting coverage data, particularly
on Windows.
2019-12-31 16:23:41 -08:00
James R. Barlow 96ee21aee9 Try to set up subprocess coverage better 2019-12-31 15:39:45 -08:00
James R. Barlow 4b759af6ff tests: fix problems with ghostscript spoofers 2019-12-31 15:33:03 -08:00
James R. Barlow 25d2b0cda4 test: environment warnings/cleanup 2019-12-30 22:38:50 -08:00
James R. Barlow 16dd8b54a8 ghostscript: don't delete output_file that will never exist
We stream output now, so no point in deleting.
2019-12-30 22:38:38 -08:00
James R. Barlow c4dc5269d2 tests: remove some obscure things from coverage 2019-12-30 21:16:16 -08:00
James R. Barlow c36e9950ae tests: test TqdmConsole 2019-12-30 17:51:09 -08:00
James R. Barlow 0c0d53b10f tests: AcroForm test case did not work correctly; fixed 2019-12-30 17:50:32 -08:00
James R. Barlow 63de7e1677 Improve error message for unreadable input files 2019-12-30 16:14:52 -08:00
James R. Barlow b0e92760a2 tests: add coverage for helpers 2019-12-30 15:52:10 -08:00
James R. Barlow 054c0773a3 Update completions 2019-12-29 02:40:55 -08:00
James R. Barlow 89aa78b724 docs: fix obsolete statement to "brew install tesseract-lang"
Closes #469
2019-12-29 02:37:10 -08:00
James R. Barlow 708113a514 Windows: Remove Program Files cache from ocrmypdf.exec
@lru_cache doesn't work here, so let's just remove it.
2019-12-29 02:36:20 -08:00
James R. Barlow 95ef5410c2 azure: tweak windows script 2019-12-28 16:12:53 -08:00
James R. Barlow 868b3b4abd exec/init: os.get_exec_path() returns list not str 2019-12-28 16:11:08 -08:00
James R. Barlow 045bdff95a azure: homebrew broke something to do with python@2? 2019-12-28 16:10:43 -08:00
James R. Barlow d12b27ac1d v9.3.0 release notes 2019-12-28 15:42:24 -08:00
James R. Barlow e4e00de79f Add improved example demonstrating watched folder functionality
Closes #466
2019-12-28 15:37:42 -08:00
James R. Barlow a53a3937c2 Fix exception on parsing Ghostscript error messages 2019-12-20 11:25:45 -08:00
James R. Barlow 343424b4d2 azure: only publish code coverage for macOS
macOS (due to Homebrew) currently has the most comprehensive code
coverage. Azure's code coverage feature does not merge code coverage,
so last task to finish wins.
2019-12-20 10:56:10 -08:00
James R. Barlow c5edff2c2f Sort imports 2019-12-19 15:31:18 -08:00
James R. Barlow 8c5f8b8ddd Add isort to precommit 2019-12-19 15:29:38 -08:00
James R. Barlow 39da931a56 Look in Program Files for executables and liblept5.dll 2019-12-19 12:11:32 -08:00
James R. Barlow 9fe354359b Generally update documentation about available platforms 2019-12-19 00:27:37 -08:00
James R. Barlow facc4750bc Remove command line qpdf from azure and travis 2019-12-12 00:14:05 -08:00
James R. Barlow 437c235738 v9.2.0 release notes and docs 2019-12-11 13:13:51 -08:00
James R. Barlow 9559b0b186 Use pikepdf to perform qpdf.check() 2019-12-11 01:21:15 -08:00
James R. Barlow 91456e19a4 pdfa.py: Fix misleading comment 2019-12-11 01:05:47 -08:00
James R. Barlow a2d89f67c4 Improve help messages for Windows 2019-12-10 01:44:00 -08:00
James R. Barlow f34130d193 Fixed case where page image was not converted to JPEG
If a preprocessing option was used,
and all original images on the page were JPEGs,
and --output-type=pdf, then
images would saved as Flate instead
of converted to JPEG.
2019-12-10 01:07:59 -08:00
James R. Barlow c5571388e2 Improve test coverage of _sync.py 2019-12-10 01:06:27 -08:00
James R. Barlow 9af59c0d6d docs: improvements for Windows 2019-12-09 21:39:01 -08:00
James R. Barlow 55ae838cb7 azure: fix extra build step 2019-12-09 21:11:50 -08:00
James R. Barlow c434b97f55 docs: more install notes 2019-12-09 16:18:01 -08:00
James R. Barlow 607eee198d tests: split out preprocessing tests 2019-12-09 16:18:01 -08:00
James R. Barlow 5e2a7f8a56 tests: speed up several slow tests 2019-12-09 16:17:57 -08:00
James R. Barlow fd9550acda Add Azure Pipelines CI/CD 2019-12-09 16:17:33 -08:00
James R. Barlow b354511ac9 ghostscript: document need to write to stdout when using txtwrite 2019-12-09 16:17:10 -08:00
James R. Barlow 7be293f628 Address tests that fail on Windows with Python 3.7 or 3.6 2019-12-09 16:17:10 -08:00
James R. Barlow 65855dc14c Fix close_fds=True on Windows Python 3.6 2019-12-09 16:17:10 -08:00
James R. Barlow cac4a8b9b6 Suppress duplicate error messages from Ghostscript 2019-12-06 15:14:15 -08:00
James R. Barlow 17d97b354a Ignore mypy cache 2019-12-06 15:11:10 -08:00
James R. Barlow 1c1b60fa9f Add typing hints for ocr() function 2019-12-06 15:10:54 -08:00
James R. Barlow 6b745d892f Merge branch 'windows' 2019-12-06 15:09:09 -08:00
James R. Barlow fbf271a3ec Remove Tesseract < 4.0 specific check 2019-12-06 15:06:02 -08:00
James R. Barlow 8077718804 Possible fix to loss of log adapter state 2019-12-06 15:05:58 -08:00
James R. Barlow 66bda3420a docs: cause about using Windows in production 2019-12-06 15:03:20 -08:00
James R. Barlow f6510e2b15 Document function of symlink shim 2019-12-06 15:00:12 -08:00
James R. Barlow 51abd79136 Tesseract no longer posts an error message if config file not found 2019-12-04 21:35:28 -08:00
James R. Barlow 5607429d9a tests: error message from tesseract change 2019-12-04 21:31:01 -08:00
James R. Barlow b8b7ecfe7f Fix DecompressionBomb related errors due to Windows process differences 2019-12-04 21:10:27 -08:00
James R. Barlow d4abe88452 docs: sketch Windows install procedure 2019-12-04 21:01:09 -08:00
James R. Barlow cb3cfaa055 Add Windows install advice 2019-12-04 21:01:09 -08:00
James R. Barlow 9db01c7ff5 Remove test_bad_utf8
Due to difficulties of getting this to work on Python 3.8, Windows, and
high probability that this behavior is now gone from Tesseract 4.0+.

Originally added in 2017.
2019-12-04 21:01:09 -08:00
James R. Barlow d0301813cc ghosttext: mention page number differences 2019-12-04 21:01:09 -08:00
James R. Barlow cff37bf681 Make test_german more Windows-friendly 2019-12-04 21:01:09 -08:00
James R. Barlow 66d04dd6e3 Don't expect filenames to be replicated on NT 2019-12-04 21:01:09 -08:00
James R. Barlow 06a1f987d4 Use _OCRMYPDF_TEST_PATH for testing and .py stubs to simulate symlinks 2019-12-04 21:01:06 -08:00
James R. Barlow e51e21c6b6 ghostscript: Refactor checking for executable name on Windows 2019-12-04 21:01:06 -08:00
James R. Barlow c5fa72bd4e ghostscript: use run(check=True) for more consistent error handling 2019-12-04 21:01:00 -08:00
James R. Barlow bf99587aa1 ghostscript: use correct executable name on Windows 2019-12-04 17:37:38 -08:00
James R. Barlow d249aef57d ghostscript: don't use NamedTemporaryFile
Temporary files are more awkward for Windows.
2019-12-04 17:37:38 -08:00
James R. Barlow 43ab7c88d7 Remove os_environ() context manager 2019-12-04 17:37:38 -08:00
James R. Barlow ca9669742d Move gs tests to test_ghostscript 2019-12-04 17:14:27 -08:00
James R. Barlow 8a1dddc3ee Don't worry about closed streams on Windows 2019-12-04 17:14:27 -08:00
James R. Barlow 0cd424ffcb Enforce str-only environment for Windows since it's more strict 2019-12-04 17:14:27 -08:00
James R. Barlow fde550f9a7 test: Replace many instances of run_ocrmypdf in subprocess with inline 2019-12-04 17:14:27 -08:00
James R. Barlow a3726e4ce3 Fix test_metadata: use mmap in a Windows and POSIX compatible way 2019-12-04 17:13:52 -08:00
James R. Barlow 4ab0a8ff35 Fix test_single_page_inline_image - remove temp file 2019-12-04 17:13:51 -08:00
James R. Barlow 37f6f72df3 tests: a few Windows fixes 2019-12-04 17:13:51 -08:00
James R. Barlow 3f92867ae6 Fix TypeError "environment can only contain strings"
Apparently Windows Python doesn't coerce pathlib.Path to str.
2019-12-04 17:13:51 -08:00
James R. Barlow e63503d64b Fix difference in Windows error message breaking test_no_languages 2019-12-04 17:13:51 -08:00
James R. Barlow 17d20309c7 leptonica: fix missing Leptonica error message for Windows
Since it has the unintuitive fix of adding Tesseract to PATH.
2019-12-04 17:13:51 -08:00
James R. Barlow fe7c69ce95 leptonica: don't open files by name; use memory buffers
Avoids encoding issues and makes error trap unnecessary in some cases.
2019-12-04 17:13:51 -08:00
James R. Barlow 9baccee8c5 leptonica: Handle API change for pixFindPageForeground 2019-12-04 17:13:51 -08:00
James R. Barlow d5bb9929f3 leptonica: Use Windows name for DLL
Thanks to @dibu28
2019-12-04 17:13:51 -08:00
James R. Barlow 72d3ee3a87 Refactor symlink usage to support Windows 2019-12-04 17:13:51 -08:00
James R. Barlow 17c419dfcb compile_leptonica: move to correct location 2019-12-04 17:04:08 -08:00
James R. Barlow 84cc49b14b black: don't reformat _leptonica.py 2019-12-04 17:04:08 -08:00
James R. Barlow b7f63bc93d Make devnull check compatible with Windows 2019-12-04 17:04:08 -08:00
James R. Barlow ad9a3b5302 Update version of pdfminer.six supported 2019-12-04 17:04:08 -08:00
James R. Barlow 4e4bcaf243 Improve pre-commit checks 2019-11-25 14:38:23 -08:00
James R. Barlow 11afe3507f black: don't reformat _leptonica.py 2019-11-25 14:34:36 -08:00
James R. Barlow 7691ba8535 v9.1.1 release notes 2019-11-18 15:17:00 -08:00
James R. Barlow b787a369ee Fix reference to Alpine apk add 2019-11-18 15:13:42 -08:00
James R. Barlow 9fb8b267af docker: use get-pip to install pip
Smaller download, needed for manylinux2010.
2019-11-16 15:21:45 -08:00
James R. Barlow 0a08d6ce1f Update version of pdfminer.six supported 2019-11-13 01:45:06 -08:00
James R. Barlow f517efe819 docs: wsl - get-pip.py 2019-11-12 15:01:15 -08:00
James R. Barlow 5f5421f23d test: further fixes to test_report_file_size 2019-11-12 01:14:21 -08:00
James R. Barlow 703b6db95c test: fix test_report_file_size 2019-11-11 23:00:08 -08:00
James R. Barlow 000040d497 v9.1.0 release notes 2019-11-11 22:39:33 -08:00
James R. Barlow 5bd6665b49 Use pikepdf 1.7.0 to improve Python 3.8 support 2019-11-11 22:36:38 -08:00
James R. Barlow 1c303afe21 docs: fix installation instructions for pikepdf manylinux2010 wheels 2019-11-11 22:22:30 -08:00
James R. Barlow 11a5c80917 travis: enable Py 3.8 2019-11-11 22:19:15 -08:00
James R. Barlow 9b2ab92913 tesseract: fix exception when logger is RootLogger 2019-11-11 22:19:01 -08:00
James R. Barlow 0c4b69ec5a Fix lint warning about missing cur_item 2019-11-09 14:56:43 -08:00
James R. Barlow 45bea1c0e0 Import and docstring cleanup 2019-11-09 14:56:30 -08:00
James R. Barlow db914d4cd1 Report missing optional dependencies as possible cause of file size increase 2019-11-09 14:21:57 -08:00
James R. Barlow df4a8faecd docs: mention systemd for batches 2019-11-08 03:24:54 -08:00
James R. Barlow 1273e7aeda docs: document optimization 2019-11-08 03:22:28 -08:00
James R. Barlow e13a673b1a docs: mention how to suppress progbar 2019-11-08 02:59:02 -08:00
James R. Barlow 979b0bcaed tesseract: refactor logging 2019-11-05 15:38:09 -08:00
James R. Barlow 3438afaffe Support pdfminer.six 20191020 2019-11-04 03:15:59 -08:00
James R. Barlow 681fa039cc Update release notes; disable Py3.8 test again 2019-11-04 03:00:15 -08:00
James R. Barlow 69e80f1545 docker-compose.test does not seem to be ready for production use 2019-11-04 02:58:57 -08:00
James R. Barlow 983835cce4 docs: add remark about optimizing without OCR 2019-11-04 02:32:29 -08:00
James R. Barlow 6c23b137e2 Docker: relocate dockerfile 2019-11-04 02:27:30 -08:00
James R. Barlow d656b2b3f2 docs: remove comment about Ubuntu image
[ci skip]
2019-11-04 02:08:42 -08:00
James R. Barlow 031b800aac Docker autotest: fix, maybe? 2019-11-04 02:04:07 -08:00
James R. Barlow 05eb85ee77 Docker: try adding automated test 2019-11-04 01:23:54 -08:00
James R. Barlow 4da5214ca9 Drop support for unpaper 6.1 on Ubuntu 14.04 2019-11-04 00:09:04 -08:00
James R. Barlow 1ee829dd59 Travis: enable Python 3.8 testing 2019-11-04 00:05:18 -08:00
James R. Barlow 99db5d91ae Fix issue "MANIFEST.in exists" by removing MANIFEST.in
MANIFEST.in is always an issue
2019-11-04 00:03:49 -08:00
James R. Barlow 3a4490ee36 Dockerfile: fix jbig2 not copied over 2019-11-03 23:52:08 -08:00
James R. Barlow a492e3b472 Dockerfile: fix errors are trying to build unneeded cached wheels 2019-11-03 23:51:55 -08:00
James R. Barlow c3719d3b72 Dockerfile: remove venv from Ubuntu image; tweak reqs 2019-11-03 23:39:40 -08:00
James R. Barlow ad48fc6415 Remove Alpine Docker image 2019-11-03 22:35:15 -08:00
James R. Barlow 7f8018ffde Mention that v9.0.4 requires a source install for Py3.8 for now, due to lack of CI availability 2019-11-03 01:49:36 -08:00
James R. Barlow 80651fe12c Fix test suite error 2019-10-24 18:17:03 -07:00
James R. Barlow a58209e895 Disable Py3.8 for now 2019-10-24 18:16:47 -07:00
James R. Barlow 775b958c55 Update release notes 2019-10-24 16:58:39 -07:00
James R. Barlow cdcdd16865 Require Pillow 6.2.0 based on security vulnerability report in older versions 2019-10-23 12:27:29 -07:00
James R. Barlow b332d76782 Mention when we default to English and the system locale is not English
Closes #337
2019-10-22 01:49:38 -07:00
James R. Barlow 3660007fc8 travis: Python 3.8, osx_image 2019-10-20 04:06:13 -07:00
James R. Barlow b55d7e57af Python 3.8 updates 2019-10-20 03:20:54 -07:00
James R. Barlow 6e99e7b346 Use lstm_use_matrix for --user-words,patterns 2019-10-20 00:49:11 -07:00
James R. Barlow 4d26867dee Delinting 2019-09-20 17:17:11 -07:00
James R. Barlow 78e8bf9cbf Use at most 3 Tesseract threads
Based on a user suggestion and
tesseract-ocr/tesseract#2611, I reviewed thread limits and found that
thread limit of 3 is still beneficial, but not 4.

> time env OMP_THREAD_LIMIT=2 tesseract omp4.png stdout >/dev/null
Warning: Invalid resolution 0 dpi. Using 70 instead.
Estimating resolution as 143
116.67user 1.67system 1:26.26elapsed 137%CPU (0avgtext+0avgdata 356752maxresident)k
2213inputs+0outputs (18major+131059minor)pagefaults 0swaps
> time env OMP_THREAD_LIMIT=3 tesseract omp4.png stdout >/dev/null
Warning: Invalid resolution 0 dpi. Using 70 instead.
Estimating resolution as 143
136.89user 1.63system 1:19.56elapsed 174%CPU (0avgtext+0avgdata 356784maxresident)k
821inputs+0outputs (0major+131080minor)pagefaults 0swaps
> time env OMP_THREAD_LIMIT=4 tesseract omp4.png stdout >/dev/null
Warning: Invalid resolution 0 dpi. Using 70 instead.
Estimating resolution as 143
161.31user 1.51system 1:18.80elapsed 206%CPU (0avgtext+0avgdata 356632maxresident)k
8477inputs+0outputs (12major+131074minor)pagefaults 0swaps
> time env OMP_THREAD_LIMIT=8 tesseract omp4.png stdout >/dev/null
Warning: Invalid resolution 0 dpi. Using 70 instead.
Estimating resolution as 143
160.30user 1.62system 1:18.01elapsed 207%CPU (0avgtext+0avgdata 356640maxresident)k
821inputs+0outputs (0major+131078minor)pagefaults 0swaps
2019-09-20 17:12:36 -07:00
James R. Barlow de61530d4d docs: fix intermediate file list for v9 2019-09-20 17:02:35 -07:00
James R. Barlow c149f860b5 Add contributing guide 2019-09-20 17:02:22 -07:00
James R. Barlow 68c852acec Remove test_tesseract_config_invalid from suite
Also causes problems in CI
2019-09-18 13:28:02 -07:00
James R. Barlow a8565bac6e Fix any False in the ocrmypdf.ocr() API being set to True 2019-09-15 01:47:31 -07:00
James R. Barlow 6e8b0c3194 Fix py36 test including 37 2019-09-15 01:47:10 -07:00
James R. Barlow ff860e8362 Fix black settings in pyproject.toml 2019-09-15 01:46:13 -07:00
James R. Barlow cf4b04c5d1 optimize: work around pikepdf 1.6.3 limitation with indexed ICCbased colorspaces 2019-09-11 12:56:27 -07:00
James R. Barlow 078bc2abe9 pdfa: assume 3 RGB channels always 2019-09-11 12:55:38 -07:00
James R. Barlow d7b7ca0574 v9.0.3 notes; Remove test_tesseract_config_notfound from suite 2019-09-05 13:39:43 -07:00
James R. Barlow 17ac9d7a9a Embed ICC profile in .ps (fixing Ghostscript 9.28 compatibility)
Previously we included the
   filename, which required Postscript to run with file access enabled. For
   security, Ghostscript 9.28 enables ``-dSAFER`` and as such, no longer
   permits access to any file by default. This fix is necessary for
   compatibility with Ghostscript 9.28.

We use ASCII85 for a slightly more compact representation.
2019-09-05 13:17:26 -07:00
James R. Barlow a2a197ce4c v9.0.2 release notes 2019-09-04 02:34:21 -07:00
James R. Barlow 944d59e5ad Fix --print-parameters issue when chi_sim is not installed 2019-09-04 01:17:52 -07:00
James R. Barlow 1c3e90a892 optimize: solve monochrome by converting to G4 2019-09-04 00:51:47 -07:00
James R. Barlow c728836956 Adjust test requirements 2019-09-04 00:50:48 -07:00
James R. Barlow 0d80fab339 Remove restriction on pytest < 5 2019-09-03 23:47:55 -07:00
James R. Barlow a650caa599 optimize: don't consider 1bpp images for PNG optimization 2019-09-03 23:47:20 -07:00
James R. Barlow c6caff90a1 optimize: only re-insert pngs after pngquant
Previously we attempted to reinsert all PNGs, but it appears to be
unlikely that Leptonica's API is actually capable of optimizing the PNG
before it inserts it.

In any event qpdf has gained image optimization capabilities as well
which we coudld borrow.
2019-09-03 23:46:25 -07:00
James R. Barlow 671c88d3b5 optimize: exclude images with custom Decode tables 2019-09-03 23:37:23 -07:00
James R. Barlow b2cfaedf91 optimize: Don't reinsert 1bpp images
There seems to be version to version inconsistencies between
Leptonica's photometric interpretation of 1bpp images, in
particular commit a0692307 introduces a change to force transcoding
in this situation.

However, I never entirely got to the bottom of where the problem
is, and in any event 1bpp images are probably better optimized
by JBIG2 than pngquant, so we're going to stop running them through
pngquant.
2019-09-03 23:26:13 -07:00
James R. Barlow 19ba3ae011 Allow test_german to xfail if deu language is not installed 2019-09-03 17:38:54 -07:00
James R. Barlow feff1e38bb Use context managers to ensure Pillow images are closed 2019-09-03 17:19:12 -07:00
James R. Barlow c8d6ea6b10 Fix tests broken by --print-parameters change 2019-09-03 17:17:24 -07:00
James R. Barlow b0d9775343 Attempt to resolve black-inversion issue 2019-08-31 01:25:36 -07:00
James R. Barlow 462bfb84fb install: affirm that we now require Tesseract beta 2019-08-31 01:24:31 -07:00
James R. Barlow 11ef78a891 Fix running without eng.traineddata installed raises exception 2019-08-27 14:54:03 -07:00
James R. Barlow 638eb556ef Reactivate user-words test that was always skipped 2019-08-27 14:52:59 -07:00
James R. Barlow fdefcd8af2 travis: Make 3.7 the build leader/deployer 2019-08-26 13:30:07 -07:00
James R. Barlow 09457edad3 alpine: use jbig2enc@community 2019-08-26 12:49:47 -07:00
James R. Barlow 6460a7eb3e docs: leptonica.com -> .org 2019-08-26 12:07:34 -07:00
James R. Barlow 707ebeb151 docs: installation updates 2019-08-11 18:48:56 -07:00
James R. Barlow e9bc093842 v9.0.1 release notes 2019-08-11 17:14:11 -07:00
James R. Barlow 2eeaca1168 travis: make minimal config even more minimal 2019-08-11 17:13:55 -07:00
James R. Barlow 7755c5c5a7 tests: fix interpretation of None as omitted argument 2019-08-11 16:58:22 -07:00
James R. Barlow 793348a47c tests: mark test as requiring pngquant 2019-08-11 16:58:22 -07:00
James R. Barlow b241f66919 travis: Add a minimal Ubuntu config 2019-08-11 16:58:06 -07:00
James R. Barlow 8ad034a678 docs: update install on FreeBSD to point to ports 2019-08-11 15:50:52 -07:00
James R. Barlow a1a7b973e9 tests: split out stdin/stdout tests 2019-08-09 01:23:49 -07:00
James R. Barlow 7bfcd0a9d5 Use pikepdf 1.6.1 2019-08-09 01:12:13 -07:00
James R. Barlow f276c4ef1e Alpine Docker: jbig2enc moved from testing to community 2019-08-09 01:09:18 -07:00
James R. Barlow 77bbc22c50 Ensure --image-dpi on non-image produces a warning 2019-08-09 01:08:16 -07:00
James R. Barlow a6805ed343 Travis: remove vestiges of pdfminer being optional on osx 2019-07-30 00:42:38 -07:00
James R. Barlow c4afc5c242 Add missing item from v9.0.0 release notes 2019-07-30 00:39:14 -07:00
James R. Barlow df32008667 Ensure test_optimize passes
Linearization sends it over the edge
2019-07-27 16:47:53 -07:00
James R. Barlow db4598f76a Add PDF linearization 2019-07-27 16:15:48 -07:00
James R. Barlow ce13431ecf Remove experimental filters.py 2019-07-27 15:47:09 -07:00
James R. Barlow eb104b405d Avoid circular imports for __version__ 2019-07-27 05:02:19 -07:00
James R. Barlow 5f00e4f9d8 Sort imports 2019-07-27 04:51:52 -07:00
James R. Barlow 1a91cd4652 pikepdf 1.6 2019-07-27 04:36:48 -07:00
James R. Barlow b0f1a55537 completions: --pages 2019-07-27 04:26:38 -07:00
James R. Barlow 0c066d1d53 Expand scope of --pages testing 2019-07-27 04:26:23 -07:00
James R. Barlow e4cfcec5f3 docs: some cleanup 2019-07-27 04:04:33 -07:00
James R. Barlow 85c90404d7 Update release notes 2019-07-27 03:23:56 -07:00
James R. Barlow 4d011c28ea Improve completions 2019-07-27 03:02:04 -07:00
James R. Barlow 5304c631ec Don't warn about --user-words in Tesseract 4.1 or later 2019-07-27 02:14:59 -07:00
James R. Barlow 6fbeb6347d Merge api (without plugins) 2019-07-27 02:04:01 -07:00
James R. Barlow f83de20c37 Remove plugins (for now)
It's holding up too many other useful,
releaseable changes.
2019-07-27 01:41:14 -07:00
James R. Barlow 423324af01 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2019-07-15 12:08:59 -07:00
James R. Barlow 016a2a01d9 docs: Notes on WSL 2019-07-13 02:06:11 -07:00
James R. Barlow 6189910c74 Fix text-image registration when mediabox contains an offset
Cropbox, trimbox not addressed... should look at those. Also rotation.
2019-07-11 02:20:04 -07:00
James R. Barlow 7117dc10de Suppress noisy empty debug messages 2019-07-11 01:23:01 -07:00
jbarlow83andGitHub b601cb0cba Fix funding.yml 2019-07-11 00:36:09 -07:00
jbarlow83andGitHub 0c781faf89 Create funding.yml
[ci skip]
2019-07-11 00:35:39 -07:00
James R. Barlow 12769b96e5 Drop support for omitting pdfminer.six 2019-07-10 13:37:01 -07:00
James R. Barlow a7b4ed9688 Double vertical bars for logo 2019-07-08 22:20:23 -07:00
James R. Barlow ee92ce8717 gitattributes: ensure afdesign is okay 2019-07-08 22:16:47 -07:00
James R. Barlow cbeddab35f rename ocrmypdf.run -> ocrmypdf.ocr 2019-07-07 02:11:44 -07:00
James R. Barlow 98050534d6 logo: once more 2019-07-07 01:19:23 -07:00
James R. Barlow 9850231e91 (Hopefully) fix logo text invisible on Github 2019-07-07 01:12:52 -07:00
James R. Barlow fd810239b5 Add a logo 2019-07-07 01:07:48 -07:00
James R. Barlow 2cff6ad2d1 Fixed blank pages produced when NULs removed from metadata 2019-07-03 02:22:50 -07:00
James R. Barlow 3ee306184b Don't overwrite input PDF when fixing NULs in metadata 2019-07-03 01:57:58 -07:00
James R. Barlow 1cc4c45b7e docs: mention WSL works
[ci skip]
2019-07-03 00:49:06 -07:00
James R. Barlow a86cb8148a Fix jbig2 not checked for special colorspaces 2019-07-02 13:50:29 -07:00
James R. Barlow eeae6f8292 test: Add syntax checks for shell completions 2019-07-02 13:49:17 -07:00
James R. Barlow 4dab299619 Fix parameterization of --verbose 2019-07-02 13:27:07 -07:00
James R. Barlow 340e2bbac6 Drop --mask-barcodes from completions 2019-07-02 13:10:05 -07:00
James R. Barlow 187283192b Fix reporting output file size skipped
Due to change to using finally for clean up
2019-06-30 15:08:10 -07:00
James R. Barlow f855bdd36b Docker: Ubuntu image should be manylinux1 compatible 2019-06-24 01:32:42 -07:00
James R. Barlow 9873d51f58 release notes: add next 2019-06-23 16:54:53 -07:00
James R. Barlow 11a57c7a17 Drop --mask-barcodes feature 2019-06-23 16:54:43 -07:00
James R. Barlow 8aa678859d Use pandoc to rewrite .rst files
Fixes all of the long lines, mainly.
2019-06-22 17:29:26 -07:00
James R. Barlow 1beb7dfd37 helpers: don't expect psutil will be installed
It's not in stdlib
2019-06-22 02:36:06 -07:00
James R. Barlow 9b60d3e285 Improve testing of _validation.py 2019-06-22 02:33:04 -07:00
James R. Barlow 3331a686fa Fix tess_threads clamped to 1 2019-06-22 00:59:33 -07:00
James R. Barlow c32ea3b374 If a page have vector content, promote to full color 2019-06-22 00:59:04 -07:00
James R. Barlow c357d4146e Restructure ocrmypdf.pdfinfo 2019-06-20 03:10:41 -07:00
James R. Barlow f47cb2fade docs: update ocrmypdf.ocrmypdf to .run 2019-06-20 02:45:14 -07:00
James R. Barlow 9c4b1aeb8d docs: plugin; renaming 2019-06-20 02:44:54 -07:00
James R. Barlow 51ed381bfc Rename weave -> graft 2019-06-13 01:16:56 -07:00
James R. Barlow 5ee45411c9 Decide on OMP_THREAD_LIMIT more intelligently 2019-06-13 01:02:07 -07:00
James R. Barlow 16990890d8 Remove "from ocrmypdf import ocrmypdf"
Messes up future imports from ocrmypdf, so don't do it.
2019-06-12 17:52:25 -07:00
James R. Barlow cfb11559d5 logging: capture warnings too 2019-06-12 17:28:02 -07:00
James R. Barlow 8b8de7cc1d Add new --pages feature to limit OCR to only specific pages 2019-06-12 17:27:47 -07:00
James R. Barlow aba293fd80 Change "Temporary working files" output message 2019-06-12 13:56:02 -07:00
James R. Barlow 066a293462 If verbose, print stacktrace on KeyboardInterrupt 2019-06-12 13:55:43 -07:00
James R. Barlow 0bbd6885e2 Make the go/no-go decision pluggable 2019-06-06 23:07:46 -07:00
James R. Barlow 5dd10c961c Docker: prefer streaming 2019-06-05 03:14:36 -07:00
James R. Barlow 81fc95556c Add progress bar for PdfInfo step 2019-06-05 03:08:04 -07:00
James R. Barlow 20ad032977 Fix some error messages that printed directly to sys.stderr instead of logging 2019-06-05 03:07:48 -07:00
James R. Barlow 93f1b73579 Fix --remove-vectors which was broken in API migration
It got dropped during the change. This feature has also been altered so that
the final visual appearance of the file is not affected, only the OCR image.
2019-06-05 02:04:45 -07:00
James R. Barlow fd427a8ec1 plugins: replace path manipulation 2019-06-05 01:46:56 -07:00
James R. Barlow 9444cf357b optimize: add divide by zero check 2019-06-04 02:01:53 -07:00
James R. Barlow 5ab69153ee Fix .coveragerc 2019-06-03 02:26:49 -07:00
James R. Barlow eb5200d26a Change most tests to use ocrmypdf API instead of subprocess
The main benefit of this is code coverage gains can actually follow it.
Also removes most ugly os.environ hacks.
2019-06-03 01:45:27 -07:00
James R. Barlow 98a3fda1f5 Drop support for Tesseract 4 alpha releases without textonly_pdf (mostly)
hocr renderer can still be used
2019-06-03 01:39:41 -07:00
James R. Barlow e73740ae9d test: remove test code that support tess3 or tess4 testing 2019-06-03 01:33:24 -07:00
James R. Barlow fb933edc0f Use newer pytest tmp_path API 2019-06-01 01:55:51 -07:00
James R. Barlow ba41ccae1b conftest: don't modify PYTEST_CURRENT_TEST when manipulating os.environ
It confuses pytest.
2019-06-01 01:41:39 -07:00
James R. Barlow df9e286e9c Make bypassed exception clearer 2019-06-01 01:35:15 -07:00
James R. Barlow b9d6e46572 shutil.rmtree: use builtin error suppression 2019-05-31 15:12:46 -07:00
James R. Barlow 8347c0d662 validation: remove dead code check_input_file 2019-05-31 01:57:08 -07:00
James R. Barlow 45a361d112 Add option to use threads instead of processes
Mainly since they are more convenient for debugging
2019-05-31 01:56:16 -07:00
James R. Barlow 522e1e948b ghostscript: don't use threads= for generate_pdfa
Not supported for pdfwrite
2019-05-31 01:55:29 -07:00
James R. Barlow 8ed4e229f3 ghostscript: avoid log=None construct 2019-05-30 13:57:38 -07:00
James R. Barlow db29cae177 Docker docs: Remove legacy images, revive Ubuntu 2019-05-28 21:36:45 -07:00
James R. Barlow d5b6cbb95e Update Ubuntu dockerfile 2019-05-28 15:36:50 -07:00
James R. Barlow 396c39978a Reorganize .docker folder so we don't have to rebuild as much 2019-05-28 14:18:54 -07:00
James R. Barlow 9d5f23e961 Rename filters to plugins 2019-05-28 02:39:25 -07:00
James R. Barlow 26a6232e1c Ignore DSStore 2019-05-28 02:33:35 -07:00
James R. Barlow 7566d4b768 Introduce plugins/filters 2019-05-27 16:55:04 -07:00
James R. Barlow 5c4c32ab3c Remove multiprocessing tests - no longer valid 2019-05-27 12:07:20 -07:00
James R. Barlow 692f7b3151 Dockerfile: with newer pip
Newer pip seems to install ocrmypdf-*.dist-info and has no problem reporting
installed version unlike -egg-info, so
skip copying.

Also move WORKDIR
2019-05-26 04:31:53 -07:00
James R. Barlow 8d0958d7ea Dockerfile: qpdf-dev needs to be requested explicitly 2019-05-26 04:30:34 -07:00
James R. Barlow e9731b6bac Docker: upgrade pip, temporarily enable community repository for qpdf 2019-05-26 04:00:24 -07:00
James R. Barlow 0628a89041 docs: mention how to use Docker image shell 2019-05-26 00:20:40 -07:00
James R. Barlow c14f62752b Tests: add an API test 2019-05-25 16:24:09 -07:00
James R. Barlow 24855045e1 Provisionally add filters 2019-05-25 16:23:39 -07:00
James R. Barlow ed236e0c27 Begin API documentation 2019-05-24 01:05:32 -07:00
James R. Barlow db6aa22eae Progress bar: unit types 2019-05-23 02:00:47 -07:00
James R. Barlow 805aa776ad Re-disable progress bar when not connected to tty 2019-05-23 02:00:35 -07:00
James R. Barlow d0efdf643c Cleanup working files when done with a particular file, rather than end of process 2019-05-23 01:25:08 -07:00
James R. Barlow 22298b31be Fix distinction between clean and clean_final lost in API refactor 2019-05-23 01:19:58 -07:00
James R. Barlow 5cecb3ecb4 Convert one test to use API 2019-05-22 23:53:48 -07:00
James R. Barlow a139e64c67 api: short-circuit exception handler, as caller should provide their own 2019-05-22 18:30:30 -07:00
James R. Barlow db69b4d11a Improve argparse behavior for its role in making the API work 2019-05-22 15:55:48 -07:00
James R. Barlow 8bcb85720c release notes: clarify 2019-05-22 15:34:23 -07:00
James R. Barlow 09ca1bee97 Add progress bar to optimize and add option to disable it 2019-05-22 15:31:48 -07:00
James R. Barlow 23dd77ce0f api: fix progress_bar_friendly=False 2019-05-22 15:31:03 -07:00
James R. Barlow 32a076c039 Refactor validation and exceptions
CLI now tracks check_options exceptions. API now works more like
an API, without an exception handler,
because the caller should provide one.
2019-05-20 18:01:17 -07:00
James R. Barlow e4baa8c0dd Remove sys.exit() calls so we don't terminate caller application 2019-05-20 15:08:20 -07:00
James R. Barlow 2fdaa76a0d Refactor configure_logging 2019-05-20 14:54:34 -07:00
James R. Barlow 7ee0c52a57 Refactor cli into basic high level api 2019-05-19 22:34:45 -07:00
James R. Barlow ac2fc9c2a0 Explain picklable logger 2019-05-19 15:06:47 -07:00
James R. Barlow 188e08e98b docs: Remove discussion of ruffus 2019-05-17 22:28:28 -07:00
James R. Barlow ef1ef1cdf0 Fix test invalidated by Python 3.6 logging fixes 2019-05-17 15:20:07 -07:00
James R. Barlow 24da92d39e Fix extra blank lines in output messages in Python 3.6 2019-05-17 14:56:33 -07:00
James R. Barlow 61afef549e Remove some now-unused code; etc 2019-05-17 14:25:17 -07:00
James R. Barlow cfd67ab6aa Fixing threading._RLock exception on Python 3.6
Issue was the usual business: objects that cross process boundaries need
to be picklable and Python 3.6 is more strict about this. The logger
object in particular interfered, so now we suppress it and rebuild it
in process.
2019-05-17 14:24:47 -07:00
James R. Barlow 56067b590b Make re_symlink() not require a log object 2019-05-17 01:59:36 -07:00
James R. Barlow 4340ad9f12 Update test cache 2019-05-17 01:45:06 -07:00
James R. Barlow 70def4a0d0 validation: eliminate print() 2019-05-17 01:43:26 -07:00
James R. Barlow 8df1ea2754 Mark some slow tests 2019-05-17 01:42:27 -07:00
James R. Barlow e528adc603 pylint removal 2019-05-17 01:09:06 -07:00
James R. Barlow c1af0fb18d Cleanup ghostscript error output 2019-05-17 00:50:29 -07:00
James R. Barlow 0cb4e854e5 Replace ProcessPoolExecutor with multiprocessing.Pool
There seems to be no reasonable way to handle Ctrl-C with a
ProcessPoolExecutor. Or at least you have to press it several times to
actually kill. Pool does the job.
2019-05-17 00:48:06 -07:00
James R. Barlow 13ab23ba54 Refactor weave_layers, introduce progress bar
Fixes a bug in this branch where --sidecar would fail by trying to iterator
the executor futures twice.
2019-05-16 14:57:31 -07:00
James R. Barlow 19263f00c6 Additional logging fixes; silence extremely verbose pdfminer logging 2019-05-16 13:44:44 -07:00
James R. Barlow 50bd129d7a logging: don't pass log object to validation 2019-05-16 01:58:48 -07:00
James R. Barlow 471cdea232 Move app specific settings a library may not want to __main__ 2019-05-16 01:29:26 -07:00
James R. Barlow 9d750828c7 Make logging format consistent with v8.3.0 2019-05-16 00:48:40 -07:00
James R. Barlow 5e025c3382 Reinstate log level in messages to be closer to old behavior 2019-05-15 15:46:36 -07:00
James R. Barlow 4410503349 More fixes to logging and disabled tests 2019-05-15 03:08:09 -07:00
James R. Barlow 486f73d5d6 Remove custom logger 2019-05-15 02:28:13 -07:00
James R. Barlow b10285d11b Fix warnings 2019-05-14 16:34:42 -07:00
James R. Barlow c904b430b6 Merge master into api branch; all test pass 2019-05-14 16:33:02 -07:00
James R. Barlow 07d4fff3d4 docs: mention FreeBSD works 2019-05-14 02:13:56 -07:00
James R. Barlow 7e388f59af Move completions to better location/Homebrew compat 2019-05-13 00:22:52 -07:00
James R. Barlow 84ec83151d v8.3.0 notes: clarify 2019-05-12 03:36:41 -07:00
James R. Barlow 0a72c12ff0 weave: add new test for link consistency 2019-05-12 03:36:33 -07:00
James R. Barlow 5adbddd1f9 ghostscript: rendering threads has no effect on pdfwrite, so remove it
See
https://www.ghostscript.com/doc/current/Language.htm
2019-05-12 03:36:07 -07:00
James R. Barlow c2fecffdb4 Require pikepdf 1.3.0 2019-05-12 02:16:05 -07:00
James R. Barlow 7dea0a8362 v8.3.0 release notes in progress 2019-05-11 12:50:44 -07:00
James R. Barlow 482cb788ed Don't use MagicMock() as a dummy logger in pytest 2019-05-11 12:44:17 -07:00
James R. Barlow 15a988b999 weave: use emplacement method, scrap TOC repair
The new emplacement method updates page objects in place without
generating new objgen numbers, meaning we no longer need to update the table
of contents to preserve links.
2019-05-11 12:40:25 -07:00
James R. Barlow 83398e54ea weave: fix corruption of certain high page count files
Corruption occurred when replacements was not incremented for multiple
consecutive pages.
2019-05-11 12:22:21 -07:00
James R. Barlow bcdd196699 ghostscript: remove unnecessary post-render resizing step 2019-05-11 12:10:50 -07:00
James R. Barlow 0cd576e701 Rename bash completions file 2019-05-11 10:52:42 -07:00
James R. Barlow 4d5e0eb749 docs: mention completions 2019-05-06 18:07:41 -07:00
Frankandjbarlow83 7ed0f8f50e Add bash completion (#384)
* Add bash completion

file must be copied to completion folder e.g. /usr/share/bash-completion/completions
2019-05-06 15:18:15 -07:00
James R. Barlow 79c84eefa3 Fix main.txt 2019-04-23 02:21:31 -07:00
James R. Barlow 5398003160 Fix test.txt 2019-04-23 00:42:40 -07:00
James R. Barlow 58b2bed99d v8.2.4 notes 2019-04-23 00:07:12 -07:00
James R. Barlow 58c29ffb5c weave: use explicit pdf.close(), drastically reduce open file handles
With the new pikepdf 1.2.0 we no longer need to hold file handles
open because of the "copy to memory" functionality. We retain
the behavior of closing/reopening the output PDF every 100 pages as
a way to limit memory usage.
2019-04-18 15:12:48 -07:00
James R. Barlow f615b6f0e8 pdfinfo: be more specific about detecting XFA we can't render 2019-04-18 15:07:25 -07:00
James R. Barlow e0c8dadcce Explicitly close most pikepdf.Pdf when done with them 2019-04-18 15:02:12 -07:00
James R. Barlow 9a86f53109 Ignore pip-wheel-metadata folder
https://github.com/pypa/pip/issues/6213
2019-04-18 10:42:10 -07:00
James R. Barlow 91cb092aa0 Remove PyCharm debugger hack 2019-04-18 10:15:02 -07:00
James R. Barlow f4b87915df Fix --redo-ocr 2019-04-15 13:11:26 -07:00
James R. Barlow 922a107b7f Remove safety traversal of PDF table of contents
qpdf fixed the danging reference issue (qpdf #240) in 8.3.0, which is
required by pikepdf 1.1.0. We no
longer need the workaround.
2019-04-13 00:24:03 -07:00
mawi 1c44fd4f3b fix: typo 2019-04-08 15:01:04 +02:00
mawi c92ccc6134 fix: tests 2019-04-08 14:57:42 +02:00
mawi 1137534e97 fix: update pytest version
Solves install error: pkg_resources.ContextualVersionConflict: (pytest 4.3.0 (/app/.eggs/pytest-4.3.0-py3.6.egg), Requirement.parse('pytest>=4.4.0'), {'pytest-xdist'})
2019-04-08 11:08:29 +02:00
mawi 39617dd739 fix: remove ruffus 2019-04-08 11:07:32 +02:00
mawi 6590875756 feat: add triage step
remove tqdm demo
2019-04-08 10:26:56 +02:00
mawi 01bbf064e0 feat: add tqdm progress bar
This is just a POC. Will be removed.
2019-04-05 19:52:38 +02:00
mawi fc1c4f12f5 feat: add concurrent.futures pipeline 2019-04-05 18:48:34 +02:00
mawi 2647382cf6 fix: most of the tests (37 failed, 133 passed, 28 skipped) 2019-04-05 14:06:07 +02:00
mawi 783a128bd1 feat: move to sync (none ETL) implementation - remove ruffus 2019-04-04 21:02:38 +02:00
Martin Wind b214aa5b38 feat: move to sync (none ETL) implementation 2019-04-03 19:59:43 +02:00
James R. Barlow 6e49bb3588 v8.2.3 notes 2019-04-03 01:19:12 -07:00
Martin Wind aa512b6181 feat: move to sync (none ETL) implementation (WIP) 2019-04-02 20:03:09 +02:00
Martin Wind a4667b5656 refactor: move ruffus related code to one file 2019-03-28 20:16:10 +01:00
Martin Wind f65a3d3762 fix import in unpaper test 2019-03-26 10:04:26 +01:00
Martin Wind 2fa43ecf09 refactor: split argparse and run_pipline 2019-03-26 08:10:20 +01:00
James R. Barlow 427afc0616 Fix LeptonicaErrorTrap when a sys.stderr.fileno() is not available
The LeptonicaErrorTrap was problematic for Celery and other
libraries that mess with stderr.

Closes #359
2019-03-17 14:22:36 -07:00
James R. Barlow 9c7ee2bf23 Better help text for --verbose 2019-03-17 13:29:25 -07:00
James R. Barlow c5cfaa950b readme: tweaks 2019-03-16 14:09:19 -07:00
James R. Barlow 4e2a98ead4 leptonica: fix junkpixt harder 2019-03-16 14:08:58 -07:00
James R. Barlow 210f134b5b Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2019-03-08 15:38:31 -08:00
James R. Barlow 696c0721a0 docs: fix broken sphinx ref
[ci skip]
2019-03-08 15:38:26 -08:00
James R. Barlow aabab95418 docs: use images folder 2019-03-08 15:38:01 -08:00
James R. Barlow 7d614dd68b docs: explain Automator workflow 2019-03-08 15:37:42 -08:00
jumbliesandjbarlow83 f57dda7939 Update batch.rst (#362)
Added docker instructions for passing "find" filenames into container.  Obviates prior incorrect flag fix.
2019-03-08 12:46:50 -08:00
James R. Barlow 1b4542aa77 Further fixes to external program version testing 2019-03-07 14:27:16 -08:00
James R. Barlow 6c7fca57ec v8.2.1 notes 2019-03-06 22:22:50 -08:00
James R. Barlow 486dc7e22c Fix some test failures missed in prev commit 2019-03-06 13:28:50 -08:00
James R. Barlow dc616bb507 Fix test suite so --clean is not requested when unpaper is not installed 2019-03-05 22:33:13 -08:00
James R. Barlow 902bda43e3 main: fix version testing unnecessarily throwing exception to itself 2019-03-05 22:32:06 -08:00
James R. Barlow f7da63f68b main: fix redundant argument test 2019-03-05 22:29:29 -08:00
James R. Barlow 5da26e4c9c Convert most uses of subprocess.Popen to subprocess.run in test suite 2019-03-05 22:25:22 -08:00
James R. Barlow c19c852705 Fix exception while attempting to print error message for missing program 2019-03-05 16:32:48 -08:00
James R. Barlow a27ee3ee8c optimize: use Decode to invert 1bpp PNGs for now 2019-03-03 17:50:12 -08:00
James R. Barlow c2f316c2c5 v8.2.0 release notes: optimizer 2019-03-03 15:26:01 -08:00
James R. Barlow 974979b0a0 Merge branch 'feature/optimization-fixes' 2019-03-03 15:00:20 -08:00
James R. Barlow 66586bdaab optimize: Disable jpg->png migration
Needs more testing before release
2019-03-03 14:59:59 -08:00
James R. Barlow 01d2ea309f Fix Predictor name and photometric flip 2019-03-03 14:57:15 -08:00
James R. Barlow e918480351 v8.2.0 release notes 2019-03-03 14:15:20 -08:00
James R. Barlow 52fd84fa95 Remove debug message 2019-03-03 13:31:10 -08:00
James R. Barlow 2c56b0935c docs: minor 2019-03-03 03:28:17 -08:00
James R. Barlow 4f69ace868 optimize: fix all JBIG2 images binned on last page
During some past refactor it appears we now end up treating
all JBIG2 images as if they appeared on the last page in the
file. This bug had no visual side ffects but probably led to
suboptimal JBIG2 encoding.
2019-03-03 03:28:17 -08:00
James R. Barlow 497c531112 optimize: update comments 2019-03-03 03:28:17 -08:00
James R. Barlow b27b92fbf3 optimize: on aggressive settings try JPG to PNG transcoding
If the color count of an image is low such as when black and white
documents are scanned in color, PNG with lossy quantization may
produce a superior encoding to JPEG. This is expensive to test however.
2019-03-03 03:28:17 -08:00
James R. Barlow 2e6ba2df8c optimize: fix recoding of PNGs
Previously we opened pngquant-compressed PNGs with transcoding
because the transcode free function in Leptonica didn't seem to
work. This mean Leptonica may have thrown away the hard of
pngquant if didn't understand the encoding.

This change resolves the issue and allows us to open PNG encoded
data and insert it into a PDF without transcoding. Should improve
encoding quality.
2019-03-03 03:28:17 -08:00
James R. Barlow 67a405c6b7 Move install-time external program checks out of setup.py
We did runtime tests for several of them anyway, and it's better to do
at runtime since config may change after installation.
2019-03-03 03:26:56 -08:00
James R. Barlow 58e6663806 Update test cache for french->german change 2019-03-03 03:23:59 -08:00
James R. Barlow 602570fcf9 Update requirements 2019-03-03 02:27:56 -08:00
James R. Barlow 691f8ce254 Docs: reorganize for new docker-alpine image 2019-03-01 23:15:32 -08:00
James R. Barlow 22812e74b9 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2019-02-26 13:01:59 -08:00
Martin Windandjbarlow83 9d824e723d Add Dockerfile based on alpine:3.9 (#354)
* Do not exclude .git from docker build

* Use multi-stage builds to keep the image size down

* Copy project files to get the test suite.

* Add webservice

* Add tesseract language data for German and Chinese Simplified
2019-02-26 13:01:38 -08:00
James R. Barlow 5dad800d85 Add version to build-system declaration 2019-02-26 12:58:44 -08:00
James R. Barlow 56a56a4dcb docs: avoid importing ocrmypdf 2019-02-26 12:57:50 -08:00
James R. Barlow 3f1d9ef99c Fix tests for move to Alpine dockerfile 2019-02-26 12:30:21 -08:00
James R. Barlow 92c8a5885e Declare build system in pyproject.toml 2019-02-26 12:23:33 -08:00
James R. Barlow 7749d14252 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2019-02-24 01:56:47 -08:00
Julien Maandjbarlow83 9b92af5aed README: install other language packs on macOS (#352)
The default homebrew formula installs only the English language pack.
Another brew formula exists to install all other language packs.
This makes it easier than having to do the whole install manually.
2019-02-19 10:13:36 -08:00
James R. Barlow 0bf26b03ae optimize: Modernize pikepdf usage 2019-02-16 14:03:10 -08:00
James R. Barlow e2847ea4c3 v8.1.0 release notes 2019-02-10 02:10:48 -08:00
James R. Barlow 19e35db2b7 Fix issue when weave handoff occurs with no OCR font present
If using --tesseract-timeout 0 and any image processing on a file with
more than 100 pages, the weave handoff will occur. Ensure this
works correctly even if no Glyphless font is present.

Closes #347
2019-02-10 02:05:59 -08:00
James R. Barlow df688742d5 Fix exception on traversing corrupt ToC entries 2019-02-10 00:50:21 -08:00
James R. Barlow 42c2925f9d Activate black precommit 2019-02-08 14:09:08 -08:00
James R. Barlow 933f0b8f9b docs: more unpaper details 2019-02-08 13:05:09 -08:00
James R. Barlow 03ab5a8ee2 If --tesseract-timeout 0, say nothing when we time out
This is our "don't actually OCR" mode. No need to mention it.
2019-02-08 13:04:48 -08:00
James R. Barlow 4f06920224 Be os.nice()-r 2019-02-07 17:24:47 -08:00
James R. Barlow a733b09623 webservice: add an optional config and larger upload limit 2019-02-07 17:24:17 -08:00
James R. Barlow 5483dacf52 Fuzz 2019-02-07 17:09:47 -08:00
James R. Barlow ae7844ad88 --clean-final implies --clean
It's never made sense to leave it out before; might as well introduce it.
2019-02-07 17:08:08 -08:00
James R. Barlow a6e7485da6 docs: --unpaper-args 2019-02-07 17:06:51 -08:00
James R. Barlow 3bcc6d6121 Merge 'feature/unpaper-args' 2019-02-07 17:06:28 -08:00
James R. Barlow 9fe067bbd9 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2019-02-07 16:31:06 -08:00
James R. Barlow f095e91cb4 unpaper-args: add test case and harden feature 2019-02-07 16:21:02 -08:00
Charles Forceyandjbarlow83 66c8d4b47a Adjust the docker pull command for webservice (#346)
Not completely sure this is correct, but I think `docker pull jbarlow83/ocrmypdf-webservice` might be the correct command for getting the web service version.  It installs as expected:

```
docker pull jbarlow83/ocrmypdf-webservice
Using default tag: latest
latest: Pulling from jbarlow83/ocrmypdf-webservice
38e2e6cd5626: Already exists 
705054bc3f5b: Already exists 
c7051e069564: Already exists 
7308e914506c: Already exists 
3977c3cd82d1: Already exists 
ec01b9573956: Already exists 
b508b5192a3c: Already exists 
ace6e737fffb: Already exists 
0a453ee84e11: Already exists 
f8cb8b66151b: Already exists 
f53c3b27b23f: Already exists 
22df51ea5473: Already exists 
e38d932f9f30: Already exists 
b9d3c1d5b53b: Already exists 
68be2088ada3: Already exists 
8b17945ab41b: Pull complete 
59c4aae491bd: Pull complete 
19dce698a07e: Pull complete 
Digest: sha256:0cc9433d490c9a65389403757bf6081a30bcd248055340a8789c23d9cdf9ac8a
Status: Downloaded newer image for jbarlow83/ocrmypdf-webservice:latest
```
2019-01-25 10:41:42 -08:00
James R. Barlow 721489a06c docs: remove reference to --skip-repair since the argument was removed 2019-01-18 05:44:11 -08:00
James R. Barlow 9a4493f211 Add --unpaper-args
Needs test code and stricter validation
2019-01-18 05:33:28 -08:00
James R. Barlow edb4d6c586 docs: Clarify ArchLinux edition is in AUR 2019-01-18 05:29:37 -08:00
James R. Barlow b8cd3acd9e v8.0.1 notes 2019-01-17 00:57:28 -08:00
James R. Barlow 03779e33da docs: Update some install procedures for v8 changes
[ci skip]
2019-01-12 00:33:36 -08:00
James R. Barlow c466483e82 docs: Explain intermediate files 2019-01-11 14:52:05 -08:00
James R. Barlow e3a58219d1 Ensure XObjects with no subtype don't cause an exception
Closes #325
2019-01-08 16:46:08 -08:00
James R. Barlow 72337094ca v8.0.0 release notes 2019-01-05 23:35:47 -08:00
James R. Barlow f472587d22 Bump pikepdf version, point to release notes 2019-01-05 16:48:13 -08:00
James R. Barlow f34b3015b2 Prevent Ghostscript from generating invalid XMP metadata
If DocumentInfo contains NULs Ghostscript will generate XMP with
NULs which is not allowed. Repair DocumentInfo before Ghostscript sees it.
2019-01-04 13:20:41 -08:00
James R. Barlow 089ece2715 use pikepdf 0.10.2 2019-01-03 12:08:43 -08:00
James R. Barlow 6438465e3f Add fish completions 2019-01-02 17:08:30 -08:00
James R. Barlow 7d330afd81 Delinting 2019-01-02 13:34:45 -08:00
James R. Barlow 68fbd9fcc9 pikepdf: version bump 2018-12-31 15:37:31 -08:00
James R. Barlow c771938907 Convert to f-strings where it makes sense 2018-12-31 15:01:19 -08:00
James R. Barlow c2a947acf4 travis: fix 2018-12-31 01:18:30 -08:00
James R. Barlow 8c0009c5c8 Make pdfminer.six optional
Mainly since the current release of pdfminer.six lacks a sdist, blocking
homebrew packaging. Also in case other distros don't accept pdfminer.six.
2018-12-31 01:08:43 -08:00
James R. Barlow cfc5cdf47d pdfa: remove a pile of deprecated code
It's now handled in pikepdf.
2018-12-31 00:05:13 -08:00
James R. Barlow 05152a8af9 Remove always-false Tess v3 tests 2018-12-30 02:01:05 -08:00
James R. Barlow 0880b16491 Sort imports with isort 2018-12-30 01:28:15 -08:00
James R. Barlow 06308a22ce Reformat with black 2018-12-30 01:27:49 -08:00
James R. Barlow 80bd7de580 Generate test cache 2018-12-30 01:02:37 -08:00
James R. Barlow 8b90c45437 Drop support for Tesseract 3 2018-12-30 00:47:12 -08:00
James R. Barlow 72b920eb16 Drop support for Python 3.5 2018-12-30 00:23:26 -08:00
James R. Barlow b4a51907d6 Detect when metadata is dropped during PDF/A conversion 2018-12-30 00:13:25 -08:00
James R. Barlow 1ca1221432 leptonica.py: Fix exception on certain types of barcode failures
Closes #322
2018-12-19 17:23:23 -08:00
James R. Barlow 40b72b0fa8 v7.4.1 release notes 2018-12-19 16:41:09 -08:00
James R. Barlow 0e55b4ad52 Travis: remove Brewfile 2018-12-19 16:40:48 -08:00
James R. Barlow 7b4f5a8fc4 docs: try to fix readthedocs
[ci skip]
2018-12-19 15:30:07 -08:00
James R. Barlow 9261a38493 Readme: more media 2018-12-19 15:27:54 -08:00
James R. Barlow cc8ff318ed New issue template 2018-12-19 15:27:44 -08:00
James R. Barlow ad15e845f9 docs: Ghostscript PDF/A XMP metadata loss; ocrmypdf-webservice
[ci skip]
2018-12-17 23:20:49 -08:00
James R. Barlow ab632f57cd v7.4.0 release notes 2018-12-15 15:27:23 -08:00
James R. Barlow 13d20bd993 pdfinfo: tolerate PDFs that overflow and underflow the graphics stack 2018-12-15 15:10:29 -08:00
James R. Barlow b973208137 Require pikepdf 0.9.1 2018-12-15 14:23:10 -08:00
James R. Barlow 942abf8074 Fix reqs/main.txt for pikepdf 0.9.0 2018-12-14 23:29:26 -08:00
James R. Barlow ed9bb985e2 Fix pikepdf 0.9.0 2018-12-14 23:21:13 -08:00
James R. Barlow 5a7a8e573b Require pikepdf 0.9.0 2018-12-14 23:06:57 -08:00
James R. Barlow ce878db913 Rename to polyglot.dockerfile 2018-12-14 23:06:29 -08:00
James R. Barlow a3d58683b2 Update webservice.py with separate license 2018-12-14 23:05:54 -08:00
James R. Barlow 039e8ca7e7 Merge branches 'feature/newer-pike' and 'feature/webapp' 2018-12-14 18:08:31 -08:00
James R. Barlow 0ebbd4e21b Don't open encrypted files, even if password is empty 2018-12-13 22:48:00 -08:00
James R. Barlow 2cb75f6076 Refactor pipeline to make PDF/A conversion a separate step 2018-12-13 20:48:48 -08:00
James R. Barlow 857d871364 Fix regression on Ghostscript path 2018-12-13 20:36:41 -08:00
James R. Barlow 632dab2cc0 Replace Ghostscript DOCINFO and fix 9.25 metadata date regression
We no longer use Ghostscript to manage PDF metadata, instead
omitting the DOCINFO segment from the pdfmark file we generate.

Instead all of the relevant metadata code has been migrated to pikepdf,
and we use that API. This should be more consistent and fixes the
Ghostscript version-depedent quirks.

Also removes our python-xmp-toolkit dependency, except for
testing.
2018-12-13 18:13:30 -08:00
James R. Barlow 7647918f2d setup: suppress XMLParser() warning - defusedxml related 2018-12-12 22:13:32 -08:00
James R. Barlow 75c5d8055c pdfinfo: fix FutureWarning 2018-12-12 22:12:14 -08:00
James R. Barlow a938bbea55 Remove more libxmp dependencies 2018-12-12 22:02:35 -08:00
James R. Barlow 414407fbd6 Deprecate encode/decode_pdf_date and remap to pikepdf version 2018-12-12 22:01:21 -08:00
James R. Barlow 076fc717df pdfa: replace PDF/A checking with pikepdf implementation 2018-12-12 21:41:16 -08:00
James R. Barlow 2a04b2d82b Rename webapp to webservice 2018-12-12 21:29:05 -08:00
James R. Barlow 065db414c0 webapp docker: Build from polyglot 2018-12-12 21:24:04 -08:00
James R. Barlow 19a054a78b Add webapp stuff 2018-12-10 20:03:52 -08:00
James R. Barlow 9df24a81b7 Fix comment in layout.py 2018-11-28 15:16:34 -08:00
James R. Barlow 40c0acd3f2 Support using --force-ocr and --threshold or --mask-barcodes together 2018-11-28 15:16:24 -08:00
James R. Barlow 20db7f0a8f leptonica: delete file junkpixt.png if created 2018-11-28 13:47:55 -08:00
James R. Barlow e54f6ee37f v7.3.1 release notes 2018-11-16 02:13:41 -08:00
James R. Barlow 2da556bf79 Fix unsupported operand Decimal, float 2018-11-16 02:13:25 -08:00
James R. Barlow b183ad8167 Fix barcodes error handling 2018-11-16 02:08:16 -08:00
James R. Barlow 9e6b54c7ed Add test case for Type3 fonts with no Unicode mapping 2018-11-15 21:54:26 -08:00
James R. Barlow d3b334c10f Test case: true type font without Unicode mapping 2018-11-15 16:22:53 -08:00
James R. Barlow 622f2c4bab More argument checking 2018-11-15 15:59:38 -08:00
James R. Barlow 07b638a394 pdfminer: detect TrueType fonts with no valid encoding information 2018-11-15 13:44:11 -08:00
James R. Barlow 9bee2405d8 Leptonica: make threshold functions more flexible 2018-11-15 13:43:34 -08:00
James R. Barlow 8f040491bf Fix erasure of undetectable barcodes 2018-11-15 12:03:51 -08:00
James R. Barlow 8a18988706 Fix 'del draw' exception 2018-11-15 12:02:53 -08:00
James R. Barlow 47a954514b Fix name2unicode ignoring certain markers 2018-11-15 12:02:30 -08:00
James R. Barlow e3b65d4288 Fix detailed page analysis enabled at wrong time 2018-11-15 12:02:08 -08:00
James R. Barlow 4704f7ed1d Add ReadTheDocs yml so we can build with Py3.6 2018-11-12 13:43:17 -08:00
James R. Barlow 3a2745445a Fix docs build 2018-11-12 13:26:04 -08:00
James R. Barlow 12e15bab15 v7.3.0 release notes 2018-11-11 02:05:52 -08:00
James R. Barlow 9593aa4fb9 Merge v7.3.0 development 2018-11-11 01:38:42 -08:00
James R. Barlow 817d520e63 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2018-11-11 01:34:00 -08:00
James R. Barlow 700abbb8a5 Documentation for OCR quality features 2018-11-10 15:48:41 -08:00
James R. Barlow 701ef1df3f Add threshold function to work around Tesseract's poor thresholding of bright backgrounds 2018-11-10 15:34:37 -08:00
James R. Barlow 0f5c484b62 Travis: only need to specify chardet because we use pip install --no-deps 2018-11-10 13:57:04 -08:00
James R. Barlow cc7f2a3f02 Fix Python 3.5 pathlib regressions 2018-11-10 02:11:23 -08:00
James R. Barlow 755b5d87e3 Add missing chardet, implied by pdfminer.six? 2018-11-10 01:50:51 -08:00
James R. Barlow e55a4115e1 Travis: pytest 3.10.0 internal error? 2018-11-10 01:44:05 -08:00
James R. Barlow 16a6fd2ea9 Update docs for --redo-ocr and --mask-barcodes 2018-11-10 01:34:33 -08:00
James R. Barlow e3fce112ed main.txt: wrong pdfminer 2018-11-10 01:32:27 -08:00
James R. Barlow eacd26a68b Mention v6.2.5 release 2018-11-10 01:10:45 -08:00
James R. Barlow 0e88b3c38a Update v7.3.0 release notes 2018-11-10 01:09:19 -08:00
James R. Barlow a2170ef8d6 test: test version check code 2018-11-10 00:56:22 -08:00
James R. Barlow eed0424390 Update requirements 2018-11-10 00:56:04 -08:00
James R. Barlow 5ed05e08b1 Fix "no languages" test and misuse of os.environ 2018-11-09 01:57:11 -08:00
James R. Barlow 58b26f6715 Leptonica: learn to despeckle 1bpp images 2018-11-07 01:49:13 -08:00
James R. Barlow 806daf4284 leptonica: reduce boilerplate for PIX (2/2) 2018-11-06 20:33:40 -08:00
James R. Barlow c64bc9329e leptonica: reduce boilerplate for wrapper classes (except PIX) 2018-11-06 20:12:09 -08:00
James R. Barlow dd01745519 Leptonica: add masked threshold fn 2018-11-06 19:31:06 -08:00
James R. Barlow 501ce726e7 Fix two failing tests 2018-11-06 11:16:08 -08:00
James R. Barlow 03076e89ce Leptonica: reduce verbosity, more error trapping, more garbage collection 2018-11-06 11:10:59 -08:00
James R. Barlow 02f37293ee Integrate barcode masking 2018-11-05 13:01:13 -08:00
James R. Barlow 590942ad14 Leptonica: Add barcode API 2018-11-05 01:48:38 -08:00
James R. Barlow 2ac028c759 test: Add a basic redo OCR test 2018-11-04 15:54:41 -08:00
James R. Barlow 2125b5bfab Remove text detection from our parser interpret_contents
It's redundant now
2018-11-04 15:47:55 -08:00
James R. Barlow b96532caa4 Only do detailed page analysis when needed by --redo-ocr 2018-11-04 15:40:49 -08:00
James R. Barlow 995fc58466 Move Ghostscript text analysis into its own module 2018-11-04 14:55:48 -08:00
James R. Barlow c023cae299 Make pdfminer Type3 patch conditional on PScript5.dll
It appears that PDFs created by this software have a bug in their BBox
which will cause us to misjudge the space occupied by the font.

Other programs probably work around this by ignoring BBox and reading
each character procedure.
2018-11-04 01:53:53 -07:00
James R. Barlow 237eaf9130 Exception message not printed in some cases
Closes #310
2018-11-03 17:10:24 -07:00
James R. Barlow 8b9ab25125 coverage: test compile leptonica 2018-11-02 01:55:25 -07:00
James R. Barlow 77e87abe8f coverage: ensure get_orientation is checked 2018-11-02 01:32:20 -07:00
James R. Barlow 3be02e1e8d coverage: improve leptonic; don't create objects with null pointers 2018-11-02 01:10:10 -07:00
James R. Barlow 64c9ede979 leptonica: barcodes, BOXA 2018-11-02 00:42:01 -07:00
James R. Barlow 5b8d197812 coverage: make it more likely timeout is tested 2018-11-02 00:41:15 -07:00
James R. Barlow 2cba62dc4f coverage: ensure rotation is actually tested 2018-11-02 00:40:56 -07:00
James R. Barlow 288e28328f coverage: add qpdf 2018-11-02 00:37:33 -07:00
James R. Barlow b8214b3c49 coverage: exclude unicodefun.py 2018-11-02 00:33:08 -07:00
James R. Barlow 8681693994 Set up code coverage (it works with multiprocessing now!) 2018-11-02 00:31:50 -07:00
James R. Barlow 1364c63b7c Fix failure to pickle file with AcroForm 2018-11-01 20:07:53 -07:00
James R. Barlow 4ba9e8fe25 Add AcroForm detection 2018-10-30 22:28:44 -07:00
James R. Barlow a195713bb4 Throw exception on corrupt text 2018-10-30 16:35:09 -07:00
James R. Barlow 600d31a907 Require pikepdf 0.3.7 2018-10-30 16:22:05 -07:00
James R. Barlow be31cec332 Add corrupt text warning (when using --redo-ocr) 2018-10-30 16:19:58 -07:00
James R. Barlow 22a7cd3421 Add argument checks for --redo-ocr 2018-10-30 16:19:13 -07:00
James R. Barlow 8b61d2d521 pdfminer: If font descent claims to be positive, treat it as negative 2018-10-30 14:40:53 -07:00
James R. Barlow 559e5269d2 Ensure inline image is parsed correctly
Requires pikepdf > 0.3.6
2018-10-29 23:30:53 -07:00
James R. Barlow ebf6acb318 pdfminer patch: Type3 font height calculation is incorrect
Not sure where it goes wrong or why it needs special treatment, but
this does address it.
2018-10-29 22:27:25 -07:00
James R. Barlow 7acd75f013 pipeline: fix bbox coordinates 2018-10-29 22:26:37 -07:00
James R. Barlow 93623b2226 Refactor TextboxInfo 2018-10-29 14:46:40 -07:00
James R. Barlow d71fd089cb layout: allow names beginning with /i0123 for now
Showed up in GGastro2.pdf. Need to check if this pattern has valid
Unicode mappings but allow for now.
2018-10-29 14:45:59 -07:00
James R. Barlow 05aa43c856 Require pdfminer 2018-10-29 12:45:15 -07:00
James R. Barlow de80fb6bc8 Fix some failing tests after --redo-ocr changes 2018-10-29 11:49:38 -07:00
James R. Barlow 8e396f4be2 Document --redo-ocr more accurately 2018-10-29 02:03:58 -07:00
James R. Barlow efec6da377 Fix error on serializing bad character markers
(Since they held a reference to their font, which in turn, had an
open file handle.)
2018-10-29 02:02:00 -07:00
James R. Barlow 00ef53195e Fix corrupt Unicode mapping detection's false positives 2018-10-29 01:30:19 -07:00
James R. Barlow f564aaf485 Remove only_ocr_text 2018-10-28 22:41:18 -07:00
James R. Barlow 5ac2d31d0d Redo OCR can now handle visible and invisible text, so adjust accordingly
Still can't filter out corrupt text
2018-10-28 14:06:25 -07:00
James R. Barlow fda890ab47 pdfinfo: further layout improvements
Rather than grouping visible/invisible in a custom analysis step,
use pdfminer's analysis and iterate.
Make iteration predicate and return more generic.
2018-10-28 14:05:50 -07:00
Stefan Weilandjbarlow83 a873278c2a Fix some recommendations from LGTM (#309)
* Fix unreachable code

This fixes an issue reported by LGTM.

Signed-off-by: Stefan Weil <sw@weilnetz.de>

* Remove unused imports

This fixes several recommendations from LGTM.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
2018-10-28 13:59:58 -07:00
James R. Barlow e6d64be890 pdfinfo: formatting 2018-10-27 23:22:44 -07:00
James R. Barlow 0e4d978d20 pdfinfo: all -> not any 2018-10-27 23:22:28 -07:00
James R. Barlow b12c2cfedf Fix handling of Type3 fonts with no ToUnicode mapping 2018-10-27 01:24:48 -07:00
James R. Barlow 58cc70725e Reorganize around getting bboxes for visible/invisible text 2018-10-26 01:07:02 -07:00
James R. Barlow 339afb02aa --redo-ocr now works in the presence of printable text 2018-10-25 16:53:47 -07:00
James R. Barlow 7ba0ff5c36 Fix strip invisible text bug: missing BT operator 2018-10-25 16:52:23 -07:00
James R. Barlow ff41fbf673 Add pdfminer based layout analysis 2018-10-25 12:42:35 -07:00
James R. Barlow 2435cd23ce Move pdfinfo into a package 2018-10-25 00:37:38 -07:00
James R. Barlow a063cff720 Rename/expose strip_invisible_text 2018-10-24 21:53:24 -07:00
James R. Barlow 0d396e1ac0 option check: Remove always-True condition
Both renderers are now lossless reconstruction-capable. (Have
been since 7.0)
2018-10-22 22:13:59 -07:00
James R. Barlow f5807a2053 Require pikepdf 0.3.5 2018-10-21 21:37:15 -07:00
James R. Barlow eb4938a36f Fix KeyError 'has_vector' 2018-10-20 01:20:22 -07:00
James R. Barlow c5ad530bbf pdfinfo: reminder about 'INLINE IMAGE' sentinel 2018-10-20 01:17:08 -07:00
James R. Barlow d11c428407 Redo OCR: disallow in cases that will damage the output PDF 2018-10-20 01:14:33 -07:00
James R. Barlow 6182b1f53e Merge branch 'feature/remove-vectors' into feature/redo-ocr 2018-10-20 01:13:24 -07:00
James R. Barlow 00fc1a12e2 optimize: should remove unreference resources too 2018-10-19 00:03:56 -07:00
James R. Barlow 16af753206 Add functional "redo OCR" feature
Needs argument validation and some other changes. Needs testing
with mixed-content PDFs.

Only really works for pure invisible text at the moment.
2018-10-19 00:02:19 -07:00
James R. Barlow fa48205bb8 Add feature to remove vector graphics objects 2018-10-18 21:46:08 -07:00
James R. Barlow f7dbf94071 pipeline: if vector graphic objects exist, ensure the DPI is reasonable 2018-10-18 01:23:31 -07:00
James R. Barlow b18e66e2ca pdfinfo: learn to detect vector graphic objects 2018-10-18 01:21:51 -07:00
James R. Barlow 7a5504dfa5 pdfinfo: fix terminology (operands, command) -> (operands, operator) 2018-10-18 01:18:30 -07:00
James R. Barlow d1cad7bc68 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2018-10-16 01:28:17 -07:00
Elliott Sales de Andradeandjbarlow83 c58d5c097c Add Fedora install instructions. (#304)
* Add Fedora install instructions.

* Fix path to fedora_rawhide badget
2018-10-14 13:28:50 -07:00
James R. Barlow 46157ca94e docs: some redundancies 2018-10-12 21:29:27 -07:00
jbarlow83andGitHub dd99511bcc Fix broken badges in README 2018-10-12 21:16:08 -07:00
M.Yasoob Ullah Khalid ☺andjbarlow83 5bc2efd3c7 Removed extra word from docs (#303) 2018-10-12 21:02:16 -07:00
James R. Barlow 1b18dbecf5 Fix filename test.txt 2018-10-11 16:03:25 -07:00
James R. Barlow 9f82c0eb6e v7.2.1 release notes 2018-10-11 15:55:01 -07:00
James R. Barlow 68bac1b177 Fix compatibility with pikepdf 0.3.5 API change 2018-10-11 15:51:34 -07:00
James R. Barlow 1495b78330 Remove cruft to support leptonica < 1.72 in test suite 2018-10-11 01:37:32 -07:00
James R. Barlow 6f777d2848 Include Debian copyright file 2018-10-10 23:55:48 -07:00
James R. Barlow 5650eba848 Cleanup MANIFEST.in, reorg requirements/*.txt, fix non-Unicode readme 2018-10-10 23:53:08 -07:00
James R. Barlow 5bc5dc93f3 v7.2.0 release notes update 2018-10-05 01:27:00 -07:00
James R. Barlow c1e18bb825 optimize: Exclude soft masks (SMasks) from optimization
Soft masks are only allowed to be of colorspace DeviceGray so we
shouldn't use pngquant on them. For now, avoid this exceptional
case by excluded soft masks from optimization.
2018-10-05 01:23:26 -07:00
James R. Barlow 58282ea0fb optimize: more refactoring
Now properly generalized/specialized where it should be
2018-10-04 13:44:51 -07:00
James R. Barlow 891da7834c optimize: refactor image extraction 2018-10-04 12:34:22 -07:00
James R. Barlow 5c229d48d5 optimize: Reorganize so JBIG2 can be performed on images reduced to 1bpp
Closes #297
2018-10-04 11:53:11 -07:00
James R. Barlow 53f660cf35 Travis: use newer macos image 2018-10-04 08:59:40 -07:00
James R. Barlow 7b66ca68f2 ...and document lossy JBIG2 2018-10-04 01:31:53 -07:00
James R. Barlow ba71c3ffbd requirements: request pikepdf 0.3.4 2018-10-04 01:22:03 -07:00
James R. Barlow 6707ad427a v7.2.0 release notes 2018-10-04 01:21:17 -07:00
James R. Barlow 5b84549716 Change JBIG2 lossy mode to require --jbig2-lossy 2018-10-04 01:20:49 -07:00
James R. Barlow c74f2ee6e8 Refactor the detailed error messages 2018-10-04 00:10:59 -07:00
James R. Barlow b32dd9f9d3 Fix lossless JBIG2 when there are multiple JBIG2 images on a single page 2018-10-03 17:40:26 -07:00
James R. Barlow fb8b161f6c Fix suppression of tesseract config error messages 2018-10-03 17:39:50 -07:00
James R. Barlow baddd6d233 Remove libtiff from Brewfile
For some reason, brew complains about it now.
2018-10-03 16:17:59 -07:00
James R. Barlow 6f554c6ae8 tesseract: account for behavior changes when params are missing
Tesseract 4.0-rc1 now accepts invalid parameters in config and
won't return an error anymore. We prefer to raise an error if this
occurs.

See: https://github.com/tesseract-ocr/tesseract/commit/741ea00d7059d8ff7c55797ffc525a461d7f3ced
2018-10-03 15:11:34 -07:00
James R. Barlow a71e4488b3 test: fix pytest warning about direct use of a fixture 2018-10-03 15:04:46 -07:00
James R. Barlow 72156b5653 Degrade more gracefully when --optimize is set but JBIG2 is not present 2018-10-03 14:24:20 -07:00
James R. Barlow 9fa471e053 Test: send stderr to stderr, why don't we? 2018-10-03 14:23:34 -07:00
James R. Barlow 31ef2fe907 test: this error message changed case in newer Tesseract 2018-10-03 13:58:20 -07:00
James R. Barlow 9a8ec4b210 optimize: only enable lossy JBIG2 for -O3 2018-10-03 00:38:58 -07:00
James R. Barlow 75aad4cc79 optimize: Refactor convert_to_jbig2 2018-10-02 23:42:12 -07:00
James R. Barlow 4b27feca98 optimize: Disable JBIG2 lossy mode, use lossless instead 2018-10-01 12:28:54 -07:00
James R. Barlow 45522cd15f weave: clarify comment about garbage data in ToC 2018-09-27 13:48:35 -07:00
James R. Barlow 677d9a4e76 Remove some unhelpful lambdas 2018-09-27 13:48:12 -07:00
James R. Barlow efa7ea4fde Fix log.error where log is None 2018-09-19 23:01:27 -07:00
James R. Barlow 137a6e45f5 ghostscript: fix missing fspath for py3.5 2018-09-19 22:57:20 -07:00
James R. Barlow 29116e1dec Change to README.md 2018-09-19 21:01:24 -07:00
James R. Barlow 87193335b9 v7.1.0 notes 2018-09-19 20:57:18 -07:00
James R. Barlow cfd4f8a850 Improve error handling for improvements to Ghostscript text extraction 2018-09-19 20:29:18 -07:00
James R. Barlow eaa324939f Upgrade to pikepdf 0.3.3
Closes #231
2018-09-19 15:30:54 -07:00
James R. Barlow ef70e538f7 Improve error message on handling KeyboardInterrupt
Closes #301
2018-09-19 01:40:26 -07:00
James R. Barlow b7b912e56a Fix test suite and blank pages 2018-09-17 01:12:58 -07:00
James R. Barlow 4615cf2f1e First cut at improving text extraction speed 2018-09-16 23:34:18 -07:00
James R. Barlow eaf772f80a Merge v6.2.4 release notes 2018-09-16 15:45:38 -07:00
James R. Barlow 96ba75eabd Ghostscript: fix issues in strict ASCII implementation 2018-09-16 15:41:54 -07:00
James R. Barlow fdfe52c1ad main: add debug option to force threads 2018-09-15 00:01:45 -07:00
James R. Barlow 932b2e2a29 main: print Ghostscript version too 2018-09-14 23:58:06 -07:00
James R. Barlow 57e489c957 main: Cleanup; support overriding sys.args in run_pipeline 2018-09-14 23:57:35 -07:00
James R. Barlow 17a3fa671c ghostscript: API docs update 2018-09-14 23:51:52 -07:00
James R. Barlow 2659afb4f6 Cleanup gitignore 2018-09-14 21:02:22 -07:00
James R. Barlow 7392115507 Blacklist Ghostscript 9.24 due to regressions
As per issue #291. Forced push to remove a copyrighted test file that was
accidentally included.
2018-09-14 20:41:13 -07:00
James R. Barlow c54d0c7eaa v7.0.5 release notes 2018-09-13 23:29:54 -07:00
James R. Barlow b95eefc65f Fix pikepdf version for Travis 2018-09-13 22:08:19 -07:00
James R. Barlow 686207ab7f Check for and reject Adobe LiveCycle Designer PDFs
These are the ones that display a "Please wait..." message.

Closes #296
2018-09-13 21:50:51 -07:00
James R. Barlow 517b385fe5 Work around loss of Unicode DOCINFO in Ghostscript 9.24+
Ghostscript no longer supports UTF-16-BE-hex strings as a way of
supplying Unicode data in pdfmark so we have lost this functionality too:
http://git.ghostscript.com/?p=ghostpdl.git;a=commit;h=e997c6836d243ab37fe3a5f0d57974af95eb5eac

For users this means setting --title, --author, etc. will not work if gs
9.24 is installed, but if the file has existing metadata it might work.

For now we enforce police-state-strict ASCII, until there's time to
implement proper metadata editing. Relevant tests set to xfail.
2018-09-13 21:33:39 -07:00
James R. Barlow 795019b0c1 Work around invalid TOC entries
Kodak Capture Desktop and probably other software creates a
/Outlines entry with /First being set to an invalid indirect reference to
an object that hasn't been created. This is legal in the PDF spec but
problematic for qpdf. The objgen will be (max valid object ID + 1, 0).
Because we create new objects in _weave, some TOC entries will end
up assigned to new objects we create. Typically /ProcSet.

We solve the issue by refactoring page traversal and then doing it
twice, once to resolve all references (eliminating the null
reference problem) and a second pass to make our changes.
2018-09-11 14:44:16 -07:00
James R. Barlow 3127a73822 Ghostscript: no need to specify ProcessColorModel when ColorConversionStrategy 2018-09-11 11:56:05 -07:00
James R. Barlow 069ee6c91f ghostscript: fix for 9.24 having jpeg passthrough available 2018-09-10 23:09:51 -07:00
James R. Barlow 3aac3a98ca tests: Migrate metadata tests to pikepdf
For some reason PyPDF2 has begun to trigger internal errors in
pytest on macOS alone. Not sure why, but nothing is wrong that I can
see. Seemed like an opportune time to switch to pikepdf; found some
new issues in the process anyway.
2018-09-10 16:06:01 -07:00
James R. Barlow 268859a304 Merge branch 'master' of github.com:jbarlow83/OCRmyPDF: docs 2018-09-10 11:52:04 -07:00
James R. Barlow a96710aa7b leptonica: update comments 2018-09-10 11:47:38 -07:00
James R. Barlow edcc58826a pdfinfo: remove some dead code 2018-09-10 11:47:00 -07:00
James R. Barlow 7077c8220a Fix rst formatting in release notes 2018-09-10 11:46:17 -07:00
Mateus Seenem Tavaresandjbarlow83 f7cbf68edd Updating Arch Linux instalation (#288)
* Updating Arch Linux instalation

And adding a workaround to a wrong dependencies definition on https://aur.archlinux.org/packages/python-pikepdf/

* Remove comment about temporary workaround
2018-08-31 12:30:18 -07:00
James R. Barlow 68a58ee8a5 docs: fix hyperlinking of jbig2 page (again) and cleanup release notes 2018-08-27 01:25:30 -07:00
James R. Barlow 3109ec5091 v7.0.4 notes 2018-08-24 12:41:53 -07:00
James R. Barlow e0599fe8d7 Require pikepdf 0.3.2 2018-08-24 12:41:43 -07:00
James R. Barlow a749240589 docs: mention pikepdf install more clearly 2018-08-22 03:19:46 -07:00
James R. Barlow 6decdaa062 Try setuptools_scm_git_archive again 2018-08-20 15:45:51 -07:00
James R. Barlow 4d5c9b8cdf Fix error in optimize.py on PNGs at -O2
Error was
TypeError: unsupported operand type(s) for -: 'tuple' and 'int'
2018-08-20 15:45:34 -07:00
James R. Barlow 1e23ea5364 Remove pikepdf < 0.3 compatibility shims since > 0.3.1 is now required 2018-08-10 17:01:03 -07:00
James R. Barlow cf9a8a91b5 Require pikepdf 0.3.1 2018-08-10 16:59:08 -07:00
James R. Barlow 05d3a65e94 docs: Fix links to JBIG2 encoder page
[ci skip]
2018-08-09 21:14:50 -07:00
James R. Barlow c043552f8b Fix travis.yml syntax 2018-08-03 14:02:20 -07:00
James R. Barlow 487ee2b6c9 Notes for v7.0.2 2018-08-03 13:37:18 -07:00
James R. Barlow 8013fd50da Draw preview image at full resolution
As reported in #281 and confirmed by the test file in #279, downsampling
the preview adversely affects quality of image rotation especially for
small font sizes and marginal scans.

Full size gets rotation accuracy. This makes rotation a little inefficient
since it rasterizes twice - to be addressed later.
2018-08-03 13:32:10 -07:00
James R. Barlow 4ec9ec12e3 docs: Describe PDF optimization 2018-08-03 13:10:18 -07:00
James R. Barlow ed96594727 Regroup installation page content around platforms
Also separate out JBIG2 encoder instructions so that distributions that
delete installation.rst won't omit this information.
2018-08-03 12:47:25 -07:00
James R. Barlow 91b7193249 Travis: use xenial for Python 3.7 2018-08-03 11:52:23 -07:00
James R. Barlow 008f65c6a9 Update pinned requirements 2018-08-03 01:15:21 -07:00
James R. Barlow 7aa4e60af2 Explain pytest --runslow 2018-08-03 00:57:59 -07:00
James R. Barlow 6ef26516d1 pipeline: revise logic of rotations to fix pages with nonzero /Rotate 2018-08-03 00:53:29 -07:00
James R. Barlow bb50501369 ghostscript: never use autorotatepages
It looks like from the gs manual that this parameter is only respected for
vector devices so it never had any effect, but to stay on the safe side,
which it off.
2018-08-03 00:44:34 -07:00
James R. Barlow 55eb481f30 Add intensive (optional) rotation test 2018-08-03 00:42:59 -07:00
James R. Barlow 4509ca6e75 pipeline: remove unused function 2018-08-03 00:41:08 -07:00
James R. Barlow c61fe9af61 Fix release notes typos 2018-08-03 00:40:44 -07:00
James R. Barlow 4181a712d1 Update setup.py for version changes 2018-08-01 15:17:49 -07:00
James R. Barlow c171cb7286 Merge img2pdf 0.3.0 fix from v6.2.3 2018-08-01 15:17:33 -07:00
James R. Barlow 1d09061130 Revert previous commit amd reject input images with alpha channel
Decided on this for simplicity of old release branch.

Modifies baiona.png by stripping
alpha, adds baiona_alpha which
includes the alpha.
2018-07-31 23:45:28 -07:00
James R. Barlow a2203b2447 Discard alpha channel when triaging images 2018-07-25 22:23:41 -04:00
James R. Barlow 755f6e0979 optimize: Use new pikepdf Object.write API 2018-07-13 23:52:36 -07:00
James R. Barlow 95b2830d17 Support removal of Null from pikepdf 2018-07-13 23:50:35 -07:00
James R. Barlow 2669d8332e Add more guidance about using Docker 2018-07-13 12:39:47 -07:00
James R. Barlow 11fbd32e6e Merge v6.2.2 (mainly to get release notes) 2018-07-13 12:38:15 -07:00
James R. Barlow 6a302fdb88 Travis: nevermind xenial, then
Gave a weird build error
2018-07-12 03:27:40 -07:00
James R. Barlow 1d9cc239ee Travis: Fix v6 build failures 2018-07-12 03:22:03 -07:00
James R. Barlow d240fc1ea6 Update release notes for v6.2.2 2018-07-12 03:07:47 -07:00
James R. Barlow e7d21dd826 Skip locale check on Python 3.7 2018-07-12 03:03:34 -07:00
James R. Barlow e774b4650b ocrmypdf.exec: trap FileNotFoundError too 2018-07-12 03:01:01 -07:00
James R. Barlow 8f8e6dcdd4 Fix problem iterating ruffus exceptions and rotate-pages-threshold parameter validation 2018-07-12 02:59:38 -07:00
James R. Barlow 5252b88f0f Travis: modernize with v7.0.0 updates
Travis: add 3.7 testing
Travis: remove deploy to testpypi since it's broken
Travis: cherry-pick change to declarative APT
From efb9572
Removed pngquant
Travis: Remove linux_before_install.sh
Eliminate Homebrew autobrewing
2018-07-12 02:47:29 -07:00
James R. Barlow ea69883386 Tests: Speed up a slow test (cherry-picked from v7) 2018-07-12 02:47:15 -07:00
James R. Barlow eb343b1e37 Tests: Add ability to disable use of cache (cherrypicked from v7) 2018-07-12 02:46:53 -07:00
James R. Barlow 9f02de55be main: do better parameter validation 2018-07-12 02:46:52 -07:00
James R. Barlow 7394a4cf49 Cherrypick warning about --user-words not having any effect
Might be available in full release of Tess4
2018-07-12 02:46:34 -07:00
James R. Barlow ed9fb110b1 Fix a comment about Tesseract behavior in certain versions 2018-07-12 02:46:34 -07:00
James R. Barlow 4650074428 Cherrypick Python 3.7 documentation updates from v7.0.0
From b0eacd6
2018-07-12 02:45:51 -07:00
James R. Barlow 70aa644c10 Backport Python 3.7 fix for ruffus 2.7.0 from ocrmypdf v7.0.0 2018-07-12 02:45:51 -07:00
James R. Barlow 2ccb3edc58 Ignore masks when deciding what color to rasterize at 2018-07-12 02:45:51 -07:00
James R. Barlow a12a5f11ff More doc updates for 7.0.0 2018-07-12 01:52:49 -07:00
James R. Barlow f46e4067f4 Dockerfile: should work now
[ci skip]
2018-07-11 00:57:20 -07:00
James R. Barlow 47bf5585e8 Dockerfile: needs leptonica 2018-07-11 00:17:04 -07:00
James R. Barlow 72fb4d39be Dockerfile: fix missing mkdir 2018-07-10 21:56:49 -07:00
James R. Barlow 9681c0e05a Eliminate Homebrew autobrewing
The Homebrew-core does such a good job of keeping things up to
date there is little to point to a private infrastructure.
2018-07-10 21:37:38 -07:00
James R. Barlow 2628575dab Adjust Dockerfile again 2018-07-10 21:14:41 -07:00
James R. Barlow f7d59068e5 Adjust Dockerfile for 7.0.0 2018-07-10 18:20:33 -07:00
James R. Barlow 4cf10d62ca More work on installation steps
[ci skip]
2018-07-10 18:20:22 -07:00
James R. Barlow 2f65e3aac8 (Hopefully) fix build on ReadTheDocs
[ci skip]
2018-07-10 13:20:33 -07:00
James R. Barlow e494cd7aa6 Update installation notes
Closes #276
2018-07-10 12:24:01 -07:00
James R. Barlow 809880f46d Brew: jbig2dec and libffi are not required 2018-07-10 11:05:36 -07:00
James R. Barlow 5925ebe31c Merge branch 'develop' (7.0.0) into master
# Conflicts:
#	.gitignore
#	.travis/linux_before_install.sh
#	docs/release_notes.rst
#	test_requirements.txt
2018-07-09 21:15:13 -07:00
James R. Barlow 316709c291 Fix typo introduced in
338593d
2018-07-09 21:07:58 -07:00
James R. Barlow 0d6a2ba554 Declare certain APIs public 2018-07-09 14:28:37 -07:00
James R. Barlow 1fcb06b1aa Update readme 2018-07-09 14:02:12 -07:00
James R. Barlow 36138c7c06 Update release notes 2018-07-09 13:56:30 -07:00
James R. Barlow 338593d9e7 Remove ruffus 2.6.3 exception special casing 2018-07-09 13:56:23 -07:00
James R. Barlow d6eb1f9578 Remove dependency on private fork of ruffus, change to official 2.7 2018-07-09 12:51:56 -07:00
James R. Barlow 1cc9d2d3d1 Fix path error on Py3.5 2018-07-08 01:01:06 -07:00
James R. Barlow 58642aa98b Fix issue #275: doesn't work when installed in non-Unicode path
Closes #275
2018-07-07 01:35:05 -07:00
James R. Barlow 7baaf00a38 Fix wrong return code tested 2018-07-05 13:49:22 -07:00
James R. Barlow 5cc23dbf24 pdfinfo: more robustness 2018-07-04 17:12:30 -07:00
James R. Barlow 216d60ea2c pdfinfo: improve the regex 2018-07-04 00:59:32 -07:00
James R. Barlow 8b0496d35e Fix invalid XML characters choking parser 2018-07-03 22:51:59 -07:00
James R. Barlow e44001641c Return a distinct error code if PDF/A fails 2018-07-03 16:59:03 -07:00
James R. Barlow 47885f4230 Remove initial qpdf.repair
Since pikepdf is doing the work the initial repair takes time and gives
little benefit.

It turns out to not be worthwhile to
save the results of PdfInfo parsing,
since the time to save this seems to exceed the costs of recalculating
it since the "weave" code. At least
for small files.
2018-07-03 16:50:05 -07:00
James R. Barlow 921767e82e ocrmypdf.exec: trap FileNotFoundError too 2018-07-03 00:05:49 -07:00
James R. Barlow 85f96b7fb0 Add test to optimize if jbig2 is present 2018-07-02 23:49:11 -07:00
James R. Barlow 890c7fd0f6 optimize: allow modification of quality settings in command line mode 2018-07-02 23:48:51 -07:00
James R. Barlow 39c44bdd2f Don't use --optimize in test since jbig2enc is not always installed 2018-07-02 23:48:23 -07:00
James R. Barlow 5f99f7f6ca Upgrade to Py3.7 locally and resolve a few issues 2018-07-02 23:47:51 -07:00
James R. Barlow 4f864bce98 Update macOS Brewfile 2018-07-02 22:25:46 -07:00
James R. Barlow 2974929b26 Make jpeg/png quality tunable args 2018-07-02 22:22:59 -07:00
James R. Barlow db837aa55c Improve release notes 2018-07-02 16:48:33 -07:00
James R. Barlow 7200623007 Fix installation for Python 3.7
Need to use private fork of ruffus for Python 3.7. Backward compatible with Python 3.6 for ruffus 2.6.3

Disable locale checking for 3.7 since the various fixes in that release should make it unnecessary.
2018-07-02 16:47:14 -07:00
James R. Barlow 73e02ae4ea Hopefully workaround Py3.5 marshal error
https://github.com/eliben/pycparser/issues/251
2018-06-29 12:54:48 -07:00
James R. Barlow d4cbef9457 Update test cache with naming rule change 2018-06-29 12:04:20 -07:00
James R. Barlow ed8ff79e10 Optimize some of our bigger test files
Only partially optimize multipage.pdf so that it hopefully
improves speed of test suite without being useless as an
optimization test.
2018-06-29 00:35:49 -07:00
James R. Barlow e725f64b6a Add test case to ensure mono is not inverted 2018-06-29 00:25:11 -07:00
James R. Barlow 0029cc4fe7 optimize: fix PNGs that were reduced to 1-bit being inverted
At some point the color gets flipped, we have to flip it again,
for mono.

Incidentally this exposed an unused
optimization. Should change the
first past to scan all images and
record monochrome xrefs, then optimize
JPEG and PNG, possibly adding
mono images to the monochrome
queue. Finally, do JBIG2 optimization.
2018-06-29 00:09:20 -07:00
James R. Barlow 9637696a54 Fix test resources naming inconsistency 2018-06-28 23:37:14 -07:00
James R. Barlow 02b3ca6862 Compress test images more heavily 2018-06-28 21:40:12 -07:00
James R. Barlow bc90f40a8f Replace all Pix.read with Pix.open 2018-06-28 15:13:26 -07:00
James R. Barlow 3d727ff4c0 Fix leptonica remove_colormap was replaced with a no-op at some point 2018-06-28 15:11:51 -07:00
James R. Barlow b0eacd6586 Add Python 3.7 support 2018-06-28 13:57:45 -07:00
James R. Barlow 7795701595 Merge branch 'test/ignore-masks' 2018-06-28 13:05:45 -07:00
James R. Barlow bf214eecb3 Use newer pikepdf API for objgen 2018-06-28 12:59:01 -07:00
James R. Barlow 434b96d734 optimize: skip incremental images if any
These are fairly rare
2018-06-24 00:18:48 -07:00
James R. Barlow b9dc109892 optimize: use new pikepdf api for objgen 2018-06-24 00:16:28 -07:00
James R. Barlow 1f40a70554 Use qpdf 8.0.2 backport, force old pytest-timeout to fix build 2018-06-23 03:14:18 -07:00
James R. Barlow e14ffbf03f v6.2.1 release notes 2018-06-23 03:01:54 -07:00
James R. Barlow 25a1dde57c Fix recent versions of tesseract not registering as textonly_pdf
This change happened sometime after the 4.0.0-beta1 release in
Ubuntu 18.04
2018-06-23 02:59:22 -07:00
James R. Barlow bf96171b65 Ignore whether or not textonly_pdf was used in cache
The difference doesn't matter in 7.0.0 anymore.
2018-06-23 02:58:26 -07:00
James R. Barlow b7ff821fa3 Fix recent versions of tesseract not registering as textonly_pdf
This change happened sometime after the 4.0.0-beta1 release in
Ubuntu 18.04
2018-06-23 02:55:58 -07:00
James R. Barlow b81daf71d1 Regenerate test cache 2018-06-23 02:02:58 -07:00
James R. Barlow faad1fc58a Reactivate two tests that weren't using their fixtures properly 2018-06-23 01:54:09 -07:00
James R. Barlow 6f48181a56 Disable a pylint 2018-06-23 01:53:04 -07:00
James R. Barlow f1305e5a37 pdfa: fix function using closure when it shouldn't 2018-06-23 01:52:36 -07:00
James R. Barlow f0e0f92776 leptonica: fix variables defined on class outside __init__ 2018-06-23 01:51:55 -07:00
James R. Barlow 807c8b0726 Trailing whitespace 2018-06-23 01:51:19 -07:00
James R. Barlow 6333ec928c Cleanup some cases where log was lazy and should be 2018-06-23 01:50:27 -07:00
James R. Barlow cd220d9ed9 pipeline: search_window variable not actually used 2018-06-23 01:48:57 -07:00
James R. Barlow 76532649b8 tesseract.get_orientation: removed unused language parameter 2018-06-23 01:48:24 -07:00
James R. Barlow b0dbaeafc5 Cleanup unused imports 2018-06-23 01:47:53 -07:00
James R. Barlow 2530d1791b Fix several pylint errors and warnings 2018-06-23 00:54:22 -07:00
James R. Barlow 94150f414a Remove qpdf.merge
We no longer need to merge pages this way. Much of the functionality
was there to implement page splitting without hitting ulimit which
will be fixed in qpdf > 8.0.2. The tests were expensive to run.

Also remove pytest-timeout since it breaks the Linux build.
2018-06-23 00:45:03 -07:00
James R. Barlow 54e74f84cc Remove special of TypeError from ruffus
split_pages would still run if repair_pdf failed, for some reason.
Since we are no longer splitting pages this is vestigial.
2018-06-23 00:41:20 -07:00
James R. Barlow 76e7e8dbbb Replace several uses of str(path) with fspath(path)
Helps make it more explicit. Did not do this to tests because use of paths
is more involved there.
2018-06-22 21:00:47 -07:00
James R. Barlow 324598e992 Remove helpers.universal_open()
This helper function only had a single usage, this was always an awkward
way to support Python 3.5 that I'd forget to use.
2018-06-22 17:56:20 -07:00
James R. Barlow 9e765ddf46 Rename _optimize to optimize.py 2018-06-22 17:51:57 -07:00
James R. Barlow 6ac9e92f17 Fix PEP8 docstring convention misuse in a few places 2018-06-22 17:51:25 -07:00
James R. Barlow faaa4a1def Ghostscript, PDF/A: support pathlib 2018-06-22 17:45:10 -07:00
James R. Barlow 0aa51f0f3a Remove fitz from Travis 2018-06-18 15:38:41 -07:00
James R. Barlow 73431d9761 Remove obsolete _naive_find_text 2018-06-13 14:00:50 -07:00
James R. Barlow 45cb4525cf Remove other references to PyMuPDF 2018-06-13 01:02:53 -07:00
James R. Barlow 8c84c515b6 Use Ghostscript for text region detection
Ghostscript txtwrite seems to be quite effective at the task.

Eliminates dependency on fitz
2018-06-13 00:58:09 -07:00
James R. Barlow 1dfbbdebf4 Adjust for pikepdf API change 2018-06-08 22:47:56 -07:00
James R. Barlow 740918daee Create debug envvar to override Creator or Producer
Note that Ghostscript always overrides Producer
2018-06-06 23:17:28 -07:00
jbarlow83andGitHub 1d10eac764 Add wiki link to issue template
[ci skip]
2018-06-06 12:59:59 -07:00
jbarlow83andGitHub 3f868118cd Remove gpg
[ci skip]
2018-06-06 12:58:02 -07:00
James R. Barlow 04d79b15b4 optimize: fix error in Py3.5 2018-06-06 12:25:32 -07:00
James R. Barlow a13c398c06 Suppress some spurious tesseract errors 2018-06-05 23:26:28 -07:00
James R. Barlow e3b3f716ee optimize: use tempdir for cmdline invocation 2018-06-05 21:20:54 -07:00
James R. Barlow cf43c06f46 Use python-xmp-toolkit for xmp check
Eliminates PyPDF2 and defusedxml as dependencies.
2018-05-29 22:00:52 -07:00
James R. Barlow 74a5a18607 Tweak release notes 2018-05-28 14:52:06 -07:00
James R. Barlow 44241c6dd5 Travis: remove deploy to testpypi since it's broken 2018-05-27 01:49:18 -07:00
James R. Barlow 8fff496ffd Fix Py3.5 not understanding os.path.exists(Path(...)) 2018-05-26 22:55:22 -07:00
James R. Barlow edf75c519c Update v7 release notes 2018-05-26 02:08:49 -07:00
James R. Barlow 9608b22d34 Remove all uses of PyPDF2 except PDF/A check
Leave PDF/A check alone for now, since pikepdf has no equivalent.
2018-05-26 02:07:18 -07:00
James R. Barlow 8ba4968c48 pdfinfo: more robustness 2018-05-26 01:54:25 -07:00
James R. Barlow ffdd78f1a5 pdfinfo: Fix text_operators type not changed in related commit 2018-05-25 02:10:39 -07:00
James R. Barlow ad9f8ca78e pdfinfo: reinstate stack normalization for q/Q 2018-05-25 01:28:26 -07:00
James R. Barlow 78a686ecb4 Consider qpdf behavior on algo4 a pass
qpdf opens files with null user password, so do the same.
2018-05-25 00:33:31 -07:00
James R. Barlow 59e786eb3c Remove old code to deal with single page only things 2018-05-25 00:32:55 -07:00
James R. Barlow 6d0461435f Use OperandGrouper whitelist 2018-05-24 22:52:33 -07:00
James R. Barlow 0a04a60f69 Document need for pdfinfo to be pickleable 2018-05-24 22:24:13 -07:00
James R. Barlow 68d8642988 Found out this test was extremely slow - no reason to actual use a large file 2018-05-24 22:22:51 -07:00
James R. Barlow 16f70ff054 Main changeset for pikepdf-based refactor pdfinfo 2018-05-24 22:22:01 -07:00
James R. Barlow c00aeafff0 Add scratch file 2018-05-24 22:20:15 -07:00
James R. Barlow 83f35e00f3 Start removing PyPDF2 2018-05-21 01:28:21 -07:00
James R. Barlow 786a2ad65a Make optimize test do a little more 2018-05-18 17:50:39 -07:00
James R. Barlow 9425506c2a Use pikepdf to handle paletted images
Removes all use of PyMuPDF in optimize
2018-05-18 17:44:29 -07:00
James R. Barlow 93b858afd1 Remove qpdf appimage support for now, check for pngquant 2018-05-18 16:24:33 -07:00
James R. Barlow 7b0a3ec365 Add notes for v7 2018-05-18 00:20:45 -07:00
James R. Barlow 083d442529 main: wording change 2018-05-18 00:20:24 -07:00
James R. Barlow b52eb95cf8 optimize: use pikepdf to save PIL images
Eliminates another usage of PyMuPDF in the main path.
2018-05-18 00:18:44 -07:00
James R. Barlow f4571e2508 Ensure we try compress anything that's not compressed when saving 2018-05-17 22:05:01 -07:00
James R. Barlow b06ef03aac pipeline: use the resolution of the OCR image rather than recalculating
(Recalculating would fail if the image is not centered.)
2018-05-17 16:51:53 -07:00
James R. Barlow 1d1962a106 weave: fix rescaling logic
rotation % 90 == 0 is always true.
2018-05-17 16:50:01 -07:00
James R. Barlow 4b98e9ff08 weave: if we don't have textonly_pdf, delete instruction to draw image 2018-05-17 16:49:20 -07:00
James R. Barlow f83ca5d8ac weave: whitespace 2018-05-17 16:06:36 -07:00
James R. Barlow 95cb4d22d7 pipeline: make /Info from indirect object as required 2018-05-17 16:06:13 -07:00
James R. Barlow 0c279b01a4 Fix test failure on missing JobContext 2018-05-17 01:16:58 -07:00
James R. Barlow 3b820ffa7b test_metadata: change from xfail to skipif without fitz 2018-05-17 00:14:57 -07:00
James R. Barlow 35cb416563 pipeline: remove fitz-based attempt to repair table of contents
Prior to unsplit, if we were rebuilding the PDF we'd lose the
table of contents. With unsplit we keep the original file and patch
the table of contents as necessary, adn that works fine.
This remaining bit of code from PyMuPDF actually damages the
table of contents and removing it fixes the test suite. G'bye.
2018-05-16 23:24:57 -07:00
James R. Barlow cdb737259c pipeline: remove old page merge strategies 2018-05-16 22:16:54 -07:00
James R. Barlow 0843b5939c pipeline: Move weave* to its own file 2018-05-16 22:08:31 -07:00
James R. Barlow 2b5f23a2d1 Add code to repair ToC with pikepdf 2018-05-16 21:39:23 -07:00
James R. Barlow 5e20d1d554 metadata: Fix failing test on __getitem__['/CreationDate'] 2018-05-16 13:46:07 -07:00
James R. Barlow 18595ca86a Use pikepdf for get_pdfmark
It does fine.
2018-05-16 12:24:35 -07:00
James R. Barlow 3e269fa188 Ubuntu 14.04 has a qpdf 8.0.2 backport, making life easier 2018-05-15 21:43:19 -07:00
James R. Barlow 65405c2cb9 Try getting qpdf from Ubuntu 18.04 2018-05-15 21:27:27 -07:00
James R. Barlow 442cf8897a Travis: maybe upgrading wheel? 2018-05-15 18:12:35 -07:00
James R. Barlow d5fb275e9e Travis: hack in qpdf appimage version
qpdf from appimage does not report its version with --version if renamed
or accessed via symlink. Use an environment variable to supply it
where needed.
2018-05-15 17:45:58 -07:00
James R. Barlow e60aec81ca Travis: why can't we use qpdf appimage? 2018-05-15 16:59:16 -07:00
James R. Barlow 398e9e535e optimize: Changed pikepdf API 2018-05-15 16:29:57 -07:00
James R. Barlow 08bf651ef2 Refactor JBIG2 path for non-CCITT monochrome images 2018-05-15 15:32:15 -07:00
James R. Barlow 6171de41bf optimize: move a lot of image scanning code to pikepdf 2018-05-14 22:21:53 -07:00
James R. Barlow f0a56592e2 Pull JobContext out of pipeline.py to avoid circular reference 2018-05-14 14:01:25 -07:00
James R. Barlow 87a7d4d1a8 Another fitz failure - incorrect object reference introduced
MuPDF/fitz changed some font references to point to table of contents
entries, corrupting the page.  It no longer gets to save.
2018-05-14 13:58:49 -07:00
James R. Barlow 05287902a2 Travis: again 2018-05-13 11:02:25 -07:00
James R. Barlow 96e453feb6 Travis: Tweak setup so it can run 2018-05-13 01:21:24 -07:00
James R. Barlow 9c0fa9fc04 Travis: again 2018-05-13 01:17:04 -07:00
James R. Barlow 3bde0715b0 Move qpdf to before_script 2018-05-13 01:01:48 -07:00
James R. Barlow e2ec3d8b9b Travis: adjust qpdf appimage 2018-05-13 00:53:31 -07:00
James R. Barlow ad91eaf8a7 Travis: try using qpdf appimage to speed up build 2018-05-13 00:42:48 -07:00
James R. Barlow b6d30214fd PyMuPDF 1.13.4 looks good, use it 2018-05-12 12:35:46 -07:00
James R. Barlow c4ab01d63d Fix "AttributeError: 'ImageInfo' object has no attribute '_type'"
Also deal with 'fixme' imagemask comment.

Also fix bpc incorrectly set to 8 by default on stencil masks.
2018-05-12 12:14:57 -07:00
James R. Barlow 4ba3b3f55a Fix rotate_pages_threshold test failure 2018-05-12 11:47:46 -07:00
James R. Barlow 52d2706a9e optimize: Fix error causing many images to be skipped 2018-05-12 01:37:30 -07:00
James R. Barlow 964afc69f6 leptonica: ErrorTrap is an implementation detail 2018-05-12 01:21:45 -07:00
James R. Barlow 3ddf545ccd optimize: leptonica can fail to open PNG
ERROR - Info in pixReadStreamPng: converting (cmap + alpha) ==> RGBA
Error in pixReadStreamPng: spp == 1, cmap, trans array, invalid depth: 4

To investigate later....
2018-05-12 01:21:19 -07:00
James R. Barlow f9374733bb optimize: process ICCBased images that declare an /Alternate we recognize 2018-05-12 00:43:36 -07:00
James R. Barlow 5930135f45 optimize: Refactor naming helpers 2018-05-12 00:42:24 -07:00
James R. Barlow f03f6bc128 optimize: document problem with transcode free compressed image data 2018-05-11 23:43:06 -07:00
James R. Barlow 6c50c70235 Try to optimize paletted images 2018-05-11 23:42:26 -07:00
James R. Barlow 8790fc2c1b optimize: add knobs to control image quality but don't show the user yet 2018-05-11 23:41:49 -07:00
James R. Barlow f86c4fccf4 optimize: don't alter >8 bpc images 2018-05-11 22:31:24 -07:00
James R. Barlow 7d0785e9ed main: do better parameter validation 2018-05-11 22:31:09 -07:00
James R. Barlow 2cac88162c Ignore masks when deciding what color to rasterize at 2018-05-11 21:27:57 -07:00
James R. Barlow 4809627d8a Fix jbig2enc name 2018-05-11 17:51:08 -07:00
James R. Barlow 871979abd6 Temporarily unbreak without fitz mode 2018-05-11 17:32:15 -07:00
James R. Barlow efb95722ca Travis: Use declarative APT for Tesseract too 2018-05-11 12:46:10 -07:00
James R. Barlow d9bbb80a6b Don't try to run jbig2 when not available 2018-05-11 12:42:00 -07:00
James R. Barlow 3254315127 Update test cache 2018-05-11 12:19:50 -07:00
James R. Barlow ca297fd26b Update tests 2018-05-11 02:33:44 -07:00
James R. Barlow ac36a43cef Warn about --user-words not having any effect
Might be available in full release of Tess4
2018-05-11 02:31:07 -07:00
James R. Barlow f00183115d Update our dependencies 2018-05-11 02:11:55 -07:00
James R. Barlow 161b29a899 Check jbig2 when optimizing is requested 2018-05-11 02:11:01 -07:00
James R. Barlow 72253d09fa Add arguments to control optimization 2018-05-10 22:23:24 -07:00
James R. Barlow 40d09ddb23 Fix merge error in Leptonica 2018-05-10 21:17:47 -07:00
James R. Barlow 3026d86a9e Remove jbig2enc.py 2018-05-10 21:15:07 -07:00
James R. Barlow 0661a7edc3 Merge optimize 2018-05-10 21:05:32 -07:00
James R. Barlow 24b0adfacc Merge branch 'master' into develop 2018-05-10 20:54:55 -07:00
James R. Barlow acc6698ab3 Make XML metadata test actually work 2018-05-10 20:37:10 -07:00
James R. Barlow 606d3e6aa1 Remove tests that exercise obsolete features (tesseract, -g) 2018-05-10 20:33:32 -07:00
James R. Barlow 687a7954d6 test_main: uses leptonica 2018-05-10 19:05:31 -07:00
James R. Barlow 36a53a7b37 Weave: Unconditionally rotate and scale the text layerThis solves two issues. First, the text layer can end up being adifferent size, probably if the DPI is not an integer; scaling helps itfit slightly better. Second, other printable text on the page can end uphorizontally scaled or misaligned if we don't all of our drawing in aq/Q pair. 2018-05-10 19:03:31 -07:00
James R. Barlow 0a5982a902 PyMuPDF tweaks: don't clean
In MuPDF 1.13 clean might be unreliable, so explicitly don't do it,
even though it doesn't cause trouble in 1.12.
2018-05-10 18:50:52 -07:00
James R. Barlow 601863f9e9 Return to PyMuPDF 1.12.5 2018-05-10 18:47:10 -07:00
James R. Barlow c9ce731119 Fix DPI mismatch between OCR page and source page 2018-05-10 17:34:08 -07:00
James R. Barlow abed8e034e Add metadata preservation test from stash 2018-05-10 16:43:28 -07:00
James R. Barlow 63032d304d Revert "Since PyMuPDF 1.13.3 corrupts text, pin 1.12.5 and work around it"
This reverts commit b0ce7c63dd.
2018-05-10 16:27:17 -07:00
James R. Barlow a57ecede78 Refactor textareas to remove duplicate code 2018-05-10 16:26:52 -07:00
James R. Barlow b0ce7c63dd Since PyMuPDF 1.13.3 corrupts text, pin 1.12.5 and work around it 2018-05-10 16:10:24 -07:00
James R. Barlow d139a11c16 Weave: periodically save to prevent indefinite growth of open file list 2018-05-10 15:08:57 -07:00
James R. Barlow aef043db0b Revise parameter validation for output-type, pdf-renderer, lang 2018-05-10 14:53:22 -07:00
James R. Barlow b8f3ead541 Remove tesseract renderer entirely
Grafting lets us work with older Tesseract versions as if they could use
sandwich, so there is no point in keeping it. It's been deprecated for a
long time now anyway.
2018-05-10 14:06:13 -07:00
James R. Barlow e0bb898f29 Remove hocr debug renderer (-g)
The fact that this produces additional pages makes it a maintenance
burden. hocr can be debugged using hocrtransform.
2018-05-10 13:48:39 -07:00
James R. Barlow 45336c7c28 textareas: filter out images 2018-05-10 01:17:28 -07:00
James R. Barlow 20aabb2e83 When deciding if there is a text on a page, ignore the margins
Margins may include watermarks or digital stamps on otherwise
text-free pages.
2018-05-10 01:16:11 -07:00
James R. Barlow 1539e24d61 Ignore masks when deciding what color to rasterize at 2018-05-10 00:49:36 -07:00
Fabian Rodriguezandjbarlow83 c7cf041e4a Fixed language option example (French) (#266)
Replace fre to fra.
2018-05-10 00:10:27 -07:00
James R. Barlow da80d3f354 Add unconditional (for now) whiteout of text areas 2018-05-07 17:37:46 -07:00
James R. Barlow 001c8d7678 Upgrade PyMuPDF version 2018-05-07 16:24:26 -07:00
James R. Barlow 38ab03655b Restore unpaper
It's a suggested/recommended dep not required in Deb/Ubu.
2018-05-06 21:36:12 -07:00
James R. Barlow 9226f8a5d1 Trap PDF/A-3 errors on old Ghostscript 2018-05-04 15:29:43 -07:00
James R. Barlow 5c8a007f3e Fix failure to prevent use of Ghostscript on /UserUnit files 2018-05-04 13:34:34 -07:00
James R. Barlow b3ad3e297d v6.2.0 fixes 2018-05-03 17:04:23 -07:00
James R. Barlow d607553e48 v6.2.0 Release notes 2018-05-03 16:47:21 -07:00
James R. Barlow 7cf83c77ca Merge branch 'feature/pdfa3' 2018-05-03 16:45:57 -07:00
James R. Barlow 8a9f174f63 Fix XMP validation issue with /CreationDate
Related to previous validation issue. If the /CreationDate had no
timezone, Ghostscript also creates invalid metadata. Work around this.
Also fix up PDF date decoding, and transcode dates to standardize them.
2018-05-03 16:30:20 -07:00
James R. Barlow 98a0786c32 Add 18.04 update procedure 2018-05-03 13:55:16 -07:00
James R. Barlow df1129724c Update Dockerfile for Ubuntu 18.04 2018-05-03 01:27:13 -07:00
James R. Barlow 423cef08bf Handle procset properly 2018-05-02 14:48:02 -07:00
James R. Barlow 04580accb4 Document aliasing of tesseract renderer 2018-05-02 14:47:38 -07:00
James R. Barlow 6376f77b8c Refactor, remove trigonometry 2018-05-02 12:30:34 -07:00
James R. Barlow e27e614ed9 Fixed rotation hard case 2018-05-02 01:32:11 -07:00
James R. Barlow b0c04704a1 Fixed all but one rotation case 2018-05-02 01:24:21 -07:00
James R. Barlow 6bb6bf8323 Fix correction angle used from wrong page 2018-05-02 01:00:30 -07:00
James R. Barlow e22fe8aefc Silence debug messages 2018-05-01 23:51:54 -07:00
James R. Barlow 76276f61e5 Split out rotation related tests 2018-05-01 23:51:35 -07:00
James R. Barlow bfd26e6ec6 Tests: confirm OCR layer copied 2018-05-01 23:16:41 -07:00
James R. Barlow d787e1ea0f ghostscript.py not saved in last commit
Given importance of last one, confirmed that when the file is saved all tests pass too.
Passing is invariant with this change.
2018-05-01 22:59:22 -07:00
James R. Barlow b5d7e9cbb0 Fix all issues with rotations
All tests now pass
2018-05-01 22:50:20 -07:00
James R. Barlow f3b6d9dcdf Fix a comment about Tesseract behavior in certain versions 2018-05-01 21:31:09 -07:00
James R. Barlow a9abe13185 Remove the old tesseract pdf_renderer 2018-05-01 17:31:34 -07:00
James R. Barlow 6b315e8315 Add ability to disable cache 2018-05-01 15:52:00 -07:00
James R. Barlow 37677de884 Fix regressions: pdfa.ps not used, PDF/A failures, handling of text layers with no font 2018-05-01 15:51:46 -07:00
James R. Barlow c7387de325 Fix auto rotate 2018-05-01 15:18:28 -07:00
James R. Barlow 2495b1e038 Refactor find font, get test cases working again 2018-05-01 14:48:41 -07:00
James R. Barlow 073ee52ce7 Use hocr and weave; eliminate old combine layers and merge pages 2018-05-01 14:21:53 -07:00
James R. Barlow 54150a14e9 Further elimination of tesseract renderer special casing
We don't need to keep a "skip page" around anymore since
skipping means just not grafting on the text layer.
2018-05-01 13:36:20 -07:00
James R. Barlow 88ff091cce Unify tesseract and sandwich renderer paths
Since the new weaving method copies the font and content
stream from the Tesseract PDF, it doesn't matter if Tesseract
happens to have an image or not.
If Tesseract is text-only capable we use that feature for efficiency,
but ignore the image either way.
2018-05-01 13:24:20 -07:00
James R. Barlow e87a5776f1 Remove now-unnecessary code to rotate pages
Track only the decision to change rotation.
2018-05-01 13:01:25 -07:00
James R. Barlow 0806ce6406 Fix rotation for unsplit (modulo --rotate-pages) 2018-04-30 20:58:42 -07:00
James R. Barlow 6409894a71 feature/unsplit-try-imagerotate 2018-04-30 20:48:59 -07:00
James R. Barlow e7286f6129 Unsplit now works with multipage, --force-ocr 2018-04-30 14:46:20 -07:00
James R. Barlow 2ab94b3151 unsplit: it's alive
First successful file output.
2018-04-28 01:57:41 -07:00
James R. Barlow 7ee90890ec Add copying of essential information from Tesseract textonly 2018-04-27 23:19:08 -07:00
James R. Barlow 383e726d65 Expand size growth reasons to other arguments that trigger transcoding 2018-04-27 19:34:57 -07:00
James R. Barlow e046f70642 Set OMP_THREAD_LIMIT unconditionally, for pngquant 2018-04-27 19:19:30 -07:00
James R. Barlow 2131ad4670 Fix --remove-background error on PDFs with colormapped images
It's unclear how exactly a
colormapped image gets to this
spot given the tendency of other
image processing tools to flatten
such images, but someone made it happen, so now we make sure
the image is okay.

Closes #262
2018-04-27 17:21:01 -07:00
James R. Barlow 219fe2155b test_pageinfo: remove duplicate import 2018-04-27 17:16:42 -07:00
James R. Barlow 4209034d20 Add gpg key to issue template 2018-04-27 15:51:26 -07:00
James R. Barlow abcae0c2a4 Fix helpers.py again 2018-04-25 22:10:51 -07:00
James R. Barlow 0934905493 Don't suppress error message from config_notfound
Since it showed up in s390x bionic
2018-04-25 21:58:18 -07:00
James R. Barlow 11cd6201d9 helpers: fix missing call to complain()
In practice this is probably unreachable.
2018-04-25 21:57:50 -07:00
James R. Barlow 8d2a917676 Page unsplit, development 2018-04-25 21:56:43 -07:00
James R. Barlow 44b4afa534 Begin conversion from page splititng to page markers 2018-04-23 22:57:50 -07:00
James R. Barlow 775be3933c Cherrypick merge_pages unification 2018-04-20 23:08:15 -07:00
James R. Barlow df87e21c85 Add support for PDF/A-3
No ability to attach files however
2018-04-20 00:06:55 -07:00
Hugoandjbarlow83 d761d80750 Use more standard __version__ rather than PILLOW_VERSION (#257) 2018-04-19 23:35:32 -07:00
James R. Barlow 8052019dde optimize: fix reporting of jbig2 groups 2018-04-19 01:54:44 -07:00
James R. Barlow a3d8950088 optimize: Don't save JPEGs if larger 2018-04-19 01:25:49 -07:00
James R. Barlow 004f5d3bf1 optimize: further improve decodeparms handling 2018-04-18 15:52:25 -07:00
James R. Barlow f5d308a156 optimize: refactor tricky /Filter and /DecodeParms handling 2018-04-18 15:30:21 -07:00
James R. Barlow 3869996758 optimize: jbig2 error 2018-04-18 01:36:38 -07:00
James R. Barlow cdb2107c4e optimize: jbigs2 fix 2018-04-18 01:31:35 -07:00
James R. Barlow 4db2b3413b optimize: more robustness 2018-04-18 01:25:34 -07:00
James R. Barlow b2f31bec79 Make optimize a lot safer 2018-04-18 00:20:06 -07:00
James R. Barlow 78f9f4a266 Be more defensive about accessing 2018-04-18 00:11:39 -07:00
James R. Barlow ad6087c342 optimize: more fixes 2018-04-17 23:58:10 -07:00
James R. Barlow 0d6ef430de optimize: fix "length not defined" 2018-04-17 23:38:00 -07:00
James R. Barlow a5942209e8 optimize: fix error on missing /Filter 2018-04-17 23:27:56 -07:00
James R. Barlow 9a60694cfc optimize: ccitt header fixes
Changed to match TIFF spec's use of unsigned types, eliminated check for
/Columns.

There is some complex behavior for /Width != /Columns and
(/Width, /Columns) mod 8 != 0
that is not described well in the PDF spec.
2018-04-17 23:27:25 -07:00
James R. Barlow 4bf13f4737 optimize: be less chatty 2018-04-17 23:25:41 -07:00
James R. Barlow 9e89b75186 Merge v6.1.5 2018-04-17 22:51:13 -07:00
James R. Barlow 0b10db91be Fix regression: Disable Ghostscript JPEG passthrough entirely 2018-04-17 17:00:24 -07:00
James R. Barlow 1a516b2af9 Fix regression: time stamp test suite failures 2018-04-17 16:59:21 -07:00
James R. Barlow 076363d78e Disable JPEG passthrough for Ghostscript 9.23
Seems to corrupt JPEGs involved in image masks?
2018-04-17 16:31:03 -07:00
James R. Barlow 5fde214290 Update notes for v6.1.5 2018-04-17 15:23:35 -07:00
James R. Barlow a620724d6a Fix PDF/A validation failure due to timezone being omitted from /ModDate 2018-04-17 15:16:48 -07:00
James R. Barlow 640b953ec7 Fix PDF/A validation failure due to timezone being omitted from /ModDate 2018-04-17 14:55:32 -07:00
James R. Barlow a009ca7597 Disable JPEG passthrough for Ghostscript 9.23
Seems to corrupt JPEGs involved in image masks?
2018-04-17 13:54:34 -07:00
James R. Barlow 7368399f8b Clarify license of two test files - https://github.com/jbarlow83/OCRmyPDF/issues/254 2018-04-17 11:56:36 -07:00
James R. Barlow c974aec934 Search for image masks too 2018-04-17 02:06:08 -07:00
James R. Barlow 3033f03f64 Iterate images with pikepdf / fix mono PNG corruption
To work around PNG corruption problem in PyMuPDF for monochrome images,
extract and save monochrome CCITT with synthetic TIFF header.

Works better but currently skips /ImageMask due to qpdf
implementation, which affects many files.
2018-04-17 01:50:37 -07:00
James R. Barlow 72723e0bb5 optimize: be quieter 2018-04-16 18:06:02 -07:00
James R. Barlow 2fb6ab3939 Trap writePNG error 2018-04-16 17:29:10 -07:00
James R. Barlow 25c1c160b8 Move optimize to new file 2018-04-16 17:22:06 -07:00
James R. Barlow 7e92895471 Parallelize pngquant 2018-04-16 12:37:51 -07:00
James R. Barlow d291d48991 PNG palette: parse PDF string from leptonica instead
Seems better to accept whatever leptonica rather than make detailed
assumptions about how it encodes the palette.

Experimented with setting FlateDecode on the palette but it seems to
expand it.
2018-04-16 12:16:13 -07:00
James R. Barlow 0e6b8042b0 Implement PNG palettization 2018-04-16 11:18:52 -07:00
James R. Barlow 34c78a892a Fix list table for tests/resources
[ci skip]
2018-04-15 23:52:19 -07:00
James R. Barlow 9d28879505 Update Ubuntu 14.04 instructions
Closes #252
2018-04-14 17:30:33 -07:00
James R. Barlow 2482296e2b hocr: avoid division by zero
Issue #253 - PDF that produces the error is not available, but if font_width
is zero, chances are the text is nonprinting characters, so suppress it.
2018-04-14 17:24:21 -07:00
James R. Barlow f755fb76ee Try pngquant 2018-04-14 01:37:14 -07:00
James R. Barlow c61b5dcb62 Fix PDF/A validation error from setting /Predictor 0 2018-04-14 01:36:46 -07:00
James R. Barlow fae893b9d9 Reinstate transcoding of PNG 2018-04-14 00:19:24 -07:00
James R. Barlow 10aadefd6a Document return codes 2018-04-14 00:18:58 -07:00
James R. Barlow e75b6280fd Try reading compressed data directly to see if Leptonica will add predictor
Turns out it does not transcode at all in this case, so probably going
to revert to transcoding PNG -> PNG. However if pngquant or similar is
done, this API will still be useful.
2018-04-13 23:55:23 -07:00
James R. Barlow 8c4023165a Release L_COMP_DATA properly 2018-04-13 23:53:41 -07:00
James R. Barlow b7d403f106 Deprecate Pix.read() behaving as an open function 2018-04-13 23:52:46 -07:00
James R. Barlow b069de0caa Use Leptonica to rewrite all PNGs with predictor
Leptonica does a better job of encoding them than Ghostscript, about -15%.
For a test file 450k worth of
PNGs was reduced to 388k with no loss of quality.
2018-04-13 16:35:50 -07:00
James R. Barlow 136da74bfa Update branch with v6.1.4 2018-04-13 12:57:21 -07:00
James R. Barlow 7fc897e6dc Fix NameError 'ghostscript' 2018-04-12 21:24:05 -07:00
James R. Barlow 9b731d63b8 Set Ghostscript -sColorConversionStrategy the way old/new versions expect 2018-04-12 16:28:48 -07:00
James R. Barlow 10aa59f674 v6.1.4 fix test suite regression with Ghostscript 9.23 2018-04-12 15:16:54 -07:00
James R. Barlow 1f7837e7b1 v6.1.4 release notes update 2018-04-12 00:55:45 -07:00
James R. Barlow ba0535e3fb Update test cache to account for unpaper --layout none change 2018-04-12 00:48:21 -07:00
James R. Barlow 49fa7f6b5c tesseract_cache: don't reveal host system file paths in manifest file 2018-04-12 00:47:28 -07:00
James R. Barlow c95db246d4 v6.1.4 merge 2018-04-11 15:58:00 -07:00
James R. Barlow 1ba93371ce docs: Update installation to reflect qpdf 7.0.0 requirement 2018-04-11 15:40:50 -07:00
James R. Barlow fedbbdb575 Travis: compile qpdf from source
The older version in Travis's Ubuntu 14.04 can't pass the test suite anymore.
2018-04-11 15:40:45 -07:00
James R. Barlow 85ebba72bc Fix setup.py syntax 2018-04-10 18:30:48 -07:00
James R. Barlow b6cd436d5d setup: Blacklist Pillow 5.1.0 on macos
https://github.com/python-pillow/Pillow/issues/3068
2018-04-10 18:15:37 -07:00
James R. Barlow ec170c7e1e Travis: use setup.py for requirements, don't override with .txt 2018-04-10 17:52:19 -07:00
James R. Barlow f6399eb90f optimize: use Leptonica to compact JPEGs
Pillow could do it too, but Leptonica is somewhat more PDF aware.
2018-04-10 17:45:05 -07:00
James R. Barlow 77f2448e59 Leptonica: add L_COMP_DATA compressed data manager 2018-04-10 17:44:03 -07:00
James R. Barlow 3d69b46fca Release notes 2018-04-10 15:53:02 -07:00
James R. Barlow 4b6153ad18 Use defusedxml for XML parsing when reading XMP 2018-04-10 14:25:13 -07:00
James R. Barlow 75d37eb103 docs: expand ocr of image usage 2018-04-09 13:06:09 -07:00
James R. Barlow 11b6f77df0 unpaper: close images on error paths 2018-04-09 13:05:12 -07:00
James R. Barlow db8b0319dd get_version: repeat system error messages if the process exists with a signal 2018-04-09 13:04:51 -07:00
James R. Barlow c9dd330766 JBIG2: refactor, don't recompress existing JBIG2 2018-04-09 13:04:10 -07:00
James R. Barlow e40228102c JBIG2: Streams created in this manner are already indirect objects 2018-04-06 17:11:17 -07:00
James R. Barlow 7889c6fb4c Parallelize JBIG2 execution with thread pools 2018-04-06 17:00:23 -07:00
James R. Barlow 6eb1773110 Fix JBIG2Globals included multiple times in output 2018-04-06 17:00:03 -07:00
James R. Barlow 1d25823746 Implement functional, single threaded optimize
Passes verapdf
2018-04-06 15:49:16 -07:00
James R. Barlow d1d4f1e198 Add issue links to release notes 2018-04-06 14:52:40 -07:00
James R. Barlow 709c01c7a1 Regroup three merge steps into a single step
All take the same inputs and deliver similar outputs, so it makes sense.
2018-04-06 01:07:02 -07:00
James R. Barlow 4a341c9034 Merge branch 'master' into feature/jbig2-2018 2018-04-05 21:29:39 -07:00
James R. Barlow be41ff6d54 Update flowchart
[ci skip]
2018-04-05 21:26:37 -07:00
James R. Barlow 1dbb6f1746 Notes on relevant envvars, repology 2018-04-05 02:15:01 -07:00
James R. Barlow 753e6274ab Tell unpaper to use --layout none so it won't blank out multi column text 2018-04-05 02:14:33 -07:00
James R. Barlow 7f462c618b v6.1.3 notes 2018-04-03 00:11:20 -07:00
James R. Barlow d8ac6e28ab Convert monochrome images to JBIG2
Awkwardly using fitz and pikepdf, transcode monochrome to CCITT.
This requires _OCRMYPDF_NO_FITZ=1.

00000x.opt.pdf can be checked for JBIG2 to confirm, but this file is
not passed to the output since it's not all wired up yet.
2018-04-03 00:00:53 -07:00
James R. Barlow a95ffcdc46 Experimental add jbig2
It appears that fitz forces conversion of jbig2 to ccitt no matter what,
so pikepdf will be needed to patch jbig2 images.
2018-04-03 00:00:53 -07:00
James R. Barlow 1b01d45dd2 Warn about Python 3.5 page count issue 2018-04-02 19:29:17 -07:00
James R. Barlow 7a1cd39b21 Fix creation date metadata lost from input
Closes #247
2018-04-02 17:53:39 -07:00
James R. Barlow 1c1fd9616a Don't depend on pytest-xdist in setup.cfg 2018-04-02 11:45:03 -07:00
Sean Whittonandjbarlow83 11e19e4085 remove addopts key from tool:pytest section of setup.cfg (#246)
The '-n' command line argument is not supported by recent pytest.
2018-04-02 14:43:38 -04:00
James R. Barlow 2a43f73228 Update installation.rst, further info on fitz 2018-04-02 11:32:57 -07:00
James R. Barlow b1d1310a75 Dockerfile: use fitz 2018-04-02 11:08:03 -07:00
James R. Barlow 0e7fa78e65 Remove inaccurate statement from setup.py 2018-04-01 13:20:17 -07:00
James R. Barlow 4032570d97 Change docs for fitz/PyMuPDF 2018-04-01 13:19:57 -07:00
James R. Barlow 90644a3017 pipeline: refactoring, use with block for images 2018-03-31 13:26:40 -07:00
James R. Barlow 4f6bffb477 Update copyrights 2018-03-31 11:54:38 -07:00
James R. Barlow 158f902c3b Fixed setup.py syntax error 2018-03-30 14:00:36 -07:00
James R. Barlow 6dc25ddc6e v6.1.2: add license to wheels, depend on defusedxml 2018-03-30 13:22:35 -07:00
James R. Barlow 7f6aaeaecf v6.1.2 2018-03-30 12:39:33 -07:00
James R. Barlow ace439910e Remove PyMuPDF 1.12.4 shim 2018-03-30 12:33:27 -07:00
James R. Barlow 7f038568de Add envvar to ease testing without PyMuPDF 2018-03-30 12:32:48 -07:00
James R. Barlow af777c0b6a Test macos without fitz too 2018-03-30 00:13:09 -07:00
James R. Barlow fc299032a4 v6.1.1 release notes
Better get the last one out
2018-03-30 00:11:52 -07:00
James R. Barlow e0f3f07907 Fix text reported as found on all pages when PyMuPDF is not available 2018-03-30 00:10:53 -07:00
James R. Barlow b36df9cf9e pdfa: codecs.encode -> hexlify (simpler) 2018-03-29 22:17:23 -07:00
James R. Barlow 81c3f780d4 Travis: Should test 3.6 Linux without fitz too 2018-03-28 23:54:43 -07:00
James R. Barlow b51efdd3e3 Travis: don't upload to legacy PyPI anymore, it will stop working soon 2018-03-28 23:40:29 -07:00
James R. Barlow 610b769df9 Update release notes 2018-03-28 23:33:34 -07:00
James R. Barlow 527f4d0101 Workaround fitz not escaping parentheses
Closes #239
2018-03-28 23:23:34 -07:00
James R. Barlow 8d9be43c60 test_bookmarks_preserved won't raise ImportError any more
Due to trapping this in ocrmypdf.lib
2018-03-28 23:22:55 -07:00
James R. Barlow 40ef4f0bbe Add new argument --skip-repair to skip the repair step 2018-03-28 00:54:58 -07:00
James R. Barlow d0271d5049 More debug messages on repair; update notes 2018-03-28 00:39:38 -07:00
James R. Barlow 5becfcf8ea Refactor fitz ImportError trap 2018-03-27 21:38:02 -07:00
James R. Barlow 112e8d6c18 Fix regression: PDF/A broken without fitz 2018-03-27 21:33:10 -07:00
James R. Barlow 1d8d49a01d Add PyMuPDF to preamble 2018-03-27 21:32:38 -07:00
James R. Barlow 5050155685 Add warning for large file size increases 2018-03-27 15:49:16 -07:00
James R. Barlow a9bd494cc0 Merge branch 'optional-fitz' 2018-03-27 13:36:33 -07:00
James R. Barlow 6a4df78bc0 Add _naive_find_text to search for text when fitz is not available 2018-03-27 13:36:17 -07:00
James R. Barlow 530eae3898 Fix test_main missing file_claims_pdfa 2018-03-26 15:33:53 -07:00
James R. Barlow 3e444f6a90 Make fitz optional 2018-03-26 13:22:09 -07:00
James R. Barlow 45dbff6401 Fix table of contents not preserved in PDF/A 2018-03-26 02:23:19 -07:00
James R. Barlow bc56b8e058 Move metadata tests to new test_metadata 2018-03-26 01:49:25 -07:00
James R. Barlow d86e315c48 v6.0.1 start release notes 2018-03-26 01:44:01 -07:00
James R. Barlow 746969207a Remove deprecated --pdf-renderer tess4, which was renamed to sandwich
Should have been cut in v6.0.0
2018-03-26 01:17:22 -07:00
James R. Barlow 1caebaefb5 tesseract: Fix FileExistsError on if output file was created at timeout 2018-03-25 21:38:20 -07:00
James R. Barlow 2d10fdcf0f Fix typo in release notes 2018-03-25 21:37:06 -07:00
James R. Barlow 355ec70a80 Note other web frontends 2018-03-25 21:36:39 -07:00
James R. Barlow a2f499de01 Remove pageinfo.py which release notes said was gone for v6 2018-03-25 12:16:56 -07:00
James R. Barlow f4bca89722 Remove Tesseract 4 message 2018-03-25 12:16:31 -07:00
James R. Barlow 9fbc69df3f v6.0.0 release 2018-03-25 01:34:26 -07:00
James R. Barlow 230d301268 conftest: py3.5 path issue 2018-03-25 00:52:45 -07:00
James R. Barlow 1ce7b02d94 Travis: don't cache tests/cache anymore, you get it with git 2018-03-25 00:52:19 -07:00
James R. Barlow a2d00f5f1d tess cache: fix tess3 error for -psm instead of --psm 2018-03-25 00:43:02 -07:00
James R. Barlow f68eaa3b46 Fix PyMuPDF version for Travis 2018-03-25 00:36:26 -07:00
James R. Barlow 0199ab220e Tweak Manifest and .travis once more
Travis "do_not_include" moving around no longer needed, thankfully.
Manifest needed LICENSE.
2018-03-25 00:19:45 -07:00
James R. Barlow 656045610a Update release notes 2018-03-25 00:17:23 -07:00
James R. Barlow 8c1c61f207 test cache: fix Path + str error 2018-03-25 00:02:03 -07:00
James R. Barlow af085b79dd Move ocrmypdf to src/ocrmypdf 2018-03-24 23:59:08 -07:00
James R. Barlow 77476965ae test cache: use .bin extension, fix .gitignore .gitattributes 2018-03-24 23:54:16 -07:00
James R. Barlow 961c1365f9 Update manifest.in 2018-03-24 23:50:58 -07:00
James R. Barlow ca51514046 Add test cache 2018-03-24 23:50:41 -07:00
James R. Barlow 8975b72a01 Fix test_testonly_pdf generating an output file in pwd 2018-03-24 22:34:35 -07:00
James R. Barlow 874ec6a87f Add missing fixture to test_unpaper 2018-03-24 22:24:14 -07:00
James R. Barlow 909eaeeead spoof: Allow tesseract cache to share cache
Previous incarnation was only suitable for generating a local cache
where the suite was executed repeatedly. Now the cache ignores
differences, so it can be checked into Github and shared.
2018-03-24 22:17:36 -07:00
James R. Barlow c138161fae Tests: more cleanup 2018-03-24 15:35:57 -07:00
James R. Barlow e48590d66c Refactor out unpaper-specific tests 2018-03-24 15:21:44 -07:00
James R. Barlow 5b1c8541fc Review some skipped tests to make sure reasons still valid 2018-03-24 15:13:23 -07:00
James R. Barlow e5e011021b Remove the OCRMYPDF_program environment variables
Really, this was just replicating the functionality of the PATH
environment variable, and users probably do that anyway.
2018-03-24 15:09:08 -07:00
James R. Barlow 11d74dea09 Remove the OCRMYPDF_program environment variables
Really, this was just replicating the functionality of the PATH
environment variable, and users probably do that anyway.
2018-03-24 15:07:02 -07:00
James R. Barlow cbdf9c88c5 Update requirements 2018-03-24 14:03:34 -07:00
James R. Barlow 46601b1350 setup: skip 1.12.4.1 since it does not provide wheels 2018-03-24 02:59:58 -07:00
James R. Barlow 6f1a40b2ca v6.0.0 notes, build machinery changes 2018-03-24 02:52:56 -07:00
James R. Barlow a2b1f54eb2 Update documentation license info 2018-03-24 02:33:24 -07:00
James R. Barlow 6756016572 Add license notice to all files
Source files to GPL3

Exceptions:
-tests/spoof/* to MIT
-hocrtransform.py
-_unicodefun.py

Test resources to CC BY-SA 4.0 except when otherwise noted.

Add GPL license.
2018-03-24 02:33:24 -07:00
James R. Barlow f42123afc3 pipeline: make removal of merge_qpdf more explicit 2018-03-24 02:30:05 -07:00
James R. Barlow 1425ffd274 pipeline: Merge branch 'feature/mumerge' into test
Replaces qpdf page merging
2018-03-24 02:26:01 -07:00
James R. Barlow d700154e0e Fix regressions after --skip-text improvements 2018-03-24 02:24:45 -07:00
James R. Barlow efecf42566 Add PyMuPDF and use to detect text on pages 2018-03-24 02:16:53 -07:00
James R. Barlow 74bdfc07fb mumerge: fix regressions 2018-03-24 01:18:22 -07:00
James R. Barlow 376dfdba1c Fix text/image files not closed in combine_layers 2018-03-23 13:48:37 -07:00
James R. Barlow 3795d6720f Try out pymupdf merging
With garbage collection it reduces waste on the worst case file.
That's nice. 1 MB -> 105 MB -> 1.5 MB.

Indicates really problem is using PyPDF2 to watermark.

Currently hacked into --output-type pdf.
2018-03-23 13:39:16 -07:00
James R. Barlow 537aaf56d7 Remove duplication between page merge functions 2018-03-23 13:38:53 -07:00
James R. Barlow 34d51b5d3d Merge branch 'feature/faster-split' 2018-03-23 13:10:53 -07:00
James R. Barlow dea8fcfb5b Optimize page splitting by multiprocessing
Previously page splitting occurred in a single process because it was
not believed to affect performance much. It turned out to be an expensive
operation.

It now scales better with large page sizes although this has a negative
effect on small files.

Overall time changes as follows:

7 page file, 9.02s -> 9.56s
731 page file, 213s -> 97s

WITH --tesseract-timeout 0 --output-type pdf --skip-text

i.e. you don't get a 2.2x speed gain when OCR is available.

Squashed a commit to ix test suite failure on --rotate-pages
Squashed a commit to remove debug code
2018-03-23 13:07:51 -07:00
James R. Barlow 4f1f3b9b51 Move available_cpu_count to helpers 2018-03-23 13:07:51 -07:00
James R. Barlow dfeb8812ad Document some instances of 0 vs 1-based page numbering, import cleanup 2018-03-23 13:07:31 -07:00
James R. Barlow 63e2b4273a Travis: avoid using set -e since it interferes with Travis
https://github.com/travis-ci/docs-travis-ci-com/issues/1672
2018-03-23 12:49:12 -07:00
James R. Barlow 5790dbc085 Merge commit '9e2105e08d5fc765dbf636d108809bb66ab562a5' 2018-03-20 18:15:57 -07:00
jbarlow83andGitHub 9e2105e08d Update readme shields
Drop Docker Hub for now, add homebrew
2018-03-20 17:16:28 -07:00
James R. Barlow 22582bbd1c Travis: don't trigger Docker Hub anymore
Docker Cloud is set up to build on pushes to master and tagged releases.
Hopefully that will work out.
2018-03-19 21:09:31 -07:00
James R. Barlow e5f27b7a12 Solve text detection issue with PyMuPDF 2018-03-15 22:29:56 -07:00
James R. Barlow e88ec9822b Tweak release notes 2018-03-15 17:09:43 -07:00
James R. Barlow 5ffd2f5c96 Not ending Py3.5 support just yet 2018-03-15 17:06:11 -07:00
James R. Barlow 11fdb4c5d8 Update release notes for v5.7.0 2018-03-15 17:06:04 -07:00
James R. Barlow 319aff6d09 Merge better-hocr 2018-03-15 16:59:59 -07:00
James R. Barlow a614fa3400 hocr: simplify some math expressions and add comments 2018-03-14 17:05:40 -07:00
endolithandjbarlow83 8d691391ac Fix typos in advanced.rst (#228) 2018-03-14 15:54:55 -04:00
James R. Barlow 0089a84c94 hocr: Make interword spaces default and non-optional for hocr
Update documentation to match.
2018-03-13 14:51:47 -07:00
James R. Barlow 90676e1c6a hocr: Remove baseline dashes 2018-03-13 14:45:31 -07:00
James R. Barlow 062901be43 Some cleanup and variable renaming 2018-03-13 14:29:22 -07:00
James R. Barlow b195d79b50 Refactoring 2018-03-13 11:04:34 -07:00
James R. Barlow 6d7ee98721 Force Tesseract 4 to be single threaded
Gives better performance (throughput basis) than the existing solution
and scales better on powerful boxes.
2018-03-13 08:54:52 -07:00
James R. Barlow fc0800ed5d v5.6.3 notes 2018-03-12 03:41:12 -07:00
James R. Barlow f4e3a0e5b2 v5.6.2 notes 2018-03-09 15:37:08 -08:00
James R. Barlow d631c80024 Suppress debug message when merging large files 2018-03-09 11:10:45 -08:00
James R. Barlow f1f0033875 Suppress spurious debug message in --output-type pdf 2018-03-09 11:07:28 -08:00
James R. Barlow 84d120e850 v5.6.1 notes 2018-03-09 08:00:42 -08:00
James R. Barlow 8159cc6b88 Skip one test that fails for qpdf 8.0.[0,1], due to qpdf regression 2018-03-09 07:57:22 -08:00
James R. Barlow 995f8c106b hocr: account for baseline offset to position text more accurately 2018-03-09 07:45:41 -08:00
Jim Barlow 7cc104b138 hocr: account for skewed baseline 2018-03-05 11:16:40 -05:00
James R. Barlow b3a7299a62 hocr: refactor/improve PEP8 a bit 2018-03-05 10:47:36 -05:00
James R. Barlow 0e7a4deaec hocr: add baseline function, hocr doc link 2018-03-05 10:47:36 -05:00
James R. Barlow b4d66650bd hocr: adjust text cursor with relative moves 2018-03-05 10:47:36 -05:00
James R. Barlow 4986afca28 hocr: Refactor use of text object
We don't need to declare the font on each word.

No improvement for removing trailing space or adding \n
2018-03-05 10:47:36 -05:00
James R. Barlow 2b6004a82b hocr: Make words on line use the line height
Seems to improve the behavior and appearance
of selected text a fair bit.
2018-03-05 10:47:36 -05:00
James R. Barlow 04c54a7c31 Suppress spurious debug message in --output-type pdf 2018-03-05 10:47:36 -05:00
James R. Barlow 7ae6c5ae87 Trial merge interword-spaces 2018-03-02 23:47:06 -08:00
James R. Barlow 4046766ca5 Fix Python 3.5 test suite failure on symlinks
Did not account for API difference in pathlib
2018-03-02 16:57:46 -08:00
James R. Barlow 810390df0b pipeline: Refactor duplicate with clause 2018-03-02 16:57:15 -08:00
James R. Barlow de749bc7ae Fix regression - output to stdout broken 2018-03-02 15:35:51 -08:00
James R. Barlow 9965b8800c Move Dockerfiles out of the way 2018-03-02 15:26:41 -08:00
James R. Barlow ab870fddd6 Dockerignore: glob supported now 2018-03-02 15:24:42 -08:00
James R. Barlow a79d6807cf Dockerfiles: remove deprecated MAINTAINER tag 2018-03-02 15:24:28 -08:00
James R. Barlow 8185fb7e43 Migrate Travis CI setup to Brewfile 2018-03-02 15:16:05 -08:00
Tucker Barbourandjbarlow83 4b10929b25 Fix Homebrew python package (#227)
Homebrew removed python3 and python now defaults to version 3. Here we
use `brew upgrade python` to upgrade the pre-installed version of
python to python3.
2018-03-02 14:59:23 -08:00
Tucker Barbour f6c70312c9 Fix Homebrew python package
Homebrew removed python3 and python now defaults to version 3. Here we
use `brew upgrade python` to upgrade the pre-installed version of
python to python3.
2018-03-02 14:26:13 +00:00
Tucker Barbour 9fd9c7a51f Scale BoundingBox and Text elements to account for additional space.
Here we are manually scaling the pt width used for the BoundingBox and
the Text element when manually adding whitespace to account for
limitations of the PDF.js viewer. This fixes an initial regression
noticed when selecting text elements in Chrome and PDFium. The width
of the Text element and BoundBox had not been adjusted for the
additional whitespace so the highlighting was offset slightly.
2018-03-02 11:18:47 +00:00
Charles Forcey 422e619978 Add a note to the documentation about interword-spaces 2018-03-01 13:15:03 -05:00
Tucker BarbourandCharles Forcey e6e34251c6 Add option to explicitly add interword spaces to HOCR pdf-renderer
This commit includes an optional work around for limitations of the
PDF.js viewer described in
https://github.com/jbarlow83/OCRmyPDF/issues/133. Here is explicitly
add an addition space to text elements before drawing them on the PDF
canvas when using the HOCR renderer. This option does not apply to
other pdf renderers in OCRmyPDF and is turned off by default.
2018-03-01 13:15:03 -05:00
James R. Barlow 2d8aad1086 Improve docs 2018-03-01 00:24:38 -08:00
James R. Barlow 74ca736333 Issue #223: improve text of encrypted PDF error message 2018-02-27 15:08:22 -08:00
James R. Barlow 5e4fd8b0b9 compile_leptonica should have shebang 2018-02-24 12:49:36 -08:00
James R. Barlow 8ab8132411 lint: unused variables, wildcard imports 2018-02-24 12:48:52 -08:00
James R. Barlow 6899dd46e4 lint: remove extraneous backslash 2018-02-24 12:42:37 -08:00
James R. Barlow 8ad0697a20 lint: remove duplicate property definition 2018-02-24 12:42:03 -08:00
James R. Barlow b47e5672e6 Remove old test case that no longer works 2018-02-24 12:40:14 -08:00
James R. Barlow 45c7bd9a60 lint: Remove shebangs from non-executable files 2018-02-24 12:38:58 -08:00
James R. Barlow e7bcb95635 Fix pylint errors 2018-02-24 11:59:01 -08:00
James R. Barlow 3de83627a9 Handle output to /dev/null or directory (#219)
Previously we threw an exception if the output name was a directory (only after doing OCR) and would trigger a PermissionError on trying to flip permission bits of /dev/null due to shutil.copyfile implementation. Instead of copying file use shutil.copyfileobj which should also respect umask etc.
2018-02-19 22:15:07 -08:00
James R. Barlow f248576994 Change instructions to point away from private tap 2018-02-19 17:33:58 -08:00
James R. Barlow aac5b6de3b Update autobrew script to match final changes 2018-02-17 00:12:03 -08:00
James R. Barlow 24435f11e0 We are now in homebrew 2018-02-15 17:42:16 -08:00
James R. Barlow a9da839c39 Add vector-only PDF test case 2018-02-08 00:17:35 -08:00
James R. Barlow fa2c0296d6 v5.6.0 release notes, docs 2018-02-07 16:48:04 -08:00
James R. Barlow 1dfc32d7e6 Preserve "text as curves" vector content
Never updated the checking logic to deal with a pure vector file with no text that needs an OCR layer. This is doable, so allow it.
2018-02-07 16:05:48 -08:00
James R. Barlow 4a61beae41 autobrew: use homebrew's built-in test fixture 2018-02-05 11:10:33 -08:00
James R. Barlow bd30587bf1 Update depends_on order 2018-01-29 13:16:07 -08:00
James R. Barlow e0070e3e18 Update Dockerfile to use Ubuntu 17.10 (issue #214) 2018-01-28 15:30:41 -08:00
James R. Barlow 019513696b Ghostscript spoof scripts did not report their --version correctly 2018-01-10 17:08:14 -08:00
James R. Barlow ad7a4476db hugemono.pdf needs --max-image-mpixels to pass with Pillow 5.0 2018-01-10 16:55:18 -08:00
James R. Barlow 4812b20fb2 Fix tesseract_noop.py generating wrong size of output PDF in tests
This caused trouble before with test_deskew
2018-01-10 16:35:31 -08:00
James R. Barlow f5e07c9427 Fix Ghostscript parameter order 2018-01-10 16:33:26 -08:00
James R. Barlow 75dcb90621 Niceties: when environment variable overload is used clarify we're not checking the PATH 2018-01-10 16:33:03 -08:00
James R. Barlow dfc0434cc2 Update requirements to set Pillow to 5.0 2018-01-10 15:45:23 -08:00
James R. Barlow 882fc2257c Add --max-image-mpixels argument to support Pillow 5.0 2018-01-10 15:43:59 -08:00
James R. Barlow 41e83b52fc Document process for redoing OCR 2018-01-10 15:39:58 -08:00
James R. Barlow 47758b4d8f Reactivate autobrew 2018-01-10 15:39:36 -08:00
James R. Barlow 6bf1f970a0 Fix some parameter validation for --output-type pdfa-1 and pdfa-2 2018-01-10 11:50:08 -08:00
James R. Barlow 7d451f101f Detect old versions of Ghostscript and warn about them (#208) 2018-01-10 11:47:39 -08:00
James R. Barlow 7edbfe0e40 Update autobrew template 2018-01-09 12:36:14 -08:00
James R. Barlow 91b42cbfa8 Fix issue in sandwich renderer when skipping OCR on a rotated and deskewed page
If OCR is skipped due to --tesseract-timeout or similar, and the skip page is rotated with /Rotate, and the skip page was deskewed or had other image processing, then the skip page was created with the wrong dimensions causing the output page to be cropped.
2018-01-09 00:17:53 -08:00
James R. Barlow 6907df41b4 Disable autobrew until homebrew accepts the official release 2018-01-08 23:26:58 -08:00
James R. Barlow 2cebd90cbd Fix brew audit --strict warnings 2017-12-09 12:14:28 -08:00
James R. Barlow 376a121aaa Re-enable macos 2017-11-29 15:06:19 -08:00
James R. Barlow da11fd17ee qpdf dummy: needs to return version now 2017-11-29 14:35:37 -08:00
James R. Barlow a40689a0ff tesseract: handle return of bytes properly in error cases 2017-11-29 14:35:26 -08:00
James R. Barlow 44a45fc3fb Add "bad UTF8 output from Tesseract" test 2017-11-29 14:08:07 -08:00
James R. Barlow ec4bb5359a Read tesseract's output as binary to avoid UnicodeDecodeErrors if it messes up 2017-11-29 13:44:40 -08:00
James R. Barlow d2217632df Rename _verify_python3_env 2017-11-29 13:43:18 -08:00
James R. Barlow 2cc044feed Move qpdf complaint to after options checking so that it won't break ocrmypdf --version 2017-11-29 13:42:55 -08:00
James R. Barlow 64fd0cb54f Remove test_qpdf.py only from travis 2017-11-29 12:54:48 -08:00
James R. Barlow c5a1d22e81 That fixed it. Complain about old versions of qpdf now 2017-11-29 12:53:34 -08:00
James R. Barlow a7b307af04 Looks like issue was negzero.pdf with qpdf 5.1.1 on travis, which is why osx passes
Reorganize and see if this is better now
2017-11-29 12:47:09 -08:00
James R. Barlow 3269eba16c Is it negzero.pdf? 2017-11-29 12:02:37 -08:00
James R. Barlow d472860e3b Try to diagnose travis-only failure of qpdf test 2017-11-29 11:41:06 -08:00
James R. Barlow 731c9ea55e Set timeouts on the tests that seem to be stalling on travis (but not elsewhere) 2017-11-27 14:46:10 -08:00
James R. Barlow 0b6af8d965 Clarify hocrtransform license/copyright 2017-11-27 13:41:46 -08:00
James R. Barlow 92ca9e954c Fix test warning/failures, hopefully 2017-11-27 13:41:32 -08:00
James R. Barlow 67aa00e2c5 Remove py3.5 deprecation
It'll be in Debian stable for a long time, so maybe not such a good idea
2017-11-27 01:10:34 -08:00
James R. Barlow a1bf161eb7 Update v5.4.4 release notes 2017-11-27 01:09:35 -08:00
James R. Barlow 56614fcaa4 Add support and tests for handling page count > ulimit - fixes issue #181 2017-11-27 00:32:35 -08:00
James R. Barlow 4d9169e15f Add merge ulimit test case 2017-11-26 23:34:36 -08:00
James R. Barlow 3a167af2c4 Nearly smallest possible PDF-1.3 with all required fields 2017-11-26 23:32:21 -08:00
James R. Barlow 64b9005030 v5.4.4 release notes; mention deprecation of 3.5 2017-11-26 23:08:55 -08:00
James R. Barlow 965de3a235 Test case for issue #200 2017-11-26 22:52:53 -08:00
James R. Barlow 2040ae4856 Fix issue #200, uncommon but valid decimal syntax treated as error
Also replace check_output() calls with run() in qpdf.py
2017-11-26 22:52:43 -08:00
James R. Barlow 31a1f9f5ea pytest-runner should be a setup requirement 2017-11-26 22:51:51 -08:00
James R. Barlow 7eadd82048 Update docker script based on uid issue
issue #180, fix reported by @Atredis76
2017-11-26 15:11:17 -08:00
James R. Barlow 31d0eaac8e Ensure intermediate metadata holder PDF has same version as its input file
While not known to cause problems, the absence of this could confuse parsers
2017-11-24 00:09:13 -08:00
James R. Barlow f45d012635 Update docs: security, pip install --user 2017-11-21 16:50:14 -08:00
James R. Barlow 5a777ee9bc v5.4.3 release notes 2017-11-17 02:28:02 -08:00
James R. Barlow 7bbf6bc7f4 Travis didn't like LANG, use LC_ALL 2017-11-16 20:37:30 -08:00
James R. Barlow 9bfc45702d Update copyright in license 2017-11-16 17:18:29 -08:00
James R. Barlow 40aa82ab41 Check that the locale is sane before allowing OCR to proceed 2017-11-16 17:18:02 -08:00
James R. Barlow 5d2c67c62b Declare in yet another place that this shouldn't run on py2 2017-11-16 14:54:28 -08:00
James R. Barlow a00ef4836d autobrew: declare conflict with mactex
As reported in https://github.com/jbarlow83/homebrew-ocrmypdf/issues/4#issuecomment-345029899
2017-11-16 14:53:42 -08:00
James R. Barlow 3ef766bb93 Remove bare 'except:' 2017-11-01 01:52:13 -07:00
James R. Barlow 44b5a18462 Declare our __version__ properly 2017-11-01 01:49:51 -07:00
James R. Barlow 4df716f0e8 Update requirements.txt 2017-11-01 01:49:35 -07:00
James R. Barlow fcbf34a4d3 Refactor obtaining version from subprocesses
Issue #196 raised the need to deal with linker warnings on --version.
2017-11-01 01:44:36 -07:00
James R. Barlow c7b8b6e18b Fix issue #194 - --sidecar creates blank txt file 2017-10-26 18:15:31 -07:00
James R. Barlow 9fd165bf4b Add 'zlib' to travis recipe 2017-10-26 18:06:12 -07:00
James R. Barlow 2061092ec5 Describe how to stream with Docker
[ci skip]
2017-10-18 12:37:27 -07:00
James R. Barlow 57466a31ba v5.4.1 release notes 2017-10-12 14:04:45 -07:00
James R. Barlow 4213c69bef Try to make travis actually deploy macOS 2017-10-12 12:45:40 -07:00
James R. Barlow cc5578488a Remove workaround for OSD crash and use explicit -l osd
Suggested by amitdo in https://github.com/tesseract-ocr/tesseract/issues/1167
2017-10-12 12:45:18 -07:00
James R. Barlow 4b7135f0e5 Add option to produce PDF/A-1B 2017-10-11 14:32:58 -07:00
James R. Barlow 984df023b1 Fixup manifest 2017-10-11 13:12:14 -07:00
James R. Barlow 6f8682f558 Track .coveragerc for future reference 2017-10-11 13:05:19 -07:00
James R. Barlow 51defa6d66 Import cleanup and some pylint fixes 2017-10-11 12:58:35 -07:00
James R. Barlow 7d73098d6e ghostscript.py: fix missing imports 2017-10-11 12:39:16 -07:00
James R. Barlow c580aa4683 Revert to pre-build stages travis.yml
It's still in beta for a reason apparently
2017-10-11 11:32:47 -07:00
James R. Barlow 47d1199360 Fix travis.yml typo 2017-10-11 11:26:08 -07:00
James R. Barlow 61ab435b31 Try disabling the matrix and osx for build stages 2017-10-11 11:18:32 -07:00
James R. Barlow aa8f534b45 pipeline: fix variable not defined in __init__ 2017-10-11 11:17:33 -07:00
James R. Barlow dc9589aca8 travis: Explicitly set python versions 2017-10-11 00:37:13 -07:00
James R. Barlow dd6eaacc6b More travis build tweaks 2017-10-11 00:28:33 -07:00
James R. Barlow 37dc03eec6 Add workaround for tess4 form feed behavior change 2017-10-10 23:33:54 -07:00
James R. Barlow 6b478172f6 Try using plain tessdata instead of tessdata_best 2017-10-10 20:52:03 -07:00
James R. Barlow c6e73bcfd6 Still failing, did the cp work at all? 2017-10-10 15:15:26 -07:00
James R. Barlow a2d62938ce cp overwrite needs sudo 2017-10-10 14:54:52 -07:00
James R. Barlow d7ae1f3cca travis: Replacement problematic traineddate file 2017-10-10 14:39:05 -07:00
James R. Barlow 70219581c4 Disable tesseract 4 so tests can succeed
tess4 -psm 0 is broken right now
2017-10-10 14:18:02 -07:00
James R. Barlow f70ac9fb89 Workaround travis issues in build stages... maybe 2017-10-10 12:36:50 -07:00
James R. Barlow 235b9fbaf0 Resolve merge conflicts 2017-10-10 12:24:10 -07:00
James R. Barlow ebda7f42db travis: need script for each stage 2017-10-10 12:22:23 -07:00
James R. Barlow 0b04e4b977 Try out travis build matrix 2017-10-10 12:14:50 -07:00
James R. Barlow 9498601a37 Add docs on adding to docker iamge 2017-10-10 12:13:20 -07:00
James R. Barlow ef5d320e06 Ignore .vscode too 2017-10-09 16:19:41 -07:00
James R. Barlow b00c9a562d Remove meaningless version from Dockerfile.polyglot 2017-10-09 16:18:02 -07:00
James R. Barlow 5372656893 Don't say tess4 support is experimental - it's pretty good now 2017-10-09 16:17:42 -07:00
James R. Barlow 571de0e368 Update release notes 2017-10-08 12:41:03 -07:00
James R. Barlow 82cea2fd85 Update batch processing docs to include Synology script 2017-10-08 12:34:36 -07:00
James R. Barlow aed9814345 Use Ubuntu 17.04 instead of 16.10 for Docker image (issue #191)
Due to 16.10 PPAs no longer being generated by alex-p
2017-10-08 12:13:20 -07:00
James R. Barlow 34fc1f5fd7 Add reminder that blank.pdf is not trivial 2017-09-13 01:19:18 -07:00
James R. Barlow 87c2ed8b27 Improve clarity of --pdf-renderer=tesseract deprecation warning 2017-09-12 14:34:53 -07:00
James R. Barlow 1467d118ab Add more leptonica functions 2017-09-06 00:27:02 -07:00
James R. Barlow 922dbe83c3 Update MANIFEST rules 2017-09-02 20:05:57 -07:00
James R. Barlow 6af7d61ee5 Fix CI failure due to spoofers not being updated to Tesseract 3.05 strings 2017-09-01 16:17:26 -07:00
James R. Barlow bafd08391d Update release notes 2017-09-01 12:50:45 -07:00
James R. Barlow 82ebd8ef1a Fix missing error message about trying to use sandwich on old tesseract 2017-09-01 12:50:36 -07:00
James R. Barlow 4ed1aa4d23 Release notes: fix indentation 2017-09-01 12:47:22 -07:00
James R. Barlow d04e43d46d Update copyright info for test files
[ci skip]
2017-09-01 01:00:32 -07:00
James R. Barlow 952f0cca15 Dockerfiles: set LANG=C.UTF-8
Issue #184 to avoid issue with printing UTF-8 text to sidecar
2017-08-30 13:25:54 -07:00
James R. Barlow f6a4d8f1f8 Fix Ubuntu 14.04 install instructions to account for dropping Py3.4 support
[ci skip]
2017-08-27 13:53:36 -07:00
James R. Barlow b3097a2384 Fix broken test case related to language packs 2017-08-24 13:01:02 -07:00
James R. Barlow 6d9ddbe98b v5.3.1 notes 2017-08-24 01:09:19 -07:00
James R. Barlow 9bb42c0229 Wrong error type used for missing language 2017-08-24 01:07:23 -07:00
James R. Barlow bd7226b27a Merge branch 'master' of github.com:jbarlow83/OCRmyPDF 2017-08-23 23:30:19 -07:00
James R. Barlow 5b413e3873 Cookbook: add "don't OCR" examples 2017-08-23 23:29:41 -07:00
James R. Barlow be5831a629 Offer the readme as a long description for new PyPI 2017-08-23 23:29:21 -07:00
jbarlow83andGitHub 084d2bf8e2 More badges 2017-08-23 23:19:29 -07:00
James R. Barlow da79e6bac7 macos: Skip brew audit because it seems to crash ruby on travis 2017-07-27 16:00:41 -07:00
James R. Barlow c4831ac00c v5.3 release notes 2017-07-27 00:11:12 -07:00
James R. Barlow 93a954ef9f Fix missing import for Py3.5 2017-07-26 23:40:01 -07:00
James R. Barlow f7ce8f44e9 Weaken the --user-words test so it will pass on Travis 2017-07-26 21:03:51 -07:00
James R. Barlow 0b012697e5 Whitelist the Latin-1 languages that work with HOCR
Omitted French because the rare 'oe' and 'ÿ' glyphs are not in Latin-1.
Basically steer people away from HOCR renderer but avoid a potential
disruptive behavior change.
2017-07-26 21:03:18 -07:00
James R. Barlow 58e357c992 Report location of attempted output_file that fails to write 2017-07-22 17:49:56 -07:00
James R. Barlow 71fbad83ad Fix py3.5 test 2017-07-21 17:01:06 -07:00
James R. Barlow 52483072dc Add a differential test that checks tesseract uses supplied word list 2017-07-21 16:40:20 -07:00
James R. Barlow 7f0b8621f3 Tests: accept rich path objects without having to str() everything 2017-07-21 16:39:22 -07:00
James R. Barlow cd8db60b06 Crash test all renderers, not just two 2017-07-21 14:10:02 -07:00
James R. Barlow 1aa34f5d2e Make some interfaces accepting of both str-paths and Path objects 2017-07-21 13:28:30 -07:00
James R. Barlow dfa1d88ce9 Fix missing user_words/user_patterns from textonly_pdf case 2017-07-20 17:14:04 -07:00
James R. Barlow dd38519f07 Merge branch 'feature/user-words' into develop
# Conflicts:
#	ocrmypdf/exec/tesseract.py
2017-07-20 16:25:20 -07:00
James R. Barlow 098f5d4f0b docs: remove deprecated example of pdftotext 2017-07-20 16:20:17 -07:00
James R. Barlow ffc685d536 docs: envvar markup 2017-07-20 16:19:57 -07:00
James R. Barlow cd1a99a0de Refactor int(os.path.basename(s)[0:6]) -> page_number(s) 2017-06-26 13:29:40 -07:00
James R. Barlow 48e3b267fc Accept PDFs with whitespace ahead of %PDF marker
Noticed in @aagahi 's fork
2017-06-26 13:17:47 -07:00
James R. Barlow 3a7c3417bb Don’t check tags and branch at the same time as Travis doesn’t get this
Travis is weird
2017-06-13 13:14:34 -07:00
James R. Barlow d792ef7222 Give the ‘auto’ renderer setting more test covfefe 2017-06-13 13:13:58 -07:00
James R. Barlow 2c24f67deb Rename “tess4” renderer to “sandwich” and make it default in Tess 3.05.01
Tesseract 3.05.01 backported the textonly_pdf=1 which allows the use
of this superior PDF renderer prior to 4.00 alpha. This means that
the tess4 name is no longer accurate, so call it a sandwich because of
its merge-preserve characteristic. Preserve the tess4 name. Fix the
documentation and tests to reflect this.

Make it the default, because it’s better. It does not have the issues
the “tesseract” renderer does prior to Tess 3.05.00 with rendering
PDFs that Ghostscript corrupts, and it produces better output without
re-rastering.

Deprecate some old stuff to avoid the test suite growing obscenely
large.
2017-06-13 13:09:12 -07:00
James R. Barlow 9e75e28d0c Homebrew needs x11 to compile Pillow 2017-06-13 11:03:26 -07:00
James R. Barlow 3232643809 Support “textonly PDF” renderer in Tesseract 3.05.01 2017-06-13 10:18:08 -07:00
James R. Barlow f7ee9e90ce Document what is meant by the ocrmypdf “API” 2017-06-13 10:15:11 -07:00
James R. Barlow 47298be132 Remove Python <3.5 test 2017-06-13 10:14:28 -07:00
James R. Barlow a88fa83515 Travis: fix deploy conditions for homebrew autobrew 2017-05-31 02:29:32 -07:00
James R. Barlow 12bfe20385 v5.1 release notes 2017-05-29 14:36:50 -07:00
James R. Barlow 3d2f6f0772 Fix tess4 test using old-style pageinfo API 2017-05-29 13:51:21 -07:00
James R. Barlow 1cb607f64b Merge UserUnit 2017-05-29 13:22:55 -07:00
James R. Barlow d3c54fbbde For —rotate-pages, rasterize preview at half DPI instead of 200 DPI
Ensures that time is not wasted on previews at higher resolution than
the input as was sometimes the case
2017-05-29 13:01:18 -07:00
James R. Barlow 28341b755f Refactor common test fixtures 2017-05-29 12:47:55 -07:00
James R. Barlow 4b5cd420e1 Add new test file 2017-05-29 12:16:08 -07:00
James R. Barlow 1d57bcc99e Fix Ghostscript rasterizing of UserUnit pages and related sizing issues 2017-05-29 12:14:10 -07:00
James R. Barlow facdd13879 Ghostscript: refactor image output resizing 2017-05-29 11:42:27 -07:00
James R. Barlow 6e891f91d3 ghostscript, qpdf: Restore API backward compatibility 2017-05-29 11:13:06 -07:00
James R. Barlow 9b50ede977 Partially solve ghostscript rasterize_pdf producing wrong file size
Kludge. Assumes JPEG for now. Messy.
2017-05-25 01:17:43 -07:00
James R. Barlow 82cf010333 Error out if trying to produce PDF/A >200” due to Ghostscript limitation 2017-05-25 00:07:29 -07:00
James R. Barlow 6ff6c8614f —output-type=pdf now outputs /UserUnit PDFs at the correct size
This currently distorts the output size because Tesseract assumes it
 knows the DPI better than we do.

Does not work for Ghostscript, because it emerges that Ghostscript
honors /UserUnit for rasterizing but not in pdfwrite (resolve/wontfix).

https://bugs.ghostscript.com/show_bug.cgi?id=690781

Ghostscript’s output would need to be patched in a PDF/A safe way for
this to work. Temporary route may be to block Ghostscript if
/UserUnit.
2017-05-24 23:26:07 -07:00
James R. Barlow eb1cd38f6c Add an open helper that is compatible with pathlib 2017-05-24 16:19:15 -07:00
James R. Barlow 148b632b4f Prove multiprocessing works, although it is still racy in some places 2017-05-23 16:32:13 -07:00
James R. Barlow 591e213713 Add more dependencies for autobrew 2017-05-23 13:52:28 -07:00
James R. Barlow 75f2262659 Ensure JobContext stuff is actually tested for IPC consistency 2017-05-19 17:57:07 -07:00
James R. Barlow d9005a1074 pdfinfo: replace most remaining dict-style access 2017-05-19 16:17:36 -07:00
James R. Barlow 3e73fa81bf pageinfo: deprecation warning 2017-05-19 16:17:07 -07:00
James R. Barlow ba6e290231 Restore old pageinfo.py to avoid breaking compatibility 2017-05-19 15:49:23 -07:00
James R. Barlow 08e47117a3 Rename pageinfo to pdfinfo 2017-05-19 15:48:23 -07:00
James R. Barlow 532ef38157 /UserUnit is a scalar, not an array 2017-05-19 14:19:50 -07:00
James R. Barlow 4c09875890 docs: upload unpaper Dropbox link, .rst typo blocking macOS install
[ci skip]
2017-05-19 12:18:09 -07:00
James R. Barlow 0e98139712 Upload to upload.pypi.org/legacy as recommend by PyPA
https://github.com/pypa/warehouse/issues/1996#issuecomment-302784126
2017-05-19 12:06:24 -07:00
James R. Barlow 4c04d802d7 Introduce /UserUnit checking 2017-05-19 12:01:19 -07:00
James R. Barlow b3dc404571 Update unpaper.deb link (fixes #171)
*Shakes fist a Dropbox*
2017-05-19 11:28:45 -07:00
James R. Barlow 8694f8d2eb Replace magic strings colorspace and encoding with Enums 2017-05-18 22:32:27 -07:00
James R. Barlow 263f9b79f4 pageinfo: debug stuff 2017-05-18 21:52:55 -07:00
James R. Barlow 56d2aae963 Refactor from ImageInfo index to attribute accessing 2017-05-18 18:39:14 -07:00
James R. Barlow 127706153d Refactor dictionary based image info to ImageInfo 2017-05-18 18:26:31 -07:00
James R. Barlow caee5b1428 Access PageInfo instance variables instead of dictionary 2017-05-18 17:12:04 -07:00
James R. Barlow 6c12e7e944 Refactor pageinfo dictionary to PageInfo() 2017-05-18 16:53:38 -07:00
James R. Barlow cd04ae6949 Refactor PdfInfo(str(filename)) -> PdfInfo(filename) 2017-05-18 16:43:50 -07:00
James R. Barlow 6a0b68298f Refactor pdf_get_all_pageinfo to PdfInfo 2017-05-18 16:31:18 -07:00
James R. Barlow 0a2f732267 docs: Fix restructured text typos 2017-05-16 23:27:10 -07:00
James R. Barlow 4bade99f27 docs: Remark that someone got bash on Windows working 2017-05-16 23:24:34 -07:00
James R. Barlow 0b048cd24e Join the build badge club 2017-05-16 23:24:05 -07:00
James R. Barlow c69ee63d82 Travis, true is a program, not a keyword 2017-05-15 15:12:14 -07:00
James R. Barlow 744fa104d7 v5.0.1 release notes (anticipating) 2017-05-14 23:59:09 -07:00
James R. Barlow e24ff0fd64 Travis: don’t update the homebrew version because we pushed to testpypi 2017-05-14 23:55:40 -07:00
James R. Barlow 5de107d44c tesseract_cache: update explanatory notes 2017-05-14 23:54:09 -07:00
James R. Barlow 131a5b741d tesseract.py: update canned HOCR template to tess 3.05 output
Seems better to not claim the existence of several entities that don’t
exist as the older one does
2017-05-14 23:40:09 -07:00
James R. Barlow 65b89687a9 ghostscript: fix missing “import sys”, only applicable for an exception 2017-05-14 23:38:52 -07:00
James R. Barlow 048ae40e75 Update copyrights 2017-05-14 23:38:28 -07:00
James R. Barlow 234183ecd2 Fix: Tesseract 3.04 is sensitive to order of configuration commands
“txt hocr” is not acceptable and does not produce expected output .txt
while “hocr text” works fine, so switch the order everywhere.

Should fix #169
2017-05-14 23:27:46 -07:00
James R. Barlow fb067dc97b cookbook: more on improving OCR 2017-05-14 23:16:47 -07:00
James R. Barlow a1fea0ce16 docs: link to OCRmyPDF-web 2017-05-14 23:16:30 -07:00
James R. Barlow e1e9135e93 Test suite: tidy up imports 2017-05-14 23:15:29 -07:00
James R. Barlow aff982036b autobrew: fix brew audit error on double blank line 2017-05-14 23:15:02 -07:00
James R. Barlow d087649eab Remove “null deploy script” since “/usr/bin/true” is equivalent 2017-05-12 15:37:02 -07:00
James R. Barlow 7f3fa46a40 v5.0 release notes 2017-05-12 14:14:28 -07:00
James R. Barlow b1f79e4d97 Disable other use redo_ocr 2017-05-12 13:24:30 -07:00
James R. Barlow 115d6df94f Warn user when —image-dpi is supplied but ignored 2017-05-12 12:09:53 -07:00
James R. Barlow 559af9635f —redo-ocr is not implemented, so disable 2017-05-12 12:08:16 -07:00
James R. Barlow cb06359c0b Turn on Tesseract 4 cache in test suite
Travis is too slow without it, and perhaps it’s overly paranoid to
never cache Tess4. Maybe nuke the cache occasionally to be safe…
2017-05-12 11:42:27 -07:00
James R. Barlow 5e26bb29d9 Update requirements files 2017-05-12 11:41:15 -07:00
James R. Barlow b0e95842b8 Fix Travis CI errors while looking around for Tess4 2017-05-12 00:40:00 -07:00
James R. Barlow 08e678f21f rst: Clean up indentation 2017-05-12 00:12:06 -07:00
James R. Barlow c17817810f Update documentation for 3.03 support removal 2017-05-12 00:08:22 -07:00
James R. Barlow ff5c38b1f7 Tell Travis to download Tesseract 4.00 from a PPA for testing 2017-05-11 23:52:13 -07:00
James R. Barlow 64314c1b82 Insist on Python 3.5 wherever we check for it 2017-05-11 23:51:45 -07:00
James R. Barlow 83230097ae Insist on Tesseract 3.04 wherever we check for it 2017-05-11 23:51:28 -07:00
James R. Barlow 8f91acf956 Remove Tesseract 3.02 and 3.03 compatibility shims 2017-05-11 23:50:52 -07:00
James R. Barlow d211722a2f .gitignore the docs Makefile 2017-05-11 23:28:52 -07:00
James R. Barlow 56e6ed1249 Fix missing import; all tests passing! 2017-05-11 23:28:05 -07:00
James R. Barlow 21982cf1cb baiona_gray remove alpha channel 2017-05-11 23:23:37 -07:00
James R. Barlow edc01408da Update the .png files, again, hopefully without corruption 2017-05-11 23:20:50 -07:00
James R. Barlow aee33c87ed Merge release notes 2017-05-11 23:11:12 -07:00
James R. Barlow 0dae1602c7 Fix missing import PIPE 2017-05-11 23:07:20 -07:00
James R. Barlow d926f07ac1 Stop git from corrupting .pngs
Grrr.
2017-05-11 23:07:06 -07:00
James R. Barlow 96045e98f4 Update develop with master changes
We’re well out of the “trivial updates” zone
2017-05-11 22:54:27 -07:00
James R. Barlow 01b7205e2c Ensure skipped pages are explained in sidecars 2017-05-11 00:43:36 -07:00
James R. Barlow c8a4cbcf17 Fix test suite breakage after sidecar feature added
Forgot to update tesseract spoofers to account for change in tesseract
parameters.  Also the change to outputting multiple files in the collate
steps affected how ruffus passes information into downstream consumers
of those files.
2017-05-11 00:17:24 -07:00
James R. Barlow 16b6442b23 Add changes to __main__.py that should have been in last commit 2017-05-10 17:55:42 -07:00
James R. Barlow 183eafa587 Implement sidecar text files (#126) 2017-05-10 15:22:44 -07:00
James R. Barlow 47a2997538 Reorganize —help text 2017-05-10 12:19:56 -07:00
James R. Barlow 37ebcadfa1 Implement —user-words, —user-patterns 2017-05-09 17:54:56 -07:00
James R. Barlow 74d98216f1 Update documentation for Ghostscript behavior 2017-05-09 17:43:39 -07:00
James R. Barlow 4bdebf573e Tell Travis CI to use multiple cores
Let’s see if this helps the build go faster
2017-05-09 17:24:32 -07:00
James R. Barlow 1606b6a383 Add —quiet (fixes #143), stop using ruffus to partially generate argparser 2017-05-09 17:24:06 -07:00
James R. Barlow 2a61902df5 Merge commit 'c4f01de231d22da5cea02c25aa581a965a37640b' 2017-05-09 16:37:55 -07:00
James R. Barlow 01a1c2b576 Implement —pdfa-image-compression to control Ghostscript’s compression
Fixes #163
2017-05-09 16:37:29 -07:00
Ingo Feinererandjbarlow83 c4f01de231 Fix typo "cutput" -> "output" (#164)
[ci skip]
2017-05-09 16:22:10 -07:00
James R. Barlow 63a4a761dd Revert "v4.5.7 release notes"
The change introduced regressions, so find another way to fix.

This reverts commit d077c03686.

[ci skip]
2017-05-08 14:38:04 -07:00
James R. Barlow d077c03686 v4.5.7 release notes 2017-05-06 22:34:54 -07:00
James R. Barlow c97ea1f2a9 Update high DPI test case to confirm the output image is not downsampled 2017-05-06 22:34:01 -07:00
James R. Barlow fd27df2abb Update documentation to warn that transparency is not tested 2017-05-06 22:33:24 -07:00
James R. Barlow bf04f03c4c Fix corrupt test file “typewriter.png”
This file is not currently used in any tests, but could be, so replace
corrupt version with a useful one.
2017-05-06 22:28:34 -07:00
James R. Barlow 93e802f473 Fix issue #163, color and grayscale images JPEG compressed when not needed 2017-05-06 22:27:25 -07:00
James R. Barlow 1464b9087a Try Travis again with null deploy for OSX 2017-05-01 17:37:59 -07:00
James R. Barlow e8cc8fc879 Add travis null_deploy for osx 2017-05-01 17:24:43 -07:00
James R. Barlow fae2119b1e v4.5.6 release notes 2017-05-01 15:55:02 -07:00
James R. Barlow aa859a4139 Fix #156 - NoneType has no ‘getObject’ for pages with no /Contents 2017-05-01 15:46:15 -07:00
James R. Barlow b9b12e2879 Ensure that ocrmypdf stops and reports an error if Ghostscript fails
Past behavior was to continue and let ruffus puke eventually
2017-05-01 15:44:21 -07:00
James R. Barlow cf643c9f43 Fix argparse.ArgumentError needs two positional args 2017-05-01 14:55:47 -07:00
James R. Barlow 5b1a7880a9 Switch to Travis triggering Docker build to skip race condition with PyPI 2017-04-28 17:18:01 -07:00
James R. Barlow 474b6b0500 v4.5.5 release notes 2017-04-28 15:27:41 -07:00
James R. Barlow 6c8c1d8173 Fix #154: KeyError ‘/Contents’ on blank pages with /Contents record 2017-04-28 15:22:05 -07:00
James R. Barlow 6a91fa637f Squash merge improvements to auto-homebrewing macOS version 2017-04-28 15:17:38 -07:00
Jakub Wilkandjbarlow83 2846fb4e31 Remove misplaced flags from re.sub() call (#153)
The 4th argument of re.sub() is maximum number of substitutions,
not flags.

Moreover, re.MULTILINE affects only semantics of ^ and $,
so it wouldn't have any effect on this regular expression.
2017-04-20 00:38:23 -07:00
James R. Barlow a1033cdc64 osx_brew: show output before letting “brew audit” check it 2017-04-18 23:38:07 -07:00
James R. Barlow 204336e1a5 Move release notes into the rest of documentation 2017-04-18 18:26:31 -07:00
James R. Barlow 8954e6c3b9 v4.5.4 Update release notes 2017-04-18 18:09:15 -07:00
James R. Barlow fee22b6b0b Fix #151, cannot write mode P as JPEG
all(<empty generator>) is True.
2017-04-18 18:08:45 -07:00
James R. Barlow 2b82c31b85 Update documentation 2017-04-18 18:07:19 -07:00
James R. Barlow 9a4813089c autobrew: remove homebrew dependency “zlib”, causes audit failure 2017-04-18 15:28:30 -07:00
James R. Barlow 554fcc8b9d Add test case for #152 2017-04-18 15:20:25 -07:00
Tomandjbarlow83 345256ee99 Fix --skip-big when there are no images in pdf (#152)
* fixed skip-big when there are no images in pdf

* added only_text pdf

* updated only_text pdf

* Delete only_text.pdf
2017-04-18 15:19:24 -07:00
James R. Barlow 58d1042147 v4.5.3 release notes update 2017-03-29 23:50:31 -07:00
James R. Barlow 7b7e3a3e03 Enable lossless reconstruction for —pdf-renderer tess4 where appropriate 2017-03-29 23:44:12 -07:00
James R. Barlow 1e7fbd4202 Fix issues with —pdf-renderer tess4 page skipping
If tess4 renderer needed to skip OCR on a page it would end up
duplicating the page contents onto the new page, rather than creating
a blank OCR layer and placing it on the output page. This created
duplicated content in output files.
2017-03-29 23:43:26 -07:00
James R. Barlow 6e907856f2 Refresh requirements 2017-03-29 18:15:29 -07:00
James R. Barlow 8bc6019172 Begin adding new option to redo ocr 2017-03-29 18:15:20 -07:00
James R. Barlow 059f79242e Phase out subprocess.Popen 2017-03-29 18:15:02 -07:00
James R. Barlow 89599b4812 Drop Python 3.4 compatibility 2017-03-29 15:46:53 -07:00
James R. Barlow a9f4047a97 Remove backward compatible API deprecations from v4.x 2017-03-29 15:42:54 -07:00
James R. Barlow 23227ae763 Deprecate old files 2017-03-29 15:41:52 -07:00
James R. Barlow 4a9e9e9db2 v4.5.3 release notes 2017-03-29 13:19:34 -07:00
James R. Barlow 88ef2718f1 Reject high Unicode metadata at command line
Ghostscript 9.21 does not seem to accept Unicode above U+FFFF. Previous
versions did, but it now exits with a rangecheck error (-15).

Reject on the command line for now. Complete fix would also need to
check input PDF’s metadata.
2017-03-28 11:08:38 -07:00
James R. Barlow e71e8ca3ad Workaround for GS VMerror -25 bug
Avoid inserting docinfo keys that would be translated to null strings,
to avoid running afoul of
https://bugs.ghostscript.com/show_bug.cgi?id=697684
2017-03-28 11:05:43 -07:00
James R. Barlow 45e9257d6e Don’t use filename “pdfa_def.ps” for GS file
At recommendation of Artifex people, don’t use the filename pdfa_def.ps
because if given without a full path Ghostscript will prioritize
searching its own resources over the current directory, and will pick up
 the example file pdfa_def.ps instead.  We used the full path anyway,
but this is a prudent defensive change to avoid odd Ghostscript behavior.
2017-03-28 11:02:46 -07:00
James R. Barlow 2954e72652 Some examples of Ghostscript and Tesseract warnings/errors were not tagged properly 2017-03-28 10:59:53 -07:00
James R. Barlow 199de96cff Ghostcript 9.21 seems to have a regression related to Unicode metadata 2017-03-24 15:15:46 -07:00
James R. Barlow 8ddbe81513 Fix issue #147: unpaper loses DPI information, affects —pdf-renderer tess4 2017-03-24 13:23:03 -07:00
James R. Barlow a3e26e0498 Make —pdf-renderer tess4 more informative, less FUD 2017-03-24 12:58:40 -07:00
James R. Barlow 4ad129d8d8 docs: Don't recommend system-site-packages anymore (Ubuntu 16.04)
Not needed since reportlab 3.4 comes with a wheel, and that was the main difficulty.
2017-03-23 08:13:28 -07:00
James R. Barlow dfb9fa0736 autobrew: missing deps 2017-03-16 00:18:07 -07:00
James R. Barlow eb036898e9 Update documentation with macOS homebrew tap
[ci skip]
2017-03-13 22:25:25 -07:00
James R. Barlow 7c6aa76a2a Fix brew audit failure 2017-03-13 17:58:31 -07:00
James R. Barlow f035cb1088 Fixed issue #142 — closed streams raise an exception on fork attempt 2017-03-13 15:52:57 -07:00
James R. Barlow 35162166c5 Fix UnboundLocalError in autobrew.py 2017-03-13 15:51:39 -07:00
James R. Barlow 107f6abcb1 Fix autobrew build issues - missing deps 2017-03-12 17:07:38 -07:00
James R. Barlow 760a939e7d Further autobrew tweaks 2017-03-11 17:05:26 -08:00
James R. Barlow 72660d0dec MacOS skip the one test that needs poppler, to save installing poppler 2017-03-11 17:03:26 -08:00
James R. Barlow 8444a8f211 before_deploy doesn’t run unless something is going to be deployed 2017-03-11 17:03:20 -08:00
James R. Barlow 4a1fec8328 Improvements to macOS test and work on homebrew tap autobrew
Squashed commits:
[3f06c1e] Try setting up homebrew tap autobuilding
[01532f1] Strict mode error in brew
2017-03-11 17:00:54 -08:00
James R. Barlow 42547f6017 Revert "Finalize Dockerfile move; unfortunately not supported by Docker Hub"
Unfortunately because of this issue
https://github.com/docker/hub-feedback/issues/292

Docker Hub effectively automatically changes directory to the location
of Dockerfile, meaning needed files are not in the right places, so
this has to be reverted.

This reverts commit 65c9a07dde.

[ci skip]
2017-03-10 22:49:08 -08:00
James R. Barlow 0ccf564f03 Revert "Move Dockerfiles out of root"
This reverts commit 3d3b3abc1b.
2017-03-10 22:46:03 -08:00
James R. Barlow 65c9a07dde Finalize Dockerfile move; unfortunately not supported by Docker Hub
Unfortunately because of this issue
https://github.com/docker/hub-feedback/issues/292

Docker Hub effectively automatically changes directory to the location
of Dockerfile, meaning needed files are not in the right places, so
this has to be reverted.
2017-03-10 22:45:31 -08:00
James R. Barlow 4700a19322 Move pipeline.svg out of root 2017-03-10 17:18:38 -08:00
James R. Barlow 3d3b3abc1b Move Dockerfiles out of root 2017-03-10 17:17:36 -08:00
James R. Barlow 7cd2770a13 Fix issue #137 - proportions of non-square resolution distorted
Distortion mainly affected —force-ocr
2017-02-26 17:13:16 -08:00
James R. Barlow 7b94129d9e v4.5 notes 2017-02-14 13:03:48 -08:00
James R. Barlow d1a0065ef8 Create test case for Form XObjects 2017-02-14 12:51:15 -08:00
James R. Barlow 5a817370fd Warn more strongly about —pdf-renderer tesseract until fix is widely propagated 2017-02-14 11:33:07 -08:00
James R. Barlow ab0a210763 Update dockerfile.tess4 yet again
Installing Tess4 PPA over Tess3 proved too much pain, so sever the link
between this and the jbarlow83/ocrmypdf image, starting each from
scratch. Also the complete set of language packs proves too much - the
build seems likely to fail when trying to install so many.
2017-02-14 11:32:05 -08:00
James R. Barlow 9f800736bc Fix running_in_docker() check failing on newer Docker
This test has to work to ensure spoof/tesseract_cache.py has a writable
directory to put cache into. Otherwise those tests fail.
2017-02-13 02:16:06 -08:00
James R. Barlow c9a83afad6 Improve batch processing examples 2017-02-13 02:14:32 -08:00
James R. Barlow 5e14274f10 pageinfo: learn to extract image information from Form XObjects 2017-02-11 16:48:59 -08:00
James R. Barlow 167470b4bd Re-fix Dockerfile.tess4
[ci skip]
2017-02-10 15:43:49 -08:00
James R. Barlow f06d3c2ec2 Fix tesseract 3.04 on tesseract 4 on image
[skip ci]
2017-02-10 08:38:22 -08:00
James R. Barlow 74c99a8a77 v4.4.2 release notes 2017-02-06 21:56:55 -08:00
James R. Barlow 0e4d312ee2 Adjust Travis deploy to PyPI settings
-only on master branch
-only Python 3.6 build uploads, so the others don’t compete
-don’t upload docs to PyPI
2017-02-06 21:27:59 -08:00
James R. Barlow 589f19559d Rewrite Dockerfiles to use ubuntu 16.10 base system
Debian now has a few disadvantages:
-there is no convenient PPA for Debian tesseract 4.0, but there is for
Ubuntu
-Ubuntu sets locale to UTF-8 automatically removing the need to do this

All three ocrmypdf docker images are now based on a common Ubuntu
16.10 image, derived from the one used to build ocrmypdf-tess4.
-polyglot now differs from -tess4 only by opting into the tess4 PPA.

Both Ubuntu 16.10 and Debian stretch use tesseract 3.04.01 now making
the sharp.ttf patch unnecessary. /etc/apt/sources has been unused for a
while now both have newer Ghostscripts.
2017-02-06 14:39:29 -08:00
James R. Barlow f28bc25dc0 Configure travis to handle deployment to PyPI; also lint .travis.yml 2017-02-06 13:50:53 -08:00
James R. Barlow a0657ad937 Prevent use of —pdf-renderer tess4 on tesseract 3 2017-02-06 13:49:43 -08:00
James R. Barlow 5b8d88af4c Suggest use of aliases to hide docker run 2017-01-30 15:08:02 -08:00
James R. Barlow fa82b50340 Adding missing file Dockerfile.tess4 2017-01-29 18:34:01 -08:00
James R. Barlow 005216bc57 Support ocrmypdf-tess4 2017-01-29 18:26:52 -08:00
James R. Barlow e748fdcf6f v4.4.1 release notes 2017-01-28 22:23:35 -08:00
James R. Barlow 8c17c9918e Add documentation and test cases for —tesseract-config
This parameter has existed for along time but never really got any
attention.
2017-01-28 22:06:51 -08:00
James R. Barlow ea0dd99d0b More documentation updates 2017-01-28 15:35:59 -08:00
James R. Barlow e0cc67afae docs: suggest —oem 1 2017-01-28 14:58:25 -08:00
James R. Barlow 04f9cbe364 Describe how to use tesseract 4.0 while 3.04 is installed 2017-01-27 18:13:59 -08:00
James R. Barlow 99afebd033 tesseract jobs_limit(2)
At least on macOS with my quadcore performance improves with two
tesseracts in parallel (20% gain). Hard to say how this will affect
Linux, but stepping up to 2 jobs seems justifiable.
2017-01-27 18:13:12 -08:00
James R. Barlow a6feacc810 travis: fix ‘pip install’ by moving working code out of the way 2017-01-27 14:33:23 -08:00
James R. Barlow 65e4b1672f cffi: verbose=True 2017-01-27 14:17:13 -08:00
James R. Barlow 46cc0dd190 Revert "Do we need to exclude ocrmypdf.lib?"
This reverts commit 678b9fb603.
2017-01-27 13:51:30 -08:00
James R. Barlow 678b9fb603 Do we need to exclude ocrmypdf.lib? 2017-01-27 13:49:11 -08:00
James R. Barlow 49ab0c1f0b setup.py: cffi is definitely needed in setup_requires 2017-01-27 13:43:47 -08:00
James R. Barlow ab490a7736 Experiment: update *requirements.txt, use more current travis build steps
Perhaps this works around the pip/setup.py asymmetry that broke the
4.4 release.
2017-01-27 13:13:14 -08:00
James R. Barlow e4ce1dae35 setup.py: for some reason, subpackages must be explicitly specified 2017-01-27 00:37:05 -08:00
James R. Barlow 179b812acb Fix readthedocs build error 2017-01-26 23:57:51 -08:00
jbarlow83andGitHub 7f170517ec Note about pytest-helpers-namespace 2017-01-26 23:15:32 -08:00
James R. Barlow 5480da4f04 Additional docs updates for v4.4 2017-01-26 23:02:44 -08:00
James R. Barlow 9a15a4db10 Ensure specified destination is writable before starting pipeline process 2017-01-26 22:08:24 -08:00
James R. Barlow 55aeaec293 Autorotation check: Replace duplicated tests with parameterized test 2017-01-26 18:07:59 -08:00
James R. Barlow f6df1fb40c Fix test suite regression: output files dumped in tests/resources 2017-01-26 18:07:09 -08:00
James R. Barlow b889a89c36 Fix remaining 3.4/3.5 regressions 2017-01-26 17:53:27 -08:00
James R. Barlow 1976dc6f30 Fix issue #121 “pop from empty list” (content stream parsing error) 2017-01-26 17:24:40 -08:00
James R. Barlow e864c65d26 (Hopefully) Fix Path <-> py.path conversion on Py3.4/3.5 2017-01-26 17:19:15 -08:00
James R. Barlow 02fba02d31 Refactor test suite to use fixtures to manage paths 2017-01-26 16:38:59 -08:00
James R. Barlow fb9e7c82f6 Move duplicate test code into common namespace 2017-01-26 13:36:52 -08:00
James R. Barlow 77d31bf646 Add renderers page (missed from previous) 2017-01-26 13:20:44 -08:00
James R. Barlow 29ca799bcf Move pytest.ini into setup.cfg 2017-01-26 12:45:38 -08:00
James R. Barlow 467b7f0163 Update docs for eventual v4.4 release 2017-01-26 12:29:11 -08:00
James R. Barlow bad67c6dc5 Rename ‘tesstop’ to ‘tess4’
There’s no reason text-only PDF shouldn’t become the default for
tesseract 4.
2017-01-26 12:28:51 -08:00
James R. Barlow ac40426971 Implement “tesstop” (tesseract v4 text-only pages - working name) 2017-01-20 17:16:01 -08:00
James R. Barlow 7acfaf6d34 pipeline: rename some of the stages, for clarity 2017-01-20 17:15:00 -08:00
James R. Barlow 99e47c9c04 tesseract: add support for using v4 textonly_pdf feature 2017-01-20 17:06:23 -08:00
James R. Barlow d7904e2251 Travis now has Python 3.6, test against it 2017-01-20 14:26:17 -08:00
James R. Barlow 68aef489de Merge branch 'master' (4.3.5, Python 3.6 support) into develop
# Conflicts:
#	dev_requirements.txt
#	requirements.txt
2017-01-20 14:25:28 -08:00
James R. Barlow 3f9adcd5e0 Document idea for producing companion text files 2017-01-19 16:48:05 -08:00
James R. Barlow 6cc5135d2d Output to stdout: ensure stdout is flushed to prevent truncation errors 2017-01-19 16:41:10 -08:00
James R. Barlow d4c72b371f Forward --oem argument to tesseract 4 2017-01-18 21:37:50 -08:00
James R. Barlow 18b6f05657 Resolve issue #124 - poor performance with Tesseract v4
It seems that Tesseract v4 on a platform with OpenMP working correctly
while perform poorly with ocrmypdf because each will also soak up all
available CPUs. Running N^2 processes/threads on a N-core CPU where
each wants 100% of CPU turns out to be detrimental.

So, we restrict ocrmypdf w/tessv4 to a single Tesseract process at a
time, for now. Alternative may be to limit OpenMP threads if throughput
is higher.
2017-01-18 17:52:12 -08:00
James R. Barlow c42d9baa26 tesseract: for v4, use --psm while keeping -psm for v3
At the moment v4 accepts both but who knows if this will get dropped,
so do as document for each version.
2017-01-18 17:43:47 -08:00
James R. Barlow 6e27ecd2b9 Finalize ‘exec’ migration and make it backward compatibility for now 2017-01-18 17:40:50 -08:00
James R. Barlow 482692396e Add installation instructions for Ubuntu 16.04 2017-01-17 08:17:56 -08:00
James R. Barlow c48acf165a v4.3.5: Python 3.6 compatibility 2017-01-03 00:45:33 -08:00
James R. Barlow 9e004c3ec0 Another attempt at py 3.4/3.5
Revert to exactly what the previous passing build specified.
2017-01-03 00:34:26 -08:00
James R. Barlow 7be4e9c919 fix setuptools-scm for py 3.4, 3.5 2017-01-03 00:25:57 -08:00
James R. Barlow 5ec38a4bed Update requirements files and documentation for Python 3.6 - no code changes 2017-01-03 00:11:34 -08:00
James R. Barlow f246779b8e pdfa: documentation, remove from __future__ 2016-12-12 15:10:10 -08:00
James R. Barlow a7d8cdf061 Don’t copy pageinfo - job manager already provides a copy of real pdfinfo 2016-12-12 15:09:41 -08:00
James R. Barlow 620745c812 pipeline: don’t use qpdf to check page count again
We already know the number of pages at this stage.
2016-12-12 15:09:11 -08:00
James R. Barlow b8767e5ba9 Rename exe -> exec, more Unix-y and suggestive 2016-12-10 15:34:00 -08:00
James R. Barlow d33a50660d Replace most sys.exit() with raising exceptions
Because ruffus doesn’t handle exceptions well I tended to call sys.exit
to make sure we got out of dodge when needed.  However, sys.exit is not
ideal for the Python API this is moving towards, so this introduces
proper exceptions for the various cases that retain suggested error
codes. Only __main__.py should call sys.exit now, everyone else has to
throw an exception.

For now the worker raising a fatal exception is logging messages rather
than passing an exception object with the fatal error message, mainly
because ruffus doesn’t properly marshall the exception object so we
just check “what is the name of the exception class that caused ruffus
to thrown an RethrownJobError”?

Also fixed along the way was the wrong return code being shown for
encrypted PDF checking, and incorrect use of str.find (e.output.find)
in boolean logic (str.find returns -1 on failure to find, which is True).
2016-12-10 15:24:24 -08:00
James R. Barlow 4ee9658e97 Move external program wrappers to ocrmypdf.exe package 2016-12-09 16:54:24 -08:00
James R. Barlow dd1b84e7ba More refactoring - helpers.py 2016-12-09 16:31:08 -08:00
James R. Barlow 4c677e6c47 Extract pipeline out of __main__.py and into pipeline.py
This leaves __main__.py to handle command line arguments while pipeline.py
runs the pipeline - mostly. They are still somewhat intertwined, with
__main__.py doing essential things for pipeline.py, etc., and some
helper functions that could go in their own module.

All tests pass after this major refactor.
2016-12-09 16:17:12 -08:00
James R. Barlow f0f889440b Merge branch 'master' into feature/ooruffus 2016-12-08 16:36:03 -08:00
James R. Barlow cc9ceaeb74 v4.3.4: release notes 2016-12-08 16:34:09 -08:00
James R. Barlow ad2fa8d1d7 Fix MANIFEST for .png 2016-12-08 16:25:04 -08:00
James R. Barlow adc1580742 Help py.test collect output in more cases 2016-12-08 16:21:07 -08:00
James R. Barlow 4d3b44d6df ghostscript: cleanup harmless error message printed for overprint
Redirect stderr->stdout to hopefully make GS output easier to work with
overall, since the previous code didn’t seem to account for mixed used
properly.
2016-12-08 16:19:15 -08:00
James R. Barlow e57aa0eee2 pageinfo: fix “decimal.InvalidOperation: quantize result has too many digits”
And add new test case for this.
2016-12-08 16:06:53 -08:00
James R. Barlow 1ae1d116c7 Make setup.py license internally consistent 2016-12-08 16:06:31 -08:00
James R. Barlow 097a69d07f pageinfo: fix “decimal.InvalidOperation: quantize result has too many digits”
And add new test case for this.
2016-12-08 16:04:14 -08:00
James R. Barlow a81ce87a50 Remove non-reentrant options checking and logging setup 2016-12-05 14:13:36 -08:00
James R. Barlow 88be0d43a0 Make setup.py license internally consistent 2016-12-03 21:37:24 -08:00
James R. Barlow ff16a00a3d Remove test for Pillow JPEG and PNG
As of 3.1.1, our minimum version, these codecs are now required by
default for a successful installation, effectively solving the problem
of Pillow installed without libjpeg/libpng.
2016-12-03 14:25:46 -08:00
James R. Barlow 8982b3e1e2 Update requirements
-update requirements.txt and dev_requirements.txt to more recent version
-setup.py updated to Ubuntu 14.04 rather than 12.04 backports
-request at least Pillow 3.1.1 now (since this makes jpeg/png mandatory)
2016-12-03 14:14:07 -08:00
James R. Barlow be0fa35d14 Merge branch 'master' into feature/ooruffus 2016-12-03 14:02:43 -08:00
James R. Barlow 9f51ed9d01 Finalize v4.3.3 release notes 2016-12-03 00:39:24 -08:00
James R. Barlow 731e6792c7 Add test cases for Ghostscript PDF/A warnings 2016-12-03 00:32:09 -08:00
James R. Barlow c35ec0b4aa ghostscript: more effort at error logging 2016-12-03 00:22:03 -08:00
James R. Barlow 03aaf575dc v4.3.3 release notes, fix more gs 9.20 issues 2016-12-02 16:26:34 -08:00
James R. Barlow 9a060579ba Move work_folder into multiprocessing manager 2016-12-02 01:39:17 -08:00
James R. Barlow d40a5c4f7a Remove all remaining traces of ‘options’ global state from task runners 2016-12-02 01:31:57 -08:00
James R. Barlow 21f7dc3377 Distribute ‘options’ to worker processes via the multiprocessing manager 2016-12-02 01:06:11 -08:00
James R. Barlow 43c13a1ed9 Replace pdfinfo, pdfinfo_lock with multiprocessing manager
Using a context manager to guard the pdfinfo list makes the lock
unnecessary. (Although it was probably unnecessary in the first place
anyway.)
2016-12-01 23:36:30 -08:00
James R. Barlow 6bc3f189e1 Remove “WrappedLogger” - does not do anything useful
Never really investigated the reason why ruffus returns a mutex to go
along with its logger. It seems that the mutex is only needed if one
wanted to make multiple successive calls to a log function and have
them appear appear atomically. It is not needed to protect the logger
proxy because accessing the proxy triggers IPC in the child process
that handles the multiprocessing.Manager() object.

The logging wrapper only logs one line at a time, so the mutex does not
actually protect logging sequence. Cut it.

Also manager.Lock() returns a threading.Lock object so the purpose of it
is actually to help processes share a thread-level lock. It would be
more appropriate to use a semaphore based multiprocessing.Lock.
2016-12-01 15:27:07 -08:00
James R. Barlow 2c5437135c Remove temporary re_symlink logging shim 2016-12-01 00:31:42 -08:00
James R. Barlow 444da02523 Fix mistake made in converting pipeline; incredibly, all tests pass now 2016-12-01 00:30:19 -08:00
James R. Barlow 00e8af2381 Reactivate the pipeline; surprisingly works in quick test 2016-12-01 00:03:03 -08:00
James R. Barlow 401b21864f Convert to object oriented ruffus syntax (does not run)
I experimented with the idea of using asyncio-based processing but
realized that that does not solve the import time binding problem
that is the real issue. Therefore the simpler refactoring is to convert
to ruffus-oo syntax and get things working again.

build_pipeline() is really ugly at the moment. The old syntax had its
advantages.

This test reproduces the complete pipeline graph but does not work
otherwise.
2016-11-30 23:58:26 -08:00
James R. Barlow de939951d4 Record version in debug log 2016-11-29 15:30:50 -08:00
James R. Barlow 7725d16a26 Fix exception on inline stencil masks with no /CS attribute 2016-11-24 22:37:00 -08:00
James R. Barlow 8a74408d83 Add security suggestions 2016-11-21 20:58:31 -08:00
James R. Barlow 3d0dc95a06 Moved venvs 2016-11-21 20:40:22 -08:00
James R. Barlow 04a57a3cc2 OS X -> macOS 2016-11-21 20:40:06 -08:00
James R. Barlow d0c22ce01d v4.3.2 release notes 2016-11-10 23:16:08 -08:00
James R. Barlow 23c95e9660 ghostscript: elide overprinting to fix PDF/A errors in GS 9.20
It looks like GS 9.19 can incorrectly set overprinting for the text layer
even though this makes no sense in PDF/A, or at least someone produced
PDFs that have this after a Tesseract PDF -> GS PDF/A conversion. GS 9.20
complains about this. Instead of aborting, elide the feature.

See
http://git.ghostscript.com/?p=ghostpdl.git;a=commitdiff;h=094d5a1880f1cb9ed320ca9353eb69436e09b594
and
issue #107.

It looks like it is better to elide features and warn about elision rather
than abort with an error.
2016-11-10 14:48:02 -08:00
James R. Barlow eecab9b95d pdfa: fix KeyError on pdfa_dict if document has some xmp metadata but
not exactly what we’re looking for
2016-11-09 05:41:12 -08:00
James R. Barlow 8abc2f113c Merge branch 'develop' 2016-11-07 14:36:50 -08:00
James R. Barlow 949d2ff1c2 v4.3.1 release notes 2016-11-07 14:36:08 -08:00
James R. Barlow 1c8b763d53 test_pageinfo: Remove bits per component test
The behavior of this test will ultimately depend on what version of
img2pdf is installed, since after my patch it will be able to produce
1bpp images.
2016-11-07 14:35:54 -08:00
James R. Barlow bb91393b85 Fix “deskew-rotate” bug.
Turns out this occurred in any case where pdf-renderer hocr was used
and a tesseract timeout or error occurred. We created a replacement
page based on the unrotated page dimensions instead of the input image’s
dimensions.
2016-11-07 14:17:31 -08:00
James R. Barlow cc9c0d819e Add test case for documents that get rotated incorrectly after deskew 2016-11-07 14:15:03 -08:00
James R. Barlow a72b8caf47 Update documentation on other languages, multilingual documents 2016-11-07 14:14:06 -08:00
James R. Barlow fdd9b8b8ce Optimize some of the test resources to reduce file sizes
Mostly by reducing RGB -> monochrome and applying JBIG2 compression
2016-11-07 14:01:23 -08:00
James R. Barlow c096b4ca8c Make debug dump of pageinfo at the end of processing readable 2016-11-04 02:23:02 -07:00
James R. Barlow 427add3008 Add @posttask debug hooks 2016-11-03 18:15:21 -07:00
James R. Barlow c45871700d Fix bug: LeptonicaErrorTrap() leaks file handles 2016-11-03 15:51:27 -07:00
Sean Whittonandjbarlow83 6821e8eeb2 disable mathjax sphinx extension (#103)
Mathjax isn't actually needed for OCRmyPDF's docs, but enabling this
extension causes the browser to download a copy of mathjax.js from
cdn.mathjax.org anyway.

I have to disable this for the offline docs bundled with Debian, but
since you're not using mathjax, it would be nice to have the diff merged
upstream.
2016-11-01 21:56:57 -07:00
James R. Barlow a4f07756a5 tesseract caching: don't transcode tesseract's output, hash source file
For sanity's sake, deal with tesseract streams in binary without
transcoding (via universal_newlines, etc.). The only differences are
printing messages regarding spoofing.

Also hash the source file so that changes to the cache mechanism
invalidate old cache automatically. That is probably too aggressive,
but simple and safer than the previous approach.
2016-10-28 16:44:12 -07:00
James R. Barlow f24fb0e0c5 Obligatory MANIFEST.in repair 2016-10-28 01:28:46 -07:00
James R. Barlow 73b88a0a6f More work on documentation 2016-10-28 01:22:40 -07:00
James R. Barlow c42f39e2d4 Update README to point to ReadTheDocs 2016-10-28 00:33:17 -07:00
James R. Barlow 5e5fe3175f docs: OS X -> macOS branding change 2016-10-28 00:32:57 -07:00
James R. Barlow cab65d1f11 pageinfo: add a python3.4 implementation of isclose() 2016-10-28 00:31:04 -07:00
James R. Barlow 245f05d5f4 docs: allow python setup.py install --force to bypass checks
ReadTheDocs needs this.
2016-10-28 00:07:26 -07:00
James R. Barlow dda751f9e3 Merge branch 'feature/docs' into develop
# Conflicts:
#	ocrmypdf/__main__.py
2016-10-27 23:50:08 -07:00
James R. Barlow 3d37ae988a Update release notes for 4.3 2016-10-27 23:48:12 -07:00
James R. Barlow 717acd9855 Prevent dumping binary PDFs to stdout 2016-10-27 16:20:53 -07:00
James R. Barlow 2e4431cc63 Allow piping output to stdout 2016-10-27 16:14:42 -07:00
James R. Barlow f7387b0859 test_stdin: simplify this test
No need to involve 'cat', just hook the file up to stdin.
2016-10-27 16:01:07 -07:00
James R. Barlow a09f6b8977 Test cases: check that stdout is clear of output
To ensure piping to stdout is possible.
2016-10-27 15:58:24 -07:00
James R. Barlow d63449c214 main: don't print output file location to stdout, use stderr 2016-10-27 15:57:33 -07:00
James R. Barlow a86805f0d9 Remove possibly non-free page from "multipage.pdf" 2016-10-27 15:56:43 -07:00
James R. Barlow 7d2009ccef ghostscript: log errors from stdout 2016-10-27 15:36:20 -07:00
James R. Barlow 18ae5db06d ghostscript: ensure raster resolution is specified in integer units 2016-10-27 15:35:33 -07:00
James R. Barlow 9a1838f102 pageinfo: accept "cm/Do" image drawing without the usual "q/Q" wrapper
Some PDFs omit the traditional q/Q wrapper and alter ctm with a stack
depth of zero, so make our test for stack depth specifically test for
the case where the PDF calls for rendering to an uninitialized ctm.

Probably related to #97.
2016-10-27 15:35:00 -07:00
James R. Barlow e20346032d leptonica: add color testing functions for future experiments 2016-10-27 14:49:49 -07:00
James R. Barlow 693a27d76c leptonica: add iPython display hook and equality test 2016-10-26 14:44:41 -07:00
James R. Barlow 203966d86b leptonica: fix Pillow conversion for 1-bit and 8-bit gray images 2016-10-26 13:10:13 -07:00
James R. Barlow 7eca8508fd Implement new preprocessing feature, background removal 2016-10-14 17:23:34 -07:00
James R. Barlow b85270df1c Merge branch 'master' into develop 2016-10-14 15:56:58 -07:00
James R. Barlow aff597cef4 v4.2.5: update release notes, fix silly typo in pageinfo.py 2016-10-13 13:26:39 -07:00
James R. Barlow 61b05b3dee Fix issue: BitsPerComponent is an optional field, sometimes omitted 2016-10-13 13:15:27 -07:00
Julian Kahnertandjbarlow83 453c4ef602 Update README.rst (#98)
`brew install tesseract` just installed the english language pack not French, German or Spanish
2016-10-12 11:20:58 -07:00
James R. Barlow cf4b04f92d The main 'quick' test should be a file that OCRs to recognizable text 2016-10-07 16:25:34 -07:00
James R. Barlow 06c6999987 Merge commit '07891d994aab92e7a14aebe1ac509aab2d4f170c' 2016-10-07 12:45:56 -07:00
James R. Barlow 013c5a369f Replace redacted file with an OCR-able file 2016-10-07 12:45:22 -07:00
James R. Barlow 07891d994a Replace redacted file with an OCR-able file 2016-10-07 12:44:49 -07:00
James R. Barlow 6baf8668a6 Replace with non-free file milk.pdf with free equivalent 2016-10-06 13:10:28 -07:00
James R. Barlow 4ba2962c56 Comment on non-free files 2016-10-05 16:48:16 -07:00
James R. Barlow 7ad92f5db4 Merge branch 'master' of https://github.com/jbarlow83/OCRmyPDF 2016-10-05 16:39:00 -07:00
James R. Barlow 4dad09cc91 resources/README: replace the other large table with a list table 2016-10-05 16:38:51 -07:00
Sean Whittonandjbarlow83 7b2e0c7a7a also exclude .git in pytest.ini (#94) 2016-09-15 08:56:14 -07:00
Sean Whittonandjbarlow83 7f08f15fc9 pytest skipif for milk.pdf test (#95)
Skip the test if the fair use restricted milk.pdf is not present.
2016-09-15 08:55:31 -07:00
James R. Barlow 825c0f8b2a Note that milk.pdf is non-free, start using list-tables 2016-09-10 14:44:00 -07:00
James R. Barlow dbe880bc41 Update tesseract supported languages 2016-09-09 12:55:07 -07:00
James R. Barlow 2ec516b6ff leptonica: learn a few new tricks
Found some interesting options for background norm.
2016-09-09 12:54:36 -07:00
James R. Barlow 7942a01e50 leptonica: This is not a Py2 module anymore 2016-09-08 20:59:38 -07:00
James R. Barlow df684f9344 Update tesseract supported languages 2016-09-08 15:52:25 -07:00
James R. Barlow ae16e95e42 leptonica: scale should be a tuple for consistency 2016-09-08 11:38:10 -07:00
James R. Barlow 2ac8e8a0cc More doc tweaks, mainly introduction 2016-09-08 11:29:53 -07:00
James R. Barlow 0a0ceda71f Start the documentation 2016-09-06 13:52:40 -07:00
James R. Barlow 220f1ce161 tasks.py: stop tracking this file for now
This helper script is still in development and needs to be changed each
release, which breaks the release.

It shouldn't be in MANIFEST.in at all because it's not part of a
distribution.
2016-09-04 10:55:57 -07:00
James R. Barlow c62a8a97c9 v4.2.4 release notes 2016-09-01 21:33:38 -07:00
James R. Barlow f8a1136979 tasks: show logging info 2016-09-01 21:24:13 -07:00
James R. Barlow 9ca29c787b Update description of masks.pdf to reflect what it actually tests 2016-09-01 21:21:14 -07:00
James R. Barlow 6af748a251 pageinfo: regression - didn't add inline images to list 2016-09-01 15:27:51 -07:00
James R. Barlow 9041867f86 pageinfo: exclude images from DPI calculation if drawn at stack depth 0
More thorough testing showed that Acrobat do not presume that images
fill the page if the CTM is unspecified, as tests/resources/masks.pdf
seems to want.  Instead they treat it literally and draw the image
as 1x1 PDF units or 1/72" square in the bottom left corner of the page.

Seems like the best thing to do is ignore any such images for the purpose
of DPI calculation.  masks.pdf still works out okay because it has
other images.

For more robustness we could consider invalidating any DPI above some
limit, or warning the user about these microdot thumbnails.
2016-09-01 14:23:31 -07:00
James R. Barlow 04099b087c pageinfo: handle stencil masks when stack depth > 0 2016-09-01 14:03:30 -07:00
James R. Barlow 6d6234714c tasks: fix logic error and make magic numbers disappear 2016-09-01 14:03:08 -07:00
James R. Barlow 520be23481 Add release helper script 2016-08-31 20:33:04 -07:00
James R. Barlow 346c3c8dd3 Start tracking development requirements 2016-08-31 20:31:31 -07:00
James R. Barlow bd534c3313 main.py -> __main__.py
Executing a package with python -m packagename will check for
__main__.py inside the package.  In other words main.py should have
always been named __main__.py.

In the unlikely event that someone depends on "import ocrmypdf.main"
being meaningful, main.py continues to exist and replicates the
behavior of __main__.  (It's unlikely because import ocrmypdf.main does
unpythonic ruffus-related things at things import time, essentially
configuring itself to work with sys.argv.  To fix another day.)

This should solve the problem of Debian needing to run test suites
before installation and afterwards for continuous integration without
having to patch either file, as python -m ocrmypdf will follow import
order.  That is, if the current directory contains "ocrmypdf/" (e.g.
staging a new version) then that will be tested, else sys.path will
be checked.
2016-08-31 17:01:42 -07:00
James R. Barlow 2625368aed link: more MANIFEST.in tweaks 2016-08-31 16:28:39 -07:00
James R. Barlow 8ac94879f1 lint: no need to check for DEVNULL; all supported versions have it 2016-08-31 16:28:18 -07:00
James R. Barlow dd8c0f3756 Merge branch 'master' of https://github.com/jbarlow83/OCRmyPDF 2016-08-31 13:19:46 -07:00
James R. Barlow 010f353a5e v4.2.3 release notes 2016-08-31 13:19:27 -07:00
James R. Barlow e0a18edb92 Fix MANIFEST.in, as Python packages require 2016-08-31 13:19:17 -07:00
James R. Barlow c6f2eea058 Reinstate OCRmyPDF.sh with a deprecation warning 2016-08-31 11:57:02 -07:00
James R. Barlow bf89e38c69 Add milk.pdf test case 2016-08-31 11:42:21 -07:00
jbarlow83andGitHub e1f0640d42 Create issue template 2016-08-31 11:26:29 -07:00
James R. Barlow 71b54035ba Bug fix issue #89: trying to perform arithmetic on IndirectObject
TypeError: bad operand type for unary -: 'IndirectObject'
2016-08-31 10:25:58 -07:00
James R. Barlow 325cc0beca Allow test cases to run without installing first
As @spwhitton found:

The test suite needs to call "python3 -m ocrmypdf.main" instead of
just "ocrmypdf" because this /usr/bin/ocrmypdf script has not yet been
generated when dh runs the test suite.

---

Seems reasonable to perform in-place testing independent of installation.

Source:
https://sources.debian.net/src/ocrmypdf/4.2.1%2Bgit.20160824.1.5d67cc7-1/debian/patches/0001-patch-test-suite-executable.patch/
2016-08-26 15:23:26 -07:00
James R. Barlow 1a9f09c4d5 Remove OCRmyPDF.sh and its usage in all test cases 2016-08-26 15:18:38 -07:00
James R. Barlow 4fed4e2af3 tests: don't try to pass Unicode arguments on command line on Linux
Depends on locale being configured properly, and it's not necessary
to be able to do this.
2016-08-26 15:08:56 -07:00
James R. Barlow 74cc2346a5 pytest.ini: apply patch from Debian to exclude .pc dir
https://sources.debian.net/src/ocrmypdf/4.2.1%2Bgit.20160824.1.5d67cc7-1/debian/patches/0003-pytest-exclude-.pc-dir.patch/

From: Sean Whitton <spwhitton@spwhitton.name>
Date: Thu, 12 May 2016 08:21:39 -0700
Subject: pytest exclude .pc dir
Forwarded: not-needed

Patches to the test suite present in .pc confuse pytest.
2016-08-26 15:07:11 -07:00
James R. Barlow cc7e328358 Improve some documentation for tests 2016-08-26 15:04:08 -07:00
James R. Barlow d25397e2b0 Add test case for PDFs with masks and stencil masks 2016-08-26 15:03:27 -07:00
James R. Barlow bc11454e1c Help text: example of shell pipeline with img2pdf 2016-08-25 14:58:25 -07:00
James R. Barlow 2025a096c3 Test case for stdin streaming 2016-08-25 14:46:54 -07:00
James R. Barlow 38fe14b108 Make final PDF/A output message less obtuse 2016-08-25 14:46:40 -07:00
James R. Barlow 1b7b2f3695 v4.2.2 release notes, documentation improvements 2016-08-25 14:46:09 -07:00
James R. Barlow 5d67cc76cc Update 4.2.1 release notes 2016-08-24 14:16:22 -07:00
James R. Barlow 27a3813207 Recover input filename from symlink on error message
The recent commit to accept files from stdin broken the feature of
returning the input filename on an error, returning the temp filename
instead, which is confusing.
2016-08-23 17:38:28 -07:00
James R. Barlow b06e0bfdcd Merge branch 'develop' 2016-08-23 16:03:07 -07:00
James R. Barlow d616f25324 Implement DPI checking for stencil masks 2016-08-23 15:59:34 -07:00
James R. Barlow b03028e31f setup.py -> license is MIT 2016-08-19 10:14:33 -07:00
James R. Barlow e08c42fd3d Tweak pipeline again 2016-08-09 22:40:29 -07:00
James R. Barlow 16901f7134 Accept input from stdin if input filename is '-' 2016-08-09 15:46:24 -07:00
James R. Barlow dffceedd85 Update the pipeline image 2016-08-09 15:45:19 -07:00
James R. Barlow e5541e435c New test to confirm we can emit JBIG2 with appropriate settings 2016-08-03 11:35:48 -07:00
James R. Barlow b969aad67b Tweak release notes 2016-08-03 03:36:45 -07:00
James R. Barlow e70387b1af Add a simple test for image to PDF 2016-08-03 03:35:30 -07:00
James R. Barlow 44f47fba21 PDF/A: handle case of no XMP metadata gracefully 2016-08-03 02:57:25 -07:00
James R. Barlow 02584094a1 Suppress NUL bytes in metadata from input files 2016-08-03 02:47:44 -07:00
James R. Barlow 91d715ac93 Add test cases for --output-type 2016-08-03 02:47:18 -07:00
James R. Barlow 35addb8a33 Complain if Chinese is requested with settings known to not work
Should extend test for other Asian languages
2016-08-03 01:29:12 -07:00
James R. Barlow d32ea8d0dd Remove dead code from qpdf merge + PyPDF2 metadata patching
I tried "qpdf merge + PyPDF2 metadata patching" first. The problem is
that PyPDF2 produces a 1.3 by default and generally I have less
confidence it.

New approach is to stuff the Document Info metadata in the first page
with PyPdf2, cross fingers and use qpdf to merge. It's not quite as
clean and might harm the first page, but it's better than shipping
files produced by PyPDF2.
2016-08-03 01:28:27 -07:00
James R. Barlow 12575d594a Improve PDF/A validity checking at end 2016-08-03 01:26:16 -07:00
James R. Barlow 0746083301 Fix failing test case - unbound local variable in finally block 2016-08-03 01:00:38 -07:00
James R. Barlow 5c99acf6d1 Experimental change to use qpdf to merge files (disables Ghostscript)
All but one tests pass, test_input_file_not_a_pdf

Not sure if PyPDF2 metadata generation will mangle the first page.
2016-08-03 00:56:44 -07:00
James R. Barlow 2b10df7b74 leptonica: note about when it may be safe to drop <1.72 workaround 2016-08-03 00:54:37 -07:00
James R. Barlow ebe68de4ff Functional qpdfmerge with PyPDF2 for DocumentInfo block
Tests mostly passing. For the moment this is the new default.

Although PyPDF2 produces a PDF-1.3 which will be wrong for some contents
and possible should be repaired with qpdf. Again.

Looks like it could work better to merge PyPDF2 and fix everything
with qpdf.
2016-08-02 16:48:13 -07:00
James R. Barlow b17c6a146d Experimental qpdf merging
Does not copy /Catalog metadata, but otherwise functional
2016-08-02 02:19:02 -07:00
James R. Barlow 46d837c866 Clarify trusty/precise stuff 2016-08-02 01:29:33 -07:00
James R. Barlow 24856b61e4 Fix typo in readme 2016-08-02 01:29:22 -07:00
James R. Barlow 8d0c6ff616 pyvenv -> python3 -m venv
Sadly the Python developers are removing this script
2016-08-02 01:27:50 -07:00
James R. Barlow 0b24f971cd ocrmyimage: complain about ICC profiles being presumed 2016-08-02 01:22:36 -07:00
James R. Barlow bc5d3824bd Don't overload --oversample, use --image-dpi instead for images 2016-07-31 02:09:30 -07:00
James R. Barlow 4356983707 Suppress overly long stack traces on traverse_ruffus_exception 2016-07-31 02:06:44 -07:00
James R. Barlow 2414b79ee6 More cleanup of exception related errors 2016-07-31 01:48:13 -07:00
James R. Barlow 968e1546f0 Refactor image file triage 2016-07-31 01:47:57 -07:00
James R. Barlow 48213c9c3f Update release notes and readme 2016-07-29 15:25:16 -07:00
James R. Barlow f385772d21 Refactor "is this an iterable that's not a string?" test 2016-07-29 15:25:02 -07:00
James R. Barlow d257c83520 Most tests were failing at split_pages()
It seems that ruffus sometimes decides to send a ['inputfile.pdf']
instead of a bare string.
2016-07-29 14:59:17 -07:00
James R. Barlow 7b72ffec4f ocrmyimage: better handling of missing/invalid DPI 2016-07-29 14:38:07 -07:00
James R. Barlow 757f6826dc ocrmyimage - Attempt conversion to PDF if input file is not a PDF
First cut.

May have broken ruffus errors again too.
2016-07-29 14:03:19 -07:00
James R. Barlow 5df83a0d30 Travis: use Python 3.5 too 2016-07-29 13:31:40 -07:00
James R. Barlow d70e3d3753 ruffus exceptions: for clarity only, don't iterate strings
It's a good habit to ensure any iterator test is explicit about
allowing or disallowing strings.
2016-07-29 13:31:24 -07:00
James R. Barlow 0dfceedcfb Remove old OCRmyPDF 2.x from release notes; update 4.2 notes 2016-07-29 03:08:59 -07:00
James R. Barlow 2c30f4bfc5 Travis: build partly working on trusty; tweak requirements again
The build is #122
https://travis-ci.org/jbarlow83/OCRmyPDF/builds/148255615

Errors seem to be related to either Ghostscript or leptonica? Maybe
-dSAFER?
2016-07-29 03:08:01 -07:00
James R. Barlow 9e7fb52b47 Travis: add PPA to support unpaper 2016-07-29 01:57:12 -07:00
James R. Barlow bb5fd38e38 Remove additional PPA's and try again 2016-07-29 01:47:56 -07:00
James R. Barlow 7c8cf5cfa2 Try travis-trusty
This removes some backports for packages that Ubuntu trusty offers but
for which Ubuntu precise needed help.
2016-07-29 01:44:57 -07:00
James R. Barlow fef35e4eb2 Fix handling of DPI for rare case of JPEG recompression after deskew/clean
This test is exercised by page 4 of multipage.pdf. If all images are
JPEGs, and one of deskew/clean removes DPI information, make sure that
we can get the right information back and that the DPI stays square.
2016-07-29 01:34:52 -07:00
James R. Barlow 8f77576dc4 Fix non-square image resolution for "hocr" case; use img2pdf 0.2.1
Tesseract renderer not immediately fixable.
2016-07-28 16:43:51 -07:00
James R. Barlow b3fcf24a26 Refactor DPI: fix regressions in test suite
Some called functions are particular about the data format of DPI and
don't like to deal with the Decimal() returned by PyPDF2. Convert to
float and int where needed.
2016-07-28 00:19:32 -07:00
James R. Barlow 16e4d342d2 Bug fix: --force-ocr should still run on pages with no images
Useful for people who want to reprocess text.

This also requires --oversample because DPI is undefined. To be fixed
in next commit.
2016-07-27 15:06:49 -07:00
James R. Barlow 8458a51860 Tighten requirements and dependencies 2016-07-27 14:47:59 -07:00
James R. Barlow 636d1903b3 Ghostscript: do raster output with -dSAFER
-dSAFER does not work when rendering PDF/A, because that needs to load
the ICC file, and -dSAFER prevents access to external files.
2016-07-27 00:54:40 -07:00
jbarlow83andGitHub 514efa36fc Readme: Add table of contents, brew install tesseract --with-language packs 2016-07-24 11:21:46 -07:00
James R. Barlow bd48f40d3d v4.1.4 release notes 2016-07-17 00:35:06 -07:00
James R. Barlow c02dbc809a Merge commit '68cf9cbd87c188823027f9d1bfe9029017e7281f' into develop 2016-07-17 00:29:48 -07:00
James R. Barlow 410111d6fb Bug fix: Monochrome images with ICC treated as full color images
Issue #79.
User submitted PDF with ICC profile attached to the monochrome image
in the input file, which is not common but useful for PDFs that want to
define how light the paper is or how dark the black is. The code was
written to assume unusual images are full color unless it can prove
otherwise. Handle this simple case. Other ICC cases should be tested.
2016-07-17 00:29:32 -07:00
jbarlow83andGitHub 68cf9cbd87 .rst: add code-block markup 2016-07-05 14:03:55 -07:00
jbarlow83andGitHub c9b2540d9d Fix some .rst formatting errors 2016-07-05 13:48:19 -07:00
jbarlow83andGitHub 1bacf35a2c Update license information for encrypted_algo4.pdf 2016-06-24 14:25:15 -07:00
jbarlow83andGitHub 8aef0d9277 Merge pull request #76 from Jmuccigr/patch-2
Adding explicit reference to help
2016-06-24 14:21:23 -07:00
John MuccigrossoandGitHub b2fa8645ba Adding explicit reference to help 2016-06-24 13:44:12 -05:00
James R. Barlow c96823a648 v4.1.3 release notes 2016-06-23 13:47:56 -07:00
James R. Barlow 3807b7d655 Merge branch 'feature/leptfun' into develop 2016-06-23 13:45:35 -07:00
James R. Barlow a45505cf1d Fix order of operations in matrix multiplication
Issue #73. The order of operations happens to not matter for scaling
but does matter for translation. We only need scaling to find the DPI,
so the error was not noticed. Mainly useful to other uses of this
library.
2016-06-23 13:36:23 -07:00
James R. Barlow b4a734fc0d Test case for "algorithm 4" test
Algorithm 4 -> PDF version 1.6
2016-06-23 13:21:26 -07:00
James R. Barlow bbd02926e1 Add helpful error message for PDFs that use algorithm 4 2016-06-23 13:13:17 -07:00
jbarlow83andGitHub 5022ded276 Update Windows directions 2016-06-16 15:15:46 -07:00
James R. Barlow c7612152ef leptonica: pillow interop 2016-06-06 23:55:23 -07:00
James R. Barlow af91642cd1 lept: fix __getstate/__setstate 2016-06-06 23:55:23 -07:00
James R. Barlow 9c66334c38 Leptonica - ortho rotate, background norm 2016-06-06 23:55:23 -07:00
James R. Barlow 8d79b94b84 cpix -> _pix 2016-06-06 23:55:23 -07:00
James R. Barlow d7f60b96c1 More leptonica functions for page manipulation 2016-06-06 23:55:23 -07:00
James R. Barlow b964999427 Update filename references from sRGB_IEC to sRGB 2016-05-10 21:58:04 -07:00
James R. Barlow 3473345ea6 Replace sRGB_IEC with MIT license compatible sRGB
New file is from Debian package icc-profiles-free
2016-05-10 21:48:32 -07:00
James R. Barlow 349ec5c81f Provide more helpful error message if pypdf can't merge pages 2016-04-28 14:02:12 -07:00
James R. Barlow ff78d7c56c v4.1 release notes 2016-04-28 00:46:16 -07:00
James R. Barlow ff092c8629 Fix race condition between these tests when run in parallel 2016-04-28 00:39:15 -07:00
James R. Barlow fe14cb57c0 Fix ruffus exception output
I found this issue in ruffus 2.6.3
https://github.com/bunbun/ruffus/issues/65
also discussed here
https://github.com/bunbun/ruffus/pull/67

ruffus 2.6.3 RethrownJobError don't follow the normal conventions and
so its exception causes problems when they cross process boundaries.
This change carefully examines the various forms of ruffus exception
objects that can appear in 2.6.3 and parses them more carefully. It
also removes any direct posting of the exception to the logger because
this triggers another serializing of the exception object, mutating it
further.
2016-04-28 00:38:50 -07:00
James R. Barlow 507fbc01d5 Refactor _find_page_images 2016-04-27 13:13:43 -07:00
James R. Barlow 325479e5be Fix test failure: inline images with multiple image filters specified 2016-04-27 13:09:41 -07:00
James R. Barlow e926ecb8b2 Fuzzing: check for graphics stack overflow
Very unlikely to occur
2016-04-18 13:08:35 -07:00
James R. Barlow d0cb6c0e92 Replace private hypotenuse formula with hypot() 2016-04-18 13:07:53 -07:00
James R. Barlow 5b7c8cf5d3 Remove check for /ImageMask
/ImageMask means the the image is a stencil mask for a grayscale or
color image. From issue #63 a user has a PDF apparently with only a
stencil mask and no other images. According to PDF spec you're supposed
to use masks to help draw other images rather than draw masks
explicitly, although drawing masks is not forbidden.

In the event that the image mask is higher resolution than the input
 image, the composite image should be drawn at the highest DPI to a
ccurately capture the effect of the mask. It looks like the right to
 do is take masks into account rather than ignore them (a holdover
from earlier, less robust versions), or don't treat it different.
2016-04-14 14:24:33 -07:00
James R. Barlow 40baab32ac Remove dead code "import stuff in testcase" 2016-04-14 14:22:34 -07:00
James R. Barlow e877d37ac8 --rotate-pages: Only apply rotation if we're reasonable confident
Take the threshold from tesseract's default value for -psm 1.
2016-04-14 13:49:44 -07:00
James R. Barlow 5a9f77e438 Merge commit '1605408c23fa1b9252c5d3f10f279b43733b0728' into develop 2016-04-14 13:07:10 -07:00
James R. Barlow 8ddd67d1e2 Check encoding of inline images 2016-03-24 15:11:10 -07:00
jbarlow83 1605408c23 README: add libffi-dev 2016-03-22 23:39:02 -07:00
James R. Barlow 2d3b1ebf6e Simplify DPI calculation with algebraic derivation
Needs testing
2016-03-20 12:35:41 -07:00
James R. Barlow c74eaab7f5 Update license: sRGB ICC 2016-03-20 12:34:34 -07:00
James R. Barlow c21d231388 Merge commit 'a73afc4e769202b916d35dee481d741cf6bb7224' 2016-03-20 12:33:15 -07:00
jbarlow83 a73afc4e76 Merge pull request #59 from spwhitton/apt-get
README: Debian and Ubuntu installation option
2016-03-14 00:19:59 -07:00
Sean Whitton 76c364150d README: Debian and Ubuntu installation option 2016-03-13 23:15:00 -07:00
James R. Barlow 94a3e447cc Add otsu threshold to leptonica 2016-03-12 00:09:20 -08:00
James R. Barlow 12868b461a Travis: install unpaper.deb instead of compiling from source 2016-03-11 21:16:01 -08:00
James R. Barlow 322085933b unpaper: fix check for missing and old versions, add test case 2016-03-10 15:37:09 -08:00
James R. Barlow 3fed94bb79 v4.0.7 2016-03-02 06:27:01 -08:00
James R. Barlow 8c877482bd Fix leptonica initializers 2016-03-02 06:26:25 -08:00
James R. Barlow b17d589e84 Don't set -sOutputICCProfile
Ghostscript dev advised against. It appears that this is for
creating target for a device that colors in a particular format.
2016-03-02 06:25:34 -08:00
James R. Barlow 368252a243 setuptools_scm_git_archive seems suddenly broken 2016-03-01 02:09:45 -08:00
James R. Barlow ccefda1bee v4.0.6 notes 2016-03-01 01:58:32 -08:00
James R. Barlow 3d0e8c9629 Provide our own sRGB profile instead of Ghostscript's 2016-03-01 01:27:40 -08:00
James R. Barlow 313bbbb94c setup_scm_git_archive: add additional files 2016-02-29 12:46:27 -08:00
James R. Barlow 0360f078de get_postscript_icc_path: don't check the same path multiple times 2016-02-29 12:45:58 -08:00
James R. Barlow c8901666c4 Merge branch 'master' of https://github.com/jbarlow83/OCRmyPDF 2016-02-29 00:06:07 -08:00
James R. Barlow 7430006596 Improve install instructions for OS X (unpaper) 2016-02-29 00:05:31 -08:00
James R. Barlow f3e06b2dbd Add bookmarks to file for more testing 2016-02-29 00:05:07 -08:00
jbarlow83 e97df307ff Merge pull request #54 from stweil/master
Replace broken link to c't article by permalink
2016-02-28 07:18:40 -08:00
Stefan Weil 1443354aa2 Replace broken link to c't article by permalink
Update also the 2nd article link to use a permalink, too.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
2016-02-28 13:57:42 +01:00
James R. Barlow 250e68c1cd v4.0.5 release notes 2016-02-27 01:01:38 -08:00
James R. Barlow 6a380ee99c Fix temporary file placed in wrong folder 2016-02-27 00:51:47 -08:00
James R. Barlow 3c90bd96a9 Remove extraneous debug print() messages 2016-02-27 00:50:58 -08:00
James R. Barlow 06a7ceb25a v4.0.4 Updates release notes 2016-02-27 00:22:37 -08:00
James R. Barlow 733a8e7d58 Merge branch 'feature/parsecontent' 2016-02-27 00:19:19 -08:00
James R. Barlow 570bbe9a05 Add comments and remove debugging, improve inline handling
Squashed commits:
[bfff3c9] pageinfo, have a main()
2016-02-27 00:18:36 -08:00
James R. Barlow 5cc3adb39a Add support for inline images 2016-02-27 00:18:36 -08:00
James R. Barlow 3957a0606c Compute image pixel density without performing rectangle intersection (+5 squashed commits)
Squashed commits:
[0e27904] Partially implement DPI calculation with rotation of the image

Fixes test suite
[a64f662] pageinfo: all tests pass
[c5b811a] Fix typos
[cdd2286] Can now find inline images for efficiently
[60dde8d] First cut at implementing intelligent DPI detection based on content stream

Broke many of the test cases
2016-02-27 00:18:36 -08:00
James R. Barlow 11a561dbce v4.0.3 release notes 2016-02-26 01:12:15 -08:00
James R. Barlow dad2198394 Log information about detected page orientations in a summary line 2016-02-26 01:07:59 -08:00
James R. Barlow e40fdc502d Always dump stack trace for unexpected errors 2016-02-26 01:06:59 -08:00
James R. Barlow d446fe5922 Fix "too few characters" reported as error by tesseract -psm 0 2016-02-21 08:53:34 -08:00
James R. Barlow 4ca90c106d Docker: fix blank JPEG2000 PDF issue 2016-02-21 04:24:21 -08:00
James R. Barlow 7c5e58a497 Fix test cases that break in Docker, improve test for running in Docker 2016-02-20 23:47:37 -08:00
James R. Barlow 323b9a5f8e Add other missing files 2016-02-20 05:34:21 -08:00
James R. Barlow cab381a339 Add JPEG 2000 test case 2016-02-20 05:13:19 -08:00
James R. Barlow fe4d4c39cd Merge commit '6f3ac46b1c176d48782347cfa14d9ef6ce773f37' into develop 2016-02-20 04:56:12 -08:00
James R. Barlow ad188d7ae1 Docker: supply openjpeg to address JPXDecode errors 2016-02-20 04:54:55 -08:00
James R. Barlow 8246cc0538 Gracefully recover from tesseract's failure to process very large images
And test cases to check this
2016-02-20 04:53:23 -08:00
James R. Barlow 6f3ac46b1c Gracefully recover from tesseract's failure to process very large images
And test cases to check this
2016-02-20 04:53:02 -08:00
James R. Barlow ac71c3be63 4.0.2rc1 - release notes, add missing file caught by Travis 2016-02-20 03:36:37 -08:00
James R. Barlow ecc0ac9b19 Fix error on --tesseract-timeout timing out 2016-02-20 03:13:23 -08:00
James R. Barlow ea4e6bf67d leptonica: serialization tweaks, memory handling 2016-02-20 02:54:53 -08:00
James R. Barlow 46c204f533 Fix leptonica pickling 2016-02-20 02:35:34 -08:00
James R. Barlow 71fbda8bf6 Adjust page orientation parsing to deal with change in Tess 3.04.01 2016-02-20 01:32:56 -08:00
James R. Barlow 9b79b4a7c8 Leptonica: documentation, helper functions 2016-02-20 01:20:06 -08:00
James R. Barlow c04cc853d7 leptonica: remove special PNM handling
We no longer use PNM as an intermediate format, so there's no need to
handle leptonica's PNM quirks.
2016-02-19 15:13:14 -08:00
James R. Barlow dd41e70ccc leptonica: nit 2016-02-19 15:11:48 -08:00
James R. Barlow 4206e74f42 tests: also check that monochrome correlation correctly detects matches 2016-02-19 14:35:31 -08:00
James R. Barlow 68c3ce56a9 Don't do chmod unless necessarily (breaks py.test on Docker) 2016-02-19 14:09:56 -08:00
James R. Barlow ab0e5fa425 Improve error checking for tesseract -psm 0 (orientation) errors 2016-02-19 03:58:39 -08:00
James R. Barlow f3b0434a87 Improve ability to capture error messages from tesseract on a crash 2016-02-19 03:48:49 -08:00
James R. Barlow aa394440db Just use the PyPI version of ocrmypdf in dockerfile
Apparently setuptools_scm_git_archive is ineffective on hub.docker.com
automatic build, it still can't find a version.
2016-02-17 15:14:23 -08:00
James R. Barlow 3b98a1a04b Fix KeyError on unexpected tess output 2016-02-17 06:05:27 -08:00
James R. Barlow fcb89b0c58 Forgot to save release notes 2016-02-17 01:48:25 -08:00
James R. Barlow ac65d6a03a v4.0: release notes 2016-02-17 01:21:17 -08:00
James R. Barlow 2103f60906 Merge branch 'release/v4.0.0' 2016-02-17 01:13:24 -08:00
James R. Barlow e3c3d848c1 Save Dockerfile comment 2016-02-17 01:11:41 -08:00
James R. Barlow d4ef3411e0 Suppress --pdf-renderer tesseract warning in Docker image
Since the corrected font is provided in the Docker image, there's no
reason to show the warning.
2016-02-17 01:03:20 -08:00
James R. Barlow 71d616e413 Restore Dockerfile on local and probably on automated build as well 2016-02-17 00:13:45 -08:00
James R. Barlow fe651d1bf5 Overwrite Tesseract 3.04 default pdf font with better pdf font 2016-02-16 21:45:44 -08:00
James R. Barlow 582ba8cfad Provide sharp2.ttf for Docker images 2016-02-16 21:45:17 -08:00
James R. Barlow d23291650a Remove duplicate line from documentation 2016-02-16 14:30:15 -08:00
James R. Barlow 812fd745b6 Remove redundant line from resources 2016-02-16 14:29:56 -08:00
James R. Barlow a87aa71d85 Remove old documentation about Pillow not linking jpeg, zlib
As of Pillow 3.0.0 this is fixed, so make Pillow 3 a requirement
2016-02-16 14:29:31 -08:00
James R. Barlow 60b2eb1455 Fix JPEG DPI: Pillow expects dpi=(x,y) 2016-02-16 07:29:20 -08:00
James R. Barlow ab3c1988c1 Work around Leptonica < 1.72 bug that breaks Travis 2016-02-16 07:03:50 -08:00
James R. Barlow ee5223eea8 Travis again: are invalid correlation measurements a use-after-free?
Try explicitly casting the value to a float.
2016-02-16 06:09:48 -08:00
James R. Barlow edd2185268 Fix pytest-runner not understanding 'norecursedirs'
As discussed here
https://github.com/pytest-dev/pytest-runner/issues/7
and sort of
https://github.com/pytest-dev/pytest/issues/567
2016-02-16 05:44:55 -08:00
James R. Barlow 35b1ca2be2 Travis: try replacing non-standard invocation of py.test
It seems the normal thing to wire up python setup.py test to invoke
the test suite rather than py.test. This may be the reason for the
past chain of cffi-related commits.
2016-02-16 05:36:14 -08:00
James R. Barlow 71e493a810 Fix case of JPEG missing DPI field 2016-02-16 05:29:32 -08:00
James R. Barlow 6178e22e7f Travis: force compile leptonica? 2016-02-16 05:24:46 -08:00
James R. Barlow ef0aab060a Make debug output more verbose on failure 2016-02-16 05:17:18 -08:00
James R. Barlow d70ce61cfd Travis: maybe it's just the missing __init__.py? 2016-02-16 05:09:12 -08:00
James R. Barlow 8cd84afac8 Revert "Try moving leptonica build script, playing with wheels a bit"
This reverts commit ec2c6c312b.
2016-02-16 05:04:20 -08:00
James R. Barlow ec2c6c312b Try moving leptonica build script, playing with wheels a bit 2016-02-16 04:05:58 -08:00
James R. Barlow 3946bba318 Too soon, try again 2016-02-16 03:51:08 -08:00
James R. Barlow 2ed0b78a7b Travis: are you creating _leptonica.py? 2016-02-16 03:47:35 -08:00
James R. Barlow ed346d032c Does Travis need explicit install libffi-dev? 2016-02-16 03:41:11 -08:00
James R. Barlow acd645f192 Fix travis syntax error 2016-02-16 02:42:47 -08:00
James R. Barlow 88433e4c34 Fiddle with travis, try to get better debug output
Essentially cffi failed somehow, not clear how
2016-02-16 02:12:14 -08:00
James R. Barlow 1224af1780 Update test resources to address files with unknown source
-Remove Test_Issue_28.pdf (inherited from fritz-hh, source unknown)
-Replace missing_docinfo.pdf (received from user, but it's a printout of
a website; unclear status, so created a new PDF with the same effect)
-Others are okay
2016-02-16 00:28:28 -08:00
James R. Barlow ab13342931 Revise rotation tests in prep for adding a few more 2016-02-15 17:17:43 -08:00
James R. Barlow d7913da484 Test case: remove filename conflict 2016-02-15 16:49:28 -08:00
James R. Barlow c50e3f1329 Complain about older tesseracts that don't have sharp2.ttf installed 2016-02-15 16:43:41 -08:00
James R. Barlow a62f86dbd7 Update release notes 2016-02-15 16:43:14 -08:00
James R. Barlow 33b88b18db Update the notes 2016-02-15 14:03:59 -08:00
James R. Barlow 7c691c21ab Fix image layer rotation for pages with nonzero crop boxes 2016-02-10 17:48:33 -08:00
James R. Barlow 4ec51729d8 Partial fix for images not anchored to (0, 0) 2016-02-10 17:14:48 -08:00
James R. Barlow 07b41e479a Cleaner access to mediabox 2016-02-09 02:19:05 -08:00
James R. Barlow 6510bcad19 DPI information not transferred automatically from PNG to JPEG 2016-02-09 02:18:54 -08:00
James R. Barlow 265d2ce39b Better skewed image 2016-02-08 23:44:46 -08:00
James R. Barlow 1928a64cae Better logging output for autorotation 2016-02-08 23:42:25 -08:00
James R. Barlow 11e575a5a3 leptonica: suppress debug output 2016-02-08 23:41:45 -08:00
James R. Barlow 7fbc0d6460 tesseract: unify logging function 2016-02-08 23:40:36 -08:00
James R. Barlow 1ba8b1aa4b unpaper is lousy at deskewing, so let leptonica do it 2016-02-08 15:26:33 -08:00
James R. Barlow 3569c76c0f Also include cardinal.pdf 2016-02-08 15:23:04 -08:00
James R. Barlow 16c7ac2582 Fix test_deskew for new Leptonica API 2016-02-08 15:20:01 -08:00
James R. Barlow 4ceb59215f Leptonica: classes are better 2016-02-08 15:14:44 -08:00
James R. Barlow 2e6879ee51 Introduce Leptonica class for Pix 2016-02-08 14:52:01 -08:00
James R. Barlow 66fc2e9d7d Add rotate 180 correlation sanity check 2016-02-08 13:10:11 -08:00
James R. Barlow 2c7a6e574f Shorten names of _make_input/output 2016-02-08 12:57:26 -08:00
James R. Barlow 78c3bf5dba Check autorotate using leptonica correlation 2016-02-08 12:55:50 -08:00
James R. Barlow 98c115e3bb Cache wasn't enabled properly for test_autorotate 2016-02-08 12:55:28 -08:00
James R. Barlow 2752bda80b Merge branch 'feature/leptdeskew' into feature/logging
Need leptonica for testing now, I think
# Conflicts:
#	ocrmypdf/tesseract.py
#	requirements.txt
#	setup.py
2016-02-08 12:34:48 -08:00
James R. Barlow 7c0940609a Take a stab at writing test case for autorotate 2016-02-08 12:32:39 -08:00
James R. Barlow d30a879e2d Fix test suite by running select_image_for_pdf unconditionally
The purpose of this change that caused the problem was a minor
optimization for the tesseract renderer path that had it pull an image
from select_image_for_pdf so that it could use a JPEG instead of PNG,
instead of taking it from preprocess_clean where it would only get a PNG
and make large files.
2016-02-08 02:33:03 -08:00
James R. Barlow b907234d5c Update tesseract spoofing to cache orientation and script detection checks
No cache: 269 s
With cache: 144 s

test_oversample[tesseract] now fails, all others good
2016-02-08 02:21:56 -08:00
James R. Barlow b0114c9174 More logging improvements 2016-02-08 01:31:15 -08:00
James R. Barlow d2ba8c501f Restore invisibletext for normal output 2016-02-08 01:14:39 -08:00
James R. Barlow 6a7ed7d359 Make logging output a lot more useful 2016-02-08 00:58:14 -08:00
James R. Barlow 6289afa1a6 Better: custom logging factory to avoid whatever ruffus is doing 2016-02-08 00:18:52 -08:00
James R. Barlow 9bb6fa04cb Return logging to a semblance of normalcy 2016-02-08 00:09:31 -08:00
James R. Barlow afb6f6f5c9 Render preview as .jpg instead of .png
Smaller file size of JPEG seems to help performance, although the
difference is only about 1%.
2016-02-07 15:49:10 -08:00
James R. Barlow 8a69671dbd Suppress debug message 2016-02-07 15:43:57 -08:00
James R. Barlow 178aee4687 Make rotation optional (for now it's off, possibly should be on) 2016-02-07 15:43:45 -08:00
James R. Barlow 8484caddfb Tweak pipeline, allowing --pdf-renderer to use JPEGs instead of PNGs 2016-02-07 15:36:51 -08:00
James R. Barlow 08313316de Cleanup auto-rotation 2016-02-07 15:06:54 -08:00
James R. Barlow 1d0eca5c63 All four rotation directions working 2016-02-07 06:09:01 -08:00
James R. Barlow fe89232a30 Fix autorotate for some lossless cases 2016-02-07 05:59:46 -08:00
James R. Barlow 4b51b521e2 Implement autorotate (provided lossless reconstruction is disabled)
Works for a single page file, probably

Although arguably rotation is not quite lossless, and the two could be
mutually exclusive anyway, so maybe this is it. Did not check in some
debugging changes (lossless=False, text debugging=True)

PyPDF seems to get merging wrong when one of the pages is rotated.
2016-02-07 03:27:33 -08:00
James R. Barlow e9ec458304 tesseract: add command to access OSD values 2016-02-07 03:21:32 -08:00
James R. Barlow 54b0ddd787 ghostscript: don't try to "help" autorotation
It uses text direction alone -- unreliable guide.
2016-02-07 03:20:42 -08:00
jbarlow83 93bec22f9c README: mention polyglot, fix container vs image 2016-02-07 00:32:20 -08:00
James R. Barlow 0dc96442d8 Fix img2pdf usage in test case (to make Travis CI happy again) 2016-02-06 23:41:32 -08:00
James R. Barlow 58f4582517 More Dockerfile repair
I'm not fully happy with this arrangement, as it effectively downloads
OCRmyPDF twice, not to mention the lengthy setup time overall.

Will need to try separate build/run images in the future, but now just
get it working again.
2016-02-06 23:13:16 -08:00
James R. Barlow 2d15c09cca Merge branch 'develop' 2016-02-06 18:18:49 -08:00
James R. Barlow 04cb8865b0 Fetch application from PyPI instead of local
setuptools_scm barfs because it can't find the version, because Docker hub
retrieves the application from Github in a way that omits the necessary
details.

I suppose there is a certain logic to Docker only using the tagged
released versions from PyPI, so go with it.  The other attractive option
is to nix setuptools_scm.
2016-02-06 18:18:30 -08:00
James R. Barlow 6fe32bbaf7 v3.2.1 2016-02-05 16:10:18 -08:00
James R. Barlow 4abb20390d Bump Dockerfile versions 2016-02-05 16:08:26 -08:00
James R. Barlow daa3916430 Fix img2pdf 0.2 usage
All tests pass when forced to rely on img2pdf, so seems okay
2016-02-05 15:13:26 -08:00
James R. Barlow e9b87cefcc Try img2pdf 0.2 2016-02-05 14:38:37 -08:00
James R. Barlow 60593b5ad3 Tighten up package requirements to deal with incompatible img2pdf 0.2 release 2016-02-05 14:37:05 -08:00
James R. Barlow f708b11ea4 Fix Python 2.7 warning 2016-02-05 02:34:49 -08:00
James R. Barlow 7982f58b2e Try tweaking Dockerfile for automated build again 2016-02-05 01:38:59 -08:00
James R. Barlow e805c1908a Minor fix for Dockerfile polyglot 2016-02-05 00:52:27 -08:00
James R. Barlow cb3ba8e973 Merge branch 'release/v3.2' into develop 2016-02-05 00:10:41 -08:00
James R. Barlow 344fc40cbc Merge branch 'release/v3.2' 2016-02-05 00:10:41 -08:00
James R. Barlow 7e5c37137b Merge branch 'develop' into release/v3.2 2016-02-04 23:42:06 -08:00
James R. Barlow 1aae11714b Update release notes for v3.2 2016-02-04 23:41:33 -08:00
James R. Barlow d82f14a7aa Update .gitignore 2016-02-04 18:51:41 -08:00
James R. Barlow 4b65e0b093 Set JPEG output quality to 95 for better transcoding 2016-02-04 18:49:09 -08:00
James R. Barlow 43b0faa830 Bug in tesseract_noop spoof: produced wrong page sizes
Now checks input image to ensure the implied page size of its .hocr file
matches the rest of the PDF.
2016-02-04 18:48:22 -08:00
James R. Barlow 8674c9fb20 Merge commit 'ccfbb54e8c26784e438ba2fcac2179f21e7d857b' into release/v3.2 2016-02-04 17:39:36 -08:00
jbarlow83 ccfbb54e8c Update release notes for v3.2
Fix the notes
2016-02-04 17:37:30 -08:00
James R. Barlow 9893ebf889 Suppress tesseract argument printout 2016-02-04 17:26:36 -08:00
James R. Barlow 303eb3e93a Merge commit 'ca546d70e5bff9e9b115371f7813f3c326822bd8' into release/v3.2 2016-02-04 17:25:56 -08:00
jbarlow83 ca546d70e5 Merge pull request #45 from spwhitton/hocrtransform-shebang-fix
fix shebang in hocrtransform.py
2016-02-04 17:21:33 -08:00
Sean Whitton 6a5ea2d64a fix shebang in hocrtransform.py 2016-02-03 17:48:35 -07:00
James R. Barlow ec3d92ad8e Reorg gitignore 2016-01-30 15:28:24 -08:00
James R. Barlow 66a095d7de Improve organization of CFFI setup 2016-01-30 15:19:40 -08:00
James R. Barlow 411981efbc Experiment with CFFI instead of ctypes 2016-01-30 15:06:25 -08:00
James R. Barlow 350ad5210e Leptonica: convert to CFFI 2016-01-20 15:03:07 -08:00
James R. Barlow f3b588764e Suppress tesseract argument printout 2016-01-20 15:02:48 -08:00
James R. Barlow b49f5a7d77 Support optionally using leptonica to deskew
unpaper doesn't seem to be good at deskewing. It fails on test case
with a lot of italics. I think it also struggles on pages with a lot
of whitespace. Leptonica continues to shine here.

However, this is only a first crack at Leptonica. The leptonica module
should be redone to use cffi (more extensible).

Also considering the possibility of making all Lept calls in a forked
process to insulate the calling process from C code crashes and the
messy redirect of stdout/stderr to read Leptonica's errors.

I don't think the redirect is a huge problem as long as multiprocesses
rather than multithreads are used. The ruffus child process that is
handling a page is single threaded and will not be affected by the
redirection. It just feels dirty. The main reason to consider a child
process is crash isolation.
2016-01-19 17:43:40 -08:00
James R. Barlow bacbcba58a Merge branch 'release/v3.2-rc1' 2016-01-19 16:58:37 -08:00
James R. Barlow 52e8aa434f Update release notes for v3.2-rc1 2016-01-19 16:49:49 -08:00
James R. Barlow 37c508f3f8 Better versioning: no silly version files, but wrong ver in development
Small price to pay.
2016-01-19 16:07:52 -08:00
James R. Barlow 26e36422cc More fiddling with version 2016-01-19 15:07:21 -08:00
James R. Barlow f82cb002bc Try automatic versioning with setuptools_scm 2016-01-19 13:27:18 -08:00
James R. Barlow c1eb047a4b Fix name of pdfa_def.ps
Used to include a copy of the parent dir's name.
2016-01-19 13:11:03 -08:00
James R. Barlow 626ca18f5c Remove stale comment 2016-01-19 13:02:35 -08:00
James R. Barlow 9058dedfbe New tests for ccitt, jbig2 encodings 2016-01-19 13:01:56 -08:00
James R. Barlow a0952bfca3 Optimize: use img2pdf stream instead of repeated copies 2016-01-18 20:24:46 -08:00
James R. Barlow 354e61946e Use os.makedirs for test output directories
Broke Travis
2016-01-16 02:47:56 -08:00
James R. Barlow fd6d1d748a Merge branch 'feature/pypdf-page-merge' into develop 2016-01-16 02:33:23 -08:00
James R. Barlow 360acd1e2c Adjust test_oversample test case
Add -f to force generation of the background image at the desired
oversample resolution.  Our new behavior is to only send the oversampled
image to Tesseract while leaving the main page intact unless asked to
deskew, clean, etc.
2016-01-15 15:55:23 -08:00
James R. Barlow fc0479f110 Fix all but test_oversample[hocr] 2016-01-15 15:46:47 -08:00
James R. Barlow 62728205b6 Implement image+text merging in other cases
5 failed, 28 passed

failures:
test_oversample[hocr], test_skip_ocr, test_skip_big, test_maximum_options[hocr],
test_blank_input_pdf,
2016-01-15 15:38:08 -08:00
James R. Barlow dc0fb25e64 Render hocr page: no longer needs an image as input 2016-01-15 15:16:47 -08:00
James R. Barlow f3e04cce56 Update pipeline.svg 2016-01-15 14:56:16 -08:00
James R. Barlow 7067110308 Add safety check to prevent merge from running when not sensible 2016-01-15 14:54:45 -08:00
James R. Barlow 599d889703 Implement "perfect reconstruction" - transfer page and watermark OCR layer
Works, does not account for changes to clean/deskew, etc.
Surprisingly, it works. PyPDF2 fixes since last attempt?
2016-01-15 14:39:12 -08:00
James R. Barlow 2fa8366632 Merge branch 'feature/test-pageinfo-cleanup' into develop 2016-01-15 14:18:01 -08:00
James R. Barlow c368c51bad New hocrtransform test 2016-01-15 14:14:08 -08:00
James R. Barlow 7c558b3713 Move pageinfo test into tests folder 2016-01-11 17:40:44 -08:00
James R. Barlow 8d323ae510 Merge branch 'feature/pagesegmode' into develop 2016-01-11 17:23:00 -08:00
James R. Barlow 3b53e9adac Use tesseract cache for -psm 2016-01-11 17:22:50 -08:00
James R. Barlow 074c1d71b4 Activate --tesseract-pagesegmode 2016-01-11 17:19:32 -08:00
James R. Barlow 1fca9a004d Adjust command line parameters
Was splitting each argument to --tesseract-config into a list of single
character strings
2016-01-11 16:57:19 -08:00
James R. Barlow b485a1ef78 Override ruffus' handling of --jobs
Ruffus treats omitted parameter as -j1. For our purposes it makes more
sense for omitting the parameter to mean "use all CPUs". As such we
must be able to distinguish -j1 from the parameter -j being omitted.

Telling ruffus to ignore the argument actually just makes it not auto
generate the argument. We can add an argument back with the same name.
2016-01-09 19:07:48 -08:00
James R. Barlow 326ef7a3ac Merge branch 'hotfix/v3.1.1' into develop
# Conflicts:
#	RELEASE_NOTES.rst
2016-01-09 18:55:04 -08:00
James R. Barlow 12bc58b5b6 Merge branch 'hotfix/v3.1.1' 2016-01-09 18:45:40 -08:00
James R. Barlow 6af0815681 Bump version 2016-01-09 18:45:06 -08:00
James R. Barlow 66c2b9b78e Merge branch 'hotfix/v3.1.1' into develop 2016-01-09 18:38:09 -08:00
James R. Barlow d03c056cb1 Supporting all languages bloats the image by an extra 1 GB
Make it a special image
2016-01-04 16:49:06 -08:00
James R. Barlow 3f94d628fa Dockerfile: remove manual build of unpaper
Fortunately unpaper now exists as binary package, eliminating the need
to install all of the build machinery and build it from source.
2016-01-04 15:07:12 -08:00
James R. Barlow a64c7dbe99 Update dockerfile: include all languages
Also update ignore files
2016-01-04 14:27:16 -08:00
James R. Barlow 61b3ccb57c Place ruffus database in temporary folder
Because we don't really use ruffus checkpoint feature, putting the
database in a permanent location does not help anything, but does cause
large database files and problems if the .ruffus_history.sqlite wanted
to be in a writable location.
2016-01-04 13:23:47 -08:00
James R. Barlow 424b4b33b1 Just go right ahead and demand Python 3.4 2016-01-04 12:56:51 -08:00
James R. Barlow e510f89792 Python 2 warning message 2015-12-21 09:38:38 -08:00
James R. Barlow 49cd6cc619 Off by one error in page info calculation 2015-12-21 09:35:02 -08:00
James R. Barlow 9aa3d340d4 Tell Travis about the cache 2015-12-17 14:02:13 -08:00
James R. Barlow 09782242c8 Adjust test cases to use cache and noop more effectively
This reduces total execution time to 164s on my machine, down from
about double that.
2015-12-17 14:00:17 -08:00
James R. Barlow 9ec4aa039d Add tesseract caching to speed up tests 2015-12-17 12:52:12 -08:00
James R. Barlow ecebe2f24b Let some tests use the spoofed tesseract
Where getting OCR doesn't matter
2015-12-17 11:56:09 -08:00
James R. Barlow 7313a77c2a Implement pdf renderer side of tess spoof 2015-12-17 11:41:54 -08:00
James R. Barlow 45113676a3 Add Tesseract spoofing 2015-12-17 11:36:47 -08:00
James R. Barlow 102bd07019 Check for encrypted PDF and complain appropriately 2015-12-17 10:37:54 -08:00
James R. Barlow 9622e31da9 Use envvars in a new test case
And get rid of the messy binary replacement spoofing
2015-12-17 09:29:01 -08:00
James R. Barlow 1731ce2a44 Environment variables can now override default programs 2015-12-17 09:05:10 -08:00
James R. Barlow 276f421c44 Did a quick test of Ghostscript vs QPDF at PDF page splitting
qpdf won so hard it wasn't funny, even though it must be called once
per page to do the job. Perhaps Ghostscript interprets it as a call to
render the page?

time bash qpdf-test.fish ../tests/resources/multipage.pdf
        0.07 real         0.02 user         0.03 sys

time gs -sDEVICE=pdfwrite -dSAFER -o '%06d.pdf' ../tests/resources/multipage.pdf
        5.12 real         5.06 user         0.04 sys
2015-12-17 08:49:08 -08:00
James R. Barlow 133357779a All subprocess invocations refactored out of main.py 2015-12-17 08:31:18 -08:00
James R. Barlow 5d8167b232 Move PDF validation check to qpdf.py 2015-12-17 08:28:00 -08:00
James R. Barlow e76ae8c46c Move more qpdf calls into qpdf.py 2015-12-17 08:24:48 -08:00
James R. Barlow 53a7c0e668 Refactor qpdf subprocess calls into module 2015-12-17 08:19:53 -08:00
James R. Barlow 4ca243e490 Merge commit '9f374461559460527e47237323e511123f31b6b0' into feature/envvars 2015-12-17 07:27:26 -08:00
jbarlow83 9f37446155 Merge pull request #34 from shemgp/master
Don't exit when qpdf repairs the file successfully but displays warning
2015-12-16 20:46:47 -08:00
Shem Pasamba d7c7559b05 Use boolean instead of integers 2015-12-17 11:23:27 +08:00
Shem Pasamba b2b66d1344 Don't exit when qpdf repair was successful 2015-12-17 11:20:20 +08:00
James R. Barlow 5d111a3c04 Refactor tesseract --pdfrenderer calls to tesseract.py 2015-12-16 17:48:26 -08:00
James R. Barlow 10416f847f Migrate tesseract-hocr code to tesseract module, because modularity 2015-12-16 17:36:11 -08:00
James R. Barlow 79b3472b26 All tests passed, bump version 2015-12-04 04:31:01 -08:00
James R. Barlow f1b2f1ae08 Merge branch 'feature/pdfa-2' into develop 2015-12-04 04:04:08 -08:00
James R. Barlow ee7d97ae8c Trivial 2015-12-04 04:03:38 -08:00
James R. Barlow 7d9f473bb1 Remove eval() call by introspecting ExitCode 2015-12-04 03:34:53 -08:00
James R. Barlow e77a5e5e75 We don't want threads. Really. Do. Not. Want. 2015-12-04 03:11:38 -08:00
James R. Barlow 6ab19af122 Comments 2015-12-04 03:09:39 -08:00
James R. Barlow 276fe49867 Better error messages for input file not found or invalid
Not as good finding a general way to deal with ruffus exceptions, but
better than nil.
2015-12-04 03:07:53 -08:00
James R. Barlow acb31abe86 Fix issue #20 - fails on uppercase .PDF 2015-12-04 02:14:09 -08:00
James R. Barlow 4f964a3c8a Introduce --pdf-renderer auto
Tess 3.03's has various quality problems like wrong DPI that are fixed
in Tess 3.04. Idea here is to introduce an option to let OCRmyPDF
select the rendering backend based on the options and system.

However, we're not ready for tesseract as the main renderer.
Setting pdf-renderer to tesseract does not pass all test cases, mainly
the one where --tesseract-timeout is triggered, and some others.
2015-12-02 23:20:31 -08:00
James R. Barlow df1fda7438 pageinfo: workaround PyPDF extractText limitations on hidden text
It appears that extractText() does not find all text. At a glance it
may be that Tesseract's PDF renderer generates a font and uses glyphs
that map to different Unicode code points that PyPDF expects, so it
discards the content and finds nothing. As a proxy in lieu of better
PDF parsing, assume that a "GlyphLessFont" means there is a text there.

I had previously found it does not work to check for the presence of a
font on page. Some PDF generators create a font resource entry even if
the font is never called for.
2015-12-02 23:16:36 -08:00
James R. Barlow d6124c1787 pageinfo: improve robustness of text test for Tesseract produced PDFs 2015-12-02 03:12:52 -08:00
James R. Barlow 80d89b5420 Set /Creator metadata to OCRmyPDF
with reference to Tess version and settings
2015-12-02 02:19:39 -08:00
James R. Barlow 74059eecf1 Choose PDF/A-2b by default instead of A-1b 2015-12-02 01:48:10 -08:00
James R. Barlow 78697341a2 pytest: don't run tests that happened to be part of pyvenv 2015-12-02 01:19:43 -08:00
James R. Barlow cfb56dd8ff Merge commit 'b1769cbe18e6380ddfe96b3b22e6d02cb603338b' into develop 2015-12-01 00:40:43 -08:00
jbarlow83 b1769cbe18 README: El Capitan supported now, Py3.5 supported 2015-11-26 16:31:33 -08:00
James R. Barlow 955b801e7f Merge branch 'master' into develop 2015-09-14 00:34:21 -07:00
James R. Barlow 3cea3f1afe Try to work around git binary file bug again 2015-09-14 00:34:16 -07:00
James R. Barlow fd4a227ccb Force this file to stop thinking it was modified 2015-09-13 17:53:01 -07:00
James R. Barlow 19c3097483 Update notes 2015-09-13 17:51:18 -07:00
James R. Barlow cdd1a6d03c Suppress failing test 2015-09-10 07:01:14 -07:00
James R. Barlow 5fb8411571 Try new PPA for libav 2015-09-10 06:01:59 -07:00
James R. Barlow 334a15b8c7 typo fix 2015-09-10 05:01:44 -07:00
James R. Barlow 6390736577 ffmpeg-dev instead? 2015-09-10 04:27:57 -07:00
James R. Barlow d55a214516 Autoreconf? 2015-09-10 04:10:12 -07:00
James R. Barlow 0994164b9a travis: apt-get install in wrong place 2015-09-06 01:43:47 -07:00
James R. Barlow 54ee0dd147 travis: fix typo 2015-09-06 01:39:54 -07:00
James R. Barlow 47c7990fb3 travis: build unpaper with cache 2015-09-06 01:38:01 -07:00
James R. Barlow 997e95de4d travis: build unpaper 2015-09-06 01:29:07 -07:00
James R. Barlow 44204be256 Fix order of PPAs 2015-09-06 00:54:50 -07:00
James R. Barlow 9b1d9aa88a travis: improve, add new PPA, etc. 2015-09-06 00:41:23 -07:00
James R. Barlow b775762f6a travis: doesn't like gcc-4.8, try just gcc 2015-09-06 00:23:05 -07:00
James R. Barlow df1a28e319 Travis needs sudo mode 2015-09-06 00:21:20 -07:00
James R. Barlow c300b2802a travis: tabs -> spaces 2015-09-06 00:08:25 -07:00
James R. Barlow 01040ace4c More complete travis.yml 2015-09-06 00:02:58 -07:00
James R. Barlow 8367172e0b Start setting up Travis CI 2015-09-05 23:44:43 -07:00
James R. Barlow 09afd8d25d Move to my repo: github.com/fritz-hh => jbarlow83
I made several efforts to contact fritz but he is no longer
communicating, and to set up Github integrations with Docker and Travis
CI I need admin access. Which I don't have. So I'm moving it to my own
and aiming the old one at me.
2015-09-05 01:14:54 -07:00
James R. Barlow 7ed60429b3 Test case: No longer using JHOVE
So JHOVE will not claim this is an invalid PDF and we should see it
reported as valid.
2015-09-05 01:12:33 -07:00
James R. Barlow 281eafada0 bump to v3.0 and move repos 2015-09-05 00:53:14 -07:00
James R. Barlow c14e10128a Bump version to -rc9 2015-08-29 16:43:22 -07:00
James R. Barlow 3270635192 ghostscript: quiet startup on rasterize 2015-08-28 04:51:36 -07:00
James R. Barlow 3d26257710 Add test cases for additional image formats 2015-08-28 04:51:11 -07:00
James R. Barlow c4f134d694 Prevent running validation on missing file after an exception is thrown 2015-08-28 04:48:29 -07:00
James R. Barlow 83f9dfbac4 Use png256 raster device when possible
Someone reported a bug where the .png input to unpaper ended up being
type 'P' (palette) for some reason, which was not supported in unpaper.

Not sure how it happened, but seemed easier to fix by explicitly
supporting. Here we use png256 if it would capture all colors in the
input file. It's up to tesseract/reportlab to make use of the palette
PNG when rendering.
2015-08-28 04:47:57 -07:00
James R. Barlow 3a445ad5f7 unpaper: support paletted files by conversion instead of bailing 2015-08-28 04:44:26 -07:00
James R. Barlow c6d106ec33 Throw exception if iccprofiles not found instead of returning None
So far iccprofiles were only missing for a user who had a custom and
possibly broken ghostscript installation.
2015-08-28 03:59:35 -07:00
James R. Barlow 2ce6834be4 Bump to -rc8 2015-08-24 01:25:01 -07:00
James R. Barlow b376672dbc Bug fix: exception thrown if input PDF was missing DocumentInfo block 2015-08-24 01:23:30 -07:00
James R. Barlow d07db8547f Merge branch 'master' of https://github.com/fritz-hh/OCRmyPDF 2015-08-23 12:30:46 -07:00
James R. Barlow aab08bfcc7 Fix requirements.txt problem 2015-08-23 12:30:40 -07:00
jbarlow83 e0a25494ee Explain the need for multi core, etc 2015-08-22 13:34:42 -07:00
James R. Barlow fd876d5e4e Merge branch 'develop' 2015-08-22 01:51:44 -07:00
James R. Barlow ee7f008ff5 Require unpaper 6.1; no messing around with broken versions 2015-08-22 01:51:08 -07:00
jbarlow83 d9161a6ddb Update README: docker run instructions 2015-08-22 01:50:13 -07:00
jbarlow83 f8d66768e3 Update README with docker install instructions 2015-08-22 01:33:12 -07:00
James R. Barlow 4f3673d14d Update notes for -rc6 2015-08-22 00:40:07 -07:00
James R. Barlow 1712fdb74a Merge branch 'feature/docker-debian' 2015-08-22 00:32:27 -07:00
James R. Barlow 3a5ffc79e0 Stock debian unpaper is no good; replace with 6.1 built from source
debian and ubuntu both install unpaper 0.4.2 or so. No .deb packages
available at higher version numbers although ArchLinux had something.
Considered making a separate image to handle building and install but
decided that was a premature optimization at this point, so just build
the unpaper that works. All tests pass.
2015-08-22 00:30:39 -07:00
James R. Barlow 859b063444 Fixup other docker test suite errors
Outstanding failures:
test_pageinfo::test_jpeg
tests involving unpaper due to version <6.1 failures
2015-08-20 02:37:03 -07:00
James R. Barlow bd61e7c644 dockerignore *.pyc
https://github.com/docker/docker/issues/13113
Docker kinda sucks. No recursive exclusion.
2015-08-20 02:27:07 -07:00
James R. Barlow c9abf282b5 Set docker locale to utf-8
Shocked, shocked, that there's a Linux distribution out that there isn't
doing the right thing and setting up utf-8 by default. (Many tests failed)
2015-08-20 01:44:30 -07:00
James R. Barlow 9dad40b5a3 Major overhaul of the Dockerfile
Switched from Ubuntu to debian:stretch because stretch has more recent
versions of our binary packages and starts smaller.  In particular,
stretch has both pillow==2.9.0 and reportlab==3.2.0 available as system
packages which saves the considerable hassle of install a toolchain.

Instead, a pyvenv is set up with access to system's site-packages (note:
needs two steps), making the binary-dependent packages available.  Then
the remaining packages are installed into the pyvenv with --no-cache-dir
to avoid saving files. And there we are.

Image is still very large (>500 MB), but programs like reportlab require
font rendering capabilities so they pull in large portions of the Linux
graphics stack. Not much will shrink that.
2015-08-20 01:25:31 -07:00
James R. Barlow 8e2d690cb0 Rework Dockerfile, setup.py to work with wheels for better cache use 2015-08-19 13:43:32 -07:00
James R. Barlow c132e091e1 Dockerfile: use local copy of application 2015-08-19 13:10:58 -07:00
James R. Barlow 630e6cbf1e pip chokes on Unicode filenames? 2015-08-18 23:56:30 -07:00
James R. Barlow 83ff5760a8 Dockerfile comment cleanup 2015-08-18 23:41:41 -07:00
James R. Barlow fed0ee638e Fix ruffus writing to RO directory in container 2015-08-18 23:30:06 -07:00
James R. Barlow cc161780df Replace fileinput with regular open-replace
fileinput is supposed to save time in these cases but it's not capable
of doing both in-place rewrites and working with a non-ascii encoding.
This was not noticed until characters outside of ASCII were picked up
by tesseract and saved in a HOCR file. Rework some surrounding code as
well and add multilingual test cases.
2015-08-18 23:27:50 -07:00
James R. Barlow 898b2b000a Works 2015-08-18 05:38:05 -07:00
James R. Barlow b3ee743ed7 WIP on docker 2015-08-18 04:46:25 -07:00
James R. Barlow ef17b669fe README needs ghostscript 2015-08-18 03:27:39 -07:00
James R. Barlow 2dff3e07ce Drop libxml2 dependency
It seems that Python's internal XML parser is good enough to do the job.
2015-08-17 15:26:07 -07:00
James R. Barlow 53c88093ad Bump to -rc5 2015-08-16 02:19:04 -07:00
James R. Barlow 0ec13d3a17 Fix test cases: minor issues
-os.environ directly modified when whole suite run, breaking subsequent
tests
-no longer trusting JHOVE for PDF/A validation
2015-08-16 01:57:35 -07:00
jbarlow83 0d5104049a Update README with better install instructions 2015-08-16 01:28:28 -07:00
James R. Barlow ce8fa69785 Update readme 2015-08-16 00:59:57 -07:00
James R. Barlow 30072e0c70 Pillow sucks
Far from being fluffy or friendly, Pillow silently allows installation
of itself without support for major image types.  Reportlab calls for
pillow 2.4.0.  On Ubuntu 14.04 LTS this will trigger an upgrade of
pillow that will be built without JPEG or ZLIB so it is effectively
neutered, and unfortunately Pillow will not detect this situation at
install time and guide users to a resolution.  Instead, you see nasty
stack traces.

So add a run-time check to ensure that Pillow is sane and capable of JPEG
and PNG support since both may be used internally.
2015-08-16 00:54:03 -07:00
James R. Barlow eb04a890b2 Relax Pillow requirement for Ubuntu 14.04 LTS 2015-08-15 15:55:56 -07:00
James R. Barlow 0c53adb04f setup: rollback lxml version to 3.3.3 - that's the latest in Ubuntu 14.04 2015-08-15 15:25:58 -07:00
James R. Barlow ee5a43fd47 setup: suppress jhove errors 2015-08-15 15:25:30 -07:00
James R. Barlow c43d6c2cbe Merge branch 'develop' of https://github.com/fritz-hh/OCRmyPDF into develop
Conflicts:
	setup.py
2015-08-15 15:18:41 -07:00
James R. Barlow 87aeeacb04 Fix erroneous instruction to "apt-get install tesseract"
Should be tesseract-ocr
2015-08-15 15:17:38 -07:00
James R. Barlow 6b26e9cad6 Fix erroneous instruction to "apt-get install tesseract"
Should be tesseract-ocr
2015-08-15 15:12:05 -07:00
James R. Barlow 85af0f0d03 Add test case for blank PDF page 2015-08-14 00:46:50 -07:00
James R. Barlow f6f4705ea3 Remove Java from setup.py 2015-08-14 00:44:56 -07:00
James R. Barlow a4702bff22 Possible fix for issue #111 2015-08-13 23:10:22 -07:00
James R. Barlow 73c5c48f79 Update notes 2015-08-13 23:08:29 -07:00
James R. Barlow adf495e8cc Remove JHOVE
JHOVE is not an effective PDF/A validator, as detailed in this article:
http://www.pdfa.org/2014/12/ensuring-long-term-access-pdf-validation-with-jhove/

In short, it's buggy. Out of 670 invalid PDF/A files in a test suite,
it only flagged 5.  It only looks for certain problems that Ghostscript
generated PDFs are unlikely to have.  So use qpdf as a final check for
general ill-formed PDF problems since it is quite reliable.

JHOVE 1 is no longer maintained. There's a JHOVE 2 but it has no PDF
support.  I also don't know if it's appropriate to bundle JHOVE, with an
LGPL, under this project and its current license.

Removing a dependency on Java is a huge win.  A world with less Java is
a world with less AbstractFactoryConstructorInterfaces.
2015-08-11 15:31:32 -07:00
James R. Barlow 9247ea00bf Improve ruffus exception handling
ruffus swallows the return code if the process of handling an exception
we hit an error in ruffus' own code, which can happen.  So pick through
its error stack and find out if there's an interesting return code in
there.  Had to use eval() of all things.

Also suppress the stack trace for normal error conditions that don't
need one.
2015-08-11 02:19:46 -07:00
James R. Barlow a1238d7bf9 Document override binary test 2015-08-11 00:44:43 -07:00
James R. Barlow 2d63268f0f Work around JHOVE bug for now, so that the test passes 2015-08-11 00:23:48 -07:00
James R. Barlow 1cb5f6a90d Refactor exit codes; test for missing tessdata
Some versions of tesseract installed by homebrew end up without a
functional tessdata folder, and tesseract is not helpful in this
situation, so add a new test to make sure our output is at least
indicative of the problem.

In the process of properly handling return codes I discovered
test_override_metadata triggers a NPE inside JHOVE probably due to the
Unicode character checking.  This could be specific to my JRE (1.6.0_65,
Oracle) but it's probably JHOVE's fault.  A valid PDF/A (per Acrobat)
is still generated.
2015-08-11 00:17:02 -07:00
James R. Barlow 8d848284df Fix code, test case: complain when GS fails to produce PDF/A
Modified pipeline to fix regression and return the proper error code if
we did not produce a PDF/A as expected.  The wrapper forces the output
to be PDF 1.3 which is not PDF/A compliant.

The funny thing is that in some cases JHOVE incorrectly states that a
file is PDF/A-1b compliant, well formed and valid, even when it is not
according to Acrobat XI and is missing the PDF/A metadata marker, as
far as I can tell.  JHOVE may not be as beneficial as hoped.
2015-08-10 16:05:00 -07:00
James R. Barlow 8fe54d1a5c Add new test case to check invalid PDF/A case
It revealed a regression - return code not the same as v2.x for invalid
PDF/A.  It's also not easy to get the return code out of ruffus.  Will
need to tweak the final step of the pipeline.
2015-08-10 13:57:28 -07:00
James R. Barlow 11dd9f14c3 setup.py: block unsafe 'upload', say to use twine instead 2015-08-09 14:16:30 -07:00
519 changed files with 51110 additions and 42629 deletions
+90
View File
@@ -0,0 +1,90 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
FROM ubuntu:22.04 as base
ENV LANG=C.UTF-8
ENV TZ=UTC
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
RUN apt-get update && apt-get install -y --no-install-recommends \
python3 \
libqpdf-dev \
zlib1g \
liblept5
FROM base as builder
# Note we need leptonica here to build jbig2
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential autoconf automake libtool \
libleptonica-dev \
zlib1g-dev \
python3-dev \
python3-distutils \
libffi-dev \
ca-certificates \
curl \
git \
libcairo2-dev \
pkg-config
# Get the latest pip (Ubuntu version doesn't support manylinux2010)
RUN \
curl https://bootstrap.pypa.io/get-pip.py | python3
# Compile and install jbig2
# Needs libleptonica-dev, zlib1g-dev
RUN \
mkdir jbig2 \
&& curl -L https://github.com/agl/jbig2enc/archive/ea6a40a.tar.gz | \
tar xz -C jbig2 --strip-components=1 \
&& cd jbig2 \
&& ./autogen.sh && ./configure && make && make install \
&& cd .. \
&& rm -rf jbig2
COPY . /app
WORKDIR /app
RUN pip3 install --no-cache-dir .[test,webservice,watcher]
FROM base
# For Tesseract 5
RUN apt-get update && apt-get install -y --no-install-recommends \
software-properties-common gpg-agent
RUN add-apt-repository -y ppa:alex-p/tesseract-ocr-devel
RUN apt-get update && apt-get install -y --no-install-recommends \
ghostscript \
fonts-droid-fallback \
jbig2dec \
img2pdf \
libsm6 libxext6 libxrender-dev \
pngquant \
python-is-python3 \
tesseract-ocr \
tesseract-ocr-chi-sim \
tesseract-ocr-deu \
tesseract-ocr-eng \
tesseract-ocr-fra \
tesseract-ocr-por \
tesseract-ocr-spa \
unpaper \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY --from=builder /usr/local/lib/ /usr/local/lib/
COPY --from=builder /usr/local/bin/ /usr/local/bin/
COPY --from=builder /app/misc/webservice.py /app/
COPY --from=builder /app/misc/watcher.py /app/
# Copy minimal project files to get the test suite.
COPY --from=builder /app/pyproject.toml /app/README.md /app/
COPY --from=builder /app/tests /app/tests
ENTRYPOINT ["/usr/local/bin/ocrmypdf"]
+83
View File
@@ -0,0 +1,83 @@
# SPDX-FileCopyrightText: 2023 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
FROM alpine:3.19.1 as base
ENV LANG=C.UTF-8
ENV TZ=UTC
RUN apk add --no-cache \
python3 \
zlib
FROM base as builder
RUN apk add --no-cache \
ca-certificates \
git \
python3-dev \
py3-pip
# On arm64, we need to build cffi from source.
ARG TARGETPLATFORM
RUN if [ "${TARGETPLATFORM}" == "linux/arm64" ]; then \
apk add --no-cache \
build-base \
autoconf \
automake \
libtool \
zlib-dev \
libffi-dev \
cairo-dev \
pkgconfig \
; \
fi
COPY . /app
WORKDIR /app
RUN python3 -m venv .venv
RUN source .venv/bin/activate \
&& python3 -m pip install --no-cache-dir --upgrade pip \
&& python3 -m pip install --no-cache-dir wheel \
&& python3 -m pip install --no-cache-dir .[test,webservice,watcher]
FROM base
RUN apk add --no-cache \
ghostscript \
jbig2dec \
jbig2enc \
pngquant \
tesseract-ocr \
tesseract-ocr-data-chi_sim \
tesseract-ocr-data-deu \
tesseract-ocr-data-eng \
tesseract-ocr-data-fra \
tesseract-ocr-data-osd \
tesseract-ocr-data-por \
tesseract-ocr-data-spa \
ttf-droid \
unpaper \
&& rm -rf /var/cache/apk/*
WORKDIR /app
COPY --from=builder /usr/local/lib/ /usr/local/lib/
COPY --from=builder /usr/local/bin/ /usr/local/bin/
COPY --from=builder /app/.venv/ /app/.venv/
COPY --from=builder /app/misc/webservice.py /app/
COPY --from=builder /app/misc/watcher.py /app/
# Copy minimal project files to get the test suite.
COPY --from=builder /app/pyproject.toml /app/README.md /app/
COPY --from=builder /app/tests /app/tests
ENV PATH="/app/.venv/bin:${PATH}"
ENTRYPOINT ["/app/.venv/bin/ocrmypdf"]
+47
View File
@@ -0,0 +1,47 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
# dotfiles
.*
!.coveragerc
!.dockerignore
!.git_archival.txt
!.gitattributes
!.gitignore
!.pre-commit-config.yaml
!.readthedocs.yml
# Dev scratch
*.ipynb
**/*.pyc
/*.pdf
/*.qdf
/*.png
/scratch.py
IDEAS
log/
tests/resources/private/
tmp/
venv*/
/debug_tests.py
*.traineddata
/private
# Package building
*.egg-info/
build/
dist/
wheelhouse/
pip-wheel-metadata/
# Code coverage
htmlcov/
# Docker specific
bin/
docs/
include/
lib/
# Docker include .git/
!.git/
+4
View File
@@ -0,0 +1,4 @@
node: $Format:%H$
node-date: $Format:%cI$
describe-name: $Format:%(describe:tags=true)$
ref-names: $Format:%D$
+10 -1
View File
@@ -1,3 +1,6 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
# Always use Unix convention for new lines
* text eol=lf
@@ -5,4 +8,10 @@
# (binary is a macro for -text -diff)
*.jar binary
*.pdf binary
*.PDF binary
*.PDF binary
*.png binary
*.jpg binary
*.bin binary
*.afdesign binary
.git_archival.txt export-subst
+15
View File
@@ -0,0 +1,15 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
# These are supported funding model platforms
github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
patreon: # Replace with a single Patreon username
open_collective: james-barlow
ko_fi: # Replace with a single Ko-fi username
tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
liberapay: # Replace with a single Liberapay username
issuehunt: # Replace with a single IssueHunt username
otechie: # Replace with a single Otechie username
custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']
@@ -0,0 +1,72 @@
name: Installation, packaging, dependencies
description: Installation, packages, dependencies, "nothing works", test suite failures...
title: "[Bug]: "
labels: ["triage"]
assignees:
- jbarlow83
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to fill out this bug report!
If your issue involves using OCRmyPDF on specific file(s) and not getting
good results, this is the *wrong* issue template. Please use the recommended
template to ensure we have enough information to help.
- type: textarea
id: what-happened
attributes:
label: What were you trying to do?
description: Also tell us, what did you expect to happen?
placeholder: Tell us what you see!
validations:
required: true
- type: dropdown
id: packaging-system
attributes:
label: Where are you installing/running from?
multiple: true
options:
- PyPI (pip, poetry, pipx, etc.)
- Linux package manager (apt, dnf, etc.)
- Wndows package manager (chocolatey, etc.)
- Homebrew
- Docker container
- Ubuntu snap
- Conda
- source build
validations:
required: true
- type: input
id: version
attributes:
label: OCRmyPDF version
description: Paste "ocrmypdf --version" here
- type: dropdown
id: operating-system
attributes:
label: What operating system are you working on?
multiple: true
options:
- Linux
- Windows
- macOS
- BSD
- type: input
id: os_version
attributes:
label: Operating system details and version
- type: checkboxes
attributes:
label: Simple sanity checks
description: Select all that apply
options:
- label: Operating system is currently supported by its vendor (not end of life)
- label: Python version is compatible with OCRmyPDF
- label: This issue is not about a specific input file
- type: textarea
id: logs
attributes:
label: Relevant log output
description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
render: plain text
@@ -0,0 +1,75 @@
name: Problem with specific file
description: Something went wrong while trying to OCR a specific file
title: "[Bug]: "
labels: ["triage"]
assignees:
- jbarlow83
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to describe this issue with a particular file.
- type: textarea
id: what-happened
attributes:
label: Describe the bug
description: A clear and concise description of what the bug is.
placeholder: Tell us what you see!
validations:
required: true
- type: textarea
id: reproduce
attributes:
label: Steps to reproduce
description: Please include steps to reproduce.
value: |
1. Run ocrmypdf -v1 ...arguments... input.pdf output.pdf
2. Open output.pdf
3. ...
render: plain text
- type: textarea
id: files
attributes:
label: Files
description: |
Please attach the input and output files, or any screenshots that may be helpful.
If you cannot provide a test file, we probably won't be able to help with the issue.
PDF is a complex file format, and there may be technical details in the PDF that are
causing the issue. There's really no substitute for a test file.
We understand files may contain personal or sensitive information. Here are some options:
- Try reproducing the issue with a file from the OCRmyPDF test suite. (See tests/resources)
- Try to create another file in the same way as your private file.
- Encrypt the file to OCRmyPDF's private GPG key, and then zip the GPG file.
- Use ``qpdf --json yourfile.pdf`` to produce a JSON representation of your file that
omits personal information.
placeholder: |
Drag and drop files here.
- type: dropdown
id: packaging-system
attributes:
label: How did you download and install the software?
multiple: true
options:
- PyPI (pip, poetry, pipx, etc.)
- Linux package manager (apt, dnf, etc.)
- Windows package manager (chocolatey, etc.)
- Homebrew
- Docker container
- Ubuntu snap
- Conda
- source build
- type: input
id: version
attributes:
label: OCRmyPDF version
description: Paste "ocrmypdf --version" here
placeholder: ocrmypdf --version
- type: textarea
id: logs
attributes:
label: Relevant log output
description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
placeholder: Run OCRmyPDF with verbosity `-v1` to get more detailed logging output.
render: plain text
+83
View File
@@ -0,0 +1,83 @@
name: Problem with third party app that uses OCRmyPDF
description: |
For PDF generation issues with third party software such as Paperless-ngx that
uses OCRmyPDF to perform OCR or generate PDFs.
title: "[3rdparty]: "
labels: ["triage"]
assignees:
- jbarlow83
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to describe this issue with a particular file
and third party app.
If you are comfortable using OCRmyPDF, please trying to install OCRmyPDF,
run it on your file, and see if it works. It's easier for everyone
if you can confirm that the issue occurs with OCRmyPDF and not with
the third party app.
- type: checkboxes
attributes:
label: Simple sanity checks
description: Select all that apply
options:
- label: This is an issue with an app that uses OCRmyPDF for OCR
- label: I am using a recent version of the third party app
- label: I will include a file that reproduces the issuse
- type: input
id: thirdparty-app-name-version
attributes:
label: Third party app name and version
description: e.g. Paperless-ngx 2.9.0
- type: textarea
id: what-happened
attributes:
label: Describe the bug
description: A clear and concise description of what the bug is.
placeholder: Tell us what you see!
validations:
required: true
- type: textarea
id: reproduce
attributes:
label: Steps to reproduce
description: Please include steps to reproduce.
value: |
1. Import attached file into Paperless-ngx
2. Trigger OCR
3. Check log file
4. ...
render: plain text
- type: textarea
id: files
attributes:
label: Files
description: |
Please attach the input and output files, or any screenshots that may be helpful.
If you cannot provide a test file, we probably won't be able to help with the issue.
PDF is a complex file format, and there may be technical details in the PDF that are
causing the issue. There's really no substitute for a test file.
We understand files may contain personal or sensitive information. Here are some options:
- Try reproducing the issue with a file from the test suite. (See tests/resources)
- Try to create another file in the same way as your private file.
- Encrypt the file to OCRmyPDF's private GPG key, and then zip the GPG file.
- Use ``qpdf --json yourfile.pdf`` to produce a JSON representation of your file that
omits personal information.
placeholder: |
Drag and drop files here.
- type: input
id: version
attributes:
label: OCRmyPDF version
description: Paste "ocrmypdf --version" here
placeholder: ocrmypdf --version
- type: textarea
id: logs
attributes:
label: Relevant log output
description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
placeholder: Run OCRmyPDF with verbosity `-v1` to get more detailed logging output.
render: plain text
@@ -0,0 +1,12 @@
name: Feature request
description: Suggest an idea for this project
title: "[Feature]: "
labels: ["enhancement", "triage"]
assignees:
- jbarlow83
body:
- type: textarea
id: feature
attributes:
label: Describe the proposed feature
description: A clear and concise description of what the desired is.
+14
View File
@@ -0,0 +1,14 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for all configuration options:
# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
version: 2
updates:
- package-ecosystem: "github-actions" # See documentation for possible values
directory: "/" # Location of package manifests
schedule:
interval: "weekly"
+397
View File
@@ -0,0 +1,397 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
name: Test and deploy
on:
push:
branches:
- main
- ci
- release/*
- feature/*
tags:
- v*
paths-ignore:
- README*
pull_request:
jobs:
test_linux:
name: Test ${{ matrix.os }} with Python ${{ matrix.python }}
runs-on: ${{ matrix.os }}
continue-on-error: ${{ matrix.experimental }}
strategy:
matrix:
experimental: [false]
include:
- os: ubuntu-22.04
python: "3.10"
- os: ubuntu-22.04
python: "3.11"
- os: ubuntu-22.04
python: "3.10"
tesseract5: true
- os: ubuntu-latest
python: "3.12"
tesseract5: true
- os: ubuntu-latest
python: "pypy3.10"
- os: ubuntu-22.04
python: "3.13"
experimental: true
env:
OS: ${{ matrix.os }}
PYTHON: ${{ matrix.python }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: "0" # 0=all, needed for setuptools-scm to resolve version tags
- uses: actions/setup-python@v5
name: Setup Python
with:
python-version: ${{ matrix.python }}
cache: "pip"
allow-prereleases: ${{ matrix.experimental }}
- name: Install Tesseract 5
if: matrix.tesseract5
run: |
sudo add-apt-repository -y ppa:alex-p/tesseract-ocr-devel
- name: Install common packages
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
curl \
ghostscript \
jbig2dec \
img2pdf \
libexempi8 \
libffi-dev \
libsm6 libxext6 libxrender-dev \
pngquant \
poppler-utils \
tesseract-ocr \
tesseract-ocr-deu \
tesseract-ocr-eng \
tesseract-ocr-osd \
unpaper \
zlib1g
- name: Install Ubuntu packages for PyPy
if: startsWith(matrix.python, 'pypy')
run: |
sudo apt-get install -y --no-install-recommends \
libxml2-dev \
libxslt1-dev \
pypy3-dev
- name: Install Python packages
run: |
python -m pip install --upgrade pip wheel
python -m pip install --prefer-binary .[test]
- name: Report versions
run: |
tesseract --version
gs --version
pngquant --version
unpaper --version
img2pdf --version
- name: Test
run: |
python -m pytest --cov-report xml --cov=ocrmypdf --cov=tests/ -n0 tests/
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
with:
files: ./coverage.xml
env_vars: OS,PYTHON
test_macos:
name: Test macOS
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [macos-latest, macos-13] # macos-latest is arm64, macos-13 is x86_64
python: ["3.10", "3.11", "3.12"]
env:
OS: ${{ matrix.os }}
PYTHON: ${{ matrix.python }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: "0" # 0=all, needed for setuptools-scm to resolve version tags
- name: Install Homebrew deps
continue-on-error: true
run: |
brew update
brew install \
exempi \
ghostscript \
jbig2enc \
openjpeg \
pngquant \
tesseract
- uses: actions/setup-python@v5
name: Setup Python
with:
python-version: ${{ matrix.python }}
cache: "pip"
- name: Install Python packages
run: |
python -m pip install --upgrade pip wheel
python -m pip install --prefer-binary .[test]
- name: Report versions
run: |
tesseract --version
gs --version
pngquant --version
img2pdf --version
- name: Test
run: |
python -m pytest --cov-report xml --cov=ocrmypdf --cov=tests/ -n0 tests/
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
with:
files: ./coverage.xml
env_vars: OS,PYTHON
test_windows:
name: Test Windows
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [windows-latest]
python: ["3.10", "3.11", "3.12"]
env:
OS: ${{ matrix.os }}
PYTHON: ${{ matrix.python }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: "0" # 0=all, needed for setuptools-scm to resolve version tags
- uses: actions/setup-python@v5
name: Setup Python
with:
python-version: ${{ matrix.python }}
cache: "pip"
- name: Install system packages
run: |
choco install --yes --no-progress --pre tesseract
choco install --yes --no-progress --ignore-checksums ghostscript --version 9.56.1
- name: Install Python packages
run: |
python -m pip install --upgrade pip wheel
python -m pip install --prefer-binary .[test]
- name: Test
run: |
python -m pytest --cov-report xml --cov=ocrmypdf --cov=tests/ -n0 tests/
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
with:
files: ./coverage.xml
env_vars: OS,PYTHON
wheel_sdist_linux:
name: Build sdist and wheels
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: "0" # 0=all, needed for setuptools-scm to resolve version tags
- uses: actions/setup-python@v5
name: Setup Python
with:
python-version: "3.10"
cache: "pip"
- name: Make wheels and sdist
run: |
python -m pip install --upgrade pip wheel build
python -m build --sdist --wheel
- uses: actions/upload-artifact@v4
with:
name: artifact
path: |
./dist/*.whl
./dist/*.tar.gz
upload_pypi:
name: Deploy artifacts to PyPI
needs: [wheel_sdist_linux, test_linux, test_macos, test_windows]
runs-on: ubuntu-latest
environment: release
permissions:
id-token: write # mandatory for PyPI publishing
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags/v')
steps:
- uses: actions/download-artifact@v4
with:
name: artifact
path: dist
- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
create_release:
name: Create GitHub release
needs: [upload_pypi]
runs-on: ubuntu-latest
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags/v')
permissions:
# Required to create a release
contents: write
id-token: write
steps:
- uses: actions/download-artifact@v4
with:
name: artifact
path: dist
- name: Sign the dists with Sigstore
uses: sigstore/gh-action-sigstore-python@v2.1.1
with:
inputs: >-
./dist/*.tar.gz
./dist/*.whl
- name: Create GitHub Release
env:
GITHUB_TOKEN: ${{ github.token }}
run: >-
gh release create
'${{ github.ref_name }}'
--repo '${{ github.repository }}'
--notes ""
- name: Upload artifact signatures to GitHub Release
env:
GITHUB_TOKEN: ${{ github.token }}
# Upload to GitHub Release using the `gh` CLI.
# `dist/` contains the built packages, and the
# sigstore-produced signatures and certificates.
run: >-
gh release upload
'${{ github.ref_name }}' dist/**
--repo '${{ github.repository }}'
docker_ubuntu:
name: Build Ubuntu-based Docker image
needs: [wheel_sdist_linux, test_linux, test_macos, test_windows]
runs-on: ubuntu-latest
if: github.event_name != 'pull_request'
steps:
- name: Set image tag to release or branch
run: echo "DOCKER_IMAGE_TAG=${GITHUB_REF##*/}" >> $GITHUB_ENV
- name: If main, set to latest
run: echo 'DOCKER_IMAGE_TAG=latest' >> $GITHUB_ENV
if: env.DOCKER_IMAGE_TAG == 'main'
- name: Set Docker Hub repository to username
run: echo "DOCKER_REPOSITORY=jbarlow83" >> $GITHUB_ENV
- name: Set image name
run: echo "DOCKER_IMAGE_NAME=ocrmypdf" >> $GITHUB_ENV
- uses: actions/checkout@v4
with:
fetch-depth: "0" # 0=all, needed for setuptools-scm to resolve version tags
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: jbarlow83
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v3
- name: Print image tag
run: echo "Building image ${DOCKER_REPOSITORY}/${DOCKER_IMAGE_NAME}:${DOCKER_IMAGE_TAG}"
- name: Build
run: |
docker buildx build \
--push \
--platform linux/arm64/v8,linux/amd64 \
--tag "${DOCKER_REPOSITORY}/${DOCKER_IMAGE_NAME}:${DOCKER_IMAGE_TAG}" \
--tag "${DOCKER_REPOSITORY}/${DOCKER_IMAGE_NAME}-ubuntu:${DOCKER_IMAGE_TAG}" \
--file .docker/Dockerfile .
docker_alpine:
name: Build Alpine-based Docker images
needs: [wheel_sdist_linux, test_linux, test_macos, test_windows]
runs-on: ubuntu-latest
if: github.event_name != 'pull_request'
steps:
- name: Set image tag to release or branch
run: echo "DOCKER_IMAGE_TAG=${GITHUB_REF##*/}" >> $GITHUB_ENV
- name: If main, set to latest
run: echo 'DOCKER_IMAGE_TAG=latest' >> $GITHUB_ENV
if: env.DOCKER_IMAGE_TAG == 'main'
- name: Set Docker Hub repository to username
run: echo "DOCKER_REPOSITORY=jbarlow83" >> $GITHUB_ENV
- name: Set image name
run: echo "DOCKER_IMAGE_NAME=ocrmypdf-alpine" >> $GITHUB_ENV
- uses: actions/checkout@v4
with:
fetch-depth: "0" # 0=all, needed for setuptools-scm to resolve version tags
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: jbarlow83
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v3
- name: Print image tag
run: echo "Building image ${DOCKER_REPOSITORY}/${DOCKER_IMAGE_NAME}:${DOCKER_IMAGE_TAG}"
- name: Build
run: |
docker buildx build \
--push \
--platform linux/amd64,linux/arm64 \
--tag "${DOCKER_REPOSITORY}/${DOCKER_IMAGE_NAME}:${DOCKER_IMAGE_TAG}" \
--file .docker/Dockerfile.alpine .
+44 -14
View File
@@ -1,17 +1,47 @@
tmp/
log/
*.pyc
tests/output/
.ruffus_history.sqlite
*.sublime-*
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: CC-BY-SA-4.0
# dotfiles
.coverage
.venv*/
.tox/
.vscode/
.hypothesis/
.ipynb_checkpoints/
.mypy_cache/
.pytest_cache/
# Dev scratch
*.ipynb
**/*.pyc
/*.pdf
/*.qdf
/*.png
/scratch.py
IDEAS
log/
tests/resources/private/
tmp/
venv*/
/debug_tests.py
*.traineddata
/private
/coverage.xml
/issuepdf
# Package building
*.egg-info/
build/
dist/
*.egg-info/
venv/
*/test/output
bin/
include/
lib/
pip-selfcheck.json
pyvenv.cfg
wheelhouse/
pip-wheel-metadata/
# Code coverage
htmlcov/
# Automatically generated files
docs/_build/
docs/_static/
docs/_templates/
docs/Makefile
src/ocrmypdf/_version.py
+32
View File
@@ -0,0 +1,32 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: check-case-conflict
- id: check-merge-conflict
- id: check-toml
- id: check-yaml
- id: debug-statements
- repo: https://github.com/charliermarsh/ruff-pre-commit
rev: "v0.0.261"
hooks:
- id: ruff
files: "src/.*\\.pyi?$"
args: [--fix, --exit-non-zero-on-fix]
- repo: https://github.com/psf/black
rev: 23.3.0
hooks:
- id: black
language_version: python
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.2.0
hooks:
- id: mypy
additional_dependencies:
- types-toml
- types-setuptools
- types-requests
- types-Pillow
+29
View File
@@ -0,0 +1,29 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: docs/conf.py
# Optionally build your docs in additional formats such as PDF
formats:
- pdf
# Optionally set the version of Python and requirements required to build your docs
build:
os: ubuntu-22.04
tools:
python: "3.10"
python:
install:
- method: pip
path: .
extra_requirements:
- docs
+140
View File
@@ -0,0 +1,140 @@
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: OCRmyPDF
Upstream-Contact: James R. Barlow <james@purplerock.ca>
Source: https://github.com/ocrmypdf/OCRmyPDF
Files:
.git_archival.txt
docs/images/logo-social.png
docs/images/logo-square-256.svg
docs/images/logo-square.png
docs/images/logo-square.svg
docs/images/logo.svg
Copyright: (C) 2022 James R. Barlow
License: MPL-2.0
Files:
.github/ISSUE_TEMPLATE/*.yml
docs/images/macos-workflow.png
Copyright: (C) 2023 James R. Barlow
License: CC-BY-SA-4.0
Files:
tests/resources/acroform.pdf
tests/resources/aspect.pdf
tests/resources/blank.pdf
tests/resources/cmyk.pdf
tests/resources/crom.png
tests/resources/enormous.pdf
tests/resources/formxobject.pdf
tests/resources/francais.pdf
tests/resources/hugemono.pdf
tests/resources/invalid.pdf
tests/resources/kcs.pdf
tests/resources/livecycle.pdf
tests/resources/meta.pdf
tests/resources/missing_docinfo.pdf
tests/resources/negzero.pdf
tests/resources/no_contents.pdf
tests/resources/tagged*
tests/resources/toc.pdf
tests/resources/trivial.pdf
tests/resources/truetype_font_nomapping.pdf
tests/resources/type3_font_nomapping.pdf
Copyright: (C) 2023 James R. Barlow
License: CC-BY-SA-4.0
Files:
tests/resources/graph.pdf
tests/resources/graph_ocred.pdf
Copyright: (C) 2012 SmokeyJoe
License: GFDL-1.2-or-later or CC-BY-SA-3.0
Files: tests/resources/c02-22.pdf
tests/resources/multipage.pdf
Copyright: Public domain
License: public-domain
Copyright on these files has expired.
Files: docs/images/bitmap_vs_svg.svg
Copyright: (C) 2006 Yug
License: CC-BY-SA-2.5
Files: tests/cache/*
Copyright: (C) 2022 James R. Barlow
License: CC-BY-SA-4.0
Files: tests/resources/linn.png
tests/resources/linn.pdf
tests/resources/linn.txt
tests/resources/ccitt.pdf
tests/resources/cardinal.pdf
tests/resources/jbig2.pdf
tests/resources/jbig2_baddevicen.pdf
tests/resources/skew.pdf
tests/resources/rotated_skew.pdf
tests/resources/poster.pdf
Copyright: (C) 1985 Forat Electronics
License: GFDL-1.2-or-later or CC-BY-SA-3.0
Files: tests/resources/lichtenstein.pdf
Copyright: (C) 2001 Andreas Tille
(C) 2007 Alessio Damato
License: GFDL-1.2-or-later or CC-BY-SA-3.0
Files: tests/resources/masks.pdf
Copyright: held by the contributors to the German Wikipedia article "Linux"
see: https://de.wikipedia.org/w/index.php?title=Linux&action=history
(masks.pdf generated from Wikipedia article as of 2016-08-24)
License: CC-BY-SA-3.0
Files: tests/resources/epson.pdf
Copyright: held by the contributors to the Wikipedia article "Optical character recognition"
see: https://en.wikipedia.org/w/index.php?title=Optical_character_recognition&action=history
(epson.pdf generated from Wikipedia article as of 2016-09-14)
License: CC-BY-SA-3.0
Files: tests/resources/typewriter.png tests/resources/2400dpi.pdf
Copyright: (C) 2005 Ellywa
License: GFDL-1.2-or-later or CC-BY-SA-1.0 or CC-BY-SA-2.0 or CC-BY-SA-2.5 or CC-BY-SA-3.0
Comment:
Obtained from: https://commons.wikimedia.org/wiki/File:Triumph.typewriter_text_Linzensoep.gif
Files: tests/resources/overlay.pdf
Copyright: (C) 2017 Max Anderson
License: MIT
Files:
tests/resources/baiona*.png
tests/resources/baiona*.jpg
tests/resources/link.pdf
tests/resources/palette.pdf
Copyright: (C) 2014 Euskaldunaa
License: CC-BY-SA-4.0
Files: tests/resources/vector.pdf
Copyright: (C) 2018 Catscratch
License: MIT
Files: src/ocrmypdf/data/sRGB.icc
Copyright: Kai-Uwe Behrmann <www.behrmann.name>
Marti Maria <www.littlecms.com>
Photogamut <www.photogamut.org>
Graeme Gill <www.argyllcms.com>
ColorSolutions <www.basICColor.com>
License: Zlib
Files: src/ocrmypdf/data/pdf.ttf
Copyright: (C) 2014 Ray Smith
(C) 2015 Ken Sharp
(C) 2016 James R. Barlow
(C) 2016 Jeff Breidenbach
(C) 2017 Zdenko Podobný
License: Apache-2.0
Files: tests/resources/3small.pdf
Copyright: (C) 2014 Euskaldunaa
(C) 2017 James R. Barlow
(C) 2005 Ellywa
License: CC-BY-SA-4.0 and (GFDL-1.2-or-later or CC-BY-SA-1.0 or CC-BY-SA-2.0 or CC-BY-SA-2.5 or CC-BY-SA-3.0)
Comment: concatenation of baiona_gray.png, crom.png and typewriter.png/2400dpi.pdf
+373
View File
@@ -0,0 +1,373 @@
Mozilla Public License Version 2.0
==================================
1. Definitions
--------------
1.1. "Contributor"
means each individual or legal entity that creates, contributes to
the creation of, or owns Covered Software.
1.2. "Contributor Version"
means the combination of the Contributions of others (if any) used
by a Contributor and that particular Contributor's Contribution.
1.3. "Contribution"
means Covered Software of a particular Contributor.
1.4. "Covered Software"
means Source Code Form to which the initial Contributor has attached
the notice in Exhibit A, the Executable Form of such Source Code
Form, and Modifications of such Source Code Form, in each case
including portions thereof.
1.5. "Incompatible With Secondary Licenses"
means
(a) that the initial Contributor has attached the notice described
in Exhibit B to the Covered Software; or
(b) that the Covered Software was made available under the terms of
version 1.1 or earlier of the License, but not also under the
terms of a Secondary License.
1.6. "Executable Form"
means any form of the work other than Source Code Form.
1.7. "Larger Work"
means a work that combines Covered Software with other material, in
a separate file or files, that is not Covered Software.
1.8. "License"
means this document.
1.9. "Licensable"
means having the right to grant, to the maximum extent possible,
whether at the time of the initial grant or subsequently, any and
all of the rights conveyed by this License.
1.10. "Modifications"
means any of the following:
(a) any file in Source Code Form that results from an addition to,
deletion from, or modification of the contents of Covered
Software; or
(b) any new file in Source Code Form that contains any Covered
Software.
1.11. "Patent Claims" of a Contributor
means any patent claim(s), including without limitation, method,
process, and apparatus claims, in any patent Licensable by such
Contributor that would be infringed, but for the grant of the
License, by the making, using, selling, offering for sale, having
made, import, or transfer of either its Contributions or its
Contributor Version.
1.12. "Secondary License"
means either the GNU General Public License, Version 2.0, the GNU
Lesser General Public License, Version 2.1, the GNU Affero General
Public License, Version 3.0, or any later versions of those
licenses.
1.13. "Source Code Form"
means the form of the work preferred for making modifications.
1.14. "You" (or "Your")
means an individual or a legal entity exercising rights under this
License. For legal entities, "You" includes any entity that
controls, is controlled by, or is under common control with You. For
purposes of this definition, "control" means (a) the power, direct
or indirect, to cause the direction or management of such entity,
whether by contract or otherwise, or (b) ownership of more than
fifty percent (50%) of the outstanding shares or beneficial
ownership of such entity.
2. License Grants and Conditions
--------------------------------
2.1. Grants
Each Contributor hereby grants You a world-wide, royalty-free,
non-exclusive license:
(a) under intellectual property rights (other than patent or trademark)
Licensable by such Contributor to use, reproduce, make available,
modify, display, perform, distribute, and otherwise exploit its
Contributions, either on an unmodified basis, with Modifications, or
as part of a Larger Work; and
(b) under Patent Claims of such Contributor to make, use, sell, offer
for sale, have made, import, and otherwise transfer either its
Contributions or its Contributor Version.
2.2. Effective Date
The licenses granted in Section 2.1 with respect to any Contribution
become effective for each Contribution on the date the Contributor first
distributes such Contribution.
2.3. Limitations on Grant Scope
The licenses granted in this Section 2 are the only rights granted under
this License. No additional rights or licenses will be implied from the
distribution or licensing of Covered Software under this License.
Notwithstanding Section 2.1(b) above, no patent license is granted by a
Contributor:
(a) for any code that a Contributor has removed from Covered Software;
or
(b) for infringements caused by: (i) Your and any other third party's
modifications of Covered Software, or (ii) the combination of its
Contributions with other software (except as part of its Contributor
Version); or
(c) under Patent Claims infringed by Covered Software in the absence of
its Contributions.
This License does not grant any rights in the trademarks, service marks,
or logos of any Contributor (except as may be necessary to comply with
the notice requirements in Section 3.4).
2.4. Subsequent Licenses
No Contributor makes additional grants as a result of Your choice to
distribute the Covered Software under a subsequent version of this
License (see Section 10.2) or under the terms of a Secondary License (if
permitted under the terms of Section 3.3).
2.5. Representation
Each Contributor represents that the Contributor believes its
Contributions are its original creation(s) or it has sufficient rights
to grant the rights to its Contributions conveyed by this License.
2.6. Fair Use
This License is not intended to limit any rights You have under
applicable copyright doctrines of fair use, fair dealing, or other
equivalents.
2.7. Conditions
Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
in Section 2.1.
3. Responsibilities
-------------------
3.1. Distribution of Source Form
All distribution of Covered Software in Source Code Form, including any
Modifications that You create or to which You contribute, must be under
the terms of this License. You must inform recipients that the Source
Code Form of the Covered Software is governed by the terms of this
License, and how they can obtain a copy of this License. You may not
attempt to alter or restrict the recipients' rights in the Source Code
Form.
3.2. Distribution of Executable Form
If You distribute Covered Software in Executable Form then:
(a) such Covered Software must also be made available in Source Code
Form, as described in Section 3.1, and You must inform recipients of
the Executable Form how they can obtain a copy of such Source Code
Form by reasonable means in a timely manner, at a charge no more
than the cost of distribution to the recipient; and
(b) You may distribute such Executable Form under the terms of this
License, or sublicense it under different terms, provided that the
license for the Executable Form does not attempt to limit or alter
the recipients' rights in the Source Code Form under this License.
3.3. Distribution of a Larger Work
You may create and distribute a Larger Work under terms of Your choice,
provided that You also comply with the requirements of this License for
the Covered Software. If the Larger Work is a combination of Covered
Software with a work governed by one or more Secondary Licenses, and the
Covered Software is not Incompatible With Secondary Licenses, this
License permits You to additionally distribute such Covered Software
under the terms of such Secondary License(s), so that the recipient of
the Larger Work may, at their option, further distribute the Covered
Software under the terms of either this License or such Secondary
License(s).
3.4. Notices
You may not remove or alter the substance of any license notices
(including copyright notices, patent notices, disclaimers of warranty,
or limitations of liability) contained within the Source Code Form of
the Covered Software, except that You may alter any license notices to
the extent required to remedy known factual inaccuracies.
3.5. Application of Additional Terms
You may choose to offer, and to charge a fee for, warranty, support,
indemnity or liability obligations to one or more recipients of Covered
Software. However, You may do so only on Your own behalf, and not on
behalf of any Contributor. You must make it absolutely clear that any
such warranty, support, indemnity, or liability obligation is offered by
You alone, and You hereby agree to indemnify every Contributor for any
liability incurred by such Contributor as a result of warranty, support,
indemnity or liability terms You offer. You may include additional
disclaimers of warranty and limitations of liability specific to any
jurisdiction.
4. Inability to Comply Due to Statute or Regulation
---------------------------------------------------
If it is impossible for You to comply with any of the terms of this
License with respect to some or all of the Covered Software due to
statute, judicial order, or regulation then You must: (a) comply with
the terms of this License to the maximum extent possible; and (b)
describe the limitations and the code they affect. Such description must
be placed in a text file included with all distributions of the Covered
Software under this License. Except to the extent prohibited by statute
or regulation, such description must be sufficiently detailed for a
recipient of ordinary skill to be able to understand it.
5. Termination
--------------
5.1. The rights granted under this License will terminate automatically
if You fail to comply with any of its terms. However, if You become
compliant, then the rights granted under this License from a particular
Contributor are reinstated (a) provisionally, unless and until such
Contributor explicitly and finally terminates Your grants, and (b) on an
ongoing basis, if such Contributor fails to notify You of the
non-compliance by some reasonable means prior to 60 days after You have
come back into compliance. Moreover, Your grants from a particular
Contributor are reinstated on an ongoing basis if such Contributor
notifies You of the non-compliance by some reasonable means, this is the
first time You have received notice of non-compliance with this License
from such Contributor, and You become compliant prior to 30 days after
Your receipt of the notice.
5.2. If You initiate litigation against any entity by asserting a patent
infringement claim (excluding declaratory judgment actions,
counter-claims, and cross-claims) alleging that a Contributor Version
directly or indirectly infringes any patent, then the rights granted to
You by any and all Contributors for the Covered Software under Section
2.1 of this License shall terminate.
5.3. In the event of termination under Sections 5.1 or 5.2 above, all
end user license agreements (excluding distributors and resellers) which
have been validly granted by You or Your distributors under this License
prior to termination shall survive termination.
************************************************************************
* *
* 6. Disclaimer of Warranty *
* ------------------------- *
* *
* Covered Software is provided under this License on an "as is" *
* basis, without warranty of any kind, either expressed, implied, or *
* statutory, including, without limitation, warranties that the *
* Covered Software is free of defects, merchantable, fit for a *
* particular purpose or non-infringing. The entire risk as to the *
* quality and performance of the Covered Software is with You. *
* Should any Covered Software prove defective in any respect, You *
* (not any Contributor) assume the cost of any necessary servicing, *
* repair, or correction. This disclaimer of warranty constitutes an *
* essential part of this License. No use of any Covered Software is *
* authorized under this License except under this disclaimer. *
* *
************************************************************************
************************************************************************
* *
* 7. Limitation of Liability *
* -------------------------- *
* *
* Under no circumstances and under no legal theory, whether tort *
* (including negligence), contract, or otherwise, shall any *
* Contributor, or anyone who distributes Covered Software as *
* permitted above, be liable to You for any direct, indirect, *
* special, incidental, or consequential damages of any character *
* including, without limitation, damages for lost profits, loss of *
* goodwill, work stoppage, computer failure or malfunction, or any *
* and all other commercial damages or losses, even if such party *
* shall have been informed of the possibility of such damages. This *
* limitation of liability shall not apply to liability for death or *
* personal injury resulting from such party's negligence to the *
* extent applicable law prohibits such limitation. Some *
* jurisdictions do not allow the exclusion or limitation of *
* incidental or consequential damages, so this exclusion and *
* limitation may not apply to You. *
* *
************************************************************************
8. Litigation
-------------
Any litigation relating to this License may be brought only in the
courts of a jurisdiction where the defendant maintains its principal
place of business and such litigation shall be governed by laws of that
jurisdiction, without reference to its conflict-of-law provisions.
Nothing in this Section shall prevent a party's ability to bring
cross-claims or counter-claims.
9. Miscellaneous
----------------
This License represents the complete agreement concerning the subject
matter hereof. If any provision of this License is held to be
unenforceable, such provision shall be reformed only to the extent
necessary to make it enforceable. Any law or regulation which provides
that the language of a contract shall be construed against the drafter
shall not be used to construe this License against a Contributor.
10. Versions of the License
---------------------------
10.1. New Versions
Mozilla Foundation is the license steward. Except as provided in Section
10.3, no one other than the license steward has the right to modify or
publish new versions of this License. Each version will be given a
distinguishing version number.
10.2. Effect of New Versions
You may distribute the Covered Software under the terms of the version
of the License under which You originally received the Covered Software,
or under the terms of any subsequent version published by the license
steward.
10.3. Modified Versions
If you create software not governed by this License, and you want to
create a new license for such software, you may create and use a
modified version of this License if you rename the license and remove
any references to the name of the license steward (except to note that
such modified license differs from this License).
10.4. Distributing Source Code Form that is Incompatible With Secondary
Licenses
If You choose to distribute Source Code Form that is Incompatible With
Secondary Licenses under the terms of this version of the License, the
notice described in Exhibit B of this License must be attached.
Exhibit A - Source Code Form License Notice
-------------------------------------------
This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.
If it is not possible or desirable to put the notice in a particular
file, then You may include the notice in a location (such as a LICENSE
file in a relevant directory) where a recipient would be likely to look
for such a notice.
You may add additional accurate notices of copyright ownership.
Exhibit B - "Incompatible With Secondary Licenses" Notice
---------------------------------------------------------
This Source Code Form is "Incompatible With Secondary Licenses", as
defined by the Mozilla Public License, v. 2.0.
-20
View File
@@ -1,20 +0,0 @@
Copyright (c) 2013-2015, The OCRmyPDF Authors
Permission is hereby granted, free of charge, to any person obtaining a
copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+235
View File
@@ -0,0 +1,235 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
Preamble
The GNU Affero General Public License is a free, copyleft license for software and other kinds of works, specifically designed to ensure cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, our General Public Licenses are intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users.
When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License which gives you legal permission to copy, distribute and/or modify the software.
A secondary benefit of defending all users' freedom is that improvements made in alternate versions of the program, if they receive widespread use, become available for other developers to incorporate. Many developers of free software are heartened and encouraged by the resulting cooperation. However, in the case of software used on network servers, this result may fail to come about. The GNU General Public License permits making a modified version and letting the public access it on a server without ever releasing its source code to the public.
The GNU Affero General Public License is designed specifically to ensure that, in such cases, the modified source code becomes available to the community. It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version.
An older license, called the Affero General Public License and published by Affero, was designed to accomplish similar goals. This is a different license, not a version of the Affero GPL, but Affero has released a new version of the Affero GPL which permits relicensing under this license.
The precise terms and conditions for copying, distribution and modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based on the Program.
To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work.
A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source.
The Corresponding Source for a work in source code form is that same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures.
When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified it, and giving a relevant date.
b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices".
c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so.
A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways:
a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b.
d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d.
A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product.
"Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made.
If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM).
The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or authors of the material; or
e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors.
All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11).
However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice.
Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party.
If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it.
A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software. This Corresponding Source shall include the Corresponding Source for any work covered by version 3 of the GNU General Public License that is incorporated pursuant to the following paragraph.
Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the work with which it is combined will remain governed by version 3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of the GNU Affero General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU Affero General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU Affero General Public License, you may choose any version ever published by the Free Software Foundation.
If the Program specifies that a proxy can decide which future versions of the GNU Affero General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program.
Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer network, you should also make sure that it provides a way for users to get its source. For example, if your program is a web application, its interface could display a "Source" link that leads users to an archive of the code. There are many ways you could offer source, and different solutions will be better for different programs; see section 13 for the specific requirements.
You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU AGPL, see <http://www.gnu.org/licenses/>.
+81
View File
@@ -0,0 +1,81 @@
Creative Commons Attribution-ShareAlike 1.0
CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE LEGAL SERVICES. DISTRIBUTION OF THIS DRAFT LICENSE DOES NOT CREATE AN ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES REGARDING THE INFORMATION PROVIDED, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM ITS USE.
License
THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVE COMMONS PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS PROTECTED BY COPYRIGHT AND/OR OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS AUTHORIZED UNDER THIS LICENSE IS PROHIBITED.
BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND AGREE TO BE BOUND BY THE TERMS OF THIS LICENSE. THE LICENSOR GRANTS YOU THE RIGHTS CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND CONDITIONS.
1. Definitions
a. "Collective Work" means a work, such as a periodical issue, anthology or encyclopedia, in which the Work in its entirety in unmodified form, along with a number of other contributions, constituting separate and independent works in themselves, are assembled into a collective whole. A work that constitutes a Collective Work will not be considered a Derivative Work (as defined below) for the purposes of this License.
b. "Derivative Work" means a work based upon the Work or upon the Work and other pre-existing works, such as a translation, musical arrangement, dramatization, fictionalization, motion picture version, sound recording, art reproduction, abridgment, condensation, or any other form in which the Work may be recast, transformed, or adapted, except that a work that constitutes a Collective Work will not be considered a Derivative Work for the purpose of this License.
c. "Licensor" means the individual or entity that offers the Work under the terms of this License.
d. "Original Author" means the individual or entity who created the Work.
e. "Work" means the copyrightable work of authorship offered under the terms of this License.
f. "You" means an individual or entity exercising rights under this License who has not previously violated the terms of this License with respect to the Work, or who has received express permission from the Licensor to exercise rights under this License despite a previous violation.
2. Fair Use Rights. Nothing in this license is intended to reduce, limit, or restrict any rights arising from fair use, first sale or other limitations on the exclusive rights of the copyright owner under copyright law or other applicable laws.
3. License Grant. Subject to the terms and conditions of this License, Licensor hereby grants You a worldwide, royalty-free, non-exclusive, perpetual (for the duration of the applicable copyright) license to exercise the rights in the Work as stated below:
a. to reproduce the Work, to incorporate the Work into one or more Collective Works, and to reproduce the Work as incorporated in the Collective Works;
b. to create and reproduce Derivative Works;
c. to distribute copies or phonorecords of, display publicly, perform publicly, and perform publicly by means of a digital audio transmission the Work including as incorporated in Collective Works;
d. to distribute copies or phonorecords of, display publicly, perform publicly, and perform publicly by means of a digital audio transmission Derivative Works;
The above rights may be exercised in all media and formats whether now known or hereafter devised. The above rights include the right to make such modifications as are technically necessary to exercise the rights in other media and formats. All rights not expressly granted by Licensor are hereby reserved.
4. Restrictions. The license granted in Section 3 above is expressly made subject to and limited by the following restrictions:
a. You may distribute, publicly display, publicly perform, or publicly digitally perform the Work only under the terms of this License, and You must include a copy of, or the Uniform Resource Identifier for, this License with every copy or phonorecord of the Work You distribute, publicly display, publicly perform, or publicly digitally perform. You may not offer or impose any terms on the Work that alter or restrict the terms of this License or the recipients' exercise of the rights granted hereunder. You may not sublicense the Work. You must keep intact all notices that refer to this License and to the disclaimer of warranties. You may not distribute, publicly display, publicly perform, or publicly digitally perform the Work with any technological measures that control access or use of the Work in a manner inconsistent with the terms of this License Agreement. The above applies to the Work as incorporated in a Collective Work, but this does not require the Collective Work apart from the Work itself to be made subject to the terms of this License. If You create a Collective Work, upon notice from any Licensor You must, to the extent practicable, remove from the Collective Work any reference to such Licensor or the Original Author, as requested. If You create a Derivative Work, upon notice from any Licensor You must, to the extent practicable, remove from the Derivative Work any reference to such Licensor or the Original Author, as requested.
b. You may distribute, publicly display, publicly perform, or publicly digitally perform a Derivative Work only under the terms of this License, and You must include a copy of, or the Uniform Resource Identifier for, this License with every copy or phonorecord of each Derivative Work You distribute, publicly display, publicly perform, or publicly digitally perform. You may not offer or impose any terms on the Derivative Works that alter or restrict the terms of this License or the recipients' exercise of the rights granted hereunder, and You must keep intact all notices that refer to this License and to the disclaimer of warranties. You may not distribute, publicly display, publicly perform, or publicly digitally perform the Derivative Work with any technological measures that control access or use of the Work in a manner inconsistent with the terms of this License Agreement. The above applies to the Derivative Work as incorporated in a Collective Work, but this does not require the Collective Work apart from the Derivative Work itself to be made subject to the terms of this License.
c. If you distribute, publicly display, publicly perform, or publicly digitally perform the Work or any Derivative Works or Collective Works, You must keep intact all copyright notices for the Work and give the Original Author credit reasonable to the medium or means You are utilizing by conveying the name (or pseudonym if applicable) of the Original Author if supplied; the title of the Work if supplied; in the case of a Derivative Work, a credit identifying the use of the Work in the Derivative Work (e.g., "French translation of the Work by Original Author," or "Screenplay based on original Work by Original Author"). Such credit may be implemented in any reasonable manner; provided, however, that in the case of a Derivative Work or Collective Work, at a minimum such credit will appear where any other comparable authorship credit appears and in a manner at least as prominent as such other comparable authorship credit.
5. Representations, Warranties and Disclaimer
a. By offering the Work for public release under this License, Licensor represents and warrants that, to the best of Licensor's knowledge after reasonable inquiry:
i. Licensor has secured all rights in the Work necessary to grant the license rights hereunder and to permit the lawful exercise of the rights granted hereunder without You having any obligation to pay any royalties, compulsory license fees, residuals or any other payments;
ii. The Work does not infringe the copyright, trademark, publicity rights, common law rights or any other right of any third party or constitute defamation, invasion of privacy or other tortious injury to any third party.
b. EXCEPT AS EXPRESSLY STATED IN THIS LICENSE OR OTHERWISE AGREED IN WRITING OR REQUIRED BY APPLICABLE LAW, THE WORK IS LICENSED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES REGARDING THE CONTENTS OR ACCURACY OF THE WORK.
6. Limitation on Liability. EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE LAW, AND EXCEPT FOR DAMAGES ARISING FROM LIABILITY TO A THIRD PARTY RESULTING FROM BREACH OF THE WARRANTIES IN SECTION 5, IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANY SPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES ARISING OUT OF THIS LICENSE OR THE USE OF THE WORK, EVEN IF LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
7. Termination
a. This License and the rights granted hereunder will terminate automatically upon any breach by You of the terms of this License. Individuals or entities who have received Derivative Works or Collective Works from You under this License, however, will not have their licenses terminated provided such individuals or entities remain in full compliance with those licenses. Sections 1, 2, 5, 6, 7, and 8 will survive any termination of this License.
b. Subject to the above terms and conditions, the license granted here is perpetual (for the duration of the applicable copyright in the Work). Notwithstanding the above, Licensor reserves the right to release the Work under different license terms or to stop distributing the Work at any time; provided, however that any such election will not serve to withdraw this License (or any other license that has been, or is required to be, granted under the terms of this License), and this License will continue in full force and effect unless terminated as stated above.
8. Miscellaneous
a. Each time You distribute or publicly digitally perform the Work or a Collective Work, the Licensor offers to the recipient a license to the Work on the same terms and conditions as the license granted to You under this License.
b. Each time You distribute or publicly digitally perform a Derivative Work, Licensor offers to the recipient a license to the original Work on the same terms and conditions as the license granted to You under this License.
c. If any provision of this License is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this License, and without further action by the parties to this agreement, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable.
d. No term or provision of this License shall be deemed waived and no breach consented to unless such waiver or consent shall be in writing and signed by the party to be charged with such waiver or consent.
e. This License constitutes the entire agreement between the parties with respect to the Work licensed here. There are no understandings, agreements or representations with respect to the Work not specified here. Licensor shall not be bound by any additional provisions that may appear in any communication from You. This License may not be modified without the mutual written agreement of the Licensor and You.
Creative Commons is not a party to this License, and makes no warranty whatsoever in connection with the Work. Creative Commons will not be liable to You or any party on any legal theory for any damages whatsoever, including without limitation any general, special, incidental or consequential damages arising in connection to this license. Notwithstanding the foregoing two (2) sentences, if Creative Commons has expressly identified itself as the Licensor hereunder, it shall have all rights and obligations of Licensor.
Except for the limited purpose of indicating to the public that the Work is licensed under the CCPL, neither party will use the trademark "Creative Commons" or any related trademark or logo of Creative Commons without the prior written consent of Creative Commons. Any permitted use will be in compliance with Creative Commons' then-current trademark usage guidelines, as may be published on its website or otherwise made available upon request from time to time.
Creative Commons may be contacted at http://creativecommons.org/.
+85
View File
@@ -0,0 +1,85 @@
Creative Commons Attribution-ShareAlike 2.0
CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE LEGAL SERVICES. DISTRIBUTION OF THIS LICENSE DOES NOT CREATE AN ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES REGARDING THE INFORMATION PROVIDED, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM ITS USE.
License
THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVE COMMONS PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS PROTECTED BY COPYRIGHT AND/OR OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS AUTHORIZED UNDER THIS LICENSE OR COPYRIGHT LAW IS PROHIBITED.
BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND AGREE TO BE BOUND BY THE TERMS OF THIS LICENSE. THE LICENSOR GRANTS YOU THE RIGHTS CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND CONDITIONS.
1. Definitions
a. "Collective Work" means a work, such as a periodical issue, anthology or encyclopedia, in which the Work in its entirety in unmodified form, along with a number of other contributions, constituting separate and independent works in themselves, are assembled into a collective whole. A work that constitutes a Collective Work will not be considered a Derivative Work (as defined below) for the purposes of this License.
b. "Derivative Work" means a work based upon the Work or upon the Work and other pre-existing works, such as a translation, musical arrangement, dramatization, fictionalization, motion picture version, sound recording, art reproduction, abridgment, condensation, or any other form in which the Work may be recast, transformed, or adapted, except that a work that constitutes a Collective Work will not be considered a Derivative Work for the purpose of this License. For the avoidance of doubt, where the Work is a musical composition or sound recording, the synchronization of the Work in timed-relation with a moving image ("synching") will be considered a Derivative Work for the purpose of this License.
c. "Licensor" means the individual or entity that offers the Work under the terms of this License.
d. "Original Author" means the individual or entity who created the Work.
e. "Work" means the copyrightable work of authorship offered under the terms of this License.
f. "You" means an individual or entity exercising rights under this License who has not previously violated the terms of this License with respect to the Work, or who has received express permission from the Licensor to exercise rights under this License despite a previous violation.
g. "License Elements" means the following high-level license attributes as selected by Licensor and indicated in the title of this License: Attribution, ShareAlike.
2. Fair Use Rights. Nothing in this license is intended to reduce, limit, or restrict any rights arising from fair use, first sale or other limitations on the exclusive rights of the copyright owner under copyright law or other applicable laws.
3. License Grant. Subject to the terms and conditions of this License, Licensor hereby grants You a worldwide, royalty-free, non-exclusive, perpetual (for the duration of the applicable copyright) license to exercise the rights in the Work as stated below:
a. to reproduce the Work, to incorporate the Work into one or more Collective Works, and to reproduce the Work as incorporated in the Collective Works;
b. to create and reproduce Derivative Works;
c. to distribute copies or phonorecords of, display publicly, perform publicly, and perform publicly by means of a digital audio transmission the Work including as incorporated in Collective Works;
d. to distribute copies or phonorecords of, display publicly, perform publicly, and perform publicly by means of a digital audio transmission Derivative Works.
e. For the avoidance of doubt, where the work is a musical composition:
i. Performance Royalties Under Blanket Licenses. Licensor waives the exclusive right to collect, whether individually or via a performance rights society (e.g. ASCAP, BMI, SESAC), royalties for the public performance or public digital performance (e.g. webcast) of the Work.
ii. Mechanical Rights and Statutory Royalties. Licensor waives the exclusive right to collect, whether individually or via a music rights society or designated agent (e.g. Harry Fox Agency), royalties for any phonorecord You create from the Work ("cover version") and distribute, subject to the compulsory license created by 17 USC Section 115 of the US Copyright Act (or the equivalent in other jurisdictions).
f. Webcasting Rights and Statutory Royalties. For the avoidance of doubt, where the Work is a sound recording, Licensor waives the exclusive right to collect, whether individually or via a performance-rights society (e.g. SoundExchange), royalties for the public digital performance (e.g. webcast) of the Work, subject to the compulsory license created by 17 USC Section 114 of the US Copyright Act (or the equivalent in other jurisdictions).
The above rights may be exercised in all media and formats whether now known or hereafter devised. The above rights include the right to make such modifications as are technically necessary to exercise the rights in other media and formats. All rights not expressly granted by Licensor are hereby reserved.
4. Restrictions. The license granted in Section 3 above is expressly made subject to and limited by the following restrictions:
a. You may distribute, publicly display, publicly perform, or publicly digitally perform the Work only under the terms of this License, and You must include a copy of, or the Uniform Resource Identifier for, this License with every copy or phonorecord of the Work You distribute, publicly display, publicly perform, or publicly digitally perform. You may not offer or impose any terms on the Work that alter or restrict the terms of this License or the recipients' exercise of the rights granted hereunder. You may not sublicense the Work. You must keep intact all notices that refer to this License and to the disclaimer of warranties. You may not distribute, publicly display, publicly perform, or publicly digitally perform the Work with any technological measures that control access or use of the Work in a manner inconsistent with the terms of this License Agreement. The above applies to the Work as incorporated in a Collective Work, but this does not require the Collective Work apart from the Work itself to be made subject to the terms of this License. If You create a Collective Work, upon notice from any Licensor You must, to the extent practicable, remove from the Collective Work any reference to such Licensor or the Original Author, as requested. If You create a Derivative Work, upon notice from any Licensor You must, to the extent practicable, remove from the Derivative Work any reference to such Licensor or the Original Author, as requested.
b. You may distribute, publicly display, publicly perform, or publicly digitally perform a Derivative Work only under the terms of this License, a later version of this License with the same License Elements as this License, or a Creative Commons iCommons license that contains the same License Elements as this License (e.g. Attribution-ShareAlike 2.0 Japan). You must include a copy of, or the Uniform Resource Identifier for, this License or other license specified in the previous sentence with every copy or phonorecord of each Derivative Work You distribute, publicly display, publicly perform, or publicly digitally perform. You may not offer or impose any terms on the Derivative Works that alter or restrict the terms of this License or the recipients' exercise of the rights granted hereunder, and You must keep intact all notices that refer to this License and to the disclaimer of warranties. You may not distribute, publicly display, publicly perform, or publicly digitally perform the Derivative Work with any technological measures that control access or use of the Work in a manner inconsistent with the terms of this License Agreement. The above applies to the Derivative Work as incorporated in a Collective Work, but this does not require the Collective Work apart from the Derivative Work itself to be made subject to the terms of this License.
c. If you distribute, publicly display, publicly perform, or publicly digitally perform the Work or any Derivative Works or Collective Works, You must keep intact all copyright notices for the Work and give the Original Author credit reasonable to the medium or means You are utilizing by conveying the name (or pseudonym if applicable) of the Original Author if supplied; the title of the Work if supplied; to the extent reasonably practicable, the Uniform Resource Identifier, if any, that Licensor specifies to be associated with the Work, unless such URI does not refer to the copyright notice or licensing information for the Work; and in the case of a Derivative Work, a credit identifying the use of the Work in the Derivative Work (e.g., "French translation of the Work by Original Author," or "Screenplay based on original Work by Original Author"). Such credit may be implemented in any reasonable manner; provided, however, that in the case of a Derivative Work or Collective Work, at a minimum such credit will appear where any other comparable authorship credit appears and in a manner at least as prominent as such other comparable authorship credit.
5. Representations, Warranties and Disclaimer
UNLESS OTHERWISE AGREED TO BY THE PARTIES IN WRITING, LICENSOR OFFERS THE WORK AS-IS AND MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE MATERIALS, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF TITLE, MERCHANTIBILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY, OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT DISCOVERABLE. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO SUCH EXCLUSION MAY NOT APPLY TO YOU.
6. Limitation on Liability. EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE LAW, IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANY SPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES ARISING OUT OF THIS LICENSE OR THE USE OF THE WORK, EVEN IF LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
7. Termination
a. This License and the rights granted hereunder will terminate automatically upon any breach by You of the terms of this License. Individuals or entities who have received Derivative Works or Collective Works from You under this License, however, will not have their licenses terminated provided such individuals or entities remain in full compliance with those licenses. Sections 1, 2, 5, 6, 7, and 8 will survive any termination of this License.
b. Subject to the above terms and conditions, the license granted here is perpetual (for the duration of the applicable copyright in the Work). Notwithstanding the above, Licensor reserves the right to release the Work under different license terms or to stop distributing the Work at any time; provided, however that any such election will not serve to withdraw this License (or any other license that has been, or is required to be, granted under the terms of this License), and this License will continue in full force and effect unless terminated as stated above.
8. Miscellaneous
a. Each time You distribute or publicly digitally perform the Work or a Collective Work, the Licensor offers to the recipient a license to the Work on the same terms and conditions as the license granted to You under this License.
b. Each time You distribute or publicly digitally perform a Derivative Work, Licensor offers to the recipient a license to the original Work on the same terms and conditions as the license granted to You under this License.
c. If any provision of this License is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this License, and without further action by the parties to this agreement, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable.
d. No term or provision of this License shall be deemed waived and no breach consented to unless such waiver or consent shall be in writing and signed by the party to be charged with such waiver or consent.
e. This License constitutes the entire agreement between the parties with respect to the Work licensed here. There are no understandings, agreements or representations with respect to the Work not specified here. Licensor shall not be bound by any additional provisions that may appear in any communication from You. This License may not be modified without the mutual written agreement of the Licensor and You.
Creative Commons is not a party to this License, and makes no warranty whatsoever in connection with the Work. Creative Commons will not be liable to You or any party on any legal theory for any damages whatsoever, including without limitation any general, special, incidental or consequential damages arising in connection to this license. Notwithstanding the foregoing two (2) sentences, if Creative Commons has expressly identified itself as the Licensor hereunder, it shall have all rights and obligations of Licensor.
Except for the limited purpose of indicating to the public that the Work is licensed under the CCPL, neither party will use the trademark "Creative Commons" or any related trademark or logo of Creative Commons without the prior written consent of Creative Commons. Any permitted use will be in compliance with Creative Commons' then-current trademark usage guidelines, as may be published on its website or otherwise made available upon request from time to time.
Creative Commons may be contacted at http://creativecommons.org/.
+85
View File
@@ -0,0 +1,85 @@
Creative Commons Attribution-ShareAlike 2.5
CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE LEGAL SERVICES. DISTRIBUTION OF THIS LICENSE DOES NOT CREATE AN ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES REGARDING THE INFORMATION PROVIDED, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM ITS USE.
License
THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVE COMMONS PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS PROTECTED BY COPYRIGHT AND/OR OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS AUTHORIZED UNDER THIS LICENSE OR COPYRIGHT LAW IS PROHIBITED.
BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND AGREE TO BE BOUND BY THE TERMS OF THIS LICENSE. THE LICENSOR GRANTS YOU THE RIGHTS CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND CONDITIONS.
1. Definitions
a. "Collective Work" means a work, such as a periodical issue, anthology or encyclopedia, in which the Work in its entirety in unmodified form, along with a number of other contributions, constituting separate and independent works in themselves, are assembled into a collective whole. A work that constitutes a Collective Work will not be considered a Derivative Work (as defined below) for the purposes of this License.
b. "Derivative Work" means a work based upon the Work or upon the Work and other pre-existing works, such as a translation, musical arrangement, dramatization, fictionalization, motion picture version, sound recording, art reproduction, abridgment, condensation, or any other form in which the Work may be recast, transformed, or adapted, except that a work that constitutes a Collective Work will not be considered a Derivative Work for the purpose of this License. For the avoidance of doubt, where the Work is a musical composition or sound recording, the synchronization of the Work in timed-relation with a moving image ("synching") will be considered a Derivative Work for the purpose of this License.
c. "Licensor" means the individual or entity that offers the Work under the terms of this License.
d. "Original Author" means the individual or entity who created the Work.
e. "Work" means the copyrightable work of authorship offered under the terms of this License.
f. "You" means an individual or entity exercising rights under this License who has not previously violated the terms of this License with respect to the Work, or who has received express permission from the Licensor to exercise rights under this License despite a previous violation.
g. "License Elements" means the following high-level license attributes as selected by Licensor and indicated in the title of this License: Attribution, ShareAlike.
2. Fair Use Rights. Nothing in this license is intended to reduce, limit, or restrict any rights arising from fair use, first sale or other limitations on the exclusive rights of the copyright owner under copyright law or other applicable laws.
3. License Grant. Subject to the terms and conditions of this License, Licensor hereby grants You a worldwide, royalty-free, non-exclusive, perpetual (for the duration of the applicable copyright) license to exercise the rights in the Work as stated below:
a. to reproduce the Work, to incorporate the Work into one or more Collective Works, and to reproduce the Work as incorporated in the Collective Works;
b. to create and reproduce Derivative Works;
c. to distribute copies or phonorecords of, display publicly, perform publicly, and perform publicly by means of a digital audio transmission the Work including as incorporated in Collective Works;
d. to distribute copies or phonorecords of, display publicly, perform publicly, and perform publicly by means of a digital audio transmission Derivative Works.
e. For the avoidance of doubt, where the work is a musical composition:
i. Performance Royalties Under Blanket Licenses. Licensor waives the exclusive right to collect, whether individually or via a performance rights society (e.g. ASCAP, BMI, SESAC), royalties for the public performance or public digital performance (e.g. webcast) of the Work.
ii. Mechanical Rights and Statutory Royalties. Licensor waives the exclusive right to collect, whether individually or via a music rights society or designated agent (e.g. Harry Fox Agency), royalties for any phonorecord You create from the Work ("cover version") and distribute, subject to the compulsory license created by 17 USC Section 115 of the US Copyright Act (or the equivalent in other jurisdictions).
f. Webcasting Rights and Statutory Royalties. For the avoidance of doubt, where the Work is a sound recording, Licensor waives the exclusive right to collect, whether individually or via a performance-rights society (e.g. SoundExchange), royalties for the public digital performance (e.g. webcast) of the Work, subject to the compulsory license created by 17 USC Section 114 of the US Copyright Act (or the equivalent in other jurisdictions).
The above rights may be exercised in all media and formats whether now known or hereafter devised. The above rights include the right to make such modifications as are technically necessary to exercise the rights in other media and formats. All rights not expressly granted by Licensor are hereby reserved.
4. Restrictions. The license granted in Section 3 above is expressly made subject to and limited by the following restrictions:
a. You may distribute, publicly display, publicly perform, or publicly digitally perform the Work only under the terms of this License, and You must include a copy of, or the Uniform Resource Identifier for, this License with every copy or phonorecord of the Work You distribute, publicly display, publicly perform, or publicly digitally perform. You may not offer or impose any terms on the Work that alter or restrict the terms of this License or the recipients' exercise of the rights granted hereunder. You may not sublicense the Work. You must keep intact all notices that refer to this License and to the disclaimer of warranties. You may not distribute, publicly display, publicly perform, or publicly digitally perform the Work with any technological measures that control access or use of the Work in a manner inconsistent with the terms of this License Agreement. The above applies to the Work as incorporated in a Collective Work, but this does not require the Collective Work apart from the Work itself to be made subject to the terms of this License. If You create a Collective Work, upon notice from any Licensor You must, to the extent practicable, remove from the Collective Work any credit as required by clause 4(c), as requested. If You create a Derivative Work, upon notice from any Licensor You must, to the extent practicable, remove from the Derivative Work any credit as required by clause 4(c), as requested.
b. You may distribute, publicly display, publicly perform, or publicly digitally perform a Derivative Work only under the terms of this License, a later version of this License with the same License Elements as this License, or a Creative Commons iCommons license that contains the same License Elements as this License (e.g. Attribution-ShareAlike 2.5 Japan). You must include a copy of, or the Uniform Resource Identifier for, this License or other license specified in the previous sentence with every copy or phonorecord of each Derivative Work You distribute, publicly display, publicly perform, or publicly digitally perform. You may not offer or impose any terms on the Derivative Works that alter or restrict the terms of this License or the recipients' exercise of the rights granted hereunder, and You must keep intact all notices that refer to this License and to the disclaimer of warranties. You may not distribute, publicly display, publicly perform, or publicly digitally perform the Derivative Work with any technological measures that control access or use of the Work in a manner inconsistent with the terms of this License Agreement. The above applies to the Derivative Work as incorporated in a Collective Work, but this does not require the Collective Work apart from the Derivative Work itself to be made subject to the terms of this License.
c. If you distribute, publicly display, publicly perform, or publicly digitally perform the Work or any Derivative Works or Collective Works, You must keep intact all copyright notices for the Work and provide, reasonable to the medium or means You are utilizing: (i) the name of the Original Author (or pseudonym, if applicable) if supplied, and/or (ii) if the Original Author and/or Licensor designate another party or parties (e.g. a sponsor institute, publishing entity, journal) for attribution in Licensor's copyright notice, terms of service or by other reasonable means, the name of such party or parties; the title of the Work if supplied; to the extent reasonably practicable, the Uniform Resource Identifier, if any, that Licensor specifies to be associated with the Work, unless such URI does not refer to the copyright notice or licensing information for the Work; and in the case of a Derivative Work, a credit identifying the use of the Work in the Derivative Work (e.g., "French translation of the Work by Original Author," or "Screenplay based on original Work by Original Author"). Such credit may be implemented in any reasonable manner; provided, however, that in the case of a Derivative Work or Collective Work, at a minimum such credit will appear where any other comparable authorship credit appears and in a manner at least as prominent as such other comparable authorship credit.
5. Representations, Warranties and Disclaimer
UNLESS OTHERWISE AGREED TO BY THE PARTIES IN WRITING, LICENSOR OFFERS THE WORK AS-IS AND MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE MATERIALS, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF TITLE, MERCHANTIBILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY, OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT DISCOVERABLE. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO SUCH EXCLUSION MAY NOT APPLY TO YOU.
6. Limitation on Liability. EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE LAW, IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANY SPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES ARISING OUT OF THIS LICENSE OR THE USE OF THE WORK, EVEN IF LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
7. Termination
a. This License and the rights granted hereunder will terminate automatically upon any breach by You of the terms of this License. Individuals or entities who have received Derivative Works or Collective Works from You under this License, however, will not have their licenses terminated provided such individuals or entities remain in full compliance with those licenses. Sections 1, 2, 5, 6, 7, and 8 will survive any termination of this License.
b. Subject to the above terms and conditions, the license granted here is perpetual (for the duration of the applicable copyright in the Work). Notwithstanding the above, Licensor reserves the right to release the Work under different license terms or to stop distributing the Work at any time; provided, however that any such election will not serve to withdraw this License (or any other license that has been, or is required to be, granted under the terms of this License), and this License will continue in full force and effect unless terminated as stated above.
8. Miscellaneous
a. Each time You distribute or publicly digitally perform the Work or a Collective Work, the Licensor offers to the recipient a license to the Work on the same terms and conditions as the license granted to You under this License.
b. Each time You distribute or publicly digitally perform a Derivative Work, Licensor offers to the recipient a license to the original Work on the same terms and conditions as the license granted to You under this License.
c. If any provision of this License is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this License, and without further action by the parties to this agreement, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable.
d. No term or provision of this License shall be deemed waived and no breach consented to unless such waiver or consent shall be in writing and signed by the party to be charged with such waiver or consent.
e. This License constitutes the entire agreement between the parties with respect to the Work licensed here. There are no understandings, agreements or representations with respect to the Work not specified here. Licensor shall not be bound by any additional provisions that may appear in any communication from You. This License may not be modified without the mutual written agreement of the Licensor and You.
Creative Commons is not a party to this License, and makes no warranty whatsoever in connection with the Work. Creative Commons will not be liable to You or any party on any legal theory for any damages whatsoever, including without limitation any general, special, incidental or consequential damages arising in connection to this license. Notwithstanding the foregoing two (2) sentences, if Creative Commons has expressly identified itself as the Licensor hereunder, it shall have all rights and obligations of Licensor.
Except for the limited purpose of indicating to the public that the Work is licensed under the CCPL, neither party will use the trademark "Creative Commons" or any related trademark or logo of Creative Commons without the prior written consent of Creative Commons. Any permitted use will be in compliance with Creative Commons' then-current trademark usage guidelines, as may be published on its website or otherwise made available upon request from time to time.
Creative Commons may be contacted at http://creativecommons.org/.
+99
View File
@@ -0,0 +1,99 @@
Creative Commons Attribution-ShareAlike 3.0 Unported
CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE LEGAL SERVICES. DISTRIBUTION OF THIS LICENSE DOES NOT CREATE AN ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES REGARDING THE INFORMATION PROVIDED, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM ITS USE.
License
THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVE COMMONS PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS PROTECTED BY COPYRIGHT AND/OR OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS AUTHORIZED UNDER THIS LICENSE OR COPYRIGHT LAW IS PROHIBITED.
BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND AGREE TO BE BOUND BY THE TERMS OF THIS LICENSE. TO THE EXTENT THIS LICENSE MAY BE CONSIDERED TO BE A CONTRACT, THE LICENSOR GRANTS YOU THE RIGHTS CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND CONDITIONS.
1. Definitions
a. "Adaptation" means a work based upon the Work, or upon the Work and other pre-existing works, such as a translation, adaptation, derivative work, arrangement of music or other alterations of a literary or artistic work, or phonogram or performance and includes cinematographic adaptations or any other form in which the Work may be recast, transformed, or adapted including in any form recognizably derived from the original, except that a work that constitutes a Collection will not be considered an Adaptation for the purpose of this License. For the avoidance of doubt, where the Work is a musical work, performance or phonogram, the synchronization of the Work in timed-relation with a moving image ("synching") will be considered an Adaptation for the purpose of this License.
b. "Collection" means a collection of literary or artistic works, such as encyclopedias and anthologies, or performances, phonograms or broadcasts, or other works or subject matter other than works listed in Section 1(f) below, which, by reason of the selection and arrangement of their contents, constitute intellectual creations, in which the Work is included in its entirety in unmodified form along with one or more other contributions, each constituting separate and independent works in themselves, which together are assembled into a collective whole. A work that constitutes a Collection will not be considered an Adaptation (as defined below) for the purposes of this License.
c. "Creative Commons Compatible License" means a license that is listed at http://creativecommons.org/compatiblelicenses that has been approved by Creative Commons as being essentially equivalent to this License, including, at a minimum, because that license: (i) contains terms that have the same purpose, meaning and effect as the License Elements of this License; and, (ii) explicitly permits the relicensing of adaptations of works made available under that license under this License or a Creative Commons jurisdiction license with the same License Elements as this License.
d. "Distribute" means to make available to the public the original and copies of the Work or Adaptation, as appropriate, through sale or other transfer of ownership.
e. "License Elements" means the following high-level license attributes as selected by Licensor and indicated in the title of this License: Attribution, ShareAlike.
f. "Licensor" means the individual, individuals, entity or entities that offer(s) the Work under the terms of this License.
g. "Original Author" means, in the case of a literary or artistic work, the individual, individuals, entity or entities who created the Work or if no individual or entity can be identified, the publisher; and in addition (i) in the case of a performance the actors, singers, musicians, dancers, and other persons who act, sing, deliver, declaim, play in, interpret or otherwise perform literary or artistic works or expressions of folklore; (ii) in the case of a phonogram the producer being the person or legal entity who first fixes the sounds of a performance or other sounds; and, (iii) in the case of broadcasts, the organization that transmits the broadcast.
h. "Work" means the literary and/or artistic work offered under the terms of this License including without limitation any production in the literary, scientific and artistic domain, whatever may be the mode or form of its expression including digital form, such as a book, pamphlet and other writing; a lecture, address, sermon or other work of the same nature; a dramatic or dramatico-musical work; a choreographic work or entertainment in dumb show; a musical composition with or without words; a cinematographic work to which are assimilated works expressed by a process analogous to cinematography; a work of drawing, painting, architecture, sculpture, engraving or lithography; a photographic work to which are assimilated works expressed by a process analogous to photography; a work of applied art; an illustration, map, plan, sketch or three-dimensional work relative to geography, topography, architecture or science; a performance; a broadcast; a phonogram; a compilation of data to the extent it is protected as a copyrightable work; or a work performed by a variety or circus performer to the extent it is not otherwise considered a literary or artistic work.
i. "You" means an individual or entity exercising rights under this License who has not previously violated the terms of this License with respect to the Work, or who has received express permission from the Licensor to exercise rights under this License despite a previous violation.
j. "Publicly Perform" means to perform public recitations of the Work and to communicate to the public those public recitations, by any means or process, including by wire or wireless means or public digital performances; to make available to the public Works in such a way that members of the public may access these Works from a place and at a place individually chosen by them; to perform the Work to the public by any means or process and the communication to the public of the performances of the Work, including by public digital performance; to broadcast and rebroadcast the Work by any means including signs, sounds or images.
k. "Reproduce" means to make copies of the Work by any means including without limitation by sound or visual recordings and the right of fixation and reproducing fixations of the Work, including storage of a protected performance or phonogram in digital form or other electronic medium.
2. Fair Dealing Rights. Nothing in this License is intended to reduce, limit, or restrict any uses free from copyright or rights arising from limitations or exceptions that are provided for in connection with the copyright protection under copyright law or other applicable laws.
3. License Grant. Subject to the terms and conditions of this License, Licensor hereby grants You a worldwide, royalty-free, non-exclusive, perpetual (for the duration of the applicable copyright) license to exercise the rights in the Work as stated below:
a. to Reproduce the Work, to incorporate the Work into one or more Collections, and to Reproduce the Work as incorporated in the Collections;
b. to create and Reproduce Adaptations provided that any such Adaptation, including any translation in any medium, takes reasonable steps to clearly label, demarcate or otherwise identify that changes were made to the original Work. For example, a translation could be marked "The original work was translated from English to Spanish," or a modification could indicate "The original work has been modified.";
c. to Distribute and Publicly Perform the Work including as incorporated in Collections; and,
d. to Distribute and Publicly Perform Adaptations.
e. For the avoidance of doubt:
i. Non-waivable Compulsory License Schemes. In those jurisdictions in which the right to collect royalties through any statutory or compulsory licensing scheme cannot be waived, the Licensor reserves the exclusive right to collect such royalties for any exercise by You of the rights granted under this License;
ii. Waivable Compulsory License Schemes. In those jurisdictions in which the right to collect royalties through any statutory or compulsory licensing scheme can be waived, the Licensor waives the exclusive right to collect such royalties for any exercise by You of the rights granted under this License; and,
iii. Voluntary License Schemes. The Licensor waives the right to collect royalties, whether individually or, in the event that the Licensor is a member of a collecting society that administers voluntary licensing schemes, via that society, from any exercise by You of the rights granted under this License.
The above rights may be exercised in all media and formats whether now known or hereafter devised. The above rights include the right to make such modifications as are technically necessary to exercise the rights in other media and formats. Subject to Section 8(f), all rights not expressly granted by Licensor are hereby reserved.
4. Restrictions. The license granted in Section 3 above is expressly made subject to and limited by the following restrictions:
a. You may Distribute or Publicly Perform the Work only under the terms of this License. You must include a copy of, or the Uniform Resource Identifier (URI) for, this License with every copy of the Work You Distribute or Publicly Perform. You may not offer or impose any terms on the Work that restrict the terms of this License or the ability of the recipient of the Work to exercise the rights granted to that recipient under the terms of the License. You may not sublicense the Work. You must keep intact all notices that refer to this License and to the disclaimer of warranties with every copy of the Work You Distribute or Publicly Perform. When You Distribute or Publicly Perform the Work, You may not impose any effective technological measures on the Work that restrict the ability of a recipient of the Work from You to exercise the rights granted to that recipient under the terms of the License. This Section 4(a) applies to the Work as incorporated in a Collection, but this does not require the Collection apart from the Work itself to be made subject to the terms of this License. If You create a Collection, upon notice from any Licensor You must, to the extent practicable, remove from the Collection any credit as required by Section 4(c), as requested. If You create an Adaptation, upon notice from any Licensor You must, to the extent practicable, remove from the Adaptation any credit as required by Section 4(c), as requested.
b. You may Distribute or Publicly Perform an Adaptation only under the terms of: (i) this License; (ii) a later version of this License with the same License Elements as this License; (iii) a Creative Commons jurisdiction license (either this or a later license version) that contains the same License Elements as this License (e.g., Attribution-ShareAlike 3.0 US)); (iv) a Creative Commons Compatible License. If you license the Adaptation under one of the licenses mentioned in (iv), you must comply with the terms of that license. If you license the Adaptation under the terms of any of the licenses mentioned in (i), (ii) or (iii) (the "Applicable License"), you must comply with the terms of the Applicable License generally and the following provisions: (I) You must include a copy of, or the URI for, the Applicable License with every copy of each Adaptation You Distribute or Publicly Perform; (II) You may not offer or impose any terms on the Adaptation that restrict the terms of the Applicable License or the ability of the recipient of the Adaptation to exercise the rights granted to that recipient under the terms of the Applicable License; (III) You must keep intact all notices that refer to the Applicable License and to the disclaimer of warranties with every copy of the Work as included in the Adaptation You Distribute or Publicly Perform; (IV) when You Distribute or Publicly Perform the Adaptation, You may not impose any effective technological measures on the Adaptation that restrict the ability of a recipient of the Adaptation from You to exercise the rights granted to that recipient under the terms of the Applicable License. This Section 4(b) applies to the Adaptation as incorporated in a Collection, but this does not require the Collection apart from the Adaptation itself to be made subject to the terms of the Applicable License.
c. If You Distribute, or Publicly Perform the Work or any Adaptations or Collections, You must, unless a request has been made pursuant to Section 4(a), keep intact all copyright notices for the Work and provide, reasonable to the medium or means You are utilizing: (i) the name of the Original Author (or pseudonym, if applicable) if supplied, and/or if the Original Author and/or Licensor designate another party or parties (e.g., a sponsor institute, publishing entity, journal) for attribution ("Attribution Parties") in Licensor's copyright notice, terms of service or by other reasonable means, the name of such party or parties; (ii) the title of the Work if supplied; (iii) to the extent reasonably practicable, the URI, if any, that Licensor specifies to be associated with the Work, unless such URI does not refer to the copyright notice or licensing information for the Work; and (iv) , consistent with Ssection 3(b), in the case of an Adaptation, a credit identifying the use of the Work in the Adaptation (e.g., "French translation of the Work by Original Author," or "Screenplay based on original Work by Original Author"). The credit required by this Section 4(c) may be implemented in any reasonable manner; provided, however, that in the case of a Adaptation or Collection, at a minimum such credit will appear, if a credit for all contributing authors of the Adaptation or Collection appears, then as part of these credits and in a manner at least as prominent as the credits for the other contributing authors. For the avoidance of doubt, You may only use the credit required by this Section for the purpose of attribution in the manner set out above and, by exercising Your rights under this License, You may not implicitly or explicitly assert or imply any connection with, sponsorship or endorsement by the Original Author, Licensor and/or Attribution Parties, as appropriate, of You or Your use of the Work, without the separate, express prior written permission of the Original Author, Licensor and/or Attribution Parties.
d. Except as otherwise agreed in writing by the Licensor or as may be otherwise permitted by applicable law, if You Reproduce, Distribute or Publicly Perform the Work either by itself or as part of any Adaptations or Collections, You must not distort, mutilate, modify or take other derogatory action in relation to the Work which would be prejudicial to the Original Author's honor or reputation. Licensor agrees that in those jurisdictions (e.g. Japan), in which any exercise of the right granted in Section 3(b) of this License (the right to make Adaptations) would be deemed to be a distortion, mutilation, modification or other derogatory action prejudicial to the Original Author's honor and reputation, the Licensor will waive or not assert, as appropriate, this Section, to the fullest extent permitted by the applicable national law, to enable You to reasonably exercise Your right under Section 3(b) of this License (right to make Adaptations) but not otherwise.
5. Representations, Warranties and Disclaimer
UNLESS OTHERWISE MUTUALLY AGREED TO BY THE PARTIES IN WRITING, LICENSOR OFFERS THE WORK AS-IS AND MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE WORK, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF TITLE, MERCHANTIBILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY, OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT DISCOVERABLE. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO SUCH EXCLUSION MAY NOT APPLY TO YOU.
6. Limitation on Liability. EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE LAW, IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANY SPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES ARISING OUT OF THIS LICENSE OR THE USE OF THE WORK, EVEN IF LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
7. Termination
a. This License and the rights granted hereunder will terminate automatically upon any breach by You of the terms of this License. Individuals or entities who have received Adaptations or Collections from You under this License, however, will not have their licenses terminated provided such individuals or entities remain in full compliance with those licenses. Sections 1, 2, 5, 6, 7, and 8 will survive any termination of this License.
b. Subject to the above terms and conditions, the license granted here is perpetual (for the duration of the applicable copyright in the Work). Notwithstanding the above, Licensor reserves the right to release the Work under different license terms or to stop distributing the Work at any time; provided, however that any such election will not serve to withdraw this License (or any other license that has been, or is required to be, granted under the terms of this License), and this License will continue in full force and effect unless terminated as stated above.
8. Miscellaneous
a. Each time You Distribute or Publicly Perform the Work or a Collection, the Licensor offers to the recipient a license to the Work on the same terms and conditions as the license granted to You under this License.
b. Each time You Distribute or Publicly Perform an Adaptation, Licensor offers to the recipient a license to the original Work on the same terms and conditions as the license granted to You under this License.
c. If any provision of this License is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this License, and without further action by the parties to this agreement, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable.
d. No term or provision of this License shall be deemed waived and no breach consented to unless such waiver or consent shall be in writing and signed by the party to be charged with such waiver or consent.
e. This License constitutes the entire agreement between the parties with respect to the Work licensed here. There are no understandings, agreements or representations with respect to the Work not specified here. Licensor shall not be bound by any additional provisions that may appear in any communication from You. This License may not be modified without the mutual written agreement of the Licensor and You.
f. The rights granted under, and the subject matter referenced, in this License were drafted utilizing the terminology of the Berne Convention for the Protection of Literary and Artistic Works (as amended on September 28, 1979), the Rome Convention of 1961, the WIPO Copyright Treaty of 1996, the WIPO Performances and Phonograms Treaty of 1996 and the Universal Copyright Convention (as revised on July 24, 1971). These rights and subject matter take effect in the relevant jurisdiction in which the License terms are sought to be enforced according to the corresponding provisions of the implementation of those treaty provisions in the applicable national law. If the standard suite of rights granted under applicable copyright law includes additional rights not granted under this License, such additional rights are deemed to be included in the License; this License is not intended to restrict the license of any rights under applicable law.
Creative Commons Notice
Creative Commons is not a party to this License, and makes no warranty whatsoever in connection with the Work. Creative Commons will not be liable to You or any party on any legal theory for any damages whatsoever, including without limitation any general, special, incidental or consequential damages arising in connection to this license. Notwithstanding the foregoing two (2) sentences, if Creative Commons has expressly identified itself as the Licensor hereunder, it shall have all rights and obligations of Licensor.
Except for the limited purpose of indicating to the public that the Work is licensed under the CCPL, Creative Commons does not authorize the use by either party of the trademark "Creative Commons" or any related trademark or logo of Creative Commons without the prior written consent of Creative Commons. Any permitted use will be in compliance with Creative Commons' then-current trademark usage guidelines, as may be published on its website or otherwise made available upon request from time to time. For the avoidance of doubt, this trademark restriction does not form part of the License.
Creative Commons may be contacted at http://creativecommons.org/.
+170
View File
@@ -0,0 +1,170 @@
Creative Commons Attribution-ShareAlike 4.0 International
Creative Commons Corporation (“Creative Commons”) is not a law firm and does not provide legal services or legal advice. Distribution of Creative Commons public licenses does not create a lawyer-client or other relationship. Creative Commons makes its licenses and related information available on an “as-is” basis. Creative Commons gives no warranties regarding its licenses, any material licensed under their terms and conditions, or any related information. Creative Commons disclaims all liability for damages resulting from their use to the fullest extent possible.
Using Creative Commons Public Licenses
Creative Commons public licenses provide a standard set of terms and conditions that creators and other rights holders may use to share original works of authorship and other material subject to copyright and certain other rights specified in the public license below. The following considerations are for informational purposes only, are not exhaustive, and do not form part of our licenses.
Considerations for licensors: Our public licenses are intended for use by those authorized to give the public permission to use material in ways otherwise restricted by copyright and certain other rights. Our licenses are irrevocable. Licensors should read and understand the terms and conditions of the license they choose before applying it. Licensors should also secure all rights necessary before applying our licenses so that the public can reuse the material as expected. Licensors should clearly mark any material not subject to the license. This includes other CC-licensed material, or material used under an exception or limitation to copyright. More considerations for licensors.
Considerations for the public: By using one of our public licenses, a licensor grants the public permission to use the licensed material under specified terms and conditions. If the licensors permission is not necessary for any reasonfor example, because of any applicable exception or limitation to copyrightthen that use is not regulated by the license. Our licenses grant only permissions under copyright and certain other rights that a licensor has authority to grant. Use of the licensed material may still be restricted for other reasons, including because others have copyright or other rights in the material. A licensor may make special requests, such as asking that all changes be marked or described.
Although not required by our licenses, you are encouraged to respect those requests where reasonable. More considerations for the public.
Creative Commons Attribution-ShareAlike 4.0 International Public License
By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution-ShareAlike 4.0 International Public License ("Public License"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions.
Section 1 Definitions.
a. Adapted Material means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image.
b. Adapter's License means the license You apply to Your Copyright and Similar Rights in Your contributions to Adapted Material in accordance with the terms and conditions of this Public License.
c. BY-SA Compatible License means a license listed at creativecommons.org/compatiblelicenses, approved by Creative Commons as essentially the equivalent of this Public License.
d. Copyright and Similar Rights means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights.
e. Effective Technological Measures means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements.
f. Exceptions and Limitations means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material.
g. License Elements means the license attributes listed in the name of a Creative Commons Public License. The License Elements of this Public License are Attribution and ShareAlike.
h. Licensed Material means the artistic or literary work, database, or other material to which the Licensor applied this Public License.
i. Licensed Rights means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license.
j. Licensor means the individual(s) or entity(ies) granting rights under this Public License.
k. Share means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them.
l. Sui Generis Database Rights means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world.
m. You means the individual or entity exercising the Licensed Rights under this Public License. Your has a corresponding meaning.
Section 2 Scope.
a. License grant.
1. Subject to the terms and conditions of this Public License, the Licensor hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, irrevocable license to exercise the Licensed Rights in the Licensed Material to:
A. reproduce and Share the Licensed Material, in whole or in part; and
B. produce, reproduce, and Share Adapted Material.
2. Exceptions and Limitations. For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions.
3. Term. The term of this Public License is specified in Section 6(a).
4. Media and formats; technical modifications allowed. The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section 2(a)(4) never produces Adapted Material.
5. Downstream recipients.
A. Offer from the Licensor Licensed Material. Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License.
B. Additional offer from the Licensor Adapted Material. Every recipient of Adapted Material from You automatically receives an offer from the Licensor to exercise the Licensed Rights in the Adapted Material under the conditions of the Adapters License You apply.
C. No downstream restrictions. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material.
6. No endorsement. Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section 3(a)(1)(A)(i).
b. Other rights.
1. Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives and/or agrees not to assert any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise.
2. Patent and trademark rights are not licensed under this Public License.
3. To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases the Licensor expressly reserves any right to collect such royalties.
Section 3 License Conditions.
Your exercise of the Licensed Rights is expressly made subject to the following conditions.
a. Attribution.
1. If You Share the Licensed Material (including in modified form), You must:
A. retain the following if it is supplied by the Licensor with the Licensed Material:
i. identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated);
ii. a copyright notice;
iii. a notice that refers to this Public License;
iv. a notice that refers to the disclaimer of warranties;
v. a URI or hyperlink to the Licensed Material to the extent reasonably practicable;
B. indicate if You modified the Licensed Material and retain an indication of any previous modifications; and
C. indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License.
2. You may satisfy the conditions in Section 3(a)(1) in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information.
3. If requested by the Licensor, You must remove any of the information required by Section 3(a)(1)(A) to the extent reasonably practicable.
b. ShareAlike.In addition to the conditions in Section 3(a), if You Share Adapted Material You produce, the following conditions also apply.
1. The Adapters License You apply must be a Creative Commons license with the same License Elements, this version or later, or a BY-SA Compatible License.
2. You must include the text of, or the URI or hyperlink to, the Adapter's License You apply. You may satisfy this condition in any reasonable manner based on the medium, means, and context in which You Share Adapted Material.
3. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, Adapted Material that restrict exercise of the rights granted under the Adapter's License You apply.
Section 4 Sui Generis Database Rights.
Where the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material:
a. for the avoidance of doubt, Section 2(a)(1) grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database;
b. if You include all or a substantial portion of the database contents in a database in which You have Sui Generis Database Rights, then the database in which You have Sui Generis Database Rights (but not its individual contents) is Adapted Material, including for purposes of Section 3(b); and
c. You must comply with the conditions in Section 3(a) if You Share all or a substantial portion of the contents of the database.
For the avoidance of doubt, this Section 4 supplements and does not replace Your obligations under this Public License where the Licensed Rights include other Copyright and Similar Rights.
Section 5 Disclaimer of Warranties and Limitation of Liability.
a. Unless otherwise separately undertaken by the Licensor, to the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.
b. To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.
c. The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability.
Section 6 Term and Termination.
a. This Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically.
b. Where Your right to use the Licensed Material has terminated under Section 6(a), it reinstates:
1. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or
2. upon express reinstatement by the Licensor.
c. For the avoidance of doubt, this Section 6(b) does not affect any right the Licensor may have to seek remedies for Your violations of this Public License.
d. For the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License.
e. Sections 1, 5, 6, 7, and 8 survive termination of this Public License.
Section 7 Other Terms and Conditions.
a. The Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed.
b. Any arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License.
Section 8 Interpretation.
a. For the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License.
b. To the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions.
c. No term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor.
d. Nothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority.
Creative Commons is not a party to its public licenses. Notwithstanding, Creative Commons may elect to apply one of its public licenses to material it publishes and in those instances will be considered the “Licensor.” Except for the limited purpose of indicating that material is shared under a Creative Commons public license or as otherwise permitted by the Creative Commons policies published at creativecommons.org/policies, Creative Commons does not authorize the use of the trademark “Creative Commons” or any other trademark or logo of Creative Commons without its prior written consent including, without limitation, in connection with any unauthorized modifications to any of its public licenses or any other arrangements, understandings, or agreements concerning use of licensed material. For the avoidance of doubt, this paragraph does not form part of the public licenses.
Creative Commons may be contacted at creativecommons.org.
+130
View File
@@ -0,0 +1,130 @@
GNU Free Documentation License
Version 1.2, November 2002
Copyright (C) 2000,2001,2002 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
0. PREAMBLE
The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.
This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.
We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.
1. APPLICABILITY AND DEFINITIONS
This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.
A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.
A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document's overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.
The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.
The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.
A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not "Transparent" is called "Opaque".
Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.
The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work's title, preceding the beginning of the body of the text.
A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled XYZ" according to this definition.
The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.
2. VERBATIM COPYING
You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.
You may also lend copies, under the same conditions stated above, and you may publicly display copies.
3. COPYING IN QUANTITY
If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document's license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.
If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.
If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.
It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.
4. MODIFICATIONS
You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:
A. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission.
B. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has fewer than five), unless they release you from this requirement.
C. State on the Title page the name of the publisher of the Modified Version, as the publisher.
D. Preserve all the copyright notices of the Document.
E. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices.
F. Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of this License, in the form shown in the Addendum below.
G. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document's license notice.
H. Include an unaltered copy of this License.
I. Preserve the section Entitled "History", Preserve its Title, and add to it an item stating at least the title, year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no section Entitled "History" in the Document, create one stating the title, year, authors, and publisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence.
J. Preserve the network location, if any, given in the Document for public access to a Transparent copy of the Document, and likewise the network locations given in the Document for previous versions it was based on. These may be placed in the "History" section. You may omit a network location for a work that was published at least four years before the Document itself, or if the original publisher of the version it refers to gives permission.
K. For any section Entitled "Acknowledgements" or "Dedications", Preserve the Title of the section, and preserve in the section all the substance and tone of each of the contributor acknowledgements and/or dedications given therein.
L. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section numbers or the equivalent are not considered part of the section titles.
M. Delete any section Entitled "Endorsements". Such a section may not be included in the Modified Version.
N. Do not retitle any existing section to be Entitled "Endorsements" or to conflict in title with any Invariant Section.
O. Preserve any Warranty Disclaimers.
If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's license notice. These titles must be distinct from any other section titles.
You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties--for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.
You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.
The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.
5. COMBINING DOCUMENTS
You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.
The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.
In the combination, you must combine any sections Entitled "History" in the various original documents, forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sections Entitled "Dedications". You must delete all sections Entitled "Endorsements".
6. COLLECTIONS OF DOCUMENTS
You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.
You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.
7. AGGREGATION WITH INDEPENDENT WORKS
A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the compilation is not used to limit the legal rights of the compilation's users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.
If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document's Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.
8. TRANSLATION
Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.
If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.
9. TERMINATION
You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.
10. FUTURE REVISIONS OF THIS LICENSE
The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.
Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.
ADDENDUM: How to use this License for your documents
To use this License in a document you have written, include a copy of the License in the document and put the following copyright and license notices just after the title page:
Copyright (c) YEAR YOUR NAME. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".
If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the "with...Texts." line with this:
with the Invariant Sections being LIST THEIR TITLES, with the Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.
If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.
If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.
+9
View File
@@ -0,0 +1,9 @@
MIT License
Copyright (c) <year> <copyright holders>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+373
View File
@@ -0,0 +1,373 @@
Mozilla Public License Version 2.0
==================================
1. Definitions
--------------
1.1. "Contributor"
means each individual or legal entity that creates, contributes to
the creation of, or owns Covered Software.
1.2. "Contributor Version"
means the combination of the Contributions of others (if any) used
by a Contributor and that particular Contributor's Contribution.
1.3. "Contribution"
means Covered Software of a particular Contributor.
1.4. "Covered Software"
means Source Code Form to which the initial Contributor has attached
the notice in Exhibit A, the Executable Form of such Source Code
Form, and Modifications of such Source Code Form, in each case
including portions thereof.
1.5. "Incompatible With Secondary Licenses"
means
(a) that the initial Contributor has attached the notice described
in Exhibit B to the Covered Software; or
(b) that the Covered Software was made available under the terms of
version 1.1 or earlier of the License, but not also under the
terms of a Secondary License.
1.6. "Executable Form"
means any form of the work other than Source Code Form.
1.7. "Larger Work"
means a work that combines Covered Software with other material, in
a separate file or files, that is not Covered Software.
1.8. "License"
means this document.
1.9. "Licensable"
means having the right to grant, to the maximum extent possible,
whether at the time of the initial grant or subsequently, any and
all of the rights conveyed by this License.
1.10. "Modifications"
means any of the following:
(a) any file in Source Code Form that results from an addition to,
deletion from, or modification of the contents of Covered
Software; or
(b) any new file in Source Code Form that contains any Covered
Software.
1.11. "Patent Claims" of a Contributor
means any patent claim(s), including without limitation, method,
process, and apparatus claims, in any patent Licensable by such
Contributor that would be infringed, but for the grant of the
License, by the making, using, selling, offering for sale, having
made, import, or transfer of either its Contributions or its
Contributor Version.
1.12. "Secondary License"
means either the GNU General Public License, Version 2.0, the GNU
Lesser General Public License, Version 2.1, the GNU Affero General
Public License, Version 3.0, or any later versions of those
licenses.
1.13. "Source Code Form"
means the form of the work preferred for making modifications.
1.14. "You" (or "Your")
means an individual or a legal entity exercising rights under this
License. For legal entities, "You" includes any entity that
controls, is controlled by, or is under common control with You. For
purposes of this definition, "control" means (a) the power, direct
or indirect, to cause the direction or management of such entity,
whether by contract or otherwise, or (b) ownership of more than
fifty percent (50%) of the outstanding shares or beneficial
ownership of such entity.
2. License Grants and Conditions
--------------------------------
2.1. Grants
Each Contributor hereby grants You a world-wide, royalty-free,
non-exclusive license:
(a) under intellectual property rights (other than patent or trademark)
Licensable by such Contributor to use, reproduce, make available,
modify, display, perform, distribute, and otherwise exploit its
Contributions, either on an unmodified basis, with Modifications, or
as part of a Larger Work; and
(b) under Patent Claims of such Contributor to make, use, sell, offer
for sale, have made, import, and otherwise transfer either its
Contributions or its Contributor Version.
2.2. Effective Date
The licenses granted in Section 2.1 with respect to any Contribution
become effective for each Contribution on the date the Contributor first
distributes such Contribution.
2.3. Limitations on Grant Scope
The licenses granted in this Section 2 are the only rights granted under
this License. No additional rights or licenses will be implied from the
distribution or licensing of Covered Software under this License.
Notwithstanding Section 2.1(b) above, no patent license is granted by a
Contributor:
(a) for any code that a Contributor has removed from Covered Software;
or
(b) for infringements caused by: (i) Your and any other third party's
modifications of Covered Software, or (ii) the combination of its
Contributions with other software (except as part of its Contributor
Version); or
(c) under Patent Claims infringed by Covered Software in the absence of
its Contributions.
This License does not grant any rights in the trademarks, service marks,
or logos of any Contributor (except as may be necessary to comply with
the notice requirements in Section 3.4).
2.4. Subsequent Licenses
No Contributor makes additional grants as a result of Your choice to
distribute the Covered Software under a subsequent version of this
License (see Section 10.2) or under the terms of a Secondary License (if
permitted under the terms of Section 3.3).
2.5. Representation
Each Contributor represents that the Contributor believes its
Contributions are its original creation(s) or it has sufficient rights
to grant the rights to its Contributions conveyed by this License.
2.6. Fair Use
This License is not intended to limit any rights You have under
applicable copyright doctrines of fair use, fair dealing, or other
equivalents.
2.7. Conditions
Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
in Section 2.1.
3. Responsibilities
-------------------
3.1. Distribution of Source Form
All distribution of Covered Software in Source Code Form, including any
Modifications that You create or to which You contribute, must be under
the terms of this License. You must inform recipients that the Source
Code Form of the Covered Software is governed by the terms of this
License, and how they can obtain a copy of this License. You may not
attempt to alter or restrict the recipients' rights in the Source Code
Form.
3.2. Distribution of Executable Form
If You distribute Covered Software in Executable Form then:
(a) such Covered Software must also be made available in Source Code
Form, as described in Section 3.1, and You must inform recipients of
the Executable Form how they can obtain a copy of such Source Code
Form by reasonable means in a timely manner, at a charge no more
than the cost of distribution to the recipient; and
(b) You may distribute such Executable Form under the terms of this
License, or sublicense it under different terms, provided that the
license for the Executable Form does not attempt to limit or alter
the recipients' rights in the Source Code Form under this License.
3.3. Distribution of a Larger Work
You may create and distribute a Larger Work under terms of Your choice,
provided that You also comply with the requirements of this License for
the Covered Software. If the Larger Work is a combination of Covered
Software with a work governed by one or more Secondary Licenses, and the
Covered Software is not Incompatible With Secondary Licenses, this
License permits You to additionally distribute such Covered Software
under the terms of such Secondary License(s), so that the recipient of
the Larger Work may, at their option, further distribute the Covered
Software under the terms of either this License or such Secondary
License(s).
3.4. Notices
You may not remove or alter the substance of any license notices
(including copyright notices, patent notices, disclaimers of warranty,
or limitations of liability) contained within the Source Code Form of
the Covered Software, except that You may alter any license notices to
the extent required to remedy known factual inaccuracies.
3.5. Application of Additional Terms
You may choose to offer, and to charge a fee for, warranty, support,
indemnity or liability obligations to one or more recipients of Covered
Software. However, You may do so only on Your own behalf, and not on
behalf of any Contributor. You must make it absolutely clear that any
such warranty, support, indemnity, or liability obligation is offered by
You alone, and You hereby agree to indemnify every Contributor for any
liability incurred by such Contributor as a result of warranty, support,
indemnity or liability terms You offer. You may include additional
disclaimers of warranty and limitations of liability specific to any
jurisdiction.
4. Inability to Comply Due to Statute or Regulation
---------------------------------------------------
If it is impossible for You to comply with any of the terms of this
License with respect to some or all of the Covered Software due to
statute, judicial order, or regulation then You must: (a) comply with
the terms of this License to the maximum extent possible; and (b)
describe the limitations and the code they affect. Such description must
be placed in a text file included with all distributions of the Covered
Software under this License. Except to the extent prohibited by statute
or regulation, such description must be sufficiently detailed for a
recipient of ordinary skill to be able to understand it.
5. Termination
--------------
5.1. The rights granted under this License will terminate automatically
if You fail to comply with any of its terms. However, if You become
compliant, then the rights granted under this License from a particular
Contributor are reinstated (a) provisionally, unless and until such
Contributor explicitly and finally terminates Your grants, and (b) on an
ongoing basis, if such Contributor fails to notify You of the
non-compliance by some reasonable means prior to 60 days after You have
come back into compliance. Moreover, Your grants from a particular
Contributor are reinstated on an ongoing basis if such Contributor
notifies You of the non-compliance by some reasonable means, this is the
first time You have received notice of non-compliance with this License
from such Contributor, and You become compliant prior to 30 days after
Your receipt of the notice.
5.2. If You initiate litigation against any entity by asserting a patent
infringement claim (excluding declaratory judgment actions,
counter-claims, and cross-claims) alleging that a Contributor Version
directly or indirectly infringes any patent, then the rights granted to
You by any and all Contributors for the Covered Software under Section
2.1 of this License shall terminate.
5.3. In the event of termination under Sections 5.1 or 5.2 above, all
end user license agreements (excluding distributors and resellers) which
have been validly granted by You or Your distributors under this License
prior to termination shall survive termination.
************************************************************************
* *
* 6. Disclaimer of Warranty *
* ------------------------- *
* *
* Covered Software is provided under this License on an "as is" *
* basis, without warranty of any kind, either expressed, implied, or *
* statutory, including, without limitation, warranties that the *
* Covered Software is free of defects, merchantable, fit for a *
* particular purpose or non-infringing. The entire risk as to the *
* quality and performance of the Covered Software is with You. *
* Should any Covered Software prove defective in any respect, You *
* (not any Contributor) assume the cost of any necessary servicing, *
* repair, or correction. This disclaimer of warranty constitutes an *
* essential part of this License. No use of any Covered Software is *
* authorized under this License except under this disclaimer. *
* *
************************************************************************
************************************************************************
* *
* 7. Limitation of Liability *
* -------------------------- *
* *
* Under no circumstances and under no legal theory, whether tort *
* (including negligence), contract, or otherwise, shall any *
* Contributor, or anyone who distributes Covered Software as *
* permitted above, be liable to You for any direct, indirect, *
* special, incidental, or consequential damages of any character *
* including, without limitation, damages for lost profits, loss of *
* goodwill, work stoppage, computer failure or malfunction, or any *
* and all other commercial damages or losses, even if such party *
* shall have been informed of the possibility of such damages. This *
* limitation of liability shall not apply to liability for death or *
* personal injury resulting from such party's negligence to the *
* extent applicable law prohibits such limitation. Some *
* jurisdictions do not allow the exclusion or limitation of *
* incidental or consequential damages, so this exclusion and *
* limitation may not apply to You. *
* *
************************************************************************
8. Litigation
-------------
Any litigation relating to this License may be brought only in the
courts of a jurisdiction where the defendant maintains its principal
place of business and such litigation shall be governed by laws of that
jurisdiction, without reference to its conflict-of-law provisions.
Nothing in this Section shall prevent a party's ability to bring
cross-claims or counter-claims.
9. Miscellaneous
----------------
This License represents the complete agreement concerning the subject
matter hereof. If any provision of this License is held to be
unenforceable, such provision shall be reformed only to the extent
necessary to make it enforceable. Any law or regulation which provides
that the language of a contract shall be construed against the drafter
shall not be used to construe this License against a Contributor.
10. Versions of the License
---------------------------
10.1. New Versions
Mozilla Foundation is the license steward. Except as provided in Section
10.3, no one other than the license steward has the right to modify or
publish new versions of this License. Each version will be given a
distinguishing version number.
10.2. Effect of New Versions
You may distribute the Covered Software under the terms of the version
of the License under which You originally received the Covered Software,
or under the terms of any subsequent version published by the license
steward.
10.3. Modified Versions
If you create software not governed by this License, and you want to
create a new license for such software, you may create and use a
modified version of this License if you rename the license and remove
any references to the name of the license steward (except to note that
such modified license differs from this License).
10.4. Distributing Source Code Form that is Incompatible With Secondary
Licenses
If You choose to distribute Source Code Form that is Incompatible With
Secondary Licenses under the terms of this version of the License, the
notice described in Exhibit B of this License must be attached.
Exhibit A - Source Code Form License Notice
-------------------------------------------
This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at https://mozilla.org/MPL/2.0/.
If it is not possible or desirable to put the notice in a particular
file, then You may include the notice in a location (such as a LICENSE
file in a relevant directory) where a recipient would be likely to look
for such a notice.
You may add additional accurate notices of copyright ownership.
Exhibit B - "Incompatible With Secondary Licenses" Notice
---------------------------------------------------------
This Source Code Form is "Incompatible With Secondary Licenses", as
defined by the Mozilla Public License, v. 2.0.
+11
View File
@@ -0,0 +1,11 @@
zlib License
This software is provided 'as-is', without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this software.
Permission is granted to anyone to use this software for any purpose, including commercial applications, and to alter it and redistribute it freely, subject to the following restrictions:
1. The origin of this software must not be misrepresented; you must not claim that you wrote the original software. If you use this software in a product, an acknowledgment in the product documentation would be appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be misrepresented as being the original software.
3. This notice may not be removed or altered from any source distribution.
-4
View File
@@ -1,4 +0,0 @@
recursive-include ocrmypdf/jhove/bin *.jar
recursive-include ocrmypdf/jhove/lib *.jar
recursive-include ocrmypdf/jhove/conf *.conf
recursive-exclude tests/output *
-6
View File
@@ -1,6 +0,0 @@
#!/bin/sh
##############################################################################
# Copyright (c) 2013-14: fritz-hh from Github (https://github.com/fritz-hh)
##############################################################################
python3 -m ocrmypdf.main "$@"
+143
View File
@@ -0,0 +1,143 @@
<!-- SPDX-FileCopyrightText: 2014 Julien Pfefferkorn -->
<!-- SPDX-FileCopyrightText: 2015 James R. Barlow -->
<!-- SPDX-License-Identifier: CC-BY-SA-4.0 -->
<img src="docs/images/logo.svg" width="240" alt="OCRmyPDF">
[![Build Status](https://github.com/ocrmypdf/OCRmyPDF/actions/workflows/build.yml/badge.svg)](https://github.com/ocrmypdf/OCRmyPDF/actions/workflows/build.yml) [![PyPI version][pypi]](https://pypi.org/project/ocrmypdf/) ![Homebrew version][homebrew] ![ReadTheDocs][docs] ![Python versions][pyversions]
[pypi]: https://img.shields.io/pypi/v/ocrmypdf.svg "PyPI version"
[homebrew]: https://img.shields.io/homebrew/v/ocrmypdf.svg "Homebrew version"
[docs]: https://readthedocs.org/projects/ocrmypdf/badge/?version=latest "RTD"
[pyversions]: https://img.shields.io/pypi/pyversions/ocrmypdf "Supported Python versions"
OCRmyPDF adds an OCR text layer to scanned PDF files, allowing them to be searched or copy-pasted.
```bash
ocrmypdf # it's a scriptable command line program
-l eng+fra # it supports multiple languages
--rotate-pages # it can fix pages that are misrotated
--deskew # it can deskew crooked PDFs!
--title "My PDF" # it can change output metadata
--jobs 4 # it uses multiple cores by default
--output-type pdfa # it produces PDF/A by default
input_scanned.pdf # takes PDF input (or images)
output_searchable.pdf # produces validated PDF output
```
[See the release notes for details on the latest changes](https://ocrmypdf.readthedocs.io/en/latest/release_notes.html).
## Main features
- Generates a searchable [PDF/A](https://en.wikipedia.org/?title=PDF/A) file from a regular PDF
- Places OCR text accurately below the image to ease copy / paste
- Keeps the exact resolution of the original embedded images
- When possible, inserts OCR information as a "lossless" operation without disrupting any other content
- Optimizes PDF images, often producing files smaller than the input file
- If requested, deskews and/or cleans the image before performing OCR
- Validates input and output files
- Distributes work across all available CPU cores
- Uses [Tesseract OCR](https://github.com/tesseract-ocr/tesseract) engine to recognize more than [100 languages](https://github.com/tesseract-ocr/tessdata)
- Keeps your private data private.
- Scales properly to handle files with thousands of pages.
- Battle-tested on millions of PDFs.
<img src="misc/screencast/demo.svg" alt="Demo of OCRmyPDF in a terminal session">
For details: please consult the [documentation](https://ocrmypdf.readthedocs.io/en/latest/).
## Motivation
I searched the web for a free command line tool to OCR PDF files: I found many, but none of them were really satisfying:
- Either they produced PDF files with misplaced text under the image (making copy/paste impossible)
- Or they did not handle accents and multilingual characters
- Or they changed the resolution of the embedded images
- Or they generated ridiculously large PDF files
- Or they crashed when trying to OCR
- Or they did not produce valid PDF files
- On top of that none of them produced PDF/A files (format dedicated for long time storage)
...so I decided to develop my own tool.
## Installation
Linux, Windows, macOS and FreeBSD are supported. Docker images are also available, for both x64 and ARM.
| Operating system | Install command |
| ----------------------------- | ------------------------------|
| Debian, Ubuntu | ``apt install ocrmypdf`` |
| Windows Subsystem for Linux | ``apt install ocrmypdf`` |
| Fedora | ``dnf install ocrmypdf`` |
| macOS (Homebrew) | ``brew install ocrmypdf`` |
| macOS (MacPorts) | ``port install ocrmypdf`` |
| macOS (nix) | ``nix-env -i ocrmypdf`` |
| LinuxBrew | ``brew install ocrmypdf`` |
| FreeBSD | ``pkg install py-ocrmypdf`` |
| Conda | ``conda install ocrmypdf`` |
| Ubuntu Snap | ``snap install ocrmypdf`` |
For everyone else, [see our documentation](https://ocrmypdf.readthedocs.io/en/latest/installation.html) for installation steps.
## Languages
OCRmyPDF uses Tesseract for OCR, and relies on its language packs. For Linux users, you can often find packages that provide language packs:
```bash
# Display a list of all Tesseract language packs
apt-cache search tesseract-ocr
# Debian/Ubuntu users
apt-get install tesseract-ocr-chi-sim # Example: Install Chinese Simplified language pack
# Arch Linux users
pacman -S tesseract-data-eng tesseract-data-deu # Example: Install the English and German language packs
# brew macOS users
brew install tesseract-lang
```
You can then pass the `-l LANG` argument to OCRmyPDF to give a hint as to what languages it should search for. Multiple languages can be requested.
OCRmyPDF supports Tesseract 4.1.1+. It will automatically use whichever version it finds first on the `PATH` environment variable. On Windows, if `PATH` does not provide a Tesseract binary, we use the highest version number that is installed according to the Windows Registry.
## Documentation and support
Once OCRmyPDF is installed, the built-in help which explains the command syntax and options can be accessed via:
```bash
ocrmypdf --help
```
Our [documentation is served on Read the Docs](https://ocrmypdf.readthedocs.io/en/latest/index.html).
Please report issues on our [GitHub issues](https://github.com/ocrmypdf/OCRmyPDF/issues) page, and follow the issue template for quick response.
## Requirements
In addition to the required Python version (3.8+), OCRmyPDF requires external program installations of Ghostscript and Tesseract OCR. OCRmyPDF is pure Python, and runs on pretty much everything: Linux, macOS, Windows and FreeBSD.
## Press & Media
- [Going paperless with OCRmyPDF](https://medium.com/@ikirichenko/going-paperless-with-ocrmypdf-e2f36143f46a)
- [Converting a scanned document into a compressed searchable PDF with redactions](https://medium.com/@treyharris/converting-a-scanned-document-into-a-compressed-searchable-pdf-with-redactions-63f61c34fe4c)
- [c't 1-2014, page 59](https://heise.de/-2279695): Detailed presentation of OCRmyPDF v1.0 in the leading German IT magazine c't
- [heise Open Source, 09/2014: Texterkennung mit OCRmyPDF](https://heise.de/-2356670)
- [heise Durchsuchbare PDF-Dokumente mit OCRmyPDF erstellen](https://www.heise.de/ratgeber/Durchsuchbare-PDF-Dokumente-mit-OCRmyPDF-erstellen-4607592.html)
- [Excellent Utilities: OCRmyPDF](https://www.linuxlinks.com/excellent-utilities-ocrmypdf-add-ocr-text-layer-scanned-pdfs/)
- [LinuxUser Texterkennung mit OCRmyPDF und Scanbd automatisieren](https://www.linux-community.de/ausgaben/linuxuser/2021/06/texterkennung-mit-ocrmypdf-und-scanbd-automatisieren/)
- [Y Combinator discussion](https://news.ycombinator.com/item?id=32028752)
## Business enquiries
OCRmyPDF would not be the software that it is today without companies and users choosing to provide support for feature development and consulting enquiries. We are happy to discuss all enquiries, whether for extending the existing feature set, or integrating OCRmyPDF into a larger system.
## License
The OCRmyPDF software is licensed under the Mozilla Public License 2.0 (MPL-2.0). This license permits integration of OCRmyPDF with other code, included commercial and closed source, but asks you to publish source-level modifications you make to OCRmyPDF.
Some components of OCRmyPDF have other licenses, as indicated by standard SPDX license identifiers or the DEP5 copyright and licensing information file. Generally speaking, non-core code is licensed under MIT, and the documentation and test files are licensed under Creative Commons ShareAlike 4.0 (CC-BY-SA 4.0).
## Disclaimer
The software is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-113
View File
@@ -1,113 +0,0 @@
OCRmyPDF
========
OCRmyPDF adds an OCR text layer to scanned PDF files, allowing them to
be searched.
Main features
-------------
- Generates a searchable
`PDF/A <https://en.wikipedia.org/?title=PDF/A>`__ file from a PDF
file containing only images
- Places OCRed text accurately below the image to ease copy / paste
- Keeps the exact resolution of the original embedded images
- or if requested oversamples the images before OCRing so as to get
better results
- If requested deskews and/or cleans the image before performing OCR
- Validates the generated file against the PDF/A-1b specification using
`JHOVE <http://jhove.sourceforge.net/>`__
- Provides debug mode to enable easy verification of the OCR results
- Processes several pages in parallel when more than one CPU core is
available
- Uses Tesseract OCR engine
For details: please consult the `release notes <RELEASE_NOTES.rst>`__
Motivation
----------
I searched the web for a free command line tool to OCR PDF files on
Linux/UNIX: I found many, but none of them were really satisfying.
- Either they produced PDF files with misplaced text under the image (making copy/paste impossible)
- Or they did not display correctly some escaped HTML characters located in the hOCR file produced by the OCR engine
- Or they changed the resolution of the embedded images
- Or they generated PDF files having a ridiculous big size
- Or they crashed when trying to OCR some of my PDF files
- Or they did not produce valid PDF files (even though they were readable with my current PDF reader)
- On top of that none of them produced PDF/A files (format dedicated for long time storage)
... so I decided to develop my own tool (using various existing scripts
as an inspiration)
Install
-------
Download OCRmyPDF here: https://github.com/fritz-hh/OCRmyPDF/releases
You can install it within a virtual environment or system-wide.
To install the HEAD revision from sources in development mode::
git clone -b master https://github.com/fritz-hh/OCRmyPDF.git
cd OCRmyPDF
pip3 install -e .
On certain Linux/UNIX platforms such as Ubuntu, you may need to use
run the install command as superuser::
sudo pip3 install -e .
Note that this will alter your system's Python distribution. If you prefer
to not install as superuser, you can install the package in a Python virtual environment::
git clone -b master https://github.com/fritz-hh/OCRmyPDF.git
pyvenv venv
source venv/bin/activate
cd OCRmyPDF
pip3 install -e .
If your platform does not have ``pip3``, make sure that Python 3.4+ and the `pip`
package are installed.
To run the program::
ocrmypdf --help
If not yet installed, the script will notify you about dependencies that
need to be installed. The script requires specific versions of the
dependencies. Older version than the ones mentioned in the release notes
are likely not to be compatible to OCRmyPDF.
Support
-------
In case you detect an issue, please:
- Check if your issue is already known
- If no problem report exists on github, please create one here:
https://github.com/fritz-hh/OCRmyPDF/issues
- Describe your problem thoroughly
- Append the console output of the script when running the debug mode
(-v 1 option)
- If possible provide your input PDF file as well as the content of the
temporary folder (using a file sharing service like
www.file-upload.net)
Press & Media
-------------
- `c't 1-2014, page 59 <http://www.heise.de/ct/inhalt/2014/1/58/>`__:
Detailed presentation of OCRmyPDF v1.0 in the leading German IT
magazine c't
- `heise Open Source, 09/2014: Texterkennung mit
OCRmyPDF <http://www.heise.de/-2356670>`__
Disclaimer
----------
The software is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied.
-478
View File
@@ -1,478 +0,0 @@
RELEASE NOTES
=============
Please always read this file before installing the package
Download software here: https://github.com/fritz-hh/OCRmyPDF/tags
v3.0:
=====
New features
------------
- Easier installation with Python's package manager
- Now installs ``ocrmypdf`` to ``/usr/local/bin`` or equivalent for system-wide
access
- Tesseract 3.03 PDF page can be used instead for better positioning
of recognized text (``--pdf-renderer tesseract``)
- Improved command line syntax and usage help (``--help``)
- PDF metadata (title, author, keywords) are now transferred to the
output PDF
- PDF metadata can also be set from the command line (``--title``, etc.)
- Added test cases to confirm everything is working
- Added option to skip extremely large pages that take too long to OCR and are
often not OCRable (e.g. large scanned maps or diagrams); other pages are still
processed (``--skip-big``)
- Added option to kill Tesseract OCR process if it seems to be taking too long on
a page, while still processing other pages (``--tesseract-timeout``)
Changes
-------
- New, robust rewrite in Python 3.4+ with ruffus_ pipelines
- Now uses Ghostscript 9.14's improved color conversion model
- All "tasks" in the pipeline can be executed in parallel on any
available CPUs, increasing performance
- The ``-o DPI`` argument has been phased out, in favor of ``--oversample DPI``, in
case we need ``-o OUTPUTFILE`` in the future
- Removed several dependencies, so it's easier to install. We no
longer use:
- GNU parallel_
- ImageMagick_
- Python 2.7
- Poppler
- MuPDF_ tools
- shell scripts
- Some new external dependencies are required or optional, compared to v2.x:
- Ghostscript 9.14+
- qpdf_ 5.0.0+
- Unpaper_ 6.1 (optional)
- some automatically managed Python packages
.. _ruffus: http://www.ruffus.org.uk/index.html
.. _parallel: https://www.gnu.org/software/parallel/
.. _ImageMagick: http://www.imagemagick.org/script/index.php
.. _MuPDF: http://mupdf.com/docs/
.. _qpdf: http://qpdf.sourceforge.net/
.. _Unpaper: https://github.com/Flameeyes/unpaper
Release candidates
------------------
- rc4:
- dropped MuPDF in favour of qpdf
- fixed some installer issues and errors in installation instructions
- improve performance: run Ghostscript with multithreaded rendering
- improve performance: use multiple cores by default
- bug fix: checking for wrong exception on process timeout
- rc3: skipping version number intentionally to avoid confusion with Tesseract
- rc2: first release for public testing to test-PyPI, Github
- rc1: testing release process
Compatibility notes
-------------------
- ``./OCRmyPDF.sh`` script is still available for now
- Stacking the verbosity option like ``-vvv`` is no longer supported
- The configuration file ``config.sh`` has been removed. Instead, you can
feed a file to the arguments for common settings:
::
ocrmypdf input.pdf output.pdf @settings.txt
where ``settings.txt`` contains *one argument per line*, for example:
::
-l
deu
--author
A. Merkel
--pdf-renderer
tesseract
Fixes
-----
- Handling of filenames containing spaces: fixed
Notes
-----
- Some dependencies may work with lower versions than tested, so try
overriding dependencies if they are "in the way" to see if they work.
v2.2-stable (2014-09-29):
=========================
New features
------------
- None
Changes
-------
- Update to jhove v1.11
- Request the python library reportlab v3.0 or newer (So that we could remove a patch to the previous version of reportlab leading to issues for some users)
Fixes
-----
- Fix bug on Mac OS X (resolution of simlink to OCRmyPDF.sh script) (thanks to jbarlow83)
- Check if the input pdf file exists before to continue
Tested with
-----------
- Operating system: FreeBSD 9.2
- Dependencies:
- parallel 20140822
- poppler-utils 0.24.5
- ImageMagick 6.8.9-4 2014-09-17
- Unpaper 0.3
- tesseract 3.02.02
- Python 2.7.8
- ghostcript (gs): 9.06
- java: openjdk version "1.7.0_65"
v2.1-stable (2014-09-20):
=========================
New features
------------
- None
Changes
-------
- None
Fixes
-----
- Allow execution via simlink
- Add support for tesseract 3.03
- Add support for newer version of reportlab
- Lowered minimum version of gnu parallel
- Various typo
Tested with
-----------
- Operating system: FreeBSD 9.1
- Dependencies:
- parallel 20130222
- poppler-utils 0.22.2
- ImageMagick 6.8.0-7 2013-03-30
- Unpaper 0.3
- tesseract 3.02.02
- Python 2.7.3
- ghoscript (gs): 9.06
- java: openjdk version "1.7.0\_17"
v2.0-stable (2014-01-25):
=========================
New features
------------
- Check if the language(s) passed using the -l option is supported by
tesseract (fixes #60)
Changes
-------
- Allow OCRmyPDF to be used with tesseract 3.02.01, even though OCR
might fail for few PDF file (see issue #28). Rationale: For some
linux distribution, no newer version than tesseract 3.02.01 is
available
Fixes
-----
- More robust algorithm for checking the version of the installed
tesseract package
Tested with
-----------
- Operating system: FreeBSD 9.1
- Dependencies:
- parallel 20130222
- poppler-utils 0.22.2
- ImageMagick 6.8.0-7 2013-03-30
- Unpaper 0.3
- tesseract 3.02.02
- Python 2.7.3
- ghoscript (gs): 9.06
- java: openjdk version "1.7.0\_17"
v2.0-rc2 (2014-01-16):
======================
New features
------------
- None
Changes
-------
- Size reduction of final PDF file: (fixes #50)
- Support for monochrome (Black&White) images (massive size reduction
in final PDF: >80%)
- Reduced size of grayscale images (by 13% on test PDF file)
- Preventing fi, fl ligatures does not require anymore to pass an
additional config file to tesseract using the -C option (fixes #58)
- Location of temporary folder according to content of environment
variable TMPDIR.
- Dependency to pdftk removed
- Check for compatible versions of dependencies: (fixes #51)
- parallel and tesseract
- python libraries reportlab and lxml
Fixes
-----
- Improved portability with various shells (dash, bash, tcsh) and OS
(FreeBSD, MAC OSX, Linux) (fixes #59)
- Corrected bug in case the input PDF file contains a space character
(fixes #48)
- Prevent spurious error message in case there is no image in a PDF
page
- Prevent collision of temporary folder names (fixes #57)
Tested with
-----------
- Operating system: FreeBSD 9.1
- Dependencies:
- parallel 20130222
- poppler-utils 0.22.2
- ImageMagick 6.8.0-7 2013-03-30
- Unpaper 0.3
- tesseract 3.02.02
- Python 2.7.3
- ghoscript (gs): 9.06
- java: openjdk version "1.7.0\_17"
v2.0-rc1 (2014-01-07):
======================
New features
------------
- Huge performance improvement on machines having multiple CPU/cores
(processing of several pages concurrently) (fixes #18)
- By default prevent from processing a PDF file already containing
fonts (i.e. text)(it can be overridden with the -f flag) (fixes #16)
- Warn if the resolution is too low to get reasonable OCR results
(fixes #37)
- New option (-o) to perform automatic oversampling if the image
resolution is too low. This can improve OCR results.
- Warn if using a tesseract version older than v3.02.02 (as older
versions are known to produce invalid output) (fixes #41)
- Echo version of the installed dependencies (e.g. tesseract) in debug
mode in order to ease support (fixes #35)
- Echo the arguments passed to the script in debug mode to ease support
Changes
-------
- In debug mode: The debug page is now placed after the respective
"normal" page
- Reduced disk space usage in temporary folder if -d (deskew) or -c
(cleanup) options are not selected
- New file src/config.sh containing various configuration parameters
- Documentation of the tesseract config file "tess-cfg/no\_ligature"
improved
- Improved consistency of the temporary file names
Fixes
-----
- Improved robustness:
- in case vertical resolution differs from horizontal resolution (fixes
#38)
- in case a PDF page contains more than one image (fixes #36)
- Fix a problem occurring if python 3 is the standard interpreter
(fixes #33)
- Fix a problem occurring if the input PDF file contains special
characters like "#" (fixes #34)
Tested with
-----------
- Operating system: FreeBSD 9.1
- Dependencies:
- parallel 20130222
- poppler-utils 0.22.2
- ImageMagick 6.8.0-7 2013-03-30
- Unpaper 0.3
- tesseract 3.02.02
- Python 2.7.3
- pdftk 1.45
- ghoscript (gs): 9.06
- java: openjdk version "1.7.0\_17"
v1.1-stable (2014-01-06):
=========================
New features
------------
- N/A
Changes
-------
- N/A
Fixes
-----
- Fixed syntax error (bashism) leading to an error message on certain
systems (fixes #42)
Tested with
-----------
- Operating system: FreeBSD 9.1
- Dependencies:
- poppler-utils 0.22.2
- ImageMagick 6.8.0-7 2013-03-30
- Unpaper 0.3
- tesseract 3.02.02
- Python 2.7.3
- pdftk 1.45
- ghoscript (gs): 9.06
- java: openjdk version "1.7.0\_17"
v1.0-stable (2013-05-06):
=========================
New features
------------
- In debug mode: compute and echo time required for processing (fixes
#26)
Changes
-------
- Removed feature to add metadata in final pdf file (because it lead to
to final PDF file that does not comply to the PDF/A-1 format)
- Removed feature to set same owner & permissions in final PDF file
than in input file
- Removed many unused jhove files (e.g. documentation, \*.java and
\*.class files)
Fixes
-----
- Correction to handle correctly path and input PDF files having spaces
(fixes #31)
- Resolutions (x/y) that are nearly equal are now supported (fixes #25)
- Fix compatibility issue with Ubuntu server 12.04 / Ubuntu server
10.04 / Linux Mint 13 Maya and probably other Linux distributions
(fixes #27)
- Commit missing jhove files (\*.jar mainly) due to wrong .gitignore
Tested with
-----------
- Operating system: FreeBSD 9.1
- Dependencies:
- poppler-utils 0.22.2
- ImageMagick 6.8.0-7 2013-03-30
- Unpaper 0.3
- tesseract 3.02.02
- Python 2.7.3
- pdftk 1.45
- ghoscript (gs): 9.06
- java: openjdk version "1.7.0\_17"
v1.0-rc2 (2013-04-29):
======================
New features
------------
- Keep temporary files if debug mode is set (fixes #22)
- Set same owner & permissions in final PDF file than in input file
(fixes #9)
- Added metadata in final pdf file (fixes #4)
Changes
-------
- N/A
Fixes
-----
- Fixed wrong image cropping when deskew option is activated
- Exit with error message if page size is not found in hocr file (fixes
#21)
- Various minor fixes in log messages
Tested with
-----------
- Operating system: FreeBSD 9.1
- Dependencies:
- poppler-utils 0.22.2
- ImageMagick 6.8.0-7 2013-03-30
- Unpaper 0.3
- tesseract 3.02.02
- Python 2.7.3
- pdftk 1.45
- ghoscript (gs): 9.06
- java: openjdk version "1.7.0\_17"
v1.0-rc1 (2013-04-26):
======================
New features
------------
- First release candidate
Changes
-------
- N/A
Fixes
-----
- N/A
Tested with
-----------
- Operating system: FreeBSD 9.1
- Dependencies:
- poppler-utils 0.22.2
- ImageMagick 6.8.0-7 2013-03-30
- Unpaper 0.3
- tesseract 3.02.02
- Python 2.7.3
- pdftk 1.45
- ghoscript (gs): 9.06
- java: openjdk version "1.7.0\_17"
+482
View File
@@ -0,0 +1,482 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
.. SPDX-License-Identifier: CC-BY-SA-4.0
=================
Advanced features
=================
Control of unpaper
==================
OCRmyPDF uses ``unpaper`` to provide the implementation of the
``--clean`` and ``--clean-final`` arguments.
`unpaper <https://github.com/Flameeyes/unpaper/blob/main/doc/basic-concepts.md>`__
provides a variety of image processing filters to improve images.
By default, OCRmyPDF uses only ``unpaper`` arguments that were found to
be safe to use on almost all files without having to inspect every page
of the file afterwards. This is particularly true when only ``--clean``
is used, since that instructs OCRmyPDF to only clean the image before
OCR and not the final image.
However, if you wish to use the more aggressive options in ``unpaper``,
you may use ``--unpaper-args '...'`` to override the OCRmyPDF's defaults
and forward other arguments to unpaper. This option will forward
arguments to ``unpaper`` without any knowledge of what that program
considers to be valid arguments. The string of arguments must be quoted
as shown in the examples below. No filename arguments may be included.
OCRmyPDF will assume it can append input and output filename of
intermediate images to the ``--unpaper-args`` string.
In this example, we tell ``unpaper`` to expect two pages of text on a
sheet (image), such as occurs when two facing pages of a book are
scanned. ``unpaper`` uses this information to deskew each independently
and clean up the margins of both.
.. code-block:: bash
ocrmypdf --clean --clean-final --unpaper-args '--layout double' input.pdf output.pdf
ocrmypdf --clean --clean-final --unpaper-args '--layout double --no-noisefilter' input.pdf output.pdf
.. warning::
Some ``unpaper`` features will reposition text within the image.
``--clean-final`` is recommended to avoid this issue.
.. warning::
Some ``unpaper`` features cause multiple input or output files to be
consumed or produced. OCRmyPDF requires ``unpaper`` to consume one
file and produce one file; errors will result if this assumption is not
met.
.. note::
``unpaper`` uses uncompressed PBM/PGM/PPM files for its intermediate
files. For large images or documents, it can take a lot of temporary
disk space.
Control of OCR options
======================
OCRmyPDF provides many features to control the behavior of the OCR
engine, Tesseract.
When OCR is skipped
-------------------
If a page in a PDF seems to have text, by default OCRmyPDF will exit
without modifying the PDF. This is to ensure that PDFs that were
previously OCRed or were "born digital" rather than scanned are not
processed.
If ``--skip-text`` is issued, then no image processing or OCR will be
performed on pages that already have text. The page will be copied to
the output. This may be useful for documents that contain both "born
digital" and scanned content, or to use OCRmyPDF to normalize and
convert to PDF/A regardless of their contents.
If ``--redo-ocr`` is issued, then a detailed text analysis is performed.
Text is categorized as either visible or invisible. Invisible text (OCR)
is stripped out. Then an image of each page is created with visible text
masked out. The page image is sent for OCR, and any additional text is
inserted as OCR. If a file contains a mix of text and bitmap images that
contain text, OCRmyPDF will locate the additional text in images without
disrupting the existing text. Some PDF OCR solutions render text as
technically printable or visible in some way, perhaps by drawing it and
then painting over it. OCRmyPDF cannot distinguish this type of OCR
text from real text, so it will not be "redone".
If ``--force-ocr`` is issued, then all pages will be rasterized to
images, discarding any hidden OCR text, rasterizing any printable
text, and flattening form fields or interactive objects into their visual
representation. This is useful for redoing OCR, for fixing OCR text
with a damaged character map (text is selectable but not searchable),
and destroying redacted information.
Time and image size limits
--------------------------
By default, OCRmyPDF permits tesseract to run for three minutes (180
seconds) per page. This is usually more than enough time to find all
text on a reasonably sized page with modern hardware.
If a page is skipped, it will be inserted without OCR. If preprocessing
was requested, the preprocessed image layer will be inserted.
If you want to adjust the amount of time spent on OCR, change
``--tesseract-timeout``. You can also automatically skip images that
exceed a certain number of megapixels with ``--skip-big``. (A 300 DPI,
8.5×11" page image is 8.4 megapixels.)
.. code-block:: bash
# Allow 300 seconds for OCR; skip any page larger than 50 megapixels
ocrmypdf --tesseract-timeout 300 --skip-big 50 bigfile.pdf output.pdf
OCR for huge images
-------------------
Tesseract has internal limits on the size
of images it will process. By default,
``--tesseract-downsample-large-images`` is enabled, and OCRmyPDF will
downsample images to fit Tesseract limits. (The limits are usually encountered
only for scanned images of oversized media, such as large maps or blueprints exceeding
110 cm or 43 inches in either dimension, and at high DPI.) This feature can disabled
using ``--no-tesseract-downsample-large-images``.
``--tesseract-downsample-above Npixels`` adjusts the threshold at which images
will be downsampled. By default, only images that exceed any of Tesseract's
internal limits are downsampled (32767 pixels on either dimension).
You will also need to set ``--tesseract-timeout`` high enough to allow
for processing.
Only the image sent for OCR is downsampled. The original image is
preserved.
.. code-block:: bash
# Allow 600 seconds for OCR on huge images
ocrmypdf --tesseract-timeout 600 \
--tesseract-downsample-large-images \
bigfile.pdf output.pdf
# Downsample images above 5000 pixels on the longest dimension to
# 5000 pixels
ocrmypdf --tesseract-timeout 120 \
--tesseract-downsample-large-images \
--tesseract-downsample-above 5000 \
bigfile.pdf output_downsampled_ocr.pdf
Overriding default tesseract
----------------------------
OCRmyPDF checks the system ``PATH`` for the ``tesseract`` binary.
Some relevant environment variables that influence Tesseract's behavior
include:
.. envvar:: TESSDATA_PREFIX
Overrides the path to Tesseract's data files. This can allow
simultaneous installation of the "best" and "fast" training data
sets. OCRmyPDF does not manage this environment variable.
.. envvar:: OMP_THREAD_LIMIT
Controls the number of threads Tesseract will use. OCRmyPDF will
manage this environment variable if it is not already set.
For example, if you have a development build of Tesseract don't wish to
use the system installation, you can launch OCRmyPDF as follows:
.. code-block:: bash
env \
PATH=/home/user/src/tesseract/api:$PATH \
TESSDATA_PREFIX=/home/user/src/tesseract \
ocrmypdf input.pdf output.pdf
In this example ``TESSDATA_PREFIX`` is required to redirect Tesseract to
an alternate folder for its "tessdata" files.
Overriding other support programs
---------------------------------
In addition to tesseract, OCRmyPDF uses the following external binaries:
- ``gs`` (Ghostscript)
- ``unpaper``
- ``pngquant``
- ``jbig2``
In each case OCRmyPDF will search the ``PATH`` environment variable to
locate the binaries. By modifying the ``PATH`` environment variable, you
can override the binaries that OCRmyPDF uses.
Changing Tesseract configuration variables
------------------------------------------
You can override Tesseract's default `control
parameters <https://tesseract-ocr.github.io/tessdoc/tess3/ControlParams.html>`__
with a configuration file.
As an example, this configuration will disable Tesseract's dictionary
for current language. Normally the dictionary is helpful for
interpolating words that are unclear, but it may interfere with OCR if
the document does not contain many words (for example, a list of part
numbers).
Create a file named "no-dict.cfg" with these contents:
::
load_system_dawg 0
language_model_penalty_non_dict_word 0
language_model_penalty_non_freq_dict_word 0
then run ocrmypdf as follows (along with any other desired arguments):
.. code-block:: bash
ocrmypdf --tesseract-config no-dict.cfg input.pdf output.pdf
.. warning::
Some combinations of control parameters will break Tesseract or break
assumptions that OCRmyPDF makes about Tesseract's output.
Changing page segmentation mode
-------------------------------
The directive ``--tesseract-pagesegmode Nmode`` forwards the desired page segmentation
mode to Tesseract OCR. The default is 3.
Page segmentation can improve OCR results when you know that a PDF ought to be
analyzed a particular way, such as PDFs whose pages contain only a single line of
text. For the vast majority of users, changing the page segmentation mode will only
make things worse.
As of June 2024, the Tesseract page segmentation modes are:
+-----+----------------------------------------------------------------------------------+
| ID | Description |
+=====+==================================================================================+
| 0 | Orientation and script detection (OSD) only. |
+-----+----------------------------------------------------------------------------------+
| 1 | Automatic page segmentation with OSD. |
+-----+----------------------------------------------------------------------------------+
| 2 | Automatic page segmentation, but no OSD, or OCR. (not implemented) |
+-----+----------------------------------------------------------------------------------+
| 3 | Fully automatic page segmentation, but no OSD. (Default) |
+-----+----------------------------------------------------------------------------------+
| 4 | Assume a single column of text of variable sizes. |
+-----+----------------------------------------------------------------------------------+
| 5 | Assume a single uniform block of vertically aligned text. |
+-----+----------------------------------------------------------------------------------+
| 6 | Assume a single uniform block of text. |
+-----+----------------------------------------------------------------------------------+
| 7 | Treat the image as a single text line. |
+-----+----------------------------------------------------------------------------------+
| 8 | Treat the image as a single word. |
+-----+----------------------------------------------------------------------------------+
| 9 | Treat the image as a single word in a circle. |
+-----+----------------------------------------------------------------------------------+
| 10 | Treat the image as a single character. |
+-----+----------------------------------------------------------------------------------+
| 11 | Sparse text. Find as much text as possible in no particular order. |
+-----+----------------------------------------------------------------------------------+
| 12 | Sparse text with OSD. |
+-----+----------------------------------------------------------------------------------+
| 13 | Raw line. Treat the image as a single text line, bypassing hacks that are |
| | Tesseract-specific. |
+-----+----------------------------------------------------------------------------------+
Modes 0, 1, 2, and 12 (all of those that enable orientation and script detection)
are not compatible with OCRmyPDF, which performs OSD in a separate step from OCR.
Their use may interfere with ``--rotate-pages`` and other features.
It is currently not possible to use advanced Tesseract OCR features, such as creating
OCR information, when using Tesseract through OCRmyPDF.
Changing the PDF renderer
=========================
rasterizing
Converting a PDF to an image for display.
rendering
Creating a new PDF from other data (such as an existing PDF).
OCRmyPDF has these PDF renderers: ``sandwich`` and ``hocr``. The
renderer may be selected using ``--pdf-renderer``. The default is
``auto`` which lets OCRmyPDF select the renderer to use. Currently,
``auto`` always selects ``hocr``.
The ``hocr`` renderer
---------------------
.. versionchanged:: 16.0.0
In both renderers, a text-only layer is rendered and sandwiched (overlaid)
on to either the original PDF page, or newly rasterized version of the
original PDF page (when ``--force-ocr`` is used). In this way, loss
of PDF information is generally avoided. (You may need to disable PDF/A
conversion and optimization to eliminate all lossy transformations.)
The current approach used by the new hOCR renderer is a re-implementation
of Tesseract's PDF renderer, using the same Glyphless font and general
ideas, but fixing many technical issues that impeded it. The new hocr
provides better text placement accuracy, avoids issues with word
segmentation, and provides better positioning of skewed text.
Using the experimental API, it is also possible to edit the OCR output
from Tesseract, using any tool that is capable of editing hOCR files.
Older versions of this renderer did not support non-Latin languages, but
it is now universal.
The ``sandwich`` renderer
-------------------------
The ``sandwich`` renderer uses Tesseract's text-only PDF feature,
which produces a PDF page that lays out the OCR in invisible text.
Currently some problematic PDF viewers like Mozilla PDF.js and macOS
Preview have problems with segmenting its text output, and
mightrunseveralwordstogether. It also does not implement right to left
fonts (Arabic, Hebrew, Persian). The output of this renderer cannot
be edited. The sandwich renderer is retained for testing.
When image preprocessing features like ``--deskew`` are used, the
original PDF will be rendered as a full page and the OCR layer will be
placed on top.
Rendering and rasterizing options
=================================
.. versionadded:: 14.3.0
The ``--continue-on-soft-render-error`` option allows OCRmyPDF to
proceed if a page cannot be rasterized/rendered. This is useful if you are
trying to get the best possible OCR from a PDF that is not well-formed,
and you are willing to accept some pages that may not visually match the
input, and that may not OCR well.
Color conversion strategy
=========================
.. versionadded:: 15.0.0
OCRmyPDF uses Ghostscript to convert PDF to PDF/A. In some cases, this
conversion requires color conversion. The default strategy is to convert
using the ``LeaveColorUnchanged`` strategy, which preserves the original
color space wherever possible (some rare color spaces might still be
converted).
Usually document scanners produce PDFs in the sRGB color space, and do
not need to be converted, so the default strategy is appropriate.
Suppose that you have a document that was prepared for professional
printing in a Separation or CMYK color space, and text was converted to
curves. In this case, you may want to use a different color conversion
strategy. The ``--color-conversion-strategy`` option allows you to select a
different strategy, such as ``RGB``.
Return code policy
==================
OCRmyPDF writes all messages to ``stderr``. ``stdout`` is reserved for
piping output files. ``stdin`` is reserved for piping input files.
The return codes generated by the OCRmyPDF are considered part of the
stable user interface. They may be imported from
``ocrmypdf.exceptions``.
.. list-table:: Return codes
:widths: 5 35 60
:header-rows: 1
* - Code
- Name
- Interpretation
* - 0
- ``ExitCode.ok``
- Everything worked as expected.
* - 1
- ``ExitCode.bad_args``
- Invalid arguments, exited with an error.
* - 2
- ``ExitCode.input_file``
- The input file does not seem to be a valid PDF.
* - 3
- ``ExitCode.missing_dependency``
- An external program required by OCRmyPDF is missing.
* - 4
- ``ExitCode.invalid_output_pdf``
- An output file was created, but it does not seem to be a valid PDF. The file will be available.
* - 5
- ``ExitCode.file_access_error``
- The user running OCRmyPDF does not have sufficient permissions to read the input file and write the output file.
* - 6
- ``ExitCode.already_done_ocr``
- The file already appears to contain text so it may not need OCR. See output message.
* - 7
- ``ExitCode.child_process_error``
- An error occurred in an external program (child process) and OCRmyPDF cannot continue.
* - 8
- ``ExitCode.encrypted_pdf``
- The input PDF is encrypted. OCRmyPDF does not read encrypted PDFs. Use another program such as ``qpdf`` to remove encryption.
* - 9
- ``ExitCode.invalid_config``
- A custom configuration file was forwarded to Tesseract using ``--tesseract-config``, and Tesseract rejected this file.
* - 10
- ``ExitCode.pdfa_conversion_failed``
- A valid PDF was created, PDF/A conversion failed. The file will be available.
* - 15
- ``ExitCode.other_error``
- Some other error occurred.
* - 130
- ``ExitCode.ctrl_c``
- The program was interrupted by pressing Ctrl+C.
.. _tmpdir:
Changing temporary storage location
===================================
OCRmyPDF generates many temporary files during processing.
To change where temporary files are stored, change the ``TMPDIR``
environment variable for ocrmypdf's environment. (Python's
``tempfile.gettempdir()`` returns the root directory in which temporary
files will be stored.) For example, one could redirect ``TMPDIR`` to a
large RAM disk to avoid wear on HDD/SSD and potentially improve
performance.
On Windows, the ``TEMP`` environment variable is used instead.
Debugging the intermediate files
================================
OCRmyPDF normally saves its intermediate results to a temporary folder
and deletes this folder when it exits, whether it succeeded or failed.
If the ``--keep-temporary-files`` (``-k```) argument is issued on the
command line, OCRmyPDF will keep the temporary folder and print the location,
whether it succeeded or failed. An example message is:
.. code-block:: none
Temporary working files retained at:
/tmp/ocrmypdf.io.u20wpz07
The organization of this folder is an implementation detail and subject
to change between releases. However the general organization is that
working files on a per page basis have the page number as a prefix
(starting with page 1), an infix indicates the processing stage, and a
suffix indicates the file type. Some important files include:
- ``_rasterize.png`` - what the input page looks like
- ``_ocr.png`` - the file that is sent to Tesseract for OCR; depending
on arguments this may differ from the presentation image
- ``_pp_deskew.png`` - the image, after deskewing
- ``_pp_clean.png`` - the image, after cleaning with unpaper
- ``_ocr_tess.pdf`` - the OCR file; appears as a blank page with invisible
text embedded
- ``_ocr_tess.txt`` - the OCR text (not necessarily all text on the page,
if the page is mixed format)
- ``fix_docinfo.pdf`` - a temporary file created to fix the PDF DocumentInfo
data structure
- ``graft_layers.pdf`` - the rendered PDF with OCR layers grafted on
- ``pdfa.pdf`` - ``graft_layers.pdf`` after conversion to PDF/A
- ``pdfa.ps`` - a PostScript file used by Ghostscript for PDF/A conversion
- ``optimize.pdf`` - the PDF generated before optimization
- ``optimize.out.pdf`` - the PDF generated by optimization
- ``origin`` - the input file
- ``origin.pdf`` - the input file or the input image converted to PDF
- ``images/*`` - images extracted during the optimization process; here
the prefix indicates a PDF object ID not a page number
+128
View File
@@ -0,0 +1,128 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
======================
Using the OCRmyPDF API
======================
OCRmyPDF originated as a command line program and continues to have this
legacy, but parts of it can be imported and used in other Python
applications.
Some applications may want to consider running ocrmypdf from a
subprocess call anyway, as this provides isolation of its activities.
Example
=======
OCRmyPDF provides one high-level function to run its main engine from an
application. The parameters are symmetric to the command line arguments
and largely have the same functions.
.. code-block:: python
import ocrmypdf
if __name__ == '__main__': # To ensure correct behavior on Windows and macOS
ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
With some exceptions, all of the command line arguments are available
and may be passed as equivalent keywords.
A few differences are that ``verbose`` and ``quiet`` are not available.
Instead, output should be managed by configuring logging.
Parent process requirements
---------------------------
The :func:`ocrmypdf.ocr` function runs OCRmyPDF similar to command line
execution. To do this, it will:
- create worker processes or threads
- manage the signal flags of its worker processes
- execute other subprocesses (forking and executing other programs)
The Python process that calls :func:`ocrmypdf.ocr()` must be sufficiently
privileged to perform these actions.
There currently is no option to manage how jobs are scheduled other
than the argument ``jobs=`` which will limit the number of worker
processes.
Creating a child process to call :func:`ocrmypdf.ocr()` is suggested. That
way your application will survive and remain interactive even if
OCRmyPDF fails for any reason. For example:
.. code-block:: python
from multiprocessing import Process
def ocrmypdf_process():
ocrmypdf.ocr('input.pdf', 'output.pdf')
def call_ocrmypdf_from_my_app():
p = Process(target=ocrmypdf_process)
p.start()
p.join()
Programs that call :func:`ocrmypdf.ocr()` should also install a SIGBUS signal
handler (except on Windows), to raise an exception if access to a memory
mapped file fails. OCRmyPDF may use memory mapping.
:func:`ocrmypdf.ocr()` will take a threading lock to prevent multiple runs of itself
in the same Python interpreter process. This is not thread-safe, because of how
OCRmyPDF's plugins and Python's library import system work. If you need to parallelize
OCRmyPDF, use processes.
.. warning::
On Windows and macOS, the script that calls :func:`ocrmypdf.ocr()` must be
protected by an "ifmain" guard (``if __name__ == '__main__'``). If you do
not take at least one of these steps, process semantics will prevent
OCRmyPDF from working correctly.
Logging
-------
OCRmyPDF will log under loggers named ``ocrmypdf``. In addition, it
imports ``pdfminer`` and ``PIL``, both of which post log messages under
those logging namespaces.
You can configure the logging as desired for your application or call
:func:`ocrmypdf.configure_logging` to configure logging the same way
OCRmyPDF itself does. The command line parameters such as ``--quiet``
and ``--verbose`` have no equivalents in the API; you must use the
provided configuration function or do configuration in a way that suits
your use case.
Progress monitoring
-------------------
OCRmyPDF uses the ``rich`` package to implement its progress bars.
:func:`ocrmypdf.configure_logging` will set up logging output to
``sys.stderr`` in a way that is compatible with the display of the
progress bar. Use ``ocrmypdf.ocr(...progress_bar=False)`` to disable
the progress bar.
Standard output
---------------
OCRmyPDF is strict about not writing to standard output so that
users can safely use it in a pipeline and produce a valid output
file. A caller application will have to ensure it does not write to
standard output either, if it wants to be compatible with this
behavior and support piping to a file. Another benefit of running
OCRmyPDF in a child process, as recommended above, is that it will
not interfere with the parent process's standard output.
Exceptions
----------
OCRmyPDF may throw standard Python exceptions, ``ocrmypdf.exceptions.*``
exceptions, some exceptions related to multiprocessing, and
:exc:`KeyboardInterrupt`. The parent process should provide an exception
handler. OCRmyPDF will clean up its temporary files and worker processes
automatically when an exception occurs.
When OCRmyPDF succeeds conditionally, it returns an integer exit code.
+71
View File
@@ -0,0 +1,71 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
=============
API reference
=============
This page summarizes the rest of the public API. Generally speaking this
should be mainly of interest to plugin developers.
ocrmypdf
========
.. autoclass:: ocrmypdf.PageContext
:members:
.. autoclass:: ocrmypdf.PdfContext
:members:
.. autoclass:: ocrmypdf.Verbosity
:members:
:undoc-members:
.. autofunction:: ocrmypdf.configure_logging
.. autofunction:: ocrmypdf.ocr
.. autofunction:: ocrmypdf.pdf_to_hocr
.. autofunction:: ocrmypdf.hocr_to_ocr_pdf
ocrmypdf.exceptions
===================
.. automodule:: ocrmypdf.exceptions
:members:
:undoc-members:
ocrmypdf.helpers
================
.. automodule:: ocrmypdf.helpers
:members:
:noindex: deprecated
.. autodecorator:: deprecated
ocrmypdf.hocrtransform
======================
.. automodule:: ocrmypdf.hocrtransform
:members:
ocrmypdf.pdfa
=============
.. automodule:: ocrmypdf.pdfa
:members:
ocrmypdf.quality
================
.. automodule:: ocrmypdf.quality
:members:
ocrmypdf.subprocess
===================
.. automodule:: ocrmypdf.subprocess
:members:
+228
View File
@@ -0,0 +1,228 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
================
Batch processing
================
This article provides information about running OCRmyPDF on multiple
files or configuring it as a service triggered by file system events.
Batch jobs
==========
Consider using the excellent `GNU
Parallel <https://www.gnu.org/software/parallel/>`__ to apply OCRmyPDF
to multiple files at once.
Both ``parallel`` and ``ocrmypdf`` will try to use all available
processors. To maximize parallelism without overloading your system with
processes, consider using ``parallel -j 2`` to limit parallel to running
two jobs at once.
This command will run ``ocrmypdf`` on all files named ``*.pdf`` in the
current directory and write them to the previously created ``output/``
folder. It will not search subdirectories.
The ``--tag`` argument tells parallel to print the filename as a prefix
whenever a message is printed, so that one can trace any errors to the
file that produced them.
.. code-block:: bash
parallel --tag -j 2 ocrmypdf '{}' 'output/{}' ::: *.pdf
OCRmyPDF automatically repairs PDFs before parsing and gathering
information from them.
Directory trees
===============
This will walk through a directory tree and run OCR on all files in
place, and printing each filename in between runs:
.. code-block:: bash
find . -printf '%p\n' -name '*.pdf' -exec ocrmypdf '{}' '{}' \;
This only runs one ``ocrmypdf`` process at a time. This variation uses
``find`` to create a directory list and ``parallel`` to parallelize runs
of ``ocrmypdf``, again updating files in place.
.. code-block:: bash
find . -name '*.pdf' | parallel --tag -j 2 ocrmypdf '{}' '{}'
In a Windows batch file, use
.. code-block:: bat
for /r %%f in (*.pdf) do ocrmypdf %%f %%f
With a Docker container, you will need to stream through standard input and output:
.. code-block:: bash
find . -name '*.pdf' -print0 | xargs -0 | while read pdf; do
pdfout=$(mktemp)
docker run --rm -i jbarlow83/ocrmypdf - - <$pdf >$pdfout && cp $pdfout $pdf
done
Sample script
-------------
This user contributed script also provides an example of batch
processing.
.. literalinclude:: ../misc/batch.py
:caption: misc/batch.py
Synology DiskStations
---------------------
Synology DiskStations (Network Attached Storage devices) can run the
Docker image of OCRmyPDF if the Synology `Docker
package <https://www.synology.com/en-global/dsm/packages/Docker>`__ is
installed. Attached is a script to address particular quirks of using
OCRmyPDF on one of these devices.
At the time this script was written, it only worked for x86-based Synology
products. It is not known if it will work on ARM-based Synology products.
Further adjustments might be needed to deal with the Synology's relatively
limited CPU and RAM.
.. literalinclude:: ../misc/synology.py
:caption: misc/synology.py - Sample script for Synology DiskStations
Huge batch jobs
---------------
If you have thousands of files to work with, contact the author.
Consulting work related to OCRmyPDF helps fund this open source project
and all inquiries are appreciated.
Hot (watched) folders
=====================
Watched folders with watcher.py
-------------------------------
OCRmyPDF has a folder watcher called watcher.py, which is currently included in source
distributions but not part of the main program. It may be used natively or may run
in a Docker container. Native instances tend to give better performance. watcher.py
works on all platforms.
Users may need to customize the script to meet their requirements.
.. code-block:: bash
pip3 install ocrmypdf[watcher]
env OCR_INPUT_DIRECTORY=/mnt/input-pdfs \
OCR_OUTPUT_DIRECTORY=/mnt/output-pdfs \
OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1 \
python3 watcher.py
.. csv-table:: watcher.py environment variables
:header: "Environment variable", "Description"
:widths: 50, 50
"OCR_INPUT_DIRECTORY", "Set input directory to monitor (recursive)"
"OCR_OUTPUT_DIRECTORY", "Set output directory (should not be under input)"
"OCR_ARCHIVE_DIRECTORY", "Set archive directory for processed originals (should not be under input, requires ``OCR_ON_SUCCESS_ARCHIVE`` to be set)"
"OCR_ON_SUCCESS_DELETE", "This will delete the input file if the exit code is 0 (OK)"
"OCR_ON_SUCCESS_ARCHIVE", "This will move the processed original file to ``OCR_ARCHIVE_DIRECTORY`` if the exit code is 0 (OK). Note that ``OCR_ON_SUCCESS_DELETE`` takes precedence over this option, i.e. if both options are set, the input file will be deleted."
"OCR_OUTPUT_DIRECTORY_YEAR_MONTH", "This will place files in the output in ``{output}/{year}/{month}/{filename}``"
"OCR_DESKEW", "Apply deskew to crooked input PDFs"
"OCR_JSON_SETTINGS", "A JSON string specifying any other arguments for ``ocrmypdf.ocr``, e.g. ``'OCR_JSON_SETTINGS={""rotate_pages"": true, ""optimize"": "3"}'``."
"OCR_POLL_NEW_FILE_SECONDS", "Polling interval"
"OCR_LOGLEVEL", "Level of log messages to report"
One could configure a networked scanner or scanning computer to drop files in the
watched folder.
Watched folders with Docker
---------------------------
The watcher service is included in the OCRmyPDF Docker image. To run it:
.. code-block:: bash
docker run \
--volume <path to files to convert>:/input \
--volume <path to store results>:/output \
--volume <path to store processed originals>:/processed \
--env OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1 \
--env OCR_ON_SUCCESS_ARCHIVE=1 \
--env OCR_DESKEW=1 \
--env PYTHONUNBUFFERED=1 \
--interactive --tty --entrypoint python3 \
jbarlow83/ocrmypdf \
watcher.py
This service will watch for a file that matches ``/input/\*.pdf``,
convert it to a OCRed PDF in ``/output/``, and move the processed
original to ``/processed``. The parameters to this image are:
.. csv-table:: watcher.py parameters for Docker
:header: "Parameter", "Description"
:widths: 50, 50
"``--volume <path to files to convert>:/input``", "Files placed in this location will be OCRed"
"``--volume <path to store results>:/output``", "This is where OCRed files will be stored"
"``--volume <path to store processed originals>:/processed``", "Archive processed originals here"
"``--env OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1``", "Define environment variable ``OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1`` to place files in the output in ``{output}/{year}/{month}/{filename}``"
"``--env OCR_ON_SUCCESS_ARCHIVE=1``", "Define environment variable ``OCR_ON_SUCCESS_ARCHIVE`` to move processed originals"
"``--env OCR_DESKEW=1``", "Define environment variable ``OCR_DESKEW`` to apply deskew to crooked input PDFs"
"``--env PYTHONBUFFERED=1``", "This will force ``STDOUT`` to be unbuffered and allow you to see messages in docker logs"
This service relies on polling to check for changes to the filesystem. It
may not be suitable for some environments, such as filesystems shared on a
slow network.
A configuration manager such as Docker Compose could be used to ensure that the
service is always available.
.. literalinclude:: ../misc/docker-compose.example.yml
:language: yaml
:caption: misc/docker-compose.example.yml
Caveats
-------
- ``watchmedo`` may not work properly on a networked file system,
depending on the capabilities of the file system client and server.
- This simple recipe does not filter for the type of file system event,
so file copies, deletes and moves, and directory operations, will all
be sent to ocrmypdf, producing errors in several cases. Disable your
watched folder if you are doing anything other than copying files to
it.
- If the source and destination directory are the same, watchmedo may
create an infinite loop.
- On BSD, FreeBSD and older versions of macOS, you may need to increase
the number of file descriptors to monitor more files, using
``ulimit -n 1024`` to watch a folder of up to 1024 files.
Alternatives
------------
- On Linux, `systemd user services <https://wiki.archlinux.org/index.php/Systemd/User>`__
can be configured to automatically perform OCR on a collection of files.
- `Watchman <https://facebook.github.io/watchman/>`__ is a more
powerful alternative to ``watchmedo``.
macOS Automator
===============
You can use the Automator app with macOS, to create a Workflow or Quick
Action. Use a *Run Shell Script* action in your workflow. In the context
of Automator, the ``PATH`` may be set differently your Terminal's
``PATH``; you may need to explicitly set the PATH to include
``ocrmypdf``. The following example may serve as a starting point:
.. figure:: images/macos-workflow.png
:alt: Example macOS Automator workflow
You may customize the command sent to ocrmypdf.
+92
View File
@@ -0,0 +1,92 @@
.. SPDX-FileCopyrightText: 2023 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
.. _ocr-service:
==================
Online deployments
==================
OCRmyPDF is designed to be used as a command line tool, but it can be
used in a web service. This document describes some considerations for
doing so.
A basic web service implementation is provided in the source code
repository, as ``misc/webservice.py``. It is only demonstration quality
and is not intended for production use.
OCRmyPDF is not designed for use as a public web service where a
malicious user could upload a chosen PDF. In particular, it is not
necessarily secure against PDF malware or PDFs that cause denial of
service. For further discussino of security, see :ref:`security`.
OCRmyPDF relies on Ghostscript, and therefore, if deployed
online one should be prepared to comply with Ghostscript's Affero GPL
license, and any other licenses.
Setting aside these concerns, a side effect of OCRmyPDF is that it may
incidentally sanitize PDFs containing certain types of malware. It
repairs the PDF with pikepdf/libqpdf, which could correct malformed PDF
structures that are part of an attack. When PDF/A output is selected
(the default), the input PDF is partially reconstructed by Ghostscript.
When ``--force-ocr`` is used, all pages are rasterized and reconverted
to PDF, which could remove malware in embedded images.
Limiting CPU usage
------------------
OCRmyPDF will attempt to use all available CPUs and storage, so
executing ``nice ocrmypdf`` or limiting the number of jobs with the
``--jobs`` argument may ensure the server remains responsive. Another option
would be to run OCRmyPDF jobs inside a Docker container, a virtual machine,
or a cloud instance, which can impose its own limits on CPU usage and be
terminated "from orbit" if it fails to complete.
Temporary storage requirements
------------------------------
OCRmyPDF will use a large amount of temporary storage for its work,
proportional to the total number of pixels needed to rasterize the PDF.
The raster image of a 8.5×11" color page at 300 DPI takes 25 MB
uncompressed; OCRmyPDF saves its intermediates as PNG, but that still
means it requires about 9 MB per intermediate based on average
compression ratios. Multiple intermediates per page are also required,
depending on the command line given. A rule of thumb would be to allow
100 MB of temporary storage per page in a file meaning that a small
cloud servers or small VM partitions should be provisioned with plenty
of extra space, if say, a 500 page file might be sent.
To change the temporary directory, see :ref:`tmpdir`.
On Amazon Web Services or other cloud vendors, consider setting your
temporary directory to `empheral
storage <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html>`__.
Timeouts
--------
To prevent excessively long OCR jobs consider setting
``--tesseract-timeout`` and/or ``--skip-big`` arguments. ``--skip-big``
is particularly helpful if your PDFs include documents such as reports
on standard page sizes with large images attached - often large images
are not worth OCR'ing anyway.
Document management systems
---------------------------
If you are looking for a full document management system, consider
`paperless-ngx <https://github.com/paperless-ngx/paperless-ngx>`__,
which is a web application that uses OCRmyPDF to automatically OCR and
archive documents.
Commercial OCR alternatives
---------------------------
The author also provides professional services that include OCR and
building databases around PDFs, and is happy to provide consultation.
Abbyy Cloud OCR is viable commercial alternative with a web services
API. Amazon Textract, Google Cloud Vision, and Microsoft Azure
Computer Vision provide advanced OCR but have less PDF rendering capability.
Executable
+382
View File
@@ -0,0 +1,382 @@
#!/usr/bin/env python3
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: CC-BY-SA-4.0
# ruff: noqa: E402
# ocrmypdf documentation build configuration file, created by
# sphinx-quickstart on Sun Sep 4 14:29:43 2016.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.intersphinx',
'sphinx.ext.autosummary',
'sphinx.ext.napoleon',
'sphinx_issues',
]
# Extension settings
intersphinx_mapping = {'https://docs.python.org/': None}
napoleon_use_rtype = False
issues_github_path = "ocrmypdf/OCRmyPDF"
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
# The encoding of source files.
#
# source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = 'ocrmypdf'
copyright = (
'2023, James R. Barlow. Licensed under Creative Commons Attribution-ShareAlike 4.0.'
)
author = 'James R. Barlow'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
import os
from importlib.metadata import version as package_version
on_rtd = os.environ.get('READTHEDOCS') == 'True'
if on_rtd:
# Help ReadTheDocs avoid having to install any binary extension modules
import sys
from unittest.mock import MagicMock
class Mock(MagicMock):
@classmethod
def __getattr__(cls, name):
return MagicMock()
MOCK_MODULES = [
'pikepdf',
'pikepdf.models',
'pikepdf.models.metadata',
]
sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)
# The full version, including alpha/beta/rc tags.
release = package_version('ocrmypdf')
version = '.'.join(release.split('.')[:2])
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#
# today = ''
#
# Else, today_fmt is used as the format for a strftime call.
#
today_fmt = '%Y-%m-%d'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This patterns also effect to html_static_path and html_extra_path
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# The reST default role (used for this markup: `text`) to use for all
# documents.
#
# default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#
# add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#
# add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#
# show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
# modindex_common_prefix = []
# If true, keep warnings as "system message" paragraphs in the built documents.
# keep_warnings = False
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False
# -- Options for HTML output ----------------------------------------------
import sphinx_rtd_theme
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
html_theme_options = {'display_version': False}
# Add any paths that contain custom themes here, relative to this directory.
# html_theme_path = []
# The name for this set of Sphinx documents.
# "<project> v<release> documentation" by default.
#
# html_title = 'ocrmypdf v4.2'
# A shorter title for the navigation bar. Default is the same as html_title.
#
# html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#
# html_logo = "images/logo.svg" # looks bad
# The name of an image file (relative to this directory) to use as a favicon of
# the docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#
# html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
#
# html_extra_path = []
# If not None, a 'Last updated on:' timestamp is inserted at every page
# bottom, using the given strftime format.
# The empty string is equivalent to '%b %d, %Y'.
#
# html_last_updated_fmt = None
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#
# html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#
# html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#
# html_additional_pages = {}
# If false, no module index is generated.
#
# html_domain_indices = True
# If false, no index is generated.
#
# html_use_index = True
# If true, the index is split into individual pages for each letter.
#
# html_split_index = False
# If true, links to the reST sources are added to the pages.
#
# html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#
# html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#
# html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#
# html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
# html_file_suffix = None
# Language to be used for generating the HTML full-text search index.
# Sphinx supports the following languages:
# 'da', 'de', 'en', 'es', 'fi', 'fr', 'h', 'it', 'ja'
# 'nl', 'no', 'pt', 'ro', 'r', 'sv', 'tr', 'zh'
#
# html_search_language = 'en'
# A dictionary with options for the search language support, empty by default.
# 'ja' uses this config value.
# 'zh' user can custom change `jieba` dictionary path.
#
# html_search_options = {'type': 'default'}
# The name of a javascript file (relative to the configuration directory) that
# implements a search results scorer. If empty, the default will be used.
#
# html_search_scorer = 'scorer.js'
# Output file base name for HTML help builder.
htmlhelp_basename = 'ocrmypdfdoc'
# -- Options for LaTeX output ---------------------------------------------
latex_elements = { # type: ignore
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'ocrmypdf.tex', 'ocrmypdf Documentation', 'James R. Barlow', 'manual')
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#
# latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#
# latex_use_parts = False
# If true, show page references after internal links.
#
# latex_show_pagerefs = False
# If true, show URL addresses after external links.
#
# latex_show_urls = False
# Documents to append as an appendix to all manuals.
#
# latex_appendices = []
# It false, will not define \strong, \code, itleref, \crossref ... but only
# \sphinxstrong, ..., \sphinxtitleref, ... To help avoid clash with user added
# packages.
#
# latex_keep_old_macro_names = True
# If false, no module index is generated.
#
# latex_domain_indices = True
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [(master_doc, 'ocrmypdf', 'ocrmypdf Documentation', [author], 1)]
# If true, show URL addresses after external links.
#
# man_show_urls = False
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(
master_doc,
'ocrmypdf',
'ocrmypdf Documentation',
author,
'ocrmypdf',
'One line description of project.',
'Miscellaneous',
)
]
# Documents to append as an appendix to all manuals.
#
# texinfo_appendices = []
# If false, no module index is generated.
#
# texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
#
# texinfo_show_urls = 'footnote'
# If true, do not generate a @detailmenu in the "Top" node's menu.
#
# texinfo_no_detailmenu = False
+77
View File
@@ -0,0 +1,77 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
=======================
Contributing guidelines
=======================
Contributions are welcome!
Big changes
===========
Please open a new issue to discuss or propose a major change. Not only is it fun
to discuss big ideas, but we might save each other's time too. Perhaps some of the
work you're contemplating is already half-done in a development branch.
Code style
==========
We use PEP8, ``black`` for code formatting and ``ruff`` for everything else. The
settings for these programs are in ``pyproject.toml``. Pull
requests should follow the style guide. One difference we use from "black" style
is that strings shown to the user are always in double quotes (``"``) and strings
for internal uses are in single quotes (``'``).
Tests
=====
New features should come with tests that confirm their correctness.
New dependencies
================
If you are proposing a change that will require a new dependency, we
prefer dependencies that are already packaged by Debian or Red Hat. This makes
life much easier for our downstream package maintainers. A package that is only
available on PyPI or GitHub, and not more widely packaged, may not be accepted.
We are unlikely to accept a dependency on CUDA or other GPU-based libraries,
because these are still difficult to package and install on many systems.
We recommend implementing these changes as plugins.
Python dependencies must also be license-compatible. GPLv3 or AGPLv3 are likely
incompatible with the project's license, but LGPLv3 is compatible.
New non-Python dependencies
===========================
OCRmyPDF uses several external programs (Tesseract, Ghostscript and others) for
its functionality. In general we prefer to avoid adding new external programs,
and if we are to add external programs, we prefer those that are already
packaged by Debian or Red Hat.
Plugins
=======
Some new features may be a good fit for a plugin. Plugins are a way to add
features to OCRmyPDF without adding them to the core program. Plugins are
installed separately from OCRmyPDF. They are written in Python and can be
installed from PyPI. See the `plugin documentation <https://ocrmypdf.readthedocs.io/en/latest/plugins.html>`_.
We are happy to link users to your plugin from the documentation.
Style guide: Is it OCRmyPDF or ocrmypdf?
========================================
The program/project is OCRmyPDF and the name of the executable or library is ocrmypdf.
Copyright and license
=====================
For contributions over 10 lines of code, please add your name to list of
copyright holders for that file. The core program is licensed under MPL-2.0,
test files and documentation under CC-BY-SA 4.0, and miscellaneous files under
MIT, with a few minor exceptions. Please contribute only content that you own
or have the right to contribute under these licenses.
+410
View File
@@ -0,0 +1,410 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
========
Cookbook
========
Basic examples
==============
Help!
-----
ocrmypdf has built-in help.
.. code-block:: bash
ocrmypdf --help
Add an OCR layer and convert to PDF/A
-------------------------------------
.. code-block:: bash
ocrmypdf input.pdf output.pdf
Add an OCR layer and output a standard PDF
------------------------------------------
.. code-block:: bash
ocrmypdf --output-type pdf input.pdf output.pdf
Create a PDF/A with all color and grayscale images converted to JPEG
--------------------------------------------------------------------
.. code-block:: bash
ocrmypdf --output-type pdfa --pdfa-image-compression jpeg input.pdf output.pdf
Modify a file in place
----------------------
The file will only be overwritten if OCRmyPDF is successful.
.. code-block:: bash
ocrmypdf myfile.pdf myfile.pdf
Correct page rotation
---------------------
OCR will attempt to automatic correct the rotation of each page. This
can help fix a scanning job that contains a mix of landscape and
portrait pages.
.. code-block:: bash
ocrmypdf --rotate-pages myfile.pdf myfile.pdf
You can increase (decrease) the parameter ``--rotate-pages-threshold``
to make page rotation more (less) aggressive. The threshold number is the ratio
of how confidence the OCR engine is that the document image should be changed,
compared to kept the same. The default value is quite conservative; on some files
it may not attempt rotations at all unless it is very confident that the current
rotation is wrong. A lower value of ``2.0`` will produce more rotations, and
more false positives. Run with ``-v1`` to see the confidence level for each
page to see if there may be a better value for your files.
If the page is "just a little off horizontal", like a crooked picture,
then you want ``--deskew``. ``--rotate-pages`` is for when the cardinal
angle is wrong.
OCR languages other than English
--------------------------------
OCRmyPDF assumes the document is in English unless told otherwise. OCR
quality may be poor if the wrong language is used.
.. code-block:: bash
ocrmypdf -l fra LeParisien.pdf LeParisien.pdf
ocrmypdf -l eng+fra Bilingual-English-French.pdf Bilingual-English-French.pdf
Language packs must be installed for all languages specified. See
:ref:`Installing additional language packs <lang-packs>`.
Unfortunately, the Tesseract OCR engine has no ability to detect the
language when it is unknown.
Produce PDF and text file containing OCR text
---------------------------------------------
This produces a file named "output.pdf" and a companion text file named
"output.txt".
.. code-block:: bash
ocrmypdf --sidecar output.txt input.pdf output.pdf
.. note::
The sidecar file contains the **OCR text** found by OCRmyPDF. If the document
contains pages that already have text, that text will not appear in the
sidecar. If the option ``--pages`` is used, only those pages on which OCR
was performed will be included in the sidecar. If certain pages were skipped
because of options like ``--skip-big`` or ``--tesseract-timeout``, those pages
will not be in the sidecar.
If you don't want to generate the output PDF, use ``--output-type=none`` to
avoid generating one. Set the output filename to ``-`` (i.e. redirect to stdout).
To extract all text from a PDF, whether generated from OCR or otherwise,
use a program like Poppler's ``pdftotext`` or ``pdfgrep``.
OCR images, not PDFs
--------------------
Option: use Tesseract
~~~~~~~~~~~~~~~~~~~~~
If you are starting with images, you can just use Tesseract directly to
convert images to PDFs:
.. code-block:: bash
tesseract my-image.jpg output-prefix pdf
.. code-block:: bash
# When there are multiple images
tesseract text-file-containing-list-of-image-filenames.txt output-prefix pdf
Tesseract's PDF output is quite good  OCRmyPDF uses it internally, in
some cases. However, OCRmyPDF has many features not available in
Tesseract like image processing, metadata control, and PDF/A generation.
Option: use img2pdf
~~~~~~~~~~~~~~~~~~~
You can also use a program like
`img2pdf <https://gitlab.mister-muffin.de/josch/img2pdf>`__ to convert
your images to PDFs, and then pipe the results to run ocrmypdf. The
``-`` tells ocrmypdf to read standard input.
.. code-block:: bash
img2pdf my-images*.jpg | ocrmypdf - myfile.pdf
``img2pdf`` is recommended because it does an excellent job at
generating PDFs without transcoding images.
Option: use OCRmyPDF (single images only)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For convenience, OCRmyPDF can also convert single images to PDFs on its
own. If the resolution (dots per inch, DPI) of an image is not set or is
incorrect, it can be overridden with ``--image-dpi``. (As 1 inch is 2.54
cm, 1 dpi = 0.39 dpcm).
.. code-block:: bash
ocrmypdf --image-dpi 300 image.png myfile.pdf
If you have multiple images, you must use ``img2pdf`` to convert the
images to PDF.
Not recommended
~~~~~~~~~~~~~~~
We caution against using ImageMagick or Ghostscript to convert images to
PDF, since they may transcode images or produce downsampled images,
sometimes without warning.
Image processing
================
OCRmyPDF perform some image processing on each page of a PDF, if
desired. The same processing is applied to each page. It is suggested
that the user review files after image processing as these commands
might remove desirable content, especially from poor quality scans.
- ``--rotate-pages`` attempts to determine the correct orientation for
each page and rotates the page if necessary.
- ``--remove-background`` attempts to detect and remove a noisy
background from grayscale or color images. Monochrome images are
ignored. This should not be used on documents that contain color
photos as it may remove them.
- ``--deskew`` will correct pages that were scanned at a skewed angle by
rotating them back into place.
- ``--clean`` uses
`unpaper <https://www.flameeyes.eu/projects/unpaper>`__ to clean up
pages before OCR, but does not alter the final output. This makes it
less likely that OCR will try to find text in background noise.
- ``--clean-final`` uses unpaper to clean up pages before OCR and
inserts the page into the final output. You will want to review each
page to ensure that unpaper did not remove something important.
.. note::
In many cases image processing will rasterize PDF pages as images,
potentially losing quality.
.. warning::
``--clean-final`` and ``--remove-background`` may leave undesirable
visual artifacts in some images where their algorithms have
shortcomings. Files should be visually reviewed after using these
options.
Example: OCR and correct document skew (crooked scan)
-----------------------------------------------------
Deskew:
.. code-block:: bash
ocrmypdf --deskew input.pdf output.pdf
Image processing commands can be combined. The order in which options
are given does not matter. OCRmyPDF always applies the steps of the
image processing pipeline in the same order (rotate, remove background,
deskew, clean).
.. code-block:: bash
ocrmypdf --deskew --clean --rotate-pages input.pdf output.pdf
Don't actually OCR my PDF
=========================
If you set ``--tesseract-timeout 0`` OCRmyPDF will apply its image
processing without performing OCR (by causing OCR to time out). This works
if all you want to is to apply image processing or PDF/A conversion.
.. code-block:: bash
ocrmypdf --tesseract-timeout=0 --remove-background input.pdf output.pdf
.. versionchanged:: v14.1.0
Prior to this version, ``--tesseract-timeout 0`` would prevent other
uses of Tesseract, such as deskewing, from working. This is no longer
the case. Use ``--tesseract-non-ocr-timeout`` to control the timeout
for non-OCR operations, if needed.
Remove all text or OCR from my PDF
----------------------------------
This is getting ridiculous, but OCRmyPDF can complete strip all textual
information from a PDF and reconstruct it as a "bag of images" PDF.
.. code-block::
ocrmypdf --tesseract-timeout 0 --force-ocr input.pdf output.pdf
Why would you want to do this? Perhaps you have a PDF where OCR
fails to produce useful results, and just want to get rid of all OCR information.
This command also removes OCR generated by third party tools.
Optimize images without performing OCR
--------------------------------------
You can also optimize all images without performing any OCR:
.. code-block:: bash
ocrmypdf --tesseract-timeout=0 --optimize 3 --skip-text input.pdf output.pdf
Process only certain pages
--------------------------
You can ask OCRmyPDF to only apply `image processing <#image-processing>`__
and OCR to certain pages.
.. code-block:: bash
ocrmypdf --pages 2,3,13-17 input.pdf output.pdf
Hyphens denote a range of pages and commas separate page numbers. If you prefer
to use spaces, quote all of the page numbers: ``--pages '2, 3, 5, 7'``.
OCRmyPDF will warn if your list of page numbers contains duplicates or
overlapping pages. OCRmyPDF does not currently account for document page numbers,
such as an introduction section of a book that uses Roman numerals. It simply
counts the number of virtual pieces of paper since the start. If your list of
pages is out of numerical order, OCRmyPDF will sort it for you.
Regardless of the argument to ``--pages``, OCRmyPDF will optimize all pages/images
in the file and convert it to PDF/A, unless you disable those options. Both of these
steps are "whole file" operations. In this example, we want to OCR only the title
and otherwise change the PDF as little as possible:
.. code-block:: bash
ocrmypdf --pages 1 --output-type pdf --optimize 0 input.pdf output.pdf
Redo existing OCR
=================
To redo OCR on a file OCRed with other OCR software or a previous
version of OCRmyPDF and/or Tesseract, you may use the ``--redo-ocr``
argument. (Normally, OCRmyPDF will exit with an error if asked to modify
a file with OCR.)
This may be helpful for users who want to take advantage of accuracy
improvements in Tesseract for files they previously OCRed with an
earlier version of Tesseract and OCRmyPDF.
.. code-block:: bash
ocrmypdf --redo-ocr input.pdf output.pdf
This method will replace OCR without rasterizing, reducing quality or
removing vector content. If a file contains a mix of pure digital text
and OCR, digital text will be ignored and OCR will be replaced. As such
this mode is incompatible with image processing options, since they
alter the appearance of the file.
In some cases, existing OCR cannot be detected or replaced. Files
produced by OCRmyPDF v2.2 or earlier, for example, are internally
represented as having visible text with an opaque image drawn on top.
This situation cannot be detected.
If ``--redo-ocr`` does not work, you can use ``--force-ocr``, which will
force rasterization of all pages, potentially reducing quality or losing
vector content.
Improving OCR quality
=====================
The `Image processing <#image-processing>`__ features can improve OCR
quality.
Rotating pages and deskewing helps to ensure that the page orientation
is correct before OCR begins. Removing the background and/or cleaning
the page can also improve results. The ``--oversample DPI`` argument can
be specified to resample images to higher resolution before attempting
OCR; this can improve results as well.
OCR quality will suffer if the resolution of input images is not correct
(since the range of pixel sizes that will be checked for possible fonts
will also be incorrect).
PDF optimization
================
By default OCRmyPDF will attempt to perform lossless optimizations on
the images inside PDFs after OCR is complete. Optimization is performed
even if no OCR text is found.
The ``--optimize N`` (short form ``-O``) argument controls optimization,
where ``N`` ranges from 0 to 3 inclusive, analogous to the optimization
levels in the GCC compiler.
.. list-table::
:widths: auto
:header-rows: 1
* - Level
- Comments
* - ``--optimize 0``
- Disables optimization.
* - ``--optimize 1``
- Enables lossless optimizations, such as transcoding images to more
efficient formats. Also compress other uncompressed objects in the
PDF and enables the more efficient "object streams" within the PDF.
(If ``--jbig2-lossy`` is issued, then lossy JBIG2 optimization is used.
The decision to use lossy JBIG2 is separate from standard optimization
settings.)
* - ``--optimize 2``
- All of the above, and enables lossy optimizations and color quantization.
* - ``--optimize 3``
- All of the above, and enables more aggressive optimizations and targets lower image quality.
Optimization is improved when a JBIG2 encoder is available and when
``pngquant`` is installed. If either of these components are missing,
then some types of images cannot be optimized.
The types of optimization available may expand over time. By default,
OCRmyPDF compresses data streams inside PDFs, and will change
inefficient compression modes to more modern versions. A program like
``qpdf`` can be used to change encodings, e.g. to inspect the internals
for a PDF.
.. code-block:: bash
ocrmypdf --optimize 3 in.pdf out.pdf # Make it small
Some users may consider enabling lossy JBIG2. See: :ref:`jbig2-lossy`.
.. note::
Image processing and PDF/A conversion can also introduce lossy transformations
to your PDF images, even when ``--optimize 1`` is in use.
Digitally signed PDFs
=====================
OCRmyPDF cannot preserve digital signatures in PDFs and also add to OCR to them.
By default, it will refuse to modify a signed PDF regardless of other settings. You can
override this behavior with ``--invalidate-digital-signatures``; as the name suggests,
any digital signatures will be invalidated.
OCRmyPDF cannot open documents that are encrypted with a digital certificate.
Versions of OCRmyPDF prior to 14.4.0 would invalidate existing digital signatures
without warning.
+32
View File
@@ -0,0 +1,32 @@
.. SPDX-FileCopyrightText: 2023 James R. Barlow
.. SPDX-License-Identifier: CC-BY-SA-4.0
============
Design notes
============
Why doesn't OCRmyPDF use PyTesseract?
=====================================
PyTesseract is a Python wrapper around the Tesseract OCR engine. When OCRmyPDF was
first written, PyTesseract used ABI bindings to call the Tesseract library. This
was not a good fit for OCRmyPDF because ABI bindings can be fragile.
PyTesseract has since evolved calling the Tesseract executable, abandoning the ABI
approach and using the CLI instead, just like OCRmyPDF does. If it were written from
scratch today, OCRmyPDF might use PyTesseract.
PyTesseract has more features don't particularly need PDF output, but less features
than OCRmyPDF's API for creating PDFs.
What is ``executor()``?
=======================
OCRmyPDF uses a custom concurrent executor which can support either threads or
processes with the same interface. This is useful because OCRmyPDF can use
either threads or processes to parallelize work, whichever is more appropriate
for the task at hand.
The interface is currently private and subject to change. In particular, if
experiments with asyncio and anyio are successful, the interface will change.
+231
View File
@@ -0,0 +1,231 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
.. _docker:
=====================
OCRmyPDF Docker image
=====================
OCRmyPDF is also available in Docker images that packages recent
versions of all dependencies.
For users who already have Docker installed this may be an easy and
convenient option.
On platforms other than Linux, Docker runs in a virtual machine, and so may
be less performant. You may also want to adjust the Docker virtual machine's
memory and CPU allocation. On Linux, the Docker image runs natively and
performance is comparable to a system installation.
.. _docker-install:
Installing the Docker image
===========================
If you have `Docker <https://docs.docker.com/>`__ installed on your
system, you can install a Docker image of the latest release.
If you can run this command successfully, your system is ready to download and
execute the image:
.. code-block:: bash
docker run hello-world
.. list-table:: Docker images
:width: 30 20 50
:header-rows: 1
* - Image
- Architecture
- Description
* - ``jbarlow83/ocrmypdf-alpine``
- x86_64 and arm64
- Recommended image, based on Alpine Linux.
* - ``jbarlow83/ocrmypdf-ubuntu``
- x86_64 and arm64
- Alternate image, based on Ubuntu. When the Alpine image is considered
stable and available for arm64, this image will be deprecated.
* - ``jbarlow83/ocrmypdf``
- x86_64 and arm64
- Currently an alias for ocrmypdf-ubuntu. When the Alpine image is
considered stable and available for arm64, this name point to the
Alpine image. If you don't about the difference between Alpine and
Ubuntu, use this image.
To install:
.. code-block:: bash
docker pull jbarlow83/ocrmypdf-alpine
The ``ocrmypdf`` image is also available, but is deprecated and will be removed
in the future.
OCRmyPDF will use all available CPU cores. See the Docker documentation for
`adjusting memory and CPU on other platforms <https://docs.docker.com/config/containers/resource_constraints/>`__.
Using the Docker image on the command line
==========================================
**Unlike typical Docker containers**, in this section the OCRmyPDF Docker
container is ephemeral it runs for one OCR job and terminates, just like a
command line program. We are using Docker to deliver an application (as opposed
to the more conventional case, where a Docker container runs as a server).
For that reason we usually use the ``--rm`` argument to delete the container
when it exits.
To start a Docker container (instance of the image):
.. code-block:: bash
docker tag jbarlow83/ocrmypdf-alpine ocrmypdf
docker run --rm -i ocrmypdf (... all other arguments here...) - -
For convenience, create a shell alias to hide the Docker command. It is
easier to send the input file as stdin and read the output from
stdout **this avoids the messy permission issues with Docker entirely**.
.. code-block:: bash
alias docker_ocrmypdf='docker run --rm -i ocrmypdf'
docker_ocrmypdf --version # runs docker version
docker_ocrmypdf - - <input.pdf >output.pdf
Or in the wonderful `fish shell <https://fishshell.com/>`__:
.. code-block:: fish
alias docker_ocrmypdf 'docker run --rm ocrmypdf'
funcsave docker_ocrmypdf
Alternately, you could mount the local current working directory as a
Docker volume:
.. code-block:: bash
alias docker_ocrmypdf='docker run --rm -i --user "$(id -u):$(id -g)" --workdir /data -v "$PWD:/data" ocrmypdf'
docker_ocrmypdf /data/input.pdf /data/output.pdf
.. _docker-lang-packs:
Adding languages to the Docker image
====================================
By default the Docker image includes English, German, Simplified Chinese,
French, Portuguese and Spanish, the most popular languages for OCRmyPDF
users based on feedback. You may add other languages by creating a new
Dockerfile based on the public one.
.. code-block:: dockerfile
FROM jbarlow83/ocrmypdf
# Example: add Italian
RUN apt install tesseract-ocr-ita
To install language packs (training data) such as the
`tessdata_best <https://github.com/tesseract-ocr/tessdata_best>`_ suite or
custom data, you first need to determine the version of Tesseract data files, which
may differ from the Tesseract program version. Use this command to determine the data
file version:
.. code-block:: bash
docker run -i --rm --entrypoint /bin/ls jbarlow83/ocrmypdf /usr/share/tesseract-ocr
As of 2021, the data file version is probably ``4.00``.
You can then add new data with either a Dockerfile:
.. code-block:: dockerfile
FROM jbarlow83/ocrmypdf:{TAG}
# Example: add a tessdata_best file
COPY chi_tra_vert.traineddata /usr/share/tesseract-ocr/<data version>/tessdata/
When creating your own image, you should always pin a specific version of the
OCRmyPDF Docker image. This ensures that your image will not break when a new
version of OCRmyPDF is released.
Alternately, you can copy training data into a Docker container as follows:
.. code-block:: bash
docker cp mycustomtraining.traineddata name_of_container:/usr/share/tesseract-ocr/<tesseract version>/tessdata/
Extending the Docker image
==========================
You can extend the Docker image with your own customizations, similar to the way
it is extended to add language packs.
Note that the Docker image is subject to change at any time. For example, the base
image may be updated to a newer version of Ubuntu or Debian. Such changes will be
noted in the release notes but might occur at minor versions releases, unless the
way a "casual" user of the Docker image is affected.
If you extend the Docker image, you should pin a specific version of the OCRmyPDF
Docker image.
Executing the test suite
========================
The OCRmyPDF test suite is installed with image. To run it:
.. code-block:: bash
docker run --rm --entrypoint python jbarlow83/ocrmypdf -m pytest
Accessing the shell
===================
To use the shell in the Docker image:
.. code-block:: bash
docker run -it --entrypoint sh jbarlow83/ocrmypdf
Using the OCRmyPDF web service wrapper
======================================
The OCRmyPDF Docker image includes an example, barebones HTTP web
service. The webservice may be launched as follows:
.. code-block:: bash
docker run --entrypoint python -p 5000:5000 jbarlow83/ocrmypdf webservice.py
We omit the ``--rm`` parameter so that the container will not be
automatically deleted when it exits.
This will configure the machine to listen on port 5000. On Linux machines
this is port 5000 of localhost. On macOS or Windows machines running
Docker, this is port 5000 of the virtual machine that runs your Docker
images. You can find its IP address using the command ``docker-machine ip``.
Unlike command line usage this program will open a socket and wait for
connections.
.. warning::
The OCRmyPDF web service wrapper is intended for demonstration or
development. It provides no security, no authentication, no
protection against denial of service attacks, and no load balancing.
The default Flask WSGI server is used, which is intended for
development only. The server is single-threaded and so can respond to
only one client at a time. While running OCR, it cannot respond to
any other clients.
Clients must keep their open connection while waiting for OCR to
complete. This may entail setting a long timeout; this interface is more
useful for internal HTTP API calls.
Unlike the rest of OCRmyPDF, this web service is licensed under the
Affero GPLv3 (AGPLv3) since Ghostscript is also licensed in this way.
In addition to the above, please read our
:ref:`general remarks on using OCRmyPDF as a service <ocr-service>`.
+57
View File
@@ -0,0 +1,57 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
=====================
Common error messages
=====================
Page already has text
=====================
.. code-block::
ERROR - 1: page already has text! aborting (use --force-ocr to force OCR)
You ran ocrmypdf on a file that already contains printable text or a
hidden OCR text layer (it can't quite tell the difference). You probably
don't want to do this, because the file is already searchable.
As the error message suggests, your options are:
- ``ocrmypdf --force-ocr`` to :ref:`rasterize <raster-vector>` all
vector content and run OCR on the images. This is useful if a
previous OCR program failed, or if the document contains a text
watermark.
- ``ocrmypdf --skip-text`` to skip OCR and other processing on any
pages that contain text. Text pages will be copied into the output
PDF without modification.
- ``ocrmypdf --redo-ocr`` to scan the file for any existing OCR
(non-printing text), remove it, and do OCR again. This is one way
to take advantage of improvements in OCR accuracy. Printable vector
text is excluded from OCR, so this can be used on files that contain
a mix of digital and scanned files.
Input file 'filename' is not a valid PDF
========================================
OCRmyPDF checks files with pikepdf, a library that in turn uses libqpdf to fixes
errors in PDFs, before it tries to work on them. In most cases this happens
because the PDF is corrupt and truncated (incomplete file copying) and not much
can be done.
You can try rewriting the file with Ghostscript:
.. code-block:: bash
gs -o output.pdf -dSAFER -sDEVICE=pdfwrite input.pdf
``pdftk`` can also rewrite PDFs:
.. code-block:: bash
pdftk input.pdf cat output output.pdf
Sometimes Acrobat can repair PDFs with its `Preflight
tool <https://helpx.adobe.com/acrobat/using/correcting-problem-areas-preflight-tool.html>`__.
+35
View File
@@ -0,0 +1,35 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" width="100%" height="100%" viewBox="0 0 915 585" xmlns:xlink="http://www.w3.org/1999/xlink">
<defs>
<linearGradient id="b" y2="445" gradientUnits="userSpaceOnUse" y1="179" gradientTransform="translate(0 -2.06)" x2="-29.7" x1="322">
<stop stop-color="#333" offset="0"/>
<stop stop-color="#fff" stop-opacity="0" offset="1"/>
</linearGradient>
<linearGradient id="a" y2="414" gradientUnits="userSpaceOnUse" y1="159" x2="490" x1="815">
<stop stop-color="#33f" offset="0"/>
<stop stop-color="#3f3fff" stop-opacity="0" offset="1"/>
</linearGradient>
</defs>
<path fill="url(#b)" d="m403 247c-12 115-135 122-368 123-4.3-1.07-7.32-7.33-6-41 76-37 151-124 167-236 93 123 201 40.9 207 154z"/>
<g stroke-width="3.45" fill="none">
<path stroke="#000" d="m11.8 11.8h411v411l-411 0.01v-411z"/>
<path stroke="#448" d="m489 11.7h415v411h-415v-411z"/>
</g>
<path d="m876 244c-12 115-133 120-366 121-6-14-10-40-3-43 76-37.3 136-106 152-218 38 48 209 101 217 140z" fill="url(#a)"/>
<g id="RasterLarge" transform="matrix(1.36 0 0 1.28 -161 -636)">
<path fill="#999" d="m287 730h-30v-60h20v20h10v10h10v10h20v10h40v-10h20v-20h10v-20h-10v-10h-10v-10h-40v-10h-30v-10h-20v-10h-10v-10h-10v-60h10v-10h10v-10h10v-10h90v10h30v60h-20v-10h-10v-20h-10v-10h-20v-10h-30v10h-20v10h-10v20h10v20h30v10h30v10h30v10h10v10h10v10h10v60h-10v10h-10v10h-10v10h-100v-10z"/>
<path fill="#555" d="m297 730h-30v-10h-10v-50h20v20h10v20h10 10v10h60v-10h10v-10h10v-40h-20v-10h-20v-10h-40v-10h-20v-10h-20v-20h-10v-50h10v-10h10v-10h20v-10h70v10h30v10h10v50h-20v-20h-10v-20h-20v-10h-50v10h-10v10h-10v30h10v10h20v10h30v10h30v10h20v10h10v10h10v50h-10v20h-10v10h-20v10h-80z"/>
<path d="m307 730h-30v-10h-20v-50h20v30h10v10h10v10h70v-10h20v-50h-10v-10h-20v-10h-40v-10h-30v-10h-10v-10h-10v-20h-10v-30h10v-20h20v-10h20v-10h50v10h30v10h20v50h-20v-30h-10v-10h-10v-10h-70v10h-10v40h10v10h10v10h30v10h40v10h20v10h10v20h10v40h-10v20h-20v10h-30v10h-50v-10z"/>
</g>
<g font-size="40" font-family="sans-serif" text-anchor="middle">
<g font-size="100">
<text y="518" x="210">Raster</text>
<text y="518" x="695" fill="#338">Vector</text>
</g>
<text y="563" x="210">.jpeg .gif .png</text>
<text y="563" x="696" fill="#338">.svg</text>
</g>
<path id="VectorLarge" fill="#005" d="m661 294v-62.5l23.4 0.184c0.678 20.8 7.32 36.3 19.9 46.4 12.7 9.93 32.1 14.9 57.9 14.9 24.1 0 42.5-4.29 55.1-12.9 12.7-8.71 19.1-21.3 19.1-37.9 0-13.2-3.86-23.4-11.6-30.5-7.59-7.11-23.7-14-48.4-20.8l-40.1-10.9c-29-7.97-49.5-17.9-61.4-29.8-11.8-11.9-17.7-28.2-17.7-48.9 0-23.3 9.15-41.4 27.4-54.3 18.3-12.9 44-19.3 77.1-19.3 14.1 0 29.6 1.41 46.4 4.23 16.8 2.7 34.7 6.68 53.7 12v58.5h-23c-2.3-19.4-9.49-33.4-21.6-41.9-11.9-8.71-30.2-13.1-54.7-13.1-21.4 0-37.7 3.99-49 12-11.1 7.85-16.7 19.3-16.7 34.4 0 13.1 4.2 23.4 12.6 30.9 8.4 7.48 26.2 14.9 53.5 22.3l37.6 10.1c27.5 7.48 47.1 17 58.8 28.7 11.8 11.5 17.7 27 17.7 46.5 0 26.6-9.42 46.7-28.3 60.2s-46.9 20.2-84.2 20.2c-16.7 0-33.7-1.53-51-4.6-18-3-35-7-53-13z"/>
<use xlink:href="#VectorLarge" transform="matrix(.17 0 0 .17 392 313)"/>
<use xlink:href="#RasterLarge" transform="matrix(.173 0 0 .173 -8.25 314)" height="100%" width="100%" y="0" x="0"/>
</svg>

After

Width:  |  Height:  |  Size: 3.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

+239
View File
@@ -0,0 +1,239 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg
width="256"
height="256"
viewBox="0 0 256 256.00001"
version="1.1"
xml:space="preserve"
style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"
id="svg270"
sodipodi:docname="logo-square-256.svg"
inkscape:export-filename="/home/jb/src/ocrmypdf/docs/images/logo-square.png"
inkscape:export-xdpi="96"
inkscape:export-ydpi="96"
inkscape:version="1.1.2 (0a00cf5339, 2022-02-04)"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns="http://www.w3.org/2000/svg"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:serif="http://www.serif.com/"><metadata
id="metadata276"><rdf:RDF><cc:Work
rdf:about=""><dc:format>image/svg+xml</dc:format><dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" /></cc:Work></rdf:RDF></metadata><defs
id="defs274" /><sodipodi:namedview
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1"
objecttolerance="10"
gridtolerance="10"
guidetolerance="10"
inkscape:pageopacity="0"
inkscape:pageshadow="2"
inkscape:window-width="2396"
inkscape:window-height="1691"
id="namedview272"
showgrid="false"
lock-margins="false"
inkscape:zoom="2.0079523"
inkscape:cx="189.74554"
inkscape:cy="54.533168"
inkscape:window-x="26"
inkscape:window-y="23"
inkscape:window-maximized="0"
inkscape:current-layer="svg270"
inkscape:pagecheckerboard="0"
width="256px"
fit-margin-top="0"
fit-margin-left="0"
fit-margin-right="0"
fit-margin-bottom="0" />
<g
id="svg"
transform="matrix(0.48534351,0,0,0.4057699,1.8106874,71.192214)">
<rect
x="0"
y="0"
width="520"
height="280"
style="fill:#ffffff"
id="rect188" />
<g
transform="matrix(1.03522,0,0,1.23823,-69.7528,-83.422)"
id="g267">
<g
transform="translate(243.977,20.0703)"
id="g218">
<g
id="Page">
<g
transform="matrix(0.961773,0,0,1.05962,6.19811,-3.01071)"
id="g192">
<path
d="m 328.5,97.682 c 0,-1.217 -0.517,-2.386 -1.444,-3.264 -7.03,-6.66 -37.614,-35.638 -44.828,-42.474 -0.977,-0.925 -2.327,-1.448 -3.738,-1.448 -13.997,0 -90.407,0 -111.151,0 -2.871,0 -5.198,2.113 -5.198,4.718 0,27.837 0,170.351 0,198.186 0,2.605 2.327,4.717 5.197,4.717 24.904,0 131.821,0 156.2,0 2.74,0 4.962,-2.016 4.962,-4.504 0,-24.345 0,-139.717 0,-155.931 z"
style="fill:#fdfdfd;stroke:#333333;stroke-width:3.95px"
id="path190" />
</g>
<g
id="Dog-ear"
serif:id="Dog ear"
transform="translate(-4,2)">
<path
d="m 277.072,48.496 v 45.352 c 0,1.324 0.526,2.593 1.462,3.529 0.936,0.936 2.205,1.462 3.529,1.462 12.485,0 44.078,0 44.078,0"
style="fill:#f5f5f5;stroke:#333333;stroke-width:4px"
id="path194" />
</g>
</g>
<g
transform="translate(-29.6816,-0.395178)"
id="g216">
<g
transform="matrix(1.00243,0,0,1.11818,-144.72,-8.80181)"
id="g200">
<path
d="m 465.73,119.654 c 0,-2.049 -1.856,-3.713 -4.142,-3.713 H 310.259 c -2.286,0 -4.142,1.664 -4.142,3.713 v 63.454 c 0,2.049 1.856,3.713 4.142,3.713 h 151.329 c 2.286,0 4.142,-1.664 4.142,-3.713 z"
style="fill:#f80000;stroke:#ffffff;stroke-width:3.77px"
id="path198" />
</g>
<g
transform="matrix(1.24571,0,0,1.35864,116.812,84.3924)"
id="g214">
<g
transform="matrix(64,0,0,64,42.1437,77.6203)"
id="g204">
<path
d="m 0.084,0 v -0.68 h 0.213 c 0.074,0 0.137,0.017 0.19,0.05 0.053,0.034 0.079,0.09 0.079,0.168 0,0.077 -0.028,0.134 -0.085,0.17 -0.057,0.037 -0.121,0.055 -0.193,0.055 H 0.213 V 0 Z m 0.209,-0.572 h -0.08 v 0.228 h 0.082 c 0.039,0 0.07,-0.009 0.094,-0.027 0.024,-0.017 0.037,-0.045 0.04,-0.083 0,-0.044 -0.012,-0.075 -0.036,-0.092 -0.024,-0.017 -0.057,-0.026 -0.1,-0.026 z"
style="fill:#ffffff;fill-rule:nonzero"
id="path202" />
</g>
<g
transform="matrix(64,0,0,64,79.7117,77.6203)"
id="g208">
<path
d="M 0.332,0 H 0.084 v -0.68 h 0.252 c 0.105,0 0.182,0.032 0.233,0.095 0.051,0.063 0.076,0.144 0.076,0.241 0,0.105 -0.027,0.189 -0.082,0.251 C 0.508,-0.031 0.431,0 0.332,0 Z M 0.337,-0.57 H 0.213 v 0.461 H 0.33 c 0.055,0 0.099,-0.018 0.132,-0.054 C 0.495,-0.199 0.511,-0.259 0.511,-0.344 0.511,-0.415 0.497,-0.47 0.469,-0.51 0.441,-0.55 0.397,-0.57 0.337,-0.57 Z"
style="fill:#ffffff;fill-rule:nonzero"
id="path206" />
</g>
<g
transform="matrix(64,0,0,64,123.424,77.6203)"
id="g212">
<path
d="M 0.405,-0.288 H 0.213 V 0 H 0.084 v -0.68 h 0.385 l 0.02,0.102 H 0.213 v 0.189 h 0.173 z"
style="fill:#ffffff;fill-rule:nonzero"
id="path210" />
</g>
</g>
</g>
</g>
<g
transform="matrix(1,0,0,1.52217,67.3796,10.7507)"
id="g222">
<rect
x="23.500999"
y="81.300003"
width="162.30499"
height="61.77"
style="fill:#b4d5ff"
id="rect220" />
</g>
<g
transform="matrix(0.967536,0,0,0.961535,5.90498,47.9703)"
id="g236">
<g
transform="matrix(90.4804,0,0,90.4804,82.6698,167.705)"
id="g226">
<path
d="m 0.057,-0.337 c 0,-0.105 0.027,-0.19 0.082,-0.257 0.055,-0.066 0.132,-0.1 0.231,-0.102 0.107,0 0.186,0.034 0.237,0.103 0.051,0.069 0.077,0.152 0.077,0.249 0,0.105 -0.027,0.191 -0.082,0.258 -0.055,0.067 -0.133,0.1 -0.232,0.1 C 0.264,0.014 0.185,-0.02 0.134,-0.089 0.083,-0.157 0.057,-0.24 0.057,-0.337 Z m 0.135,-0.001 c 0,0.071 0.014,0.13 0.043,0.175 0.029,0.045 0.073,0.068 0.134,0.068 0.055,0 0.098,-0.02 0.131,-0.061 0.033,-0.041 0.049,-0.103 0.049,-0.188 0,-0.071 -0.014,-0.129 -0.043,-0.174 -0.029,-0.045 -0.073,-0.068 -0.134,-0.068 -0.053,0 -0.097,0.022 -0.13,0.067 -0.033,0.045 -0.05,0.105 -0.05,0.181 z"
style="fill:#333333;fill-rule:nonzero"
id="path224" />
</g>
<g
transform="matrix(90.4804,0,0,90.4804,147.906,167.705)"
id="g230">
<path
d="M 0.505,-0.557 C 0.473,-0.567 0.448,-0.574 0.429,-0.579 0.41,-0.583 0.388,-0.585 0.361,-0.585 c -0.054,0 -0.096,0.022 -0.125,0.066 -0.029,0.044 -0.044,0.104 -0.044,0.181 0,0.066 0.012,0.123 0.037,0.171 0.025,0.048 0.066,0.072 0.124,0.072 0.029,0 0.056,-0.003 0.081,-0.009 0.025,-0.006 0.047,-0.013 0.068,-0.022 L 0.551,-0.03 C 0.525,-0.017 0.494,-0.006 0.457,0.002 0.42,0.01 0.388,0.014 0.36,0.014 0.254,0.014 0.177,-0.02 0.129,-0.088 0.081,-0.156 0.057,-0.239 0.057,-0.337 c 0,-0.105 0.027,-0.19 0.08,-0.257 0.053,-0.067 0.129,-0.1 0.228,-0.1 0.02,0 0.048,0.003 0.083,0.01 0.035,0.007 0.068,0.018 0.097,0.034 z"
style="fill:#333333;fill-rule:nonzero"
id="path228" />
</g>
<g
transform="matrix(90.4804,0,0,90.4804,199.751,167.705)"
id="g234">
<path
d="m 0.293,-0.572 h -0.08 v 0.208 h 0.082 c 0.039,0 0.071,-0.008 0.096,-0.024 0.025,-0.015 0.038,-0.041 0.038,-0.077 0,-0.038 -0.012,-0.065 -0.036,-0.082 -0.024,-0.017 -0.057,-0.025 -0.1,-0.025 z M 0.479,0 0.335,-0.26 C 0.328,-0.259 0.32,-0.259 0.312,-0.259 0.304,-0.258 0.296,-0.258 0.288,-0.258 H 0.213 V 0 H 0.084 v -0.68 h 0.213 c 0.074,0 0.137,0.017 0.19,0.051 0.053,0.034 0.079,0.087 0.079,0.158 0,0.042 -0.011,0.078 -0.032,0.108 -0.022,0.031 -0.05,0.054 -0.084,0.071 L 0.617,0 Z"
style="fill:#333333;fill-rule:nonzero"
id="path232" />
</g>
</g>
<g
transform="matrix(0.916882,0,0,1,121.475,-32.6535)"
id="g246">
<g
transform="matrix(86.953,0,0,86.953,152.996,241.878)"
id="g240">
<path
d="M 0.479,-0.428 C 0.5,-0.451 0.527,-0.47 0.562,-0.484 c 0.034,-0.013 0.065,-0.02 0.092,-0.02 0.066,0 0.113,0.019 0.141,0.058 0.027,0.039 0.041,0.086 0.041,0.142 V 0 H 0.705 v -0.298 c 0,-0.031 -0.007,-0.054 -0.022,-0.071 -0.015,-0.016 -0.036,-0.024 -0.064,-0.024 -0.019,0 -0.038,0.005 -0.059,0.015 -0.021,0.01 -0.039,0.021 -0.056,0.034 0.001,0.007 0.001,0.013 0.002,0.02 0.001,0.007 0.001,0.013 0.001,0.02 V 0 H 0.376 v -0.298 c 0,-0.031 -0.007,-0.054 -0.022,-0.071 -0.015,-0.016 -0.036,-0.024 -0.063,-0.024 -0.017,0 -0.033,0.003 -0.05,0.01 -0.017,0.007 -0.034,0.016 -0.049,0.027 V 0 H 0.062 V -0.485 H 0.13 l 0.032,0.044 c 0.022,-0.02 0.049,-0.035 0.08,-0.047 0.031,-0.011 0.058,-0.016 0.083,-0.016 0.038,0 0.07,0.007 0.095,0.02 0.025,0.014 0.045,0.033 0.059,0.056 z"
style="fill:#333333;fill-rule:nonzero"
id="path238" />
</g>
<g
transform="matrix(86.953,0,0,86.953,228.906,241.878)"
id="g244">
<path
d="M 0.156,0.023 0.179,-0.034 0.006,-0.467 0.14,-0.485 0.252,-0.191 0.358,-0.485 H 0.495 L 0.278,0.064 C 0.263,0.103 0.236,0.137 0.197,0.165 0.158,0.193 0.118,0.212 0.075,0.222 L 0.029,0.115 C 0.052,0.105 0.077,0.093 0.104,0.079 0.13,0.064 0.147,0.046 0.156,0.023 Z"
style="fill:#333333;fill-rule:nonzero"
id="path242" />
</g>
</g>
<g
id="Selectors"
transform="matrix(0.965977,0,0,0.807602,67.3796,67.3718)">
<g
id="Right-selector"
serif:id="Right selector">
<g
transform="matrix(1.03522,0,0,1.23823,2.07044,0)"
id="g250">
<path
d="M 185.806,161.156 V 67.132"
style="fill:none;stroke:#4c9fff;stroke-width:4px;stroke-linecap:butt"
id="path248" />
</g>
<g
transform="matrix(1.03522,0,0,1.23823,161.788,169.469)"
id="g254">
<circle
cx="31.523001"
cy="34.313999"
r="10.021"
style="fill:#4c9fff;stroke:#4c9fff;stroke-width:4px;stroke-linecap:butt"
id="circle252" />
</g>
</g>
<g
id="Left-selector"
serif:id="Left selector">
<g
transform="matrix(1.03522,0,0,1.23823,-170.092,0)"
id="g259">
<path
d="M 185.806,161.156 V 67.132"
style="fill:none;stroke:#4c9fff;stroke-width:4px;stroke-linecap:butt"
id="path257" />
</g>
<g
transform="matrix(1.03522,0,0,1.23823,-10.3742,28.2274)"
id="g263">
<circle
cx="31.523001"
cy="34.313999"
r="10.021"
style="fill:#4c9fff;stroke:#4c9fff;stroke-width:4px;stroke-linecap:butt"
id="circle261" />
</g>
</g>
</g>
</g>
</g>
</svg>

After

Width:  |  Height:  |  Size: 11 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

+233
View File
@@ -0,0 +1,233 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg
xmlns:serif="http://www.serif.com/"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
width="503"
height="503"
viewBox="0 0 503 503"
version="1.1"
xml:space="preserve"
style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"
id="svg270"
sodipodi:docname="logo-square.svg"
inkscape:export-filename="/home/jb/src/ocrmypdf/docs/images/logo-square.png"
inkscape:export-xdpi="96"
inkscape:export-ydpi="96"
inkscape:version="1.0.2 (e86c870879, 2021-01-15)"><metadata
id="metadata276"><rdf:RDF><cc:Work
rdf:about=""><dc:format>image/svg+xml</dc:format><dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" /><dc:title></dc:title></cc:Work></rdf:RDF></metadata><defs
id="defs274" /><sodipodi:namedview
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1"
objecttolerance="10"
gridtolerance="10"
guidetolerance="10"
inkscape:pageopacity="0"
inkscape:pageshadow="2"
inkscape:window-width="2396"
inkscape:window-height="1691"
id="namedview272"
showgrid="false"
lock-margins="false"
inkscape:zoom="2.0079523"
inkscape:cx="251.5"
inkscape:cy="193.18317"
inkscape:window-x="26"
inkscape:window-y="23"
inkscape:window-maximized="0"
inkscape:current-layer="svg270" />
<g
id="svg"
transform="matrix(0.965977,0,0,0.807602,0,138.43572)">
<rect
x="0"
y="0"
width="520"
height="280"
style="fill:#ffffff"
id="rect188" />
<g
transform="matrix(1.03522,0,0,1.23823,-69.7528,-83.422)"
id="g267">
<g
transform="translate(243.977,20.0703)"
id="g218">
<g
id="Page">
<g
transform="matrix(0.961773,0,0,1.05962,6.19811,-3.01071)"
id="g192">
<path
d="m 328.5,97.682 c 0,-1.217 -0.517,-2.386 -1.444,-3.264 -7.03,-6.66 -37.614,-35.638 -44.828,-42.474 -0.977,-0.925 -2.327,-1.448 -3.738,-1.448 -13.997,0 -90.407,0 -111.151,0 -2.871,0 -5.198,2.113 -5.198,4.718 0,27.837 0,170.351 0,198.186 0,2.605 2.327,4.717 5.197,4.717 24.904,0 131.821,0 156.2,0 2.74,0 4.962,-2.016 4.962,-4.504 0,-24.345 0,-139.717 0,-155.931 z"
style="fill:#fdfdfd;stroke:#333333;stroke-width:3.95px"
id="path190" />
</g>
<g
id="Dog-ear"
serif:id="Dog ear"
transform="translate(-4,2)">
<path
d="m 277.072,48.496 v 45.352 c 0,1.324 0.526,2.593 1.462,3.529 0.936,0.936 2.205,1.462 3.529,1.462 12.485,0 44.078,0 44.078,0"
style="fill:#f5f5f5;stroke:#333333;stroke-width:4px"
id="path194" />
</g>
</g>
<g
transform="translate(-29.6816,-0.395178)"
id="g216">
<g
transform="matrix(1.00243,0,0,1.11818,-144.72,-8.80181)"
id="g200">
<path
d="m 465.73,119.654 c 0,-2.049 -1.856,-3.713 -4.142,-3.713 H 310.259 c -2.286,0 -4.142,1.664 -4.142,3.713 v 63.454 c 0,2.049 1.856,3.713 4.142,3.713 h 151.329 c 2.286,0 4.142,-1.664 4.142,-3.713 z"
style="fill:#f80000;stroke:#ffffff;stroke-width:3.77px"
id="path198" />
</g>
<g
transform="matrix(1.24571,0,0,1.35864,116.812,84.3924)"
id="g214">
<g
transform="matrix(64,0,0,64,42.1437,77.6203)"
id="g204">
<path
d="m 0.084,0 v -0.68 h 0.213 c 0.074,0 0.137,0.017 0.19,0.05 0.053,0.034 0.079,0.09 0.079,0.168 0,0.077 -0.028,0.134 -0.085,0.17 -0.057,0.037 -0.121,0.055 -0.193,0.055 H 0.213 V 0 Z m 0.209,-0.572 h -0.08 v 0.228 h 0.082 c 0.039,0 0.07,-0.009 0.094,-0.027 0.024,-0.017 0.037,-0.045 0.04,-0.083 0,-0.044 -0.012,-0.075 -0.036,-0.092 -0.024,-0.017 -0.057,-0.026 -0.1,-0.026 z"
style="fill:#ffffff;fill-rule:nonzero"
id="path202" />
</g>
<g
transform="matrix(64,0,0,64,79.7117,77.6203)"
id="g208">
<path
d="M 0.332,0 H 0.084 v -0.68 h 0.252 c 0.105,0 0.182,0.032 0.233,0.095 0.051,0.063 0.076,0.144 0.076,0.241 0,0.105 -0.027,0.189 -0.082,0.251 C 0.508,-0.031 0.431,0 0.332,0 Z M 0.337,-0.57 H 0.213 v 0.461 H 0.33 c 0.055,0 0.099,-0.018 0.132,-0.054 C 0.495,-0.199 0.511,-0.259 0.511,-0.344 0.511,-0.415 0.497,-0.47 0.469,-0.51 0.441,-0.55 0.397,-0.57 0.337,-0.57 Z"
style="fill:#ffffff;fill-rule:nonzero"
id="path206" />
</g>
<g
transform="matrix(64,0,0,64,123.424,77.6203)"
id="g212">
<path
d="M 0.405,-0.288 H 0.213 V 0 H 0.084 v -0.68 h 0.385 l 0.02,0.102 H 0.213 v 0.189 h 0.173 z"
style="fill:#ffffff;fill-rule:nonzero"
id="path210" />
</g>
</g>
</g>
</g>
<g
transform="matrix(1,0,0,1.52217,67.3796,10.7507)"
id="g222">
<rect
x="23.500999"
y="81.300003"
width="162.30499"
height="61.77"
style="fill:#b4d5ff"
id="rect220" />
</g>
<g
transform="matrix(0.967536,0,0,0.961535,5.90498,47.9703)"
id="g236">
<g
transform="matrix(90.4804,0,0,90.4804,82.6698,167.705)"
id="g226">
<path
d="m 0.057,-0.337 c 0,-0.105 0.027,-0.19 0.082,-0.257 0.055,-0.066 0.132,-0.1 0.231,-0.102 0.107,0 0.186,0.034 0.237,0.103 0.051,0.069 0.077,0.152 0.077,0.249 0,0.105 -0.027,0.191 -0.082,0.258 -0.055,0.067 -0.133,0.1 -0.232,0.1 C 0.264,0.014 0.185,-0.02 0.134,-0.089 0.083,-0.157 0.057,-0.24 0.057,-0.337 Z m 0.135,-0.001 c 0,0.071 0.014,0.13 0.043,0.175 0.029,0.045 0.073,0.068 0.134,0.068 0.055,0 0.098,-0.02 0.131,-0.061 0.033,-0.041 0.049,-0.103 0.049,-0.188 0,-0.071 -0.014,-0.129 -0.043,-0.174 -0.029,-0.045 -0.073,-0.068 -0.134,-0.068 -0.053,0 -0.097,0.022 -0.13,0.067 -0.033,0.045 -0.05,0.105 -0.05,0.181 z"
style="fill:#333333;fill-rule:nonzero"
id="path224" />
</g>
<g
transform="matrix(90.4804,0,0,90.4804,147.906,167.705)"
id="g230">
<path
d="M 0.505,-0.557 C 0.473,-0.567 0.448,-0.574 0.429,-0.579 0.41,-0.583 0.388,-0.585 0.361,-0.585 c -0.054,0 -0.096,0.022 -0.125,0.066 -0.029,0.044 -0.044,0.104 -0.044,0.181 0,0.066 0.012,0.123 0.037,0.171 0.025,0.048 0.066,0.072 0.124,0.072 0.029,0 0.056,-0.003 0.081,-0.009 0.025,-0.006 0.047,-0.013 0.068,-0.022 L 0.551,-0.03 C 0.525,-0.017 0.494,-0.006 0.457,0.002 0.42,0.01 0.388,0.014 0.36,0.014 0.254,0.014 0.177,-0.02 0.129,-0.088 0.081,-0.156 0.057,-0.239 0.057,-0.337 c 0,-0.105 0.027,-0.19 0.08,-0.257 0.053,-0.067 0.129,-0.1 0.228,-0.1 0.02,0 0.048,0.003 0.083,0.01 0.035,0.007 0.068,0.018 0.097,0.034 z"
style="fill:#333333;fill-rule:nonzero"
id="path228" />
</g>
<g
transform="matrix(90.4804,0,0,90.4804,199.751,167.705)"
id="g234">
<path
d="m 0.293,-0.572 h -0.08 v 0.208 h 0.082 c 0.039,0 0.071,-0.008 0.096,-0.024 0.025,-0.015 0.038,-0.041 0.038,-0.077 0,-0.038 -0.012,-0.065 -0.036,-0.082 -0.024,-0.017 -0.057,-0.025 -0.1,-0.025 z M 0.479,0 0.335,-0.26 C 0.328,-0.259 0.32,-0.259 0.312,-0.259 0.304,-0.258 0.296,-0.258 0.288,-0.258 H 0.213 V 0 H 0.084 v -0.68 h 0.213 c 0.074,0 0.137,0.017 0.19,0.051 0.053,0.034 0.079,0.087 0.079,0.158 0,0.042 -0.011,0.078 -0.032,0.108 -0.022,0.031 -0.05,0.054 -0.084,0.071 L 0.617,0 Z"
style="fill:#333333;fill-rule:nonzero"
id="path232" />
</g>
</g>
<g
transform="matrix(0.916882,0,0,1,121.475,-32.6535)"
id="g246">
<g
transform="matrix(86.953,0,0,86.953,152.996,241.878)"
id="g240">
<path
d="M 0.479,-0.428 C 0.5,-0.451 0.527,-0.47 0.562,-0.484 c 0.034,-0.013 0.065,-0.02 0.092,-0.02 0.066,0 0.113,0.019 0.141,0.058 0.027,0.039 0.041,0.086 0.041,0.142 V 0 H 0.705 v -0.298 c 0,-0.031 -0.007,-0.054 -0.022,-0.071 -0.015,-0.016 -0.036,-0.024 -0.064,-0.024 -0.019,0 -0.038,0.005 -0.059,0.015 -0.021,0.01 -0.039,0.021 -0.056,0.034 0.001,0.007 0.001,0.013 0.002,0.02 0.001,0.007 0.001,0.013 0.001,0.02 V 0 H 0.376 v -0.298 c 0,-0.031 -0.007,-0.054 -0.022,-0.071 -0.015,-0.016 -0.036,-0.024 -0.063,-0.024 -0.017,0 -0.033,0.003 -0.05,0.01 -0.017,0.007 -0.034,0.016 -0.049,0.027 V 0 H 0.062 V -0.485 H 0.13 l 0.032,0.044 c 0.022,-0.02 0.049,-0.035 0.08,-0.047 0.031,-0.011 0.058,-0.016 0.083,-0.016 0.038,0 0.07,0.007 0.095,0.02 0.025,0.014 0.045,0.033 0.059,0.056 z"
style="fill:#333333;fill-rule:nonzero"
id="path238" />
</g>
<g
transform="matrix(86.953,0,0,86.953,228.906,241.878)"
id="g244">
<path
d="M 0.156,0.023 0.179,-0.034 0.006,-0.467 0.14,-0.485 0.252,-0.191 0.358,-0.485 H 0.495 L 0.278,0.064 C 0.263,0.103 0.236,0.137 0.197,0.165 0.158,0.193 0.118,0.212 0.075,0.222 L 0.029,0.115 C 0.052,0.105 0.077,0.093 0.104,0.079 0.13,0.064 0.147,0.046 0.156,0.023 Z"
style="fill:#333333;fill-rule:nonzero"
id="path242" />
</g>
</g>
<g
id="Selectors"
transform="matrix(0.965977,0,0,0.807602,67.3796,67.3718)">
<g
id="Right-selector"
serif:id="Right selector">
<g
transform="matrix(1.03522,0,0,1.23823,2.07044,0)"
id="g250">
<path
d="M 185.806,161.156 V 67.132"
style="fill:none;stroke:#4c9fff;stroke-width:4px;stroke-linecap:butt"
id="path248" />
</g>
<g
transform="matrix(1.03522,0,0,1.23823,161.788,169.469)"
id="g254">
<circle
cx="31.523001"
cy="34.313999"
r="10.021"
style="fill:#4c9fff;stroke:#4c9fff;stroke-width:4px;stroke-linecap:butt"
id="circle252" />
</g>
</g>
<g
id="Left-selector"
serif:id="Left selector">
<g
transform="matrix(1.03522,0,0,1.23823,-170.092,0)"
id="g259">
<path
d="M 185.806,161.156 V 67.132"
style="fill:none;stroke:#4c9fff;stroke-width:4px;stroke-linecap:butt"
id="path257" />
</g>
<g
transform="matrix(1.03522,0,0,1.23823,-10.3742,28.2274)"
id="g263">
<circle
cx="31.523001"
cy="34.313999"
r="10.021"
style="fill:#4c9fff;stroke:#4c9fff;stroke-width:4px;stroke-linecap:butt"
id="circle261" />
</g>
</g>
</g>
</g>
</g>
</svg>

After

Width:  |  Height:  |  Size: 11 KiB

+75
View File
@@ -0,0 +1,75 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg width="100%" height="100%" viewBox="0 0 503 227" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xml:space="preserve" xmlns:serif="http://www.serif.com/" style="fill-rule:evenodd;clip-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5;">
<g id="svg" transform="matrix(0.965977,0,0,0.807602,0,0)">
<rect x="0" y="0" width="520" height="280" style="fill:white;"/>
<g transform="matrix(1.03522,0,0,1.23823,-69.7528,-83.422)">
<g transform="matrix(1,0,0,1,243.977,20.0703)">
<g id="Page">
<g transform="matrix(0.961773,0,0,1.05962,6.19811,-3.01071)">
<path d="M328.5,97.682C328.5,96.465 327.983,95.296 327.056,94.418C320.026,87.758 289.442,58.78 282.228,51.944C281.251,51.019 279.901,50.496 278.49,50.496C264.493,50.496 188.083,50.496 167.339,50.496C164.468,50.496 162.141,52.609 162.141,55.214C162.141,83.051 162.141,225.565 162.141,253.4C162.141,256.005 164.468,258.117 167.338,258.117C192.242,258.117 299.159,258.117 323.538,258.117C326.278,258.117 328.5,256.101 328.5,253.613C328.5,229.268 328.5,113.896 328.5,97.682Z" style="fill:rgb(253,253,253);stroke:rgb(51,51,51);stroke-width:3.95px;"/>
</g>
<g id="Dog-ear" serif:id="Dog ear" transform="matrix(1,0,0,1,-4,2)">
<path d="M277.072,48.496L277.072,93.848C277.072,95.172 277.598,96.441 278.534,97.377C279.47,98.313 280.739,98.839 282.063,98.839C294.548,98.839 326.141,98.839 326.141,98.839" style="fill:rgb(245,245,245);stroke:rgb(51,51,51);stroke-width:4px;"/>
</g>
</g>
<g transform="matrix(1,0,0,1,-29.6816,-0.395178)">
<g transform="matrix(1.00243,0,0,1.11818,-144.72,-8.80181)">
<path d="M465.73,119.654C465.73,117.605 463.874,115.941 461.588,115.941L310.259,115.941C307.973,115.941 306.117,117.605 306.117,119.654L306.117,183.108C306.117,185.157 307.973,186.821 310.259,186.821L461.588,186.821C463.874,186.821 465.73,185.157 465.73,183.108L465.73,119.654Z" style="fill:rgb(248,0,0);stroke:white;stroke-width:3.77px;"/>
</g>
<g transform="matrix(1.24571,0,0,1.35864,116.812,84.3924)">
<g transform="matrix(64,0,0,64,42.1437,77.6203)">
<path d="M0.084,0L0.084,-0.68L0.297,-0.68C0.371,-0.68 0.434,-0.663 0.487,-0.63C0.54,-0.596 0.566,-0.54 0.566,-0.462C0.566,-0.385 0.538,-0.328 0.481,-0.292C0.424,-0.255 0.36,-0.237 0.288,-0.237L0.213,-0.237L0.213,0L0.084,0ZM0.293,-0.572L0.213,-0.572L0.213,-0.344L0.295,-0.344C0.334,-0.344 0.365,-0.353 0.389,-0.371C0.413,-0.388 0.426,-0.416 0.429,-0.454C0.429,-0.498 0.417,-0.529 0.393,-0.546C0.369,-0.563 0.336,-0.572 0.293,-0.572Z" style="fill:white;fill-rule:nonzero;"/>
</g>
<g transform="matrix(64,0,0,64,79.7117,77.6203)">
<path d="M0.332,0L0.084,0L0.084,-0.68L0.336,-0.68C0.441,-0.68 0.518,-0.648 0.569,-0.585C0.62,-0.522 0.645,-0.441 0.645,-0.344C0.645,-0.239 0.618,-0.155 0.563,-0.093C0.508,-0.031 0.431,0 0.332,0ZM0.337,-0.57L0.213,-0.57L0.213,-0.109L0.33,-0.109C0.385,-0.109 0.429,-0.127 0.462,-0.163C0.495,-0.199 0.511,-0.259 0.511,-0.344C0.511,-0.415 0.497,-0.47 0.469,-0.51C0.441,-0.55 0.397,-0.57 0.337,-0.57Z" style="fill:white;fill-rule:nonzero;"/>
</g>
<g transform="matrix(64,0,0,64,123.424,77.6203)">
<path d="M0.405,-0.288L0.213,-0.288L0.213,0L0.084,0L0.084,-0.68L0.469,-0.68L0.489,-0.578L0.213,-0.578L0.213,-0.389L0.386,-0.389L0.405,-0.288Z" style="fill:white;fill-rule:nonzero;"/>
</g>
</g>
</g>
</g>
<g transform="matrix(1,0,0,1.52217,67.3796,10.7507)">
<rect x="23.501" y="81.3" width="162.305" height="61.77" style="fill:rgb(180,213,255);"/>
</g>
<g transform="matrix(0.967536,0,0,0.961535,5.90498,47.9703)">
<g transform="matrix(90.4804,0,0,90.4804,82.6698,167.705)">
<path d="M0.057,-0.337C0.057,-0.442 0.084,-0.527 0.139,-0.594C0.194,-0.66 0.271,-0.694 0.37,-0.696C0.477,-0.696 0.556,-0.662 0.607,-0.593C0.658,-0.524 0.684,-0.441 0.684,-0.344C0.684,-0.239 0.657,-0.153 0.602,-0.086C0.547,-0.019 0.469,0.014 0.37,0.014C0.264,0.014 0.185,-0.02 0.134,-0.089C0.083,-0.157 0.057,-0.24 0.057,-0.337ZM0.192,-0.338C0.192,-0.267 0.206,-0.208 0.235,-0.163C0.264,-0.118 0.308,-0.095 0.369,-0.095C0.424,-0.095 0.467,-0.115 0.5,-0.156C0.533,-0.197 0.549,-0.259 0.549,-0.344C0.549,-0.415 0.535,-0.473 0.506,-0.518C0.477,-0.563 0.433,-0.586 0.372,-0.586C0.319,-0.586 0.275,-0.564 0.242,-0.519C0.209,-0.474 0.192,-0.414 0.192,-0.338Z" style="fill:rgb(51,51,51);fill-rule:nonzero;"/>
</g>
<g transform="matrix(90.4804,0,0,90.4804,147.906,167.705)">
<path d="M0.505,-0.557C0.473,-0.567 0.448,-0.574 0.429,-0.579C0.41,-0.583 0.388,-0.585 0.361,-0.585C0.307,-0.585 0.265,-0.563 0.236,-0.519C0.207,-0.475 0.192,-0.415 0.192,-0.338C0.192,-0.272 0.204,-0.215 0.229,-0.167C0.254,-0.119 0.295,-0.095 0.353,-0.095C0.382,-0.095 0.409,-0.098 0.434,-0.104C0.459,-0.11 0.481,-0.117 0.502,-0.126L0.551,-0.03C0.525,-0.017 0.494,-0.006 0.457,0.002C0.42,0.01 0.388,0.014 0.36,0.014C0.254,0.014 0.177,-0.02 0.129,-0.088C0.081,-0.156 0.057,-0.239 0.057,-0.337C0.057,-0.442 0.084,-0.527 0.137,-0.594C0.19,-0.661 0.266,-0.694 0.365,-0.694C0.385,-0.694 0.413,-0.691 0.448,-0.684C0.483,-0.677 0.516,-0.666 0.545,-0.65L0.505,-0.557Z" style="fill:rgb(51,51,51);fill-rule:nonzero;"/>
</g>
<g transform="matrix(90.4804,0,0,90.4804,199.751,167.705)">
<path d="M0.293,-0.572L0.213,-0.572L0.213,-0.364L0.295,-0.364C0.334,-0.364 0.366,-0.372 0.391,-0.388C0.416,-0.403 0.429,-0.429 0.429,-0.465C0.429,-0.503 0.417,-0.53 0.393,-0.547C0.369,-0.564 0.336,-0.572 0.293,-0.572ZM0.479,0L0.335,-0.26C0.328,-0.259 0.32,-0.259 0.312,-0.259C0.304,-0.258 0.296,-0.258 0.288,-0.258L0.213,-0.258L0.213,0L0.084,0L0.084,-0.68L0.297,-0.68C0.371,-0.68 0.434,-0.663 0.487,-0.629C0.54,-0.595 0.566,-0.542 0.566,-0.471C0.566,-0.429 0.555,-0.393 0.534,-0.363C0.512,-0.332 0.484,-0.309 0.45,-0.292L0.617,0L0.479,0Z" style="fill:rgb(51,51,51);fill-rule:nonzero;"/>
</g>
</g>
<g transform="matrix(0.916882,0,0,1,121.475,-32.6535)">
<g transform="matrix(86.953,0,0,86.953,152.996,241.878)">
<path d="M0.479,-0.428C0.5,-0.451 0.527,-0.47 0.562,-0.484C0.596,-0.497 0.627,-0.504 0.654,-0.504C0.72,-0.504 0.767,-0.485 0.795,-0.446C0.822,-0.407 0.836,-0.36 0.836,-0.304L0.836,0L0.705,0L0.705,-0.298C0.705,-0.329 0.698,-0.352 0.683,-0.369C0.668,-0.385 0.647,-0.393 0.619,-0.393C0.6,-0.393 0.581,-0.388 0.56,-0.378C0.539,-0.368 0.521,-0.357 0.504,-0.344C0.505,-0.337 0.505,-0.331 0.506,-0.324C0.507,-0.317 0.507,-0.311 0.507,-0.304L0.507,0L0.376,0L0.376,-0.298C0.376,-0.329 0.369,-0.352 0.354,-0.369C0.339,-0.385 0.318,-0.393 0.291,-0.393C0.274,-0.393 0.258,-0.39 0.241,-0.383C0.224,-0.376 0.207,-0.367 0.192,-0.356L0.192,0L0.062,0L0.062,-0.485L0.13,-0.485L0.162,-0.441C0.184,-0.461 0.211,-0.476 0.242,-0.488C0.273,-0.499 0.3,-0.504 0.325,-0.504C0.363,-0.504 0.395,-0.497 0.42,-0.484C0.445,-0.47 0.465,-0.451 0.479,-0.428Z" style="fill:rgb(51,51,51);fill-rule:nonzero;"/>
</g>
<g transform="matrix(86.953,0,0,86.953,228.906,241.878)">
<path d="M0.156,0.023L0.179,-0.034L0.006,-0.467L0.14,-0.485L0.252,-0.191L0.358,-0.485L0.495,-0.485L0.278,0.064C0.263,0.103 0.236,0.137 0.197,0.165C0.158,0.193 0.118,0.212 0.075,0.222L0.029,0.115C0.052,0.105 0.077,0.093 0.104,0.079C0.13,0.064 0.147,0.046 0.156,0.023Z" style="fill:rgb(51,51,51);fill-rule:nonzero;"/>
</g>
</g>
<g id="Selectors" transform="matrix(0.965977,0,0,0.807602,67.3796,67.3718)">
<g id="Right-selector" serif:id="Right selector">
<g transform="matrix(1.03522,0,0,1.23823,2.07044,0)">
<path d="M185.806,161.156L185.806,67.132" style="fill:none;stroke:rgb(76,159,255);stroke-width:4px;stroke-linecap:butt;"/>
</g>
<g transform="matrix(1.03522,0,0,1.23823,161.788,169.469)">
<circle cx="31.523" cy="34.314" r="10.021" style="fill:rgb(76,159,255);stroke:rgb(76,159,255);stroke-width:4px;stroke-linecap:butt;"/>
</g>
</g>
<g id="Left-selector" serif:id="Left selector">
<g transform="matrix(1.03522,0,0,1.23823,-170.092,0)">
<path d="M185.806,161.156L185.806,67.132" style="fill:none;stroke:rgb(76,159,255);stroke-width:4px;stroke-linecap:butt;"/>
</g>
<g transform="matrix(1.03522,0,0,1.23823,-10.3742,28.2274)">
<circle cx="31.523" cy="34.314" r="10.021" style="fill:rgb(76,159,255);stroke:rgb(76,159,255);stroke-width:4px;stroke-linecap:butt;"/>
</g>
</g>
</g>
</g>
</g>
</svg>

After

Width:  |  Height:  |  Size: 9.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 21 KiB

+56
View File
@@ -0,0 +1,56 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
OCRmyPDF documentation
======================
.. figure:: images/logo.svg
OCRmyPDF adds an optical character recognition (OCR) text layer to scanned PDF
files, allowing them to be searched.
PDF is the best format for storing and exchanging scanned documents.
Unfortunately, PDFs can be difficult to modify. OCRmyPDF makes it easy to apply
image processing and OCR (recognized, searchable text) to existing PDFs.
.. toctree::
:maxdepth: 1
introduction
release_notes
installation
languages
jbig2
.. toctree::
:caption: Usage
:maxdepth: 2
cookbook
optimizer
docker
advanced
batch
cloud
performance
pdfsecurity
errors
.. toctree::
:caption: Developers
:maxdepth: 2
api
plugins
apiref
design_notes
contributing
maintainers
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
+719
View File
@@ -0,0 +1,719 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
===================
Installing OCRmyPDF
===================
.. |latest| image:: https://img.shields.io/pypi/v/ocrmypdf.svg
:alt: OCRmyPDF latest released version on PyPI
|latest|
The easiest way to install OCRmyPDF is to follow the steps for your operating
system/platform. This version may be out of date, however.
These platforms have one-liner installs:
+-------------------------------+-----------------------------------------+
| Debian, Ubuntu | ``apt install ocrmypdf`` |
+-------------------------------+-----------------------------------------+
| Windows Subsystem for Linux | ``apt install ocrmypdf`` |
+-------------------------------+-----------------------------------------+
| Fedora | ``dnf install ocrmypdf tesseract-osd`` |
+-------------------------------+-----------------------------------------+
| macOS (Homebrew) | ``brew install ocrmypdf`` |
+-------------------------------+-----------------------------------------+
| macOS (MacPorts) | ``port install ocrmypdf`` |
+-------------------------------+-----------------------------------------+
| LinuxBrew | ``brew install ocrmypdf`` |
+-------------------------------+-----------------------------------------+
| FreeBSD | ``pkg install textproc/py-ocrmypdf`` |
+-------------------------------+-----------------------------------------+
| Conda (WSL, macOS, Linux) | ``conda install ocrmypdf`` |
+-------------------------------+-----------------------------------------+
| Snap (snapcraft packaging) | ``snap install ocrmypdf`` |
+-------------------------------+-----------------------------------------+
More detailed procedures are outlined below. If you want to do a manual
install, or install a more recent version than your platform provides, read on.
.. contents:: Platform-specific steps
:depth: 2
:local:
Installing on Linux
===================
Debian and Ubuntu 20.04 or newer
--------------------------------
.. |deb-11| image:: https://repology.org/badge/version-for-repo/debian_11/ocrmypdf.svg
:alt: Debian 11
.. |deb-12| image:: https://repology.org/badge/version-for-repo/debian_12/ocrmypdf.svg
:alt: Debian 12
.. |deb-unstable| image:: https://repology.org/badge/version-for-repo/debian_unstable/ocrmypdf.svg
:alt: Debian unstable
.. |ubu-2004| image:: https://repology.org/badge/version-for-repo/ubuntu_20_04/ocrmypdf.svg
:alt: Ubuntu 20.04 LTS
.. |ubu-2204| image:: https://repology.org/badge/version-for-repo/ubuntu_22_04/ocrmypdf.svg
:alt: Ubuntu 22.04 LTS
+-----------------------------------------------+
| **OCRmyPDF versions in Debian & Ubuntu** |
+-----------------------------------------------+
| |latest| |
+-----------------------------------------------+
| |deb-11| |deb-12| |deb-unstable| |
+-----------------------------------------------+
| |ubu-2004| |ubu-2204| |
+-----------------------------------------------+
Users of Debian or Ubuntu may simply
.. code-block:: bash
apt install ocrmypdf
As indicated in the table above, Debian and Ubuntu releases may lag
behind the latest version. If the version available for your platform is
out of date, you could opt to install the latest version from source.
See `Installing HEAD revision from
sources <#installing-head-revision-from-sources>`__.
For full details on version availability for your platform, check the
`Debian Package Tracker <https://tracker.debian.org/pkg/ocrmypdf>`__ or
`Ubuntu launchpad.net <https://launchpad.net/ocrmypdf>`__.
.. note::
OCRmyPDF for Debian and Ubuntu currently omit the JBIG2 encoder.
OCRmyPDF works fine without it but will produce larger output files.
If you build jbig2enc from source, ocrmypdf will
automatically detect it (specifically the ``jbig2`` binary) on the
``PATH``. To add JBIG2 encoding, see :ref:`jbig2`.
Fedora
------
.. |fedora-38| image:: https://repology.org/badge/version-for-repo/fedora_38/ocrmypdf.svg
:alt: Fedora 38
.. |fedora-39| image:: https://repology.org/badge/version-for-repo/fedora_39/ocrmypdf.svg
:alt: Fedora 39
.. |fedora-rawhide| image:: https://repology.org/badge/version-for-repo/fedora_rawhide/ocrmypdf.svg
:alt: Fedore Rawhide
+-----------------------------------------------+
| **OCRmyPDF version** |
+-----------------------------------------------+
| |latest| |
+-----------------------------------------------+
| |fedora-38| |fedora-39| |fedora-rawhide| |
+-----------------------------------------------+
Users of Fedora may simply
.. code-block:: bash
dnf install ocrmypdf tesseract-osd
For full details on version availability, check the `Fedora Package
Tracker <https://packages.fedoraproject.org/pkgs/ocrmypdf/ocrmypdf/>`__.
If the version available for your platform is out of date, you could opt
to install the latest version from source. See `Installing HEAD revision
from sources <#installing-head-revision-from-sources>`__.
.. note::
OCRmyPDF for Fedora currently omits the JBIG2 encoder due to patent
issues. OCRmyPDF works fine without it but will produce larger output
files. If you build jbig2enc from source, ocrmypdf 7.0.0 and later
will automatically detect it on the ``PATH``. To add JBIG2 encoding,
see :ref:`Installing the JBIG2 encoder <jbig2>`.
.. _ubuntu-lts-latest:
RHEL 9
------
Prepare the environment by getting Python 3.11:
.. code-block:: bash
dnf install python3.11 python3.11-pip
Then, follow `Requirements for pip and HEAD install <#requirements-for-pip-and-head-install>`__ to instal dependencies:
.. code-block:: bash
dnf install ghostscript tesseract
and build ocrmypdf in virtual environment:
.. code-block:: bash
python3.11 -m venv .venv
To add JBIG2 encoding, see :ref:`Installing the JBIG2 encoder <jbig2>`.
Note Fedora packages for language data haven't been branched for RHEL/EPEL, but you can get traineddata files directly from `tesseract
<https://github.com/tesseract-ocr/tessdata/>`__ and place them in ``/usr/share/tesseract/tessdata``.
Installing the latest version on Ubuntu 22.04 LTS
-------------------------------------------------
Ubuntu 22.04 includes ocrmypdf 13.4.0 - you can install that with
``apt install ocrmypdf``. To install a more recent version for the current
user, follow these steps:
.. code-block:: bash
sudo apt-get update
sudo apt-get -y install ocrmypdf python3-pip
pip install --user --upgrade ocrmypdf
If you get the message ``WARNING: The script ocrmypdf is installed in
'/home/$USER/.local/bin' which is not on PATH.``, you may need to re-login
or open a new shell, or manually adjust your PATH.
To add JBIG2 encoding, see :ref:`jbig2`.
Ubuntu 20.04 LTS
----------------
Ubuntu 20.04 includes ocrmypdf 9.6.0 - you can install that with ``apt``. The
most convenient way to install recent OCRmyPDF on older Ubuntu is to use
Homebrew on Linux (Linuxbrew).
.. code-block:: bash
brew install ocrmypdf
Arch Linux (AUR)
----------------
.. image:: https://repology.org/badge/version-for-repo/aur/ocrmypdf.svg
:alt: ArchLinux
:target: https://repology.org/metapackage/ocrmypdf
There is an `Arch User Repository (AUR) package for OCRmyPDF
<https://aur.archlinux.org/packages/ocrmypdf/>`__.
Installing AUR packages as root is not allowed, so you must first `setup a
non-root user
<https://wiki.archlinux.org/index.php/Users_and_groups#User_management>`__ and
`configure sudo <https://wiki.archlinux.org/index.php/Sudo#Configuration>`__.
The standard Docker image, ``archlinux/base:latest``, does **not** have a
non-root user configured, so users of that image must follow these guides. If
you are using a VM image, such as `the official Vagrant image
<https://app.vagrantup.com/archlinux/boxes/archlinux>`__, this work may already
be completed for you.
Next you should install the `base-devel package group
<https://archlinux.org/packages/core/any/base-devel/>`__. This includes the
standard tooling needed to build packages, such as a compiler and binary tools.
.. code-block:: bash
sudo pacman -S --needed base-devel
Now you are ready to install the OCRmyPDF package.
.. code-block:: bash
curl -O https://aur.archlinux.org/cgit/aur.git/snapshot/ocrmypdf.tar.gz
tar xvzf ocrmypdf.tar.gz
cd ocrmypdf
makepkg -sri
At this point you will have a working install of OCRmyPDF, but the Tesseract
install wont include any OCR language data. You can install `the
tesseract-data package group
<https://www.archlinux.org/groups/any/tesseract-data/>`__ to add all supported
languages, or use that package listing to identify the appropriate package for
your desired language.
.. code-block:: bash
sudo pacman -S tesseract-data-eng
As an alternative to this manual procedure, consider using an `AUR helper
<https://wiki.archlinux.org/index.php/AUR_helpers>`__. Such a tool will
automatically fetch, build and install the AUR package, resolve dependencies
(including dependencies on AUR packages), and ease the upgrade procedure.
If you have any difficulties with installation, check the repository package
page.
.. note::
The OCRmyPDF AUR package currently omits the JBIG2 encoder. OCRmyPDF works
fine without it but will produce larger output files. The encoder is
available from `the jbig2enc-git AUR package
<https://aur.archlinux.org/packages/jbig2enc-git/>`__ and may be installed
using the same series of steps as for the installation OCRmyPDF AUR
package. Alternatively, it may be built manually from source following the
instructions in :ref:`Installing the JBIG2 encoder <jbig2>`. If JBIG2 is
installed, OCRmyPDF 7.0.0 and later will automatically detect it.
Alpine Linux
------------
.. image:: https://repology.org/badge/version-for-repo/alpine_edge/ocrmypdf.svg
:alt: Alpine Linux
:target: https://repology.org/metapackage/ocrmypdf
To install OCRmyPDF for Alpine Linux:
.. code-block:: bash
apk add ocrmypdf
Gentoo Linux
------------
.. image:: https://repology.org/badge/version-for-repo/gentoo_ovl_guru/ocrmypdf.svg
:alt: Gentoo Linux
:target: https://repology.org/metapackage/ocrmypdf
To install OCRmyPDF on Gentoo Linux, use the following commands:
.. code-block:: bash
eselect repository enable guru
emaint sync --repo guru
emerge --ask app-text/OCRmyPDF
Other Linux packages
--------------------
See the
`Repology <https://repology.org/metapackage/ocrmypdf/versions>`__ page.
In general, first install the OCRmyPDF package for your system, then
optionally use the procedure `Installing with Python
pip <#installing-with-python-pip>`__ to install a more recent version.
Installing on macOS
===================
Homebrew
--------
.. image:: https://img.shields.io/homebrew/v/ocrmypdf.svg
:alt: homebrew
:target: https://formulae.brew.sh/formula/ocrmypdf
OCRmyPDF is now a standard `Homebrew <https://brew.sh>`__ formula. To
install on macOS:
.. code-block:: bash
brew install ocrmypdf
This will include only the English language pack. If you need other
languages you can optionally install them all:
.. code-block:: bash
brew install tesseract-lang # Optional: Install all language packs
MacPorts
--------
.. image:: https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fports.macports.org%2Fapi%2Fv1%2Fports%2Focrmypdf%2F%3Fformat%3Djson&query=version&label=MacPorts
:alt: Macports Version Information
:target: https://ports.macports.org/port/ocrmypdf
OCRmyPDF is includes in MacPorts:
.. code-block:: bash
sudo port install ocrmypdf
Note that while this will install tesseract you will need to install
the appropriate tesseract `language ports <https://ports.macports.org/search/?selected_facets=categories_exact%3Atextproc&installed_file=&q=tesseract&name=on>`__.
Manual installation on macOS
----------------------------
These instructions probably work on all macOS supported by Homebrew, and are
for installing a more current version of OCRmyPDF than is available from
Homebrew. Note that the Homebrew versions usually track the release versions
fairly closely.
If it's not already present, `install Homebrew <http://brew.sh/>`__.
Update Homebrew:
.. code-block:: bash
brew update
Install or upgrade the required Homebrew packages, if any are missing.
To do this, use ``brew edit ocrmypdf`` to obtain a recent list of Homebrew
dependencies. You could also check the ``.workflows/build.yml``.
This will include the English, French, German and Spanish language
packs. If you need other languages you can optionally install them all:
.. _macos-all-languages:
.. code-block:: bash
brew install tesseract-lang # Option 2: for all language packs
Update the homebrew pip:
.. code-block:: bash
pip install --upgrade pip
You can then install OCRmyPDF from PyPI for the current user:
.. code-block:: bash
pip install --user ocrmypdf
The command line program should now be available:
.. code-block:: bash
ocrmypdf --help
Installing on Windows
=====================
Native Windows
--------------
.. note::
Administrator privileges will be required for some of these steps.
You must install the following for Windows:
* Python 64-bit
* Tesseract 64-bit
* Ghostscript 64-bit
Using the `winget <https://docs.microsoft.com/en-us/windows/package-manager/winget/>`_
package manager:
* ``winget install -e --id Python.Python.3.11``
* ``winget install -e --id UB-Mannheim.TesseractOCR``
You will need to install Ghostscript manually, `since it does not support automated
installs anymore <https://artifex.com/news/ghostscript-10.01.0-disabling-silent-install-option>`_.
* `Ghostscript download page <https://ghostscript.com/releases/gsdnld.html>`_.`
(Or alternately, using the `Chocolatey <https://chocolatey.org/>`_ package manager, install
the following when running in an Administrator command prompt):
* ``choco install python3``
* ``choco install --pre tesseract``
* ``choco install pngquant`` (optional)
Either set of commands will install the required software. At the moment there is no
single command to install Windows.
You may then use ``pip`` to install ocrmypdf. (This can performed by a user or
Administrator.):
* ``python3 -m pip install ocrmypdf``
OCRmyPDF will check the Windows Registry and standard locations in your Program Files
for third party software it needs (specifically, Tesseract and Ghostscript). To
override the versions OCRmyPDF selects, you can modify the ``PATH`` environment
variable. `Follow these directions <https://www.computerhope.com/issues/ch000549.htm#dospath>`_
to change the PATH.
.. warning::
As of early 2021, users have reported problems with the Microsoft Store version of
Python and OCRmyPDF. These issues affect many other third party Python packages.
Please download Python from Python.org or a package manager instead of the
Microsoft Store version.
.. warning::
32-bit Windows is not supported.
Windows Subsystem for Linux
---------------------------
#. Install Ubuntu 22.04 for Windows Subsystem for Linux, if not already installed.
#. Follow the procedure to install :ref:`OCRmyPDF on Ubuntu 22.04 <ubuntu-lts-latest>`.
#. Open the Windows command prompt and create a symlink:
.. code-block:: powershell
wsl sudo ln -s /home/$USER/.local/bin/ocrmypdf /usr/local/bin/ocrmypdf
Then confirm that the expected version from PyPI (|latest|) is installed:
.. code-block:: powershell
wsl ocrmypdf --version
You can then run OCRmyPDF in the Windows command prompt or Powershell, prefixing
``wsl``, and call it from Windows programs or batch files.
Cygwin64
--------
First install the the following prerequisite Cygwin packages using ``setup-x86_64.exe``::
python310 (or later)
python3?-devel
python3?-pip
python3?-lxml
python3?-imaging
(where 3? means match the version of python3 you installed)
gcc-g++
ghostscript
libexempi3
libexempi-devel
libffi6
libffi-devel
pngquant
qpdf
libqpdf-devel
tesseract-ocr
tesseract-ocr-devel
Then open a Cygwin terminal (i.e. ``mintty``), run the following commands. Note
that if you are using the version of ``pip`` that was installed with the Cygwin
Python package, the command name will be ``pip3``. If you have since updated
``pip`` (with, for instance ``pip3 install --upgrade pip``) the the command is
likely just ``pip`` instead of ``pip3``:
.. code-block:: bash
pip3 install wheel
pip3 install ocrmypdf
The optional dependency "unpaper" that is currently not available under Cygwin.
Without it, certain options such as ``--clean`` will produce an error message.
However, the OCR-to-text-layer functionality is available.
Docker
------
You can also :ref:`Install the Docker <docker>` container on Windows. Ensure that
your command prompt can run the docker "hello world" container.
Installing on FreeBSD
=====================
.. image:: https://repology.org/badge/version-for-repo/freebsd/ocrmypdf.svg
:alt: FreeBSD
:target: https://repology.org/project/ocrmypdf/versions
.. code-block:: bash
pkg install textproc/py-ocrmypdf
To install a more recent version, you could attempt to first install the system
version with ``pkg``, then use ``pip install --user ocrmypdf``.
Installing the Docker image
===========================
For some users, installing the Docker image will be easier than
installing all of OCRmyPDF's dependencies.
See :ref:`docker` for more information.
Installing with Python pip
==========================
OCRmyPDF is delivered by PyPI because it is a convenient way to install
the latest version. However, PyPI and ``pip`` cannot address the fact
that ``ocrmypdf`` depends on certain non-Python system libraries and
programs being installed.
For best results, first install `your platform's
version <https://repology.org/metapackage/ocrmypdf/versions>`__ of
``ocrmypdf``, using the instructions elsewhere in this document. Then
you can use ``pip`` to get the latest version if your platform version
is out of date. Chances are that this will satisfy most dependencies.
Use ``ocrmypdf --version`` to confirm what version was installed.
Then you can install the latest OCRmyPDF from the Python wheels. First
try:
.. code-block:: bash
pip install --user ocrmypdf
(If the message appears ``Requirement already satisfied: ocrmypdf in...``,
you will need to use ``pip install --user --upgrade ocrmypdf``.)
You should then be able to run ``ocrmypdf --version`` and see that the
latest version was located.
Installing with pipx
====================
Some users may prefer pipx. As with the method above, you will need to
satisfy all non-Python dependencies. Then if pipx is installed, you
can use
.. code-block:: bash
pipx run ocrmypdf
(If not installed, pipx will install first.)
Requirements for pip and HEAD install
-------------------------------------
OCRmyPDF currently requires these external programs and libraries to be
installed, and must be satisfied using the operating system package
manager. ``pip`` cannot provide them.
The following versions are required:
- Python 3.10 or newer
- Ghostscript 9.54 or newer
- Tesseract 4.1.1 or newer
- jbig2enc 0.29 or newer
- pngquant 2.5 or newer
- unpaper 6.1
We recommend 64-bit versions of all software. (32-bit versions are not
supported, although on Linux, they may still work.)
jbig2enc, pngquant, and unpaper are optional. If missing certain
features are disabled. OCRmyPDF will discover them as soon as they are
available.
**jbig2enc**, if present, will be used to optimize the encoding of
monochrome images. This can significantly reduce the file size of the
output file. It is not required.
`jbig2enc <https://github.com/agl/jbig2enc>`__ is not generally
available for Ubuntu or Debian due to lingering concerns about patent
issues, but can easily be built from source. To add JBIG2 encoding, see
:ref:`jbig2`.
**pngquant**, if present, is optionally used to optimize the encoding of
PNG-style images in PDFs (actually, any that are that losslessly
encoded) by lossily quantizing to a smaller color palette. It is only
activated then the ``--optimize`` argument is ``2`` or ``3``.
**unpaper**, if present, enables the ``--clean`` and ``--clean-final``
command line options.
These are in addition to the Python packaging dependencies, meaning that
unfortunately, the ``pip install`` command cannot satisfy all of them.
Installing HEAD revision from sources
=====================================
If you have ``git`` and Python 3.10 or newer installed, you can install
from source. When the ``pip`` installer runs, it will alert you if
dependencies are missing.
If you prefer to build every from source, you will need to `build
pikepdf from
source <https://pikepdf.readthedocs.io/en/latest/installation.html#building-from-source>`__.
First ensure you can build and install pikepdf.
To install the HEAD revision from sources in the current Python 3
environment:
.. code-block:: bash
pip install git+https://github.com/ocrmypdf/OCRmyPDF.git
Or, to install in editable mode
allowing customization of OCRmyPDF, use the ``-e`` flag:
.. code-block:: bash
pip install -e git+https://github.com/ocrmypdf/OCRmyPDF.git
You may find it easiest to install in a virtual environment, rather than
system-wide:
.. code-block:: bash
git clone -b main https://github.com/ocrmypdf/OCRmyPDF.git
python3 -m venv .venv
source .venv/bin/activate
cd OCRmyPDF
pip install .
However, ``ocrmypdf`` will only be accessible on the system PATH when
you activate the virtual environment.
To run the program:
.. code-block:: bash
ocrmypdf --help
If not yet installed, the script will notify you about dependencies that
need to be installed. The script requires specific versions of the
dependencies. Older version than the ones mentioned in the release notes
are likely not to be compatible to OCRmyPDF.
For development
---------------
To install all of the development and test requirements:
.. code-block:: bash
git clone -b main https://github.com/ocrmypdf/OCRmyPDF.git
python -m venv .venv
source .venv/bin/activate
cd OCRmyPDF
pip install -e .[test]
To add JBIG2 encoding, see :ref:`jbig2`.
Shell completions
=================
Completions for ``bash`` and ``fish`` are available in the project's
``misc/completion`` folder. The ``bash`` completions are likely ``zsh``
compatible but this has not been confirmed. Package maintainers, please
install these at the appropriate locations for your system.
To manually install the ``bash`` completion, copy
``misc/completion/ocrmypdf.bash`` to ``/etc/bash_completion.d/ocrmypdf``
(rename the file).
To manually install the ``fish`` completion, copy
``misc/completion/ocrmypdf.fish`` to
``~/.config/fish/completions/ocrmypdf.fish``.
Note on 32-bit support
======================
Many Python libraries no longer provide 32-bit binary wheels for Linux. This
includes many of the libraries that OCRmyPDF depends on, such as
Pillow. The easiest way to express this to end users is to say we don't
support 32-bit Linux.
However, if your Linux distribution still supports 32-bit binaries, you
can still install and use OCRmyPDF. A warning message will appear.
In practice, OCRmyPDF may need more than 32-bit memory space to run when
large documents are processed, so there are practical limitations to what
users can accomplish with it. Still, for the common use case of an 32-bit
ARM NAS or Raspberry Pi processing small documents, it should work.
+225
View File
@@ -0,0 +1,225 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
============
Introduction
============
OCRmyPDF is a Python application and library that adds text "layers" to images in
PDFs, making scanned image PDFs searchable. It uses OCR to guess the text
contained in images. OCRmyPDF also supports plugins
that enable customization of its processing steps, and it is highly tolerant
of PDFs containing scanned images and "born digital" content that doesn't
require text recognition.
About OCR
=========
`Optical character
recognition <https://en.wikipedia.org/wiki/Optical_character_recognition>`__
is a technology that converts images of typed or handwritten text, such as
in a scanned document, into computer text that can be selected, searched and copied.
OCRmyPDF uses
`Tesseract <https://github.com/tesseract-ocr/tesseract>`__, a widely
available open source OCR engine, to perform OCR.
.. _raster-vector:
About PDFs
==========
PDFs are page description files that attempt to preserve a layout
exactly. They contain `vector
graphics <http://vector-conversions.com/vectorizing/raster_vs_vector.html>`__
that can contain raster objects, such as scanned images. Because PDFs can
contain multiple pages (unlike many image formats) and can contain fonts
and text, they are a suitable format for exchanging scanned documents.
|image|
A PDF page may contain multiple images, even if it appears to have only
one image. Some scanners or scanning software may segment pages into
monochromatic text and color regions, for example, to enhance the compression
ratio and appearance of the page.
Rasterizing a PDF is the process of generating corresponding raster images.
OCR engines like Tesseract work with images, not scalable vector graphics
or mixed raster-vector-text graphics such as PDF.
About PDF/A
===========
`PDF/A <https://en.wikipedia.org/wiki/PDF/A>`__ is an ISO-standardized
subset of the full PDF specification that is designed for archiving (the
'A' stands for Archive). PDF/A differs from PDF primarily by omitting
features that could complicate future file readability,
such as embedded Javascript, video, audio and references to external
fonts. All fonts and resources needed to interpret the PDF must be
contained within it. Because PDF/A disables Javascript and other types
of embedded content, it is likely more secure.
There are various conformance levels and versions, such as "PDF/A-2b".
In general, the preferred format for scanned documents is PDF/A. Some
governments and jurisdictions, US Courts in particular, `mandate the use
of PDF/A <https://pdfblog.com/2012/02/13/what-is-pdfa/>`__ for scanned
documents.
Since most individuals scanning documents aim for long-term readability,
OCRmyPDF defaults to generating PDF/A-2b.
PDF/A does have a few drawbacks. Some PDF viewers display an alert
indicating that the file is in PDF/A format, which may confuse some users.
Additionally, it tends to result in larger files than standard PDFs because
it embeds certain resources, even if they are widely available. PDF/A
files can be digitally signed but may not be encrypted to ensure future
readability. Fortunately, converting from PDF/A to a regular PDF is
straightforward, and any PDF viewer can handle PDF/A files.
What OCRmyPDF does
==================
OCRmyPDF analyzes each page of a PDF to determine the required colorspace
and resolution (DPI) for capturing all the information on that page without
losing content. It uses
`Ghostscript <http://ghostscript.com/>`__ to rasterize each page and subsequently
performs OCR on the rasterized image to generate an OCR "layer." This layer
is then integrated back into the original PDF.
While it is possible to use a program like Ghostscript or ImageMagick to
obtain an image and then run that image through Tesseract OCR, this process
actually generates a new PDF, potentially resulting in the loss of various
details (such as the document's metadata). In contrast, OCRmyPDF can produce
a minimally altered PDF as the output.
OCRmyPDF also offers several image processing options, such as deskew, which
enhances the visual quality of files and the accuracy of OCR. When these
options are utilized, the OCR layer is integrated into the processed image.
By default, OCRmyPDF generates archival PDFs in the PDF/A format, which is
a more rigid subset of PDF features designed for long-term archives. If you
prefer regular PDFs, you can disable this feature using the
``--output-type pdf`` option.
Why you shouldn't do this manually
==================================
A PDF is similar to an HTML file, in that it contains document structure
along with images. While some PDFs may solely display a full-page image,
they often contain additional content that would be forfeited if not preserved.
A manual process could take one of these approaches:
1. Rasterize each page as an image, perform OCR on the images, and then merge the
output into a PDF. This method preserves the layout of each page, but
resamples all images potentially leading to quality loss, increased file size,
and the introduction of compression artifacts, among other issues.
2. Extract each image, OCR, and combine the output into a PDF. This approach
loses the context in which images are used in the PDF, potentially resulting
in loss of information related to scaling and position of images. Some scanned
PDFs contain multiple images segmented into black and white, grayscale
and color regions, with stencil masks to prevent overlap, as this can
enhance the appearance of a file while reducing file size.
Reassembling these images can be challenging, and risks losing vector art
or text that is not part of an image.
In cases where a PDF solely serves as a container for images without any
rotation, scaling, or cropping, the second approach can be lossless.
OCRmyPDF uses various strategies depending on input options and the input PDF
itself. Generally, it rasterizes a page for OCR and then integrates the OCR
data back into the original PDF. This approach allows it to handle complex
PDFs and preserve their content as much as possible.
Furthermore, OCRmyPDF supports a wide range of edge cases that have emerged
during several years of development. It accommodates PDF features like
images within Form XObjects and pages with UserUnit scaling. It also
supports less common image formats like non-monochrome 1-bit images and
provides warnings about files you may not want to OCR. Thanks to tools
like pikepdf and QPDF, it can auto-repair damaged PDFs. You don't need to
understand the intricacies of these issues; you should be able to use
OCRmyPDF with any PDF file, and expect reasonable results.
Limitations
===========
OCRmyPDF is subject to limitations imposed by the Tesseract OCR engine.
These limitations are inherent to any software relying on Tesseract:
- The OCR accuracy may not match that of commercial OCR solutions.
- It is incapable of recognizing handwriting.
- It may detect gibberish and report it as OCR output.
- Results may be subpar when a document contains languages not specified
in the ``-l LANG`` argument.
- Tesseract may struggle to analyze the natural reading order of documents.
For instance, it might fail to recognize two columns in a document and
attempt to join text across columns.
- Poor quality scans can result in subpar OCR quality. In other words, the
quality of the OCR output depends on the quality of the input.
- Tesseract does not provide information about the font family to which text
belongs.
- Tesseract does not divide text into paragraphs or headings. It only provides
the text and its bounding box. As such, the generated PDF does not
contain any information about the document's structure.
Ghostscript also imposes some limitations:
- PDFs containing JPEG 2000-encoded content may be converted to JPEG
encoding, which may introduce compression artifacts, if Ghostscript
PDF/A is enabled.
- Ghostscript may transcode grayscale and color images, potentially
lossily, based on an internal algorithm. This
behavior can be suppressed by setting ``--pdfa-image-compression`` to
``jpeg`` or ``lossless`` to set all images to one type or the other.
Ghostscript lacks an option to maintain the input image's format.
(Modern Ghostscript can copy JPEG images without transcoding them.)
- Ghostscript's PDF/A conversion removes any XMP metadata that is not
one of the standard XMP metadata namespaces for PDFs. In particular,
PRISM Metadata is removed.
- Ghostscript's PDF/A conversion may remove or deactivate
hyperlinks and other active content.
You can use ``--output-type pdf`` to disable PDF/A conversion and produce
a standard, non-archival PDF.
Regarding OCRmyPDF itself:
- PDFs using transparency are not currently represented in the test
suite
Similar programs
================
To the author's knowledge, OCRmyPDF is the most feature-rich and
thoroughly tested command line OCR PDF conversion tool. If it does not
meet your needs, contributions and suggestions are welcome.
Ghostscript recently added three "pdfocr" output devices. They work by
rasterizing all content and converting all pages to a single colour space.
Web front-ends
==============
The Docker image of OCRmyPDF provides a web service front-end
that allows files to submitted over HTTP, and the results can be downloaded.
This is an HTTP server intended to demonstrate how OCRmyPDF can be
integrated into a web service. It is not intended to be deployed on the
public internet and does not provide any security measures.
In addition, the following third-party integrations are available:
- `Paperless-ngx <https://docs.paperless-ngx.com/>`__ is a free software
document management system that uses OCRmyPDF to perform OCR on
uploaded documents.
- `Nextcloud OCR <https://github.com/janis91/ocr>`__ is a free software
plugin for the Nextcloud private cloud software.
OCRmyPDF is not designed to be secure against malware-bearing PDFs (see
`Using OCRmyPDF online <ocr-service>`__). Users should ensure they
comply with OCRmyPDF's licenses and the licenses of all dependencies. In
particular, OCRmyPDF requires Ghostscript, which is licensed under
AGPLv3.
.. |image| image:: images/bitmap_vs_svg.svg
+83
View File
@@ -0,0 +1,83 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
.. _jbig2:
============================
Installing the JBIG2 encoder
============================
Most Linux distributions do not include a JBIG2 encoder since JBIG2
encoding was patented for a long time. All known JBIG2 US patents have
expired as of 2017, but it is possible that unknown patents exist.
JBIG2 encoding is recommended for OCRmyPDF and is used to losslessly
create smaller PDFs. If JBIG2 encoding is not available, lower quality
CCITT encoding will be used for monochrome images.
JBIG2 decoding is not patented and is performed automatically by most
PDF viewers. It is widely supported and has been part of the PDF
specification since 2001.
JBIG encoding is automatically provided by these OCRmyPDF packages:
- Docker image (both Ubuntu and Alpine)
- Snap package
- ArchLinux AUR package
- Alpine Linux package
- Homebrew on macOS
For all other platforms, you would need to build the JBIG2 encoder from source:
.. code-block:: bash
git clone https://github.com/agl/jbig2enc
cd jbig2enc
./autogen.sh
./configure && make
[sudo] make install
.. _jbig2-lossy:
Dependencies include libtoolize and libleptonica, which on Ubuntu systems
are packaged as libtool and libleptonica-dev. On Fedora (35) they are packaged
as libtool and leptonica-devel. For this to work, please make sure to install
``autotools``, ``automake``, ``libtool`` and ``leptonica`` first if not already
installed.
.. code-block:: bash
[sudo] apt install autotools-dev automake libtool libleptonica-dev
..
Lossy mode JBIG2
================
OCRmyPDF provides lossy mode JBIG2 as an advanced and potentially dangerous
feature. Users should
`review the technical concerns with JBIG2 in lossy
mode <https://en.wikipedia.org/wiki/JBIG2#Disadvantages>`__
and decide if this feature is acceptable for their use case. In general,
this mode should not be used for archival purposes, should not be used when
the original document is not available or will be destroyed, and should
not be used when numbers present in the document are important, because
there is a risk of 6/8 and 8/6 substitution errors.
JBIG2 lossy mode does achieve higher compression ratios than any other
monochrome (bitonal) compression technology; for large text documents
the savings are considerable. JBIG2 lossless still gives great
compression ratios and is a major improvement over the older CCITT G4
standard.
To turn on JBIG2 lossy mode, add the argument ``--jbig2-lossy``.
``--optimize {1,2,3}`` are necessary for the argument to take effect
also required. Also, a JBIG2 encoder must be installed as described in
the previous section.
You can adjust the threshold for JBIG2 compression with the
``--jbig2-threshold``. The default is 0.85, meaning that if two symbols
are 85% similar, they will be compressed together.
*Due to an oversight, ocrmypdf v7.0 and v7.1 used lossy mode by
default.*
+141
View File
@@ -0,0 +1,141 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
.. _lang-packs:
====================================
Installing additional language packs
====================================
OCRmyPDF uses Tesseract for OCR, and relies on its language packs for all languages.
On most platforms, English is installed with Tesseract by default, but not always.
Tesseract supports `most
languages <https://github.com/tesseract-ocr/tesseract/blob/main/doc/tesseract.1.asc#languages>`__.
Languages are identified by standardized three-letter codes (called ISO 639-2 Alpha-3).
Tesseract's documentation also lists the three-letter code for your language.
Some are anglicized, e.g. Spanish is ``spa`` rather than ``esp``, while others
are not, e.g. German is ``deu`` and French is ``fra``.
Language packs (strictly speaking, Tesseract "traineddata" files) generally correspond
to the language in question, but different language packs are used in certain
situations. For German, the "Fraktur" language pack can assist with reading older
materials in the Fraktur typeface family (``deu_frak``). Some communities have changed
their script from Cyrillic to Latin; the Cyrillic version of Uzbek is available
as ``uzb_cyrl`` and the Latin version is ``uzb``.
After you have installed a language pack, you can use it with ``ocrmypdf -l <language>``,
for example ``ocrmypdf -l spa``. For multilingual documents, you can specify
all languages to be expected, e.g. ``ocrmypdf -l eng+fra`` for English and French.
English is assumed by default unless other language(s) are specified.
For Linux users, you can often find packages that provide language
packs.
Platform install steps
======================
Debian and Ubuntu (apt)
-----------------------
.. code-block:: bash
# Display a list of all Tesseract language packs
apt-cache search tesseract-ocr
# Install Chinese Simplified language pack
apt-get install tesseract-ocr-chi-sim
You can then pass the ``-l LANG`` argument to OCRmyPDF to give a hint as
to what languages it should search for. Multiple languages can be
requested using either ``-l eng+fra`` (English and French) or
``-l eng -l fra``.
Fedora
------
.. code-block:: bash
# Display a list of all Tesseract language packs
dnf search tesseract
# Install Chinese Simplified language pack
dnf install tesseract-langpack-chi_sim
You can then pass the ``-l LANG`` argument to OCRmyPDF to give a hint as
to what languages it should search for. Multiple languages can be
requested using either ``-l eng+fra`` (English and French) or
``-l eng -l fra``.
Archlinux
------
.. code-block:: bash
# Display a list of all Tesseract language packs
pacman -Ss tesseract-data
# Install German language pack
pacman -S tesseract-data-deu
You can then pass the ``-l LANG`` argument to OCRmyPDF to give a hint as
to what languages it should search for. Multiple languages can be
requested using either ``-l eng+fra`` (English and French) or
``-l eng -l fra``.
Gentoo
------
On Gentoo the package ``app-text/tessdata_fast``, which ``app-text/tesseract`` depends on, handles Tesseract languages.
It accepts USE flags to select what languages should be installed, these can be set in ``/etc/portage/package.use``.
Alternatively one can globally set the `L10N use extension <https://wiki.gentoo.org/wiki/Localization/Guide#L10N>`__ in ``/etc/portage/make.conf``.
This enables these languages for all packages (e.g. including aspell).
.. code-block:: bash
# Display a list of all Tesseract language packs
equery uses app-text/tessdata_fast
# Add English and German language support for Tesseract only
echo 'app-text/tessdata_fast l10n_de l10n_en' >> /etc/portage/package.use
# Add global English and German language support (the `l10n_` from equery has to be omitted)
echo L10N="de en" >> /etc/portage/make.conf
# update system to reflect changed USE flags
emerge --update --deep --newuse @world
You can then pass the ``-l LANG`` argument to OCRmyPDF to give a hint as
to what languages it should search for. Multiple languages can be
requested using either ``-l eng+fra`` (English and French) or
``-l eng -l fra``.
macOS
-----
You can install additional language packs by
:ref:`installing Tesseract using Homebrew with all language packs <macos-all-languages>`.
Docker
------
Users of the OCRmyPDF Docker image should install language packs into a
derived Docker image as
:ref:`described in that section <docker-lang-packs>`.
Windows
-------
The Tesseract installer provided by Chocolatey currently includes only English language.
To install other languages, download the respective language pack (``.traineddata`` file)
from https://github.com/tesseract-ocr/tessdata/ and place it in
``C:\\Program Files\\Tesseract-OCR\\tessdata`` (or wherever Tesseract OCR is installed).
Custom language packs
=====================
If you have fine-tuned or trained Tesseract and generated custom trained data, you can
copy your ``customlang.traineddata`` file into your Tesseract "tessdata" folder, and
then use the ``-l customlang`` argument to tell OCRmyPDF to pass that language on to
Tesseract.
+77
View File
@@ -0,0 +1,77 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
================
Maintainer notes
================
This is for those who package OCRmyPDF for downstream use. (Thank you
for your hard work.)
Known ports/packagers
=====================
OCRmyPDF has been ported to many platforms already. If you are interesting in
porting to a new platform, check with
`Repology <https://repology.org/projects/?search=ocrmypdf>`__ to see the status
of that platform.
Make sure you can package pikepdf
---------------------------------
pikepdf, created by the same author, is a mixed Python and C++14 package with
much stiffer build requirements. If you want to use OCRmyPDF on some novel platform
or distribution, first make sure you can package pikepdf.
Non-Python dependencies
-----------------------
Note that we have non-Python dependencies. In particular, OCRmyPDF requires
Ghostscript and Tesseract OCR to be installed and needs to be able to locate their
binaries on the system PATH. On Windows, OCRmyPDF will also check the registry
for their locations.
Tesseract OCR relies on SIMD for performance and only has proper support for this
on ARM and x86_64. Performance may be poor on other processor architectures.
Versioning scheme
-----------------
OCRmyPDF uses hatch-vcs for versioning, which derives the version from
Git as a single source of truth. This may be unsuitable for some distributions, e.g.
to indicate that your distribution modifies OCRmyPDF in some way.
You can patch the ``__version__`` variable in ``src/ocrmypdf/_version.py`` if
necessary, or set the environment variable ``SETUPTOOLS_SCM_PRETEND_VERSION``
to the required version, if you need to override versioning for some reason.
jbig2enc
--------
OCRmyPDF will use jbig2enc, a JBIG2 encoder, if one can be found. Some distributions
have shied away from packaging JBIG2 because it contains patented algorithms, but
all patents have expired since 2017. If possible, consider packaging it too to
improve OCRmyPDF's compression.
Command line completions
------------------------
Please ensure that command line completions are installed, as described in the
installation documentation.
32-bit Linux support
--------------------
If you maintain a Linux distribution that supports 32-bit x86 or ARM, OCRmyPDF
should continue to work as long as all of its dependencies continue to be
available in 32-bit form. Please note we do not test on 32-bit platforms.
HEIF/HEIC
---------
OCRmyPDF defaults to installing the pi-heif PyPI package, which supports converting
HEIF (High Efficiency Image File Format) images to PDF from the command line.
If your distribution does not have this library available, you can exclude it and
OCRmyPDF will gracefully degrade automatically, losing only support for this
feature.
+100
View File
@@ -0,0 +1,100 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
================
PDF optimization
================
OCRmyPDF includes an image-oriented PDF optimizer. By default, the optimizer
runs with safe settings with the goal of improving compression at no loss of
quality. At higher optimization levels, lossy optimizations may be applied and
tuned. Optimization occurs after OCR, and only if OCR succeeded. It does not
perform other possible optimizations such as deduplicating resources,
consolidating fonts, simplifying vector drawings, or anything of that nature.
.. list-table:: Title
:widths: 33 6 60
:header-rows: 1
* - Optimization level
- Shorthand
- Description
* - ``--optimize 0``
- ``-O0``
- Disable most optimizations.
* - ``--optimize 1`` (default)
- ``-O1``
- Safe and lossless optimizations.
* - ``--optimize 2``
- ``-O2``
- Safe and lossy optimizations.
* - ``--optimize 3``
- ``-O3``
- Aggressive lossy optimizations.
The exact type of optimizations performed will vary over time, and depend on
the availability of third-party tools.
Despite optimizations, OCRmyPDF might still increase the overall file size,
since it must embed information about the recognized text, and depending on the
settings chosen, may not be able to represent the output file as compactly as
the input file.
Optimizations that always occurs
================================
OCRmyPDF will automatically replace obsolete or inferior compression schemes
such as RLE or LZW with superior schemes such as Deflate, and convert
monochrome images to CCITT G4. Since this is lossless, it always occurs and there
is no way to disable it. Other non-image compressed objects are compressed as
well.
Fast web view
=============
OCRmyPDF automatically optimizes PDFs for "fast web view" in Adobe Acrobat's
parlance, or equivalently, linearizes PDFs so that the resources they reference
are presented in the order a viewer needs them for sequential display. This
reduces the latency of viewing a PDF both online and from local storage, in
exchange for a slight increase in file size.
To disable this optimization and all others, use ``ocrmypdf --optimize 0 ...``
or the shorthand ``-O0``.
Adobe Acrobat might not report the file as being "fast web view".
Lossless optimizations
======================
At optimization level ``-O1`` (the default), OCRmyPDF will also attempt lossless
image optimization.
If a JBIG2 encoder is available, then monochrome images will be converted to
JBIG2, with the potential for huge savings on large black and white images,
since JBIG2 is far more efficient than any other monochrome (bi-level)
compression. (All known US patents related to JBIG2 have probably expired, but
it remains the responsibility of the user to supply a JBIG2 encoder such as
`jbig2enc <https://github.com/agl/jbig2enc>`__. OCRmyPDF does not implement
JBIG2 encoding on its own.)
OCRmyPDF currently does not attempt to recompress losslessly compressed objects
more aggressively.
Lossy optimizations
===================
At optimization level ``-O2`` and ``-O3``, OCRmyPDF will some attempt lossy
image optimization.
If ``pngquant`` is installed, OCRmyPDF will use it to perform quantize paletted
images to reduce their size.
The quality of JPEGs may be lowered, on the assumption that a lower quality
image may be suitable for storage after OCR.
It is not possible to optimize all image types. Uncommon image types may be
skipped by the optimizer.
OCRmyPDF provides :ref:`lossy mode JBIG2 <jbig2-lossy>` as an advanced feature
that additional requires the argument ``--jbig2-lossy``.
+125
View File
@@ -0,0 +1,125 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
===================
PDF security issues
===================
OCRmyPDF should only be used on PDFs you trust. It is not designed to
protect you against malware.
Recognizing that many users have an interest in handling PDFs and
applying OCR to PDFs they did not generate themselves, this article
discusses the security implications of PDFs and how users can protect
themselves.
The disclaimer applies: this software has no warranties of any kind.
PDFs may contain malware
========================
PDF is a rich, complex file format. The official PDF 1.7 specification,
ISO 32000:2008, is hundreds of pages long and references several annexes
each of which are similar in length. PDFs can contain video, audio, XML,
JavaScript and other programming, and forms. In some cases, they can
open internet connections to pre-selected URLs. All of these are possible
attack vectors.
In short, PDFs `may contain
viruses <https://security.stackexchange.com/questions/64052/can-a-pdf-file-contain-a-virus>`__.
If you do not trust a PDF or its source, do not open it or use OCRmyPDF
on it. Consider using a Docker container or virtual machine to isolate
an untrusted PDF from your system.
How OCRmyPDF processes PDFs
===========================
OCRmyPDF must open and interpret your PDF in order to insert an OCR
layer. First, it runs all PDFs through
`pikepdf <https://github.com/pikepdf/pikepdf>`__, a library based on
`QPDF <https://github.com/qpdf/qpdf>`__, a program that repairs PDFs
with syntax errors. This is done because, in the author's experience, a
significant number of PDFs in the wild, especially those created by
scanners, are not well-formed files. QPDF makes it more likely that
OCRmyPDF will succeed, but offers no security guarantees. QPDF is also
used to split the PDF into single page PDFs.
Finally, OCRmyPDF rasterizes each page of the PDF using
`Ghostscript <http://ghostscript.com/>`__ in ``-dSAFER`` mode.
Depending on the options specified, OCRmyPDF may graft the OCR layer
into the existing PDF or it may essentially reconstruct ("re-fry") a
visually identical PDF that may be quite different at the binary level.
That said, OCRmyPDF is not a tool designed for sanitizing PDFs.
Password protected PDFs
=======================
Password protected PDFs usually have two passwords, and owner and user
password. When the user password is set to empty, PDF readers will open
the file automatically and mark it as "(SECURED)". Password security can
also request certain restrictions on the PDF, but anyone can remove these
restrictions if they have either the owner *or* user password. Passwords
mainly present a barrier for casual users.
OCRmyPDF cannot remove passwords from PDFs. If you want to remove a
password from a PDF, you must use other software, such as ``qpdf``.
If the owner and user password are set, a
password is required for ``qpdf``. If only the owner password is set, then the
password can be stripped, even if one does not have the owner password. To
remove the password from a using QPDF, use:
.. code-block:: bash
qpdf --decrypt --password='abc123' input.pdf no_password.pdf
Then you can run OCRmyPDF on the file.
In its default mode, OCRmyPDF generates PDF/A. Passwords may not be set on PDF/A
documents. If you want to set a password on the output PDF, you must
specify ``--output-type pdf``.
Signature images
================
Many programs exist which are capable of inserting an image of someone's
signature. On its own, this offers no security guarantees. It is trivial
to remove the signature image and apply it to other files. This practice
offers no real security.
Digital signatures
==================
Important documents can be digitally signed and certified to attest to
their authorship, approval or execution of a legal agreement. OCRmyPDF
will detect signed PDFs and will not modify them, unless the
``--invalidate-digital-signatures`` option is used, which will
invalidate any signatures. (The signature may still be present in the PDF
if opened, but PDF readers will not validate it.)
A digital signature adds a cryptographic hash of the document to the
document, so tamper protection is provided. That also precludes OCRmyPDF
from modifying the document and preserving the signature.
Digital signatures are not the same as a signature image. A digital
signature is a cryptographic hash of the document that is encrypted with
the author's private key. The signature is decrypted with the author's
public key. The public key is usually distributed by a certificate
authority. The signature is then verified by the PDF reader. If the
document is modified, the signature will be invalidated.
Certificate-encrypted PDFs
==========================
PDFs can be encrypted with a certificate. This is a more secure form of
encryption than a password. The certificate is usually issued by a
certificate authority. A certificate is used to encrypt the document using
the public key for the benefit of a specific recipient who possesses
the private key.
OCRmyPDF cannot open certificate-encrypted PDFs. If you have the
certificate, you can use other PDF software, such as Acrobat, to
decrypt the PDF.
+26
View File
@@ -0,0 +1,26 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
===========
Performance
===========
Some users have noticed that current versions of OCRmyPDF do not run as quickly
as some older versions (specifically 6.x and older). This is because OCRmyPDF
added image optimization as a postprocessing step, and it is enabled by default.
Speed
=====
If running OCRmyPDF quickly is your main goal, you can use settings such as:
* ``--optimize 0`` to disable file size optimization
* ``--output-type pdf`` to disable PDF/A generation
* ``--fast-web-view 999999`` to disable fast web view optimization
* ``--skip-big`` to skip large images, if some pages have large images
You can also avoid:
* ``--force-ocr``
* Image preprocessing
+232
View File
@@ -0,0 +1,232 @@
.. SPDX-FileCopyrightText: 2022 James R. Barlow
..
.. SPDX-License-Identifier: CC-BY-SA-4.0
=======
Plugins
=======
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
RFC 2119.
You can use plugins to customize the behavior of OCRmyPDF at certain points of
interest.
Currently, it is possible to:
- add new command line arguments
- override the decision for whether or not to perform OCR on a particular file
- modify the image is about to be sent for OCR
- modify the page image before it is converted to PDF
- replace the Tesseract OCR with another OCR engine that has similar behavior
- replace Ghostscript with another PDF to image converter (rasterizer) or
PDF/A generator
OCRmyPDF plugins are based on the Python ``pluggy`` package and conform to its
conventions. Note that: plugins installed with as setuptools entrypoints are
not checked currently, because OCRmyPDF assumes you may not want to enable
plugins for all files.
See [OCRmyPDF-EasyOCR](https://github.com/ocrmypdf/OCRmyPDF-EasyOCR) for an
example of a straightforward, fully working plugin.
Script plugins
==============
Script plugins may be called from the command line, by specifying the name of a file.
Script plugins may be convenient for informal or "one-off" plugins, when a certain
batch of files needs a special processing step for example.
.. code-block:: bash
ocrmypdf --plugin ocrmypdf_example_plugin.py input.pdf output.pdf
Multiple plugins may be installed by issuing the ``--plugin`` argument multiple times.
Packaged plugins
================
Installed plugins may be installed into the same virtual environment as OCRmyPDF
is installed into. They may be invoked using Python standard module naming.
If you are intending to distribute a plugin, please package it.
.. code-block:: bash
ocrmypdf --plugin ocrmypdf_fancypants.pockets.contents input.pdf output.pdf
OCRmyPDF does not automatically import plugins, because the assumption is that
plugins affect different files differently and you may not want them activated
all the time. The command line or ``ocrmypdf.ocr(plugin='...')`` must call
for them.
Third parties that wish to distribute packages for ocrmypdf should package them
as packaged plugins, and these modules should begin with the name ``ocrmypdf_``
similar to ``pytest`` packages such as ``pytest-cov`` (the package) and
``pytest_cov`` (the module).
.. note::
We recommend plugin authors name their plugins with the prefix
``ocrmypdf-`` (for the package name on PyPI) and ``ocrmypdf_`` (for the
module), just like pytest plugins. At the same time, please make it clear
that your package is not official.
Plugins
=======
You can also create a plugin that OCRmyPDF will always automatically load if both are
installed in the same virtual environment, using a project entrypoint.
OCRmyPDF uses the entrypoint namespace "ocrmypdf".
For example, ``pyproject.toml`` would need to contain the following, for a plugin named
``ocrmypdf-exampleplugin``:
.. code-block:: toml
[project]
name = "ocrmypdf-exampleplugin"
[project.entry-points."ocrmypdf"]
exampleplugin = "exampleplugin.pluginmodule"
Plugin requirements
===================
OCRmyPDF generally uses multiple worker processes. When a new worker is started,
Python will import all plugins again, including all plugins that were imported earlier.
This means that the global state of a plugin in one worker will not be shared with
other workers. As such, plugin hook implementations should be stateless, relying
only on their inputs. Hook implementations may use their input parameters to
to obtain a reference to shared state prepared by another hook implementation.
Plugins must expect that other instances of the plugin will be running
simultaneously.
The ``context`` object that is passed to many hooks can be used to share information
about a file being worked on. Plugins must write private, plugin-specific data to
a subfolder named ``{options.work_folder}/ocrmypdf-plugin-name``. Plugins MAY
read and write files in ``options.work_folder``, but should be aware that their
semantics are subject to change.
OCRmyPDF will delete ``options.work_folder`` when it has finished OCRing
a file, unless invoked with ``--keep-temporary-files``.
The documentation for some plugin hooks contain a detailed description of the
execution context in which they will be called.
Plugins should be prepared to work whether executed in worker threads or worker
processes. Generally, OCRmyPDF uses processes, but has a semi-hidden threaded
argument that simplifies debugging.
Plugin hooks
============
A plugin may provide the following hooks. Hooks must be decorated with
``ocrmypdf.hookimpl``, for example:
.. code-block:: python
from ocrmpydf import hookimpl
@hookimpl
def add_options(parser):
pass
The following is a complete list of hooks that are available, and when
they are called.
.. _firstresult:
**Note on firstresult hooks**
If multiple plugins install implementations for this hook, they will be called in
the reverse of the order in which they are installed (i.e., last plugin wins).
When each hook implementation is called in order, the first implementation that
returns a value other than ``None`` will "win" and prevent execution of all other
hooks. As such, you cannot "chain" a series of plugin filters together in this
way. Instead, a single hook implementation should be responsible for any such
chaining operations.
Examples
========
* OCRmyPDF's test suite contains several plugins that are used to simulate certain
test conditions.
* `ocrmypdf-papermerge <https://github.com/papermerge/OCRmyPDF_papermerge>`_ is
a production plugin that integrates OCRmyPDF and the Papermerge document
management system.
Suppressing or overriding other plugins
---------------------------------------
.. autofunction:: ocrmypdf.pluginspec.initialize
Custom command line arguments
-----------------------------
.. autofunction:: ocrmypdf.pluginspec.add_options
.. autofunction:: ocrmypdf.pluginspec.check_options
Execution and progress reporting
--------------------------------
.. autoclass:: ocrmypdf.pluginspec.ProgressBar
:members:
:special-members: __init__, __enter__, __exit__
.. autoclass:: ocrmypdf.pluginspec.Executor
:members:
:special-members: __call__
.. autofunction:: ocrmypdf.pluginspec.get_logging_console
.. autofunction:: ocrmypdf.pluginspec.get_executor
.. autofunction:: ocrmypdf.pluginspec.get_progressbar_class
Applying special behavior before processing
-------------------------------------------
.. autofunction:: ocrmypdf.pluginspec.validate
PDF page to image
-----------------
.. autofunction:: ocrmypdf.pluginspec.rasterize_pdf_page
Modifying intermediate images
-----------------------------
.. autofunction:: ocrmypdf.pluginspec.filter_ocr_image
.. autofunction:: ocrmypdf.pluginspec.filter_page_image
.. autofunction:: ocrmypdf.pluginspec.filter_pdf_page
OCR engine
----------
.. autofunction:: ocrmypdf.pluginspec.get_ocr_engine
.. autoclass:: ocrmypdf.pluginspec.OcrEngine
:members:
.. automethod:: __str__
.. autoclass:: ocrmypdf.pluginspec.OrientationConfidence
PDF/A production
----------------
.. autofunction:: ocrmypdf.pluginspec.generate_pdfa
PDF optimization
----------------
.. autofunction:: ocrmypdf.pluginspec.optimize_pdf
.. autofunction:: ocrmypdf.pluginspec.is_optimization_enabled
File diff suppressed because it is too large Load Diff
+91
View File
@@ -0,0 +1,91 @@
#!/usr/bin/env python3
# SPDX-FileCopyrightText: 2016 findingorder <https://github.com/findingorder>
# SPDX-FileCopyrightText: 2024 nilsro <https://github.com/nilsro>
# SPDX-License-Identifier: MIT
"""Example of using ocrmypdf as a library in a script.
This script will recursively search a directory for PDF files and run OCR on
them. It will log the results. It runs OCR on every file, even if it already
has text. OCRmyPDF will detect files that already have text.
You should edit this script to meet your needs.
"""
from __future__ import annotations
import filecmp
import logging
import os
import posixpath
import shutil
import sys
from pathlib import Path
import ocrmypdf
# pylint: disable=logging-format-interpolation
# pylint: disable=logging-not-lazy
def filecompare(a, b):
try:
return filecmp.cmp(a, b, shallow=True)
except FileNotFoundError:
return False
script_dir = Path(__file__).parent
# set archive_dir to a path for backup original documents. Leave empty if not required.
archive_dir = "/pdfbak"
if len(sys.argv) > 1:
start_dir = Path(sys.argv[1])
else:
start_dir = Path(".")
if len(sys.argv) > 2:
log_file = Path(sys.argv[2])
else:
log_file = script_dir.with_name("ocr-tree.log")
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(message)s",
filename=log_file,
filemode="a",
)
logging.info(f"Start directory {start_dir}")
ocrmypdf.configure_logging(ocrmypdf.Verbosity.default)
for filename in start_dir.glob("**/*.pdf"):
logging.info(f"Processing {filename}")
if ocrmypdf.pdfa.file_claims_pdfa(filename)["pass"]:
logging.info("Skipped document because it already contained text")
else:
archive_filename = archive_dir + str(filename)
if len(archive_dir) > 0 and not filecompare(filename, archive_filename):
logging.info(f"Archiving document to {archive_filename}")
try:
shutil.copy2(filename, posixpath.dirname(archive_filename))
except OSError:
os.makedirs(posixpath.dirname(archive_filename))
shutil.copy2(filename, posixpath.dirname(archive_filename))
try:
result = ocrmypdf.ocr(filename, filename, deskew=True)
logging.info(result)
except ocrmypdf.exceptions.EncryptedPdfError:
logging.info("Skipped document because it is encrypted")
except ocrmypdf.exceptions.PriorOcrFoundError:
logging.info("Skipped document because it already contained text")
except ocrmypdf.exceptions.DigitalSignatureError:
logging.info("Skipped document because it has a digital signature")
except ocrmypdf.exceptions.TaggedPDFError:
logging.info(
"Skipped document because it does not need ocr as it is tagged"
)
except Exception:
logging.error("Unhandled error occured")
logging.info("OCR complete")
+313
View File
@@ -0,0 +1,313 @@
# SPDX-FileCopyrightText: 2021 Frank Pille
# SPDX-FileCopyrightText: 2020 Alex Willner
# SPDX-License-Identifier: MIT
set -o errexit
__ocrmypdf_arguments()
{
local arguments="\
--help (show help message)
--language (language(s) of the file to be OCRed)
--image-dpi (assume this DPI if input image DPI is unknown)
--output-type (select PDF output options)
--sidecar (write OCR to text file)
--version (print program version and exit)
--jobs (how many worker processes to use)
--quiet (suppress INFO messages)
--verbose (set verbosity level)
--title (set metadata)
--author (set metadata)
--subject (set metadata)
--keywords (set metadata)
--rotate-pages (rotate pages to correct orientation)
--remove-background (attempt to remove background from pages)
--deskew (fix small horizontal alignment skew)
--clean (clean document images before OCR)
--clean-final (clean document images and keep result)
--unpaper-args (a quoted string of arguments to pass to unpaper)
--oversample (oversample images to this DPI)
--remove-vectors (don\'t send vector objects to OCR)
--threshold (threshold images before OCR)
--force-ocr (OCR documents that already have printable text)
--skip-text (skip OCR on any pages that already contain text)
--redo-ocr (redo OCR on any pages that seem to have OCR already)
--invalidate-digital-signatures (remove digital signatures from PDF)
--skip-big (skip OCR on pages larger than this many MPixels)
--optimize (select optimization level)
--jpeg-quality (JPEG quality [0..100])
--png-quality (PNG quality [0..100])
--jbig2-lossy (enable lossy JBIG2 (see docs))
--jbig2-threshold (set JBIG2 threshold (see docs))
--pages (apply OCR to only the specified pages)
--max-image-mpixels (image decompression bomb threshold)
--pdf-renderer (select PDF renderer options)
--rotate-pages-threshold (page rotation confidence)
--pdfa-image-compression (set PDF/A image compression options)
--fast-web-view (if file size if above this amount in MB linearize PDF)
--plugin (name of plugin to import)
--keep-temporary-files (keep temporary files (debug)
--tesseract-config (set custom tesseract config file)
--tesseract-pagesegmode (set tesseract --psm)
--tesseract-oem (set tesseract --oem)
--tesseract-thresholding (set tesseract image thresholding)
--tesseract-timeout (maximum number of seconds to wait for OCR)
--user-words (specify location of user words file)
--user-patterns (specify location of user patterns file)
--no-progress-bar (disable the progress bar)
--color-conversion-strategy (select color conversion strategy)
"
COMPREPLY=( $( compgen -W "$arguments" -- "$cur") )
# Remove description if only one completion exists
if [[ ${#COMPREPLY[*]} -eq 1 ]]; then
COMPREPLY=( ${COMPREPLY[0]%% *} )
fi
}
__ocrmypdf_output-type()
{
local choices="pdfa (output a PDF/A (default))
pdf (output a standard PDF)
pdfa-1 (output a PDF/A-1b)
pdfa-2 (output a PDF/A-2b)
pdfa-3 (output a PDF/A-3b)
none (do not produce an output PDF (for example, if you only care about --sidecar))"
COMPREPLY=( $( compgen -W "$choices" -- "$cur") )
# Remove description if only one completion exists
if [[ ${#COMPREPLY[*]} -eq 1 ]]; then
COMPREPLY=( ${COMPREPLY[0]%% *} )
fi
}
__ocrmypdf_verbose()
{
local choices="0 (standard output messages)
1 (troubleshooting output messages)
2 (debugging output messages)"
COMPREPLY=( $( compgen -W "$choices" -- "$cur") )
# Remove description if only one completion exists
if [[ ${#COMPREPLY[*]} -eq 1 ]]; then
COMPREPLY=( ${COMPREPLY[0]%% *} )
fi
}
__ocrmypdf_optimize()
{
local choices="0 (do not optimize)
1 (do safe, lossless optimizations (default))
2 (do some lossy optimizations)
3 (do aggressive lossy optimizations (including lossy JBIG2))"
COMPREPLY=( $( compgen -W "$choices" -- "$cur") )
# Remove description if only one completion exists
if [[ ${#COMPREPLY[*]} -eq 1 ]]; then
COMPREPLY=( ${COMPREPLY[0]%% *} )
fi
}
__ocrmypdf_pdf-renderer()
{
local choices="auto (auto select PDF renderer)
hocr (use hOCR renderer)
hocrdebug (uses hOCR renderer in debug mode, showing recognized text)
sandwich (use sandwich renderer)"
COMPREPLY=( $( compgen -W "$choices" -- "$cur") )
# Remove description if only one completion exists
if [[ ${#COMPREPLY[*]} -eq 1 ]]; then
COMPREPLY=( ${COMPREPLY[0]%% *} )
fi
}
__ocrmypdf_pdfa-image-compression()
{
local choices="auto (let Ghostscript decide how to compress images)
jpeg (convert color and grayscale images to JPEG)
lossless (convert color and grayscale images to lossless (PNG))"
COMPREPLY=( $( compgen -W "$choices" -- "$cur") )
# Remove description if only one completion exists
if [[ ${#COMPREPLY[*]} -eq 1 ]]; then
COMPREPLY=( ${COMPREPLY[0]%% *} )
fi
}
__ocrmypdf_tesseract-pagesegmode()
{
local choices="0 (orientation and script detection (OSD) only)
1 (automatic page segmentation with OSD)
2 (automatic page segmentation, but no OSD, or OCR)
3 (fully automatic page segmentation, but no OSD (default))
4 (assume a single column of text of variable sizes)
5 (assume a single uniform block of vertically aligned text)
6 (assume a single uniform block of text)
7 (treat the image as a single text line)
8 (treat the image as a single word)
9 (treat the image as a single word in a circle)
10 (treat the image as a single character)
11 (sparse text - find as much text as possible in no particular order)
12 (sparse text with OSD)
13 (raw line - treat the image as a single text line)"
COMPREPLY=( $( compgen -W "$choices" -- "$cur") )
# Remove description if only one completion exists
if [[ ${#COMPREPLY[*]} -eq 1 ]]; then
COMPREPLY=( ${COMPREPLY[0]%% *} )
fi
}
__ocrmypdf_tesseract-oem()
{
local choices="0 (legacy engine only)
1 (neural nets LSTM engine only)
2 (legacy + LSTM engines)
3 (default, based on what is available)"
COMPREPLY=( $( compgen -W "$choices" -- "$cur") )
# Remove description if only one completion exists
if [[ ${#COMPREPLY[*]} -eq 1 ]]; then
COMPREPLY=( ${COMPREPLY[0]%% *} )
fi
}
__ocrmypdf_tesseract-thresholding()
{
local choices="auto (let OCRmyPDF pick thresholding - current always uses otsu)
otsu (use hOCR renderer)
adaptive-otsu (use adaptive Otsu thresholding)
sauvola (use Sauvola thresholding)"
COMPREPLY=( $( compgen -W "$choices" -- "$cur") )
# Remove description if only one completion exists
if [[ ${#COMPREPLY[*]} -eq 1 ]]; then
COMPREPLY=( ${COMPREPLY[0]%% *} )
fi
}
__ocrmypdf_color-conversion-strategy()
{
local choices="LeaveColorUnchanged (default)
CMYK (convert to CMYK)
Gray (convert to grayscale)
RGB (convert to RGB)
UseDeviceIndependentColor (convert with device independent color)"
COMPREPLY=( $( compgen -W "$choices" -- "$cur") )
# Remove description if only one completion exists
if [[ ${#COMPREPLY[*]} -eq 1 ]]; then
COMPREPLY=( ${COMPREPLY[0]%% *} )
fi
}
__ocrmypdf_check_previous()
{
case $prev in
-h|--help|--version)
return 0
;;
-l|--language)
COMPREPLY=$( command tesseract --list-langs 2>/dev/null )
COMPREPLY=( $( compgen -W '${COMPREPLY[@]##*:}' -- "$cur" ) )
return 0
;;
--output-type)
__ocrmypdf_output-type
return 0
;;
-j|--jobs)
COMPREPLY=( $( compgen -W '{1..'$( _ncpus )'}' -- "$cur" ) )
return 0
;;
-v|--verbose)
__ocrmypdf_verbose
return 0
;;
-O|--optimize)
__ocrmypdf_optimize
return 0
;;
--pdf-renderer)
__ocrmypdf_pdf-renderer
return 0
;;
--pdfa-image-compression)
__ocrmypdf_pdfa-image-compression
return 0
;;
--tesseract-pagesegmode)
__ocrmypdf_tesseract-pagesegmode
return 0
;;
--tesseract-oem)
__ocrmypdf_tesseract-oem
return 0
;;
--tesseract-thresholding)
__ocrmypdf_tesseract-thresholding
return 0
;;
--title|--author|--subject|--keywords|--unpaper-args|--pages|--plugin|\
--jpeg-quality|--png-quality|--image-dpi|--oversample|--skip-big|--max-image-mpixels|\
--tesseract-timeout|--rotate-pages-threshold|--fast-web-view)
# argument required but no completions available
return 0
;;
--tesseract-config|--user-words|--user-patterns|--sidecar)
_filedir
return 0
;;
--color-conversion-strategy)
__ocrmypdf_color-conversion-strategy
return 0
;;
esac
return 1
}
_ocrmypdf()
{
local OLDIFS="$IFS"
local IFS=$'\n'
local cur prev
# Homebrew on Macs have version 1.3 of bash-completion which doesn't include - see #502
if declare -F _init_completion >/dev/null 2>&1; then
_init_completion || return
else
COMPREPLY=()
_get_comp_words_by_ref cur prev
fi
if __ocrmypdf_check_previous -ne 0; then
return
fi
if [[ "$cur" == -* ]]; then
__ocrmypdf_arguments
else
_filedir
fi
IFS="$OLDIFS"
return
} &&
complete -F _ocrmypdf ocrmypdf
set +o errexit
# ex: filetype=sh
+156
View File
@@ -0,0 +1,156 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MIT
complete -c ocrmypdf -x -n __fish_is_first_arg -l version
complete -c ocrmypdf -x -n __fish_is_first_arg -s h -s "?" -l help
complete -c ocrmypdf -r -l sidecar -d "write OCR to text file"
complete -c ocrmypdf -x -s q -l quiet
complete -c ocrmypdf -s r -l rotate-pages -d "rotate pages to correct orientation"
complete -c ocrmypdf -s d -l deskew -d "fix small horizontal alignment skew"
complete -c ocrmypdf -s c -l clean -d "clean document images before OCR"
complete -c ocrmypdf -s i -l clean-final -d "clean document images and keep result"
complete -c ocrmypdf -l remove-vectors -d "don't send vector objects to OCR"
complete -c ocrmypdf -s f -l force-ocr -d "OCR documents that already have printable text"
complete -c ocrmypdf -s s -l skip-text -d "skip OCR on any pages that already contain text"
complete -c ocrmypdf -l redo-ocr -d "redo OCR on any pages that seem to have OCR already"
complete -c ocrmypdf -l invalidate-digital-signatures -d "invalidate digital signatures and allow OCR to proceed"
complete -c ocrmypdf -s k -l keep-temporary-files -d "keep temporary files (debug)"
function __fish_ocrmypdf_languages
set langs (tesseract --list-langs ^/dev/null)
set arr (string split '\n' $langs)
for lang in $arr[2..-1]
echo $lang
end
end
complete -c ocrmypdf -x -s l -l language -a '(__fish_ocrmypdf_languages)' -d language
complete -c ocrmypdf -x -l image-dpi -d "assume this DPI if input image DPI is unknown"
function __fish_ocrmypdf_output_type
echo -e "pdfa\t"(_ "output a PDF/A (default)")
echo -e "pdf\t"(_ "output a standard PDF")
echo -e "pdfa-1\t"(_ "output a PDF/A-1b")
echo -e "pdfa-2\t"(_ "output a PDF/A-2b")
echo -e "pdfa-3\t"(_ "output a PDF/A-3b")
echo -e "none\t"(_ "do not produce an output PDF (for example, if you only care about --sidecar)")
end
complete -c ocrmypdf -x -l output-type -a '(__fish_ocrmypdf_output_type)' -d "select PDF output options"
function __fish_ocrmypdf_pdf_renderer
echo -e "auto\t"(_ "auto select PDF renderer")
echo -e "hocr\t"(_ "use hOCR renderer")
echo -e "hocrdebug\t"(_ "uses hOCR renderer in debug mode, showing recognized text")
echo -e "sandwich\t"(_ "use sandwich renderer")
end
complete -c ocrmypdf -x -l pdf-renderer -a '(__fish_ocrmypdf_pdf_renderer)' -d "select PDF renderer options"
function __fish_ocrmypdf_optimize
echo -e "0\t"(_ "do not optimize")
echo -e "1\t"(_ "do safe, lossless optimizations (default)")
echo -e "2\t"(_ "do some lossy optimizations")
echo -e "3\t"(_ "do aggressive lossy optimizations (including lossy JBIG2)")
end
complete -c ocrmypdf -x -s O -l optimize -a '(__fish_ocrmypdf_optimize)' -d "select optimization level"
function __fish_ocrmypdf_verbose
echo -e "0\t"(_ "standard output messages")
echo -e "1\t"(_ "troubleshooting output messages")
echo -e "2\t"(_ "debugging output messages")
end
complete -c ocrmypdf -x -s v -l verbose -a '(__fish_ocrmypdf_verbose)' -d "set verbosity level"
complete -c ocrmypdf -x -l no-progress-bar -d "disable the progress bar"
function __fish_ocrmypdf_pdfa_compression
echo -e "auto\t"(_ "let Ghostscript decide how to compress images")
echo -e "jpeg\t"(_ "convert color and grayscale images to JPEG")
echo -e "lossless\t"(_ "convert color and grayscale images to lossless (PNG)")
end
complete -c ocrmypdf -x -l pdfa-image-compression -a '(__fish_ocrmypdf_pdfa_compression)' -d "set PDF/A image compression options"
complete -c ocrmypdf -x -s j -l jobs -d "how many worker processes to use"
complete -c ocrmypdf -x -l title -d "set metadata"
complete -c ocrmypdf -x -l author -d "set metadata"
complete -c ocrmypdf -x -l subject -d "set metadata"
complete -c ocrmypdf -x -l keywords -d "set metadata"
complete -c ocrmypdf -x -l oversample -d "oversample images to this DPI"
complete -c ocrmypdf -x -l skip-big -d "skip OCR on pages larger than this many MPixels"
complete -c ocrmypdf -x -l jpeg-quality -d "JPEG quality [0..100]"
complete -c ocrmypdf -x -l png-quality -d "PNG quality [0..100]"
complete -c ocrmypdf -x -l jbig2-lossy -d "enable lossy JBIG2 (see docs)"
complete -c ocrmypdf -x -l jbig2-threshold -d "JBIG2 compression threshold (see docs)"
complete -c ocrmypdf -x -l max-image-mpixels -d "image decompression bomb threshold"
complete -c ocrmypdf -x -l pages -d "apply OCR to only the specified pages"
complete -c ocrmypdf -x -l tesseract-config -d "set custom tesseract config file"
function __fish_ocrmypdf_tesseract_pagesegmode
echo -e "0\t"(_ "orientation and script detection (OSD) only")
echo -e "1\t"(_ "automatic page segmentation with OSD")
echo -e "2\t"(_ "automatic page segmentation, but no OSD, or OCR")
echo -e "3\t"(_ "fully automatic page segmentation, but no OSD (default)")
echo -e "4\t"(_ "assume a single column of text of variable sizes")
echo -e "5\t"(_ "assume a single uniform block of vertically aligned text")
echo -e "6\t"(_ "assume a single uniform block of text")
echo -e "7\t"(_ "treat the image as a single text line")
echo -e "8\t"(_ "treat the image as a single word")
echo -e "9\t"(_ "treat the image as a single word in a circle")
echo -e "10\t"(_ "treat the image as a single character")
echo -e "11\t"(_ "sparse text - find as much text as possible in no particular order")
echo -e "12\t"(_ "sparse text with OSD")
echo -e "13\t"(_ "raw line - treat the image as a single text line")
end
complete -c ocrmypdf -x -l tesseract-pagesegmode -a '(__fish_ocrmypdf_tesseract_pagesegmode)' -d "set tesseract --psm"
function __fish_ocrmypdf_tesseract_oem
echo -e "0\t"(_ "legacy engine only")
echo -e "1\t"(_ "neural nets LSTM engine only")
echo -e "2\t"(_ "legacy + LSTM engines")
echo -e "3\t"(_ "default, based on what is available")
end
complete -c ocrmypdf -x -l tesseract-oem -a '(__fish_ocrmypdf_tesseract_oem)' -d "set tesseract --oem"
function __fish_ocrmypdf_tesseract_thresholding
echo -e "auto\t"(_ "let OCRmyPDF pick thresholding (current always uses otsu)")
echo -e "otsu\t"(_ "legacy Otsu thresholding")
echo -e "adaptive-otsu\t"(_ "use adaptive Otsu thresholding")
echo -e "sauvola\t"(_ "use Sauvola thresholding")
end
complete -c ocrmypdf -x -l tesseract-thresholding -a '(__fish_ocrmypdf_tesseract_thresholding)' -d "set tesseract thresholding method (needs Tesseract 5.x)"
complete -c ocrmypdf -x -l tesseract-timeout -d "maximum number of seconds to wait for OCR"
complete -c ocrmypdf -x -l rotate-pages-threshold -d "page rotation confidence"
complete -c ocrmypdf -r -l user-words -d "specify location of user words file"
complete -c ocrmypdf -r -l user-patterns -d "specify location of user patterns file"
complete -c ocrmypdf -x -l fast-web-view -d "if file size if above this amount in MB, linearize PDF"
function __fish_ocrmypdf_color_conversion_strategy
echo -e "LeaveColorUnchanged\t"(_ "do not convert color spaces (default)")
echo -e "CMYK\t"(_ "convert all color spaces to CMYK")
echo -e "Gray\t"(_ "convert all color spaces to grayscale")
echo -e "RGB\t"(_ "convert all color spaces to RGB")
echo -e "UseDeviceIndependentColor\t"(_ "convert all color spaces to ICC-based color spaces")
end
complete -c ocrmypdf -x -l color-conversion-strategy -a '(__fish_ocrmypdf_color_conversion_strategy)' -d "set color conversion strategy"
function __fish_ocrmypdf_input_file_given
set -l tokens (commandline -opc)
for token in $tokens
if string match -q -r '^-' -- $token
continue
end
if test -f "$token"
return 0
end
end
return 1
end
complete -c ocrmypdf -x -n 'not __fish_ocrmypdf_input_file_given' -a "(__fish_complete_suffix .pdf)" -d "input file"
+17
View File
@@ -0,0 +1,17 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MIT
---
version: "3.3"
services:
ocrmypdf:
restart: always
container_name: ocrmypdf
image: jbarlow83/ocrmypdf
volumes:
- "/media/scan:/input"
- "/mnt/scan:/output"
environment:
- OCR_OUTPUT_DIRECTORY_YEAR_MONTH=0
user: "<SET TO YOUR USER ID>:<SET TO YOUR GROUP ID>"
entrypoint: python3
command: watcher.py
+68
View File
@@ -0,0 +1,68 @@
# SPDX-FileCopyrightText: 2022 James R Barlow: https://github.com/jbarlow83
# SPDX-License-Identifier: MIT
"""An example of an OCRmyPDF plugin.
This plugin adds two new command line arguments
--grayscale-ocr: converts the image to grayscale before performing OCR on it
(This is occasionally useful for images whose color confounds OCR. It only
affects the image shown to OCR. The image is not saved.)
--mono-page: converts pages all pages in the output file to black and white
To use this from the command line:
ocrmypdf --plugin path/to/example_plugin.py --mono-page input.pdf output.pdf
To use this as an API:
import ocrmypdf
ocrmypdf.ocr('input.pdf', 'output.pdf',
plugins=['path/to/example_plugin.py'], mono_page=True
)
"""
from __future__ import annotations
import logging
from PIL import Image
from ocrmypdf import hookimpl
log = logging.getLogger(__name__)
@hookimpl
def add_options(parser):
parser.add_argument('--grayscale-ocr', action='store_true')
parser.add_argument('--mono-page', action='store_true')
@hookimpl
def prepare(options):
pass
@hookimpl
def validate(pdfinfo, options):
pass
@hookimpl
def filter_ocr_image(page, image):
if page.options.grayscale_ocr:
log.info("graying")
return image.convert('L')
return image
@hookimpl
def filter_page_image(page, image_filename):
if page.options.mono_page:
with Image.open(image_filename) as im:
im = im.convert('1')
im.save(image_filename)
return image_filename
else:
output = image_filename.with_suffix('.jpg')
with Image.open(image_filename) as im:
im.save(output)
return output
+31
View File
@@ -0,0 +1,31 @@
<!-- SPDX-License-Identifier: CC-BY-SA-4.0 -->
To regenerate
=============
Using asciinema and svg-term (`npm install -g svg-term-cli`).
Create `~/.config/asciinema/config` to disable prompt.
```
[record]
command = fish --init-command 'alias fish_prompt="echo \>\ "'
```
Run asciinema
```
asciinema rec new_input.cast
```
Re-record faster version with fewer pauses
```
asciinema rec demo.cast -c "asciinema play new_input.cast --speed 2 --idle-time-limit 0.5"
```
Convert to SVG
```
svg-term --in=misc/screencast/demo.cast --out=misc/screencast/demo.svg --window
```
+65
View File
@@ -0,0 +1,65 @@
{"version": 2, "width": 131, "height": 24, "timestamp": 1687247006, "env": {"SHELL": "/usr/bin/fish", "TERM": "xterm-256color"}}
[0.103649, "o", "\u001b[?2004h\u001b]7; \u0007"]
[0.104223, "o", "\u001b]0;fish \u0007\u001b[30m\u001b(B\u001b[m\r> \u001b[K\r\u001b[C\u001b[C"]
[0.604542, "o", "o\r\u001b[3C\b\u001b[38;2;255;0;0mo\r\u001b[3C\u001b[30m\u001b(B\u001b[m\u001b[38;2;85;85;85mcrmypdf multipage.pdf multipage_with_ocr.pdf\r\u001b[3C\u001b[30m\u001b(B\u001b[m"]
[0.679571, "o", "\u001b[38;2;255;0;0mc\u001b[38;2;85;85;85mrmypdf multipage.pdf multipage_with_ocr.pdf\r\u001b[4C\u001b[30m\u001b(B\u001b[m"]
[0.767271, "o", "\u001b[38;2;255;0;0mr\u001b[38;2;85;85;85mmypdf multipage.pdf multipage_with_ocr.pdf\r\u001b[5C\u001b[30m\u001b(B\u001b[m"]
[0.814505, "o", "\u001b[38;2;255;0;0mm\u001b[38;2;85;85;85mypdf multipage.pdf multipage_with_ocr.pdf\r\u001b[6C\u001b[30m\u001b(B\u001b[m"]
[0.938919, "o", "\u001b[38;2;255;0;0my\u001b[38;2;85;85;85mpdf multipage.pdf multipage_with_ocr.pdf\r\u001b[7C\u001b[30m\u001b(B\u001b[m"]
[0.967347, "o", "\u001b[38;2;255;0;0mp\u001b[38;2;85;85;85mdf multipage.pdf multipage_with_ocr.pdf\r\u001b[8C\u001b[30m\u001b(B\u001b[m"]
[1.009954, "o", "\u001b[38;2;255;0;0md\u001b[38;2;85;85;85mf multipage.pdf multipage_with_ocr.pdf\r\u001b[9C\u001b[30m\u001b(B\u001b[m"]
[1.034488, "o", "\u001b[38;2;255;0;0mf\u001b[38;2;85;85;85m multipage.pdf multipage_with_ocr.pdf\r\u001b[10C\u001b[30m\u001b(B\u001b[m\b\b\b\b\b\b\b\b\u001b[38;2;0;95;215mocrmypdf\u001b[38;2;85;85;85m multipage.pdf multipage_with_ocr.pdf\r\u001b[10C\u001b[30m\u001b(B\u001b[m"]
[1.069226, "o", "\u001b[38;2;0;95;215m \u001b[38;2;85;85;85mmultipage.pdf multipage_with_ocr.pdf\r\u001b[11C\u001b[30m\u001b(B\u001b[m\b \u001b[38;2;85;85;85mmultipage.pdf multipage_with_ocr.pdf\r\u001b[11C\u001b[30m\u001b(B\u001b[m"]
[1.569682, "o", "-\u001b[K\r\u001b[12C\u001b[38;2;85;85;85m-version\r\u001b[12C\u001b[30m\u001b(B\u001b[m\b\u001b[38;2;0;175;255m-\u001b[38;2;85;85;85m-version\r\u001b[12C\u001b[30m\u001b(B\u001b[m"]
[1.642096, "o", "\u001b[38;2;0;175;255m-\u001b[38;2;85;85;85mversion\r\u001b[13C\u001b[30m\u001b(B\u001b[m"]
[1.71793, "o", "\u001b[38;2;0;175;255ms\u001b[30m\u001b(B\u001b[m\u001b[K\r\u001b[14C"]
[1.771483, "o", "\u001b[38;2;0;175;255mk\r\u001b[15C\u001b[30m\u001b(B\u001b[m"]
[1.864664, "o", "\u001b[38;2;0;175;255mi\r\u001b[16C\u001b[30m\u001b(B\u001b[m"]
[1.876085, "o", "\u001b[38;2;0;175;255mp\r\u001b[17C\u001b[30m\u001b(B\u001b[m"]
[2.092979, "o", "\u001b[38;2;0;175;255m-\r\u001b[18C\u001b[30m\u001b(B\u001b[m"]
[2.138821, "o", "\u001b[38;2;0;175;255mt\r\u001b[19C\u001b[30m\u001b(B\u001b[m"]
[2.18017, "o", "\u001b[38;2;0;175;255me\r\u001b[20C\u001b[30m\u001b(B\u001b[m"]
[2.268222, "o", "\u001b[38;2;0;175;255mx\r\u001b[21C\u001b[30m\u001b(B\u001b[m"]
[2.277031, "o", "\u001b[38;2;0;175;255mt\r\u001b[22C\u001b[30m\u001b(B\u001b[m"]
[2.322469, "o", "\u001b[38;2;0;175;255m \r\u001b[23C\u001b[30m\u001b(B\u001b[m\b \r\u001b[23C"]
[2.824696, "o", "m\r\u001b[24C\b\u001b[38;2;0;175;255m\u001b[4mm\r\u001b[24C\u001b[30m\u001b(B\u001b[m\u001b[38;2;85;85;85masks.pdf \r\u001b[24C\u001b[30m\u001b(B\u001b[m"]
[2.923234, "o", "\u001b[38;2;0;175;255m\u001b[4mu\u001b[30m\u001b(B\u001b[m\u001b[K\r\u001b[25C\u001b[38;2;85;85;85mltipage.pdf \r\u001b[25C\u001b[30m\u001b(B\u001b[m"]
[2.960685, "o", "\u001b[38;2;0;175;255m\u001b[4ml\u001b[38;2;85;85;85m\u001b[24mtipage.pdf \r\u001b[26C\u001b[30m\u001b(B\u001b[m"]
[3.03365, "o", "\u001b[38;2;0;175;255m\u001b[4mt\u001b[38;2;85;85;85m\u001b[24mipage.pdf \r\u001b[27C\u001b[30m\u001b(B\u001b[m"]
[3.479338, "o", "\u001b[38;2;0;175;255m\u001b[4mipage.pdf \r\u001b[37C\u001b[30m\u001b(B\u001b[m\b \r\u001b[37C"]
[3.754818, "o", "m\r\u001b[38C\b\u001b[38;2;0;175;255m\u001b[4mm\r\u001b[38C\u001b[30m\u001b(B\u001b[m\u001b[38;2;85;85;85masks.pdf \r\u001b[38C\u001b[30m\u001b(B\u001b[m"]
[3.873318, "o", "\u001b[38;2;0;175;255m\u001b[4mu\u001b[30m\u001b(B\u001b[m\u001b[K\r\u001b[39C\u001b[38;2;85;85;85mltipage.pdf \r\u001b[39C\u001b[30m\u001b(B\u001b[m"]
[3.926829, "o", "\u001b[38;2;0;175;255m\u001b[4ml\u001b[38;2;85;85;85m\u001b[24mtipage.pdf \r\u001b[40C\u001b[30m\u001b(B\u001b[m"]
[4.272251, "o", "\u001b[38;2;0;175;255m\u001b[4mtipage.pdf \r\u001b[51C\u001b[30m\u001b(B\u001b[m\b \r\u001b[51C"]
[4.343464, "o", "\r\u001b[50C"]
[4.416286, "o", "\r\u001b[49C"]
[4.490574, "o", "\r\u001b[48C"]
[4.564115, "o", "\r\u001b[47C"]
[4.630398, "o", "\r\u001b[46C"]
[4.76825, "o", "\u001b[38;2;0;175;255m\u001b[4m_.pd\u001b[30m\u001b(B\u001b[mf \r\u001b[47C\u001b[10D\u001b[38;2;0;175;255mmultipage_.pdf\u001b[30m\u001b(B\u001b[m \r\u001b[47C"]
[5.012506, "o", "\u001b[38;2;0;175;255mo.pd\u001b[30m\u001b(B\u001b[mf \r\u001b[48C\u001b[3C\u001b[38;2;0;175;255mf\u001b[30m\u001b(B\u001b[m \r\u001b[48C"]
[5.053615, "o", "\u001b[38;2;0;175;255mc.pd\u001b[30m\u001b(B\u001b[mf \r\u001b[49C\u001b[3C\u001b[38;2;0;175;255mf\u001b[30m\u001b(B\u001b[m \r\u001b[49C"]
[5.103957, "o", "\u001b[38;2;0;175;255mr.pd\u001b[30m\u001b(B\u001b[mf \r\u001b[50C\u001b[3C\u001b[38;2;0;175;255mf\u001b[30m\u001b(B\u001b[m \r\u001b[50C"]
[5.226183, "o", "\r\u001b[55C"]
[5.728321, "o", "\r\n\u001b[30m\u001b(B\u001b[m\u001b[?2004l\u001b]0;ocrmypdf --skip-text multipage.pdf multipage_ocr.pdf /home/jb/src/ocrmypdf/tests/resources\u0007\u001b[30m\u001b(B\u001b[m\r"]
[5.801032, "o", "\rScanning contents: 0%| | 0/6 [00:00<?, ?page/s]"]
[5.802664, "o", "\rScanning contents: 100%|█████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 1270.68page/s]\r\n"]
[5.802747, "o", "Start processing 6 pages concurrently\r\n"]
[5.803488, "o", "\rOCR: 0%| | 0.0/6.0 [00:00<?, ?page/s]"]
[5.804896, "o", "\r \r 4 skipping all processing on this page\r\n\rOCR: 0%| | 0.0/6.0 [00:00<?, ?page/s]"]
[5.896969, "o", "\rOCR: 25%|█████████████████████▎ | 1.5/6.0 [00:00<00:00, 8.12page/s]"]
[6.170021, "o", "\rOCR: 42%|███████████████████████████████████▍ | 2.5/6.0 [00:00<00:01, 3.05page/s]"]
[6.292338, "o", "\rOCR: 58%|█████████████████████████████████████████████████▌ | 3.5/6.0 [00:00<00:00, 3.39page/s]"]
[6.586017, "o", "\rOCR: 75%|███████████████████████████████████████████████████████████████▊ | 4.5/6.0 [00:01<00:00, 2.49page/s]"]
[7.087058, "o", "\rOCR: 92%|█████████████████████████████████████████████████████████████████████████████▉ | 5.5/6.0 [00:06<00:00, 1.98s/page]\rOCR: 100%|█████████████████████████████████████████████████████████████████████████████████████| 6.0/6.0 [00:06<00:00, 1.09s/page]\r\nPostprocessing...\r\n"]
[7.104927, "o", "\rPDF/A conversion: 0%| | 0/6 [00:00<?, ?page/s]"]
[7.607392, "o", "\rPDF/A conversion: 50%|██████████████████████████████████████ | 3/6 [00:01<00:01, 1.61page/s]"]
[7.653781, "o", "\rPDF/A conversion: 83%|███████████████████████████████████████████████████████████████▎ | 5/6 [00:01<00:00, 2.90page/s]"]
[7.774532, "o", "\rPDF/A conversion: 100%|████████████████████████████████████████████████████████████████████████████| 6/6 [00:02<00:00, 2.71page/s]\r\n"]
[7.778252, "o", "\u001b[33mSome input metadata could not be copied because it is not permitted in PDF/A. You may wish to examine the output PDF's XMP metadata.\u001b[0m\r\n"]
[8.280789, "o", "\rRecompressing JPEGs: 0image [00:00, ?image/s]\rRecompressing JPEGs: 0image [00:00, ?image/s]\r\n\rDeflating JPEGs: 0%| | 0/4 [00:00<?, ?image/s]\rDeflating JPEGs: 100%|███████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 238.28image/s]\r\n"]
[8.28149, "o", "\rJBIG2: 0item [00:00, ?item/s]\rJBIG2: 0item [00:00, ?item/s]\r\n"]
[8.289998, "o", "Image optimization ratio: 1.01 savings: 1.3%\r\nTotal file size ratio: 1.02 savings: 1.6%\r\n"]
[8.291209, "o", "Output file is a PDF/A-2B (as expected)\r\n"]
[8.361316, "o", "\u001b[2m⏎\u001b(B\u001b[m \r⏎ \r\u001b[K\u001b[?2004h\u001b]0;fish /home/jb/src/ocrmypdf/tests/resources\u0007\u001b[30m\u001b(B\u001b[m> \u001b[K\r\u001b[C\u001b[C"]
[8.862206, "o", "\r\n\u001b[30m\u001b(B\u001b[m\u001b[30m\u001b(B\u001b[m\u001b[?2004l"]
File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 29 KiB

+76
View File
@@ -0,0 +1,76 @@
#!/bin/env python3
# SPDX-FileCopyrightText: 2017 Enantiomerie
# SPDX-License-Identifier: MIT
"""Example OCRmyPDF for Synology NAS."""
from __future__ import annotations
# This script must be edited to meet your needs.
import logging
import os
import shutil
import subprocess
import sys
import time
# pylint: disable=logging-format-interpolation
# pylint: disable=logging-not-lazy
script_dir = os.path.dirname(os.path.realpath(__file__))
timestamp = time.strftime("%Y-%m-%d-%H%M_")
log_file = script_dir + '/' + timestamp + 'ocrmypdf.log'
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s %(message)s',
filename=log_file,
filemode='w',
)
start_dir = sys.argv[1] if len(sys.argv) > 1 else '.'
for dir_name, _subdirs, file_list in os.walk(start_dir):
logging.info(dir_name)
os.chdir(dir_name)
for filename in file_list:
file_stem, file_ext = os.path.splitext(filename)
if file_ext != '.pdf':
continue
full_path = os.path.join(dir_name, filename)
timestamp_ocr = time.strftime("%Y-%m-%d-%H%M_OCR_")
filename_ocr = timestamp_ocr + file_stem + '.pdf'
# create string for pdf processing
# the script is processed as root user via chron
cmd = [
'docker',
'run',
'--rm',
'-i',
'jbarlow83/ocrmypdf',
'--deskew',
'-',
'-',
]
logging.info(cmd)
full_path_ocr = os.path.join(dir_name, filename_ocr)
with (
open(filename, 'rb') as input_file,
open(full_path_ocr, 'wb') as output_file,
):
proc = subprocess.run(
cmd,
stdin=input_file,
stdout=output_file,
stderr=subprocess.PIPE,
check=False,
text=True,
errors='ignore',
)
logging.info(proc.stderr)
os.chmod(full_path_ocr, 0o664)
os.chmod(full_path, 0o664)
full_path_ocr_archive = sys.argv[2]
full_path_archive = sys.argv[2] + '/no_ocr'
shutil.move(full_path_ocr, full_path_ocr_archive)
shutil.move(full_path, full_path_archive)
logging.info('Finished.\n')
+336
View File
@@ -0,0 +1,336 @@
#!/usr/bin/env python3
# SPDX-FileCopyrightText: 2019 Ian Alexander <https://github.com/ianalexander>
# SPDX-FileCopyrightText: 2020 James R Barlow <https://github.com/jbarlow83>
# SPDX-License-Identifier: MIT
"""Watch a directory for new PDFs and OCR them."""
# Do not enable annotations!
# https://github.com/tiangolo/typer/discussions/598
import json
import logging
import shutil
import sys
import time
from datetime import datetime
from enum import Enum
from pathlib import Path
from typing import Annotated, Any
import pikepdf
import typer
from dotenv import load_dotenv
from watchdog.events import PatternMatchingEventHandler
from watchdog.observers import Observer
from watchdog.observers.polling import PollingObserver
import ocrmypdf
load_dotenv()
# pylint: disable=logging-format-interpolation
app = typer.Typer(name="ocrmypdf-watcher")
log = logging.getLogger('ocrmypdf-watcher')
class LoggingLevelEnum(str, Enum):
"""Enum for logging levels."""
DEBUG = "DEBUG"
INFO = "INFO"
WARNING = "WARNING"
ERROR = "ERROR"
CRITICAL = "CRITICAL"
def get_output_path(root: Path, basename: str, output_dir_year_month: bool) -> Path:
assert '/' not in basename, "basename must not contain '/'"
if output_dir_year_month:
today = datetime.today()
output_directory_year_month = root / str(today.year) / f'{today.month:02d}'
if not output_directory_year_month.exists():
output_directory_year_month.mkdir(parents=True, exist_ok=True)
output_path = Path(output_directory_year_month) / Path(basename).with_suffix(
'.pdf'
)
else:
output_path = root / Path(basename).with_suffix('.pdf')
return output_path
def wait_for_file_ready(
file_path: Path, poll_new_file_seconds: int, retries_loading_file: int
):
# This loop waits to make sure that the file is completely loaded on
# disk before attempting to read. Docker sometimes will publish the
# watchdog event before the file is actually fully on disk, causing
# pikepdf to fail.
tries = retries_loading_file + 1
while tries:
try:
with pikepdf.Pdf.open(file_path) as pdf:
log.debug(f"{file_path} ready with {pdf.pages} pages")
return True
except (FileNotFoundError, OSError) as e:
log.info(f"File {file_path} is not ready yet")
log.debug("Exception was", exc_info=e)
time.sleep(poll_new_file_seconds)
tries -= 1
except pikepdf.PdfError as e:
log.info(f"File {file_path} is not full written yet")
log.debug("Exception was", exc_info=e)
time.sleep(poll_new_file_seconds)
tries -= 1
return False
def execute_ocrmypdf(
*,
file_path: Path,
archive_dir: Path,
output_dir: Path,
ocrmypdf_kwargs: dict[str, Any],
on_success_delete: bool,
on_success_archive: bool,
poll_new_file_seconds: int,
retries_loading_file: int,
output_dir_year_month: bool,
):
output_path = get_output_path(output_dir, file_path.name, output_dir_year_month)
log.info("-" * 20)
log.info(f'New file: {file_path}. Waiting until fully written...')
if not wait_for_file_ready(file_path, poll_new_file_seconds, retries_loading_file):
log.info(f"Gave up waiting for {file_path} to become ready")
return
log.info(f'Attempting to OCRmyPDF to: {output_path}')
log.debug(
f'OCRmyPDF input_file={file_path} output_file={output_path} '
f'kwargs: {ocrmypdf_kwargs}'
)
exit_code = ocrmypdf.ocr(
input_file=file_path,
output_file=output_path,
**ocrmypdf_kwargs,
)
if exit_code == 0:
if on_success_delete:
log.info(f'OCR is done. Deleting: {file_path}')
file_path.unlink()
elif on_success_archive:
log.info(f'OCR is done. Archiving {file_path.name} to {archive_dir}')
shutil.move(file_path, f'{archive_dir}/{file_path.name}')
else:
log.info('OCR is done')
else:
log.info('OCR is done')
class HandleObserverEvent(PatternMatchingEventHandler):
def __init__( # noqa: D107
self,
patterns=None,
ignore_patterns=None,
ignore_directories=False,
case_sensitive=False,
settings={},
):
super().__init__(
patterns=patterns,
ignore_patterns=ignore_patterns,
ignore_directories=ignore_directories,
case_sensitive=case_sensitive,
)
self._settings = settings
def on_any_event(self, event):
if event.event_type in ['created']:
execute_ocrmypdf(file_path=Path(event.src_path), **self._settings)
@app.command()
def main(
input_dir: Annotated[
Path,
typer.Argument(
envvar='OCR_INPUT_DIRECTORY',
exists=True,
file_okay=False,
dir_okay=True,
readable=True,
resolve_path=True,
),
] = '/input',
output_dir: Annotated[
Path,
typer.Argument(
envvar='OCR_OUTPUT_DIRECTORY',
exists=True,
file_okay=False,
dir_okay=True,
writable=True,
resolve_path=True,
),
] = '/output',
archive_dir: Annotated[
Path,
typer.Argument(
envvar='OCR_ARCHIVE_DIRECTORY',
exists=True,
file_okay=False,
dir_okay=True,
writable=True,
resolve_path=True,
),
] = '/processed',
output_dir_year_month: Annotated[
bool,
typer.Option(
envvar='OCR_OUTPUT_DIRECTORY_YEAR_MONTH',
help='Create a subdirectory in the output directory for each year/month',
),
] = False,
on_success_delete: Annotated[
bool,
typer.Option(
envvar='OCR_ON_SUCCESS_DELETE',
help='Delete the input file after successful OCR',
),
] = False,
on_success_archive: Annotated[
bool,
typer.Option(
envvar='OCR_ON_SUCCESS_ARCHIVE',
help='Archive the input file after successful OCR',
),
] = False,
deskew: Annotated[
bool,
typer.Option(
envvar='OCR_DESKEW',
help='Deskew the input file before OCR',
),
] = False,
ocr_json_settings: Annotated[
str,
typer.Option(
envvar='OCR_JSON_SETTINGS',
help='JSON settings to pass to OCRmyPDF (JSON string or file path)',
),
] = None,
poll_new_file_seconds: Annotated[
int,
typer.Option(
envvar='OCR_POLL_NEW_FILE_SECONDS',
help='Seconds to wait before polling a new file',
min=0,
),
] = 1,
use_polling: Annotated[
bool,
typer.Option(
envvar='OCR_USE_POLLING',
help='Use polling instead of filesystem events',
),
] = False,
retries_loading_file: Annotated[
int,
typer.Option(
envvar='OCR_RETRIES_LOADING_FILE',
help='Number of times to retry loading a file before giving up',
min=0,
),
] = 5,
loglevel: Annotated[
LoggingLevelEnum,
typer.Option(
envvar='OCR_LOGLEVEL',
help='Logging level',
),
] = LoggingLevelEnum.INFO,
patterns: Annotated[
str,
typer.Option(
envvar='OCR_PATTERNS',
help='File patterns to watch',
),
] = '*.pdf,*.PDF',
):
ocrmypdf.configure_logging(
verbosity=(
ocrmypdf.Verbosity.default
if loglevel != LoggingLevelEnum.DEBUG
else ocrmypdf.Verbosity.debug
),
manage_root_logger=True,
)
log.setLevel(loglevel.value)
log.info(
f"Starting OCRmyPDF watcher with config:\n"
f"Input Directory: {input_dir}\n"
f"Output Directory: {output_dir}\n"
f"Output Directory Year & Month: {output_dir_year_month}\n"
f"Archive Directory: {archive_dir}"
)
log.debug(
f"INPUT_DIRECTORY: {input_dir}\n"
f"OUTPUT_DIRECTORY: {output_dir}\n"
f"OUTPUT_DIRECTORY_YEAR_MONTH: {output_dir_year_month}\n"
f"ARCHIVE_DIRECTORY: {archive_dir}\n"
f"ON_SUCCESS_DELETE: {on_success_delete}\n"
f"ON_SUCCESS_ARCHIVE: {on_success_archive}\n"
f"DESKEW: {deskew}\n"
f"ARGS: {ocr_json_settings}\n"
f"POLL_NEW_FILE_SECONDS: {poll_new_file_seconds}\n"
f"RETRIES_LOADING_FILE: {retries_loading_file}\n"
f"USE_POLLING: {use_polling}\n"
f"LOGLEVEL: {loglevel.value}"
)
if ocr_json_settings and Path(ocr_json_settings).exists():
json_settings = json.loads(Path(ocr_json_settings).read_text())
else:
json_settings = json.loads(ocr_json_settings or '{}')
if 'input_file' in json_settings or 'output_file' in json_settings:
log.error(
'OCR_JSON_SETTINGS (--ocr-json-settings) may not specify input/output file'
)
sys.exit(1)
handler = HandleObserverEvent(
patterns=patterns.split(','),
settings={
'archive_dir': archive_dir,
'output_dir': output_dir,
'ocrmypdf_kwargs': json_settings | {'deskew': deskew},
'on_success_delete': on_success_delete,
'on_success_archive': on_success_archive,
'poll_new_file_seconds': poll_new_file_seconds,
'retries_loading_file': retries_loading_file,
'output_dir_year_month': output_dir_year_month,
},
)
if use_polling:
observer = PollingObserver()
else:
observer = Observer()
observer.schedule(handler, input_dir, recursive=True)
observer.start()
typer.echo(f"Watching {input_dir} for new PDFs. Press Ctrl+C to exit.")
try:
while True:
time.sleep(30)
except KeyboardInterrupt:
observer.stop()
observer.join()
if __name__ == "__main__":
app()
+107
View File
@@ -0,0 +1,107 @@
#!/usr/bin/env python3
# SPDX-FileCopyrightText: 2019 James R. Barlow
# SPDX-License-Identifier: AGPL-3.0-or-later
"""This is a simple web service/HTTP wrapper for OCRmyPDF.
This may be more convenient than the command line tool for some Docker users.
Note that OCRmyPDF uses Ghostscript, which is licensed under AGPLv3+. While
OCRmyPDF is under GPLv3, this file is distributed under the Affero GPLv3+ license,
to emphasize that SaaS deployments should make sure they comply with
Ghostscript's license as well as OCRmyPDF's.
"""
from __future__ import annotations
import os
import shlex
from subprocess import run
from tempfile import TemporaryDirectory
from flask import Flask, Response, request, send_from_directory
from werkzeug.utils import secure_filename
app = Flask(__name__)
app.secret_key = "secret"
app.config['MAX_CONTENT_LENGTH'] = 50_000_000
app.config.from_envvar("OCRMYPDF_WEBSERVICE_SETTINGS", silent=True)
ALLOWED_EXTENSIONS = {"pdf"}
def allowed_file(filename):
return "." in filename and filename.rsplit(".", 1)[1].lower() in ALLOWED_EXTENSIONS
def do_ocrmypdf(file):
uploaddir = TemporaryDirectory(prefix="ocrmypdf-upload")
downloaddir = TemporaryDirectory(prefix="ocrmypdf-download")
filename = secure_filename(file.filename)
up_file = os.path.join(uploaddir.name, filename)
file.save(up_file)
down_file = os.path.join(downloaddir.name, filename)
cmd_args = [arg for arg in shlex.split(request.form["params"])]
if "--sidecar" in cmd_args:
return Response("--sidecar not supported", 501, mimetype='text/plain')
ocrmypdf_args = ["ocrmypdf", *cmd_args, up_file, down_file]
proc = run(ocrmypdf_args, capture_output=True, encoding="utf-8", check=False)
if proc.returncode != 0:
stderr = proc.stderr
return Response(stderr, 400, mimetype='text/plain')
return send_from_directory(downloaddir.name, filename)
@app.route("/", methods=["GET", "POST"])
def upload_file():
if request.method == "POST":
if "file" not in request.files:
return Response("No file in POST", 400, mimetype='text/plain')
file = request.files["file"]
if file.filename == "":
return Response("Empty filename", 400, mimetype='text/plain')
if not allowed_file(file.filename):
return Response("Invalid filename", 400, mimetype='text/plain')
if file and allowed_file(file.filename):
return do_ocrmypdf(file)
return Response("Some other problem", 400, mimetype='text/plain')
return """
<!doctype html>
<title>OCRmyPDF webservice</title>
<h1>Upload a PDF (debug UI)</h1>
<form method=post enctype=multipart/form-data>
<label for="args">Command line parameters</label>
<input type=textbox name=params>
<label for="file">File to upload</label>
<input type=file name=file>
<input type=submit value=Upload>
</form>
<h4>Notice</h2>
<div style="font-size: 70%; max-width: 34em;">
<p>This is a webservice wrapper for OCRmyPDF.</p>
<p>Copyright 2019 James R. Barlow</p>
<p>This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
</p>
<p>This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
</p>
<p>
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see &lt;http://www.gnu.org/licenses/&gt;.
</p>
</div>
"""
if __name__ == "__main__":
app.run(host='0.0.0.0', port=5000)
-52
View File
@@ -1,52 +0,0 @@
#!/usr/bin/env python3
# © 2015 James R. Barlow: github.com/jbarlow83
from tempfile import NamedTemporaryFile
from subprocess import Popen, PIPE, check_call
from shutil import copy
def rasterize_pdf(input_file, output_file, xres, yres, raster_device, log):
with NamedTemporaryFile(delete=True) as tmp:
args_gs = [
'gs',
'-dBATCH', '-dNOPAUSE',
'-sDEVICE=%s' % raster_device,
'-o', tmp.name,
'-r{0}x{1}'.format(str(xres), str(yres)),
input_file
]
p = Popen(args_gs, close_fds=True, stdout=PIPE, stderr=PIPE,
universal_newlines=True)
stdout, stderr = p.communicate()
if stdout:
log.debug(stdout)
if stderr:
log.error(stderr)
if p.returncode == 0:
copy(tmp.name, output_file)
else:
log.error('Ghostscript rendering failed')
def generate_pdfa(pdf_pages, output_file, threads=1):
with NamedTemporaryFile(delete=True) as gs_pdf:
args_gs = [
"gs",
"-dQUIET",
"-dBATCH",
"-dNOPAUSE",
'-dNumRenderingThreads=' + str(threads),
"-sDEVICE=pdfwrite",
"-sColorConversionStrategy=/RGB",
"-sProcessColorModel=DeviceRGB",
"-dPDFA",
"-sPDFACompatibilityPolicy=2",
"-sOutputICCProfile=srgb.icc",
"-sOutputFile=" + gs_pdf.name,
]
args_gs.extend(pdf_pages)
check_call(args_gs)
copy(gs_pdf.name, output_file)
-231
View File
@@ -1,231 +0,0 @@
#!/usr/local/bin/python3
##############################################################################
# Copyright (c) 2013-14: fritz-hh from Github
# (https://github.com/fritz-hh)
#
# Copyright (c) 2010: Jonathan Brinley from Github
# (https://github.com/jbrinley/HocrConverter)
# Initial version by Jonathan Brinley, jonathanbrinley@gmail.com
##############################################################################
from reportlab.pdfgen.canvas import Canvas
from reportlab.lib.units import inch
from lxml import etree as ElementTree
from PIL import Image
from collections import namedtuple
import re
import argparse
Rect = namedtuple('Rect', ['x1', 'y1', 'x2', 'y2'])
class HocrTransformError(Exception):
pass
class HocrTransform():
"""
A class for converting documents from the hOCR format.
For details of the hOCR format, see:
http://docs.google.com/View?docid=dfxcv4vc_67g844kf
"""
def __init__(self, hocrFileName, dpi):
self.dpi = dpi
self.boxPattern = re.compile(r'bbox((\s+\d+){4})')
self.hocr = ElementTree.ElementTree()
self.hocr.parse(hocrFileName)
# if the hOCR file has a namespace, ElementTree requires its use to
# find elements
matches = re.match(r'({.*})html', self.hocr.getroot().tag)
self.xmlns = ''
if matches:
self.xmlns = matches.group(1)
# get dimension in pt (not pixel!!!!) of the OCRed image
self.width, self.height = None, None
for div in self.hocr.findall(
".//%sdiv[@class='ocr_page']" % (self.xmlns)):
coords = self.element_coordinates(div)
pt_coords = self.pt_from_pixel(coords)
self.width = pt_coords.x2 - pt_coords.x1
self.height = pt_coords.y2 - pt_coords.y1
# there shouldn't be more than one, and if there is, we don't want
# it
break
if self.width is None or self.height is None:
raise HocrTransformError("hocr file is missing page dimensions")
def __str__(self):
"""
Return the textual content of the HTML body
"""
if self.hocr is None:
return ''
body = self.hocr.find(".//%sbody" % (self.xmlns))
if body:
return self._get_element_text(body)
else:
return ''
def _get_element_text(self, element):
"""
Return the textual content of the element and its children
"""
text = ''
if element.text is not None:
text += element.text
for child in element.getchildren():
text += self._get_element_text(child)
if element.tail is not None:
text += element.tail
return text
def element_coordinates(self, element):
"""
Returns a tuple containing the coordinates of the bounding box around
an element
"""
out = (0, 0, 0, 0)
if 'title' in element.attrib:
matches = self.boxPattern.search(element.attrib['title'])
if matches:
coords = matches.group(1).split()
out = Rect._make(int(coords[n]) for n in range(4))
return out
def pt_from_pixel(self, pxl):
"""
Returns the quantity in PDF units (pt) given quantity in pixels
"""
return Rect._make(
(c / self.dpi * inch) for c in pxl)
def replace_unsupported_chars(self, s):
"""
Given an input string, returns the corresponding string that:
- is available in the helvetica facetype
- does not contain any ligature (to allow easy search in the PDF file)
"""
# The 'u' before the character to replace indicates that it is a
# unicode character
s = s.replace(u"", "fl")
s = s.replace(u"", "fi")
return s
def to_pdf(self, outFileName, imageFileName=None, showBoundingboxes=False,
fontname="Helvetica", invisibleText=False):
"""
Creates a PDF file with an image superimposed on top of the text.
Text is positioned according to the bounding box of the lines in
the hOCR file.
The image need not be identical to the image used to create the hOCR
file.
It can have a lower resolution, different color mode, etc.
"""
# create the PDF file
# page size in points (1/72 in.)
pdf = Canvas(
outFileName, pagesize=(self.width, self.height), pageCompression=1)
# draw bounding box for each paragraph
# light blue for bounding box of paragraph
pdf.setStrokeColorRGB(0, 1, 1)
# light blue for bounding box of paragraph
pdf.setFillColorRGB(0, 1, 1)
pdf.setLineWidth(0) # no line for bounding box
for elem in self.hocr.findall(
".//%sp[@class='%s']" % (self.xmlns, "ocr_par")):
elemtxt = self._get_element_text(elem).rstrip()
if len(elemtxt) == 0:
continue
pxl_coords = self.element_coordinates(elem)
pt = self.pt_from_pixel(pxl_coords)
# draw the bbox border
if showBoundingboxes:
pdf.rect(
pt.x1, self.height - pt.y2, pt.x2 - pt.x1, pt.y2 - pt.y1,
fill=1)
# check if element with class 'ocrx_word' are available
# otherwise use 'ocr_line' as fallback
elemclass = "ocr_line"
if self.hocr.find(
".//%sspan[@class='ocrx_word']" % (self.xmlns)) is not None:
elemclass = "ocrx_word"
# itterate all text elements
# light green for bounding box of word/line
pdf.setStrokeColorRGB(1, 0, 0)
pdf.setLineWidth(0.5) # bounding box line width
pdf.setDash(6, 3) # bounding box is dashed
pdf.setFillColorRGB(0, 0, 0) # text in black
for elem in self.hocr.findall(
".//%sspan[@class='%s']" % (self.xmlns, elemclass)):
elemtxt = self._get_element_text(elem).rstrip()
elemtxt = self.replace_unsupported_chars(elemtxt)
if len(elemtxt) == 0:
continue
pxl_coords = self.element_coordinates(elem)
pt = self.pt_from_pixel(pxl_coords)
# draw the bbox border
if showBoundingboxes:
pdf.rect(
pt.x1, self.height - pt.y2, pt.x2 - pt.x1, pt.y2 - pt.y1,
fill=0)
text = pdf.beginText()
fontsize = pt.y2 - pt.y1
text.setFont(fontname, fontsize)
if invisibleText:
text.setTextRenderMode(3) # Invisible (indicates OCR text)
# set cursor to bottom left corner of bbox (adjust for dpi)
text.setTextOrigin(pt.x1, self.height - pt.y2)
# scale the width of the text to fill the width of the bbox
text.setHorizScale(
100 * (pt.x2 - pt.x1) / pdf.stringWidth(
elemtxt, fontname, fontsize))
# write the text to the page
text.textLine(elemtxt)
pdf.drawText(text)
# put the image on the page, scaled to fill the page
if imageFileName is not None:
pdf.drawImage(imageFileName, 0, 0,
width=self.width, height=self.height)
# finish up the page and save it
pdf.showPage()
pdf.save()
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Convert hocr file to PDF')
parser.add_argument('-b', '--boundingboxes', action="store_true",
default=False, help='Show bounding boxes borders')
parser.add_argument('-r', '--resolution', type=int,
default=300,
help='Resolution of the image that was OCRed')
parser.add_argument('-i', '--image', default=None,
help='Path to the image to be placed above the text')
parser.add_argument('hocrfile', help='Path to the hocr file to be parsed')
parser.add_argument(
'outputfile', help='Path to the PDF file to be generated')
args = parser.parse_args()
hocr = HocrTransform(args.hocrfile, args.resolution)
hocr.to_pdf(args.outputfile, args.image, args.boundingboxes)
-502
View File
@@ -1,502 +0,0 @@
GNU LESSER GENERAL PUBLIC LICENSE
Version 2.1, February 1999
Copyright (C) 1991, 1999 Free Software Foundation, Inc.
59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
[This is the first released version of the Lesser GPL. It also counts
as the successor of the GNU Library Public License, version 2, hence
the version number 2.1.]
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
Licenses are intended to guarantee your freedom to share and change
free software--to make sure the software is free for all its users.
This license, the Lesser General Public License, applies to some
specially designated software packages--typically libraries--of the
Free Software Foundation and other authors who decide to use it. You
can use it too, but we suggest you first think carefully about whether
this license or the ordinary General Public License is the better
strategy to use in any particular case, based on the explanations below.
When we speak of free software, we are referring to freedom of use,
not price. Our General Public Licenses are designed to make sure that
you have the freedom to distribute copies of free software (and charge
for this service if you wish); that you receive source code or can get
it if you want it; that you can change the software and use pieces of
it in new free programs; and that you are informed that you can do
these things.
To protect your rights, we need to make restrictions that forbid
distributors to deny you these rights or to ask you to surrender these
rights. These restrictions translate to certain responsibilities for
you if you distribute copies of the library or if you modify it.
For example, if you distribute copies of the library, whether gratis
or for a fee, you must give the recipients all the rights that we gave
you. You must make sure that they, too, receive or can get the source
code. If you link other code with the library, you must provide
complete object files to the recipients, so that they can relink them
with the library after making changes to the library and recompiling
it. And you must show them these terms so they know their rights.
We protect your rights with a two-step method: (1) we copyright the
library, and (2) we offer you this license, which gives you legal
permission to copy, distribute and/or modify the library.
To protect each distributor, we want to make it very clear that
there is no warranty for the free library. Also, if the library is
modified by someone else and passed on, the recipients should know
that what they have is not the original version, so that the original
author's reputation will not be affected by problems that might be
introduced by others.
Finally, software patents pose a constant threat to the existence of
any free program. We wish to make sure that a company cannot
effectively restrict the users of a free program by obtaining a
restrictive license from a patent holder. Therefore, we insist that
any patent license obtained for a version of the library must be
consistent with the full freedom of use specified in this license.
Most GNU software, including some libraries, is covered by the
ordinary GNU General Public License. This license, the GNU Lesser
General Public License, applies to certain designated libraries, and
is quite different from the ordinary General Public License. We use
this license for certain libraries in order to permit linking those
libraries into non-free programs.
When a program is linked with a library, whether statically or using
a shared library, the combination of the two is legally speaking a
combined work, a derivative of the original library. The ordinary
General Public License therefore permits such linking only if the
entire combination fits its criteria of freedom. The Lesser General
Public License permits more lax criteria for linking other code with
the library.
We call this license the "Lesser" General Public License because it
does Less to protect the user's freedom than the ordinary General
Public License. It also provides other free software developers Less
of an advantage over competing non-free programs. These disadvantages
are the reason we use the ordinary General Public License for many
libraries. However, the Lesser license provides advantages in certain
special circumstances.
For example, on rare occasions, there may be a special need to
encourage the widest possible use of a certain library, so that it becomes
a de-facto standard. To achieve this, non-free programs must be
allowed to use the library. A more frequent case is that a free
library does the same job as widely used non-free libraries. In this
case, there is little to gain by limiting the free library to free
software only, so we use the Lesser General Public License.
In other cases, permission to use a particular library in non-free
programs enables a greater number of people to use a large body of
free software. For example, permission to use the GNU C Library in
non-free programs enables many more people to use the whole GNU
operating system, as well as its variant, the GNU/Linux operating
system.
Although the Lesser General Public License is Less protective of the
users' freedom, it does ensure that the user of a program that is
linked with the Library has the freedom and the wherewithal to run
that program using a modified version of the Library.
The precise terms and conditions for copying, distribution and
modification follow. Pay close attention to the difference between a
"work based on the library" and a "work that uses the library". The
former contains code derived from the library, whereas the latter must
be combined with the library in order to run.
GNU LESSER GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License Agreement applies to any software library or other
program which contains a notice placed by the copyright holder or
other authorized party saying it may be distributed under the terms of
this Lesser General Public License (also called "this License").
Each licensee is addressed as "you".
A "library" means a collection of software functions and/or data
prepared so as to be conveniently linked with application programs
(which use some of those functions and data) to form executables.
The "Library", below, refers to any such software library or work
which has been distributed under these terms. A "work based on the
Library" means either the Library or any derivative work under
copyright law: that is to say, a work containing the Library or a
portion of it, either verbatim or with modifications and/or translated
straightforwardly into another language. (Hereinafter, translation is
included without limitation in the term "modification".)
"Source code" for a work means the preferred form of the work for
making modifications to it. For a library, complete source code means
all the source code for all modules it contains, plus any associated
interface definition files, plus the scripts used to control compilation
and installation of the library.
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running a program using the Library is not restricted, and output from
such a program is covered only if its contents constitute a work based
on the Library (independent of the use of the Library in a tool for
writing it). Whether that is true depends on what the Library does
and what the program that uses the Library does.
1. You may copy and distribute verbatim copies of the Library's
complete source code as you receive it, in any medium, provided that
you conspicuously and appropriately publish on each copy an
appropriate copyright notice and disclaimer of warranty; keep intact
all the notices that refer to this License and to the absence of any
warranty; and distribute a copy of this License along with the
Library.
You may charge a fee for the physical act of transferring a copy,
and you may at your option offer warranty protection in exchange for a
fee.
2. You may modify your copy or copies of the Library or any portion
of it, thus forming a work based on the Library, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) The modified work must itself be a software library.
b) You must cause the files modified to carry prominent notices
stating that you changed the files and the date of any change.
c) You must cause the whole of the work to be licensed at no
charge to all third parties under the terms of this License.
d) If a facility in the modified Library refers to a function or a
table of data to be supplied by an application program that uses
the facility, other than as an argument passed when the facility
is invoked, then you must make a good faith effort to ensure that,
in the event an application does not supply such function or
table, the facility still operates, and performs whatever part of
its purpose remains meaningful.
(For example, a function in a library to compute square roots has
a purpose that is entirely well-defined independent of the
application. Therefore, Subsection 2d requires that any
application-supplied function or table used by this function must
be optional: if the application does not supply it, the square
root function must still compute square roots.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Library,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Library, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote
it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Library.
In addition, mere aggregation of another work not based on the Library
with the Library (or with a work based on the Library) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may opt to apply the terms of the ordinary GNU General Public
License instead of this License to a given copy of the Library. To do
this, you must alter all the notices that refer to this License, so
that they refer to the ordinary GNU General Public License, version 2,
instead of to this License. (If a newer version than version 2 of the
ordinary GNU General Public License has appeared, then you can specify
that version instead if you wish.) Do not make any other change in
these notices.
Once this change is made in a given copy, it is irreversible for
that copy, so the ordinary GNU General Public License applies to all
subsequent copies and derivative works made from that copy.
This option is useful when you wish to copy part of the code of
the Library into a program that is not a library.
4. You may copy and distribute the Library (or a portion or
derivative of it, under Section 2) in object code or executable form
under the terms of Sections 1 and 2 above provided that you accompany
it with the complete corresponding machine-readable source code, which
must be distributed under the terms of Sections 1 and 2 above on a
medium customarily used for software interchange.
If distribution of object code is made by offering access to copy
from a designated place, then offering equivalent access to copy the
source code from the same place satisfies the requirement to
distribute the source code, even though third parties are not
compelled to copy the source along with the object code.
5. A program that contains no derivative of any portion of the
Library, but is designed to work with the Library by being compiled or
linked with it, is called a "work that uses the Library". Such a
work, in isolation, is not a derivative work of the Library, and
therefore falls outside the scope of this License.
However, linking a "work that uses the Library" with the Library
creates an executable that is a derivative of the Library (because it
contains portions of the Library), rather than a "work that uses the
library". The executable is therefore covered by this License.
Section 6 states terms for distribution of such executables.
When a "work that uses the Library" uses material from a header file
that is part of the Library, the object code for the work may be a
derivative work of the Library even though the source code is not.
Whether this is true is especially significant if the work can be
linked without the Library, or if the work is itself a library. The
threshold for this to be true is not precisely defined by law.
If such an object file uses only numerical parameters, data
structure layouts and accessors, and small macros and small inline
functions (ten lines or less in length), then the use of the object
file is unrestricted, regardless of whether it is legally a derivative
work. (Executables containing this object code plus portions of the
Library will still fall under Section 6.)
Otherwise, if the work is a derivative of the Library, you may
distribute the object code for the work under the terms of Section 6.
Any executables containing that work also fall under Section 6,
whether or not they are linked directly with the Library itself.
6. As an exception to the Sections above, you may also combine or
link a "work that uses the Library" with the Library to produce a
work containing portions of the Library, and distribute that work
under terms of your choice, provided that the terms permit
modification of the work for the customer's own use and reverse
engineering for debugging such modifications.
You must give prominent notice with each copy of the work that the
Library is used in it and that the Library and its use are covered by
this License. You must supply a copy of this License. If the work
during execution displays copyright notices, you must include the
copyright notice for the Library among them, as well as a reference
directing the user to the copy of this License. Also, you must do one
of these things:
a) Accompany the work with the complete corresponding
machine-readable source code for the Library including whatever
changes were used in the work (which must be distributed under
Sections 1 and 2 above); and, if the work is an executable linked
with the Library, with the complete machine-readable "work that
uses the Library", as object code and/or source code, so that the
user can modify the Library and then relink to produce a modified
executable containing the modified Library. (It is understood
that the user who changes the contents of definitions files in the
Library will not necessarily be able to recompile the application
to use the modified definitions.)
b) Use a suitable shared library mechanism for linking with the
Library. A suitable mechanism is one that (1) uses at run time a
copy of the library already present on the user's computer system,
rather than copying library functions into the executable, and (2)
will operate properly with a modified version of the library, if
the user installs one, as long as the modified version is
interface-compatible with the version that the work was made with.
c) Accompany the work with a written offer, valid for at
least three years, to give the same user the materials
specified in Subsection 6a, above, for a charge no more
than the cost of performing this distribution.
d) If distribution of the work is made by offering access to copy
from a designated place, offer equivalent access to copy the above
specified materials from the same place.
e) Verify that the user has already received a copy of these
materials or that you have already sent this user a copy.
For an executable, the required form of the "work that uses the
Library" must include any data and utility programs needed for
reproducing the executable from it. However, as a special exception,
the materials to be distributed need not include anything that is
normally distributed (in either source or binary form) with the major
components (compiler, kernel, and so on) of the operating system on
which the executable runs, unless that component itself accompanies
the executable.
It may happen that this requirement contradicts the license
restrictions of other proprietary libraries that do not normally
accompany the operating system. Such a contradiction means you cannot
use both them and the Library together in an executable that you
distribute.
7. You may place library facilities that are a work based on the
Library side-by-side in a single library together with other library
facilities not covered by this License, and distribute such a combined
library, provided that the separate distribution of the work based on
the Library and of the other library facilities is otherwise
permitted, and provided that you do these two things:
a) Accompany the combined library with a copy of the same work
based on the Library, uncombined with any other library
facilities. This must be distributed under the terms of the
Sections above.
b) Give prominent notice with the combined library of the fact
that part of it is a work based on the Library, and explaining
where to find the accompanying uncombined form of the same work.
8. You may not copy, modify, sublicense, link with, or distribute
the Library except as expressly provided under this License. Any
attempt otherwise to copy, modify, sublicense, link with, or
distribute the Library is void, and will automatically terminate your
rights under this License. However, parties who have received copies,
or rights, from you under this License will not have their licenses
terminated so long as such parties remain in full compliance.
9. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Library or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Library (or any work based on the
Library), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Library or works based on it.
10. Each time you redistribute the Library (or any work based on the
Library), the recipient automatically receives a license from the
original licensor to copy, distribute, link with or modify the Library
subject to these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties with
this License.
11. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Library at all. For example, if a patent
license would not permit royalty-free redistribution of the Library by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Library.
If any portion of this section is held invalid or unenforceable under any
particular circumstance, the balance of the section is intended to apply,
and the section as a whole is intended to apply in other circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
12. If the distribution and/or use of the Library is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Library under this License may add
an explicit geographical distribution limitation excluding those countries,
so that distribution is permitted only in or among countries not thus
excluded. In such case, this License incorporates the limitation as if
written in the body of this License.
13. The Free Software Foundation may publish revised and/or new
versions of the Lesser General Public License from time to time.
Such new versions will be similar in spirit to the present version,
but may differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the Library
specifies a version number of this License which applies to it and
"any later version", you have the option of following the terms and
conditions either of that version or of any later version published by
the Free Software Foundation. If the Library does not specify a
license version number, you may choose any version ever published by
the Free Software Foundation.
14. If you wish to incorporate parts of the Library into other free
programs whose distribution conditions are incompatible with these,
write to the author to ask for permission. For software which is
copyrighted by the Free Software Foundation, write to the Free
Software Foundation; we sometimes make exceptions for this. Our
decision will be guided by the two goals of preserving the free status
of all derivatives of our free software and of promoting the sharing
and reuse of software generally.
NO WARRANTY
15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO
WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.
EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR
OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY
KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE
LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME
THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN
WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY
AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU
FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR
CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE
LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING
RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A
FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF
SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Libraries
If you develop a new library, and you want it to be of the greatest
possible use to the public, we recommend making it free software that
everyone can redistribute and change. You can do so by permitting
redistribution under these terms (or, alternatively, under the terms of the
ordinary General Public License).
To apply these terms, attach the following notices to the library. It is
safest to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least the
"copyright" line and a pointer to where the full notice is found.
<one line to give the library's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
Also add information on how to contact you by electronic and paper mail.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the library, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the
library `Frob' (a library for tweaking knobs) written by James Random Hacker.
<signature of Ty Coon>, 1 April 1990
Ty Coon, President of Vice
That's all there is to it!
-17
View File
@@ -1,17 +0,0 @@
JHOVE - JSTOR/Harvard Object Validation Environment
Copyright 2003-2008 by JSTOR and the President and Fellows of Harvard College
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU Lessor General Public License as
published by the Free Software Foundation; either version 2.1 of the
License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
USA
-227
View File
@@ -1,227 +0,0 @@
JHOVE - JSTOR/Harvard Object Validation Environment
Copyright 2003-2012 by JSTOR and the President and Fellows of Harvard College
JHOVE is made available under the GNU Lesser General Public License (LGPL;
see the file LICENSE for details)
Rev. 1.11, 2013-09-29
JHOVE (the JSTOR/Harvard Object Validation Environment, pronounced "jhove")
is an extensible software framework for performing format identification,
validation, and characterization of digital objects.
o Format identification is the process of determining the format to which a
digital object conforms: "I have a digital object; what format is it?"
o Format validation is the process of determining the level of compliance of a
digital object to the specification for its purported format: "I have an
object purportedly of format F; is it?"
o Format characterization is the process of determing the format-specific
significant properties of an object of a given format: "I have an object of
format F; what are its salient properties?"
These actions are frequently necessary during routine operation of digital
repositories and for digital preservation activities.
The output from JHOVE is controlled by output handlers. JHOVE uses an
extensible plug-in architecture; it can be configured at the time of its
invocation to include whatever specific format modules and output handlers
that are desired. The initial release of JHOVE includes modules for
arbitrary byte streams, ASCII and UTF-8 encoded text, AIFF and WAVE audio,
GIF, JPEG, JPEG 2000, TIFF, and PDF; and text and XML output handlers.
The JHOVE project is a collaboration of JSTOR and the Harvard University
Library. Development of JHOVE was funded in part by the Andrew W. Mellon
Foundation. JHOVE is made available under the GNU Lesser General Public
License (LGPL; see the file LICENSE for details).
JHOVE is currently being maintained by indpendent developers.
REQUIREMENTS
1. Java J2SE 1.5
(JHOVE was originally implemented using the Sun J2SE SDK 1.4.1 and has
been tested to work with 1.5)
2. If you would like to compile the JHOVE source code, then
Apache Ant, a Java-based build tool <http://ant.apache.org/> is necessary.
Note that the JAVA_HOME environment variable must be set appropriately for
Ant to work properly.
(JHOVE was implemented and tested using Ant 1.5.1.)
DISTRIBUTION
The JHOVE distribution package includes:
jhove/ # JHOVE home directory
COPYING # GNU Lesser General Public License
LICENSE # JHOVE license information
README
RELEASENOTES # JHOVE release notes
bin/
jhove.jar # JHOVE API package
jhove-handler.jar # Standard output handler package
jhove-module.jar # Standard module package
JhoveApp.jar # JHOVE command line application
JhoveView.jar # JHOVE with Swing GUI front-end
build.xml # Ant configuration file
classes/
build.xml # Ant configuration file
edu/ ... # JHOVE API packages
ADump.* # AIFF dump utility class
GDump.* # GIF dump utility class
Jhove.* # JHOVE main class
JDump.* # JPEG dump utility class
J2Dump.* # JPEG 2000 dump utility class
PDump.* # PDF dump utility class
TDump.* # TIFF dump utility class
UserHome.* # user.home property utility class
WDump.* # WAVE dump utility class
conf/
jhove.conf # JHOVE configuration file
jhove.xsd # JHOVE output schema
jhoveConfig.xsd # JHOVE configuration file schema
doc/
*.html # API documentation
...
examples/ # Sample files
ascii/ ...
gif/ ...
jpeg/ ...
jpeg2000/ ...
pdf/ ...
tiff/ ...
utf-8/ ...
adump* # AIFF dump Bourne shell driver
adump.bat* # AIFF dump DOS shell driver script
gdump* # GIF dump Bourne shell driver
gdump.bat* # GIF dump DOS shell driver script
jdump* # JPEG dump Bourne shell driver
jdump.bat* # JPEG dump DOS shell driver script
j2dump* # JPEG 2000 dump Bourne shell driver
j2dump.bat* # JPEG 2000 dump DOS shell driver
jhove.tmpl* # Template for JHOVE Bourne shell driver script
jhove_bat.tmpl* # Template for JHOVE DOS shell driver script
pdump* # PDF dump Bourne shell driver
pdump.bat* # PDF dump DOS shell driver script
tdump* # TIFF dump Bourne shell driver
tdump.bat* # TIFF dump DOS shell driver script
userhome* # user.home Bourne shell driver
userhome.bat* # user.home DOS shell driver script
wdump* # WAVE dump Bourne shell driver
wdump.bat* # WAVE dump DOS shell driver script
INSTALLATION
Edit the configuration file, jhove/conf/jhove.conf, and set the absolute
pathname of the JHOVE home directory and the temporary directory (in which
temporary files are created):
<jhoveHome>jhove-home-directory</jhoveHome>
<tempDirectory>temporary-directory</tempDirectory>
The JHOVE home directory is the top-most directory in the distribution TAR
or ZIP file. On Unix systems, "/var/tmp" is an appropriate temporary
directory; on Windows, "C:\Temp". For example, if the distribution TAR
file is disaggregated on a Unix system in the directory "/users/stephen/
projects", then the configuration file should read:
<jhoveHome>/users/stephen/projects/jhove</jhoveHome>
<tempDirectory>/var/tmp</jhoveHome>
In the JHOVE home directory, copy the JHOVE Bourne shell driver script
template, "jhove.tmpl", to "jhove" (or the equivalent Windows shell
script, "jhove_bat.tmpl" to "jhove.bat"), and set the
JHOVE home directory, Java home directory, and Java interpreter:
JHOVE_HOME=jhove-home-directory
JAVA_HOME=java-home-directory
JAVA=java-interpreter
The JAVA_HOME property should provide the absolute pathname of the Java
runtime or SDK installation; JAVA should provide the absolute pathname of the
Java interpreter. For example:
JHOVE_HOME=/users/stephen/projects/jhove
JAVA_HOME=/usr/local/j2re1.4.1_02
JAVA=$JAVA_HOME/bin/java
In the DOS shell driver script, jhove.bat, the equivalent three
variables are:
SET JHOVE_HOME=jhove-home-directory
SET JAVA_HOME=java-home-directory
SET JAVA=%JAVA_HOME%\bin\java
For example:
SET JHOVE_HOME="C:\Program Files\jhove"
SET JAVA_HOME="C:\Program Files\java\j2re1.4.1_02"
SET JAVA=%JAVA_HOME%\bin\java
The quotation marks are necessary because of the embedded space characters.
On Windows platforms it may also be necessary to add the Java bin subdirectory
to the System PATH environment variable:
PATH=C:\Program Files\java\j2re1.4.1_02\bin;...
(For information on setting a Windows environment variable, consult your local
documentation or system administrator.)
USAGE
java Jhove [-c config] [-m module] [-h handler] [-e encoding] [-H handler]
[-o output] [-x saxclass] [-t tempdir] [-b bufsize]
[-l loglevel] [[-krs] dir-file-or-uri [...]]
where -c config Configuration file pathname
-m module Module name
-h handler Output handler name (defaults to TEXT)
-e encoding Character encoding used by output handler (defaults to UTF-8)
-H handler About handler name
-o output Output file pathname (defaults to standard output)
-x saxclass SAX parser class (defaults to J2SE default)
-t tempdir Temporary directory in which to create temporary files
-b bufsize Buffer size for buffered I/O (defaults to J2SE 1.4 default)
-l loglevel Logging level
-k Calculate CRC32, MD5, and SHA-1 checksums
-r Display raw data flags, not textual equivalents
-s Format identification based on internal signatures only
dir-file-or-uri Directory or file pathname or URI of formated content
stream
All named modules and output handlers must be found on the Java CLASSPATH at
the time of invocation. The JHOVE driver script, jhove/jhove, automatically
sets the CLASSPATH and invokes the Jhove main class:
jhove [-c config] [-m module] [-h handler] [-e encoding] [-H handler]
[-o output] [-x saxclass] [-t tempdir] [-b bufsize] [-l loglevel]
[[-krs] dir-file-or-uri [...]]
The following additional programs are available, primarily for testing
and debugging purposes. They display a minimally processed, human-readable
version of the contents of AIFF, GIF, JPEG, JPEG 2000, PDF, TIFF, and WAVE
files:
java ADump aiff-file
java GDump gif-file
java JDump jpeg-file
java J2Dump jpeg2000-file
java PDump pdf-file
java TDump tiff-file
java WDump wave-file
For convenience, the following driver scripts are also available:
adump aiff-file
gdump gif-file
jdump jpeg-file
j2dump jpeg2000-file
pdump pdf-file
tdump tiff-file
wdump wave-file
The JHOVE Swing-based GUI interface can be invoked from a command shell from
the jhove/bin sub-directory:
java -jar JhoveView.jar -c <configFile>
where <configFile> is the pathname of the JHOVE configuration file.
File diff suppressed because it is too large Load Diff
Binary file not shown.
Binary file not shown.
-25
View File
@@ -1,25 +0,0 @@
JHOVE - JSTOR/Harvard Object Validation Environment
Copyright 2003 by JSTOR and the President and Fellows of Harvard College
JHOVE is made available under the GNU General Public License (see the file
LICENSE for details)
Rev. 2003-11-25
The following jar files are meant to be used for embedding JHOVE functionality
into new applications or systems.
jhove.jar Contains the JHOVE API interfaces and classes
jhove-module.jar Contains the standard JHOVE modules ()
jhove-handler.jar Contains the standard JHOVE output handlers (TEXT and XML)
The following jar file is meant to be used with the stand-alone JHOVE
application using a command-line interface. It contains the main Jhove class
and the contents of jhove.jar, jhove-module.jar, and jhove-handler.jar.
JhoveApp.jar
The following jar file is meant to be used with the stand-alone JHOVE
application using a Swing GUI interface. It contains the main JhoveView class
and the contents of jhove.jar, jhove-module.jar, and jhove-handler.jar.
JhoveView.jar
Binary file not shown.
Binary file not shown.
Binary file not shown.
-78
View File
@@ -1,78 +0,0 @@
<project name="Jhove" default="dist" basedir=".">
<description>Project build file
Jhove - JSTOR/Harvard Object Validation Environment
Version 1.0 2004-09-10
Copyright 2004 by JSTOR and the President and Fellows of Harvard College
</description>
<!-- ant (or ant dist) Build everything
ant debug Build everything with debug enabled
ant clean Delete backup files
ant cleanclass Delete backup and class files
ant cleandist Delete backup, class, and jar files
ant javadoc Build javadocs
-->
<!-- set global properties for this build -->
<property name="bin" location="bin"/>
<property name="classes" location="classes"/>
<property name="doc" location="doc"/>
<target name="dist" description="Create distribution">
<ant dir="${classes}" inheritAll="false">
<property name="dbg" value="off"/>
</ant>
<chmod file="jhove" perm="ugo+x"/>
<chmod file="${bin}/JhoveView.jar" perm="ugo+x"/>
</target>
<target name="debug" description="Create distribution with debug enabled">
<ant dir="${classes}" inheritAll="false">
<property name="dbg" value="on"/>
</ant>
</target>
<target name="view" description="Create JhoveView application">
<ant dir="${classes}" target="view" inheritAll="false">
<property name="dbs" value="on"/>
</ant>
</target>
<target name="clean" description="Delete backup files">
<ant dir="${classes}" target="main-clean" inheritAll="false"/>
</target>
<target name="cleanclass" depends="clean">
<ant dir="${classes}" target="main-cleanclass" inheritAll="false"/>
</target>
<target name="cleandist" depends="cleanclass">
<delete file="${bin}/jhove.jar"/>
<delete file="${bin}/jhove-handler.jar"/>
<delete file="${bin}/jhove-module.jar"/>
<delete file="${bin}/JhoveApp.jar"/>
<delete file="${bin}/JhoveView.jar"/>
</target>
<target name="javadoc">
<javadoc sourcepath="${classes}" destdir="${doc}"
windowtitle="JHOVE Documentation"
Overview="${classes}/overview.html">
<package name="edu.harvard.hul.ois.jhove"/>
<package name="edu.harvard.hul.ois.jhove.handler"/>
<package name="edu.harvard.hul.ois.jhove.handler.audit"/>
<package name="edu.harvard.hul.ois.jhove.module"/>
<package name="edu.harvard.hul.ois.jhove.module.aiff"/>
<package name="edu.harvard.hul.ois.jhove.module.gif"/>
<package name="edu.harvard.hul.ois.jhove.module.html"/>
<package name="edu.harvard.hul.ois.jhove.module.iff"/>
<package name="edu.harvard.hul.ois.jhove.module.jpeg"/>
<package name="edu.harvard.hul.ois.jhove.module.jpeg2000"/>
<package name="edu.harvard.hul.ois.jhove.module.pdf"/>
<package name="edu.harvard.hul.ois.jhove.module.tiff"/>
<package name="edu.harvard.hul.ois.jhove.module.wave"/>
<package name="edu.harvard.hul.ois.jhove.module.xml"/>
<package name="edu.harvard.hul.ois.jhove.viewer"/>
</javadoc>
</target>
</project>
-63
View File
@@ -1,63 +0,0 @@
JHOVE - JSTOR/Harvard Object Validation Environment
Copyright 2003-2007 by JSTOR and the President and Fellows of Harvard College
JHOVE is made available under the GNU General Public License (see the file
LICENSE for details)
Rev. 2007-08-30
Edit the configuration file, jhove.conf, and set the JHOVE home
directory:
<jhoveHome>jhove-home-directory</jhoveHome>
and temporary directory:
<tempDirectory>temporary-directory</tempDirectory>
On most Unix systems, a reasonable temporary directory is "/var/tmp";
on Windows, "C:\temp".
The optional
<bufferSize>buffer-size</bufferSize>
element defines the buffer size used for buffer I/O operations.
The optional
<mixVersion>1.0</mixVersion>
element specifies that the XML output handler should conform to the
MIX 1.0 schema. The default behavior is for handler output to conform
to the MIX 0.2 schema.
The optional
<sigBytes>n</sigBytes>
element specifies that JHOVE modules will look for format signatures
in the first <n> bytes of the file. The default value is 1024.
All class names must be fully qualified with their package name:
<module>
<class>fully-package-qualified-class-name</class>
<init>optional-initialization-argument</init>
<param>optional-invocation-argument</param>
</module>
The optional <init> argument is passed to the module once at the time
its class is instantiated. See module-specific documentation for a
description of any initialization options.
The optional <param> argument is passed to the module every time it is
invoked. See module-specific documentation for a description of any
invocation options.
The order in which format modules are defined is important; when
performing a format identification operation, JHOVE will search for a
matching module in the order in which the modules are defined in the
configuration file. In general, the modules for more generic formats
should come later in the list. For example, the standard module ASCII
should be defined before the UTF-8 module, since all ASCII objects
are, by definition, UTF-8 objects, but not vice versa.
@@ -1,47 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<jhoveConfig version="1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://hul.harvard.edu/ois/xml/ns/jhove/jhoveConfig"
xsi:schemaLocation="http://hul.harvard.edu/ois/xml/ns/jhove/jhoveConfig
http://hul.harvard.edu/ois/xml/xsd/jhove/1.6/jhoveConfig.xsd">
<jhoveHome>/users/stephen/projects/jhove</jhoveHome>
<defaultEncoding>utf-8</defaultEncoding>
<tempDirectory>/var/tmp</tempDirectory>
<bufferSize>131072</bufferSize>
<mixVersion>1.0</mixVersion>
<sigBytes>1024</sigBytes>
<module>
<class>edu.harvard.hul.ois.jhove.module.AiffModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.WaveModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.PdfModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.Jpeg2000Module</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.JpegModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.GifModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.TiffModule</class>
<param>byteoffset=true</param>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.XmlModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.HtmlModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.AsciiModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.Utf8Module</class>
</module>
</jhoveConfig>
-51
View File
@@ -1,51 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<jhoveConfig version="1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://hul.harvard.edu/ois/xml/ns/jhove/jhoveConfig"
xsi:schemaLocation="http://hul.harvard.edu/ois/xml/ns/jhove/jhoveConfig
http://hul.harvard.edu/ois/xml/xsd/jhove/1.6/jhoveConfig.xsd">
<jhoveHome>/users/stephen/projects/jhove</jhoveHome>
<defaultEncoding>utf-8</defaultEncoding>
<tempDirectory>/var/tmp</tempDirectory>
<bufferSize>131072</bufferSize>
<mixVersion>1.0</mixVersion>
<sigBytes>1024</sigBytes>
<module>
<class>edu.harvard.hul.ois.jhove.module.AiffModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.WaveModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.PdfModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.Jpeg2000Module</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.JpegModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.GifModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.TiffModule</class>
<param>byteoffset=true</param>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.XmlModule</class>
<param>withTextMD=true</param>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.HtmlModule</class>
<param>withTextMD=true</param>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.AsciiModule</class>
<param>withTextMD=true</param>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.Utf8Module</class>
<param>withTextMD=true</param>
</module>
</jhoveConfig>
-45
View File
@@ -1,45 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<jhoveConfig version="1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://hul.harvard.edu/ois/xml/ns/jhove/jhoveConfig"
xsi:schemaLocation="http://hul.harvard.edu/ois/xml/ns/jhove/jhoveConfig
http://hul.harvard.edu/ois/xml/xsd/jhove/1.6/jhoveConfig.xsd">
<jhoveHome>./jhove/</jhoveHome>
<defaultEncoding>utf-8</defaultEncoding>
<tempDirectory>/var/tmp</tempDirectory>
<bufferSize>131072</bufferSize>
<module>
<class>edu.harvard.hul.ois.jhove.module.AiffModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.WaveModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.PdfModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.Jpeg2000Module</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.JpegModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.GifModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.TiffModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.XmlModule</class>
<param>schema=http://www.example.com/schema;/home/schemas/exampleschema.xsd</param>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.HtmlModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.AsciiModule</class>
</module>
<module>
<class>edu.harvard.hul.ois.jhove.module.Utf8Module</class>
</module>
</jhoveConfig>
-88
View File
@@ -1,88 +0,0 @@
#!/usr/bin/perl
########################################################################
# Jhove - JSTOR/Harvard Object Validation Environment
# Copyright 2004 by JSTOR and the President and Fellows of Harvard College
#
# A Perl script for plugging local path information into the
# various script files of JHOVE, as well as conf/jhove.conf.
#
# This is configured only for Unix (including OS X).
#
# Usage: configure.pl jhove_home_directory [java_home_directory [java_runtime_directory]]
#
# If invoked with no arguments, it will output a usage message.
#
########################################################################
use File::Copy;
sub mung {
my $f = $_[0];
my $bak = $f . "~";
#If there is no backup file, copy the file to the
#backup. Otherwise work from the backup.
if (!(-e $bak)) {
rename ($f, $bak);
}
open (INFILE, $bak);
open (OUTFILE, ">" . $f);
#Walks through each line of file, making substitutions.
#Remember that the JAVA_HOME and JAVA arguments are optional.
while (<INFILE>) {
s/^JHOVE_HOME=.*/JHOVE_HOME=$ARGV[0]/;
if ($narg >= 2) {
s/^JAVA_HOME=.*/JAVA_HOME=$ARGV[1]/;
}
if ($narg >= 3) {
s/^JAVA=.*/JAVA=$ARGV[2]/;
}
print OUTFILE;
}
close (INFILE);
close (OUTFILE);
if (-e $f) {
print ("Fixed " . $f . "\n");
}
}
$narg = $#ARGV + 1;
if ($narg <= 0) {
print "Usage: configure.pl jhove_home_directory [java_home_directory [java_runtime_directory]]\n";
exit;
}
print "JHOVE_HOME will be set to " . $ARGV[0] . "\n";
if ($narg >= 2) {
print "JAVA_HOME will be set to " . $ARGV[1] . "\n";
}
if ($narg >= 3) {
print "JAVA will be set to " . $ARGV[2] . "\n";
}
mung ("jhove");
mung ("adump");
mung ("gdump");
mung ("jdump");
mung ("j2dump");
mung ("pdump");
mung ("tdump");
mung ("wdump");
#Fix up the config file. We assume that the <jhoveHome>
#element is all on one line.
if (!(-e "conf/jhove.conf~")) {
rename ("conf/jhove.conf", "conf/jhove.conf~");
}
open (INFILE, "conf/jhove.conf~");
open (OUTFILE, ">conf/jhove.conf");
while (<INFILE>) {
s!<jhoveHome>.*</jhoveHome>!<jhoveHome>$ARGV[0]</jhoveHome>!;
print OUTFILE;
}
close (INFILE);
close (OUTFILE);
if (-e "conf/jhove.conf") {
print "Fixed conf/jhove.conf\n";
}
exit;
-37
View File
@@ -1,37 +0,0 @@
#!/bin/sh
########################################################################
# gdump - JSTOR/Harvard Object Validation Environment
# Copyright 2004-2005 by the President and Fellows of Harvard College
# JHOVE is made available under the GNU General Public License (see the
# file LICENSE for details)
#
# Driver script for the GIF dump utility
#
# Usage: gdump file
#
# where file is a GIF file
#
# Configuration constants:
JHOVE_HOME=/users/stephen/projects/jhove
JAVA_HOME=/usr/java # Java JRE directory
JAVA=$JAVA_HOME/bin/java # Java interpreter
EXTRA_JARS= # Extra .jar files to add to CLASSPATH
# NOTE: Nothing below this line should be edited
########################################################################
CP=${JHOVE_HOME}/bin/JhoveApp.jar:${EXTRA_JARS}
# Retrieve a copy of all command line arguments to pass to the application.
ARGS=""
for ARG do
ARGS="$ARGS $ARG"
done
# Set the CLASSPATH and invoke the Java loader.
${JAVA} -classpath $CP GDump $ARGS
-37
View File
@@ -1,37 +0,0 @@
#!/bin/sh
########################################################################
# j2dump - JSTOR/Harvard Object Validation Environment
# Copyright 2004-2005 by the President and Fellows of Harvard College
# JHOVE is made available under the GNU General Public License (see the
# file LICENSE for details)
#
# Driver script for the JPEG 2000 dump utility
#
# Usage: j2dump file
#
# where file is a JPEG file
#
# Configuration constants:
JHOVE_HOME=/users/stephen/projects/jhove
JAVA_HOME=/usr/java # Java JRE directory
JAVA=$JAVA_HOME/bin/java # Java interpreter
EXTRA_JARS= # Extra .jar files to add to CLASSPATH
# NOTE: Nothing below this line should be edited
########################################################################
CP=${JHOVE_HOME}/bin/JhoveApp.jar:${EXTRA_JARS}
# Retrieve a copy of all command line arguments to pass to the application.
ARGS=""
for ARG do
ARGS="$ARGS $ARG"
done
# Set the CLASSPATH and invoke the Java loader.
${JAVA} -classpath $CP J2Dump $ARGS
-37
View File
@@ -1,37 +0,0 @@
#!/bin/sh
########################################################################
# jdump - JSTOR/Harvard Object Validation Environment
# Copyright 2004-2005 by the President and Fellows of Harvard College
# JHOVE is made available under the GNU General Public License (see the
# file LICENSE for details)
#
# Driver script for the JPEG dump utility
#
# Usage: jdump file
#
# where file is a JPEG file
#
# Configuration constants:
JHOVE_HOME=/users/stephen/projects/jhove
JAVA_HOME=/usr/java # Java JRE directory
JAVA=$JAVA_HOME/bin/java # Java interpreter
EXTRA_JARS= # Extra .jar files to add to CLASSPATH
# NOTE: Nothing below this line should be edited
########################################################################
CP=${JHOVE_HOME}/bin/JhoveApp.jar:${EXTRA_JARS}
# Retrieve a copy of all command line arguments to pass to the application.
ARGS=""
for ARG do
ARGS="$ARGS $ARG"
done
# Set the CLASSPATH and invoke the Java loader.
${JAVA} -classpath $CP JDump $ARGS
-58
View File
@@ -1,58 +0,0 @@
#!/bin/sh
########################################################################
# JHOVE - JSTOR/Harvard Object Validation Environment
# Copyright 2003-2005 by JSTOR and the President and Fellows of Harvard College
# JHOVE is made available under the GNU General Public License (see the
# file LICENSE for details)
#
# Usage: jhove [-c config] [-m module] [-h handler] [-e encoding] [-H handler]
# [-o output] [-x saxclass] [-t tempdir] [-b bufsize]
# [-l loglevel] [[-krs] dir-file-or-uri [...]]
#
# where -c config Configuration file pathname
# -m module Module name
# -h handler Output handler name (defaults to TEXT)
# -e encoding Character encoding of output handler (defaults to UTF-8)
# -H handler About handler name
# -o output Output file pathname (defaults to standard output)
# -x saxclass SAX parser class (defaults to J2SE 1.4 default)
# -t tempdir Temporary directory in which to create temporary files
# -b bufsize Buffer size for buffered I/O (defaults to J2SE 1.4 default)
# -k Calculate CRC32, MD5, and SHA-1 checksums
# -r Display raw data flags, not textual equivalents
# -s Format identification based on internal signatures only
# dir-file-or-uri Directory, file pathname or URI of formatted content
#
# CHANGE for JHOVE 1.8:
# You no longer have to figure out where JAVA_HOME is; that's the
# operating system's job. If the OS tells you it can't find Java,
# adjust your shell's path or revert to the old way (commented out).
# Configuration constants:
#JHOVE_HOME=/users/gary/dev/jhove
JHOVE_HOME=[fill in path to jhove directory]
JAVA_HOME=/usr/java # Java JRE directory -- change to your local java home
JAVA=$JAVA_HOME/bin/java # Java interpreter -- usually won't need change
#XTRA_JARS=/users/stephen/xercesImpl.jar
EXTRA_JARS= # Extra .jar files to add to CLASSPATH
# NOTE: Nothing below this line should be edited
########################################################################
CP=${JHOVE_HOME}/bin/JhoveApp.jar:${EXTRA_JARS}
# Retrieve a copy of all command line arguments to pass to the application.
ARGS=""
for ARG do
ARGS="$ARGS $ARG"
done
# Set the CLASSPATH and invoke the Java loader.
#{JAVA} -classpath $CP Jhove $ARGS -x org.apache.xerces.parsers.SAXParser
#${JAVA} -classpath $CP Jhove $ARGS
# New way, doesn't require you to use JAVA_HOME.
java -classpath $CP Jhove $ARGS
-54
View File
@@ -1,54 +0,0 @@
#!/bin/sh
########################################################################
# JHOVE - JSTOR/Harvard Object Validation Environment
# Copyright 2003-2004 by JSTOR and the President and Fellows of Harvard College
# JHOVE is made available under the GNU General Public License (see the
# file LICENSE for details)
#
# Copy jhove.tmpl to jhove, and replace the value of JHOVE_HOME with
# the path to your jhove directory.
#
# Usage: jhove [-c config] [-m module [-p param]] [-h handler [-P param]]
# [-e encoding] [-H handler] [-o output] [-x saxclass]
# [-t tempdir] [-b bufsize] [[-krs] dir-file-or-uri [...]]
#
# where -c config Configuration file pathname
# -m module Module name
# -p param Module-specific parameter
# -h handler Output handler name (defaults to TEXT)
# -P param Handler-specific parameter
# -o output Output file pathname (defaults to standard output)
# -x saxclass SAX parser class (defaults to J2SE 1.4 default)
# -t tempdir Temporary directory in which to create temporary files
# -b bufsize Buffer size for buffered I/O (defaults to J2SE 1.4 default)
# -k Calculate CRC32, MD5, and SHA-1 checksums
# -r Display raw data flags, not textual equivalents
# -s Format identification based on internal signatures only
# dir-file-or-uri Directory, file pathname or URI of formatted content
#
# Configuration constants:
JHOVE_HOME=[your directory path]/jhove
JAVA_HOME=/usr/java
JAVA=/usr/bin/java
#XTRA_JARS=/users/stephen/xercesImpl.jar
EXTRA_JARS= # Extra .jar files to add to CLASSPATH
# NOTE: Nothing below this line should be edited
########################################################################
CP=${JHOVE_HOME}/bin/JhoveApp.jar:${EXTRA_JARS}
# Retrieve a copy of all command line arguments to pass to the application.
ARGS=""
for ARG do
ARGS="$ARGS $ARG"
done
# Set the CLASSPATH and invoke the Java loader.
#{JAVA} -classpath $CP Jhove $ARGS -x org.apache.xerces.parsers.SAXParser
${JAVA} -classpath $CP Jhove $ARGS
-60
View File
@@ -1,60 +0,0 @@
@ECHO OFF
REM JHOVE - JSTOR/Harvard Object Validation Environment
REM Copyright 2003-2005 by JSTOR and the President and Fellows of Harvard College
REM JHOVE is made available under the GNU General Public License (see the
REM file LICENSE for details)
REM
REM Usage: jhove [-c config] [-m module] [-h handler] [-e encoding]
REM [-H handler] [-o output] [-x saxclass] [-t tempdir]
REM [-b bufsize] [-l loglevel] [[-krs] dir-file-or-uri [...]]
REM
REM For Windows systems, copy jhove_bat.tmpl to jhove.bat and change
REM the value of JHOVE_HOME to the path to your jhove directory.
REM
REM where -c config Configuration file pathname
REM -m module Module name
REM -h handler Output handler name (defaults to TEXT)
REM -e encoding Character encoding of output handler (defaults to UTF-8)
REM -H handler About handler name
REM -o output Output file pathname (defaults to standard output)
REM -x saxclass SAX parser class (defaults to J2SE 1.4 default)
REM -t tempdir Temporary directory in which to create temporary files
REM -b bufsize Buffer size for buffered I/O (defaults to J2SE default)
REM -l loglevel Logging level
REM -k Calculate CRC32, MD5, and SHA-1 checksums
REM -r Display raw data flags, not textual equivalents
REM -s Format identification based on internal signatures only
REM dir-file-or-uri Directory, file pathname, or URI of formatted content
REM
REM Configuration constants:
REM JHOVE_HOME Jhove installation directory
REM JAVA_HOME Java JRE directory
REM JAVA Java interpreter
REM EXTRA_JARS Extra jar files to add to CLASSPATH
REM SET JHOVE_HOME="C:\Program Files\jhove"
SET JHOVE_HOME="[your directory path]\jhove"
SET EXTRA_JARS=
REM NOTE: Nothing below this line should be edited
REM #########################################################################
SET CP=%JHOVE_HOME%\bin\JhoveApp.jar
IF "%EXTRA_JARS%"=="" GOTO FI
SET CP=%CP%:%EXTRA_JARS
:FI
REM Retrieve a copy of all command line arguments to pass to the application
SET ARGS=
:WHILE
IF "%1"=="" GOTO LOOP
SET ARGS=%ARGS% %1
SHIFT
GOTO WHILE
:LOOP
REM Set the CLASSPATH and invoke the Java loader
JAVA -classpath %CP% Jhove %ARGS%
Binary file not shown.
Binary file not shown.
-14
View File
@@ -1,14 +0,0 @@
#!/usr/bin/perl -w
# Generate MD5 checksum
use Digest::MD5;
die "usage: md5.pl file\n" if $#ARGV < 0;
open (FILE, "<$ARGV[0]") or die "can't open file\"$ARGV[0]\"!\n";
$ctx = Digest::MD5->new->addfile (*FILE);
close (FILE);
$digest = $ctx->hexdigest;
print "$digest\n";

Some files were not shown because too many files have changed in this diff Show More