Tweak release notes

Should extend test for other Asian languages

.travis.yml

@@ -1,3 +1,5 @@
+sudo: required
+dist: trusty
 language: python
 
 cache: pip
@@ -10,16 +12,15 @@ cache:
 
 python:
     - 3.4
+    - 3.5
 
 before_cache:
     - rm -f $HOME/.cache/pip/log/debug.log
 
 before_install:
     # Ubuntu packages
-    - sudo add-apt-repository ppa:evl.ms/precise -y  # for Ghostscript 9.15
-    - sudo add-apt-repository ppa:lyrasis/precise-backports -y  # for Tesseract 3.03
-    - sudo add-apt-repository ppa:b-eltzner/qpdfview-exp -y  # for QPDF 5
-    - sudo add-apt-repository ppa:itachi-san/ffmpeg -y  # for libav 11.2 (for unpaper)
+    - sudo add-apt-repository ppa:vshn/ghostscript -y  # for ghostscript 9.16 (trusty has 9.10)
+    - sudo add-apt-repository ppa:heyarje/libav-11 -y  # for libav11, which is for unpaper
     - sudo apt-get update -qq  # must go after all add-apt-repo
     - sudo apt-get install -y ghostscript tesseract-ocr tesseract-ocr-deu tesseract-ocr-eng tesseract-ocr-fra qpdf poppler-utils libavformat56 libavcodec56 libavutil54 libffi-dev
 

README.rst

@@ -190,7 +190,14 @@ The command line program should now be available:
 Installing on Ubuntu 14.04 LTS
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Installing on Ubuntu 14.04 LTS (trusty) is more difficult than other options, because of certain bugs in Python package installation.
+Installing on Ubuntu 14.04 LTS (trusty) is more difficult than some other options, because of bugs in Python package installation.
+
+Add new "apt" repositories needed for backports of Ghostscript 9.16 and libav-11, which supports unpaper 6.1. This will replace Ghostscript on your system.
+
+.. code-block:: bash
+
+   sudo add-apt-repository ppa:vshn/ghostscript -y 
+   sudo add-apt-repository ppa:heyarje/libav-11 -y
 
 Update apt-get:
 
@@ -207,10 +214,10 @@ Install system dependencies:
       zlib1g-dev \
       libjpeg-dev \
       libffi-dev \
+      libavformat56 libavcodec56 libavutil54 \
       ghostscript \
       tesseract-ocr \
       qpdf \
-      unpaper \
       python3-pip \
       python3-pil \
       python3-pytest \
@@ -223,7 +230,7 @@ into your system Python, which could interfere with other programs):
 
    sudo pip3 install ocrmypdf
    
-If you wish to install OCRmyPDF to a virtual environment to isolate system Python from modified, you can
+If you wish to install OCRmyPDF to a virtual environment to isolate the system Python, you can
 follow these steps.  This includes a workaround `for a known, unresolved issue in Ubuntu 14.04's ensurepip
 package <http://www.thefourtheye.in/2014/12/Python-venv-problem-with-ensurepip-in-Ubuntu.html>`_:
 
@@ -234,11 +241,17 @@ package <http://www.thefourtheye.in/2014/12/Python-venv-problem-with-ensurepip-i
    source venv-ocrmypdf/bin/activate
    wget -O - -o /dev/null https://bootstrap.pypa.io/get-pip.py | python
    deactivate
-   pyvenv --system-site-packages venv-ocrmypdf
+   python3 -m venv --system-site-packages venv-ocrmypdf
    source venv-ocrmypdf/bin/activate
    pip install ocrmypdf
 
-Ubuntu 14.04 only installs ``unpaper`` version 0.4.2, which is not supported by OCRmyPDF because it is produces invalid output. This program is an optional dependency, and provides page deskewing and cleaning. See `Dockerfile <Dockerfile>`_ for an example of how to building unpaper 6.1 from source. If you choose to install unpaper later, OCRmyPDF will use the foremost version on the system PATH.
+These installation instructions omit the optional dependency ``unpaper``, which is only available at version 0.4.2 in Ubuntu 14.04. The author could not find a backport of ``unpaper`` and is not motivated to figure how to set up a Ubuntu PPA to distribute it. You can create a .deb package to do the job of installing unpaper 6.1 (for x86 64-bit only):
+
+.. code-block:: bash
+
+   wget -q https://dl.dropboxusercontent.com/u/28971240/unpaper_6.1-1.deb -O unpaper_6.1-1.deb
+   sudo dpkg -i unpaper_6.1-1.deb
+
 
 Installing on Windows
 ~~~~~~~~~~~~~~~~~~~~~
@@ -287,7 +300,7 @@ to not install as superuser, you can install the package in a Python virtual env
 .. code-block:: bash
 
    git clone -b master https://github.com/jbarlow83/OCRmyPDF.git
-   pyvenv venv
+   python3 -m venv
    source venv/bin/activate
    cd OCRmyPDF
    pip3 install .

RELEASE_NOTES.rst

@@ -1,10 +1,39 @@
 RELEASE NOTES
 =============
 
-Please always read this file before installing the package
 
-Download software here: https://github.com/jbarlow83/OCRmyPDF/tags
+v4.2:
+=====
 
+-  ocrmypdf will now try to convert single image files to PDFs if they are provided as input (#15)
+
+   +  This is a basic convenience feature. It only supports a single image and always makes the image fill the whole page.
+   +  For better control over image to PDF conversion, use ``img2pdf`` (one of ocrmypdf's dependencies)
+
+-  New argument ``--output-type {pdf|pdfa}`` allows disabling Ghostscript PDF/A generation
+
+   +  ``pdfa`` is the default, consistent with past behavior
+   +  ``pdf`` provides a workaround for users concerned about the increase in file size from Ghostscript forcing JBIG2 images to CCITT and transcoding JPEGs
+   +  ``pdf`` preserves as much as it can about the original file, including problems that PDF/A conversion fixes
+
+-  PDFs containing images with "non-square" pixel aspect ratios, such as 200x100 DPI, are now handled and converted properly (fixing a bug that caused to be cropped)
+-  ``--force-ocr`` rasterizes pages even if they contain no images
+
+   +  supports users who want to use OCRmyPDF to reconstruct text information in PDFs with damaged Unicode maps (copy and paste text does not match displayed text)
+   +  supports reinterpreting PDFs where text was rendered as curves for printing, and text needs to be recovered
+   +  fixes issue #82
+
+-  Fixes an issue where, with certain settings, monochrome images in PDFs would be converted to 8-bit grayscale, increasing file size (#79)
+-  Support for Ubuntu 12.04 LTS "precise" has been dropped in favor of (roughly) Ubuntu 14.04 LTS "trusty" 
+
+   +  Some Ubuntu "PPAs" (backports) are needed to make it work
+      
+-  Support for some older dependencies dropped
+
+   +  Ghostscript 9.15 or later is now required (available in Ubuntu trusty with backports)
+   +  Tesseract 3.03 or later is now required (available in Ubuntu trusty)
+
+-  Ghostscript now runs in "safer" mode where possible
 
 v4.1.4:
 =======
@@ -52,9 +81,6 @@ v4.0.6:
 v4.0.5:
 =======
 
-Fixes
------
-
 -  Remove some verbose debug messages from v4.0.4
 -  Fixed temporary that wasn't being deleted
 -  DPI is now calculated correctly for cropped images, along with other image transformations
@@ -78,7 +104,7 @@ New features
 Fixes
 -----
 
--  Show stack trace if unexpect errors occur
+-  Show stack trace if unexpected errors occur
 -  Treat "too few characters" error message from Tesseract as a reason to skip that page rather than
    abort the file
 -  Docker: fix blank JPEG2000 issue by insisting on Ghostscript versions that have this fixed
@@ -351,363 +377,6 @@ Notes and known issues
 v2.2-stable (2014-09-29):
 =========================
 
-New features
-------------
+OCRmyPDF versions 1 and 2 were implemented as shell scripts. OCRmyPDF 3.0+ is a fork that gradually replaced all shell scripts with Python while maintaining the existing command line arguments. No one is maintaining old versions.
 
-- None
-
-Changes
--------
-
-- Update to jhove v1.11
-- Request the python library reportlab v3.0 or newer (So that we could remove a patch to the previous version of reportlab leading to issues for some users)
-
-Fixes
------
-
-- Fix bug on Mac OS X (resolution of simlink to OCRmyPDF.sh script) (thanks to jbarlow83)
-- Check if the input pdf file exists before to continue
-
-Tested with
------------
-
-- Operating system: FreeBSD 9.2
-- Dependencies:
-
-   - parallel 20140822
-   - poppler-utils 0.24.5
-   - ImageMagick 6.8.9-4 2014-09-17
-   - Unpaper 0.3
-   - tesseract 3.02.02
-   - Python 2.7.8
-   - ghostcript (gs): 9.06
-   - java: openjdk version "1.7.0_65"
-
-
-v2.1-stable (2014-09-20):
-=========================
-
-New features
-------------
-
--  None
-
-Changes
--------
-
--  None
-
-Fixes
------
-
--  Allow execution via simlink
--  Add support for tesseract 3.03
--  Add support for newer version of reportlab
--  Lowered minimum version of gnu parallel
--  Various typo
-
-Tested with
------------
-
--  Operating system: FreeBSD 9.1
--  Dependencies:
--  parallel 20130222
--  poppler-utils 0.22.2
--  ImageMagick 6.8.0-7 2013-03-30
--  Unpaper 0.3
--  tesseract 3.02.02
--  Python 2.7.3
--  ghoscript (gs): 9.06
--  java: openjdk version "1.7.0\_17"
-
-v2.0-stable (2014-01-25):
-=========================
-
-New features
-------------
-
--  Check if the language(s) passed using the -l option is supported by
-   tesseract (fixes #60)
-
-Changes
--------
-
--  Allow OCRmyPDF to be used with tesseract 3.02.01, even though OCR
-   might fail for few PDF file (see issue #28). Rationale: For some
-   linux distribution, no newer version than tesseract 3.02.01 is
-   available
-
-Fixes
------
-
--  More robust algorithm for checking the version of the installed
-   tesseract package
-
-Tested with
------------
-
--  Operating system: FreeBSD 9.1
--  Dependencies:
--  parallel 20130222
--  poppler-utils 0.22.2
--  ImageMagick 6.8.0-7 2013-03-30
--  Unpaper 0.3
--  tesseract 3.02.02
--  Python 2.7.3
--  ghoscript (gs): 9.06
--  java: openjdk version "1.7.0\_17"
-
-v2.0-rc2 (2014-01-16):
-======================
-
-New features
-------------
-
--  None
-
-Changes
--------
-
--  Size reduction of final PDF file: (fixes #50)
--  Support for monochrome (Black&White) images (massive size reduction
-   in final PDF: >80%)
--  Reduced size of grayscale images (by 13% on test PDF file)
--  Preventing fi, fl ligatures does not require anymore to pass an
-   additional config file to tesseract using the -C option (fixes #58)
--  Location of temporary folder according to content of environment
-   variable TMPDIR.
--  Dependency to pdftk removed
--  Check for compatible versions of dependencies: (fixes #51)
--  parallel and tesseract
--  python libraries reportlab and lxml
-
-Fixes
------
-
--  Improved portability with various shells (dash, bash, tcsh) and OS
-   (FreeBSD, MAC OSX, Linux) (fixes #59)
--  Corrected bug in case the input PDF file contains a space character
-   (fixes #48)
--  Prevent spurious error message in case there is no image in a PDF
-   page
--  Prevent collision of temporary folder names (fixes #57)
-
-Tested with
------------
-
--  Operating system: FreeBSD 9.1
--  Dependencies:
--  parallel 20130222
--  poppler-utils 0.22.2
--  ImageMagick 6.8.0-7 2013-03-30
--  Unpaper 0.3
--  tesseract 3.02.02
--  Python 2.7.3
--  ghoscript (gs): 9.06
--  java: openjdk version "1.7.0\_17"
-
-v2.0-rc1 (2014-01-07):
-======================
-
-New features
-------------
-
--  Huge performance improvement on machines having multiple CPU/cores
-   (processing of several pages concurrently) (fixes #18)
--  By default prevent from processing a PDF file already containing
-   fonts (i.e. text)(it can be overridden with the -f flag) (fixes #16)
--  Warn if the resolution is too low to get reasonable OCR results
-   (fixes #37)
--  New option (-o) to perform automatic oversampling if the image
-   resolution is too low. This can improve OCR results.
--  Warn if using a tesseract version older than v3.02.02 (as older
-   versions are known to produce invalid output) (fixes #41)
--  Echo version of the installed dependencies (e.g. tesseract) in debug
-   mode in order to ease support (fixes #35)
--  Echo the arguments passed to the script in debug mode to ease support
-
-Changes
--------
-
--  In debug mode: The debug page is now placed after the respective
-   "normal" page
--  Reduced disk space usage in temporary folder if -d (deskew) or -c
-   (cleanup) options are not selected
--  New file src/config.sh containing various configuration parameters
--  Documentation of the tesseract config file "tess-cfg/no\_ligature"
-   improved
--  Improved consistency of the temporary file names
-
-Fixes
------
-
--  Improved robustness:
--  in case vertical resolution differs from horizontal resolution (fixes
-   #38)
--  in case a PDF page contains more than one image (fixes #36)
--  Fix a problem occurring if python 3 is the standard interpreter
-   (fixes #33)
--  Fix a problem occurring if the input PDF file contains special
-   characters like "#" (fixes #34)
-
-Tested with
------------
-
--  Operating system: FreeBSD 9.1
--  Dependencies:
--  parallel 20130222
--  poppler-utils 0.22.2
--  ImageMagick 6.8.0-7 2013-03-30
--  Unpaper 0.3
--  tesseract 3.02.02
--  Python 2.7.3
--  pdftk 1.45
--  ghoscript (gs): 9.06
--  java: openjdk version "1.7.0\_17"
-
-v1.1-stable (2014-01-06):
-=========================
-
-New features
-------------
-
--  N/A
-
-Changes
--------
-
--  N/A
-
-Fixes
------
-
--  Fixed syntax error (bashism) leading to an error message on certain
-   systems (fixes #42)
-
-Tested with
------------
-
--  Operating system: FreeBSD 9.1
--  Dependencies:
--  poppler-utils 0.22.2
--  ImageMagick 6.8.0-7 2013-03-30
--  Unpaper 0.3
--  tesseract 3.02.02
--  Python 2.7.3
--  pdftk 1.45
--  ghoscript (gs): 9.06
--  java: openjdk version "1.7.0\_17"
-
-v1.0-stable (2013-05-06):
-=========================
-
-New features
-------------
-
--  In debug mode: compute and echo time required for processing (fixes
-   #26)
-
-Changes
--------
-
--  Removed feature to add metadata in final pdf file (because it lead to
-   to final PDF file that does not comply to the PDF/A-1 format)
--  Removed feature to set same owner & permissions in final PDF file
-   than in input file
--  Removed many unused jhove files (e.g. documentation, \*.java and
-   \*.class files)
-
-Fixes
------
-
--  Correction to handle correctly path and input PDF files having spaces
-   (fixes #31)
--  Resolutions (x/y) that are nearly equal are now supported (fixes #25)
--  Fix compatibility issue with Ubuntu server 12.04 / Ubuntu server
-   10.04 / Linux Mint 13 Maya and probably other Linux distributions
-   (fixes #27)
--  Commit missing jhove files (\*.jar mainly) due to wrong .gitignore
-
-Tested with
------------
-
--  Operating system: FreeBSD 9.1
--  Dependencies:
--  poppler-utils 0.22.2
--  ImageMagick 6.8.0-7 2013-03-30
--  Unpaper 0.3
--  tesseract 3.02.02
--  Python 2.7.3
--  pdftk 1.45
--  ghoscript (gs): 9.06
--  java: openjdk version "1.7.0\_17"
-
-v1.0-rc2 (2013-04-29):
-======================
-
-New features
-------------
-
--  Keep temporary files if debug mode is set (fixes #22)
--  Set same owner & permissions in final PDF file than in input file
-   (fixes #9)
--  Added metadata in final pdf file (fixes #4)
-
-Changes
--------
-
--  N/A
-
-Fixes
------
-
--  Fixed wrong image cropping when deskew option is activated
--  Exit with error message if page size is not found in hocr file (fixes
-   #21)
--  Various minor fixes in log messages
-
-Tested with
------------
-
--  Operating system: FreeBSD 9.1
--  Dependencies:
--  poppler-utils 0.22.2
--  ImageMagick 6.8.0-7 2013-03-30
--  Unpaper 0.3
--  tesseract 3.02.02
--  Python 2.7.3
--  pdftk 1.45
--  ghoscript (gs): 9.06
--  java: openjdk version "1.7.0\_17"
-
-v1.0-rc1 (2013-04-26):
-======================
-
-New features
-------------
-
--  First release candidate
-
-Changes
--------
-
--  N/A
-
-Fixes
------
-
--  N/A
-
-Tested with
------------
-
--  Operating system: FreeBSD 9.1
--  Dependencies:
--  poppler-utils 0.22.2
--  ImageMagick 6.8.0-7 2013-03-30
--  Unpaper 0.3
--  tesseract 3.02.02
--  Python 2.7.3
--  pdftk 1.45
--  ghoscript (gs): 9.06
--  java: openjdk version "1.7.0\_17"
+For details on older versions, see the `final version of its release notes <https://github.com/fritz-hh/OCRmyPDF/blob/7fd3dbdf42ca53a619412ce8add7532c5e81a9d1/RELEASE_NOTES.md>`_.

ocrmypdf/__init__.py

@@ -1,5 +1,6 @@
 from enum import IntEnum
 import os
+from collections.abc import Iterable
 
 
 class ExitCode(IntEnum):
@@ -7,12 +8,13 @@ class ExitCode(IntEnum):
     bad_args = 1
     input_file = 2
     missing_dependency = 3
-    invalid_output_pdfa = 4
+    invalid_output_pdf = 4
     file_access_error = 5
     already_done_ocr = 6
     child_process_error = 7
     encrypted_pdf = 8
     other_error = 15
+    ctrl_c = 130
 
 
 def get_program(name):
@@ -22,3 +24,7 @@ def get_program(name):
 
 def page_number(input_file):
     return int(os.path.basename(input_file)[0:6])
+
+
+def is_iterable_notstr(thing):
+    return isinstance(thing, Iterable) and not isinstance(thing, str)

ocrmypdf/ghostscript.py

@@ -14,6 +14,7 @@ def rasterize_pdf(input_file, output_file, xres, yres, raster_device, log,
         args_gs = [
             get_program('gs'),
             '-dQUIET',
+            '-dSAFER',
             '-dBATCH',
             '-dNOPAUSE',
             '-sDEVICE=%s' % raster_device,

ocrmypdf/leptonica.py

@@ -362,7 +362,8 @@ class Pix:
         if get_leptonica_version() < 'leptonica-1.72':
             # Older versions of Leptonica (pre-1.72) have a buggy
             # implementation of pixCorrelationBinary that overflows on larger
-            # images.
+            # images.  Ubuntu trusty has 1.70. Ubuntu PPA
+            # ppa:rebuntu16/avidemux+unofficial has "leptonlib" 1.73.
             pix1_count = ffi.new('l_int32 *')
             pix2_count = ffi.new('l_int32 *')
             pixn_count = ffi.new('l_int32 *')

ocrmypdf/main.py

@@ -27,11 +27,12 @@ import ruffus.proxy_logger as proxy_logger
 
 from .hocrtransform import HocrTransform
 from .pageinfo import pdf_get_all_pageinfo
-from .pdfa import generate_pdfa_def
+from .pdfa import generate_pdfa_def, file_claims_pdfa
 from . import ghostscript
 from . import tesseract
 from . import qpdf
-from . import ExitCode, page_number
+from . import ExitCode, page_number, is_iterable_notstr
+from collections.abc import Sequence
 
 import pkg_resources
 
@@ -42,6 +43,8 @@ warnings.simplefilter('ignore', pypdf.utils.PdfReadWarning)
 
 BASEDIR = os.path.dirname(os.path.realpath(__file__))
 
+VECTOR_PAGE_DPI = 400
+
 
 # -------------
 # External dependencies
@@ -120,6 +123,16 @@ parser.add_argument(
 parser.add_argument(
     '-j', '--jobs', metavar='N', type=int,
     help="Use up to N CPU cores simultaneously (default: use all)")
+parser.add_argument(
+    '--image-dpi', metavar='DPI', type=int,
+    help="for input image instead of PDF, use this DPI instead of file's")
+parser.add_argument(
+    '--output-type', choices=['pdfa', 'pdf'], default='pdfa',
+    help="Choose output type. 'pdfa' creates a PDF/A-2b compliant file for "
+         "long term archiving (default, recommended) but may not suitable "
+         "for users who want their file altered as little as possible. 'pdfa' "
+         "also has problems with full Unicode text. 'pdf' attempts to "
+         "preserve file contents as much as possible.")
 
 metadata = parser.add_argument_group(
     "Metadata options",
@@ -269,6 +282,13 @@ if options.clean and not options.clean_final \
         "Tesseract PDF renderer cannot render --clean pages without "
         "also performing --clean-final, so --clean-final is assumed.")
 
+if set(options.language) & {'chi_sim', 'chi_tra'} \
+        and (options.pdf_renderer == 'hocr' or options.output_type == 'pdfa'):
+    complain(
+        "Your settings are known to cause problems with OCR of Chinese text. "
+        "Try adding these arguments: "
+        "    ocrmypdf --pdf-renderer tesseract --output-type pdf")
+
 lossless_reconstruction = False
 if options.pdf_renderer == 'hocr':
     if not options.deskew and not options.clean_final and not options.force_ocr:
@@ -384,10 +404,87 @@ def cleanup_working_files(*args):
             shutil.rmtree(work_folder)
 
 
+def triage_image_file(input_file, output_file, log):
+    try:
+        log.info("Input file is not a PDF, checking if it is an image...")
+        im = Image.open(input_file)
+    except EnvironmentError as e:
+        log.error(e)
+        sys.exit(ExitCode.input_file)
+        return
+    else:
+        log.info("Input file is an image")
+
+        if 'dpi' in im.info:
+            if im.info['dpi'] <= (96, 96) and not options.image_dpi:
+                log.info("Image size: (%d, %d)" % im.size)
+                log.info("Image resolution: (%d, %d)" % im.info['dpi'])
+                log.error(
+                    "Input file is an image, but the resolution (DPI) is "
+                    "not credible.  Estimate the resolution at which the "
+                    "image was scanned and specify it using --image-dpi.")
+                sys.exit(ExitCode.input_file)
+        elif not options.image_dpi:
+            log.info("Image size: (%d, %d)" % im.size)
+            log.error(
+                "Input file is an image, but has no resolution (DPI) "
+                "in its metadata.  Estimate the resolution at which "
+                "image was scanned and specify it using --image-dpi.")
+            sys.exit(ExitCode.input_file)
+
+        if 'iccprofile' not in im.info:
+            if im.mode == 'RGB':
+                log.info('Input image has no ICC profile, assuming sRGB')
+            elif im.mode == 'CMYK':
+                log.info('Input CMYK image has no ICC profile, not usable')
+                sys.exit(ExitCode.input_file)
+        im.close()
+
+    try:
+        log.info("Image seems valid. Try converting to PDF...")
+        layout_fun = img2pdf.default_layout_fun
+        if options.image_dpi:
+            layout_fun = img2pdf.get_fixed_dpi_layout_fun(
+                (options.image_dpi, options.image_dpi))
+        with open(output_file, 'wb') as outf:
+            img2pdf.convert(
+                input_file,
+                layout_fun=layout_fun,
+                with_pdfrw=False,
+                outputstream=outf)
+        log.info("Successfully converted to PDF, processing...")
+    except img2pdf.ImageOpenError as e:
+        log.error(e)
+        sys.exit(ExitCode.input_file)
+
+
 @transform(
     input=options.input_file,
-    filter=formatter('(?i)\.pdf'),
-    output=os.path.join(work_folder, '{basename[0]}.repaired.pdf'),
+    filter=formatter('(?i)'),
+    output=os.path.join(work_folder, '{basename[0]}.pdf'),
+    extras=[_log])
+def triage(
+        input_file,
+        output_file,
+        log):
+    try:
+        with open(input_file, 'rb') as f:
+            signature = f.read(4)
+            if signature == b'%PDF':
+                re_symlink(input_file, output_file)
+                return
+    except EnvironmentError as e:
+        log.error(e)
+        sys.exit(ExitCode.input_file)
+
+    triage_image_file(input_file, output_file, log)
+
+
+@transform(
+    input=triage,
+    filter=suffix('.pdf'),
+    output='.repaired.pdf',
+    output_dir=work_folder,
     extras=[_log, _pdfinfo, _pdfinfo_lock])
 def repair_pdf(
         input_file,
@@ -409,31 +506,60 @@ def get_pageinfo(input_file, pdfinfo, pdfinfo_lock):
     return pageinfo
 
 
+def get_page_dpi(pageinfo):
+    "Get the DPI when nonsquare DPI is tolerable"
+    xres = max(pageinfo.get('xres', VECTOR_PAGE_DPI), options.oversample or 0)
+    yres = max(pageinfo.get('yres', VECTOR_PAGE_DPI), options.oversample or 0)
+    return (float(xres), float(yres))
+
+
+def get_page_square_dpi(pageinfo):
+    "Get the DPI when we require xres == yres"
+    return float(max(
+        pageinfo.get('xres', VECTOR_PAGE_DPI),
+        pageinfo.get('yres', VECTOR_PAGE_DPI),
+        options.oversample or 0))
+
+
 def is_ocr_required(pageinfo, log):
     page = pageinfo['pageno'] + 1
     ocr_required = True
     if not pageinfo['images']:
-        # If the page has no images, then it contains vector content or text
-        # or both. It seems quite unlikely that one would find meaningful text
-        # from rasterizing vector content. So skip the page.
-        log.info(
-            "{0:4d}: page has no images - skipping OCR".format(page)
-        )
-        ocr_required = False
+        if options.force_ocr and options.oversample:
+            # The user really wants to reprocess this file
+            log.info(
+                "{0:4d}: page has no images - "
+                "rasterizing at {1} DPI because "
+                "--force-ocr --oversample was specified".format(
+                    page, options.oversample))
+        elif options.force_ocr:
+            # Warn the user they might not want to do this
+            log.warning(
+                "{0:4d}: page has no images - "
+                "all vector content will be "
+                "rasterized at {1} DPI, losing some resolution and likely "
+                "increasing file size. Use --oversample to adjust the "
+                "DPI.".format(page, VECTOR_PAGE_DPI))
+        else:
+            log.info(
+                "{0:4d}: page has no images - "
+                "skipping all processing on this page".format(page))
+            ocr_required = False
+
     elif pageinfo['has_text']:
-        s = "{0:4d}: page already has text! – {1}"
+        msg = "{0:4d}: page already has text! – {1}"
 
         if not options.force_ocr and not options.skip_text:
-            log.error(s.format(page,
-                               "aborting (use --force-ocr to force OCR)"))
+            log.error(msg.format(page,
+                                 "aborting (use --force-ocr to force OCR)"))
             sys.exit(ExitCode.already_done_ocr)
         elif options.force_ocr:
-            log.info(s.format(page,
-                              "rasterizing text and running OCR anyway"))
+            log.info(msg.format(page,
+                                "rasterizing text and running OCR anyway"))
             ocr_required = True
         elif options.skip_text:
-            log.info(s.format(page,
-                              "skipping all processing on this page"))
+            log.info(msg.format(page,
+                                "skipping all processing on this page"))
             ocr_required = False
 
     if ocr_required and options.skip_big:
@@ -452,12 +578,17 @@ def is_ocr_required(pageinfo, log):
     os.path.join(work_folder, '*.page.pdf'),
     extras=[_log, _pdfinfo, _pdfinfo_lock])
 def split_pages(
-        input_file,
+        input_files,
         output_files,
         log,
         pdfinfo,
         pdfinfo_lock):
 
+    if is_iterable_notstr(input_files):
+        input_file = input_files[0]
+    else:
+        input_file = input_files
+
     for oo in output_files:
         with suppress(FileNotFoundError):
             os.unlink(oo)
@@ -606,11 +737,14 @@ def rasterize_with_ghostscript(
             device = 'pnggray'
 
     log.debug("Rasterize {0} with {1}".format(
-            os.path.basename(input_file), device))
-    xres = max(pageinfo['xres'], options.oversample or 0)
-    yres = max(pageinfo['yres'], options.oversample or 0)
+              os.path.basename(input_file), device))
 
-    ghostscript.rasterize_pdf(input_file, output_file, xres, yres, device, log)
+    # Produce the page image with square resolution or else deskew and OCR
+    # will not work properly
+    dpi = get_page_square_dpi(pageinfo)
+    ghostscript.rasterize_pdf(
+        input_file, output_file, xres=dpi, yres=dpi, raster_device=device,
+        log=log)
 
 
 @transform(
@@ -630,7 +764,7 @@ def preprocess_deskew(
         return
 
     pageinfo = get_pageinfo(input_file, pdfinfo, pdfinfo_lock)
-    dpi = int(pageinfo['xres'])
+    dpi = get_page_square_dpi(pageinfo)
 
     from . import leptonica
     leptonica.deskew(input_file, output_file, dpi)
@@ -653,7 +787,7 @@ def preprocess_clean(
         return
 
     pageinfo = get_pageinfo(input_file, pdfinfo, pdfinfo_lock)
-    dpi = int(pageinfo['xres'])
+    dpi = get_page_square_dpi(pageinfo)
 
     unpaper.clean(input_file, output_file, dpi, log)
 
@@ -704,13 +838,20 @@ def select_image_for_pdf(
     image = next(ii for ii in infiles if ii.endswith(image_suffix))
 
     pageinfo = get_pageinfo(image, pdfinfo, pdfinfo_lock)
-    if all(image['enc'] == 'jpeg' for image in pageinfo['images']):
+    if all(orig_image['enc'] == 'jpeg' for orig_image in pageinfo['images']):
         # If all images were JPEGs originally, produce a JPEG as output
         im = Image.open(image)
-        dpi = im.info.get(
-            'dpi',
-            (int(pageinfo['xres']), int(pageinfo['yres']))
-        )
+
+        # At this point the image should be a .png, but deskew, unpaper might
+        # have removed the DPI information. In this case, fall back to square
+        # DPI used to rasterize. When the preview image was rasterized, it
+        # was also converted to square resolution, which is what we want to
+        # give tesseract, so keep it square.
+        fallback_dpi = get_page_square_dpi(pageinfo)
+        dpi = im.info.get('dpi', (fallback_dpi, fallback_dpi))
+
+        # Pillow requires integer DPI
+        dpi = round(dpi[0]), round(dpi[1])
         im.save(output_file, format='JPEG', dpi=dpi)
     else:
         re_symlink(image, output_file)
@@ -738,18 +879,16 @@ def select_image_layer(
         re_symlink(page_pdf, output_file)
     else:
         pageinfo = get_pageinfo(image, pdfinfo, pdfinfo_lock)
-        dpi = round(max(pageinfo['xres'], pageinfo['yres'],
-                        options.oversample))
-        imgsize = ((img2pdf.ImgSize.dpi, dpi), (img2pdf.ImgSize.dpi, dpi))
-
-        layout_fun = img2pdf.get_layout_fun(None, imgsize, None, None, None)
+        dpi = get_page_dpi(pageinfo)
+        dpi = float(dpi[0]), float(dpi[1])
+        layout_fun = img2pdf.get_fixed_dpi_layout_fun(dpi)
 
         with open(image, 'rb') as imfile, \
                 open(output_file, 'wb') as pdf:
             rawdata = imfile.read()
-            pdf.write(img2pdf.convert(
-                rawdata, producer="img2pdf", with_pdfrw=False,
-                layout_fun=layout_fun))
+            img2pdf.convert(
+                rawdata, with_pdfrw=False,
+                layout_fun=layout_fun, outputstream=pdf)
 
 
 @active_if(options.pdf_renderer == 'hocr')
@@ -766,7 +905,7 @@ def render_hocr_page(
         pdfinfo_lock):
     hocr = input_file
     pageinfo = get_pageinfo(hocr, pdfinfo, pdfinfo_lock)
-    dpi = round(max(pageinfo['xres'], pageinfo['yres'], options.oversample))
+    dpi = get_page_square_dpi(pageinfo)
 
     hocrtransform = HocrTransform(hocr, dpi)
     hocrtransform.to_pdf(output_file, imageFileName=None,
@@ -790,7 +929,7 @@ def render_hocr_debug_page(
     image = next(ii for ii in infiles if ii.endswith('.image'))
 
     pageinfo = get_pageinfo(image, pdfinfo, pdfinfo_lock)
-    dpi = round(max(pageinfo['xres'], pageinfo['yres'], options.oversample))
+    dpi = get_page_square_dpi(pageinfo)
 
     hocrtransform = HocrTransform(hocr, dpi)
     hocrtransform.to_pdf(output_file, imageFileName=None,
@@ -840,7 +979,7 @@ def add_text_layer(
     y2 = page_image.mediaBox.getUpperRight_y()
 
     # Rotation occurs about the page's (0, 0). Most pages will have the media
-    # box at (0, 0) will all content in the first quadrant but some cropped
+    # box at (0, 0) with all content in the first quadrant but some cropped
     # files may have an offset mediabox. We translate the page so that its
     # bottom left corner after rotation is pinned to (0, 0) with the image
     # in the first quadrant.
@@ -866,7 +1005,6 @@ def add_text_layer(
         if 'writeToStream' in str(e) or 'invalid literal' in str(e):
             raise PdfMergeFailedError() from e
 
-
     pdf_output = pypdf.PdfFileWriter()
     pdf_output.addPage(page_text)
 
@@ -905,6 +1043,40 @@ def tesseract_ocr_and_render_pdf(
         log=log)
 
 
+def get_pdfmark(base_pdf):
+    def from_document_info(key):
+        # pdf.documentInfo.get() DOES NOT behave as expected for a dict-like
+        # object, so call with precautions.  TypeError may occur if the PDF
+        # is missing the optional document info section.
+        try:
+            s = base_pdf.documentInfo[key]
+            return str(s)
+        except (KeyError, TypeError):
+            return ''
+
+    pdfmark = {
+        '/Title': from_document_info('/Title'),
+        '/Author': from_document_info('/Author'),
+        '/Keywords': from_document_info('/Keywords'),
+        '/Subject': from_document_info('/Subject'),
+    }
+    if options.title:
+        pdfmark['/Title'] = options.title
+    if options.author:
+        pdfmark['/Author'] = options.author
+    if options.keywords:
+        pdfmark['/Keywords'] = options.keywords
+    if options.subject:
+        pdfmark['/Subject'] = options.subject
+
+    pdfmark['/Creator'] = '{0} {1} / Tesseract OCR{2} {3}'.format(
+            parser.prog, VERSION,
+            '+PDF' if options.pdf_renderer == 'tesseract' else '',
+            tesseract.version())
+    return pdfmark
+
+
+@active_if(options.output_type == 'pdfa')
 @transform(
     input=repair_pdf,
     filter=formatter(r'\.repaired\.pdf'),
@@ -916,37 +1088,7 @@ def generate_postscript_stub(
         log):
 
     pdf = pypdf.PdfFileReader(input_file)
-
-    def from_document_info(key):
-        # pdf.documentInfo.get() DOES NOT behave as expected for a dict-like
-        # object, so call with precautions.  TypeError may occur if the PDF
-        # is missing the optional document info section.
-        try:
-            s = pdf.documentInfo[key]
-            return str(s)
-        except (KeyError, TypeError):
-            return ''
-
-    pdfmark = {
-        'title': from_document_info('/Title'),
-        'author': from_document_info('/Author'),
-        'keywords': from_document_info('/Keywords'),
-        'subject': from_document_info('/Subject'),
-    }
-    if options.title:
-        pdfmark['title'] = options.title
-    if options.author:
-        pdfmark['author'] = options.author
-    if options.keywords:
-        pdfmark['keywords'] = options.keywords
-    if options.subject:
-        pdfmark['subject'] = options.subject
-
-    pdfmark['creator'] = '{0} {1} / Tesseract OCR{2} {3}'.format(
-            parser.prog, VERSION,
-            '+PDF' if options.pdf_renderer == 'tesseract' else '',
-            tesseract.version())
-
+    pdfmark = get_pdfmark(pdf)
     generate_pdfa_def(output_file, pdfmark)
 
 
@@ -967,12 +1109,13 @@ def skip_page(
     re_symlink(input_file, output_file, log)
 
 
+@active_if(options.output_type == 'pdfa')
 @merge(
     input=[add_text_layer, render_hocr_debug_page, skip_page,
            tesseract_ocr_and_render_pdf, generate_postscript_stub],
     output=os.path.join(work_folder, 'merged.pdf'),
     extras=[_log, _pdfinfo, _pdfinfo_lock])
-def merge_pages(
+def merge_pages_ghostscript(
         input_files,
         output_file,
         log,
@@ -997,26 +1140,66 @@ def merge_pages(
     ghostscript.generate_pdfa(pdf_pages, output_file, options.jobs or 1)
 
 
-@transform(
-    input=merge_pages,
-    filter=formatter(),
-    output=options.output_file,
+@active_if(options.output_type == 'pdf')
+@merge(
+    input=[add_text_layer, render_hocr_debug_page, skip_page,
+           tesseract_ocr_and_render_pdf, repair_pdf],
+    output=os.path.join(work_folder, 'merged.pdf'),
     extras=[_log, _pdfinfo, _pdfinfo_lock])
-def copy_final(
-        input_file,
+def merge_pages_qpdf(
+        input_files,
         output_file,
         log,
         pdfinfo,
         pdfinfo_lock):
+
+    metadata_file = next(
+        (ii for ii in input_files if ii.endswith('.repaired.pdf')))
+    input_files.remove(metadata_file)
+
+    def input_file_order(s):
+        '''Sort order: All rendered pages followed
+        by their debug page.'''
+        key = int(os.path.basename(s)[0:6]) * 10
+        if 'debug' in os.path.basename(s):
+            key += 1
+        return key
+
+    pdf_pages = sorted(input_files, key=input_file_order)
+    log.debug("Final pages: " + "\n".join(pdf_pages))
+
+    reader_metadata = pypdf.PdfFileReader(metadata_file)
+    pdfmark = get_pdfmark(reader_metadata)
+    pdfmark['/Producer'] = 'qpdf ' + qpdf.version()
+
+    first_page = pypdf.PdfFileReader(pdf_pages[0])
+
+    writer = pypdf.PdfFileWriter()
+    writer.appendPagesFromReader(first_page)
+    writer.addMetadata(pdfmark)
+    writer_file = pdf_pages[0].replace('.pdf', '.metadata.pdf')
+    with open(writer_file, 'wb') as f:
+        writer.write(f)
+
+    pdf_pages[0] = writer_file
+
+    qpdf.merge(pdf_pages, output_file)
+
+
+@merge(
+    input=[merge_pages_ghostscript, merge_pages_qpdf],
+    output=options.output_file,
+    extras=[_log, _pdfinfo, _pdfinfo_lock])
+def copy_final(
+        input_files,
+        output_file,
+        log,
+        pdfinfo,
+        pdfinfo_lock):
+    input_file = next((ii for ii in input_files if ii.endswith('.pdf')))
     shutil.copy(input_file, output_file)
 
 
-def validate_pdfa(
-        input_file,
-        log):
-    return qpdf.check(input_file, log)
-
-
 def available_cpu_count():
     try:
         return multiprocessing.cpu_count()
@@ -1062,6 +1245,9 @@ def do_ruffus_exception(ruffus_five_tuple):
             _log.error("Input file '{0}' is not a valid PDF".format(
                 options.input_file))
             return ExitCode.input_file
+    elif exc_name == 'builtins.KeyboardInterrupt':
+        _log.error("Interrupted by user")
+        return ExitCode.ctrl_c
     elif exc_name == 'subprocess.CalledProcessError':
         # It's up to the subprocess handler to report something useful
         msg = "Error occurred while running this command:"
@@ -1091,21 +1277,23 @@ def do_ruffus_exception(ruffus_five_tuple):
 
             """))
         return ExitCode.encrypted_pdf
-    elif not options.verbose:
+
+    if not options.verbose:
         _log.error(exc_stack)
-        return ExitCode.other_error
+    return ExitCode.other_error
 
 
-def traverse_ruffus_exception(e):
+def traverse_ruffus_exception(e_args):
     """Walk through a RethrownJobError and find the first exception.
 
     The exit code will be based on this, even if multiple exceptions occurred
     at the same time."""
 
-    if isinstance(e[0], str) and len(e) == 5:
-        return do_ruffus_exception(e)
-    elif hasattr(e, '__iter__'):
-        for exc in e:
+    if isinstance(e_args, Sequence) and isinstance(e_args[0], str) and \
+            len(e_args) == 5:
+        return do_ruffus_exception(e_args)
+    elif is_iterable_notstr(e_args):
+        for exc in e_args:
             return traverse_ruffus_exception(exc)
 
 
@@ -1146,9 +1334,17 @@ def run_pipeline():
         _log.error(e)
         return ExitCode.other_error
 
-    if not validate_pdfa(options.output_file, _log):
-        _log.warning('Output file: The generated PDF/A file is INVALID')
-        return ExitCode.invalid_output_pdfa
+    if options.output_type == 'pdfa':
+        pdfa_info = file_claims_pdfa(options.output_file)
+        if pdfa_info['pass']:
+            _log.info(pdfa_info['message'])
+        else:
+            _log.warning(pdfa_info['message'])
+            return ExitCode.invalid_output_pdf
+
+    if not qpdf.check(options.output_file, _log):
+        _log.warning('Output file: The generated PDF is INVALID')
+        return ExitCode.invalid_output_pdf
 
     with _pdfinfo_lock:
         _log.debug(_pdfinfo)

ocrmypdf/pdfa.py

@@ -7,6 +7,7 @@ from __future__ import print_function, absolute_import, division
 from string import Template
 import codecs
 import pkg_resources
+import PyPDF2 as pypdf
 
 ICC_PROFILE_RELPATH = 'data/sRGB.icc'
 
@@ -74,8 +75,14 @@ def encode_text_string(s: str) -> str:
     Postscript file to be completely ASCII and no escaping of Postscript
     characters is necessary.
     '''
+
+    # Sometimes lazy C programmer leave their NULs at the end of strings
+    # tests/resources/aspect.pdf is one example (created by ImageMagick)
+    s = s.replace('\x00', '')
+
     if s == '':
         return ''
+
     utf16_bytes = s.encode('utf-16be')
     ascii_hex_bytes = codecs.encode(b'\xfe\xff' + utf16_bytes, 'hex')
     ascii_hex_str = ascii_hex_bytes.decode('ascii').lower()
@@ -88,11 +95,11 @@ def _get_pdfa_def(icc_profile, icc_identifier, pdfmark):
     t = Template(pdfa_def_template)
     result = t.substitute(icc_profile=icc_profile,
                           icc_identifier=icc_identifier,
-                          title=pdfmark_utf16.get('title', ''),
-                          author=pdfmark_utf16.get('author', ''),
-                          subject=pdfmark_utf16.get('subject', ''),
-                          creator=pdfmark_utf16.get('creator', ''),
-                          keywords=pdfmark_utf16.get('keywords', ''))
+                          title=pdfmark_utf16.get('/Title', ''),
+                          author=pdfmark_utf16.get('/Author', ''),
+                          subject=pdfmark_utf16.get('/Subject', ''),
+                          creator=pdfmark_utf16.get('/Creator', ''),
+                          keywords=pdfmark_utf16.get('/Keywords', ''))
     return result
 
 
@@ -104,7 +111,48 @@ def generate_pdfa_def(target_filename, pdfmark, icc='sRGB'):
 
     ps = _get_pdfa_def(icc_profile, icc, pdfmark)
 
-    # Since PostScript might not handle UTF-8 (it's hard to get a clear
-    # answer), insist on ascii
+    # We should have encoded everything to pure ASCII by this point, and
+    # to be safe, only allow ASCII in PostScript
     with open(target_filename, 'w', encoding='ascii') as f:
         f.write(ps)
+
+
+def file_claims_pdfa(filename):
+    """Determines if the file claims to be PDF/A compliant
+
+    Checking if a file is a truly compliant PDF/A is a massive undertaking
+    that no open source tool does properly.  Some commercial tools are
+    generally reliable (Acrobat).
+
+    This checks if the XMP metadata contains a PDF/A marker.
+    """
+
+    pdf = pypdf.PdfFileReader(filename)
+    xmp = pdf.getXmpMetadata()
+
+    try:
+        pdfa_nodes = xmp.getNodesInNamespace(
+            aboutUri='',
+            namespace='http://www.aiim.org/pdfa/ns/id/')
+    except AttributeError:
+        return {'pass': False, 'output': 'pdf', 'message': 'No XMP metadata'}
+
+    pdfa_dict = {attr.localName: attr.value for attr in pdfa_nodes}
+    pdfa_dict['pass'] = False
+    pdfa_dict['output'] = 'pdf'
+    if pdfa_dict:
+        part_conformance = pdfa_dict['part'] + pdfa_dict['conformance']
+        valid_part_conforms = {'1A', '1B', '2A', '2B', '2U', '3A', '3B', '3U'}
+
+        message = 'File claims to be PDF/A-{}'.format(
+            part_conformance)
+
+        if part_conformance in valid_part_conforms:
+            pdfa_dict['pass'] = True
+            pdfa_dict['output'] = 'pdfa'
+        pdfa_dict['message'] = message
+    else:
+        pdfa_dict['message'] = 'File is a regular PDF'
+
+    return pdfa_dict
+

ocrmypdf/qpdf.py

@@ -2,12 +2,32 @@
 # © 2015 James R. Barlow: github.com/jbarlow83
 
 from subprocess import CalledProcessError, check_output, STDOUT, check_call
+from functools import lru_cache
 import sys
 import os
+import re
 
 from . import ExitCode, get_program
 
 
+@lru_cache(maxsize=1)
+def version():
+    args_qpdf = [
+        get_program('qpdf'),
+        '--version'
+    ]
+    try:
+        versions = check_output(
+            args_qpdf, close_fds=True, universal_newlines=True,
+            stderr=STDOUT)
+    except CalledProcessError:
+        print("Could not find qpdf executable on system PATH.")
+        sys.exit(ExitCode.missing_dependency)
+
+    qpdf_version = re.match(r'qpdf version (.+)', versions).group(1)
+    return qpdf_version
+
+
 def check(input_file, log):
     args_qpdf = [
         get_program('qpdf'),
@@ -20,7 +40,7 @@ def check(input_file, log):
     except CalledProcessError as e:
         if e.returncode == 2:
             log.error("{0}: not a valid PDF, and could not repair it.".format(
-                    input_file))
+                input_file))
             log.error("Details:")
             log.error(e.output)
         elif e.returncode == 3:
@@ -85,3 +105,15 @@ def split_pages(input_file, work_folder, npages):
             os.path.join(work_folder, '{0:06d}.page.pdf'.format(n + 1))
         ]
         check_call(args_qpdf)
+
+
+def merge(input_files, output_file):
+    """Merge the list of input files (all filenames) into the output file.
+
+    The input files may contain one or more pages.
+    """
+    args_qpdf = [
+        get_program('qpdf'), input_files[0], '--pages'
+    ] + input_files + ['--', output_file]
+    check_call(args_qpdf)
+

requirements.txt

@@ -1,6 +1,9 @@
+# requirements.txt can be used to replicate the developer's build environment
+# setup.py lists a separate set of requirements that are looser to simplify
+# installation
 ruffus==2.6.3
-Pillow==3.1.1
+Pillow==3.3.0
 reportlab==3.2.0
-PyPDF2==1.25.1
-img2pdf==0.2
+PyPDF2==1.26
+img2pdf==0.2.1
 cffi==1.5.2

setup.py

@@ -154,23 +154,23 @@ if command.startswith('install') or \
         command in ['check', 'test', 'nosetests', 'easy_install']:
     check_external_program(
         program='tesseract',
-        need_version='3.02.02',
+        need_version='3.03',  # limited by Travis CI / Ubuntu 12.04 backports
         package={'darwin': 'tesseract', 'linux': 'tesseract-ocr'}
     )
     check_external_program(
         program='gs',
-        need_version='9.14',
+        need_version='9.15',  # limited by Travis CI / Ubuntu 12.04 backports
         package='ghostscript'
     )
     check_external_program(
         program='unpaper',
-        need_version='6.1',
+        need_version='6.1',   # latest sane version
         package='unpaper',
         optional=True
     )
     check_external_program(
         program='qpdf',
-        need_version='5.0.0',
+        need_version='5.0.0',   # limited by Travis CI / Ubuntu 12.04 backports
         package='qpdf',
         version_check_args=['--version']
     )
@@ -217,12 +217,12 @@ setup(
         'ocrmypdf/lib/compile_leptonica.py:ffi'
     ],
     install_requires=[
-        'ruffus==2.6.3',
-        'Pillow>=3.0.0',
-        'reportlab>=3.1.44',
-        'PyPDF2>=1.25.1',
-        'img2pdf>=0.2.0',
-        'cffi>=1.5.0'
+        'ruffus==2.6.3',        # pinned - ocrmypdf implements a 2.6.3 workaround
+        'Pillow>=3.1.0',        # Pillow is pretty stable
+        'reportlab>=3.2.0',     # oldest released version with sane image handling
+        'PyPDF2>=1.26',         # pure Python, so track HEAD closely
+        'img2pdf>=0.2.1',       # pure Python, so track HEAD closely
+        'cffi>=1.5.0'           # oldest version ever tested
     ],
     tests_require=tests_require,
     entry_points={

tests/resources/README.rst

@@ -14,9 +14,9 @@ In some cases they were converted from one image format to another without other
 +=====================+================================================================================+
 | c02-22.pdf          | `Project Gutenberg`_, Adventures of Huckleberry Finn, page 22                  |
 +---------------------+--------------------------------------------------------------------------------+
-| congress.jpg        | `US Congressional Records`_                                                    |
+| congress.jpg        | `US Congressional Records`_ (Public Domain)                                    |
 +---------------------+--------------------------------------------------------------------------------+
-| graph.pdf           | `Wikimedia: Pandas text analysis.png`_                                         |
+| graph.pdf           | `Wikimedia: Pandas text analysis.png`_ (Public Domain)                         |
 +---------------------+--------------------------------------------------------------------------------+
 | lichtenstein.pdf    | `Wikimedia: JPEG2000 Lichtenstein`_ (Creative Commons BY-SA 3.0)               |
 +---------------------+--------------------------------------------------------------------------------+
@@ -31,6 +31,7 @@ Files generated for this project
 The following test resources were crafted specifically for this project, and can be used
 under the terms of the license in LICENSE.rst.
 
+- aspect.pdf (a page with an image with 200 x 100 DPI resolution)
 - blank.pdf (a blank PDF page)
 - cmyk.pdf (a CMYK image created in Photoshop)
 - enormous.pdf (a very lage page)

tests/test_main.py

@@ -12,6 +12,7 @@ from ocrmypdf.pageinfo import pdf_get_all_pageinfo
 import PyPDF2 as pypdf
 from ocrmypdf import ExitCode
 from ocrmypdf import leptonica
+from ocrmypdf.pdfa import file_claims_pdfa
 
 
 if sys.version_info.major < 3:
@@ -159,34 +160,34 @@ def test_deskew(spoof_tesseract_noop):
 
 
 def test_clean(spoof_tesseract_noop):
-    check_ocrmypdf('skew.pdf', 'test_clean.pdf', '-c', env=spoof_tesseract_noop)
+    check_ocrmypdf('skew.pdf', 'test_clean.pdf', '-c',
+                   env=spoof_tesseract_noop)
 
 
-@pytest.mark.parametrize("pdf,renderer", [
-    ('palette.pdf', 'hocr'),
-    ('palette.pdf', 'tesseract'),
-    ('cmyk.pdf', 'hocr'),
-    ('cmyk.pdf', 'tesseract'),
-    ('ccitt.pdf', 'hocr'),
-    ('ccitt.pdf', 'tesseract'),
-    ('jbig2.pdf', 'hocr'),
-    ('jbig2.pdf', 'tesseract'),
-    ('lichtenstein.pdf', 'hocr'),
-    ('lichtenstein.pdf', 'tesseract')
-])
-def test_exotic_image(spoof_tesseract_cache, pdf, renderer):
+# This will run 5 * 2 * 2 = 20 test cases
+@pytest.mark.parametrize(
+    "pdf",
+    ['palette.pdf', 'cmyk.pdf', 'ccitt.pdf', 'jbig2.pdf', 'lichtenstein.pdf'])
+@pytest.mark.parametrize("renderer", ['hocr', 'tesseract'])
+@pytest.mark.parametrize("output_type", ['pdf', 'pdfa'])
+def test_exotic_image(spoof_tesseract_cache, pdf, renderer, output_type):
     check_ocrmypdf(
         pdf,
         'test_{0}_{1}.pdf'.format(pdf, renderer),
         '-dc',
         '-v', '1',
+        '--output-type', output_type,
         '--pdf-renderer', renderer, env=spoof_tesseract_cache)
 
 
-def test_preserve_metadata(spoof_tesseract_noop):
+@pytest.mark.parametrize("output_type", [
+    'pdfa', 'pdf'
+    ])
+def test_preserve_metadata(spoof_tesseract_noop, output_type):
     pdf_before = pypdf.PdfFileReader(_infile('graph.pdf'))
 
     output = check_ocrmypdf('graph.pdf', 'test_metadata_preserve.pdf',
+                            '--output-type', output_type,
                             env=spoof_tesseract_noop)
 
     pdf_after = pypdf.PdfFileReader(output)
@@ -194,8 +195,14 @@ def test_preserve_metadata(spoof_tesseract_noop):
     for key in ('/Title', '/Author'):
         assert pdf_before.documentInfo[key] == pdf_after.documentInfo[key]
 
+    pdfa_info = file_claims_pdfa(output)
+    assert pdfa_info['output'] == output_type
 
-def test_override_metadata(spoof_tesseract_noop):
+
+@pytest.mark.parametrize("output_type", [
+    'pdfa', 'pdf'
+    ])
+def test_override_metadata(spoof_tesseract_noop, output_type):
     input_file = _infile('c02-22.pdf')
     output_file = _outfile('test_override_metadata.pdf')
 
@@ -208,6 +215,7 @@ def test_override_metadata(spoof_tesseract_noop):
         '--title', german,
         '--author', chinese,
         '--subject', high_unicode,
+        '--output-type', output_type,
         env=spoof_tesseract_noop)
 
     assert p.returncode == ExitCode.ok
@@ -226,6 +234,9 @@ def test_override_metadata(spoof_tesseract_noop):
     assert pdfinfo['Subject'] == high_unicode
     assert pdfinfo.get('Keywords', '') == ''
 
+    pdfa_info = file_claims_pdfa(output_file)
+    assert pdfa_info['output'] == output_type
+
 
 @pytest.mark.parametrize('renderer', [
     'hocr',
@@ -386,16 +397,15 @@ def test_skip_big(spoof_tesseract_cache):
     assert not pdfinfo[0]['has_text']
 
 
-@pytest.mark.parametrize('renderer', [
-    'hocr',
-    'tesseract',
-    ])
-def test_maximum_options(spoof_tesseract_cache, renderer):
+@pytest.mark.parametrize('renderer', ['hocr', 'tesseract'])
+@pytest.mark.parametrize('output_type', ['pdf', 'pdfa'])
+def test_maximum_options(spoof_tesseract_cache, renderer, output_type):
     check_ocrmypdf(
         'multipage.pdf', 'test_multipage%s.pdf' % renderer,
         '-d', '-c', '-i', '-g', '-f', '-k', '--oversample', '300',
         '--skip-big', '10', '--title', 'Too Many Weird Files',
         '--author', 'py.test', '--pdf-renderer', renderer,
+        '--output-type', output_type,
         env=spoof_tesseract_cache)
 
 
@@ -420,6 +430,17 @@ def test_blank_input_pdf():
     assert p.returncode == ExitCode.ok
 
 
+def test_force_ocr_on_pdf_with_no_images(spoof_tesseract_crash):
+    # As a correctness test, make sure that --force-ocr on a PDF with no
+    # content still triggers tesseract. If tesseract crashes, then it was
+    # called.
+    p, _, err = run_ocrmypdf_env(
+        'blank.pdf', 'wont_be_created.pdf', '--force-ocr',
+        env=spoof_tesseract_crash)
+    assert p.returncode == ExitCode.child_process_error, err
+    assert not os.path.exists(_outfile('wontwork.pdf'))
+
+
 def test_french(spoof_tesseract_cache):
     p, out, err = run_ocrmypdf_env(
         'francais.pdf', 'francais.pdf', '-l', 'fra', env=spoof_tesseract_cache)
@@ -551,3 +572,28 @@ def test_old_unpaper():
 def test_algo4():
     sh, _, _ = run_ocrmypdf_env('encrypted_algo4.pdf', 'wontwork.pdf')
     assert sh.returncode == ExitCode.encrypted_pdf
+
+
+@pytest.mark.parametrize('renderer', [
+    'hocr'])  # tesseract cannot pass this test yet
+def test_non_square_resolution(renderer, spoof_tesseract_cache):
+    # Confirm input image is non-square resolution
+    in_pageinfo = pdf_get_all_pageinfo(_infile('aspect.pdf'))
+    assert in_pageinfo[0]['xres'] != in_pageinfo[0]['yres']
+
+    out = 'aspect_%s.pdf' % renderer
+    check_ocrmypdf(
+        'aspect.pdf', out,
+        '--pdf-renderer', renderer, env=spoof_tesseract_cache)
+
+    out_pageinfo = pdf_get_all_pageinfo(_outfile(out))
+
+    # Confirm resolution was kept the same
+    assert in_pageinfo[0]['xres'] == out_pageinfo[0]['xres']
+    assert in_pageinfo[0]['yres'] == out_pageinfo[0]['yres']
+
+
+def test_image_to_pdf(spoof_tesseract_noop):
+    check_ocrmypdf(
+        'LinnSequencer.jpg', 'image_to_pdf.pdf', '--image-dpi', '200',
+        env=spoof_tesseract_noop)