v6.1.3 notes

Closes #247

.docker/Dockerfile

@@ -28,7 +28,7 @@ RUN python3 -m venv --system-site-packages /appenv
 # (i.e. "pip install ."). It's unclear to me why this is the case.
 RUN . /appenv/bin/activate; \
   pip install --upgrade pip \
-  && pip install ocrmypdf
+  && pip install ocrmypdf[fitz]
 
 # Now copy the application in, mainly to get the test suite.
 # Do this now to make the best use of Docker cache.

.docker/Dockerfile.tess4

@@ -38,7 +38,7 @@ RUN python3 -m venv --system-site-packages /appenv
 # (i.e. "pip install ."). It's unclear to me why this is the case.
 RUN . /appenv/bin/activate; \
   pip install --upgrade pip \
-  && pip install ocrmypdf
+  && pip install ocrmypdf[fitz]
 
 # Now copy the application in, mainly to get the test suite.
 # Do this now to make the best use of Docker cache.

.travis.yml

@@ -27,6 +27,10 @@ matrix:
       language: python
       python: 3.6
       env: EXTRAS=[fitz]
+    - os: osx
+      osx_image: xcode8
+      language: generic
+      env: EXTRAS=
     - os: osx
       osx_image: xcode8
       language: generic
@@ -51,7 +55,7 @@ install:
 script:
 - tesseract --version
 - qpdf --version
-- pytest
+- pytest -n auto
 
 deploy:
   # release for main pypi

docs/batch.rst

@@ -18,7 +18,7 @@ The ``--tag`` argument tells parallel to print the filename as a prefix whenever
 
 	parallel --tag -j 2 ocrmypdf '{}' 'output/{}' ::: *.pdf
 
-OCRmyPDF automaticaly repairs PDFs before parsing and gathering information from them.  If you are already repairing PDFs with ``qpdf`` prior to attempting OCR, or you can use ``--skip-repair`` to skip this step.  It may improve performance for large files, since repairing PDFs is single-threaded.
+OCRmyPDF automatically repairs PDFs before parsing and gathering information from them.  If you are already repairing PDFs with ``qpdf`` prior to attempting OCR, or you can use ``--skip-repair`` to skip this step.  It may improve performance for large files, since repairing PDFs is single-threaded.
 
 Directory trees
 ---------------

docs/installation.rst

@@ -1,7 +1,16 @@
 Installation
 ============
 
-OCRmyPDF requires Python 3.5 (or newer) and Tesseract 3.04 (or newer).
+OCRmyPDF requires Python 3.5 (or newer) and Tesseract 3.04 (or newer). 
+
+Python 3.6 and Tesseract 4.x are recommended for best OCR results and best performance.
+
+OCRmyPDF 6.x adds a dependency on PyMuPDF ("fitz"). This library is not widely available in platform distributions, and it improves OCRmyPDF in certain conditions. Consider installing OCRmyPDF from the Python binary wheels, which include a precompiled version of this library.
+
+
+.. contents:: Platform-specific steps
+    :depth: 1
+    :local:
 
 Installing on Debian and Ubuntu 16.10 or newer
 ----------------------------------------------
@@ -12,8 +21,14 @@ Users of Debian 9 ("stretch") or later or Ubuntu 16.10 or later may simply
 
     apt-get install ocrmypdf
 
-Installing on macOS
--------------------
+To see what versions are available, check the `Debian Package Tracker <https://tracker.debian.org/pkg/ocrmypdf>`_ or `Ubuntu launchpad.net <https://launchpad.net/ocrmypdf>`_.
+
+Installing on macOS with Homebrew
+---------------------------------
+
+.. image:: https://img.shields.io/homebrew/v/ocrmypdf.svg   
+    :alt: homebrew
+    :target: http://brewformulas.org/Ocrmypdf
 
 OCRmyPDF is now a standard `Homebrew <https://brew.sh>`_ formula. To install on macOS:
 
@@ -36,11 +51,9 @@ Installing the Docker image
 
 For many users, installing the Docker image will be easier than installing all of OCRmyPDF's dependencies. For Windows, it is the only option.
 
-If you have `Docker <https://docs.docker.com/>`_ installed on your system, you can install
-a Docker image of the latest release.
+If you have `Docker <https://docs.docker.com/>`_ installed on your system, you can install a Docker image of the latest release.
 
-Follow the Docker installation instructions for your platform.  If you can run this command
-successfully, your system is ready to download and execute the image:
+Follow the Docker installation instructions for your platform.  If you can run this command successfully, your system is ready to download and execute the image:
 
 .. code-block:: bash
 
@@ -58,16 +71,22 @@ OCRmyPDF will use all available CPU cores.  By default, the VirtualBox machine i
 
 Assuming you have a Docker engine running, you can download one of the three available images:
 
-+-----------------------------+---------------------------------------------+---------------------------------------------------------------------------------+
-| Image name                  | Download command                            | Notes                                                                           |
-+-----------------------------+---------------------------------------------+---------------------------------------------------------------------------------+
-| ocrmypdf                    | ``docker pull jbarlow83/ocrmypdf``          | Latest ocrmypdf with Tesseract 3.04. Includes English, French, German, Spanish. |
-+-----------------------------+---------------------------------------------+---------------------------------------------------------------------------------+
-| ocrmypdf-polyglot           | ``docker pull jbarlow83/ocrmypdf-polyglot`` | As above, with all available language packs.                                    |
-+-----------------------------+---------------------------------------------+---------------------------------------------------------------------------------+
-| ocrmypdf-tess4              | ``docker pull jbarlow83/ocrmypdf-tess4``    | Latest ocrmypdf with Tesseract 4.00.00alpha and English, French, German,        |
-|                             |                                             | Spanish, Portuguese, Chinese Simplified, Arabic and Russian (the top 8).        |
-+-----------------------------+---------------------------------------------+---------------------------------------------------------------------------------+
+.. list-table::
+    :widths: auto
+    :header-rows: 1
+
+    *   - Image name
+        - Download command
+        - Notes
+    *   - ocrmypdf
+        - ``docker pull jbarlow83/ocrmypdf``
+        - Latest ocrmypdf with Tesseract 3.x. Includes English, French, German, Spanish.
+    *   - ocrmypdf-polyglot
+        - ``docker pull jbarlow83/ocrmypdf-polyglot``
+        - As above, with all available language packs.
+    *   - ocrmypdf-tess4
+        - ``docker pull jbarlow83/ocrmypdf-tess4``
+        - Latest ocrmypdf with Tesseract 4.x and English, French, German, Spanish, Portuguese, Chinese Simplified, Arabic and Russian (the top 8).
 
 For example:
 
@@ -169,24 +188,25 @@ Install the required Tesseract OCR engine with the language packs you plan to us
 
     brew install tesseract --with-all-languages  # Option 2: for all language packs
 
-Update the homebrew pip and install Pillow:
+Update the homebrew pip:
 
 .. code-block:: bash
 
     pip3 install --upgrade pip
-    pip3 install --upgrade pillow
 
 You can then install OCRmyPDF from PyPI, for the current user:
 
 .. code-block:: bash
 
-    pip3 install --user ocrmypdf
+    pip3 install --user ocrmypdf[fitz]
 
 or system-wide:
 
 .. code-block:: bash
 
-    pip3 install ocrmypdf
+    pip3 install ocrmypdf[fitz]
+
+``[fitz]`` includes the optional dependency on PyMuPDF, which improves OCRmyPDF's output in many cases.
 
 The command line program should now be available:
 
@@ -215,29 +235,29 @@ If you wish install OCRmyPDF for the current user:
 
 .. code-block:: bash
 
-    pip3 install --user ocrmypdf
+    pip3 install --user ocrmypdf[fitz]
 
 Alternately, system-wide. Note that this may modify the system Python environment:
 
 .. code-block:: bash
 
-    sudo pip3 install ocrmypdf
+    sudo pip3 install ocrmypdf[fitz]
 
-If you wish to install OCRmyPDF to a virtual environment to isolate the system Python, you can
-follow these steps.
+If you wish to install OCRmyPDF to a virtual environment to isolate the system Python, you can follow these steps.
 
 .. code-block:: bash
 
     python3 -m venv venv-ocrmypdf
     source venv-ocrmypdf/bin/activate
-    pip3 install ocrmypdf
+    pip3 install ocrmypdf[fitz]
+
+``[fitz]`` includes the optional dependency on PyMuPDF, which improves OCRmyPDF's output in many cases.
 
 
 Installing on Ubuntu 14.04 LTS
 ------------------------------
 
-Installing on Ubuntu 14.04 LTS (trusty) is more difficult than some other options,
-because it is older and does not provide ``pip``.
+Installing on Ubuntu 14.04 LTS (trusty) is more difficult than some other options, because it is older and does not provide ``pip``.
 
 Update apt-get:
 
@@ -256,10 +276,7 @@ Install system dependencies:
         libffi-dev \
         qpdf
 
-We will need backports of Ghostscript 9.16, libav-11 (for unpaper 6.1),
-Tesseract 4.00 (alpha), and Python 3.6. This will replace Ghostscript and
-Tesseract 3.x on your system. Python 3.6 will be installed alongside the system
-Python 3.
+We will need backports of Ghostscript 9.16, libav-11 (for unpaper 6.1), Tesseract 4.00 (alpha), and Python 3.6. This will replace Ghostscript and Tesseract 3.x on your system. Python 3.6 will be installed alongside the system Python 3.
 
 If you prefer to not modify your system in this matter, consider using a Docker container.
 
@@ -285,7 +302,7 @@ Now we need to install ``pip`` and let it install ocrmypdf:
 .. code-block:: bash
 
     wget -O - -o /dev/null https://bootstrap.pypa.io/get-pip.py | python3.6
-    pip3.6 install ocrmypdf
+    pip3.6 install ocrmypdf[fitz]
 
 The ``wget`` command will download a program and run it.
 
@@ -312,6 +329,9 @@ Installing on Windows
 
 Direct installation on Windows is not possible.  Install the _`Docker` container as described above.  Ensure that your command prompt can run the docker "hello world" container.
 
+It would probably not be too difficult to run on Windows.  The main reason this has been avoided is the difficulty of packaging and installing the various non-Python dependencies: Tesseract, QPDF, Ghostscript, Leptonica.  Pull requests to add or improve Windows support would be quite welcome.
+
+
 Running on Windows
 ~~~~~~~~~~~~~~~~~~
 
@@ -343,15 +363,7 @@ Or, to install in `development mode <https://pythonhosted.org/setuptools/setupto
 
     pip3 install -e git+https://github.com/jbarlow83/OCRmyPDF.git
 
-On certain Linux distributions such as Ubuntu, you may need to use 
-run the install command as superuser:
-
-.. code-block:: bash
-
-    sudo pip3 install [-e] git+https://github.com/jbarlow83/OCRmyPDF.git
-
-Note that this will alter your system's Python distribution. If you prefer 
-to not install as superuser, you can install the package in a Python virtual environment:
+You may find it easiest to install in a virtual environment, rather than system-wide:
 
 .. code-block:: bash
 

docs/release_notes.rst

@@ -5,6 +5,26 @@ OCRmyPDF uses `semantic versioning <http://semver.org/>`_ for its command line i
 
 The OCRmyPDF package itself does not contain a public API, although it is fairly stable and breaking changes are usually timed with a major release. A future release will clearly define the stable public API.
 
+v6.1.3
+------
+
+-   Fix issue #247, ``/CreationDate`` metadata not copied from input to output.
+
+
+v6.1.2
+------
+
+-   Upgrade to PyMuPDF v1.12.5 which includes a more complete fix to #239.
+
+-   Add ``defusedxml`` dependency.
+
+
+v6.1.1
+------
+
+-   Fix text being reported as found on all pages if PyMuPDF is not installed.
+
+
 v6.1.0
 ------
 

setup.cfg

@@ -11,4 +11,6 @@ ignore =
 [tool:pytest]
 norecursedirs = lib .pc .git output cache resources
 testpaths = tests
-addopts = -n auto
+
+[metadata]
+license_file = LICENSE

setup.py

@@ -190,9 +190,6 @@ if not forced and command.startswith('install') or \
         package='qpdf',
         version_check_args=['--version']
     )
-else:
-    print('Skipping external program tests because of --force')
-
 
 if 'upload' in sys.argv[1:]:
     print('Use twine to upload the package - setup.py upload is insecure')
@@ -232,26 +229,27 @@ setup(
         "Topic :: Text Processing :: Indexing",
         "Topic :: Text Processing :: Linguistic",
         ],
-    python_requires='>=3.5',
+    python_requires=' >= 3.5',
     setup_requires=[
+        'cffi >= 1.9.1',        # to build the leptonica module
+        'pytest-runner',        # to enable python setup.py test
         'setuptools_scm',       # so that version will work
-        'cffi>=1.9.1',          # to build the leptonica module
-        'pytest-runner'         # to enable python setup.py test
     ],
     use_scm_version={'version_scheme': 'post-release'},
     cffi_modules=[
         'src/ocrmypdf/lib/compile_leptonica.py:ffibuilder'
     ],
     install_requires=[
-        'ruffus==2.6.3',        # pinned - ocrmypdf implements a 2.6.3 workaround
-        'Pillow>=4.0.0',        # Pillow < 4 has BytesIO/TIFF bug w/img2pdf 0.2.3
-        'reportlab>=3.3.0',     # oldest released version with sane image handling
-        'PyPDF2>=1.26',         # pure Python, so track HEAD closely
-        'img2pdf>=0.2.3',       # pure Python, so track HEAD closely
-        'cffi>=1.9.1',          # must be a setup and install requirement
+        'cffi >= 1.9.1',          # must be a setup and install requirement
+        'defusedxml >= 0.5.0',    # pure Python, so track HEAD closely
+        'img2pdf >= 0.2.4',       # pure Python, so track HEAD closely
+        'Pillow >= 4.0.0',        # Pillow < 4 has BytesIO/TIFF bug w/img2pdf 0.2.3
+        'PyPDF2 >= 1.26',         # pure Python, so track HEAD closely
+        'reportlab >= 3.3.0',     # oldest released version with sane image handling
+        'ruffus == 2.6.3',        # pinned - ocrmypdf implements a 2.6.3 workaround
     ],
     extras_require={
-        'fitz': ['PyMuPDF == 1.12.4']     # pinned to avoid problems with 1.12.4.x
+        'fitz': ['PyMuPDF >= 1.12.5']     # for table of contents bug
     },
     tests_require=tests_require,
     entry_points={

src/ocrmypdf/lib/__init__.py

@@ -17,7 +17,12 @@
 
 """Bindings to external libraries"""
 
+import os as _os
+
 try:
     import fitz
 except ImportError:
+    fitz = None
+
+if _os.environ.get('_OCRMYPDF_NO_FITZ'):
     fitz = None

src/ocrmypdf/pdfa.py

@@ -18,7 +18,9 @@
 # Generate a PDFA_def.ps file for Ghostscript >= 9.14
 
 from string import Template
-import codecs
+from binascii import hexlify
+from datetime import datetime
+from xml.parsers.expat import ExpatError
 import pkg_resources
 import PyPDF2 as pypdf
 
@@ -93,12 +95,84 @@ def encode_text_string(s: str) -> str:
         return ''
 
     utf16_bytes = s.encode('utf-16be')
-    ascii_hex_bytes = codecs.encode(b'\xfe\xff' + utf16_bytes, 'hex')
+    ascii_hex_bytes = hexlify(b'\xfe\xff' + utf16_bytes)
     ascii_hex_str = ascii_hex_bytes.decode('ascii').lower()
     return ascii_hex_str
 
 
+def encode_pdf_date(d: datetime) -> str:
+    """Encode Python datetime object as PDF date string
+
+    From Adobe pdfmark manual:    
+    (D:YYYYMMDDHHmmSSOHH'mm')
+    D: is an optional prefix. YYYY is the year. All fields after the year are
+    optional. MM is the month (01-12), DD is the day (01-31), HH is the
+    hour (00-23), mm are the minutes (00-59), and SS are the seconds
+    (00-59). The remainder of the string defines the relation of local
+    time to GMT. O is either + for a positive difference (local time is
+    later than GMT) or - (minus) for a negative difference. HH' is the
+    absolute value of the offset from GMT in hours, and mm' is the
+    absolute value of the offset in minutes. If no GMT information is
+    specified, the relation between the specified time and GMT is
+    considered unknown. Regardless of whether or not GMT
+    information is specified, the remainder of the string should specify
+    the local time.
+    """
+
+    pdfmark_date_fmt = r'%Y%m%d%H%M%S'
+    s = d.strftime(pdfmark_date_fmt)
+
+    tz = d.strftime('%z')
+    if tz == 'Z':
+        s += "+00'00'"
+    elif tz != '':
+        sign, tz_hours, tz_mins = tz[0], tz[1:3], tz[3:5]
+        s += "{}{}'{tz}'".format(sign, tz_hours, tz_mins)
+    return s
+
+
+def decode_pdf_date(s: str) -> datetime:
+    pdfmark_date_fmts = (
+        r'%Y%m%d%H%M%S%z',  # +0430 etc
+        r'%Y%m%d%H%M%S',    # no time zone
+        r'%Y%m%d%H%M%SZ')   # trailing Z
+
+    if s.startswith('D:'):
+        s = s[2:]
+    for fmt in pdfmark_date_fmts:
+        try:
+            return datetime.strptime(s, fmt)
+        except ValueError:
+            continue
+    return None
+
+
+def _get_pdfmark_dates(pdfmark):
+    """Encode dates for pdfmark Postscript.  The best way to deal with a
+    missing date entry is set it to null, because if the key is omitted 
+    Ghostscript will set it to now - we do not want to erase the fact that
+    the value was unknown.  Setting to an empty string breaks Ghostscript
+    9.22 as reported here:
+    https://bugs.ghostscript.com/show_bug.cgi?id=699182
+    """
+
+    for key in ('/CreationDate', '/ModDate'):
+        if key not in pdfmark:
+            continue
+        if pdfmark[key].strip() == '':
+            yield '  {} null'.format(key)
+            continue
+        date_str = pdfmark[key]
+        if date_str.startswith('D:'):
+            date_str = date_str[2:]
+        yield '  {} (D:{})'.format(key, date_str)
+
+
 def _get_pdfa_def(icc_profile, icc_identifier, pdfmark):
+    """Create a Postscript file for Ghostscript.  pdfmark contains the various
+    objects as strings; these must be encoded in ASCII, and dates have a 
+    special format."""
+
     # Ghostscript <= 9.21 has a bug where null entries in DOCINFO might produce
     # ERROR: VMerror (-25) on closing pdfwrite device.
     # https://bugs.ghostscript.com/show_bug.cgi?id=697684
@@ -107,12 +181,12 @@ def _get_pdfa_def(icc_profile, icc_identifier, pdfmark):
     docinfo_line_template = '  {key} <{value}>'
 
     def docinfo_gen():
+        yield from _get_pdfmark_dates(pdfmark)
         for key in docinfo_keys:
             if key in pdfmark and pdfmark[key].strip() != '':
                 line = docinfo_line_template.format(
                     key=key, value=encode_text_string(pdfmark[key]))
                 yield line
-
     docinfo = '\n'.join(docinfo_gen())
 
     t = Template(pdfa_def_template)
@@ -145,9 +219,12 @@ def file_claims_pdfa(filename):
 
     This checks if the XMP metadata contains a PDF/A marker.
     """
-
     pdf = pypdf.PdfFileReader(filename)
-    xmp = pdf.getXmpMetadata()
+    try:
+        xmp = pdf.getXmpMetadata()
+    except ExpatError:
+        return {'pass': False, 'output': 'pdf',
+                'conformance': 'Invalid XML metadata'}
 
     try:
         pdfa_nodes = xmp.getNodesInNamespace(

src/ocrmypdf/pdfinfo.py

@@ -205,7 +205,7 @@ def _interpret_contents(contentstream, initial_shorthand=UNIT_SQUARE):
     return ContentsInfo(
         xobject_settings=xobject_settings,
         inline_images=inline_images,
-        found_text=True)
+        found_text=found_text)
 
 
 def _get_dpi(ctm_shorthand, image_size):

src/ocrmypdf/pipeline.py

@@ -18,6 +18,7 @@
 from contextlib import suppress
 from shutil import copyfileobj
 from pathlib import Path
+from datetime import datetime
 import sys
 import os
 import shutil
@@ -31,7 +32,7 @@ from ruffus import formatter, regex, Pipeline, suffix
 
 from .hocrtransform import HocrTransform
 from .pdfinfo import PdfInfo, Encoding, Colorspace
-from .pdfa import generate_pdfa_ps
+from .pdfa import generate_pdfa_ps, encode_pdf_date
 from .helpers import re_symlink, is_iterable_notstr, page_number
 from .exec import ghostscript, tesseract, qpdf
 from .lib import fitz
@@ -230,6 +231,12 @@ def repair_and_parse_pdf(
             "output these files.)  Use --output-type=pdf instead."
         )
         raise InputFileError()
+    
+    if len(pdfinfo.pages) > 2000 and sys.version_info[0:2] <= (3, 5):
+        log.warning(
+            "Performance regressions are known occur with Python 3.5 for "
+            "high page count files.  Python 3.6 or newer is recommended."
+        )
 
     context.set_pdfinfo(pdfinfo)
     log.debug(pdfinfo)
@@ -624,19 +631,18 @@ def select_visible_page_image(
         log.debug('{:4d}: JPEG input -> JPEG output'.format(
             page_number(image)))
         # If all images were JPEGs originally, produce a JPEG as output
-        im = Image.open(image)
+        with Image.open(image) as im:
+            # At this point the image should be a .png, but deskew, unpaper
+            # might have removed the DPI information. In this case, fall back to
+            # square DPI used to rasterize. When the preview image was
+            # rasterized, it was also converted to square resolution, which is
+            # what we want to give tesseract, so keep it square.
+            fallback_dpi = get_page_square_dpi(pageinfo, options)
+            dpi = im.info.get('dpi', (fallback_dpi, fallback_dpi))
 
-        # At this point the image should be a .png, but deskew, unpaper might
-        # have removed the DPI information. In this case, fall back to square
-        # DPI used to rasterize. When the preview image was rasterized, it
-        # was also converted to square resolution, which is what we want to
-        # give tesseract, so keep it square.
-        fallback_dpi = get_page_square_dpi(pageinfo, options)
-        dpi = im.info.get('dpi', (fallback_dpi, fallback_dpi))
-
-        # Pillow requires integer DPI
-        dpi = round(dpi[0]), round(dpi[1])
-        im.save(output_file, format='JPEG', dpi=dpi)
+            # Pillow requires integer DPI
+            dpi = round(dpi[0]), round(dpi[1])
+            im.save(output_file, format='JPEG', dpi=dpi)
     else:
         re_symlink(image, output_file, log)
 
@@ -658,25 +664,25 @@ def select_image_layer(
         log.debug("{:4d}: page eligible for lossless reconstruction".format(
             page_number(page_pdf)))
         re_symlink(page_pdf, output_file, log)
-    else:
-        pageinfo = get_pageinfo(image, context)
+        return
 
-        # We rasterize a square DPI version of each page because most image
-        # processing tools don't support rectangular DPI. Use the square DPI
-        # as it accurately describes the image. It would be possible to
-        # resample the image at this stage back to non-square DPI to more
-        # closely resemble the input, except that the hocr renderer does not
-        # understand non-square DPI. The sandwich renderer would be fine.
-        dpi = get_page_square_dpi(pageinfo, options)
-        layout_fun = img2pdf.get_fixed_dpi_layout_fun((dpi, dpi))
+    pageinfo = get_pageinfo(image, context)
 
-        with open(image, 'rb') as imfile, \
-                open(output_file, 'wb') as pdf:
-            log.debug('{:4d}: convert'.format(page_number(page_pdf)))
-            img2pdf.convert(
-                imfile, with_pdfrw=False,
-                layout_fun=layout_fun, outputstream=pdf)
-            log.debug('{:4d}: convert done'.format(page_number(page_pdf)))
+    # We rasterize a square DPI version of each page because most image
+    # processing tools don't support rectangular DPI. Use the square DPI as it
+    # accurately describes the image. It would be possible to resample the image
+    # at this stage back to non-square DPI to more closely resemble the input,
+    # except that the hocr renderer does not understand non-square DPI. The
+    # sandwich renderer would be fine.
+    dpi = get_page_square_dpi(pageinfo, options)
+    layout_fun = img2pdf.get_fixed_dpi_layout_fun((dpi, dpi))
+
+    with open(image, 'rb') as imfile, open(output_file, 'wb') as pdf:
+        log.debug('{:4d}: convert'.format(page_number(page_pdf)))
+        img2pdf.convert(
+            imfile, with_pdfrw=False,
+            layout_fun=layout_fun, outputstream=pdf)
+        log.debug('{:4d}: convert done'.format(page_number(page_pdf)))
 
 
 def render_hocr_page(
@@ -872,12 +878,8 @@ def get_pdfmark(base_pdf, options):
         except (KeyError, TypeError):
             return ''
 
-    pdfmark = {
-        '/Title': from_document_info('/Title'),
-        '/Author': from_document_info('/Author'),
-        '/Keywords': from_document_info('/Keywords'),
-        '/Subject': from_document_info('/Subject'),
-    }
+    pdfmark = {k: from_document_info(k) for k in 
+        ('/Title', '/Author', '/Keywords', '/Subject', '/CreationDate')}
     if options.title:
         pdfmark['/Title'] = options.title
     if options.author:
@@ -898,6 +900,7 @@ def get_pdfmark(base_pdf, options):
         PROGRAM_NAME, VERSION,
         renderer_tag,
         tesseract.version())
+    pdfmark['/ModDate'] = encode_pdf_date(datetime.utcnow())
     return pdfmark
 
 
@@ -1031,7 +1034,7 @@ def merge_pages_mupdf(
     reader_metadata = pypdf.PdfFileReader(metadata_file)
     pdfmark = get_pdfmark(reader_metadata, options)
     pdfmark['/Producer'] = 'PyMuPDF ' + fitz.version[0]
-    pymupdf_metadata = {k[1:].lower() : v for k, v in pdfmark.items()}
+    pymupdf_metadata = {(k[1].lower() + k[2:]) : v for k, v in pdfmark.items()}
 
     for pdf_page in pdf_pages:
         page = fitz.open(pdf_page)
@@ -1039,12 +1042,7 @@ def merge_pages_mupdf(
 
     metadata = fitz.open(metadata_file)
     toc = metadata.getToC(simple=False)
-    def filter_toc_pages():
-        "fitz does not escape parens properly"
-        for entry in toc:
-            entry[1] = entry[1].replace('(', '').replace(')', '')
-            yield entry
-    doc.setToC([item for item in filter_toc_pages()])
+    doc.setToC(toc)
     doc.setMetadata(pymupdf_metadata)
     doc.save(output_file, garbage=4, deflate=True)
 

tests/resources/README.rst

@@ -46,80 +46,97 @@ The following test resources were crafted specifically for this project, and are
 licensed under the specified license.
 
 .. list-table:: 
-    :widths: 20 20 60
+    :widths: 20 40 15 15 10
     :header-rows: 1
 
     *   - File
-        - Contributor
         - Purpose
+        - Contributor
+        - Copyright Holder
         - License
     *   - aspect.pdf
-        - @jbarlow83
         - test image with 200 x 100 DPI resolution
-        - CC BY-SA 4.0
+        - @jbarlow83
+        - @jbarlow83
+        - CC-BY-SA 4.0
     *   - blank.pdf
-        - @jbarlow83
         - blank PDF generated by Adobe Illustrator CC 17, containing a lot of application-specific metadata/bloat
-        - CC BY-SA 4.0
+        - @jbarlow83
+        - @jbarlow83
+        - CC-BY-SA 4.0
     *   - cmyk.pdf
-        - @jbarlow83
         - a CMYK image created in Photoshop
-        - CC BY-SA 4.0        
+        - @jbarlow83
+        - @jbarlow83
+        - CC-BY-SA 4.0        
     *   - crom.png
-        - @jbarlow83
         - test for non-dictionary words
-        - CC BY-SA 4.0        
+        - @jbarlow83
+        - @jbarlow83
+        - CC-BY-SA 4.0        
     *   - enormous.pdf
-        - @jbarlow83
         - very large PDF page
-        - CC BY-SA 4.0
+        - @jbarlow83
+        - @jbarlow83
+        - CC-BY-SA 4.0
     *   - epson.pdf
+        - a linearized PDF containing some unusual indirect objects, created by an Epson printer; printout of a Wikipedia article (CC-BY-SA)
         - @lowesjam
-        - a linearized PDF containing some unusual indirect objects, created by an Epson printer; printout of a Wikipedia article (CC BY-SA)
-        - MIT
+        - Wikipedia authors
+        - CC-BY-SA 3.0
     *   - formxobject.pdf
-        - @jbarlow83
         - hand-crafted PDF containing an image inside a Form XObject
-        - CC BY-SA 4.0
+        - @jbarlow83
+        - @jbarlow83
+        - CC-BY-SA 4.0
     *   - francais.pdf
-        - @jbarlow83
         - a page containing French accents (diacritics)  
-        - CC BY-SA 4.0
+        - @jbarlow83
+        - @jbarlow83
+        - CC-BY-SA 4.0
     *   - hugemono.pdf
-        - @jbarlow83
         - large monochrome 35000x35000 image in JBIG2 encoding 
-        - CC BY-SA 4.0
-    *   - invalid.pdf
         - @jbarlow83
+        - @jbarlow83
+        - CC-BY-SA 4.0
+    *   - invalid.pdf
         - a PDF file header followed by EOF marker
-        - CC BY-SA 4.0
+        - @jbarlow83
+        - @jbarlow83
+        - CC-BY-SA 4.0
     *   - masks.pdf
+        - file containing explicit masks and a stencil mask drawn without a proper transformation matrix; printout of a German Wikipedia article (CC-BY-SA)
         - @supergrobi
-        - file containing explicit masks and a stencil mask drawn without a proper transformation matrix; printout of a German Wikipedia article (CC BY-SA)
-        - MIT
+        - Wikipedia authors
+        - CC-BY-SA 3.0
     *   - missing_docinfo.pdf
         - @jbarlow83
         - PDF file with no /DocumentInfo section 
-        - CC BY-SA 4.0
+        - CC-BY-SA 4.0
     *   - overlay.pdf
-        - @maxandersen
         - PDF file generated by PDFPen pro that triggered content stream parse errors
+        - @maxandersen
+        - @maxandersen
         - MIT
     *   - negzero.pdf
-        - @jbarlow83
         - copy of formxobject.pdf with token that qpdf doesn't like
-        - CC BY-SA 4.0
+        - @jbarlow83
+        - @jbarlow83
+        - CC-BY-SA 4.0
     *   - no_contents.pdf
-        - @jbarlow83
         - synthetic PDF with a blank page that has no /Contents entry
-        - CC BY-SA 4.0
-    *   - trivial.pdf
         - @jbarlow83
+        - @jbarlow83
+        - CC-BY-SA 4.0
+    *   - trivial.pdf
         - smallest possible valid PDF-1.3 with all required fields
-        - CC BY-SA 4.0
+        - @jbarlow83
+        - @jbarlow83
+        - CC-BY-SA 4.0
     *   - vector.pdf
-        - @Catscratch
         - a PDF with vector art and text rendered as curves with no fonts
+        - @Catscratch
+        - @Catscratch
         - MIT
 
 

tests/test_metadata.py

@@ -18,8 +18,9 @@
 
 import pytest
 import PyPDF2 as pypdf
+import datetime
 
-from ocrmypdf.pdfa import file_claims_pdfa
+from ocrmypdf.pdfa import file_claims_pdfa, encode_pdf_date, decode_pdf_date
 from ocrmypdf.exceptions import ExitCode
 from ocrmypdf.lib import fitz
 
@@ -115,3 +116,40 @@ def test_bookmarks_preserved(spoof_tesseract_noop, output_type, ocr_option,
     print(before_toc)
     print(after_toc)
     assert before_toc == after_toc
+
+
+def seconds_between_dates(date1, date2):
+    return (date2 - date1).total_seconds()
+
+
+@pytest.mark.parametrize('infile', ['trivial.pdf', 'jbig2.pdf'])
+@pytest.mark.parametrize('output_type', ['pdf', 'pdfa'])
+def test_creation_date_preserved(spoof_tesseract_noop, output_type, resources,
+                                 infile, outpdf):
+    input_file = resources / infile
+
+    before = pypdf.PdfFileReader(str(input_file)).getDocumentInfo()
+    check_ocrmypdf(
+        input_file, outpdf, '--output-type', output_type, 
+        env=spoof_tesseract_noop)
+    after = pypdf.PdfFileReader(str(outpdf)).getDocumentInfo()
+
+    if not before:
+        # If there was input creation date, none should be output
+        # because of Ghostscript quirks we set it to null
+        # This test would be better if we had a test file with /DocumentInfo but
+        # no /CreationDate, which we don't
+        assert not after['/CreationDate'] or \
+                isinstance(after['/CreationDate'], pypdf.generic.NullObject)
+    else:
+        # We expect that the creation date stayed the same
+        date_before = decode_pdf_date(before['/CreationDate'])
+        date_after = decode_pdf_date(after['/CreationDate'])
+        assert seconds_between_dates(date_before, date_after) < 1000
+
+    # We expect that the modified date is quite recent
+    date_after = decode_pdf_date(after['/ModDate'])
+    assert seconds_between_dates(
+        date_after, datetime.datetime.utcnow()) < 1000
+
+